ENCYCLOPEDIA OF STATISTICAL SCIENCES Volume 10 Preference Mapping to Recovery of Interblock Information
ENCYCLOPEDIA O...
14 downloads
1503 Views
32MB Size
Report
This content was uploaded by our users and we assume good faith they have the permission to share this book. If you own the copyright to this book and it is wrongfully on our website, we offer a simple DMCA procedure to remove your content from our site. Start by pressing the button below!
Report copyright / DMCA form
ENCYCLOPEDIA OF STATISTICAL SCIENCES Volume 10 Preference Mapping to Recovery of Interblock Information
ENCYCLOPEDIA OF STATISTICAL SCIENCES Second Edition
Volume 10 Preference Mapping to Recovery of Interblock Information
The Encyclopedia of Statistical Sciences is available Online at http://www.mrw.interscience.wiley.com/ess
A John Wiley & Sons, Inc., Publication
CONTRIBUTORS J. R. Abernathy, University of North Carolina, Chapel Hill, NC. Randomized Response Robert J. Adler, The Technion, Haifa, Israel. Random Fields Munir Ahmad, KFUPM, Dhahran, Saudi Arabia. PValues, Combination of James H. Albert, Bowling Green State University, Bowling Green, OH. Probit Models, Bayesian Analysis of Erling B. Andersen, University of Copenhagen, Copenhagen, Denmark. Rasch, Georg J. R. Ashford, University of Exeter, Exeter, United Kingdom. Probit Analysis R. A. Bailey, Rothamsted Experimental Station, Harpenden, United Kingdom. Randomization, Constrained N. Balakrishnan, McMaster University, Hamilton, ON, Canada. Record Values and Record Statistics L. A. Baxter, State University of New York, Stony Brook, NY. Quasi Life Tables Betsy Jane Becker, Michigan State University, East Lansing, MI. P-Values, Combination of M. L. Berenson, Baruch College, New York, NY. Puri’s Expected Normal Scores Test Matilde Bini, Universita degli Studi di Firenze, Firenze, Italy. Raked (Adjusted) Kappa ¨ , Institut fur ¨ Statistik und Okonometrie, Fred Boker Gottingen, Germany. P-Values and Power of Tests K. R. W. Brewer, Commonwealth Schools Commission, Woden, Australia. Primary Sampling Unit; Proportional Allocation Jacob Burbea, University of Pittsburgh, Pittsburgh, PA. Rao Distance S. Cambanis, University of North Carolina, Chapel Hill, NC. Radon–Nikodym Theorem
H. Caussinus, Universit´ee, Paul Sabatier, Toulouse, France. Principal Component Analysis, Generalized David Chappell, University of Sheffield, Sheffield, United Kingdom. Randomness, BDS Test for V. Chew, University of Florida, Gainesville, FL. Radial Error Raj S. Chhikara, National Aeronautics and Space Center, Houston, TX. Proportion Estimation in Surveys using Remote Sensing Siddhartha Chib, Washington University, St. Louis, MO. Probit Models, Bayesian Analysis of Norman Cliff, Carnegie Mellon University, Pittsburgh, PA. Psychometrika Clifford C. Clogg, Pennsylvania State University, University Park, PA. Quasi-Independence Marc-David Cohen, SAS Institute, Cary, NC. PseudoRandom Number Generators Peter Coughlin, University of Maryland, College Park, MD. Probabilistic Voting Models Keren L. Coxe, Washington, DC. Principal Components Regression Analysis ´ Csorg ˝ o˝ , Carleton University, Ottawa, ON, Miklos Canada. Quantile Processes J. N. Darroch, The Flinders University, Bedford Park, South Australia. Quasi-Symmetry C. E. Davis, University of North Carolina, Chapel Hill, NC. Quantile Estimation A. P. Dawid, University College, London, United Kingdom. Prequential Analysis; Probability Forecasting Morris H. Degroot, Carnegie Mellon University, Pittsburgh, PA. Record Linkage and Matching Systems W. E. Deming, Washington, DC. Principles of Professional Statistical Practice B. Dennis, Pennsylvania State University, University Park, PA. Profiles of Diversity v
vi
CONTRIBUTORS
D. Disch, Rose-Hulman Institute of Technology, Terre Haute, IN. Ramsey’s Prior Constantin A. Drossos, University of Patras, Patras, Greece. Probability and Logic Acheson J. Duncan, Johns Hopkins University, Baltimore, MD. Quality Control, Statistical John F. Early, US Bureau of Labor Statistics, Washington, DC. Producer Price Indexes Eugene S. Edgington, University of Calgary, Calgary, AB, Canada. Randomization Tests A. W. F. Edwards, University of Cambridge, Cambridge, United Kingdom. Problem of Points Regina C. Elandt-Johnson, University of North Carolina, Chapel Hill, NC. Rates, Standardized; Rates Eva Elvers, Statistics Sweden, Stockholm, Sweden. Quality Concept for Official Statistics A. K. Md. Eshanes Saleh, Carleton University, Ottawa, ON, Canada. Rank Tests, Grouped Data in Randall L. Eubank, Southern Methodist University, Dallas, TX. Quantile Brian Everitt, Institute of Psychiatry, London, United Kingdom. Psychology, Statistics in Jean-Claude Falmagne, New York University, New York, NY. Psychophysics, Statistical Methods in Robert E. Fay, US Bureau of the Census, Washington, DC. Rao and Scott (Type) Tests Stephen E. Fienberg, Carnegie Mellon University, Pittsburgh, PA. Rasch Model Peter D. Finch, Monash University, Clayton, Australia. Randomization—I Terrence L. Fine, Cornell University, Ithaca, NY. Probability, Foundations of—II; Probability, Foundations of—I David Firth, University of Oxford, Oxford, United Kingdom. Quasiscore Bernard D. Flury, Indiana University, Bloomington, IN. Principal Components; Principal Points G. H. Freeman, University of Warwick, Coventry, United Kingdom. Quasi-Factorial Designs Janos Galambos, Temple University, Philadelphia, PA. Probabilistic Number Theory; Raikov’s Theorem J. L. Gastwirth, George Washington University, Washington, DC. Pyramid Scheme Models Jean Dickinson Gibbons, University of Alabama, Tuscaloosa, AL. Randomized Tests; Randomness, Tests of; Ranking Procedures; P-Values; Rank Tests Richard Gill, University of Utrecht, Utrecht, The Netherlands. Product Integration Michael Goldstein, University of Hull, Hull, United Kingdom. Prevision B. G. Greenberg, University of North Carolina, Chapel Hill, NC. Randomized Response
Raymond S. Greenberg, Emory University, Decatur, GA. Prospective Studies Kenneth H. Gross, California Survey Research, Sherman Oaks, CA. Random-Digit Dialing, Sampling Methods for Emil Grosswald, Temple University, Philadelphia, PA. Rademacher Functions Rudolph Haag, University of Hamburg, Hamburg, Germany. Quantum Mechanics, Statistical Interpretation of C. C. Heyde, University of Melbourne, Parkville, Australia. Probability Theory: An Outline; Quantile Transformation Methods; Random Matrices; Random Sum Distributions Katuomi Hirano, Institute of Statistical Mathematics, Tokyo, Japan. Randomized Gamma Distribution; Rayleigh Distribution Bruce Hoadley, Bell Communications Research, Holmdel, NJ. Quality Measurement Plan (QMP) David C. Hoaglin, Harvard University, Cambridge, MA. Random Number Generation W. Hoeffding, University of North Carolina, Chapel Hill, NC. Probability Inequalities for Sums of Bounded Random Variables; Range-Preserving Estimators D. G. Horvitz, Research Triangle Institute, Research Triangle, NC. Randomized Response J. R. M. Hosking, Institute of Hydrology, Wallingford, United Kingdom. Quenouille’s Test A. Glen Houston, University of Houston, Houston, TX. Proportion Estimation in Surveys using Remote Sensing Nuri T. Jazairi, York University, Toronto, ON, Canada. Productivity Measurement; Purchasing Power Parity Lyle V. Jones, Center for Advanced Study in the Behavioral Sciences, Stanford, CA. Psychological Scaling Eleanor W. Jordan, University of Texas, Austin, TX. Problem Solving in Statistics Brian Junker, Carnegie Mellon University, Pittsburgh, PA. Psychometrika John D. Kalbfleisch, University of Western Ontario, London, ON, Canada. Pseudo-Likelihood Graham Kalton, University of Michigan, Ann Arbor, MI. Question-Wording Effects in Surveys D. Kannan, University of Georgia, Athens, GA. Processes, Discrete ´ , Adam Mickiewicz University, PozMichal Karonski nan, Poland. Random Graphs Richard Kay, University of Sheffield, Sheffield, United Kingdom. Proportional Hazards Model, Cox’s Oscar Kempthorne, Iowa State University, Ames, IA. Randomization—II
CONTRIBUTORS D. G. Kendall, University of Cambridge, Cambridge, United Kingdom. Quantum Hunting; Random Sets of Points K. B. Kulasekera, Clemson University, Clemson, SC. Pseudo Residuals; Quasi Residuals James M. Landwehr, AT & T Bell Laboratories, Murray Hill, NJ. Projection Plots Louis Lefeber, York University, Toronto, ON, Canada. Productivity Measurement A. M. Liebetrau, Batelle Pacific Northwest Laboratories, Richmond, WA. Proportional Reduction in Error (PRE) Measures of Association Frederic M. Lord, Educational Testing Service, Princeton, NJ. Psychological Testing Theory David W. Matula, Southern Methodist University, Dallas, TX. Random Graphs P. McCullagh, Imperial College, London, United Kingdom. Quasi-Likelihood Functions Paul W. Mielke, Jr., Colorado State University, Fort Collins, CO. Quantit Analysis R. E. Miles, Australian National University, Canberra, Australia. Random Tessellations David Moher, University of Ottawa, Ottawa, ON, Canada. Quality Assessment for Clinical Trials Stephen G. Nash, Johns Hopkins University, Baltimore, MD. Quasi-Random Sampling; QuasiRandom Sequences Rajeshwari Natarajan, Southern Methodist University, Dallas, TX. Reciprocal Inverse Gaussian Distribution Bent Natvig, University of Oslo, Oslo, Norway. Priority Queue Valery Nevzorov, St. Petersburg State University, St. Petersburg, Russia. Record Values and Record Statistics J. V. Noble, University of Virginia, Charlottesville, VA. Quantum Mechanics and Probability W. R. S. North, Northwick Park Hospital, Harrow, United Kingdom. Quangle David J. Pack, Union Carbide Corporation, Oak Ridge, TN. Prior Distributions; Prior Probabilities Anthony G. Pakes, University of Western Australia, Nedlands, Western Australia. Queueing Theory S. Panchapakesan, Southern Illinois University, Carbondale, IL. Ranking and Selection Procedures G. P. Patil, Pennsylvania State University, University Park, PA. Profiles of Diversity B. F. Pendleton, University of Akron, Akron, OH. Ratio Correlation A. N. Pettitt, University of Technology, Loughborough, United Kingdom. Rank Likelihood E. C. Pielou, University of Lethbridge, Lethbridge, AB, Canada. Quadrat Sampling
vii
Harry O. Posten, University of Connecticut, Storrs, CT. Quincunx Albert Prat, Universitat Politecnica de Catalunya, Barcelona, Spain. Questii ¨ o´ Sandra Pruzansky, AT & T Bell Laboratories, Murray Hill, NJ. Proximity Data C. P. Quesenberry, North Carolina State University, Raleigh, NC. Probability Integral Transformations Damaraju Raghavarao, Temple University, Philadelphia, PA. Random Balance Designs James O. Ramsay, McGill University, Montreal, PQ, Canada. Principal Differential Analysis C. Radhakrishna Rao, University of Pittsburgh, Pittsburgh, PA. Rao’s Axiomatization of Diversity Measures J. N. K. Rao, Carleton University, Ottawa, ON, Canada. Ratio Estimators—I Poduri S. R. S. Rao, University of Rochester, Rochester, NY. Ratio Estimators—II Amnon Rapoport, University of Haifa, Haifa, Israel. Psychological Decision Making Campbell B. Read, Southern Methodist University, Dallas, TX. Ranges; Rao–Blackwell Theorem Bengt Ros´en, Statistics Sweden, Stockholm, Sweden. Quality Concept for Official Statistics James L. Rosenberger, Pennsylvania State University, University Park, PA. Recovery of Interblock Information William D. Rowe, American University, Washington, DC. Rare-Event Risk Analysis Bernhard Ruger ¨ , University of Munich, Munich, Germany. Profile-a Tests A. Ruiz-Gazen, Universite Toulouse 1-Sciences Sociales, Toulouse, France. Principal Component Analysis, Generalized Harold Sackrowitz, Rutgers University, Piscataway, NJ. Rationality Criterion Ester Samuel-Cahn, Hebrew University, Jerusalem, Israel. Prophet Inequalities Hans T. Schreuder, U.S. Forest Service, Fort Collins, CO. Ratio Estimators, Tin’s; Ratio and Regression Estimators, Mickey’s; Ratio Estimator, Pascual’s P. K. Sen, University of North Carolina, Chapel Hill, NC. Progressive Censoring Schemes; Progressively Censored Data Analysis; Rank Statistics E. Seneta, University of Sydney, Sydney, Australia. Probability, History of L. R. Shenton, University of Georgia, Athens, GA. Quasibinomial Distributions Andrew F. Siegel, University of Washington, Seattle, WA. Rarefaction Curves Gerald L. Sievers, Western Michigan University, Kalamazoo, MI. Probability Plotting
viii
CONTRIBUTORS
Barry Simon, California Institute of Technology, Pasadena, CA. Quantum Physics and Functional Integration C. J. Skinner, University of Southampton, Southampton, United Kingdom. Probability Proportional to Size (PPS) Sampling Grant W. Somes, University of Tennessee, Memphis, TN. Proportions, Cochran’s Q-Statistic for S. M. Steinberg, University of North Carolina, Chapel Hill, NC. Quantile Estimation Stephen M. Stigler, University of Chicago, Chicago, IL. Quetelet, Adolphe S. Lynne Stokes, University of Texas, Austin, TX. Ranked Set Sampling David J. Strauss, University of California, Riverside, CA. Random Utility Models Heinrich Strecker, University of Tuubingen, ¨ Tuubingen, ¨ Germany. Quotient Method Donna F. Stroup, University of Texas, Austin, TX. Problem Solving in Statistics Seymour Sudman, University of Illinois, Urbana, IL. Public Opinion Polls Jiayang Sun, Case Western Reserve University, Cleveland, OH. Projection Pursuit
W. Y. Tan, Memphis State University, Memphis, TN. Quadratic Forms Thaddeus Tarpey, Wright State University, Dayton, OH. Principal Points Rasesh Thakkar, York University, Toronto, ON, Canada. Purchasing Power Parity ¨ Tubingen, ¨ Gerhard Tintner, University of Tuubingen, Germany. Quotient Method R. L. Trader, University of Maryland, College Park, MD. Quasi-Bayesian Inference Brent M. Troutman, U.S. Geological Survey, Lakewood, CO. Rainfall, Landforms, and Streamflow Dietrich Von Rosen, Uppsala University, Uppsala, Sweden. Profile Analysis Gernot Wassmer, Universitat zu Kooeln, ¨ Kooeln, ¨ Germany. Profile-a Tests D. S. Watkins, Washington State University, Pullman, WA. QR Algorithm George H. Weiss, National Institutes of Health, Bethesda, MD. Random Walks D. G. Wellington, Environmental Protection Agency, Washington, DC. Rai and Van Ryzin Dose–Response Model
P
(Continued)
PREFERENCE MAPPING. See GRAPHICAL
consider various properties, such as consistency of our estimator, as n → ∞. However, study of the properties of prequential schemes in the original context of probability forecasting [4,6] suggested that it could be fruitful to reformulate many such nonprequential activities in the prequential setting. The general framework for such ‘‘prequential inference’’ was laid down by Dawid [5]. In it, estimation, for example, is treated merely as an aid to prediction of the next observation. Success at estimation is then measured by the quality of the associated prequential forecasting system, which, in turn, is judged directly by comparing forecasts with actual outcomes. Apart from the pleasing mathematical properties that, it turns out, are possessed by such procedures, it is intuitively more satisfying to formulate inferences in terms of future observables rather than, say, unobservable parameters: for only in such a case can we ever know whether or not we are being successful. Prediction of observables has always been a practically important and theoretically challenging task for probability and statistics: see the Encyclopedia of Statistical Science articles by Kallianpur [15], Ledolter [18], and Geisser [13]. Prequential analysis shares the general concerns of all predictive analysis, but with special emphasis on the assessment of predictive performance through comparison of outcomes with their predictions. In this, it is superficially similar to assessment by cross-validation, or predictive sample reuse [12,26], in which, for a fixed sample, a forecast is made for each data point on the basis of an analysis of all the other points in the sample omitting that one, this forecast then being compared with the omitted value. In prequential analysis, by contrast, the forecast is based on previous values only. This avoids any problems as further data accrue and, more important, eliminates dependence between the assessments for different data points, thus simplifying analysis and interpretation, and permitting the development of a very general and powerful underlying theory.
REPRESENTATION OF DATA
PREFERENCES. See BAYESIAN
INFERENCE;
DECISION THEORY
PREGIBON’S LINK TEST. See LINK TESTS PREMATURE AGGREGATION. See HALF-NORMAL PLOTS
PREQUENTIAL ANALYSIS There are many applied problem areas where data are observed sequentially, and where at any time it is required to issue a forecast, make an inference, or take an action, related to the next observation, on the basis of past and current data. Examples include weather forecasting, playing the stock market, or thermostatic control of a refrigerator. Such problems may be termed prequential, i.e., predictive sequential. In cases where only a point forecast, or some action, is required, it is nonetheless often useful to think of the forecaster as first formulating a probability distribution, Pi+1 say, representing uncertainty about the next observable Xi+1 in the light of the accumulated data Xi = xi [where Xi denotes the initial sequence (X1 , . . . , Xi ) of the variables, and xi := (x1 , . . . , xi ) are their observed values], and then applying this to solve the specific decision problem at hand. A variety of methods is available to assess the quality of such sequential probability forecasts, or their implied decisions, in the light of the actual outcomes [8]. Much of the impetus for these arose from the specific application to meteorology [2,19]. In probability and statistics more generally, we often meet problems in which variables (X1 , X2 , . . .) arise sequentially, but our goals are typically not expressed in this prequential form. For example, we might think of estimating an underlying parameter θ of the joint distribution of X := (X1 , X2 , . . .) on the basis of data Xi = xi , and would then 6391
6392
PREQUENTIAL ANALYSIS
PREQUENTIAL ASSESSMENT OF PROBABILITY FORECASTS Suppose that, for a realized sequence x := (x1 , x2 , . . .) of outcomes of variables X := (X1 , X2 , . . .), a string of probability forecasts P := (P1 , P2 , . . .) has been issued prequentially. Thus Pi+1 is a distribution for Xi+1 , formulated when the data Xi = xi are in hand. One might think of Pi+1 as being formed, as the appropriate conditional distribution for Xi+1 given Xi = xi , from some underlying joint distribution P for X; but this is not necessary. Such an underlying P could be termed a probability forecasting∗ system (PFS), since it determines the forecast for Xi+1 for any i and any data string xi . The general methodology of probability forecasting (see ref. [8], which could usefully be read in conjunction with this article) supplies a number of ways in which we can assess the performance of the issued forecasts P in the light of the actual observations x. In the case of binary (Xi ), we might assess ‘‘calibration’’ up to time n by examining quantities of the form νn−1 {(xi − pi ) : 1 i n; i ∈ ξ }, where pi is the assessed probability Pi (Xi = 1); ξ is some appropriately constructed subsequence (which could be the whole sequence, or, for example, contain just those terms i for which |pi − π0 | < , for some fixed π0 and ); and νn is the number of terms in the summation. For a sequence P to be ‘‘valid,’’ such quantities should tend to zero as n → ∞, since if P is a distribution underlying P this property must hold with P-probability 1 [4]. By considering enough subsequences simultaneously, we can show that valid forecasts are essentially uniquely determined, in the sense that, if P and P are both valid forecast sequences for the same outcome sequence x, we must have pi − pi → 0 as i → ∞ [6]. For continuous (Xi ), we can form, from P and x, the conditional probability integral transform (CPIT) ui := Pi (Xi xi ). Then P may be considered a valid forecast sequence, for data x, if u := (u1 , u2 , . . .) looks like a random sample from the uniform distribution U[0, 1]: again, if P underlies P, this must hold with P-probability 1, thus justifying such a criterion. In particular, for any t and suitable subsequences, we could require that the relative frequency, in the subsequence,
of occurrence of the events Ai : ui t should converge to t; again forecasts valid under this criterion will be essentially unique. Another approach to forecast assessment is to award a ‘‘performance score’’ si ≡ si (xi , Pi ) to the forecast Pi for Xi when in fact Xi = xi , and to compare different sequences Pn of forecasts for the first n outcomes xn by means of their cumulative scores Sn (x, P) := n s (x i i , Pi ). A simple and important special i=1 case has si ≡ log pi (xi ), withpi (·) the density of Pi . Then Sn (x, P) = ni=1 log pi (xi ) = log p(xn ), the overall log likelihood based on data xn for an underlying joint distribution having joint density p(·). We call it the prequential log likelihood for Pn based on xn . Thus, in this case, comparison is simply likelihood-based. If X is generated from a joint distribution P, and Q is any other distribution for X, then Sn (X, P) − Sn (X, Q) is, with P-probability one, bounded below; typically it will tend to infinity. In this sense, forecasts constructed from the true distribution will be optimal. PREQUENTIAL PRINCIPLES The preceding validity criteria have the property that they can be calculated knowing only the realized sequences x and P; in particular, no other features of any joint distribution, or PFS, P underlying the actual forecasts issued are required. This is clearly desirable in applications such as weather forecasting, where it would be unreasonable to expect that such a P has even been formulated. Any criterion with this property will be said to satisfy the weak prequential principle (WPP). The justifications we have given for using such measures, however, have rested on their probabilistic properties under an underlying distribution P (i.e., when the same P used for constructing forecasts is assumed to have generated the data). Nevertheless, these properties, in fact, did not depend on the detailed specification of P. Such probabilistic properties may be said to satisfy the strong prequential principle (SPP). Although indeed strong, it turns out that this is satisfiable for a wide range of interesting problems. Suppose, for example, that, for binary (Xi ), data sequence x, and forecast sequence given by p = (pi ), where pi = Pi (Xi = 1), we wish to
PREQUENTIAL ANALYSIS
assess whether the observed value of the simple calibration measure, n−1 ni=1 (xi − pi ), is ‘‘significant.’’ A natural test statistic, satisfying WPP, is 1/2 n n (xi − pi ) pi (1 − pi ) . yn := i=1
i=1
This is the observed value of 1/2 n n (Xi − pi ) pi (1 − pi ) . Yn := i=1
i=1
If we consider the specific underlying P for which the (Xi ) are independent, with P(Xi = 1) = pi , then Yn has mean 0 and variance 1, and (so long as its denominator tends to ∞) is asymptotically standard normal. For a different underlying P, however, we might have dependent (Xi ); then pi will not be a constant but random, being a function of Xi−1 . Nevertheless, it turns out that the asymptotic standard normal distribution for Yn will still hold, essentially under any such P [24]. So a significance test that refers the observed yn to tables of the standard normal distribution satisfies, asymptotically, (WPP and) SPP. Similarly, many other tests with this property can be constructed. Some understanding of such phenomena can be grounded in the theory of global probability, introduced by Vovk [28], which is essentially determined by realized forecast sequences alone, and is consistent with ordinary Kolmogorov probability under any underlying distribution.
6393
on Xi , and define Pi+1 = Pθ˜ ,i+1 . This is the i plug-in SFS determined by the estimator sequence (θ˜i ). Alternatively, we might assign to θ a prior distribution, say with density π0 (·), construct the mixture P = Pθ π0 (θ )dθ , and use this to generate forecasts. This gives a Bayes SFS. It has Pi+1 = Pθ ,i πi (θ )dθ , where πi (·) is the posterior density of θ given Xi . Example 1. Suppose that, under Pθ , the (Xi ) are independent exponential E(θ ) variables, with density pθ (x) = θ e−θ x (x > 0, θ > 0). The maximum likelihood estimator based on xi is θˆi = i/ti , with ti = ij=1 xj , yielding the MLE plug-in SFS under which Xi+1 ∼ Pi+1 = E(i/ti ). If we give θ a gamma (a, b) prior distribution, (a + i, b + ti ) is the posterior given xi , and the Bayes SFS forecast distribution Pi+1 for Xi+1 is [(b + ti )/(a + i)] × F2,2(a+i) , F·,· denoting the F-distribution. In either case, the underlying P incorporates dependence between the Xi , since Pi+1 depends on Xi through Ti := ij=1 Xj . As i, ti → ∞, the two SFSs produce essentially identical forecasts. An alternative ‘‘fiducial’’ SFS may be constructed by noting that, for any θ , Xi+1 /Ti ∼ (1/i)F2,2i , suggesting Pi+1 = (ti /i)F2,2i , a limiting form of the above Bayes SFS as a, b → 0. (Note that both this and the MLE plug-in SFS require ad hoc modification for i = 0.) As the construction of an SFS is essentially arbitrary, we need to identify those that are ‘‘good,’’ in the sense of having desirable behavior under any Pθ ∈ P. PREQUENTIAL CONSISTENCY
STATISTICAL FORECASTING SYSTEMS Now suppose we have a parametrized family of forecast distribution sequences Pθ = (Pθ ,i , i = 1, 2, . . .), Pθ ,i+1 being the probability forecast made for Xi+1 , in the light of past data xi , under parameter value θ . These might be constructed from an underlying statistical model P = (Pθ ). The statistician is allowed to make use of this model, or its forecasts, in forming her own forecast Pi+1 . A system for so doing, for any data string xi , is termed a statistical forecasting system (SFS) for P. A simple class of SFSs may be constructed as follows. Let θ˜i be an estimator of θ based
In Example 1, under Pθ , Ti → θ −1 almost surely. It follows that any of the forecasts Pi+1 constructed above will be asymptotically indistinguishable from the true forecast distribution E(θ ) under Pθ , no matter what θ is. This desirable property is termed prequential consistency. Typically it will hold even in cases where consistent estimation of θ , in the usual parametric sense, is impossible: see ref. [5, Example 3]. PREQUENTIAL EFFICIENCY Let P be a Bayes SFS for P, with everywhere positive prior density π0 (·), and let Q
6394
PREQUENTIAL ANALYSIS
be an arbitrary PFS. The likelihood ratio in favor of Q as against P remains bounded (and will typically tend to 0) as the number of observations tends to infinity, with probability 1 under Pθ ∈ P, for almost every θ . In this sense, if P is a valid model, a Bayes SFS P cannot be beaten. This property may be termed prequential efficiency. It will also typically (but not universally) hold for the MLE plug-in SFS. Any prequentially efficient SFS P provides, in an optimal way, a single joint distribution which, for predictive purposes, can replace the full family P. MODEL TESTING Suppose we wish to test the validity of the family P of exponential distributions in Example 1. As a simple and concrete example, we might construct, for each i, the discrepancy Xi − µi , where µi is the expectation of Xi under the forecast distribution Pi made by some prequentially efficient SFS; and then assess overall departure, up to time n, by the statistic ni=1 (Xi − µi )/( ni=1 σi2 )1/2 , where σi2 is the variance of Xi under Pi . Using, for simplicity, the MLE plug-in SFS (and omitting the problematic but ignorable term for P1 ) yields n−1 1/2 . It can be (X − T /i)]/[ [ n−1 i+1 i i=1 i=1 (Ti /i)] shown that, for any θ , this will be asymptotically standard normal when the (Xi ) are independent E(θ ) variables. We thus obtain a simple test of the model. In more general models, such test statistics, and more complex variations on them, will again be asymptotically standard normal whenever prequentially efficient forecasts are utilized, under some regularity conditions [25]. Cowell et al. [3] have applied such tests to assess the performance of probabilistic expert systems∗ . MODEL CHOICE [11, § 6.4] set of foreIf (Pk ) is a finite or countable P casting systems, and P = α k k , with αk > 0, αk = 1, is a Bayes SFS for (Pk ), then P is prequentially efficient and consistent for (Pk ). Given data xn , P’s forecast for Xn+1 is αkn Pk,n+1 , where αkn ∝ αk pk (xn ) is the posterior probability of Pk for priors (αk ). Typically, the (Pk ) will, as distributions over sequence
space, be mutually singular. In this case αkn → 1 as n → ∞ with Pk -probability 1, so P will ‘‘home in’’ on the true distribution. Then ˆ where Pˆ n+1 = Pk ,n+1 if kn the plug-in SFS P, n maximizes αkn over k, will likewise be consistent. Now consider a collection (Pk ) of parametric statistical models, where Pk = (Pk,θk ) has finite-dimensional parameter θk with prior density πk (·) > 0. Let Pk bea prequentially efficient SFS for Pk and P = αk Pk , as above. Under mutual singularity of the (Pk ), which will typically hold for distinct models, we then have αkn → 1 with Pk,θk -probability 1 for almost all Pk,θk ∈ Pk . Hence choosing k to maximize the ‘‘adjusted prequential likelihood’’ (or posterior probability) αk pk (xn ) (where we may omit αk if the number of models is finite) will yield a consistent strategy for model selection. Even when the various derived (Pk ) are not mutually singular, the plug-in SFS, or the full mixture SFS αk Pk , will give asymptotically correct forecasts, no matter which Pk,θk holds. In regular models, the prequential log likelihood log pk (xn ) can be approximated by the Jeffreys–Schwarz criterion [26] log pk,θˆk (xn ) − 12 dk log n, where θˆk is the maximum likelihood estimate of θk in Pk based on xn , and dk is the dimensionality of θk ; more refined approximations are given by Phillips and Ploberger [21]. Note that, in the direct prequential approach, any required ‘‘penalization’’ of the likelihood is effected automatically. ROBUSTNESS [9, § 8] Suppose that, although we are using a model P = (Pθ ) to aid prediction, the actual unknown joint distribution generating X is / P. We can still find a SFS which will P∗ ∈ perform effectively as well as will any Pθ ∈ P, with probability 1 under P∗ , for any P∗ . However, the construction of this will depend on the specific decision problem at hand. Suppose that, just before time i + 1, we have to choose an act a, and will suffer a loss depending on a and the value xi+1 of Xi+1 . Without loss of generality we can assume that the act a involves specifying a probability distribution Q for Xi+1 , and that the loss, if in fact Xi+1 = x, is given by a proper scoring rule
PREQUENTIAL ANALYSIS
S ≡ S(x, P) [8]; that is, when X ∼ P, S(P, Q) := E{S(X, Q)} is minimized, in Q, by quoting the true distribution, Q = P. The excess loss in quoting another Q rather than Q = P is then D(P, Q) := S(P, Q) − S(P, P), and this can be used as a measure of discrepancy between distributions, appropriate for the decision problem at hand. For the logarithmic scoring rule S(x, P) ≡ − log p(x), for example, this yields the Kullback–Liebler∗ discrepancy between P and Q. Alternatively, a quadratic loss function yields S(x, P) = [x − EP (X)]2 and the discrepancy D(P, Q) = [EP (X) − EQ (X)]2 . For any infinite data sequence x, we can consider, for each θ , thelimiting average discrepancy, limn→∞ n−1 ni=1 D(P∗i , Pθ ,i ), between the ‘‘true’’ forecasts, based on P∗ , and those based on Pθ ∈ P. A value θ ∗ minimizing this quantity may be regarded as giving the closest approximation to P∗ obtainable from within P (note that θ ∗ may depend on the data x). Since P∗ is unknown, so is θ ∗ . Howthe minimizer of the actual ever, consider θn† , cumulative score ni=1 S(xi , Pθ ,i ) for the initial data sequence xn . Then θn† is calculable from the data, and moreover, under weak conditions we can show that we have θn† → θ ∗ with P∗ -probability 1. That is, choosing θ = θn† by minimizing the actual total score will yield, asymptotically, the closest approximation to the true, unknown P∗ ; and the plug-in SFS formed from the estimator sequence (θn† ) will deliver, asymptotically, the same forecasts as this closest approximation. STRATEGIES In attempting to describe a data sequence x, we might undertake various more or less complex modeling exercises. Almost any standard statistical technique, for example factor analysis∗ , nonlinear time-series∗ analysis, or nonparametric density estimation∗ , may be applied to the partial data xn and used to generate a suitable probability forecast Pn+1 for Xn+1 . Thus any data analysis strategy may be converted into a PFS, and its predictive success assessed accordingly. We can feed this assessment back into the strategy in various ways, to produce a more complex strategy. This in its turn can be converted into a PFS, and its performance assessed.
6395
Suppose, for example, we have fitted a Gaussian AR(1) model, leading to an SFS P. At time n we could examine the CPIT values ui = Fi (xi ), i = 1, . . . , n, with Fi the distribution function of Xi under Pi , for uniformity and independence. Suppose we judge that independence and identical distribution are not at issue, but that the (ui ) do not appear uniformly distributed. Our analysis leading to P thus appears, empirically, to have been unsuccessful. We might however try to estimate the actual distribution of the (ui ), say by nonparametric density estimation, obtaining the estimated distribution function Gn (u). The past data are thus reasonably well described by taking Fi (Xi ) ∼ Gn , independently. If we now apply this ‘‘recalibration’’ correction to the next observation Xn+1 , we might take Fn+1 (Xn+1 ) ∼ Gn (rather than U[0, 1], as under Pn+1 ); this assertion can be transformed into a new predictive distribution, Qn+1 say, for Xn+1 . By continual reassessment of and correction for past performance, we can thus produce a new PFS Q, which we hope will describe the data better. However, we cannot be certain that it will do so, since the correction at any time is based on the past, but applied to the future, which might not continue the same pattern. (Dawid [7] considers some philosophical implications of this lack of guarantee.) In a variety of economic applications, Kling and Bessler [16] found that recalibration generally led to improved performance; but Bessler and Kling [1] gave examples of performance degrading with recalibration. As another application, we might have a collection of candidate models or strategies, as in the section ‘‘Model Choice,’’ and use some strategy based on their forecasts and the outcomes to data to select from or combine these, so as to produce a forecast for the next observation. (Bayes or maximum adjusted likelihood strategies might seem attractive, although their performance need not necessarily be good when the data cannot be regarded as arising from any of the candidate models.) Phillips [20] has applied this approach to 14 historical time series of the U.S. economy, the candidate models being autoregressions of order up to 6, with or without additional linear trend. For 10 of the series, an approximate maximum
6396
PREQUENTIAL ANALYSIS
prequential likelihood strategy had higher prequential likelihood than the generally favored AR(3)+linear trend model, often substantially. Hart [14] uses what he calls ‘‘time-series cross-validation,’’ essentially a prequential least-squares criterion, to choose the bandwith∗ for nonparametric kernel estimation of trend, in the presence of autocorrelated errors, so as to optimize forecast performance. The use of prequential assessment to choose, at each step, among models or strategies is close in spirit to cross-validation [27] or stacked generalization [29], but with the important difference that its output is itself simply another forecasting strategy. This, in turn, can be assessed prequentially and compared with others; and so on to ever deeper levels. At any level, prequential assessment supplies a valid measure of performance, unaffected by overfitting or data dredging. For further discussion, examples, and applications, see ref. [10]. CONNECTIONS WITH OTHER THEORIES Stochastic Complexity The total negative logarithmic prequential likelihood—log p(xn ) of a PFS P, for any data xn , can be regarded as defining the length function of a prefix code for encoding xn ; moreover, this will be, essentially, the shortest possible code when xn can be regarded as generated by the distribution P [11]. When P is prequentially efficient for a model P, this code length defines the stochastic complexity∗ (ref. [23]: see also ref. [22]) of xn relative to P. The theory of stochastic complexity may thus be treated as a special case of prequential analysis. Algorithmic Complexity Kolmogorov and Uspenskii [17] review this approach to probability theory, which attempts to explicate the idea that a given data string x can be considered ‘‘random’’ with respect to some joint distribution P. Equivalently, we might reexpress this as requiring that P be a ‘‘valid’’ PFS for x, as considered above. This reformulation then renders the insights and techniques of each approach, prequential analysis and algorithmic complexity, available to the other.
REFERENCES 1. Bessler, D. A. and Kling, J. L. (1990). Prequential analysis of cattle prices. Appl. Statist., 39, 95–106. 2. Brier, G. W. (1950). Verification of forecasts expressed in terms of probability. Mon. Weather Rev., 78, 1–3. 3. Cowell, R. G., Dawid, A. P., and Speigelhalter, D. J. (1993). Sequential model criticism in probabilistic expert systems. IEEE Trans. Pattern Recognition and Machine Intell., 15, 209–219. 4. Dawid, A. P. (1982). The well-calibrated Bayesian (with discussion). J. Amer. Statist. Ass., 77, 605–613. 5. Dawid, A. P. (1984). Statistical theory. The prequential approach (with discussion). J. Roy. Statist. Soc. A, 147, 278–292. 6. Dawid, A. P. (1985). Calibration-based empirical probability (with discussion). Ann. Statist., 13, 1251–1285. 7. Dawid, A. P. (1985). The impossibility of inductive inference. (Comments on ‘‘Selfcalibrating priors do not exist,’’ by David Oakes.) J. Amer. Statist. Ass., 80, 340–341. 8. Dawid, A. P. (1986). Probability forecasting. In Encyclopedia of Statistical Sciences, Vol. 7, S. Kotz, N. L. Johnson, and C. B. Read, eds. Wiley-Interscience, pp. 210–218. 9. Dawid, A. P. (1991). Fisherian inference in likelihood and prequential frames of reference (with discussion). J. R. Statist. Soc. B, 53, 79–109. 10. Dawid, A. P. (1992). Prequential data analysis. In Current Issues in Statistical Inference: Essays in Honor of D. Basu, M. Ghosh and P. K. Pathak, eds. IMS Lecture Notes–Monograph Ser. 17, pp. 113–126. 11. Dawid, A. P. (1992). Prequential analysis, stochastic complexity and Bayesian inference (with discussion). In Bayesian Statistics 4, J. M. Bernardo, J. Berger, A. P. Dawid, and A. F. M. Smith, eds. Oxford University Press, pp. 109–125. 12. Geisser, S. (1975). The predictive sample reuse method with applications. J. Amer. Statist. Ass., 70, 320–328. 13. Geisser, S. (1980). Predictive analysis. In Encyclopedia of Statistical Sciences, Vol. 7, S. Kotz, N. L. Johnson, and C. B. Read, eds. Wiley-Interscience, pp. 158–170. 14. Hart, J. D. (1994). Automated kernel smoothing of dependent data using time series crossvalidation. J. R. Statist. Soc. B, 56, 529–542.
PREVISION 15. Kallianpur, G. (1986). Prediction and filtering, linear. In Encyclopedia of Statistical Sciences, Vol. 7, S. Kotz, N. L. Johnson, and C. B. Read, eds. Wiley-Interscience, pp. 139– 148. 16. Kling, J. L. and Bessler, D. A. (1989). Calibration-based predictive distributions: an application of prequential analysis to interest rates, money, prices, and output. J. Business, 62, 477–499. 17. Kolmogorov, A. N. and Uspenskii, V. A. (1987). Algorithms and randomness. Theory Probab. Appl., 32, 389–412. 18. Ledolter, J. (1986). Prediction and forecasting. In Encyclopedia of Statistical Sciences, Vol. 7, S. Kotz, N. L. Johnson, and C. B. Read, eds. Wiley-Interscience, pp. 148–158. 19. Murphy, A. H. and Epstein, E. S. (1967). Verification of probabilistic predictions: a brief review. J. Appl. Meteorol., 6, 748–755. 20. Phillips, P. C. B. (1995). Bayesian model selection and prediction with empirical applications. J. Econometrics, 69, 289–331. 21. Phillips, P. C. B. and Ploberger, W. (1995). An asymptotic theory of Bayesian inference for time series. Econometrica, 64, 381–412. 22. Rissanen, J. (1985). Minimum-descriptionlength principle. In Encyclopedia of Statistical Sciences, Vol. 5, S. Kotz, N. L. Johnson, and C. B. Read, eds. Wiley-Interscience, pp. 513–527. 23. Rissanen, J. (1989). Stochastic Complexity in Statistical Inquiry. World Scientific Publishing, Singapore. 24. Seillier-Moiseiwitsch, F. and Dawid, A. P. (1993). On testing the validity of sequential probability forecasts. J. Amer. Statist. Ass., 88, 355–359. 25. Seillier-Moiseiwitsch, F., Sweeting, T. J., and Dawid, A. P. (1992). Prequential tests of model fit. Scand. J. Statist., 19, 45–60. 26. Smith, A. F. M. (1988). Schwarz criterion. In Encyclopedia of Statistical Sciences, Vol. 8, S. Kotz, N. L. Johnson, and C. B. Read, eds. Wiley-Interscience, pp. 289–290. 27. Stone, M. (1974). Cross-validatory choice and assessment of statistical predictions (with discussion). J. R. Statist. Soc. B, 36, 111– 147. 28. Vovk, V. G. (1993). A logic of probability, with application to the foundations of Statistics (with discussion). J. R. Statist. Soc. B, 55, 317–351. 29. Wolpert, D. H. (1992). Stacked generalization. Neural Networks, 5, 241–259.
6397
See also CONSISTENCY; EFFICIENCY, SECOND-ORDER; LIKELIHOOD; PREDICTION AND FORECASTING; PREDICTIVE ANALYSIS; and PROBABILITY FORECASTING.
A. P. DAWID
PREVAILING ITEM An obsolete term for mode (see MEAN, MEDIAN, AND MODE).
PREVISION Prevision is a concept introduced by B. de Finetti to establish the notion of expectation as the fundamental element of the subjectivist theory of probability. Your prevision, P(X), for the random quantity X may be thought of, intuitively, as the ‘‘fair price’’ for the random gain X [i.e., so that you are indifferent between the random gain X and the certain gain P(X)]. (Prevision, as a subjectivist concept, always expresses the judgment of a specified individual, here termed ‘‘you.’’) To make this precise, P(X) must be defined in terms of an operational criterion. One such criterion is as follows. For a general random quantity X, your prevision for X, P(X), is the value x that you would choose if you were subsequently to suffer a penalty L given by L = k(X − x)2 , where k is a constant defining the units of loss. Your previsions x1 , . . . , xm for the random quantities X1 , . . . , Xm are said to be coherent if there is no other choice x∗1 , . . . , x∗m which would lead to a reduction in your overall penalty for all possible combinations of values of X1 , . . . , Xm . (Thus, if you are not coherent, you are a ‘‘sure loser.’’) Coherence is here used in the same intuitive sense as for the axiomatic development of inference and decision-making (see COHERENCE), namely as a criterion to identify intrinsically contradictory beliefs. For any random quantities X and Y, coherence imposes the following requirements on your specifications: (1) infX P(X) sup X, and (2) P(X + Y) = P(X) + P(Y). These properties reflect the ‘‘fair price’’ interpretation. (Notice that prevision is finitely additive but not necessarily countably additive.) [(1) and (2) are special cases of the general result that a specification P(Yi ) = yi , i = 1, . . . , m, is
6398
PRIESTLEY’S TEST FOR HARMONIC COMPONENT(S)
coherent if and only if the point (y1 , . . . , ym ) lies in the closed convex hull of the set of possible values of the random vector (Y1 , . . . , Ym ) in m-dimensional Euclidean space.] Prevision can be considered to be expectation, but defined directly as a primitive quantity, rather than in terms of a prespecified probability distribution. For any event H, if you also use H to denote the indicator function of the event (i.e., H = 1 if H is true, H = 0 if false), then you may interpret P(H) as the probability of H. In this development there is no distinction between probability and expectation, and the notation is chosen to emphasize this. If you have specified a full probability distribution for X, your prevision for X will agree with the usual definition of expectation (at least if X is bounded; there are subtleties for unbounded X). The essential difference arises when you do not want to assess probabilities for all the possible outcomes, in which case P(X), being assigned directly, is well defined, as opposed to your expectation for X, which is defined only through the associated probability distribution for X. Thus you may consider ‘‘average’’ properties without exhaustively considering some limiting partition of possibilities, so that you are free to choose the level of detail at which you wish to describe your beliefs. Using H interchangeably as the event and the indicator function of your event, your prevision of the random quantity X, conditional on the event H, P(X|H), is defined exactly as for P(X), but with the penalty L replaced by the ‘‘called off’’ penalty HL. Coherence imposes the following requirement: (3) P(HX) = P(H)P(X|H). The essential reference on prevision is de Finetti [1,2], where all the foregoing material is developed. Much of the importance of prevision lies in the depth and thoroughness of the treatment which de Finetti provides in these two volumes. Some statistical implications of this formulation are developed in Goldstein [3] as follows. In the usual Bayesian formulation, your beliefs are collected into a prior probability distribution which is modified by Bayes’ theorem in the light of new information to yield a posterior∗ probability distribution. When working with prevision rather than probability, it is shown
that your beliefs should be collected into a prior inner product space, which is modified by a linear transformation, in the light of new information, to yield a posterior inner product space. The relationship between the conditional probability approach and the linear transformation approach is discussed in detail in Goldstein [5]. These developments rely on the condition that if PT (X) is the actual value that you will assign for P(X) at the future time T, then your current previsions should satisfy the relation: (4) P(PT (X)) = P(X). [This result would not, in general, be true if PT (X) were replaced by conditional prevision.] A coherence argument for (4) is given in Goldstein [4]. REFERENCES 1. de Finetti, B. (1974). Theory of Probability, Vol. 1. Wiley, London. 2. de Finetti, B. (1975). Theory of Probability, Vol. 2. Wiley, London. 3. Goldstein, M. (1981). J. R. Statist. Soc. B, 43, 105–130. 4. Goldstein, M. (1983). J. Amer. Statist. Ass., 78, 817–819. 5. Goldstein, M. (1984). In Bayesian Econometrics and Statistics, Vol. 3 (in honour of B. de Finetti), Prem. K. Goel and A. Zellner, eds. North-Holland, Amsterdam (in press) See also COHERENCE; CONDITIONAL PROBABILITY AND EXPECTATION; EXPECTED VALUE; and SUBJECTIVE PROBABILITIES.
MICHAEL GOLDSTEIN
PRIESTLEY’S TEST FOR HARMONIC COMPONENT(S). See TIME SERIES, PRIESTLEY’S HARMONIC COMPONENT TEST FOR
PRIMARY SAMPLING UNIT This is a term used in multistage sampling. At the first stage of sampling it is necessary to divide the entire population or universe into a convenient number of primary sampling units (PSUs). In large-scale area samples∗ it is usual to form a few thousand PSUs, from which a few hundred would be selected
PRIMARY SAMPLING UNIT
to constitute the first-stage sample. Thus in area samples of the United States it is not uncommon for the PSUs to be counties. The second stage of selection in multistage sampling takes place only within those PSUs which form the first-stage sample. Secondary sampling units, for instance census tracts, need to be listed only within those selected PSUs. If the number of selected PSUs is held constant, the consequences of forming large PSUs are that many second-stage units need to be listed and that those selected are scattered over relatively large areas. Conversely, if the PSUs are made small, the task of selecting at the second stage involves forming relatively short lists of second-stage units, and those selected are geographically concentrated. It is usual, particularly in area sampling, for primary sampling units to have unequal probabilities of selection or of inclusion in a sample. These selection or inclusion probabilities are usually proportional to some measure of size, such as the number of persons or dwellings recorded in the PSU at the latest available census. There are mathematical and operational advantages in selecting PSUs ‘‘with replacement.’’ To select with unequal probabilities with replacement, the PSUs within each (usually geographical) stratum are assigned selection probabilities which sum to unity. A series of n sample draws is then made using these probabilities. It is possible for a PSU to be selected more than once, hence the expression ‘‘with replacement.’’ (Because the same probabilities are used at each draw, the distribution of the numbers of times each PSU is selected in a sample is multinomial∗ . Since ‘‘sampling with replacement’’ does not necessarily imply that the same probabilities of selection are used at each draw, this type of sampling is occasionally referred to as ‘‘multinomial.’’) If sampling is to be without replacement at the first stage, the probabilities of inclusion in the sample are usually chosen to sum to n, an integer, which is the required sample number or sample size. A procedure then has to be specified for selecting the sample in accordance with these inclusion probabilities. Either this procedure itself will be complicated, or the variance estimation will be
6399
complicated, inefficient or biased, or the sample must be limited to two units. Sampling ‘‘with replacement’’ (multinomial sampling) is less efficient but can be distinctly more convenient. In particular, if the PSUs are selected ‘‘with replacement,’’ then regardless of the selection methods used at subsequent stages, it is possible to obtain an unbiased estimator of variance (of the unbiased estimator of mean or total) by comparing the estimates based on the individual PSUs. Since these were selected independently, the estimates based on them are also independent. This applies only to estimation of the variance over all stages of sampling combined. To obtain simple unbiased estimators of the variances at the individual stages, sampling must be ‘‘with replacement’’ at the subsequent stages as well as the first (see STRATIFIED MULTISTAGE SAMPLING). In the event that a PSU is selected twice or three times, it is usual to double or triple the number of second-stage sample units to be selected within it. This may in some cases involve multiple selection of second- or even lower-stage units. If final-stage units are selected more than once, unequal weighting must be used in the sample estimator. Since area sampling effects a great simplification in the sampling procedure and the resulting clustered samples∗ are much cheaper to enumerate than unclustered samples, this technique is widely used in largescale household surveys such as the Current Population Survey∗ and those of the major private polling organizations in the United States and other developed countries. Although the term ‘‘primary sampling unit’’ is seldom used outside the context of area sampling, it is not confined to it. Thus if a two-stage sample is used in which the first stage consists of a sample of boxes and the second of cards within those boxes, the boxes would be the PSU’s. BIBLIOGRAPHY Barnett, V. (1974). Elements of Sampling Theory. Hodder & Stoughton, London, Chap. 5. (Concise treatment at an intermediate mathematical level.)
6400
PRIMITIVE ROOT
Sudman, S. (1976). Applied Sampling. Academic Press, New York. Chap. 7. (For survey users with limited statistical backgrounds.)
properties of their proposals. One of these proposals consists in letting T=
See also AREA SAMPLING; STRATIFIED MULTISTAGE SAMPLING; and SURVEY SAMPLING.
K. R. W. BREWER
PRIMITIVE ROOT In any finite field (Galois field∗ ), there is at least one element in the field, different powers of which give all the nonzero elements of the field. Such an element is called a primitive root of the field. For example, if the number of elements in the field is p = 7, then 3 is a primitive root since (modulo p) 31 = 3,
32 = 2, 35 = 5,
33 = 6, and
34 = 4,
36 = 1.
(Actually, X p−1 = 1 when X is any element of the field.) See also CONFOUNDING; FRACTIONAL FACTORIAL DESIGNS; and GALOIS FIELDS.
PRINCIPAL COMPONENT ANALYSIS, GENERALIZED Let V be the variance matrix of the pdimensional data X 1 , . . . , X n and T be a positive definite p × p matrix. While classical principal component analysis∗ (PCA) rests on the spectral decomposition of V, generalized PCA rests on the eigen analysis of VT −1 and the T −1 -orthogonal projection (instead of the canonical projection) of the centered data on the subspace Eq of Rp spanned by the q (q < p) eigenvectors associated with the q largest eigenvalues. Various choices of T give rise to interesting projections of the data. In order to display clusters, Art et al. [1] consider a withincluster dispersion matrix T estimated without knowledge of the clusters. In a similar approach displaying outliers∗ [4], the matrix T is a robust estimate of the total variance. Caussinus and Ruiz-Gazen [2,3] propose to simplify these methods and give some
n−1 n i=1
exp[−(β/2)||X i −X j ||2 −1 ](X i −X j )(X i −X j ) j=i+1 V , n−1 n exp[−(β/2)||X i −X j ||2 −1 ] j=i+1 i=1 V
where the squared norm ||x||2V −1 = x V −1 x for any column vector x of Rp . The use of an exponential weighting of the pairwise differences in the definition of T permits the authors to develop theoretical results. These results are obtained for large samples from a mixture distribution∗ Np (x, W)dP(x), where P, the nonnormal distribution of interest, is concentrated on an unknown k-dimensional subspace Fk of Rp (k < p), and Np (x, W), the p-dimensional normal density with mean x and covariance W, is the noise. For instance, Ek tends to Fk when n goes to infinity if P is a discrete distribution defining clusters, W is small enough with respect to the variance of P, and β is suitably chosen (in practice, 1 β 2). Since k is unknown, means are also provided to assess an appropriate dimension for a useful projection. All the displays are affine invariant. These techniques can be considered as alternatives to projection pursuit∗ . REFERENCES 1. Art, D., Gnanadesikan, R., and Kettenring, J. R. (1982). Data based metrics for cluster analysis. Utilitas Math., 21, 75–99. 2. Caussinus, H. and Ruiz-Gazen, A. (1994). Projection pursuit and generalized principal component analyses. In New Directions in Statistical Data Analysis and Robustness, S. Morgen¨ thaler et al., eds. Birkhauser, Basel, pp. 35–46. 3. Caussinus, H. and Ruiz-Gazen, A. (1995). Metrics for finding typical structures by means of principal component analysis. In Data Science and its Applications, Y. Escoufier et al., eds. Academic Press, pp. 177–192. 4. Yenyukov, I. S. (1988). Detecting structures by means of projection pursuit. COMPSTAT 88, Physica-Verlag, Heidelberg, pp. 47–58. See also COMPONENT ANALYSIS; PRINCIPAL COMPONENTS; and PROJECTION PURSUIT.
H. CAUSSINUS A. RUIZ-GAZEN
PRINCIPAL COMPONENTS
PRINCIPAL COMPONENTS This entry is an update to COMPONENT ANALYSIS. It focuses on developments during the past decade, highlighting mostly modelbased generalizations of principal component analysis. The topics discussed are: (1) generalizations to several groups; (2) nonlinear generalizations; (3) principal components in classification and regression; (4) patterned covariance matrices; and (5) varia, including testing problems and robust estimation. Other recent developments are treated in three books [9,19,23] that deal exclusively with principal component analysis. GENERALIZATIONS TO SEVERAL GROUPS When multivariate data fall into natural groups, such as males and females of the same species, the investigator may be interested in comparing principal components across groups. A descriptive method achieving this goal consists of computing angles between the linear subspaces spanned by the first q (out of p) principal components in each group [27]. In a model setup, assume that the principal component directions (eigenvectors of the covariance matrices) are identical across groups, while the variances may differ. Specifically, let i denote the covariance matrix of a p-variate random vector X in the ith group, and let i = Bi i Bi be its spectral decomposition, where the columns of the orthogonal matrix Bi are the eigenvectors of i , and i is a diagonal matrix with the eigenvalues of i as diagonal entries. The common principal component (CPC) model [9] assumes that all Bi are equal, i.e., all i can be diagonalized simultaneously by a single orthogonal matrix B. With B = (β 1 · · · β p ) and i = diag(λi1 , . . . , λip ), the CPC model can be written as i = Bi B =
p
λij β j β j .
6401
estimates of the parameters, based on k sample covariance matrices Si , can be computed using a Jacobitype algorithm for simultaneous diagonalization [10]. The first principal component of logtransformed measurements is often used in morphometrics to estimate constants of allometric growth [22,24]. In the case of several related species, the investigator may want to estimate identical constants of allometric growth for all groups under consideration [1,9], without imposing the constraint of equality of all eigenvectors of the k covariance matrices. A modification of the CPC model, the partial CPC model, assumes that i = λi1 β 1 β 1 +
p
(i) λij β (i) j βj ,
i = 1, . . . , k.
j=2
Here, the orthogonal matrices Bi = (β 1 β (i) 2 ··· β (i) ) have one vector in common, while the p remaining ones are allowed to differ between groups. In a variation of this model, q 2 eigenvectors of each covariance matrix span the same subspace [9,31,34]. Another modification of the CPC model imposes constraints λij = ci λj on the diagonal matrices i ; this results in the model of proportional covariance matrices i = ci 1 ,
i = 2, . . . , k,
for proportionality constants c2 , . . . , ck [8,9,21,30]. Related approaches in the psychometric literature are called three-mode∗ principal component analysis [26]. They are based on three- and higher-way data matrices (e.g., data arranged by variables, objects, and experimental conditions), and originated in the psychometric literature, but have been further developed in chemometrics∗ [17,18,42,43]. These methods rely on algebraic decompositions of higherorder data matrices and are not explicitly based on parametric models (see LINEAR ALGEBRA, COMPUTATIONAL). Solutions are usually obtained by a least squares criterion.
j=1
NONLINEAR GENERALIZATIONS The CPC model holds trivially if all covariance matrices are equal or proportional. Normal-theory maximum likelihood
A principal curve, or self-consistent curve, of a p-variate random vector X is a smooth
6402
PRINCIPAL COMPONENTS
curve in Rp such that each point x on the curve is the conditional mean of X in the subset of all points in Rp that are closer to x than to any other point on the curve [16]. For elliptical distributions, principal-component axes are self-consistent curves; thus principal curves are a genuine generalization of ordinary linear principal components. For a bivariate spherically symmetric random vector X, the circle of radius E[ X ] centered at the mean of X is also self-consistent. Self-consistent curves other than lines can typically only be found by iterative computations similar to a k-means algorithm∗ . In addition, for estimation of principal curves based on observed data one needs a scatterplot smoothing technique [2,16,39]. Principal curves can be generalized to principal surfaces, i.e., smooth manifolds∗ that generalize subspaces spanned by subsets of eigenvectors in linear principal component analysis [16]. While principal curves generalize principal components using self-consistency∗ as the defining criterion, smallest additive principal components [7] generalize the principal components associated with the smallest eigenvalues by constructing additive funcp tions i=1 i , where the i (Xi ) are nonlinear functions of the original variables, such that var[ i ] is a minimum subject to the constraint var[i ] = 1. PRINCIPAL COMPONENTS IN CLASSIFICATION AND REGRESSION In discriminant analysis∗ , principal components are sometimes used to reduce the dimensionality of the data, but the success of such a procedure depends on the assumption that discarding principal components with small eigenvalues will not, at the same time, discard valuable information on differences between the groups. A different application of principal components in discriminant analysis imposes constraints like the ones discussed above in connection with the covariance matrices [3,14,30]. This leads to classification∗ rules that are quadratic but reduce the number of parameters estimated, which may improve the expected error rates. Principal components are also used frequently in cluster analysis∗ to detect
homogeneous subsets of data; in particular, a scatterplot of the first two components often reveals structure. Principal points of a p-variate random vector X are points that approximate the distribution of X optimally in the least squares sense; if the distribution of X is elliptical, then principal points lie in the subspace spanned by the first eigenvectors of the covariance matrix of X, thus justifying the use of principal components [36,37]. The principal components of regressor variables may be used in multiple regression to combat the effects of collinearity∗ ; see PRINCIPAL COMPONENTS REGRESSION ANALYSIS. By omitting components with small eigenvalues the stability of parameter estimates is improved while a bias may be introduced, similarly to ridge regression∗ . Nonlinear functions of the principal components, including their products and powers, may be used to improve prediction [29]. PATTERNED COVARIANCE MATRICES Suppose a pk-variate random vector X is partitioned into k vectors X1 , . . . , Xk of dimension p each; for instance, Xi may be a set of p variables measured in the ith stadium of growth of an insect. The pk × pk covariance matrix of X, partitioned into submatrices ij of dimension p × p each, may then display some distinct structure, and models similar to those of the first section can be defined [12]. For instance, one may assume that the same orthogonal p × p matrix B diagonalizes all ij simultaneously; this leads to the CPC model for dependent random vectors [25] used in studies of static and ontogenetic allometry∗ . For example, for k = 2 the model is 11 12 B O 11 12 B O = , O B 21 22 21 22 O B where B is an orthogonal p × p matrix, and all ij are diagonal. If this model holds, then the canonical correlations between X1 and X2 are the same as the correlations between the common principal components [12]. Assuming that B is nonsingular but not necessarily orthogonal leads to the common canonical variates model [28]. See also the references on three-way principal component analysis given at the end of the first section.
PRINCIPAL COMPONENTS
VARIA Robust estimation∗ of principal components is discussed in refs. 5, 20; for common principal components, in ref. [13]. References [6, 43] use projection pursuit∗ to define robust principal components, and to detect clusters and outliers∗ . The influence function∗ [4,20] provides a tool to assess the effect of single observations on estimates of eigenvalues and eigenvectors. See ref. [4] for a comprehensive review. Various tests for hypotheses are developed in [32,33,40,41]. The T 2 statistic for equality of the mean vectors of two p-variate populations can be decomposed into p parts corresponding to principal components [35]. This decomposition can be generalized by assuming that the two populations differ in location and in variability, but only in q < p common principal component directions [11]. The same approach allows us to test the model of allometric extension. The biometric, chemometric, and psychometric literature abounds with applications of principal component analysis and its various extensions, for example, to process monitoring in paper production [42], water chemistry [18], soil contamination [43], and regional variation in growth of insects [24]. Future directions for research should include models for high-dimensional data, in particular data measured in continuous time, and Bayesian estimation. On the latter topic there is only a very limited amount of literature available [15,38], and none related to the various generalizations presented in this update. REFERENCES 1. Airoldi, J. P. and Flury, B. (1988). An application of common principal component analysis to cranial morphometry of Microtus californicus and M. ochrogaster (Mammalia, Rodentia). J. Zool. (London), 216, 21–36. 2. Banfield, J. D. and Raftery, A. E. (1992). Ice floe identification in satellite images using mathematical morphology and clustering about principal curves. J. Amer. Statist. Ass., 87, 7–15. 3. Bensmail, H. and Celeux, G. (1996). Regularized Gaussian discriminant analysis through eigenvalue decomposition. J. Amer. Statist. Ass., 91, 1743–1748.
6403
4. Brooks, S. P. (1994). Diagnostics for principal components: influence functions as diagnostic tools. Statistician, 43, 483–494. 5. Campbell, N. A. (1980). Robust procedures in multivariate analysis I: robust covariance estimation. Appl. Statist., 29, 231–237. 6. Caussinus, H. and Ruiz-Gazen, A. (1993). Projection pursuit and generalized principal component analysis. In New Directions in Statistical Data Analysis and Robustness, S. Morgenthaler, E. Ronchetti, and W. A. Stahel, eds. ¨ Birkhauser, Basel, pp. 35–46. 7. Donnell, D. J., Buja, A., and Stuetzle, W. (1994). Analysis of additive dependencies and concurvities using smallest additive principal components. Ann. Statist., 22, 1635–1673. 8. Eriksen, P. S. (1987). Proportionality of covariance matrices. Ann. Statist., 15, 732–748. 9. Flury, B. (1988). Common Principal Components and Related Multivariate Models. Wiley, New York. (Model-based introduction to principal component analysis, with extensions to common principal components, partial common principal components, and proportional covariance matrices, including distribution theory and numerical algorithms.) 10. Flury, B. and Gautschi, W. (1986). An algorithm for simultaneous orthogonal transformation of several positive definite symmetric matrices to nearly diagonal form. SIAM J. Sci. Statist. Comput., 7, 169–184. 11. Flury, B., Nel, D., and Pienaar, I. (1995). Simultaneous detection of shift in means and variances. J. Amer. Statist. Ass., 90, 1474–1481. 12. Flury, B. and Neuenschwander, B. (1994). Principal component models for patterned covariance matrices, with applications to canonical correlation analysis of several sets of variables. In Descriptive Multivariate Analysis, W. J. Krzanowski, ed. Oxford University Press, pp. 90–112. 13. Flury, B. and Novi Inverardi, P. L. (1992). Robust estimation of common principal components. Quad. Statist. e Mat. Appl. Sci. Econ.–Soc., 14, 49–80. 14. Flury, B., Schmid, M., and Narayanan, A. (1994). Error rates in quadratic discrimination with constraints on the covariance matrices. J. Classification, 11, 101–120. 15. Geisser, S. (1965). Bayesian estimation in multivariate analysis. Ann. Math. Statist., 36, 150–159. 16. Hastie, T. J. and Stuetzle, W. (1989). Principal curves. J. Amer. Statist. Ass., 84, 502–516.
6404
PRINCIPAL COMPONENTS
17. Henrion, R. (1993). Body diagonalization of core matrices in three-way principal component analysis: theoretical bounds and simulation. J. Chemometrics, 7, 477–494. 18. Henrion, R. (1994). N-way principal component analysis: theory, algorithms, and applications. Chemometrics and Intell. Lab. Systems, 25, 1–23. 19. Jackson, J. E. (1991). A User’s Guide to Principal Components. Wiley, New York. (An easyto-read introduction to principal component analysis and its applications, including many real and synthetic data sets, and connections to other multivariate methods.) 20. Jaupi, L. and Saporta, G. (1993). Using the influence function in robust principal component analysis. In New Directions in Statistical Data Analysis and Robustness, S. Morgenthaler, E. Ronchetti, and W. A. Stahel, eds. ¨ Birkhauser, Basel, pp. 147–156. 21. Jensen, S. T. and Johansen, S. (1987). Estimation of proportional covariances. Statist. Probab. Lett., 6, 83–85. 22. Jolicoeur, P. C. (1963). The multivariate generalization of the allometry equation. Biometrics, 19, 497–499. 23. Jolliffe, I. T. (1986). Principal Component Analysis. Springer, New York. (A very readable introduction to principal-component analysis with many illustrative examples, including applications to factor analysis and regression.) 24. Klingenberg, C. P. (1996). Multivariate allometry. In Advances in Morphometrics, L. F. Marcus et al., eds. Plenum, New York, pp. 23–49. 25. Klingenberg, C. P., Neuenschwander, B., and Flury, B. (1996). Ontogeny and individual variation: analysis of patterned covariance matrices with common principal components. Systematic Biol., 45, 135–150. 26. Kroonenberg, P. M. (1983). Three-Mode Principal Component Analysis. DSWO Press, Leiden, The Netherlands. (A comprehensive review of the psychometric literature on extensions of principal component analysis to threeway data up to 1983.) 27. Krzanowski, W. J. (1979). Between-groups comparison of principal components. J. Amer. Statist. Ass., 74, 703–707. Correction, 76, 1022. 28. Neuenschwander, B. and Flury, B. (1995). Common canonical variates. Biometrika, 82, 553–560. 29. Oman, S. D., Næs, T., and Zube, A. (1993). Detecting and adjusting for non-linearities
in calibration of near-infrared data using principal components. J. Chemometrics, 7, 195–212. 30. Owen, A. (1984). A neighbourhood-based classifier for LANDSAT data. Can. J. Statist., 12, 191–200. 31. Schott, J. R. (1988). Common principal component subspaces in two groups. Biometrika, 75, 229–236. 32. Schott, J. R. (1988). Testing the equality of the smallest latent roots of a correlation matrix. Biometrika, 75, 794–796. 33. Schott, J. R. (1991). A test for a specific principal component of a correlation matrix. J. Amer. Statist. Ass., 86, 747–751. 34. Schott, J. R. (1991). Some tests for common principal component subspaces in several groups. Biometrika, 78, 771–777. 35. Takemura, A. (1985). A principal decomposition of Hotelling’s T 2 statistic. In Multivariate Analysis—VI, P. R. Krishnaiah, ed. Elsevier, Amsterdam, pp. 583–597. 36. Tarpey, T. (1995). Principal points and selfconsistent points of symmetric multivariate distributions. J. Multivariate Anal., 53, 39–51. 37. Tarpey, T., Li, L., and Flury, B. (1995). Principal points and self-consistent points of elliptical distributions. Ann. Statist., 23, 103–112. 38. Tiao, G. C. and Fienberg, S. (1969). Bayesian estimation of latent roots and vectors with special reference to the bivariate normal distribution. Biometrika, 56, 97–108. 39. Tibshirani, R. (1992). Principal curves revisited. Statist. and Comput., 2, 183–190. 40. Tyler, D. E. (1981). Asymptotic inference for eigenvectors. Ann. Statist., 9, 725–736. 41. Tyler, D. E. (1983). A class of asymptotic tests for principal component vectors. Ann. Statist., 11, 1243–1250. 42. Wold, S. (1994). Exponentially weighted moving principal components analysis and projections to latent structures. Chemometrics and Intell. Lab. Systems, 23, 149–161. 43. Xie, Y., Wang, J., Liang, Y., Sun, L., Song, X., and Yu, R. (1993). Robust principal component analysis by projection pursuit. J. Chemometrics, 7, 527–541. See also ALLOMETRY; CLASSIFICATION —I; COMPONENT ANALYSIS; DISCRIMINANT ANALYSIS; PRINCIPAL COMPONENTS REGRESSION ANALYSIS; and THREE-MODE ANALYSIS.
BERNARD D. FLURY
PRINCIPAL COMPONENTS REGRESSION ANALYSIS
r = 1, 2, . . . , p, where a r = (a1r , a2r , . . . , apr ) with constraints a r as = δrs , where δrs = 1 if r = s and is zero otherwise.
PRINCIPAL COMPONENTS REGRESSION ANALYSIS Principal components (PC) regression is often used to solve the problem of multicollinearity∗ [3–5, 8], especially when deleting independent variables to break up the multicollinearity is not a feasible option [3]. One also uses PC regression when there are more independent variables than observations [5]. Basically, the way that PC regression works is that one first finds the set of orthonormal eigenvectors of the correlation matrix of the independent variables. The matrix of PCs is then the product of the eigenmatrix with the matrix of independent variables. The matrix of PCs is such that the first PC will have the maximum possible variance, the second the maximum possible variance among those uncorrelated with the first, and so on. The set of PCs then becomes the set of new regressor variables. Upon completion of this regression, one transforms back to the original coordinate system. Let the ordinary least squares∗ (OLS) model be defined (the vector and matrix dimensions are given in parentheses) as Y =
(n×1)
+
γ0 1 (1×1)(n×1)
Xγ (n×p)(p×1)
+ . (n×1)
i
(Xij − X j )2
λ1 λ2 · · · λp 0.
(5)
Then, if A = (a1 , . . . , ap ), A x xA = Z Z =
and
x x = AA , (6)
since A is orthogonal, where Z = xA is the PC matrix and Z Z is the new design matrix. Then r = 1, . . . , p, (7) var(zr ) = z r zr = λr , tr(x x) = p = z r zr = λr implies that λi /p represents the proportionate contribution of the ith PC to the total variation of the x’s. McCallum [8] has noted that y = xβ + = ZA β + ,
(8)
which implies that h = A β
1/2
√ = (Xij − X j )/(sj n − 1)
1/2 (Yi − Y)2 yi = (Yi − Y)
= diag[λ1 , λ2 , . . . , λp ] such that
(1)
To compute the correlation matrix∗ , it is useful to standardize the variables: xij = (Xij − X j )
6405
(2)
or
β = hA,
(9)
where h is the coefficient vector of the regression on the PCs. Hence the transformed coefficients for model (1) are obtained from (9) and (3) as γi =
p
√ aij hj /(si n − 1),
i = 1, . . . , p, (10)
j=1
i
√ = (Yi − Y)/(sy n − 1).
where i j
Then (1) becomes y = xβ + ,
√ βj = γj sj n − 1.
γi (3)
The correlation matrix x x has eigenvalues and eigenvectors defined by |x x − λr I| = 0
and
(x x − λr I)ar = 0, (4)
aij hj
si
corresponds to the independent variable number, corresponds to the PC number is the transformed coefficient, model (1) are elements of the eigenmatrix are the coefficients obtained from the regression on the PCs is the standard deviation of independent variable i
6406
PRINCIPAL COMPONENTS REGRESSION ANALYSIS
(In ridge regression∗ , h is referred to as the vector of canonical coefficients.) If a subset of the PCs is selected, some of the hj are zero. The constant term in the transformed equation can be obtained from γ0 = Y −
p
γi X i .
(11)
i=1
McCallum [8] has also observed that var b = σ 2 (x x)−1 = σ 2 (A−1 A ),
(12)
where b is the least squares estimator of β. Therefore, the standard errors of the estimators of the coefficients for model (1) (but not the constant term) are, from (12) and (3), SE(γi ) =
σ
p a2ij
1/2
√ si n − 1 j=1 λj
,
i = 1, . . . , p.
(13)
When a subset of the PCs is selected, the appropriate aij and λj are deleted from (13). One does not need to be concerned with the interpretation of the PCs or the regression on the PCs, because this is just an intermediate step and one can transform back to the original coordinate system (as shown above). Three selection rules for picking PCs are [2,3,8]: 1. Keeping all p PCs yields zero bias [because this yields OLS, as shown in (8)]. 2. Keeping those PCs with the largest eigenvalues minimizes the variances of the estimators of the coefficients. (The proof is given in Greenberg [3] and Coxe [2, p. 226]. One simply needs to substitute A for Z and Z for W.) 3. Keeping those PCs which are highly correlated with the dependent variable tends to minimize the mean square errors of the estimators of the coefficients. (Coxe [2] has given an intuitive argument why this rule holds, but it has yet to be proven.)
The reasons given are: (a) McCallum [8] has shown that one can obtain smaller MSEs by eliminating some of the PCs. (b) Since rule 1 yields zero bias and rule 2 yields minimum coefficient variances, minimum MSE should fall somewhere in between. (c) Selecting those PCs that are strongly correlated with the dependent variable is similar to selecting independent variables by stepwise regression∗ in OLS; that is, both use F-values to determine which terms to retain. It should be noted that selection rules 2 and 3 do not necessarily yield the same PCs model, and hence the resulting coefficients in model 1 can be quite different, especially when the degree of multicollinearity is severe. There is no definitive source for PC regression analysis. In going from paper to paper, one must be alert to the different standardizations of the variables. Some √ use sj in the denominator, others use sj n − 1, and still others do not standardize at all. REFERENCES 1. Anderson, T. W. (1958). An Introduction to Multivariate Statistics. Wiley, New York, pp. 272–287. (A rigorous derivation of PC.) 2. Coxe, K. (1982). Amer. Statist. Ass., Proc. Bus. & Econ. Statist. Sec., pp. 222–227. (Selection rules for PC regression. Comparison with latent root regression. Long on math symbols, short on words. Good list of references.) 3. Greenberg, E. (1975). J. Amer. Statist. Ass., 70, 194–197. [Deals with selection rule (2). Good, fairly technical paper.] 4. Gunst, R. F. and Mason, R. L. (1980). Regression Analysis and Its Application. Marcel Dekker, New York, pp. 317–328. (One of the better treatments of the subject in a textbook.) 5. Johnston, J. (1972). Econometric Methods, 2nd ed. McGraw-Hill, New York, pp. 322–331. (A good, simple derivation of PC regression.) 6. Karson, M. J. (1982). Multivariate Statistical Methods. Iowa State University Press, Ames, IA, pp. 191–224. (Many numerical examples. Excellent list of references.) 7. Massy, W. F. (1965). J. Amer. Statist. Ass., 60, 234–256. (Derivation of PC regression. Fairly technical paper and one of the first.) 8. McCallum, B. T. (1970). Rev. Econ. Statist., 52, 110–113. (Good paper. Easy to read.) 9. Morrison, D. F. (1967). Multivariate Statistical Methods. McGraw-Hill, New York, pp.
PRINCIPAL DIFFERENTIAL ANALYSIS 221–258. (A rather simple approach. Long list of references.) 10. Pidot, G. B., Jr. (1969). Rev. Econ. Statist., 51, 176–188. (Suggests using a combination of the X’s and the PC’s.) See also COMPONENT ANALYSIS; DISCRIMINANT ANALYSIS; LATENT ROOT REGRESSION; MULTICOLLINEARITY; and MULTIVARIATE ANALYSIS.
KEREN L. COXE
PRINCIPAL COMPONENTS, ¨ SCHONEMANN’S THEOREM FOR The following theorem by Sch¨onemann [7] dealing with principal components* has been used by various authors [1,4,5] in connection with the so-called Spearman’s hypothesis (see, e.g., ref. 3) of the positive relationship between the black–white differences in means on cognitive tests and the loading of these tests on the general common factor identified as ‘‘general cognitive ability’’. Sch¨onemann [6,7] asserts that this relationship is ascribable to a statistical artifact and is simply just a mathematical necessity. The statement of the theorem is as follows: If the range Rp —a p-dimensional Euclidean space—of a p-variate normal vector y ∼ N(0, ) is partitioned into ahigh set (H) and a low set (L) by the plane k yk = 0 and both within-covariance matrices H and L remain positive-definite, then (i) the mean difference vector d = E(y|H) − E(y|L) is collinear with the largest principal components (pc’s) of , and (ii) d is also collinear with the pc1’s (eigenvectors corresponding to the largest eigenvalues) of L and H . In other words, if the test correlations are positive in the parent population, they remain so in both selected subpopulations. Parts (i) and (ii) of the theorem do not generally hold for the samples created by selection on the basis of a variable k yk [2]. (It may be noteworthy that Americans are not usually sorted into categories of ‘‘white’’ and ‘‘black’’ by an observed selection variable that is a sum of test scores. Moreover, principal components analysis is not well suited to assess cognitive abilities, since it does not distinguish between factors common to the tests under consideration and factors specific to the tests.)
6407
REFERENCES 1. Braden, J. P. (1989). Fact or artifact. An empirical test of Spearman’s hypothesis. Intelligence, 13, 149–155. 2. Dolan, C. V. (1997). A note on Sch¨onemann’s refutation of Spearman’s hypothesis. Multivariate Behav. Res., 32, 319–328. 3. Jensen, A. R. (1987). Further evidence for Spearman’s hypothesis concerning black-white differences on psychometric tests. Brain Behav. Sci., 10, 512–519. 4. Jensen, A. R. (1992). Spearman’s hypothesis: methodology and evidence. Multivariate Behav. Res., 27, 225–233. 5. Loehlin, J. C. (1992). On Sch¨onemann, on Guttman, on Jensen, via Lewontin. Multivariate Behav. Res., 27, 261–263. 6. Sch¨onemann, P. H. (1989). Some new results on the Spearman hypothesis artifact. Bull. Psychon. Soc., 27, 462–464. 7. Sch¨onemann, P. H. (1992). Extension of Guttman’s results from g to PC1. Multivariate Behav. Res., 27, 219–223. See also COMPONENT ANALYSIS; PRINCIPAL COMPONENTS; and PSYCHOLOGICAL TESTING THEORY.
PRINCIPAL DIFFERENTIAL ANALYSIS Principal differential analysis (PDA) is used in the analysis of functional data to estimate a linear differential equation (linear DIFE) that describes the observed functions. The main References are 4, 8, and 9, and the entry FUNCTIONAL DATA ANALYSIS can be consulted for a discussion of functional data.
DIFFERENTIAL EQUATION MODELS It is often natural to describe a function of time or space in terms of a relationship between the function’s rate of change and its value. More generally, a relation may be specified among derivatives of various orders, the function itself, and other functions. Differential equations are powerful tools for modeling functional data that offer capabilities difficult to achieve with conventional parametric and nonparametric approaches. Because a DIFE involves rates of change, it is often called a
6408
PRINCIPAL DIFFERENTIAL ANALYSIS
dynamic equation, and is said to offer a model for the dynamics of a process rather than just its static behavior. In this article, lower case Greek and Roman letters such as x and β indicate functions of a single argument t unless otherwise indicated, instead of the usual notation β(t) and x(t). This simplifies expressions considerably. Also, we use Dx to indicate the first derivative of function x, Dm x for the derivative of order m, and specifically, D0 x = x. The article will only consider functions of a single argument, but principal differential analysis can also be applied to functions of multiple variables. There are many famous examples of differential equation models. Newton’s Second Law is D2 x = βf , where x is the position of a moving body through time, f is a force applied to the body, and the constant β is the reciprocal of the body’s mass. The differential equation for harmonic motion is D2 x = −ω2 x, where spring constant ω2 determines the motion’s period. Exponential growth, decay, and approach to an asymptote are all described by D2 x = βDx when β is a constant, and if β is a function, this simple equation describes any smooth strictly monotone curve [5]. See reference 6 for other examples of differential equations that are useful in statistics. A linear differential equation of order m has the form Dm x = β0 x + β1 Dx + · · · + βm−1 Dm−1 x + f , (1) and the equation is said to be homogeneous or nonhomogeneous depending on whether the function f is zero or not respectively. The examples given above are all linear DIFEs because of their linearity in the coefficients βj , j = 0, . . . , m − 1. The forcing function f is often considered to represent external or exogenous input to a system whose closed state is described by the homogeneous equation. Linear DIFEs share with other types of linear equations a mathematical structure that is relatively easy to describe and techniques for solving them that are fairly routine. Moreover, a linear differential equation has a solution for any value of t, a feature that
does not always hold for nonlinear DIFEs. This article will focus on linear DIFEs, but nonlinear DIFEs are playing an increasing role in statistics because of the much richer range of functional behaviors that they can define (see Ref. 1 for a readable account). You are encouraged to regard (1) as an equation defining a multiple regression model. The dependent variable is Dm x and the independent variables or covariates are Dj x. The forcing function f can, if not observed directly, be viewed as the error or residual term. The main distinction between a conventional linear model and a linear DIFE is that each variable Dj x and each regression coefficient βj is a function of t. That is, Equation 1 is a t-indexed multiple regression model. In fact, if a sample of functions xi , i = 1, . . . , N are available, then performing a multiple regression for each of a fine mesh of values of t is a reasonable strategy for estimating the coefficient functions βj . It is not surprising, therefore, that the theory associated with linear DIFEs uses the same matrix theory that underlies regular regression (see Ref. 2). Describing functional observations by differential equations brings several useful features that are not provided by conventional approaches. By conventional we mean parametric models; expansions in terms of Fourier series, B-splines, and other basis systems; or other methods that smooth the data. For example, exponential behavior in a function x can either be expressed by the equation x(t) = c1 + c2 eβt where c1 and c2 are arbitrary constants, or more compactly as the DIFE D2 x = βDx. First of all, a DIFE simultaneously defines the behavior of the m derivatives Dj x and the behavior of the function itself, while conventional methods focus all the attention on estimating x. In applications where rates of change are of theoretical interest, as is the case in mechanics, a DIFE requires that derivatives also be well estimated because of their explicit role in the model. Secondly, a single differential equation model can simultaneously fit a whole sample of distinct functions xi . Homogeneous linear DIFEs of order m have an m-dimensional space of solutions, like the exponential two-dimensional exponential equation given
PRINCIPAL DIFFERENTIAL ANALYSIS
above. That is, any solution x can berepresented as a linear combination x = m j cj uj of m linearly independent specific solutions uj , where the coefficients cj are arbitrary constants. The homogeneous case for Newton’s Law, D2 x = 0, has as its solution x(t) = c1 + c2 t. Solutions of the DIFE D3 x = −ω2 Dx are all linear combinations of 1, sin ωt, and cos ωt. Moreover, nonhomogeneous equations have the same solution space translated by a functional of the forcing function f . Finally, as was noted above, specific constraints on x can often be expressed by a specialized DIFE. For example, Dm x = βDm−1 x constrains x to entirely positive/negative, strictly monotonic, or strictly convex/concave depending on whether m is 1, 2, or 3 respectively. The solutions to the nonlinear DIFE Dx = βx(1 − x) are the set of probability functions taking values in (0,1).
ESTIMATING A LINEAR DIFFERENTIAL OPERATOR L The basic idea behind PDA is to estimate a linear differential equation of order m from one or more observed functions xi . The mdimensional solution space of the estimated DIFE is designed to contain most of the variation in these functions. PDA has much in common with principal components analysis (PCA) since, if the DIFE is successful, a set of m linearly independent functions uj will be identified that can be combined linearly to reproduce well the original functional data. But unlike PCA, PDA aims to give a good account of each of m − 1 derivatives at the same time. In PDA, we seek a linear differential operator L of the form Lx =
m−1
βj Dj x + Dm x.
(2)
Now, viewing the forcing function f as an unknown residual term, we can ask whether it is possible to use the data to estimate an operator L such that the least squares fitting criterion N
(Lxi )2 (t)dt =
i=1
N
fi2 (t)dt
(3)
i=1
is minimized. That is, we are aiming to get as close as possible to a homogeneous model Lx = 0 that has solutions that well describe the data. Similarly, in PCA, we essentially define the analysis as a least squares problem in which a linear projection operator Q is defined that minimizes the variation in the residual vectors Qxi . The projection of the observations into the m-dimensional space spanned by the principal components is defined by m eigenvalues. But the linear differential operator L is also defined by the m coefficient functions βj , j = 0, . . . , m − 1; and the essential problem in PDA is also to estimate these coefficient functions by minimizing a least squares criterion. A number of extensions of PDA are possible. We can augment Lx by adding to it terms αk zk where each zk , k = 1, . . . , K, is a known function, and each coefficient function αk also needs to be estimated. If we do so, we do not change the nature of the problem in any qualitative way; the result is still an operator that is linear in the unknown coefficient functions βj and αk . We can also work with systems of linear differential equations, where we have an equation for each of a set of variables, x , = 1, . . . , L. The actual estimation of the weight coefficients βj and αk requires the use of a basis function expansion. That is, when we have a set of K basis functions φk , which are often terms in a Fourier series, or a set of B-splines, then
j=0
The operator L defines the homogeneous part of Equation 1, and we can think of it as a generalization of the notion of the specific derivative Dm . Using this definition, we can express the DIFE model (1) more compactly as Lx = f .
6409
βj =
K
cjk φk = c j φ.
(4)
k=1
The fitting criterion (3) is then quadratic in the coefficients of the expansion, and its minimization merely requires the solution of a set of linear equation. PDA also works fine
6410
PRINCIPAL DIFFERENTIAL ANALYSIS
if there is only a single observed function x provided that the numbers of basis functions in the weight coefficient expansions are small relative to that of x. Finally, we can easily incorporate both specific constraints on certain βj ’s, such as being zero, and can also apply roughness penalties to them that ensure that they are as smooth in some sense as is required. The functions xi being analyzed are usually the result of some smoothing process applied to a set of discrete observed values. It is essential that the smoothing procedure gives a reasonable estimate of the derivatives up to order m at the same time, and consequently PDA is most effective for functional observations where the observational error is fairly small relative to the size of the variation in the function that the smoothing technique is trying to estimate. Methods are under development, however, that will permit using the discrete noisy data directly rather than passing through a smoothing stage. These are based on the work of Reference 3. Because a linear DIFE is essentially a tindexed regression equation, the inferential and diagnostic tools of regression analysis can be adapted to the results of a PDA. We can, for example, examine squared multiple correlation and F-ratio functions R2 and F 0.30
D3 x = βxx1 Dx + βxx2 D2 x + βxy1 Dy + βxy2 D2 y + βxz1 Dz + βxz2 D2 z D3 y = βyx1 Dx + βyx2 D2 x + βyy1 Dy + βyy2 D2 y + βyz1 Dz + βyz2 D2 z D3 z = βzx1 Dx + βzx2 D2 x + βzy1 Dy + βzy2 D2 y + βzz1 Dz + βzz2 D2 z.
(5)
This is a second-order linear system where pen velocities Dx, Dy, and Dz are the
•• •• • •
••••••••••••••• •••• •••• •••• ••• • •• ••••••• ••• ••• ••• ••••••••••••••• ••• • • •• •••••••••••••• • •• •••••••••••••••• ••• •••••••• • • • • • •• • • •• • • ••• • • ••• •• •• • • • •• • •• • •• ••• •• •••••• • • •• • ••••• •• •••••••••••••• • • • • • • • • • •• ••• •• • • • • • • • • • • • • ••• •• • • ••• • ••••••• • • • ••••• • ••••••••• • • • • • • • ••••• •• • ••••••••••• ••• • • • ••• • • • • •• • • • • • • • • • • •• ••• • • • • •••••••••• •• • ••• • • •• • • • •••••••• • • •• •• • • •• •• • •• ••• • •••• ••••••••••••••• • • • • • •• • • • • •• • • ••• ••• • •••••• • ••••• • •• •• • • • • ••••••• •• • •••••• •••• • • • • • •• • • • • • • • • • • • • • • • •• •• •• ••• • •• •• •••••• • • • • •••• • • • ••• ••• • •• ••• •••• •• ••• • ••••
0.25
Y (m)
that are themselves functions of time so that we can identify where the linear DIFE fits the data well and where it does not. As in regression analysis, an inspection of the residual functions fi can offer many useful insights. Examples of PDA can be found in references 8 and 9. Figure 1 is taken from reference 7 reporting an analysis of 50 replications of writing ‘‘statistical science’’ in Mandarin. The dynamics of handwriting involve sequences of high-energy strokes occurring about every 120 milliseconds with levels of acceleration that would send a rocket into orbit in a few minutes. In this application, we have as functional observations the three coordinate functions xi , yi , and zi , i = 1, . . . , 50. The differential equation model was the system
0.20
0.15
0.10 −0.2
−0.1
0.0 X (m)
0.1
0.2
Figure 1. A sample of Mandarin script for the phrase ‘‘statistical science.’’ The dots are pen positions at 200-ms intervals, and the solid line is the reconstruction of the script using the second-order DIFE [5].
PRINCIPAL POINTS
functional observations. It is remarkable that such a simple equation can reconstruct the static trace of the original script so well as well as describing accurately its first and second time-derivative. Acknowledgments The author’s work was prepared under a grant from the Natural Science and Engineering Research Council of Canada.
REFERENCES 1. Alligood, K. T., Sauer, T. D., and Yorke, J. A. (1996). Chaos: An Introduction to Dynamical Systems. Springer, New York. 2. Coddington, E. A. and Carlson, R. (1997). Linear Ordinary Differential Equations. SIAM, Philadelphia, Pa. 3. Heckman, N. and Ramsay, J. O. (2000). Penalized regression with model-based penalties. Can. J. Stat., 28, 241–258. 4. Ramsay, J. O. (1995). Principal differential analysis: data reduction by differential operators. J. R. Stat. Soc., Ser. B, 58, 495– 508. 5. Ramsay, J. O. (1998). Estimating smooth monotone functions. J. R. Stat. Soc., Ser. B, 60, 365–375. 6. Ramsay, J. O. (2000). Differential equation models for statistical functions. Can. J. Stat., 28, 225–240. 7. Ramsay, J. O. (2000). Functional components of variation in handwriting. J. Am. Stat. Assoc., 95, 9–15. 8. Ramsay, J. O. and Silverman, B. W. (1997). Functional Data Analysis. Springer, New York. 9. Ramsay, J. O. and Silverman, B. W. (2002). Applied Functional Data Analysis. Springer, New York.
JAMES O. RAMSAY
PRINCIPAL MOMENTS An obsolete term for central moments∗ (moments about the mean).
PRINCIPAL POINTS A set of k principal points of a distribution is the best k-point approximation of the distribution in terms of a quadratic loss. Principal
6411
points can be motivated by the following practical problem. Suppose a clothing manufacturer wishes to produce a new type of swimsuit in four different sizes. Some relevant measurements such as stature, bust size, etc., are taken on a large number of people. The statistician is asked to identify four ‘‘typical’’ people whose dimensions will be used to model the swimsuits. The term principal points was defined in a similar problem of determining sizes and shapes of new gas masks [9]. Another problem to which principal-point theory has been applied is determining a few representative curves from a large collection of curves [11,15].
DEFINITION AND PROPERTIES OF PRINCIPAL POINTS A set of k points ξ 1 , . . . , ξ k are called k principal points of a random vector X if E[minj=1,...,k X − ξ j 2 ] E[minj=1,...,k X − yj 2 ] for any set of k points {y1 , . . . , yk } ⊂ Rp . Here · denotes the Euclidean norm. Thus principal points provide a least squares∗ approximation by projecting the values of a random vector on a finite number of points, similar to Pearson’s definition of principal components∗ [22]. Principal points are a theoretical analogue to k-cluster means [12] (k-MEANS ALGORITHMS), and have also been called k-centers [21]. For k = 1, the single principal point is the mean of the distribution. The k 2 principal points of a distribution are a generalization of the mean from one to several points. Usually numerical methods are required to determine the principal points except in some simple cases. For instance, for the N (µ, σ 2 ) distribution, the two principal points are µ ± σ 2/π [2,9]. For any k 1, the k principal points of the uniform distribution in [a, b] are a + (2j − 1)(b − a)/2k, j = 1, . . . , k [31]. Determining the principal points of a distribution is aided by studying self-consistent points, particularly for multivariate distributions. For k points y1 , . . . , yk in Rp , let Sj ⊂ Rp denote the set of all points in Rp closer to yj than to all other yi ’s. If E[X|X ∈ Sj ] = yj for j = 1, . . . , k, then the yj are called k self-consistent points of X [10]. Self-consistent points have also been called ‘‘unbiased’’ [19].
6412
PRINCIPAL POINTS
Principal points are self-consistent [10]. Multivariate distributions have typically several sets of k self-consistent points but only one set of k principal points. For example, consider the bivariate normal distribution∗ centered at the origin with covariance matrix diag (σ 2 , 1), where σ > 1. Then there exist two sets of two self-consistent points: {(0, 2/π ) , (0, − 2/π ) } and {(σ 2/π , 0) , (−σ 2/π , 0) }. The second set corresponds to two principal points [9]. An important property of self-consistent points (and hence principal points) is the following: if y1 , . . . , yk are k self-consistent points of X, then E[X] lies in the convex hull of the yj [28]. See also SELF-CONSISTENCY.
UNIVARIATE RESULTS Principal points for the standard normal distribution have appeared in numerous publications, e.g., refs. 2, 9, 18, and for k up to 31 in ref. [6]. Computational methods for finding principal points of univariate continuous distributions are given in refs. 18, 25, 30. Symmetric univariate distributions always have a set of k symmetric selfconsistent points [31], but the k principal points may not be symmetric to the mean even if the distribution is continuous and unimodal [9]. However, if the distribution has a log-concave density, then there exists a unique set of k self-consistent points which must be the k principal points of the distribution [8,16,26,29]. Under rather general conditions on the density f of a random variable X, asymptotically as k → ∞, the k principal points tend towards the k quantiles of the distribution whose density is proportional to f 1/3 [23].
MULTIVARIATE RESULTS Suppose X follows a p-variate elliptical distribution∗ such as a multivariate normal with mean zero. If k principal points of X span a subspace of dimension q < p, then this subspace is also spanned by q eigenvectors associated with the q largest eigenvalues of the covariance matrix [28]. Similar results relating principal component subspaces to
principal points and self-consistent points hold for a larger class of symmetric multivariate distributions [27]. For the bivariate normal, numerical results indicate that principal points and self-consistent points form highly symmetrical patterns [9,10,17]. For the uniform distribution on the unit square, asymptotically the optimal configuration of points as k → ∞ corresponds to the tessellation by regular hexagons with principal points at their centers ([7,20]; see RANDOM TESSELLATIONS). ESTIMATION OF PRINCIPAL POINTS To apply the theory of principal points to real data, various estimators have been defined, ranging from normal theory maximum likelihood∗ [10] to the nonparametric k-means clustering algorithm∗ [12]. The cluster means from the k-means algorithm are by construction a set of k self-consistent points of the empirical distribution. Under rather general conditions, the cluster means from the kmeans algorithm are consistent estimators of principal points, and are asymptotically normal [1,13,19,24]. Convergence results have also been established for the more general case of distributions on Banach spaces [3,21]. Semiparametric∗ estimators of principal points have been defined under the constraint that they form particular symmetric patterns in the subspace of the first few principal components∗ , and by applying the k-means algorithm to data reduced to the first few principal components [10], as suggested by the theory of principal points of elliptical distributions [28]. EXAMPLE Principal point methodology has been applied in a problem of determining shapes and sizes of new gas masks for the Swiss Army [10]. Due to the considerable variability in human heads, a single mask would not be sufficient to fit all its users. The dimensions of k = 2 to 5 ‘‘typical’’ heads were requested for constructing masks of different sizes and shapes. Six head measurements were taken on a sample of Swiss soldiers. Assuming multiˆ denote the variate normality, let µ ˆ and
PRINCIPAL POINTS
maximum likelihood estimates of the mean vector and the covariance matrix respectively. Then the k principal points of the ˆ distribution are maximum likeliˆ ) N6 (µ, hood estimates of k principal points. Figure 1 shows the normal theory maximum likelihood estimates of k = 4 principal points for the head dimension data. The points are plotted in the subspace spanned by the first two principal components. These estimates were based on the assumption that the four principal points form a ‘‘cross’’ pattern in the subspace spanned by the first two principal components. RELATED TOPICS Since a set of k principal points partition the space into k Voronoi regions or strata according to minimal distance, the mathematical problem of determining principal points is related to the problem of optimum stratification∗ [4] and optimal grouping [2]. For a unified approach to these and several other related problems that share a common mathematical framework, see [5]. Principal points are also related to quantization in signal processing∗ . A common form of quantization is rounding off, which occurs in all digital systems. A quantizer partitions
6413
the sample space of a random variable and maps the random variable to representative points so as to minimize the average distortion. Determining an optimal quantizer is often equivalent to finding a set of representative (principal) points of a distribution [14]. IEEE published a special issue on the topic of quantization (IEEE Transactions on Information Theory, 1982). Several of the articles that appear in that issue are noted in the references below. REFERENCES 1. Bryant, P. (1991). Large-sample results for optimization-based clustering methods. J. Classification, 8, 31–44. 2. Cox, D. (1957). Note on grouping. J. Amer. Statist. Ass., 52, 543–547. 3. Cuesta, J. and Matran, C. (1988). The strong law of large numbers for k-means and best possible nets of Banach valued random variables. Probab. Theory and Related Fields, 78, 523–534. 4. Dalenius, T. (1950). The problem of optimum stratification. Skand. Aktuarietidskrift, 33, 203–213. 5. Eubank, R. (1988). Optimal grouping, spacing, stratification, and piecewise constant approximation. SIAM Rev., 30, 404–420.
Figure 1. Normal theory maximum likelihood estimates of four principal points (denoted •) for the head dimensions data. Scatter plot shows 50 observations in the coordinate system of the first two principal components. Broken lines indicate the boundaries of the Voronoi regions associated with the principal points.
6414
PRINCIPLE OF PARSIMONY
6. Fang, K. and He, S. (1982). The problem of selecting a given number of representative points in a normal population and a generalized Mills’ ratio. Tech. Rep., Department of Statistics, Stanford University. 7. Fejes T´oth, L. (1953). Lagerungen in der Ebene, auf der Kugel und im Raum. SpringerVerlag, Berlin. 8. Fleischer, P. (1964). Sufficient conditions for achieving minimum distortion in a quantizer. IEEE Int. Convention Record Part I, 12, 104–111. 9. Flury, B. (1990). Principal points. Biometrika, 77, 33–41. 10. Flury, B. (1993). Estimation of principal points, Appl. Statist., 42, 139–151. 11. Flury, B. and Tarpey, T. (1993). Representing a large collection of curves: a case for principal points, Amer. Statist., 47, 304–306. 12. Hartigan, J. (1975). Clustering Algorithms. Wiley, New York. 13. Hartigan, J. (1978). Asymptotic distributions for clustering criteria. Ann. Statist., 6, 117–131. 14. Iyengar, S. and Solomon, H. (1983). Selecting representative points in normal populations. In Recent Advances in Statistics: Papers in Honor of Herman Chernoff on his 60th Birthday. Academic Press, New York, pp. 579– 591.
22. Pearson, K. (1901). On lines and planes of closest fit to systems of points in space. Phil. Mag., 2, 559–572. 23. P¨otzelberger, K. and Felsenstein, K. (1994). An asymptotic result on principal points for univariate distributions. Optimization, 28, 397–406. 24. Pollard, D. (1981). Strong consistency of kmeans clustering. Ann. Statist., 9, 135–140. 25. Rowe, S. (1996). An algorithm for computing principal points with respect to a loss function in the unidimensional case, Statist. and Comput., 6, 187–190. 26. Tarpey, T. (1994). Two principal points of symmetric, strongly unimodal distributions. Statist. and Probab. Lett., 20, 253–257. 27. Tarpey, T. (1995). Principal points and selfconsistent points of symmetric multivariate distributions. J. Multivariate Anal., 53, 39–51. 28. Tarpey, T., Li, L., and Flury, B. (1995). Principal points and self-consistent points of elliptical distributions. Ann. Statist., 23, 103–112. 29. Trushkin, A. (1982). Sufficient conditions for uniqueness of locally optimal quantizer for a class of convex error weighting functions. IEEE Trans. Inf. Theory, IT-28, 187–198. 30. Zopp`e, A. (1995). Principal points of univariate continuous distributions, Statist. and Comput., 5, 127–132.
15. Jones, M. and Rice, J. (1992). Displaying the important features of large collections of similar curves. Amer. Statist., 46, 140–145.
31. Zopp`e, A. (1997). On uniqueness and symmetry of self-consistent points of univariate continuous distributions, J. Classification, 14, 147–158.
16. Kieffer, J. (1982). Exponential rate of convergence for Lloyd’s method I. IEEE Trans. Inf. Theory, IT-28, 205–210.
See also COMPONENT ANALYSIS; k-MEANS ALGORITHMS; SELF-CONSISTENCY; and SIZE AND SHAPE ANALYSIS.
¨ 17. Kipper, S. and Parna, K. (1992). Optimal kcentres for a two-dimensional normal distribution. Acta et Comment. Univ. Tartuensis, 942, 21–27. 18. Lloyd, S. (1982). Least squares quantization in PCM. IEEE Trans. Inf. Theory, IT-28, 129–136. 19. MacQueen, J. (1966). Some methods for classification and analysis of multivariate observations. Proc. Fifth Berkeley Symp. Math. Statist. and Probab. University of California Press, vol. 1, pp. 281–297.
BERNARD D. FLURY THADDEUS TARPEY
PRINCIPLE OF PARSIMONY. See PARSIMONY, PRINCIPLE OF ; and SCIENTIFIC METHOD STATISTICS
AND
PRINCIPLES OF PROFESSIONAL STATISTICAL PRACTICE
20. Newman, D. (1982). The hexagon theorem. IEEE Trans. Inf. Theory, IT-28, 137–139.
PURPOSE AND SCOPE
¨ 21. Parna, K. (1990). On the existence and weak convergence of k-centres in Banach spaces. Acta et Comment. Univ. Tartuensis, 893, 17–28.
1. The statistician’s job. The statistician’s responsibility, whether as a consultant on a part-time basis, or on regular salary is to
PRINCIPLES OF PROFESSIONAL STATISTICAL PRACTICE
find problems that other people could not be expected to perceive. The statistician’s tool is statistical theory (theory of probability). It is the statistician’s use of statistical theory that distinguishes him from other experts. Statistical practice is a collaborative venture between statistician and experts in subject matter. Experts who work with a statistician need to understand the principles of professional statistical practice as much as the statistician needs them. Challenges face statisticians today as never before. The whole world is talking about safety in mechanical and electrical devices (in automobiles, for example); safety in drugs; reliability; due care; pollution; poverty; nutrition; improvement of medical practice; improvement of agricultural practice; improvement in quality of product; breakdown of service; breakdown of equipment; tardy buses, trains and mail; need for greater output in industry and in agriculture; enrichment of jobs. These problems cannot be understood and cannot even be stated, nor can the effect of any alleged solution be evaluated, without the aid of statistical theory and methods. One cannot even define operationally such adjectives as reliable, safe, polluted, unemployed, on time (arrivals), equal (in size), round, random, tired, red, green, or any other adjective, for use in business or in government, except in statistical terms. To have meaning for business or legal purposes, a standard (as of safety, or of performance or capability) must be defined in statistical terms. 2. Professional practice. The main contribution of a statistician in any project or in any organization is to find the important problems, the problems that other people cannot be expected to perceive. An example is the 14 points that top management learned in Japan in 1950. The problems that walk in are important, but are not usually the important problems (4). The statistician also has other obligations to the people with whom he works, examples being to help design studies, to construct an audit by which to discover faults in procedure before it is too late to make alterations, and at the end, to evaluate the statistical reliability of the results.
6415
The statistician, by virtue of experience in studies of various kinds, will offer advice on procedures for experiments and surveys, even to forms that are to be filled out, and certainly in the supervision of the study. He knows by experience the dangers of (1) forms that are not clear, (2) procedures that are not clear, (3) contamination, (4) carelessness, and (5) failure of supervision. Professional practice stems from an expanding body of theory and from principles of application. A professional man aims at recognition and respect for his practice, not for himself alone, but for his colleagues as well. A professional man takes orders, in technical matters, from standards set by his professional colleagues as unseen judges, never from an administrative superior. His usefulness and his profession will suffer impairment if he yields to convenience or to opportunity. A professional man feels an obligation to provide services that his client may never comprehend or appreciate. A professional statistician will not follow methods that are indefensible, merely to please someone, or support inferences based on such methods. He ranks his reputation and profession as more important than convenient assent to interpretations not warranted by statistical theory. Statisticians can be trusted and respected public servants. Their careers as expert witnesses will be shattered if they indicate concern over which side of the case the results seem to favor. ‘‘As a statistician, I couldn’t care less’’ is the right attitude in a legal case or in any other matter. LOGICAL BASIS FOR DIVISION OF RESPONSIBILITIES 3. Some limitations in application of statistical theory. Knowledge of statistical theory is necessary but not sufficient for successful operation. Statistical theory does not provide a road map toward effective use of itself. The purpose of this article is to propose some principles of practice, and to explain their meaning in some of the situations that the statistician encounters. Statisticians have no magic touch by which they may come in at the stage of tabulation
6416
PRINCIPLES OF PROFESSIONAL STATISTICAL PRACTICE
and make something of nothing. Neither will their advice, however wise in the early stages of a study, ensure successful execution and conclusion. Many a study launched on the ways of elegant statistical design, later boggled in execution, ends up with results to which the theory of probability can contribute little. Even though carried off with reasonable conformance to specifications, a study may fail through structural deficiencies in the method of investigation (questionnaire, type of test, technique of interviewing) to provide the information needed. Statisticians may reduce the risk of this kind of failure by pointing out to their clients in the early stages of the study the nature of the contributions that they themselves must put into it. (The word client will denote the expert or group of experts in a substantive field.) The limitations of statistical theory serve as signposts to guide a logical division of responsibilities between statistician and client. We accordingly digress for a brief review of the power and the limitations of statistical theory. We note first that statistical inferences (probabilities, estimates, confidence limits, fiducial limits, etc.), calculated by statistical theory from the results of a study, will relate only to the material, product, people, business establishments, and so on, that the frame was in effect drawn from, and only to the environment of the study, such as the method of investigation, the date, weather, rate of flow and levels of concentration of the components used in tests of a production process, range of voltage, or of other stress specified for the tests (as of electrical equipment). Empirical investigation consists of observations on material of some kind. The material may be people; it may be pigs, insects, physical material, industrial product, or records of transactions. The aim may be enumerative, which leads to the theory for the sampling of finite populations∗ . The aim may be analytic, the study of the causes that make the material what it is, and which will produce other material in the future. A definable lot of material may be divisible into identifiable sampling units. A list of these sampling units, with identification of each unit, constitutes a frame. In some physical
and astronomical investigations, the sampling unit is a unit of time. We need not detour here to describe nests of frames for multistage sampling. The important point is that without a frame there can be neither a complete coverage of a designated lot of material nor a sample of any designated part thereof. Stephan introduced the concept of the frame, but without giving it a name (7). Objective statistical inferences in respect to the frame are the speciality of the statistician. In contrast, generalization to cover material not included in the frame, nor to ranges, conditions, and methods outside the scope of the experiment, however essential for application of the results of a study, are a matter of judgment and depend on knowledge of the subject matter (5). For example, the universe in a study of consumer behavior might be all the female homemakers in a certain region. The frame might therefore be census blocks, tracts or other small districts, and the ultimate sampling unit might be a segment of area containing households. The study itself will, of course, reach only the people that can be found at home in the segments selected for the sample. The client, however, must reach generalizations and take action on the product or system of marketing with respect to all female homemakers, whether they be the kind that are almost always at home and easy to reach, or almost never at home and therefore in part omitted from the study. Moreover, the female homemakers on whom the client must take action belong to the future, next year, and the next. The frame only permits study of the past. For another example, the universe might be the production process that will turn out next week’s product. The frame for study might be part of last week’s product. The universe might be traffic in future years, as in studies needed as a basis for estimating possible losses or gains as a result of a merger. The frame for this study might be records of last year’s shipments. Statistical theory alone could not decide, in a study of traffic that a railway is making, whether it would be important to show the movement of (for example) potatoes, separately from other agricultural products in
PRINCIPLES OF PROFESSIONAL STATISTICAL PRACTICE
the northwestern part of the United States; only he who must use the data can decide. The frame for a comparison of two medical treatments might be patients or other people in Chicago with a specific ailment. A pathologist might, on his own judgment, without further studies, generalize the results to people that have this ailment anywhere in the world. Statistical theory provides no basis for such generalizations. No knowledge of statistical theory, however profound, provides by itself a basis for deciding whether a proposed frame would be satisfactory for a study. For example, statistical theory would not tell us, in a study of consumer research, whether to include in the frame people that live in trailer parks. The statistician places the responsibility for the answer to this question where it belongs, with the client: Are trailer parks in your problem? Would they be in it if this were a complete census? If yes, would the cost of including them be worthwhile? Statistical theory will not of itself originate a substantive problem. Statistical theory cannot generate ideas for a questionnaire or for a test of hardness, nor can it specify what would be acceptable colors of dishes or of a carpet; nor does statistical theory originate ways to teach field workers or inspectors how to do their work properly or what characteristics of workmanship constitute a meaningful measure of performance. This is so in spite of the fact that statistical theory is essential for reliable comparisons of questionnaires and tests. 4. Contributions of statistical theory to subject matter. It is necessary, for statistical reliability of results, that the design of a survey or experiment fit into a theoretical model. Part of the statistician’s job is to find a suitable model that he can manage, once the client has presented his case and explained why statistical information is needed and how the results of a study might be used. Statistical practice goes further than merely to try to find a suitable model (theory). Part of the job is to adjust the physical conditions to meet the model selected. Randomness∗ , for example, in sampling a frame, is not just a word to assume; it must be created by use of random numbers for the selection of sampling units. Recalls to find people at home,
6417
or tracing illegible or missing information back to source documents, are specified so as to approach the prescribed probability of selection. Statistical theory has in this way profoundly influenced the theory of knowledge. It has given form and direction to quantitative studies by molding them into the requirements of the theory of estimation∗ , and other techniques of inference. The aim is, of course, to yield results that have meaning in terms that people can understand. The statistician is often effective in assisting the substantive expert to improve accepted methods of interviewing, testing, coding, and other operations. Tests properly designed will show how alternative test or field procedures really perform in practice, so that rational changes or choices may be possible. It sometimes happens, for example, that a time-honored or committee-honored method of investigation, when put to statistically designed tests, shows alarming inherent variances between instruments, between investigators, or even between days for the same investigator. It may show a trend, or heavy intraclass correlation∗ between units, introduced by the investigators. Once detected, such sources of variation may then be corrected or diminished by new rules, which are of course the responsibility of the substantive expert. Statistical techniques provide a safe supervisory tool to help to reduce variability in the performance of worker and machine. The effectiveness of statistical controls, in all stages of a survey, for improving supervision, to achieve uniformity, to reduce errors, to gain speed, reduce costs, and to improve quality of performance in other ways is well known. A host of references could be given, but two will suffice [2,3]. 5. Statistical theory as a basis for division of responsibilities. We may now, with this background, see where the statistician’s responsibilities lie. In the first place, his specialized knowledge of statistical theory enables him to see which parts of a problem belong to substantive knowledge (sociology, transportation, chemistry of a manufacturing process, law), and which parts are statistical. The responsibility as logician in a study falls to him by
6418
PRINCIPLES OF PROFESSIONAL STATISTICAL PRACTICE
default, and he must accept it. As logician, he will do well to designate, in the planning stages, which decisions will belong to the statistician and which to the substantive expert. This matter of defining specifically the areas of responsibility is a unique problem faced by the statistician. Business managers and lawyers who engage an expert on corporate finance, or an expert on steam power plants, know pretty well what such experts are able to do and have an idea about how to work with them. It is different with statisticians. Many people are confused between the role of theoretical statisticians (those who use theory to guide their practice) and the popular idea that statisticians are skillful in compiling tables about people or trade, or who prophesy the economic outlook for the coming year and which way the stock market will drift. Others may know little about the contributions that statisticians can make to a study or how to work with them. Allocation of responsibilities does not mean impervious compartments in which you do this and I’ll do that. It means that there is a logical basis for allocation of responsibilities and that it is necessary for everyone involved in a study to know in advance what he or she will be accountable for. A clear statement of responsibilities will be a joy to the client’s lawyer in a legal case, especially at the time of cross-examination. It will show the kind of question that the statistician is answerable for, and what belongs to the substantive experts. Statisticians have something to learn about professional practice from law and medicine. 6. Assistance to the client to understand the relationship. The statistician must direct the client’s thoughts toward possible uses of the results of a statistical study of the entire frame without consideration of whether the entire frame will be studied or only a small sample of the sampling units in the frame. Once these matters are cleared, the statistician may outline one or more statistical plans and explain to the client in his own language what they mean in respect to possible choices of frame and environmental conditions, choices of sampling unit, skills, facilities and supervision required. The statistician will urge the client to foresee
possible difficulties with definitions or with the proposed method of investigation. He may illustrate with rough calculations possible levels of precision to be expected from one or more statistical plans that appear to be feasible, along with rudimentary examples of the kinds of tables and inferences that might be forthcoming. These early stages are often the hardest part of the job. The aim in statistical design is to hold accuracy and precision to sensible levels, with an economic balance between all the possible uncertainties that afflict data—built-in deficiencies of definition, errors of response, nonresponse, sampling variation, difficulties in coding, errors in processing, difficulties of interpretation. Professional statistical practice requires experience, maturity, fortitude, and patience, to protect the client against himself or against his duly appointed experts in subject matter who may have in mind needless but costly tabulations in fine classes (fiveyear age groups, small areas, fine mileage brackets, fine gradations in voltage, etc.), or unattainable precision in differences between treatments, beyond the capacity of the skills and facilities available, or beyond the meaning inherent in the definitions. These first steps, which depend heavily on guidance from the statistician, may lead to important modifications of the problem. Advance considerations of cost, and of the limitations of the inferences that appear to be possible, may even lead clients to abandon a study, at least until they feel most confident of the requirements. Protection of a client’s bank account, and deliverance from more serious losses from decisions based on inconclusive or misleading results, or from misinterpretation, is one of the statistician’s greatest services. Joint effort does not imply joint responsibility. Divided responsibility for a decision in a statistical survey is as bad as divided responsibility in any venture—it means that no one is responsible. Although they acquire superficial knowledge of the subject matter, the one thing that statisticians contribute to a problem, and which distinguishes them from other experts, is knowledge and ability in statistical theory.
PRINCIPLES OF PROFESSIONAL STATISTICAL PRACTICE
SUMMARY STATEMENT OF RECIPROCAL OBLIGATIONS AND RESPONSIBILITIES 7. Responsibilities of the client. The client will assume responsibility for those aspects of the problem that are substantive. Specifically, he will stand the ultimate responsibility for: a. The type of statistical information to be obtained. b. The methods of test, examination, questionnaire, or interview, by which to elicit the information from any unit selected from the frame. c. The decision on whether a proposed frame is satisfactory. d. Approval of the probability model proposed by the statistician (statistical procedures, scope, and limitations of the statistical inferences that may be possible from the results). e. The decision on the classes and areas of tabulation (as these depend on the uses that the client intends to make of the data); the approximate level of statistical precision or protection that would be desirable in view of the purpose of the investigation, skills and time available, and costs. The client will make proper arrangements for: f. The actual work of preparing the frame for sampling, such as serializing and identifying sampling units at the various stages. g. The selection of the sample according to procedures that the statistician will prescribe, and the preparation of these units for investigation. h. The actual investigation; the training for this work, and the supervision thereof. i. The rules for coding; the coding itself. j. The processing, tabulations and computations, following procedures of estimation that the sampling plans prescribe. The client or his representative has the obligation to report at once any departure from instructions, to permit the statistician
6419
to make a decision between a fresh start or an unbiased adjustment. The client will keep a record of the actual performance. 8. Responsibilities of the statistician. The statistician owes an obligation to his own practice to forestall disappointment on the part of the client, who if he fails to understand at the start that he must exercise his own responsibilities in the planning stages, and in execution, may not realize in the end the fullest possibility of the investigation, or may discover too late that certain information that he had expected to get out of the study was not built into the design. The statistician’s responsibility may be summarized as follows: a. To formulate the client’s problem in statistical terms (probability model), subject to approval of the client, so that a proposed statistical investigation may be helpful to the purpose. b. To lay out a logical division of responsibilities for the client, and for the statistician, suitable to the investigation proposed. c. To explain to the client the advantages and disadvantages of various frames and possible choices of sampling units, and of one or more feasible statistical plans of sampling or experimentation that seem to be feasible. d. To explain to the client, in connection with the frame, that any objective inferences that one may draw by statistical theory from the results of an investigation can only refer to the material or system that the frame was drawn from, and only to the methods, levels, types, and ranges of stress presented for study. It is essential that the client understand the limitations of a proposed study, and of the statistical inferences to be drawn therefrom, so that he may have a chance to modify the content before it is too late. e. To furnish statistical procedures for the investigation—selection, computation of estimates and standard errors, tests, audits, and controls as seem warranted for detection and evaluation of important possible departures from specifications, variances between
6420
PRINCIPLES OF PROFESSIONAL STATISTICAL PRACTICE
investigators, nonresponse, and other persistent uncertainties not contained in the standard error; to follow the work and to know when to step in. f. To assist the client (on request) with statistical methods of supervision, to help him to improve uniformity of performance of investigators, gain speed, reduce errors, reduce costs, and to produce a better record of just what took place. g. To prepare a report on the statistical reliability of the results. 9. The statistician’s report or testimony. The statistician’s report or testimony will deal with the statistical reliability of the results. The usual content will cover the following points: a. A statement to explain what aspects of the study his responsibility included, and what it excluded. It will delimit the scope of the report. b. A description of the frame, the sampling unit, how defined and identified, the material covered, and a statement in respect to conditions of the survey or experiment that might throw light on the usefulness of the results. c. A statement concerning the effect of any gap between the frame and the universe for important conclusions likely to be drawn from the results. (A good rule is that the statistician should have before him a rough draft of the client’s proposed conclusions.) d. Evaluation (in an enumerative study) of the margin of uncertainty, for a specified probability level, attributable to random errors of various kinds, including the uncertainty introduced by sampling, and by small independent random variations in judgment, instruments, coding, transcription, and other processing. e. Evaluation of the possible effects of other relevant sources of variation, examples being differences between investigators, between instruments, between days, between areas.
f. Effect of persistent drift and conditioning of instruments and of investigators; changes in technique. g. Nonresponse and illegible or missing entries. h. Failure to select sampling units according to the procedure prescribed. i. Failure to reach and to cover sampling units that were designated in the sampling table. j. Inclusion of sampling units not designated for the sample but nevertheless covered and included in the results. k. Any other important slips and departures from the procedure prescribed. l. Comparisons with other studies, if any are relevant. In summary, a statement of statistical reliability attempts to present to readers all information that might help them to form their own opinions concerning the validity of conclusions likely to be drawn from the results. The aim of a statistical report is to protect clients from seeing merely what they would like to see; to protect them from losses that could come from misuse of the results. A further aim is to forestall unwarranted claims of accuracy that a client’s public might otherwise accept. Any printed description of the statistical procedures that refer to this participation, or any evaluation of the statistical reliability of the results, must be prepared by the statistician as part of the engagement. If a client prints the statistician’s report, it will be printed in full. The statistician has an obligation to institute audits and controls as a part of the statistical procedures for a survey or experiment, to discover any departure from the procedures prescribed. The statistician’s full disclosure and discussion of all the blemishes and blunders that took place in the survey or experiment will build for him a reputation of trust. In a legal case, this disclosure of blunders and blemishes and their possible effects on conclusions to be drawn from the data are a joy to the lawyer.
PRINCIPLES OF PROFESSIONAL STATISTICAL PRACTICE
A statistician does not recommend to a client any specific administrative action or policy. Use of the results that come from a survey or experiment are entirely up to the client. Statisticians, if they were to make recommendations for decision, would cease to be statisticians. Actually, ways in which the results may throw light on foreseeable problems will be settled in advance, in the design, and there should be little need for a client or for anyone else to reopen a question. However, problems sometimes change character with time (as when a competitor of the client suddenly comes out with a new model), and open up considerations of statistical precision and significance of estimates that were not initially in view. The statistician may describe in a professional or scientific meeting the statistical methods that he develops in an engagement. He will not publish actual data or substantive results or other information about a client’s business without permission. In other words, the statistical methods belong to the statistician: the data to the client. A statistician may at times perform a useful function by examining and reporting on a study in which he did not participate. A professional statistician will not write an opinion on another’s procedures or inferences without adequate time for study and evaluation. SUPPLEMENTAL REMARKS 10. Necessity for the statistician to keep in touch. A statistician, when he enters into a relationship to participate in a study, accepts certain responsibilities. He does not merely draft instructions and wait to be called. The people whom he serves are not statisticians, and thus cannot always know when they are in trouble. A statistician asks questions and will probe on his own account with the help of proper statistical design, to discover for himself whether the work is proceeding according to the intent of the instructions. He must expect to revise the instructions a number of times in the early stages. He will be accessible by mail, telephone, telegraph, or in person, to answer questions and to listen to suggestions.
6421
He may, of course, arrange consultations with other statisticians on questions of theory or procedure. He may engage another statistician to take over certain duties. He may employ other skills at suitable levels to carry out independent tests or reinvestigation of certain units, to detect difficulties and departures from the prescribed procedure, or errors in transcription or calculation. It must be firmly understood, however, that consultation or assistance is in no sense a partitioning of responsibility. The obligations of a statistician to his client, once entered into, may not be shared. 11. What is an engagement? Dangers of informal advice. It may seem at first thought that statisticians ought to be willing to give to the world informally and impromptu the benefit of their knowledge and experience, without discussion or agreement concerning participation and relationships. Anyone who has received aid from a doctor of medicine who did his best without a chance to make a more thorough examination can appreciate how important the skills of a professional man can be, even under handicap. On second thought, most statisticians can recall instances in which informal advice backfired. It is the same in any professional line. A statistician who tries to be helpful and give advice under adverse circumstances is in practice and has a client, whether he intended it so or not; and he will later find himself accountable for the advice. It is important to take special precaution under these circumstances to state the basis of understanding for any statements or recommendations, and to make clear that other conditions and circumstances could well lead to different statements. 12. When do the statistician’s obligations come to a close? A statistician should make it clear that his name may be identified with a study only as long as he is active in it and accountable for it. A statistical procedure, contrary to popular parlance, is not installed. One may install new furniture, a new carpet, or a new dean, but not a statistical procedure. Experience shows that a statistical procedure deteriorates rapidly when left completely to nonprofessional administration.
6422
PRINCIPLES OF PROFESSIONAL STATISTICAL PRACTICE
A statistician may draw up plans for a continuing study, such as for the annual inventory of materials in process in a group of manufacturing plants, or for a continuing national survey of consumers. He may nurse the job into running order, and conduct it through several performances. Experience shows, however, that if he steps out and leaves the work in nonstatistical hands, he will shortly find it to be unrecognizable. New people come on the job. They may think that they know better than their predecessor how to do the work; or they may not be aware that there ever were any rules or instructions, and make up their own. What is even worse, perhaps, is that people that have been on a job a number of years think that they know it so well that they cannot go wrong. This type of fault will be observed, for example, in a national monthly sample in which households are to be revisited a number of times; when left entirely to nonstatistical administration, it will develop faults. Some interviewers will put down their best guesses about the family, on the basis of the preceding month, without an actual interview. They will forget the exact wording of the question, or may think that they have something better. They will become lax about calling back on people not at home. Some of them may suppose that they are following literally the intent of the instructions, when in fact (as shown by a control), through a misunderstanding, they are doing something wrong. Or they may depart wilfully, thinking that they are thereby improving the design, on the supposition that the statistician did not really understand the circumstances. A common example is to substitute an averagelooking sampling unit when the sampling unit designated is obviously unusual in some respect (1). In the weighing and testing of physical product, people will in all sincerity substitute their judgment for the use of random numbers. Administration at the top will fail to rotate areas in the manner specified. Such deterioration may be predicted with confidence unless the statistician specifies statistical controls that provide detective devices and feedback. 13. The single consultation. It is wise to avoid a single consultation with a commercial
concern unless satisfactory agenda are prepared in advance and satisfactory arrangements made for absorbing advice offered. This requirement, along with an understanding that there will be a fee for a consultation, acts as a shield against a hapless conference which somebody calls in the hope that something may turn up. It also shows that statisticians, as professional people, although eager to teach and explain statistical methods, are not on the lookout for chances to demonstrate what they might be able to accomplish. Moreover, what may be intended as a single consultation often ends up with a request for a memorandum. This may be very difficult to write, especially in the absence of adequate time to study the problem. The precautions of informal advice apply here. 14. The statistician’s obligation in a muddle. Suppose that fieldwork is under way. Then the statistician discovers that the client or duly appointed representatives have disregarded the instructions for the preparation of the frame, or for the selection of the sample, or that the fieldwork seems to be falling apart. ‘‘We went ahead and did so and so before your letter came, confident that you would approve,’’ is a violation of relationship. That the statistician may approve the deed done does not mitigate the violation. If it appears to the statistician that there is no chance that the study will yield results that he could take professional responsibility for, this fact must be made clear to the client, at a sufficiently high management level. It is a good idea for the statistician to explain at the outset that such situations, while extreme, have been known. Statisticians should do all in their power to help clients to avoid such a catastrophe. There is always the possibility that a statistician may be partly to blame for not being sufficiently clear nor firm at the outset concerning his principles of participation, or for not being on hand at the right time to ask questions and to keep apprised of what is happening. Unfortunate circumstances may teach the statistician a lesson on participation. 15. Assistance in interpretation of non-probability samples. It may be humiliating, but statisticians must face the fact
PRINCIPLES OF PROFESSIONAL STATISTICAL PRACTICE
that many accepted laws of science have come from theory and experimentation without benefit of formal statistical design. Vaccination for prevention of smallpox is one; John Snow’s discovery of the source of cholera in London is another (6). So is the law F = ma in physics; also Hooke’s law, Boyle’s law, Mendel’s findings, Keppler’s laws, Darwin’s theory of evolution, the Stefan-Boltzmann law of radiation (first empirical, later established by physical theory). All this only means, as everyone knows, that there may well be a wealth of information in a nonprobability sample. Perhaps the main contribution that the statistician can make in a nonprobability sample is to advise the experimenter against conclusions based on meaningless statistical calculations. The expert in the subject matter must take the responsibility for the effects of selectivity and confounding∗ of treatments. The statistician may make a positive contribution by encouraging the expert in the subject matter to draw whatever conclusions he believes to be warranted, but to do so over his own signature, and not to attribute conclusions to statistical calculations or to the kind of help provided by a statistician. 16. A statistician will not agree to use of his name as advisor to a study, nor as a member of an advisory committee, unless this service carries with it explicit responsibilities for certain prescribed phases of the study. 17. A statistician may accept engagements from competitive firms. His aim is not to concentrate on the welfare of a particular client, but to raise the level of service of his profession. 18. A statistician will prescribe in every engagement whatever methods known to him seem to be most efficient and feasible under the circumstances. Thus, he may prescribe for firms that are competitive methods that are similar or even identical word for word in part or in entirety. Put another way, no client has a proprietary right in any procedures or techniques that a statistician prescribes. 19. A statistician will, to the best of his ability, at his request, lend technical assistance to another statistician. In rendering this assistance, he may provide copies of procedures that he has used, along with
6423
whatever modification seems advisable. He will not, in doing this, use confidential data. REFERENCES 1. Bureau of the Census (1954). Measurement of Employment and Unemployment. Report of Special Advisory Committee on Employment Statistics, Washington, D.C. 2. Bureau of the Census (1964). Evaluation and research program of the censuses of population and housing. 1960 Series ER60. No. 1. 3. Bureau of the Census (1963). The Current Population Survey and Re-interview Program. Technical Paper No. 6. 4. Center for Advanced Engineering Study. (1983). Management for Quality, Productivity, and Competitive Position. Massachusetts Institute of Technology, Cambridge, Mass. 5. Deming, W. E. (1960). Sample Design in Business Research. Wiley, New York, Chap. 3. 6. Hill, A. B. (1953). Observation and experiment. N. Engl. J. Med., 248, 995–1001. 7. Stephan, F. F. (1936). Practical problems of sampling procedure. Amer. Sociol. Rev., 1, 569–580.
FURTHER READING See the following works, as well as the references just given, for more information on the topic of principles of professional statistical practice. Brown, T. H. (1952). The statistician and his conscience. Amer. Statist., 6(1), 14–18. Burgess, R. W. (1947). Do we need a ‘‘Bureau of Standards’’ for statistics? J. Marketing, 11, 281–282. Chambers, S. P. (1965). Statistics and intellectual integrity. J. R. Statist. Soc. A, 128, 1–16. Court, A. T. (1952). Standards of statistical conduct in business and in government. Amer. Statist., 6, 6–14. Deming, W. E. (1954). On the presentation of the results of samples as legal evidence. J. Amer. Statist. Ass., 49, 814–825. Deming, W. E. (1954). On the Contributions of Standards of Sampling to Legal Evidence and Accounting. Current Business Studies, Society of Business Advisory Professions, Graduate School of Business Administration, New York University, New York. Eisenhart, C. (1947). The role of a statistical consultant in a research organization. Proc. Int. Statist. Conf., 3, 309–313.
6424
PRIOR DENSITY
Freeman, W. W. K. (1952). Discussion of Theodore Brown’s paper [see Brown, 1952]. Amer. Statist., 6(1), 18–20. Freeman, W. W. K. (1963). Training of statisticians in diplomacy to maintain their integrity. Amer. Statist., 17(5), 16–20. Gordon, R. A. et al. (1962). Measuring Employment and Unemployment Statistics. President’s Committee to Appraise Employment and Unemployment Statistics. Superintendent of Documents, Washington, D.C. Hansen, M. H. (1952). Statistical standards and the Census. Amer. Statist., 6(1), 7–10. Hotelling, H. et al. (1948). The teaching of statistics (a report of the Institute of Mathematical Statistics Committee on the teaching of statistics). Ann. Math. Statist., 19, 95–115. Jensen, A. et al. (1948). The life and works of A. K. Erlang. Trans. Danish Acad. Tech. Sci., No. 2 (Copenhagen). International Statistical Institute (1985). Declaration on Professional Ethics, website www.cbs.nl/isi/ethics.htm Molina, E. C. (1913). Computation formula for the probability of an event happening at least c times in n trials. Amer. Math. Monthly, 20, 190–192. Molina, E. C. (1925). The theory of probability and some applications to engineering problems. J. Amer. Inst. Electr. Eng., 44, 1–6. Morton, J. E. (1952). Standards of statistical conduct in business and government. Amer. Statist., 6(1), 6–7. Shewhart, W. A. (1931). Economic Control of Quality of Manufactured Product. D. Van Nostrand, New York. Shewhart, W. A. (1939). Statistical Method from the Viewpoint of Quality Control. The Graduate School, Department of Agriculture, Washington, D.C. Shewhart, W. A. (1958). Nature and origin of standards of quality. Bell Syst. Tech. J., 37, 1–22. Society of Business Advisory Professions (1957). Report of Committee on Standards of Probability Sampling for Legal Evidence. Graduate School of Business Administration, New York University, New York. Zirkle, C. (1954). Citation of fraudulent data. Science, 120, 189–190. See also CONSULTING, STATISTICAL.
W.E. DEMING
PRIOR DENSITY. See PRIOR DISTRIBUTIONS
PRIOR DISTRIBUTIONS Prior distributions are probability distributions or probability density functions (PDFs) that summarize information about a random variable or parameter known or assumed at a given point in time, prior to obtaining further information from empirical data, and so on. The use of the designation ‘‘prior distribution’’ occurs almost entirely within the context of statistical methodology generally labeled ‘‘Bayesian inference’’∗ . Bayes’ theorem∗ and a 1763 paper by Thomas Bayes [2] are generally cited as the beginnings of this methodology. However, Stigler [12] cites evidence that the littleknown English mathematician Saunderson may have been the true builder of the foundations of what is now termed Bayesian inference. Suppose that Y and X are two random variables with PDFs fY (y) and fX (x) (Y and X may be discrete or continuous). Denoting the joint distribution of Y and X by fX,Y (x, y) and conditional distributions of Y given X and X given Y by fY|X (y|x) and fX|Y (x|y), respectively, it is known by definition that fY|X (y|x)fX (x) = fX,Y (x, y) = fX|Y (x|y)fY (y). Then, for example, fX|Y (x|y) =
fY|X (y|x)fX (x) . fY (y)
This result is one representation of the socalled ‘‘Bayes’ theorem,’’ true by definition and not an object of debate among statisticians. fX (x) tells us what is known about X before or prior to the observation of Y, and is thus termed the prior distribution of X. Alternatively, fX|Y (x|y) tells us what is known about X after or posterior to the observation of Y (see POSTERIOR DISTRIBUTIONS). The representation of Bayes’ theorem that is associated with Bayesian inference and the typical reference to prior distribution is f|Y (θ |y) =
fY| (y|θ)f (θ ) , fY (y)
PRIOR DISTRIBUTIONS
where Y = (Y1 , Y2 , . . . , Yn ) is a vector of n independent Yi representing sample observations, each with probability distribution fYi | (yi |θ) dependent on the value of m parameters in the vector . For example, for normally distributed Yi , 1 1 yi − µ 2 fYi | (yi |θ) = √ , exp − 2 σ 2π σ −∞ < yi < ∞, where = (µ, σ ), the mean and standard deviation, and fY| (y|θ) = fYi | (yi |θ). The focus of this representation is inference about the typically unknown parameter vector , where the prior distribution f (θ) is modified by the empirical data y to form the posterior distribution f|Y (θ|y). The prior distribution f (θ) is an object of debate among statisticians. Classical statisticians perceive parameters as fixed, even if unknown, and reject the idea of any ‘‘distribution’’ for . They also complain that even if one accepts the concept of such a distribution, its exact nature is almost inevitably subjective, based only on some vague degree-of-belief∗ probability concept rather than the scientifically sound relative-frequency-in-repeatedexperiments probability concept. Alternatively, Bayesian statisticians are comfortable using prior distributions for unknown parameters, even if there is considerable subjectivity involved. There have been attempts to reduce subjectivity by developing procedures for estimating prior distributions. Martz and Waller [9] discuss associating distribution parameters with characteristics of the distribution that one may feel relatively well known, such as chosen percentiles, the distribution mean, and so on. O’Bryan and Walter [10] and Walter [13] discuss specific prior distribution estimation procedures. Some commonly encountered categorizations of prior distributions are: Improper Prior: Distribution does not sum or integrate to 1. Otherwise, the prior is proper. Improper priors are often used as approximations to proper prior distributions which are nearly flat in the range of indicated to be relevant by the empirical data y. Akaike [1]
6425
and Eaton [5] discuss the interpretation and evaluation of improper priors. Reference Prior: A prior distribution used as a standard in a particular area of study [3, p. 22]. Locally Uniform Prior: Typically refers to a prior density function for a continuous parameter which is at least approximately constant over a limited relevant parameter range, thus expressing equal probability of the parameter falling in any equal-length subrange. Noninformative Prior: A prior distribution which expresses little knowledge about the parameter relative to the information expected in a size n sample of empirical data. Locally uniform priors are commonly used as noninformative priors. There is a problem, however, in selecting the metric (i.e., itself, log , −1 , etc.) in which the prior f (θ ) is locally uniform to best reflect true noninformativeness. Box and Tiao [3, p. 25] discuss this in depth. Jeffrey’s rule [7] is a rule guiding the choice of noninformative prior distributions. See JEFFREYS’ PRIOR DISTRIBUTION. Conjugate Prior: A prior distribution f (θ ) which combines fY| (y|θ) to produce a posterior distribution f|Y (θ |y) which is in the same family as the prior distribution; for example, both are normal distributions. See Martz and Waller [9, p. 226], De Groot [4, p. 164], Raiffa and Schlaifer [11, p. 53], and CONJUGATE FAMILIES OF DISTRIBUTIONS. Exchangeable Prior: A prior distribution f (θ ) which is not altered by any permutation in the multiparameter . Exchangeability∗ has been examined in the context of parameters from multiple populations and multinomial parameters of a single population. See Lindley and Smith [8] and Eaves [6]. REFERENCES 1. Akaike, H. (1980). J. R. Statist. Soc. B, 42, 46–52. (On the interpretation of improper prior distributions as limits of data-dependent proper prior distributions.) 2. Bayes, T. (1958). Biometrika, 45, 298–315. (Reprint of the famous 1763 paper generally
6426
3.
4. 5.
6.
7. 8.
9.
10. 11.
12.
13.
PRIORITY QUEUE
credited with giving birth to Bayesian statistical inference.) Box, G. E. P. and Tiao, G. C. (1973). Bayesian Inference in Statistical Analysis. AddisonWesley, Reading, Mass. (An excellent general statistical reference on Bayesian inference.) DeGroot, M. H. (1970). Optimal Statistical Decisions. McGraw-Hill, New York. Eaton, M. L. (1982). In Statistical Decision Theory and Related Topics III, Vol. 1, S. Gupta and J. Berger, eds. Academic Press, New York, pp. 1329–1352. (Evaluating improper prior distributions.) Eaves, D. M. (1980). J. R. Statist. Soc. B, 42, 88–93. (Good reference list for work with exchangeable priors.) Jeffreys, H. (1961). Theory of Probability, 3rd ed. Clarendon Press, Oxford, England. Lindley, D. V. and Smith, A. F. M. (1972). J. R. Statist. Soc. B, 34, 1–41. (Exchangeable priors in the context of the usual linear statistical model.) Martz, H. F. and Waller, R. A. (1982). Bayesian Reliability Analysis. Wiley, New York. (Chapters 5 and 6 contain good general knowledge about Bayesian inference.) O’Bryan, T. and Walter, G. (1979). Sankhya A, 41, 95–108. Raiffa, H. and Schlaifer, R. (1961). Applied Statistical Decision Theory. Harvard University Press, Cambridge, Mass. Stigler, S. M. (1983). Amer. Statist., 37, 290–296. (A witty search into who really discovered Bayes’ theorem.) Walter, G. (1981). Sankhya A, 43, 228–245.
BIBLIOGRAPHY See the following works, as well as the references just given, for more information on the topic of prior distributions. Lindley, D. V. (1965). Introduction to Probability and Statistics from a Bayesian Viewpoint, Part 1: Probability; Part 2: Inference, Cambridge University Press, Cambridge, England. Savage, L. J. (1954). The Foundations of Statistics. Wiley, New York. (General reference with philosophical basis for Bayesian statistical inference.) See also BAYESIAN INFERENCE; BAYES’ THEOREM; BELIEF, DEGREES OF; CONJUGATE FAMILIES OF DISTRIBUTIONS; POSTERIOR DISTRIBUTIONS; POSTERIOR PROBABILITIES; and PRIOR PROBABILITIES.
DAVID J. PACK
PRIORITY QUEUE As a general background regarding ideas and notation, we refer the reader to QUEUEING THEORY. The theory for priority queueing systems is the branch of queueing theory dealing with systems where high-priority customers get faster through the system at the expense of others. The basic reference seems still to be ref. 4, giving a series of different models, with a variety of priority disciplines, and a solid mathematical treatment. In ref. 5, priority queueing systems of special interest in computer applications are considered. Since the applicability of queueing theory has often been questioned, we will here give reviews of three papers having specific applications in mind. We are not claiming that these are either the best or the most general in the area. One should also remember that even a complex queueing system is more sympathetic to mathematical modeling than is a series of other real-life systems. Hence queueing theory has been and will still be an experimental area to develop tools which are useful in other areas of applied probability. CLASSICAL PRIORITY QUEUEING MODELS Assume that the customers are divided into p different priority classes each having a priority index 1, 2, . . . , p. The index 1 corresponds to the highest-priority class and p corresponds to the lowest. For simplicity we assume a single server and an infinite waiting-room capacity. Within each class we have a first-in-first-out (FIFO) queueing discipline. Denoting the interarrival-time distribution and the service-time distribution within the ith class by the symbols Ai and Bi , respectively, we can use the notation A1 / · · · /Ap /B1 / · · · /Bp /1 for this class of models. However, we have to specify the queueing discipline between customers from different priority classes. Compare the ith and jth classes, where 1 i < j p. In the queue all customers from the ith class are standing in front of all customers from the jth class and will hence be served first. If, on the other hand, a customer from the jth class is being served on the arrival of a customer from the
PRIORITY QUEUE
ith class, there are different alternatives. The priority discipline is preemptive∗ if the current service is immediately interrupted and service is started for the arriving customer. It is nonpreemptive if service is continued to completion, and it is discretionary if the server may use his or her discretion to decide which of the two former strategies to use in each case. For instance, he or she may use the nonpreemptive discipline if the amount of service received exceeds a certain level [4], or if remaining service time is sufficiently small [3]. APPOINTMENT SYSTEM Consider the GI/M/1 queueing model with infinite waiting-room capacity. The customer arriving at t = 0 will find k − 1 customers waiting. The latter customers belong to the second priority class, whereas the ones arriving in [0, ∞) belong to the first. The priority discipline is nonpreemptive, whereas within each class we have a FIFO queueing discipline. As a motivation for studying the present model, consider the following specialization of the arrival pattern above which is realistic when, for example, doctors, dentists, or lawyers are consulted. Let the intervals between possible arrivals have fixed length 1/λ and let the probability of a customer not turning up be 1 − p. Customers turn up or not independently of each other. The number of intervals of length 1/λ between the arrivals of two customers are then geometrically distributed with parameter p. The k − 1 customers in the second priority class do not have an appointment but are allowed to queue up, for instance, either before the office is opened in the morning or before it is reopened after the lunch break. One is now interested in: 1. Waiting times for customers from both priority classes not to be too long. 2. The initial busy period (starting with k customers in the system) not to be too short. Small values of k will satisfy 1, whereas large values satisfy 2.
6427
In ref. 1 the transient waiting times for customers belonging to both priority classes are arrived at for the general model. Using these results on the special arrival pattern above, optimal values for k have been tabulated for various values of p, λ and service intensity. COMPUTER TIME SHARING In ref. 2 a modification of the so-called roundrobin priority discipline is treated in an M/M/ 1 queueing model. Each program receives a quantum q of service at a time from a single central processor. If this completes its service requirement, it leaves the system. If not, and there is a new arrival during the service of the quantum, it is given an additional quantum. Otherwise, the program joins the end of the queue to await its next turn. The model is analyzed under the assumption of a constant, nonzero overhead when the processor swaps one program for another. Obviously, during periods of high arrival rates, this algorithm has the effect of reducing the system’s swapping activities. On the other hand, during periods of low arrival rates the discipline is similar to the conventional round robin, which automatically gives priority to programs with lesser service time requirements. Reference 2 arrives at expressions for the mean waiting time in queue as a function of the quanta required and for the mean system cost due to waiting. Numerical comparisons with the conventional round-robin discipline are performed. TELEPHONE EXCHANGE In ref. 6 a telephone exchange is considered, handling the calls to and from a minor group of subscribers. Let the latter group belong to the second priority class, whereas people from the rest of the world belong to the first class. The Electronic Selector Bar Operator (ESBO) serves customers from both classes. The model one wished to be able to analyse is a modified version of GI/G/M/G/ 1 with a nonpreemptive priority discipline. However, customers of the higher priority are just
6428
PRIOR PROBABILITIES
allowed to wait a fixed length of time before they hear the busy signal and are lost. This is not the case for customers of lower priority. Second, whereas we have a FIFO queueing discipline within the first class, it is random within the second (i.e., all customers waiting have equal chance of being served next). Unfortunately, there seems no way of arriving at the stationary waiting-time distributions for customers of the two classes by using queueing theory of today. As a first approximation ref. 6 starts out from an M1 /G/M2 /G/1 model where Laplace transforms of the distributions above are well known. Furthermore, in this case it is possible to get a look behind the ‘‘Laplace curtain.’’ REFERENCES 1. Dalen, G. and Natvig, B. (1980). J. Appl. Prob., 17, 227–234. 2. Heacox, H. and Purdom, P. (1972). J. ACM, 19, 70–91. 3. Hokstad, P. (1978). INFOR, 16, 158–170. 4. Jaiswal, N. K. (1968). Priority Queues. Academic Press, New York. 5. Kleinrock, L. (1976). Queueing Systems, Vol. 2: Computer Applications. Wiley, New York. 6. Natvig, B. (1977). On the Waiting-Time for a Priority Queueing Model. Tech. Rep. No. 3, Dept. of Mathematical Statistics, University of Trondheim, Trondheim, Norway. Prepared for the Norwegian Telecommunications Administration Research Establishments. See also PREEMPTIVE DISCIPLINE and QUEUEING THEORY.
BENT NATVIG
PRIOR PROBABILITIES Prior probabilities are probabilities that are applicable at a point in time which is prior to one obtaining additional information provided by empirical studies, experiments, surveys, and so on, yet to be executed. In the context of a particular random variable or unknown parameter, prior probabilities follow directly from the choice of a prior distribution∗ , as the assumption of a probability distribution or probability density function in any statistical endeavor automatically leads to the ability to determine
probabilities of various events of interest. See PRIOR DISTRIBUTIONS for a full discussion and references. Suppose for the sake of illustration that you are concerned with the probability of the event E that a particular person known to you only by name is a college graduate. Knowing nothing else, you might reason that about 1 in 20 people in the general population is a college graduate, and thus your probability for E is Pr[E] = 0.05. This is a prior probability. It is based on your knowledge at this time and is not conditioned on any other event (explicitly). It is a probability that you would use prior to getting any other information about the person. If, for example, you knew the person read Newsweek or a similar news magazine, you might feel the chances of the person being a college graduate were more than the 1 in 20 in the general population. The illustration above suggests an element that is frequently present in discussions of prior probabilities—subjective probability assessment. Pr[E] = 0.05 is a statement of degree of belief∗ , not an objective statement based on repeated experiments or some scientific fact. Prior probabilities may have some objectivity, particularly when they are the posterior probabilities from previous empirical observations (see POSTERIOR DISTRIBUTIONS), but in most cases there is inevitably a significant subjective component. The discipline called applied statistical decision theory∗ as discussed by Raiffa and Schlaifer [11] and Raiffa [10] is one that particularly illustrates the utilization of subjective prior probabilities. There is a substantial literature on the quantification of degree-of-belief information in a prior probability. Martz and Waller [8] contains a good bibliographic overview. Hampton et al. [3] and Hogarth [4] are good general technical references. The issue of encoding prior probabilities from one expert source is dealt with in Lichtenstein et al. [6] and Lindley et al. [7]. One may also be concerned with forming the prior probabilities from a number of expert sources into a consensus probability, as discussed by DeGroot [2] and Morris [9]. Adams [1] and Kazakos [5] provide other perspectives on the
PROBABILISTIC NUMBER THEORY
general problem of quantifying prior probabilities. REFERENCES 1. Adams, E. W. (1976). In Foundations of Probability Theory, Statistical Inference and Statistical Theories of Science, Vol. 1, W. L. Harper and C. A. Hooker, eds. D. Reidel, Boston, pp. 1–22. 2. DeGroot, M. H. (1974). J. Amer. Statist. Ass., 69, 118–121. (Consensus prior probabilities via prior distributions which are mixtures of individual expert priors.) 3. Hampton, J. M., Moore, P. G., and Thomas, H. (1973). J. R. Statist. Soc. A, 136, 21–42. (General reference on the measurement of subjective probability.) 4. Hogarth, R. M. (1975). J. Amer. Statist. Ass., 70, 271–289. (General reference on the measurement of subjective probability.) 5. Kazakos, D. (1977). IEEE Trans. Inf. Theory, 23, 203–210. (Estimation of prior probabilities using a mixture.) 6. Lichtenstein, S., Fischhoff, B., and Phillips, L. D. (1977). In Decision Making and Change in Human Affairs, H. Jungerman and G. de Zeeuv, eds. D. Reidel, Dordrecht, The Netherlands, pp. 275–324. (Encoding prior probabilities.) 7. Lindley, D. V., Tversky, A., and Brown, R. V. (1979). J. R. Statist. Soc. A, 142, 146–180. (Reconciling inconsistent prior probability assessments.) 8. Martz, H. F. and Waller, R. A. (1982). Bayesian Reliability Analysis. Wiley, New York. [Section 6.4 gives an overview of prior probability (distribution) assessment.] 9. Morris, P. A. (1977). Manag. Sci., 23, 679–693. (Consensus prior probabilities.) 10. Raiffa, H. (1968). Decision Analysis, Introductory Lectures on Choices under Uncertainty. Addison-Wesley, Reading, Mass. (Introduction to simple statistical decision theory.) 11. Raiffa, H. and Schlaifer, R. (1961). Applied Statistical Decision Theory. Harvard University Press, Cambridge, Mass. See also BELIEF, DEGREES OF; POSTERIOR DISTRIBUTIONS; POSTERIOR PROBABILITIES; and PRIOR DISTRIBUTIONS.
DAVID J. PACK
PROBABILISTIC GRAMMAR. See LINGUISTICS, STATISTICS IN
6429
PROBABILISTIC NUMBER THEORY Some questions and results of number theory can easily be expressed in probabilistic terms; for others, proofs can be simplified by probabilistic arguments. Probabilistic number theory can therefore be defined as a branch of number theory in which questions are of interest to number theorists, but the proofs utilize techniques of probability theory. There are two major subfields of probabilistic number theory: (a) distribution of values of arithmetical functions, and (b) metric (measure theoretic) results concerning expansions of real numbers. These are discussed in detail in the first two sections. The present article is special in that it is somewhat outside the expertise of most readers. Therefore, a detailed survey of the subject is not aimed at; rather, an expository introduction is given in which the major trends, but not the fine points, are presented. DISTRIBUTION OF VALUES OF ARITHMETICAL FUNCTIONS A function f (n) on the positive integers n = 1, 2, . . ., is called an arithmetical function. In other words, an arithmetical function is a sequence in which the emphasis on the variable (index) n is its arithmetical structure rather than its magnitude. Let us start with an example. Let f (n) be the number of all (distinct) prime divisors of n. Hence, to determine the value of f (n), one first has to express n as the product of powers of primes, and then the number of factors in this product is f (n). One can hardly find any regularity of this f (n), because it is frequently equal to 1 (at prime numbers), but it also exceeds any prescribed value infinitely many times (e.g., when n is a factorial). Here is where probabilistic number theory comes in: One can prove, by an appeal to the Chebyshev inequality∗ , that, for ‘‘most numbers n,’’ f (n) is approximately loglog n. More precisely, if A(N) denotes the number of positive integers n N such that |f (n) − log log n| < (log log n)3/4 , then A(N)/N → 1 as N → +∞. To prove this claim, let us introduce the probability spaces
6430
PROBABILISTIC NUMBER THEORY
where p < q are prime numbers and [y] signifies the integer part of y, we have
that log log n ∼ log log N for all n satisfying √ N n N, say, the Chebyshev inequality now implies our claim on A(N) concerning the closeness of f (n) to log log n. This computation represents well the essence of probabilistic number theory. The question, namely the behavior of f (n), originates from number theory; then the basic idea of applying the Chebyshev inequality is a tool of probability theory, but finally, in computing E(f (n)), the application of the prime number theorem is unavoidable. This procedure is generally the case in connection with number-theoretic functions. Only the basic ideas come from probability theory; in the actual computations number-theoretic results are frequently utilized. Let us now consider a whole family of functions f (n) satisfying
PN (ep (n) = 1) = 1/p + O(1/N)
f (nm) = f (n) + f (m)
(N , AN , PN ), where N = {1, 2, . . . , N}, AN is the set of all subsets of N , and PN is the uniform distribution on N . Now the function fN (n) = f (n), 1 n N, is a random variable, which can be represented as f (n) = ep (n), where the summation is over primes p N, and ep (n) = 1 if p divides n and ep (n) = 0 otherwise. The expectation and the variance of f (n) can easily be computed by means of this representation. Namely, because PN (ep (n) = 1) = [N/p]/N and PN (ep (n) = eq (n) = 1) = [N/(pq)]/N,
and PN (ep (n) = eq (n) = 1) = 1/(pq) + O(1/N). Thus E(f (n)) =
1/p + O(π (N)/N),
pN
where π (N) is the number of primes not exceeding N. However, from number theory we know that π (N) ∼ N/ log N (the so-called prime number theorem), which yields E(f (n)) =
(1)
for all n and m which are relatively prime (i.e., they do not have any common divisor other than 1). Functions satisfying (1) are additive arithmetical functions. If, in addition, the additive function f (n) satisfies f (pk ) = f (p) for all primes p and integers k 1, then f (n) is strongly additive. In what follows there is no difference between additive and strongly additive functions. However, computations are much simpler with strongly additive functions, so we restrict ourselves to these. Now, for such functions f (n), (1) implies that
N π (m) − π (m − 1) 1 +O m log N
f (n) =
f (p)ep (n),
(2)
m=1 N
1 1 π (m) = − m m+1 m=1 1 +O log N =
N m=1
1 + O(1) (m + 1) log m
= log log N + O(1). The variance of f (n) can be computed similarly; it also equals log log N. Finally, observing that log log N is so slowly increasing
where summation is again over all primes p (when n is restricted to N , then p N). One easily gets from (2) that E(f (n)) ∼ AN and V(f (n)) ∼ B2N , where AN =
f (p) p
pN
and
B2N =
f 2 (p) . (3) p
pN
Repeating the argument with the Chebyshev inequality, we get that f (n) ∼ AN for ‘‘most n’’ whenever BN → +∞ with N. More accurate results can also be deduced from (2). Because the indicator variables ep (n)
PROBABILISTIC NUMBER THEORY
are almost independent, that is, for arbitrary primes p1 < p2 < · · · < pk , PN (ep1 (n) = ep2 (n) = · · · = epk (n) = 1) = 1/(p1 p2 · · · pk ) + O(1/N), (4) the sums in (2) and (3) suggest that the distribution of (f (n) − AN )/BN converges weakly under conditions similar to sums of independent random variables. Although most results for sums of independent random variables are indeed valid for strongly additive functions, unfortunately these cannot be deduced directly from probability theory because the error term O(1/N) in (4) is more troublesome than it looks at first. Namely, if √ only two primes in (4) are around N, the major term is already smaller than the error term, and it is not easy to show that one can limit the primes in (2) to remain below powers of N. This is why one is forced to consider f (p)ep (n) fr (n) = pr
in place of f (n), where r = r(N) is some slowly increasing function of N, and then one uses probability theory to determine the asymptotic behavior of fr (n), and number theory (so-called sieve methods) to show that the difference f (n) − fr (n) is small compared with BN . This line of attack proves very successful. A few basic results will be quoted as we now turn to the history of the subject. The references are far from complete, even when one adds all references in the survey papers and books listed later. The probabilistic theory of arithmetic functions started with the paper of ` [43], who gave a new proof, essentially Turan through the Chebyshev inequality (although without reference to probability theory), for the asymptotic relation between the number of prime divisors and log log n. The result itself had been known in number theory, but it was obtained by analytic methods. Both P. Erd¨os and M. Kac were influenced ´ by Turan’s proof. They, first independently, and later jointly, contributed at the early stages the most significant results of the distribution theory of additive functions. It was Kac who approached the subject from
6431
a purely probabilistic point of view by recognizing in the representation at (2) the sum of almost independent random variables. Kac’s early work is well set out in his popular monograph [24]. Erd¨os [7] and Erd¨os and Wintner [10] found necessary and sufficient conditions for the weak convergence of PN (f (n) < x) for arbitrary additive functions f (n). Erd¨os and Kac [9] established that if the sequence f (p) is bounded (over primes p), then, for the strongly additive function f (n), PN (f (n) < AN + BN x) is asymptotically standard normal, where AN and BN are defined at (3). It turned out that the boundedness of f (p) is not necessary for asymptotic normality∗ . In a series of papers, J. Kubilius developed a probabilistic theory of distributions for the truncated functions fr (n), in which his only requirement on r is that r → +∞ with N and log r/log N should converge to zero as N → +∞. If in addition Br /BN → 1 and BN → +∞ as N → +∞, then f (n) and fr (n) have the same asymptotic laws. Later, Kubilius published his work in a book which has had several editions; we quote here the English language edition [30]. His contribution to the theory of distributions of arithmetical functions is immeasurable because he not only published significant papers and books on the subject, but established a school at Vilnius which produced a large number of scientists working on the subject and also made a considerable influence on both individual scientists all over the world and the literature. Some of the work of this school is presented in the survey papers of Galambos [11,14], Kubilius [31], Manstavicius [34], and Schwarz [41], and in the books of Kubilius [30] and Elliott [6]. In the Kubilius model, to show that (f (n) − fr (n))/BN converges to zero (‘‘in probability’’), quite deep number theoretic results are employed. This induced investigations aimed at reobtaining at least some of the results of Kubilius by purely probabilistic arguments. Several models have been set up, which to various degrees have achieved the desired results, but none of them would have come close to the generality of the Kubilius model. However, through the models of Novoselov [35], Paul [36], Galambos [12], and Philipp [37], interesting probabilistic proofs are given for one result or another in the distribution theory of additive
6432
PROBABILISTIC NUMBER THEORY
functions. The model of Galambos has the added advantage that it presents a unified treatment of additive functions when the variable is not limited to the set of successive integers. Such investigations are extensive ` in the literature; see Kubilius [30], Katai [26,27], and Indlekofer [21]. For the truncated function fr (n) Kubilius was able to use the theory of sums of independent random variables. However, for extending his results to additive functions for which a truncation is not possible, other methods are necessary. The method of characteristic functions∗ proved fruitful, although it required new deep number-theoretic investigations. Namely, since PN is the uniform distribution over a finite set, the characteristic function of an arithmetical function f (n) is the arithmetical mean of the function g(n) = exp(itf (n)). Now, if f (n) is additive, then, in view of (1), g(n) satisfies the equation g(mn) = g(m)g(n)
(5)
for all m and n which are relatively prime. A function satisfying (5) is multiplicative. Thus asymptotic mean value theorems were needed for multiplicative functions whose absolute value is bounded by 1. Such results were established by Delange [5] and ` [18], who employed tools of analytic Halasz number theory (see also R´enyi [40]). These mean value theorems gave new life to the distribution theory of arithmetical functions. In a series of papers, Elliott, C. Ryavec, ` Delange, Halasz, and the members of the Vilnius school brought the theory of additive functions to a very advanced level. The twovolume set of books by Elliott [6] is based on this new development. The distribution theory of multiplicative functions has also been studied in detail. Because multiplicative functions which can take both positive and negative values cannot be related to real-valued additive functions, their distribution theory requires a completely different approach. The combined result of Bakstys [1] and Galambos [13] gives necessary and sufficient conditions for the weak convergence of PN (g(n) < x) for multiplicative functions g(n), and the weak convergence of the normalized multiplicative function (g(n) − CN )/DN is characterized in Levin et al. [33].
For arithmetical functions which are neither additive nor multiplicative, approximation methods have been developed: by additive functions in Galambos [15] and Erd¨os and Galambos [8], and by so-called Ramanujan sums in a series of papers by W. Schwarz and his collaborators; see the survey [41]. The distribution theory of arithmetical functions has several applications within number theory. For example, for estimating certain sums, see De Koninck and Ivic [4], and for its role in characterizing log n among all additive functions, see Indlekofer [22]. For how much joy the study of this theory can give, one should read the recollections of Kac [25]. METRIC THEORY OF EXPANSIONS OF REAL NUMBERS When a sequence of ‘‘digits’’ is assigned to real numbers by a specific algorithm, we speak of an expansion. The best known expansion is the decimal, or more generally, the q-adic expansions, in which, when q is an integer, the digits are independent and identically distributed random variables with respect to Lebesgue measure. Hence, although no regularity of the digits of irrational numbers seems to hold, the law of large numbers implies that these digits are quite regular for almost all x. Other theorems of probability theory are directly applicable as well. The situation is completely different for most other series expansions. As an example, let us look at the so-called Engel series. For a given real number 0 < x 1, the ‘‘digits’’ dj are defined by the following algorithm: Every dj is an integer such that 1/dj < xj 1/(dj − 1), where x = x1 and xj+1 = dj xj − 1. This algorithm leads to the series representation x = 1/d1 + 1/(d1 d2 ) + 1/(d1 d2 d3 ) + · · · , where 2 d1 d2 d3 · · · . The digits dj are strongly dependent in view of their monotonicity. However, the ratios
PROBABILISTIC NUMBER THEORY
dj /(dj−1 − 1) can well be approximated by independent and identically distributed random variables, which can be utilized for discovering general properties of the sequence dj , valid for almost all x. Similar situations arise in connection with most series expansions. That is, the digits are dependent but after some transformation an approximation by independent random variables becomes possible. This is not obvious, though, in most instances, and the challenge in these problems usually is probabilistic: to extend known theorems of probability theory to some weakly dependent random variables. These investigations did indeed contribute to probability theory itself. Most results on series expansions are contained in the books of Galambos [17] and Vervaat [44], although for some expansions there are newer results as well (see, e.g., Deheuvels [3] concerning the Engel series). Among the expansions which do not result in a series most attention was paid to continued fraction expansions. This is obtained by repeated application of the transformation Tx = x − [1/x], where, once again, [y] signifies the integer part of y. As far back as in the nineteenth century, Gauss recognized that, if instead of length (i.e., Lebesgue measure), an absolutely continuous measure is assigned to the unit interval whose derivative is 1/{(log 2)(1 + x)}, then the sequence an , defined by the algorithm aj = [1/xj ], where x = x1 and Txj = xj+1 , is stationary. More recent studies revealed that this sequence is stationary mixing, which fact induced intensive studies of mixing sequences of random variables. This benefited both probability theory and number theory; see Ibragimov and Linnik [20], Billingsley [2], and Philipp [37]. For a more classical approach, see Khintchine [29]. On the speed of convergence of the distribution of Txn , see Kuzmin [32] and ¨ [42]. Galambos [16], Philipp [38], and Szusz Iosifescu [23] deal with the stochastic behavior of the largest of the digits an . The work of R´enyi [39] shows the connection between series expansions and continued fractions. We conclude by noting that there are several works in probabilistic number theory which do not belong to either topic covered in the preceding two sections. These are, however, individual problems rather than a
6433
part of a systematic study. We mention two ` of these. Katai [28] is probably the purest among all papers in probabilistic number theory in that he shows that for certain sequences of integers, the number of divisors √ of n in the neighborhood of n is normally distributed, in which argument no number theory is applied. The other area we wish to mention is the proof of the existence of certain types of numbers by showing that their probability is positive. Although most of the results are due to Erd¨os, the most convenient source for consultation is Halberstam and Roth [19]. REFERENCES 1. Bakstys, A. (1968). Litovsk. Mat. Sb., 8, 5–20. 2. Billingsley, P. (1965). Ergodic Theory and Information. Wiley, New York. 3. Deheuvels, P. (1982). C. R. Acad. Sci. Paris, 295, 21–24. 4. De Koninck, J. -M. and Ivic, A. (1980). Topics in Arithmetical Functions. North-Holland, Amsterdam. ´ 5. Delange, H. (1961). Ann. Sci. Ecole Norm. Sup., 78, 273–304. 6. Elliott, P. D. T. A. (1979/1980). Probabilistic Number Theory, Vols. I and II. SpringerVerlag, Berlin. 7. Erd¨os, P. (1938). J. Lond. Math. Soc., 13, 119–127. 8. Erd¨os, P. and Galambos, J. (1974). Proc. Amer. Math. Soc., 46, 1–8. 9. Erd¨os, P. and Kac, M. (1940). Amer. J. Math., 62, 738–742. 10. Erd¨os, P. and Wintner, A. (1939). Amer. J. Math., 61, 713–721. 11. Galambos, J. (1970). Ann. Inst. Henri Poincar´e B, 6, 281–305. 12. Galambos, J. (1971). Z. Wahrscheinl. verw. Geb., 18, 261–270. 13. Galambos, J. (1971). Bull. Lond. Math. Soc., 3, 307–312. 14. Galambos, J. (1971). In The Theory of Arithmetic Functions, A. A. Gioia et al., eds. Springer-Verlag, Berlin, pp. 127–139. 15. Galambos, J. (1973). Proc. Amer. Math. Soc., 39, 19–25. 16. Galambos, J. (1974). Acta Arith., 25, 359–364. 17. Galambos, J. (1976). Representations of Real Numbers by Infinite Series. Springer-Verlag, Berlin.
6434
PROBABILISTIC VOTING MODELS
` 18. Halasz, G. (1968). Acta Math. Acad. Sci. Hung., 19, 365–403. 19. Halberstam, H. and Roth, K. F. (1966). Sequences 1. Oxford University Press, Oxford, England. 20. Ibragimov, I. A. and Linnik, Yu. V. (1971). Independent and Stationary Sequences of Random Variables. Wolters-Noordhoff, Groningen. 21. Indlekofer, K. -H. (1974). In Proc. Coll. Bolyai Math. Soc. Debrecen, B. Gyires, ed. NorthHolland, Amsterdam, pp. 111–128. 22. Indlekofer, K. -H. (1981). Ill. J. Math., 25, 251–257. 23. Iosifescu, M. (1978). Trans. 8th Prague Conf. Inf. Theory, Vol. A. D. Reidel, Dordrecht, The Netherlands, pp. 27–40. 24. Kac, M. (1959). Statistical Independence in Probability, Analysis and Number Theory. Carus Monographs. Wiley, New York. 25. Kac, M. (1979). Probability, Number Theory and Statistical Physics. MIT Press, Cambridge, Mass. ` 26. Katai, I. (1968). Compositio Math., 19, 278–289. ` 27. Katai, I. (1969). Acta Math. Acad. Sci. Hung., 20, 69–87. ` 28. Katai, I. (1977). Publ. Math. Debrecen, 24, 91–96. 29. Khintchine, A. (1964). Continued Fractions. Wolters-Noordhoff, Groningen, The Netherlands. 30. Kubilius, J. (1964). Probabilistic Methods in the Theory of Numbers. Transl. Math. Monogr., Vol. 11. American Mathematical Society, Providence, R.I. 31. Kubilius, J. (1974). In Current Problems of Analytic Number Theory, Izd. Nauka i Techn., Minsk, pp. 81–118. 32. Kuzmin, R. O. (1928). Rep. Pol. Acad. Sci. (A), 375–380. 33. Levin, B. V., Timofeev, N. M., and Tuljaganov, S. T. (1973). Litovsk. Mat. Sb., 13, 87–100. 34. Manstavicius, E. (1980). Litovsk. Mat. Sb., 20, 39–52. 35. Novoselov, E. V. (1964). Izv. Akad. Nauk SSSR, Ser. Mat., 28, 307–364. 36. Paul, E. M. (1962). Sankhya A, 24, 103–114, 209–212; ibid., 25, 273–280 (1963). 37. Philipp, W. (1971). Mem. Amer. Math. Soc., 114, 1–102. 38. Philipp, W. (1976). Acta Arith., 28, 379–386.
39. R´enyi, A. (1957). Acta Math. Acad. Sci. Hung., 8, 477–493. 40. R´enyi, A. (1965). Publ. Math. Debrecen, 12, 323–330. 41. Schwarz, W. (1976). Jb. Dtsch. Math.-Ver., 78, 147–167. ¨ 42. Szusz, P. (1961). Acta Math. Acad. Sci. Hung., 12, 447–453. ` 43. Turan, P. (1934). J. Lond. Math. Soc., 9, 274–276. 44. Vervaat, W. (1972). Success Epochs in Bernoulli Trials with Applications in Number Theory. Math. Centre Tracts No. 42, Mathematische Centrum, Amsterdam. See also FIRST-DIGIT PROBLEM; LAWS OF LARGE NUMBERS; ¨ FUNCTION; and PROBABILITY THEORY: AN MOBIUS OUTLINE).
JANOS GALAMBOS
PROBABILISTIC VOTING MODELS In many nations and at many levels of government, the question of who should hold various political offices is settled by voting. As a consequence, there are many public officials, and many politicians who would like to become public officials, who are concerned about the implications of their choices for the decisions that voters will make. In particular, they are concerned about which voters will vote in the next election and who these people will vote for. Two choices that elected officials and officeseeking politicians make that are especially important in this regard are: Their positions on the leading policy questions of the day and the way in which they allocate their campaign resources. As a consequence, researchers have developed mathematical models of the relation between these choices and the decisions that public officials and office-seeking politicians can expect the voters to make. These models can be used to address questions such as: In a given situation in which campaign resources have already been allocated, are there ‘‘best’’ policy positions that can be taken? If so, what are they? In a given situation in which policy positions have already been selected, is there a ‘‘best’’ campaign resource allocation that can be chosen? If so, what is it? In a given situation in which both policy positions
PROBABILISTIC VOTING MODELS
and an allocation of resources can be selected, is there a ‘‘best’’ possible combination? If so, what is it? The answers are useful for social scientists who want to know what policy positions and campaign resource allocations can be expected from public officials and officeseeking politicians. They are also potentially useful for campaign strategists who advise political candidates.
TWO EXAMPLES Before stating the features that differ in the two examples in this section, the characteristics that they have in common will be covered: There are two candidates for a particular political office. They will be indexed by the elements in the set C = {1, 2}. There are three voters. They will be indexed by the elements in the set N = {1, 2, 3}. Each candidate has to decide on what policy proposals he or she wants to make. The only characteristic of a candidate’s policy proposals that matters to the voters is the distribution of income that they expect the policies to lead to. For each vector of policy proposals that a candidate can make, each voter has a unique distribution of income that he or she expects the policies to lead to. These expectations are the same for all three voters. As a consequence, the candidate’s decision can be viewed as the choice of an income distribution∗ . It will be assumed that each candidate can, in particular, choose any ψc ∈ X = {(x1 , x2 , x3 ) ∈ R3 : x1 + x2 + x3 = 1, x1 0.01, x2 0.01, x3 0.01} (by a suitable choice of policy proposals). Each particular voter cares only about his or her own income. More specifically, (for any given i ∈ N) for each pair x, y ∈ X : x is at least as good as y for i if and only if xi yi . This implies that i prefers x to y (i.e., x is at least as good as y for i, but y is not at least as good as x for i) if and only if xi > yi . It also implies that i is indifferent between x and y (i.e., x is at least as good as y for i and y is at least as good as x for i) if and only if xi = yi . Thus i’s preferences on X can be represented by the utility function Ui (x) = xi ; for each pair x, y ∈ X, Ui (x) Ui (y) if and only if x is at least as good as y for i. In addition, assume
6435
that Ui (x) measures the intensity of i’s preferences. More specifically, the values assigned by Ui (x) have the following interpretations: Ui (x)/Ui (y) = 12 if and only if i likes x only half as much as he or she likes y; Ui (x)/Ui (y) = 2 if and only if i likes x twice as much as he or she likes y; Ui (x)/Ui (y) = 3 if and only if i likes x three times as much as he or she likes y; and so on. This property implies that each voter’s utility∗ function is unique up to multiplication by a positive scalar. (It is, accordingly, called a ratio-scale utility function.) In the examples, Pci (ψ1 , ψ2 ) will be used to denote the probability that the individual indexed by i will vote for candidate c when c = 1 chooses ψ1 and c = 2 chooses ψ2 . Thus [at any given (ψ1 , ψ2 ) ∈ X 2 ] the expected vote for a given c ∈ C can be written as EVc (ψ1 , ψ2 ) =
3
Pci (ψ1 , ψ2 ).
i=1
Each candidate c is concerned solely about his or her expected plurality. Plc (ψ1 , ψ2 ) = EVc (ψ1 , ψ2 ) − EVk (ψ1 , ψ2 ) (where k is the index for the other candidate), that is, with his or her expected margin of victory (or, phrased differently, how much he or she expects to win or lose by). Furthermore, each candidate wants to maximize his or her expected plurality; in a candidate’s view, the larger the expected margin of victory, the better. This implies that, for any specification of the Pci (·) functions, the decisions that the two candidates have to make can be appropriately modeled as a two-person, noncooperative game, (X, X; Pl1 , Pl2 ), in which (1) the two players are the two candidates, (2) the strategy set for each candidate is X, and (3) the payoff functions are Pl1 : X × X → R1 and Pl2 : X × X → R1 , respectively. By the definitions of Pl1 and Pl2 , Pl1 (ψ1 , ψ2 ) + Pl2 (ψ1 , ψ2 ) = 0∀(ψ1 , ψ2 ) ∈ X 2 . Hence the game is zero-sum. See GAME THEORY or NASH EQUILIBRIUM. Example 1. Consider the case in which, for each i ∈ N and each pair (ψ1 , ψ2 ) ∈ X 2 , 1 if Ui (ψ1 ) > Ui (ψ2 ) P1i (ψ1 , ψ2 ) = 12 if Ui (ψ1 ) = Ui (ψ2 ) 0 if Ui (ψ1 ) < Ui (ψ2 )
6436
PROBABILISTIC VOTING MODELS
and P2i (ψ1 , ψ2 ) = 1 − P1i (ψ1 , ψ2 ). The resulting game, (X, X; Pl1 , Pl2 ), has no Nash equilibrium. Since the game is zerosum, this can also be phrased as: The game has no saddle point. The fact that this game has no Nash equilibrium (or, equivalently, has no saddle point) can be seen quite easily. Choose any (x, y) ∈ X 2 . Since the game is zero-sum, there are three possibilities: (a)
Pl1 (x, y) = Pl2 (x, y) = 0,
(b)
1
(c)
2
Pl (x, y) > 0 > Pl (x, y), or Pl1 (x, y) < 0 < Pl2 (x, y).
Suppose that (a) or (b) holds. Identify a voter, i, who gets at least as much income at x as anyone else (i.e., for whom we have xi xj , ∀j = i). Let z be the alternative in which this individual gets zi = 0.01 and the others get zj = xj + 12 (xi − 0.01) (i.e., the others split the decrease in i’s income). Since xi > 0.01, we have Uj (z) = zj > xj = Uj (x) for each j = i and Ui (z) = zi < xi = Ui (x). Therefore, P2j (x, z) = 1 and P1j (x, z) = 0 for each j = i and, in addition, P2i (x, z) = 0 and P1i (x, z) = 1. Therefore, Pl2 (x, z) = +1 > 0 Pl2 (x, y). Therefore, if (a) or (b) holds, (x, y) is not a Nash equilibrium. Similar reasoning applies when (c) holds. Example 1 is a game-theoretic version of the voting paradox∗ , the fact that when majority rule is used to make collective decisions, a society can find itself in a situation where, for each social choice that could be made, there is a feasible alternative that a majority of the voters prefer (and, hence, any particular choice can be overturned by a majority vote). More specifically, this example illustrates how easy it is for the paradox of voting (and the corresponding absence of equilibrium policies) to occur when the issues that are to be resolved involve the distribution of income in the society (see also VOTING PARADOX, Arrow [1] and Sen [30]). Example 2. Consider the case in which the probabilities that describe the voters’ choices
satisfy the following version of Luce’s axiom of ‘‘independence from irrelevant alternatives’’ (see CHOICE AXIOM, LUCE’S): For each i ∈ N and (ψ1 , ψ2 ) ∈ X 2 , P1i (ψ1 , ψ2 ) P2i (ψ1 , ψ2 )
=
Ui (ψ1 ) ; Ui (ψ2 )
(1)
assume also (implicit in Example 1) that each voter is going to vote; that is, for each i ∈ N and (ψ1 , ψ2 ) ∈ X 2 , P1i (ψ1 , ψ2 ) + P2i (ψ1 , ψ2 ) = 1.
(2)
Equations (1) and (2) imply that for each i ∈ N and (ψ1 , ψ2 ) ∈ X 2 , P1i (ψ1 , ψ2 ) =
Ui (ψ1 ) , Ui (ψ1 ) + Ui (ψ2 )
P2i (ψ1 , ψ2 ) =
Ui (ψ2 ) . Ui (ψ1 ) + Ui (ψ2 )
This time, the resulting game (X, X; Pl1 , Pl2 ) does have a Nash equilibrium. Since the game is zero sum, this can also be phrased as: The game has a saddle point (see GAME THEORY or NASH EQUILIBRIUM). The fact that this game has a Nash equilibrium (or equivalently, has a saddle point) can be seen quite easily. For each i ∈ N and (x, y) ∈ X 2 , P1i (x, y) = xi /(xi + yi ). Therefore, each P1i (x, y) is a concave function of x = (x1 , x2 , x3 ), and each −P1i (x, y) is a concave function of y = (y1 , y2 , y3 ). This, in turn, implies that EV1 (x, y) is a concave function of x and −EV1 (x, y) is a concave function of y. Similarly, EV2 (x, y) is a concave function of y and −EV2 (x, y) is a concave function of x. Therefore, candidate 1’s payoff function, Pl1 (x, y) = EV1 (x, y) − EV2 (x, y), is a concave function of x and candidate 2’s payoff function, Pl2 (x, y) = EV2 (x, y) − EV1 (x, y), is a concave function of y. By a similar argument, Pl1 (x, y) and Pl2 (x, y) are continuous functions of (x, y). Finally, from its definition, X is a compact, convex subset of R3 . Hence all of the assumptions in the premise of one of the theorems that has been labeled ‘‘Nash’s Theorem’’ are satisfied (see NASH EQUILIBRIUM). Therefore, there is a Nash equilibrium in this example.
PROBABILISTIC VOTING MODELS
Where is (or are) the Nash equilibrium (or equilibria) located? This question can be answered by solving the problem: Find the x ∈ X that maximize(s) U1 (x) · U2 (x) · U3 (x) = x1 · x2 · x3 over the set X. There is a unique x which solves this problem: x = ( 13 , 13 , 13 ). This implies that there is a unique Nash equilibrium in the game: ψ1 = ψ2 = ( 13 , 13 , 13 ). (See Theorem 1 in Coughlin and Nitzan [9].) GENERAL MODEL Models similar to the two examples given above have been studied in the references listed at the end of this entry. They are all special cases of the following general model: There are two candidates for a particular political office. They will be indexed by the elements in the set C = {1, 2}. There is a set of possible ‘‘locations,’’ X, for the candidates. The set of possible locations is the same for both of them. A particular x ∈ X could, for instance, specify a position on the policy issues in the election, an allocation of campaign resources, or both of these. ψc will be used to denote a particular location for c. There is an index set, N, for the individuals who can vote in the election. A particular i ∈ N could be a number (e.g., if the voters are labeled as voters 1, . . . , n), a specification of various characteristics that a voter can have (such as location of residence, income, age, etc.), a vector that specifies the ‘‘ideal’’ positions on the various policy issues for a voter, or something else. For each i ∈ N and c ∈ C, there is a function Pci : X × X → [0, 1] that assigns to each (ψ1 , ψ2 ) ∈ X × X a probability for the event ‘‘a voter randomly drawn from the individuals labeled by i ∈ N will vote for c if candidate 1 is located at ψ1 , and candidate 2 is located at ψ2 .’’ For each (ψ1 , ψ2 ) ∈ X 2 , P0i (ψ1 , ψ2 ) = 1 − P1i (ψ1 , ψ2 ) − P2i (ψ1 , ψ2 ) will be the probability for the event ‘‘a voter randomly drawn from the individuals labeled by i ∈ N will not vote (i.e., will abstain from voting), if candidate 1 is located at ψ1 and candidate 2 is located at ψ2 .’’ These can be objective
6437
probabilities or they can be subjective probabilities that are believed by both of the candidates. There is a probability distribution on N which assigns to each measurable set, B ⊆ N, a probability for the event ‘‘a voter randomly drawn from the individuals who can vote in the election has an index i ∈ B.’’ These probabilities can also be either objective probabilities or subjective probabilities that are believed by both of the candidates. Each candidate is concerned solely about his or her (a) expected vote, EVc (ψ1 , ψ2 ), (b) expected plurality, Plc (ψ1 , ψ2 ), or (c) probability of winning W c (ψ1 , ψ2 ). DETERMINISTIC VOTING MODELS An important special case for the general model given above is the deterministic voting model. This terminology comes from analyses of models in which each candidate wants to maximize his or her expected plurality (as in the examples). When [for a given i ∈ N and (ψ1 , ψ2 ) ∈ X 2 ] P1i (ψ1 , ψ2 ) = P2i (ψ1 , ψ2 ), the expected vote from the individual(s) indexed by ‘‘i’’ is split evenly between the two candidates. When this occurs (since each candidate’s objective is to maximize his or her expected plurality), the expected votes corresponding to the index i cancel each other out—and, therefore, have no effect on Pl1 (ψ1 , ψ2 ) or Pl2 (ψ1 , ψ2 ). For instance, in Example 1, if U3 (ψ1 ) = U3 (ψ2 ), then P13 (ψ1 , ψ2 ) − P23 (ψ1 , ψ2 ) = 12 − 1 2 = 0. Therefore, the expected plurality for candidate 1 when only voters 1 and 2 are counted is the same as when voters 1, 2, and 3 are counted. That is, [P11 (ψ1 , ψ2 ) + P12 (ψ1 , ψ2 )] − [P21 (ψ1 , ψ2 ) + P22 (ψ1 , ψ2 )] = [P11 (ψ1 , ψ2 ) + P12 (ψ1 , ψ2 ) + P13 (ψ1 , ψ2 )] − [P21 (ψ1 , ψ2 ) + P22 (ψ1 , ψ2 ) + P23 (ψ1 , ψ2 )]. Similarly, the expected plurality for candidate 2 when only voters 1 and 2 are counted is the same as when voters 1, 2, and 3 are counted. From the preceding observations it is thus clear that at any given (ψ1 , ψ2 ), the
6438
PROBABILISTIC VOTING MODELS
only voter indices that matter (to expected plurality maximizing candidates) are ones with P1i (ψ1 , ψ2 ) = P2i (ψ1 , ψ2 ). When, in fact, P1i (ψ1 , ψ2 ) = 1 or P2i (ψ1 , ψ2 ) = 1 at a given i ∈ N and (ψ1 , ψ2 ) ∈ X 2 , one of two things must be true: (a) there is one voter with the index i and the candidates believe that his or her decision will be completely determined once they choose the strategies ψ1 and ψ2 , respectively, or (b) there is more than one voter with the index i and the decisions made by all these voters will be completely determined (and the same) once the candidates choose the strategies ψ1 and ψ2 , respectively. Because of this, any model which satisfies the assumptions of the general voting model given above and is such that, at each (ψ1 , ψ2 ) ∈ X 2 and i ∈ N where P1i (ψ1 , ψ2 ) = P2i (ψ1 , ψ2 ), either (i) P1i (ψ1 , ψ2 ) = 1 or (ii) P2i (ψ1 , ψ2 ) = 1, is called a ‘‘deterministic voting model.’’ Example 1, for instance, is such a model, whereas Example 2 is not. The defining characteristic of a deterministic voting model can be restated as: For each (ψ1 , ψ2 ) ∈ X 2 and i ∈ N, either the expected votes corresponding to the index i cancel each other out (and, therefore, have no effect on the candidates’ expected pluralities) or the decision(s) of the voter(s) corresponding to the index i will be completely determined (and identical when candidate 1 chooses ψ1 and candidate 2 chooses ψ2 . A third way of stating this characteristic is: For each (ψ1 , ψ2 ) ∈ X 2 and i ∈ N, (a) P1i (ψ1 , ψ2 ) = 1, (b) P2i (ψ1 , ψ2 ) = 1, or (c) P1i (ψ1 , ψ2 ) = P2i (ψ1 , ψ2 ) = 12 (1 − P0i (ψ1 , ψ2 )). The deterministic voting models that have received the most attention are ones in which (1) each index corresponds to one voter, (2) each voter, i, has a utility function, Ui (x), and (3a) for each voter, i, and each (ψ1 , ψ2 ) ∈ X 2 , 1 if Ui (ψ1 ) > Ui (ψ2 ), 1 Pi (ψ1 , ψ2 ) = 12 if Ui (ψ1 ) = Ui (ψ2 ), 0 if Ui (ψ1 ) < Ui (ψ2 ), P2i (ψ1 , ψ2 ) = 1 − P1i (ψ1 , ψ2 ) (as in Example 1) or (3b) for each voter, i, and each (ψ1 , ψ2 ) ∈ X 2 1 if Ui (ψ1 ) = Ui (ψ2 ), P0i (ψ1 , ψ2 ) = 0 if Ui (ψ1 ) = Ui (ψ2 ),
P1i (ψ1 , ψ2 )
1 if Ui (ψ1 ) > Ui0 (ψ2 ), = 0 if Ui (ψ1 ) Ui (ψ2 ),
P2i (ψ1 , ψ2 ) = 1 − P1i (ψ1 , ψ2 ) − P0i (ψ1 , ψ2 ). An exception is McKelvey [26], where it is assumed that (a) for each index, α, all of the voters who are labeled by this index have the same utility function, Uα (x), and (b) for each index α and each (ψ1 , ψ2 ) ∈ X 2 , (P0α (ψ1 , ψ2 ), P1α (ψ1 , ψ2 ), P2α (ψ1 , ψ2 )) ∈ {(1, 0, 0), (0, 1, 0), (0, 0, 1)}, P1α (ψ1 , ψ2 ) = 1 ⇒ Uα (ψ1 ) > Uα (ψ2 ), P2α (ψ1 , ψ2 ) = 1 ⇒ Uα (ψ1 ) < Uα (ψ2 ). This formulation provides a deterministic voting model in which abstentions can occur more frequently than just when Uα (ψ1 ) = Uα (ψ2 ). The seminal work on deterministic voting models was done by Hotelling [21], Downs [16], Black [2], and Davis and Hinich [11,12]. Surveys of results that have been derived in these and other analyses [using either Plc (ψ1 , ψ2 ) or W c (ψ1 , ψ2 ) as the objective functions for the candidates] can be found in Plott [29], Kramer [22], and Mueller [27,28]. PROBABILISTIC VOTING MODELS Any model which satisfies the assumptions of the general voting model given above, but is not a deterministic voting model, is called a probabilistic voting model. Example 2, for instance, is such a model. The terminology reflects the fact that in any such model, there is at least one pair of strategies (ψ1 , ψ2 ) ∈ X 2 at which there is at least one index that matters (to expected plurality maximizing candidates) where the candidates’ beliefs about the voter(s) corresponding to the index are probabilistic in a nontrivial way; the random variable that describes them is nondegenerate. The defining characteristic for these models can be restated as: There is at least one (ψ1 , ψ2 ) ∈ X 2 and i ∈ N where P1i (ψ1 , ψ2 ) = P2i (ψ1 , ψ2 ) and 0 < P1i (ψ1 , ψ2 ) < 1 or 0 < P2i (ψ1 , ψ2 ) < 1 (or both).
PROBABILISTIC VOTING MODELS
Any model that satisfies all the assumptions of the general voting model given above will be a probabilistic voting model if there is an index i that corresponds to two or more voters and at least one pair of possible locations (ψ1 , ψ2 ) ∈ X 2 such that the choices of the voters corresponding to index i are completely determined when candidate 1 chooses ψ1 and candidate 2 chooses ψ2 , but (a) they will not all make the same choice, and (b) those who will vote are not split evenly between the candidates. Indeed, almost any deterministic voting model can be converted into a probabilistic voting model of this sort by appropriately regrouping the indices in the deterministic voting model into indices for the probabilistic voting model. For instance, let all three voters in Example 1 have the same index i. Then, for the strategies ψ1 = ( 31 , 13 , 13 ) and ψ2 = (0, 12 , 12 ), we have P1i (ψ1 , ψ2 ) = 13 and P2i (ψ1 , ψ2 ) = 23 . The resulting model is a probabilistic voting model. This approach is the basis for the models analyzed in McKelvey [26]. A voting model that satisfies all the assumptions of the general voting model given above can, alternatively, be a probabilistic voting model if there is at least one voter whose choice of whether to vote and/or which candidate to vote for (if he or she votes) is probabilistic in nature, as in Example 2. (See CHOICE AXIOM, LUCE’S for a discussion of theories of probabilistic choice behavior.) This approach is the basis for the models analyzed in Brams [3], Brams and Davis [4], Comaner [5], Coughlin and Nitzan [9,10], Davis and Hinich [13], Hinich [17], Hinich and Ordeshook [18], Hinich et al. [19,20], and Lake [23]. A voting model that satisfies all the assumptions given above could be a probabilistic voting model because the candidates are uncertain about the choices that voters will make and use subjective probabilities to summarize their expectations about these choices (see DECISION THEORY and UTILITY THEORY). This is the basis for the model analyzed in Ledyard [24,25]. Analyses that apply to models in which any of these three interpretations can arise have been carried out in Coughlin [7,8]. The most closely scrutinized probabilistic voting models can be grouped into three basic categories. The first consists of models
6439
in which the description of the candidates’ expectations about the voters’ decisions to vote (or abstain) are probabilistic, but the description of their expectations about the choices that the voters make between the candidates are deterministic (i.e., they believe that, for each (ψ1 , ψ2 ) ∈ X × X and i ∈ N, when candidate 1 chooses ψ1 and candidate 2 chooses ψ2 , the voters corresponding to i who vote will all vote for the same candidate and the candidate who so benefits is completely determined). This category includes the models in Davis and Hinich [13], Hinich and Ordeshook [18], Hinich et al. [19,20], and McKelvey [26]. The second category consists of models in which there are no abstentions, but the description of the candidates’ expectations about the choices that the voters will make in choosing between them is probabilistic: (a) P0i (ψ1 , ψ2 ) = 0, ∀(ψ1 , ψ2 ) ∈ X × X, ∀i ∈ N and (b) ∃i ∈ N and (ψ1 , ψ2 ) ∈ X × X, where P1i (ψ1 , ψ2 ) = 12 . This category includes Brams [3], Brams and Davis [4], Hinich [17], Coughlin and Nitzan [9,10], and Coughlin [8]. The third category consists of models in which the description of the candidates’ expectations both about voter abstentions and about the choices that the voters will make in choosing between them are probabilistic. This category includes Denzau and Kats [15], Coughlin [6,7], and Ledyard [24,25]. For further discussions, see the references.
REFERENCES 1. Arrow, K. (1963). Social Choice and Individual Values, 2nd ed. Yale University Press, New Haven, Conn. 2. Black, D. (1958). The Theory of Committees and Elections. Cambridge University Press, Cambridge, England. 3. Brams, S. (1975). Game Theory and Politics. MacMillan, London. 4. Brams, S. and Davis, M. (1982). Math. Social Sci., 3, 373–388. 5. Comaner, W. 169–178.
(1976).
J. Public
6. Coughlin, P. 427–433.
(1982).
Public
Econ.,
Choice,
5, 39,
7. Coughlin, P. (1983). Math. Social Sci., 4, 275–292.
6440
PROBABILITIES, ASSESSMENT OF
8. Coughlin, P. (1984). J. Econ. Theory, 34, 1–12. 9. Coughlin, P. and Nitzan, S. (1981). J. Public Econ., 15, 113–122. 10. Coughlin, P. and Nitzan, S. (1981). J. Econ. Theory, 24, 226–240. 11. Davis, O. and Hinich, M. (1966). In Mathematical Applications in Political Science, Vol. II, J. Bernd, ed. Southern Methodist University Press, Dallas, Tex. 12. Davis, O. and Hinich, M. Choice, 5, 59–72.
(1968).
Public
13. Davis, O. and Hinich, M. (1972). In Probability Models of Collective Decision Making, R. Niemi and H. Weisberg, eds. Charles E. Merrill, Columbus, Ohio. 14. Davis, O., DeGroot, M., and Hinich, M. (1972). Econometrica, 40, 147–157. 15. Denzau, A. and Kats, A. (1977). Rev. Econ. Stud., 44, 227–233. 16. Downs, A. (1957). An Economic Theory of Democracy. Harper, New York. 17. Hinich, M. 208–219.
(1977).
J. Econ.
Theory,
16,
18. Hinich, M. and Ordeshook, P. (1969). Public Choice, 7, 81–106. 19. Hinich, M., Ledyard, J., and Ordeshook, P. (1972). J. Econ. Theory, 4, 144–153. 20. Hinich, M., Ledyard, J., and Ordeshook, P. (1973). J. Politics, 35, 154–193. 21. Hotelling, H. (1929). Econ. J., 39, 41–57. 22. Kramer, G. (1977). In Frontiers of Quantitative Economics, Vol. III, M. Intriligator, ed. North-Holland, Amsterdam. 23. Lake, M. (1979). In Applied Game Theory, S. Brams, A. Schotter, and G. Schwodiauer, eds. ¨ Physica-Verlag, Wurzburg, Federal Republic of Germany. 24. Ledyard, J. (1981). In Essays in Contemporary Fields of Economics, G. Horwich and J. Quirk, eds. Purdue University Press, West Lafayette, Ind.
30. Sen, A. (1970). Collective Choice and Social Welfare. Holden-Day, San Francisco. See also CHOICE AXIOM, LUCE’S; DECISION THEORY; GAME THEORY; NASH EQUILIBRIUM; POLITICAL SCIENCE, STATISTICS IN; UTILITY THEORY; and VOTING PARADOX.
PETER COUGHLIN
PROBABILITIES,
ASSESSMENT
OF.
See ASSESSMENT OF PROBABILITIES
PROBABILITY AND LOGIC This term usually deals with formalized theories for probability and more generally the study of relationships between logic and probability. The metamathematical structure of probability theory comes out of the necessity to study the formal linguistic structure which goes into the description of the concepts of probability. The method is a glueing of probability and logic methods, intended to give us a better understanding of the foundations of probability∗ and statistics; it may be a new starting point for incorporating theories such as those of fuzzy sets∗ , probabilistic metric spaces, Boolean and Heyting-valued models, nonstandard analysis, topos theory, and so on. As a result we may also have a general theory of uncertainty not necessarily probabilistic in nature. We review the more mathematical aspects of the subject. The reader who is interested in foundational problems and different axiomatic theories for probability should see ref. 7. The article is divided into three sections: Assigning Probabilities to Logical Formulas, Hyperfinite Model Theory, and Final Remarks.
25. Ledyard, J. (1984). Public Choice, 44, 7– 41. 26. McKelvey, R. 815–844.
(1975).
Econometrica,
43,
27. Mueller, D. (1976). J. Econ. Lit., 14, 395– 433. 28. Mueller, D. (1979). Public Choice. Cambridge University Press, Cambridge, England. 29. Plott, C. (1971). In Frontiers of Quantitative Economics, Vol. I, M. Intriligator, ed. NorthHolland, Amsterdam.
ASSIGNING PROBABILITIES TO LOGICAL FORMULAS Historically this idea goes back to the Cambridge philosopher W. E. Johnson, who influenced both Keynes and Jeffreys in their wellknown treatises on probability [10]. Also, one can find similar ideas in Koopman’s comparative logical probability [7, pp. 183–186].
PROBABILITY AND LOGIC
Carnap was the first to study probabilities on formalized sentences [7, pp. 187–204]. His study, however, was restricted to a portion of a formalized language and to symmetric probabilities. It was Gaifman [8] who first used the full power of a first-order language and arbitrary probabilities. His work was extended considerably by Scott and Krauss [19] and others [4–6], most recently Gaifman and Snir [9]. Let L be a first-order language and let C be the set of its constants with C ⊆ T. A probability model for L(T) is an ordered pair (T, µ) where µ : S0 → [0, 1] is a probability on the quantifier-free sentences S0 (T) such that, for φ, ψ ∈ S0 (T), (i) µ(φ) = 1 if φ is provable in L and µ(φ) = µ(ψ) if φ and ψ are logically equivalent in L, (ii) µ(φ ∨ ψ) + µ(φ ∧ ψ) = µ(φ) + µ(ψ), (iii) µ(φ) = 1 − µ(ϕ). Gaifman [8] first proved the following interesting extension property. Theorem. Let (T, µ) be a probability model. Then there exists a unique probability µ∗ on the set S(T), of all closed sentences of L(T), which extends µ and satisfies the Gaifman condition: φ(t) , (G)µ∗ (υφ(υ)) = sup µ∗ F∈T (ω)
t∈F
where T (ω) is the set of all finite subsets of T. It follows that Doob’s separability condition is a special case of the (G) condition. Another important aspect of assigning probabilities to logical formulas is that it is a special case of assigning truth values in a complete Boolean algebra B. More precisely, if we take as the Boolean algebra B the quotient algebra of the σ -field A of a probability space (, A, P), modulo the σ -ideal of sets of probability zero, then assigning probabilities to logical formulas is the same as assigning truth values in B. This gives us a connection to Boolean-valued models [1,23] and finally to more general Heytingvalued models [13] and topos theory. These structures are generalizations toward covering uncertainties which are not necessarily probabilistic in nature.
6441
HYPERFINITE MODEL THEORY Hyperfinite model theory was introduced by Keisler [16] and is aimed at models that arise in probability and applied mathematics, in contrast with classical model theory which is more appropriate for problems in algebra and set theory. Hyperfinite models are appropriate for the study of the limiting behavior of finite models through the use of finite combinatorial arguments in infinite settings. Keisler’s work [16], along with Hoover’s [14,15], constitutes a novel approach to the metamathematics of probability. It is based on hyperfinite probability [21], which in turn is based on Robinson’s infinitesimal analysis. A hyperfinite model is a structure A = A, µ, ai , Sj , i ∈ I, j ∈ J, where A, µ is a hyperfinite probability space, aj ∈ A, and each Sj is an internal relation on A. The logics LωP , Lω1 P involved are similar to the ordinary logics but instead of the quantifiers ∃x and ∀x, one uses the probability quantifiers (Px > r), (Px r), where (Px > r) means that the set {x, φ(x)} has probability greater than r in the Loeb measure involved. An interpretation of a hyperfinite model can be given in terms of the so-called opinion poll model [16]. We can think of A as a very large set of people, Sj as a statement about the tuple y, and Sj (a, b) is true if the person a agrees with statement Sj about the n-tuple of people b. Thus (Px > 12 )(Py > 12 )S(x, y) means that a majority of x’s agree with S about a majority of y’s. The sentence (Py > 12 )(Px > 12 )S(x, y) means that for a majority of y’s, a majority of x’s agree with S about y. Neither of the foregoing sentences implies the other. In addition to the above logics, an internal logic L is introduced with hyperfinite truth values and three quantifiers (max), (min), and . . . dx. This internal logic is needed to get deeper model-theoretic results about the probability logic Lωp, but it is also of interest in its own right. An interesting discovery is the connection between apparently unrelated theorems from model theory and
6442
PROBABILITY AND LOGIC
probability theory. Downward L¨owenheim Skolem theorems for L are closely related to the weak law of large numbers∗ , whereas the elementary chain theorem is related to the strong law of large numbers. In the opinion-poll model an elementary submodel B of A is a sample which is good in the sense that, for any statement Sj and any n-tuple b ∈ An , the percentage of a ∈ B who agree with Sj about b is infinitely close to the percentage of a ∈ A who agree with Sj about b. In a similar way one can interpret the L¨owenheim-Skolem theorem. FINAL REMARKS In addition to the above two mainstreams of metamathematics of probability there are interesting connections between logic and probability. Hyperfinite Stochastic Analysis This is based on Robinson’s infinitesimal analysis and is a formal way of reducing continuous problems to discrete ones. For example, Brownian motion∗ is treated as a hyperfinite random walk∗ , stochastic differential equations∗ as stochastic difference equations, stochastic integrals∗ as nonstandard Stieltjes integrals, etc. For details, extensions and further references, see ref. 21. Statistical Model Theory This idea starts with Suppes [22], who studies models of (statistical) theories, models of experiments, and models of data. Then we ´ ´ have the book of Hajek and Harvanek [11], ´ and Harvanek’s work [12] toward a model theory of statistical theories. These works can be considered a starting point for a metamathematical structure of statistics. From a Bayesian viewpoint, Gaifman and Snir [9] study a language L which has also an empirical part L0 , consisting of finitely many so-called empirical predicates and/or empirical function symbols. Recursion Theory and Randomness The basic issue here is what it means for a sequence to be random. R. von Mises∗ based his axiomatic treatment of probability on the notion of ‘‘kollektiv,’’ a notion of
randomness∗ for a sequence; see ref. 7, pp. 92–100. Church [2] was the first to apply recursion theory to von Mises’ concept of a kollektiv. The names of Ville, Kolmogorov, Martin-L¨of, Knuth, Schnorr, and others [17] are connected with the efforts to develop a consistent theory of random sequences, with the help of recursion theory. The most recent developments of the notion of randomness can be found in Gaifman and Snir [9]. For details and exact references that are not mentioned here, see refs. 7, 9, 12, and 17. Fuzzy Sets, Heyting-Valued Models, and Topos Theory Examining the work of Higgs [13] and other developments of topos theory one can easily convince oneself that Heyting-valued models and topos theory constitute a formalization of the theory of fuzzy sets. The logic involved is of course intuitionistic. On the other hand some of the work on elementary topoi was conducted much in the spirit of nonstandard analysis. The elementary topoi is the appropriate frame to do nonstandard mathematics. The development of nonstandard models (not only Robinson’s infinitesimal models [1,13,21,23]), has shown that they contain the concept of ‘‘randomness’’ or more generally the concept of ‘‘uncertainty’’ as a basic element. For example, in Takeuti [23, Chap. 2] the nonstandard reals are random variables, and in Solovay [20] the concept of a ‘‘random real’’ is introduced for the adoption of Cohen’s forcing into measure-theoretic problems. Furthermore, Boolean algebras and probabilities on them represent the qualitative and quantitative aspects of ‘‘randomness’’ respectively, whereas Heyting algebras represent more general aspects of uncertainty, not necessarily probabilistic in nature, e.g., fuzzy sets [3]. It is believed that fuzzy sets∗ [3] and probabilistic metric spaces [18] can be expressed formally using Heyting-valued and Booleanvalued models. REFERENCES 1. Bell, J. L. (1977). Boolean-Valued Models. Clarendon, Oxford, England. (A standard textbook on the subject. Presupposes acquaintance with mathematical logic and axiomatic set theory.)
PROBABILITY AND MATHEMATICAL STATISTICS 2. Church, A. (1940). Bull. Amer. Math. Soc., 46, 130–135. 3. Dubois, D. and Prade, H. (1980). Fuzzy Sets and Systems: Theory and Applications. Academic Press, New York. 4. Eisele, K. Th. (1976). Booleschwertige Modelle mit Wahrscheinlichkeitsmasses und Darstellungen Stochastischer Prozesse. Ph.D. Thesis, University of Heidelberg, Germany. 5. Fenstad, J. E. (1967). In Sets, Models and Recursion Theory, J. N. Crossley, ed. NorthHolland, Amsterdam, pp. 156–172. 6. Fenstad, J. E. (1968). Synth`ese, 18, 1–23. 7. Fine, T. (1973). Theories of Probability. Academic Press, New York. (Examines the foundations of probability. Perhaps the only textbook of this sort.) 8. Gaifman, H. (1964). Israel J. Math., 2, 1–18. (Original work; deals with finitary formulas.) 9. Gaifman, H. and Snir, M. (1982). J. Symbolic Logic, 47, 495–548. (A recent landmark paper which introduces new concepts; results from a Bayesian viewpoint and summarizes and extends the works in refs. 4, 5, 16, and 14.) 10. Good, I. J. (1965). The Estimation of Probabilities. MIT Press, Cambridge, MA. ´ ´ 11. Hajek, P. and Havranek, T. (1978). Mechanizing Hypothesis Formation: Mathematical Foundations for a General Theory. Universitexte, Springer-Verlag, Heidelberg. ´ 12. Havranek, T. (1977). Synth`ese, 36, 441–458. (Technical paper, worth reading by those interested in model theory of statistical theories. Also, there are further references on this subject.) 13. Higgs, D. (1973). A Category Approach to Boolean-Valued Set Theory. University of Waterloo, preprint. (Excellent technical paper for Heyting-valued models; can be considered as a formalization of the theory of fuzzy sets.) 14. Hoover, D. N. (1978). Ann. Math. Logic, 14, 287–313. (Technical paper on probability logic extending Keisler’s work in ref. 11.) 15. Hoover, D. N. (1982). J. Symbolic Logic, 47, 605–624. (Recent extensions of refs. 11 and 13.) 16. Keisler, H. J. (1977). In Logic Colloquium 76. North-Holland, Amsterdam, pp. 5–110. (Source paper; develops hyperfinite model theory, based on hyperfinite probabilities; see ref. 16.) 17. Schnorr, C. P. (1971). Zufalligkeit ¨ und Wahrscheinlichkeit. Lecture Notes in Math., No.
218, Springer, Berlin. (Studies sequences using recursion theory.)
6443 random
18. Schweizer, B. and Sklar, A. (1983). Probabilistic Metric Spaces. North-Holland, Amsterdam. (The only introductory text to this very interesting subject.) 19. Scott, D. and Krauss, P. (1966). In Aspects of Inductive Logic, J. Hintikka and P. Suppes, eds. North-Holland, Amsterdam, pp. 219–264. (Extends ref. 7 to infinitary languages.) 20. Solovay R. M. (1970). Ann. Math., 92, 1–59. (Source paper; adopts forcing to measure theoretic problems.) 21. Stroyan, K. D. and Bayod, J. M. (1983). Foundations of Infinitesimal Stochastic Analysis. North-Holland, Amsterdam. (A graduate introduction to hyperfinite probability and infinitesimal stochastic analysis.) 22. Suppes, P. (1962). In Logic, Methodology and Philosophy of Science; Proceedings of the 1960 International Congress, E. Nagel, P. Suppes, and A. Tarski, eds. Stanford University Press, Stanford, CA, pp. 252–261. 23. Takeuti, G. (1978). Two Applications of Logic to Mathematics. Iwanami Shoten, Tokyo; Princeton University Press, Princeton, NJ. (A technical book that develops Boolean analysis.) See also AXIOMS OF PROBABILITY; BELIEF FUNCTIONS; PROBABILITY, FOUNDATIONS OF —I; FUZZY SET THEORY; NONADDITIVE PROBABILITY; and SUBJECTIVE RANDOMNESS.
CONSTANTIN A. DROSSOS
PROBABILITY AND MATHEMATICAL STATISTICS This journal is published by the Institute of Mathematics, Polish Academy of Sciences, Wroclaw University, Poland. The Editor-inChief is Kazimierz Urbanik. The main purpose of the journal is to foster development in fields of probability theory of special relevance to statistics. Volume 7 appeared in 1986. Papers may be in English or French. Most papers are in English. The editorial address is Pl. Grunwaldzki 2 / 4, 50-384 Wroclaw, Poland. The website is www.math.uni.wroc.pl/mathbank/ pms.html
6444
PROBABILITY BOUNDS, WORSLEY–HUNTER
PROBABILITY BOUNDS, WORSLEY–HUNTER This is an improved Bonferroni upper bound∗ proposed by Worsley [8] in 1982. A similar result was obtained independently by Hunter [3]. A somewhat different approach for obtaining Bonferroni-type bounds was developed by Galambos [1]. In its general formulation, the Worsley bound is given by the following result: Let events A1 , . . . , An be represented as vertices υ1 , . . . , υn of a graph∗ G, where υi and υj are joined by an edge eij if and only if the events Ai and Aj are not disjoint. Let T be a spanning tree of G. Then n n Ai Pr(Ai ) − Pr(Ai ∩ Aj ). Pr i=1
i=1
(i,j):eij ∈T
(1) The sharpest bound is obtained by finding the tree for which the second term on the right hand side of (1) is maximum. [Note that the upper Bonferroni bound simply states that n n Ai Pr(Ai ).] Pr i=1
i=1
Ai = Ap1 ∪ (Ap2 \Ap2 ) ∪ · · ·
i=1
∪ (Apn \Apn ).
(2)
Applying the Bonferroni bound to (2) we obtain (1). If T ≡ G, then the events on the right-hand side of (2) are disjoint and (1) becomes equality (by the well-known properties of the union operator ∪). As a corollary, letting T = (e12 , e23 , . . . , en−1,n ), we have n n n−1 Ai Pr(Ai ) − Pr(Ai ∩ Ai+1 ). Pr i=1
i=1
i=1
n
Pr(Ai ) − (2/n)
i=1
Pr(Ai ∩ Aj )
i<j
is a particular case of the Worsley bound, as is the more general Kounias bound [4]: n Pr Ai i=1
n i=1
Pr(Ai ) −
m
Pr(Ai ∩ Ajk ),
k=1 i∈Jk \jk
where (J1 , . . . , Jm ) is a partition of {1, . . . , n} and jk ∈ Jk , k = 1, . . . , m. See Worsley [8] for more details and applications. Also compare with the Galambos [1] approach. There are enlightening discussions of the Worsley-Hunter bound in [2], [6] and, more recently [7]. In the latter paper, Seneta presents a sharpening of the bounds, and an illuminating numerical example. REFERENCES
If T is a tree it is always possible to find a permutation p1 , . . . , pn of {1, . . . , n} so that υpi is joined to some υpj = υpi say, for some j < i, i = 2, . . . , n, and ni=1 Ai can be written as n
Kwerel’s [3] refinement of the Bonferroni bound given by n Pr Ai
i=1
1. Galambos, J. (1977). Ann. Prob., 5, 577–581. 2. Hochberg, Y. and Tamhane, A. C. (1987). Multiple Comparison Procedures, Wiley, New York. 3. Hunter, D. (1976). J. Appl. Prob., 13, 597–603. 4. Kounias, E. G. (1968). Ann. Math. Statist., 39, 2154–2158. 5. Kwerel, S. M. (1975). J. Amer. Statist. Ass., 70, 472–479. ˇ aritescu, ˇ 6. Marg E. (1986). Biom. J., 28, 937–943. 7. Seneta, E. (1988). Austral. J. Statist., 30A, 27–38. 8. Worsley, K. J. (1982). Biometrika, 69, 297–302. See also BONFERRONI INEQUALITIES AND INTERVALS and GRAPH THEORY.
PROBABILITY DENSITIES, COMPATIBILITY OF This concept was introduced by Abrahams and Thomas [1]. Let p1 (x|y) and p2 (y|x) be
PROBABILITY FORECASTING
given conditional density functions. If there exists a bivariate density a (x, y) with the given conditional densities, these densities are said to be compatible. A necessary and sufficient condition for compatibility of a pair of conditional densities over the region of the (x, y) space where these densities are nonzero is that g(x) p1 (x|y) is of the form , p2 (y|x) h(y) where g and h are nonnegative (integrable) functions satisfying
g(x)dx =
h(x) dy.
Examples of beta (yielding bivariate Dirichlet density), gamma, exponential, and Gaussian conditional densities are provided in ref. [1]. This result has been used and refined later by subsequent authors for characterization of bivariate densities in terms of their conditionals. See BIVARIATE DISTRIBUTIONS, SPECIFICATION OF.
REFERENCE 1. Abrahams, J. and Thomas, J. B. (1984). A note on the characterization of bivariate densities by conditional densities. Commun. Statist. Theory Methods, 13, 395–400.
PROBABILITY FORECASTING At its simplest, probability forecasting refers to the process of attaching a numerical probability to an uncertain event. Surprisingly, this most basic requirement for any realworld application of probability theory was for long ignored by most statisticians. The motivation for its development as a discipline in its own right has largely arisen from the needs of the field of probabilistic weather forecasting∗ , and much of the theoretical work underpinning it originally appeared in the meteorological literature. This special application also forms a convenient framework on which to hang a discussion of general principles.
6445
In many countries the weather service still issues its weather forecasts in purely verbal terms, either as a categorical prediction (‘‘It will be dry and sunny’’) or with imprecise indications of uncertainty (‘‘A chance of light showers’’). But many decisions can often only be taken rationally when the uncertain nature of the weather is admitted and quantified as precisely as possible. To meet this need, the National Weather Service in the United States has, since 1965, formulated and issued probability of precipitation (PoP) forecasts. Such a forecast is a number between 0 and 1, intended to be interpreted as the probability that measurable precipitation ( 0.01 inch) will occur during a specific period (generally 12 hours) at a particular point in the area of concern. These PoP forecasts are now popularly accepted by the American public as meaningful and informative. The forecasts themselves are usually arrived at intuitively by the weather forecaster, after taking account of all the information available. This information may include, as a decision aid, an ‘‘objective’’ probability produced by a computerized statistical analysis of weather records, but the forecast issued is usually interpreted, in the last analysis, as a personal or subjective probability∗ statement for that forecaster. Questions thus arise concerning the motivation and integrity of forecasters in forming and stating their subjective probabilities, the empirical validity of the forecasts issued, the extent of the agreement between different forecasters, and the way in which the public should use the forecasts. This article will concentrate on the first two issues. A very readable guide is that of Sta¨el von Holstein [30]. The work of Savage [27] on elicitation of subjective probabilities is particularly elegant. ENCOURAGING HONESTY [28] To motivate a forecaster F to divulge honestly his personal probability p for the uncertain event A, we can present him with a decision problem having economic consequences to him which depend on his decision∗ d and the outcome a of A (where a = 1 if A occurs, a = 0 if not). These consequences may be expressed
6446
PROBABILITY FORECASTING
by a loss function L(a, d). If L is measured on a utility∗ scale, or is small overall in relation to the forecaster’s assets, the theory of coherence∗ implies that he will choose d = dp , where dp minimizes his expected loss pL(1, d) + (1 − p)L(0, d) = L(p, d), say. Suppose now that F is asked to quote a value q ∈ [0, 1], on the understanding that this will commit him to the decision dq appropriate if q were his ‘‘true’’ probability. Then he clearly has no incentive to quote any q = p, and should definitely quote q = p if dp yields the unique minimum of L(p,d). His loss, if he quotes q and A = a, is S(a, q) = L(a, dq ), so that his expected loss, if he quotes q but believes Pr(A) = p, is S(p, q) = L(p, dq ), which is minimized if q = p. Any such function S(a, q), for which S(p, ˜ q)], where a˜ = 1 with q) (defined as E[S(a, probability p, 0 with probability 1 − p) is minimized in q when q = p, is a proper scoring rule; it is strictly proper if this minimum is unique. Such a rule can always be regarded as arising, as above, from a decision problem, for example that having d ∈ [0, 1] and L(a, d) = S(a, d). Using any such scoring rule as a loss function encourages honesty. (However, difficulties in their use arise when the forecaster’s utility is nonlinear in S, or he is motivated to maximize the probability of getting a bigger score than rival forecasters.) In practice, if the elicitor of F’s forecast intends to use it as his own in a decision problem he faces, presenting F with a ‘‘share’’ in his own business will yield a scoring rule which is likely to provide the best motivation for F to concentrate his attention on good assessment where it matters most. The formula D(p, q) = S(p, q) − S(p, p)(= L(p, dq ) − L(p, dp )) defines a directed distance of q from p associated with the decision problem or scoring rule, from which S(a, q) may be effectively recovered as D(a, q). Then D(p, p) = 0, and D(p, q) is nondecreasing as q departs from p in either direction [27]. Important strictly proper scoring rules are the quadratic Brier score [3], with S(a, q) = (a − q)2 , and the logarithmic score [12] S(1, q) = − log q, S(0, q) = − log(1 − q). The scoring rule S(a, q) = |a − q| is improper, however, having optimal qp = 1 for p > 12 , 0 for p < 12 . Murphy and Epstein [23]
consider the optimal ‘‘hedges’’ qp for other scoring rules. We can introduce an operational definition of F’s personal probability p as the solution of dp = d∗ , where d∗ is F’s chosen decision in his decision problem, and the solution is supposed existent and unique. Equivalently, we take p = q, where q is F’s quote under a strictly proper scoring rule. General coherence arguments (see especially ref. 27) imply that this solution should not depend on the specific decision problem. An early approach along these lines was that of de Finetti [7,8], based on the Brier score. Multipurpose Forecasts The users of a weather forecast face many diverse decision problems. A particularly simple one, the cost-loss problem [31,32], is that of the fruit farmer, say, who can take protective measures (decision d1 ) to guard her plants against frost (event A), at a cost of C; if she does not protect (d0 ), she faces a loss L if there is frost, 0 if not. Her expected loss when Pr(A) = p is L(p, d1 ) = C, L(p, d0 ) = pL, so that dp = d1 if p > k = C/L, dp = d0 if p < k. The associated proper scoring rule S(a, q) = L(a, dq ) is not, however, strictly proper, q being an optimal quote whenever q and p both lie on the same side of k, and thus lead to the same decision. To motivate F to meet the needs of all such decision makers, one might proceed as follows [16,17]. After F states his forecast q, a value k is generated, independently of A, from a known distribution v on [0, 1], and he is faced with the cost-loss decision problem having L = 1, C = k, in which he must use his implied optimal decision (i.e., dq = d1 if k < q, d0 if k q). His overall expected loss on following this strategy for quote q when A = a is
q
S(a, q) = 0
1
kdv(k) + a
dv(k). q
This formula must therefore yield a proper scoring rule [18], and this will be strictly proper if and only if v has the whole of [0, 1] as its support (so that p and q have to be on the same side of any k ∈ [0, 1]).
PROBABILITY FORECASTING
With no essential change, 1 we can replace S(1, q) above by S(1, q) − 0 kdv(k), yielding the standard form q kdv(k), S(0, q) =
0 1
S(1, q) =
(1 − k)dv(k).
(1)
q
It can, in fact, be shown [27–29], under weak conditions, that any proper scoring rule is equivalent to one having the form (1), where v is an arbitrary measure on [0, 1]. (No additional generality is obtained by allowing L above to vary also, as in [25].) In choosing an ‘‘all-purpose’’ scoring rule to elicit a probability in practice, it makes sense to use one whose associated v gives strongest weight to values of k regarded as most typical of the decision problems faced by users. Taking v to be the uniform distribution on [0, 1], dv(k) = dk, effectively recovers the Brier score [16]. FORECASTING CONTINUOUS AND MULTISTATE QUANTITIES Many of the ideas above extend to the more general case in which the forecaster has to assess a whole distribution Q for a quantity (or quantities) X. Again we can consider a decision problem with loss function L(x, d), define dQ to minimize EQ [L(X, d)], and so construct a proper scoring rule S(x, Q) = L(x, dQ ). One important strictly proper scoring rule is the logarithmic score S(x, Q) = − log q(x), where q(·) is the density of Q with respect to a dominating measure µ. It may be shown [1,15,27] that whenever X can take more than two values, this is essentially the only proper scoring rule which is local [i.e., depends on q(·) only through its value at the point x which materializes]. The associated distance function is D(P, Q) = p(x) log[p(x)/q(x)]dµ(x), the Kullback–Leibler information distance of Q from P. In practice one often confines attention to a finite set A = (As : s = 1, 2, . . . , k) of events
6447
determined by X, and their corresponding probability forecasts q = (qs : s = 1, 2, . . . , k). Let these be judged by a strictly proper scoring rule S(a, q) when A = a. It may be that S is defined for values of q that violate the probability axioms. However, considerations of coherence imply [27] that any elicited probabilities should obey the axioms, so that if, for example, A3 is the disjoint union of A1 and A2 , then q3 = q1 + q2 (see also ref. 14 for coherence of deduced probabilities when the loss function is given by an improper scoring rule). The kBrier score2 in the context above is now s=1 (as − qs ) and is strictly proper. It is normally applied in the case that the (As ) form a partition [e.g., wind-speed rating (A9 = gale, force 9, etc.)]. Then q is a ‘‘multistate’’ forecast. However, when, as in this example, the various states have a natural ordering, the Brier score appears somewhat unsatisfactory, since no credit is given for assessing high probabilities for states close to that materializing. This end can be accomplished by an analog of the cost-loss decision problem: A suitable loss function would have the property that if ds is the optimal decision when As occurs, then the loss in using dr instead of ds increases with the distance between r and s. Suitable random selection of such a decision problem will yield a strictly proper scoring rule which is ‘‘sensitive to distance.’’ Epstein [11] used this argument to derive his ‘‘ranked probability score.’’ EMPIRICAL VALIDITY: RELIABILITY A single noncategorical probability forecast (i.e., not 0 or 1) can never be said to have been ‘‘right’’ or ‘‘wrong.’’ But when a forecaster has issued a long string of such forecasts, it becomes possible to apply checks of external validity. Suppose that of n sequential forecasts, the ith is pi and the realized outcome of the associated event Ai (‘‘rain on day i’’) is ai (= 0 or 1). Then we can compare the overall aver age forecast probability pn = n−1 ni=1 pi with the overall relative frequency of occurrence an = n−1 ni=1 ai . If pn an , the set of forecasts may be regarded as approximately valid on an overall basis. Murphy and Epstein [22]
6448
PROBABILITY FORECASTING
refer to this property as ‘‘unbiasedness in the large.’’ A more incisive test looks at that subset of occasions i for which the forecast probability pi was at, or suitably close to, some preassigned value p∗ , and compares the observed relative frequency in this subset, a(p∗ ) say, with p∗ . If a(p∗ ) p∗ for all p∗ , the forecasts have been variously termed ‘‘unbiased in the small,’’ ‘‘reliable,’’ ‘‘valid,’’ or ‘‘well calibrated.’’ [Of course, sampling variability will lead to unavoidable departures of a(p∗ ) from p∗ , particularly if based on small numbers.] The plot of a(p) against p is the forecaster’s reliability diagram or Calibration curve. Consider, for illustration, the following probability forecasts and outcomes: Outcome, ai 0 0 1 0 1 0 1 Probability, pi 0.4 0.6 0.3 0.2 0.6 0.3 0.4
Outcome, ai 1 1 0 1 0 0 1 Probability, pi 0.5 0.6 0.2 0.6 0.4 0.3 0.5 Table 1 details the calculation of a(p∗ ) for these data. For example, of the three occasions on which a probability forecast of 0.3 was given, just one of the three associated events occurred, yielding a(0.3) = 13 . The reliability diagram for these forecasts and outcomes is plotted in Fig. 1. The diagonal line corresponds, in the absence of sampling fluctuations, to perfect reliability. With such small numbers, there can be little strong evidence against this. Figure 2 displays the reliability diagram for a sequence of 2820 12-hour PoP forecasts issued by a single weather forecaster in Chicago during the period 1972–1976. These results, which are typical of experienced forecasters, clearly indicate excellent reliability.
Figure 1. Reliability diagram corresponding to Table 1.
For assessing multistate forecasts (pi ), with pi = (pi1 , · · · , pik ), one might concentrate on reliability for each state separately, or for all states jointly (by extracting the subset for which pi p∗ ). The latter (‘‘vector reliability’’) is more common, and provides a more demanding test. Another possibility, not generally recommended, is ‘‘scalar reliability,’’ in which the whole set (pis : i = 1, . . . , n; s = 1, . . . , k) is regarded as a single sequence, thus ignoring the identity of the state being forecast. For a continuous quantity X, such as maximum temperature over 12 hours, credible interval forecasting involves specifying an interval (l, u) with the interpretation that Pr(l < X < u) is some fixed preassigned value γ (e.g., 75%). Reliability of a set of such forecasts is equivalent to requiring that a proportion γ of the intervals turn out to contain their specified quantity. If one also requires Pr(X l) = Pr(X u) = 1 − γ /2, say, vector
Table 1. Calibration Calculations Probability, p∗ 0.2 0.3 0.4 0.5 0.6
Number of Instances
Number of 1’s
Relative Frequency of 1’s, a(p∗ )
2 3 3 2 4
0 1 1 2 3
0 0.33 0.33 1 0.75
PROBABILITY FORECASTING
Figure 2. Reliability diagram of a weather forecaster. Reproduced from [24] with the permission of the authors. The number by each point is the number of occasions on which the associated forecast probability was issued.
reliability can be applied to this multistate problem. EMPIRICAL VALIDITY: RESOLUTION Even when the calibration criterion above is satisfied, the forecasts can be clearly unsatisfactory. Suppose that the weather in fact alternates—dry, wet, dry, wet, . . .. Consider two forecasters assigning precipitation probabilities. F1 assigns pi = 12 always, while F2 ’s forecast probabilities alternate 0, 1, 0, 1, . . .. Then both F1 and F2 are well calibrated; but F2 ’s forecasts are perfect, whereas those of F1 are almost useless. Now the assignment of probability forecasts to a sequence of events can be considered as comprising two separate tasks, sorting and labeling [26]. In sorting, the sequence of events is divided into disjoint subsequences, such that all the events in any given subsequence are considered (approximately) equally probable. Labeling then assigns a numerical value to the common probability in each subsequence. Reliability measures how well the labeling has been carried out but does not address the sorting aspect at all. In the example above, F1 and F2 were both
6449
completely reliable, but F2 was clearly the more successful sorter. Another forecaster F3 , who assigns alternating probabilities 1, 0, 1, 0, . . . is also sorting as well as F2 , although the reliability of these forecasts is abominable. In many ways, good sorting is more important than good labeling: F3 ’s forecasts are more valuable than those of F1 once we have obtained enough data to see how to recalibrate them. In effect, ability at sorting depends on substantive (e.g., meteorological) expertise, and ability at labeling on normative expertise (i.e., skill at expressing uncertainty) [34]. The quality of the sorting process, termed resolution, might be judged by the forecaster’s ability to create a set of subsequences in which the relative frequencies are close to 0 or 1 rather than near the overall relative frequency in the whole sequence. However, it is unreasonable, in general, to expect perfect sorting, which is equivalent to absolutely correct (or absolutely incorrect!) categorical forecasting. When, then, can we say that sorting has been carried out successfully? Curtiss [4] suggests that this has been achieved when within each subsequence of events corresponding to some fixed forecast value, the sequence of actual outcomes exhibits no pattern, but is essentially a ‘‘random sequence,’’ or collective in the sense of von Mises [33]. Informally, this means that it should not be possible to extract, from the subsequence, a sub-subsequence in which the relative frequency differs from that in the subsequence. Forecaster F1 above has clearly failed, and F2 and F3 succeeded, by this criterion. Dawid [5] introduced an extended calibration criterion. Consider a subsequence Sn of (1, 2, . . . , n), extracted arbitrarily, subject to the constraint that the decision on whether or not i is to be included in Sn should be determined only by the previous outcomes (a1 , . . . , ai−1 ), not by ai or any later outcomes. Let pn be the average forecast probability, and an the empirical relative frequency, for the events in Sn . Then we require that pn an . This is justified by showing that if the (pi ) are constructed sequentially as appropriate conditional probabilities from a joint distribution for the events (A1 , A2 , . . .), so that pi = (Ai |a1 , a2 , . . . , ai−1 ), then pn − an → 0 except on a set of -probability zero.
6450
PROBABILITY FORECASTING
Consequently, if pn and an are not close enough together, there is a suggestion that, in fact, pn − an →0, which serves to discredit ’s probability assignments. This extended calibration criterion subsumes the earlier one, since the decision to include i in Sn whenever pi is close enough to some preassigned value p∗ depends only on pi , which in turn depends only on (a1 , a2 , . . . , ai−1 ). It also allows us to extract a suitable sub-subsequence of this subsequence and still require matching of forecasts and frequencies, as demanded by Curtiss. A sequence of forecasts which satisfies this criterion for every one of a large enough collection of subsequences (e.g., all computable subsequences) is called completely calibrated. Such a sequence has both perfect reliability and maximum attainable resolution. It is shown by Dawid [6] that if (p1i ) and (p2i ) are each completely calibrated computable forecast sequences, then p1i − p2i → 0(i → ∞). Consequently, this criterion of complete calibration is strong enough to exclude all but one limiting assignment of probabilities, as being empirically invalid. Unfortunately, there is no general way of constructing the asymptotically unique valid probability assignments. In practice, one must allow departures from complete calibration ascribable to sampling variation in finite sequences, so that the criterion is not overrestrictive. Refinement. DeGroot and Fienberg [9,10] introduce a partial ordering between forecasters, which is closely related to comparative resolution. Each combination of forecast p and observed outcome a can be formally regarded as one observation from a bivariate distribution for two random variables, p˜ ˜ From a sequence of such forecasts and and a. their outcomes we can build up an empirical ˜ a): ˜ Q may be bivariate distribution Q for (p, regarded as constructed by randomly selecting one of the occasions in the sequence, with equal probability for each, and setting ˜ a) ˜ to the values (pi , ai ) associated with the (p, selected occasion. Then Q serves as a partial description of the sequence. (To be usable, this empirical distribution may need smoothing or discretisation.) Nature determines the ˜ The forecaster’s empirical distribution of a. input into Q is thus contained in the con˜ given a˜ = 0 and ditional distributions of p,
given a˜ = 1. (For good resolution, these two distributions should be very different.) We can now apply concepts from the theory of comparison of experiments [2]. Forecaster A is sufficient for, or more refined than, forecaster B if B’s conditional distribution for p˜ given a˜ can be obtained by first generating p˜ ∗ from A’s conditional distribution and then generating p˜ from some fixed conditional distribution given p˜ ∗ . In other words, B’s forecast distributions can be produced by passing those of A through a noisy channel, independently of the actual outcome. Consequently, A’s forecasts might be considered more valuable than B’s. ECONOMIC EVALUATION OF FORECASTS Another approach to assessing the validity of a sequence of forecasts p = (p1 , p2 , . . . , pn ) in the light of outcomes a = (a1 , a2 , . . . , an ) is in terms of the economic consequences which would have arisen from using these forecasts to solve a sequence of decision problems. Suppose that before each event Ai is observed, a decision di must be made, leading to loss L(ai , di ) if Ai = ai (we suppose the decision space and loss function independent of i, for simplicity). Using the forecasts above corresponds to taking di = dpi , the decision optimal under Pr(Ai ) = pi , and thus leads to loss S(ai , pi ) on the ith event, where S is the proper scoring rule associated with this decision problem. Comparisons between different forecasters might thus be based on their average scores S = n−1 ni=1 S(ai , pi ). (Since any suitably regular proper scoring rule can be regarded as having arisen from choosing a random cost-loss decision problem, forecaster A will do at least as well as forecaster B for any decision problem if he does so for every simple cost-loss problem.) The average score S can be split into components of independent interest. Thus letting E denote expectation under the (perhaps discretized or smoothed) joint empirical ˜ a) ˜ based on the observed distribution Q for (p, ˜ p˜ = forecasts and outcomes, define ρ(p) = E(a| p) = Q(a˜ = 1|p˜ = p). This is the ‘‘recalibrated’’ version of p, giving the proportion of events occurring when the forecast p is quoted. We ˜ = E(a| ˜ p). ˜ Then S can be introduce r˜ = ρ(p)
PROBABILITY FORECASTING
˜ p)]. ˜ But expressed as E[S(a,
a proportion ρj results in the event occurring. Then
˜ p)] ˜ = E[E{S(a, ˜ p)| ˜ p}] ˜ E[S(a, ˜ p), ˜ p)] ˜ = E[S(˜r, p)]. ˜ = E[S(E(a| Recalling that the directed distance D(p, q) ≡ S(p, q) − S(p, p)( 0), ˜ we obtain S = S1 + S2 , where S1 = E[D(˜r, p)] and S2 = E[S(˜r, r˜ )]. Note that S1 0, with equality (if S is strictly proper) only when ρ(p) ≡ p (almost surely under Q)—that is, when the forecasts are empirically calibrated. Consequently, S1 can be regarded as a penalty score for poor calibration. On the other hand, S2 is just the average score accruing to the recalibrated forecasts. Since S(r, r) is concave in r [27], S2 will tend to be smaller when these are concentrated near the endpoints of [0, 1]. If forecaster A is more refined than forecaster B, A will have a smaller value for S2 than B for any proper scoring rule [10]. Thus S2 may be regarded as a penalty for poor resolution. Schervish [28] points out that A will be more refined than B if and only if the recalibrated forecasts of A perform at least as well as those of B in every simple cost-loss problem. ˜ = Q(a˜ = 1) denote the Now let π = E(a) empirical relative frequency of 1’s in the outcome sequence (called the climatological probability in meteorology). Then ˜ p)] ˜ = E(a) ˜ = π . So E[S(˜r, π )] = E(˜r) = E[E(a| S(E(˜r), π ) = S(π , π ). Hence S2 = E[S(˜r, r˜ )] = E[S(˜r, π ) − D(˜r, π )] = S0 − S3 , with S0 = S(π , π ), S3 = E[D(˜r, π )] 0. Thus S = S0 + S1 − S3 . In this decomposition, S0 is the score attained by the least refined wellcalibrated forecaster, who always gives probability forecast π ; S3 measures the improvement in resolution over these constant forecasts. Again, S1 penalizes poor calibration. As an example, let S be the Brier score, so that S(a, p) = (a − p)2 , D(p, q) = (p − q)2 . Suppose that the probability forecasts are confined to a set (pj : j = 1, 2, . . . , k) of values, and that forecast pj is issued on nj occasions, of which
6451
nS =
k j=1
nj {ρj (1 − pj )2 + (1 − ρj )p2j },
S0 = π (1 − π ) (with nπ =
k j=1
nS1 = nS2 = nS3 =
nj ρj ), k j=1
k j=1
k j=1
nj (ρj − pj )2 , nj ρj (1 − ρj ) nj (ρj − π )2 .
For this case, the decomposition S = S1 + S2 was first obtained by Sanders [26], and the formula S = S0 + S1 − S3 by Murphy [21]. The former decomposition was extended, as above, to general proper scoring rules by DeGroot and Fienberg [10]. The extension of the latter decomposition appears to be new. If a completely calibrated set of forecasts q exists, these will be more refined than any other set p, and lead to the minimum achievable average loss, for any decision problem. For in any subsequence for which pi and qi take on values (approximately) equal, respectively, to some preassigned p∗ and q∗ , the proportion of events occurring will be q∗ . Thus q is sufficient for the pair (p, q) [10], so is more refined than p. The average loss on applying the p’s will be L(q∗ , dp∗ ) L(q∗ , dq∗ ), the average loss for the q’s. Extensions of the decompositions above to multistate forecasts are straightforward, but can be based on either a scalar or a vector approach [20]. Indeed, even in the two-state case, scalar partitions will differ from the foregoing essentially vector partitions [19]. The vector partitions appear more relevant in general. EMPIRICAL EVIDENCE Many studies of actual forecasting performance have been carried out, investigating reliability, average scores, related ‘‘skill scores,’’ and so on. A good review is given by Lichtenstein et al. [13]. The principal finding is that naive forecasters, or experienced
6452
PROBABILITY, FOUNDATIONS OF—I
experts departing even slightly from their usual tasks, find probability assessment difficult and perform poorly, generally showing overconfidence, with ρ(p) much closer to 12 than p. Feedback sessions, in which penalties arising from applying a proper scoring rule are revealed to the forecaster (or docked from his pay!), can sharpen forecasting ability remarkably. With such practice, U.S. weather forecasters, for example, find themselves well able to discern small differences in probability and are very well calibrated. REFERENCES 1. Bernardo, J. M. (1979). Ann. Statist., 7, 686–690. 2. Blackwell, D. (1951). Proc. 2nd Berkeley Symp. Math. Statist. Prob. University of California Press, Berkeley, Calif., pp. 93–102. 3. Brier, G. W. (1950). Monthly Weather Rev., 78, 1–3. 4. Curtiss, J. H. (1968). J. Appl. Meteorol., 7, 3–17. 5. Dawid, A. P. (1982). J. Amer. Statist. Ass., 77, 605–613. 6. Dawid, A. P. (1985). Ann. Statist., 13, 1251–1274. 7. de Finetti, B. (1962). In The Scientist Speculates, I. J. Good, ed. Heinemann, London, 357–364. 8. de Finetti, B. (1975). Theory of Probability (English transl.). 2 vols. Wiley, New York. 9. DeGroot, M. H. and Fienberg, S. E. (1982). In Statistical Decision Theory and Related Topics III, Vol. 1, S. S. Gupta and J. O. Berger, eds. Academic Press, New York, pp. 291–314. 10. DeGroot, M. H. and Fienberg, S. E. (1983). Statistician, 32, 12–22. 11. Epstein, E. S. (1969). J. Appl. Meteorol., 8, 985–987. 12. Good, I. J. (1952). J. R. Statist. Soc. B, 14, 107–114. 13. Lichtenstein, S., Fischhoff, B., and Phillips, L. D. (1982). In Judgment under Uncertainty: Heuristics and Biases, D. Kahneman, P. Slovic, and A. Tversky, eds. Cambridge University Press, Cambridge, England. 14. Lindley, D. V. (1982). Int. Statist. Rev., 50, 1–26. 15. McCarthy, J. (1956). Proc. Natl. Acad. Sci. (USA), pp. 654–655. 16. Murphy, A. H. (1966). J. Appl. Meteorol., 5, 534–537.
17. Murphy, A. H. (1969). J. Appl. Meteorol., 8, 863–873. 18. Murphy, A. H. (1969). J. Appl. Meteorol., 8, 989–991. 19. Murphy, A. H. (1972). J. Appl. Meteorol., 11, 273–282. 20. Murphy, A. H. (1972). J. Appl. Meteorol., 11, 1183–1192. 21. Murphy, A. H. (1973). J. Appl. Meteorol., 12, 595–600. 22. Murphy, A. H. and Epstein, E. S. (1967). J. Appl. Meteorol., 6, 748–755. 23. Murphy, A. H. and Epstein, E. S. (1967). J. Appl. Meteorol., 6, 1002–1004. 24. Murphy, A. H. and Winkler, R. L. (1977). Appl. Statist., 26, 41–47. 25. Pearl, J. (1978). Int. J. Man-Machine Stud., 10, 175–183. 26. Sanders, F. (1963). J. Appl. Meteorol., 2, 191–201. 27. Savage, L. J. (1971). J. Amer. Statist. Ass., 66, 783–801. 28. Schervish, M. J. (1983). A General Method for Comparing Probability Assessors. Tech. Rep. No. 275, Dept. of Statistics, Carnegie-Mellon University, Pittsburgh, Pa. 29. Shuford, E. H., Albert, A., and Massengill, H. E. (1966). Psychometrika, 31, 125–145. 30. Sta¨el von Holstein, C. A. S. (1970). Assessment and Evaluation of Subjective Probability Distributions. Economic Research Institute, Stockholm School of Economics, Stockholm. 31. Thompson, J. C. (1950). Monthly Weather Rev., 78, 113–124. 32. Thompson, J. C. (1952). Bull. Amer. Meteorol. Soc., 33, 223–226. 33. von Mises, R. (1957). Probability, Statistics and Truth. Allen & Unwin, London. 34. Winkler, R. L. and Murphy, A. H. (1968). J. Appl. Meteorol., 7, 751–758. See also METEOROLOGY, STATISTICS IN; SUBJECTIVE PROBABILITIES; WEATHER FORECASTING, BRIER SCORE IN; and WEATHER FORECASTING, EPSTEIN SCORING RULE IN.
A. P. DAWID
PROBABILITY, FOUNDATIONS OF—I Foundations of Probability As A Discipline Any account of the foundations of probability (hereinafter designated F. of P.) must
PROBABILITY, FOUNDATIONS OF—I
be read skeptically. The objectives are to understand and develop probabilistic reasoning. The scale of the effort in the F. of P. is broad, usually conceptual, and generally not concerned with the more narrowly focused technical advances in the familiar numerical probability concept. Efforts in the F. of P. are likely to be critical, expository, and/or argumentative, exhibiting a tradition inherited from the philosophy of science. The F. of P. lacks widely accepted guidelines as to important problems and acceptable solutions; this lack has been hospitable to the development of several schools of thought, each housing a variety of positions. Contributions to the F. of P. come from many disciplines, including philosophy of science, economics, mathematical probability, physics, and engineering, with philosophers of science in the modal but not the majority position. There is no single journal that is a reliable source for current literature. Some journals that occasionally publish articles on the F. of P. are: Annals of Probability∗ , Annals of Statistics∗ , British Journal of the Philosophy of Science, Journal of the Royal Statistical Society∗ (Series B), Synth`ese, Theory and Decision, and Theoria. Mathematical Reviews can be profitably consulted under 60A05. Brief Historical Background While we are likely to learn little to improve the mathematics of numerical probability by consulting the past, history is of importance to the F. of P., for it provides a perspective on the conceptual assumptions that lie at the root of our present-day notions of probability. This perspective can enlighten us as to alternative assumptions, as well as enable us to better perceive the consequences of commitments to particular probability concepts and statistical methodologies. Standard historical works include David [8], Maistrov [34], Todhunter [45], and more recently the work of Hacking [21] and Shafer [43]; these latter two works are of interest for their greater concern with key conceptual issues, although their historical accuracy has been challenged. Probability ideas, as expressed by scholars and not just as implied by gambling∗ or religious practices, can be traced back to
6453
the ancient Egyptians and Greeks and then forward through the Middle Ages (e.g., the writings of Aquinas) to the appearance of recognizably modern probability in the last half of the seventeenth century. Scholarly views up to the onset of the Renaissance held that it did not admit refined degrees and that the basis for a probability assertion was the opinions of qualified individuals. Numerical probability on a scale from 0 to 1 first appears in the seventeenth century (if we omit scattered precursors who were ignored in their time), at a time when a belief in the desirability of mathematical precision is replacing the belief in the possibility of certain knowledge about the world, the latter possibility having been called into question by Cartesian skepticism. At the close of the seventeenth century we find not only numerical probability and expectation but also Bernoulli’s law of large numbers∗ (published posthumously in 1713). The latter supplied the basis for a frequentist view of probability that largely came to supplant the subjective/epistemic view predominant hitherto (see FREQUENCY INTERPRETATION IN PROBABILITY AND STATISTICAL INFERENCE). The eighteenth century saw the introduction of conditional probability∗ and Bayes’∗ formula, great advances in the calculus and limit laws, the elimination of the belief held by Bernoulli∗ that the sum of the probability of an event and the probability of its complement need not be 1, and the beginnings of the scientific use of probability. The use of probability in the physical and social sciences grew greatly in the nineteenth century under the influence of the works of Boltzmann∗ , Laplace∗ , Quetelet∗ , and others, and increasing attention was paid to a frequentist view [48]. Major contributions to the F. of P. in the first part of the twentieth century include the following: 1. A frequentist interpretation by R. von Mises∗ based on a notion of randomness and repeated experiments modeled by the sample space and collective [49] (a posthumous work with references to earlier work) 2. A. N. Kolmogorov’s axiomatization of probability as a normed measure [25]
6454
PROBABILITY, FOUNDATIONS OF—I
3. B. de Finetti’s and L. J. Savage’s development of the subjective∗ view of probability [9–11, 38] 4. R. Carnap’s development of logical probability, as an objective assessment of the degree to which an evidence statement (inductively) supports a hypothesis statement [4] 5. J. M. Keynes’ [24], B. O. Koopman’s [27], and B. de Finetti’s studies of comparative probability. Role of The F. of P. in Statistics The gap between the F. of P. and applied probability∗ and statistics is narrower than might be expected. Knowledge of the different mathematical and interpretive concepts of probability treated in the F. of P. can guide the selection of families of probability models (not necessarily numerical ones) so as to better reflect the indeterminate, uncertain, or chance phenomena being treated, and can clarify a choice among the divergent, conflicting statistical methodologies now current. The relevance of the F. of P. to statistical methodology is established when we realize that the positions of the NeymanPearsonians, Bayesians, structuralists, fiducialists, and maximum entropists are better understood when it is seen that these schools rely on different concepts of probability, albeit this difference is obscured by common agreement on the mathematical structure (excepting countable additivity) of probability. Neyman–Pearsonians postulate that a class of uncertain/indeterminate phenomena (i.e., the ‘‘unknown’’ parameter) cannot be given a probability model. Bayesians/subjectivists in turn insist upon giving this class an overly precise numerical probability model, but allow great latitude in the subjectively based choice of model. Structuralists/fiducialists and maximum entropists carry the modeling process one step further by claiming to provide objective, rational grounds for the selection of a unique numerical probability model to describe, say, the ‘‘unknown’’ parameter. (Barnett [1] provides some of the details of the different statistical methodologies we have mentioned.) The F. of P. enables us to understand the actual claims and
presuppositions of these methodologies (not just the proffered written and verbal explanations of proponents and opponents), and it provides a basis for the improvement and selective utilization of these methodologies. Lest the reader misunderstand, the F. of P. is far from a panacea for our statistical and modeling ills. It is an undeveloped subject that, like philosophy, will always promise more than it delivers. Yet what it does deliver is essential to the development of statistical methodology. Outline of Contents Our presentation is calculated to broaden the reader’s view by first detaching him or her from the notion that probability is only numerical (see AXIOMS OF PROBABILITY) and then by supplying some examples of nonnumerical probability concepts. It is also our intent to expose the reader to a variety of interpretations that are available for probability. In the following section, we comment on probabilistic reasoning, the subject of the F. of P., locate it with respect to science, mathematics, and rationality, and observe that a probability concept lies at the center of probabilistic reasoning. The third section introduces a five-element framework for analyzing and presenting a probability concept. Elements of the framework are illustrated by reference to several current concepts of probability, with details reserved to the references. Most entries in the reference list pertain to specific theories of probability. A broader coverage is available in Black [3], Fine [13], and Kyburg [29]. The references should also assist the reader to establish his or her own perspective on the F. of P. PROBABILISTIC REASONING The F. of P. has probabilistic reasoning as its subject. This is a mode of rational thought, engaged in by diverse entities having various aims. Furthermore, rationality itself evolves in time, varies with the reasoner and with the goals of the reasoner [35]. The diversity of conditions under which probabilistic reasoning is engaged in makes a prima facie case for a parallel diversity in its forms.
PROBABILITY, FOUNDATIONS OF—I
Probabilistic reasoning, like rationality, has the status of a methodology and as such is neither an empirical discipline nor a branch of deductive or mathematical logic. Probability is neither contained within science nor within mathematics; it is in fact part of the methodology of science, although the appropriateness of specific probability models for categories of empirical phenomena is the subject of scientific investigations. Probability is no more a branch of mathematics than is physics, although it owes a great debt to mathematics for its formulation and development. Hence neither science nor mathematics can be relied on to confirm or refute the broad probabilistic ideas that are the concern of the F. of P. This status makes it difficult to determine generally agreed-on canons of acceptability for proposed or familiar modes of probabilistic reasoning, and it partially explains the controversial nature of the F. of P. A concept of probability is the armature around which one constructs a form of probabilistic reasoning. We thus turn to present a framework for the classification and analysis of probability concepts and aspects of the associated system of probabilistic reasoning. FRAMEWORK FOR THE ANALYSIS OF PROBABILITY CONCEPTS We identify five elements (reasoner, domain of application, semantics/interpretation, syntax/mathematical structure, and working basis) that form a framework for the analysis and selection of a probability concept. The Reasoner The first element is the entity having aims or goals that lead it to engage in probabilistic reasoning. Examples of such entities, hereafter referred to as reasoners, are: 1. Individual 2. Organization (e.g., a business planning ahead, a family group) 3. Society (e.g., the inductive reasoning and rhetoric of the medical profession differs from that of the legal profession, which in turn differs from that of engineering).
6455
Goals can include: 1. Summarization of data 2. Understanding (e.g., inference in science, generation/selection of a model) 3. Choice of action (e.g., decision making by an individual or formal organization) 4. Communication (e.g., between individuals or between an individual scientist and his or her colleagues, say). One also needs to take into account the knowledge and information available to the reasoner and the limits to the ‘‘intelligence’’ of the reasoner (bounded rationality). Domain of Application The domain of application over which the reasoner seeks some mastery is a milieu isolated from its surroundings by the reasoner’s interest in the occurrence of events (we shall use ‘‘events’’ to refer both to events as commonly conceived and to propositions describing events) in some collection and by the reasoner’s, perhaps tacit, identification of an available setting or background that has a bearing on the collection of events. Analysis of the domain guides the choice of an appropriate form of probabilistic reasoning; such analyses can point the way to the need for new theories. Examples of backgrounds or settings drawn from the logical, mental, and physical realms include: 1. Logical. Formally presented information corpus 2. Mental. Individual’s or group’s state of knowledge or belief 3. Physical. Category of empirical phenomena having nearly identical characteristics Descriptors associated with the foregoing setting are, respectively: 1. Indeterminate 2. Uncertain 3. Chance or aleatory. The term epistemic is used to jointly refer to indeterminate and uncertain settings; it
6456
PROBABILITY, FOUNDATIONS OF—I
also often suggests that the issue is one of inductive reasoning from the setting to the events, with probability inhering in the link between the two. Aleatory, on the propensity account, suggests that the reasoner is concerned with a physical property of some empirical phenomenon and not just with his or her knowledge about this property; probability inheres in the phenomenon itself. It is also common for ‘‘chance’’ or ‘‘aleatory’’ to refer to a summarization of empirical data through relative frequencies of occurrence. Further classification of domains can be made along the ‘‘subjective’’/‘‘objective’’ axis. Objective domains are those that are in principle available to more than one individual and that can be described independently of the particular observer. Subjective domains are to a significant extent private and cannot be described without reference to a particular individual. In practice, one rarely encounters pure cases of chance, indeterminate, uncertain, objective, or subjective domains (e.g., consider the bases for the choice of a reference class for a frequentist probability model). Examples of domains of Types 1 and 3 often differ more in the view of probability than in the phenomena themselves. For example, the events of interest may concern the outcome of the nth toss of a particular coin when we have recorded the results of the first (n − 1) tosses. The explicit corpus would then be the record of (n − 1) tosses and perhaps some generalities about coins. One can then construe probability either as a physical property of the coin, with the corpus providing relevant measurement data, or one can construe probability as representing the linkage or relation between the given record of (n − 1) tosses and background on coins and the nth toss; the first case is what we mean by type 3 and the second case is of Type 1. Examples of Type 2 domains are common in the subjectivist/personalist/Bayesian decision-making area [11,18,23,33,38,39]. An individual confronted with a choice of actions introspectively and seriously weights his or her beliefs or strength of conviction as to the probabilities of the consequences attendant on each choice of action. He or she then assesses the expected utility∗ for each action and decides on an action with maximal expected utility. It is an important feature of
the subjective account that there need not be formal knowledge of the contents or even of the extent of the domain. Semantics/Interpretations A domain thus contains both a collection of events whose occurrences are of interest to a reasoner and a setting identified by the reasoner as informative about the occurrence of events and as relevant to achieving its goals. In some fashion, the reasoner decides that it can perhaps identify which of the events are, say, probable, or which events are more probable than other events, or even assign a numerical probability to each event. Implicit in this process is an initial determination as to what provides the evaluative basis for the probability concept being invoked (e.g., what weather data and theory can we use to calculate the probability of rain tomorrow). The evaluative basis largely, but not completely, fixes the meaning of the probability concept, which must have meaning extending beyond its evaluative basis if it is to serve a role other than that of data summarization. Nevertheless, we discuss semantics in terms of this basis, for it provides the least controversial access to elements of meaning. Possible evaluative bases for probability are: 1. Past occurrences of other events of the same type 2. The phenomenon/experiment generating the events 3. The ‘‘state of mind’’ of the reasoner as exhibited by his or her behavior 4. The inductive linkage or relation between the setting and the events. The interpretation of probability most closely associated with Basis 1 is that of converging relative frequencies of occurrence for each event A in the collection; typically, the domain type is 3 (see FREQUENCY INTERPRETATION IN PROBABILITY AND STATISTICAL INFERENCE). Reflection on Basis 1, though, reveals that it is less understandable than it seems at first glance. One possibility is that probability is just descriptive of actual occurrences without regard to the random experiment or setup generating the events. Typically, the descriptive approach is elaborated in a frequentist context in terms of the proportion of
PROBABILITY, FOUNDATIONS OF—I
occurrences of an event in repeated performances of a random experiment. However, probability as a description of actual occurrences is an easily discredited view; we have no grounds for determining such a probability in advance of the actual occurrences. Hence, the common frequentist view of proabability is an ensemble one that carefully refrains from committing itself to what will actually occur. Generally, one would ask of a property that it have known relations to other properties and terms. Yet, if probability is just a property of occurrences of events, and not say, of the underlying random experiment, then what is our basis for determining the value of probability, and how is it related to other probabilities and properties? Basis 2 is a somewhat clearer basis for, say, the common converging relativefrequency-of-occurrence interpretation (see Giere [15], Gillies [17], and Mellor [36]). A random phenomenon or experiment is now said to have a ‘‘physical’’ characteristic: namely, its disposition, propensity, or tendency to produce, say, A with probability Pr(A), or its greater tendency to produce A than to produce B, and this characteristic manifests itself through occurrence rates for the events that are the outcomes of the performance of the experiment; i.e., the propensity is large that the observed relative frequency of event A will be close to the propensity of the experiment to produce A. Probability on this account is a property of the apparatus and not of its output, with implications for the outputs that make it easy to travel the road to Basis 1. Basis 3 is the one held by proponents of the subjectivist/personalist/Bayesian interpretation (see BELIEF, DEGREES OF) as well as by some holders of a subjective epistemic theory [16,32,42]. Subjective probability describes the strength of belief of the reasoner (an individual) concerning the truth of propositions or occurrence of events. Strength of belief is determined through a process of introspection and manifests itself through overt choice/betting behavior. Basis 3 might also include Toulmin’s conception of probability as guarding or qualifying an assertion [46]. Although Toulmin’s interpretation has few adherents, it could
6457
have a role in communications; an individual can signal reservations about a statement in a graded manner that can, perhaps, be made interpersonally meaningful. Basis 4 is maintained by proponents of objective epistemic theories [4,5,7,30], who hold that the probability of an event is relative to a given background corpus (domain Type 1), and it represents the rational degree of inductive support lent by the corpus to the truth/occurrence of the event. It is perhaps also held by some subjective epistemic theorists (e.g., Shafer [42]). Semantic position 4 and domain Type 1 also seem to be appropriate as locations for the following: computational complexity-based induction methods and universal probability constructions proposed by Kolmogorov [26], Solomonoff [44], and Chaitin [6] and developed also by Schnorr [40] and others; Fisher’s fiducial inference∗ (see Seidenfeld [41]); Fraser’s structural inference∗ [14]; and some treatments of likelihood-based statistical support [2,12,20]. At any rate, fiducial and structural inference do not seem to directly admit frequentist or subjectivist interpretations. Other issues in the consideration of the semantics of probability include those aspects of the domains referred to in the section ‘‘Domain of Application’’ by the descriptions ‘‘indeterminate,’’ ‘‘uncertain,’’ and ‘‘chance,’’ as well as by the objective/subjective classifications. Furthermore, there is the generally overlooked issue of the extent to which distinctions or degrees of precision are meaningful. The prevalent commitment to numerical probability has been supported by, and in turn supports, a view that at least in principle, degrees of indeterminacy, uncertainty, and chance∗ can be arbitrarily finely graded. This is a remarkable hypothesis having little justification except that of its widespread acceptance. We can address this issue more clearly when we turn to the mathematical structures we adopt for probability. Syntax/Mathematical Structure
Viewpoint of Measurement Theory. Once the reasoner has adopted a concept of probability supported by the domain of application, he or she then wishes to move this system into a formal mathematical domain so as better
6458
PROBABILITY, FOUNDATIONS OF—I
to determine the implications of the position. From the viewpoint of the theory of measurement [28] the domain and the selected probability concept form an ‘‘empirical relational system.’’ The reasoner homomorphically maps this system into a ‘‘mathematical relational system.’’ The collection of events becomes a collection A of subsets A, B, C, . . ., of a sample space . Relations are then identified between elements of the mathematical domain A that homomorphically represent the relational properties holding between events, and these relations characterize the adopted probability concept. For example, if the reasoner has only classified events as to whether or not they are probable, this is represented by partitioning A so as to specify those sets that are probable. If the reasoner has instead considered binary comparisons of the likelihood of events, this is > represented by a binary relation ∼ such that > A ∼ B represents the claim ‘‘the event represented by A is at least as probable as the event > represented by B.’’ One might expect that ∼ is an order relation for a reasonable probability concept. If, finally, the reasoner holds that his or her goals and setting justify a numerical assignment, he or she may adopt the familiar probability measure representation µ : A → [0, 1], with µ(A) µ(B) if and only if A represents an event at least as probable as the event represented by B. A numerical probability assignment usually also carries the implications that such statements as ‘‘A is x times as probable as B’’ and ‘‘A is more probable than B by x’’ are meaningful in the empirical domain. It is common in discussions of the meaning of probability to reverse the process we have described and start with a well-defined mathematical structure for a probability concept. One then attempts to coordinate the given mathematical structure with an application domain through an interpretation so as to give meaning to what is otherwise merely a collection of symbols. The possible interpretations are constrained by the axiomatic structure of the probability concept but are not determined by it. (See AXIOMS OF PROBABILITY.) This reverse process has
seemed reasonable only because of the prevalent belief that the only mathematical structure for a probability concept is that of Kolmogorov for numerical probability [25]. We hope that our discussion of syntactical variety in the section ‘‘Hierarchy of Probability Concepts’’ will encourage the study and use of new mathematical structures for probability. Event Collections. The events of interest in the domain are represented either by sets or by propositions. The collection of events is typically assumed to generate a Boolean algebra or σ -algebra of sets or propositions, although there is some interest, largely motivated by quantum logic, in lattices∗ . Too little attention has been paid to the mathematical structure of event collection representations [37]. It is generally not possible to enumerate all possible events (complex systems occasionally surprise us by behaving in unforeseen ways), and therefore the sample space is at best a list of practical possibilities. (The device of logically completing the space by introducing a new event, ‘‘everything else,’’ has problems of its own.) Furthermore, it has been argued in discussions of quantum logic [22] that representations of event collections need not be closed under the usual Boolean operations. Nevertheless, almost all work in the F. of P. presumes at least a Boolean algebra of sets or propositions. Hierarchy of Probability Concepts. The recognition that probabilistic reasoning must confront a wide range of domains and levels of information, knowledge, belief, and empirical regularity can lead us to an acceptance of an hierarchy of increasingly precise mathematical concepts of probability. This hierarchy has been little explored, as almost all of the effort, even in the F. of P., has been devoted to the acknowledgedly vital case of familiar numerical probability. That numerical probability may be inadequate to the full range of uses of probabilistic reasoning is suggested by the following observations: 1. There are categories of empirical phenomena (e.g., speech and picture information sources, earthquakes, weather) for which there is no obvious stability of relative frequency for all events of interest.
PROBABILITY, FOUNDATIONS OF—I
2. An information corpus may lack information; the resulting indeterminacy should be respected and not papered over by dubious hypotheses (e.g., ‘‘If you know nothing about the parameter, then adopt a uniform or maximum entropy∗ prior for it.’’). 3. Self-knowledge of individuals is intrinsically limited, and attempts to force belief or conviction to fit the mold of a particular ‘‘rational’’ theory can only yield results of unknown value. An attempt to accommodate to the preceding observations leads to the following hierarchy of concepts: 1. 2. 3. 4.
‘‘Possibly A.’’ ‘‘Probably A’’ or ‘‘not improbably A.’’ ‘‘A is at least as probable as B.’’ ‘‘A has interval-valued probability (P(A), P(A)’’ where ‘‘P(A)’’ is called the lower probability of A and ‘‘P(A)’’ is the upper probability of A. 5. ‘‘A has probability x.’’
Conditional versions of each of the foregoing concepts are also available. The use of a statistical hypothesis (family of probability measures) to specify our knowledge of the domain phenomena can lead to upper and lower probabilities generated by the upper and lower envelopes of the family of measures; a special case of such hypotheses are the so-called ‘‘indeterminate probabilities’’ [31,32]. It is shown in Walley and Fine [50] that the subclass of concept 4 called belief functions∗ can serve to represent concept 3, which in turn can represent concept 2. Furthermore, the belief functions are a subclass of lower probabilities and are therefore representable by them. These representations (e.g., ‘‘probably A’’ iff ‘‘A is at least as probable as not A’’) are generally nonunique but establish the hierarchy of concepts. The probability concepts just introduced must then be given structure through a set of axioms and definitions of significant terms (e.g., independence or unlinkedness, expectation). The most familiar axiom set is that given by Kolmogorov for concept 5. Some discussion of axioms for the other concepts is
6459
given in Walley and Fine [50]. Here we will only present the basic axioms for lower and upper probabilities [19]. Let A be an algebra of subsets of a set and let A, B denote elements of A. A1. P(A) 0. A2. P() = 1. A3. If A ∩ B = Ø (empty set), then P(A) + P(B) P(A ∪ B) (superadditivity) P(A) + P(B) P(A ∪ B) (subadditivity). A4. P(A) + P(Ac ) = 1 (Ac is the complement of A). Elementary consequences of these axioms include: P(Ø) = P(Ø) = 0. P(A) P(A). P() = 1. A ⊃ B implies that P(A) P(B), P(A) P(B). 5. If A ∩ B = Ø, then P(A ∪ B) P(A) + P(B) P(A ∪ B).
1. 2. 3. 4.
If P ≡ P or if in A3 either superadditivity or subadditivity is replaced by additivity, A1 to A4 reduce to the Kolmogorov axioms for the finite case. In one is interested in algebras containing infinitely many events, the basis axioms would need to be supplemented. In fact, there is almost no work in the F. of P. that proceeds solely in terms of the basic axiom set we have given (see BELIEF FUNCTIONS). Working Basis While it is the role of interpretation to coordinate the mathematical, axiomatically constrained concept with the domain of events of interest to the reasoner, this coordination is typically idealized and not itself a working basis for probabilistic reasoning. Statistics is the discipline that supplies the working basis for numerical probability with a frequencyof-occurrence interpretation. Statistics is also of value in supplying the basis for numerical probability in the personalist/subjectivist setting. In this case there is also a literature concerned with the actual capabilities of
6460
PROBABILITY, FOUNDATIONS OF—I
humans to supply probabilities and consistently/rationally/coherently use probabilities to choose actions [47]. Any truly rational theory of partial belief or conviction must take account of the capabilities of the agent in question. Psychology and business school students of these capabilities have shown that human performance is somewhat at variance with the idealized requirements of current personal probability theories. Little is known at present about the practical issues connected either with formal concepts of probability other than the numerical one or with interpretations of the objective epistemic variety. Walley and Fine [51] outline the elements of a frequentist basis for interval-valued probabilities. Acknowledgment Preparation of this article partially supported by National Science Foundation Grant SOC-7812278.
REFERENCES 1. Barnett, V. (1973). Comparative Statistical Inference. Wiley, New York. 2. Birnbaum, A. (1977). Synthese, 36, 19–49. 3. Black, M. (1967). Probability. In The Encyclopedia of Philosophy, Vol. 6, P. Edwards, ed. Macmillan, New York, pp. 464–479. 4. Carnap, R. (1962). The Logical Foundations of Probability, 2nd ed. University of Chicago Press, Chicago. 5. Carnap, R. and Jeffrey, R. C. (1971). Studies in Logic and Probability, Vol. 1. University of California Press, Berkeley, Calif. 6. Chaitin, G. J. (1969). J. ACM, 16, 145–149. 7. Cohen, L. J. (1977). The Probable and the Provable. Clarendon Press, Oxford. 8. David, F. N. (1962). Games, Gods and Gambling. Charles Griffin, London. 9. de Finetti, B. (1931). Fundam. Math., 17, 298–329. 10. de Finetti, B. (1937). Foresight: its logical laws, its subjective sources, transl. in H. Kyburg and H. Smokler, eds., Studies in Subjective Probability. Wiley, New York, 1964, pp. 93–158. 11. de Finetti, B. (1974). Theory of Probability, Vols. 1, 2. Wiley, New York. 12. Edwards, A. W. F. (1973). Likelihood. Cambridge University Press, Cambridge.
13. Fine, T. L. (1973). Theories of Probability: An Examination of Foundations. Academic Press, New York. 14. Fraser, D. A. S. (1968). The Structure of Inference. Wiley, New York. 15. Giere, R. N. (1976). In Foundations of Probability Theory, Statistical Inference and Statistical Theories of Science, Vol. 2, W. L. Harper and C. A. Hooker, eds. D. Reidel, Dordrecht, Holland, pp. 63–101. 16. Giles, R. (1976). In Foundations of Probability Theory, Statistical Inference and Statistical Theories of Science, Vol. 1. W. L. Harper and C. A. Hooker, eds. D. Reidel, Dordrecht, Holland, pp. 41–72. 17. Gillies, D. A. (1973). An Objective Theory of Probability. Methuen, London. 18. Good, I. J. (1950). Probability and the Weighing of Evidence. Charles Griffin, New York. 19. Good, I. J. (1962). In Logic, Methodology and Philosophy of Science, E. Nagel, P. Suppes, and A. Tversky, eds. Stanford University Press, Stanford, Calif., pp. 319–329. 20. Hacking, I. (1965). The Logic of Statistical Inference. Cambridge University Press, Cambridge. 21. Hacking, I. (1975). The Emergence of Probability. Cambridge University Press, Cambridge. 22. Jauch, J. M. (1968). Foundations of Quantum Mechanics. Addison-Wesley, Reading, Mass. 23. Jeffrey, R. C. (1965). The Logic of Decision. McGraw-Hill, New York. 24. Keynes, J. M. (1921). A Treatise on Probability, reprinted 1962. Harper and Row, New York. 25. Kolmogorov, A. N. (1933). Grundbegriffe der Wahrscheinlichkeitrechnung. Transl. by N. Morrison, Foundations of the Theory of Probability, 2nd ed. Chelsea, New York, 1956. 26. Kolmogorov, A. N. (1965). Problemy peredachii informatsii, 1, 4–7 (Soviet journal translated as Problems of Information Transmission). 27. Koopman, B. O. (1940). Bull. Amer. Math. Soc., 46, 763–774. 28. Krantz, D. H., Luce, R. D., Suppes, P., and Tversky, A. (1971). Foundations of Measurement, Vol. 1. Academic Press, New York. 29. Kyburg, H. E., Jr. (1970). Probability and Inductive Logic. Macmillan, Toronto. 30. Kyburg, H. E., Jr. (1974). The Logical Foundations of Statistical Inference. D. Reidel, Dordrecht, Holland. 31. Levi, I. (1974). J. Philos., 71, 391–418.
PROBABILITY, FOUNDATIONS OF—II 32. Levi, I. (1980). The Enterprise of Knowledge. MIT Press, Cambridge, Mass. 33. Lindley, D. (1965). Introduction to Probability and Statistics from a Bayesian Viewpoint, Vols. 1, 2. Cambridge University Press, Cambridge. 34. Maistrov, L. E. (1974). Probability Theory: A Historical Sketch, 2nd ed., S. Kotz, trans. Academic Press, New York. 35. March, J. (1978). Bell J. Econ., Autumn, 587–608. 36. Mellor, D. H. (1971). The Matter of Change. Cambridge University Press, Cambridge. 37. Randall, C. H. and Foulis, D. J. (1976). In Foundations of Probability Theory, Statistical Inference and Statistical Theories of Science, Vol. 3, W. L. Harper and C. A. Hooker, eds. D. Reidel, Dordrecht, Holland. 38. Savage, L. J. (1954). The Foundations of Statistics. Wiley, New York. 39. Savage, L. J. (1962). The Foundations of Statistical Inference: A Discussion. Wiley, New York. 40. Schnorr, C. P. (1971). Zufalligskeit und Wahrscheinlichkeit. Springer-Verlag, Berlin. 41. Seidenfeld, T. (1979). Philosophical Problems of Statistical Inference. D. Reidel, Dordrecht, Holland. 42. Shafer, G. (1976). A Mathematical Theory of Evidence. Princeton University Press, Princeton, N. J. 43. Shafer, G. (1978). Arch. History Exact Sci., 19, 309–370. 44. Solomonoff, R. J. (1978). IEEE Trans. Information Th. IT-24, 422–443. 45. Todhunter, I. (1865). A History of the Mathematical Theory of Probability: From the Time of Pascal to That of Laplace. Reprinted by Macmillan, London, 1949. 46. Toulmin, S. E. (1958). The Uses of Argument. Cambridge University Press, Cambridge. 47. Tversky, A. and Kahneman, D. (1974). Science, 185, 1124–1131. 48. Venn, J. (1888). The Logic of Chance, 3rd ed. Macmillan, London. 49. von Mises, R. (1964). Mathematical Theory of Probability and Statistics, ed. and complemented by H. Geiringer. Academic Press, New York. 50. Walley, P. and Fine, T. L. (1979). Synthese, 41, 321–374. 51. Walley, P. and Fine, T. L. Statist., 10, 741–761.
(1982).
Ann.
6461
See also AXIOMS OF PROBABILITY; BAYESIAN INFERENCE; BELIEF FUNCTIONS; CAUSATION —I; CHANCE —I; BELIEF, DEGREES OF; FREQUENCY INTERPRETATION IN PROBABILITY AND STATISTICAL INFERENCE; LOGIC OF STATISTICAL REASONING; PROBABILITY, HISTORY OF; and PROBABILITY THEORY: AN OUTLINE.
TERRENCE L. FINE
PROBABILITY, FOUNDATIONS OF—II MathSci, the online version of Mathematical Reviews, lists over 1000 items (several with duplications) under foundations of probability and statistics (AMS codes 60A05, 62A99) in the period 1980–1996, roughly covering the time since the original appearance of PROBABILITY, FOUNDATIONS OF —I in vol. 4 of this Encyclopedia. These papers can be grouped under the headings: Bayesian/subjectivist Decision theory/utility Logical, inductive, knowledge representation Conditioning and updating Complexity-based randomness and frequentist Quantum-mechanical issues Formalist/mathematical—finite additivity, upper—lower Dempster—Shafer belief functions Objectivist and not the above Fuzzy and possibilistic As probability is central to statistics, issues in the foundations of probability are also issues in the foundations of statistics and have potential, when sufficiently matured, to change the conduct of statistics. The brevity of this update and the wide variety of approaches listed above dictate selectivity in our presentation. Our choices reflect our tastes and our judgement as to those issues in the foundations of probability that have developed to where they can have an influence on statistics. We focus below on Kolmogorov complexitybased approaches to randomness* and
6462
PROBABILITY, FOUNDATIONS OF—II
simplicity of explanation, the Bayesian program, and the alternative nonadditive probability model of interval-valued or upper—lower probability. Kolmogorov complexity has provided the strongest focus on frequentist phenomena, and it and intervalvalued probability depart the farthest from the conventional circle of ideas. The Bayesian program, largely subjective but significantly objective as well, is in the conventional center and among the most readily applicable of developments.
FREQUENTIST DEVELOPMENTS BASED UPON COMPLEXITY Defenses and explications of frequentist approaches to probability have included those based on Kolmogorov complexity (see below), the prequential* probability approach of Dawid [9] and Vovk [43], and uses of interval-valued probability. Perhaps the most penetrating efforts to elucidate an objective, frequentist-related conception of probability and to explicate a logical mathematical basis for induction have come from what is now called Kolmogorov complexity although it was founded independently in the 1960s by A. N. Kolmogorov, R. J. Solomonoff, and G. Chaitin [6]. This pioneering work, reported in Fine [12] (see COMPLEXITY), was then pursued by the three founders (that Kolmogorov took it seriously is evident [1]) as well as by a number of other researchers in the 1970s (e.g., P. Gacs, L. Levin, C.P. Schnorr) and developed in detail. The major themes, of which we address only the first, were: a computational complexity basis for determining the randomness of an individual finite or infinite sequence; construction of a universal prior probability* which, when combined with Bayes’ Theorem*, provides an objective foundation for induction; an algorithmic basis for the notion of information. For an excellent overview of this area with a comprehensive citation of results see Li and Vitanyi [26]; a brief survey is given
by Shen [39]. The complexity program has enabled us to understand issues of randomness, induction, and information that have hitherto been opaque. The major flaw in this program has been the intrinsic ineffective computability of the ideas put forward, rendering them not applicable in statistics. Rissanen’s revitalization of the principle of minimum description length* renders accessible to statistics some of the key concepts developed in the complexity program. The notion of randomness∗ is explicated through a notion of the complexity K(x) of, say, a binary-valued string x ∈ {0, 1}∗ , the set of all finite length strings. Let l(x) denote the length of the string x. The complexity K(x) is taken to be the length l(p) of the shortest program p (another finite length, binary-valued string) that can precisely reproduce x when input to an appropriate universal computing machine U (e.g., a universal Turing machine capable of emulating the computation of any other Turing machine or partial recursive function): K(x) = min{l(p) : x = U(p)}. The important proviso added in the 1970s was that such machines be self-delimiting, in that they do not allow a program p that leads to a defined output (the machine eventually halts to signal the completion of the calculation) to be the prefix, i.e., initial segment, of another such program q yielding a different result. Since we expect random sequences to be incompressible, we would expect that K(x) ≈ l(x); one cannot do much better than a 1 : 1 transformation of x itself to represent x. One could then show that the class of infinite length random strings could be well defined in terms of the complexitybased randomness of all of their initial segments. More precisely, if x is an infinite length string and x1:n its initial segment of length n, then we have the following definition.
Definition: Random Sequence. The infinite random sequence x is random if and only if for K a universal prefix complexity measure (∃c finite)(∀n)K(x1:n ) > n − c.
PROBABILITY, FOUNDATIONS OF—II
In effect, an infinite sequence is random if and only if all of its initial segments are also random in a precise sense. The necessary fly in this ointment is that the complexity measure K cannot be effectively computable (recursive). In fact, K is coenumerable, meaning that it can be effectively approximated only from above; e.g., if U(p) = x then K(x) l(p). There is an interesting axiomatic complement to this approach [40,41]. By allowing defects in incompressibility (deviations of initial segment complexities from the maximal) one can develop families of ‘‘random’’ sequences that have some, but not all of the properties traditionally associated with full randomness. One of the easiest properties to satisfy is that of the strong law of large numbers, and this is taken to hold for any concept worthy of being called random. The property that infinitely often a sequence has an equal number of 0’s and 1’s holds when the complexity deficiency is less than or equal to ( 21 − ) log n, but not for a deficiency of (2 + ) log n [42]. For the familiar law of the iterated logarithm∗ , however, the deficiency cannot exceed log n, but can exceed o(log log n). A reasonable recasting of the complexity approach in an effective and statistically plausible form uses the minimumdescription-length principle∗ (MDL), reinvigorated by the 1980s work of Rissanen in stochastic complexity* [33,34,35] (see also Dawid [9]). Rissanen replaces encoding by short programs that will produce the desired sequence on a universal self-delimiting partial recursive function by computation or encoding of data X (finding a short program) with respect to a parametrized family of probability measures {Pθ : θ ∈ } that need not contain a ‘‘true’’ model generating X. The key is the well-known result (Huffman coding) that, given a probability mass function P(x) for x ∈ χ , we can encode, in a prefix-free manner, the elements of χ using codewords of lengths nearly − log P(x). One way to approximate the MDL calculation is first to estimate (e.g., via maximum likelihood) the parameter θ from X, encoding it as θn with a precision depending upon the sample size n. Then one encodes the sample X using the model Pθn , resulting in an overall short description of the pair X, θn .
6463
BAYESIAN APPROACH Focus The past fifteen years has seen a significant number (about 100 cited in Mathematical Reviews) of articles and books concerned with the Bayesian approach, primarily in its subjective interpretation (see SUBJECTIVE PROBABILITIES; BELIEF, DEGREES OF), but also regarding objective methods for determining priors (e.g., maximum entropy∗ methods), and studies of finitely additive probability, motivated by de Finetti. The needs of artificial intelligence for knowledge representation and processing have been an interesting driver for developments in the Bayesian approach [11,23,27,32,38]. Three developments that stand out are selection of priors; rendering feasible the multidimensional integrations required by Bayesian calculations to evaluate posteriors and parameter estimates through expectations; the dogma of precision and the attempts to more realistically characterize the nature of prior knowledge by intervalvalued alternatives to measures. Choosing Priors The Bayesian program is of no value unless one can specify the required prior probability laws. Carnap made early efforts to identify the unique rationally correct Bayesian prior that would drive his program of logical probability. A more fundamental attack was made by Solomonoff using computational complexity ideas to establish a universal prior [26,8]. Jaynes [18] and his followers, using the socalled maximum entropy principle∗ , have constructed unique priors that have been the basis of useful applications such as estimation of power spectra. You are urged to take as a prior the one having the maximum entropy, subject to the constraints reflecting one’s prior knowledge (e.g., of certain moments). Other approaches with a significant following in the Bayesian community include the use of conjugate and reference priors [2,4].
6464
PROBABILITY, FOUNDATIONS OF—II
Nor can we rest content with toy problems in which priors are assigned to a handful of parameters. Bayesian approaches to applications such as neural networks∗ [27,29] have forced a confrontation with the need to choose complex priors to describe very many parameters (typically in hundreds or thousands). The difficulty of constructing priors on spaces of infinitely many outcomes (even those as small as the set of integers) is addressed by Diaconis and Freedman [10]. They establish that posterior distributions∗ based upon priors that indeed support the ‘‘true’’ underlying distribution need not converge to this distribution. A consistent prior must be chosen with care for the Bayesian program to yield sensible results. The Bayes methodology, while enjoying good properties (e.g., admissibility* and consistency), is peculiar, in that it requires the user to postulate a prior distribution that is basically as complex as the quantities being inferred, if not more so. One must assume at least as much as one wishes to learn! Practical approaches finesse this difficulty either by adopting relatively simply parametrized families of priors (e.g., jointly normal distributions over parameters) or by the device of hyperparameters∗ . A relatively small number of the latter specify a distribution that in turn randomly selects the actual prior. However, there remains the chronic difficulty of rooting the choice of prior in the prior knowledge of the user. Discussion of the essentials and philosophy of the Bayesian position has been plentiful (see BELIEF, DEGREES OF), and here we note only refs. 2, 3, 4, 5, 19, 32, 44, 48. Calculating Posterior Parameter Estimates Calculations of posterior estimates of parameters, given likelihoods and priors, often involve high-dimensional integrations, and the resulting calculations cannot be carried out analytically. Those who feel a need for analytical calculations are constrained to take especially simple (e.g., conjugate) priors that may have little relationship to any prior knowledge. Failing analytical methods, we can estimate a parameter as an expected value of a suitable random variable with respect to
a posterior distribution if we can generate efficiently a large number of (independent, if practicable) samples drawn from this distribution. The expectation is then estimated as an arithmetical average of these results. Thus there has been interest in Monte Carlobased calculations suited to computers. An original discussion of the computation of posterior expectations is provided by Neal [29] in the context of applications to neural network design. Neal introduces a hybrid Markov chain Monte Carlo∗ calculational method drawn from elementary particle physics [29, Chap. 3]. The resulting samples come from a Markov chain, and thus are dependent, but the chain has the desired limiting stationary distribution. The presence of dependence increases the number of samples that need to be taken and may require a large number of iterations of the Markov chain before a good approximation to the limiting distribution appears. While calculations with complex models then become possible, computation times are still enormous. The Dogma of Precision The Bayesian program suffers from what has been called the dogma of precision [24; 44, Section 5.7] The desire to avoid incoherence (e.g., Dutch books) is claimed to force prior knowledge into a representation by a precise probability distribution. That beliefs and our personal knowledge are hardly ever of this kind is evident upon introspection. While the Bayesian position enjoys significant properties of rationality, none of this is to our good if it can only be maintained through self-deception. A welldeveloped alternative to precise probabilistic knowledge, which yet preserves a strong notion of coherence∗ , is available from Walley’s theory of prevision∗ [44], and it is noted below. INTERVAL-VALUED PROBABILITY Classification By interval-valued or upper—lower probability we refer to nonadditive mathematical models of probability that differ significantly from the familiar real-valued probability that
PROBABILITY, FOUNDATIONS OF—II
is either countably additive (in the celebrated formulation by Kolmogorov) or finitely additive (as espoused by de Finetti and many who are his disciples) (see also NONADDITIVE PROBABILITY). The probability of an event A is now represented by the pair [P(A), P(A)], where P(A) P(A). Since all of these theories share the relationship P(A) = 1 − P(Ac ), it is possible to proceed in terms of just the lower probability P. Interval-valued probability has been used most commonly to represent epistemic probability—probability based upon knowledge/evidence that may either be uniquely determined by this knowledge (as in logical probability), unconstrained by anything more than a form of self-consistency* or coherence* (as in personalist/subjective probability representing degrees of belief of an individual), or lying somewhere in between. Proponents of an unconstrained view of epistemic probability include I.J. Good, C.A.B. Smith, and P. Suppes; those favoring constraints include F. Hampel, H.E. Kyburg, I. Levi, and P. Walley. An objective, frequentist account of interval-valued probability is central to the work of Fine and his students, and possibly also to that of P. Huber. A hierarchical classification of lower probabilities has been given [30,7]. For present purposes a lower probability is said to be dominated, P ∈ D , if there exists a measure µ such that (∀A ∈ A)P(A) µ(A). If there does not exist a dominating measure, the lower probability is undominated, P ∈ U. Given a family M of finitely additive measures, we can define the lower expectation E and its restriction to the lower envelope P ∈ L through EX = inf Eµ X, µ∈M
EIA = P(A) = inf µ(A). µ∈M
The corresponding upper probability P is derived by replacing inf by sup. Thus, for example, a lower envelope can be derived from the usual relative frequencies rn (A)
6465
for the occurrence of event A in the first n repeated, unlinked random experiments, through P(A) = lim inf rn (A), n→x
P(A) = lim sup rn (A). n→x
The upper and lower probabilities will agree, with a finitely additive measure, if the relative frequencies are all convergent, but in the theory of interval-valued probability we no longer need to postulate this at the outset. A special lower envelope is a 2-monotone lower probability satisfying P(A ∪ B) + P(A ∩ B) P(A) + P(B). Applications of Interval-Valued Probability We consider briefly applications to problems of representation of belief and knowledge, inference, stochastic modeling. Applications of epistemic interval-valued probability have been concerned primarily with the first two items, and we start with them. Levi [24] and Walley both use lower envelopes but prefer to focus on families of measures and expectation, called prevision∗ by Walley in apparent homage to de Finetti. Levi’s credal probabilities are a special case of lower envelopes in which the family of measures M is required to be convex, whereas Walley allows arbitrary families of measures. The importance of M is that it is not uniquely identifiable from the lower envelope P, although it is identifiable from the lower expectation E. Levi is concerned both with the representation of belief and mechanisms for changing belief [25]. In Walley’s approach to ‘‘statistical reasoning with imprecise probabilities’’ [44] we interpret EX as the highest price we would pay for the gamble with outcome X (assuming the payoffs of X to be linear in utility), and EX as the lowest price we would sell it for. As in the marketplace, buying and selling prices are typically unequal, although in the unrealistic standard Bayesian approach they are required to be equal. Walley introduces a notion of coherence* and proves that lower envelopes and lower expectations are precisely the right representations to characterize coherence. He then extends these concepts to
6466
PROBABILITY, FOUNDATIONS OF—II
include conditioning. In this way he refounds Bayesianism while denying the implausible dogma of precision. Walley [44] is a masterpiece both for its clear and thorough presentation of a wide circle of ideas on the foundations of probability and statistics, and for the alternative it develops carefully and in detail. While he does not see his efforts as restricted to the Bayesian program, anyone seriously concerned with Bayesian methods will benefit from ref. 44. Hampel [14,15] makes use of conditional upper and lower probability in constructing his related notion of successful bets; he interprets them as epistemic and not physical/frequentist concepts. Let X be an observed random variable and Y one we desire to predict, particularly whether Y ∈ A. Assume that both random variables are defined on a parametrized family of measure spaces with {µθ , θ ∈ }, and some unknown θ0 ∈ determines the actual distribution generating X, Y. We interpret P(A|X) as that we are ‘‘just barely willing to bet’’ at odds of P(A|X) : [1 − P(A|X)] on A when we have observed X. Letting EX|θ denote expectation over X using µθ , the lower probability P is successful if it leads to bets (the Xdependent gain for a unit stake being given by −P(A|X)µθ (Ac ) + [1 − P(A|X)]µθ (A)) that satisfy (∀θ ∈ ) EX|θ P(A|X) µθ (A). As Hampel [15, p. 12] observes, ‘‘The operational interpretation is that with many independent such bets in different situations (different parametric models), my average gain will most probably be −.’’ Belief functions∗ , whose structure and application owe most to the efforts of Shafer, are also special 2-monotone lower envelopes. An interesting analysis of belief functions in the context of modal logics is provided by Mongin [28]. Belief functions continue to attract interest in the artificial intelligence community [38]. Kyburg [22,23], in a long-standing research program (e.g., ref. 21), develops logical probability, which relates strongly to (reconstructs) frequentist statistics in the form of epistemic probabilities that are interval-valued. The basic setup is that we
have a rational corpus K of sentences that inter alia contains statistical statements giving ranges for the number of instances of, say, a B being a C; thus, in K we may have the sentence ‘‘the frequency of B ∈ C is in [p, q].’’ We can say that the epistemic probability that A ∈ C is [p, q] if we have the additional sentence in K informing us that A ∈ B and A is a random member of B with respect to C. A notion of A being a random member of B is essential to this project. We also want to identify the ‘‘strongest’’ such claims that can be made relative to K. Kyburg asserts that the lower probability is dominated [22, p. 49]. Applications of interval-valued probability to inference problems have been considered systematically by Walley [44,45] using lower envelopes, and indeed this is the raisond eˆ tre for his work. He uses lower envelopes [45] to treat the fundamental problem of inference in a multinomial setting when little is known a priori either about the number of kinds of possible outcomes (the elements of the sample space) or about their likelihoods. In this case we have an urn of unknown composition from which we draw samples with replacement. The vacuous lower probability [(∀A = ) P(A) = 0] provides a plausible starting point, and conditioning on successive observations through an imprecise Dirichlet model leads to a sequence of more informative (narrower intervals) lower envelopes. An early use of 2-monotone lower envelopes for robust statistical models (e.g., -contamination) was made by Huber [16]. Kadane and Wasserman [20,46] have studied the mathematical properties of upper envelopes, calling them coherent capacities. In ref. 46 they develop Bayes tests for 2monotone lower envelopes. In ref. 47 they attempted to extend the maximum entropy principle to the selection of a set of measures, and hence a lower envelope, but concluded that this could not be done. The major attempt at a frequentist-based notion of interval-valued probability has been in the context of time series* modeling by Fine and his students P. Walley, A. Kumar, Y. L. Grize, A. Papamarcou, and A. Sadrolhefazi, in chronological order. The example of a frequentist-based lower envelope given in the preceding subsection suggests that one can make time series
PROBABILITY, FOUNDATIONS OF—II
models that indulge in fewer metaphysical assumptions about the unobservable infinitely long run. The goal was to construct time series models incorporating stationarity (an unvarying random generating mechanism for the series) and a limited ability to infer from the directly observable events (cylinder sets) to some limiting or tail events (those determined by monotone limits of cylinder sets). For bounded random variables, the celebrated stationarity convergence theorem asserts that the event D, say, of the divergence of relative frequencies must be assigned probability 0 by any stationary, continuous (countably additive) measure. Standard probability theory commits us to a position concerning a tail event (D) on the basis of what is seen on the finite-dimensional cylinder sets. We are unable either to suspend judgment or assert the contrary that µ(D) = 1. Papamarcou and Fine [31] established that it is indeed possible to have stationary lower probabilities that are monotonely continuous along the cylinder sets yet assign P(D) = 1, albeit such a lower probability has to be undominated. Sadrolhefazi and Fine [36] then determined that for any > 0 and n one can find an undominated lower probability P ∈ U that is noncommittal on all tail events, while at the same time agreeing to within with a conventional countably additive measure on cylinder sets of size no greater than n. Hence, while we might have good evidence for a standard probability model on small enough cylinder sets, there yet exists an unconventional extension using lower probability that is noncommittal with respect to more metaphysical claims.
4. 5.
6.
7.
8.
9.
10.
11.
12.
13.
REFERENCES 1. Bassalygo, L. A., Dobrushin, R. L., and Pinsker, M. S. (1988). Kolmogorov remembered. IEEE Trans. Inf. Theory, IT-34, 175. 2. Berger, J. O. and Bernardo, J. M. (1992). On the development of reference priors. In Bayesian Statistics 4. Oxford Univ. Press, New York, pp. 35–60. 3. Berger, J. O. (1993). Statistical Decision Theory and Bayesian Analysis, corrected reprint of 2nd (1985) ed. Springer-Verlag, New York.
14.
15.
6467
(A good textbook on the subject by an eminent statistician who became a Bayesian only after the first version of this significantly rewritten text.) Berger, J. O. (1994). An overview of robust Bayesian analysis. Test, 3, 5–124. Bernardo, J. M. and Smith, A. F. M. (1994). Bayesian Theory. Wiley, New York. (Another good textbook on the subject by persons committed to Bayesianism.) Chaitin, G. J. (1987). Algorithmic Information Theory. Cambridge Univ. Press, Cambridge. (Papers by a highly original thinker on the subject.) Chateauneuf, A. and Jaffray, J. Y. (1989). Some characterizations of lower probabilities and other monotone capacities through the use of Mobius inversion. Math. Soc. Sci., 17, 263–283. Chuaqui, R. (1991). Truth, Possibility and Probability: New Logical Foundations of Probability and Statistical Inference. NorthHolland/Elsevier, Amsterdam. Dawid, A. P. (1991). Prequential analysis, stochastic complexity and Bayesian inference. In Bayesian Statistics 4. Oxford Univ. Press, New York, pp. 109–125. Diaconis, P. and Freedman, D. (1986). On the consistency of Bayes estimates (including discussion). Ann. Statist., 14, 1–67. Faigin, R. and Halpern, J. (1994). Reasoning about knowledge and probability. J. Ass. Comput. Mach., 41, 340–367. (Two leading figures in artificial intelligence who take epistemological questions seriously and proceed as logicians.) Fine, T. L. (1973). Theories of Probability: An Examination of Foundations. Academic Press, New York. Fine, T. L. (1994). Upper and lower probability. In ref. 17, pp. 109–129. (An overview of efforts to use lower probability in time-series modeling and nonadditive probability in a frequentist setting.) Hampel, F. (1996). On the philosophical foundations of statistics: bridges to Huber’s work and recent results. In Robust Statistics Data Analysis and Computer Intensive Methods: in Honor of Peter Huber’s 60th Birthday, H. Rieder, ed. Lecture Notes in Statistics. Springer, New York, pp. 185–196. Hampel, F. (1997). What can the foundations discussion contribute to data analysis? And what may be some of the future directions in robust methods and data analysis? J. Statist. Plann. Inference, 57, 7–19. (A summary of
6468
PROBABILITY, FOUNDATIONS OF—II
Hampel’s position, noting issues in the foundations of probability such as the need for upper and lower probability to represent prior knowledge and the importance of distinguishing between epistemic and so-called physical probabilities. This paper provides several references to Hampel’s work.) 16. Huber, P. (1980). Robust Statistics. WileyInterscience, New York. 17. Humphreys, P., ed. (1994). Patrick Suppes: Scientific Philosopher, vol. 1. Kluwer, Dordrecht. (A Festchrift for Suppes including many valuable contributions on a range of foundational issues.) 18. Jaynes, E. T. (1989). E.T. Jaynes: Papers on Probability, Statistics, and Statistical Physics, R. D. Rosenkrantz, ed. Kluwer Academic, Dordrecht. (Reprint of 1983 collection of papers by the foremost exponent of the maximum entropy principle for determining priors.) 19. Jeffrey, R. C. (1992). Probability and the Art of Judgment. Princeton Univ. Press, Princeton. (Reprints of several papers by Jeffrey, who has been highly influential as a philosopher concerned with Bayesianism and the issue of updating or conditioning probabilities.) 20. Kadane, J. B. and Wasserman, L. (1996). Symmetric, coherent, Choquet capacities. Ann. Statist., 24, 1250–1264. 21. Kyburg, H. E., Jr. (1974). The Logical Foundations of Statistical Inference. Reidel, Dordrecht. (An extended development of Kyburg’s epistemic approach to probability and its role in statistics; uses formalism that will challenge the reader.) 22. Kyburg, H. E., Jr. (1990). Science and Reason. Oxford Univ. Press. (Contains a brief, but accessible introduction to Kyburg’s formulation of interval-valued epistemic probability.) 23. Kyburg, H. E., Jr. (1997). Combinatorial semantics: semantics for frequent validity. Comput. Intell., 13, 215–257. 24. Levi, I. (1985). Imprecision and indeterminacy in probability judgment. Phil. Sci., 52, 390–409. 25. Levi, I. (1991). The Fixation of Belief and Its Undoing: Changing Beliefs through Inquiry. Cambridge Univ. Press, Cambridge. 26. Li, M. and Vitanyi, P. (1997). An Introduction to Kolmogorov Complexity and Its Applications. Springer-Verlag, New York. (An excellent and comprehensive review of the subject notable for an uncommon interest in the history of and contributors to computational complexitybased approaches to randomness, induction, and information.)
27. MacKay, D. (1992). A practical Bayesian framework for backpropagation networks. Neural Comput., 4, 448–472. 28. Mongin, P. (1994). Some connections between epistemic logic and the theory of nonadditive probability. In ref. 17, pp. 135–171. 29. Neal, R. (1996). Bayesian Learning for Neural Networks. Springer-Verlag, New York. (An accessible treatment of effective Bayesian calculations in the realistic setting of many parameters. While neural networks are the focus, they can be thought of as another parametrized family of flexible regression functions.) 30. Papamarcou, A. and Fine, T. L. (1986). A note on undominated lower probabilities. Ann. Probab., 14, 710–723. 31. Papamarcou, A. and Fine, T. L. (1991). Stationary and almost sure divergence of time averages in interval-valued probability. J. Theor. Probab., 4, 239–260. 32. Pearl, J. (1988). Probabilistic Reasoning in Intelligent Systems. Morgan Kaufmann, San Mateo, CA. 33. Rissanen, J. (1984). Universal coding, information, prediction, and estimation. IEEE Trans. Inf. Theory, IT-30, 629–636. 34. Rissanen, J. (1987). Stochastic complexity (with discussion). J. R. Statist. Soc. B, 49, 223–239, 252–265. 35. Rissanen, J. (1989). Stochastic Complexity in Statistical Inquiry. World Scientific, Singapore. (The most comprehensive presentation of stochastic complexity.) 36. Sadrolhefazi, A. and Fine, T. L. (1994). Finitedimensional distributions and tail behavior in stationary interval-valued probability models. Ann. Statist., 22, 1840–1870. 37. Shafer, G. (1987). Probability judgment in artificial intelligence and expert systems. Statist. Sci., 2, 3–16. 38. Shafer, G. and Pearl, J. (1990). Readings in Uncertain Reasoning. Morgan Kaufmann, San Mateo, CA. 39. Shen, A. (1992). Algorithmic complexity and randomness: recent developments. Theory Probab. Appl., 37, 92–97 (Russian original, Teor. Veroyatnost. i Primenen., 37, 124–131). 40. van Lambalgen, M. (1992). Independence, randomness and the axiom of choice. J. Symbolic Logic, 57, 1274–1304. 41. van Lambalgen, M. (1993). Axioms for randomness. Theory Probab. Appl., 38, 562–566 (Russian original, Teor. Veroyatnost. i Primenen., 38, 670–674).
PROBABILITY, HISTORY OF 42. Vovk, V. G. (1988). The law of the iterated logarithm for Kolmogorov random, or chaotic, sequences (in Russian). Teor. Veroyatnost. i Primenen., 32, 413–425. 43. Vovk, V. G. and Vyugin, V. V. (1994). Prequential level of impossibility with some applications. J. R. Statist. Soc. B, 56, 115–123. 44. Walley, P. (1991). Statistical Reasoning with Imprecise Probabilities. Chapman and Hall, London. (A masterful approach to decision making based upon upper and lower expectation that also contains thorough discussion of competing approaches.) 45. Walley, P. (1996). Inferences from multinomial data: learning about a bag of marbles. J. R. Statist. Soc. B, 58, 3–57. (A clear application of the approach.) 46. Wasserman, L. and Kadane, J. B. (1990). Bayes’ theorem for Choquet capacities. Ann. Statist., 18, 1328–1339. 47. Wasserman, L. and Kadane, J. B. (1996). Entropy when probabilities are imprecise. In Bayesian Analysis in Statistics and Econometrics. Wiley, New York, pp. 549–555. 48. Wright, G. and Ayton, P. (1994). Subjective Probability. Wiley, Chichester. (A compendium of recent essays on many aspects of subjective probability.) See also BELIEF, DEGREES OF; BELIEF FUNCTIONS; COHERENCE —I; COMPLEXITY; DECISION THEORY; IMPRECISE PROBABILITIES; MAXIMUM ENTROPY PRINCIPLE: CLASSICAL APPROACH; MINIMUM DESCRIPTION LENGTH PRINCIPLE; NONADDITIVE PROBABILITY; POSTERIOR PROBABILITIES; PREVISION; PRIOR PROBABILITIES; PROBABILITY, HISTORY OF; PROBABILITY THEORY: AN OUTLINE; STOCHASTIC COMPLEXITY; SUBJECTIVE PROBABILITIES; SUBJECTIVE RANDOMNESS; and UTILITY THEORY.
TERRENCE L. FINE
PROBABILITY GENERATING FUNCTIONS. See GENERATING FUNCTIONS PROBABILITY, HISTORY OF The history of probability is in essence separate from the history of statistics∗ , although the statistics of sample data as a mathematical discipline relies on probability as the foundation of quantitative inference. The synthesis of probability with statistics to
6469
form mathematical statistics was ultimately achieved at the beginning of the twentieth century, largely due to the English Biometric School∗ . In this article we adhere to the history of probability proper up to the end of the nineteenth century, with a brief incursion into the twentieth. Thus we shall not mention names as eminent as that of Gauss∗ , more pertinent to the ‘‘theory of errors.’’ The history of probability theory is traditionally taken to begin in France in 1654, the year of the correspondence between Blaise Pascal∗ (1623–1662) and Pierre de Fermat∗ (1601–1665), primarily on the problem of equitable division of stakes in games of chance∗ . Although there is a ‘‘prehistory’’ of probability, it is due to the enormous prestige of these two workers that later developments crystallized round their work. It is also characteristic of the history that until the nineteenth century its motivation remained, very substantially, games of chance. The frequently mentioned significant names of the prehistory of probability in the context of gaming are Cardano∗ , Galileo∗ , Pacioli, and Tartaglia in Italy. In a book of Fra Luca Pacioli (ca. 1445–ca. 1517) printed in 1494, there occurs a simple instance of the problem of points∗ which was later taken up by Tartaglia (Niccol`o Fontana, ca. 1500–1557) and Girolamo Cardano (1501–1576), and then solved by Fermat and Pascal. In Cardano’s manuscript Liber de Ludo Aleae we find (perhaps for the first time) the fundamental notion, in the context of dice games, that the probability of an event in the situation where each of a finite set of possible outcomes has equal weight is the ratio of the favorable cases to the total number of the cases. Galileo Galilei (1564–1642) showed, inter alia, a thorough understanding of this principle in reference to a throw of three fair dice. The keynote of the Fermat–Pascal correspondence is the equitable division of total stake when a game is interrupted before its agreed completion (‘‘probl`eme de partis’’), and a solution to the problem of points is produced by an ingenious extension of the sample space to equally likely points. The correspondence contains a numerical instance of the gambler’s ruin problem (random walk∗ with absorbing barriers). Pascal’s posthumously
6470
PROBABILITY, HISTORY OF
published Trait´e du triangle arithmetique is the corresponding treatise; his Pens´ees and portions, disputably written by Pascal, of the Logique de Port-Royal (1662) contain elements of the calculus of probabilities also. The next landmark is the publication in 1657 of the first text on probability: De Ratiociniis in Ludo Aleae by the Dutch scientist Christiaan Huygens∗ (1629–1695). Huygens had learned of the problem of points on a visit to Paris in 1655, and his systematization of arguments regarding this problem, and some exercises including the gambler’s ruin problem, were inspired directly by the activities of the French mathematicians. His booklet contains a clear allusion to expectation and notions of sampling without and with replacement. Its main function was as a popularizing medium for the calculus of chances until superseded by the very substantial works Essai d’analyse sur les jeux de hasard (1708) of Pierre de Montmort∗ (1678–1719), Ars Conjectandi (1713, posthumous) of Jacob (James) Bernoulli∗ (1654–1705), and The Doctrine of Chances (1718) of Abraham De Moivre∗ (1667–1754). Bernoulli’s book, written perhaps about 1690, incorporated Huygens’s treatise, with notes, in its first part and the ideas in its fourth part were suggested at least partially by the Logique de Port-Royal. In the first part, the gambler’s ruin problem is generalized to a totally algebraic setting. In the whole, the general rules of probability calculus are made clear. The most significant achievement, however, is in the fourth part, where it is proved under some restrictions that if n−1 Sn is the proportion of successes in n binomial∗ trials in which the success probability is p, then as n → ∞ for any > 0, Pr(|n−1 Sn − p| ) → 1, which has come to be known as Bernoulli’s theorem∗ and is the first version of the weak law of large numbers∗ . In the preface to Ars Conjectandi, James Bernoulli’s nephew Nicholas asks that Montmort and De Moivre turn their attention to further applications of probability. One of the results, in part, was the second enlarged edition in 1713 of Montmort’s treatise; Montmort had been aware at the time of his first edition of the work
of the Bernoullis. Inasmuch as he writes for mathematicians rather than gamblers, using ‘‘modern’’ analysis, this work is historically significant. He worked on the game of Rencontre (matching pairs), resolved the problem of points in general terms, and worked with Nicholas Bernoulli (1687–1759) on the problem of duration of play in the gambler’s ruin problem. The most important contribution of De Moivre is the refinement of Bernoulli’s theorem by obtaining the normal approximation∗ to the binomial distribution∗ for general p (‘‘De Moivre’s theorem’’), thereby establishing the first instance of the central limit theorem∗ . This work appears in the second and third editions of his Doctrine, which also contains the elements of the Poisson approximation to the binomial distribution. The book of the Marquis De Condorcet (1743–1794), Essai sur l’Application de l’Analyse a` la Probabilit`e des D´ecisions Rendues a` la Pluralit´e des Voix (1785), is important in that it is concerned with social questions and in that it stimulated the fundamental work of Pierre Simon de Laplace∗ (1749–1827), Th´eorie analytique des probabilit´es (1812), which is often taken to mark the end of an era. Laplace’s book may be regarded as describing the then state of the art not only for probability but for statistics in a systematic mathematical fashion. Perhaps two of Laplace’s greatest contributions for probability theory proper are recognition of the mathematical power of transforms (generating functions∗ , characteristic functions∗ ) and an early general version of the central limit theorem. The nineteenth century sees work in probability by French mathematicians on the foundations laid by Laplace and the development of a strong probabilistic school in the Russian Empire as the two most subsequently influential streams. (German developments were rather in the direction of statistics.) Of the French school, a significant landmark was the publication in 1837 of the book by Sim´eon Poisson∗ (1781–1840), Recherches sur la Probabilit´e des Jugements en Mati`ere Criminelle et en Mati`ere Civile, Pr´ec´ed´ees des R`egles G´en´erales du Calcul des Probabilit´es (translated into German in 1841), which, as well as taking up the social context treated
PROBABILITY, HISTORY OF
by Condorcet and Laplace, is in part a probability text. Poisson considered in particular binomial trials with probability of success which may vary deterministically from trial to trial. This situation gives rise to Poisson’s law of large numbers∗ for the proportion of successes, and a corresponding normal approximation, possibly the first suggestion of a central limit statement for sums of independent but nonidentically distributed random variables. Poisson also arrived at the Poisson distribution as the limit of a negative binomial∗ , and the probability distribution function makes possibly its first appearance. Augustin Cauchy∗ (1789–1857) in 1853 discovered the symmetric stable laws∗ and supplied almost a complete rigorous proof of the central limit theorem by characteristic function methods within the context of least squares, while Jules Bienaym´e∗ (1796–1878) proved by the simple techniques still used today the Bienaym´e –Chebyshev inequality∗ , used it to prove a general weak law of large numbers for sums of i.i.d. random variables, and gave a completely correct statement of the criticality theorem∗ of branching processes∗ . The book of Antoine Cournot∗ (1801–1877), Exposition de la Th´eorie des Chances et des Probabilit´es (1834) (translated into German in 1849), is largely philosophical and is notable for an early frequentist view of the subject, while that of Joseph Bertrand (1822–1900), entitled Calcul des Probabilit´es (1889), is a significant and sensibly written ‘‘modern’’ textbook on probability and its applications. The flowering of probability theory in particular (as well as mathematics in general) in the Russian Empire is due in part to the return of the young mathematicians V. Ya. Buniakovsky (1804–1889) and M. V. Ostrogradsky (1801–1861) from Paris, where in the early 1820s both had had contact with Laplace, Cauchy, Fourier, and Poisson. Both popularized probability theory in St. Petersburg, together with A. Yu. Davydov (1823–1885) in Moscow. Buniakovsky wrote the first Russian-language text (heavily influenced by Laplace’s Th´eorie) on probability, Osnovania Matematicheskoi Teorii Veroiatnostei, and thereby introduced the still-current mathematical terminology into that language. The stage had been
6471
set for the founding of the St. Petersburg School by P. L. Chebyshev∗ (1821–1894), whose probabilistic contributions, with those of his pupils A.A. Markov∗ (1856–1922) and A. M. Liapunov∗ (1857–1918), in particular, established it as another center of the subject area and gave rise to much of the direction of modern probability theory. Chebyshev proved rigorously Poisson’s weak law of large numbers and popularized the applicability of the Bienaym´e –Chebyshev inequality as a tool for proving weak laws. All three worked on rigorous proofs of the central limit theorem for nonidentically distributed independent summands under general conditions, the culmination being Liapunov’s theorem∗ . Markov developed the concept of Markov chains∗ and thereby gave impetus to the development of stochastic processes∗ . His textbook Ischislenie veroiatnostei (1900, 1908), in its German version in 1912 and in its third (substantially revised) edition of 1913 (and the posthumous edition of 1924), to a large extent laid the foundation for the further development of probability theory. In regard to stochastic processes, further foundations were laid, again in France, by the Calcul des probabilit´es (1896) of Henri Poincar´e (1854–1912), and by Louis Bachelier (1870–1946) through his Th´eorie de la speculation (1900), for the theory of Markov processes. In England some early impetus for probabilistic work was provided by Augustus de Morgan (1806–1871). The Logic of Chance (1876) of John Venn (1834–1923) gave impetus to the study of probability from the standpoint of philosophy, while that of William Whitworth (1840–1905), entitled Choice and Chance, of which the first edition appeared in 1867, even now continues to provide exercises on combinatorial probability. Other eminent contributors to the subject were Robert Ellis (1817–1859), Morgan Crofton (1826–1915), and James Glaisher (1848–1928). In the context of statistical mechanics∗ , important contributors were James Maxwell∗ (1831–1879) and Ludwig Boltzmann∗ (1844–1906). For the prehistory of probability see refs. 3, 6, 9, 10, 13, and 15. The Fermat–Pascal correspondence is thoroughly discussed in refs. 3 and 19, while work from this point up to and including Laplace is well covered in the
6472
PROBABILITY, HISTORY OF
classic ref. 19. Other relevant sources for this period are refs. 1–4, 6, 7, 9, 11–14, and 16. Unlike the preceding period, the history of probability in the nineteenth century is not adequately represented in monograph form. The most comprehensive is Czuber [2], with the supplement, ref. 21. Modern monograph sources are refs. 5, 8, 9 and 20. For specialized aspects, consult also refs. 1, 13, 17, and 18. The modern era in the development of probability theory begins with its axiomatization. The first attempts in this direction are due to Sergei Bernstein∗ (1880–1968), Richard von Mises∗ (1883–1953), and Emile Borel (1871–1956). A. N. Kolmogorov’s Grundbegriffe der Wahrscheinlichkeitsrechnung (1933) introduced the now generally accepted axiomatization suitable not only for classical probability theory but also for the theory of stochastic processes∗ . A few of the significant books which provided an impetus to the development of modern probability are P. L´evy’s Th´eorie de l’addition des variables al´eatoires (1937); H. Cram´er’s Random Variables and Probability Distributions (1937) and Mathematical Methods of Statistics (1946); Limit Distributions for Sums of Independent Random Variables (Russian edition 1949; first English edition 1954) by B. V. Gnedenko and A. N. Kolmogorov; W. Feller’s An Introduction to Probability Theory and Its Applications, Vol. 1 (1950); and M. Lo`eve’s Probability Theory (1955).
REFERENCES 1. Adams, W. J. (1974). The Life and Times of the Central Limit Theorem. Caedmon, New York. (Chapter 2 contains a careful bibliographical analysis of the origin of De Moivre’s theorem.) 2. Czuber, E. (1899). Jb. Dtsch. Math.-Ver., 7, Part 2, 1–279. (Together with W¨olffing [21], the best bibliographical source on probability and statistics in the nineteenth century.) 3. David, F. N. (1962). Games, Gods and Gambling. Charles Griffin, London. 4. Dupont, P. (1979). Atti. Accad. Sci. Torino Cl. Sci. Fis. Mat. Natur., 113, 243–261. 5. Gnedenko, B. V. and Sheynin, O. B. (1978). In Matematika XIX veka (Mathematics of the
nineteenth century). Nauka, Moscow, Chap. 4. (Chapter 4, pp. 184–240, deals with the history of probability, beginning with Laplace.) 6. Gouraud, C. (1848). Histoire du calcul des probabilit´es. A. Durand, Paris. 7. Hacking, I. (1975). The Emergence of Probability. Cambridge University Press, Cambridge, England, Chap. 8. 8. Heyde, C. C. and Seneta, E. (1977). I. J. Bienaym´e: Statistical Theory Anticipated. Springer-Verlag, New York. (A modern source on probability and statistics in the nineteenth century.) 9. Maistrov, L. E. (1967). Teoriya Veroiatnostei: Istoricheskii Ocherk. Nauka, Moscow. (English transl.: Probability Theory. A Historical Sketch, S. Kotz, trans./ed. Academic Press, New York, 1974.) 10. Ore, O. (1953). Cardano, the Gambling Scholar. Princeton University Press, Princeton, N.J. (This book contains a translation by S. H. Gould of Liber de Ludo Aleae.) 11. Ore, O. (1960). Amer. Math. Monthly, 67, 409–419. 12. Pearson, K. (1978). The History of Statistics in the 17th and 18th Centuries. (Lectures by Karl Pearson 1921–1933, edited by E. S. Pearson.) Charles Griffin, London. 13. Pearson, E. S. and Kendall, M. G., eds. (1970). Studies in the History of Statistics. Charles Griffin, London; Hafner, Darien, Conn. (Vol. II, M. G. Kendall and R. L. Plackett, eds., 1977). 14. Seneta, E. (1979). In Interactive Statistics, D. McNeil, ed. North-Holland, Amsterdam, pp. 225–233. 15. Sheynin, O. B. (1974). Arch. History Exact Sci., 12, 97–141. 16. Sheynin, O. B. (1977). Arch. History Exact Sci., 17, 201–259. 17. Sheynin, O. B. (1978). Arch. History Exact Sci., 18, 245–300. 18. Stigler, S. M. 503–517.
(1975).
Biometrika,
62,
19. Todhunter, I. (1865). A History of the Mathematical Theory of Probability from the Time of Pascal to That of Laplace. Cambridge University Press, Cambridge, England. (Reprinted in 1949 and 1961 by Chelsea, New York.) 20. Ulbricht, K. (1980). Wahrscheinlichkeitsfunktionen im neunzehnten Jahrhundert. Minerva, Munich. (Contains a good bibliography and is valuable in presenting a German standpoint, possibly under-represented in the article.)
PROBABILITY INEQUALITIES FOR SUMS OF BOUNDED RANDOM VARIABLES 21. W¨olffing, E. (1899). Math. Naturwiss. Ver. Wurttemberg ¨ (Stuttgart), Mitt., (2), 1, 76–84. (Supplement to Czuber [2].) See also BERNOULLIS, THE; BERNOULLI’S LIMIT THEOREM; BINOMIAL DISTRIBUTION; CHEBYSHEV’S INEQUALITY; ´ ), PAFNUTY LVOVICH; CHEBYSHEV (or TCHEBICHEF COMBINATORICS; CONVERGENCE OF SEQUENCES OF RANDOM VARIABLES; DE MOIVRE, ABRAHAM; DE MOIVRE –LAPLACE THEOREM, GLOBAL; FERMAT, PIERRE DE; GAMES OF CHANCE; LAPLACE, PIERRE SIMON; LIMIT THEOREM, CENTRAL; MARKOV, ANDREI ANDREEVICH; ´ MONTMORT, PIERRE REMOND DE; PASCAL, BLAISE; ´ -DENIS; PROBLEM OF POINTS; POISSON, SIMEON RANDOM WALKS; and STATISTICS, HISTORY OF).
E. SENETA
PROBABILITY INEQUALITIES FOR SUMS OF BOUNDED RANDOM VARIABLES
Pr[|S − ES| x(var S)1/2 ] x−2 .
(1)
If S is the sum of n independent, identically distributed random variables, then, by the central limit theorem∗ , as n → ∞, the probability on the left approaches 2(−x), where (x) is the standard normal distribution function. For x large, (−x) behaves as const. x−1 exp(−x2 /2). S. Bernstein∗ [2,3] has shown that under an additional assumption, which is satisfied when the summands of S are uniformly bounded, the upper bound in (1) can be replaced by one which, for n large, behaves as exp(−x2 /2). We first discuss Bernstein’s inequality∗ and related results for sums of independent random variables and random vectors, and then extensions of some of these results to certain sums of dependent random variables. SUMS OF INDEPENDENT RANDOM VARIABLES Let X1 , . . . , Xn be n independent real-valued random variables with finite means, S = X1 + · · · + Xn ,
µ = ES/n.
We begin with inequalities that involve no moments other than µ. Theorem 1. If 0 Xi 1, i = 1, . . . , n, then for 0 < t < 1 − µ, Pr[S − ES nt] µ+t 1−µ−t n µ 1−µ µ+t 1−µ−t
(2)
(3)
exp{−g(µ)nt2 }
(4)
exp{−2nt2 },
(5)
where
g(µ) =
If S is a random variable with finite mean and variance, the Bienaym´e –Chebyshev inequality states that for x > 0,
6473
1 1−µ ln 1 − 2µ µ
1 2µ(1 − µ)
for 0 < µ < 12 , (6)
for
1 2
µ < 1.
(7)
[Note: If t > 1 − µ, the probability in (3) is zero.] The assumption 0 Xi 1 has been made to give the bounds a simple form. If, instead, we assume that a Xi b, then µ and t in the statement of Theorem 1 are to be replaced by (µ − a)/(b − a) and t/(b − a), respectively. Upper bounds for Pr[S − ES −nt] = Pr[−S + ES nt] and for Pr[|S − ES| nt] = Pr[S − ES nt] + Pr[S − ES −nt] follow from Theorem 1 by appropriate substitutions. Similar remarks apply to the inequalities that follow. Let σ 2 = n−1 var S.
(8)
Under the conditions of Theorem 1, σ 2 µ(1 − µ), with equality holding only in the case Pr[Xi = 0] = 1 − µ,
Pr[Xi = 1] = µ, i = 1, . . . , n.
(9)
6474
PROBABILITY INEQUALITIES FOR SUMS OF BOUNDED RANDOM VARIABLES
For this (binomial) case inequality (3) is implicitly contained in Chernoff [5], and the other bounds of Theorem 1, except for (4) with µ < 12 , are due to Okamoto [7]. In the general case Theorem 1 was proved by Hoeffding [6]. If the inequalities of Theorem 1 are written as Pr[S − ES nt] A1 A2 A3 , then each Ai is of the form ani , where ai is a function of µ and t only. Each of these bounds is simpler but cruder than the preceding. We have A1 = A2 if t = 1 − 2µ and A2 = A3 if µ = 12 . If t and µ are fixed as n → ∞, in which case Pr[S − ES nt] → 0 exponentially fast, and if the inequality Ai < Ai+1 is strict, then the bound Ai is appreciably smaller than Ai+1 when n is large. On the other hand, if we put t = yn−1/2 and hold y and µ fixed, then, as n → ∞, A1 → exp −
y2 , 2µ(1 − µ)
A2 = exp(−g(µ)y2 ), A3 = exp(−2y2 ). Note that the limit of A1 is equal to A2 for 1 2 µ < 1. When the central limit theorem applies to S, Pr[S − ES n1/2 y] → (−y/σ ) (−y/ µ(1 − µ)). The following is an extension of bound (5); see Hoeffding [6]. Theorem 2. If X1 , . . . , Xn are independent and ai Xi bi (i = 1, . . . , n), then for t > 0, Pr[S − ES nt] n exp −2n2 t2 / (bi − ai )2 . 1
(10)
We now assume that the Xi have a common mean. For simplicity the mean is taken to be zero. Theorem 3. If X1 , . . . , Xn are independent, σ 2 < ∞, EXi = 0, Xi b(i = 1, . . . , n), then for 0 < t < b,
Pr[S nt]
1+
bt σ2
−(1+bt/σ 2 )σ 2 /(b2 +σ 2 )
2 2 2 n t −(1−t/b)b /(b +σ ) × 1− b σ2 nt 1+ exp − b bt !" bt × ln 1 + 2 − 1 σ " nt2 exp − . 2(σ 2 + bt/3)
(11)
(12) (13)
Here the summands are assumed to be bounded only from above. However, to obtain from Theorem 3 upper bounds for Pr[|S| t], we must assume that they are bounded on both sides. The bounds in (3) and (11) are related as follows. If the assumptions of Theorem 3 are satisfied and the Xi are also bounded from below, a Xi b (where a < 0 < b), then σ 2 −ab. The bound in (11) is an increasing function of σ 2 . If we replace σ 2 by its upper bound −ab, we obtain from (11) the inequality which results from (3) when the appropriate substitutions mentioned after the statement of Theorem 1 are made. Note, however, that Theorem 1 does not require that the Xi have a common mean. The bounds for Pr[S nt] in (11) to (13) are due to Hoeffding [6], Bennett [1], and Bernstein [3], respectively. For proofs of Bernstein’s inequality, see also Uspensky [8] and Bennett [1]. For an early version of (13), see Bernstein [2]. Bernstein [3] derived an upper bound for Pr[|S| nt] similar to (13) without assuming that the Xi are bounded. Instead, he assumed that the moments of the Xi satisfy |EXim | 12 (EXi2 )m!cm−2 , m = 3, 4, . . . ;
i = 1, . . . , n,
(14)
with some constant c > 0. If |Xi | b for all i, then (14) is satisfied with c = b/3. The inequalities of Theorem 3 can be written Pr[S nt] B1 B2 B3 .
PROBABILITY INEQUALITIES FOR SUMS OF BOUNDED RANDOM VARIABLES
Each Bi is of the form bni , where bi is a function of t/b and σ/b only. If t, b, and σ are fixed as n → ∞, remarks similar to those concerning Theorem 1 apply. If we put t = xσ n−1/2 and hold x, b, and σ fixed, then, as n → ∞, Bi → exp(−x2 /2),
i = 1, 2, 3.
When the central limit theorem applies to S, Pr[S xσ n1/2 ] → (−x). Thus in this case the simple bound (13) is nearly as good as the sharper bounds (11) and (12) when n is sufficiently large. For further comparisons between different bounds, see Bennett [1] and Hoeffding [6]. Bernstein’s exponential bound (13) has been extended to the cases of sums of independent random vectors taking values in a Euclidean space, a Hilbert space, and a Banach space. See Yurinskiˇı [9], where references to earlier work are given. SUMS OF DEPENDENT RANDOM VARIABLES Some of the inequalities discussed above have been extended to certain types of dependent random variables. (a) Martingales∗ . Let X1 , . . . , Xn be random variables with finite means, Xi = j Xi − EXi , Sj = 1 Xi , and suppose that the sequence S1 , . . . , Sn is a martingale, that is, |X1 , . . . , Xj ] = 0, j = 1, . . . , n − 1, with E[Xj+1 probability 1. Bernstein [4] showed that a version of the inequality named for him holds in the present case. Theorems 1 and 2 remain true if the Sj sequence is a martingale [6, p. 18]. (b) U-statistics∗ . Let X1 , . . . , Xn be i.i.d., and consider the U-statistic U=
1 φ(Xi , . . . , Xir ), n,r n(r)
n r,
where φ is a measurable function,n(r) = n(n − 1) . . . (n − r + 1), and the sum n,r is taken over all r-tuples i1 , . . . , ir of distinct positive integers not exceeding n. If 0 φ(x1 , . . . , xr ) 1, then the bounds of Theorem 1 with n replaced by [n/r] (the largest integer n/r) and µ = Eφ(X1 , . . . , Xr )
6475
are upper bounds for Pr[U − EU t]. If σ 2 = var φ(X1 , . . . , Xr ) exists and φ(x1 , . . . , xr ) EU + b, then the bounds of Theorem 3 with n replaced by [n/r] are upper bounds for Pr[U − EU t]. Similar results hold for k-sample Ustatistics and for statistics of the form V = n−r
n i1 =1
···
n
φ(Xi1 , . . . , Xir ).
ir =1
(c) Sums of finitely dependent random variables. Let the random variables X1 , . . . , Xn be (r − 1)-dependent; that is, the random vectors (X1 , . . . , Xi ) and (Xj , . . . , Xn ) are independent if j − i r. Suppose also that X1 , . . . , Xn are identically distributed. Let µ = EX1 , S = X1 + · · · + Xn . Under the appropriate boundedness conditions the bounds of Theorems 1 and 3 with n replaced by [n/r], are upper bounds for Pr[S − ES nt]. (d) Sampling from a finite population. Consider a finite population∗ P = {1, . . . , N} of size N and a real-valued function f defined on P. Let (Y1 , . . . , Yn ) be a simple random sample without replacement from P, Xi = f (Yi ), S = X1 + · · · + Xn . Let µ = 2 EX1 , σ 2 = N −1 N j=1 (f (j) − µ) . If the appropriate boundedness conditions on f (j) are satisfied, the inequalities of Theorems 1 and 3 hold in this case. REFERENCES 1. Bennett, G. (1962). J. Amer. Statist. Ass., 57, 33–45. 2. Bernstein, S. N. (1924). Uˇcen. Zap. Nauc.ˇ Issled. Kafedr Ukrainy, Otd. Mat., 1, 30–49 (in Russian). [Reprinted in S. N. Bernstein, Collected Works, Vol. IV. Nauka, Moscow, 1964 (in Russian).] 3. Bernstein, S. N. (1927). Probability Theory (in Russian). (Referred to in Uspensky [8].) 4. Bernstein, S. N. (1937). Dokl. Akad. Nauk SSSR, 17, 275–277 (in Russian). [Reprinted in S. N. Bernstein, Collected Works, Vol. IV. Nauka, Moscow, 1964 (in Russian).] 5. Chernoff, H. (1952). Ann. Math. Statist., 23, 493–507. 6. Hoeffding, W. (1963). J. Amer. Statist. Ass., 58, 13–30. 7. Okamoto, M. (1958). Ann. Inst. Statist. Math., 10, 20–35.
6476
PROBABILITY INTEGRAL TRANSFORMATIONS
8. Uspensky, J. V. (1937). Introduction to Mathematical Probability. McGraw-Hill, New York. 9. Yurinskiˇı, V. V. (1976). J. Multivariate Anal., 6, 473–499. See also BERNSTEIN’S INEQUALITY; BIRNBAUM-RAYMOND-ZUCKERMAN INEQUALITY; and CHEBYSHEV’S INEQUALITY.
W. HOEFFDING
PROBABILITY INTEGRAL TRANSFORMATIONS The classic probability integral transformation (PIT) theorem can be stated as follows. Theorem 1. If a random variable X has a continuous distribution function F(x), then the random variable U = F(X) has a uniform distribution on the interval (0, 1), that is, is a U(0, 1) random variable (see UNIFORM DISTRIBUTION MODULO 1). This result is given in a number of recent books, including refs. 9 (p. 408) and 19 (p. 203). If the distribution function is assumed to be absolutely continuous, so that a density function exists, then the PIT is easily established using the transformation of densities technique. The earliest use of this result of which we are aware was by R. A. Fisher∗ [4] in his famous 1930 paper in which he introduced the theory of fiducial∗ limits or intervals. Although he did not discuss the PIT explicitly as such, his implicit use of it to construct fiducial intervals leaves no doubt that he understood it (see FIDUCIAL INFERENCE). Let T be a statistic with continuous distribution function Fθ (t), and suppose that we wish to estimate θ . For fixed values of t1 , u1 , and u2 (0 u1 u2 1), put t = {θ : u1 Fθ (t) u2 }. Then if t is an interval, it is called a 100(u2 − u1 )% fiducial interval for θ . This construction was clearly motivated by the PIT. Fisher [5] again used the PIT in 1932 in the fourth edition of Statistical Methods for Research Workers, when he proposed a method for combining tests of significance. Again, he did not discuss the PIT explicitly, and this omission caused considerable
confusion among readers (see ref. 21). His combined test of significance is developed as follows. Suppose that T1 , T2 , . . . , Tk are k test statistics that are independent when the k corresponding null hypotheses hold. Let p1 , . . . , pk denote the observed significance levels or p values∗ of these tests. Then by Theorem 1, the p values are independent, identically distributed (i.i.d.) U(0, 1) random variables. If it is desired to test that all the null hypotheses are correct simultaneously against the alternative that at least one is incorrect, then an intuitively appealing test can be made by rejecting when the product of the p-values, say Q = p1 · · · pk , is too small, or equivalently, when −2 ln Q is too large. However, since p1 , . . . , pk are i.i.d. U(0, 1) random variables, it is readily shown that −2 ln p1 , . . . , −2 ln pk are i.i.d. χ 2 (2) random variables; therefore, −2 ln Q = −2 ln pi is a χ 2 (2k) random variable, and the test is easily made by rejecting when −2 ln Q exceeds χα2 (2k). The principal uses of the PIT in statistics consider a random sample X1 , . . . , Xn from a parent distribution function F(·), and the i.i.d. U(0, 1) random variables U1 = F(X1 ), . . . , Un = F(Xn ). If X(1) , . . . , X(n) and U(1) , . . . , U(n) , denote the order statistics∗ from the samples X1 , . . . , Xn and U1 , . . . , Un , respectively, then it follows that U(j) = F(X(j) ) for j = 1, . . . , n, by the monotonicity of F. These results are fundamental to the development of important fields of statistics such as the distribution theory of order statistics∗ (see ref. 3) and tolerance regions∗ theory (see ref. 6). The PIT result has an important ‘‘inverse’’ which we consider next. If X is a random variable with distribution function F(x)—continuous or not—mapping the real number line to the unit interval (0, 1), we define an inverse function, say F − , that maps the unit interval to the real number line, by F − (u) = inf{x : F(x) u} for 0 < u < 1. If U is a U(0, 1) random variable and if Y = F − (U), then X and Y are identically distributed random variables [see ref. 19 (p. 203)]. This result also has many important applications in statistics. One use is to generate observations from an arbitrary distribution in simulation∗ studies by drawing
PROBABILITY INTEGRAL TRANSFORMATIONS
observations on a U(0, 1) random variable and transforming them with this result. By combining the two transformations above, we can transform a random variable with any particular continuous distribution to one with any other distribution. That is, if X has continuous distribution function F and Y has any distribution function G, then Y and the random variable W = G− (F(X)) are identically distributed. If F and G are both continuous, this is sometimes called a quantile transformation∗ , since it transforms the pth quantile of the distribution of X to the pth quantile of the distribution of Y for 0 < p < 1. Note the generality of these transformation results. They hold for all continuous distributions! We note also a point that will arise later in different contexts. The results hold even if the distribution functions depend on additional variables; those that will be of interest later will be either distribution parameters or random variables that are conditionally fixed.
MULTIVARIATE PROBABILITY INTEGRAL TRANSFORMATION (MPIT) The PIT is sometimes used to transform the elements of a given sample to a sample from a U(0, 1) distribution. We now consider a result due to Rosenblatt [20] that shows how to transform a set of possibly dependent continuous random variables to a set of i.i.d. U(0, 1) random variables. Let (X1 , . . . , Xk ) be a random vector with an absolutely continuous distribution function F(x1 , . . . , xk ). Then put (u1 , . . . , uk ) = H(x1 , . . . , xk ), where the transformation H is defined by u1 = P{X1 x1 } ≡ F1 (x1 ), u2 = P{X2 x2 |X1 = x1 } ≡ F2 (x2 |x1 ), .. . uk = P{Xk xk |X1 = x1 , . . . , Xk−1 = xk−1 } ≡ Fk (xk |x1 , . . . , xk−1 ). For u = (u1 , . . . , uk ) , put Su = {(u1 , . . . , : uj uj , j = 1, . . . , k}, and the distribution function of U ≡ H(X1 , . . . , Xk ) is given by uk )
6477
P(Su ) = P{Uj uj , j = 1, . . . , k}, = P{X1 x1 }P{X2 x2 |x1 } · · · ×P{Xk xk |x1 , . . . , xk−1 } = u1 u2 . . . uk , and 0 uj 1; j = 1, . . . , k. We have the following result. Theorem 2. If (X1 , . . . , Xk ) is a random vector with absolutely continuous distribution function F(x1 , . . . , xk ), then U1 = F1 (X1 ), U2 = F2 (X2 |X1 ), . . . , Uk = Fk (Xk |X1 , . . . , Xk−1 ) are i.i.d. U(0, 1) random variables. This result can be used in problems in statistical inference for multivariate distributions. If X1 , . . . , Xn is a sample of k-variate independent vector observations from an absolutely continuous multivariate distribution, then Theorem 2 shows how these observations can be transformed to nk i.i.d. U(0, 1) random variables. If, for example, the distribution function is known under a simple goodnessof-fit∗ null hypothesis, then a test for this hypothesis can be constructed by utilizing the i.i.d. U(0, 1) distributions of the nk U’s under the null hypothesis. It should be observed that the random variables U1 , . . . , Uk in Theorem 2 depend on the order of X1 , . . . , Xk . If these X’s are written in a different order, then a set of i.i.d. U(0, 1) random variables will be obtained, but they will, in general, be different random variables. This lack of invariance to permutations of the X’s can be important in some contexts. CONDITIONAL PROBABILITY INTEGRAL TRANSFORMATIONS (CPIT) When an observed sample is at hand, in order to actually apply either the classic PIT of Theorem 1 or the multivariate PIT of Theorem 2, it is necessary to know the distribution function required exactly. This requirement limits the use of such transformations in many statistical problems, since we often can assume only that the distribution function is a member of some specified class of distribution functions. This fact led Johnson and David [7] in an early pioneering
6478
PROBABILITY INTEGRAL TRANSFORMATIONS
paper to replace the parameters in a distribution function (df) by estimators. They showed that for location-scale classes, the transformed sample values, using this df with estimated parameters, have distributions that do not depend on the values of the parameters. In this section we give some transformations which can be used when the parent distribution function for a sample is assumed only to be a member of a parametric class of distribution functions. Although the transformations we consider here generalize to multivariate or more complex families of distributions, such as distributions dependent on structural variables, the initial presentation will be for i.i.d. univariate samples. Let X1 , . . . , Xn be i.i.d. random variables with common distribution function F assumed to be a member of a parametric class F = {Fθ : θ ∈ } of continuous distribution functions. Suppose that the family F admits a v-component sufficient statistic∗ Tn , which we assume, without loss of generality, is a symmetric function∗ of X1 , . . . , Xn , or, equivalently, that Tn is a function of the order statistics of this sample. Let F˜ n (x1 , . . . , xα ) denote the conditional distribution function defined by
is a maximum when Tn is a minimal sufficient statistic. This theorem is most often useful when F is a parametric family with, say, v functionally independent parameters. In many common cases the maximum value of α for which F˜ n (x1 , . . . , xα ) is absolutely continuous is then α = n − v, that is, the sample size reduced by the number of parameters. Even though Theorem 3 gives general transformation results, in particular cases writing out the transformations can itself be a difficult task, so it is worthwhile to consider particular techniques and results that are helpful. The next result is given for this purpose. A sequence of statistics (Tn )n1 is doubly transitive if for each n we can compute Tn+1 from Tn and Xn+1 ; and conversely, if we can compute Tn from Tn+1 and Xn+1 . Many important families of distributions have doubly transitive minimal sufficient statistics. When this is the case, the following result from [10] gives the transformations of Theorem 3 in a form that is easier to find in many problems.
F˜ n (x1 , . . . , xα ) = E{I[X1 x1 ,...,Xα xα ] |Tn },
U1 = F˜ n−α+1 (Xn−α+1 ), . . . , Uα = F˜ n (Xn )
α ∈ {1, . . . , n}. This is an unbiased estimating distribution function for F(x1 , . . . , xα ); when Tn is complete as well as sufficient, it is the minimum variance unbiased∗ (MVU) estimating distribution function. It is sometimes called the Rao-Blackwell (R-B) distribution function. The basic result is stated using these R-B distribution functions. Theorem 3. If F˜ n (x1 , . . . , xα ) is absolutely continuous for an α ∈ {1, . . . , n}, then the α random variables U1 = F˜ n (X1 ), U2 = F˜ n (X2 |X1 ), Uα = F˜ n (Xα |X1 , . . . , Xα−1 )
···,
are i.i.d. U(0, 1) random variables. This result is proved in O’Reilly and Quesenberry [10], where it is shown also that α
Theorem 4. If F˜ n (x1 , . . . , xα ) is absolutely continuous for α ∈ {1, . . . , n}, and (Tn )n1 is doubly transitive, then
is a set of α i.i.d. U(0, 1) random variables. The R-B distribution functions are known for many classes of distributions, and the transformations of Theorem 4 can be readily obtained explicitly for these classes. Example: Normal Distributions. Suppose that X1 , . . . , Xn is a random sample 2 distribution. Then from a N(µ, σ ) parent (X n , S2n ) = ( Xj /n, (Xj − X)2 /n) is a doubly transitive minimal sufficient statistic, since one can compute (X n+1 , S2n+1 , Xn+1 ) from (X n , S2n , Xn+1 ) and conversely. Moreover, the R-B distribution function is given (see ref. 8) for r > 2 by z − X r < −(r − 1)1/2 Sr , 0, ˜ Fr (z) = 1, (r − 1)1/2 Sr < z − X r , Gr−2 (Ar ), elsewhere,
PROBABILITY INTEGRAL TRANSFORMATIONS
(r − 2)1/2 (z − X) Ar =
1/2 , (r − 1)S2r − (z − X r )2 where Gr−2 is a Student t-distribution∗ function with (r − 2) degrees of freedom. Using this result in Theorem 4, we see that the n − 2 random variables given by Ur−2 = Gr−2 (Br ), (r − 2)1/2 (Xr − X r ) Br =
1/2 , (r − 1)S2r − (Xr − X r )2 for r = 3, . . . , n, are i.i.d. U(0, 1) random variables. The conditioning approach discussed above can be used in some cases for models more complex than i.i.d. univariate samples. Multivariate normal distributions were considered in ref. 10 and in more detail in refs. 17 and 18. In these papers the transformations were used to establish goodness-of-fit tests for multivariate normality. The assumption that the original values X1 , . . . , Xn are identically distributed can also be relaxed in some important models such as regression or design models with normal errors. Some design models have been considered in ref. 15, which gives a multiple samples test of normality, and in ref. 16, which gives a test for a one-way analysis of variance∗ model. The independence assumption of the variables X1 , . . . , Xn may also be relaxed in Theorem 3; however, the relatively simple approach of Theorem 4 is not then available, and finding the transformations is in most cases a formidable task. One place where this approach is potentially helpful is the following. It was pointed out earlier that the MPIT is not a symmetric function of X1 , . . . , Xn , and depends on the order of the variables. The same is true for the CPIT, in general; this has a number of practical consequences. First, when the transformations are used to establish a test of fit, it is possible to obtain different test results from different orderings of the same data. The probability of agreement of tests based on different randomly selected permutations of the sample has been considered in ref. 13, where
6479
this probability was found to be high in a large class of cases. Another approach is to replace (X1 , . . . , Xα ) in Theorem 3 by a subset of α order statistics, say (X(1) , . . . , X(α) ), where X(1) X(2) · · · X(n) . Then the transformed values will be functions only of the order statistics and will therefore be symmetric functions of the original sample. The task of finding the actual transformation is difficult with this approach, and has been solved only for two rather simple cases of uniform classes [12] and scale parameter exponential classes [11]. The transformations obtained for the exponential distribution∗ are characterizations of the distribution that are closely related to other characterizing transformations studied in ref. 2. A second consequence of the order dependence of the CPIT transformations is that it may be exploited to investigate the dependence of a data set on an ordering variable, such as time. FURTHER REMARKS ON CPITS The CPIT approach is useful mainly for parametric classes of distributions that admit fixed-dimension sufficient statistics∗ . Such classes are, under regularity conditions, exponential classes, so that it appears that this method is useful largely just for exponential classes. However, in ref. 12 an extension of the foregoing conditioning approach is considered which gives results for many important families that are not exponential classes because their support is a function of parameters, such as uniform or Pareto distributions∗ . The transformations of Theorem 4 possess interesting and useful properties. They have important invariance properties [14]. For a parent class of distribution functions F that has an associated class of transformations G, say, these transformations will themselves be invariant under the members of G. To illustrate, suppose that F is a location-scale parametric family of distributions. Then the transformations will be invariant under linear transformations: g(xj ) = axj + b; j = 1, . . . , n; a > 0. This invariance can readily be verified for the transformations of the normal distribution in the preceding section. The CPITs possess
6480
PROBABILITY INTEGRAL TRANSFORMATIONS
similar invariance properties quite generally. This invariance property can sometimes be helpful in the derivation of most powerful invariant tests for separate hypothesis testing∗ problems (see ref. 14), or, to obtain optimal invariant selection statistics (see MODEL CONSTRUCTION: SELECTION OF DISTRIBUTIONS). If the sufficient statistic T is also complete, then T = (T1 , . . . , Tv ) and U = (U1 , . . . , Un−v ) are independent, since the distribution of U is the same for every parent distribution in the class F, for which T is sufficient and complete [1] (see also BASU THEOREMS). This result has important implications. T contains all the information about the parameters of F, that is, all information on which inferences can be made within the class F. Thus U can be used to make inferences about the class F, perhaps to make goodness-of-fit tests or conduct model selection procedures, and T can be used to make a parametric test or other inference within F, and the independence of U and T exploited to assess overall error rates. The remarks above suggest that the transformations are useful largely for investigating the correctness of the parent class F, which will usually be posed under a null hypothesis. It is likely that additional applications of the conditioning approach to transformations and inference described here will be developed. REFERENCES 1. Basu, D. (1955). Sankhya, ¯ 15, 377–380. (The now well-known condition for a statistic to be independent of a complete sufficient statistic was given.) 2. Cs¨org¨o, M. and Seshadri, V. (1971). Z. Wahrschlein. verw. Geb., 18, 333–339. (‘‘Characterizing’’ transformations to uniformity are considered for exponential and normal distributions. See also references to other papers in this paper.) 3. David, H. A. (1981). Order Statistics, 2nd ed. Wiley, New York. (A general reference for order statistics.) 4. Fisher, R. A. (1930). Proc. Camb. Philos. Soc., 26, Part. 4, 528–535. (Fisher introduced ‘‘fiducial’’ intervals and bounds for a parameter. The distribution function was used as a ‘‘pivotal’’ function, and this was the first use of the PIT.)
5. Fisher, R. A. (1932). Statistical Methods for Research Workers, 4th ed. Oliver & Boyd, Edinburgh. (Fisher implicitly used the PIT to define his ‘‘combined test for significance.’’) 6. Guttman, I. (1970). Statistical Tolerance Regions: Classical and Bayesian. Hafner, Darien, Conn. (A general reference for tolerance regions.) 7. Johnson, N. L. and David, F. N. (1948). Biometrika, 35, 182–190. (The members of a sample are transformed using a distribution function with parameters replaced by estimates.) 8. Lieberman, G. J. and Resnikoff, G. J. (1955). J. Amer. Statist. Ass., 50, 457–516. 9. Lindgren, B. W. (1968). Statistical Theory, 2nd ed. Macmillan, New York. 10. O’Reilly, F. J. and Quesenberry, C. P. (1973). Ann. Statist., 1, 79–83. (This paper introduces the important general principle of conditioning a set of RVs on a sufficient statistic and then using this conditional distribution to transform the RVs themselves—the conditional probability integral transformation, CPIT. Results that are helpful for performing these transformations are given for the special case when the original RVs constitute a random sample from an exponential class of distributions.) 11. O’Reilly, F. J. and Stephens, M. A. (1982). J. R. Statist. Soc. B, 44, 353–360. (Obtain the CPIT for order statistics for the scale parameter exponential distribution.) 12. Quesenberry, C. P. (1975). Commun. Statist. A, 4, 1149–1155. (This paper extends the CPIT theory for samples to distributions with ranges dependent on parameters.) 13. Quesenberry, C. P. and Dietz, J. (1983). J. Statist. Comp. Simul., 17, 125–131. (Studies agreement probabilities for CPIT-Neyman smooth tests made on randomly selected permutations of a sample.) 14. Quesenberry, C. P. and Starbuck, R. R. (1976). Commun. Statist. A, 5, 507–524. (A theoretical paper that studies optimality properties of CPITs.) 15. Quesenberry, C. P., Whitaker, T. B., and Dickens, J. W. (1976). Biometrics, 32, 753–759. (Gives a test for normality for multiple samples.) 16. Quesenberry, C. P., Giesbrecht, F. G., and Burns, J. C. (1983). Biometrics, 39, 735–739. (Consider testing normality in a one-way ANOVA model.) 17. Rincon-Gallardo, S. and Quesenberry, C. P. (1982). Commun. Statist. Theor. Meth., 11,
PROBABILITY MEASURES, TIGHTNESS OF
18.
19.
20.
21.
343–358. (An applied paper that considers applying the multivariate normality test of ref. 18.) Rincon-Gallardo, S., Quesenberry, C. P., and O’Reilly, F. J. (1979). Ann. Statist., 7, 1052–1057. (Proposes a multiple samples test for multivariate normality.) Rohatgi, V. K. (1976). An Introduction to Probability Theory and Mathematical Statistics. Wiley, New York. Rosenblatt, M. (1952). Ann. Math. Statist., 23, 470–472. (Brief but important paper that gives the multivariate probability integral transformation.) Wallis, W. A. (1942). Econometrica, 10, 229–248. (Interesting historically since it explains why Fisher’s combined test of significance works. Some earlier writers had given fallacious accounts.)
See also DISTRIBUTION-FREE METHODS; GOODNESS OF FIT; MULTIVARIATE NORMAL DISTRIBUTIONS; QUANTILE TRANSFORMATION METHODS; RANDOM NUMBER GENERATION; and UNIFORM DISTRIBUTION MODULO 1.
C. P. QUESENBERRY
PROBABILITY IN THE ENGINEERING AND INFORMATIONAL SCIENCES This journal commenced publication in 1987; the Editor is Sheldon M. Ross and the Editorial Office is Department of Industrial Engineering and Operations Research, University of California, Berkeley, CA 94720. All papers must be in English. ‘‘This journal will be concerned with articles detailing innovative uses of probability in a variety of areas. Among others, we will be especially interested in applications to computer and information science, the engineering sciences, applied statistics, and reliability theory.’’ [From the Introductory Editorial, Vol. 1, No. 1 (1987).] The journal is published by Cambridge University Press, New York. There is a link at www.cambridge.org/uk/journals/
PROBABILITY, INVERSE. See BAYESIAN INFERENCE
PROBABILITY MEASURE Formally, a probability measure is defined as a measure∗ µ on an arbitrary space such
6481
that µ() = 1. We then define an event as a measurable set and the probability of an event E as µ(E). The space is called ‘‘space of elementary events’’ or ‘‘sample space.’’ In applications, a probability measure is associated with our conceptions of the likelihoods of occurrences of various physical events. Probability measures which are defined on the real line (R1 ) result in univariate (one-dimensional) probability distributions, while those on ndimensional Euclidean space yield multivariate (n-dimensional) probability distributions. See MEASURE THEORY IN PROBABILITY AND STATISTICS and PROBABILITY THEORY: AN OUTLINE and entries on specific distributions for more details. BIBLIOGRAPHY Cram´er, H. (1946). Mathematical Methods of Statistics. Princeton University Press, Princeton, N.J., Chaps. 4 to 9. Lo`eve, M. (1979). Probability Theory, 4th ed. Springer-Verlag, New York. (See especially Chaps. I to III.) See also MEASURE THEORY IN PROBABILITY AND STATISTICS.
PROBABILITY MEASURES, TIGHTNESS OF A collection of probability measures {µn } on the real line is tight if for all > 0 there exist a and b, a < b, with µn ([a, b]) 1 − for all n. In other words, all the measures give most of their mass to the same finite interval; the probability mass does not ‘‘escape to infinity.’’ Any finite collection of probability measures is tight. A subcollection of a tight collection is tight and, finally, the union of two tight collections of probability measures must again be tight. The concept is of special importance in continuity theorems for characteristic functions* [1].
6482
PROBABILITY MINIMUM REPLACEMENT SAMPLING DESIGN
REFERENCE
and
1. Rosenthal, J. S. (2000). A First Look at Rigorous Probability Theory. World Scientific Publishers, Singapore. See also MEASURE THEORY IN PROBABILITY AND STATISTICS.
PROBABILITY MINIMUM REPLACEMENT SAMPLING DESIGN We will use the following notation for the parameters of a sampling design: N:
number of sampling units
S(i) :
size measure associated with
sampling unit(i) N S(+) = S(i)
P{n(i) = m(i)} = 1 − f (i). [Observe that PNR sample designs are a special case of PMR designs with all m(i) equal to zero.] A selection algorithm for a sequential procedure for a PMR sampling and associated unbiased estimation procedures was developed by J. R. Chromy [1]. REFERENCE 1. Chromy, J. R. (1981). In Current Topics in Survey Sampling, D. Krewski, R. Platek, and J. N. K. Rao, eds. Academic Press, New York, pp. 329–347. See also PROBABILITY PROPORTIONAL TO SIZE (PPS) SAMPLING and SURVEY SAMPLING.
i=1
n:
total sample size
n(i) :
number of times sampling unit
PROBABILITY MOMENTS AND FREQUENCY MOMENTS
(i)is selected Probability nonreplacement (PNR) sample designs are (usually) restricted to sampling frames with nS(i)/S(+) < 1, i = 1, . . . , N. When this condition holds, a PNR sample design is one for which Pr(n(i) = 1) = nS(i)/S(+)
For a random variable X having a discrete distribution, with Pr[X = xj ] = pj ,
j = . . . , −1, 0, 1, . . . ,
the rth probability moment is ωr =
∞
prj .
(1)
j=−∞
and Pr(n(i) = 0) = 1 − Pr(n(i) = 1). Probability minimum replacement (PMR) sample designs, to be defined below, allow any positive size measures S(i). The standardized size measure may be written in terms of two components nS(i)/S(+) = m(i) + f (i), where m(i) is a nonnegative integer, while 0 f (i) 1. Now, using this notation, a PMR sample design is one for which Pr{n(i) = m(i) + 1} = f (i)
If the distribution is continuous with probability density function f (x), it is defined as r =
∞ −∞
{f (x)}r dx.
(Note that ω1 = 1 = 1.) These quantities were introduced by Sichel [1,2], who used sample analogs of these parameters in fitting distributions. He also defined ratios r+2 /2r+1 , r = 1, 2, 3, . . . , αr = (2r+1)/2 (2r+5)/2 /3/2 , r = 12 , 32 , 52 , . . . , as ‘‘measures of kurtosis∗ .’’ For continuous distributions he suggested using ωr , as
PROBABILITY PLOTTING
defined in (1) with a ‘‘discretized’’ (grouped) form of the distribution, as an approximation to r , and termed the sample analogs ‘‘working’’ probability moments. The rth working frequency moment for a sample of size n, is nr times the rth working probability moment. Sichel [2] gives an account of accuracy to be expected using probability moments to estimate parameters of a distribution.
Fn (x) = (number of observations x)/n. It is a step function increasing from 0 to 1 with jumps of size 1/n at each observation (see EMPIRICAL DISTRIBUTION FUNCTION (EDF) STATISTICS). Fn (x) is a natural, nonparametric estimate of F(x) and its graph can be studied informally in making inferences about F(x). A simultaneous (1 − α)100% confidence band for F(x) is Fn (x) − kα F(x) Fn (x) + kα
REFERENCES 1. Sichel, H. S. (1947). J. R. Statist. Soc. Ser. A, 110, 337–347. 2. Sichel, H. S. (1949). Biometrika, 36, 404–425. See also ESTIMATION: METHOD OF MOMENTS and PROBABILITY-WEIGHTED MOMENTS.
PROBABILITY PLOTTING Plots and graphs of a wide variety are used in the statistical literature for descriptive purposes. Visual representations have high impact and they provide insight and understanding not readily found by other means. This article will consider methods for the oneand two-sample problems that have plots as their central focus. These methods are useful for informal data analysis and model assessment, while more formal statistical inferences can also be made. Fisher [12] has given an excellent review. See GRAPHICAL REPRESENTATION OF DATA and GRAPHICAL REPRESENTATION, COMPUTER AIDED for more examples of plots. CUMULATIVE DISTRIBUTION FUNCTION PLOTS The cumulative distribution function (CDF) of a random variable X is F(x) = Pr(X x). Many features of the distribution of X are apparent in the CDF, such as likelihood regions, location, scale, percentiles, and skewness. A random sample of size n on X is a collection of independent random variables, each having CDF F(x). The observed, ordered observations for the sample are denoted by x1 x2 · · · xn . The empirical CDF is
6483
for all x,
where Pr(D kα ) = 1 − α and D = supx |Fn (x) − F(x)|, the Kolmogorov–Smirnov statistic∗ . The confidence band should be plotted with Fn (x) to assess variability. A formal level α test of a hypothesis H0 : F(x) = F0 (x), for a specified CDF F0 (x), can be carried out with the rule: Accept H0 if F0 (x) lies in the confidence band for all x. The confidence band thus contains all CDFs that would be accepted in this testing problem. Doksum [9] has improved on this graphical method by developing a confidence band using sup aFn (x)b
|Fn (x) − F(x)| {F(x)(1 − F(x))}1/2
!
as a pivot, where 0 < a < b < 1. The weight function in the denominator serves to equalize the variance of the empirical CDF and the interval (a, b) restricts the scope of the band. The resulting confidence band is slightly wider in the middle but is much narrower in the tails when compared to the first band. QUANTILE AND PERCENT PLOTS Suppose that two CDFs F(x) and G(x) are given. These may be empirical or population CDFs. A graphical comparison of the two distributions can be made by plotting the quantiles∗ of one distribution against those of the other. The quantile of order p for the CDF F(x) is a number xp which satisfies F(xp ) = p when F(x) is strictly increasing, but to cover other cases define F −1 (p) = inf {x : F(x) p} and use xp = F −1 (p). Similarly, let yp be the quantile of order p for G(x). Then a quantile plot, called a Q-Q plot, is the graph of (xp , yp ) for 0 < p < 1.
6484
PROBABILITY PLOTTING
If G(x) = F(x), the Q-Q plot will be a 45◦ line through the origin. If the two distributions differ only in location and scale, G(x) = F((x − µ)/σ ) and yp = µ + σ xp . Thus the Q-Q plot is a straight line with intercept µ and slope σ . This is an important prototype case since the eye can readily perceive straight lines and identify slope and intercept. Departures from linearity can be easily spotted and interpreted. In a similar manner, a percent plot, called a P-P plot, is constructed by graphing the points (F(x), G(x)) for −∞ < x < ∞. This plot will trace a path from (0, 0) to (1, 1). If G(x) = F(x), the plot will be a straight line. Unlike the Q-Q plot, the P-P plot will not be linear in the case of only location-scale differences. Differences in the middles of distributions are usually more apparent for P-P plots than for Q-Q plots. Generalization to multivariate cases is easier with percent plots. The basic concepts behind these plots can be used in other creative ways. For example, ref. 16 has some good examples of plots of functions h1 (xp , yp ) versus h2 (xp , yp ) or just p. Probability plots are a useful tool in data analysis. One can assess whether a single random sample arises from a distribution with CDF F0 (x) by constructing a Q-Q or a P-P plot of F0 (x) against the empirical CDF Fn (x) of the sample. The Q-Q plot would be represented by points (F0−1 [(i − 1/2)/n], xi ) plotted for i = 1, . . . , n, where x1 · · · xn are the ordered observations. The first coordinates are quantiles of order (i − 12 )/n for F0 (x). Alternatively, quantiles of order i/(n + 1) can be used. If the plot follows a 45◦ line through the origin, the CDF F0 (x) would informally be accepted. Departures from this line would indicate differences between F0 (x) and the actual CDF. Location or scale differences, heavier tails, existence of outliers∗ , regions of poor fit, and other peculiarities may be noticed. The Kolmogorov-Smirnov test could be used for a more formal test. Probability plotting has been widely used for location-scale models, where the observations are assumed to arise from a distribution with CDF F0 ((x − µ)/σ ), with F0 known but µ and σ unknown. A Q-Q plot of F0 (x) against Fn (x), as given in the preceding paragraph, will informally test the adequacy of the model. When F0 (x) is the standard
normal CDF, this is called a normal probability plot. If the location-scale model is correct, the plot should lie along a straight line with intercept µ and slope σ . Departures from linearity would provide insights for choosing an alternative model. See GRAPHICAL REPRESENTATION OF DATA for some examples. Doksum [9] has discussed a formal confidence band that can be graphed and used to test the adequacy of the locationscale model. His procedure is based on a weighted Kolmogorov–Smirnov statistic and has an advantage over the usual Kolmogorov–Smirnov statistic in that µ and σ need not be estimated from the data. OPTIMAL PLOTTING POSITIONS With the probability plot for the locationscale model, the slope σ and intercept µ can be crudely estimated from the graph. To improve on this it is natural to use the leastsquares∗ fitted line. Let the plotted points be denoted by (vi , xi ), where the xi are the ordered observations and the vi are called the plotting positions. The basic Q-Q plot uses quantiles from F0 for the plotting positions. Chernoff and Lieberman [3] considered the problem of determining optimal plotting positions in the sense of minimum mean square error or of unbiased, minimum variance estimates. The resulting optimal plotting positions, given in their paper, are functions of the expectations, variances, and covariances of the order statistics of a sample from F0 (x). An undesirable feature is that the optimal plotting positions need not be in an increasing order. The order statistics∗ used in the plot are correlated and have unequal variances, and as a result a weighted least-squares∗ procedure would be preferable to the ordinary least-squares fit. Cran [4] gave the weighted least-squares solution to the optimal plotting position problem. He also obtained conditions on the weights for the plotting positions to be in increasing order. Barnett [1] noted that the previous results require knowledge of the first and second moments of order statistics, and this information is available only for selected sample sizes and distributions F0 . He suggested an alternative approach which
PROBABILITY PLOTTING
does not require this information and which yields estimates of µ and σ that are nearly optimal. GAMMA PROBABILITY PLOTS The gamma probability distribution is widely used in life testing∗ and reliability∗ models. It is indexed on three parameters: location, scale, and shape. A graphical method for assessing a gamma probability model consists of making an initial estimate of the shape parameter and then constructing a Q-Q plot to check for linearity as in the preceding section. Reference 17 contains several interesting examples of this approach. It illustrates an important application in assessing the assumption of homogeneity of variance in analysis-of-variance∗ problems. Under the usual assumptions of independence, normality, and equal variance, the within-cell sample variances should have a gamma distribution∗ and the gamma plot can be used to spot any discrepancies. HALF-NORMAL PLOTS∗ The half-normal distribution is the distribution of the absolute value of a standard normal random variable; its CDF is 0 (x) = 2(x) − 1 for x > 0, where is the standard normal CDF (see FOLDED DISTRIBUTIONS). A half-normal probability plot of a sample is a regular Q-Q plot of the quantiles of the absolute values of the data |x1 | < · · · < |xn | against the corresponding quantiles of the half-normal distribution, that is, a plot 1 of (−1 0 [(i − 2 )/n], |xi |) for i = 1, . . . , n. If the sample arises from a normal distribution with mean 0 and variance σ 2 , this plot should follow a straight line through the origin with slope σ . Daniel [5] presented an interesting application of this plot in the analysis of 2p factorial experiments∗ . The usual set of 2p − 1 orthogonal sample contrasts∗ are used for the half-normal plot. Departures from linearity may indicate heteroscedasticity∗ , outliers, or real effects. Formal tests for real effects are given; see also Birnbaum [2]. Zahn [18] gave several improvements for Daniel’s procedure and corrected the critical limits for the test.
6485
GRAPHICAL ANALYSIS FOR TWO-SAMPLE PROBLEMS Plots can be used to detect similarities and differences between two distributions. Interest may focus on locations, scales, shapes, tail weight, symmetries, skewness, stochastic ordering, or other features. Assuming two independent samples, let x1 · · · xm be the ordered sample from a distribution with CDF F(x) and let Fm (x) be the empirical CDF of the sample; let y1 · · · yn be the ordered sample from a distribution with CDF G(y) and let Gn (y) be the empirical CDF of this sample. The two samples can be compared with a Q-Q plot of the empiricals Fm versus Gn . If the sample sizes are equal, this is simply a plot of the points (xi , yi ) for i = 1, . . . , n. For −1 (p) = unequal sample sizes, recall that Fm inf {x : Fm (x) p} will take the value xi if (i − 1)/m < p i/m and similarly for G−1 n . The Q-Q plot is a plot of the points arising from −1 (p), G−1 (p)) for 0 < p < 1 and this will (Fm n consist of points (xi , yi ) for suitable subscripts. THE SHIFT FUNCTION FOR TWO-SAMPLE PROBLEMS The Kolmogorov–Smirnov statistic D = supx |Fm (x) − Gn (x)|, or a weighted version of it, can be used to test the hypothesis of identical CDFs for the two-sample problem. This test can be carried out with an additional emphasis on plotting and graphical analysis to provide more information and insight than the basic accept–reject decision. A discussion of this can be found in Doksum and Sievers [9] and Doksum [6,7,9]. To motivate this approach, consider the widely used shift model F(x) = G(x + ) for all x, where the Y-population is shifted a uniform amount from the X-population. Comparing populations by a simple shift may often be unrealistic, however. More generally, a shift function (x) can be defined by the equation F(x) = G(x + (x)) for all x. It follows that (x) = G−1 (F(x)) − x (see Fig. 1). Under general conditions (x) is the only function for which X + (X) has the same distribution as Y. We say that (X) is the amount of shift to bring the X variable up to
6486
PROBABILITY PLOTTING
be graphed to explore its properties. The plot of G−1 n (Fm (x)) at the values xi is basically the regular Q-Q plot. A simultaneous confidence band for (x) can be constructed for more formal inferences. One such band is based on the Kolmogorov–Smirnov statistic D . With kα chosen so that Pr(D kα |F = G) = 1 − α, a simultaneous (1 − α)100% confidence band for (x) is given by yi1 − xi (x) < yi2 − xi Figure 1. Shift function (x) between two CDFs F and G.
for xi x < xi+1 , where x0 = −∞, xm+1 = ∞,
the Y. When X is a control response and Y a treatment response, it may be reasonable to interpret (x) as the amount that the treatment adds to a control response x. This effect need not be constant. The set {x : (x) > 0} identifies the part of the population where the treatment has a positive effect. When the pure shift model holds, (x) = , a constant. When a location-scale model holds, G(x) = F((x − µ)/σ ) and (x) = µ + (σ − 1)x, a linear function. A natural, nonparametric estimate of (x) ˆ is (x) = G−1 n (Fm (x)) − x and this function can
ˆ Figure 2. Shift function (x) and an 80% confidence band for simulated samples of sizes n = m = 50, where X and Y are N(0, 1) and N(1, 4), respectively.
i1 = n((i/m) − kα ), i2 = [n((i/m) + kα )] + 1, t is the least integer greater than or equal to t, and [t] is the greatest integer less than or equal to t. An example is given in Fig. 2. The confidence band can be used to compare the two distributions in various ways. Does it contain a horizontal line (a pure shift model is accepted), a line through the origin (0, 0) (a pure scale model is accepted), or any straight line (a location-scale model is accepted)? A confidence set will be visible
PROBABILITY PLOTTING
for the set {x : (x) > 0} and similarly other conclusions can be reached, as many in number as desired because of the simultaneous nature of the band. Doksum [9,10] gave several other nonparametric bands for (x) that can be more efficient than the band discussed here. Also, special bands were developed for the locationscale case. In life testing and reliability studies, multiplicative effects have more direct interest than additive effects. For such cases a scale function θ (x) = G−1 (F(x))/x can be useful. Doksum [8] has developed appropriate graphical analyses and interesting applications for the scale function, including methods for the Weibull∗ and exponential∗ models. PLOTTING FOR SYMMETRY Suppose that X is a random variable with a continuous CDF F(x). When the distribution is symmetric about a point θ , this point is clearly the measure of location or center. In asymmetric cases, the location is not so clear. Many statistical procedures require symmetric distributions and it is useful to have methods to check on this assumption. Doksum [7] has discussed some useful graphical methods. He defined a symmetry function by θ (x) = (x + F −1 (1 − F(x)))/2; it is essentially the only function for which X − θ (X) and −(X − θ (X)) have the same distributions. For symmetric distributions θ (x) = θ , a constant. When the distribution is skewed to the right (left), θ (x) is U-shaped (upsidedown U-shaped). He also discussed a location interval [θ, θ ], where θ = infx θ (x) and θ = supx θ (x), and showed that all reasonable location measures lie in this interval. The empirical CDF Fn (x) of a sample is used to estimate θ (x) with θˆ (x) = (x + Fn−1 (1 − Fn (x)))/2. Using the ordered data x1 < · · · < xn , the graph of θˆ (x) is basically given by a plot of the points (xi , (xi + xn+1−i )/2) for i = 1, . . . , n. This plot and a simultaneous confidence band should be plotted to assess asymmetry and the location interval.
6487
REFERENCES 1. Barnett, V. (1976). Appl. Statist., 25, 47–50. (A paper on plotting positions.) 2. Birnbaum, A. (1959). Technometrics, 1, 343–357. (On theory of hypothesis testing for 2p factorials, half-normal plots.) 3. Chernoff, H. and Lieberman, G. (1956). Ann. Math. Statist., 27, 806–818. (Basic paper on estimation of location and scale; theoretical results.) 4. Cran, G. W. (1975). J. Amer. Statist. Ass., 70, 229–232. (Use of weighted least-squares line.) 5. Daniel, C. (1959). Technometrics, 1, 311–341. (Basic reference on half-normal plots for 2p factorials; readable, examples.) 6. Doksum, K. (1974). Ann. Statist., 2, 267–277. (First paper on shift function, theoretical.) 7. Doksum, K. (1975). Scand. J. Statist., 2, 11–22. (On plots for asymmetry, theoretical, has examples.) 8. Doksum, K. (1975). In Reliability and Fault Tree Analysis. SIAM, Philadelphia, pp. 427–449. (Focus on plotting with life testing, reliability data; lots of examples and applications.) 9. Doksum, K. (1977). Statist. Neerlandica, 31, 53–68. (Good survey on a wide range of plotting methods; examples and applications, readable.) 10. Doksum, K. and Sievers, G. (1976). Biometrika, 63, 421–434. (Basic paper on plotting for shift functions, location/scale model; theory and discussion.) 11. Fisher, N. I. (1981). Austral. J. Statist., 23, 352–359. 12. Fisher, N. I. (1983). Int. Statist. Rev., 51, 25–58. (Good review of many graphical methods; extensive bibliography.) 13. Hoaglin, D. C. (1980). Amer. Statist., 34, 146–149. (Plots for Poisson and other discrete distributions; readable.) 14. Nair, V. N. (1981). Biometrika, 68, 99–103. (Plots for censored data.) 15. Schweder, T. and Spjøtvoll, E. (1982). Biometrika, 69, 493–502. (Plots of p values for many tests.) 16. Wilk, M. B. and Gnanadesikan, R. (1968). Biometrika, 55, 1–17. (Basic paper on Q-Q and P-P plots; lots of examples and applications, readable.) 17. Wilk, M. B., Gnanadesikan, R., and Huyett, M. J. (1962). Technometrics, 4, 1–15. (Basic paper on gamma probability plots.)
6488
PROBABILITY PROPORTIONAL TO SIZE (PPS) SAMPLING
18. Zahn, D. A. (1975). Technometrics, 17, 189–200. (On modification and corrections for Daniel’s half-normal plots.)
BIBLIOGRAPHY (EDITORS’ ADDITION) Harter, H. L. (1984). Commun. Statist. Theor. Meth., 13, 1613–1633. (Contains 107 references.) See also GRAPHICAL REPRESENTATION, COMPUTER AIDED; GRAPHICAL REPRESENTATION OF DATA; HALF-NORMAL PLOTS; KOLMOGOROV –SMIRNOV STATISTICS; and QUANTILE.
GERALD L. SIEVERS
PROBABILITY PROPORTIONAL TO SIZE (PPS) SAMPLING USES PPS sampling is used principally in two contexts. Multi-Stage Sampling Samples are often selected in a multistage manner [16, Chap. 10]. For example, in a two-stage national household survey, a sample of areas, the primary sampling units∗ (PSUs), may be selected within each of which, subsamples of households, the subunits, are drawn. A common procedure is to sample the PSUs with probability proportional to size, that is, the number of subunits within each PSU, and to select a fixed number of subunits within each PSU by simple random sampling∗ (SRS). This yields a self-weighting sample; that is, each subunit has equal probability of selection so that no reweighting is necessary for estimation purposes. The PPS strategy above has practical advantages over other self-weighting procedures, such as SRS of PSUs, and fixed sampling fractions within PSUs, since it yields a fixed overall sample size (which is usually desirable, given budget constraints) and a fixed number of subunits in each PSU, giving equal work loads to PSU-based interviewers. In terms of sampling precision, the PPS strategy will usually not differ greatly from its competitors [8, p. 310]. A further advantage of the PPS strategy occurs for PSUs (e.g.,
schools or hospitals) which are of interest for their own sake, when equal subsample sizes may be preferred for a between-PSU analysis [16, p. 418]. For multistage sampling with more than two stages, the use of PPS sampling at every stage except the last, where SRS of fixed size is used, is again self-weighting and the comments above also apply (see also STRATIFIED MULTISTAGE SAMPLING). Single-Stage Designs For populations of units such as farms or businesses, measures of size xi may be available for each unit i in the population, and xi may be expected to be approximately proportionately related to a variable yi to be measured in a survey. For example, xi may be crop acreage, known for all farms i, and yi may be wheat yield in a given year. The procedure of sampling units by PPS may then permit more efficient estimation of the population total of the yi than is possible using SRS with the same sample size (see the section ‘‘Estimation Theory’’). PPS sampling may here be viewed as a limiting form of size-stratified sampling. If the population is divided into strata within which xi is approximately constant and if the within-stratum standard deviation of yi is proportional to xi , then optimal Neyman allocation∗ for stratified random sampling implies that units are selected by PPS. One problem with PPS sampling is in multiobjective surveys, where, for example, sampling with probability proportional to crop acreage may be more efficient than SRS for estimating wheat yield but may be much less efficient for estimating number of sheep, say. One practical advantage of PPS sampling in this case occurs in area sampling∗ , where areas may be selected simply using random points on a map [20, p. 207]. ESTIMATION THEORY For ‘‘Single-Stage Designs,’’ let x1 , . . . , xN be known size values for the population (xi > 0) and y1 , . . . , yN be the values of the variable of interest, where, without loss of generality, y1 , . . . , yn are the values observed in the sample. In with-replacement PPS sampling
PROBABILITY PROPORTIONAL TO SIZE (PPS) SAMPLING
(PPSWR) the sample is formed from n independent draws, at each of which the ith unit of the population is selected with probability pi = xi /X, where X = x1 + · · · + xN . In this case the usual estimator [14,25] of the population total, Y = y1 + · · · + yN , is Yˆ WR =
n
6489
where ij is the probability that both units i and j are included in the sample. Assuming that ij > 0, i, j = 1, . . . , N, two unbiased estimators of V(Yˆ HT ) are n
Vˆ HT (Yˆ HT ) =
2 −2 i (1 − i )yi
i=1
yi /(npi ).
+
i=1
n
−1 −1 −1 i j ij
i =j
This estimator is unbiased with variance V(Yˆ WR ) = n−1
N
2 pi (p−1 i yi − Y) .
× (ij − i j )yi yj , (1)
proposed by Horvitz and Thompson [15], and Yˆ SYG (Yˆ HT ) = ni<j −1 ij (i j − ij )
i=1
An unbiased estimator of V(Yˆ WR ) is ˆ Yˆ WR ) = [n(n − 1)]−1 V(
n
−1 2 ×(−1 i yi − j yj ) ,
2 ˆ (p−1 i yi − YWR ) .
i=1
(2) In PPSWR sampling the same unit may appear more than once in the sample. This is avoided in without-replacement PPS sampling (PPSWOR), where the ith unit of the population has probability i = nxi /X of inclusion in the sample (assuming that nxmax X). Here the usual estimator of Y is the Horvitz-Thompson estimator∗ [15] Yˆ HT =
n
−1 i yi .
i=1
The estimators Yˆ HT and Yˆ WR are formally identical, since i = npi = nxi /X, but are distinguished because they have different distributions under the two sampling schemes. Under PPSWOR, Yˆ HT is still unbiased but now has variance V(Yˆ HT ) =
N
2 −1 i (1 − i )yi
i=1
+
N
−1 −1 i j (ij − i j )yi yj
i =j
=
N
−1 2 (i j − ij )(−1 i yi − j yj ) ,
i<j
(3)
proposed by Sen [24] and Yates and Grundy [26]. Of the two estimators, Vˆ SYG seems to perform better in empirical comparisons [6, p. 8]. In comparing PPSWR and PPSWOR the ratio V(Yˆ HT )/V(Yˆ WR ) is (N − n)/(N − 1) roughly, as in SRS [8, p. 310]. Comparisons between different PPSWOR strategies (see the section ‘‘Methods’’) are given in refs. 2 and 22. The efficiency of the PPS estimators depends on how closely the yi are proportional to the xi , as can be seen by examining (1) and (3). Under the regression g model, E(yi |xi ) = α + βxi , V(yi |xi ) = σ 2 xi , the PPS strategy may be compared with the use of SRS and the ratio estimator∗ Yˆ R = X
n
yi /
i=1
n
xi .
i=1
The PPS strategy is generally more efficient if g > 1 and less efficient if g < 1 [8, p. 258]. For example, if g = 2 and α = 0, . then VSRS (Yˆ R )/VPPS (Yˆ HT ) = 1 + C2x , where Cx is the coefficient of variation of the xi . This ratio becomes closer to unity when α = 0. For two-stage sampling (see the section ‘‘Multistage Sampling’’) suppose that n PSUs are selected with probability proportional to size, Mi . Under PPSWR an unbiased estimate of the population total, Y, is Yˆ TS = M0 n−1
n i=1
yi ,
6490
PROBABILITY PROPORTIONAL TO SIZE (PPS) SAMPLING
where M0 = M1 + · · · + MN and the yi are the PSU subsample means. The variance of Yˆ TS can be expressed as the sum of a betweenPSU component of the form of (1) and a within-PSU component VW [8, p. 307]. An unbiased estimator of V(Yˆ TS ) is ˆ Yˆ TS ) = [n(n − 1)]−1 V(
n (M0 yi − Yˆ TS )2 . i=1
Under PPSWOR with i = Mi /M0 the estimator Yˆ TS is again unbiased and its variance is the sum of a between-PSU component of the form of (3) and the same withinPSU component, VW [8, p. 309]. Unbiased estimation of this variance is relatively complicated, although simplified procedures have been suggested in refs. 10 and 5. The discussion above is based on the conventional approach of making inferences with respect to sampling distributions induced by the randomized sample design. Alternative discussions of PPS sampling from Bayesian and prediction theory viewpoints are given in refs. 1, 9, and 18. METHODS The implementation of PPSWR is in principle straightforward [14], although a simplified procedure for large N has been suggested by Lahiri [17; 20, p. 202]. For PPSWOR (i.e., the selection of a sample of size n with inclusion probabilities given by i = nxi /X), many methods are available; 50 are listed in ref. 6. The case n = 2 has been especially studied because it corresponds to the common choice of two PSUs per stratum. Methods differ in respect of criteria such as (a) simplicity of implementation, (b) ease of computing the ij , (c) Vˆ SYG (Yˆ HT ) is always nonnegative, and (d) Yˆ HT and Vˆ SYG (Yˆ HT ) are efficient estimators. We assume that i = npi < 1 for i = 1, . . . , N, since any unit for which npi 1 is included in the sample with certainty and for our present purpose can therefore be excluded from the population. The simplest procedures to implement are those of systematic sampling∗ [19,12; 20, p. 215], but these suffer because the ij may be zero and hence no unbiased estimate of V(Yˆ HT ) may exist.
For n = 2 various simple PPSWOR procedures are available for which the ij are nonzero and easy to compute. In Brewer’s method [3] the first unit is selected with probability proportional to pi (1 − pi )/(1 − 2pi ), where pi = xi /X, and the second with probability pi /(1 − pj ), where unit j was drawn first. In durbin’s method [10] the first unit is selected with probability pi and the second with probability pi.j =
pi [(1 − 2pi )−1 + (1 − 2pj )−1 ] , −1 1+ N k=1 pk (1 − 2pk ) i = j,
where unit j was drawn first. Rao’s method [21], a case of rejective sampling∗ [13], involves selecting the first unit with probability pi and the second with replacement with probability proportional to pi /(1 − 2pi ). If both units are the same, the sample is rejected and the procedure above begun again until a sample of two distinct units is obtained. For all three procedures above, the joint inclusion probabilities are given by ij = 2pi pj.i . Also in each case Yˆ HT is more efficient than Yˆ WR under PPSWR [3] and Vˆ SYG is always nonnegative [21]. For n > 2 Rao’s method is generalized by Sampford [23], who proposes selecting units 2, 3, . . . , n with replacement with probability proportional to pi /(1 − npi ) and again rejecting samples not consisting of distinct units (see REJECTIVE SAMPLING). Brewer’s method has also been extended in ref. 4. The computation of the ij for these methods, however, does become more complicated as n increases. A straightforward algorithmic approach for general n which obeys at least criteria (a), (b), and (c) above is given by Chao [7]. A method for continuous surveys involving rotation of the PSUs has been suggested by Fellegi [11]. Further methods are reviewed in ref. 6. REFERENCES 1. Basu, D. (1971). In Foundations of Statistical Inference, V. P. Godambe and D. A. Sprott, eds. Holt, Rinehart and Winston, Toronto, pp. 203–242.
PROBABILITY THEORY AND RELATED FIELDS 2. Bayless, D. L. and Rao, J. N. K. (1970). J. Amer. Statist. Ass., 65, 1645–1667.
6491
3. Brewer, K. R. W. (1963). Aust. J. Statist., 5, 5–13.
25. Sukhatme, P. V. and Sukhatme, B. V. (1970). Sampling Theory of Surveys with Applications. Asia Publishing House, London, Chap. 2.
4. Brewer, K. R. W. (1975). Aust. J. Statist., 17, 166–172.
26. Yates, F. and Grundy, P. M. J. R. Statist. Soc. B, 15, 253–261.
5. Brewer, K. R. W. and Hanif, M. J. R. Statist. Soc. B, 32, 302–311.
(1970).
6. Brewer, K. R. W. and Hanif, M. (1983). Sampling with Unequal Probabilities. SpringerVerlag, New York. (The most comprehensive reference on methods and estimators.) 7. Chao, M. T. (1982). Biometrika, 69, 653– 656. 8. Cochran, W. G. (1977). Sampling Techniques, 3rd ed. Wiley, New York. (A good reference for the estimation theory.) 9. Cumberland, W. G. and Royall, R. M. (1981). J. R. Statist. Soc. B, 43, 353–367.
(1953).
See also AREA SAMPLING; POPULATION SIZE, HORVITZ –THOMPSON ESTIMATOR FOR; OPTIMUM STRATIFICATION; PRIMARY SAMPLING UNIT; REJECTIVE SAMPLING; REPRESENTATIVE SAMPLING; SIMPLE RANDOM SAMPLING; STRATIFIED DESIGNS; STRATIFIED MULTISTAGE SAMPLING; and SYSTEMATIC SAMPLING.
C. J. SKINNER
PROBABILITY SPACES, METRICS AND DISTANCES ON. See METRICS AND DISTANCES ON PROBABILITY SPACES
10. Durbin, J. (1967). Appl. Statist., 16, 152–164. 11. Fellegi, I. P. (1963). J. Amer. Statist. Ass., 58, 183–201. 12. Grundy, P. M. (1954). J. R. Statist. Soc. B, 16, 236–238. 13. Hajek, J. (1981). Sampling from a Finite Population. Marcel Dekker, New York. (Includes asymptotic results on various PPS schemes not available elsewhere.) 14. Hansen, M. H. and Hurwitz, W. N. (1943). Ann. Math. Statist., 14, 333–362. (The original PPS paper.) 15. Horvitz, D. G. and Thompson, D. J. (1952). J. Amer. Statist. Ass., 47, 663–685. 16. Kish, L. (1965). Survey Sampling. Wiley, New York, Chap. 7. (Good on the practical context of PPS sampling.) 17. Lahiri, D. B. (1951). Bull. Int. Statist. Inst., 33, Bk. 2, 133–140. 18. Little, R. J. A. (1983). J. Amer. Statist. Ass., 78, 596–604. 19. Madow, W. G. (1949). Ann. Math. Statist., 20, 333–354. 20. Murthy, M. N. (1967). Sampling Theory and Methods. Statistical Publishing Society, Calcutta, Chap. 6. 21. Rao, J. N. K. (1965). J. Indian Statist. Ass., 3, 173–180. 22. Rao, J. N. K. and Bayless, D. L. J. Amer. Statist. Ass., 64, 540–559. 23. Sampford, M. R. 499–513.
(1967).
(1969).
Biometrika,
24. Sen, A. R. (1953). J. Indian Statist., 5, 119–127.
Soc.
54,
Agric.
PROBABILITY THEORY AND RELATED FIELDS The journal Probability Theory and Related Fields (PTRF) is published by Springer Verlag in Heidelberg, Germany, in 12 issues per year, each set of 4 issues constituting one volume. PTRF was founded in 1962 with the title Zeitschrift fur ¨ Wahrscheinlichkeitstheorie und Verwandte Gebiete, which changed to the current title in 1986 with Volume 71. A website link for the journal is http://link. springer.de/link/service/journals/00440. PTRF is directed by a Managing Editor with the assistance of ten or so members of an international Editorial Board. All papers submitted to the journal for publication are refereed. Quoting from the website, ‘‘The first Managing Editor was L. Schmetterer, who wrote thus in the first issue: ‘Until about 30 years ago, the theory of probability and its applications were somewhat isolated mathematical disciplines. Then, following a period of remarkable development that is not yet completed by any means, probability theory became an important domain of mathematics, intimately connected with other domains, especially with analysis and number theory. In addition, it leads a life of its own, all by itself and in its manifold applications.’’’
6492
PROBABILITY THEORY: AN OUTLINE
‘‘The Founding Managing Editor was succeeded by K. Krickeberg in 1971, and by H. Rost in 1985. Subsequent Managing Editors [have been] O. Kallenberg (1991), E. Bolthausen (1994) and G. R. Grimmett (2000).’’ ‘‘Probability Theory and Related Fields publishes research papers in modern probability theory, its relation to analysis, geometry and other areas in mathematics, and its various fields of application. Thus, subjects of interest include: statistical mechanics, ergodic theory, mathematical biology, filtering theory, mathematical statistics, theoretical computer science, optimization and control, stochastic geometry, and stochastic algorithms. Survey papers on emerging areas of importance may be considered for publication. The main languages of publication are English, French and German.’’
PROBABILITY THEORY: AN OUTLINE Probability theory provides a mathematical framework for the description of phenomena whose outcomes are not deterministic but rather are subject to chance∗ . This framework, historically motivated by a frequency interpretation∗ of probability, is provided by a formal set of three innocent-looking axioms and the consequences thereof. These are farreaching and a rich and diverse theory has been developed. It provides a flexible and effective model of physical reality and furnishes, in particular, the mathematical theory needed for the study of the discipline of statistics. THE FORMAL FRAMEWORK Elementary (non-measure-theoretic) treatments of probability can readily deal with situations in which the set of distinguishable outcomes of an experiment subject to chance (called elementary events) is finite. This encompasses most simple gambling situations, for example. Let the set of all elementary events be (called the sample space∗ ) and let F be the Boolean field of subsets of (meaning just that (a) A ∈ F implies that A ∈ F, the bar
denoting complementation with respect to and (b) A1 , A2 ∈ F implies that A1 ∪ A2 ∈ F). Then the probability P is a set function on F satisfying the axioms: A1 P(A) 0, A ∈ F. A2 If A1 , A2 ∈ F and A1 ∩ A2 = Ø (the empty set), then P(A1 ∪ A2 ) = P(A1 ) + P(A2 ). A3 P() = 1. These axioms∗ enable probabilities to be assigned to events in simple games of chance∗ . For example, in the toss of an ordinary six-sided die, consists of six equiprobable elementary events and for A ∈ F, P(A) = n(A)/6, where n(A) is the number of elementary events in A. The formulation above can be extended straightforwardly to the case where the number of elementary events in the sample space is at most countable, by extending Axiom A2 to deal with countable unions of disjoint sets. However, a full measure-theoretic formulation is necessary to deal with general sample spaces and to provide a framework which is rich enough for a comprehensive study of limit results on combinations of events (sets); see also MEASURE THEORY IN PROBABILITY AND STATISTICS. The measure-theoretic formulation has a very similar appearance to the elementary version. We start from a general sample space and let F be the Borel σ -field of subsets of [meaning that (a) A ∈ F implies that A ∈ F and (b) Ai ∈ F, 1 i < ∞, implies that ∞ i=1 Ai ∈ F]. Then as axioms for the probability measure P we have Axioms A1 and A3 as before, while A2 is replaced by: A2 If Ai ∈ F, 1 i < ∞, and Ai ∩ Aj = Ø, i = j, then P
∞ i=1
Ai
=
∞
P(Ai ).
i=1
The triplet (, F, P) is called a probability space. This framework is due to Kolmogorov [14] in 1933 and is very widely accepted, although some authors reject the countable additivity axiom A2 and rely on
PROBABILITY THEORY: AN OUTLINE
the finite additivity version (A2) (e.g., Dubins and Savage [8]). An extension of the Kolmogorov theory in which unbounded measures are allowed and conditional probabilities are taken as the fundamental concept has been provided by R´enyi [18, Chap. 2]. For A, B ∈ F with P(B) > 0, the conditional probability of A given B is defined as P(A|B) = P(A ∩ B)/P(B). Suppose that the sample space has points ω. A real-valued function X(ω) defined on the probability space (, F, P) is called a random variable [or, in the language of measure theory, X(ω) is a measurable function] if the set {ω : X(ω) x} belongs to F for every x ∈ R (the real line). Sets of the form {ω : X(ω) x} are usually written as {X x} for convenience and we shall follow this practice. Random variables and their relationships are the principal objects of study in much of probability and statistics. IMPORTANT CONCEPTS Each random variable X has an associated distribution function defined for each real x by F(x) = P(X x). The function F is nondecreasing, rightcontinuous, and has at most a countable set of points of discontinuity. It also uniquely specifies the probability measure which X induces on the real line. Random variables are categorized as having a discrete distribution if the corresponding distribution function is a step function and as having a continuous distribution if it is continuous. Most continuous distributions of practical interest are absolutely continuous, meaning that the distribution function F(x) has a representation of the form F(x) =
6493
important are the binomial∗ , which is discrete and for which n r P(X = r) = p (1 − p)n−r , r 0 r n,
0 < p < 1,
∗
and the (unit) normal , which is absolutely continuous and for which x 2 e−(1/2)u du, P(X x) = (2π )−1/2 −∞
−∞ < x < ∞. The binomial random variable represents the numbers of successful outcomes in n trials, repeated under the same conditions, where the probability of success in each individual trial is p. Normal distributions, or close approximations thereto, occur widely in practice, for example in such measurements as heights or weights or examination scores, where a large number of individuals are involved. Two other particularly important distributions are the Poisson∗ , for which P(X = r) = e−λ λr /r!, r = 0, 1, 2, . . . ,
λ > 0,
and the exponential∗ , for which x P(X x) = λ e−λu du, x > 0,
λ > 0;
0
= 0 otherwise. These commonly occur, respectively, in situations of rare events and as waiting times between phenomena. A sequence of random variables {X1 , X2 , . . .} defined on the same probability space is called a stochastic process∗ . For a finite collection of such random variables {X1 , X2 , . . . , Xm } we can define the joint distribution function F(x1 , x2 , . . . , xm )
x
f (u) du −∞
for some f 0; f is called the probability density function. A relatively small number of families of distributions are widely used in applications of probability theory. Among the most
= P(X1 x1 , X2 x2 , . . . , Xm xm ) for xi ∈ R, 1 i m, and the random variables are said to be independent∗ if P(X1 x1 , . . . , Xm xm ) = P(X1 x1 ) · · · P(Xm xm )
6494
PROBABILITY THEORY: AN OUTLINE
for all x1 , . . . , xm . Independence is often an essential basic property and much of probability and statistics is concerned with random sampling, namely independent observations (random variables) with identical distributions. One step beyond independence is the Markov chain {X1 , X2 , . . .}, in which case P(Xm xm |X1 = i1 , . . . , Xn = in ) = P(Xm xm |Xn = in ) for any n < m and all i1 , . . . , in and xm . That is, dependence on a past history involves only the most recent of the observations. Such random sequences with a natural time scale belong to the domain of stochastic processes; see MARKOV PROCESSES. Considerable summary information about random variables and their distributions is contained in quantities called moments. For a random variable X with distribution function F, the rth moment (r > 0) is defined by EX r =
∞
xr dF(x)
to that of a random variable X (written d Xn →X) if the sequence of distribution functions {Fn (x) = P(Xn x)} converges to the distribution function F(x) = P(X x) at all points x of continuity of F. Convergence in distribution does not involve the random variables explicitly, only their distributions. Other commonly used modes of convergence are: (a) {Xn } converges in probability to X (writp ten Xn →X) if, for every > 0, lim P(|Xn − X| > ) = 0.
n→∞
(b) For r > 0, {Xn } converges in the mean of r order r to X (written Xn →X) if lim E|Xn − X|r = 0.
n→∞
(c) {Xn } converges almost surely to X (writa.s. ten Xn → X) if for every > 0, lim P sup |Xk − X| >
n→∞
−∞
∞ provided that −∞ |x|r dF(x) < ∞. E|X|r is called the rth absolute moment. The operator E is called Expectation and the first moment (r = 1) is termed the mean. Moments about the mean E(X − EX)r , r = 2, 3, . . . are widely used and E(X − EX)2 is termed the variance∗ of X, while (E(X − EX)2 )1/2 is its standard deviation∗ . The mean and variance of X are, respectively, measures of the location and spread of its distribution. CONVERGENCE Much of probability theory is concerned with the limit behavior of sequences of random variables or their distributions. The great diversity of possible probabilistic behavior when the sample size is small is often replaced by unique and well-defined probabilistic behavior for large sample sizes. In this lies the great virtue of the limit theory: statistical regularity emerging out of statistical chaos as the sample size increases. A sequence of random variables {Xn } is said to converge in distribution (or in law)
= 0.
kn
See also CONVERGENCE OF SEQUENCES OF RANDOM VARIABLES. It is not difficult to show that (b) implies (a), which in turn implies convergence in distribution while (c) implies (a). None of the converse implications hold in general. Convergence in distribution is frequently established by the method of characteristic functions∗ . The characteristic function of a random variable X with distribution function F is defined by ∞ eitx dF(x), f (t) = EeitX = √
−∞
where i = −1. If fn (t) and f (t) are, respectively, the characteristic functions of Xn and d
X, then Xn →X is equivalent to fn (t) → f (t). Characteristic functions are particularly convenient for dealing with sums of independent random variables as a consequence of the multiplicative property Eeit(X+Y) = EeitX EeitY for independent X and Y. They have been extensively used in most treatments of
PROBABILITY THEORY: AN OUTLINE
convergence in distribution for sums of independent random variables (the classical central limit problem). The basic central limit theorem∗ (of which there are many generalizations) deals with independent and identically distributed random variables Xi with 2 mean µ and variance σ . It gives, upon writing Sn = ni=1 Xi , that lim P(σ −1 n−1/2 (Sn − nµ) x) x 2 = (2π )−1/2 e−(1/2)u du,
n→∞
(1)
−∞
the limit being the distribution function of the unit normal law. Convergence in probability results are often obtained with the aid of Markov’s inequality∗ P(|Xn | > c) c−r E|Xn |r for r > 0 and a corresponding convergence in rth mean result. The useful particular case r = 2 of Markov’s inequality is ordinarily called Chebyshev’s inequality∗ . For example, in the notation of (1), P(|n−1 Sn − µ| > ) −2 n−2 E(Sn − nµ)2 → 0 (2) as n → ∞, which is a version of the weak law p of large numbers∗ , n−1 Sn →µ. Almost sure convergence results rely heavily on the Borel-Cantelli lemmas∗ . These deal with#a sequence of events (sets) {En } and ∞ E , the set of elements comE= ∞ n r=1 n=r mon to infinitely many of the En . Then (a) P(En ) < ∞ implies that P(E) = 0. (b) P(En ) = ∞ and the events En are independent implies that P(E) = 1. With the aid of (a), (2) can be strengthened to the strong law of large numbers∗ , a.s. n−1 Sn → µ
(3)
as n → ∞. The strong law of large numbers embodies the idea of a probability as a strong limit of relative frequencies; if we set Xi = Ii (A), the indicator function of the set A at the ith trial,
6495
then µ = P(A). This property is an important requirement of a physically realistic theory. The law of large numbers, together with the central limit theorem, provides a basis for a fundamental piece of statistical theory. The former yields an estimator of the location parameter µ and the latter enables approximate confidence intervals∗ for µ to be prescribed and tests of hypotheses∗ constructed.
OTHER APPROACHES The discussion above is based on a frequency interpretation of probability, and this provides the richest theory. There are, however, significant limitations to a frequency interpretation in describing uncertainty in some contexts and various other approaches have been suggested. Examples are the development of subjective probability∗ by de Finetti and Savage (e.g., refs. 7 and 19) and the so-called logical probability of Carnap [3]. The former is the concept of probability used in Bayesian inference∗ and the latter is concerned with objective assessment of the degree to which evidence supports a hypothesis (see BAYESIAN INFERENCE and PROBABILITY, FOUNDATIONS OF —I). A detailed comparative discussion of the principal theories that have been proposed is given in Fine [11]. The emergence of probability theory as a scientific discipline dates from around 1650; a comprehensive sketch of its history is given in PROBABILITY, HISTORY OF. THE LITERATURE Systematic accounts of probability theory have been provided in book form by many authors. These are necessarily at two distinct levels since a complete treatment of the subject has a measure theory prerequisite. Elementary discussions, not requiring measure theory, are given for example in Chung [5], Feller [9], Gnedenko [12], and Parzen [16]. A full treatment can be found in Billingsley [1], Breiman [2], Chow and Teicher [4], Chung [6], Feller [10], Hennequin and Tortrat [13], Lo´eve [15], and R´enyi [17].
6496
PROBABILITY-WEIGHTED MOMENTS
REFERENCES 1. Billingsley, P. (1979). Probability and Measure. Wiley, New York. 2. Breiman, L. (1968). Probability. AddisonWesley, Reading, Mass. 3. Carnap, R. (1962). The Logical Foundations of Probability, 2nd ed. University of Chicago Press, Chicago. 4. Chow, Y. S. and Teicher, H. (1978). Probability Theory. Independence, Interchangeability, Martingales. Springer-Verlag, New York. 5. Chung, K. L. (1974). Elementary Probability Theory with Stochastic Processes. SpringerVerlag, Berlin. 6. Chung, K. L. (1974). A Course in Probability Theory, 2nd ed. Academic Press, New York. 7. de Finetti, B. (1974). Theory of Probability, Vols. 1 and 2. Wiley, New York. 8. Dubins, L. E. and Savage, L. J. (1965). How to Gamble If You Must. McGraw-Hill, New York. 9. Feller, W. (1968). An Introduction to Probability Theory and Its Applications, Vol. 1, 3rd ed. Wiley, New York. 10. Feller, W. (1971). An Introduction to Probability Theory and Its Applications, Vol. 2, 2nd ed. Wiley, New York. 11. Fine, T. L. (1973). Theories of Probability. An Examination of the Foundations. Academic Press, New York. 12. Gnedenko, B. V. (1967). The Theory of Probability, 4th ed. (Translated from the Russian by B. D. Seckler.) Chelsea, New York. 13. Hennequin, P. L. and Tortrat, A. (1965). Th´eorie des Probabilit´es et Quelques Applications. Masson, Paris. 14. Kolmogorov, A. (1933). Grundbegriffe der Wahrscheinlichkeitsrechnung. SpringerVerlag, Berlin. (English trans.: N. Morrison, in Foundations of the Theory of Probability. Chelsea, New York, 1956.) 15. Lo`eve, M. (1977). Probability Theory, Vols. 1 and 2, 4th ed. Springer-Verlag, Berlin. 16. Parzen, E. (1960). Modern Probability Theory and Its Applications. Wiley, New York. 17. R´enyi, A. (1970). Probability Theory. NorthHolland, Amsterdam. (Earlier versions of this book appeared as Wahrscheinlichkeitsrechnung, VEB Deutscher Verlag, Berlin, 1962; and Calcul des probabilit´es, Dunod, Paris, 1966.) 18. R´enyi, A. (1970). Foundations of Probability. Holden-Day, San Francisco.
19. Savage, L. J. (1962). The Foundations of Statistical Inference: A Discussion. Wiley, New York. See also AXIOMS OF PROBABILITY; BAYESIAN INFERENCE; CHANCE —II; CONVERGENCE OF SEQUENCES OF RANDOM VARIABLES; FREQUENCY INTERPRETATION IN PROBABILITY AND STATISTICAL INFERENCE; LAWS OF LARGE NUMBERS; LIMIT THEOREM, CENTRAL; NONADDITIVE PROBABILITY; PROBABILITY, HISTORY OF; PROBABILITY, FOUNDATIONS OF —I; and SUBJECTIVE PROBABILITIES.
C. C. HEYDE
PROBABILITY-WEIGHTED MOMENTS The (r, s, t) probability-weighted moment of a random variable X, having a cumulative distribution function (CDF) F(x) is Mr,s,t = E[X r {F(X)}s {1 − F(X)}t ] 1 = {x(F)}r F s (1 − F)t d F, 0
where x(F) is the inverse of the CDF. Equivalently, Mr,s,t is (s!t!)/(s + t + 1)! times the expected value of the rth power of the (s + 1)th order statistic∗ in a random sample of size (s + t + 1). The quantities {Mr,0,0 } are the moments about zero of X. The quantities Mr,s,t were introduced by Greenwood et al. [1]. Equating sample to population values may be used for estimation of parameters. [Note, however, that the sample size (n) need not be equal to (s + t + 1).] Hosking et al. [2] consider use of {M1,s,0 } in this connection. Hosking [3] gives detailed theory. REFERENCES 1. Greenwood, J. A., Landwehr, J. M., Matalar, N. C., and Wallis, J. R. (1979). Water Resources Res., 15, 1049–1054. 2. Hosking, J. R. M., Wallis, J. R., and Wood, E. F. (1985). Technometrics, 27, 251–261. 3. Hosking, J. R. M. (1986). The Theory of Probability-Weighted Moments, IBM Res. Rep. RC 12210 (#54860). See also ESTIMATION, CLASSICAL; ESTIMATION: METHOD OF MOMENTS; ORDER STATISTICS; PROBABILITY MOMENTS AND FREQUENCY MOMENTS; and U-STATISTICS.
PROBIT ANALYSIS
PROBABLE ERROR Francis Galton∗ expressed the values in a normal distribution∗ as a function of probable error [4], which he described as ‘‘the value that one-half of the Errors exceed and the other half fall short of.’’ It is an archaic term for the semi-interquartile range (see QUARTILE DEVIATION), for normal variables equal to 0.67449 × (standard deviation∗ ). Probable error as a measure of spread was usually applied to normal variables [6], the probability of obtaining a value within one probable error of the population mean being 0.50, but it was also used for nonnormal data [1]. The term appeared in the early nineteenth century among German astronomers. It was apparently first used explicitly by F. W. Bessel [2] in 1818, was developed subsequently by Gauss∗ , and was used extensively by Adolphe Quetelet∗ [5]. Its origin, however, can be traced [3] to Abraham De Moivre’s The Doctrine of Chances (1756). Galton [4, pp. 57–58] objected to the name ‘‘probable error’’: ‘‘It is astonishing that mathematicians, who are the most precise and perspicacious of men, have not long since revolted against this cumbrous, slip-shod, and misleading phrase.’’ He then introduced the term ‘‘probable deviation’’ instead, but it was not adopted by others. Although probable error is still used as a measure occasionally in the physical sciences, it has been replaced in modern statistics by the standard deviation. The term ‘‘error’’ reflects the earlier convention of describing the normal law as the error distribution (see LAWS OF ERROR —I, II, III). REFERENCES 1. Aitken, A. C. (1949). Statistical Mathematics, 6th ed. Oliver & Boyd, Edinburgh. (See Chapter II.) ¨ 2. Bessel, F. W. (1818). Uber der Ort des Polarsterns. Berliner Astronomische Jahrbuch ¨ 1818. Berlin. fur 3. De Moivre, A. (1756). The Doctrine of Chances, 3rd (final) ed. A. Millar, London. (Reprinted by Chelsea, New York, 1967. 4. Galton, F. (1889). Natural Inheritance. Macmillan, London, pp. 57–58.
6497
5. Quetelet, L. A. (1849). Letters on the Theory of Probabilities, O. G. Downes, trans. Layton, London. 6. Yule, G. U. and Kendall, M. G. (1937). An Introduction to the Theory of Statistics, 11th ed. Charles Griffin, London. (See Section 19.10.) See also INTERQUARTILE RANGE; NORMAL DISTRIBUTION; QUARTILE DEVIATION; and STANDARD DEVIATION.
PROBABLE ITEM An obsolete term for mode (see MEAN, MEDIAN, AND MODE).
PROBIT ANALYSIS ‘Quantal response analysis is concerned with the relationship between one or more dependent or response variables on the one hand and a set of independent or explanatory variables on the other, in the situation in which the response is assessed in terms of a discrete, qualitative, or categorical∗ variable. The simplest form of quantal variable involves a dichotomy, with two response categories which may be assigned the conventional labels 0 and 1, respectively. More generally, response may be recorded in terms of a polychotomy with several classes or categories, which may or may not reflect some implied underlying ordering (see also NOMINAL DATA). APPLICATIONS In the past, the main applications of quantal response analysis have been in biometry and medical statistics, including pharmacology, toxicology, occupational medicine, and follow-up∗ studies of groups of patients with particular characteristics, conditions, or treatment regimes (see BIOSTATISTICS, CLASSICAL and STATISTICS IN MEDICINE). Quantal models have commonly been concerned with the death of a human being or other biological organism or the appearance of a particular symptom, in response to a stimulus such as a drug, a poison, a period of exposure to a hazardous environment, or merely the passage of time. Many biometric
6498
PROBIT ANALYSIS
applications involve just one response variable and just one stimulus, although some consideration has been given both to multiple quantal responses and mixtures of several stimuli administered jointly. See also BIOASSAY, STATISTICAL METHODS IN. More recently, there have been an increasing number and range of economic applications, in a variety of contexts which include labor force participation, choice of occupation, migration∗ , transportation, housing, education, legislation, and criminology. Many of the variables of interest in this field are either naturally expressed in a discrete or categorical form or are recorded in this way. Because of the complexities of the situation, large numbers of explanatory variables are commonly used; in addition, the growing availability and sophistication of survey data has led to a sharp increase in the raw material for such studies. Both developments reflect the growing use and power of modern computing facilities. There is an extensive literature on the applications of quantal response analysis in biometry; for example, see refs. 6, 8, and 13. Economic analysis is less well served in this context, although Daganzo [7] dealt with one specific application—transport model choice—and Amemiya [1] produced an excellent survey of the use of qualitative response models in both econometrics∗ and biometry. FRAMEWORK FOR QUANTAL RESPONSE MODELS A general framework for the formulation of quantal response models in biometry was provided by Ashford [3], who proposed that a biological organism should be regarded as a system comprising several subsystems, S1 , S2 , . . . , Sk . When a stimulus consisting of d components, D1 , D2 , . . . , Dd , is applied at level z = (z1 , z2 , . . . , zd ), the organism manifests a reaction y = (y1 , y2 , . . . , yk ), where yi , i = 1, 2, . . . , k, refers to the underlying response of subsystem Si ; yi may in principle be expressed in quantitative terms, but is not necessarily observable in this form. For example, human subjects are assessed in terms of several distinct respiratory symptoms, each of which corresponds
to the quantal response of a particular subsystem. The same framework may also be employed in economic analysis, although the interpretation of the components reflects the field of application. The underlying stimulus-response relation may then be represented in terms of an additive model of the form y = x(z) + .
(1)
In this expression, x(z) is the vector effects function of the stimulus z. The random vector represents the ‘‘tolerance’’ or ‘‘susceptibility’’ of the individual subject and is distributed with joint cumulative distribution function (CDF) T() and joint probability density function (PDF) t() in the population of subjects. In general, the tolerance components (i , j ), (i, j = 1, 2, . . . , k; i = j), will be positively correlated, since they reflect different characteristics of the same subject. RELATION BETWEEN QUANTITATIVE AND QUANTAL RESPONSES The value of the underlying reaction y will determine the particular quantal or semiquantal category of the observed system response. Suppose that the k-variate reaction space is divided into mutually exclusive and exhaustive regions corresponding to the various categories. In the simplest situation, just one subsystem S is involved; for such a subsystem quantal response, the two regions correspond to the subdivision of the y-axis at a point which, without loss of generality, may be taken as zero. The probability of subsystem quantal response may then be expressed as P(z) = P{y 0} = P{x(z) + 0} = 1 − T(−x(z)),
(2)
where T() is the (marginal) CDF of . If we make the plausible assumption that T() is symmetrical, (2) may be expressed in the form P(z) = T(x(z)).
(3)
PROBIT ANALYSIS
The form of the right-hand side of (3) emphasizes that, on the basis of a quantal response alone [i.e., using only the information contained in P(z)], it is not possible to determine the forms of both the tolerance distribution and the effects function without ambiguity, since for any given P(z) and specified marginal CDF T(), the form of x(z) is uniquely determined. A semiquantal response involving s ordered categories C1 , C2 , . . . , Cs may also be represented in terms of the response of a single subsystem S. The boundaries of the various categories may be defined as the points η1 , η2 , . . . , ηs−1 on the y-axis, where η1 < η2 < · · · < ηs−1 , the rth response category Cr corresponding to all points y in the interval (ηr−1 , ηr ), where η0 and ηs are taken as −∞ and +∞, respectively. Without loss of generality, any one of the ηi may be set as equal to zero. The probability that the response falls within category Cr is then Pr (z) = P{ηr−1 y1 < ηr } = P{ηr−1 − x(z) ηr − x(z)} = T{ηr − x(z)} − T{ηr−1 − x(z)}. (4) Quantal and semiquantal responses defined in relation to a complete system may also be represented in terms of the underlying reaction of more than one subsystem. Thus a ‘‘weakest link’’ type of system might involve the quantal response of any one of k constituent subsystems. The corresponding probability is P(z) = P{any yi > 0} = 1 − P{all yi 0} = 1 − T(−x1 (z), −x2 (z), . . . , −xk (z)).(5) In general, if the quantal response corresponds to a region R in the underlying reaction space, the corresponding probability is p(z) = · · · t(x(z) − )d1 d2 . . . dk . R
(6) Semiquantal responses may similarly be defined in terms of a mutually exclusive and exhaustive set of regions. Such categories will not in general reflect any intrinsic ordering of
6499
the various response variables, although, for example, regions defined by a set of parallel hyperplanes would in some sense imply an underlying gradation. THE TOLERANCE DISTRIBUTION: PROBITS AND LOGITS Various algebraic forms have been proposed for the distribution of the tolerances in (1). Turning first to the marginal distribution of a particular component, say , the following types of distribution have been employed in the analysis of quantal data. Linear Probability Model 0, 0, T() = , 0 < 1, 1, > 1.
(7)
This model restricts the range of variation of the tolerance component, which may well be unrealistic in many applications. This deficiency must be balanced against the advantage of computational simplicity. The model has been found useful in the preliminary analysis of economic data. Probit Model The scale used for the measurement of the underlying response is arbitrary. Given that the mean of the tolerance distribution∗ has already been set equal to zero, the variance may be equated to unity and the tolerance may be assumed to be a standardized variable. The most widely used tolerance distribution of this type is the standardized normal, √ 2 (1/ 2π ) −(1/2)t dt, (8) T() = () = −∞
which forms the basis of probit analysis (see also BIOASSAY, STATISTICAL METHODS IN). The probit of the proportion p is defined in terms of the inverse of as p∗ = −1 (p) + 5.
(9)
The factor 5 was originally included in (9) for convenience of manual computation, to avoid the occurrence of negative values of the
6500
PROBIT ANALYSIS
probit. In the more modern usage of the term ‘‘probit,’’ this factor is excluded from the definition (see NORMAL EQUIVALENT DEVIATE). Logistic Model
the extreme minimal value (m1 = 1, m2 → ∞) and the extreme maximum value (m1 → ∞, m2 = 1). MULTIVARIATE TOLERANCE DISTRIBUTIONS
An alternative ‘‘smooth’’ standardized distribution to the normal is the logistic distribution∗ , employed in the form T() = L(λ, ) = −λ /(1 + −λ ).
(10)
√ Taking λ = π/ 3, the variance of the tolerance distribution may be set equal to unity. However, some authors consider that closer overall agreement with the standardized normal distribution may be obtained by setting λ = 1.6. The inverse of the relation L(1, ) defines the logit∗ transformation (or the logarithm of the odds ratio; see ODDS RATIO ESTIMATORS) l = ln[p/(1 − p)].
(11)
Cox [6, p. 33] has suggested a modified form of (11) which avoids infinite values of the logit corresponding to p = 0 and p = 1. The logit and probit transformations are similar for values of p that lie toward the center of the range and differ substantially only for values close to zero or unity. In practical applications, extremely large numbers of observations are required to differentiate between the two transformations [5]. The logit transformation has the advantage of greater ease of computation, which must be set against the fact that the probit, by virtue of its derivation from the normal distribution, is closer to the mainstream of statistical analysis. A two-parameter class of tolerance distributions has been proposed [14] which includes both the normal and logistic distributions as special cases, exp(m1 ){1 + exp }−(m1 +m2 ) t(m1 , m2 , ) = β(m1 , m2 ) (12) where β represents the beta function. The logistic model is given by m1 = m2 = 1, while (12) converges to a normal distribution as m1 , m2 → ∞. Other limiting special cases are
The assumption of a standardized multivariate normal distribution for the joint distribution of several tolerance components corresponds to multivariate probit analysis [2]. This has the advantage of standardized normal univariate marginal distributions, as used in probit analysis, coupled with a representation of the association between pairs of tolerance components in terms of the correlation coefficient. Representations based on a multivariate logistic distribution have also been proposed. Morimune [11] compared the Ashford–Sowden bivariate probit model with the Nerlove–Press [12] multivariate logit model in an application concerned with home ownership and number of rooms in house, and concluded that the probit was to be preferred. ORDERED RESPONSE VARIABLES: RANKITS In circumstances such as sensory tests for degrees of preference, the response is measured by a variable that can be placed in rank order, but cannot be assigned a quantitative value (see ORDINAL DATA). The (subjective) ranks may be replaced by the corresponding rankits [10]. The rankit∗ of the rth in rank order from a sample of n observations is defined as the expected value of the rthorder statistic∗ of a random sample of size n from a standardized normal distribution. This transformation is in a sense a generalization of the probit transformation and produces a response variable that may be regarded as quantitative in form. MIXED DISCRETE-CONTINUOUS RESPONSE VARIABLES: TOBITS Response may be recorded in terms of a variable that is capable of being measured continuously over only part of the real line. For example, household expenditures on luxury goods are zero at low income levels. Individuals available for employment may
PROBIT ANALYSIS
be working full-time or for a certain number of hours part-time or may be unemployed during a particular week. The general ‘‘twolimit’’ Tobit model envisages an underlying quantitative response which is only measurable in categorical form at the lower and upper limits of the range of variation. If one or other of the extreme categories is not present, the Tobit model is called ‘‘right-truncated’’ or ‘‘left-truncated,’’ as appropriate. Tobit models invariably involve a standardized normal distribution of tolerances and the probabilities of obtaining one or other of the categorical responses may be calculated on the basis of (2) and (8); see also ECONOMETRICS. THE EFFECTS FUNCTION The effects function x(z) in (1) represents the average contribution of the stimulus z to the response y of the subject. In general, this function and its individual components will include parameters θ = (θ1 , θ2 , . . . , θs ) which must be estimated. The form of x(z) and its components xi (z) reflects the particular application. In most but not all applications it is reasonable to suppose that any given component of the effects function will be a monotonically increasing function of the level of any particular component of the stimulus, for fixed levels of any other components. Usually, in the absence of any stimulus all components of the effects function will be zero. Linear functions of the form z θ of the components zi of the stimulus z have been commonly used, conforming to the classical general linear model∗ . Nonlinear functional forms may, however, be appropriate in certain circumstances. The simplest situation, commonly used in the biological assay of a single drug, involves a single subsystem S for which the effects function is expressed in terms of the logarithm of the dosage of a single drug. The underlying dosage–response relation then takes the form y = θ1 + θ2 ln z1 + ,
(13)
where is a standardized variable and the corresponding probability of a quantal response is, from eq. (3), P(z1 ) = T(θ1 + θ2 ln z1 ).
(14)
6501
Both the response y and the stimulus z may be subject to errors of observation. Suppose that the observed value of y (which forms the basis of the observed quantal response) is denoted y = y + φy and that the observed stimulus (ln z1 ) = ln z1 + φz . The underlying dosage–response relation (13) may then be expressed in the form y = θ1 + θ2 ln z1 + ,
(15)
where = 1 + φy − θ2 φz . Given that φy and φz follow independent normal distributions with zero mean and variances σy2 and σz2 , respectively, is itself normally distributed with zero mean and variance (1 + σy2 + θz2 σz2 ). This means that for a standardized normal distribution of tolerances the corresponding observed dosage–response relation is (1 + σy2 + θ q2z σz2 ) y = y = (θ1 + θ2 ln z) (1 + σy2 + θz2 σz2 ) + . (16) Comparison of (13) and (16) with (14) shows that the effect of errors of observation is to reduce the slope of the quantal response relationship between the inverse of the CDF of the tolerance distribution and the log dose, but the position corresponding to a 50% response, the LD50 (see BIOASSAY, STATISTICAL METHODS IN), is unchanged. In bioassay more complex forms of the effects function have been proposed to represent stimuli with several components, corresponding to mixtures of drugs or poisons, which are becoming increasingly important in both pharmacology and toxicology (see, e.g., ref. 9). An approach based on a simplified representation of the underlying mode of biological action of the stimulus has been put forward by Ashford (see ref. 3), presenting models based on drug-receptor theory that permit both monotonic and nonmonotonic dosage–response relationships for a single-component stimulus and a simple representation of the phenomena of synergism and antagonism. The majority of examples in economics arise from the representation of events whose outcome is determined by the decisions of a customer, a product, or some other economic
6502
PROBIT ANALYSIS
unit. There is an implicit assumption that rational decisions are made which optimize some inherent utility [15]. An effects function x θ is commonly thought to be appropriate. ESTIMATION AND HYPOTHESIS TESTING The main objective of the statistical analysis of quantal response data is the assessment of the parameters θ contained within the effects function, although in certain applications further parameters relating to the tolerance distribution are also of interest, including the correlation coefficients associated with a multivariate normal distribution of tolerances. As far as the raw data are concerned, a distinction is often made between structures in which only one or a few observations correspond to any given level of stimulus and situations in which many observations are involved. If possible, the number of observations at any stimulus level should be sufficient for large-sample theory∗ to apply in order to minimize the effects of sampling variations on the observed proportions of quantal or semiquantal responses. The most commonly used estimation procedure is maximum likelihood∗ (ML), which may be applied to both data structures. Suppose that for any particular value zi of the stimulus or independent variable, there are ni observations, of which ri manifest the quantal response of some subsystem S. (If there is only one observation at any particular stimulus level, ri will be zero or unity.) Assuming that the observations are independent and applying the binomial theorem on the basis of (3), the log likelihood may be written (apart from a constant term) in the form {ri ln T(x(θ, zi )) + (ni − ri ) L= i
× ln(1 − T(x(θ, zi )))}.
(17)
The ML estimator of θ is given by the solution of the simultaneous equations {ri − ni T(x(θ , zi ))} ∂L = ∂θk T(x(θ, zi )){1 − T(x(θ, zi ))} i
× t(x(θ, zi )) where k = 1, 2, . . . , s.
∂x(θ, zi ) = 0, ∂θk
(18)
In general, explicit algebraic solutions of (18) do not exist and the ML estimators must be obtained by an iterative method. For the probit or logit models, L is globally concave, which implies that the ML estimator is unique if it is bounded; hence any iterative procedure that is guaranteed to converge to a stationary point must also converge to the global maximum. Various methods of iteration have been used. Powell’s method has the advantage of involving the calculation of the function L, but not of its derivatives, in contrast to the Newton–Raphson method∗ and the method of scoring, which both involve the first derivatives. See ITERATED MAXIMUM LIKELIHOOD ESTIMATES. For a semiquantal response, the log likelihood involves the application of the multinomial distribution∗ in conjunction with expressions of the form (4) in a generalization of (17). For the Tobit model, the log likelihood involves the sum of components corresponding to the discrete and continuous elements. The log likelihoods corresponding to multiple subsystem and system quantal responses (which reflect the state of several subsystems) may also be defined in terms of the binomial or multinomial distributions. Under certain general conditions the ML estimator is consistent and asymptotically normal. The large-sample variance–covariance matrix of the ML estimators may be calculated as the inverse of the matrix [E{−∂ 2 L/∂θk ∂θl }], where k, l = 1, 2, . . . , s. Tests of hypotheses about the parameters θ may be based on the ML estimators using Wold’s test and the likelihood ratio test∗ . Other forms of estimation have been proposed, largely in an attempt to reduce the computational difficulties associated with the ML method. The most widely used is the modified minimum logit χ 2 method [4] (see MINIMUM CHI-SQUARE). REFERENCES 1. Amemiya, T. (1981). J. Econ. Lit., 1483–1536. 2. Ashford, J. R. (1970). Biometrics, 535–546. 3. Ashford, J. R. (1981). Biometrics, 457–474.
19, 26, 37,
PROBIT MODELS, BAYESIAN ANALYSIS OF 4. Berkson, J. (1955). J. Amer. Statist. Ass., 50, 130–162. 5. Chambers, E. A. and Cox, D. R. (1967). Biometrika, 54, 573–578. 6. Cox, D. R. (1970). The Analysis of Binary Data. Methuen, London. 7. Daganzo, C. (1979). Multinomial Probit. Academic Press, New York. 8. Finney, D. J. (1971). Probit Analysis, 3rd ed. Cambridge University Press, Cambridge, England. 9. Hewlett, P. S. and Plackett, R. L. (1979). The Interpretation of Quantal Response in Biology. Edward Arnold, London. 10. Ipsen, J. and Jerne, N. K. (1944). Acta Pathol., 21, 343–361. 11. Morimune, K. (1979). Econometrica, 47, 957–976. 12. Nerlove, M. and Press, S. J. (1973). Univariate and Multivariate Log-Linear and Logistic Models. Mimeograph No. R-1306-EDA/NIH, Rand Corporation, Santa Monica, Calif. 13. Plackett, R. L. (1974). The Analysis of Categorical Data. Charles Griffin, London. 14. Prentice, R. L. (1976). Biometrics, 32, 761–768. 15. Thurstone, L. (1927). Psychol. Rev., 34, 273–286. See also BIOASSAY, STATISTICAL METHODS IN; ECONOMETRICS; ITERATED MAXIMUM LIKELIHOOD ESTIMATES; NORMAL EQUIVALENT DEVIATE; PHARMACEUTICAL INDUSTRY, STATISTICS IN; QUANTIT ANALYSIS; and RANKIT.
J. R. ASHFORD
PROBIT MODELS, BAYESIAN ANALYSIS OF A practical Bayesian fitting procedure for ordinal response regression models has been proposed [2]. This approach relies on the innovative use of latent data, in conjunction with Markov chain Monte Carlo∗ methods [9,12], to simplify the simulation of the posterior distribution. Consider the binary probit regression model in which n independent measurements yi ∈ {0, 1} are modeled in terms of covariates xi and an unknown parameter vector β. The probit model specifies that pi = Pr(yi = 1) = (xi β), where is the standard normal
6503
cumulative distribution function. Because β appears nonlinearly in this specification, the posterior distribution of β, given data y = (y1 , . . . , yn ), is difficult to simplify by standard methods, for any given prior. The posterior distribution can, however, be analyzed by the following device. Let zi = xi β + εi where εi is N (0, 1), and let yi = I[zi > 0], where I[·] is the indicator function. This two-equation specification is equivalent to the binary probit regression model, since Pr(yi = 1) = Pr(zi > 0) = (xi β). The Albert–Chib algorithm [2] now consists of simulating the joint posterior distribution of Z and β, where Z = (z1 , . . . , zn ). The Markov chain Monte Carlo (MCMC) simulation is implemented by sampling the conditional distributions [β|y, Z] and [Z|y, β]. The first of these distributions depends only on Z (and not on y) and is obtained by standard Bayesian results on normal general linear models∗ . The second factors into a product of independent distributions [zi |yi , β]. The knowledge of yi merely restricts the support of the distribution zi |β; if yi = 0, then zi 0, and if yi = 1, then zi > 0. Suppose that a priori β ∼ Nk (β 0 , B0 ), where β 0 and B0 are known hyperparameters. Then a sample of β and Z from the posterior distribution is obtained by iterating the following steps: 1. Sample βˆ from the distribution ˆ B−1 ), where βˆ = B−1 (B−1 β 0 + N (β, 0 n n −1 i=1 xi zi ), B = (B0 + i=1 xi xi ). 2. Sample zi from T N (−∞,0) (xi β, 1) if yi = 0 and from T N (0,∞) (xi β, 1) if yi = 1, where T N (a,b) (µ, σ 2 ) denotes the N (µ, σ 2 ) distribution truncated to the region (a, b). Example. To illustrate this fitting algorithm, consider the data in Table 1 from [10]. The first column contains the number of days of radio-therapy received by each of 24 patients, and the second column shows the absence (1) or presence (0) of a disse three years after treatment. The probit model pi = (β0 + β1 xi ) is estimated, where pi is the probability that the disease is absent for the ith patient and xi is the number of days of treatment.
6504
PROBIT MODELS, BAYESIAN ANALYSIS OF
Table 1. Distribution of pi Days 21 24 25 25 26 28 29 31 33 34 35 37 43 43 44 46 46 49 51 51 55 55 56 58
Disease
5th
50th
95th
1 1 1 0 1 1 0 1 1 1 1 1 1 0 0 0 0 1 1 0 1 0 0 0
.64 .62 .61 .61 .60 .59 .58 .56 .53 .51 .50 .46 .33 .33 .31 .26 .26 .19 .15 .15 .08 .08 .07 .05
.89 .86 .85 .85 .83 .81 .79 .76 .72 .71 .69 .65 .52 .52 .50 .46 .46 .40 .36 .36 .28 .28 .26 .23
.99 .97 .97 .97 .96 .94 .93 .91 .88 .86 .85 .81 .70 .70 .69 .66 .66 .62 .60 .60 .56 .56 .55 .53
Using a uniform prior, 5000 values were simulated from the joint posterior distribution of β = (β0 , β1 ). For each simulated value of β, we compute pi = (β0 + β1 xi ). This procedure gives a simulated sample from the marginal posterior distribution of the probability pi . In Table 1, the 5th, 50th, and 95th percentiles of the posterior distribution of pi are displayed for each patient. Consider next the cumulative probit model for independent ordinal data yi ∈ {0, 1, . . . , J}. Under this model, Pr(yi = j) = (γj − xi β) − (γj−1 − xi β) where γ = (γ0 , γ1 , . . . , γJ ) are ordered category-specific parameters. The posterior distribution of the parameters (β, γ ) can again be simulated by defining a latent variable zi distributed as N (xi β, 1), but now letting yi = j if zi ∈ (γj−1 , γj ). In this case, the posterior simulation of zi is from a normal distribution truncated to the interval that is implied by the observed value of yi . The simulation of β is as in the binary probit model. Finally, γ is sampled from γ |y,
as recommended in refs. 8, 13, using the Metropolis–Hastings algorithm [6]. The correspondence of the probit model with an underlying continuous latent normal model allows for many generalizations. Albert and Chib [2] extend the probit link by letting zi follow a scale mixture of normal distributions. Hierarchical models∗ and residual analysis have been discussed [2,3], as well as the comparison of alternative probit models via Bayes factors [5,7]. Some of the subsequent applications have involved item response models [1,11], longitudinal data models [4,9], and correlated binary responses using the multivariate probit model [7]. REFERENCES 1. Albert, J. H. (1992). Bayesian estimation of normal ogive response curves using Gibbs sampling. J. Educ. Statist., 17, 251–269. 2. Albert, J. H. and Chib, S. (1993). Bayesian analysis of binary and polychotomous response data. J. Amer. Statist. Ass., 88, 669–679. 3. Albert, J. H. and Chib, S. (1995). Bayesian residual analysis for binary response regression models. Biometrika, 82, 747–759. 4. Albert, J. H. and Chib, S. (1996). Bayesian probit modeling of binary repeated measures data with an application to a cross-over trial. In Bayesian Biostatistics, D. A. Berry and D. K. Stangl, eds. Marcel Dekker, New York, pp. 577–599. 5. Chib, S. (1995). Marginal likelihood from the Gibbs output. J. Amer. Statist. Ass., 90, 1313–1321. 6. Chib, S. and Greenberg, E. (1995). Understanding the Metropolis–Hastings algorithm. Amer. Statist., 49, 327–335. 7. Chib, S. and Greenberg, E. (1998). Analysis of multivariate probit models. Biometrika, 85, 347–361. 8. Cowles, M. K. (1996). Accelerating Monte Carlo Markov chain convergence for cumulative-link generalized linear models. Statist. and Comput., 6, 101–111. 9. Cowles, M. K., Carlin, B. P., and Connett, J. E. (1996). Bayesian tobit modeling of longitudinal ordinal clinical trial compliance data with nonignorable missingness. J. Amer. Statist. Ass., 91, 86–98. 10. Gelfand, A. E. and Smith, A. F. M. (1990). Sampling-based approaches to calculating
PROBLEM OF POINTS
11.
12.
13.
14.
marginal densities. J. Amer. Statist. Ass., 85, 398–409. Johnson, V. E. (1996). On Bayesian analysis of multirater ordinal data: an application to automated essay grading. J. Amer. Statist. Ass., 91, 42–51. Mendenhall, W. M., Parsons, J. T., Stringer, S. P., Cassissi, N. J., and Million, R. R. (1989). T2 oral tongue carcinoma treated with radiotherapy: analysis of local control and complications. Radiotherapy and Oncol., 16, 275–282. Nandram, B. and Chen, M. -H. (1996). Accelerating Gibbs sampler convergence in the generalized linear models via a reparameterization. J. Statist. Comput. Simulation, 54, 129–144. Tierney, L. (1994). Markov chains for exploring posterior distributions. Ann. Statist., 22, 1701–1762. (With discussion.)
See also BINARY DATA; GIBBS SAMPLING; MARKOV CHAIN MONTE CARLO ALGORITHMS; ORDINAL DATA; and PROBIT ANALYSIS.
JAMES H. ALBERT SIDDHARTHA CHIB
PROBLEM OF POINTS The ‘‘problem of points’’ is the name given to a simple gaming problem that came into prominence in the Renaissance and was not correctly solved until the seventeenth-century French mathematicians Fermat∗ and Pascal∗ considered it. Their famous correspondence about the problem, and Pascal’s treatment of it in his Trait´e du Triangle Arithm´etique, are important milestones in the theory of probability. The problem, also known as the ‘‘division problem,’’ involves determining how the total stake should be equitably divided when a game is terminated prematurely. Suppose that two players A and B stake equal money on being the first to win n points in a game in which the winner of each point is decided by the toss of a fair coin, heads for A and tails for B. If such a game is interrupted when A still lacks a points and B lacks b, how should the total stake be divided between them? Particular cases of the problem have been noted in Italian mathematical manuscripts
6505
as early as 1380, and the Renaissance mathematicians Pacioli, Tartaglia, and Peverone all made unsuccessful efforts at solutions. It was brought to Blaise Pascal’s attention by Antoine Gombaud, Chevalier de M´er´e, and led to a correspondence with Pierre de Fermat in the late summer of 1654. Unfortunately, not all the letters have survived, but it is possible to reconstruct the sequence of events from the remainder. Pascal sought Fermat’s opinion, and Fermat replied in a letter which, although missing, like Pascal’s, must have contained solutions to two gaming problems, one of which was the problem of points, for in his reply on July 29, Pascal expressed his agreement with Fermat’s solution to the latter, which had also occurred to him. If A wants a points and B wants b, they argued, then the game must be over in (a + b − 1) further tosses at most, which may occur in 2a+b−1 equiprobable ways. Each of these ways may be seen, on examination, to be a win either for A or for B, and on counting them the probability (as we should now say) of A winning is found, and hence the equitable division of the stakes given that the game is not in fact continued. In other words, the actual game has been embedded in a hypothetical game of fixed length which is easy to solve. In the same letter Pascal offered as an alternative solution an analysis of the possible games in which, working backwards, he uses recursively the idea that if expectations of gain of X and Y units are equally probable, the expectation of gain is 12 (X + Y) units, thereby introducing into probability theory the notion of expectation. Then, having appreciated that with only two players the order of occurrence of heads and tails in the hypothetical game does not matter, he makes use of the fact that the hypothetical game is solved by the partial sum of a binomial distribution∗ for equal chances. In his contemporary Trait´e du Triangle Arithm´etique he proves this result by applying mathematical induction to the method of expectations. The result itself is often attributed to Fermat, and the method of expectations to Huygens∗ . Not only are they Pascal’s but he has also introduced the binomial distribution for equal chances, albeit by
6506
PROBLEM OF THE NILE
reference to the arithmetical triangle rather than algebraically. The remainder of the Fermat–Pascal correspondence is devoted mostly to the case of three players, which Fermat had suggested could readily be solved by the original ‘‘combinatorial’’ method. Pascal thought that by this Fermat meant assigning each of the possible hypothetical games of fixed length to the appropriate player according to the numbers of points each had won without regard to order, as was correct with two players, and he demurred. Fermat reassured him that he had intended, rather, that each possible sequence should be assigned, which leads to the correct solution, and he added for good measure the direct probability solution to an example with three players, in which he simply analyses the possible games in a forward direction (as a modern student would do with the help of a tree diagram) in contrast to Pascal’s backward ‘‘expectation’’ solution. Apart from some minor generalizations introduced by Montmort∗ , the Bernoulli∗ brothers, and De Moivre∗ half a century later, the correspondence between Fermat and Pascal and the account in the latter’s Trait´e du Triangle Arithm´etique essentially disposed of the problem of points and in doing so created an incipient theory of probability where previously there had only been simple enumerations. The important methodological contributions of operating on expectations and of constructing the binomial distribution for equal chances and applying it to the problem of points were Pascal’s alone, however. BIBLIOGRAPHY The account above relies on the primary sources directly, as follows. For the Fermat–Pascal correspondence, see either Oeuvres de Fermat, edited by P. Tannery and C. Henry, Vol. II, Correspondance (Gauthier-Villars, Paris, 1894), or B. Pascal, Oeuvres Compl`etes, edited by J. Mesnard, Volume II, (Descl´ee de Brouwer, Paris, 1970). English translations (which should both be treated with reserve) are available in A Source Book in Mathematics, by D. E. Smith (McGraw-Hill, New York, 1929), and Gods, Games and Gambling, by F. N. David (Charles Griffin, London, 1962). Pascal’s Trait´e du Triangle Arithm´etique was originally published by Desprez in Paris in 1665 and is
contained in his Oeuvres cited above. There is a partial English translation in D. E. Smith (see above) and an even lesser part in A Source Book in Mathematics, by D. J. Struik (Harvard University Press, Cambridge, Mass., 1969), but neither includes the treatment of the problem of points. There are many secondary accounts, but we mention only the invaluable A History of the Mathematical Theory of Probability, by I. Todhunter (Macmillan, Cambridge, 1865; reprinted by Chelsea, New York, 1965). See also FERMAT, PIERRE DE; GAMES OF CHANCE; PASCAL, BLAISE; and PROBABILITY, HISTORY OF.
A. W. F. EDWARDS
PROBLEM OF THE NILE. See FISHER’S PROBLEM OF THE NILE
PROBLEM SOLVING IN STATISTICS A basic approach to problem solving in statistics is to compare sets of numbers: to compare observations with theoretical models, past experience, or other observations in numerical form [9]. Another approach is to make estimations based on models, experience, or observations. Such comparisons and estimations are valid and informative only if the observations and models are appropriately (and measurably) representative of some phenomenon or population. Therefore, a working knowledge of statistics is important in the planning stages of investigations as well as in determining what statistical comparisons and models are appropriate. A common complaint of practicing statisticians is that often statistical consultants are not called in until after the data have been collected [25]. One of the purposes of this article is to stress the full range of the statistical process in order to correct a narrow public image of the discipline. To provide a systematic framework for a discussion of problem solving in statistics, a conceptual model has been adapted from information systems literature, including three distinct phases: (1) study design, (2) data collection∗ , and (3) data analysis. These phases, outlined in Table 1, have been presented as an alternative to an experimental model in a business context [16]; the process
PROBLEM SOLVING IN STATISTICS
6507
Table 1. Phases and Steps of Statistical Problem Solving Phase 1. Study design
2. Data collection
3. Data analysis
Step a. Question definition b. Alternative strategies development c. Strategy evaluation d. Strategy selection a. Sample design b. Measurement a. Statistical analysis b. Reliability assessment c. Report generation
Activity The question is defined in terms of the need for information. Alternative strategies are developed and specified for sampling, data collection, and analysis. The advantages and disadvantages of the feasible alternatives are evaluated. A strategy is selected on the basis of costs and the importance of the information to the organization. Sampling procedures are planned on the basis of work done in step 1b and the selection made in step 1d. Observations are chosen and recorded in a form that facilitates analysis Statistical methods are used for estimating or summarizing. Measures of possible error in results are calculated. The results are reported in a form useful to others.
model has been revised here to encompass the general procedures required in any application area. PHASE 1: STUDY DESIGN The first phase involves determining the optimum strategy to solve the problem. The steps from the beginning of the process until final selection of a strategy are illustrated in Fig. 1. These are important steps since inadequate planning means no data, or worse, meaningless data, to analyze later. The experimenter should consider all the possibilities of misfortune as well as unexpected interactions of the variables, because results are seldom exactly what is anticipated. If a statistical consultant is involved, he or she should take an active part in planning the methodology of the study itself and not wait to help with the analysis of data that may or may not be meaningful [14,15]. Problem Definition In the first step of the study design phase the question or problem must be defined in terms of the need for information. Tukey [28] cautions that ‘‘finding the question is often more important than finding the answer.’’ This question-finding step can be a very
difficult one with none of the tidiness of a well-defined mathematical problem [15,23]; however, it is crucial to the meaningfulness of the subsequent phases and steps. If the scientist, economist, demographer, or whoever who wants the problem solved has the required statistical skills, the first phase is mainly one of defining the question clearly in order to proceed with an efficient design. ‘‘What is the real question I am interested in, and how can it logically be phrased so that it is answerable with data that can be obtained with available resources?’’ If a statistician is assisting one or more experts in a consulting∗ situation, the statistician should aid in the problem definition process by discussing the problem and determining the exact question of interest. Consultants have reported that their clients have found this initial problem definition assistance to be the most valuable service provided by the statistician [3,7,14]. Several articles [3,14,21] specifically focus on the help that statisticians can provide in developing the objectives of a study. Salsburg [23] describes his role in this step as one of asking stupid questions and worrying out loud about everything that could go wrong in the data collection stage. Statistics: A Guide to the Unknown [27] provides
6508
PROBLEM SOLVING IN STATISTICS
Figure 1. Steps in the study design phase.
reasonable examples of statistical designs and analysis to answer such ill-defined, difficult, and interesting questions as: Are police effective? Can people control when they die? In these examples and other real problems, the definition of the logical issues and the experimental design are the main challenge, rather than the choice and execution of analysis techniques. To be effective in the planning stages of an investigation, the statistician must know enough of the background subject (or ask enough questions) to be sure that the study fits the real problem [13,24]. Working with a client to determine the logical issues of a problem also may take special interpersonal skills; suggestions on this topic can be found in a discussion of the use of transaction analysis by statistical consultants [3], an identification of consulting roles [14,25], and a reading list on the training of consultants [2]. Identification of Alternatives After the problem is well defined, the statistician develops alternative procedures for the
experiment, sample survey, or other procedure appropriate for data collection. The value of the information gained in the final phase depends largely on the thoroughness of the data collection and data analysis phases, so the statistician must plan ahead. Failing to allow for dropped test tubes or uncooperative subjects may mean that there will be no usable data for the analysis stage. Evaluation of Alternatives For each alternative procedure the statistician should present a cost estimate and an error range indicating the reliability of the answer. The optimum strategy is not necessarily the one that provides the most information. Within any organization cost factors must be weighed against benefits, so the ‘‘best’’ strategy is one that provides adequate information with a justifiable cost and a reasonable amount of time. (A statistician who does not appreciate this fact is unlikely to enjoy a successful career in industry.) Each feasible alternative must be evaluated. In the final step of phase 1 a strategy is selected on the basis of cost and the
PROBLEM SOLVING IN STATISTICS
importance of the information to the organization. If the statistician has done a good job, the chosen strategy will provide a specified quantity of information at a minimum cost. At the end of phase 1 the project is approved with an understanding of how much time the study will take, how much it will cost, and the quality and type of inferences that can be made from the results. A written statement of the objectives of the study and the time and budget constraints should be prepared by the statistician before the study proceeds to the data collection stage. See CONSULTING, STATISTICAL; INDUSTRY, STATISTICS IN; PRINCIPLES OF PROFESSIONAL STATISTICAL PRACTICE. PHASE 2: DATA COLLECTION The data collection phase includes two steps: design of the sampling procedure and measurement of observations or collection of data. The time requirements and difficulty level of these steps depend on the problem. The first step consists of planning what will be done in the second step. Time-consuming, complicated observations and measurements will usually require time-consuming, detailed planning in the first step. Sample Design The total group to be studied was determined in the problem definition step of the process. The sample may be birds in a field [13], treated water [5], filtered air [21], or earthworms [26], to name a few examples from the literature that discuss the entire problemsolving process. In a quality control∗ study the target population may be all the machine parts delivered by a particular supplier. Testing millions of machine parts supposedly checked by the original manufacturer would be infeasible for the manufacturer who wishes to use these parts in assembling a product; instead, sampling is used. In a study of the environmental impact of a factory process the population or universe under investigation may be treated water in a lake, filtered air in a plant facility, or shrimp in a bay. Again it would be infeasible to measure impurities in all the air and water or to check every shrimp for contamination. The sampling procedures
6509
for the machine parts, air, water, and shrimp are based on probability theory∗ . The planning for the sample is done in the first step of this data collection phase. Measurement The complexity of the planning step varies considerably with the problem. For projects that require complicated data collection procedures, the second step of this phase requires careful planning. The design to select the air, water, and shrimp samples takes considerable time, as to the actual measurement and observation procedures in the second step of this phase. Accountants rely on sampling to audit large organizations. Courts depend on sampling to determine compensation in law suits involving thousands of plaintiffs. Market research surveys always involve sampling. An effective sampling design can ensure a valid audit, appropriate compensation, and invaluable market forecasts, without sacrificing economy (see MARKETING, STATISTICS IN). In the easiest case the sample includes all company computer records pertinent to the question and available at a particular time. In this situation the planning for data collection requires the choice of which file of information is appropriate for the question of interest; this might be the choice of annual, monthly, or end-of-the-day data on a particular date. The next step, the actual data collection, is completed by one command in a computer job that accesses the chosen data file. For example, airlines regularly analyze route schedules and passenger loads. The data source is the company file that is created and updated in the reservation process that goes on every hour of the day in airports and travel agencies around the world. Making this file of passenger information available with one computer command is all that is necessary before proceeding to the analysis phase. In the other extreme the Bureau of the Census takes years to plan and carry out the data collection for the U.S. census and to make available the results. Demographic studies are used to choose the sample of individuals asked to complete more detailed census forms. The actual collection of data
6510
PROBLEM SOLVING IN STATISTICS
and recording of survey answers involves thousands of full-time and part-time employees of the Bureau of the Census∗ . Issues in Data Collection The role of the statistician in developing sound methodology is stressed in the study design phase. The logical issues of problem solving in the definition of the research question are critical to the value of any study, even though they are not basically a mathematical concern. In the data collection∗ phase, the role of the statistician as a scientific advisor is just as important. The difficulties of designing the sample and problems with the actual measurement are considerable. Insofar as statistics is an applied science, justified by the ‘‘help it can give in solving a problem . . . [reaching] a decision, on a probabilistic basis’’ [18], the problem-solving methodology must be repeatable and reproducible [1]. Such repeatability and reproducibility are dependent on the quality of statistical data. Even the most sophisticated multivariate analysis∗ cannot correct for data that are inconsistent or clearly wrong. Kruskal [17] discusses fascinating problems arising in the public policy application of statistics which are often neglected since they fall in the cracks between disciplinary labels: (1) wrong data (e.g., 1960 census data showing 62 women aged 15 through 19 with 12 or more children) identified by lack of smoothness, logical inconsistencies, or transgressions of general knowledge; (2) ambiguity of classification (when might an unemployed person be said to be in the labor force?); (3) lack of data responsibility (unwillingness to publish the raw data); which relates to (4) confidentiality (can we learn information about specific individuals from a published table of cross classifications?). Traditionally, the ‘‘quality of statistical data’’ has referred to its accuracy [1], which is often judged by insisting on high precision and low bias. In judging this quality, we must realize that smoothness of repeated observations does not guarantee that the estimate is close to the ‘‘truth.’’ When we report ‘‘estimate ± something,’’ we imply an emphasis on precision, which does not necessarily imply accuracy. (See Eisenhart [11] for a comprehensive discussion of
this distinction.) Therefore, a complete and clear disclosure of the measurement process is necessary for those who will evaluate the results. (See CENSUS; MEASUREMENT ERROR; SAMPLING PLANS; SURVEY SAMPLING; TELEPHONE SURVEYS, COMPUTER ASSISTED).
PHASE 3: DATA ANALYSIS The data analysis phase includes three steps: statistical analysis, reliability assessment, and report generation. The sequence of steps is shown in Fig. 2. The methods employed in this phase are the main concern of statistics. Statistical Analysis Articles on specific statistical techniques are a major portion of this encyclopedia, so no attempt at a complete list will be made here. Summary lists of the most frequently used techniques have been complied for industrial statisticians [22], consultants [19,20], and statisticians in the federal government [12]. Basic analysis strategies include hypothesis testing∗ (the approach used in experimentation), estimation∗ techniques, and exploratory data analysis∗ (for a general understanding of relationships that exist in data). Summaries of statistical applications in various fields are also provided in this encyclopedia; for example, see AGRICULTURE, STATISTICS IN; BIOASSAY, STATISTICAL METHODS IN; BIOSTATISTICS, CLASSICAL; CHEMISTRY, STATISTICAL METHODS IN, and so on. Impact of Computers on Analysis The availability and accessibility of computers has had a pervasive impact on statistics as complicated statistical procedures come into routine use in analysis of large data sets (see COMPUTERS AND STATISTICS). Multivariate techniques of stepwise regression∗ , discriminant analysis∗ , and categorical techniques are now feasible as exploratory tools due to the extensive calculating ability of computers. Statistical packages have made data analysis (and misanalysis) widely accessible with multidimensional computer graphics and quick results of data transformation.
PROBLEM SOLVING IN STATISTICS
6511
Figure 2. Steps in the data analysis phase.
In addition to extending the accessibility of established statistical methodology, computers have played a vital role in the development of new statistical techniques. Diaconis and Efron [8] observe that the most commonly used statistical methods were developed when computation was slow and expensive; they point out that many new methods are free of distributional assumptions and are feasible, as a result of the use of computers. Monte Carlo∗ simulation techniques are easily employed to examine the performance of a new estimator or test. Efron [9] asserts that ‘‘computer assisted theory is no less ‘mathematical’ than the theory of the past; it is just less constrained by the limitations of the human brain.’’ The empirical Bayes∗ procedure is an example of a technique feasible only with the advent of computer technology. Empirical Bayes’ inference differs from Bayesian inference∗ in that a prior distribution∗ is assumed but not known, thus allowing the applied statistician to derive from the data a compromise between classical frequentist and Bayesian estimators. This procedure has been shown to provide accurate estimates in practical multivariate applications and is an example of a procedure whose performance has been assessed without formal proof, but with the aid of a computer. The computer makes feasible nonparametric estimators involving massive amounts of computation. Consider, for example, the simple problem of determining whether two
samples have been drawn from a single population. A frequentist addresses this question with a t-test, which uses only a few calculations but assumes the (common) population is Gaussian. Computers make possible the randomization∗ approach of deriving all possible ways of partitioning the combined data into two like-sized samples and assessing the likelihood of the original partition among all possible samples. Additionally, computers make possible the use of the jack-knife∗ and bootstrap∗ procedures to measure accuracy of estimators of population parameters without the Gaussian assumption (see SAMPLE REUSE). Cross-validation, combinations of robustness and efficiency, and analysis of censored data∗ additionally illustrate the impact of computers on previously impossible data analytic problems. Thus an aid to statistical analysis has now begun to shape the statistical methods themselves. Reliability Assessment The second step of the analysis phase is to determine how much error can be expected in the estimations or inferences made from a statistical procedure. Each technique presented in this encyclopedia includes a measure of reliability to indicate how much confidence can be placed in a decision based on the statistical results. In the final step of the analysis phase, the results must be reported in a form that is intelligible to the user. Stating the results of a statistical investigation in terms of the
6512
PROBLEM SOLVING IN STATISTICS
original objectives of the study is analogous to turning the numerical results of an arithmetic word problem back into words. [13]. When complicated statistical procedures have been used, this is a very important step; when controversial issues are involved, it is a very difficult step [22]. Ehrenberg [10] provides tips for effective statistical report writing. Hunter [15] and Salsburg [23] provide motivation. See also GRAPHICAL REPRESENTATION OF DATA. POSTPROCESS RESPONSIBILITIES At the conclusion of any worthwhile study a decision is made based on the results. The decision may be to market a new drug that has been shown to have no adverse effects, reallocate federal funds to states due to shifts in population, continue with present policies of proven worth, or gather more information for further study. This postprocess decision is usually out of the hands of the statistician. Technically, it should not be the responsibility of the statistician to determine what the implications of a 0.05 significance level are in each of the examples just named or even to set the significance level. But, perhaps, professional responsibilities require some continued interest in the actions taken based on statistical results. Kruskal [17] outlines several problems that affect the value of results of statistical studies for which no one will take responsibility—statistician or client. The ethical guidelines proposed in 1983 have also generated some questions about the responsibility of the statistician as opposed to the responsibility of the organization that acts on the study results. For example, should a statistician allow his or her work to be represented as investigating ‘‘cancer cure rates’’ when success was measured as a cancer victim’s failure to die [6]? CONCLUSIONS Statistics is a broad problem-solving process used for organizing and thinking about logical issues as well as modeling data to arrive at inferences about the true state of nature. As educators, statisticians should be concerned about teaching statistical problemsolving skills as well as methods. Managers,
engineers, and anyone else who requires the services of a statistician should at least learn that the proper time to hire is at the beginning of a statistical study, not after the data are gathered. No amount of probability theory or formula solving with clean data will communicate this knowledge effectively. It is necessary to convey a sense of the importance of structuring real problems so that they can be solved, as well as to teach statistical theory and methods. Acknowledgments We are grateful to G. J. Hahn, W. G. Hunter, and M. M. Whiteside for helpful comments on an early draft of this article. We also thank South-Western Publishing Co. for permission to use copyrighted material, including Figs. 1 and 2, from Introduction to Business and Economic Statistics (7th ed.) by C. T. Clark and E. W. Jordan.
REFERENCES 1. Bailar, B. A. (1983). Bull. Int. Statist. Inst., 813–834. (A complete discussion of concepts related to the quality of statistical data.) 2. Baskerville, J. C. (1981). Amer. Statist., 35, 121–123. (Problem solving is discussed briefly in the context of a course in statistical consulting; bibliography for course included.) 3. Bishop, T., Petersen, B., and Trayser, D. (1982). Amer. Statist., 36, 387–390. (Illustrations of types of information that should be obtained during experimental planning stages.) 4. Boen, J. and Fryd, D. (1978). Amer. Statist., 32, 58–60. (Interactions between statisticians and clients are explained in terms of transactional analysis; valuable for both parties.) 5. Box, G. E. P., Hunter, W. G. and Hunter, J. S. (1978). Statistics for Experimenters. Wiley, New York. (Chapter 1 is a readable overview of the role of statistics in experimental design; text is oriented toward scientific and engineering problem solving with topics such as ‘‘lurking variables’’ and reducing data models to ‘‘white noise.’’) 6. Bross, I. D. J. (1983). Amer. Statist., 37, 12–13. (Examples of statistical results that are misrepresented by policy makers; relevant to issue of boundary of responsibilities of statistician in reporting phase.) 7. Daniel, C. (1969). Technometrics, 11, 241–245. (Discussion of elements required
PROBLEM SOLVING IN STATISTICS
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
for successful statistical problem solving in consulting situations.) Diaconis, P. and Efron, B. (1983). Science, Apr., 116–130. (Summary of statistical methods made usable by computers.) Efron, B. (1979). SIAM Rev., 21, 460–480. (Very brief summary of standard pre-1950 statistical problem solving, followed by examples of statistical approaches that have changed dramatically due to computers.) Ehrenberg, A. S. C. (1982). A Primer in Data Reduction. Wiley, New York. (Chapters 16 and 17 cover the use of tables and graphs in reports; Chapter 18 includes recommendations on organizations and writing style.) Eisenhart, C. (1963). J. Res. Nat. Bur. Stand. C. Eng. Instrum., 67, 161–187. (A discussion of issues relating to precision and accuracy.) Eldridge, M. D., Wallman, K. K., Wulfsberg, R. M., Bailar, B. A., Bishop, B. A., Bishop, Y. M., Kibler, W. E., Orleans, B. S., Rice, D. P., Schaible, W., Selig, S. M., and Sirken, M. G. (1982). Amer. Statist., 36, 69–81. (Describes problem solving in a wide variety of positions for statisticians in the federal government.) Hahn, G. J. (1984). Technometrics, 26, 19–31. (Examples of problem solving in engineering and science with recommendations on how to improve the practice and use of statistics in industry.) Hunter, W. G. (1981). Amer. Statist., 35, 72–76. (Discussion of three roles played by statistical consultants; valuable for practicing statisticians and educators in applied statistics programs.) Hunter, W. G. (1981). Statistician, 30, 107–117. (A very enjoyable series of stories about the reality of applied statistics.) Jordan, E. W. (1982). Amer. Statist. Ass., 1982 Proc. Statist. Educ. Sec. (Examples of problem solving in business with recommendations on how to improve education in statistics for managers.) Kruskal, W. H. (1981). J. Amer. Statist. Ass., 76, 1–8. (Discussion of problems, such as data integrity in large data sets, that affect the validity of results of statistical studies, but that are not acknowledged as the responsibility of statisticians.) Mahalanobis, P. C. (1950). Sankhya, ¯ 10, 195–228. (A delightful essay entitled ‘‘Why Statistics?’’) Marquardt, D. W. (1979). Amer. Statist., 33, 102–107. (Very broad view outlined for responsibilities of statisticians in industry.)
6513
20. Marquardt, D. W. (1981). Amer. Statist., 35, 216–219. (Argument for total involvement in problem solving by statisticians in industry.) 21. Price, B. (1982). Amer. Statist. Ass., 1982 Proc. Phys. Eng. Sci. Sec. (The need to clarify objectives and specify realistic achievements for any study is discussed within the context of an environmental regulations problem.) 22. Raiffa, H. (1982). Amer. Statist., 36, 225–231. (Discussion of the difficulty of maintaining statistical objectivity in controversial application areas of risk analysis for problems such as chemical waste disposal; includes examples that could divest anyone of an image of statistics as a cold science.) 23. Salsburg, D. S. (1973). Amer. Statist., 27, 152–154. (Delightful article about how a practicing statistician earns a living.) 24. Snee, R. D. (1982). Amer. Statist., 36, 22–87. (Argument for broader view of the field of statistics, including topics that should be added and roles that should be acknowledged.) 25. Snee, R. D., Boardman, T. J., Hahn, G. J., Hill, W. J., Hocking, R. R., Hunter, W. G., Lawton, W. H., Ott, R. L., and Strawderman, W. E. (1980). Amer. Statist., 34, 65–75. (Includes list of statistical techniques that are used most often in problem solving by statisticians in industry; discussion of personal characteristics and training necessary for industrial statisticians.) 26. Sprent, W. G. (1981). J. R. Statist. Soc. B, 133, 139–164. (Difficulties in consulting are discussed; necessity of consultant being involved in projects is stressed by Sprent; wide variety of comments by members follow, including argument about role of statistician in problem solving.) 27. Tanur, J. M., Mosteller, F., Kruskal, W. H., Link, R. F., Pieters, R. S., Rising, G. R., and Lehmann, E. L., eds. (1978). Statistics: A Guide to the Unknown, 2nd ed. Holden-Day, San Francisco. (Series of examples from all fields of applied statistics.) 28. Tukey, J. W. (1980). Amer. Statist., 34, 23–25. (An argument for both exploratory and confirmatory methods in data analysis.) See also COMPUTERS AND STATISTICS; CONSULTING, STATISTICAL; DATA COLLECTION; DESIGN OF EXPERIMENTS; EXPLORATORY DATA ANALYSIS; HYPOTHESIS TESTING; INDUSTRY, STATISTICS IN; PRINCIPLES OF PROFESSIONAL STATISTICAL PRACTICE; QUALITY CONTROL, STATISTICAL; and SAMPLING PLANS.
ELEANOR W. JORDAN DONNA F. STROUP
6514
PROCESS CAPABILITY INDEX, WRIGHT’S
PROCESS CAPABILITY INDEX, WRIGHT’S
2. Wright, P. A. (1995). A process capability index sensitive to skewness. J. Statist. Comput. and Simul., 52, 195–208.
A process capability index which takes skewness into account was proposed by Wright [2]. The index is defined as
See also CAPABILITY and MEAN, MEDIAN, AND MODE.
min(USL − µ, µ − LSL) Cs = 3 σ 2 + (µ − T)2 + |µ3 /σ | d − |µ − M| , = 3 σ 2 + (µ − T)2 + |µ3 /σ | where USL and LSL are the upper and lower specification limits, µ and σ are the mean and the standard deviation of the process, respectively, T is the target value, d = (USL − LSL)/2 is the half length of the specifications interval, M = (USL + LSL)/2 is the midpoint of this interval, and µ3 = (X − µ)3 is the third central moment. It is implicitly assumed that the target value is at the midpoint. It seems to be the only index of capability which accounts for skewness that is proposed in the literature. A natural estimator of Cs is given by d − |X − T| ˆs = , C 3 S2 + (X − T)2 + |m3 /S| where m3 = (Xi − X)3 /n, X = Xi /n, and S2 = (Xi − X)2 /n. Wright also proposes a somewhat more complicated estimator of Cs to achieve efficient small sample behavior. ˆ s have been Asymptotic properties of C investigated by Chen and Kotz [1]. The index and its estimator are quite sensitive to skewness; the estimator has an asymptotic normal distribution only when µ = T and µ3 = 0, which indicates that when the process is ‘‘deviating’’ in both shape and the target, ˆ s is indeed a natural choice of a meaC sure to assess the process capability. However, if the process stays symmetric, the simpler canonical index Cp could be recommended. REFERENCES 1. Chen, H. F. and Kotz, S. (1996). An asymptotic distribution of Wright’s process capability index sensitive to skewness. J. Statist. Comput. and Simul., 54, 27–38.
PROCESSES, DISCRETE All physical and natural phenomena are subject to environmental and other types of fluctuations or noises. These phenomena are therefore of stochastic nature. A stochastic process∗ arises when one observes the evolution of such a random phenomenon as a function of time. When the process is observed at discrete time points, one obtains a discrete process. More precisely, a discrete process is a sequence Xn , n 1, of random variables (RVs) defined on an appropriate sample space . A space X containing all values of the RVs Xn , n 1, is called the state space of the process. In most practical cases X is a Euclidean space Rm . Consider, for example, the study of a population growth at nonoverlapping generations. Then the population size Xn at generation n is a RV, and Xn , n 1, is a discrete process. Discrete processes arise in the theories of statistical estimation and optimal filtering. For example, let θ (t) be a deterministic signal transmitted at time points t = 1, 2, . . ., and ξ (t, ω) be random noises entering the communication channel at these time points. Then the signal received is given by X(t) = θ (t) + ξ (t). In estimation theory, one needs to determine operations on the signal X that reproduces θ as precisely as possible. We seek an operation that produces a discrete process θ (t) such that E[X(t) − θ (t)] = 0 and E[X(t) − θ (t)]2 is minimum. (We will return to this problem after some mathematical preliminaries.) Let Xt be a discrete process with state space X . Let us first clarify the meaning of the adjective ‘‘discrete’’. We began by assuming that the time parameter set T is discrete (such as T = 1, 2, . . .). In defining discrete processes, some authors let t vary continuously while assuming that the state space is discrete. While noting that a countable state space is a particular case of a general state space, we also see that discrete time is not a particular case of continuous time.
PROCESSES, DISCRETE
To be more precise, consider the cases where T = {0, 1, 2, . . .} = N0 , and T = R+ . While R+ has plenty of limit points, N0 has ∞ as its only limit point. Consequently, in studying the limit behavior of a discrete-time process one considers the asymptotic behavior at ∞, whereas in the continuous-time process case many new properties such as the local path properties arise. Furthermore, most of the results in the discrete-time case concern the asymptotic behavior at ∞ of functions of the process. Therefore, it is more appropriate to name the discrete-time case as the discrete process; of course, the state space X can be discrete or continuous. Let X = {Xn (ω), n 1} be a discrete process. It is common to suppress the variable ω in Xn (ω) and write it only as Xn . If we fix ω ∈ , we get a sequence Xn in X , called a sample path or sample sequence. Thus each sample point in represents a path in the state space X . To keep the presentation simple we shall assume henceforth that X = R = R1 . For every integer d 1, let us arbitrarily choose time points n1 < n2 < · · · < nd . The joint distribution of the RVs Xn1 , Xn2 , . . . , Xnd is called a d-dimensional distribution of the process X = {Xn }. The collection of all these finite-dimensional distributions for all possible choices of n1 < · · · < nd and d 1 governs the probabilistic laws of X. Stochastic processes are classified according to the special properties of their finite-dimensional distributions. We need not distinguish between two processes X = {Xn } and Y = {Yn } that have the same finite-dimensional distributions; that is, the multivariate distributions of Xn1 , . . . , Xnd and Yn1 , . . . , Ynd are the same for all d 1 and all choices of n1 < · · · < nd ∈ T. The researcher is therefore free to choose, from a class of processes having the same finite-dimensional distributions, any process that he or she wishes to work with. In many practical situations the researcher knows the finite-dimensional distributions of a stochastic phenomenon and would like to associate a discrete process with them. Such an association is possible under Kolmogorov’s extension theorem [13]. Let Fn1 ...nd (ξ1 , . . . , ξd ) denote the finite-dimensional distributions of the RVs Xn1 , . . . , Xnd of a process X. Then the set of all finite-dimensional distributions
6515
satisfies the consistency condition lim Fn1 ...nd (ξ1 , . . . , ξd )
ξk ↑∞
= Fn1 ...nk−1 nk+1 ···nd (ξ1 , . . . , ξk−1 , ξk+1 , . . . , ξd ). Conversely, given a sequence {Fn } of ndimensional distributions satisfying the consistency condition limξn ↑∞ Fn (ξ1 , . . . , ξn ) = Fn−1 (ξ1 , . . . , ξn−1 ), one can find a sample space , an appropriate probability P on , and a process X = {Xn } such that Fn (ξ1 , . . . , ξn ) = P{X1 ξ1 , . . . , Xn ξn }, n 1. In place of the finite-dimensional distributions Fn of a process X, one can work with the probabilities P{X1 ∈ B1 , . . . , Xn ∈ Bn }, Bk ∈ B, where B is the Borel σ -algebra on R. Kolmogorov’s extension theorem will transform a process X into a probability measure on = RT . Treating a discrete process as a probability measure finds nice applications in statistics, for example, in maximum likelihood estimation∗ . Let θ = θt be an unknown parameter, and X(t) = θt + ξt , t = 0, ±1, . . ., where ξt is a suitable stationary Gaussian process∗ . Let µθ and µ0 be the distributions or measures corresponding to the processes Xt and ξt , respectively. To obtain the maximum likelihood estimator of θ , one needs to find the Radon–Nikodym∗ derivative dµθ /dµ0 , the density function of µθ with respect to µ0 (see Ibragimov and Rozanov [9]). A discrete process X = {Xn } is a secondorder process if EXn2 < ∞ for all n. For such a process the functions µ(n) = EXn and γij = EXi Xj form the mean sequence and correlation matrix of X, respectively. The covariance matrix of X is defined analogously. Secondorder processes can very profitably be studied since, on one hand, the (equivalence class of) second-order RVs form a Hilbert space L2 (, T , P) and, on the other hand, for each second-order process with mean sequence µ(n) (= 0, without loss of generality) and covariance matrix C = [Cij ], we can find a process Y = {yn }, possibly defined on a different probability space, such that Y has the same mean sequence, the same covariance matrix C, and such that the finitedimensional distributions of Y are all Gaussian (see Ibragimov and Rozanov [9]). Hence,
6516
PROCESSES, DISCRETE
if one has a second-order process to work with, one can assume without much loss of generality that it is a Gaussian process as well. Gaussian processes arise most often in statistical applications. Some of the other useful discrete processes are processes with uncorrelated RVs, processes with orthogonal RVs, processes with independent increments, Markov chains, stationary processes, martingales∗ , and mixing processes. We now present some fairly general examples of discrete processes that arise in statistics and in other applications. Defining a discrete process in a recursive fashion is quite common in statistical estimation theory and learning theory (in psychology). 1. Let us begin with a simple case of Tmaze learning in psychology. On each trial a rat is placed at the bottom of a T-maze. The rat makes a choice, at the fork of the T-maze, and proceeds to the end of the left or right arm (response e0 or e1 ). There the rat may or may not find food. Finding food will reinforce the response just made. Consider now a state space X and a response space E. Corresponding to each response e ∈ E, there is a contraction or distance diminishing function fe : X → X . If the experiment is in state xn at trial n, a response en is determined and xn+1 is given by xn+1 = fen (xn ). The process xn , n 1, is a discrete process roughly describing the learning; for details, see Norman [14] and Cypkin [6]. 2. Some statistical estimates are discrete processes defined as functions of other discrete processes. Let θ be a Gaussian signal N(m, σ 2 ). What one observes is yk = θ + ξk , k 1, where ξk , k 1, are additive noise consisting of independent RVs with a common density f (x). Then the best estimate xn (in the sense of standard error) is xn = E(θ |y1 , . . . , yn ). In principle the estimate xn can be computed by Bayes’ formula. This method is too tedious and does not render itself to practical applications and the study of the solution. For computational purposes one can follow the recursive procedure. For example, yk+1 = ξk+1 + A(y1 , . . . , yk )θ + B(y1 , . . . , yk ). Here A and B are the so-called ‘‘controlling’’ functions. Using such a recursive procedure and appealing to the theorem on normal correlation (stated below) one can obtain the estimates xn , the tracking errors
ek = E(θ − xk )2 , and the conditional variance σk2 = E[(θ − xk )2 |y1 , . . . , yk ]. (Further details are given below.) 3. Finally, we give a random walk∗ model of a collision diffusion process∗ that arises very commonly in the study of particle transport in dilute gas. In particular, the discrete process we define below approximates the random motion of two elastic particles of identical mass which move in such a way that they exchange velocities when they collide. Let P1 and P2 be two particles moving on the same one dimensional lattice such that (a) initially P1 is an even number of units of distance to the left of P2 ; (b) each particle moves independently of each other with steps +1 or −1; (c) if at any time n the two particles are not at the same place, then at time n + 1 each moves +1 or −1 step with probability 12 ; and (d) if at any time n both are at the same point x0 , then at time n + 1 they move according to P(both at x0 − 1) = 14 , P(both at x0 + 1) = 14 , P(P1 at x0 − 1, P2 at x0 + 1) = 12 . If xn and yn , n 0, are two independent simple random walks with probability 12 for steps of +1 or −1 unit, and x(0) − y(0) is a negative even integer, let ξn = min(xn , yn ) and ηn = max(xn , yn ). Then the random walk with collision is given by the discrete process Xn = (ξn , ηn ). As remarked earlier, an important class of results of a discrete process {Xn } is the asymptotic behavior as n → ∞. The sequence Xn can converge to a limit X in several senses: convergence∗ in distribution, in probability, in mean, in mean square, almost sure convergence, and so on. The notions lead to different classification of estimates. For example, let ξ1 , . . . , ξn be independent RVs following the same distribution whose density f (y, x) depends on a parameter x belonging to a class X . Let Xn , n > 1, be a process obtained in the procedure of estimating x. Let Px and Ex be the corresponding probability measure and expectation respectively. An estimate Xn is called asymptotically unbiased (respectively, strongly consistent, consistent, asymptotically normal, strongly asymptotically efficient, or asymptotically efficient) if (as n → ∞), Xn → x in mean (respectively, almost surely, √ n(Xn − x) → N(0, σ 2 (x)) in in probability,
PROCESSES, DISCRETE
law, nEx [Xn − x]2 → I−1 (x), or σ 2 (x) = I−1 (x), where I(x) is the Fisher information∗ ). This classification list clearly indicates the importance of these notions of convergence and the corresponding limit theorems∗ . (See the annotations in the references at the end of this article.) Finally, for completeness, we present some statistical results connected with discrete processes that arise as estimates. We will be sketchy for the simple reason that more general results appear elsewhere. Having found the estimate Xn , let Xn+1 be determined recursively by Xn+1 = g(Xn , ξn+1 ), where ξn+1 is the new observation. The first result is basic in the theory of recursive estimation. (i) Let {ξn , n 1} be independent observations from a population with a common density f (ξ , x0 ); x0 , ξ ∈ R1 . Let m(x) and D(x) be the common mean and variance with x0 = x. If D(x0 ) < ∞ and m(x) is strictly increasing such that |m(x)| increases at most linearly as |x| → ∞, then the recursive estimates defined successively by Xn+1 − Xn = an (ξn+1 − m(xn )), X0 ≡ constant, with an = ∞, a2n < ∞, give a strongly consistent estimate of x0 . Assume furthermore that m (x0 ) > 0 and 2am (x0 ) > 1. Then the recursive procedure
of recursive estimates or filtering (basic filtering equations are by-products of this theorem); see Liptser and Shiryayev [12]. If (θ , X) = ([θ1 , . . . , θk ], [X1 , . . . , Xm ]) is a Gaussian vector with mθ = Eθ , mX = EX, Dθ = cov(θ , θ ), Dθ X = cov(θ , X), DX = cov(X, X), and if D+ X denotes the pseudo-inverse matrix of DX , then the conditional expectation E[θ |X] and the conditional covariance cov(θ , θ |X) are given by E[θ |X] = mθ + Dθ X D+ X (X − mX ), ∗ cov(θ , θ |X) = Dθ − Dθ X D+ X Dθ X ,
respectively. In particular, let the RVs θ , X1 , . . . , Xm together form a Gaussian vector such that X1 , . . . , Xm are independent and var(Xk ) > 0. Then E[θ |X1 , . . . , Xm ] = Eθ +
m cov(θ , Xi ) i=1
var(Xi )
An (y) = A(y1 , . . . , yn ),
Xn+1 − Xn = (ξn+1 − m(Xn ))/(n + 1)
Bn (y) = B(y1 , . . . , yn ),
dX(t) =
− m (x0 ) X(t)dt + D(x0 )dβ(t),
1 2
where β(t) is a standard Brownian motion∗ . (ii) The following theorem on normal correlation is fundamental in the study
(Xi − EXi ).
In example 2 presented above, let the additive noises ξn [occurring with the signal θ ∈ N(m, σ 2 )] be independent unit normal RVs. Then it follows from (ii) that xn = (m + ni=1 yi )/(1 + σ 2 n) and the tracking error en = σ 2 /(1 + σ 2 n). To consider the case of yn defined in terms of the controlling functions A and B, put
X0 = constant,
is asymptotically normal, and for any k > 0, ti > 0, 1 i k, and n < n1 < · · · < nk such that log[ni /n]√→ ti as n → ∞, the joint dis√ n(Xn − x0 ), n1 (Xn1 − x0 ), . . . , tribution of √ nk (Xnk − x0 ) converges to the finite-dimensional distribution of X(0), X(t1 ), . . . , X(tk ) corresponding to a diffusion process∗ X(t) satisfying the Ito equation
6517
xn = xn+1 − xn ,
x0 = m,
σn = σn+1 − σn ,
σ0 = σ 2 ,
D = 1/(1 + σn A2n (y)). Applying the theorem on normal correlation, we get σn = −DA2n (y)σn2 , xn = Dσn An (y)(yn+1 − Bn (y) − An (y)xn ). For further details, see Liptser and Shiryayev [12, Chap. 13]. (iii) The following result also plays a central role in obtaining a maximum likelihood estimate∗ . Let Y(t), t = 0, ±1, . . ., be a
6518
PROCESSES, DISCRETE
wide-sense stationary process with spectral representation Y(t) =
π −π
where k(t + 1) = k(t) +
eiut [Pn−1 (eiu )/Qn (Eiu )]dZ(u),
where dZ(u) is an orthogonal random 2 measure with = 0, E|dZ| ), n−1E(dZ) n = du/(2π k Pn−1 (s) = k=0 bk s , Qn (s) = k=1 ak sk , an = 1, ak , bk ∈ R1 , and with the roots of Qn (s) = 0 in the unit circle. Then Y(t) = Y1 (t), where (Y1 (t), . . . , Yn (t)) is an n-dimensional widesense stationary process satisfying the recurrent equations,
9 2 Also, EY12 (t) = 12 5 , EY1 (t)Y2 (t) = − 15 , EY2 (t) 7 12 2 = 5 , E[X(t) − θ ] = 5 , and E[(X(t) − θ )Y2 (t)] 9 . Now the theorem on normal correla= − 15 tion yields
mθ (0) = (θ − X(0))/4,
t−1 i=0
j = 1, . . . , n − 1; Yn (t + 1) = −
n−1
+ aj Yj+1 (t) + Bn V(t + 1),
j=0
with V(t) = E[Yi (γ )V(t)] = 0,
γ < t,
π
Bj = bn−1 −
3 + 4k(i) X(0) − 2(1 + k(i))
t−1 t−1
−
i=0 j=i+1
3 + 4k(j) 2(1 + k(j))
X(i) 1 + 2k(i) + X(i + 1) , 2 2(1 + k(i)) t−1 3 + 4k(i) − g(t) = 4−1 2(k + k(i)) ×
eiu(t−1) Z(du),
−π
i = 1, . . . , n;
i=0
B1 = bn−1 , j−1
k(0) = 54 .
Hence mθ (t) = f (t, x) + g(t)θ , where f (t, x) = 4−1
Yj (t + 1) = Yj+1 (t) + Bj V(t + 1),
1 (1 + 2k(t))2 − . 4 4(1 + k(t))
−
t−1 t−1 i=0 j=i+1
Bi an−j+1 ,
j=1
j = 2, . . . , n.
−
3 + 4k(j) . 2(1 + k(i))
For t 1, set {1 + g(i + 1)}2 5 + . 12 1 + k(i − 1) t
As an application, consider X(t) = θ + Y(t), where −∞ < θ < ∞ is an unknown parameter and Y(t) is stationary Gaussian with EY(t) = 0 and spectral density (or power spectrum) f (u) = (eiu + 1)/(e2iu + eiu + 12 ). By the result above, Y(t) = Y1 (t), where
Dt =
i=1
The maximum likelihood estimate θˆ (t) is obtained by maximizing the right side of the Radon-Nikodym derivative dµθX
Y1 (t + 1) = Y2 (t) + V(t + 1), Y2 (t + 1) = − 21 Y1 (t) − Y2 (t) + 12 V(t + 1).
dµ0X
= exp
+
If mθ (t) = E[Y2 (t)|At (X)], then mθ (t) solves mθ (t + 1) =
5X(0)θ 25θ 2 − 12 248 t
[F(i, x)θ − G(i)θ ] , 2
i=1
where
θ − X(t) θ m (t) 2 1 + 2k(t) + 2(1 + k(t))
F(i, x) =
×(X(t + 1) − θ − m (t)), θ
G(i) =
{1 + g(i − 1)}{X(i) − f (i − 1, x)} , 1 + k(i − 1) {1 + g(i − 1)}2 . 2{1 + k(i − 1)}
PROCRUSTES TRANSFORMATION
Thus θˆ (t) = D−1 t
t 5X(0) F(i, x) . + 12 i=1
The books by Cox and Miller [5], Feller [8], Kannan [10], and Karlin and Taylor [11] are of introductory nature and include basic properties and various applications of different types of discrete processes. The first part of Chung [4] is devoted to the theory of discrete Markov chains. For an extensive treatment of theory (and potential theory) of random walks consult Spitzer [16], and for applications, see Barber and Ninham [2]. Even though published in 1953, Doob [7] is a very good advanced level treatise on various types of stochastic processes. Lo`eve [13] and Petrov [15] extensively study various types of limit theorems that are particularly useful in obtaining different types of estimates. Convergence in law is the topic to which Billingsley [3] is devoted; here one can find applications to exchangeable RVs. For detailed statistical study of discrete processes, consult Albert and Gardner [1], Ibragimov and Rozanov [9], and Liptser and Shiryayev [12].
6519
9. Ibragimov, I. A. and Rozanov, Y. A. (1978). Gaussian Random Processes. SpringerVerlag, New York. 10. Kannan, D. (1979). An Introduction to Stochastic Processes. North-Holland, New York. 11. Karlin, S. and Taylor, H. M. (1975). A First Course in Stochastic Processes, 2nd ed. Academic Press, New York. 12. Liptser, R. S. and Shiryayev, A. N. (1977/1978). Statistics of Random Processes, Vols. I and II. Springer-Verlag, New York. 13. Lo`eve, M. (1977/1978). Probability Theory, Vols. I and II, 4th ed. Springer-Verlag, New York. 14. Norman, F. (1972). Markov Process and Learning Model. Academic Press, New York. 15. Petrov, V. V. (1975). Sums of Independent Random Variables. Springer-Verlag, New York. 16. Spitzer, F. (1964). Principles of Random Walk. Van Nostrand, Reinhold, New York. See also DIFFUSION PROCESSES; GAUSSIAN PROCESSES; MARKOV PROCESSES; RANDOM WALKS; and STOCHASTIC PROCESSES.
D. KANNAN
PROCRUSTES TRANSFORMATION REFERENCES 1. Albert, A. E. and Gardner, L. A. (1967). Stochastic Approximation and Nonlinear Regression. MIT Press, Cambridge, Mass. 2. Barber, M. N. and Ninham, B. W. (1970). Random and Restricted Walks. Gordon and Breach, New York. 3. Billingsley, P. (1968). Convergence of Probability Measures. Wiley, New York. 4. Chung, K. L. (1967). Markov Chains with Stationary Transition Probabilities, 2nd ed. Springer-Verlag, New York. 5. Cox, D. R. and Miller, H. D. (1970). The Theory of Stochastic Processes. Methuen, London. 6. Cypkin, Ja. Z. (1973). Foundations of the Theory of Learning Systems. Academic Press, New York. 7. Doob, J. L. (1953). Stochastic Processes. Wiley, New York. 8. Feller, W. (1971). Introduction to Probability Theory and Its Applications, Vol. I, 3rd ed. Wiley, New York (Vol. II, 2nd ed., 1966).
A procedure used in factor analysis∗ for transforming a matrix to obtain a matrix approximately of prespecified form, in particular to transform factors so that the factor loading analysis approximates to some desired form (a ‘‘target’’ matrix). Let F denote the r × r factor loading matrix obtained from data and H denote a target matrix, also of order r × r. We seek an s × s ˜ = FS ‘‘best’’ matches H. matrix S such that H ‘‘Best’’ is conventionally defined as meaning ‘‘minimizes the sum of squares of differences between corresponding elements of H and ˜ The required S is H.’’ S = (FF )−1 F H. It represents a regression of each column of H on the corresponding column of F. Cliff [2] restricts transforming matrices (S) to be orthogonal∗ . References 1 and 3 to 6 give further details on construction and generalization; see also ref. 7.
6520
PRODUCER PRICE INDEXES
REFERENCES 1. Browne, M. W. (1967). Psychometrika, 32, 125–132. 2. Cliff, N. (1966). Psychometrika, 31, 33–42. 3. Gower, J. C. (1975). Psychometrika, 40, 33–51. 4. Hurley, J. L. and Cattell, R. B. (1962). Behav. Sci., 7, 258–262. 5. Kristof, W. and Wingersky, B. (1971). Proc. 79th Annu. Conv. Amer. Psychol. Ass., pp. 81–90. 6. Sch¨onemann, P. H. (1966). Psychometrika, 31, 1–10. 7. Sch¨onemann, P. H. and Carroll, R. H. (1970). Psychometrika, 35, 245–256. See also FACTOR ANALYSIS.
PRODUCER PRICE INDEXES The Producer Price Indexes (PPI) are calculated and released by the Bureau of Labor Statistics∗ (BLS). They began publication in 1902 as the Wholesale Price Indexes. They measure the change in prices received by producers for their goods and services. In 1980, BLS published the first installment of these indexes based on a comprehensively revised basis. The January 1986 indexes reflect the final major installment of that revision. The following discussion focuses on the indexes from the revised survey, but the old indexes will be referenced as appropriate (see also INDEX NUMBERS). UNIVERSE AND CLASSIFICATION Historically, these indexes have measured price changes for selected products of agriculture, manufacturing, and mining. The revision systematically measures the changes in prices received for all products sold by establishments classified in mining or manufacturing according to the Standard Industrial Classification (SIC). The basic building block of the survey is the four-digit industry. Generally following the Census of Manufactures product codes, products of each industry are coded to a seven- or nine-digit level. (The old indexes use an eight-digit product coding scheme that is unique to the PPI.)
There are three families of indexes produced from the same basic data base. The first are the output price indexes for each four-digit industry (493 in mining and manufacturing). These indexes reflect the price change of the output which leaves the particular industry. Sales among establishments within the industry are not included. This so-called net-output approach can also be applied to aggregations of these four-digit industries. The stage-of-processing price indexes are the most useful summary-level indexes for macroeconomic analysis of inflation. In the old indexes they are calculated for crude, intermediate, and finished goods with appropriate subindexes for specific product areas (food, energy) or end markets. Each product is classified by its degree of fabrication and type of buyers and may be included in more than one stage-of-processing index. The new stage-of-processing indexes will be constructed from aggregates of industries, each industry being classified in total based on the pattern of its purchases and sales as a producer of crude materials, primary manufactures, intermediate manufactures, or finished goods. The third set consists of price indexes for detailed products. In 1986, there are about 6,000 such detailed indexes published each month. THE INDEXES Producer Price Indexes are calculated using a modified Laspeyres index of the general form P1,i Q0,i × 100 I1 = i i P0,i Q0,i where I1 = index for period 1 P1,i , P0,i = prices for item i in periods 1 and 0 respectively Q0,i = quantity of item i sold in period 0 (the base period) These indexes may be viewed as measuring the price change of a constant set of production through time.
PRODUCER PRICE INDEXES
PRICES In order to price the output of a production unit, BLS requests the price which the producer actually received at the time of shipment for the item being priced—the transaction price. The desired price will reflect all applicable discounts and all applicable extras and surcharges. The price is f.o.b. the seller’s freight dock and excludes all direct excise taxes and transportation charges. They seek to price all the different types of transactions and types of customers relevant to a particular item. All the conditions of the sale must be identified and held constant through time. When an item is no longer being sold, it is necessary to replace it in the index. This must be done in such a way that only pure price change is captured by the index, and it remains as close as possible to the concept of pricing a constant set of production through time. This substitution procedure is generally referred to as ‘‘quality adjustment.’’ It consists of two steps: (1) identifying all the changes in the specification of the item being priced, and (2) placing a value on each change. It is the second step which presents the major difficulty. Traditionally, it has been the objective to value specification changes as the sum of the producer’s variable costs, fixed costs, and standard markup associated with the item. Increasingly, there are cases in which the producer cost methodology cannot be applied (e.g., when there is product improvement at reduced cost). Computers, electronic calculators, and other high-technology goods are the prime examples. A technique called hedonic∗ regression, which has been a research tool for some time, is being investigated for some of these cases. This approach views a particular good as composed of a large number of independent characteristics which are bought in bundles. For example, a computer is composed of certain cycle times, add times, memory size, and so on. A regression equation (in any number of possible functional forms) is estimated with the price of the computer as the dependent variable and the characteristics as the independent variables. The coefficients (or transformations of them) are
6521
then implicit prices for the characteristics. The coefficient for memory size may set an implicit value of $100 for each ‘‘K’’ of memory. A new computer with 4K more memory than an old one would thus have a quality increase of $400. Although its application has been somewhat limited, the hedonic approach has yielded encouraging results. Three major limitations are associated with its application. First, all price-determining characteristics must be specified, which makes it a costly, time-consuming process. Second, it is of no assistance when an entirely new characteristic appears (e.g., a smog-control device on a car). Third, it is necessary to assume that the value of a characteristic is independent of the bundle in which it occurs. This assumption is believed to be approximately true for most cases, but requires evaluation.
USES Producer Price Indexes are widely used by the general public as indicators of inflation. They are, in conjunction with the Consumer Price Index∗ , the unemployment rate, and the gross national product∗ , the chief statistical inputs to national fiscal and monetary policy. They are also heavily relied on by the business and research communities for economic and market analyses and projections. Most major measures of real economic activity (GNP, orders, shipments, inventories, productivity) are derived by using appropriate Consumer and Producer Price Indexes to deflate current dollar aggregates. Finally, increasing numbers of contract sales are using price escalation based on one or more Producer Price Indexes. In 1976, more than $100 billion in contracts had such escalators.
SAMPLING The universe of interest is the set of all transactions originating in mining and manufacturing establishments. Each transaction is characterized by the industry of the seller and the product sold. It can be portrayed as follows:
6522
PRODUCT INTEGRATION
Industry/Product 1 2 3 . . . 10,000 1 2 .. . 493 Each four-digit SIC industry (row in the table above) is sampled and estimated independently. A multistage probabilityproportional-to-size∗ sample of transactions is selected for pricing through time. In the first stage, an average of 74 firms in the industry are selected from the unemployment insurance file of all employers in the country. These firms are visited by BLS economists, who use a multistage sampling technique called disaggregation to select an average of five specific items for pricing along with a particular type of transaction if any are sold in more than one way at different prices. Output price indexes can be constructed for each industry with data from the appropriate row in the matrix above. Product indexes (wherever made) can be constructed from data in the appropriate column. Weights of major product groups and for industries are from the Census of Manufacturers. Within major product groups, weights are developed from the sampling process. Net output weights are developed from the input–output tables of the Bureau of Economic Analysis. ERRORS Because the old indexes were not based on probability samples, variances have not been available. Variances are now under development for the revised indexes. A total error profile will also be developed for the program covering such factors of errors in the sampling frame∗ , nonresponse∗ bias, and processing errors. Rigorous error control procedures are being installed at all critical stages in the process. The difficulties associated with quality adjustment may contribute a major portion of the total error in price indexes. This kind of error is unique to estimating price indexes. As a result, the existing literature on errors in surveys is largely not applicable. This topic has received considerable attention from both BLS and academic researchers, but a generally acceptable solution has been elusive.
BIBLIOGRAPHY Archibald, R. B. (1977). Ann. Econ. Social Meas., Winter, 57–72. [Extends Fisher and Schell (below). The direct theoretical basis for the PPI.] Bureau of Labor Statistics (1982). BLS Handbook of Methods, Bulletin 2134, Vol. 1. Bureau of Labor Statistics, Washington, D.C., pp. 43–61. (Official, general description of PPI. Updated editions issued periodically.) Early, J. F. (1978). Monthly Labor Rev., Apr., 7–15. (General audience description of the theory and methods for the revision.) Early, J. F. and Sinclair, J. (1981). In The U.S. National Income and Product Accounts: Selected Topics, M. Foss, ed. Chicago. (Extensive treatment of actual quality adjustment in the PPI. Illustrative example of hedonic approach.) Fisher, F. M. and Schell, K. (1972). The Economic Theory of Price Indexes. New York. (Rigorous mathematical economic theory underlying modern price indexes.) Triplett, J. E. (1971). The Theory of Hedonic Quality Measurement and Its Use in Price Indexes. Bureau of Labor Statistics, Washington, D.C. (A little dated, but the best introduction to the topic.) Triplett, J. E. (1975). In P. H. Earl, Analysis of Inflation. Lexington Books, Lexington, Mass. (Basic, moderately technical discussion of the problems associated with quality adjustment.) See also CONSUMER PRICE INDEX; GROSS NATIONAL PRODUCT DEFLATOR; HEDONIC INDEX NUMBERS; INDEX NUMBERS; INDEX OF INDUSTRIAL PRODUCTION; LABOR STATISTICS; and PAASCHE –LASPEYRES INDEX.
JOHN F. EARLY
PRODUCT INTEGRATION All statisticians arefamiliar with the sum and product signs and , and with the integral sign . Also they are aware that there is a certain analogy between summation and integration; indeed the integral sign is in origin a stretched-out s for ‘‘summation’’. Strange therefore that not many people are aware of the existence of the product integral , invented by the Italian mathematician Vito Volterra in 1887, which bears exactly the same relation to the ordinary product as the integral does to summation.
π
PRODUCT INTEGRATION
The mathematical theory of productintegration is not terribly difficult and not terribly deep, which is perhaps one of the reasons it was out of fashion by the time survival analysis∗ came into being in the 1950s. However, it is terribly useful, and it is a pity that E. L. Kaplan and P. Meier [10], the inventors (1956) of the product-limit or Kaplan–Meier estimator∗ (the nonparametric maximum likelihood estimator∗ of an unknown distribution function based on a sample of censored survival times), did not make the connection, and neither did the authors of the classic 1967 and 1974 papers on this estimator, respectively by B. Efron [5] and N. Breslow and J. Crowley [3]. Only with the 1978 paper of O. O. Aalen and S. Johansen [1] was the connection between the Kaplan–Meier estimator and product integration made explicit. It took several more years before the connection was put to use to derive new largesample properties of the Kaplan–Meier estimator (e.g., the asymptotic normality of the Kaplan–Meier mean) with the 1983 paper of R. D. Gill [8]. Modern treatments of the theory of product integration with a view toward statistical applications can be found in refs. 2, 6, 9. The Kaplan–Meier estimator is the product-integral of the Nelson–Aalen estimator∗ (see COUNTING PROCESSES) of the cumulative or integrated hazard function∗ ; these two estimators bear the same relation to one another as the actual survival function∗ (one minus the distribution function) and the actual cumulative hazard function. There are many other applications of product integration in statistics, for instance in the study of multistate processes (connected with the theory of Markov processes∗ ) [1] and in the theory of partial likelihood∗ (see PROPORTIONAL HAZARDS MODEL, COX’S); see ref. [2]. Product integration also turns up in extremevalue∗ theory, where again the hazard rate plays an important role, and in stochastic analysis and martingale∗ theory (stochastic integration∗ ), where it turns up under the name Dol´eans–Dades exponential martingale. Properties of integrals are often easily guessed by thinking of them as sums of many, many (usually very small) terms. Similarly,
6523
product integrals generalize the taking of products, making their properties easy to guess and to understand. Let us define product integration at a slightly higher level of generality than Volterra’s original definition (corresponding to the transition from Lebesgue to Lebesgue–Stieltjes integration). Suppose X (t) is a p × p matrix-valued function of time t ∈ [0, ∞). Suppose also X (or each component of X) is right-continuous with left-hand limits. Let 1 denote the identity matrix. The product integral of X over the interval (0, t] is now defined as t
π [1 + dX(s)] 0
=
lim
max |ti −ti−1 |→0
{1 + [X(ti ) − X(ti−1 )]},
where the limit is taken over a sequence of ever finer partitions 0 = t0 < t1 < . . . < tk = t of the time interval [0, t]. For the limit to exist, X has to be of bounded variation; equivalently, each component of X is the difference of two increasing functions. A very obvious property of product integration is multiplicativity. Defining the product integral over an arbitrary time interval in the natural way, we have for 0 < s < t t
s
t
0
0
s
π (1 + dX) = π (1 + dX) π (1 + dX). We can guess (for proofs, see ref. [6] or preferably [9]) many other useful properties by looking at various simple identities for finite products. For instance, in deriving asymptotic statistical theory it is often important to study the difference between two productintegrals. If a1 , . . . , ak and b1 , . . . , bk are two sequences of numbers, we have the identity (1 + ai ) − (1 + bi ) (1 + ai )(aj − bj ) (1 + bi ). =
j
i<j
i>j
This can be easily proved by replacing the middle factor on the right-hand side of the equation, (aj − bj ), by (1 + aj ) − (1 + bj ).
6524
PRODUCT INTEGRATION
Expanding about this difference, the righthand side becomes (1 + ai ) (1 + bi ) j
ij
−
i>j
(1 + ai )
ij−1
(1 + bi ) .
i>j−1
This is a telescoping sum; writing out the terms one by one, the whole expression collapses to the two outside products, giving the left-hand side of the identity. The same manipulations work for matrices. It is therefore no surprise, replacing sums by integrals and products by product-integrals, that t
t
π (1 + dX) − π (1 + dY) 0
=
0
t
hazard function A corresponding to S with the Nelson–Aalen estimator ˆ = A(t)
dj tj t
rj
.
This is a discrete cumulative hazard function, corresponding to the discrete estimated ˆ = 0 for t not an hazard α(t ˆ j ) = dj /rj , with α(t) observed death time. The product integral of ˆ is then A dj ˆ = ˆS(t) = t (1 − dA) 1− , 0 rj t t j
which is just the Kaplan–Meier estimator of the true survival function S. The Duhamel equation now becomes the identity t S(t) ˆ ˆ ˆ − S(t) = , S(s−)[d A(s) − dA(s)] S(t) S(s) s=0
s−
π (1 + dX)[dX(s) − dY(s)]
s=0 0
t
×
π (1 + dY). s+
This valuable identity is the Duhamel equation (the name refers to a classical identity for the derivative with respect to a parameter of the solution of a differential equation). As an example, consider the scalar case (p = 1); let A be a cumulative hazard function ˆ the Nelson–Aalen estimator based on and A a sample of censored survival times. Here we have the statistical problem of estimating the survival curve S(t) = Pr{T t} given a sample of independently censored i.i.d. survival times T1 , . . . , Tn . The cumulative hazard rate A(t) is defined by
t
A(t) = 0
dS(t) ; S(t−)
which can be exploited to get both smallsample and asymptotic results [2,6,7,8,9]. The same identity pays off in studying Dabrowska’s multivariate product-limit estimator [4,9,11], and in studying Aalen and Johansen’s estimator of the transition matrix of an inhomogeneous Markov chain [1,2]. It can be rewritten [9] as a so-called van der Laan identity [11], expressing Sˆ − S as a function-indexed empirical process (see EMPIRICAL PROCESSES), evaluated at a random argument, so that the classical largesample results for Kaplan–Meier (consistency and asymptotic normality∗ ) follow from a two-line proof: without further calculations, simply invoke the modern forms of the Glivenko–Cantelli theorem∗ and the Donsker theorem, i.e., the functional versions of the classical law of large numbers∗ and the central limit theorem∗ , respectively. Taking Y identically equal to zero in the Duhamel equation yields the formula t
A is just the integrated hazard rate in the absolutely continuous case, the cumulative sum of discrete hazards in the discrete case. Let t1 < t2 < . . . denote the distinct times when deaths are observed; let rj denote the number of individuals at risk just before time tj , and let dj denote the number of observed deaths at time tj . We estimate the cumulative
π (1 + dX) − 1 =
0
t
s−
π (1 + dX)dX(s).
s=0 0
This is the integral version of Kolmogorov’s forward differential equation from the theory of Markov processes, of the type
t
Y(t) = 1 +
Y(s−)dX(s) 0
PRODUCTION SCHEDULING
(in unknown Y, given X), that motivated Volterra to invent product integration. Y(t) = t
π (1 + dX) 0
is the unique solution of this
equation.
7.
REFERENCES 1. Aalen, O.O. and Johansen, S. (1978). An empirical transition matrix for nonhomogenous Markov chains based on censored observations. Scand. J. Statist., 5, 141–150. (Introduced counting-process theory and product integration, simultaneously, into the study of nonparametric estimation for Markov processes; the relevance to the Kaplan–Meier estimator was noted by the authors, but not noticed by the world.) 2. Andersen, P. K., Borgan, Ø., Gill, R. D., and Keiding, N. (1993). Statistical Models Based on Counting Processes. Springer-Verlag, New York. (Contains a ‘‘users’ guide’’ to product integration in the context of counting processes and generalized survival analysis.)
8.
9.
10.
3. Breslow, N. and Crowley, J. (1974). A large sample study of the life table and product limit estimates under random censorship. Ann. Statist., 2, 437–453. (First rigorous largesample results for Kaplan–Meier estimation, using the then recently developed Billingsleystyle theory of weak convergence.) 4. Dabrowska, D. (1988). Kaplan–Meier estimate on the plane. Ann. Statist., 16, 1475–1489. (Beautiful generalization of the product-limit characterization of the Kaplan–Meier estimator to higher dimensions. Other characterizations, e.g., nonparametric maximum likelihood, lead to other estimators; see refs. 9, 11.) 5. Efron, B. (1967). The two sample problem with censored data. Proc. 5th Berkeley Symp. Math. Statist. Probab., L. Le Cam and J. Neyman, eds. University of California Press, Berkeley, Calif., pp. 831–853. (This classic introduced the redistribute-to-the-right and selfconsistency properties of the Kaplan–Meier estimator and claimed, but did not prove, weak convergence of this estimator on the whole line in order to establish asymptotic normality of a new Wilcoxon generalization—results established in [7].) 6. Gill, R. D. and Johansen, S. (1990). A survey of product-integration with a view towards application in survival analysis. Ann. Statist.,
11.
6525
18, 1501–1555. (The basic theory, some history, and miscellaneous applications. Reference [9] contains some improvements and further applications.) Gill, R. D. (1980). Censoring and Stochastic Integrals, MC Tract 124. Centre for Mathematics and Computer Science (CWI), Amsterdam. (The author emphasized the counting process approach to survival analysis, using some product limit theory from [1] but not highlighting this part of the theory.) Gill, R. D. (1983). Large sample behaviour of the product limit estimator on the whole line. Ann. Statist., 11, 49–58. (Cf. comments on [7].) Gill, R. D. (1994). Lectures on survival analysis. In Lectures on Probability Theory ´ e de Probabilit´es de Saint Flour (Ecole d’Et´ XXII—1992), SLNM 1581, D. Bakry, R. D. Gill, and S. A. Molchanov, eds. SpringerVerlag, Berlin, pp. 115–241. (Perhaps cryptically brief in parts, but a yet more polished treatment of product integration and its applications in survival analysis.) Kaplan, E. L. and Meier, P. (1958). Nonparametric estimation from incomplete observations. J. Amer. Statist. Ass., 53, 457–481, 562–563. [This classic is second on a list of most cited ever papers in mathematics, statistics, and computer science. The authors never met, but submitted their independent inventions of the product-limit estimator simultaneously to J. Amer. Statist. Ass.; the resulting joint paper was the product of postal collaboration.] van der Laan (1996). Efficient and Inefficient Estimation in Semiparametric Models, CWI Tract 114. Centre for Mathematics and Computer Science, Amsterdam. (Contains a beautiful identity for the nonparametric maximum likelihood estimator in a missing-data problem: estimator minus estimand equals the empirical process of the optimal influence curve evaluated atˆ the estimator, Fˆ − F = (1/n) ni=1 ICopt (Xi /F). Applications to the bivariate censored-data∗ problem are presented, as well as treatment of other estimators for the same problem.)
See also CENSORED DATA; COUNTING PROCESSES; EMPIRICAL PROCESSES; KAPLAN-MEIER ESTIMATOR —II; PROPORTIONAL HAZARDS MODEL, COX’S; and SURVIVAL ANALYSIS.
RICHARD GILL
PRODUCTION SCHEDULING. See LINEAR PROGRAMMING
6526
PRODUCTIVITY MEASUREMENT
PRODUCTIVITY MEASUREMENT Productivity measurement is the estimation of a broad class of ratios of outputs to one input or a combination of associated inputs in particular industries or the entire economy. The aggregate output Q is associated with n inputs (factors) vi , for i = 1, 2, . . . , n, of production often aggregated into labor (L) and capital (K). The ratios, such as Q/L and Q/K, are partial factor productivity indices. The ratio Q/ ni=1 αi vi , where the αi are weights, is referred to as the multifactor or total factor productivity (TFP) index. In comparisons over time, the partial index is a measure of the rate of change in total output per unit of a particular input, without separating out the contributions of other factors to the production process. It fails to account for the effects of technological advances, substitution between inputs, scale economies, and changes in input quality and product composition. In contrast, the TFP index is the rate of change in ‘‘productive efficiency,’’ or more accurately, in the ratio of total output to a weighted combination of all inputs. The value of the TFP index must be equal to 1 if all inputs, tangible or intangible, and the specification of their combination are correctly accounted for. If the TFP index exceeds unity, as it usually does in the case of industrial economies, it is because some inputs are left out or because the relationship between output and inputs (production function) is incompletely specified. For example, if Q is assumed to be produced by L and K only, and shows an increase of 4% between two points in time, it may turn out that 25% of this increase is explained by increases in the quantities of L and K, and the remaining 75% of the 4%, which is the value of the TFP index, is due to left-out inputs or misspecification. The identification of the misspecification, the left-out inputs, and the decomposition of the TFP index between them is a complex and as yet controversial problem in productivity measurement. The left-out variables have been interpreted to be among the sources and causes of economic growth and have frequently been identified with such factors as the various forms of technological progress, scale economies, research and development, and changes in the nature and quality of
inputs, institutions, and availability of relatively cheap inputs, especially energy. PARTIAL AND TOTAL FACTOR PRODUCTIVITY INDEXES All productivity measurements before 1942 were of the simple partial factor productivity type, in particular, output per worker, or per hour worked (Q / L). They continue to be published by the statistical agencies of many countries (e.g., ref. 40) and in spite of their evident theoretical limitations, widely used. The reasons are to be found in the general availability of the relevant statistical information and the simplicity of the computational requirements. Furthermore, in interindustry, intercountry, or intertemporal comparisons the average product of labor (output per worker) is an indirect measure of the differences in capital and other resources available for a unit of labor to work with. Thus the partial factor productivity ratios are indicators, however primitive, of differences in technological and economic states, even though the causes of these differences remain unrevealed by them. It is primarily for this latter reason that the main interest in recent years has been in the TFP index.1 In 1942, Tinbergen [37] was the first to estimate TFP indexes by introducing a trend function of the form A(t) = A(0)eλt into a loglinear regression of Q on L and K, where A(t)/A(0) was the TFP index in period t relative to period 0. After the mid-1950s several other investigators independently estimated TFP indexes. In particular, in two seminal contributions Solow [32] and Kendrick [18] provided the foundations of two interrelated but conceptually different approaches—one based on the economic concept of the production function and the other on the statistical concept of index numbers∗ . These are discussed in the following two sections, while the analysis and decomposition of the TFP index are discussed in the final section of this article. The literature on the subject is large. Recent developments in productivity measurements can be found inter alia in refs. 5, 20, 21, 25, and 27. Since 1958, the U.S. Bureau of Labor Statistics∗ has published five bibliographies on productivity; the
PRODUCTIVITY MEASUREMENT
latest [39] covers 1976–1978 and contains 1200 publications. Much of this literature is related to the economic theories of capital, production and returns to productive factors (see, e.g., refs. 2, 4, 13, 29, 31, 34, 35, 38, and 42).
6527
has increased over time (see e.g., refs. 6, 9, and 32). To account for this phenomenon, assume (as in ref. 32) that the production function shifts outward as function of time. Then the function can be written as Q(t) = F[L(t), K(t), t],
THE PRODUCTION FUNCTION APPROACH
(3)
In the neoclassical economic theory of production the entire output is explained by the interaction of homogeneous and constant quality productive factors (inputs) as specified by a constant returns to scale (i.e., first-order homogeneous and additive) production function. In a typical two-input case the latter can be written as
where t designates time and the amounts of the output as well as the constant quality inputs are functions of time. As shown in ref. 26, the production function can take various explicit forms. However, if F[·] is subject to constant returns to scale as in (1), the variable t can be factored out from F and (3) can be restated as
Q = f (K, L),
Q(t) = A(t)f [L(t), K(t)],
(1)
where Q, K, and L represent output, capital, and labor, respectively. If changes in the quantities of these variables are considered in time, the percentage rate of growth of the output, as derived from (1), can be written as ˙ ˙ K L˙ Q = EK + EL , Q K L
(2)
where the dotted variables represent time derivatives, and EK and EL are defined as ∂Q K ∂K f
and
∂Q L . ∂L f
Because of the assumption of constant returns to scale, EK and EL sum to 1 (Euler’s identity), and the percentage rate of growth of the output, as shown in (2), is the weighted (linear) combination of the percentage rates of growth of the inputs. Hence the growth rate of the output must lie between, or be equal to, the growth rates of the two inputs. The empirical evidence is, however, different. In the United States, and in the advanced industrialized countries in general, the growth of labor in the long run has been exceeded by the growth of capital and output, the last two having been approximately equal to each other. In terms of partial productivity indicators, Q/L has grown over time, while Q/K has remained approximately constant. Concomitantly, the capital-labor ratio
(4)
where A(t) is a shift factor which determines the cumulative outward shift in the production function itself. A(t) is neutral with respect to, and independent of, the amounts of L and K utilized in the production process. Hence it can be arbitrarily designated as a measure of technological change of a type which is neutral with respect to input combinations and not embodied in (or independent of) the qualities of the inputs themselves. Accordingly, it has been referred to as ‘‘neutral disembodied technological change.’’ Analogously to (2), the percentage rate of change of the output over time can be obtained from (4) as ˙ ˙ ˙ K L˙ A Q = + EK + EL Q A K L and ˙ ˙ ˙ K L˙ Q A = − EK + EL , A Q K L
(5)
˙ where A/A represents the percentage rate of growth of the cumulative shift factor, or the difference between the observed growth rate of the output and that part of its growth which is imputable to the rate of growth of the amounts of factors used in production. As ˙ a consequence, A/A can also be thought of as a residual.
6528
PRODUCTIVITY MEASUREMENT
The estimation of the shift factor is based on national income and input statistics requiring the restatement of (5) in value terms. Let the prices of Q, L, and K at time t be P, PL , and PK , respectively, and in line with neoclassical purely competitive production theory assume that the prices or returns to factors are determined by the value of their contributions to output (marginal products), so that PL = A(t)(∂f /∂L)P and PK = A(t)(∂f /∂K)P. Given the assumption of competition, PQ = PL L + PK K at any point in time. We obtain ˙ ˙ Q PL L˙ + PK K˙ A . = − L A Q P L + PK K
(6)
In his original 1957 article Solow [32] assumed that A(t) = A(0)eλt , where λ rep˙ resents [A(t)/A(t)]. In calculation he set A(0) = 1 and used the formula
A(t) A(t) = A(t − 1) 1 + A(t − 1) for t = 1, 2, . . . , T, where is the first difference operator. Based on U.S. time series for 1909–1949 and using a production function of the specific form Q(t) = L(t)α K(t)β A(t),
THE INDEX NUMBER APPROACH The TFP index can be defined as an index of output divided by an index of input, an approach implicit in Mills [24], Abramovitz [1], Fabricant [14], and Kendrick [18,19]. Kendrick’s TFP index can be defined as Q(T)/Q(0) A(T) = , A(0) α[L(T)/L(0)] + β[K(T)/K(0)]
(8)
where α and β are income shares of labor and capital. The difference between Solow’s TFP index and Kendrick’s is that the latter is not explicitly based on a formal production function. However, as pointed out by Domar [11], Kendrick’s index does imply a production function of the form Q(t) = A(t)[αL(t) + βK(t)]. Also [22], under certain economic assumptions and for ‘‘small’’ changes in the quantities of the input and output, Kendrick’s and Solow’s indices are equivalent, and (8) can be rewritten as a Paasche index of ∂Q/∂L and ∂Q/∂K. Diewert [10] has proven that under certain economic assumptions, different index number formulas (e.g., those of Laspeyres, Paasche, and Fisher) are consistent with (‘‘exact’’ for) different functional forms of (3). For example, if (3) takes the form Q(t) =
2 2
1/2 aij Li (t)Kj (t)
(9)
where α and β are parameters, he found that A(1949)/A(1909) = 1.81; that is, the annual rate of technical progress in the United States over that period was 1.5%. In fact, by setting A(0) = 1, A(T) can be identified as the TFP index. Thus if t is expressed in years and A(2) = 1.0404, total productivity increased by 4.04% or at an ˙ annual rate [A(t)/A(t)] of 2%. It is in this sense that the TFP index is sometimes referred to as a measure of ‘‘technical change,’’ one of several alternative descriptions in the literature (see, e.g., ref. 12). Also, as shown by refs. 15 and 36, we obtain Q(T) X(T) A(T) = , (7) A(0) Q(0) X(0)
where the constants aij = aji , then, as shown in [10], Fisher’s ideal∗ input index is exact for (9). The problem is of course that the functional form of (3) is unknown. Caves et al. [3] proposed the use of the T¨ornqvist index for the measurement of input, output, and TFP indexes. Maddala [23] compared empirically four of Diewert’s exact indices and concluded that the differences between the underlying four functional forms result in ‘‘negligible’’ differences between the corresponding TFP indices.
where X(T)/X(0) is the Divisia input index. See INDEX NUMBERS and DIVISIA INDICES.
It is the output growth not directly attributable to the factors of production
i=1 j=1
ANALYSIS AND DECOMPOSITION OF THE TFP INDEX
PRODUCTIVITY MEASUREMENT
which is measured by the TFP index. It has also been referred to as ‘‘residual.’’ A large part of the literature on productivity measurement is devoted to attempts to decompose this residual into indices of changes of variables which can account for it. Three approaches to the analysis and decomposition of the residual are described briefly below. Growth Accounting Some studies in this vein are Denison [6,7,8,9] and Jorgensen and Griliches [16]. Denison in his implicit framework substituted a set of ‘‘explanatory variables,’’ the traditionally defined inputs in the aggregate production, such as (1). He also dropped the assumption of constant returns to scale. His productivity index can be expressed as A(t) = Q(t)/f (xi ; yj ),
(10)
where xi = x1 , x2 , . . . , xn and yj = y1 , y2 , . . . , ym , are the explanatory variables. Their rates of change over time are Denison’s ‘‘sources of growth.’’ The xi are disaggregated and quality-adjusted labor, capital and land, the contributions of which can be estimated by their respective income shares. The yj are variables which measure scale effects and improvements in resource allocation. The final residual in the growth of output is Denison’s productivity index, which he attributes to ‘‘advances of knowledge.’’ The method in ref. 16 is similar to Denison’s except that the classification of the data and perception of the required quality adjustments are different and Divisia indices are used in the calculations. Capital Embodied Technological Change A different concept of technological change than the one presented in Solow’s 1957 model is found in refs. 21, 28, 33, and 34, among others. Best known among these is the model based on a production function of the form Q(t) = A(t)L (t)J α
1−α
(t),
(11)
where J(t) is the quality-weighted capital defined by J(t) =
n i=0
Ki (t)(1 + λ)i ,
(12)
6529
Ki (t) is the gross amount of capital produced in year i and in use in year t, and λ is the rate of improvement in the productivity of new machines over those produced in the previous year. In this model, the rate of increase in the TFP index is given by ˙ ˙ ˙ Q(t) L(t) A(t) = −α A(t) Q(t) L(t)
˙ K(t) ˙ + λ − λa , (13) −(1 − α) K(t)
where a is the average age of capital. Maximum likelihood estimation∗ of the parameters of this model is discussed in ref. 41. Best Practice Technique Salter [30] attributes productivity growth to best practice technique which yields minimum production costs. He decomposes the TFP index into advances in technical knowledge, scale economies, and factor substitution. Salter examines proportionate changes in labor and capital, so as to attribute them to both neutral and nonneutral technical progress as well as to substitution between labor and capital. The discussion of Salter’s measures and their application to U.K. and U.S. industries is also contained in ref. 30. NOTE 1. A survey of partial productivity comparisons between countries can be found in [17]. The weighted mean of the ratio of labor hours worked at two points of time is called the man-hour index. The weights are usually the hourly earnings. REFERENCES 1. Abramovitz, M. (1956). Amer. Econ. Rev., 46, 5–23. 2. Arrow, K. J. (1962). Rev. Econ. Stud., 29, 155–173. 3. Caves, D. W., Christensen, L. R., and Diewert, W. E. (1982). Econometrica, 50, 1343–1414. 4. Champernowne, D. G. (1961). In The Theory of Capital, F. A. Lutz, ed. Macmillan, London.
6530
PRODUCTIVITY MEASUREMENT
5. Cowing, T. G. and Stevenson, R. E. (1981). Productivity Measurement in Regulated Industries. Academic Press, New York. (A collection of useful survey and application papers.)
23. Maddala, G. S. (1979). In ref. 27, pp. 309– 317.
6. Denison, F. (1962). The Sources of Economic Growth in the United States and the Alternatives before Us. Committee for Economic Development, New York.
25. Nadiri, M. I. (1970). J. Econ. 1137–1177. (Survey; elementary.)
7. Denison, F. (1964). Amer. Econ. Rev., 54, 90–94. 8. Denison, F. (1967). Amer. Econ. Rev., 57, 325–332. 9. Denison, F. (1967). Why Growth Rates Differ. The Brookings Institution, Washington, D.C.
24. Mills, F. C. (1952). Productivity and Economic Progress. National Bureau of Economic Research, New York. (Elementary.) Lit.,
8,
26. Nadiri, M. I. (1982). In Handbook of Mathematical Economics, Vol. 2. K. J. Arrow and M. D. Intriligator, eds. North-Holland, Amsterdam, pp. 431–490. (Economic theory of production; mathematical.) 27. National Research Council (1979). Measurement and Interpretation of Productivity. National Academy of Sciences, Washington, D.C. (Elementary.)
10. Diewert, W. E. (1976). J. Econometrics, 4, 115–145. (Economic index numbers and their use in productivity measurement; intermediate.)
28. Nelson, R. R. (1964). Amer. Econ. Rev., 54, 575–606.
11. Domar, E. D. (1961). Econ. J., 71, 709–729. (Concepts and measurement of technical progress; elementary.)
30. Salter, W. E. G. (1960). Productivity and Technical Change. Cambridge University Press, Cambridge, England.
12. Domar, E. D. (1962). J. Pol. Econ., 70, 597–608. (Reviews Kendrick’s approach to productivity measurement; elementary.)
31. Samuelson, P. A. (1962). Rev. Econ. Stud., 29, 193–206.
13. Eckaus, R. S. and Lefeber, L. (1962). Amer. Econ. Rev., 44, 113–122. 14. Fabricant, S. (1959). Basic Facts on Productivity Change. National Bureau of Economic Research, New York. (Elementary.) 15. Hillinger, C. (1970). Econometrica, 38, 773–774. (Defines Divisia form of TFP index; intermediate.) 16. Jorgenson, D. and Grilliches, Z. (1967). Rev. Econ. Stud., 34, 249–284. (Concepts, methods and application.) 17. Kravis, I. B. (1976). Econ. J., 86, 1–44. (Survey; elementary.) 18. Kendrick, J. (1961). Productivity Trends in the United States. Princeton University Press, Princeton, N.J. 19. Kendrick, J. (1973). Postwar Productivity Trends in the United States. National Bureau of Economic Research, New York. 20. Kendrick, J. and Vaccara, B. N. eds. (1980). New Developments in Productivity Measurement and Analysis. University of Chicago Press, Chicago. (Collection of survey and application papers; elementary.)
29. Phelps, E. S. (1962). Quart. J. Econ., 76, 548–567.
32. Solow, R. (1957). Rev. Econ. Statist., 39, 312–320. (Reprinted with correction in Readings in Economic Statistics and Econometrics, A. Zellner, ed. Little, Brown, Boston, 1968.) 33. Solow, R. (1959). In Mathematical Methods in the Social Sciences, K. Arrow et al., eds. Stanford University Press, Stanford, Calif., pp. 89–104. 34. Solow, R. (1962). Amer. Econ. Rev., 52, 76–86. 35. Solow, R. (1963). Capital Theory and the Rate of Return. North-Holland, Amsterdam. 36. Star, S. and Hall, R. E. (1976). Econometrica, 44, 257–263. (Divisia indices of productivity; intermediate.) 37. Tinbergen, J. (1942). In L. H. Klassen, et al. Jan Tinbergen: Selected Papers, 1959, pp. 182–221; first appeared in 1942 in German, published by North-Holland, Amsterdam. 38. Tobin, J. (1961). Amer. Econ. Rev., 52, 26–37. 39. U.S. Bureau of Labor Statistics (1980). Productivity: A Selected Annotated Bibliography, 1976–78. U.S. Government Printing Office, Washington, D.C.
21. Kennedy, C. and Thirlwall, A. P. (1972). Econ. J., 82, 11–72. (Survey paper; intermediate.)
40. U.S. Bureau of Labor Statistics (1982). Productivity Measures for Selected Industries, 1954–80. U.S. Government Printing Office, Washington, D.C.
22. Kleiman, E., Halevi, N., and Levhari, D. (1966). Rev. Econ. Statist., 48, 345–347.
41. Wickens, M. R. (1970). Rev. Econ. Statist., 52, 187–193.
PROFILE ANALYSIS
6531
42. Worswick, G. D. N. (1959). Oxf. Econ. Pap., 11, 125–142.
PROFESSIONAL STATISTICIAN, THE .
See also DIVISIA INDICES; FISHER’S IDEAL INDEX NUMBER; INDEX NUMBERS; INDEX OF INDUSTRIAL PRODUCTION; PAASCHE –LASPEYRES INDEX; and PRODUCER PRICE INDEXES.
PROFILE ANALYSIS
NURI T. JAZAIRI LOUIS LEFEBER
PRODUCT-LIMIT
ESTIMATOR. See
KAPLAN –MEIER ESTIMATOR —I
PRODUCT MOMENT The expected value of a product of random variables (X1 , X2 , . . . , Xm ) possibly raised to different powers. The usual notation is m rj µr1 r2 ...rm (X1 , X2 , . . . , Xm ) = E Xj . j=1
(For brevity the symbol µr1 r2 ...rm is often used.) The corresponding central product moment is m r j µr1 r2 ...rm = E {Xj − E[Xj ]} . j=1
To emphasize the distinction between µr1 r2 ...rm and µr1 r2 ...rm the former is sometimes called a crude product moment. Some of the r’s may be zero, but at least two should differ from zero. Another name is mixed moment. Absolute product moments are defined by m vr1 r2 ...rm = E |Xj |rj , vr1 r2 ...rm = E
j=1 m
|Xj − E[Xj ]|
rj
.
j=1
See also ABSOLUTE MOMENT and COVARIANCE.
PRODUCT, MULTINOMIAL. See LOG-LINEAR MODELS IN CONTINGENCY TABLES
See STATISTICIAN, THE
‘‘Profile analysis’’ has been used as a statistical term since the fifties. Originally it was used in the social sciences, but there exist many applications in medical, biological, and economic fields. One example concerns biological diurnal rhythms of hormone levels such as cortisol or melatonin levels. One may observe serum samples of hormone levels from human beings over a day-and-night period. Typically melatonin reaches its peak value about midnight and its lowest values during the forenoon, whereas the opposite holds for cortisol. We may be interested in comparing different groups of individuals with respect to some population averages over day-and-night cycles—for instance, between shift workers, who are on duty during at night, and nine-to-five workers. Thus, we may be dealing with data which, for each individual from each population, have been collected every fourth hour during the twenty-four hours. Hence, we may have observations at 4, 8, 12, 16, 20, and 24 o’clock, from which it is possible to construct a graph by joining averages within respective treatment groups. The primary interest is to study if the two graphs are similarly shaped. We suppose that for various reasons the mean values are of less interest than the shape, or, in other words, the profile. The so-called battery of tests may serve as another interesting example. Suppose that one intends to compare students from two schools with regard to their performance in mathematics, languages, and biology. For each of these subjects one tests the students and comes up with some kind of score (see Table 1). In Fig. 1(a) the average scores for school A and school B are plotted and joined with lines. The figure suggests that the shapes of the graphs are the same. In Fig. 1(b) are graphs that clearly differ. Note that one is not interested in the absolute levels of the scores; one is interested in their relative levels. The conclusion could be that one school is focusing more on mathematics and the other on languages. It could also be
6532
PROFILE ANALYSIS
Table 1. Student Scores in Two Schools and Three Subjects School A
School B
Individual
Math.
Lang.
Biol.
Individual
Math.
Lang.
Biol.
1 2 . .. n1
10 11 . .. 10
22 20 . .. 21
11 10 . .. 10
1 2 . .. n2
15 13 . .. 17
22 24 . .. 23
15 14 . .. 16
that the difference in scores has nothing to do with the schools themselves, but rather mirrors the selection of students. Of course, the quality of the teachers may be better in one school, but it is difficult to draw that conclusion, because the students do not constitute random samples. Let us now formalize the concept of statistical profile analysis. The purpose is to present the classical approach first given by Greenhouse and Geisser [3]. However, the analysis below will be put into the framework of multivariate analysis of variance∗ , repeated measurements∗ , and growth curve∗ analysis. It is beyond our scope to give a complete review of the subject; various generalizations can be found via a literature search. For extensions within the framework of classical multivariate linear models see refs. [5,10,13]. Other extensions, where test statistics are based on a nonparametric approach, are given by Chinchilli and Sen [2] and Bhapkar [1]. Even in the classical approach there are several open problems; one is to exploit the connections with multiple testing. Among other things, the significance levels of the tests given below are not as exact as might appear at first sight. Since experimental units are observed (measured) several times, we cannot regard all measurements as independent; hence we start from a repeated-measurements point of view where observations within experimental units are correlated and observations between experimental units are uncorrelated. Suppose that for experimental units within g treatment groups, p repeated observations are obtained, say, Xijk ,
i = 1, 2, . . . , p,
j = 1, 2, . . . , g,
k = 1, 2, . . . , nj .
(1)
We assume that the experimental units constitute random samples from their respective groups. Hence, for every i and j the expectations satisfy E[Xijk ] = µij ,
k = 1, 2, . . . , nj .
For the dispersion it is natural to assume some kind of correlation structure within experimental units, i.e., the covariances satisfy C[Xi1 j1 k1 , Xi2 j2 k2 ] " σi1 i2 = 0
for j1 = j2 , k1 = k2 , otherwise.
As with repeated measurements, the σij are functionally dependent if autocorrelation, intraclass correlation, or equicorrelation structure, etc. applies. In the following, besides symmetry (σij = σji ), we only require positive definiteness of the matrix = (σij ); no other structure will be imposed. Let us at first consider just one treatment group, i.e., g = 1 in (1). If the observations Xik are arranged into a matrix X = (Xik ) and the mean parameters µi into a p-vector B = (µi ), the model X = BC + F
(2)
emerges, where C is a vector of n ones, i.e. C = 1n = (1, 1, . . . , 1), and F is an error matrix. Since the columns of X are independently distributed, so also are the columns fi , i = 1, 2, . . . , n, of F. Up to now we have not made any assumption about the distributions for F. In the classical approach, we assume a multivariate normal distribution to hold.
PROFILE ANALYSIS
6533
where B1 C1 : B2 C2 : · · · : Bg Cg = (B1 : B2 : · · · : Bg )C and C is a g ×
nj matrix satisfying
1 ··· 0 · · · C = . .. 0 ···
1 0 .. . 0
0 ··· 1 ··· .. .. . . 0 ···
0 ··· 1 ··· .. . 0 ···
0 0 .. . . 1
Let X = (X1 : X2 : · · · : Xg ), B = (B1 : B2 : · · · : Bg ), F = (F1 : F2 : · · · : Fg ), where Fj ∼ Np,nj (0, , Inj ). Then, again, we may write X = BC + F.
(3)
However, Xisp ×
g
nj , Bisp × g,
j=1
Cisg ×
g j=1
Figure 1. (a) Average scores for school A (solid lines) and school B (dashed lines). (b) Profiles which are not parallel.
Thus, fi ∼ Np (0, ). In general, when there are g treatment groups, there will be g models of the form (2). Putting these models side by side yields X1 : X2 : · · · : Xg = B1 C1 : B2 C2 : · · · : Bg Cg + F1 : F2 : · · · : Fg ,
nj , Fisp ×
g
nj .
j=1
At first sight the model in (3) is just the multivariate analysis of variance∗ model (MANOVA). However, next we present what is typical of profile analysis. Consider the special case of two groups, i.e. g = 2, and let us return to the classical battery-of-tests example. Suppose now in Fig. 1(a) that the dots represent the mean values, i.e., for school A we have µA1 , µA2 , µA3 , and for school B, µB1 , µB2 , µB3 . According to the literature on the classical approach, there exist three different tests concerning the mean structure. The first is the so-called test of similarity of profiles, sometimes referred to as the test of parallel profiles. The null hypothesis is µ1B − µ2B µ1A − µ2A = (4) H01 : µ2A − µ3A µ2B − µ3B
6534
PROFILE ANALYSIS
This is the starting point of the classical approach. Note that the way of ordering the points µ1· , µ2· , µ3· is immaterial for the procedures given below, although the graphs such as the one in Fig. 1(a) will look different. Equivalently, in (4) we may write
H01 :
µ1A − µ1B µ2A − µ2B
=
µ2A − µ2B ; µ3A − µ3B
n
j 1 xj = xkj , nj
k=1
xkj = (X1jk , X2jk , . . . , Xpjk ) ,
j = 1, 2,
and
equality of mean differences is the same as no interaction between tests and schools. The restriction in (4) will be reformulated in terms of matrices, but instead of p = 3 a general p will be considered. Let D be (p − 1) × p, where 1 −1 0 · · · 0 1 −1 · · · D = . . .. .. .. . 0 0 0 ··· 1 E= . −1
where
0 0 .. , . 1
S=
n1 n2 (xk1 − x1 )2 + (xk2 − x2 )2 . k=1
k=1
The test of similar profiles was first given by Greenhouse and Geisser [3] and extended a result of Rao [7]. If the hypothesis of similar profiles is not rejected the classical approach explores whether the profiles are either identical (coincident) or parallel to the x-axis. The resemblance of profiles is modeled by parametric restrictions on B, given by
(5) BE = 0,
Then the hypothesis in (4) is identical to H01 : E[X] = BC,
DBE = 0.
(6)
where E was defined in (5) (see Fig. 2). However, since the hypothesis of similarity of profiles is not rejected, the testing problem is set up as
(7)
H02 : E[X] = BC, DBE = 0, BE = 0;
The alternative hypothesis will be H11 : E[X] = BC,
i.e., DBE = 0. Hence, the test of similar profiles is a MANOVA with a null hypothesis described via (6). If g = 2, i.e. two treatment groups, the hypothesis H01 is rejected if n1 + n2 − p E (CC )−1 CX D (DSD )−1 p−1
−1
× DXC (CC )
−1
E[E (CC )
E]
Fp−1,n1 +n2 −p,α ,
−1
H12
: E[X] = BC, DBE = 0.
(8) (9)
Note that H02 is identical to H02 : E[X] = BC, BE = 0. From one point of view the test formulated by (8) and (9) is more powerful than the usual MANOVA test
H02 : E[X] = BC, DBE = 0, BE = 0;
where
H12 : E[X] = BC, S = X(I − C (CC )−1 C)X .
Observe that E (CC )−1 E =
1 1 + , n1 n2
XC (CC )−1 E = x1 − x2 ,
since in this test no assumption about parallelism is made. However, to properly take into account the pretesting in (6), in order to determine the significance level, is very difficult. If the profiles are assumed to be identical, one may be interested in estimating the
PROFILE ANALYSIS
6535
or expressed in the original quantities, −1 % = 1p 1 p S (x1 − x2 ) , BE 1 p S−1 1p
since one may choose (D )0 = 1p . Furthermore, since the profiles are supposed to be parallel, we can write BE = δ1p , where δ is an unknown constant estimated by δˆ =
1 p S−1 (x1 − x2 ) . 1 p S−1 1p
From results on growth curves∗ a confidence interval for δ can be constructed: Iδ = δˆ ± tn1 +n2 −p−1,α/2
−1/2
×
n1 n2 (n1 + n2 − p − 1) n1 + n2
(1
2 + Tp−1 )1/2 (1 p S−1 1p )−1/2
,
where 2 = Tp−1
n1 n2 (x1 − x2 ) D (DSD )−1 n1 + n2 × D(x1 − x2 )
Figure 2. (a) Coincident profiles. (b) Profiles which are not coincident.
difference between them. In terms of a linear models approach, E[X] = BC,
DBE = 0,
the intention would be to estimate BE. Let (D )0 and E0 be any matrices of full rank such that (D )0 D = 0, r((D )0 ) = p − r(D), E0 E = 0 0, r(E ) = g − r(E), where r(·) denotes the rank of a matrix. Note that E0 and (D )0 form bases for the orthogonal complements to the column spaces of E and D , respectively. By either performing a reparametrization or relying on the theory for multivariate linear models [8] it follows that % = (D )0 [(D )0 S−1 (D )0 ]−1 (D )0 S−1 BE × XC (CC )−1 E,
(10)
(see HOTELLING’S T 2 ). A test that the profiles are identical given that the profiles are parallel can be performed via the confidence interval, i.e., the null hypothesis is rejected if 0 lies outside the interval. Alternatively, the hypothesis is rejected if n1 n2 (n1 + n2 − p − 1) −1 2 1 p S 1p δˆ2 (1 + Tp−1 )−1 n1 + n2 F1,n1 +n2 −p−1,α . The test and the confidence interval are due to Srivastava [11]. Commonly in use are less powerful alternative tests. For example, to test that the profiles are identical given that they are similar one tests H02 : E[X] = BC, 1 p BC = 0. The test is rejected [4] if n1 n2 (n1 + n2 − 2) (x1 − x2 ) 1p (1 p S1p )−1 1 p n1 + n2 × (x1 − x2 ) F1,n1 +n2 −2,α .
6536
PROFILE ANALYSIS
(n1 + n2 − p + 1)(n1 + n2 ) x D DSD p−1
−1 1 1 + + D(x1 − x2 )(x1 − x2 ) D n1 n2 × Dx Fp−1,n1 +n2 −p+1,α , where x = (n1 x1 + n2 x2 )/(n1 + n2 ). Sometimes, as a third step in profile analysis, one tests whether the profiles are identical and parallel to the x-axis against the alternative that the profiles are identical. In this case we have a stagewise procedure. First one tests that the profiles are similar; second, if the profiles are similar, one tests if the profiles are identical; third, if the profiles are identical, one tests if they are parallel to the x-axis. The last is precisely the test of equality of means in one sample. Formally, using the preceding notation, we may write
H03 : E[X] = BC, BE = 0, DB = 0,
H13 : E[X] = BC, BE = 0,
and H03 is rejected if n1 + n2 − p + 1 0 E CX D (DSD )−1 DXC E0 (p − 1)(n1 + n2 ) Figure 3. (a) Profiles parallel to the x-axis. (b) Profiles which are not parallel to the x-axis.
Next we present the test that the profiles are parallel to the x-axis (see Fig. 3). For this test the hypotheses are H03 : E[X] = BC, DBE = 0, DB = 0, H13 : E[X] = BC, DBE = 0, which is equivalent to H03 : E[X] = BC, DB = 0, H13 : E[X] = BC, DBE = 0. pt The test of parallelism with the x-axis is sometimes referred to as a test of condition variation, since it sometimes tests for differences in conditions. The likelihood ratio test∗ states that H03 is rejected if
Fp−1,n1 +n2 −p+1,α .
where E0 CX = n1 x1 + n2 x2 = (n1 + n2 )x. A mean structure of the form E[X] = ABC for some A and C comes within the growth curve model of Potthoff and Roy [6] (see ref. [9] for a review). All results given above follow from results for the growth-curve model, but by assuming that the profiles are parallel, the tests here usually differ from those applied in growth curve analysis. Moreover, the assumption of similar profiles, i.e. the restriction DBE = 0, can equivalently be written B = 1p 1 + D 2 1 g .
(11)
In this equation 1 and 2 have natural interpretations; (10) presents an extended growth-curve model [8]. Furthermore, to test for identical profiles is the same as to test whether 1 = 0, and in order to test for parallelism with the x-axis we test whether 2 = 0.
PROFILE ANALYSIS
We now present some results for more than two treatment groups, i.e. r(C) > 2. The distribution of the test statistic for the test of parallel profiles is then more complicated. In (6) and (7) the test of similarity of profiles was given; if we replace C in these equations by C given by (3) and the matrix E by the g × (g − 1) matrix 1 0 ··· 0 −1 1 · · · 0 E = 0 −1 · · · 0 , . . . .. .. .. 0 0 · · · −1 the hypothesis-testing problem once again is H01 : E[X] = BC, DBE = 0, H11 : E[X] = BC. From MANOVA results the hypothesis is rejected if |DSD |/|DMD | Up−1,g−1,n−g,α , where n=
g
nj ,
S = X[I − C (CC )−1 C]X ,
j=1
M = X[I − C 1g (1 g CC 1g )−1 1 g C]X , and U·,·,·,α is the α-percentile of the standard test distribution in MANOVA (e.g. ref. [12]; see also WILKS’S LAMBDA CRITERION; HOTELLING’S TRACE; PILLAI’S TRACE). In order to test that the profiles are identical given that they are parallel, the quantity λ=
|S||DMD | |M||DSD |
is used. The hypothesis is rejected if n−g−p+11−λ Fg−1,n−g−p+1,α . g−1 λ The hypothesis that all profiles are parallel to the x-axis is rejected if n−p+1 1 g CX D (DMD )−1 DXC 1g n(p − 1) Fp−1,n−p+1,α . For further details see Srivastava [11].
6537
REFERENCES
1. Bhapkar, V. P. (1984). On nonparametric profile analysis of several multivariate samples. J. Statist. Plann. Inference, 9, 285–296. (Procedures based on U-statistics and linear rank statistics are given.) 2. Chinchilli, V. M. and Sen, P. K. (1982). Multivariate linear rank statistics for profile analysis. J. Multivariate Anal., 12, 219–229. (Linear rank statistics are utilized together with Roy’s union–intersection principle.) 3. Greenhouse, S. W. and Geisser, S. (1959). On methods in the analysis of profile data. Psychometrika, 24, 95–112. (The first paper on profile analysis.) 4. Johnson, R. A. and Wichern, D. W. (1998). Applied Multivariate Statistical Analysis, 4th ed. Prentice-Hall, Englewood Cliffs, NJ. (Introductory textbook in multivariate analysis, including numerical illustrations; see Sec. 6.6.) 5. Nel, D. G. (1987). A profile analysis for two groups under the heteroscedastic assumption. Commun. Statist. Theory Methods, 16, 3577–3589. (The Behrens–Fisher problem is considered.) 6. Potthoff, R. F. and Roy, S. N. (1964). A generalized multivariate analysis of variance model useful especially for growth curve problems. Biometrika, 51, 313–326. (The first presentation of the growth-curve model.) 7. Rao, C. R. (1948). Tests of significance in multivariate analysis. Biometrika, 35, 58–79. (The analysis of dispersion is clearly presented.) 8. von Rosen, D. (1989). Maximum likelihood estimators in multivariate linear normal models. J. Multivariate Anal., 31, 187–200. (General approach for treating multivariate linear models is given.) 9. von Rosen, D. (1991). The growth curve model: a review. Commun. Statist. Theory Methods, 20, 2791–2822. (Papers which have treated the growth curve model are presented.) 10. Sparks, R. and Ballantyne, R. (1990). Interactions explored using an extension of profile analysis. S. Afr. Statist. J., 24, 127–152. (An interesting paper where profile analysis is extended.) 11. Srivastava, M. S. (1987). Profile analysis of several groups. Commun. Statist. Theory Methods, 16, 909–926. (General theoretical results for profile analysis are given.)
6538
PROFILE-a TESTS
12. Srivastava, M. S. and Carter, E. M. (1983). Applied Multivariate Statistics. NorthHolland, New York. (A solid applied work on classical multivariate analysis.) 13. Yokoyama, T. and Fujikoshi, Y. (1993). A parallel profile model with random-effects covariance structure. J. Japan Statist. Soc., 23, 83–89. (Profile analysis with certain covariance structures is considered.) See also MULTIVARIATE ANALYSIS; and MULTIVARIATE ANALYSIS OF VARIANCE (MANOVA).
DIETRICH VON ROSEN
PROFILE-a TESTS In the classical Neyman–Pearson theory of statistical hypothesis testing∗ , the probability of an erroneous rejection of a hypothesis H is controlled by a preassigned significance level α, called the size of the test. The underlying concept involves finding tests with highest power subject to a given size α. That is, size and power are treated unequally, leading to unsymmetrical test results with different interpretative meanings. In particular, not rejecting (‘‘accepting’’) H does not imply the rejection of the alternative hypothesis. This drawback in the classical theory gave rise to the development of profile-a tests, which ¨ were introduced by Ruger [2,3] and further developed by Wassmer [4]. The idea is to measure the evidence of a sample for one hypothesis against the other by the ratio of the respective probabilities of some appropriate region in the sample space∗ . The decisions are based on a level a, which does not—as in the classical theory—consist of a condition for a probability, but rather for a ratio of two probabilities. For the sake of simplicity, consider two simple hypotheses H− and H+ , and let there be given a number a > 1 (usually an integer). If an event is observed which under H+ is at least a times more likely than under H− , we accept H+ (and reject H− ). Conversely, if an event is observed which under H− is at least a times more likely than under H+ , we accept H− (and reject H+ ). That is, we do not already reject a hypothesis—as in the classical theory—if the probability of erroneous rejection is small (α), but only if the probability of a correct decision (i.e., the probability of
accepting H− if H− is true and the probability of accepting H+ if H+ is true, respectively) is at least a times greater than the probability of a false decision. Furthermore, to achieve symmetric test results the ratios of the respective probabilities should be equal. This additional symmetry condition is extracted from the idea that the hypotheses should be treated symmetrically and that a decision on H− should posses the same evidence from the data as a decision on H+ . A profile test is well defined if a no-decision region is introduced which consists of the data points where the sample shows evidence for neither H− nor H+ . Subject to an optimality criterion which maximizes the probabilities for correct decisions, these conditions lead to a unique partition of the sample space. Example 1. Let X1 , . . . , Xn be independent and distributed according to the Gaussian distribution with known σ 2 = 1, and suppose it is desired to test H− : µ = −1 and H+ : µ = +1. For n = 2, the corresponding profile test at level a = 20 becomes: if x < −0.211, accept H− ; if x > 0.211, accept H+ ; otherwise, accept neither H− nor H+ . As is easy to verify, PrH+ (X > 0.211) PrH− (X > 0.211)
=
PrH− (X < −0.211) PrH+ (X < −0.211)
= 20; 0.211 and −0.211 are the lowest and the greatest values, respectively, for which the profile condition is fulfilled. These thresholds consequently give the highest probabilities for correct decisions. For comparison, the classical Neyman–Pearson∗ test at level α = 0.05 rejects H− in favor of H+ if x > 0.170, or rejects H+ in favor of H− if x < −0.170, and hence the profile test approach leads to a more conservative decision rule than the classical test with α = 0.05. The α = 5% level of the classical test, however, ‘‘corresponds’’ with the a = 20 level, since by performing a profile-a test both probabilities of false decision are bounded by 1/a. For n = 10, the profile test becomes:
PROFILE-a TESTS
if x < 0, accept H− ; if x > 0, accept H+ . Here, PrH+ (X > 0) PrH− (X > 0)
=
PrH− (X < 0) PrH+ (X < 0)
> 20,
and 0 is the unique threshold that provides a profile test at level a = 20. In particular, the symmetry condition makes the calculation of the test possible. (Note that a lower threshold for accepting H+ and a higher threshold for accepting H− would keep the profile condition, but would fail to produce a well-defined test.) The classical test (α = 0.05) rejects H− in favor of H+ if x > −0.477, or rejects H+ in favor of H− if x < 0.477, and hence depends on the formulation of the test problem, leading to different test outcomes. Example 2. Let X be binomial (n; p) with n = 20; the hypotheses H− : p = 0.2 and H+ : p = 0.4 are tested at level a = 20. When we choose c to make PrH+ (X > c)/ PrH− (X > c) greater than and as near as possible to a = 20 (i.e., we want to decide on H+ for large values of x), we get c = 8. To get a = 20 exactly here, we have to resort to randomization∗ , yielding the rule: accept H+ with probability 1 if x = 9, . . . , 20; accept H+ with probability 0.777 if x = 8. Analogously, the rule for H− becomes: accept H− with probability 1 if x = 0, . . . , 3; accept H− with probability 0.191 if x = 4. The theory of profile tests therefore uses a test function φ(x) with three components refering to the probabilities of the outcomes ‘‘acceptance of H+ ,’’ ‘‘acceptance of H− ,’’ and ‘‘no decision.’’ The decision rule of a profilea test allows symmetric test results, thus avoiding the difficulty of the (actually unnatural) unsymmetry in the classical test results. There are alternative approaches that overcome this feature of the Neyman–Pearson theory (see MULTIPLE
6539
DECISION PROCEDURES). None of these, however, takes into account the probability ratio of the test outcomes. A second drawback in the classical theory is that the choice lies between only two test outcomes: rejecting H or not rejecting H corresponds to a simple division of the sample space into the critical region and its complement. It would be desirable, however, to find a concept which distinguishes between sample results, allowing hierarchically ordered types of inferences. Indeed, in deciding between two hypotheses H− and H+ , profile-a tests in general are characterized by permitting more than two test outcomes, which are ordered according to the evidence of a sample for either H− or H+ . GENERAL FORMULATION To progress with the general formulation of profile-a tests, consider 2k + 1 test outcomes A−k , . . . , A0 , . . . , A+k , where A−k , . . . , A−1 and A+1 , . . . , A+k are decisions on the (not necessarily simple) hypotheses H− and H+ , respectively, and A0 is the decision on neither H− nor H+ . A+i+1 shows stronger evidence for H+ than A+i , and A−i−1 analogously shows stronger evidence for H− than A−i . Accordingly, the resulting test is given by a test vec(x), . . . , φ0 (x), . . . , φ+k (x)) where tor φ(x) = (φ−k 0 φi (x) 1, φi (x) = 1 for all x in the sample space, and φi (x) refers to the probability of the decision Ai given x. If the distribution of X is Pθ, these probabilities are given by Pr θ (Ai ) = φi (x)dPθ (x), where θ denotes the unknown parameter. In the nonrandomized case such a test consists of a partition of the sample space in disjoint regions B−k , . . . , B0 , . . . , B+k with φi (x) = 1 if x ∈ Bi , / Bi . and φi (x) = 0 if x ∈ Let a = (a1 , . . . , ak ) with 1 < a1 < · · · < ak denote the preassigned (arbitrarydimensional) level. The basic requirement for the test φ to be a profile-a test is that for i = 1, . . . , k, inf θ ∈H+ Pr θ (A+i ) ai sup η∈H− Pr η (A+i ), inf η∈H− Pr η (A−i ) ai sup θ ∈H+ Pr θ (A−i ) (1) should hold. That is, in the nonrandomized case the test results in A+i if an event B+i is observed whose probability under H+ is
6540
PROFILE-a TESTS
at least ai times more likely than under H− , and an analogous condition should hold in the event B−i for the decision A−i . If sup η∈H− Pr η (A+i ) = 0 and sup θ ∈H+ Pr θ (A−i ) = 0, the symmetry condition is written as inf η∈H− Pr η (A−i ) inf θ ∈H+ Pr θ (A+i ) = , (2) sup η∈H− Pr η (A+i ) sup θ ∈H+ Pr θ (A−i ) i = 1, . . . , k. Hence, the requirements for the outcomes A+i are completely analogous to those for the outcomes A−i . A test φ that fulfills (1) and (2) is called a profile test at level a or profile-a test. Simple Hypotheses ¨ Ruger [2,3] introduced profile-a tests for the case of simple hypotheses, i.e., both H− and H+ consist of one distribution, which, without loss of generality, may be parametrized by θ− and θ+ , respectively. In the family Fa of these tests, he defined an optimality criterion for profile-a tests based on the idea of optimizing probabilities for correct test decisions where decisions with higher evidence for H− or H+ take precedence over decisions with lower evidence. More precisely, a profile-a test φ is said to be better in component i than a profile-a test ψ if φ+i (x)dPθ+ (x) ψ+i (x)dPθ+ (x),
φ−i (x)dPθ− (x)
ψ−i (x)dPθ− (x),
with strict inequality at least once. A profilea test φ is as good as ψ in component i if equality holds in the above conditions. Using this terminology, a profile-a test φ is better than ψ if there exists a j(1 j k) with φ as good as or better than ψ in the components i with j < i k, and φ better than ψ in component j. Consequently, a profile-a test φ ∗ is optimum in Fa if there exists no test in Fa better than φ ∗ . As a consequence of the Neyman–Pearson ¨ [2] showed that φ ∗ has neclemma∗ , Ruger essarily the form of a generalized likelihood ratio test∗ , which in the nonrandomized case is given by 1 if Q(x) < κ−k , ∗ (x) = φ−k 0 if Q(x) κ−k ,
∗ φ−i (x)
Q(x) κ−i−1 , 0 if = 1 if κ−i−1 < Q(x) < κ−i , 0 if Q(x) κ−i ,
for i = 1, . . . , k − 1, 0 = 1 0 0 ∗ φ+i (x) = 1 0 φ0∗ (x)
if Q(x) κ−1 , if κ−1 < Q(x) < κ+1 , if Q(x) κ+1 , if Q(x) κ+i , if κ+i < Q(x) < κ+i+1 , if Q(x) κ+i+1 ,
for i = 1, . . . , k − 1, and ∗ (x) = φ+k
0 if Q(x) κ+k , 1 if Q(x) > κ+k ,
where Q(x) denotes the likelihood ratio of H+ to H− , and κi are real numbers with 0 κ−k · · · κ+k ∞. [In order to permit randomization one has to treat the case Q(x) = κi separately.] Due to the optimality criterion, κ+k and κ−k in φ ∗ are, if possible, chosen such that Pr θ+ (Q(X) > κ+k ) = ak Pr θ− (Q(X) > κ+k ) =
Pr θ− (Q(X) < κ−k ) . Pr θ+ (Q(X) < κ−k )
Having done this, the remaining κi ’s have to be chosen so that Pr θ+ (κ+i < Q(X) < κ+i+1 ) = ai Pr θ− (κ+i < Q(X) < κ+i+1 ) =
Pr θ− (κ−i−1 < Q(X) < κ−i ) . Pr θ+ (κ−i−1 < Q(X) < κ−i )
This is done first, if possible, for i = k − 1, then for i = k − 2, etc. An optimum profile-a test φ ∗ can be definitely calculated in this way, since R(τ , ξ ) :=
Pr θ+ (τ < Q(X) < ξ ) , Pr θ− (τ < Q(X) < ξ ) 0 τ < ξ ∞,
is (weakly) isotonic in ξ given τ , and in τ given ξ .
PROFILE-a TESTS
Example 3. Let X1 , . . . , Xn be independent and Gaussian (µ, σ ) with known µ = 0, and the hypotheses tested at profile level a = (10, 20, 100) be H− : σ 2 = 1 and H+ : σ 2 = 2. depends on the sample The optimum test φ ∗ x only through s2 = x2i /n, and φ ∗ is given by the regions for s2 . For various n the optimum tests are given in Table 1. Depending on n, not all test outcomes may be realized. For n = 8, outcomes with higher evidence for H− than A−1 are not realizable. For moderate n(n = 16) all outcomes are realizable, whereas with increasing n the number of realizable outcomes decreases. If n is large enough (n = 100), only outcomes in the components with highest intensity are possible. As in this example, a general feature of the profile-a test concept is that the two distributions (under H− and H+ ) produce a specific division of the sample space and ‘‘pick out’’ their typical test outcomes.
Composite Hypotheses In order to derive profile-a tests for composite hypotheses∗ we first mention that in general, for a test to fulfill a specified a,
Table 1. Test results
Region n=8
A+3 A+2 A+1 A0 A−1 A+3 A+2 A+1 A0 A−1 A−2 A−3 A+3 A0 A−1 A−3 A+3 A−3
s2 > 3.217 2.653 < s2 < 3.217 2.436 < s2 < 2.653 0.304 < s2 < 2.436 s2 < 0.304 n = 16 s2 > 2.219 2.062 < s2 < 2.219 1.876 < s2 < 2.062 0.846 < s2 < 1.876 0.773 < s2 < 0.846 0.270 < s2 < 0.773 s2 < 0.270 n = 50 s2 > 1.535 1.238 < s2 < 1.535 1.166 < s2 < 1.238 s2 < 1.166 n = 100 s2 > 1.377 s2 < 1.377
6541
the hypotheses H− and H+ must be positive in distance. That is, assuming an indifference zone I between the hypotheses, the parameter space is partitioned into three sets belonging to H− , H+ , and I, respectively. Parameter values in H+ must differ from those in H− to the extent that the probability requirements (1) can be fulfilled at all and, accordingly, profile-a tests for testing H− and H+ are obtainable. In order to derive optimum profile-a tests for composite hypotheses, an alternative optimality criterion is formulated that demands a hierarchical maximin optimality of an optimum profile-a test. Roughly speaking, this criterion first requires maximizing inf θ ∈H+ φ+k (x)dPθ (x) (x)dP φ and inf η∈H η (x). Having done this, −k − inf θ ∈H+ φ+k−1 (x)dP θ (x) has to be maximized, and inf η∈H− φ−k+1 (x)dPη (x), etc. A solution to this maximin problem is obtained by specifying a pair of distributions in the parameter space which is least favorable∗ , i.e., which minimizes the hierarchical power of a profile-a test for the reduced (simple) hypotheses formed with this pair. In specific situations, e.g., by assuming a oneparameter exponential family∗ and by considering hypotheses that specify regions in the parameter space, the pair of least-favorable distributions∗ is given by the pair which gives mass 1 on the points in the parameter spaces of H− and H+ with the smallest distance. Optimum profile-a tests are then given by the generalized likelihood ratio test described in the preceding section. The calculation of the κi ’s is straightforward but generally different from the case where only simple hypotheses are considered. Nonetheless, the determination leads to a unique solution which first makes the regions for A+k and A−k as large as possible, then the regions for A+k−1 and A−k+1 , etc. [4]. Example 4. Suppose X1 , . . . , Xn are independent and Gaussian (µ, σ ) with known σ 2 = 4. By considering the hypotheses H− : −0.5 − c µ −0.5 and H+ : +0.5 µ +0.5 + c, given c 0, for n = 10 the maximin tests φ ∗ [profile level a = (10, 20, 100)] are based on x (see MINIMAX TESTS), and the resulting regions for the test outcomes are given in Table 2. If the specified parameter regions for H− and H+ are small, i.e., if c is
6542
PROFILE-a TESTS
Table 2. Test results A+3 A+2 A+1 A0 A−1 A−2 A−3 A+3 A+2 A0 A−2 A−3 A+3 A0 A−3
Region c = 0.5(c = 0) x > 1.640 0.973 < x < 1.640 0.873 < x < 0.973 −0.873 < x < 0.873 −0.973 < x < −0.873 −1.640 < x < −0.973 x < −1.640 c = 2.0 x > 1.640 1.360 < x < 1.640 −1.360 < x < 1.360 −1.640 < x < −1.360 x < −1.640 c = 2.5 x > 1.640 −1.640 < x < 1.640 x < −1.640
small, then φ ∗ coincides with the optimum test for the corresponding simple hypotheses (c = 0). With larger c the number of realizable outcomes decreases continually. If c is large enough and, in particular, the one-sided hypotheses H− : µ −0.5 and H+ : µ +0.5 are accepted, then the differentiation of outcomes in interior components of φ ∗ is no longer possible, and only decisions with highest evidence for H− or H+ are realizable. This last property is unappealing and might seem inappropriate to practitioners. In the light of the conceptual framework of profile-a tests, however, it is obvious and easily interpretable. Namely, in these cases the class of distributions is ‘‘too big’’ to produce a sufficiently fine division of the sample space. Furthermore, typically only rather large regions for H− and H+ make the differentiation in interior components impossible. Nevertheless, it can be argued that with a less extreme sample the intensity of the test result should gradually decrease, too. This issue is under discussion, possibly leading to an alternative profile-a test concept. In this concept, in order to define the tests and their optimality, separate test decisions A+i and A−i as described here are not considered, but rather their cumulative counterparts ∪kj=i A+j and ∪kj=i A−j , respectively. Fortunately, the
derivation of cumulative profile-a tests is straightforward, since it is reducible to the case k = 1. Clearly, in the latter case this concept leads to exactly the same tests as the one previously described. For k > 1, however, it does not. In the case of one-sided hypotheses, cumulative profile-a tests have more different outcomes in general between the two extremes A+k and A−k . Therefore, this concept might be preferred. The question, however, of whether these tests are preferable on the whole is still unanswered. The one-parameter case provides some general solutions, since here the pair of leastfavorable distributions is directly obtainable. In more general cases, e.g., by considering test problems with nuisance parameters∗ , these pairs are available only in specific situations (concerning assumed hypotheses and distributions). On the other hand, restricting the class of permitted tests, as in the classical theory [1], provides solutions for optimum profile-a tests. In particular, restricting the class to tests that are invariant under suitable transformations in the sample and parameter space leads to invariant maximin profile-a tests (see INVARIANCE). Furthermore, conditioning is also an instrument which enables the determination of profilea tests (see CONDITIONAL INFERENCE). These approaches and the way in which classical properties are transferable to profile-a tests are described in detail in Wassmer [4]. APPLICATIONS In practice profile-a tests provide a tool for evaluating and interpreting sample results. For example, these tests can be applied in bioavailability and bioequivalence∗ studies, where the tests can establish both equivalence and nonequivalence of two drugs [5] (see also BIOEQUIVALENCE CONFIDENCE INTERVALS). Among other things, in these studies profile tests display which sample size is necessary so that a decision on the equivalence of the drugs is possible at all, or which sample size is necessary so that any sample result shows evidence for either equivalence or nonequivalence. Hence, these tests provide an alternative method for sample size calculations.
PROFILES OF DIVERSITY
There are other potential fields of application where both the rejection and the acceptance of a hypothesis (in favor of an alternative hypothesis) should be taken into account, and symmetric test decisions are desirable (e.g., quality control∗ ). Furthermore, in all areas where the knowledge of both states of nature should influence the construction of the test, profile-a tests are useful to apply. REFERENCES 1. Lehmann, E. L. (1986). Testing Statistical Hypotheses, 2nd ed. Wiley, New York. ¨ 2. Ruger, B. (1978). Niveauprofiltests: Konstruktionsprinzip von Alternativtests mit endlicher Ergebnismenge (in German). Tech. Rep., Department of Statistics, University of Munich, Germany. ¨ ¨ zwei ein3. Ruger, B. (1980). Alternativtests fur fache Hypothesen mit endlich vielen Ergebnissen (in German). Metrika, 27, 73–90. (The paper that introduced profile tests for the case of simple hypotheses. Proofs of the theorems are given in ref. [2].) ¨ 4. Wassmer, G. (1993). Niveauprofiltests fur zusammengesetzte Hypothesen (in German). Doctoral dissertation, Department of Statistics, University of Munich, Germany. (Generaliza¨ tion of Ruger’s fundamental ideas to composite hypotheses.) 5. Wassmer, G. (1994). Testing equivalence in clinical trials using a new principle for constructing statistical tests. Commun. Statist. Theory and Methods, 23, 1413–1427. See also HYPOTHESIS TESTING; LIKELIHOOD RATIO TESTS; MINIMAX TESTS; and NEYMAN –PEARSON LEMMA.
GERNOT WASSMER ¨ BERNHARD RUGER
PROFILE LIKELIHOOD. See PSEUDOLIKELIHOOD
PROFILES, CIRCULAR. See SNOWFLAKES PROFILES OF DIVERSITY Diversity is an important concept in ecology and, under various names, it appears in several biological, physical, social, and
6543
management sciences∗ . The common theme is the apportionment of some quantity into a number of well-defined categories. The quantity may be in such a form as abundance, time, resource, investment, or energy. The concern in diversity analysis is about the degree of the apportionment being more diverse or less diverse, expressed variously as diversification or concentration, spread or specialization, monopoly or lack of it, and so on. As a simple example, consider the apportionment of time spent on two activities, music and mathematics. Suppose that John and Jane apportion their time as follows: Category Person
Music
Mathematics
John Jane
2 3 1 3
1 3 2 3
If we ask whether John has a different kind of specialization/diversification from Jane, the answer is yes: The subject identity matters. Instead, if we ask whether John has a different degree of specialization/diversification from Jane, the answer is no: The subject identity does not matter. The degree of specialization/diversification, or diversity, is permutation invariant; the identities of the categories are disregarded. Consider a quantity distributed among a finite set of categories labeled i = 1, 2, 3, . . . , s. Examples of such a quantity are (a) total abundance of an ecological community distributed among s species, and (b) total market sales distributed among s competing firms. Let πi be the proportion of the quantity found in the ith category, as, for example, the relative abundance of the ith species in an ecological community. We assume for this discussion that π1 π2 · · · πs > 0, and π1 + π2 + · · · + πs = 1. The relative abundance vector, π , is given by π = (π1 , π2 , . . . , πs ). Since the vast bulk of applications of diversity analysis occurs in the field of ecology (see ref. 1), we will here refer to the categories as species and define a community C as the pair C = (s, π ). The concept of diversity of a community has been defined by Patil and Taillie [9–12] independent of any particular diversity indices∗ . In a diverse community, the
6544
PROFILES OF DIVERSITY
typical species is relatively rare. Intuitively, a transfer of abundance either from a common to a rare species or from a common to a new, additional species would increase an observer’s assessment of the rarity of a typical species. Accordingly, a community C = (s , π ) is defined in [10] to be intrinsically more diverse than a community C = (s, π), written C C, if C can be constructed from C by a finite sequence of the following two operations: 1. Introducing a species in this manner: (a) subtract an amount h from the relative abundance πi of an existing species, where 0 < h < πi , and (b) add a new species to the community of relative abundance h. 2. Transferring abundance between two existing species in this manner: (a) subtract an amount h from the relative abundance πi of a species, and (b) add h to the relative abundance πj of another species, where 0 < h < πi − πj . Note that the species’ actual labels are ignored in transforming the vector π into the vector π . It is shown in ref. 12 that C is intrinsically more diverse than C if and only if every right tail-sum of π is greater than or equal to the corresponding right tail-sum of π , that is, i>k
πi
πi ,
k = 1, 2, 3, . . . , s.
(1)
i>k
Since the kth right tail-sum is the combined abundance of those species rarer than the kth ranked species, greater diversity in this sense means a greater amount of rarity. A diversity measure of a community C is defined in [10,12] as the average rarity of the community, given by (C) =
s
πi R(i, π ).
(2)
diversity measures with the following properties: (1) (C ) (C) whenever C C, and (2) (C) = 0 if C has only one species. Two general categories of diversity measures are used frequently, based on two types of rarity functions: (1) rarity functions of the dichotomous type depend only on the relative abundance of the ith species: R(i, π ) = R(πi ); and (2) rarity functions of the ranking type depend only on the (descending) rank of the ith species: R(i, π ) = R(i). One dichotomous rarity function suggested in refs. 10 and 12 is β
R(πi ) = (1 − πi )/β, where β is a constant chosen by the investigator such that β −1. The limiting form R(πi ) = − log πi is used for β = 0. The result is a parametric family of diversity measures: s β+1 β. (3) πi β (C) = 1 − i=1
Three diversity indices∗ popular in ecological studies are special cases of β : (a) −1 = s −1, the species richness index, (b) 0 = − πi log πi , the Shannon index, and (c) 1 = 1 − πi2 , the Simpson index. However, any other values of β −1 also yield diversity indices with the required properties mentioned above. Monotone transformation of β would preserve the ordering properties required of a diversity index; some have been proposed as indices. The transformation Sβ = 1/(1 − ββ )1/β is Hill’s [3] family of indices. It has the interpretation of being the numbers equivalent of β ; that is, Sβ is the number of species a totally even community must have in order for its value of β to equal that of the given community. The function log Sβ is R´enyi’s [15] entropy∗ of order β + 1, proposed by Pielou [13] as a diversity index. (See refs. 3, 13, and 15 and DIVERSITY INDICES for additional information.) Another dichotomous rarity function is R(πi ) = (1 − πi )[1 − (1 − πi )m ]/πi ,
i=1
Here R(i, π ) is a numerical measure of rarity associated with the ith species. Different forms for the function R produce different diversity measures. A natural requirement is to use only those forms of R which yield
where m is a nonnegative integer. It leads to the family of diversity indices studied in refs. 4 and 16, HSG m (C) =
s (1 − πi )[1 − (1 − πi )m ]. i=1
PROFILES OF DIVERSITY
6545
The quantity HSG + 1 has the interpretam tion of being the expected number of species obtained when m + 1 individuals are randomly selected from the community C. One ranking-type rarity function takes the kth ranked species to be the standard against which a species’ rarity is assigned the value 1 or 0: R(i) =
1, i > k . 0, i k
The diversity measure produced, denoted Tk (C), is simply the right tail-sum of the relative abundances: Tk (C) =
s
πi .
(4)
i>k
Other ranking-type indices are discussed in ref. 12. Diversity is a partial ordering in the sense that (a) C C, (b) C C whenever C C and C C, and (c) a given pair of communities need not be comparable. This explains the complaint frequently heard among ecologists (e.g., in ref. 4) that different diversity indices may give different orderings. In ref. 12, for example, a simple illustration involving two communities is given. The intrinsic diversity profile of a community C is a plot of the diversity measure Tk (C) (4) against the ranks k = 1, 2, . . . , s. By plotting the profiles of two communities on one graph, one can tell at a glance whether the communities can be ordered by diversity. By (1), C C ⇔ Tk (C ) Tk (C) for all k = 1, 2, . . . , s. Thus the profile of an intrinsically more diverse community is everywhere above or equal to that of a less diverse community (Fig. 1). The profiles will cross if the communities are not intrinsically comparable. The measure Tk (C) is a family of diversity indices parameterized by k. For larger values of k, Tk is in some sense more sensitive to changes in numbers and abundances of rare species in the community. In ref. 12 and Taillie [20] a formal definition of sensitivity of an index to rare species is provided. In particular, it is shown that the sensitivity of β to rare species is a strictly decreasing
Figure 1. Intrinsic diversity profiles of four communities (data from [11]). It is seen that C1 C2 , C1 C3 , C1 C4 , C2 C3 , C2 C4 , but that C3 and C4 are not intrinsically comparable.
function of β, while that of HSG is a strictly m increasing function of m. Other profiles of diversity can be constructed by plotting a parametric diversity index against a parameter which measures sensitivity of the index to rare species. For example, β (C) can be computed for a given community C using a variety of values of β −1. The plot of β against β is a type of diversity profile. Similarly, HSG may be comm puted and plotted for a variety of m values. If the diversity profiles of two communities cross, the communities cannot be intrinsically ordered according to diversity. Additional information on diversity analysis may be found in the references following. Ecological studies using diversity profiles include refs. 2, 11, and 8 (this last study refers to the intrinsic diversity profile as a ‘‘kdominance curve’’). In ref. 21 the β , Sβ , and
6546
PROGRESSIVE CENSORING SCHEMES
HSG families are used to explore relationm ships between genetic variation and species diversity. In ref. 19 the β family is applied to forestry. In ref. 17 estimation for diversity index families is studied. In ref. 18 a diversity ordering based on the concept of majorization is proposed. It turns out to be equivalent to the intrinsic diversity ordering (see ref. 12). In ref. 5 the log series distribution is employed as a model of species abundances, and the estimate of the parameter α is used as a diversity index (see also refs. 6 and 7). In ref. 20 it is shown that the parameter α possesses the intrinsic diversity ordering property. Finally, we mention a different approach to diversity analysis taken in ref. 14. See RAO’S AXIOMATIZATION OF DIVERSITY MEASURES. REFERENCES 1. Dennis, B., Patil, G. P., Rossi, O., Stehman, S., and Taillie, C. (1979). In Ecological Diversity in Theory and Practice, J. F. Grassle, G. P. Patil, W. Smith, and C. Taillie, eds. International Co-operative Publishing House, Fairland, Md. (Extensive bibliography of applications of diversity in ecology.) 2. Dennis, B., Patil, G. P., and Rossi, O. (1979). In Environmental Biomonitoring, Assessment, Prediction, and Management, J. Cairns, G. P. Patil, and W. E. Waters, eds. International Co-operative Publishing House, Fairland, Md. 3. Hill, M. O. (1973). Ecology, 54, 427–432. 4. Hurlbert, S. H. (1971). Ecology, 52, 577–586. 5. Kempton, R. A. and Taylor, L. R. (1974). J. Anim. Ecol., 43, 381–399. 6. Kempton, R. A. and Taylor, L. R. (1976). Nature (Lond.), 262, 818–820. 7. Kempton, R. A. and Wedderburn, R. (1978). Biometrics, 34, 25–37. 8. Lambshead, P. J. D., Platt, H. M. and Shaw, K. M. (1983). J. Nat. Hist., 17, 859–874. 9. Patil, G. P. and Taillie, C. (1976). In Proc. 9th Int. Biom. Conf., Vol. II. The Biometric Society, Washington, D.C. 10. Patil, G. P. and Taillie, C. (1979). In Ecological Diversity in Theory and Practice, J. F. Grassle, G. P. Patil, W. Smith, and C. Taillie, eds. International Co-operative Publishing House, Fairland, Md. 11. Patil, G. P. and Taillie, C. (1979). In Contemporary Quantitative Ecology and Related
Econometrics, G. P. Patil and M. Rosenzweig, eds. International Co-operative Publishing House, Fairland, Md. 12. Patil, G. P. and Taillie, C. (1982). J. Amer. Statist. Ass., 77, 548–561. (The key reference on the intrinsic diversity ordering and various diversity index families. Discussion follows, pp. 561–567.) 13. Pielou, E. C. (1975). Wiley, New York.
Ecological
Diversity.
14. Rao, C. R. (1982). Theor. Popul. Biol., 21, 24–43. 15. R´enyi, A. (1961). Proc. 6th Berkeley Symp. Math. Statist. Prob., Vol. 1. University of California Press, Berkeley, CA, pp. 547–561. 16. Smith, W. and Grassle, J. F. (1977). Biometrics, 33, 283–292. 17. Smith, W., Grassle, J. F., and Kravitz, D. (1979). In Ecological Diversity in Theory and Practice, J. F. Grassle, G. P. Patil, W. Smith, and C. Taillie, eds. International Co-operative Publishing House, Fairland, Md. 18. Solomon, D. (1979). In Ecological Diversity in Theory and Practice, J. F. Grassle, G. P. Patil, W. Smith, and C. Taillie, eds. International Co-operative Publishing House, Fairland, Md. 19. Swindel, B. F., Conde, L. F., and Smith, J. E. (1984). For. Ecol. Manag., 7. 20. Taillie, C. (1977). The Mathematical Statistics of Diversity and Abundance. Ph.D. thesis, Pennsylvania State University. 21. Taylor, C. E. and Condra, C. (1979). In Ecological Diversity in Theory and Practice, J. F. Grassle, G. P. Patil, W. Smith, and C. Taillie, eds. International Co-operative Publishing House, Fairland, Md. See also DIVERSITY INDICES and ECOLOGICAL STATISTICS.
B. DENNIS G. P. PATIL
PROGRESSIVE CENSORING SCHEMES In clinical trials∗ , survival analysis∗ , followup∗ studies, and in some problems in reliability∗ , some incompleteness and/or incompatibility of data may arise due to nonsimultaneous entry of the units into the (experimental) scheme, possible withdrawal of the units from the scheme, and more typically, due to intended termination of the
PROGRESSIVE CENSORING SCHEMES
study before all the responses become available. There may be additional complications due to incorporation of interim or repeated analysis schemes for the accumulating data (over the tenure of the study period). For example, in a clinical trial [designed to study the comparative effectiveness or performance of two (or more) drugs (or treatments) relating to a common response] yielding data on the times to failure (response) for the different units (subjects) under study, we may encounter a variety of complications due to one or more of the following factors (these call for special attention for a valid and efficient statistical analysis of the clinical trial data): 1. Censoring/truncation. Due to limitations of time, cost of experimentation, and other considerations, the clinical trial may be planned so that after a prefixed period of time, the trial may be terminated and a statistical analysis be performed on the (partial) data accumulated during this tenure of the experiment. This may be termed a truncated or Type I censored experiment. In some other situations, it may be decided to curtail experimentation after a prefixed number (or proportion) of responses becomes available. This may be termed a Type II censoring scheme. Note that in a Type I censoring scheme, the duration of the trial is prefixed, but the number of responses occurring within this period is random. In a Type II censoring scheme, this number is fixed in advance, but the duration of the trial is a random variable. In either case, at the termination of the experimentation, there may still be some surviving units whose failure times are not totally known (these are, however, known to be larger than the censoring times). These censored observations introduce some incompleteness in the response data. 2. Staggering entry. The subjects (or units) may not all be available at the beginning of experimentation, so that they may enter (possibly in batches) into the scheme at possibly different points of time. If t0 is the time point at which the study begins and t∗ the time point at which the experimentation is stopped, then for an entry at a time point t( t0 ), the exposure time may be defined as max{0, t∗ − t}. Thus a staggering entry plan for the units results in (possibly) different exposure times for these units
6547
(under a given censoring scheme), and this introduces unequal censoring patterns. In most of the cases, this entry plan may be random in nature (e.g., patients arriving in a clinic with a specific disease), so that one would have a random censoring pattern for a staggering entry plan. 3. Withdrawal. Some of the subjects may have failures due to causes other than the specific one under study, and hence, their failure times may not throw much light on the particular cause one wants to investigate. Also, due to migration or lack of interest in the participation, other subjects may even drop out of the scheme (during the experimentation). These may be regarded as withdrawals. In this setup, withdrawals may be regarded as random, although in some other plans, planned withdrawals may also be considered as a part of the experimentation protocol. For models incorporating various withdrawal patterns (random or nonrandom in nature), some information is lost due to the withdrawal and the statistical analysis scheme for the trial may become more complicated and less efficient, too. 4. Monitoring of clinical trials. In view of the fact that clinical trials involve human subjects, based on medical ethics there may be a need to monitor experimentation from the very beginning: If any significant difference among the treatments (in relation to the responses) is detected at any intermediate stage of the trial, the trial may be stopped at that point and the surviving subjects switched to the better treatment for their benefit. From the statistical point of view, this monitoring of the trial involves interim or repeated statistical analysis on accumulating data, where in a followup study, the setup of the classical sequential analysis∗ may not apply. For a proper interpretation of progressive censoring schemes (PCSs), all the concepts listed under factors 1 to 4 need to be incorporated. The term PCS has been used in the statistical literature in more than one context. Earlier uses generally refer to a (fixed) single-point censoring scheme when there is a staggering entry plan for the units, so that one essentially has an unequal pattern of censoring (see Nelson [6]). Thus, if we have n( 1) units entering into the scheme at time
6548
PROGRESSIVE CENSORING SCHEMES
points t1 , . . . , tn , respectively (where t0 t1 · · · tn ), t0 being the starting point of the study, and if t∗ be any (fixed) censoring point, then, at that time point the exposure times for these n units are t∗ − t1 , . . . , t∗ − tn , respectively (assuming that t∗ tn ). For these units and under the fixed-point censoring scheme, one has to work with the minimum of the actual failure time and the corresponding exposure time, along with the information whether this corresponds to a failure or a censored event. When t1 , . . . , tn are random variables, this may also be termed random censoring. The relevance of the term progressive censoring lies in the progressive entry of the units resulting in the unequal censoring pattern for them. In the context of life testing∗ , the term progressive censoring has also been used for a scheme to further reduce the testing time (see Klein and Basu [4]). For n items under life testing, consider a set T1 , . . . , Tm of predetermined censoring times. At time Tj , a fixed number cj ( 0) of the items are removed from the test, for i = 1, . . . , m − 1; at time Tm either a fixed number cm are removed from the test or the test is terminated with some units still surviving. Instead of predetermined time points T1 , . . . , Tm , one may also conceive of predetermined numbers r1 , . . . , rm such that at the rj th failure, an additional number cj of units may be removed and testing be continued, for j = 1, . . . , m, the modification for j = m being the same as in the case of Type I censoring. In the context of clinical trials with monitoring in mind, a comprehensive definition of progressive censoring schemes, due to Chatterjee and Sen [1] may be given. This extended definition of PCS incorporates statistical monitoring of clinical trials (i.e., factor 4) in a very natural way where various combinations of 1, 2, and 3 may be blended. For simplicity consider first simultaneous entry without any withdrawal. Here n( 1) units entering into the scheme at a common point of time t0 are followed through time and their failure times (along with other tagging variables) are recorded in order as they occur. Thus at any time point t(> t0 ), one has a composite picture of the failures occurring during (t0 , t], the censored observations [having failure times in (t, ∞)], and other design and
concomitant variables. In a fixed-point censoring scheme, at some t∗ (> t0 ), where t∗ may or may not be random depending on Type II or Type I censoring, based on the picture at t∗ , a statistical analysis of the survival data is to be made and conclusions are to be drawn therefrom. In most practical problems, unless the survival distribution is at least roughly known, a fixed-point censoring scheme may lead to considerable loss of efficiency relative to cost. This drawback is particularly felt in exploratory studies. A too early termination of the experiment may lead to an inadequately small set of observations and thus increases the risk of an incorrect decision, while unnecessary prolongation may lead to loss of valuable time with practically no extra gain in efficiency. Also, in a life-testing situation, continuous monitoring is usually adopted. As such, it seems unrealistic to wait until a given time period is over or a given number of failures has occurred, and then to draw inferences from the study. Naturally, one would like to review the results as the experiment progresses, and thereby have the scope for terminating the experiment at an early stage depending on the accumulated evidence from the observations. Such a scheme is termed a progressive censoring scheme. Here the censoring is made progressively over time; this flexibility would increase the efficiency relative to cost and would be particularly relevant in long-term studies of an exploratory nature. Although in a PCS, monitoring is done continuously, in many studies (particularly, in nonparametric ones) the picture may change only at the failure points, so that it amounts to reviewing statistically the picture at successive failure points. In this simultaneous entry plan, in a PCS, withdrawals can easily be accommodated by considering the response variable as the minimum of the failure time and the withdrawal time (along with the tagging variable whether it is a failure or censored observation). PCS may become somewhat more complicated for a staggering entry plan. Here monitoring may commence before all the subjects have entered into the scheme, and failures may also occur among the entries already in hand. Thus if t0 is the time point at which the study begins, and if t1 , . . . , tn
PROGRESSIVELY CENSORED DATA ANALYSIS
are the points of entries of the n units in the scheme, then at any time point t (not max{t1 , . . . , tn }) we have n(t) entries, where n(t) is nondecreasing in t and n(t) n, t t0 ; for t max(t1 , . . . , tn ), n(t) = n. Further, at the time point t, for the n(t) units having points of entries prior to t, the exposure times are possibly different. This picture is similar to the first interpretation of PCS, sketched earlier, but with sample size n(t) instead of n. The picture in the extended PCS is then obtained by varying t progressively over t t0 ; if at any t > t0 , a statistically significant difference among the treatments is detected (based on the picture available at that time), the trial is stopped at that time along with the rejection of the null hypothesis of equality of treatments. Otherwise, the monitoring continues. In this setup, an upper bound for the duration of the trial may also be set in advance. Also, withdrawals may be accommodated in a manner similar to that in the nonstaggering entry plan. For details of this staggering entry PCS, see Majumdar and Sen [5] and Sinha and Sen [8,9]. Statistical analysis for PCS opened up a new area of active research. Although technically allied to classical sequential analysis∗ , lack of independence of the increments over time as well as the lack of stationarity of the related stochastic processes∗ make it somewhat difficult to apply the classical theory in PCS. Some of these results are discussed in Sen [7] and Tsiatis [10].
REFERENCES 1. Chatterjee, S. K. and Sen, P. K. (1973). Calcutta Statist. Ass. Bull., 22, 13–50. 2. Elandt-Johnson, R. C. and Johnson, N. L. (1980). Survival Models and Data Analysis. Wiley, New York. 3. Gross, A. J. and Clark, V. A. (1975). Survival Distributions: Reliability Applications in the Biomedical Sciences. Wiley, New York. 4. Klein, J. P. and Basu, A. P. (1981). Commun. Statist. A, 10, 2073–2100. 5. Majumdar, H. and Sen, P. K. (1978). Commun. Statist. A, 7, 349–371. 6. Nelson, W. A. 945–966.
(1972).
Technometrics,
14,
6549
7. Sen, P. K. (1981). Sequential Nonparametrics: Invariance Principles and Statistical Inference. Wiley, New York. 8. Sinha, A. N. and Sen, P. K. (1982). Sankhya¯ B, 44, 1–18. 9. Sinha, A. N. and Sen, P. K. (1983). Proc. Golden Jubilee Conf. Statist., Indian Statistical Institute, Calcutta. 10. Tsiatis, A. A. 311–315.
(1981).
Biometrika,
68,
See also CENSORED DATA; CENSORING; CLINICAL TRIALS —II; PROGRESSIVELY CENSORED DATA ANALYSIS; and SURVIVAL ANALYSIS.
P. K. SEN
PROGRESSIVELY CENSORED DATA ANALYSIS In the entry PROGRESSIVE CENSORING SCHEMES, some statistical problems associated with progressive censoring schemes (PCS) were presented. Valid and efficient statistical analysis procedures of such progressively censored data are considered here. Consider first a staggered entry but fixedpoint censoring scheme (e.g., Nelson [18]), where n items enter at time points t1 t2 · · · tn , respectively, and the study is terminated at a point t∗ tn . Thus the effective exposure times for the n units are Ti = t∗ − ti , i = 1, . . . , n. If Xi stands for the length of life (i.e., failure time) of the ith unit (from the entry point), then this failure occurs during the tenure of the study if ti + Xi t∗ , and is otherwise censored. Thus, if Xi0 = min(Xi , Ti ) and δi = 1 or 0 according as Xi0 is equal to Xi or Ti , for i = 1, . . . , n, then the observable random vectors are (X10 , δ1 ), . . . , (Xn0 , δn ).
(1)
Note that if the entry points (ti ) are stochastic, then so are the Ti , so that we have a random censoring scheme; generally, in this context, we assume that Ti are independent and identically distributed random variables with a distribution function G, defined on [0, ∞). For a preplanned entry plan, the Ti are nonstochastic, and we have an unequal censoring plan. In either case, when the distributions of the Xi are of specified forms (involving some unknown parameters), and
6550
PROGRESSIVELY CENSORED DATA ANALYSIS
the same is the case with G, then the likelihood function for the observable random elements in (1) can be used to estimate these unknown parameters and to make suitable tests of significance. These parametric procedures are discussed in detail in books by Elandt-Johnson and Johnson [9], Kalbfleisch and Prentice [13], Gross and Clarke [12], and Lawless [15]. In the nonparametric case (typically in the context of testing the equality of two or more survival functions) Gehan [11], Efron [8], Peto and Peto [19], and others have advocated the use of some simple rank tests, while Breslow and Crowley [2] and others have incorporated the product limit (PL) or the socalled Kaplan-Meier∗ [14] estimators of the individual survival functions in the formulation of the test procedures. Under random censoring, if in (1) the information contained in the δi is ignored, the usual nonparametric tests based on the Xi0 remain applicable, although they may entail some loss of efficiency due to this sacrifice of information. Sen [24] has studied conditions under which such rank tests are locally optimal. In this context the proportional hazard (PH) model of Cox [5] has been used extensively (see PROPORTIONAL HAZARDS MODEL, COX’S). Under this model, the hazard function may be quite arbitrary though its dependence on the covariates is of a given parametric form. Thus such models can at best be regarded as quasinonparametric. Cox [5,6] has advocated the use of a partial likelihood∗ function which enables one to employ the classical parametric approach in a general setup. This has been followed up more thoroughly by a host of workers (see Tsiatis [31], Slud [30], Sen [22], Anderson and Gill [1], where other references are cited, and a conference proceedings on survival analysis, edited by Crowley and Johnson [7]). Let us next consider the second interpretation of PCS, due to Cohen [4] and others. Here also, for Type I or Type II censoring schemes, in the parametric case, the likelihood function can be written explicitly. The usual maximum likelihood∗ estimators and likelihood ratio tests∗ may be prescribed in the usual fashion. For exponential∗ , normal∗ , lognormal∗ , and logistic∗ distributions, such procedures are due to Cohen [4] and Gajjar
and Khatri [10], among others. In the nonparametric case, this PCS permits the permutational invariance to generate distributionfree∗ tests. However, closed expressions for the test statistics may be difficult to prescribe. Let us proceed to the broad interpretation of PCS, due to Chatterjee and Sen [3], and consider first nonstaggered entry plans. Essentially, we encounter here a timesequential problem which may be posed as follows. For n items under life testing, corresponding to the actual failure times X1 , . . . , Xn , the observable random vectors are (Zi , Si )i = 1, . . . , n, where Z1 < · · · < Zn stand for the order statistics∗ corresponding to the Xi , and Sn = (S1 , . . . , Sn ) is the vector of antiranks (i.e., XSi = Zi , i = 1, . . . , n). As we monitor the experiment from the beginning, we observe the failure points Z1 , . . . , Zn in order. Thus at the kth failure point Zk , the observed data relate to the set {(Z1 , S1 ), . . . , (Zk , Sk )} for k = 1, . . . , n.
(2)
Unfortunately, for different i, the vectors (Zi , Si ) are neither stochastically independent nor marginally identically distributed. Hence, if one attempts to adapt some sequential procedure based on the partial set in (2), difficulties may arise due to non-homogeneity and lack of independence of the increments. On the other hand, the PCS relates to looking at the data at each failure point (with a view to rejecting the null hypothesis if the outcome at that point so indicates), so that one essentially encounters a repeated significance testing problem on accumulating data where the increments are neither independent nor homogeneous. If we denote the likelihood function of the set of observed random variables {(Z1 , S1 ), . . . , (Zk , Sk )} by pnk (·; θ), where θ stands for the parameters associated with the underlying probability law, then the logarithmic derivative of pnk (·; θ ) (with respect to θ ) forms a martingale∗ array (for every n and k n); see Sen [21] and Sen and Tsong [25]. Based on this martingale characterization, one may then construct a random function (stochastic process∗ ) from this
PROGRESSIVELY CENSORED DATA ANALYSIS
partial sequence of the derivatives of the loglikelihood function (at the successive failure points). Under fairly general regularity conditions, such a process weakly converges to a Wiener process (in the single-parameter case) or a Bessel process (in the multiparameter case). This weak convergence result enables one to use the distribution theory of suitable functionals (such as the maximum or the maximum absolute value) of the Wiener (or Bessel) process to provide largesample approximations for the critical levels of the test statistics (in the PCS) based on the partial sequence in (2). As an illustration, consider the simple model where θ is real valued (e.g., Sen [21]). Suppose that we want to test the null hypothesis H0 : θ = θ0 against one-or twosided alternatives. Let λ0nk = (d/dθ ) log pnk (·, θ )|θ =θ0 , 0 = Eθ0 (λ02 Jnk nk ), 0 = 0). Also, let k = 0, 1, . . . , n (where λ0n0 = Jn0 0 0 /Jnn , tnk = Jnk
0 −1/2 Wn (tnk ) = λ0nk {Jnn } ,
k = 0, 1, . . . , n, and by linear interpolation we complete the definition of Wn = {Wn (t) : 0 t 1}. Then under H0 and some mild regularity conditions, as n increases, sup{Wn (t) : 0 t 1} [or sup{|Wn (t)| : 0 t 1}] converges in distribution to D+ = sup{W(t) : 0 t 1} [or D = sup{|W(t)| : 0 t 1}], where W = {W(t) : 0 t 1} is a standard Wiener process on (0, 1). Hence if D+ α (or Dα ) is the upper 100α% point of the distribution of D+ (or D), then we have for large n, P{ sup Wn (t) > D+ α |H0 } → α, 0t1
P{ sup |Wn (t)| > Dα |H0 } → α.
(3)
0t1
Operationally, at each failure point Zk , we 0 )−1/2 λ0 and if it lies above D+ compute (Jnn α nk (for one-sided alternatives) or beyond ±Dα (for the two-sided case), we stop at that point along with the rejection of the null hypothesis; if not, we continue the study until the next failure and repeat the test at that point. By virtue of (3), this repeated significance testing∗ scheme has the asymptotic level of significance α. On the other hand, if the λ0nk
6551
are tested repeatedly against their marginal critical levels, the overall significance level may be much higher than the nominal significance level for each of these marginal tests. In the nonparametric case, in the context of the simple regression model containing the two-sample model as a special case, Chatterjee and Sen [3] spotted the same martingale property of a suitable version of linear rank statistics, censored at the successive failure points. Let us denote these censored linear rank statistics by Tnk , k = 0, 1, . . . , n (where Tn0 = 0) and the corresponding array of variances (under the null hypothesis H0 of the equality of all the n distributions) by Vnk , k = 0, . . . , n. Defining tnk = Vnk /Vnn , k = 0, . . . , n, 1/2 and Wn (tnk ) = Tnk /Vnn , k = 0, . . . , n, we complete the definition of Wn = {Wn (t) : 0 t 1} by linear interpolation. As in the parametric case (3) holds, so that under a PCS the overall significance level of the repeated significance test based on the Tnk will be close to α, for large n. Under H0 the Tnk are jointly distribution-free, so that for small values of n, the exact critical value for this procedure may be obtained by direct enumeration. But these computations become prohibitively laborious, so that for large n, one would naturally use (3) as a handy tool. Majumdar and Sen [17] extended the Chatterjee–Sen [3] results to the case of a multiple regression∗ model (containing the several sample model as a special case). They were able to show that for the progressively censored rank test statistics, some Bessel process approximations hold. For local alternatives, asymptotic power properties of these rank tests for the simple and multiple regression models were also studied by Chatterjee and Sen [3] and Majumdar and Sen [16]. A detailed discussion of these results is given in Sen [23]. Instead of using linear rank statistics, one may also look at the empirical distributions for the different samples (or more generally, a weighted empirical process) and employ them for testing purposes. Note that the classical Kolmogorov–Smirnov∗ statistic is usable in a PCS for testing the identity of the two underlying distributions. Sinha and Sen [26,27] have studied this problem in detail and cited additional references. Let us finally consider the staggered entry plan. As in the case of random or unequal
6552
PROGRESSIVELY CENSORED DATA ANALYSIS
censoring (treated earlier), here also the entry pattern may or may not be stochastic in nature. Although analysis based on a specific entry pattern may be made using similar martingale characterizations, one might like to adopt some procedure which remains robust against departures from a specified pattern. With this objective in mind, Sen [20] considered a two-dimensional functional central limit theorem∗ for censored linear rank statistics (where the sample size n is allowed to vary between 1 and N and the failure numbers k between 1 and n). This theorem provides a Brownian sheet approximation for the staggered entry censored linear rank statistics. This result was used by Majumdar and Sen [17] to formulate some (conservative) tests for the two-sample as well as the simple regression model, for the staggered entry PCS; these tests remain insensitive to the entry pattern. Sinha and Sen [28,29] considered the case of weighted empirical processes∗ in the staggered entry plans, and developed the tied-down Brownian sheet approximations for these processes. Their tests are also somewhat conservative in nature (due to the fact that the entry pattern is not assumed to be precisely known in advance). Similar Brownian sheet approximations work out well for the Cox [5] proportional hazard model in the staggered entry plan as well as for the parametric likelihood ratio statistics under fairly general regularity conditions. Thus we may conclude that for the PCS, where independence, homogeneity and simultaneous entry plan may not hold, the basic tool for data analysis is to incorporate martingale theory in the formulation of suitable stochastic processes converging in distribution to appropriate Gaussian ones (in a oneor more-dimensional time parameter), and to employ the distribution theory for these Gaussian processes∗ to provide large-sample approximations for the specific ones under study. This approach is dealt with elaborately in Sen [23, Chap. 11].
2. Breslow, N. and Crowley, J. (1974). Ann. Statist., 2, 437–453. 3. Chatterjee, S. K. and Sen, P. K. (1973). Calcutta Statist. Ass. Bull., 22, 13–50. 4. Cohen, A. C. 327–339.
(1963).
Technometrics,
5,
5. Cox, D. R. (1972). J. R. Statist. Soc. B, 34, 187–220. 6. Cox, D. R. (1975). Biometrika, 62, 269–276. 7. Crowley, J. and Johnson, R. A., eds. (1982). Survival Analysis. IMS Lecture Notes–Monographs, Vol. 2. Inst. Math. Statist., Hayward, CA. 8. Efron, B. (1967). Proc. 5th Berkeley Symp. Math. Statist. Prob., Vol. 4. University of California Press, Berkeley, CA, pp. 831–853. 9. Elandt-Johnson, R. C. and Johnson, N. L. (1980). Survival Models and Data Analysis. Wiley, New York. 10. Gajjar, A. V. and Khatri, C. G. (1969). Technometrics, 11, 793–803. 11. Gehan, E. A. (1965). Biometrika, 52, 650–652. 12. Gross, A. J. and Clark, V. A. (1975). Survival Distributions: Reliability Applications in the Biomedical Sciences. Wiley, New York. 13. Kalbfleisch, J. D. and Prentice, R. L. (1980). The Statistical Analysis of Failure Time Data. Wiley, New York. 14. Kaplan, E. L. and Meier, P. (1958). J. Amer. Statist. Ass., 53, 457–481. 15. Lawless, J. F. (1982). Statistical Models and Methods for Lifetime Data. Wiley, New York. 16. Majumdar, H. and Sen, P. K. J. Multivariate Anal., 8, 73–95.
(1978).
17. Majumdar, H. and Sen, P. K. (1978). Commun. Statist. A, 7, 349–371. 18. Nelson, W. 945–965.
(1972).
Technometrics,
14,
19. Peto, R. and Peto, J. (1972). J. R. Statist. Soc. A, 135, 186–206. 20. Sen, P. K. (1976). Ann. Prob., 4, 13–26. 21. Sen, P. K. (1976). Ann. Statist., 4, 1247–1257. 22. Sen, P. K. (1981). Ann. Statist., 9, 109–121. 23. Sen, P. K. (1981). Sequential Nonparametrics: Invariance Principles and Statistical Inference. Wiley, New York. 24. Sen, P. K. (1984). J. Statist. Plan. Infer., 9, 355–366.
REFERENCES 1. Anderson, P. K. and Gill, R. D. (1982). Ann. Statist., 10, 1100–1120.
25. Sen, P. K. and Tsong, Y. (1981). Metrika, 28, 165–177. 26. Sinha, A. N. and Sen, P. K. (1979). Commun. Statist. A, 8, 871–898.
PROJECTION PLOTS 27. Sinha, A. N. and Sen, P. K. (1979). Calcutta Statist. Ass. Bull., 28, 57–82. 28. Sinha, A. N. and Sen, P. K. (1982). Sankhyn¯ B, 44, 1–18. 29. Sinha, A. N. and Sen, P. K. (1983). Proc. Golden Jubilee Conf. Statist., Indian Statistical Institute, Calcutta. 30. Slud, E. V. (1982). Biometrika, 65, 547–552.
6553
of X form an orthonormal basis for X , then X X = Ir and P = XX . Succinctly, the projection of y onto X is Py for all vectors y in En . If dim(X ) < r, and x1 , . . . , xr span X but are linearly dependent, then the uniqueness property still holds, and P = X(X X)− X ,
31. Tsiatis, A. (1981). Ann. Statist., 9, 93–108. See also CENSORED DATA; GAUSSIAN PROCESSES; KAPLAN –MEIER ESTIMATOR; LINEAR RANK TESTS; PROGRESSIVE CENSORING SCHEMES; PROPORTIONAL HAZARDS MODEL, COX’S; and SURVIVAL ANALYSIS.
P. K. SEN
PROJECTION MATRICES Let X be a vector subspace of n-dimensional Euclidean space En , and let y be any vector in En . Then the vector yx is the projection or orthogonal projection of y on X if a) yx ∈ X , b) x (y − yx ) = 0 for every vector x in X . For y = 0, its projection yx on X is unique, and the vector y − yx lies in the subspace X 1 that is orthogonal to X . An equivalent definition is that every vector y in En can be expressed uniquely via y = yx + (y − yx ) where yx ∈ X is the projection of y on X and (y − yn ) ∈ X 1 . As an illustration, when En is threedimensional Euclidean space and X is a two-dimensional plane through the origin, then y − yx is the line perpendicular to X passing through y [1, Sec. 2.6]. Suppose that x1 , x2 , . . . , xr are linearly independent vectors in En and that they span X . Then the projection yx of y onto X can be expressed as yx = X(X X)−1 X y, where the columns of X(n × r) are x1 , . . . , xr . If P = X(X X)−1 X , then P is the projection matrix for the vector space X in En , and P is unique. If the columns
where (X X)− is a generalized inverse∗ of X X [2, Sec. 8.6]. Properties The projection Py of y onto the vector space X minimizes the Euclidean distance [(y − x) (y − x)]1/2 from y to x for all vectors x in X , that is, (y − Py) (y − Py) (y − x) (y − x),
x ∈ X.
The projection matrix P is symmetric and idempotent∗ . In the general linear model∗ , y = Xβ + e,
E(e) = 0,
every least squares∗ solution βˆ for β gives rise to predicted responses yˆ = Xβˆ = X(X X)− X y = Py, where P is the projection matrix for the vector space spanned by the columns of X. P is also known as the hat matrix∗ . REFERENCES 1. Schott, J. R. (1997). Matrix Analysis for Statistics. Wiley, New York. (Secs. 2.5, 2.6.) 2. Searle, S. R. (1982). Matrix Algebra Useful for Statistics. Wiley, New York. (Sec. 8.6.) See also GENERAL LINEAR MODEL and HAT MATRIX.
PROJECTION PLOTS One of the most useful statistical techniques for interpreting bivariate data is the scatter plot∗ . It can help to identify interesting features of the data such as clusters of observations, outlying points, and possible relationships between two variables.
6554
PROJECTION PLOTS
For multivariate data with more than two variables, however, an ordinary scatter plot cannot display the entire data. Finding interesting structure in multivariate data is much more difficult; see also MULTIVARIATE GRAPHICS. There are a variety of ways to modify and enhance scatter plots for use with multivariate data. One possibility is to project the data into two dimensions and then examine a scatter plot constructed using these two dimensions (or equivalently, using these two variables). This is called a projection plot. With multivariate data there are many ways to choose the two dimensions, and projecting into them implies some loss of information from the remaining dimensions. One cannot expect interesting features of the data to show up in all projections, and there may be no single projection that reveals everything important. Some choices—such as looking at scatter plots of all pairs of the original variables—can be made without performing any calculations on the data, while other choices of dimensions can be determined from calculations performed on the specific data set. No single method will always reveal the interesting features of the data, and it is often useful to try several of the projection methods described below. PROJECTIONS ON PRECHOSEN VARIABLES Suppose that the analyst has n values (observations) on each of p variables x1 , x2 , . . . , xp , p 3, and the goal is to find scatter plots showing interesting features of the data. Consider first the ordinary scatter plot using just variables x1 and x2 , which ignores the data values on variables x3 , . . . , xp . Conceptually, the n multivariate observations can be thought of as n points lying in a p-dimensional vector space, with axes corresponding to the variables x1 , x2 , . . . , xp . The (x1 , x2 ) scatter plot corresponds to projecting the n points (orthogonally) into the x1 , x2 plane, which can also be thought of as the plane determined by the dimensions (variables) x1 and x2 . Other scatter plots will correspond to different projections. One way to construct a scatter plot that gives information on three variables rather
than just two is to change the character that is plotted. Instead of having all plotted characters being the same—such as a ∗, ·, or + as is typically done—a plot can be constructed in which the location of the character represents the (x1 , x2 ) values of an observation as usual, but the character plotted is L if the value of x3 is in its lowest quartile, M if x3 is in its middle two quartiles, and U if x3 is in its upper quartile. Clearly, there are many choices of characters for plotting and many ways of dividing the distribution on the x3 variable. What is most useful may depend on the specific application. A related idea which is possible with some plotting devices is to use only one symbol as plotting character, such as a circle, but to make its size vary depending on the value of the x3 variable. This can permit an impression of perspective on x3 ; we view the x1 x2 plane at a particular value of x3 , and points closest to this plane are plotted with the largest characters, while points farther away (in terms of the x3 variable) are plotted smaller. For this to work satisfactorily, the character size will generally have to be some nonlinear function of x3 , so that no points are so big as to completely dominate the plot or so small as to be invisible. Typically, iteration is necessary to produce a satisfying and informative plot. Another way of introducing x3 is to plot the points with different colors, depending on the value of x3 . This idea does not seem to have been used or evaluated very much, possibly because the technology permitting it is still new. There is no reason to examine only a single scatter plot. One way to construct multiple plots is to look at two variables x1 and x2 using different subsets of the observations. Using the ideas behind the preceding three paragraphs, we can first partition the observations using variable x3 , for example, into the four quartiles. Then make four scatter plots of x1 and x2 using each of these four subsets; these plots can be arranged on a page from left to right corresponding to the increase in the background variable x3 . A more complicated version of this idea is to partition the observation using two background variables, x3 and x4 , and then arrange the scatter plots of x1 and x2 for the resulting subsets in a rectangular array corresponding
PROJECTION PLOTS
to the value on x3 and x4 . In effect, this is a scatter plot of scatter plots, incorporating four variables in all. A different approach toward constructing multiple plots is to juxtapose scatter plots involving different variables in some useful way, where now each scatter plot includes different variables but all the observations. Several ways to do this, as well as ways to implement the ideas discussed previously, are described by Tukey and Tukey [7, Chap. 10; Chap. 11, pp. 231–232; Chap. 12, pp. 245–260].
6555
One way to arrange the scatter plots from all pairs of the four variables x1 , x2 , x3 , and x4 is shown in Fig. 1. For this example there appear to be positive relationships between the pairs of variables. None of these six plots indicates, however, that these 50 observations may consist of two separated groups. Now consider Fig. 2, constructed using y1 = (x1 + x2 + x3 + x4 )/2 and y2 = (x1 + x2 − x3 − x4 )/2. This is an orthogonal projection into two dimensions that do not correspond to any of the original four variables; it does suggest that these data may separate nicely
Figure 1.
6556
PROJECTION PLOTS
Figure 2.
into two groups, those with y1 < 2 and those with y1 > 2. This example points out that it can be desirable to view many different projections of the data. Simply viewing projections using each variable marginally does not guarantee finding whatever underlying structure that there may be. [The example was constructed to illustrate this point by using two multivariate normal populations with identity covariance matrices, one centered at (0,0,0,0) and the other at (2,2,2,2).] With more variables the number of scatter plots that could be examined increases greatly. Unfortunately, the chance that a prechosen projection will be sufficiently close to the ‘‘right’’ projection (assuming that there is such a projection that will reveal the structure) decreases dramatically with the number of variables. The result is that when there are a moderate to large number of variables—say, four or more—simply relying on some small number of prechosen projections is not likely to suffice to reveal the structure. Fortunately, calculations are available which can help to find useful projections. PROJECTIONS FROM MULTIVARIATE METHODS Calculations from several of the classical multivariate analysis methods give
projections which can be useful for displaying the data, although obtaining the projections need not be the only or main reason for performing these analyses. These methods, which are interrelated, include component analysis∗ (often called principal component analysis), discriminant analysis∗ , and canonical analysis∗ (canonical correlations). Component analysis can be helpful if we are treating the data as one set of observations measured on p variables. Component analysis gives those projections which preserve interpoint distances, based on distances between all pairs of observations using all p variables, as well as possible. Thus a plot of the observations projected on the first two principal components is often used to give a first, overall, two-dimensional picture of multivariate data. If the data set consists of two or more samples which are assumed Gaussian with equal covariance matrices, and the goal is to find the projections which minimize the error rates of classification, then the appropriate theory is that of linear discriminant analysis. If there are three or more groups, a plot of the data projected on the first two linear discriminant functions can help to understand relationships among the groups. Canonical analysis is designed for the situation of two sets of continuous variables and one underlying population. Again, projecting
PROJECTION PLOTS
the observations on pairs of canonical variates can help to indicate relationships. The preceding methods can all be obtained from classical multivariate normality assumptions, and all involve linear algebra eigenvector calculations. With recent advances in computational power and interest in methods that are not so closely tied to specific distributional and model assumptions, several approaches have been investigated which can give useful projections for interpreting data. The basic idea is to define a criterion that measures to what extent a given projection is ‘‘interesting,’’ and then to use numerical methods to seek those projections that maximize the criterion. Both parts of the problem can be approached in different ways [6,3]. In choosing the criterion we do not want a projection in which the observations have a more-or-less uniform scatter, but instead one in which they are clumped (clotted or clustered). There may be several local maxima to the criterion, corresponding to different sorts of structure in the data. These might all be of interest, so it can be desirable to have numerical procedures which find more than a single local maximum. The use of this idea to fit a multiple regression model was given by Friedman and Stuetzle [2]. Work in a similar spirit has been developed for cluster analysis. In clustering, generally some complicated numerical algorithm has been used to group the observations into clusters; the analyst might then need to examine various plots to interpret the results and to study relationships between and within the clusters. But, as with other multivariate data, it is not clear which projections are useful for studying the possible clusters. Gnanadesikan et al. [4] modified some of the component analysis and discriminant analysis calculations in order to produce projection plots which give as uncluttered a view of the observations and clusters as possible, by finding those projections which separate things maximally. Unlike the methods mentioned earlier in this section, these do not require iterative calculations. Instead of having the computer choose one or a small number of projection plots to examine by maximizing some criterion, a next step is to make even more use of increased
6557
computing power by showing many projection plots successively in real time, generating movie-like effects. This can now be done on certain specially designed interactive computer graphic systems. A specific projection (involving two dimensions) is shown on a CRT screen; then a third dimension is considered and the data are rotated within this subspace. This enables the user to perceive a spatial three-dimensional picture of the data, in terms of these dimensions. The rotations and dimensions can be controlled by the user. The analyst can see, quickly and interactively, many views of the data. An early such system was developed by Tukey et al. [8]; some recent work and ideas are described by Donoho et al. [1]. Such systems have not yet seen wide use, partly because of their expense; if they do become widely available, they could have a tremendous impact on analysis of multivariate data. OTHER RELATED METHODS Several graphical ways to display multivariate data have been devised which do not amount to projection plots. Instead, each multivariate data point is mapped into some appropriate symbol, and the symbols are displayed. Nearby data points should be mapped into symbols that are similar. By comparing the symbols, the analyst may be able to discover interesting relationships among the data points and variables. For Chernoff faces∗ , the symbol is a cartoon face; for Andrews function plots∗ , it is a continuous function defined on the interval (−π , π ). Other symbols that have been used include star-shaped figures and trees and castles [5]. Such methods are good for giving general impressions involving all the variables, but it often seems that specific features and details are more apparent in appropriately chosen projection plots. (It can be difficult, of course, to find the appropriate projection plots.) The biplot∗ is another graphical method, useful for displaying certain kinds of multivariate data. REFERENCES 1. Donoho, D., Huber, P. J., and Thoma, H. M. (1981). In Computer Science and Statistics:
6558
2. 3. 4.
5. 6.
7.
8.
PROJECTION PURSUIT
Proceedings of the 13th Symposium on the Interface, W. F. Eddy, ed. Springer-Verlag, New York, pp. 274–278. Friedman, J. H. and Stuetzle, W. (1981). J. Amer. Statist. Ass., 76, 817–823. Friedman, J. H. and Tukey, J. W. (1974). IEEE Trans. Computers, C-23, 881–890. Gnanadesikan, R., Kettenring, J. R., and Landwehr, J. M. (1982). In Statistics and Probability: Essays in Honor of C. R. Rao, G. Kallianpur, P. R. Krishnaiah, and J. K. Ghosh, eds. North-Holland, New York, pp. 269–280. Kleiner, B. and Hartigan, J. A. (1981). J. Amer. Statist. Ass., 76, 260–269. Kruskal, J. B. (1969). In Statistical Computation, R. C. Milton and J. A. Nelder, eds. Academic Press, New York. Tukey, P. A. and Tukey, J. W. (1981). In Interpreting Multivariate Data, V. Barnett, ed. Wiley, Chichester, England. Tukey, J. W., Friedman, J. H., and Fisherkeller, M. A. (1976). Proc. 4th Int. Conf. Stereol., Gaithersburg, Md.
See also ANDREWS FUNCTION PLOTS; BIPLOTS; CHERNOFF FACES; GRAPHICAL REPRESENTATION, COMPUTER AIDED; GRAPHICAL REPRESENTATION OF DATA; MULTIVARIATE ANALYSIS; MULTIVARIATE GRAPHICS; and TRIPLE SCATTER PLOT.
JAMES M. LANDWEHR
PROJECTION PURSUIT Many data sets are high-dimensional. It has been a common practice to use lowerdimensional linear projections of the data for visual inspection. The lower dimension is usually 1 or 2 (or maybe 3). More precisely, if X 1 , . . . , X n ∈ Rp are p-dimensional data, then for k < p, a k-dimensional linear projection is Z1 , . . . , Zn ∈ Rk where Zi = α T X i for some p × k orthonormal matrix α (so that α T α = I k , the k-dimensional identity matrix). When k = 1, α may be called a direction and the structure of the projected data can be viewed through a histogram∗ ; when k = 2, the structure can be inspected through its scatter plot∗ ; when k = 3, it can be comprehended by spinning a three-dimensional scatter plot. Since there are infinitely many projections from a higher dimension to a lower dimension, it is important to have a technique for pursuing a finite sequence of projections that can reveal the most interesting
structures of the data. The idea of combining both projection and pursuit originates from Kruskal [18] and Switzer [27]. However, the first successful implementation of the idea was by Friedman and Tukey [10], who also suggested the felicitous name projection pursuit (PP). A unified mathematical notion of PP was introduced by Huber [14], which provided the basis for further statistical research in the area. More recent papers include [7,11,22,24,25], and also [2,19,23]. TWO ELEMENTS The two basic elements of projection pursuit are a PP index and a PP algorithm. A PP index I(α) is a measure of how interesting the projection of the data by α is, where I(α) = I(α|X) implicitly depends on the data X = (X 1 , . . . , X n ). The larger the index value, the more interesting is the projection. A PP algorithm is a numerical optimization algorithm that maximizes the index over all possible α. Since there is no perfect optimizer, one hopes that the first several maxima found by a PP algorithm will provide the most interesting projections. Using a PP algorithm to seek a sequence α 1 , α 2 , . . . (which locally maximize a PP index) and then viewing structures by projecting the data to the space spanned by the columns of these α i (i = 1, 2, . . .) may be called automatic PP. This is in contrast to manual PP, in which a dynamic graphical system is employed and the interesting projections are sought by human eyes via spinning the projected data, as in Prim-9 (Tukey et al. [28]) and the grand tour (Asimov [1]). Manual PP depends on experience and great patience in a user. It can be infeasible for a nonexpert or if the data dimension is large. A more practical solution may be to combine automatic and manual PP in a suitable way, as attempted in XGobi (Swayne et al. [26]). This entry only concerns the automatic PP, as it is the PP that is used today. Clearly, nonlinear structures such as clusters, separations, and unexpected shapes are interesting. In general, the phrase ‘‘nonlinear structures’’ refers to those structures which cannot be detected easily by using a sample mean and covariance matrix. These summary
PROJECTION PURSUIT
statistics reveal linear structures such as the location, scales, and correlations of a data set. Thus a reasonable PP index should, in the first place, be location-, scale-, and rotationinvariant, i.e., I(·|sAX + β) = I(·|X)
(1)
for any s > 0, β ∈ Rp , and p × p orthonormal matrix A. This invariance can be achieved by sphering data before implementing PP. The ˜ i from X i satisfy sphered data X 1 ˜ X i = 0, n n
i=1
1 ˜ ˜T X iX i = Ip. n n
(2)
i=1
See Friedman [7] for a simple sphering scheme, and Huber [14] on the importance of sphering. How should one choose a PP index? The interestingness of a projection is given by the value of a PP index, which is a sample version (or estimate) of a distance between the distribution of the projected data and an uninteresting distribution. So, heuristically, the maximum of the index corresponds to the most interesting projection. A natural uninteresting distribution is the normal distribution. Indeed, (Diaconis and Freedman [3]), . . ., a projection will be uninteresting if it is random or unstructured. One standard measure of randomness is entropy∗ . [Thus] a projection is interesting if it has small entropy relative to other projections. . . . If the scale is fixed, maximum entropy∗ is attained by the Gaussian distribution. This suggests . . . a projection is interesting if it is far from Gaussian.
Huber [14], Friedman [7], Jones and Sibson [17], Hall [11], Cook et al. [2], and Li and Cheng [19] all considered PP indices that use the standard normal distribution as the uninteresting distribution. (However, the indices can easily be adapted to other uninteresting distributions.) Example 1. Legendre Index. Assume for the moment that E(X i ) = 0 and Cov[X i ] = I p . (Let k = 1 for simplicity.) Friedman [7] constructed his Legendre index based on the L2 distance between the densities of the projected data point Zi = α T X i and a standard
6559
normal random variable Z0 after the transformation R = 2(Z) − 1. Here (z) is the distribution function of Z0 , and the L2 distance between any two functions f and g is [f (x) − g(x)]2 dx}1/2 . Since polynomials are inexpensive to compute, Friedman expanded the L2 distance by an m-term Legendre polynomial expansion: m
{Eα [aj (R)]}2 ,
(3)
j=1
where Eα is the expectation with respect to the distribution of R for the fixed α, and aj (r) = (j + 1/2)1/2 Lj (r) is the jth orthonormal Legendre polynomial, i.e., 1 otherwise. −1 ai (r)aj (r)dr = 1 if i = j and 0 In practice, E[X i ] and Cov[X i ] are arbitrary and the distribution of X i is unknown. So the m-term Legendre index is simply an estimate ˜ i from X i : of (3) based on the sphered data X F Im (α)
m 2j + 1 = 2 j=1
1 ˜ i) L j (R n
˜ i ) − 1, ˜ i = 2(α T X R
n
2 ,
i=1
(4)
˜ denotes the transformation calcuwhere R lated from sphered data. (When k = 2, a twodimensional Legendre index can be obtained in the same way.) Example 2. Hermite Index. Hall [11] suggested an index differing from Friedman’s in that he used the first m-term Hermite polynomial expansion of the L2 distance between the densities of Zi = α T X i and the standard normal random variable directly. Again, we have taken k = 1 for simplicity. Most PP indices are derived this way: Find an estimate of some measure of the interestingness assuming E[X i ] = 0, Cov[X i ] = I p , and then take the estimate as a PP index ˜ i . In this way, the but with X i replaced by X invariance in (1) is also enforced. However, the sphered data satisfy the constraints (2), which makes the first two terms of Legendre, Hermite, and other polynomial indices much smaller than the other terms (cf. Sun [25], who suggested using 3 m 6 in polynomial indices).
6560
PROJECTION PURSUIT
Sensitivity to a particular nonlinear structure varies from one index to another. The number of terms m and the lower dimension k play nonnegligible roles in the sensitivity, too. There are some comparisons on the sensitivity in Sun [25], based on Friedman and Hall’s indices, and in Cook et al. [2], based on the population version of the indices. More studies of other indices (taking into account the fact that the indices are functions of sphered data) are yet to come. A beginner is advised to try a couple of indices. Since PP is implemented via an algorithm, the algorithm should do what it is supposed to do. Huber [15] contains an excellent discussion on ‘‘goals versus algorithms.’’ Friedman [7] gives a practical PP algorithm for his PP index, in which he uses a clever idea called structure removal; the idea is to transform the data along an ‘‘interesting’’ direction into standard normal data while keeping the structure along the other orthogonal directions unchanged. It has the effect that interesting directions already found will not be rediscovered, as normal structures are uninteresting and will therefore minimize the index. Sun [25] found that the largest of the first p maxima from Friedman’s PP algorithm is often very close to the global maximum, and based on this, she suggested a more exhaustive search algorithm. SIGNIFICANCE LEVEL After an interesting structure is found, it is useful to make a significance test∗ to decide whether the apparent structure is real or just the effect of noise. Sun [24] derived a theoretical approximation for the significance level (or P-value∗ ) associated with Friedman’s onedimensional PP index [7]:
F (α) I0 P max Im α:α T α=1
≈ κ1 ψ1 (nI0 ) + κ2 ψ2 (nI0 ),
(5)
where I0 is the observed value of the index, κ1 and κ2 are constants, ψ1 and ψ2 are incomplete gamma functions depending on p and m, and the probability is found under the null hypothesis that the data are Gaussian (i.e., uninteresting). The formula (5) works well
when nI0 is not too small. Only the first term in (5) should be used if n is not large and I0 is small. So a rule of thumb is to calculate the P-value based on this formula and adjust the value if circumstances require. A P-value like this helps ease Miller’s concern in his discussion of Huber [14]: With reference to this new technology [PP], I feel that there is a gap developing between practice and theory. While interesting shapes, clusters, and separations are now more discoverable in high dimensional data, how do we evaluate whether these are real or merely random variation? [20]
P-values for other indices and higher dimensions (k 2) are still to be studied. Zhang et al. [29] and Li and Cheng [19] are good references for bounds to the P-values associated with indices based on Kolmogorov distance (rather than the L2 distance). Based on the heuristics that a projection is uninteresting if it is random or presents normal structure, a PP index essentially amounts to testing nonnormality (cf. Huber [14]). Nevertheless, there are some subtle differences between PP and a test of multivariate normality∗ . PP aims at finding the most informative multiple views of a multivariate data set. The projections obtained from a good PP procedure usually correspond to the first several largest local maxima of the index. In testing multivariate normality, however, only the largest maximum is considered. With regard to a particular projection found by a PP algorithm, the P-value in (5) gives us an indication of whether the direction is interesting. With regard to the whole process, the P-value may be conservative, as the projection by α may only correspond to local maximum of the index. A by-product of the P-value is that it helps a user decide when to stop a PP algorithm. Consecutive large P-values indicate that there might not be much interesting structure left. In summary, with a PP index, an algorithm, and the aid of P-values, a user may be able to identify and view interesting structures by projecting a data set to some lower dimensional spaces determined by α 1 , α 2 , . . . that are found sequentially by the algorithm which maximizes the index. An artificial
PROJECTION PURSUIT
example serves as an illustration of this point. Example 3. Consider a nine-dimensional data set X 1 , . . . , X 1000 of size 1000, where the fifth coordinate is from an interesting bimodal distribution, say test scores of people from different backgrounds, while the other coordinates are noise, say other irrelevant scores and measurements. Suppose a user does not know the structure of the data and applies the PP procedure. If one of the first several α i found by the PP algorithm is close to e5 ( or − e5 ), a basis vector along the fifth coordinate, then the P-value is small in this direction and large in the other α i . Thus the user would be able to make a correct decision in classifying the data and identifying the structure. APPLICATIONS PP is a powerful tool for seeking nonlinear structures of high-dimensional data. Its ability to overcome the curse of dimensionality∗ by lifting one or two dimensional techniques to high dimensions does not stop with exploratory data analysis∗ . For example, Friedman et al. [9] and Friedman and Stuetzle [8] applied PP in density ∗ estimation∗ , classification , and regression∗ problems (PPD, PPC, and PPR, respectively). More recent applications can be found in [5,19,21] and the references therein. Projection Pursuit Regression In a typical regression model, (X, Y) is an observable pair of random variables from a distribution F, where X ∈ Rp is a pdimensional variable (called a predictor) and Y ∈ R is a response. The goal is to estimate the regression function f (x) = E[Y|X = x], or the conditional expectation of Y given X = x, using a random sample (X 1 , Y1 ), . . . , (X n , Yn ) from F. PPR approximates the regression function f (x) by a finite sum of ridge functions fˆ (x) =
m
gk (α Tk x)
(6)
k=1
where m is often determined adaptively, and the α k are p × k orthonormal matrices.
6561
The PPR model (6) is quite general. It can be used to closely approximate a large class of functions f as m → ∞ with suitable choice of α k and gk . For instance, the following models may be considered as special cases of the PPR model: 1. the simple linear regression: f (x) = α T x with k = p = 1; 2. the single-index model: f (x) = g(aT x) (cf. [12]); model with 3. the neural network∗ T one hidden layer: f (x) = m k=1 gk (ak x), where gk is a known function [usually the sigmoidal function gk (x) = 1/(1 + e−x )]; and p 4. the additive model: f (x) = k=1 gk (xk ), where xk is the kth coordinate of x (cf. [13]). Other standard models also fit into the PPR framework. A mathematical justification for (6) in terms of approximation theory is presented by Diaconis and Shahshahani [4]. A stepwise PPR algorithm can be found in Friedman and Stuetzle [8], where the gk are found by a smoothing operation. Specifically, given g(0) = 0, for j 1, the algorithm iteratively estimates the α j so as to maximize an index and the gj by a low-dimensional nonparametric regression∗ estimate based on the pro(j) (j) (j) (j) , rn ), where Zi jected data (Z1 , r1 ), . . . , (Z(j) j−1 n (j) T T = α j X i and ri = Yi − k=1 gk (α k X i ) are the residuals at the jth step (where an empty sum is to be interpreted as zero), i = 1, . . . , n. This alternating estimation is performed, sequentially, foreach k until the residual sum of squares r2i is less than a predetermined value. A different smoother for gk , or index, or fitting order may be used, yielding a different PPR algorithm. Friedman [6], Johansen and Johnstone [16], and Roosen and Hastie [23] gave various candidates. Projection Pursuit Density Estimation The idea in this application is to approximate a high-dimensional (dimension p) data density f (x) of X ∈ Rp by g(m) (x) = g0 (x)
m i=1
gi (zi ),
6562
PROJECTION PURSUIT
where zi = α Ti x ∈ Rk for a suitable choice of {α i , i = 1, . . . , m}. Here g0 is an uninteresting density or initial model; the gi are low-dimensional densities based on interesting projections of data. Huber [14, p. 458] shows that if g is any approximation to f , then the biggest possible improvement in relative entropy∗ E(f , g) := Ef log{f (X)/g(X)} that can be achieved through replacing g(x) by g∗ (x) = g(x)g1 (α T x) is g1 (α T x) = fα (α T x)/gα (α T x), where Ef is the expectation with respect to f , and fα and gα are marginal densities of Z = α T X under f and g, respectively. Among all choices of α, the largest improvement results from one maximizing E(fα , gα ). A typical g0 is the normal density with mean and covariance estimated by the sample mean and covariance of a random sample X 1 , . . . , X n from f . This suggests a stepwise PPD algorithm as follows. 1. Let g(0) = g0 . 2. Given g(l−1) , find αˆ that maximizes the index I(α) =
n fˆα (Zi ) 1 , log (l−1) ˆ n gα (Zi )
(7)
i=1
where fˆα is a marginal density estimate a denbased on Z1 , . . . , Zn , and gˆ (1−1) α sity estimate based on the projection to α of a Monte Carlo random sample from g(l−1) .
Friedman [7] approximated the relative entropy E(gα , fα )[not E(fα , gα )] by the weighted L2 distance [fα (z) − gα (z)]2 /gα (z)dz, and (Zi ) by a finite Legendre polynofˆα (Zi )/g(l−1) α mial expansion; he hence developed a different PPD algorithm. Note that there is an asymmetry in relative entropy since E(f , g) = E(g, f ). Friedman’s approximation is valid if gα and fα are close, but not necessarily when they are distant. Thus, the justification above from Huber [14] cannot be carried over to this procedure. The justification for the Friedman’s procedure in terms of approximation theory remains an open problem. CONCLUSIONS Projection pursuit seeks interesting structures of a data set through linear projections of the data. A common feature of PP, PPC, PPD, and PPR is to use an algorithm stepwise in searching over low-dimensional projections to maximize a PP index [e.g., (4) and (7)] that is sensitive to a particular structure. Projection pursuit belongs to the class of modern statistical computing techniques. With the development of statistical software∗ , it has become a useful data analysis tool. As with any modern technique, it should be used in conjunction with (and after) some simple preliminary data analysis; this would find simple structures using the summary statistics (e.g., the mean and covariances), perform a preanalytical dimension reduction via some standard procedure, and (for PP in particular) sphere the data. REFERENCES
3. Set g1 (αˆ x) = T
fˆαˆ (αˆ T x) gˆ (l−1) (αˆ T x) αˆ
,
g(l) (x) = g(l−1) (x)gl (αˆ T x). Then repeat steps 2 and 3 until maxα I(α) is less than a preselected small value. Huber [14] also showed that if f is a convolution∗ of some density f and the standard normal density φ then the g(i) obtained above converges uniformly and in L1 norm (integrated absolute deviation) to f .
1. Asimov, D. (1985). The grand tour: a tool for viewing multidimensional data. SIAM J. Sci. and Statist. Comput., 6, 128–143. (Manual PP.) 2. Cook, D. Buja, A., and Cabrera, J. (1993). Projection pursuit indexes based on orthonormal function expansions. J. Comput. and Graphical Statist., 2, 225–250. (Gives global maxima of the population version of three polynomialbased PP indices. There are some open problems here: Is the global maximum unique? What is the global maximum of the indices, and how is this captured in the sample version?)
PROJECTION PURSUIT 3. Diaconis, P. and Freedman, D. (1984). Asymptotics of graphical projection pursuit. Ann. Statist., 12, 793–815. (Motivates what uninteresting projections/distributions are.) 4. Diaconis, P. and Shahshahani, M. (1984). On nonlinear functions of linear combinations. SIAM J. Sci. and Statist. Comput., 5, 175–191. (A piece of theoretical work validating PPR modeling in terms of approximation theory.) 5. Duan, N. (1990). The adjoint projection pursuit regression. J. Amer. Statist. Ass., 85, 1029–1038. (A variant of PP.) 6. Friedman, J. H. (1984). Smart user’s guide, Tech. Rep., Standford University. (A different PPR algorithm from that in ref. [8].) 7. Friedman, J. H. (1987). Exploratory projection pursuit. J. Amer. Statist. Ass., 82, 249–266. (The first of several PP indices based on orthogonal polynomial expansions. The idea of ‘‘structure removal’’ was proposed here.) 8. Friedman, J. H. and Stuetzle, W. (1981). Projection pursuit regression. J. Amer. Statist. Ass., 76, 817–823. (The original work on PPR.) 9. Friedman, J. H., Stuetzle, W., and Schroeder, A. (1984). Projection pursuit density estimation. J. Amer. Statist. Ass., 79, 599–608. (An application of PP to density estimation.) 10. Friedman, J. H. and Tukey, J. (1974). A projection pursuit algorithm for exploratory data analysis. IEEE Trans. Comput., C-23, 881–889. (The first successful PP implementation.) 11. Hall, P. (1989). Polynomial based projection pursuit. Ann. Statist., 17, 589–605. (A generalization from Legendre index [7] to Hermite index.) ¨ 12. Hardle, W., Hall, P., and Ichimura, H. (1993). Optimal smoothing in single-index models. Ann. Statist., 21, 157–178. (Discusses how to choose the smoothing parameter in the singleindex model and its difference from that in PPR.) 13. Hastie, T. and Tibshirani, R. (1987). Generalized additive models: some applications. J. Amer. Statist. Ass., 82, 371–386. 14. Huber, P. (1985). Projection pursuit (with discussion). Ann. Statist., 13, 435–475. (An excellent review article which gives theoretical justifications for some seemingly ad hoc PP ideas.) 15. Huber, P. (1992). Goals versus algorithms. In Data Analysis and Statistical Inference, S.
6563
Schach and G. Trenkler, eds. Josef Eul Publishers, Germany, pp. 567–579. 16. Johansen, S. and Johnstone, I. (1990). Hotelling’s theorem on the volume of tubes: some illustrations in simultaneous inference and data analysis. Ann. Statist., 18, 652–684. (Shows how the powerful tube formula can be applied to complicated problems like PPR.) 17. Jones, M. C. and Sibson, R. (1987). What is projection pursuit? (with discussion). J. Roy. Statist. Soc. A, 150, 1–36. (Another review article, containing some parallel findings to that in [14].) 18. Kruskal, J. B. (1969). Toward a practical method which helps uncover the structure of a set of multivariate observations by finding the linear transformation which optimizes a new ‘‘index of condensation’’. In Statistical Computation, R. Milton and J. Nelder, eds. Academic Press, NewYork, pp. 427–440. (This seems to be the earliest work using PP.) 19. Li, G. and Cheng, P. (1993). Some recent developments in projection pursuit in China. Statist. Sinica, 3, 35–51. 20. Miller, R. G., Jr. (1985). Discussion. Ann. Statist., 13, 510–512. (Referes to Huber [14].) 21. Maller, R. A. (1989). Some consistency results on projection pursuit estimators of location and scale. Can. J. Statist., 17, 81–90. (An interesting application.) 22. Morton, S. C. (1990). Interpretable projection pursuit. Computer Science and Statistics: Proc. Symp. on the Interface. American Statistical Association, Alexandria, Virginia, pp. 470–474. (Provides a different perspective on PP—interpretability.) 23. Roosen, C. and Hastie, T. (1994). Automatic smoothing spline projection pursuit. J. Comput. and Graphical Statist., 3, 235–248. (A PPR algorithm.) 24. Sun, J. (1991). Significance levels in exploratory projection pursuit. Biometrika, 78(4), 759–769. (Gives P-values associated with the Legendre index [7]. The method can be generalized to all polynomial-based PP indices.) 25. Sun, J. (1993). Some practical aspects of exploratory projection pursuit. SIAM J. Sci. and Statist. Comput., 14(1), 68–80. (Contains discussions on choosing PP indices and algorithms.) 26. Swayne, D. F., Cook, D., and Buja, A. (1991). XGobi: Interactive dynamic graphics in the X Window system with a link to S. ASA Proc. Section on Statistical Graphics, pp. 1–8. (An
6564
PROJECTION PURSUIT, NASON’S t -INDEX FOR
attempt to mix manual and automatic PP. A fine graphical tool for viewing/analyzing data.) 27. Switzer, P. (1970). Numerical classification. In Geostatistics. Plenum, New York. (Another historical reference.) 28. Tukey, J. W., Friedman, J. H., and Fisherkeller, M. A. (1988). Prim-9, an interactive multidimensional data display and analysis system. In Dynamic Graphics for Statistics, W. S. Cleveland and M. E. McGill, eds. Wadsworth, New York, pp. 91–110. (Manual PP.) 29. Zhang, J., Zhu, L., and Cheng, P. (1993). Exponential bounds for the uniform deviation of a kind of empirical process, II. J. Multivariate Anal., 47, 250–268. (A theoretical bound for the P-value associated with a Kolmogorov-type index.)
a simple quadratic polynomial with no linear term, where Dn is a constant depending only on n. (Recall that the F-divergence from a probability density function f defined on Euclidean p-space Rp to a probability density function g defined on Rp is FF (f |g) =
f (x)F[f (x)/g(x)]dx,
where F is a function from [0, ∞) into R, which is continuous and convex on [0, ∞), finite on (0, ∞), and strictly convex at some point x, 0 < x < ∞.) REFERENCES
See also COMPUTER-INTENSIVE STATISTICAL METHODS; EXPLORATORY DATA ANALYSIS; GENERALIZED ADDITIVE MODELS; and NONPARAMETRIC REGRESSION.
JIAYANG SUN
PROJECTION PURSUIT, NASON’S t -INDEX FOR [This entry presupposes some familiarity with the content of the entry PROJECTION PURSUIT.] Nason [1,2] points out that the functional J(f ) = − f (x)1−2/(n+1) dx is minimized among all sphered distributions* by the sphered Student’s t-density tn (·) with n degrees of freedom given by
1 (n + 1) tn (x) = [(n − 2)π ]−1/2 2 &−(n+1)/2 & 1 x2 , ( n) 1 + 2 n−2 and proposes the t-index of projection pursuit J(f ) − J(tn )( 0), which can also be represented as the sum of two F-divergences (defined presently) for spherical densities f . This is connected with the property of tn (x): tn (x)−2/(n+1) = Dn [1 + (n − 2)−1 x2 ],
1. Nason, G. P. (1992). Design and Choice of Projection Indices. Ph.D. Dissertation. University of Bath, Bath, U.K. 2. Nason, G. P. (2001). Robust projection indices. J. R. Stat. Soc. B, 63, 557–567. See also J-DIVERGENCES AND RELATED CONCEPTS and PROJECTION PURSUIT.
PROJECTION PURSUIT, YENYUKOV’S INDEX Yenyukov [1,2] proposes indices for projection pursuit* based on the observation that in a planar region A, homogeneity of data can be interpreted as either uniformity or complete spatial randomness*. Thus, if A denotes the area of A, a homogeneous planar Poisson process is characterized by the properties that the number N(A) of events in any region A follows a Poisson distribution * with mean λA and that for N(A) fixed, the events in A constitute of a random sample from the uniform distribution over the region A. Under these assumptions, one of Yenyukov’s projection pursuit indices is Ypp = D/d, where D is the mean of the interpoint distances and d the average nearest neighbor* distance. For projections with fractal∗ and
PROPHET INEQUALITIES
cluster structures, the index should be large; it should be small for homogeneous structures. No practical examples have yet been provided. Other Yenyukov indices [2] are based on the inverse studentized range* and on normal scores*. REFERENCES 1. Yenyukov, I. S. (1988). ‘‘Detecting Structures by Means of Projection Pursuit’’. Proceedings of COMPSTAT 1988. Physica-Verlag, Heidelberg, Vienna, pp. 47–58. 2. Yenyukov, I. S. (1989). ‘‘Indices for Projection Pursuit’’. In Data Analysis Learning Symbolic and Numeric Knowledge, Diday, ed. Nova Science Publishers, New York. See also NEAREST-NEIGHBOR METHODS and PROJECTION PURSUIT.
PROPAGATION OF ERROR. See ERROR ANALYSIS; LAWS CONCEPT
OF
ERROR —I: DEVELOPMENT
OF
THE
PROPENSITY SCORE This is a parameter that describes one aspect of the organization of a clinical trial∗ . It is an estimate of the conditional probability of assignment to a particular treatment given a vector of values of concomitant variables∗ . Rosenbaum [1] and Rosenbaum and Rubin [2,3] discuss the use of propensity scores in correcting the analysis of observational∗ data to allow for possible biases in the assignment of individuals to treatments as indicated by the propensity scores.
6565
PROPHET INEQUALITIES Prophet inequalities are related to optimal stopping rules∗ . They compare the best return of an optimal player (the ‘‘mathematician’’, ‘‘statistician,’’ or ‘‘mortal’’) to that of a ‘‘prophet,’’ who can base his decision also on the future. Let n 2 and X n = (X1 , . . . , Xn ), X = (X1 , X2 , . . .), where the X’s have finite expectations and the distribution of X n (in the finite case) or of X (in the infinite case) is completely known to the mortal (and to the prophet). Denote by V(X n ) [V(X), respectively] the value of the sequence to the optimal player, i.e., V(X n ) = sup EXt , t
where t is a nonanticipating stopping rule such that P(t n) = 1. [A stopping rule satisfies {t = i} ∈ σ (X1 , . . . , Xi ), the σ -field generated by X i .] V(X) is defined similarly, with P(t < ∞) = 1. The value for the prophet is M(X n ) = E(max(X1 , . . . , Xn ))[M(X) = E(sup Xi ), respectively]. For a broad class C of random variables, a ratio prophet inequality is of the following form: For all X n [X] in C, M(X n ) cn V(X n ) [M(X) cV(X)].
(1)
A difference Prophet inequality has the following form: For all X n [X] in C, M(X n ) − V(X n ) dn
[M(X) − V(X) d], (2)
and a prophet region is Rn = {(x, y) : x = V(X n ), y = M(X n )
REFERENCES 1. Rosenbaum, P. R. (1984). J. Amer. Statist. Ass., 79, 565–574. 2. Rosenbaum, P. R. and Rubin, H. (1983). Biometrika, 70, 41–55. 3. Rosenbaum, P. R. and Rubin, H. (1984). J. Amer. Statist. Ass., 79, 516–524. See also CLINICAL TRIALS —II and OBSERVATIONAL STUDIES.
for some X n C},
(3)
with R defined similarly. The first and bestknown prophet inequality is when C is the class of all nonnegative independent random variables. See Krengel and Sucheston [2]. For this class the constants in (1) are cn = c = 2, and these values are sharp (i.e., cannot be improved upon). Thus for this class (1) should be read as follows: Let X1 , X2 , . . . be any finite
6566
PROPORTIONAL ALLOCATION
or infinite sequence of independent nonnegative random variables. The value of this sequence to the prophet is never more than twice its value to the mortal. In the case where C consists of nonnegative i.i.d. sequences, cn is less than 2, depends on n, and can be computed recursively. In the case where C consists of nonnegative random variables which are negatively dependent in the sense that they satisfy P(Xi < ai |X1 < a1 , . . . , Xi−1 < ai−1 ) P(Xi < ai ) for i = 2, . . . , n and all constants a1 , . . . , an , (1) again holds with cn = c = 2. If C is the class of independent random variables taking values in [0, 1], then dn = d = 14 in (2). For this class, Rn of (3) satisfies Rn = R = {(x, y) : 0 x 1, x y 2x − x2 }. If C is the class of all nonnegative random variables, the best value of cn in (1) is n, and no finite c suffices for the infinite case. The ‘‘extremal distribution’’ is a martingale∗ . If, in addition, Xi , takes values in [0, 1] for all i, then Rn = {(x, y) : 0 x 1, x y x + (n − 1)x(1 − x1/(n−1) )}. Other classes of random variables for which prophet inequalities have been considered are sequences of averages of independent nonnegative random variables, discounted independent random variables in [0, 1], independent variables in [0, 1] with cost of observations, and convex functions of sums of independent variables. Hill and Kertz [1] is an excellent reference with an almost complete bibliography of everything published in the field up till 1992. Recently, statistical prophet inequalities have been considered. In this setting, pairs (X1 , Z1 ), . . . , (Xn , Zn ) have a given joint distribution, but the Xi s are not directly observable, and only Xi ’s ‘‘noise corrupted version’’, Zi , is observable. This includes the case Zi = Xi + Yi where the Yi s are any sequence
independent of the Xi s, as well as many other examples. When the pairs are independent, all previously mentioned prophet inequalities valid for independent Xi s remain valid also for the corresponding statistical version. REFERENCES 1. Hill, T. P. and Kertz, R. P. (1992). A survey of prophet inequalities in optimal stopping theory. Contemp. Math., 125, 191–207. 2. Krengel, U. and Sucheston, L. (1978). On semiamarts, amarts and processes with finite value. In Probability on Banach Spaces, J. Kuelbs, ed. Marcel Dekker, New York. See also OPTIMAL STOPPING RULES.
ESTER SAMUEL-CAHN
PROPORTIONAL ALLOCATION This term is used in stratified designs∗ in finite population sampling∗ . When a finite population is divided into homogeneous strata and a separate sample is drawn from each stratum, the sample design is said to be stratified. For instance, a population of persons might be stratified by sex and age, a particular stratum consisting of females aged 55–59; or a population of retail establishments might be stratified by area, description, and size, a particular stratum consisting of metropolitan hardware stores with annual sales of $1,000,000–$1,999,999 at the last retail census. Proportional allocation is one method which can be used to decide how many sample units are to be selected from each stratum, given the sample number or sample size for the entire survey. In proportional allocation, the sample number for each stratum is chosen to be proportional to the number of population units in each stratum i: nh = n(Nh /N) to the nearest integer, where nh = sample number for the hth stratum n = sample number for the whole survey Nh = number of population units in the hth stratum N = number of population units in the population.
PROPORTIONAL HAZARDS MODEL, COX’S
Thus for the population of persons stratified by age and sex, the number of females aged 55–59 selected in the sample would be proportional to the number of females aged 55–59 in the population as a whole. Proportional allocation is often (not invariably) appropriate for samples of persons. It is inefficient in circumstances where the population units differ greatly in size or importance, as is the case with surveys of retail establishments. In such circumstances it is preferable to use optimal allocation (see OPTIMUM STRATIFICATION). Proportional allocation is commonly used in small surveys which do not require multistage sampling. The corresponding techniques used in multistage surveys are known as equal probability selection methods (epsem for short). BIBLIOGRAPHY Sudman, S. (1976). Applied Sampling. Academic Press, New York, pp. 112–114. (Indicates why proportional allocation is often preferred to optimum allocation on grounds of simplicity and versatility.) See also NEYMAN ALLOCATION; OPTIMUM STRATIFICATION; STRATIFIED DESIGNS; and SURVEY SAMPLING.
where β is a p × 1 vector of unknown parameters reflecting the effects of x on survival and λ0 (t) is an unspecified function of time. An extension of model (1) introduced by Kalbfleisch [7] allows strata∗ within the population to have distinct underlying hazard functions λ0j (·).
PARAMETER AND FUNCTION ESTIMATION Usually, survival data of the type envisaged here will be subject to right censoring∗ since, e.g., some individuals will not have failed on termination of the study. It will be assumed throughout that the censoring and failure mechanisms are independent [14]. For the n individuals in the study with independent variable vectors xi , i = 1, . . . , n, let t(1) < t(2) < · · · < t(k) denote the ordered uncensored failure times with corresponding independent variables x(1) , x(2) , . . . , x(n) and denote by R(t(i) ) the collection of individuals with censored or uncensored failure times t(i) . Following Prentice and Kalbfleisch [14], let Ai be the event that individual (i) fails at t(i) and Bi be the event that describes the pattern of failure and censorings to t(i) and the information that a failure occurs at t(i) . The likelihood factors to produce
K. R. W. BREWER k
PROPORTIONAL HAZARDS MODEL, COX’S
6567
p(Ai |Bi )
" k
i=1
p(Bi |Bi−1 , Ai−1 )
i=2
× p(B1 )p(Bk+1 )
THE MODEL In medical and industrial statistics, methods for evaluating the dependence of survival∗ time or response time T on independent variables or covariates x = (x1 , . . . , xp ) have received considerable attention. The independent variables could include, e.g., state of disease, duration of symptoms prior to treatment, or a binary (0/1) variable representing control/treatment group. Cox’s regression model [1] assumes that λ(t; x), the hazard function of the continuous random variable T, is given by
λ(t; x) = λ0 (t)eβ x ,
(1)
(with t(k+1) = ∞). The first of these terms does not involve λ0 (t) and reduces to
L(β) =
k i=1
eβ x(i)
j∈R(t(i) )
eβ xj
.
(2)
Cox [2], through the concept of partial likelihood∗ , indicates that inferences concerning β, using the usual large-sample likelihood methods, can be based on (2). In the absence of censoring, (2) is the marginal distribution of the ranks of the failure times.
6568
PROPORTIONAL HAZARDS MODEL, COX’S
Estimation of λ0 (t) can be achieved [9] by approximating this ‘‘underlying hazard function’’ as a step function with λ0 (t) = λi ,
t ∈ [bi−1 , bi ),
i = 1, . . . , r,
where b0 = 0 < b1 < . . . < br−1 < br = ∞ is a subdivision of the time scale. Conditional on β = βˆ (the partial likelihood estimator of β), the maximum likelihood estimator∗ of λi is given by −1 n ˆ eβxj Dij , i = 1, . . . , r, λˆ i = di
a collection of censored and uncensored survival times. Choosing τ to ensure a specified proportion, cp are censored. Kay [11] shows, e.g., in the two-group case (p = 1, xi = 0/1) that the Fisher information∗ from the exponential form of model (1) [λ0 (t) ≡ 1] is given by IE (β) =
EFFICIENCY CALCULATIONS The efficiency∗ of βˆ compared to the corresponding estimator based on parametric forms of (1) is clearly of interest. Efron [5] obtains the general element of the Cox model information matrix∗ from (2) for k, l = 1, . . . , p, as & 2 −∂ log L(β) Ikl (β) = E ∂βk ∂βl ∞ ER(t) yjk yjl λ0 (t) dt (3) = 0 j∈R(t)
where yjr = xjr −
i∈R(t)
β xi
xik e
i∈R(t)
β xi
e
−1
and ER(t) denotes expectation over the probability distribution of rankings in R (t). A censoring model of interest in the clinical trial∗ setting assumes that individuals enter the trial at random in the interval (0, τ ), termination of the trial at τ giving
1 − 12 [a1 (β, τ ) + a2 (β, τ )]
,
where −β
j=1
where Dij is the length of time spent in the interval [bi−1 , bi ) by individual j and di is the number of uncensored failures in that interval. For estimation in the stratified form of the model, see Kay [10]. In practice data may involve ties. Prentice and Gloeckler [13] have presented some results on the use of grouped form of model (1) to handle ties and have discussed other approaches to this problem.
n[1 − a1 (β, τ )][1 − a2 (β, τ )]
a1 (β, τ ) =
eβ (1 − e−τ e τ
a2 (β, τ ) =
e−β (1 − e−τ e ) τ
)
and β
and (3) reduces to 1 IC (β) = 2n β e−τ e
τ + e−β log u 2β
τ {1 + e2β u(2β−1)e }
du.
The asymptotic efficiency of βˆ under the exponential assumption is then R(β) = IC (β)/IE (β). At β = 0, R(β) = 1. In the uncensored case R(0.2) = 0.963 and R(0.4) = 0.874 while for Cp = 0.3, R(0.2) = 0.988 and R(0.4) = 0.950. Again for p = 1 and in the uncensored case Kalbfleisch [8] shows that 2 in the neighborhood of β = 0, R(β) e−u2 β , where µ2 is the second central moment of the x-values. MODEL CHECKING Some informal methods, based on residuals∗ and hazard plotting∗ , have been suggested for assessing model (1) assumptions by Kay [10], Prentice and Kalbfleisch [14], and Cox [3], among others. Defining for r = 1, 2, . . . ˆ ˆ = tj eβ xj + t(rm) eβ xj B(m) r j∈R(t / (rm) )
j∈R(t(rm) )
as the total estimated ‘‘operational’’ time at risk up to the rmth failure, Cox shows that (m) approximately Z(m) = d − log[B(m) r r − B(r−1) ], where d is a constant independent of r and B(m) 0 = 0, has mean log λr , where λr is the average underlying hazard between the failures at t((r−1)m) and t(rm) , and variance
PROPORTIONAL HAZARDS MODEL, COX’S
(m − 12 )−1 . In addition, the Z’s are indepenagainst dent and, for chosen m, plots of Z(m) r the midpoint of that interval will display the form of λ0 (t). To check that a particular independent variable x affects the hazard in the assumed multiplicative way, the sample may be divided into a small number of subsamples on the basis of the x values and the stratified form of model (1) fitted. Plots as above, separately within each stratum, can then be viewed for evidence of trends. These would indicate violation of the multiplicative assumptions with regard to the effect of x. TIME-DEPENDENT COVARIATES A generalization of (1) allows the independent variables to be time dependent and Crowley and Hu [4] and Farewell [6] provide interesting examples of their use. Prentice and Kalbfleisch [14] give a thorough discussion of their application and show how they may be used with model (1) in the analysis of competing risks∗ . AN APPLICATION The data set to be used as an illustration of the methods described above is from the Manchester (U.K.) Regional Breast Study. Further details are given in Lythgoe et al. [12]. The independent variables, recorded at entry into the study, are: " 1 tumor size > 2 cm x1 = log age x2 = 0 tumor size 2 cm " 1 tumor site lateral x3 = 0 otherwise " 1 tumor site central x4 = 0 otherwise " 1 postmenopausal < 3 years x5 = 0 otherwise " 1 postmenopausal 3 years x6 = 0 otherwise " 1 clinical state II x7 = 0 clinical stage I
6569
" 1 local + XRT, stage I y1 = 0 local, stage I " 1 local + XRT, stage II y2 = 0 radical, stage II Of the 881 patients, there were 205 deaths, of which 185 were distinct. The 10 pairs of tied values were broken at random for the purposes of analysis. Table 1 presents Cox model β estimates with estimated standard errors obtained from the inverse of the negative of the matrix of second partial derivative of ˆ Chi-square statistics (2) evaluated at β = β. were obtained by fitting reduced models with each variable omitted in turn and employing the large-sample likelihood ratio test∗ . Stage and menopausal status are the only significant factors and in particular the treatment variables y1 and y2 produce nonsignificant effects. The model checking ideas of §4 are illustrated by investigating the effect of clinical stage in the model. Defining two strata according to the value of x7 , Fig. 1 plots values, for m = 12, having fitted the Z(m) r the stratified form of the Cox model with x7 omitted from the variables listed above. The pattern of the two sets of Z values, particularly in view of the associated standard deviations, are similar, suggesting that clinical stage affects the underlying hazard function in the assumed multiplicative way. REFERENCES 1. Cox, D. R. (1972). J. R. Statist. Soc. B, 34, 187–220. 2. Cox, D. R. (1975). Biometrika, 62, 269–276. 3. Cox, D. R. (1979). Biometrika, 66, 188–190. 4. Crowley, J. and Hu, M. (1977). J. Amer. Statist. Ass., 72, 27–36. 5. Efron, B. (1977). J. Amer. Statist. Ass., 72, 557–565. 6. Farewell, V. T. (1979). Appl. Statist., 28, 136–143. 7. Kalbfleisch, J. D. (1974). Biometrics, 30, 561. 8. Kalbfleisch, J. D. (1974). Biometrika, 61, 31–38. 9. Kalbfleisch, J. D. and Prentice, R. L. (1973). Biometrika, 60, 267–278. 10. Kay, R. (1977). Appl. Statist., 26, 227–237. 11. Kay, R. (1979). Biometrika, 66, 91–96.
6570
PROPORTIONAL REDUCTION IN ERROR (PRE) MEASURES OF ASSOCIATION
Table 1. Cox Model Fit Giving Estimates, Estimated Standard Errors, and χ 21 Statistics for Testing Effects Independent Variable
Estimated Coefficient
Age, Tumor size,
Tumor site, Menopausal status, Stage,
Treatment,
Estimated Standard Error
χ12 Statistic
x1 x2 x3
0.004 0.236 0.068
0.672 0.169 0.184
0 2.01 0.14
x4 x5
0.015 0.573
0.218 0.293
0 3.56a
x6 x7 y1
0.323 0.553 −0.229
0.283 0.201 0.195
1.31 6.86b 1.39
y2
0.159
0.205
0.61
a 0.05 < p < 0.10 b p < 0.001.
Figure 1. Cox model check for effects of clinical stage. 12. Lythgoe, J. P., Leck, I., (1978). Lancet, 744–747.
and
Swindell, R.
13. Prentice, R. L. and Gloeckler, L. A. (1978). Biometrics, 34, 57–67. 14. Prentice, R. L. and Kalbfleisch, J. D. (1979). Biometrics, 35, 25–39. See also CLINICAL TRIALS; CONCOMITANT VARIABLES; HAZARD PLOTTING; and SURVIVAL ANALYSIS.
RICHARD KAY
PROPORTIONAL REDUCTION IN ERROR (PRE) MEASURES OF ASSOCIATION Proportional reduction in error measures were developed largely because of the need for measures of association∗ between two nominal∗ variables with meaningful interpretations. Unlike measures based upon the chi-square∗ statistic, which are difficult to
PROPORTIONAL REDUCTION IN ERROR (PRE) MEASURES OF ASSOCIATION
interpret, PRE measures have interpretations in terms of probabilities of misclassification which are clear and easily understood. Suppose that a population is classified on the basis of two nominal attributes, A and B. In contingency table∗ notation, let pij be the probability that a member of the population is classified at level i of the (row) variable A and at level j of the (column) variable B, for i = 1, . . . , r and j = 1, . . . , c. Then the ith row total pi+ = j pij is the marginal∗ probability that an individual falls in class i of attribute A; similarly, p+j is the probability that the individual falls in class j of B. PRE measures are motivated as follows. Some rule is used to classify a randomly selected member of the population with respect to attribute A, first using only the population information at hand, and again when the member’s B class is known as well. The associated PRE measure is the relative reduction in the probability of misclassification: PRE(A|B) = (P(E) − P(E|B))/P(E),
(1)
where P(E) is the probability of error in the first case and P(E|B) is the probability of error in the second. Thus (1) is the decrease in the probability of error in predicting (for some prediction rule) a member’s A class gained by knowledge of its B class, relative to the probability of error when its B class is unknown. If it is a priori possible to classify the member on A with certainty [i.e., P(E) = 0], then (1) is undefined. Moreover, 0 PRE(A|B) 1, with PRE(A|B) = 0 when A and B are independent, and PRE(A|B) = 1 when knowledge of the member’s B class permits errorless prediction of its A class. It can happen that PRE(A|B) = 0 even though A and B are not independent. See ref. 1 or 2 for further discussion. Each classification rule used with (1) yields a PRE measure. One rule (optimal prediction) is to assign the member to the most probable A class, given available information. When the member’s B class is unknown, it is assigned to the A class of maximum probability (i.e., to the class m such that pm+ = max{p1+ , . . . , pr+ }). When it is known to fall into B class (column) j, it is assigned to the A class (row) of maximum probability
6571
in column j (i.e., to the class m such that pmj = max{p1j , . . . , prj }). The resulting PRE measure is c pmj − pm+ (1 − pm+ ). (2) λA|B = j=1
A second rule (proportional prediction) is to assign the member to an A class according to the appropriate (marginal or conditional) probability distribution for A: The member is assigned to class i with probability pi+ when its B class is unknown, and to class i with probability pij /p+j when its B class is known to be j. This rule yields r c
τA|B =
i=1 j=1
p2ij /p+j − 1−
r i=1
r i=1
p2i+ (3)
.
p2i+
When a classification of the entire population is unavailable, it is necessary to estimate the measures (2) and (3). For a random sample of size n from a population in which xij is the number of sample members in A class i and B class j, the analogs to (2) and (3) are λˆ A|B =
c
xmj − xm+
(n − xm+ )
(4)
j=1
and (see GAMMA) τˆA|B =
GOODMAN –KRUSKAL
n
r i=1
c
TAU
AND
r
2 j=1 xij /x+j − n2 − ri=1 x2i+
2 i=1 xi+
.
(5)
Under the multinomial∗ sampling model, (4) and (5) are maximum likelihood estimators∗ of (2) and (3), respectively. Exact sampling distributions of (4) and (5) are not known, but both have asymptotically normal distributions, provided that the sample is not concentrated in a single A class and that (2) and (3) are neither 0 nor 1. Using Gini’s definition of total variability in a sample from a categorical variable, Margolin and Light [4,5] show that, when τA|B = 0, 2 UA|B = (n − 1)(r − 1)τˆA|B
(6)
6572
PROPORTIONAL SAMPLING
is asymptotically distributed as a chi-square variate with (r − 1)(c − 1) degrees of freedom. The PRE measures (2) and (3) treat A and B asymmetrically, with B playing the role of ‘‘independent’’ variable. Versions of (2) to (6) exist which treat A as ‘‘independent’’ and B as ‘‘dependent.’’ Symmetric versions of (2) to (5) have also been defined for cases where no natural asymmetry exists between A and B; see ref. 2 or 3. The measure λ was first proposed by L. Guttman and τ is attributed to W. A. Wallis.
REFERENCES 1. Bishop, Y. M. M., Fienberg, S. E., and Holland, P. (1975). Discrete Multivariate Analysis: Theory and Practice. MIT Press, Cambridge, Mass., pp. 387–393. (An exposition and derivation of the log-linear model for the analysis of contingency tables, with emphasis on iterative proportional fitting. Contains a chapter on measures of association which includes a concise discussion of the properties of λ and τ .) 2. Goodman, L. A. and Kruskal, W. H. (1979). Measures of Associations for Cross Classifications. Springer-Verlag, New York. (A reprint of four classical papers, written between 1954 and 1972, on the construction of measures of association for two-way contingency tables. The original and most extensive source of information on λ and τ ; these papers contain derivations, sample estimates and related asymptotic calculations, examples, and historical references.) 3. Liebetrau, A. M. (1983). Measures of Association. Sage, Beverly Hills, Calif. (A comprehensive and fairly elementary discussion of a large number of widely used measures of association, including λ and τ .) 4. Light, R. J. and Margolin, B. H. J. Amer. Statist. Ass., 66, 534–544.
(1971).
5. Margolin, B. H. and Light, R. J. J. Amer. Statist. Ass., 69, 755–764.
(1974).
See also ASSOCIATION, MEASURES OF; CATEGORICAL DATA; GOODMAN –KRUSKAL TAU AND GAMMA; LOG-LINEAR MODELS IN CONTINGENCY TABLES; and NOMINAL DATA.
A. M. LIEBETRAU
PROPORTIONAL PROPORTIONAL ALLOCATION
SAMPLING. See
PROPORTION ESTIMATION IN SURVEYS USING REMOTE SENSING During 1972–1984 the National Aeronautics and Space Administration (NASA) launched a series of land observatory (LANDSAT) satellites, and developed a remote sensing technology capable of monitoring certain types of earth resources. A major application of LANDSAT remote sensing is the estimation of acreage for agricultural crops. Remotely sensed data consist of radiometric measurements in each of several regions of the electromagnetic spectrum. The sensor system has a radiometer which measures the intensity of radiant energy. Two radiometers used onboard LANDSAT satellites are the multispectral scanner (MSS) and the thematic mapper (TM). Of the five LANDSAT satellites launched, the first three carried only an MSS sensor and the last two carried both the MSS and TM sensors. Both sensors image a 100-nautical mile wide swath of the earth by viewing the orbital ground track through a mirror, periodically scanning strips perpendicular to the orbital path. The analog signals recorded by the sensors are converted into digital counts onboard the satellite and transmitted to a ground station. Hence the LANDSAT coverage of an area results in a scene consisting of scanlines with a certain number of resolution elements per scanline. The TM sensor was designed to provide improved spatial and spectral resolution. It consists of six wavelength bands, compared to four for the MSS sensor. It has a smaller size resolution element, approximately 0.2 acre versus 1.1 acres for MSS. The resolution element is called a pixel. The TM data transmission rate is 85 megabits per second versus 15 megabits per second for MSS. The digitization counts are from 0 to 255 for TM and from 0 to 63 for MSS. The first three LANDSAT satellites were launched into circular, near-polar orbits 920 km (570 miles) in altitude. These orbits are ‘‘sun-synchronous’’; for a given point on the ground, it is the same local time of day when the satellite passes. These satellites pass over a given point on the ground every 18 days. Landsats 4 and 5 were launched into a sun-synchronous, circular, near-polar
PROPORTION ESTIMATION IN SURVEYS USING REMOTE SENSING
orbit 705 km (431 miles) in altitude with repetitive coverage every 16 days. Multidate acquisitions over a single location on the ground can be registered with each other and the radiance measurements can be modeled as a function of time. These measurements are influenced by the vegetation, soil type, and atmospheric conditions, and when these are statistically modeled and correlated with ground features, it is feasible to estimate agricultural crop acreages by acquiring and analyzing the remotely sensed data for an area of interest. Image analysis techniques are used to label spectral classes by crop types. A segment, several square miles in area, is required to delineate discernible patterns and identify possible crop types. A crop can be distinguished from others in a scene by monitoring the temporal development of its fields from planting through harvest and performing multitemporal analysis. This, of course, requires that multitemporal acquisitions of a segment be properly registered since misregistration, especially near field boundaries, can cause measurements to shift from one crop type to another. STATEMENT OF THE PROBLEM Let z be a p × 1 measurement vector for a pixel in an area segment containing a large number of pixels. Suppose that the segment consists of m ground cover classes for which the distribution of z is given by the probability density function fi (z), i = 1, 2, . . . , m. Then one may consider that the vectors z are independent observations sampled from a mixture∗ population with density function f (z) =
m
λi fi (z),
(1)
i=1
where λi is the proportion of the segment in ground cover class i satisfying λi 0,
m
λi = 1.
(2)
i=1
The general problem is to estimate the λi (i = 1, 2, . . . , m), given the observed vectors z for a segment without any knowledge of m or fi (z). This problem cannot be solved unless one
6573
is able to obtain information which would permit estimation, if not a complete determination, of m and fi (z). This general mixture problem has been extensively studied. A common approach is to make certain assumptions; one of the following cases is usually considered. 1. The number of classes m is known and fi (z) is from a known identifiable family of distributions, such as multivariate normal. 2. The number of classes m is known, a set of z observations from each class distribution is available, and the densities fi (z) are members of some identifiable family, usually known up to a set of parameters. For a definition of identifiability∗ and further discussion on this finite mixture problem, see Teicher [7]. In the present context, the number of classes was either fixed or estimated. An initial approach was to develop false color imagery of a segment using its spectral data and have an image analyst, trained in photointerpretation techniques, identify the potential number of ground cover classes, and label a set of pixels in the segment corresponding to the various classes. There were problems with this approach. First, the analyst was not in a position to identify all ground cover classes in a segment. Clustering the segment data can provide the possible number of classes represented in the data set, but these may not correspond to the ground cover classes in the segment. Second, labeling of pixels was highly labor intensive and time consuming, and hence not enough pixels could be labeled to estimate the class distributions nonparametrically. Generally, an analyst was trained in identifying features for a few major ground cover classes and his or her labeling of pixels was subject to error. These difficulties play a significant role in the approaches to the estimation of crop proportions using LANDSAT data. CROP PROPORTION ESTIMATION In this section we survey mainly methods of proportion estimation which were used in
6574
PROPORTION ESTIMATION IN SURVEYS USING REMOTE SENSING
practice and evaluated at the NASA Johnson Space Center. These studies were conducted under two major programs, the Large Area Crop Inventory Experiment (LACIE) [6] and the Agriculture and Resources Inventory Surveys Through Aerospace Remote Sensing (AgRISTARS). The initial approach was to assume that fi (z) in (1) represents either a parametric family of multivariate normal distributions or mixtures of multivariate normal distributions with mixing weights assumed known. For example, in LACIE, wheat was the class of interest, and therefore there were only two classes, wheat and nonwheat. Since nonwheat may consist of various ground-cover classes in a segment, its spectral distribution was assumed to be a mixture of multivariate normals. A similar model was assumed for wheat to take into account the different varieties, growth stages, and so on. Thus the density functions for the two classes were of the form f1 (z) =
m1
f2 (z) =
(3) λ2k f2k (z),
k=1
where fik (z) is the multivariate normal density with mean vector µik and covariance matrix ik , and λik is the relative size of the kth subclass in class i, i = 1, 2 and k = 1, 2, . . . , mi , satisfying λi =
mi
fˆi (z) = max fˆj (z), j
(4)
where fˆi (z) is the estimated density function for class i. The class proportion λi was estimated by * Nj , (5) λˆ i = Ni j
where Ni is the number of pixels in the segment classified into the ith class. This estimator is biased, as discussed next, and hence estimation procedures that are relatively unbiased and somewhat insensitive to errors in estimates of class distributions were necessary to obtain reliable crop acreage estimates during LACIE and AgRISTARS. ESTIMATORS
λ1k f1k (z),
k=1 m2
and the sample covariance matrices). All pixels in a segment were classified using the maximum likelihood classification∗ rule (see DISCRIMINANT ANALYSIS); a pixel with measurement vector z is classified as class i if
Let C1 represent the crop of interest and suppose a pixel either belongs to C1 or its complement C2 . Suppose that pixels in a segment are classified according to the maximum likelihood classification rule and let R1 and R2 be the classification regions corresponding to C1 and C2 . Then & Ri = z : fˆi (z) = max fˆj (z) , i = 1, 2. (6) j
λik .
k=1
The number of subclasses mi and their respective weights λik were determined by clustering the data, labeling the clusters as being wheat or nonwheat, and then finding their relative sizes within each class. If the data were not clustered, an image analyst provided a rough estimate of mi and the subclass weights were assumed equal [i.e., λik = 1/mi (i = 1, 2)]. Spectral data for the set of pixels labeled by an analyst were used to estimate fik (z), and hence fi (z), by replacing the unknown parameters with their corresponding training statistics (i.e., the sample mean vectors
Define the conditional probability pij of a pixel from class j being classified as class i by fj (z)dz. (7) pij = Ri
Suppose that y is the probability that a pixel belongs to C1 and x is the probability of classifying a pixel as C1 . Then it follows from (7) that x = p11 y + p12 (1 − y). Define the random variable φ by 1, if z ∈ R1 , φ(z) = 0, otherwise,
(8)
(9)
PROPORTION ESTIMATION IN SURVEYS USING REMOTE SENSING
where R1 is the classification region for C1 . Suppose that the segment has N pixels that are classified as C1 or C2 . Then the proportion of pixels classified as C1 , given by yˆ =
N
φ(zj )/N,
(10)
j=1
provides an estimate of y. In (10), zˆj denotes the measurement vector for pixel j. This is a biased estimator since E(ˆy) = x, where x is given in (8). The bias is given by E(ˆy) − y = p12 − (p12 + p21 ) y, where p12 and p21 are misclassification errors called the commission and omission errors, respectively. If the commission error p12 is large relative to the omission error p21 , a strong possibility when the maximum likelihood classifier is used and the true proportion y is small, then the bias may be substantial. If the classification errors are known, an unbiased estimator can be constructed by modifying (10) as follows: From (8), y = (x − p12 )/(1 − p21 − p12 ),
(11)
provided that 1 − p21 − p12 = 0. Replacing x by yˆ in (11), an unbiased estimator of y is obtained. The classification errors are generally unknown and the preceding simple modification is not possible. However, one may consider estimating p12 and p21 using an independent sample of pixels in the segment, and then estimating y by yˆˆ = (ˆy − pˆ 12 )/(1 − pˆ 21 − pˆ 12 ),
(12)
provided that 1 − pˆ 21 − pˆ 12 = 0, where yˆ is given in (10). If 1 − pˆ 21 − pˆ 12 = 0, then there is no need to modify yˆ (i.e., let yˆˆ = yˆ ). The estimator yˆˆ is then bounded between 0 and 1. Clearly, for yˆˆ to be a reliable and nearly unbiased estimator of y, the estimators pˆ 21 and pˆ 12 should be based on large samples drawn randomly from C1 and C2 , respectively. In the remote sensing application, it was not possible to select pixels randomly from individual classes C1 and C2 , since class labels were unknown for the pixels in a segment, and for a large sample pixel labeling
6575
by an analyst was too labor intensive and subject to error. There was no apparent way to deal with the latter problem; however, the problem of obtaining random samples was solved by sampling pixels from the segment without having to control sample sizes for C1 and C2 , as discussed next. Let n pixels be randomly selected from the segment and be identified for their true class labels. For a pixel with measurement vector z, define the random variable η(z) by η(z) =
1, if pixel belongs to C1 , 0, otherwise.
(13)
Clearly, E[η(z)] = y and an estimate of y is given by y=
n
η(zj )/n,
(14)
1
where zj , j = 1, 2, . . . , n, are the measurement vectors for the n sampled pixels. But y is inefficient, as it does not make use of the classification results for the segment. Suppose that S1 and S2 are the two strata of pixels classified as C1 and C2 , respectively. In other words, S1 = {uj : φ(zj ) = 1, j = 1, 2, . . . , N}, S2 = {uj : φ(zj ) = 0, j = 1, 2, . . . , N}, where uj denotes pixel j. Then yˆ given in (10) is the proportion of pixels in S1 and (1 − yˆ ) is the proportion pixels in S2 . In the sampled of n = φ(z pixels, n 1 j ) are from S1 and n2 = 1 n from S2 . Of the n1 pixels 1 [1 − φ(zj )] are from S1 , n11 = n1 η(zj )φ(zj ) belong to C1 , and from the n2 pixels in S2 , n12 = n1 η(zj )[1 − φ(zj )] belong to C1 . Thus an estimator of y is obtained by y˜ = n11 yˆ /n1 + n12 (1 − yˆ )/n2 ,
(15)
provided that ni 1, i = 1, 2. Otherwise, let y˜ = y, where y is defined in (14). Since the event that n1 or n2 is zero has nonzero probability, y˜ is in general biased. However, as n becomes large, y˜ is asymptotically unbiased. The estimator in (15) can be rewritten y˜ = y + b(X − x),
(16)
6576
PROPORTION ESTIMATION IN SURVEYS USING REMOTE SENSING
where y is given in (14), x = as defined in (10), and n
b=
n i
φ(zj )/n, X = yˆ
G(t) = G0 + A(t − T0 )α exp[−β(t − T0 )2 ], (18)
(η(zj ) − y)(φ(zj ) − x)
j=1 n
behavior of the greenness variable has been approximately modeled by
.
(17)
(φ(zj ) − x)2
j=1
If the denominator in b is zero, take b = 0. Clearly, b is an estimator of the slope in the linear regression∗ of η(z) onto φ(z). Thus y˜ can be thought of as a regression estimator in the case of a finite population where the sample mean y is adjusted to take into account the auxiliary information of the measurement vector z based on the correlation between φ(z) and η(z). Properties of this type of estimator are discussed by Cochran [2]. Heydorn [4] discusses the estimators given above in greater detail, noting further extensions of the estimator in (15). Feiveson [3] describes some of these estimators as well as others, particularly mixture decomposition estimators with mixtures of density functions as in (1), marginal distribution functions, or moments.
where G(t) is the greenness value at time t, G0 is the bare soil greenness value, T0 is the date of emergence, α is a crop specific parameter related to the rate of change of greenness in early season (‘‘green up’’), and β is another crop specific parameter related to the rate of onset of senescence or ripening of the crop (‘‘green down’’). This formulation was extremely important because it related the LANDSAT data to potentially predictable agrophysical parameters such as emergence date, rate of green development, rate of senescence, and total length of growing season. The model in (18), called the greenness profile model, has two inflection points, T1 and T2 , such that 2 = (T2 − T1 )2 = 1/(2β) + α[1 − (1 − 1/α)1/2 ]/(2β) ≈ 1/β,
(19)
and a peak at FEATURE EXTRACTION USING CROP SPECTRAL PROFILE MODELS The LANDSAT multiband data are highly correlated for both the MSS and TM sensors. A principal component analysis∗ of MSS data acquired for a segment in a single LANDSAT pass would show that these data are primarily two-dimensional; one corresponds to a measure of vegetation growth and the other to a measure of background reflectance. The two dimensions are called greenness and brightness, respectively. Recent research has shown that a similar situation holds for the six-band TM data, with the greenness and brightness space augmented by a third dimension related to soil water content. This type of analysis not only reduces the data dimensionality by a factor of 2, but also provides physically interpretable parameters. From investigations of the greenness and brightness variables, it has been observed that their behavior as a function of time is different for different crops. The temporal
t = Tp = T0 + (α/(2β))1/2
(20)
with a maximum greenness value of Gmax = G0 + A(α/(2βe))α/2 .
(21)
For corn and soybeans T1 corresponds closely to the onset of the reproductive phase of these crops and T2 corresponds to the onset of senescence. Since T1 , T2 , and thus , can be predicted using crop phenology models, this result provides an important feature for labeling crop types. Crop phenology models predict the dates of particular phases in a crop’s life cycle. The predictor variables for these models usually include daylength, temperature, and rainfall measurements. Several empirical studies showed that the variables , Tp , and Gmax given in (19) to (21) are good discriminators for crops with different crop phenology and can be used in place of the multitemporal measurement vectors z. Of course, this would not change any of the
PROPORTIONS, COCHRAN’S Q -STATISTIC FOR
methods of proportion estimation discussed earlier. The crop profile model approach to feature extraction was initially proposed by Badhwar [1]. See Houston and Hall [5] for a good review of this approach from a statistical viewpoint. Other references on the use of satellite data for crop surveys are provided in the Bibliography.
6577
Odell, P. L., ed. (1976). Commun. Statist. A, 5. (A special issue on proportion estimation using satellite data.) See also AGRICULTURE, STATISTICS IN; CLASSIFICATION; CROP AREA ESTIMATION, LANDSAT DATA ANALYSIS IN; GEOGRAPHY, STATISTICS IN; METEOROLOGY, STATISTICS IN; SPATIAL DATA ANALYSIS; and SPATIAL SAMPLING.
A. GLEN HOUSTON RAJ S. CHHIKARA
REFERENCES 1. Badhwar, G. D. (1980). Photogramm. Eng. Remote Sensing, 46, 369–377. 2. Cochran, W. G. (1977). Sampling Techniques. Wiley, New York. 3. Feiveson, A. H. (1979). Proc. Tech. Sess., LACIE Symp., Vol. 2, NASA-JSC 16015, pp. 643–646. (An excellent review of crop proportion estimation methods in remote sensing from an analytical viewpoint.) 4. Heydorn, R. P. (1984). Commun. Statist. Theor. Meth., 13, 2881–2903. (An up-to-date review of crop proportion estimation methods used in practice.) 5. Houston, A. G. and Hall, F. G. (1984). Commun. Statist. Theor. Meth., 13, 2857–2880. (Describes LANDSAT MSS and TM data and approaches for relating their temporal behavior to crop growth cycles. Also briefly describes techniques for crop identification and area and yield estimation.) 6. MacDonald, R. B. and Hall, F. G. (1980). Science, 208, 670–679. (An excellent reference on an approach to global crop forecasting using satellite data.) 7. Teicher, H. (1961). Ann. Math. Statist., 32, 244–248.
See the following works, as well as the references just given, for more information on the topic of proportion estimation in crop surveys using remotely sensed data. BIBLIOGRAPHY Bauer, M. E., ed. (1984). Remote Sensing Environ., 14 (1–3). (A special issue devoted to remote sensing research to inventory and monitor crop, soil, and land resources accomplished by the AGRISTARS program.) Chhikara, R. S., ed. (1984). Commun. Statist. Theor. Meth., 13. (A special issue on crop surveys using satellite data.)
PROPORTIONS, COCHRAN’S Q -STATISTIC FOR When comparing proportions in independent samples, a large-sample chi-square∗ statistic is quite well known. Often, however, each member of a sample is matched with a corresponding member of every other sample in order to increase the precision of comparison. When matching occurs, the samples are correlated and the ordinary Pearson chi-square statistic is no longer valid. One way of creating matched samples is to divide nc individuals into n groups of c individuals matched according to one or more characteristics or variables and then to randomize the c treatment assignments independently within each matched group. The individuals should be matched on characteristics that are associated with the response being studied. Matched samples also occur when one observes n individuals under c different treatments (e.g., c different questions or one question at c different times). No matter how the matching occurs, matched ctuples are used for comparing c samples to gain efficiency. McNemar [10] developed a test for comparing two matched samples when the response variable is a dichotomy. Cochran [6] extended McNemar’s results to the case of several matched samples. To derive Cochran’s Qstatistic, consider comparing c matched treatments (i.e., samples) with n observations per treatment. Let Xij = 1 if the outcome for the jth treatment in the ith matched observation is a ‘‘success’’ and Xij = 0 otherwise, for j = 1, 2, . . . , c and i = 1, 2, . . . , n. The total number of successes for the jth treatment is Tj = ni=1 Xij , and the total number of successes for the ith matched observation is ui = cj=1 Xij .
6578
PROPORTIONS, COCHRAN’S Q -STATISTIC FOR
In deriving the test statistic, Cochran regarded the ui as fixed. He then argued that under the ‘‘null hypothesis’’ each of the ways of distributing the ui successes among the c treatments is equally likely. Therefore, for the ith matched observation, each treatment has an equal probability of success, and this probability is dependent on the fixed ui and is allowed to vary from observation to observation. Note that Cochran’s Q-statistic reflects the idea that the probability of the response is related to the values of the matching variables. The statistic is given as c(c − 1) cj=1 (Tj − T)2 ; Q= c ni=1 ui − ni=1 u2i it provides an exact conditional test and, also, a large-sample chi-square test of the hypothesis of permutational symmetry. Bhapkar [3] formally states this hypothesis as H(c): (Xij1 , Xij2 , . . . , Xijc ) has the same conditional distribution, given uk , as (Xi1 , Xi2 , . . . , Xic ) for every permutation (j1 , j2 , . . . , jc ) of (1, 2, . . . , c), i = 1, 2, . . . , n. Although Cochran [6] never formally states H (c) as the null hypothesis of interest, he does show that Q has a limiting χ 2 (c − 1) distribution under the ‘‘null hypothesis.’’ An application of Cochran’s Q-statistic is seen in the following data originally given by Somes [13]. Shown in Table 1 are the results of observing 37 horses over a 4-day period and noting their responses as either correct or incorrect, represented by 0 or 1. The plausible alternative to rejection of H (c) would be that the horses have learned. To simplify the calculations, Q may be expressed as c 2 2 Tj − S (c − 1) c j=1 n
Q=
cS −
i=1
, u2i
where S=
c j=1
Tj =
n i=1
ui =
n c i=1 j=1
Xij .
For these data, Q has a value of 15.77, which is significant at the 0.005 level in a χ 2 (3) table. Madansky [11] mentions Cochran’s Qstatistic when he deals with a test of an hypothesis of interchangeability. He considers a more general situation of n individuals holding one of s opinions at t successive time intervals. He derives a conditional test statistic which for s = 2 reduces to that given by Cochran by letting c = t. Fleiss [7] demonstrates the applicability of Cochran’s Q-statistic for situations where the ordering of the c treatments is not necessarily random for any of the nc-tuples. Cochran’s Q-statistic has been used to test hypotheses other than the one Cochran had in mind. If the probability of success is assumed to be constant from observation to observation within a sample, then the hypothesis concerning equality of proportions in the samples may be formally stated as H0 : π1 = π2 = · · · = πc , where πj is the probability of success for the jth treatment. It is important to recognize that in H0 the probability of success is assumed to be constant from matched observation to observation, whereas in H (c) it is allowed to vary, conditional on ui . Cochran’s Q-statistic has been used to test H0 and it was erroneously assumed that Q had an asymptotic χ 2 (c − 1) distribution when H0 was true. The conditions necessary for Q to have an asymptotic χ 2 (c − 1) distribution when H0 is true were derived by Bhapkar [2]. This necessary and sufficient side condition may be expressed as H1 : π1,2 = π1,3 = π1,c = π2,3 = . . . = πc−1,c , where πj,k is the probability of simultaneous success for the jth and kth treatments. Berger and Gold [1] obtained the asymptotic distribution for Q under H0 for c = 3, and more recently Bhapkar and Somes [4] derived the asymptotic distributions of Q under H0 for any c and demonstrated that the limiting rejection value of Q under H0 is
PROPORTIONS, COCHRAN’S Q -STATISTIC FOR
always at least as large as the rejection value obtained from the χ 2 (c − 1) distribution and often much larger, depending on the violation of H1 . If the hypothesis of interest is H0 , then Bhapkar [2] offers a Wald statistic [14] which is asymptotically χ 2 (c − 1) under H0 . Even though this test statistic is more cumbersome than Cochran’s Q-statistic, it may be preferable, since the added calculations necessary to arrive at an approximation to the asymptotic distribution of Q under H0 are also considerable. Naturally, in testing H (c) or if H1 is assumed when testing H0 , then Q is the correct test statistic. When Cochran [6] introduced his Qstatistic he also indicated how it may be broken down into components for more detailed tests comparing mutually exclusive subgroups of the c treatments. It can be shown that the statistic given by Miettinen [12] for comparing a case to multiple matched controls is a special case of breaking Q into its components. That is, Miettinen’s statistic∗ [12] is identical to that one given by Cochran [6] for comparing the first treatment (say) with the (c − 1) other treatments. Along these same lines it should be noted (see, e.g., Brownlee [5]) that Cochran’s Q-statistic is a special case of Friedman’s rank test statistic∗ [9] when there are ties in the ranks. For further readings on Cochran’s Qstatistic, one should refer to Cochran’s original article [6] or possibly to the appropriate section of books by Brownlee [5] and Fleiss [8]. The latter is a good reference for many hypotheses dealing with categorical data∗ . For more information regarding the applicability of Cochran’s Q-statistic for hypotheses other than H (c), refer to Bhapkar [2,3] and Bhapkar and Somes [4].
REFERENCES 1. Berger, A. and Gold, R. F. (1973). J. Amer. Statist. Ass., 68, 984–993. 2. Bhapkar, V. P. (1970). In Random Counts in Scientific Work, Vol. 2, Patil, G. P. ed. Pennsylvania State University Press, University Park, Pa., pp. 255–267. 3. Bhapkar, V. P. 341–356.
(1973).
Sankhya¯
A,
6579
Table 1. Responses of Horses as Correct (0) or Incorrect (1) Day
Tj
1
2
3
4
1 1 1 0 1 1 0 1 0 0 0 23
1 1 0 1 1 0 1 0 0 0 0 15
1 0 1 1 0 1 1 0 1 0 0 10
1 1 1 1 0 0 0 0 0 1 0 10
Number of Cases 3 4 1 1 6 1 1 8 3 1 8
4. Bhapkar, V. P. and Somes, G. W. (1977). J. Amer. Statist. Ass., 72, 658–661. 5. Brownlee, K. A. (1965). Statistical Theory and Methodology in Science and Engineering. Wiley, New York, pp. 262–265. 6. Cochran, W. G. (1950). Biometrika, 37, 256–266. 7. Fleiss, J. L. (1965). Biometrics, 21, 1008–1010. 8. Fleiss, J. L. (1973). Statistical Methods for Rates and Proportions. Wiley, New York, pp. 83–87. 9. Friedman, M. (1937). J. Amer. Statist. Ass. 32, 675–701. 10. McNemar, Q. (1947). Psychometrika, 12, 153–157. 11. Madansky, A. (1963). J. Amer. Statist. Ass., 58, 97–119. 12. Miettinen, O. S. 339–355.
(1969).
Biometrics,
25,
13. Somes, G. W. (1975). Some Contributions to Analysis of Data from Matched Samples. Ph.D. dissertation, University of Kentucky. 14. Wald, A. (1943). Trans. Amer. Math. Soc., 54, 426–482. See also CATEGORICAL DATA; CHI-SQUARE TEST —I; FRIEDMAN’S CHI-SQUARE TEST; MATCHED PAIRS; MATCHED SAMPLES, MIETTINEN’S TEST STATISTIC FOR; and TWO-BY-TWO TABLES, MCNEMAR’S TEST FOR.
35,
GRANT W. SOMES
6580
PROSPECTIVE STUDIES
PROSPECTIVE STUDIES In epidemiology, the term ‘‘prospective study’’ usually designates a nonexperimental research design in which all the phenomena under observation occur after the onset of the investigation. This approach is known by a variety of other names, such as follow-up∗ , incidence, longitudinal∗ , and cohort studies. Regardless of the title preferred, the prospective approach may be contrasted with other nonexperimental studies that are based on historical information (see EPIDEMIOLOGICAL STATISTICS and RETROSPECTIVE STUDIES). The usual plan of prospective research is illustrated in Fig. 1. After initiation of the study, subjects are enrolled according to the level of exposure to the main independent variable. Typically, one group is defined as ‘‘exposed’’ and the comparison group consists of ‘‘unexposed’’ subjects. The group assignment in a prospective study is determined by observations on naturally occurring exposures. The reliance on natural exposures may be contrasted with experimental studies (see CLINICAL TRIALS), in which the exposure status is assigned by randomization∗ (but see also HISTORICAL CONTROLS). After the study groups are defined in a prospective investigation, the subjects are followed forward in time for the development of the outcome variable. Then the frequency of the outcome among exposed subjects is compared against the frequency of the outcome among unexposed subjects. Example. One of the most noteworthy examples of the prospective approach was the investigation of atherosclerotic heart disease in Framingham∗ , Massachusetts [1]. This study was started in 1948 by the U.S. Public Health Service for the purpose of evaluating possible determinants of heart disease. Framingham was chosen as the site of investigation because it had a stable, cooperative population, with good access to local and referral medical facilities. A sample of 6,500 persons between the ages of 30 and 60 years was chosen from a population of about 10,000 persons in that age group. Over 5,000 persons without clinical evidence of atherosclerotic heart disease agreed to participate in this research. Each subject was
examined at the outset and was reexamined at intervals of two years. The primary outcome was the development of atherosclerotic heart disease. Over the 20 years of follow-up several risk factors were identified, including hypertension, elevated serum cholesterol, and cigarette smoking [6]. THE SELECTION OF SUBJECTS A variety of factors must be considered in the choice of the study population for prospective research. In particular, the subjects must be accessible, willing to cooperate, and traceable over time. The following sources of subjects are employed commonly in prospective research. 1. General Population. A prospective study can be performed in an entire community, as illustrated by the Framingham Study. The choice of a community for investigation may be governed by the stability and demographic features of the population, public support for the study, and the availability of resources over a sufficient period of time to evaluate the outcome. 2. Special Population. A prospective study can be conducted among persons who are affiliated with a particular group. For example, Doll and Hill [2] studied the relationship between cigarette smoking and lung cancer among physicians on the British Medical Register who were living in the United Kingdom. The choice of physicians as a study population offered the advantages of a group that was interested in health research and could be traced through professional organizations. 3. Clinical Population. A prospective study can be performed among patients at specific clinical facilities. As an illustration, the Collaborative Perinatal Project was a prospective study of more than 54,000 pregnant women who received prenatal care and delivered at one of 12 cooperating hospitals between 1959 and 1966 [5]. This population was used to evaluate risk factors
PROSPECTIVE STUDIES
6581
Figure 1. Schematic diagram of a prospective study (shaded areas represent subjects exposed to the antecedent factor; unshaded areas correspond to unexposed subjects).
for perinatal and infant mortality∗ , as well as for infant morbidity∗ . THE MEASUREMENT OF EFFECT The analysis of a prospective study is based on a comparison of the frequencies of outcomes among exposed and unexposed subjects. The magnitude of effect usually is quantified with one of the following measures. 1. Incidence density ratio (IDR). The IDR is defined as the incidence rate among the exposed divided by the incidence rate among the unexposed. The standard data layout for this type of analysis is portrayed in Table 1. For this analysis, the symbols a and b represent the number of specific outcomes among exposed and unexposed subjects, respectively. The length of observation on each individual, summed over all subjects in the study group are represented by L1 and L0 , respectively. Thus, the incidence density rate of the outcome among exposed
persons is estimated by a/L1 . Similarly, the incidence density of the outcome among unexposed persons is estimated by b/L0 . The IDR is estimated from = a/L1 . IDR b/L0 When there is no association between the exposure and outcome of interest, the IDR equals unity. An IDR less than unity implies a reduced incidence of the outcome among exposed subjects, compared with the unexposed. Conversely, an IDR greater than unity indicates that exposed subjects have an elevated incidence of the outcome. A largesample test of difference from the null value can be constructed from the normal approximation to the binomial distribution∗ [3]: z=
a − m(L1 /T) . [m(L1 /T)(L0 /T)]1/2
2. Risk ratio (RR). The RR is defined as the risk of the outcome among exposed
Table 1. Data Layout for Calculation of the Incidence Density Ratio (IDR) Exposed Unexposed Subjects Subjects Total Number of Specific Outcome(s) a b m= a+b Population Time of Follow-up
L1
L0
T = L1 + L0
6582
PROSPECTIVE STUDIES Table 2. Data Layout for Calculation of the Risk Ratio (RR) Exposed Unexposed Subjects Subjects Total Number with Outcome a b m1 = a + b Number without Outcome Total
c
d
m0 = c + d
n1 = a + c
n0 = b + d
N =a+b+c+d
divided by the risk among unexposed. The standard data layout for this type of analysis is depicted in Table 2. The symbols a and b have the previously defined interpretations. The symbols c and d represent the number of subjects who do not develop the outcome among exposed and unexposed persons, respectively. The risk of the outcome among the exposed is estimated by a/n1 . Similarly, the risk of the outcome among the unexposed is estimated by b/n0 . Thus, the RR is estimated by RR
=
a/n1 . b/n0
The interpretation of the value of the RR is identical to that for the IDR. A large sample test for this measure, based on the hypergeometric distribution∗ , uses the Mantel–Haenszel∗ chi-square statistic [4]: 2 χ(1) =
(N − 1)(ad − bc)2 . m1 m0 n1 n0
As indicated, the IDR and RR have the same scale of potential values. Although there are no absolute rules for the interpretation of these measures, Table 3 may serve as a rough
Table 3. Guidelines to the Interpretation of the Incidence Density and Risk Ratios Value
Effect of Exposure
0–0.3 0.4–0.5 0.6–0.8 0.9–1.1 1.2–1.6 1.7–2.5 2.6
Strong benefit Moderate benefit Weak benefit No effect Weak hazard Moderate hazard Strong hazard
guideline. For both measures, the absolute lower limit is zero, which results when there are no outcomes among exposed subjects. The upper limit is positive infinity, which results when there are no outcomes among unexposed subjects. RELATIVE MERITS OF PROSPECTIVE STUDIES When compared with other methods of nonexperimental research, the prospective design offers several advantages. First, incidence rates of the outcome can be measured. Second, there is a clear temporal relationship between the purported cause and effect in prospective research. Third, exposures that are uncommon can be evaluated by selective sampling of exposed persons. Fourth, the amount and quality of information collected is not limited by the availability of historical records or recall of subjects. Fifth, multiple effects of a given exposure can be assessed. Nevertheless, prospective studies also have important limitations. First, large numbers of subjects may be needed to evaluate rare outcomes. Second, a prospective study may involve years of observation to evaluate chronic processes. A prolonged investigation may suffer from logistical problems, such as turnover of the research staff and variations in the methods used to measure the outcome. Third, it is difficult to follow subjects over protracted time periods and these individuals may change their exposure levels over time. Fourth, prospective research is relatively expensive when compared with alternative nonexperimental methods. Ultimately, the decision to conduct a prospective study is governed by the purpose of the investigation, the nature of the phenomena under observation, and the resources available for the study.
PROXIMITY DATA Acknowledgments The preparation of this entry was supported in part by a contract from the National Cancer Institute (N01-CN-61027) to the author, and the manuscript was typed by Ms. Vickie Thomas.
REFERENCES 1. Dawber, T. R., Meadors, G. F., and Moore, F. E. (1951). Amer. J. Public Health, 41, 279– 286. 2. Doll, R. and Hill, A. B. (1964). Brit. Med. J., 1, 1399–1410. 3. Kleinbaum, D. G., Kupper, L. L., and Morgenstern, H. (1982). Epidemiologic Research. Lifetime Learning Publications, Belmont, Calif., p. 286. 4. Mantel, N. and Haenszel, W. (1959). J. Natl. Cancer Inst., 22, 719–748. 5. Niswander, K. and Gordon, M. (1972). The Women and Their Pregnancies. National Institutes of Health Publ. 73–379. U.S. Government Printing Office, Washington, D.C. 6. Truett, J., Cornfield, J., and Kannel, W. (1967). J. Chronic Dis., 20, 511–524. See also CLINICAL TRIALS; EPIDEMIOLOGICAL STATISTICS; FOLLOW-UP; FRAMINGHAM: AN EVOLVING LONGITUDINAL STUDY; HISTORICAL CONTROLS; LONGITUDINAL DATA ANALYSIS; PROBABILITY, FOUNDATIONS OF; and RETROSPECTIVE STUDIES.
RAYMOND S. GREENBERG
PROXIMITY DATA Proximity data are numbers representing measures of relatedness of pairs of objects, for example, the judged similarity between political candidates, the confusability of speech sounds, or the number of tourists traveling between different countries. Proximities among a set of n objects are specified in an n × n matrix whose entry in row i and column j indicates the extent to which objects i and j are related to each other. Proximity matrices may be obtained directly, as in the examples above, or derived from rectangular matrices where each of a set of objects under study is described by a set of variables. Some ways of obtaining proximity data are discussed in the next section.
6583
Proximity matrices are often large and complex. Methods for summarizing these data can help to discover important relationships and patterns within the data. Much work has been reported in the past 20 years on the development of models and methods for the representation of structure in proximity data. Most of these representations fall into one of two families: continuous spatial models and discrete network models. Spatial models embed objects in some coordinate space so that the metric distances between points represent the observed proximities between the respective objects. Network models represent each object as a node in some graph so that the relations among the nodes in the graph reflect the proximities among the objects. Spatial and discrete models that assume the proximity matrices are approximately symmetric are discussed in the sections ‘‘Spatial Representations’’ and ‘‘Discrete Representations.’’ Questions about symmetry and some nonsymmetric models are touched on briefly in the section ‘‘Nonsymmetric Data.’’ WAYS OF OBTAINING PROXIMITY DATA One common procedure for obtaining proximity data is to ask subjects to judge the similarity (or dissimilarity) of the stimulus objects under study. Subjects are presented pairs of stimuli and are asked to rate the proximity of these stimuli on some scale. Although similarity or dissimilarity are most frequently used to elicit the judgments, other words have been used such as association or substitutability. However, specific attributes on which the judgments are to be made are usually not specified to the subjects. When the stimulus set under study is large, say 50 to 100 objects, a useful way of obtaining proximity data is to ask subjects to sort the stimuli into mutually exclusive and exhaustive categories; typically, subjects are asked to cluster the items so that stimuli in the same category are more similar to each other than to stimuli in the other categories. A measure of proximity, then, is the number or proportion of times each pair of stimuli is placed into the same category. Another procedure that requires somewhat less direct judgments about proximity
6584
PROXIMITY DATA
involves measurement of stimulus confusability. Subjects are asked to judge whether two stimuli are the same or different. The number or proportion of times a ‘‘same’’ response is given for two stimuli that are, in fact, different, (i.e., confusability of the stimuli) can be considered to be a proximity measure. Another type of measure for which there is a considerable body of data is the amount of mail, telephone communication, air traffic, and so on, between different locations, people, or organizational units. It is also possible to compute a derived measure of proximity from a rectangular matrix of profile data where the rows correspond to objects and the columns are variables measured on the objects. For example, a square symmetric matrix of profile proximities between variables could result from computing correlations of the variables over objects. This would measure similarity of the variables. Measures of dissimilarity of the objects could be obtained by computing distances between these objects. A variety of derived proximity measures is described in Anderberg [1, Chaps. 4 and 5] for ratio, interval, ordinal, nominal, and binary variables. Gordon [15, Chap. 2] gives a good summary of ways of computing a proximity matrix from an objects-by-variables matrix and discusses difficulties which can arise in the construction of such proximity measures. SPATIAL REPRESENTATIONS A common representation of a matrix of proximities is in terms of a spatial model; objects are represented as points in a multidimensional space and the distance between the points are related in some way to the proximity data. Multidimensional scaling∗ (MDS) is a family of models and associated methods for constructing such a space. The aim of MDS is to discover the dimensions relevant to the objects under consideration and to determine their coordinates on each dimension. There are several different approaches to MDS. The metric approach (e.g., Torgerson [33]) generally assumes that the relationship between the proximities and distances in space is linear. In nonmetric MDS (e.g., Shepard [28,29], Kruskal [21,22]), the relationship need only be monotonic. Two-way
MDS assumes that there is only one proximity matrix. Euclidean distances resulting from a two-way MDS analysis are invariant under rigid or orthogonal rotation of the coordinate system. Therefore, the configuration that results from a two-way MDS analysis that assumes Euclidean distances must usually be rotated in order to find the most interpretable dimensions. There are many different computer programs for implementing two-way MDS. One example of such a program that will do both metric and nonmetric MDS is KYST [24]. Other computer programs for two-way scaling are described in MULTIDIMENSIONAL SCALING. It is possible to have several proximity matrices, for example, one matrix for each of several different subjects or experimental conditions. Three-way MDS procedures will handle these data. The most popular threeway MDS approach, weighted MDS or INDSCAL (Individual Differences multidimensional SCALing [4,8,19]), assumes that there is a set of dimensions common to all individuals (or conditions) but that each individual (or condition) can weight these dimensions differently. Three-way MDS produces a separate space for each individual; these spaces are all related, except that the dimensions are stretched or shrunk according to how salient that dimension is for a particular individual. Unlike two-way MDS, the coordinates from a three-way weighted MDS analysis are uniquely oriented, since the model is not rotationally invariant. There are also many computer programs that implement the threeway MDS model. One example of a metric weighted MDS program for symmetric data is SINDSCAL [25]. See also MULTIDIMENSIONAL SCALING; more general models for three-way MDS are described by Tucker [34] and Harshman [17]. DISCRETE REPRESENTATIONS Spatial distance models assume that continuous dimensions are the underlying basis for the proximities. Tversky [35] and Tversky and Gati [36] question this assumption. They argue that similarity between objects often should be represented in terms of discrete features. One of the most widely used
PROXIMITY DATA
methods for discrete representation of proximity data is hierarchical cluster analysis∗ . This produces a set of clusters such that any two clusters are disjoint or one includes the other. The objects being clustered can be represented as terminal nodes of a tree and the distances between objects are the ‘‘heights’’ of the first nonterminal node at which they meet. This leads to a particular kind of metric defined on a tree, the ultra-metric. See HIERARCHICAL CLUSTER ANALYSIS for further details. Another representation of proximity data in terms of a tree is as a path-length tree, also called an additive similarity tree or free tree (see Carroll [5], Sattath and Tversky [27], and Cunningham [13]). While a hierarchical tree has a natural root, path-length trees do not, so it is not necessary to think of them as being hierarchically organized. In a path-length tree, objects are represented as nodes and there are lengths associated with branches; the distance between any two objects is the sum of lengths of the branches that connect the two nodes. A computer program implementing a path-length tree is ADDTREE [27]. Shepard and Arabie [31] describe a nonhierarchical nonspatial model and method for representing data as overlapping clusters. The model assumes that each of the clusters of objects corresponds to a discrete property shared by all and only those objects within that cluster. MAPCLUS is a computer program [2] that uses a mathematical programming∗ approach for implementing this model. INDCLUS [7] generalizes the Shepard-Arabie model to handle several proximity matrices; it assumes that there is a set of clusters that is common to all individuals, or other data sources, but that each individual can weight these clusters differently. Like MAPCLUS, the INDCLUS computer program uses a mathematical programming procedure to implement the model. In a hierarchical representation, in order for an object to be represented in more than one cluster, one of the clusters must be a subset of a larger cluster. Overlapping clustering procedures allow an object to be represented in more than one cluster. Another way of classifying an object in more than one way is to assume that the objects are organized into
6585
two or more separate hierarchies, not necessarily independent of one another. This can be done by fitting more than one tree to a single proximity matrix; objects are displayed as terminal nodes of one or more different trees (see Carroll [5] and Carroll and Pruzansky [9]). Carroll and Pruzansky [9] describe a hybrid model and fitting procedure for representing structure in proximity data that combines both spatial and discrete representations. They simultaneously fit to the data one or several hierarchical or path-length trees and an N-dimensional space. Given such a wide variety of possible models and methods, how does one select the correct model for a given set of data? Pruzansky et al. [26] investigated some properties of proximity data that might help determine the appropriate model for a particular data set. They found that the skewness and a property that measures departures from the ultrametric inequality∗ were useful in distinguishing between a nonmetric MDS model and an additive tree model. However, one should be cautious about trying to find a single correct model for representing proximities. Shepard [30] points out that different methods of analysis may bring out different but equally important aspects of the underlying structure in the data. NONSYMMETRIC DATA The models and methods described above assume that the matrix (or matrices) of proximities is symmetric. For many sets of data this is unrealistic. Asymmetries may be due to noise or they may be systematic. If it can be determined that the data are symmetric but the asymmetries are due to noise, it is reasonable to symmetrize the matrix by using some technique for averaging sij and sji (e.g., Arabie and Soli [3]). Hubert and Baker [20] describe a method for evaluating the symmetry of a proximity matrix. However, there are data sets where systematic asymmetries might be an inherent characteristic. Several approaches have been described for analyzing nonsymmetric proximity matrices in terms of both spatial and discrete models. For example, if the rows and columns of the matrix are thought of as separate entities, the row and column objects
6586
PROXIMITY DATA
can be displayed separately in the same multidimensional space (e.g., Constantine and Gower [10]). A method of this sort exists for data consisting of several nonsymmetric proximity matrices as well (see DeSarbo and Carroll [14]). Some methods deal with the symmetric and nonsymmetric portions separately [10,16,18,32]. Cunningham [13] uses an extension of a free tree representation to account for nonsymmetric proximities; he introduces bidirectional trees, allowing the length of the branch between node i and node j to be different from the length between node j and node i.
11. Coombs, C. H. (1964). A Theory of Data. Wiley, New York.
LITERATURE
12. Coxon, A. P. M. (1982). The User’s Guide to Multidimensional Scaling. Heinemann, Exeter, N.H.
A book by Coombs [11] contains a general discussion of types of data and types of models as well as the earliest formulation of nonmetric scaling. Carroll and Arabie [6] contains a comprehensive review of the work in MDS through 1979. Since they use a broad definition of MDS, the article includes discussion of discrete models as well as spatial distance models. A monograph on MDS by Kruskal and Wish [23] provides an excellent introduction to spatial distance models for representing proximity data. A book by Coxon [12] is a good introductory text on MDS that includes material about both spatial and discrete representations of proximity data. Additional references are mentioned in HIERARCHICAL CLUSTER ANALYSIS and MULTIDIMENSIONAL SCALING.
REFERENCES 1. Anderberg, M. R. (1973). Cluster Analysis for Applications. Academic Press, New York. 2. Arabie, P. and Carroll, J. D. (1980). Psychometrika, 45, 211–235. 3. Arabie, P. and Soli, S. D. (1980). In Multidimensional Analysis of Large Data Sets, R. G. Golledge and J. N. Rayner, eds. University of Minnesota Press, Minneapolis, Minn. 4. Bloxom, B. (1968). Individual Differences in Multidimensional Scaling. Res. Bull. 68-45, Educational Testing Service, Princeton, N.J. 5. Carroll, J. D. 439–463.
(1976).
Psychometrika,
41,
6. Carroll, J. D. and Arabie, P. (1980). Annu. Rev. Psychol., 31, 607–649. 7. Carroll, J. D. and Arabie, P. (1983). Psychometrika, 48, 157–169. 8. Carroll, J. D. and Chang, J. J. (1970). Psychometrika, 35, 283–319. 9. Carroll, J. D. and Pruzansky, S. (1980). In Similarity and Choice, E. D. Lantermann and J. Feger, eds. Hans Huber, Berne, Switzerland, pp. 108–139. 10. Constantine, A. G. and Gower, J. C. (1978). Appl. Statist., 27, 297–304.
13. Cunningham, J. P. (1978). J. Math. Psychol., 17, 165–188. 14. Desarbo, W. S. and Carroll, J. D. (1981). Proc. 3rd ORSA/TIMS Special Interest Conf. Market Meas. Anal., pp. 157–183. 15. Gordon, A. D. (1981). Classification. Chapman & Hall, London. 16. Gower, J. C. (1977). In Recent Developments in Statistics, J. R. Barra, F. Brodeau, G. Romier, and B. van Cutsem, eds. North-Holland, Amsterdam. 17. Harshman, R. A. (1972). PARAFAC2: Mathematical and Technical Notes. Working Paper in Phonetics No. 22, University of California, Los Angeles, CA, pp. 30–47. 18. Holman, E. W. (1979). J. Math. Psychol., 20, 1–15. 19. Horan, C. B. 139–165.
(1969).
Psychometrika,
34,
20. Hubert, L. J. and Baker, F. B. (1979). Qual. Quant., 13, 77–84. 21. Kruskal, J. B. 1–27.
(1964).
Psychometrika,
29,
22. Kruskal, J. B. 115–129.
(1964).
Psychometrika,
29,
23. Kruskal, J. B. and Wish, M. (1978). Multidimensional Scaling. Sage, Beverly Hills, Calif. 24. Kruskal, J. B., Young, F. W., and Seery, J. B. (1973). How to Use KYST 2: A Very Flexible Program to Do Multidimensional Scaling and Unfolding. Bell Laboratories, Murray Hill, N.J. 25. Pruzansky, S. (1975). How to Use SINDSCAL; A Computer Program for Individual Differences in Multidimensional Scaling. Bell Laboratories, Murray Hill, N.J.
PSEUDOCOMPONENTS
6587
26. Pruzansky, S., Tversky, A., and Carroll, J. D. (1982). Psychometrika, 47, 3–24.
REFERENCE
27. Sattath, S. and Tversky, A. (1977). Psychometrika, 42, 319–345. 28. Shepard, R. N. (1962). Psychometrika, 27, 125–140. 29. Shepard, R. N. (1962). Psychometrika, 27, 219–246.
1. Judge, G. G., Hill, R. C., Griffith, W., ¨ Lutkepohl, H., and Lee, T. C. (1982). Introduction to the Theory and Practice of Econometrics. Wiley, New York.
30. Shepard, R. N. (1980). Science, 210, 390–398. 31. Shepard, R. N. and Arabie, P. (1979). Psychol. Rev., 86, 87–123. 32. Tobler, W. (1976). J. Environ. Syst., 6, 271–301. 33. Torgerson, W. S. (1958). Theory and Methods of Scaling. Wiley, New York. 34. Tucker, L. R. (1972). Psychometrika, 37, 3–27. 35. Tversky, A. (1977). Psychol. Rev., 84, 327–352. 36. Tversky, A. and Gati, I. (1978). In Cognition and Categorization, E. Rosch and B. B. Lloyd, eds. Lawrence Erlbaum, Hillsdale, N.J., pp. 79–98. See also HIERARCHICAL CLUSTER ANALYSIS; MULTIDIMENSIONAL SCALING; and SIMILARITY, DISSIMILARITY AND DISTANCE, MEASURES OF.
SANDRA PRUZANSKY
PROXIMITY
GRAPH. See
CLUSTER
ANALYSIS, GRAPH-THEORETIC
PROXY VARIABLE In stochastic regression models∗ of the type yt = β1 + β2 x∗t + vt , where x∗t is an unobservable true variable and vt is a stochastic term, x∗t is sometimes replaced by an observable ‘‘proxy’’ variable xˆ t representing x∗t . Most commonly xˆ t is an estimated or measured (as opposed to ‘‘true’’) value of x∗t . Formally, xˆ t = x∗t + uˆ t , where uˆ t is a random measurement error. As an example (see, e.g., Judge et al. [1]), in studying the effects of education on income we need to measure the years of education. Instead, the observable proxy variable—years of schooling—is used.
See also STOCHASTIC REGRESSION MODELS.
PSEUDO-BAYESIAN INFERENCE. See QUASI-BAYESIAN INFERENCE
PSEUDOCOMPONENTS The concept of pseudocomponents was introduced explicitly by Piepel [4] and Crosier [2] into the statistical literature. It has been popularized in the quality-control literature by Gorman [3] and Snee and Rayner [5]. In mixture experiments∗ the independent variables are often proportions (X) of the g components in the mixture, having q the natural constraints 0 Xi 1 and i=1 Xi = 1; these reduce the factor space to a regular (q − 1)-dimensional simplex. Additional constraints in the form of lower and upper bounds 0 Li Xi Ui 1
(1)
are often placed on the components for various reasons. Whenever the constraints (1) are used, experiments transform original components (X) to a new set of variables called pseudocomponents. A conventional pseudocomponent transformation is Zi =
Xi − Li , q 1 − j=1 Lj
(2)
used mainly to adapt standard designs to a simplex that has been reduced by the constraints (1). Piepel [4] notes that the use of (2) is incorrect in the case of inconsistency of the lower and upper bounds—it may happen that the upper bounds make some of the lower bounds unattainable. Crosier [2] proposes pseudocomponent transformations based on the upper bounds.
PSEUDO-F
6588
These are useful not only for transforming standard designs and reducing illconditioning, but also for solving other problems arising in experiments with mixtures. A general definition of pseudocomponents as linear transformations of the original variables, which allows the use of standard results for unconstrained mixture experiments, is provided in Atkinson and Donev [1] with numerous examples.
(usually K = 1). The value of ani depends on the distribution populations. It is commonly supposed, in applications, that these distributions are normal, under which conditions the distribution of the variance ratio (S22 /S21 ) is that of
Fn1 −1,n2 −1 distribution).
1. Atkinson, A. C. and Donev, A. N. (1992). Optimum Experimental Designs. Clarendon Press, Oxford, England. 2. Crosier, R. B. (1984). Mixture experiments: geometry and pseudocomponents. Technometrics, 26, 207–210. 3. Gorman, J. W. (1970). Fitting equations to mixture data with restraints on composition. J. Qual. Technol., 2, 186–194. 4. Piepel, G. F. (1983). Defining consistent constraint regions in mixture experiments. Technometrics, 25, 97–102. 5. Snee, R. D. and Rayner, A. A. (1982). Assessing the accuracy of mixture model regression calculation. J. Qual. Technol., 14, 67–79. See also MIXTURE EXPERIMENTS.
PSEUDO-F This is a statistic (not a distribution). More precisely, it is a class of statistics formed by replacing the unbiased estimators S2i = ni (Xij − X i )2 of variance σi2 (i = (ni − 1)−1 j=1 1, 2) in the variance-ratio∗ statistic S22 /S21 by some other measure of (spread)2 . The class includes, for example, the ratio of squared mean deviations 2 n2 n1 −1 −1 |X2j − X 2 | n1 |X1j − X 1 | n2 j=1
and the ratio
ani = Kσi /E[range(Xi1 , . . . , Xini )]
(σ22 /σ12 ) × (variable having a
REFERENCES
j=1
where the factors ani are chosen so that
an2 range(X21 , . . . , X2n2 ) an1 range(X11 , . . . , X1n1 )
2 ,
See F-DISTRIBUTION.
PSEUDO-LIKELIHOOD This is a poor title for a class of procedures in which inferences are based on a derived likelihood for a subset of the parameters. We discuss here marginal, conditional, and profile likelihood; partial likelihood∗ is a fourth general class of procedures and is discussed in a separate entry. Suppose that X = (X1 , . . . , Xn ) is a vector of random variables whose distribution depends on parameters (θ , α). We suppose that θ is of interest and that θ and α may be vectors of parameters. In most cases of interest, α is of very high dimension. We consider the problem of drawing suitable likelihood-based inferences about θ when α is unknown. MARGINAL, CONDITIONAL, AND PROFILE LIKELIHOODS The joint likelihood of (θ , α) based on data X = x is L(θ , α; x) ∝ fX (x; θ , α), where fX represents the joint density function. For convenience, we consider the relative likelihood function R(θ , α; x) = L(θ , α; x)/L(θˆ , α; ˆ x),
(1)
which compares each hypothesized (θ , α) with ˆ the maximum likelihood∗ estimate (θˆ , α). Likelihood-based inferences on θ , α can be based on (1), but when (θ , α) is of high dimension, the examination and comprehension of the full likelihood surface is difficult. One
PSEUDO-LIKELIHOOD
approach to estimating θ would utilize the profile or maximized relative likelihood of θ , RP (θ ; x) = sup R(θ , α; x).
(2)
α
Note that RP (θ0 ; x) is the generalized likelihood ratio∗ statistic for the hypothesis θ = θ0 , and places an upper bound on the relative likelihood of (θ , α)-pairs with θ = θ0 . The profile likelihood can quite generally be used, whereas marginal and conditional likelihoods arise in problems with sufficient structure that the nuisance parameter∗ α can be eliminated by integration over a subset of the sample space (marginal likelihood) or by conditioning on a particular observed event (conditional likelihood). Suppose that there exists a one-to-one transformation of X to U, V such that the joint PDF of U, V factors as fU,V (u, v; θ , α) = fU|V (u|v; θ , α)fV (v; θ ),
(3)
where only the second term depends on θ . The marginal likelihood of θ based on V is LM (θ ; v) ∝ fV (v; θ ).
(4)
In general, the use of (4) for the estimation of θ will entail a loss of information, namely the information about θ in the first term on the right side of (3). When, as for example through invariance arguments, one can argue that the whole of the information about θ (when α is unknown) is contained in V, the likelihood (4) is highly recommended as the basis of likelihood inference about θ . More generally, the residual term may contain some information about θ , but (3) may still be used for inference on account of its simplicity. If, on the other hand, the transformation of X into U, V leads to a joint density which factors as fU|V (u, v; θ , α) = fU|V (u|v; θ )fV (v; θ , α), (5) LC (θ ; u|v) ∝ fU|V (u|v; θ )
(6)
is the conditional likelihood of θ based on U for the given V. Again, in certain instances, it may be argued that V contains little or no information about θ when α is unknown so
6589
that the conditional likelihood is appropriate for the estimation of θ . Marginal likelihoods were first introduced within the structural model∗ by Fraser (see refs. 6 and 7). Conditional likelihoods have been used by many authors. Bartlett [4] suggested their use and extended the argument to allow the conditioning variable V to depend on the parameter θ . Both marginal and conditional likelihoods are discussed and exemplified in refs. 9 and 10. EXAMPLES AND COMMENTS Estimation of a Normal Variance Suppose that duplicate determinations X1i , X2i are made on the ith individual, and that X1i , X2i are independent N(µi , σ 2 ) variates, i = 1, . . . , n. Here µ1 , . . . , µn and σ 2 are unknown parameters and our purpose is to estimate σ 2 . The profile or maximum relative likelihood of σ 2 is (x1i − x2i )2 , RP (σ 2 ; x) ∝ σ −2n exp − 4σ 2 which is maximized at σˆ 2 = (4)−1
(x1i − x2i )2 .
As n → ∞, σˆ 2 converges in probability to σ 2 /2. In this example, the profile likelihood is concentrated for large n about an incorrect value of σ 2 . The data for the ith pair can be written Ui = X1i − X2i and Vi = X1i + X2i . Note that Ui and Vi are independent and Vi ∼ N(2µi , σ 2 ). This suggests that Vi is uninformative about σ 2 when µi is unknown and that a marginal likelihood could be based on U = (U1 , . . . , Un ). Thus (X1i − X2i )2 (7) Lm (σ 2 ; u) ∝ σ −n exp − 4σ 2 and the marginal maximum likelihood esti2 mator is σˆ M = (X1i − X2i )2 /(2n), which is consistent. The marginal likelihood takes account of the degrees of freedom lost through the estimation of µ1 , . . . , µn . The heuristic argument which suggests the use of U1 , . . . , Un for inference about σ 2
6590
PSEUDO-LIKELIHOOD
can be formalized by group invariance∗ arguments. The group of transformations Xji → Xji∗ = Xji + ci (j = 1, 2; i = 1, . . . , n), where −∞ < ci < ∞, acts transitively on Vi and leaves Ui invariant. In the parameter space, the induced group acts transitively on µ1 , . . . , µn and leaves σ 2 invariant. This suggests that U1 , . . . , Un is sufficient for σ 2 when µ1 , . . . , µn are completely unknown. This example illustrates difficulties with the use of the profile likelihood when the dimensionality of the nuisance parameter is large. In other instances, when the nuisance parameter is of small dimension compared to the number of observations, the profile likelihood typically gives a convenient and simple summary. Estimation of β in the Proportional Hazards Model The proportional model of Cox (see ref. 5 and PROPORTIONAL HAZARDS MODEL, COX’S) with fixed covariates Z = (Z1 , . . . , Zp ) specifies the hazard function for the time to failure T as λ(t; Z) = λ0 (t) exp(Z β),
Estimation of a Ratio of Poisson Means Suppose that X and Y are independent Poisson variates with means α and θ α, respectively. Let U = X + Y and V = Y; the joint probability function of U, V factors as fU,V (u, v; θ , α) = fu (u; θ , α)fV|U (v; θ |u) u = (γ u e−γ /u!) v ×θ v (1 + θ )−u , where u v = 0, 1, 2, . . . and γ = α(1 + θ ). A conditional likelihood for θ can be based on the binomial distribution of V, given U, and yields LC (θ ; v|u) ∝ θ v (1 + θ )−u ,
0 < θ < ∞. (8)
Further, since the distribution of U depends only on γ = α(1 + θ ), it may be argued that no additional information about θ can be extracted from the residual term when α is unknown. In this example, the conditional likelihood (8) is proportional to the profile likelihood. Estimation of A Common Odds Ratio
where β = (β1 , . . . , βp ), λ0 (t) is a baseline hazard function which is left completely unspecified, and β = (β1 , . . . , βp ) is a vector of regression parameters. The model specifies that the rate at which individuals fail at time t is a constant (depending on the measured covariates Z) times a baseline rate. Consider a sample T1 , . . . , Tn with corresponding covariates Z1 , . . . , Zn , let T(1) , . . . , T(n) be the order statistics∗ and Z(1) , . . . , Z(n) the corresponding covariates. A marginal likelihood can be based on the distribution of the ranks or of the antiranks, A = ((1), . . . , (n)). This gives Z(i) β
LM (β; a) ∝ π e
n
Z(j) β
e
,
j=i
which is identical to the partial likelihood obtained by Cox (see PARTIAL LIKELIHOOD). Group invariance arguments can be used to establish the sufficiency of A for the estimation of β when λ0 (·) is completely unspecified (see ref. 8).
Let X1i and X2i be independent Bernoulli variates with parameters p1i and p2i , respectively, where log{p2i /(1 − p2i )} = log{p1i /(1 − p1i )} + θ , i = 1, . . . , n. Here exp(θ ) represents the odds ratio∗ , which is assumed common for each i. This model might be entertained when pairs of like individuals are assigned at random to two treatments and the response is binary. A conditional likelihood can be based on the distribution of V = (V1 , . . . , Vn ) given U = (U1 , . . . , Un ), where Vi = X2i and Ui = X1i + X2i . If Ui = 0 or 2, then Vi is determined, so only discordant pairs (pairs with Ui = 1) contribute and Pr{Ui = 1|Vi = 1} = 1 − Pr{Ui = 0|Vi = 1} = eθ /(1 + eθ ).
PSEUDO-MEDIAN
If λ = eθ /(1 + eθ ), it follows that the conditional likelihood of λ is proportional to λa (1 − λ)b ,
(9)
where a(b) is the number of discordant pairs with X2i = 1(X1i = 1). The conditional likelihood (9) makes no use of information in U about θ . In ref. 13 it is pointed out that some inferences on θ may be possible given only the value of U. If, for example, almost all the pairs were discordant, this would suggest that θ is not near zero. (If θ = 0, then the probability that a pair is discordant is at most 12 , which occurs when p1i = p2i = 12 .) It is not clear, however, how this information can be incorporated in the inference about θ , nor that it is desirable to do so. REMARKS The problem of assessing the information lost through use of a marginal or conditional likelihood, and of specifying conditions under which a statistic can be said to be sufficient for θ when α is unknown have been considered by several authors. See, for example, refs. 1 to 3 and 13. Both marginal and conditional likelihoods can be generalized to allow V in (3) or (5) to depend on θ . Extension of the argument to this case are considered in refs. 4, 9, and 10. Interesting applications of marginal likelihood can be found in refs. 11 and 12. REFERENCES 1. Barnard, G. A. (1963). J. R. Statist. Soc. B, 25, 111–114. (Some aspects of the fiducial argument.) 2. Barndorff-Nielsen, O. (1976). Biometrika, 63, 567–571. (Nonformation.) 3. Barndorff-Nielsen, O. (1973). Biometrika, 60, 447–455. 4. Bartlett, M. S. (1936). Proc. Camb. Philos. Soc., 32, 560–566. (The information available in small samples.) 5. Cox, D. R. (1972). J. R. Statist. Soc. B, 34, 187–220. [Regression models and life tables (with discussion).] 6. Fraser, D. A. S. (1967). Ann. Math. Statist., 38, 1456–1465. (Data transformations and the linear model.)
6591
7. Fraser, D. A. S. (1968). The Structure of Inference. Wiley, New York. 8. Kalbfleisch, J. D. and Prentice, R. L. (1973). Biometrika, 60, 267–278. (Marginal likelihoods based on Cox’s regression and life model.) 9. Kalbfleisch, J. D. and Sprott, D. A. (1970). J. R. Statist. Soc. B, 32, 175–208. [Application of likelihood methods to models involving large numbers of parameters (with discussion).] 10. Kalbfleisch, J. D. and Sprott, D. A. (1974). Sankhya A, 35, 311–328. (Marginal and conditional likelihood.) 11. Minder, C. E. and McMillan, I. (1977). Biometrics, 33, 333–341. (Estimation of compartment model parameters using marginal likelihood.) 12. Minder, C. E. and Whitney, J. B. (1975). Technometrics, 17, 463–471. (A likelihood analysis of the linear calibration problem.) 13. Sprott, D. A. (1975). Biometrika, 62, 599–605. (Marginal and conditional sufficiency.) See also ANCILLARY STATISTICS —I; LIKELIHOOD; PARTIAL LIKELIHOOD; QUASI-LIKELIHOOD FUNCTIONS; and SUFFICIENT STATISTICS.
JOHN D. KALBFLEISCH
PSEUDO-MEDIAN The pseudo-median of a distribution F is defined as a median of the distribution of 1 2 (X1 + X2 ), where X1 and X2 are independent random variables, each having the distribution F. If F is symmetric, the median and pseudomedian coincide. In general, the pseudomedian may differ from the median. Consistent estimators of pseudo-medians are discussed in Hollander and Wolfe [1]. Pseudomedians should not be confused with quasimedians∗ , which are obtained by extension of the sample median. See also ref. 2. REFERENCES 1. Hollander, M. and Wolfe, D. A. (1973). Nonparametric Statistical Methods. Wiley, New York. 2. Høyland, A. (1965). Ann. Math. Statist., 36, 174–197.
6592
PSEUDOPANELS
PSEUDOPANELS. See PANEL DATA —II PSEUDO-RANDOM NUMBER GENERATORS The need to generate a sequence of numbers {Xi } that behaves like a sample of independent uniform (0, 1) random variables arises in several areas, including discrete event simulation and Monte Carlo∗ studies. The numbers generated are called pseudorandom or quasi-random numbers and are produced from generators that yield integers on [0, m], which are then transformed to [0, 1] by division by m. In theory, a uniform deviate can be transformed to an arbitrary random variable having distribution function F(·). Numerous authors (see Fishman [8], Knuth [18], and Bratley et al. [3]) have compiled algorithms for transformations to commonly required distributions (see also RANDOM NUMBER GENERATION). Several sources of error contaminate the ideal above. The finite word length of the computer can result in errors both in the transformation and in restricting all samples to a finite subset of the rational numbers. However, these are insignificant in comparison to errors resulting from nonuniformity and dependence in the pseudo-random number sequence. The latter source of error is a focal point for research. Early approaches concentrated on physical devices such as white noise∗ in electronic circuits and tables of random numbers compiled externally to the computer; see Niederreiter [28] and Jansson [14]. Reproducing results (a necessity both for assuring that the computer representation is an accurate implementation of the model and for gaining efficiencies in the estimation technique through processes known as variance reduction techniques) when using these techniques is problematic and led researchers to consider algorithms for random number generation. RANDOMNESS Pseudo-random number generators are usually evaluated on two criteria: their speed of
execution and the degree to which the resulting number sequence ‘‘appears’’ random. A generator is acceptable if sequences it produces can pass an assortment of tests for randomness∗ . The specific tests and passing levels have not been formalized in the literature so there are no canonical measures by which to evaluate a proposed generator (see Knuth [18]). In practice, researchers seek a generator that yields a sequence {Xi } that approximates a k-distributed sequence. The infinite sequence {Wi } is k-distributed if for numbers 0 v1 < w1 1, . . . , 0 vk < wk 1, lim
n→∞
n k
I(vj Wi < wj )/n
i=1 j=1
= (w1 − v1 ) · · · (wk − vk ), where I(A) = 1 if A is true and 0 otherwise. Thus a k-distributed sequence behaves as independent samples from the joint distribution of k independent uniform deviates. A sequence is ∞-distributed if it is k-distributed for all k > 0. Tests are performed on a given finite sequence to identify deviations from kdistributed uniformity. The tests fall into two categories: empirical and theoretical. The empirical tests are the standard statistical tests one encounters in hypothesis testing∗ and are based on a sample of {Xi }. Knuth [18] and Niederreiter [28] catalog many of these; Fishman and Moore [9] demonstrate a comprehensive approach. Theoretical tests are based on examining the structure that the pseudo-random number generator imparts to the entire sequence {Xi }. As a result, they test the global properties of the generator; empirical tests evaluate local properties of the generator. The spectral test∗ [6,24] is a theoretical test that is useful for examining the degree to which {Xi } is k-distributed. The lattice test, developed by Marsaglia [25], measures the lattice structure of k-tuples from {Xi }. The degree to which the lattice is orthogonal is measured by the ratio of the lengths of the longest to shortest edges, denoted Lk . For the ideal generator Lk = 1 for all k > 1. The importance of testing a pseudorandom number generator cannot be overstated. Knuth [18] gives several examples
PSEUDO-RANDOM NUMBER GENERATORS
of algorithms which, upon cursory examination, appear to produce highly random sequences that, upon closer examination, are poor approximations to random sequences. LINEAR CONGRUENTIAL METHODS Among the most popular and well-studied methods for generating a pseudo-random sequence are the linear congruential generators. These are based on the recurrence scheme Zi+1 ≡ aZi + c (mod m),
(LCG)
where seed Z0 is an integer 0 Z0 < m, multiplier a is an integer 1 < a < m, increment c is an integer 0 c < m, and modulus m is an integer 0 < m. {Zi } is often referred to as a linear congruential sequence or a random number stream, and LCG is called a mixed congruential generator when c > 0 and a multiplicative congruential generator when c = 0. The modulo arithmetic guarantees that each element in {Zi } is an integer in the interval [0, m − 1]. Consequently, when {Zi } displays sufficiently random behavior, a pseudorandom sequence {Xi } can be obtained by normalizing Zi by the modulus m so that Xi = Zi /m. As a result, the larger the modulus m, the smaller the error that results from the finite word length of the computer. It can also be shown that the sequence {Zi } is periodic in the sense that there exists a P m such that Zi = ZP+i for all i 0. A large P (and hence a large m) is desirable for achieving apparent randomness of {Xi }. The speed of generation is also a consideration in choosing the modulus m. Consider algorithm ALCG for generating a linear congruential sequence according to LCG: Algorithm Alcg 1. W ← aZi + c. 2. Zi+1 ← W − W/M . 3. Return Zi+1 . The notation Y means the largest integer less than or equal to Y. The division W/m in line 2 (which tends to be a slow operation) can be optimized when m = ze , where
6593
z is the base of the computer’s arithmetic and e is an integer. When m has this form, division is accomplished through a shift operation which is faster than a divide. As a result, 2e (10e ) is often chosen as the modulus, where e is the number of bits (digits) in the computer’s word. The generators RANDU (formerly used in the IBM Scientific Subroutine Package), with a = 65, 539, c = 0, and m = 231 and SUPER-DUPER [25], with a = 69, 069, c = 0, and m = 232 , are two such examples. One drawback to generators having m = ze is that the low-order e bits (digits) of {Zi } form a linear congruential sequence with period at most 2e (10e ). Consequently, the low-order bits (digits) have periods shorter than P and can exhibit extreme departures from randomness. RELATIONSHIP BETWEEN A, C , Z0 , AND M Two theorems help to characterize the length of the period P in terms of a, c, Z0 , and m. The first theorem assumes that c > 0. Theorem 1. The linear congruential sequence {Zi } has full period (P = m) if and only if: (a) c is relatively prime to m. (b) a − 1 is a multiple of every prime factor of m. (c) a ≡ 1 (mod 4) when m ≡ 0 (mod 4). Example 1. When m = 2e , c is relatively prime to m, and a ≡ 1 (mod 4), LCG generates a sequence with full period 2e . A full period is no guarantee that the generator is adequate. Niederreiter [28] presents evidence questioning the use of the following full period generator. Example 2. The LCG generator on the DEC VAX-11 [7] has m = 232 , a = 69, 069, and c = 1. Since these satisfy the conditions of Theorem 1, LCG generates a sequence with full period 232 . When c = 0 there is no need to perform the addition in line 1 of ALCG. As a result, LCG with c = 0 may execute somewhat faster than one with c > 0. Theorem 2 gives the conditions on a, m, and Z0 to maximize the period of {Zi } when c = 0.
6594
PSEUDO-RANDOM NUMBER GENERATORS
Theorem 2. When c = 0 the linear congruential sequence {Zi } has maximal period λ(m) if: (a) Z0 is relatively prime to m. (b) a is a primitive root modulo m. The function λ(m) is the order of a primitive element modulo m. Knuth [18] defines λ(m) explicitly and gives conditions for a to be a primitive root modulo m. In the often encountered case when p = 2 and e > 3, they simplify to the single requirement that a ≡ 3 or 5 (mod 8). Example 3. The generator with m = 232 , a = 69, 069, and c = 0 can generate linear congruential sequences with maximal period λ(232 ) = 230 . It achieves this maximal period when Z0 is relatively prime to m, which in this case is equivalent to Z0 being odd. When this holds, every element in the sequence {Zi } is odd. Marsaglia [25] favors this multiplier on the basis of its performance on the lattice test. He claims that for m = 232 , L2 = 1.06, L3 = 1.29, L4 = 1.30, and L5 = 1.25. However, Niederreiter [28] cautions against using this generator solely on the basis of its performance on the lattice test. Example 4. The RANDU generator with m = 231 , a = 65, 539, and c = 0 has maximal period λ(231 ) = 229 . Several authors note deficiencies in the performance of this generator on the spectral and lattice tests [25,18]. PRIME MODULUS MULTIPLICATIVE GENERATORS When m is prime and c = 0, LCG is a prime modulus multiplicative generator. Theorem 2 shows that for these generators the maximal period is λ(m) = m − 1, an almost full period. Furthermore, since all integers less than m are relatively prime to m, there are no additional restrictions on the seed Z0 . Example 5. On a 32-bit machine the largest prime that can be directly represented in one word is 231 − 1. Generators with this modulus have period 231 − 2. As a result, there are several such generators.
These include (a) LLRANDOM [19,22] with a = 16, 807, used in IMSL [13], APL [15], and SIMPL/I and GPSS [12]; (b) the SIMSCRIPT II simulation language generator for which a = 630, 360, 016 (Kiviat et al. [17]); and (c) the RANUNI generator [31] and the LLRANDOMII generator [21] with a = 397, 204, 094. From an empirical point of view Fishman and Moore [9] study the performance of a multiplicative congruential generator with modulus 231 − 1 and 16 alternative multipliers; the three multipliers mentioned above are included. Although a direct implementation of the prime modulus multiplicative congruential generator following ALCG is feasible, some of the efficiencies associated with substituting a shift operation for the division in line 2 when m = ze can be gained when m is a Mersenne prime. A prime is a Mersenne prime if it can be written in the form 2e − 1. An algorithm attributed to Payne et al. [30] is such an implementation. GENERAL LINEAR CONGRUENTIAL GENERATORS A generalization of the LCG is obtained by making Zi+1 a function not only of Zi but also of Zi−1 , Zi−2 , . . . , Zi−p+1 , as Zi+1 ≡
p
aj Zi−j+1
(mod m). (GLCG)
j=1
By an extension of Theorem 2, for prime modulus m a set {aj |j = 1, . . . , p} can be found such that the linear congruential sequence generated with GLCG has period mp − 1. The coefficients {aj } must be such that the polynomial f (x) = xp + a1 xp−1 + a2 xp−2 + · · · + ap has a root that is a primitive element of the field with mp elements. Then f (x) is a primitive polynomial modulo m, which we refer to as condition PPM(m). Knuth [18] gives criteria for testing for primitivity modulo m. The additive congruential generators are the most popular of the GLCG form. These have aj = 0 or 1 for j = 1, . . . , p.
PSEUDO-RANDOM NUMBER GENERATORS
Example 6. The Fibonacci generator with p = 2, a1 = a2 = 1, and large m has been shown to produce a sequence with poor statistical properties and should be avoided; see Bratley et al. [3]. Example 7. A class of generators, which, Knuth [18] reports, produces sequences with good statistical properties but need further testing, has m ≡ 0 (mod 2), p = 55, and a24 = a55 = 1, aj = 0 for all other j. These generators have period at least as large as 255 − 1. The linear congruential generators modulo 2 are a special case of GLCG with m = 2. This restriction implies that w.l.o.g. the coefficients {aj } can be limited to {0, 1}. As a result, these generators are also a special case of additive congruential generators. When the {aj } are chosen to satisfy condition PPM(2) then {Zi } is a sequence of bits that is called a pseudo-noise sequence or maximallength shift register sequence. Because the {aj } satisfy PPM(2) and m = 2 is a prime, the sequence {Zi } has period 2p − 1. Example 8. Because the trinomial f (x) = x55 + x24 + 1 satisfies PPM(2) the generator Zi ≡ Zi−55 + Zi−24 (mod 2) has period 255 − 1. This generator is one of those in the class shown in Example 7. GENERALIZED FEEDBACK SHIFT REGISTER GENERATORS Tausworthe [34] suggests that a pseudo-noise sequence {Zi } with period 2p − 1 be used as a source of pseudo-random numbers. He examines the properties of the sequence of L-bit numbers {Xi } defined by Xi =
L
2−j Zsi+r−dj
j=1
for i = 0, . . . , (2p − 1)/s ,
(GFSRG)
where {Zi } is a pseudo-noise sequence with period 2p − 1, s L, d = 1, p L, and r is a randomly selected integer on (0, 2p − 1). This transformation of {Zi } is called the L-wise decimation of {Zi }. When s and 2n − 1 are relatively prime {Xi } has period (2p − 1)/s . In this case the transformation of {Zi } is known as proper L-wise decimation of {Zi }.
6595
The sequence {Xi } is sometimes referred to as a Tausworthe sequence and the method of generation as a Tausworthe generator or feedback shift register generator. The restriction s L is important; Knuth [18] states that for s = 1 a sequence with poor statistical properties results. In practice the sequence {Zi }is usually generated using GLCG with j aj = 2, so that Zi ≡ Zi−p + Zi−q
(mod 2).
As a result one need only determine integers p and q for which f (x) = xp + xq + 1 satisfies PPM(2). Primitive trinomials to degree 1000 are given by Zierler and Brillhart [38]. Tausworthe [34] shows that in the limit, as p → ∞, the properly decimated sequence {Xi } has the mean, variance, and autocorrelation structure that one expects from independent samples of a uniform deviate. He also shows that for k p/s the sequence {(Xi+1 , Xi+2 , . . . , Xi+k )} is approximately uniformly distributed over the unit k-hypercube. However, Tootill et al. [35] have shown that {Xi } performs better for some values of p and q than others. They recommend that s = L = q, q p/q be relatively prime to 2p − 1 and that q < p/2 be neither too small nor too close to p/2. They claim if these conditions are satisfied, then GFSRG produces sequences with good runs and uniformity properties. Example 9. Tootill et al. [36] propose a generator with p = 607 and q = 273. For d = 1, L = 23, and s = 512, {Xi } has a period of 2607 − 1. They claim that for simulations with no fixed dimensionality requirements on the pseudo-random sequence, this generator possesses 26-distributed uniformity. Lewis and Payne [20] studied sequences {Xi } generated by GFSRG with d = 100p, s = 1, and L = q. They suggest that d relatively prime to the period 2p − 1 improves the degree to which {Xi } approximates a kdistributed sequence for 1 k p/L . Furthermore, they present an algorithm that
6596
PSEUDO-RANDOM NUMBER GENERATORS
is both extremely efficient and machine independent. Its greatest drawback is an elaborate initialization process. However, Arvillias and Maritsas [1] suggest changes in the algorithm that simplify initialization when q is an integral power of 2. In Bright and Enison [4], Enison shows that with proper initialization the Lewis and Payne [20] scheme results in sequences that are 1-distributed. Fushimi and Tezuka [10] have extended this work to give necessary and sufficient conditions on GFSRG for {Xi } to be k-distributed. Example 10. Bright and Enison [4] recommend that p be a Mersenne exponent so that 2p − 1 is prime. They propose the generator having p = 521, q = 32, L = 64, d = 52, 100, and s = 1. They claim that this generator produces sequences that are 8-distributed. The attraction of these generators comes from the fact that their implementation can be machine independent. The Lewis and Payne [20] algorithm generates an L-bit integer for any value of s. Thus sequences with extremely large periods can be generated on small-word machines. However, the algorithm requires memory for pL-bit integers and a time-consuming initialization. Although initial results from these generators are encouraging, more empirical and structural study is needed before they are adopted widely [8].
The XOR indicates the Boolean exclusiveor operation and the X e(X e) indicates left (right) shift of X by e bits. In practice, methods (b) and (c) are performed on segments of the sequence {Wi } or {Vi }. Knuth [18] presents algorithms and recommends strongly the use of method (c). Brown and Solomon [5] show that method (a) results in sequences with improved k-distributed properties. They warn, however, that improvements in the k-distribution do not necessarily imply improvements in the marginal distributions of subsets. Fishman [8] reports that the LCG generator SUPERDUPER (a = 69, 069, c = 0, m = 232 ) uses method (d) to lengthen the short period evident in its low-order bits. Example 11. The combination of three multiplicative congruential generators has been suggested by Wichmann and Hill [37]. Their goal is to use three generators with short period that can be implemented on a 16bit machine to produce a pseudo-random sequence with long period. Their scheme uses the generators with multipliers 170, 171, and 172, and moduli 30,323, 30,269, and 30,307, respectively. The fractional part of the sum of three uniform deviates is also a uniform deviate; they use this result to combine the three generators. The resulting sequence has period in excess of 244 . They claim that the sequence has good statistical properties, and exhibit a FORTRAN implementation.
SHUFFLING AND SHIFTING: COMBINATIONS OF GENERATORS
LITERATURE
In an attempt to improve the degree to which a pseudo-random sequence approximates a truly random sequence, several authors suggest methods for shuffling the sequence {Zi } or shifting the bits in each Zi . If {Vi } and {Wi } are two sequences, preferably generated from independent generators, then the following schemes have been suggested:
Knuth [18] gives an excellent summary of pseudo-random number generation with emphasis on the congruential generators. Fishman [8], Kennedy and Gentle [16], and Bratley et al. [3] also contain overviews. Bratley et al. [3] go into some detail on feedback shift register generators. Bibliographies can be found in refs. 11, 27, and 29.
(a) (b) (c) (d)
Zi Zi Zi Zi
≡ (Wi + Vi ) (mod m) [18]. = Wk where k = Vi [23]. = Wk where k = Wi [2]. = (Y e1 ) XOR Y, where Y = Wi XOR (Wi e2 ) [8].
REFERENCES 1. Arvillias, A. C. and Maritsas, A. E. (1978). J. ACM, 25, 675–686. (A technical article about GFSR generators.)
PSEUDO-RANDOM NUMBER GENERATORS
6597
2. Bays, C. and Durham, S. D. (1976). ACM Trans. Math. Software, 2, 59–64. (Shifting to improve randomness.)
Mass. (A comprehensive guide to LCG, testing, and implementation, less information on GFSRG.)
3. Bratley, P., Fox, B. L., and Schrage, L. E. (1983). A Guide to Simulation. SpringerVerlag, New York. (An introductory guide that focuses on generator implementation. Several FORTRAN programs are given.)
19. Learmonth, G. and Lewis, P. A. W. (1973). Naval Postgraduate School Random Number Generator Package LLRANDOM. Rep. No. NPS55-LW-73061-A, Naval Postgraduate School, Monterey, Calif.
4. Bright, H. S. and Enison, R. L. (1979). Computing Surv., 11, 357–370.
20. Lewis, T. G. and Payne, W. H. J. ACM, 20, 456–468.
5. Brown, M. and Solomon, H. (1979). Ann. Statist., 7, 691–695.
21. Lewis, P. A. W. and Uribe, L. (1981). The New Naval Postgraduate School Random Number Package LLRANDOMII. Rep. No. NPS55-81005, Naval Postgraduate School, Monterey, Calif.
6. Coveyou, R. R. and MacPherson, R. D. (1967). J. ACM, 14, 100–119. (The basis of the spectral test.) 7. Digital Equipment Corp. (1982). VAX 11 Runtime Library Ref. Manual. AA-D036C-TE. Digital Equipment Corp., Maynard, Mass. 8. Fishman, G. S. (1978). Principles of Discrete Event Simulation. Wiley, New York. (An overview of the area with attention to implementation details, empirical and theoretical tests.) 9. Fishman, G. S. and Moore, L. R. (1982). J. Amer. Statist. Ass., 77, 129–136. (A comprehensive study of alternative prime modulus LCG.) 10. Fushimi, M. and Tezuka, S. (1983). Commun. ACM, 28, 516–523. (k-distributivity of GFSRG.) 11. Hull, T. E. and Dobell, A. R. (1962). SIAM Rev., 4, 230–254. (Early bibliography.) 12. IBM Corp. (1981). PL / I General Purpose Simulation System Program Description / Operations Manual. SH20-6181-0, IBM Corp., White Plains, N.Y. 13. International Mathematical and Statistical Libraries, Inc. (1977). IMSL Library, Houston, Tex. 14. Jansson, B. (1966). Random Number Generators. Almqvist & Wiksell, Stockholm. (Early approaches.) 15. Katzan, H., Jr. (1971). APL User’s Guide. Van Nostrand Reinhold, New York. 16. Kennedy, W. J. and Gentle, J. E. (1980). Statistical Computing. Marcel Dekker, New York. (An overview of the area.) 17. Kiviat, P. J., Villanueva, R., and Markowitz, H. (1969). The Simscript II Programming Language. Prentice-Hall, Englewood Cliffs, N.J. 18. Knuth, D. (1981). The Art of Computer Programming, Vol. 2: Seminumerical Algorithms, 2nd ed. Addison-Wesley, Reading,
(1973).
22. Lewis, P. A. W., Goodman, A. S., and Miller, J. M. (1969). IBM Syst. J., 2, 136–145. 23. MacLaren, M. D. and Marsaglia, G. (1965). J. ACM, 12, 83–89. 24. Marsaglia, G. (1968). Proc. Natl. Acad. Sci. USA, 61, 25–28. 25. Marsaglia, G. (1972). In Applications of Number Theory to Numerical Analysis, S. K. Zaremba, ed. Academic Press, New York, pp. 249–286. 26. Marsaglia, G. and Bray, T. A. (1968). Commun. ACM, 11, 757–759. 27. Nance, R. E. and Overstreet, C. (1972). Computing Rev., 13, 495–508. (Bibliography.) 28. Niederreiter, H. (1978). Bull. Amer. Math. Soc., 84, 957–1041. (Monte Carlo technique and number-theoretic aspects of linear congruential generators.) 29. Page, E. S. (1967). In The Generation of Pseudo-Random Numbers in Digital Simulation in Operational Research, S. H. Hollingsdale, ed. American Elsevier, New York, pp. 55–63. (Bibliography.) 30. Payne, W. H., Rabung, J. R., and Bogyo, T. P. (1969). Commun. ACM, 12, 85–86. 31. SAS Institute, Inc. (1982). SAS User’s Guide: Basics, 1982 Edition. SAS Institute, Inc., Cary, N.C. 32. Sowey, E. R. (1972). Int. Statist. Rev., 40, 355–371. (Bibliography.) 33. Sowey, E. R. (1978). Int. Statist. Rev., 46, 89–102. (Bibliography.) 34. Tausworthe, R. C. (1965). Math. Comp., 18, 201–209. 35. Tootill, J. P. R., Robinson, W. D., and Adams, A. G. (1971). J. ACM, 18, 381–399. 36. Tootill, J. P. R., Robinson, W. D., and Eagle, D. J. (1973). J. ACM, 20, 469–481.
6598
PSEUDO-RANDOM VARIABLE GENERATION, TAUSWERTHE METHOD FOR
37. Wichmann, B. A. and Hill, I. D. (1982). Appl. Statist., 31, 188–190.
REFERENCES
38. Zierler, N. and Brillhart, J. (1969). Inf. Control, 14, 566–569.
1. Kennedy, W. J. and Gentle, J. E. (1980). Statistical Computing. Dekker, New York. 2. Tauswerthe, R. C. (1965). Math. Comp., 19, 201–209. 3. Toothill, J. P. P., Robinson, W. D., and Adams, A. G. (1971). J. Amer. Soc. Comp. Mach., 18, 381–399. 4. Toothill, J. P. P., Robinson, W. D., and Eagle, D. J. (1973). J. Amer. Soc. Comp. Mach., 20, 469–48.
See also GENERATION OF RANDOM VARIABLES, COMPUTER; MONTE CARLO METHODS; NUMBER-THEORETIC METHODS; QUASI-RANDOM SEQUENCES; and RANDOM NUMBER GENERATION.
MARC-DAVID COHEN
PSEUDO-RANDOM VARIABLE GENERATION, TAUSWERTHE METHOD FOR
See also PSEUDO-RANDOM NUMBER GENERATORS; RANDOM NUMBER GENERATION; and RANDOM VARIABLES.
This method (Tauswerthe [2]) is based on the sequence {ak } of 0’s and 1’s produced by a linear recurrence relation
PSEUDO RESIDUALS Yi = f (xi ) + i ,
ak = cp ak−p + cp−1 ak−p+1 + · · · + c1 ak−1 (mod2)
(1)
based on a set (c1 , . . . , cp ) of 0’s and 1’s with cp = 1. The corresponding random integers (to base 2) are formed by successive L-tuples of the a’s, namely Y1 ≡ a1 a2 · · · aL
(base2),
Y2 ≡ aL+1 aL+2 · · · a2L
(base2),
and generally Yj ≡ a(j−1)L+1 a(j−1)L+2 · · · ajL
Consider a regression model of the form
(base2).
This is said to form a L-wise decimation of {ak }. If L is prime relative to 2p − 1, the decimation is said to be proper. A generator using this method is called a Tauswerthe generator. Toothill et al. [3] showed that if nL is prime relative to 2p − 1, then the sequence {Uj } with Uj = 2−L Yj has n-dimensional uniformity. Tauswerthe generators are examples of ‘‘feedback shift register’’ methods. For further details see refs. 3 and 4 and especially ref. 1.
i = 1, . . . , n,
(1)
where i (i = 1, . . . , n) are independent random errors, f is the underlying mean function, and the x’s are assumed to be ordered. Let Var( i ) = σ 2 , i = 1, . . . , n. It is customary to assume some conditions on f in the analysis of this model. Information about one of the terms on the right-hand side of (1) will be helpful in obtaining information on the other. At the outset, when often there is not much is known about f , knowledge of the error variance is helpful in the process of model building. The estimation of σ 2 in classical models has been done through an averaged squared errors of the residuals∗ of a least-squares∗ fit. However, if no parametric assumption is made on f , then the classical averaged-squared-error estimator leads to large biases, and an alternative estimator of σ 2 with smaller bias and variance is needed. Gasser et al. [1] have proposed such an estimator based on pseudo residuals, which are defined in the following manner. Let the error variance σ 2 be constant. Pseudo residuals ˜ i are obtained by taking consecutive triples of design points xi−1 , xi , xi+1 and setting xi+1 − xi xi − xi−1 ˜ i = Yi−1 + Yi+1 xi+1 − xi−1 xi+1 − xi−1 − Yi
(2)
PSEUDO-t STATISTIC, BIRNBAUM’S
This amounts to joining the two outer observations by a straight line at each triplet and computing the difference between this straight line and the middle observation. Thus ˜ i can be written as ˜ i = ai Yi−1 + bi Yi+1 − Yi , i = 1, . . . , n − 1, where ai =
xi+1 − xi , xi+1 − xi−1
bi =
xi − xi−1 . xi+1 − xi−1
If f is twice continuously differentiable, then E(˜ 2i ) = (a2i + b2i + 1)σ 2 + O(n−2 ) and |E(˜ i )| sup{|f |}/n. Hence pseudo residuals have small bias and mean squared errors of order n−1 , in contrast to O(n−4/5 ) for classical residuals obtained from a nonparametric regression curve estimator with an optimal bandwidth∗ . It was then suggested [1] that n−1 1 2 ci ˜ i , S = n−2 i=2
where ci = (a2i + b2i + 1)−1 , be used as an estimator of σ 2 . Closely related estimators of σ 2 have been proposed by Rice [3] and Hall et al. [2]. REFERENCES 1. Gasser, T., Sroka, L., and Jennen-Steinmetz, C. (1986). Residual variance and residual pattern in nonlinear regression. Biometrika, 73, 625–633. 2. Hall, P., Kay, J. W., and Titterington, D. M. (1990). Asymptotically optimal difference based estimation of variance in nonparametric regression. Biometrika, 77, 521–528. 3. Rice, J. (1984). Bandwidth choice for nonparametric regression. Ann. Statist., 12, 1215–1230. See also QUASI RESIDUALS and RESIDUALS.
K. B. KULASEKERA
6599
√ to an independent χ / ν variable (see CHI DISTRIBUTION). It applies (with ν = n − 1) to √ − example, the distribution of n(X n µ)/S, for 2 = (n − 1)−1 2 and X = (X − X) when S i i=1 n−1 ni=1 Xi , X1 , . . . , Xn being independent normal variables with expected value µ and standard deviation σ . A pseudo-t statistic is obtained by replacing S by some other estimator of σ . This estimator need not be an unbiased estimator of σ (indeed, S itself is not unbiased), but it is often felt desirable—perhaps for aesthetic purposes—to arrange that if there is any bias, it is small. Most commonly, S is replaced by an estimator based on the sample range W = max(X1 , . . . , Xn ) − min(X1 , . . . , Xn ). The corresponding pseudo-T is √ n(X − µ)/(an W), where an is chosen so that an E[W] = σ if the population is normally distributed. It is possible to use other denominators, based on the mean deviation∗ or the interquartile range r, for example. Similar modifications can be used in modifying the denominator of the two-sample t. Approximations to the distribution of pseudo-t (often by a regular t with suitable degrees of freedom, usually fractional) have been developed. See also PSEUDO-F and t-DISTRIBUTION.
PSEUDO-t STATISTIC, BIRNBAUM’S Birnbaum [1] proposes a pseudo-t statistic based on three order statistics. Let X(1) X(2) . . . X(2m+1) be the order statistics from a sample of odd size 2m + 1 of a random variable X, and consider u = X(m+1−r) , V = X(m+1) , W = X(m+1+r)
PSEUDO-t See also PSEUDO-F. The regular Student’s t∗ is the ratio of a unit normal variable
for some integer r, 1 r m. Here V is the sample median and W − U is an interquantile range between two sample quantiles.
6600
PSEUDO-VARIANCE
REFERENCES
The Birnbaum statistic is S=
V −µ , W−U
(1)
where µ is the expectation of X. A class of similar statistics was considered by David and Johnson [2]. The statistic S is invariant under linear transformations. Thus, the probability distribution of S for given m and r is the same for any bell-shaped probability distribution defined by a probability density f (x) satisfying 1. f (µ − x) = f (µ + x), x 0, and 2. f (µ + x) is nonincreasing for x 0. Birnbaum suggests using Equation 1 for the family of normal distributions, tabulating the probability distribution of S ‘‘for a reasonable range of r and m’’, and applying S in those situations in which Student’s t is used. He points out that when some of the smallest and some of the largest observations are censored, S is still valid, as long as the values of the sample median X(m+1) and of two order statistics X(m+1−r) and X(m+1+r) are known. He also shows that for any bellshaped probability density, 2m + 1 2r r m−r −r −(m+r) × λ(λ − 1) 2 , λ > 1,
Pr{|S| > λ}
which allows tabulation of the appropriate (approximate) critical values. Exact values of Pr{X > λ} are given [1]. Numerical computation of the distribution of S is quite straightforward. The definition of S can be extended using two-order statistics X(m+1−r) and X(m+1+r ) , where r and r are not necessarily equal, and which may be of practical importance when only censoring from above is considered. In this case, the numerical computation of the distribution of S is somewhat more difficult but not prohibitive. For onesided conclusions about µ, a simpler statistic S = (V − µ)/(W − V) can be used.
1. Birnbaum, Z. W. (1970). ‘‘On a Statistic Similar to Student’s t’’. In Nonparametric Techniques in Statistical Inference, M. L. Puri, ed. Cambridge University Press, Cambridge, UK. 2. David, F. N. and Johnson, N. L. (1956). Some tests of significance with ordered variables. J. R. Stat. Soc., 18, 1–20.
PSEUDO-VARIANCE The 100p% pseudo-variance of a continuous cumulative distribution function (CDF) G(x) is the square of the ratio of the difference between its upper and lower 100p% points to the corresponding difference for the standard normal CDF (x). It is thus equal to v(p) =
G−1 (1 − p) − G−1 (p) −1 (1 − p) − −1 (p)
2 .
For normal distributions, v(p) equals the variance for all p. For heavy-tailed∗ symmetric distributions, v(p) increases as p decreases to zero. Andrews et al. [1] noted that the 4.2% pseudo-variance is remarkably stable for distributions in the Pearson system∗ (see also Pearson and Tukey [3]) and so may serve as a satisfactory index for comparing variability of estimators. Further details can be found in refs. 1 and 2.