ENCYCLOPEDIA OF STATISTICAL SCIENCES Volume 10 Preference Mapping to Recovery of Interblock Information
ENCYCLOPEDIA O...
14 downloads
1500 Views
32MB Size
Report
This content was uploaded by our users and we assume good faith they have the permission to share this book. If you own the copyright to this book and it is wrongfully on our website, we offer a simple DMCA procedure to remove your content from our site. Start by pressing the button below!
Report copyright / DMCA form
ENCYCLOPEDIA OF STATISTICAL SCIENCES Volume 10 Preference Mapping to Recovery of Interblock Information
ENCYCLOPEDIA OF STATISTICAL SCIENCES Second Edition
Volume 10 Preference Mapping to Recovery of Interblock Information
The Encyclopedia of Statistical Sciences is available Online at http://www.mrw.interscience.wiley.com/ess
A John Wiley & Sons, Inc., Publication
CONTRIBUTORS J. R. Abernathy, University of North Carolina, Chapel Hill, NC. Randomized Response Robert J. Adler, The Technion, Haifa, Israel. Random Fields Munir Ahmad, KFUPM, Dhahran, Saudi Arabia. PValues, Combination of James H. Albert, Bowling Green State University, Bowling Green, OH. Probit Models, Bayesian Analysis of Erling B. Andersen, University of Copenhagen, Copenhagen, Denmark. Rasch, Georg J. R. Ashford, University of Exeter, Exeter, United Kingdom. Probit Analysis R. A. Bailey, Rothamsted Experimental Station, Harpenden, United Kingdom. Randomization, Constrained N. Balakrishnan, McMaster University, Hamilton, ON, Canada. Record Values and Record Statistics L. A. Baxter, State University of New York, Stony Brook, NY. Quasi Life Tables Betsy Jane Becker, Michigan State University, East Lansing, MI. P-Values, Combination of M. L. Berenson, Baruch College, New York, NY. Puri’s Expected Normal Scores Test Matilde Bini, Universita degli Studi di Firenze, Firenze, Italy. Raked (Adjusted) Kappa ¨ , Institut fur ¨ Statistik und Okonometrie, Fred Boker Gottingen, Germany. P-Values and Power of Tests K. R. W. Brewer, Commonwealth Schools Commission, Woden, Australia. Primary Sampling Unit; Proportional Allocation Jacob Burbea, University of Pittsburgh, Pittsburgh, PA. Rao Distance S. Cambanis, University of North Carolina, Chapel Hill, NC. Radon–Nikodym Theorem
H. Caussinus, Universit´ee, Paul Sabatier, Toulouse, France. Principal Component Analysis, Generalized David Chappell, University of Sheffield, Sheffield, United Kingdom. Randomness, BDS Test for V. Chew, University of Florida, Gainesville, FL. Radial Error Raj S. Chhikara, National Aeronautics and Space Center, Houston, TX. Proportion Estimation in Surveys using Remote Sensing Siddhartha Chib, Washington University, St. Louis, MO. Probit Models, Bayesian Analysis of Norman Cliff, Carnegie Mellon University, Pittsburgh, PA. Psychometrika Clifford C. Clogg, Pennsylvania State University, University Park, PA. Quasi-Independence Marc-David Cohen, SAS Institute, Cary, NC. PseudoRandom Number Generators Peter Coughlin, University of Maryland, College Park, MD. Probabilistic Voting Models Keren L. Coxe, Washington, DC. Principal Components Regression Analysis ´ Csorg ˝ o˝ , Carleton University, Ottawa, ON, Miklos Canada. Quantile Processes J. N. Darroch, The Flinders University, Bedford Park, South Australia. Quasi-Symmetry C. E. Davis, University of North Carolina, Chapel Hill, NC. Quantile Estimation A. P. Dawid, University College, London, United Kingdom. Prequential Analysis; Probability Forecasting Morris H. Degroot, Carnegie Mellon University, Pittsburgh, PA. Record Linkage and Matching Systems W. E. Deming, Washington, DC. Principles of Professional Statistical Practice B. Dennis, Pennsylvania State University, University Park, PA. Profiles of Diversity v
vi
CONTRIBUTORS
D. Disch, Rose-Hulman Institute of Technology, Terre Haute, IN. Ramsey’s Prior Constantin A. Drossos, University of Patras, Patras, Greece. Probability and Logic Acheson J. Duncan, Johns Hopkins University, Baltimore, MD. Quality Control, Statistical John F. Early, US Bureau of Labor Statistics, Washington, DC. Producer Price Indexes Eugene S. Edgington, University of Calgary, Calgary, AB, Canada. Randomization Tests A. W. F. Edwards, University of Cambridge, Cambridge, United Kingdom. Problem of Points Regina C. Elandt-Johnson, University of North Carolina, Chapel Hill, NC. Rates, Standardized; Rates Eva Elvers, Statistics Sweden, Stockholm, Sweden. Quality Concept for Official Statistics A. K. Md. Eshanes Saleh, Carleton University, Ottawa, ON, Canada. Rank Tests, Grouped Data in Randall L. Eubank, Southern Methodist University, Dallas, TX. Quantile Brian Everitt, Institute of Psychiatry, London, United Kingdom. Psychology, Statistics in Jean-Claude Falmagne, New York University, New York, NY. Psychophysics, Statistical Methods in Robert E. Fay, US Bureau of the Census, Washington, DC. Rao and Scott (Type) Tests Stephen E. Fienberg, Carnegie Mellon University, Pittsburgh, PA. Rasch Model Peter D. Finch, Monash University, Clayton, Australia. Randomization—I Terrence L. Fine, Cornell University, Ithaca, NY. Probability, Foundations of—II; Probability, Foundations of—I David Firth, University of Oxford, Oxford, United Kingdom. Quasiscore Bernard D. Flury, Indiana University, Bloomington, IN. Principal Components; Principal Points G. H. Freeman, University of Warwick, Coventry, United Kingdom. Quasi-Factorial Designs Janos Galambos, Temple University, Philadelphia, PA. Probabilistic Number Theory; Raikov’s Theorem J. L. Gastwirth, George Washington University, Washington, DC. Pyramid Scheme Models Jean Dickinson Gibbons, University of Alabama, Tuscaloosa, AL. Randomized Tests; Randomness, Tests of; Ranking Procedures; P-Values; Rank Tests Richard Gill, University of Utrecht, Utrecht, The Netherlands. Product Integration Michael Goldstein, University of Hull, Hull, United Kingdom. Prevision B. G. Greenberg, University of North Carolina, Chapel Hill, NC. Randomized Response
Raymond S. Greenberg, Emory University, Decatur, GA. Prospective Studies Kenneth H. Gross, California Survey Research, Sherman Oaks, CA. Random-Digit Dialing, Sampling Methods for Emil Grosswald, Temple University, Philadelphia, PA. Rademacher Functions Rudolph Haag, University of Hamburg, Hamburg, Germany. Quantum Mechanics, Statistical Interpretation of C. C. Heyde, University of Melbourne, Parkville, Australia. Probability Theory: An Outline; Quantile Transformation Methods; Random Matrices; Random Sum Distributions Katuomi Hirano, Institute of Statistical Mathematics, Tokyo, Japan. Randomized Gamma Distribution; Rayleigh Distribution Bruce Hoadley, Bell Communications Research, Holmdel, NJ. Quality Measurement Plan (QMP) David C. Hoaglin, Harvard University, Cambridge, MA. Random Number Generation W. Hoeffding, University of North Carolina, Chapel Hill, NC. Probability Inequalities for Sums of Bounded Random Variables; Range-Preserving Estimators D. G. Horvitz, Research Triangle Institute, Research Triangle, NC. Randomized Response J. R. M. Hosking, Institute of Hydrology, Wallingford, United Kingdom. Quenouille’s Test A. Glen Houston, University of Houston, Houston, TX. Proportion Estimation in Surveys using Remote Sensing Nuri T. Jazairi, York University, Toronto, ON, Canada. Productivity Measurement; Purchasing Power Parity Lyle V. Jones, Center for Advanced Study in the Behavioral Sciences, Stanford, CA. Psychological Scaling Eleanor W. Jordan, University of Texas, Austin, TX. Problem Solving in Statistics Brian Junker, Carnegie Mellon University, Pittsburgh, PA. Psychometrika John D. Kalbfleisch, University of Western Ontario, London, ON, Canada. Pseudo-Likelihood Graham Kalton, University of Michigan, Ann Arbor, MI. Question-Wording Effects in Surveys D. Kannan, University of Georgia, Athens, GA. Processes, Discrete ´ , Adam Mickiewicz University, PozMichal Karonski nan, Poland. Random Graphs Richard Kay, University of Sheffield, Sheffield, United Kingdom. Proportional Hazards Model, Cox’s Oscar Kempthorne, Iowa State University, Ames, IA. Randomization—II
CONTRIBUTORS D. G. Kendall, University of Cambridge, Cambridge, United Kingdom. Quantum Hunting; Random Sets of Points K. B. Kulasekera, Clemson University, Clemson, SC. Pseudo Residuals; Quasi Residuals James M. Landwehr, AT & T Bell Laboratories, Murray Hill, NJ. Projection Plots Louis Lefeber, York University, Toronto, ON, Canada. Productivity Measurement A. M. Liebetrau, Batelle Pacific Northwest Laboratories, Richmond, WA. Proportional Reduction in Error (PRE) Measures of Association Frederic M. Lord, Educational Testing Service, Princeton, NJ. Psychological Testing Theory David W. Matula, Southern Methodist University, Dallas, TX. Random Graphs P. McCullagh, Imperial College, London, United Kingdom. Quasi-Likelihood Functions Paul W. Mielke, Jr., Colorado State University, Fort Collins, CO. Quantit Analysis R. E. Miles, Australian National University, Canberra, Australia. Random Tessellations David Moher, University of Ottawa, Ottawa, ON, Canada. Quality Assessment for Clinical Trials Stephen G. Nash, Johns Hopkins University, Baltimore, MD. Quasi-Random Sampling; QuasiRandom Sequences Rajeshwari Natarajan, Southern Methodist University, Dallas, TX. Reciprocal Inverse Gaussian Distribution Bent Natvig, University of Oslo, Oslo, Norway. Priority Queue Valery Nevzorov, St. Petersburg State University, St. Petersburg, Russia. Record Values and Record Statistics J. V. Noble, University of Virginia, Charlottesville, VA. Quantum Mechanics and Probability W. R. S. North, Northwick Park Hospital, Harrow, United Kingdom. Quangle David J. Pack, Union Carbide Corporation, Oak Ridge, TN. Prior Distributions; Prior Probabilities Anthony G. Pakes, University of Western Australia, Nedlands, Western Australia. Queueing Theory S. Panchapakesan, Southern Illinois University, Carbondale, IL. Ranking and Selection Procedures G. P. Patil, Pennsylvania State University, University Park, PA. Profiles of Diversity B. F. Pendleton, University of Akron, Akron, OH. Ratio Correlation A. N. Pettitt, University of Technology, Loughborough, United Kingdom. Rank Likelihood E. C. Pielou, University of Lethbridge, Lethbridge, AB, Canada. Quadrat Sampling
vii
Harry O. Posten, University of Connecticut, Storrs, CT. Quincunx Albert Prat, Universitat Politecnica de Catalunya, Barcelona, Spain. Questii ¨ o´ Sandra Pruzansky, AT & T Bell Laboratories, Murray Hill, NJ. Proximity Data C. P. Quesenberry, North Carolina State University, Raleigh, NC. Probability Integral Transformations Damaraju Raghavarao, Temple University, Philadelphia, PA. Random Balance Designs James O. Ramsay, McGill University, Montreal, PQ, Canada. Principal Differential Analysis C. Radhakrishna Rao, University of Pittsburgh, Pittsburgh, PA. Rao’s Axiomatization of Diversity Measures J. N. K. Rao, Carleton University, Ottawa, ON, Canada. Ratio Estimators—I Poduri S. R. S. Rao, University of Rochester, Rochester, NY. Ratio Estimators—II Amnon Rapoport, University of Haifa, Haifa, Israel. Psychological Decision Making Campbell B. Read, Southern Methodist University, Dallas, TX. Ranges; Rao–Blackwell Theorem Bengt Ros´en, Statistics Sweden, Stockholm, Sweden. Quality Concept for Official Statistics James L. Rosenberger, Pennsylvania State University, University Park, PA. Recovery of Interblock Information William D. Rowe, American University, Washington, DC. Rare-Event Risk Analysis Bernhard Ruger ¨ , University of Munich, Munich, Germany. Profile-a Tests A. Ruiz-Gazen, Universite Toulouse 1-Sciences Sociales, Toulouse, France. Principal Component Analysis, Generalized Harold Sackrowitz, Rutgers University, Piscataway, NJ. Rationality Criterion Ester Samuel-Cahn, Hebrew University, Jerusalem, Israel. Prophet Inequalities Hans T. Schreuder, U.S. Forest Service, Fort Collins, CO. Ratio Estimators, Tin’s; Ratio and Regression Estimators, Mickey’s; Ratio Estimator, Pascual’s P. K. Sen, University of North Carolina, Chapel Hill, NC. Progressive Censoring Schemes; Progressively Censored Data Analysis; Rank Statistics E. Seneta, University of Sydney, Sydney, Australia. Probability, History of L. R. Shenton, University of Georgia, Athens, GA. Quasibinomial Distributions Andrew F. Siegel, University of Washington, Seattle, WA. Rarefaction Curves Gerald L. Sievers, Western Michigan University, Kalamazoo, MI. Probability Plotting
viii
CONTRIBUTORS
Barry Simon, California Institute of Technology, Pasadena, CA. Quantum Physics and Functional Integration C. J. Skinner, University of Southampton, Southampton, United Kingdom. Probability Proportional to Size (PPS) Sampling Grant W. Somes, University of Tennessee, Memphis, TN. Proportions, Cochran’s Q-Statistic for S. M. Steinberg, University of North Carolina, Chapel Hill, NC. Quantile Estimation Stephen M. Stigler, University of Chicago, Chicago, IL. Quetelet, Adolphe S. Lynne Stokes, University of Texas, Austin, TX. Ranked Set Sampling David J. Strauss, University of California, Riverside, CA. Random Utility Models Heinrich Strecker, University of Tuubingen, ¨ Tuubingen, ¨ Germany. Quotient Method Donna F. Stroup, University of Texas, Austin, TX. Problem Solving in Statistics Seymour Sudman, University of Illinois, Urbana, IL. Public Opinion Polls Jiayang Sun, Case Western Reserve University, Cleveland, OH. Projection Pursuit
W. Y. Tan, Memphis State University, Memphis, TN. Quadratic Forms Thaddeus Tarpey, Wright State University, Dayton, OH. Principal Points Rasesh Thakkar, York University, Toronto, ON, Canada. Purchasing Power Parity ¨ Tubingen, ¨ Gerhard Tintner, University of Tuubingen, Germany. Quotient Method R. L. Trader, University of Maryland, College Park, MD. Quasi-Bayesian Inference Brent M. Troutman, U.S. Geological Survey, Lakewood, CO. Rainfall, Landforms, and Streamflow Dietrich Von Rosen, Uppsala University, Uppsala, Sweden. Profile Analysis Gernot Wassmer, Universitat zu Kooeln, ¨ Kooeln, ¨ Germany. Profile-a Tests D. S. Watkins, Washington State University, Pullman, WA. QR Algorithm George H. Weiss, National Institutes of Health, Bethesda, MD. Random Walks D. G. Wellington, Environmental Protection Agency, Washington, DC. Rai and Van Ryzin Dose–Response Model
P
(Continued)
PREFERENCE MAPPING. See GRAPHICAL
consider various properties, such as consistency of our estimator, as n → ∞. However, study of the properties of prequential schemes in the original context of probability forecasting [4,6] suggested that it could be fruitful to reformulate many such nonprequential activities in the prequential setting. The general framework for such ‘‘prequential inference’’ was laid down by Dawid [5]. In it, estimation, for example, is treated merely as an aid to prediction of the next observation. Success at estimation is then measured by the quality of the associated prequential forecasting system, which, in turn, is judged directly by comparing forecasts with actual outcomes. Apart from the pleasing mathematical properties that, it turns out, are possessed by such procedures, it is intuitively more satisfying to formulate inferences in terms of future observables rather than, say, unobservable parameters: for only in such a case can we ever know whether or not we are being successful. Prediction of observables has always been a practically important and theoretically challenging task for probability and statistics: see the Encyclopedia of Statistical Science articles by Kallianpur [15], Ledolter [18], and Geisser [13]. Prequential analysis shares the general concerns of all predictive analysis, but with special emphasis on the assessment of predictive performance through comparison of outcomes with their predictions. In this, it is superficially similar to assessment by cross-validation, or predictive sample reuse [12,26], in which, for a fixed sample, a forecast is made for each data point on the basis of an analysis of all the other points in the sample omitting that one, this forecast then being compared with the omitted value. In prequential analysis, by contrast, the forecast is based on previous values only. This avoids any problems as further data accrue and, more important, eliminates dependence between the assessments for different data points, thus simplifying analysis and interpretation, and permitting the development of a very general and powerful underlying theory.
REPRESENTATION OF DATA
PREFERENCES. See BAYESIAN
INFERENCE;
DECISION THEORY
PREGIBON’S LINK TEST. See LINK TESTS PREMATURE AGGREGATION. See HALF-NORMAL PLOTS
PREQUENTIAL ANALYSIS There are many applied problem areas where data are observed sequentially, and where at any time it is required to issue a forecast, make an inference, or take an action, related to the next observation, on the basis of past and current data. Examples include weather forecasting, playing the stock market, or thermostatic control of a refrigerator. Such problems may be termed prequential, i.e., predictive sequential. In cases where only a point forecast, or some action, is required, it is nonetheless often useful to think of the forecaster as first formulating a probability distribution, Pi+1 say, representing uncertainty about the next observable Xi+1 in the light of the accumulated data Xi = xi [where Xi denotes the initial sequence (X1 , . . . , Xi ) of the variables, and xi := (x1 , . . . , xi ) are their observed values], and then applying this to solve the specific decision problem at hand. A variety of methods is available to assess the quality of such sequential probability forecasts, or their implied decisions, in the light of the actual outcomes [8]. Much of the impetus for these arose from the specific application to meteorology [2,19]. In probability and statistics more generally, we often meet problems in which variables (X1 , X2 , . . .) arise sequentially, but our goals are typically not expressed in this prequential form. For example, we might think of estimating an underlying parameter θ of the joint distribution of X := (X1 , X2 , . . .) on the basis of data Xi = xi , and would then 6391
6392
PREQUENTIAL ANALYSIS
PREQUENTIAL ASSESSMENT OF PROBABILITY FORECASTS Suppose that, for a realized sequence x := (x1 , x2 , . . .) of outcomes of variables X := (X1 , X2 , . . .), a string of probability forecasts P := (P1 , P2 , . . .) has been issued prequentially. Thus Pi+1 is a distribution for Xi+1 , formulated when the data Xi = xi are in hand. One might think of Pi+1 as being formed, as the appropriate conditional distribution for Xi+1 given Xi = xi , from some underlying joint distribution P for X; but this is not necessary. Such an underlying P could be termed a probability forecasting∗ system (PFS), since it determines the forecast for Xi+1 for any i and any data string xi . The general methodology of probability forecasting (see ref. [8], which could usefully be read in conjunction with this article) supplies a number of ways in which we can assess the performance of the issued forecasts P in the light of the actual observations x. In the case of binary (Xi ), we might assess ‘‘calibration’’ up to time n by examining quantities of the form νn−1 {(xi − pi ) : 1 i n; i ∈ ξ }, where pi is the assessed probability Pi (Xi = 1); ξ is some appropriately constructed subsequence (which could be the whole sequence, or, for example, contain just those terms i for which |pi − π0 | < , for some fixed π0 and ); and νn is the number of terms in the summation. For a sequence P to be ‘‘valid,’’ such quantities should tend to zero as n → ∞, since if P is a distribution underlying P this property must hold with P-probability 1 [4]. By considering enough subsequences simultaneously, we can show that valid forecasts are essentially uniquely determined, in the sense that, if P and P are both valid forecast sequences for the same outcome sequence x, we must have pi − pi → 0 as i → ∞ [6]. For continuous (Xi ), we can form, from P and x, the conditional probability integral transform (CPIT) ui := Pi (Xi xi ). Then P may be considered a valid forecast sequence, for data x, if u := (u1 , u2 , . . .) looks like a random sample from the uniform distribution U[0, 1]: again, if P underlies P, this must hold with P-probability 1, thus justifying such a criterion. In particular, for any t and suitable subsequences, we could require that the relative frequency, in the subsequence,
of occurrence of the events Ai : ui t should converge to t; again forecasts valid under this criterion will be essentially unique. Another approach to forecast assessment is to award a ‘‘performance score’’ si ≡ si (xi , Pi ) to the forecast Pi for Xi when in fact Xi = xi , and to compare different sequences Pn of forecasts for the first n outcomes xn by means of their cumulative scores Sn (x, P) := n s (x i i , Pi ). A simple and important special i=1 case has si ≡ log pi (xi ), withpi (·) the density of Pi . Then Sn (x, P) = ni=1 log pi (xi ) = log p(xn ), the overall log likelihood based on data xn for an underlying joint distribution having joint density p(·). We call it the prequential log likelihood for Pn based on xn . Thus, in this case, comparison is simply likelihood-based. If X is generated from a joint distribution P, and Q is any other distribution for X, then Sn (X, P) − Sn (X, Q) is, with P-probability one, bounded below; typically it will tend to infinity. In this sense, forecasts constructed from the true distribution will be optimal. PREQUENTIAL PRINCIPLES The preceding validity criteria have the property that they can be calculated knowing only the realized sequences x and P; in particular, no other features of any joint distribution, or PFS, P underlying the actual forecasts issued are required. This is clearly desirable in applications such as weather forecasting, where it would be unreasonable to expect that such a P has even been formulated. Any criterion with this property will be said to satisfy the weak prequential principle (WPP). The justifications we have given for using such measures, however, have rested on their probabilistic properties under an underlying distribution P (i.e., when the same P used for constructing forecasts is assumed to have generated the data). Nevertheless, these properties, in fact, did not depend on the detailed specification of P. Such probabilistic properties may be said to satisfy the strong prequential principle (SPP). Although indeed strong, it turns out that this is satisfiable for a wide range of interesting problems. Suppose, for example, that, for binary (Xi ), data sequence x, and forecast sequence given by p = (pi ), where pi = Pi (Xi = 1), we wish to
PREQUENTIAL ANALYSIS
assess whether the observed value of the simple calibration measure, n−1 ni=1 (xi − pi ), is ‘‘significant.’’ A natural test statistic, satisfying WPP, is 1/2 n n (xi − pi ) pi (1 − pi ) . yn := i=1
i=1
This is the observed value of 1/2 n n (Xi − pi ) pi (1 − pi ) . Yn := i=1
i=1
If we consider the specific underlying P for which the (Xi ) are independent, with P(Xi = 1) = pi , then Yn has mean 0 and variance 1, and (so long as its denominator tends to ∞) is asymptotically standard normal. For a different underlying P, however, we might have dependent (Xi ); then pi will not be a constant but random, being a function of Xi−1 . Nevertheless, it turns out that the asymptotic standard normal distribution for Yn will still hold, essentially under any such P [24]. So a significance test that refers the observed yn to tables of the standard normal distribution satisfies, asymptotically, (WPP and) SPP. Similarly, many other tests with this property can be constructed. Some understanding of such phenomena can be grounded in the theory of global probability, introduced by Vovk [28], which is essentially determined by realized forecast sequences alone, and is consistent with ordinary Kolmogorov probability under any underlying distribution.
6393
on Xi , and define Pi+1 = Pθ˜ ,i+1 . This is the i plug-in SFS determined by the estimator sequence (θ˜i ). Alternatively, we might assign to θ a prior distribution, say with density π0 (·), construct the mixture P = Pθ π0 (θ )dθ , and use this to generate forecasts. This gives a Bayes SFS. It has Pi+1 = Pθ ,i πi (θ )dθ , where πi (·) is the posterior density of θ given Xi . Example 1. Suppose that, under Pθ , the (Xi ) are independent exponential E(θ ) variables, with density pθ (x) = θ e−θ x (x > 0, θ > 0). The maximum likelihood estimator based on xi is θˆi = i/ti , with ti = ij=1 xj , yielding the MLE plug-in SFS under which Xi+1 ∼ Pi+1 = E(i/ti ). If we give θ a gamma (a, b) prior distribution, (a + i, b + ti ) is the posterior given xi , and the Bayes SFS forecast distribution Pi+1 for Xi+1 is [(b + ti )/(a + i)] × F2,2(a+i) , F·,· denoting the F-distribution. In either case, the underlying P incorporates dependence between the Xi , since Pi+1 depends on Xi through Ti := ij=1 Xj . As i, ti → ∞, the two SFSs produce essentially identical forecasts. An alternative ‘‘fiducial’’ SFS may be constructed by noting that, for any θ , Xi+1 /Ti ∼ (1/i)F2,2i , suggesting Pi+1 = (ti /i)F2,2i , a limiting form of the above Bayes SFS as a, b → 0. (Note that both this and the MLE plug-in SFS require ad hoc modification for i = 0.) As the construction of an SFS is essentially arbitrary, we need to identify those that are ‘‘good,’’ in the sense of having desirable behavior under any Pθ ∈ P. PREQUENTIAL CONSISTENCY
STATISTICAL FORECASTING SYSTEMS Now suppose we have a parametrized family of forecast distribution sequences Pθ = (Pθ ,i , i = 1, 2, . . .), Pθ ,i+1 being the probability forecast made for Xi+1 , in the light of past data xi , under parameter value θ . These might be constructed from an underlying statistical model P = (Pθ ). The statistician is allowed to make use of this model, or its forecasts, in forming her own forecast Pi+1 . A system for so doing, for any data string xi , is termed a statistical forecasting system (SFS) for P. A simple class of SFSs may be constructed as follows. Let θ˜i be an estimator of θ based
In Example 1, under Pθ , Ti → θ −1 almost surely. It follows that any of the forecasts Pi+1 constructed above will be asymptotically indistinguishable from the true forecast distribution E(θ ) under Pθ , no matter what θ is. This desirable property is termed prequential consistency. Typically it will hold even in cases where consistent estimation of θ , in the usual parametric sense, is impossible: see ref. [5, Example 3]. PREQUENTIAL EFFICIENCY Let P be a Bayes SFS for P, with everywhere positive prior density π0 (·), and let Q
6394
PREQUENTIAL ANALYSIS
be an arbitrary PFS. The likelihood ratio in favor of Q as against P remains bounded (and will typically tend to 0) as the number of observations tends to infinity, with probability 1 under Pθ ∈ P, for almost every θ . In this sense, if P is a valid model, a Bayes SFS P cannot be beaten. This property may be termed prequential efficiency. It will also typically (but not universally) hold for the MLE plug-in SFS. Any prequentially efficient SFS P provides, in an optimal way, a single joint distribution which, for predictive purposes, can replace the full family P. MODEL TESTING Suppose we wish to test the validity of the family P of exponential distributions in Example 1. As a simple and concrete example, we might construct, for each i, the discrepancy Xi − µi , where µi is the expectation of Xi under the forecast distribution Pi made by some prequentially efficient SFS; and then assess overall departure, up to time n, by the statistic ni=1 (Xi − µi )/( ni=1 σi2 )1/2 , where σi2 is the variance of Xi under Pi . Using, for simplicity, the MLE plug-in SFS (and omitting the problematic but ignorable term for P1 ) yields n−1 1/2 . It can be (X − T /i)]/[ [ n−1 i+1 i i=1 i=1 (Ti /i)] shown that, for any θ , this will be asymptotically standard normal when the (Xi ) are independent E(θ ) variables. We thus obtain a simple test of the model. In more general models, such test statistics, and more complex variations on them, will again be asymptotically standard normal whenever prequentially efficient forecasts are utilized, under some regularity conditions [25]. Cowell et al. [3] have applied such tests to assess the performance of probabilistic expert systems∗ . MODEL CHOICE [11, § 6.4] set of foreIf (Pk ) is a finite or countable P casting systems, and P = α k k , with αk > 0, αk = 1, is a Bayes SFS for (Pk ), then P is prequentially efficient and consistent for (Pk ). Given data xn , P’s forecast for Xn+1 is αkn Pk,n+1 , where αkn ∝ αk pk (xn ) is the posterior probability of Pk for priors (αk ). Typically, the (Pk ) will, as distributions over sequence
space, be mutually singular. In this case αkn → 1 as n → ∞ with Pk -probability 1, so P will ‘‘home in’’ on the true distribution. Then ˆ where Pˆ n+1 = Pk ,n+1 if kn the plug-in SFS P, n maximizes αkn over k, will likewise be consistent. Now consider a collection (Pk ) of parametric statistical models, where Pk = (Pk,θk ) has finite-dimensional parameter θk with prior density πk (·) > 0. Let Pk bea prequentially efficient SFS for Pk and P = αk Pk , as above. Under mutual singularity of the (Pk ), which will typically hold for distinct models, we then have αkn → 1 with Pk,θk -probability 1 for almost all Pk,θk ∈ Pk . Hence choosing k to maximize the ‘‘adjusted prequential likelihood’’ (or posterior probability) αk pk (xn ) (where we may omit αk if the number of models is finite) will yield a consistent strategy for model selection. Even when the various derived (Pk ) are not mutually singular, the plug-in SFS, or the full mixture SFS αk Pk , will give asymptotically correct forecasts, no matter which Pk,θk holds. In regular models, the prequential log likelihood log pk (xn ) can be approximated by the Jeffreys–Schwarz criterion [26] log pk,θˆk (xn ) − 12 dk log n, where θˆk is the maximum likelihood estimate of θk in Pk based on xn , and dk is the dimensionality of θk ; more refined approximations are given by Phillips and Ploberger [21]. Note that, in the direct prequential approach, any required ‘‘penalization’’ of the likelihood is effected automatically. ROBUSTNESS [9, § 8] Suppose that, although we are using a model P = (Pθ ) to aid prediction, the actual unknown joint distribution generating X is / P. We can still find a SFS which will P∗ ∈ perform effectively as well as will any Pθ ∈ P, with probability 1 under P∗ , for any P∗ . However, the construction of this will depend on the specific decision problem at hand. Suppose that, just before time i + 1, we have to choose an act a, and will suffer a loss depending on a and the value xi+1 of Xi+1 . Without loss of generality we can assume that the act a involves specifying a probability distribution Q for Xi+1 , and that the loss, if in fact Xi+1 = x, is given by a proper scoring rule
PREQUENTIAL ANALYSIS
S ≡ S(x, P) [8]; that is, when X ∼ P, S(P, Q) := E{S(X, Q)} is minimized, in Q, by quoting the true distribution, Q = P. The excess loss in quoting another Q rather than Q = P is then D(P, Q) := S(P, Q) − S(P, P), and this can be used as a measure of discrepancy between distributions, appropriate for the decision problem at hand. For the logarithmic scoring rule S(x, P) ≡ − log p(x), for example, this yields the Kullback–Liebler∗ discrepancy between P and Q. Alternatively, a quadratic loss function yields S(x, P) = [x − EP (X)]2 and the discrepancy D(P, Q) = [EP (X) − EQ (X)]2 . For any infinite data sequence x, we can consider, for each θ , thelimiting average discrepancy, limn→∞ n−1 ni=1 D(P∗i , Pθ ,i ), between the ‘‘true’’ forecasts, based on P∗ , and those based on Pθ ∈ P. A value θ ∗ minimizing this quantity may be regarded as giving the closest approximation to P∗ obtainable from within P (note that θ ∗ may depend on the data x). Since P∗ is unknown, so is θ ∗ . Howthe minimizer of the actual ever, consider θn† , cumulative score ni=1 S(xi , Pθ ,i ) for the initial data sequence xn . Then θn† is calculable from the data, and moreover, under weak conditions we can show that we have θn† → θ ∗ with P∗ -probability 1. That is, choosing θ = θn† by minimizing the actual total score will yield, asymptotically, the closest approximation to the true, unknown P∗ ; and the plug-in SFS formed from the estimator sequence (θn† ) will deliver, asymptotically, the same forecasts as this closest approximation. STRATEGIES In attempting to describe a data sequence x, we might undertake various more or less complex modeling exercises. Almost any standard statistical technique, for example factor analysis∗ , nonlinear time-series∗ analysis, or nonparametric density estimation∗ , may be applied to the partial data xn and used to generate a suitable probability forecast Pn+1 for Xn+1 . Thus any data analysis strategy may be converted into a PFS, and its predictive success assessed accordingly. We can feed this assessment back into the strategy in various ways, to produce a more complex strategy. This in its turn can be converted into a PFS, and its performance assessed.
6395
Suppose, for example, we have fitted a Gaussian AR(1) model, leading to an SFS P. At time n we could examine the CPIT values ui = Fi (xi ), i = 1, . . . , n, with Fi the distribution function of Xi under Pi , for uniformity and independence. Suppose we judge that independence and identical distribution are not at issue, but that the (ui ) do not appear uniformly distributed. Our analysis leading to P thus appears, empirically, to have been unsuccessful. We might however try to estimate the actual distribution of the (ui ), say by nonparametric density estimation, obtaining the estimated distribution function Gn (u). The past data are thus reasonably well described by taking Fi (Xi ) ∼ Gn , independently. If we now apply this ‘‘recalibration’’ correction to the next observation Xn+1 , we might take Fn+1 (Xn+1 ) ∼ Gn (rather than U[0, 1], as under Pn+1 ); this assertion can be transformed into a new predictive distribution, Qn+1 say, for Xn+1 . By continual reassessment of and correction for past performance, we can thus produce a new PFS Q, which we hope will describe the data better. However, we cannot be certain that it will do so, since the correction at any time is based on the past, but applied to the future, which might not continue the same pattern. (Dawid [7] considers some philosophical implications of this lack of guarantee.) In a variety of economic applications, Kling and Bessler [16] found that recalibration generally led to improved performance; but Bessler and Kling [1] gave examples of performance degrading with recalibration. As another application, we might have a collection of candidate models or strategies, as in the section ‘‘Model Choice,’’ and use some strategy based on their forecasts and the outcomes to data to select from or combine these, so as to produce a forecast for the next observation. (Bayes or maximum adjusted likelihood strategies might seem attractive, although their performance need not necessarily be good when the data cannot be regarded as arising from any of the candidate models.) Phillips [20] has applied this approach to 14 historical time series of the U.S. economy, the candidate models being autoregressions of order up to 6, with or without additional linear trend. For 10 of the series, an approximate maximum
6396
PREQUENTIAL ANALYSIS
prequential likelihood strategy had higher prequential likelihood than the generally favored AR(3)+linear trend model, often substantially. Hart [14] uses what he calls ‘‘time-series cross-validation,’’ essentially a prequential least-squares criterion, to choose the bandwith∗ for nonparametric kernel estimation of trend, in the presence of autocorrelated errors, so as to optimize forecast performance. The use of prequential assessment to choose, at each step, among models or strategies is close in spirit to cross-validation [27] or stacked generalization [29], but with the important difference that its output is itself simply another forecasting strategy. This, in turn, can be assessed prequentially and compared with others; and so on to ever deeper levels. At any level, prequential assessment supplies a valid measure of performance, unaffected by overfitting or data dredging. For further discussion, examples, and applications, see ref. [10]. CONNECTIONS WITH OTHER THEORIES Stochastic Complexity The total negative logarithmic prequential likelihood—log p(xn ) of a PFS P, for any data xn , can be regarded as defining the length function of a prefix code for encoding xn ; moreover, this will be, essentially, the shortest possible code when xn can be regarded as generated by the distribution P [11]. When P is prequentially efficient for a model P, this code length defines the stochastic complexity∗ (ref. [23]: see also ref. [22]) of xn relative to P. The theory of stochastic complexity may thus be treated as a special case of prequential analysis. Algorithmic Complexity Kolmogorov and Uspenskii [17] review this approach to probability theory, which attempts to explicate the idea that a given data string x can be considered ‘‘random’’ with respect to some joint distribution P. Equivalently, we might reexpress this as requiring that P be a ‘‘valid’’ PFS for x, as considered above. This reformulation then renders the insights and techniques of each approach, prequential analysis and algorithmic complexity, available to the other.
REFERENCES 1. Bessler, D. A. and Kling, J. L. (1990). Prequential analysis of cattle prices. Appl. Statist., 39, 95–106. 2. Brier, G. W. (1950). Verification of forecasts expressed in terms of probability. Mon. Weather Rev., 78, 1–3. 3. Cowell, R. G., Dawid, A. P., and Speigelhalter, D. J. (1993). Sequential model criticism in probabilistic expert systems. IEEE Trans. Pattern Recognition and Machine Intell., 15, 209–219. 4. Dawid, A. P. (1982). The well-calibrated Bayesian (with discussion). J. Amer. Statist. Ass., 77, 605–613. 5. Dawid, A. P. (1984). Statistical theory. The prequential approach (with discussion). J. Roy. Statist. Soc. A, 147, 278–292. 6. Dawid, A. P. (1985). Calibration-based empirical probability (with discussion). Ann. Statist., 13, 1251–1285. 7. Dawid, A. P. (1985). The impossibility of inductive inference. (Comments on ‘‘Selfcalibrating priors do not exist,’’ by David Oakes.) J. Amer. Statist. Ass., 80, 340–341. 8. Dawid, A. P. (1986). Probability forecasting. In Encyclopedia of Statistical Sciences, Vol. 7, S. Kotz, N. L. Johnson, and C. B. Read, eds. Wiley-Interscience, pp. 210–218. 9. Dawid, A. P. (1991). Fisherian inference in likelihood and prequential frames of reference (with discussion). J. R. Statist. Soc. B, 53, 79–109. 10. Dawid, A. P. (1992). Prequential data analysis. In Current Issues in Statistical Inference: Essays in Honor of D. Basu, M. Ghosh and P. K. Pathak, eds. IMS Lecture Notes–Monograph Ser. 17, pp. 113–126. 11. Dawid, A. P. (1992). Prequential analysis, stochastic complexity and Bayesian inference (with discussion). In Bayesian Statistics 4, J. M. Bernardo, J. Berger, A. P. Dawid, and A. F. M. Smith, eds. Oxford University Press, pp. 109–125. 12. Geisser, S. (1975). The predictive sample reuse method with applications. J. Amer. Statist. Ass., 70, 320–328. 13. Geisser, S. (1980). Predictive analysis. In Encyclopedia of Statistical Sciences, Vol. 7, S. Kotz, N. L. Johnson, and C. B. Read, eds. Wiley-Interscience, pp. 158–170. 14. Hart, J. D. (1994). Automated kernel smoothing of dependent data using time series crossvalidation. J. R. Statist. Soc. B, 56, 529–542.
PREVISION 15. Kallianpur, G. (1986). Prediction and filtering, linear. In Encyclopedia of Statistical Sciences, Vol. 7, S. Kotz, N. L. Johnson, and C. B. Read, eds. Wiley-Interscience, pp. 139– 148. 16. Kling, J. L. and Bessler, D. A. (1989). Calibration-based predictive distributions: an application of prequential analysis to interest rates, money, prices, and output. J. Business, 62, 477–499. 17. Kolmogorov, A. N. and Uspenskii, V. A. (1987). Algorithms and randomness. Theory Probab. Appl., 32, 389–412. 18. Ledolter, J. (1986). Prediction and forecasting. In Encyclopedia of Statistical Sciences, Vol. 7, S. Kotz, N. L. Johnson, and C. B. Read, eds. Wiley-Interscience, pp. 148–158. 19. Murphy, A. H. and Epstein, E. S. (1967). Verification of probabilistic predictions: a brief review. J. Appl. Meteorol., 6, 748–755. 20. Phillips, P. C. B. (1995). Bayesian model selection and prediction with empirical applications. J. Econometrics, 69, 289–331. 21. Phillips, P. C. B. and Ploberger, W. (1995). An asymptotic theory of Bayesian inference for time series. Econometrica, 64, 381–412. 22. Rissanen, J. (1985). Minimum-descriptionlength principle. In Encyclopedia of Statistical Sciences, Vol. 5, S. Kotz, N. L. Johnson, and C. B. Read, eds. Wiley-Interscience, pp. 513–527. 23. Rissanen, J. (1989). Stochastic Complexity in Statistical Inquiry. World Scientific Publishing, Singapore. 24. Seillier-Moiseiwitsch, F. and Dawid, A. P. (1993). On testing the validity of sequential probability forecasts. J. Amer. Statist. Ass., 88, 355–359. 25. Seillier-Moiseiwitsch, F., Sweeting, T. J., and Dawid, A. P. (1992). Prequential tests of model fit. Scand. J. Statist., 19, 45–60. 26. Smith, A. F. M. (1988). Schwarz criterion. In Encyclopedia of Statistical Sciences, Vol. 8, S. Kotz, N. L. Johnson, and C. B. Read, eds. Wiley-Interscience, pp. 289–290. 27. Stone, M. (1974). Cross-validatory choice and assessment of statistical predictions (with discussion). J. R. Statist. Soc. B, 36, 111– 147. 28. Vovk, V. G. (1993). A logic of probability, with application to the foundations of Statistics (with discussion). J. R. Statist. Soc. B, 55, 317–351. 29. Wolpert, D. H. (1992). Stacked generalization. Neural Networks, 5, 241–259.
6397
See also CONSISTENCY; EFFICIENCY, SECOND-ORDER; LIKELIHOOD; PREDICTION AND FORECASTING; PREDICTIVE ANALYSIS; and PROBABILITY FORECASTING.
A. P. DAWID
PREVAILING ITEM An obsolete term for mode (see MEAN, MEDIAN, AND MODE).
PREVISION Prevision is a concept introduced by B. de Finetti to establish the notion of expectation as the fundamental element of the subjectivist theory of probability. Your prevision, P(X), for the random quantity X may be thought of, intuitively, as the ‘‘fair price’’ for the random gain X [i.e., so that you are indifferent between the random gain X and the certain gain P(X)]. (Prevision, as a subjectivist concept, always expresses the judgment of a specified individual, here termed ‘‘you.’’) To make this precise, P(X) must be defined in terms of an operational criterion. One such criterion is as follows. For a general random quantity X, your prevision for X, P(X), is the value x that you would choose if you were subsequently to suffer a penalty L given by L = k(X − x)2 , where k is a constant defining the units of loss. Your previsions x1 , . . . , xm for the random quantities X1 , . . . , Xm are said to be coherent if there is no other choice x∗1 , . . . , x∗m which would lead to a reduction in your overall penalty for all possible combinations of values of X1 , . . . , Xm . (Thus, if you are not coherent, you are a ‘‘sure loser.’’) Coherence is here used in the same intuitive sense as for the axiomatic development of inference and decision-making (see COHERENCE), namely as a criterion to identify intrinsically contradictory beliefs. For any random quantities X and Y, coherence imposes the following requirements on your specifications: (1) infX P(X) sup X, and (2) P(X + Y) = P(X) + P(Y). These properties reflect the ‘‘fair price’’ interpretation. (Notice that prevision is finitely additive but not necessarily countably additive.) [(1) and (2) are special cases of the general result that a specification P(Yi ) = yi , i = 1, . . . , m, is
6398
PRIESTLEY’S TEST FOR HARMONIC COMPONENT(S)
coherent if and only if the point (y1 , . . . , ym ) lies in the closed convex hull of the set of possible values of the random vector (Y1 , . . . , Ym ) in m-dimensional Euclidean space.] Prevision can be considered to be expectation, but defined directly as a primitive quantity, rather than in terms of a prespecified probability distribution. For any event H, if you also use H to denote the indicator function of the event (i.e., H = 1 if H is true, H = 0 if false), then you may interpret P(H) as the probability of H. In this development there is no distinction between probability and expectation, and the notation is chosen to emphasize this. If you have specified a full probability distribution for X, your prevision for X will agree with the usual definition of expectation (at least if X is bounded; there are subtleties for unbounded X). The essential difference arises when you do not want to assess probabilities for all the possible outcomes, in which case P(X), being assigned directly, is well defined, as opposed to your expectation for X, which is defined only through the associated probability distribution for X. Thus you may consider ‘‘average’’ properties without exhaustively considering some limiting partition of possibilities, so that you are free to choose the level of detail at which you wish to describe your beliefs. Using H interchangeably as the event and the indicator function of your event, your prevision of the random quantity X, conditional on the event H, P(X|H), is defined exactly as for P(X), but with the penalty L replaced by the ‘‘called off’’ penalty HL. Coherence imposes the following requirement: (3) P(HX) = P(H)P(X|H). The essential reference on prevision is de Finetti [1,2], where all the foregoing material is developed. Much of the importance of prevision lies in the depth and thoroughness of the treatment which de Finetti provides in these two volumes. Some statistical implications of this formulation are developed in Goldstein [3] as follows. In the usual Bayesian formulation, your beliefs are collected into a prior probability distribution which is modified by Bayes’ theorem in the light of new information to yield a posterior∗ probability distribution. When working with prevision rather than probability, it is shown
that your beliefs should be collected into a prior inner product space, which is modified by a linear transformation, in the light of new information, to yield a posterior inner product space. The relationship between the conditional probability approach and the linear transformation approach is discussed in detail in Goldstein [5]. These developments rely on the condition that if PT (X) is the actual value that you will assign for P(X) at the future time T, then your current previsions should satisfy the relation: (4) P(PT (X)) = P(X). [This result would not, in general, be true if PT (X) were replaced by conditional prevision.] A coherence argument for (4) is given in Goldstein [4]. REFERENCES 1. de Finetti, B. (1974). Theory of Probability, Vol. 1. Wiley, London. 2. de Finetti, B. (1975). Theory of Probability, Vol. 2. Wiley, London. 3. Goldstein, M. (1981). J. R. Statist. Soc. B, 43, 105–130. 4. Goldstein, M. (1983). J. Amer. Statist. Ass., 78, 817–819. 5. Goldstein, M. (1984). In Bayesian Econometrics and Statistics, Vol. 3 (in honour of B. de Finetti), Prem. K. Goel and A. Zellner, eds. North-Holland, Amsterdam (in press) See also COHERENCE; CONDITIONAL PROBABILITY AND EXPECTATION; EXPECTED VALUE; and SUBJECTIVE PROBABILITIES.
MICHAEL GOLDSTEIN
PRIESTLEY’S TEST FOR HARMONIC COMPONENT(S). See TIME SERIES, PRIESTLEY’S HARMONIC COMPONENT TEST FOR
PRIMARY SAMPLING UNIT This is a term used in multistage sampling. At the first stage of sampling it is necessary to divide the entire population or universe into a convenient number of primary sampling units (PSUs). In large-scale area samples∗ it is usual to form a few thousand PSUs, from which a few hundred would be selected
PRIMARY SAMPLING UNIT
to constitute the first-stage sample. Thus in area samples of the United States it is not uncommon for the PSUs to be counties. The second stage of selection in multistage sampling takes place only within those PSUs which form the first-stage sample. Secondary sampling units, for instance census tracts, need to be listed only within those selected PSUs. If the number of selected PSUs is held constant, the consequences of forming large PSUs are that many second-stage units need to be listed and that those selected are scattered over relatively large areas. Conversely, if the PSUs are made small, the task of selecting at the second stage involves forming relatively short lists of second-stage units, and those selected are geographically concentrated. It is usual, particularly in area sampling, for primary sampling units to have unequal probabilities of selection or of inclusion in a sample. These selection or inclusion probabilities are usually proportional to some measure of size, such as the number of persons or dwellings recorded in the PSU at the latest available census. There are mathematical and operational advantages in selecting PSUs ‘‘with replacement.’’ To select with unequal probabilities with replacement, the PSUs within each (usually geographical) stratum are assigned selection probabilities which sum to unity. A series of n sample draws is then made using these probabilities. It is possible for a PSU to be selected more than once, hence the expression ‘‘with replacement.’’ (Because the same probabilities are used at each draw, the distribution of the numbers of times each PSU is selected in a sample is multinomial∗ . Since ‘‘sampling with replacement’’ does not necessarily imply that the same probabilities of selection are used at each draw, this type of sampling is occasionally referred to as ‘‘multinomial.’’) If sampling is to be without replacement at the first stage, the probabilities of inclusion in the sample are usually chosen to sum to n, an integer, which is the required sample number or sample size. A procedure then has to be specified for selecting the sample in accordance with these inclusion probabilities. Either this procedure itself will be complicated, or the variance estimation will be
6399
complicated, inefficient or biased, or the sample must be limited to two units. Sampling ‘‘with replacement’’ (multinomial sampling) is less efficient but can be distinctly more convenient. In particular, if the PSUs are selected ‘‘with replacement,’’ then regardless of the selection methods used at subsequent stages, it is possible to obtain an unbiased estimator of variance (of the unbiased estimator of mean or total) by comparing the estimates based on the individual PSUs. Since these were selected independently, the estimates based on them are also independent. This applies only to estimation of the variance over all stages of sampling combined. To obtain simple unbiased estimators of the variances at the individual stages, sampling must be ‘‘with replacement’’ at the subsequent stages as well as the first (see STRATIFIED MULTISTAGE SAMPLING). In the event that a PSU is selected twice or three times, it is usual to double or triple the number of second-stage sample units to be selected within it. This may in some cases involve multiple selection of second- or even lower-stage units. If final-stage units are selected more than once, unequal weighting must be used in the sample estimator. Since area sampling effects a great simplification in the sampling procedure and the resulting clustered samples∗ are much cheaper to enumerate than unclustered samples, this technique is widely used in largescale household surveys such as the Current Population Survey∗ and those of the major private polling organizations in the United States and other developed countries. Although the term ‘‘primary sampling unit’’ is seldom used outside the context of area sampling, it is not confined to it. Thus if a two-stage sample is used in which the first stage consists of a sample of boxes and the second of cards within those boxes, the boxes would be the PSU’s. BIBLIOGRAPHY Barnett, V. (1974). Elements of Sampling Theory. Hodder & Stoughton, London, Chap. 5. (Concise treatment at an intermediate mathematical level.)
6400
PRIMITIVE ROOT
Sudman, S. (1976). Applied Sampling. Academic Press, New York. Chap. 7. (For survey users with limited statistical backgrounds.)
properties of their proposals. One of these proposals consists in letting T=
See also AREA SAMPLING; STRATIFIED MULTISTAGE SAMPLING; and SURVEY SAMPLING.
K. R. W. BREWER
PRIMITIVE ROOT In any finite field (Galois field∗ ), there is at least one element in the field, different powers of which give all the nonzero elements of the field. Such an element is called a primitive root of the field. For example, if the number of elements in the field is p = 7, then 3 is a primitive root since (modulo p) 31 = 3,
32 = 2, 35 = 5,
33 = 6, and
34 = 4,
36 = 1.
(Actually, X p−1 = 1 when X is any element of the field.) See also CONFOUNDING; FRACTIONAL FACTORIAL DESIGNS; and GALOIS FIELDS.
PRINCIPAL COMPONENT ANALYSIS, GENERALIZED Let V be the variance matrix of the pdimensional data X 1 , . . . , X n and T be a positive definite p × p matrix. While classical principal component analysis∗ (PCA) rests on the spectral decomposition of V, generalized PCA rests on the eigen analysis of VT −1 and the T −1 -orthogonal projection (instead of the canonical projection) of the centered data on the subspace Eq of Rp spanned by the q (q < p) eigenvectors associated with the q largest eigenvalues. Various choices of T give rise to interesting projections of the data. In order to display clusters, Art et al. [1] consider a withincluster dispersion matrix T estimated without knowledge of the clusters. In a similar approach displaying outliers∗ [4], the matrix T is a robust estimate of the total variance. Caussinus and Ruiz-Gazen [2,3] propose to simplify these methods and give some
n−1 n i=1
exp[−(β/2)||X i −X j ||2 −1 ](X i −X j )(X i −X j ) j=i+1 V , n−1 n exp[−(β/2)||X i −X j ||2 −1 ] j=i+1 i=1 V
where the squared norm ||x||2V −1 = x V −1 x for any column vector x of Rp . The use of an exponential weighting of the pairwise differences in the definition of T permits the authors to develop theoretical results. These results are obtained for large samples from a mixture distribution∗ Np (x, W)dP(x), where P, the nonnormal distribution of interest, is concentrated on an unknown k-dimensional subspace Fk of Rp (k < p), and Np (x, W), the p-dimensional normal density with mean x and covariance W, is the noise. For instance, Ek tends to Fk when n goes to infinity if P is a discrete distribution defining clusters, W is small enough with respect to the variance of P, and β is suitably chosen (in practice, 1 β 2). Since k is unknown, means are also provided to assess an appropriate dimension for a useful projection. All the displays are affine invariant. These techniques can be considered as alternatives to projection pursuit∗ . REFERENCES 1. Art, D., Gnanadesikan, R., and Kettenring, J. R. (1982). Data based metrics for cluster analysis. Utilitas Math., 21, 75–99. 2. Caussinus, H. and Ruiz-Gazen, A. (1994). Projection pursuit and generalized principal component analyses. In New Directions in Statistical Data Analysis and Robustness, S. Morgen¨ thaler et al., eds. Birkhauser, Basel, pp. 35–46. 3. Caussinus, H. and Ruiz-Gazen, A. (1995). Metrics for finding typical structures by means of principal component analysis. In Data Science and its Applications, Y. Escoufier et al., eds. Academic Press, pp. 177–192. 4. Yenyukov, I. S. (1988). Detecting structures by means of projection pursuit. COMPSTAT 88, Physica-Verlag, Heidelberg, pp. 47–58. See also COMPONENT ANALYSIS; PRINCIPAL COMPONENTS; and PROJECTION PURSUIT.
H. CAUSSINUS A. RUIZ-GAZEN
PRINCIPAL COMPONENTS
PRINCIPAL COMPONENTS This entry is an update to COMPONENT ANALYSIS. It focuses on developments during the past decade, highlighting mostly modelbased generalizations of principal component analysis. The topics discussed are: (1) generalizations to several groups; (2) nonlinear generalizations; (3) principal components in classification and regression; (4) patterned covariance matrices; and (5) varia, including testing problems and robust estimation. Other recent developments are treated in three books [9,19,23] that deal exclusively with principal component analysis. GENERALIZATIONS TO SEVERAL GROUPS When multivariate data fall into natural groups, such as males and females of the same species, the investigator may be interested in comparing principal components across groups. A descriptive method achieving this goal consists of computing angles between the linear subspaces spanned by the first q (out of p) principal components in each group [27]. In a model setup, assume that the principal component directions (eigenvectors of the covariance matrices) are identical across groups, while the variances may differ. Specifically, let i denote the covariance matrix of a p-variate random vector X in the ith group, and let i = Bi i Bi be its spectral decomposition, where the columns of the orthogonal matrix Bi are the eigenvectors of i , and i is a diagonal matrix with the eigenvalues of i as diagonal entries. The common principal component (CPC) model [9] assumes that all Bi are equal, i.e., all i can be diagonalized simultaneously by a single orthogonal matrix B. With B = (β 1 · · · β p ) and i = diag(λi1 , . . . , λip ), the CPC model can be written as i = Bi B =
p
λij β j β j .
6401
estimates of the parameters, based on k sample covariance matrices Si , can be computed using a Jacobitype algorithm for simultaneous diagonalization [10]. The first principal component of logtransformed measurements is often used in morphometrics to estimate constants of allometric growth [22,24]. In the case of several related species, the investigator may want to estimate identical constants of allometric growth for all groups under consideration [1,9], without imposing the constraint of equality of all eigenvectors of the k covariance matrices. A modification of the CPC model, the partial CPC model, assumes that i = λi1 β 1 β 1 +
p
(i) λij β (i) j βj ,
i = 1, . . . , k.
j=2
Here, the orthogonal matrices Bi = (β 1 β (i) 2 ··· β (i) ) have one vector in common, while the p remaining ones are allowed to differ between groups. In a variation of this model, q 2 eigenvectors of each covariance matrix span the same subspace [9,31,34]. Another modification of the CPC model imposes constraints λij = ci λj on the diagonal matrices i ; this results in the model of proportional covariance matrices i = ci 1 ,
i = 2, . . . , k,
for proportionality constants c2 , . . . , ck [8,9,21,30]. Related approaches in the psychometric literature are called three-mode∗ principal component analysis [26]. They are based on three- and higher-way data matrices (e.g., data arranged by variables, objects, and experimental conditions), and originated in the psychometric literature, but have been further developed in chemometrics∗ [17,18,42,43]. These methods rely on algebraic decompositions of higherorder data matrices and are not explicitly based on parametric models (see LINEAR ALGEBRA, COMPUTATIONAL). Solutions are usually obtained by a least squares criterion.
j=1
NONLINEAR GENERALIZATIONS The CPC model holds trivially if all covariance matrices are equal or proportional. Normal-theory maximum likelihood
A principal curve, or self-consistent curve, of a p-variate random vector X is a smooth
6402
PRINCIPAL COMPONENTS
curve in Rp such that each point x on the curve is the conditional mean of X in the subset of all points in Rp that are closer to x than to any other point on the curve [16]. For elliptical distributions, principal-component axes are self-consistent curves; thus principal curves are a genuine generalization of ordinary linear principal components. For a bivariate spherically symmetric random vector X, the circle of radius E[ X ] centered at the mean of X is also self-consistent. Self-consistent curves other than lines can typically only be found by iterative computations similar to a k-means algorithm∗ . In addition, for estimation of principal curves based on observed data one needs a scatterplot smoothing technique [2,16,39]. Principal curves can be generalized to principal surfaces, i.e., smooth manifolds∗ that generalize subspaces spanned by subsets of eigenvectors in linear principal component analysis [16]. While principal curves generalize principal components using self-consistency∗ as the defining criterion, smallest additive principal components [7] generalize the principal components associated with the smallest eigenvalues by constructing additive funcp tions i=1 i , where the i (Xi ) are nonlinear functions of the original variables, such that var[ i ] is a minimum subject to the constraint var[i ] = 1. PRINCIPAL COMPONENTS IN CLASSIFICATION AND REGRESSION In discriminant analysis∗ , principal components are sometimes used to reduce the dimensionality of the data, but the success of such a procedure depends on the assumption that discarding principal components with small eigenvalues will not, at the same time, discard valuable information on differences between the groups. A different application of principal components in discriminant analysis imposes constraints like the ones discussed above in connection with the covariance matrices [3,14,30]. This leads to classification∗ rules that are quadratic but reduce the number of parameters estimated, which may improve the expected error rates. Principal components are also used frequently in cluster analysis∗ to detect
homogeneous subsets of data; in particular, a scatterplot of the first two components often reveals structure. Principal points of a p-variate random vector X are points that approximate the distribution of X optimally in the least squares sense; if the distribution of X is elliptical, then principal points lie in the subspace spanned by the first eigenvectors of the covariance matrix of X, thus justifying the use of principal components [36,37]. The principal components of regressor variables may be used in multiple regression to combat the effects of collinearity∗ ; see PRINCIPAL COMPONENTS REGRESSION ANALYSIS. By omitting components with small eigenvalues the stability of parameter estimates is improved while a bias may be introduced, similarly to ridge regression∗ . Nonlinear functions of the principal components, including their products and powers, may be used to improve prediction [29]. PATTERNED COVARIANCE MATRICES Suppose a pk-variate random vector X is partitioned into k vectors X1 , . . . , Xk of dimension p each; for instance, Xi may be a set of p variables measured in the ith stadium of growth of an insect. The pk × pk covariance matrix of X, partitioned into submatrices ij of dimension p × p each, may then display some distinct structure, and models similar to those of the first section can be defined [12]. For instance, one may assume that the same orthogonal p × p matrix B diagonalizes all ij simultaneously; this leads to the CPC model for dependent random vectors [25] used in studies of static and ontogenetic allometry∗ . For example, for k = 2 the model is 11 12 B O 11 12 B O = , O B 21 22 21 22 O B where B is an orthogonal p × p matrix, and all ij are diagonal. If this model holds, then the canonical correlations between X1 and X2 are the same as the correlations between the common principal components [12]. Assuming that B is nonsingular but not necessarily orthogonal leads to the common canonical variates model [28]. See also the references on three-way principal component analysis given at the end of the first section.
PRINCIPAL COMPONENTS
VARIA Robust estimation∗ of principal components is discussed in refs. 5, 20; for common principal components, in ref. [13]. References [6, 43] use projection pursuit∗ to define robust principal components, and to detect clusters and outliers∗ . The influence function∗ [4,20] provides a tool to assess the effect of single observations on estimates of eigenvalues and eigenvectors. See ref. [4] for a comprehensive review. Various tests for hypotheses are developed in [32,33,40,41]. The T 2 statistic for equality of the mean vectors of two p-variate populations can be decomposed into p parts corresponding to principal components [35]. This decomposition can be generalized by assuming that the two populations differ in location and in variability, but only in q < p common principal component directions [11]. The same approach allows us to test the model of allometric extension. The biometric, chemometric, and psychometric literature abounds with applications of principal component analysis and its various extensions, for example, to process monitoring in paper production [42], water chemistry [18], soil contamination [43], and regional variation in growth of insects [24]. Future directions for research should include models for high-dimensional data, in particular data measured in continuous time, and Bayesian estimation. On the latter topic there is only a very limited amount of literature available [15,38], and none related to the various generalizations presented in this update. REFERENCES 1. Airoldi, J. P. and Flury, B. (1988). An application of common principal component analysis to cranial morphometry of Microtus californicus and M. ochrogaster (Mammalia, Rodentia). J. Zool. (London), 216, 21–36. 2. Banfield, J. D. and Raftery, A. E. (1992). Ice floe identification in satellite images using mathematical morphology and clustering about principal curves. J. Amer. Statist. Ass., 87, 7–15. 3. Bensmail, H. and Celeux, G. (1996). Regularized Gaussian discriminant analysis through eigenvalue decomposition. J. Amer. Statist. Ass., 91, 1743–1748.
6403
4. Brooks, S. P. (1994). Diagnostics for principal components: influence functions as diagnostic tools. Statistician, 43, 483–494. 5. Campbell, N. A. (1980). Robust procedures in multivariate analysis I: robust covariance estimation. Appl. Statist., 29, 231–237. 6. Caussinus, H. and Ruiz-Gazen, A. (1993). Projection pursuit and generalized principal component analysis. In New Directions in Statistical Data Analysis and Robustness, S. Morgenthaler, E. Ronchetti, and W. A. Stahel, eds. ¨ Birkhauser, Basel, pp. 35–46. 7. Donnell, D. J., Buja, A., and Stuetzle, W. (1994). Analysis of additive dependencies and concurvities using smallest additive principal components. Ann. Statist., 22, 1635–1673. 8. Eriksen, P. S. (1987). Proportionality of covariance matrices. Ann. Statist., 15, 732–748. 9. Flury, B. (1988). Common Principal Components and Related Multivariate Models. Wiley, New York. (Model-based introduction to principal component analysis, with extensions to common principal components, partial common principal components, and proportional covariance matrices, including distribution theory and numerical algorithms.) 10. Flury, B. and Gautschi, W. (1986). An algorithm for simultaneous orthogonal transformation of several positive definite symmetric matrices to nearly diagonal form. SIAM J. Sci. Statist. Comput., 7, 169–184. 11. Flury, B., Nel, D., and Pienaar, I. (1995). Simultaneous detection of shift in means and variances. J. Amer. Statist. Ass., 90, 1474–1481. 12. Flury, B. and Neuenschwander, B. (1994). Principal component models for patterned covariance matrices, with applications to canonical correlation analysis of several sets of variables. In Descriptive Multivariate Analysis, W. J. Krzanowski, ed. Oxford University Press, pp. 90–112. 13. Flury, B. and Novi Inverardi, P. L. (1992). Robust estimation of common principal components. Quad. Statist. e Mat. Appl. Sci. Econ.–Soc., 14, 49–80. 14. Flury, B., Schmid, M., and Narayanan, A. (1994). Error rates in quadratic discrimination with constraints on the covariance matrices. J. Classification, 11, 101–120. 15. Geisser, S. (1965). Bayesian estimation in multivariate analysis. Ann. Math. Statist., 36, 150–159. 16. Hastie, T. J. and Stuetzle, W. (1989). Principal curves. J. Amer. Statist. Ass., 84, 502–516.
6404
PRINCIPAL COMPONENTS
17. Henrion, R. (1993). Body diagonalization of core matrices in three-way principal component analysis: theoretical bounds and simulation. J. Chemometrics, 7, 477–494. 18. Henrion, R. (1994). N-way principal component analysis: theory, algorithms, and applications. Chemometrics and Intell. Lab. Systems, 25, 1–23. 19. Jackson, J. E. (1991). A User’s Guide to Principal Components. Wiley, New York. (An easyto-read introduction to principal component analysis and its applications, including many real and synthetic data sets, and connections to other multivariate methods.) 20. Jaupi, L. and Saporta, G. (1993). Using the influence function in robust principal component analysis. In New Directions in Statistical Data Analysis and Robustness, S. Morgenthaler, E. Ronchetti, and W. A. Stahel, eds. ¨ Birkhauser, Basel, pp. 147–156. 21. Jensen, S. T. and Johansen, S. (1987). Estimation of proportional covariances. Statist. Probab. Lett., 6, 83–85. 22. Jolicoeur, P. C. (1963). The multivariate generalization of the allometry equation. Biometrics, 19, 497–499. 23. Jolliffe, I. T. (1986). Principal Component Analysis. Springer, New York. (A very readable introduction to principal-component analysis with many illustrative examples, including applications to factor analysis and regression.) 24. Klingenberg, C. P. (1996). Multivariate allometry. In Advances in Morphometrics, L. F. Marcus et al., eds. Plenum, New York, pp. 23–49. 25. Klingenberg, C. P., Neuenschwander, B., and Flury, B. (1996). Ontogeny and individual variation: analysis of patterned covariance matrices with common principal components. Systematic Biol., 45, 135–150. 26. Kroonenberg, P. M. (1983). Three-Mode Principal Component Analysis. DSWO Press, Leiden, The Netherlands. (A comprehensive review of the psychometric literature on extensions of principal component analysis to threeway data up to 1983.) 27. Krzanowski, W. J. (1979). Between-groups comparison of principal components. J. Amer. Statist. Ass., 74, 703–707. Correction, 76, 1022. 28. Neuenschwander, B. and Flury, B. (1995). Common canonical variates. Biometrika, 82, 553–560. 29. Oman, S. D., Næs, T., and Zube, A. (1993). Detecting and adjusting for non-linearities
in calibration of near-infrared data using principal components. J. Chemometrics, 7, 195–212. 30. Owen, A. (1984). A neighbourhood-based classifier for LANDSAT data. Can. J. Statist., 12, 191–200. 31. Schott, J. R. (1988). Common principal component subspaces in two groups. Biometrika, 75, 229–236. 32. Schott, J. R. (1988). Testing the equality of the smallest latent roots of a correlation matrix. Biometrika, 75, 794–796. 33. Schott, J. R. (1991). A test for a specific principal component of a correlation matrix. J. Amer. Statist. Ass., 86, 747–751. 34. Schott, J. R. (1991). Some tests for common principal component subspaces in several groups. Biometrika, 78, 771–777. 35. Takemura, A. (1985). A principal decomposition of Hotelling’s T 2 statistic. In Multivariate Analysis—VI, P. R. Krishnaiah, ed. Elsevier, Amsterdam, pp. 583–597. 36. Tarpey, T. (1995). Principal points and selfconsistent points of symmetric multivariate distributions. J. Multivariate Anal., 53, 39–51. 37. Tarpey, T., Li, L., and Flury, B. (1995). Principal points and self-consistent points of elliptical distributions. Ann. Statist., 23, 103–112. 38. Tiao, G. C. and Fienberg, S. (1969). Bayesian estimation of latent roots and vectors with special reference to the bivariate normal distribution. Biometrika, 56, 97–108. 39. Tibshirani, R. (1992). Principal curves revisited. Statist. and Comput., 2, 183–190. 40. Tyler, D. E. (1981). Asymptotic inference for eigenvectors. Ann. Statist., 9, 725–736. 41. Tyler, D. E. (1983). A class of asymptotic tests for principal component vectors. Ann. Statist., 11, 1243–1250. 42. Wold, S. (1994). Exponentially weighted moving principal components analysis and projections to latent structures. Chemometrics and Intell. Lab. Systems, 23, 149–161. 43. Xie, Y., Wang, J., Liang, Y., Sun, L., Song, X., and Yu, R. (1993). Robust principal component analysis by projection pursuit. J. Chemometrics, 7, 527–541. See also ALLOMETRY; CLASSIFICATION —I; COMPONENT ANALYSIS; DISCRIMINANT ANALYSIS; PRINCIPAL COMPONENTS REGRESSION ANALYSIS; and THREE-MODE ANALYSIS.
BERNARD D. FLURY
PRINCIPAL COMPONENTS REGRESSION ANALYSIS
r = 1, 2, . . . , p, where a r = (a1r , a2r , . . . , apr ) with constraints a r as = δrs , where δrs = 1 if r = s and is zero otherwise.
PRINCIPAL COMPONENTS REGRESSION ANALYSIS Principal components (PC) regression is often used to solve the problem of multicollinearity∗ [3–5, 8], especially when deleting independent variables to break up the multicollinearity is not a feasible option [3]. One also uses PC regression when there are more independent variables than observations [5]. Basically, the way that PC regression works is that one first finds the set of orthonormal eigenvectors of the correlation matrix of the independent variables. The matrix of PCs is then the product of the eigenmatrix with the matrix of independent variables. The matrix of PCs is such that the first PC will have the maximum possible variance, the second the maximum possible variance among those uncorrelated with the first, and so on. The set of PCs then becomes the set of new regressor variables. Upon completion of this regression, one transforms back to the original coordinate system. Let the ordinary least squares∗ (OLS) model be defined (the vector and matrix dimensions are given in parentheses) as Y =
(n×1)
+
γ0 1 (1×1)(n×1)
Xγ (n×p)(p×1)
+ . (n×1)
i
(Xij − X j )2
λ1 λ2 · · · λp 0.
(5)
Then, if A = (a1 , . . . , ap ), A x xA = Z Z =
and
x x = AA , (6)
since A is orthogonal, where Z = xA is the PC matrix and Z Z is the new design matrix. Then r = 1, . . . , p, (7) var(zr ) = z r zr = λr , tr(x x) = p = z r zr = λr implies that λi /p represents the proportionate contribution of the ith PC to the total variation of the x’s. McCallum [8] has noted that y = xβ + = ZA β + ,
(8)
which implies that h = A β
1/2
√ = (Xij − X j )/(sj n − 1)
1/2 (Yi − Y)2 yi = (Yi − Y)
= diag[λ1 , λ2 , . . . , λp ] such that
(1)
To compute the correlation matrix∗ , it is useful to standardize the variables: xij = (Xij − X j )
6405
(2)
or
β = hA,
(9)
where h is the coefficient vector of the regression on the PCs. Hence the transformed coefficients for model (1) are obtained from (9) and (3) as γi =
p
√ aij hj /(si n − 1),
i = 1, . . . , p, (10)
j=1
i
√ = (Yi − Y)/(sy n − 1).
where i j
Then (1) becomes y = xβ + ,
√ βj = γj sj n − 1.
γi (3)
The correlation matrix x x has eigenvalues and eigenvectors defined by |x x − λr I| = 0
and
(x x − λr I)ar = 0, (4)
aij hj
si
corresponds to the independent variable number, corresponds to the PC number is the transformed coefficient, model (1) are elements of the eigenmatrix are the coefficients obtained from the regression on the PCs is the standard deviation of independent variable i
6406
PRINCIPAL COMPONENTS REGRESSION ANALYSIS
(In ridge regression∗ , h is referred to as the vector of canonical coefficients.) If a subset of the PCs is selected, some of the hj are zero. The constant term in the transformed equation can be obtained from γ0 = Y −
p
γi X i .
(11)
i=1
McCallum [8] has also observed that var b = σ 2 (x x)−1 = σ 2 (A−1 A ),
(12)
where b is the least squares estimator of β. Therefore, the standard errors of the estimators of the coefficients for model (1) (but not the constant term) are, from (12) and (3), SE(γi ) =
σ
p a2ij
1/2
√ si n − 1 j=1 λj
,
i = 1, . . . , p.
(13)
When a subset of the PCs is selected, the appropriate aij and λj are deleted from (13). One does not need to be concerned with the interpretation of the PCs or the regression on the PCs, because this is just an intermediate step and one can transform back to the original coordinate system (as shown above). Three selection rules for picking PCs are [2,3,8]: 1. Keeping all p PCs yields zero bias [because this yields OLS, as shown in (8)]. 2. Keeping those PCs with the largest eigenvalues minimizes the variances of the estimators of the coefficients. (The proof is given in Greenberg [3] and Coxe [2, p. 226]. One simply needs to substitute A for Z and Z for W.) 3. Keeping those PCs which are highly correlated with the dependent variable tends to minimize the mean square errors of the estimators of the coefficients. (Coxe [2] has given an intuitive argument why this rule holds, but it has yet to be proven.)
The reasons given are: (a) McCallum [8] has shown that one can obtain smaller MSEs by eliminating some of the PCs. (b) Since rule 1 yields zero bias and rule 2 yields minimum coefficient variances, minimum MSE should fall somewhere in between. (c) Selecting those PCs that are strongly correlated with the dependent variable is similar to selecting independent variables by stepwise regression∗ in OLS; that is, both use F-values to determine which terms to retain. It should be noted that selection rules 2 and 3 do not necessarily yield the same PCs model, and hence the resulting coefficients in model 1 can be quite different, especially when the degree of multicollinearity is severe. There is no definitive source for PC regression analysis. In going from paper to paper, one must be alert to the different standardizations of the variables. Some √ use sj in the denominator, others use sj n − 1, and still others do not standardize at all. REFERENCES 1. Anderson, T. W. (1958). An Introduction to Multivariate Statistics. Wiley, New York, pp. 272–287. (A rigorous derivation of PC.) 2. Coxe, K. (1982). Amer. Statist. Ass., Proc. Bus. & Econ. Statist. Sec., pp. 222–227. (Selection rules for PC regression. Comparison with latent root regression. Long on math symbols, short on words. Good list of references.) 3. Greenberg, E. (1975). J. Amer. Statist. Ass., 70, 194–197. [Deals with selection rule (2). Good, fairly technical paper.] 4. Gunst, R. F. and Mason, R. L. (1980). Regression Analysis and Its Application. Marcel Dekker, New York, pp. 317–328. (One of the better treatments of the subject in a textbook.) 5. Johnston, J. (1972). Econometric Methods, 2nd ed. McGraw-Hill, New York, pp. 322–331. (A good, simple derivation of PC regression.) 6. Karson, M. J. (1982). Multivariate Statistical Methods. Iowa State University Press, Ames, IA, pp. 191–224. (Many numerical examples. Excellent list of references.) 7. Massy, W. F. (1965). J. Amer. Statist. Ass., 60, 234–256. (Derivation of PC regression. Fairly technical paper and one of the first.) 8. McCallum, B. T. (1970). Rev. Econ. Statist., 52, 110–113. (Good paper. Easy to read.) 9. Morrison, D. F. (1967). Multivariate Statistical Methods. McGraw-Hill, New York, pp.
PRINCIPAL DIFFERENTIAL ANALYSIS 221–258. (A rather simple approach. Long list of references.) 10. Pidot, G. B., Jr. (1969). Rev. Econ. Statist., 51, 176–188. (Suggests using a combination of the X’s and the PC’s.) See also COMPONENT ANALYSIS; DISCRIMINANT ANALYSIS; LATENT ROOT REGRESSION; MULTICOLLINEARITY; and MULTIVARIATE ANALYSIS.
KEREN L. COXE
PRINCIPAL COMPONENTS, ¨ SCHONEMANN’S THEOREM FOR The following theorem by Sch¨onemann [7] dealing with principal components* has been used by various authors [1,4,5] in connection with the so-called Spearman’s hypothesis (see, e.g., ref. 3) of the positive relationship between the black–white differences in means on cognitive tests and the loading of these tests on the general common factor identified as ‘‘general cognitive ability’’. Sch¨onemann [6,7] asserts that this relationship is ascribable to a statistical artifact and is simply just a mathematical necessity. The statement of the theorem is as follows: If the range Rp —a p-dimensional Euclidean space—of a p-variate normal vector y ∼ N(0, ) is partitioned into ahigh set (H) and a low set (L) by the plane k yk = 0 and both within-covariance matrices H and L remain positive-definite, then (i) the mean difference vector d = E(y|H) − E(y|L) is collinear with the largest principal components (pc’s) of , and (ii) d is also collinear with the pc1’s (eigenvectors corresponding to the largest eigenvalues) of L and H . In other words, if the test correlations are positive in the parent population, they remain so in both selected subpopulations. Parts (i) and (ii) of the theorem do not generally hold for the samples created by selection on the basis of a variable k yk [2]. (It may be noteworthy that Americans are not usually sorted into categories of ‘‘white’’ and ‘‘black’’ by an observed selection variable that is a sum of test scores. Moreover, principal components analysis is not well suited to assess cognitive abilities, since it does not distinguish between factors common to the tests under consideration and factors specific to the tests.)
6407
REFERENCES 1. Braden, J. P. (1989). Fact or artifact. An empirical test of Spearman’s hypothesis. Intelligence, 13, 149–155. 2. Dolan, C. V. (1997). A note on Sch¨onemann’s refutation of Spearman’s hypothesis. Multivariate Behav. Res., 32, 319–328. 3. Jensen, A. R. (1987). Further evidence for Spearman’s hypothesis concerning black-white differences on psychometric tests. Brain Behav. Sci., 10, 512–519. 4. Jensen, A. R. (1992). Spearman’s hypothesis: methodology and evidence. Multivariate Behav. Res., 27, 225–233. 5. Loehlin, J. C. (1992). On Sch¨onemann, on Guttman, on Jensen, via Lewontin. Multivariate Behav. Res., 27, 261–263. 6. Sch¨onemann, P. H. (1989). Some new results on the Spearman hypothesis artifact. Bull. Psychon. Soc., 27, 462–464. 7. Sch¨onemann, P. H. (1992). Extension of Guttman’s results from g to PC1. Multivariate Behav. Res., 27, 219–223. See also COMPONENT ANALYSIS; PRINCIPAL COMPONENTS; and PSYCHOLOGICAL TESTING THEORY.
PRINCIPAL DIFFERENTIAL ANALYSIS Principal differential analysis (PDA) is used in the analysis of functional data to estimate a linear differential equation (linear DIFE) that describes the observed functions. The main References are 4, 8, and 9, and the entry FUNCTIONAL DATA ANALYSIS can be consulted for a discussion of functional data.
DIFFERENTIAL EQUATION MODELS It is often natural to describe a function of time or space in terms of a relationship between the function’s rate of change and its value. More generally, a relation may be specified among derivatives of various orders, the function itself, and other functions. Differential equations are powerful tools for modeling functional data that offer capabilities difficult to achieve with conventional parametric and nonparametric approaches. Because a DIFE involves rates of change, it is often called a
6408
PRINCIPAL DIFFERENTIAL ANALYSIS
dynamic equation, and is said to offer a model for the dynamics of a process rather than just its static behavior. In this article, lower case Greek and Roman letters such as x and β indicate functions of a single argument t unless otherwise indicated, instead of the usual notation β(t) and x(t). This simplifies expressions considerably. Also, we use Dx to indicate the first derivative of function x, Dm x for the derivative of order m, and specifically, D0 x = x. The article will only consider functions of a single argument, but principal differential analysis can also be applied to functions of multiple variables. There are many famous examples of differential equation models. Newton’s Second Law is D2 x = βf , where x is the position of a moving body through time, f is a force applied to the body, and the constant β is the reciprocal of the body’s mass. The differential equation for harmonic motion is D2 x = −ω2 x, where spring constant ω2 determines the motion’s period. Exponential growth, decay, and approach to an asymptote are all described by D2 x = βDx when β is a constant, and if β is a function, this simple equation describes any smooth strictly monotone curve [5]. See reference 6 for other examples of differential equations that are useful in statistics. A linear differential equation of order m has the form Dm x = β0 x + β1 Dx + · · · + βm−1 Dm−1 x + f , (1) and the equation is said to be homogeneous or nonhomogeneous depending on whether the function f is zero or not respectively. The examples given above are all linear DIFEs because of their linearity in the coefficients βj , j = 0, . . . , m − 1. The forcing function f is often considered to represent external or exogenous input to a system whose closed state is described by the homogeneous equation. Linear DIFEs share with other types of linear equations a mathematical structure that is relatively easy to describe and techniques for solving them that are fairly routine. Moreover, a linear differential equation has a solution for any value of t, a feature that
does not always hold for nonlinear DIFEs. This article will focus on linear DIFEs, but nonlinear DIFEs are playing an increasing role in statistics because of the much richer range of functional behaviors that they can define (see Ref. 1 for a readable account). You are encouraged to regard (1) as an equation defining a multiple regression model. The dependent variable is Dm x and the independent variables or covariates are Dj x. The forcing function f can, if not observed directly, be viewed as the error or residual term. The main distinction between a conventional linear model and a linear DIFE is that each variable Dj x and each regression coefficient βj is a function of t. That is, Equation 1 is a t-indexed multiple regression model. In fact, if a sample of functions xi , i = 1, . . . , N are available, then performing a multiple regression for each of a fine mesh of values of t is a reasonable strategy for estimating the coefficient functions βj . It is not surprising, therefore, that the theory associated with linear DIFEs uses the same matrix theory that underlies regular regression (see Ref. 2). Describing functional observations by differential equations brings several useful features that are not provided by conventional approaches. By conventional we mean parametric models; expansions in terms of Fourier series, B-splines, and other basis systems; or other methods that smooth the data. For example, exponential behavior in a function x can either be expressed by the equation x(t) = c1 + c2 eβt where c1 and c2 are arbitrary constants, or more compactly as the DIFE D2 x = βDx. First of all, a DIFE simultaneously defines the behavior of the m derivatives Dj x and the behavior of the function itself, while conventional methods focus all the attention on estimating x. In applications where rates of change are of theoretical interest, as is the case in mechanics, a DIFE requires that derivatives also be well estimated because of their explicit role in the model. Secondly, a single differential equation model can simultaneously fit a whole sample of distinct functions xi . Homogeneous linear DIFEs of order m have an m-dimensional space of solutions, like the exponential two-dimensional exponential equation given
PRINCIPAL DIFFERENTIAL ANALYSIS
above. That is, any solution x can berepresented as a linear combination x = m j cj uj of m linearly independent specific solutions uj , where the coefficients cj are arbitrary constants. The homogeneous case for Newton’s Law, D2 x = 0, has as its solution x(t) = c1 + c2 t. Solutions of the DIFE D3 x = −ω2 Dx are all linear combinations of 1, sin ωt, and cos ωt. Moreover, nonhomogeneous equations have the same solution space translated by a functional of the forcing function f . Finally, as was noted above, specific constraints on x can often be expressed by a specialized DIFE. For example, Dm x = βDm−1 x constrains x to entirely positive/negative, strictly monotonic, or strictly convex/concave depending on whether m is 1, 2, or 3 respectively. The solutions to the nonlinear DIFE Dx = βx(1 − x) are the set of probability functions taking values in (0,1).
ESTIMATING A LINEAR DIFFERENTIAL OPERATOR L The basic idea behind PDA is to estimate a linear differential equation of order m from one or more observed functions xi . The mdimensional solution space of the estimated DIFE is designed to contain most of the variation in these functions. PDA has much in common with principal components analysis (PCA) since, if the DIFE is successful, a set of m linearly independent functions uj will be identified that can be combined linearly to reproduce well the original functional data. But unlike PCA, PDA aims to give a good account of each of m − 1 derivatives at the same time. In PDA, we seek a linear differential operator L of the form Lx =
m−1
βj Dj x + Dm x.
(2)
Now, viewing the forcing function f as an unknown residual term, we can ask whether it is possible to use the data to estimate an operator L such that the least squares fitting criterion N
(Lxi )2 (t)dt =
i=1
N
fi2 (t)dt
(3)
i=1
is minimized. That is, we are aiming to get as close as possible to a homogeneous model Lx = 0 that has solutions that well describe the data. Similarly, in PCA, we essentially define the analysis as a least squares problem in which a linear projection operator Q is defined that minimizes the variation in the residual vectors Qxi . The projection of the observations into the m-dimensional space spanned by the principal components is defined by m eigenvalues. But the linear differential operator L is also defined by the m coefficient functions βj , j = 0, . . . , m − 1; and the essential problem in PDA is also to estimate these coefficient functions by minimizing a least squares criterion. A number of extensions of PDA are possible. We can augment Lx by adding to it terms αk zk where each zk , k = 1, . . . , K, is a known function, and each coefficient function αk also needs to be estimated. If we do so, we do not change the nature of the problem in any qualitative way; the result is still an operator that is linear in the unknown coefficient functions βj and αk . We can also work with systems of linear differential equations, where we have an equation for each of a set of variables, x , = 1, . . . , L. The actual estimation of the weight coefficients βj and αk requires the use of a basis function expansion. That is, when we have a set of K basis functions φk , which are often terms in a Fourier series, or a set of B-splines, then
j=0
The operator L defines the homogeneous part of Equation 1, and we can think of it as a generalization of the notion of the specific derivative Dm . Using this definition, we can express the DIFE model (1) more compactly as Lx = f .
6409
βj =
K
cjk φk = c j φ.
(4)
k=1
The fitting criterion (3) is then quadratic in the coefficients of the expansion, and its minimization merely requires the solution of a set of linear equation. PDA also works fine
6410
PRINCIPAL DIFFERENTIAL ANALYSIS
if there is only a single observed function x provided that the numbers of basis functions in the weight coefficient expansions are small relative to that of x. Finally, we can easily incorporate both specific constraints on certain βj ’s, such as being zero, and can also apply roughness penalties to them that ensure that they are as smooth in some sense as is required. The functions xi being analyzed are usually the result of some smoothing process applied to a set of discrete observed values. It is essential that the smoothing procedure gives a reasonable estimate of the derivatives up to order m at the same time, and consequently PDA is most effective for functional observations where the observational error is fairly small relative to the size of the variation in the function that the smoothing technique is trying to estimate. Methods are under development, however, that will permit using the discrete noisy data directly rather than passing through a smoothing stage. These are based on the work of Reference 3. Because a linear DIFE is essentially a tindexed regression equation, the inferential and diagnostic tools of regression analysis can be adapted to the results of a PDA. We can, for example, examine squared multiple correlation and F-ratio functions R2 and F 0.30
D3 x = βxx1 Dx + βxx2 D2 x + βxy1 Dy + βxy2 D2 y + βxz1 Dz + βxz2 D2 z D3 y = βyx1 Dx + βyx2 D2 x + βyy1 Dy + βyy2 D2 y + βyz1 Dz + βyz2 D2 z D3 z = βzx1 Dx + βzx2 D2 x + βzy1 Dy + βzy2 D2 y + βzz1 Dz + βzz2 D2 z.
(5)
This is a second-order linear system where pen velocities Dx, Dy, and Dz are the
•• •• • •
••••••••••••••• •••• •••• •••• ••• • •• ••••••• ••• ••• ••• ••••••••••••••• ••• • • •• •••••••••••••• • •• •••••••••••••••• ••• •••••••• • • • • • •• • • •• • • ••• • • ••• •• •• • • • •• • •• • •• ••• •• •••••• • • •• • ••••• •• •••••••••••••• • • • • • • • • • •• ••• •• • • • • • • • • • • • • ••• •• • • ••• • ••••••• • • • ••••• • ••••••••• • • • • • • • ••••• •• • ••••••••••• ••• • • • ••• • • • • •• • • • • • • • • • • •• ••• • • • • •••••••••• •• • ••• • • •• • • • •••••••• • • •• •• • • •• •• • •• ••• • •••• ••••••••••••••• • • • • • •• • • • • •• • • ••• ••• • •••••• • ••••• • •• •• • • • • ••••••• •• • •••••• •••• • • • • • •• • • • • • • • • • • • • • • • •• •• •• ••• • •• •• •••••• • • • • •••• • • • ••• ••• • •• ••• •••• •• ••• • ••••
0.25
Y (m)
that are themselves functions of time so that we can identify where the linear DIFE fits the data well and where it does not. As in regression analysis, an inspection of the residual functions fi can offer many useful insights. Examples of PDA can be found in references 8 and 9. Figure 1 is taken from reference 7 reporting an analysis of 50 replications of writing ‘‘statistical science’’ in Mandarin. The dynamics of handwriting involve sequences of high-energy strokes occurring about every 120 milliseconds with levels of acceleration that would send a rocket into orbit in a few minutes. In this application, we have as functional observations the three coordinate functions xi , yi , and zi , i = 1, . . . , 50. The differential equation model was the system
0.20
0.15
0.10 −0.2
−0.1
0.0 X (m)
0.1
0.2
Figure 1. A sample of Mandarin script for the phrase ‘‘statistical science.’’ The dots are pen positions at 200-ms intervals, and the solid line is the reconstruction of the script using the second-order DIFE [5].
PRINCIPAL POINTS
functional observations. It is remarkable that such a simple equation can reconstruct the static trace of the original script so well as well as describing accurately its first and second time-derivative. Acknowledgments The author’s work was prepared under a grant from the Natural Science and Engineering Research Council of Canada.
REFERENCES 1. Alligood, K. T., Sauer, T. D., and Yorke, J. A. (1996). Chaos: An Introduction to Dynamical Systems. Springer, New York. 2. Coddington, E. A. and Carlson, R. (1997). Linear Ordinary Differential Equations. SIAM, Philadelphia, Pa. 3. Heckman, N. and Ramsay, J. O. (2000). Penalized regression with model-based penalties. Can. J. Stat., 28, 241–258. 4. Ramsay, J. O. (1995). Principal differential analysis: data reduction by differential operators. J. R. Stat. Soc., Ser. B, 58, 495– 508. 5. Ramsay, J. O. (1998). Estimating smooth monotone functions. J. R. Stat. Soc., Ser. B, 60, 365–375. 6. Ramsay, J. O. (2000). Differential equation models for statistical functions. Can. J. Stat., 28, 225–240. 7. Ramsay, J. O. (2000). Functional components of variation in handwriting. J. Am. Stat. Assoc., 95, 9–15. 8. Ramsay, J. O. and Silverman, B. W. (1997). Functional Data Analysis. Springer, New York. 9. Ramsay, J. O. and Silverman, B. W. (2002). Applied Functional Data Analysis. Springer, New York.
JAMES O. RAMSAY
PRINCIPAL MOMENTS An obsolete term for central moments∗ (moments about the mean).
PRINCIPAL POINTS A set of k principal points of a distribution is the best k-point approximation of the distribution in terms of a quadratic loss. Principal
6411
points can be motivated by the following practical problem. Suppose a clothing manufacturer wishes to produce a new type of swimsuit in four different sizes. Some relevant measurements such as stature, bust size, etc., are taken on a large number of people. The statistician is asked to identify four ‘‘typical’’ people whose dimensions will be used to model the swimsuits. The term principal points was defined in a similar problem of determining sizes and shapes of new gas masks [9]. Another problem to which principal-point theory has been applied is determining a few representative curves from a large collection of curves [11,15].
DEFINITION AND PROPERTIES OF PRINCIPAL POINTS A set of k points ξ 1 , . . . , ξ k are called k principal points of a random vector X if E[minj=1,...,k X − ξ j 2 ] E[minj=1,...,k X − yj 2 ] for any set of k points {y1 , . . . , yk } ⊂ Rp . Here · denotes the Euclidean norm. Thus principal points provide a least squares∗ approximation by projecting the values of a random vector on a finite number of points, similar to Pearson’s definition of principal components∗ [22]. Principal points are a theoretical analogue to k-cluster means [12] (k-MEANS ALGORITHMS), and have also been called k-centers [21]. For k = 1, the single principal point is the mean of the distribution. The k 2 principal points of a distribution are a generalization of the mean from one to several points. Usually numerical methods are required to determine the principal points except in some simple cases. For instance, for the N (µ, σ 2 ) distribution, the two principal points are µ ± σ 2/π [2,9]. For any k 1, the k principal points of the uniform distribution in [a, b] are a + (2j − 1)(b − a)/2k, j = 1, . . . , k [31]. Determining the principal points of a distribution is aided by studying self-consistent points, particularly for multivariate distributions. For k points y1 , . . . , yk in Rp , let Sj ⊂ Rp denote the set of all points in Rp closer to yj than to all other yi ’s. If E[X|X ∈ Sj ] = yj for j = 1, . . . , k, then the yj are called k self-consistent points of X [10]. Self-consistent points have also been called ‘‘unbiased’’ [19].
6412
PRINCIPAL POINTS
Principal points are self-consistent [10]. Multivariate distributions have typically several sets of k self-consistent points but only one set of k principal points. For example, consider the bivariate normal distribution∗ centered at the origin with covariance matrix diag (σ 2 , 1), where σ > 1. Then there exist two sets of two self-consistent points: {(0, 2/π ) , (0, − 2/π ) } and {(σ 2/π , 0) , (−σ 2/π , 0) }. The second set corresponds to two principal points [9]. An important property of self-consistent points (and hence principal points) is the following: if y1 , . . . , yk are k self-consistent points of X, then E[X] lies in the convex hull of the yj [28]. See also SELF-CONSISTENCY.
UNIVARIATE RESULTS Principal points for the standard normal distribution have appeared in numerous publications, e.g., refs. 2, 9, 18, and for k up to 31 in ref. [6]. Computational methods for finding principal points of univariate continuous distributions are given in refs. 18, 25, 30. Symmetric univariate distributions always have a set of k symmetric selfconsistent points [31], but the k principal points may not be symmetric to the mean even if the distribution is continuous and unimodal [9]. However, if the distribution has a log-concave density, then there exists a unique set of k self-consistent points which must be the k principal points of the distribution [8,16,26,29]. Under rather general conditions on the density f of a random variable X, asymptotically as k → ∞, the k principal points tend towards the k quantiles of the distribution whose density is proportional to f 1/3 [23].
MULTIVARIATE RESULTS Suppose X follows a p-variate elliptical distribution∗ such as a multivariate normal with mean zero. If k principal points of X span a subspace of dimension q < p, then this subspace is also spanned by q eigenvectors associated with the q largest eigenvalues of the covariance matrix [28]. Similar results relating principal component subspaces to
principal points and self-consistent points hold for a larger class of symmetric multivariate distributions [27]. For the bivariate normal, numerical results indicate that principal points and self-consistent points form highly symmetrical patterns [9,10,17]. For the uniform distribution on the unit square, asymptotically the optimal configuration of points as k → ∞ corresponds to the tessellation by regular hexagons with principal points at their centers ([7,20]; see RANDOM TESSELLATIONS). ESTIMATION OF PRINCIPAL POINTS To apply the theory of principal points to real data, various estimators have been defined, ranging from normal theory maximum likelihood∗ [10] to the nonparametric k-means clustering algorithm∗ [12]. The cluster means from the k-means algorithm are by construction a set of k self-consistent points of the empirical distribution. Under rather general conditions, the cluster means from the kmeans algorithm are consistent estimators of principal points, and are asymptotically normal [1,13,19,24]. Convergence results have also been established for the more general case of distributions on Banach spaces [3,21]. Semiparametric∗ estimators of principal points have been defined under the constraint that they form particular symmetric patterns in the subspace of the first few principal components∗ , and by applying the k-means algorithm to data reduced to the first few principal components [10], as suggested by the theory of principal points of elliptical distributions [28]. EXAMPLE Principal point methodology has been applied in a problem of determining shapes and sizes of new gas masks for the Swiss Army [10]. Due to the considerable variability in human heads, a single mask would not be sufficient to fit all its users. The dimensions of k = 2 to 5 ‘‘typical’’ heads were requested for constructing masks of different sizes and shapes. Six head measurements were taken on a sample of Swiss soldiers. Assuming multiˆ denote the variate normality, let µ ˆ and
PRINCIPAL POINTS
maximum likelihood estimates of the mean vector and the covariance matrix respectively. Then the k principal points of the ˆ distribution are maximum likeliˆ ) N6 (µ, hood estimates of k principal points. Figure 1 shows the normal theory maximum likelihood estimates of k = 4 principal points for the head dimension data. The points are plotted in the subspace spanned by the first two principal components. These estimates were based on the assumption that the four principal points form a ‘‘cross’’ pattern in the subspace spanned by the first two principal components. RELATED TOPICS Since a set of k principal points partition the space into k Voronoi regions or strata according to minimal distance, the mathematical problem of determining principal points is related to the problem of optimum stratification∗ [4] and optimal grouping [2]. For a unified approach to these and several other related problems that share a common mathematical framework, see [5]. Principal points are also related to quantization in signal processing∗ . A common form of quantization is rounding off, which occurs in all digital systems. A quantizer partitions
6413
the sample space of a random variable and maps the random variable to representative points so as to minimize the average distortion. Determining an optimal quantizer is often equivalent to finding a set of representative (principal) points of a distribution [14]. IEEE published a special issue on the topic of quantization (IEEE Transactions on Information Theory, 1982). Several of the articles that appear in that issue are noted in the references below. REFERENCES 1. Bryant, P. (1991). Large-sample results for optimization-based clustering methods. J. Classification, 8, 31–44. 2. Cox, D. (1957). Note on grouping. J. Amer. Statist. Ass., 52, 543–547. 3. Cuesta, J. and Matran, C. (1988). The strong law of large numbers for k-means and best possible nets of Banach valued random variables. Probab. Theory and Related Fields, 78, 523–534. 4. Dalenius, T. (1950). The problem of optimum stratification. Skand. Aktuarietidskrift, 33, 203–213. 5. Eubank, R. (1988). Optimal grouping, spacing, stratification, and piecewise constant approximation. SIAM Rev., 30, 404–420.
Figure 1. Normal theory maximum likelihood estimates of four principal points (denoted •) for the head dimensions data. Scatter plot shows 50 observations in the coordinate system of the first two principal components. Broken lines indicate the boundaries of the Voronoi regions associated with the principal points.
6414
PRINCIPLE OF PARSIMONY
6. Fang, K. and He, S. (1982). The problem of selecting a given number of representative points in a normal population and a generalized Mills’ ratio. Tech. Rep., Department of Statistics, Stanford University. 7. Fejes T´oth, L. (1953). Lagerungen in der Ebene, auf der Kugel und im Raum. SpringerVerlag, Berlin. 8. Fleischer, P. (1964). Sufficient conditions for achieving minimum distortion in a quantizer. IEEE Int. Convention Record Part I, 12, 104–111. 9. Flury, B. (1990). Principal points. Biometrika, 77, 33–41. 10. Flury, B. (1993). Estimation of principal points, Appl. Statist., 42, 139–151. 11. Flury, B. and Tarpey, T. (1993). Representing a large collection of curves: a case for principal points, Amer. Statist., 47, 304–306. 12. Hartigan, J. (1975). Clustering Algorithms. Wiley, New York. 13. Hartigan, J. (1978). Asymptotic distributions for clustering criteria. Ann. Statist., 6, 117–131. 14. Iyengar, S. and Solomon, H. (1983). Selecting representative points in normal populations. In Recent Advances in Statistics: Papers in Honor of Herman Chernoff on his 60th Birthday. Academic Press, New York, pp. 579– 591.
22. Pearson, K. (1901). On lines and planes of closest fit to systems of points in space. Phil. Mag., 2, 559–572. 23. P¨otzelberger, K. and Felsenstein, K. (1994). An asymptotic result on principal points for univariate distributions. Optimization, 28, 397–406. 24. Pollard, D. (1981). Strong consistency of kmeans clustering. Ann. Statist., 9, 135–140. 25. Rowe, S. (1996). An algorithm for computing principal points with respect to a loss function in the unidimensional case, Statist. and Comput., 6, 187–190. 26. Tarpey, T. (1994). Two principal points of symmetric, strongly unimodal distributions. Statist. and Probab. Lett., 20, 253–257. 27. Tarpey, T. (1995). Principal points and selfconsistent points of symmetric multivariate distributions. J. Multivariate Anal., 53, 39–51. 28. Tarpey, T., Li, L., and Flury, B. (1995). Principal points and self-consistent points of elliptical distributions. Ann. Statist., 23, 103–112. 29. Trushkin, A. (1982). Sufficient conditions for uniqueness of locally optimal quantizer for a class of convex error weighting functions. IEEE Trans. Inf. Theory, IT-28, 187–198. 30. Zopp`e, A. (1995). Principal points of univariate continuous distributions, Statist. and Comput., 5, 127–132.
15. Jones, M. and Rice, J. (1992). Displaying the important features of large collections of similar curves. Amer. Statist., 46, 140–145.
31. Zopp`e, A. (1997). On uniqueness and symmetry of self-consistent points of univariate continuous distributions, J. Classification, 14, 147–158.
16. Kieffer, J. (1982). Exponential rate of convergence for Lloyd’s method I. IEEE Trans. Inf. Theory, IT-28, 205–210.
See also COMPONENT ANALYSIS; k-MEANS ALGORITHMS; SELF-CONSISTENCY; and SIZE AND SHAPE ANALYSIS.
¨ 17. Kipper, S. and Parna, K. (1992). Optimal kcentres for a two-dimensional normal distribution. Acta et Comment. Univ. Tartuensis, 942, 21–27. 18. Lloyd, S. (1982). Least squares quantization in PCM. IEEE Trans. Inf. Theory, IT-28, 129–136. 19. MacQueen, J. (1966). Some methods for classification and analysis of multivariate observations. Proc. Fifth Berkeley Symp. Math. Statist. and Probab. University of California Press, vol. 1, pp. 281–297.
BERNARD D. FLURY THADDEUS TARPEY
PRINCIPLE OF PARSIMONY. See PARSIMONY, PRINCIPLE OF ; and SCIENTIFIC METHOD STATISTICS
AND
PRINCIPLES OF PROFESSIONAL STATISTICAL PRACTICE
20. Newman, D. (1982). The hexagon theorem. IEEE Trans. Inf. Theory, IT-28, 137–139.
PURPOSE AND SCOPE
¨ 21. Parna, K. (1990). On the existence and weak convergence of k-centres in Banach spaces. Acta et Comment. Univ. Tartuensis, 893, 17–28.
1. The statistician’s job. The statistician’s responsibility, whether as a consultant on a part-time basis, or on regular salary is to
PRINCIPLES OF PROFESSIONAL STATISTICAL PRACTICE
find problems that other people could not be expected to perceive. The statistician’s tool is statistical theory (theory of probability). It is the statistician’s use of statistical theory that distinguishes him from other experts. Statistical practice is a collaborative venture between statistician and experts in subject matter. Experts who work with a statistician need to understand the principles of professional statistical practice as much as the statistician needs them. Challenges face statisticians today as never before. The whole world is talking about safety in mechanical and electrical devices (in automobiles, for example); safety in drugs; reliability; due care; pollution; poverty; nutrition; improvement of medical practice; improvement of agricultural practice; improvement in quality of product; breakdown of service; breakdown of equipment; tardy buses, trains and mail; need for greater output in industry and in agriculture; enrichment of jobs. These problems cannot be understood and cannot even be stated, nor can the effect of any alleged solution be evaluated, without the aid of statistical theory and methods. One cannot even define operationally such adjectives as reliable, safe, polluted, unemployed, on time (arrivals), equal (in size), round, random, tired, red, green, or any other adjective, for use in business or in government, except in statistical terms. To have meaning for business or legal purposes, a standard (as of safety, or of performance or capability) must be defined in statistical terms. 2. Professional practice. The main contribution of a statistician in any project or in any organization is to find the important problems, the problems that other people cannot be expected to perceive. An example is the 14 points that top management learned in Japan in 1950. The problems that walk in are important, but are not usually the important problems (4). The statistician also has other obligations to the people with whom he works, examples being to help design studies, to construct an audit by which to discover faults in procedure before it is too late to make alterations, and at the end, to evaluate the statistical reliability of the results.
6415
The statistician, by virtue of experience in studies of various kinds, will offer advice on procedures for experiments and surveys, even to forms that are to be filled out, and certainly in the supervision of the study. He knows by experience the dangers of (1) forms that are not clear, (2) procedures that are not clear, (3) contamination, (4) carelessness, and (5) failure of supervision. Professional practice stems from an expanding body of theory and from principles of application. A professional man aims at recognition and respect for his practice, not for himself alone, but for his colleagues as well. A professional man takes orders, in technical matters, from standards set by his professional colleagues as unseen judges, never from an administrative superior. His usefulness and his profession will suffer impairment if he yields to convenience or to opportunity. A professional man feels an obligation to provide services that his client may never comprehend or appreciate. A professional statistician will not follow methods that are indefensible, merely to please someone, or support inferences based on such methods. He ranks his reputation and profession as more important than convenient assent to interpretations not warranted by statistical theory. Statisticians can be trusted and respected public servants. Their careers as expert witnesses will be shattered if they indicate concern over which side of the case the results seem to favor. ‘‘As a statistician, I couldn’t care less’’ is the right attitude in a legal case or in any other matter. LOGICAL BASIS FOR DIVISION OF RESPONSIBILITIES 3. Some limitations in application of statistical theory. Knowledge of statistical theory is necessary but not sufficient for successful operation. Statistical theory does not provide a road map toward effective use of itself. The purpose of this article is to propose some principles of practice, and to explain their meaning in some of the situations that the statistician encounters. Statisticians have no magic touch by which they may come in at the stage of tabulation
6416
PRINCIPLES OF PROFESSIONAL STATISTICAL PRACTICE
and make something of nothing. Neither will their advice, however wise in the early stages of a study, ensure successful execution and conclusion. Many a study launched on the ways of elegant statistical design, later boggled in execution, ends up with results to which the theory of probability can contribute little. Even though carried off with reasonable conformance to specifications, a study may fail through structural deficiencies in the method of investigation (questionnaire, type of test, technique of interviewing) to provide the information needed. Statisticians may reduce the risk of this kind of failure by pointing out to their clients in the early stages of the study the nature of the contributions that they themselves must put into it. (The word client will denote the expert or group of experts in a substantive field.) The limitations of statistical theory serve as signposts to guide a logical division of responsibilities between statistician and client. We accordingly digress for a brief review of the power and the limitations of statistical theory. We note first that statistical inferences (probabilities, estimates, confidence limits, fiducial limits, etc.), calculated by statistical theory from the results of a study, will relate only to the material, product, people, business establishments, and so on, that the frame was in effect drawn from, and only to the environment of the study, such as the method of investigation, the date, weather, rate of flow and levels of concentration of the components used in tests of a production process, range of voltage, or of other stress specified for the tests (as of electrical equipment). Empirical investigation consists of observations on material of some kind. The material may be people; it may be pigs, insects, physical material, industrial product, or records of transactions. The aim may be enumerative, which leads to the theory for the sampling of finite populations∗ . The aim may be analytic, the study of the causes that make the material what it is, and which will produce other material in the future. A definable lot of material may be divisible into identifiable sampling units. A list of these sampling units, with identification of each unit, constitutes a frame. In some physical
and astronomical investigations, the sampling unit is a unit of time. We need not detour here to describe nests of frames for multistage sampling. The important point is that without a frame there can be neither a complete coverage of a designated lot of material nor a sample of any designated part thereof. Stephan introduced the concept of the frame, but without giving it a name (7). Objective statistical inferences in respect to the frame are the speciality of the statistician. In contrast, generalization to cover material not included in the frame, nor to ranges, conditions, and methods outside the scope of the experiment, however essential for application of the results of a study, are a matter of judgment and depend on knowledge of the subject matter (5). For example, the universe in a study of consumer behavior might be all the female homemakers in a certain region. The frame might therefore be census blocks, tracts or other small districts, and the ultimate sampling unit might be a segment of area containing households. The study itself will, of course, reach only the people that can be found at home in the segments selected for the sample. The client, however, must reach generalizations and take action on the product or system of marketing with respect to all female homemakers, whether they be the kind that are almost always at home and easy to reach, or almost never at home and therefore in part omitted from the study. Moreover, the female homemakers on whom the client must take action belong to the future, next year, and the next. The frame only permits study of the past. For another example, the universe might be the production process that will turn out next week’s product. The frame for study might be part of last week’s product. The universe might be traffic in future years, as in studies needed as a basis for estimating possible losses or gains as a result of a merger. The frame for this study might be records of last year’s shipments. Statistical theory alone could not decide, in a study of traffic that a railway is making, whether it would be important to show the movement of (for example) potatoes, separately from other agricultural products in
PRINCIPLES OF PROFESSIONAL STATISTICAL PRACTICE
the northwestern part of the United States; only he who must use the data can decide. The frame for a comparison of two medical treatments might be patients or other people in Chicago with a specific ailment. A pathologist might, on his own judgment, without further studies, generalize the results to people that have this ailment anywhere in the world. Statistical theory provides no basis for such generalizations. No knowledge of statistical theory, however profound, provides by itself a basis for deciding whether a proposed frame would be satisfactory for a study. For example, statistical theory would not tell us, in a study of consumer research, whether to include in the frame people that live in trailer parks. The statistician places the responsibility for the answer to this question where it belongs, with the client: Are trailer parks in your problem? Would they be in it if this were a complete census? If yes, would the cost of including them be worthwhile? Statistical theory will not of itself originate a substantive problem. Statistical theory cannot generate ideas for a questionnaire or for a test of hardness, nor can it specify what would be acceptable colors of dishes or of a carpet; nor does statistical theory originate ways to teach field workers or inspectors how to do their work properly or what characteristics of workmanship constitute a meaningful measure of performance. This is so in spite of the fact that statistical theory is essential for reliable comparisons of questionnaires and tests. 4. Contributions of statistical theory to subject matter. It is necessary, for statistical reliability of results, that the design of a survey or experiment fit into a theoretical model. Part of the statistician’s job is to find a suitable model that he can manage, once the client has presented his case and explained why statistical information is needed and how the results of a study might be used. Statistical practice goes further than merely to try to find a suitable model (theory). Part of the job is to adjust the physical conditions to meet the model selected. Randomness∗ , for example, in sampling a frame, is not just a word to assume; it must be created by use of random numbers for the selection of sampling units. Recalls to find people at home,
6417
or tracing illegible or missing information back to source documents, are specified so as to approach the prescribed probability of selection. Statistical theory has in this way profoundly influenced the theory of knowledge. It has given form and direction to quantitative studies by molding them into the requirements of the theory of estimation∗ , and other techniques of inference. The aim is, of course, to yield results that have meaning in terms that people can understand. The statistician is often effective in assisting the substantive expert to improve accepted methods of interviewing, testing, coding, and other operations. Tests properly designed will show how alternative test or field procedures really perform in practice, so that rational changes or choices may be possible. It sometimes happens, for example, that a time-honored or committee-honored method of investigation, when put to statistically designed tests, shows alarming inherent variances between instruments, between investigators, or even between days for the same investigator. It may show a trend, or heavy intraclass correlation∗ between units, introduced by the investigators. Once detected, such sources of variation may then be corrected or diminished by new rules, which are of course the responsibility of the substantive expert. Statistical techniques provide a safe supervisory tool to help to reduce variability in the performance of worker and machine. The effectiveness of statistical controls, in all stages of a survey, for improving supervision, to achieve uniformity, to reduce errors, to gain speed, reduce costs, and to improve quality of performance in other ways is well known. A host of references could be given, but two will suffice [2,3]. 5. Statistical theory as a basis for division of responsibilities. We may now, with this background, see where the statistician’s responsibilities lie. In the first place, his specialized knowledge of statistical theory enables him to see which parts of a problem belong to substantive knowledge (sociology, transportation, chemistry of a manufacturing process, law), and which parts are statistical. The responsibility as logician in a study falls to him by
6418
PRINCIPLES OF PROFESSIONAL STATISTICAL PRACTICE
default, and he must accept it. As logician, he will do well to designate, in the planning stages, which decisions will belong to the statistician and which to the substantive expert. This matter of defining specifically the areas of responsibility is a unique problem faced by the statistician. Business managers and lawyers who engage an expert on corporate finance, or an expert on steam power plants, know pretty well what such experts are able to do and have an idea about how to work with them. It is different with statisticians. Many people are confused between the role of theoretical statisticians (those who use theory to guide their practice) and the popular idea that statisticians are skillful in compiling tables about people or trade, or who prophesy the economic outlook for the coming year and which way the stock market will drift. Others may know little about the contributions that statisticians can make to a study or how to work with them. Allocation of responsibilities does not mean impervious compartments in which you do this and I’ll do that. It means that there is a logical basis for allocation of responsibilities and that it is necessary for everyone involved in a study to know in advance what he or she will be accountable for. A clear statement of responsibilities will be a joy to the client’s lawyer in a legal case, especially at the time of cross-examination. It will show the kind of question that the statistician is answerable for, and what belongs to the substantive experts. Statisticians have something to learn about professional practice from law and medicine. 6. Assistance to the client to understand the relationship. The statistician must direct the client’s thoughts toward possible uses of the results of a statistical study of the entire frame without consideration of whether the entire frame will be studied or only a small sample of the sampling units in the frame. Once these matters are cleared, the statistician may outline one or more statistical plans and explain to the client in his own language what they mean in respect to possible choices of frame and environmental conditions, choices of sampling unit, skills, facilities and supervision required. The statistician will urge the client to foresee
possible difficulties with definitions or with the proposed method of investigation. He may illustrate with rough calculations possible levels of precision to be expected from one or more statistical plans that appear to be feasible, along with rudimentary examples of the kinds of tables and inferences that might be forthcoming. These early stages are often the hardest part of the job. The aim in statistical design is to hold accuracy and precision to sensible levels, with an economic balance between all the possible uncertainties that afflict data—built-in deficiencies of definition, errors of response, nonresponse, sampling variation, difficulties in coding, errors in processing, difficulties of interpretation. Professional statistical practice requires experience, maturity, fortitude, and patience, to protect the client against himself or against his duly appointed experts in subject matter who may have in mind needless but costly tabulations in fine classes (fiveyear age groups, small areas, fine mileage brackets, fine gradations in voltage, etc.), or unattainable precision in differences between treatments, beyond the capacity of the skills and facilities available, or beyond the meaning inherent in the definitions. These first steps, which depend heavily on guidance from the statistician, may lead to important modifications of the problem. Advance considerations of cost, and of the limitations of the inferences that appear to be possible, may even lead clients to abandon a study, at least until they feel most confident of the requirements. Protection of a client’s bank account, and deliverance from more serious losses from decisions based on inconclusive or misleading results, or from misinterpretation, is one of the statistician’s greatest services. Joint effort does not imply joint responsibility. Divided responsibility for a decision in a statistical survey is as bad as divided responsibility in any venture—it means that no one is responsible. Although they acquire superficial knowledge of the subject matter, the one thing that statisticians contribute to a problem, and which distinguishes them from other experts, is knowledge and ability in statistical theory.
PRINCIPLES OF PROFESSIONAL STATISTICAL PRACTICE
SUMMARY STATEMENT OF RECIPROCAL OBLIGATIONS AND RESPONSIBILITIES 7. Responsibilities of the client. The client will assume responsibility for those aspects of the problem that are substantive. Specifically, he will stand the ultimate responsibility for: a. The type of statistical information to be obtained. b. The methods of test, examination, questionnaire, or interview, by which to elicit the information from any unit selected from the frame. c. The decision on whether a proposed frame is satisfactory. d. Approval of the probability model proposed by the statistician (statistical procedures, scope, and limitations of the statistical inferences that may be possible from the results). e. The decision on the classes and areas of tabulation (as these depend on the uses that the client intends to make of the data); the approximate level of statistical precision or protection that would be desirable in view of the purpose of the investigation, skills and time available, and costs. The client will make proper arrangements for: f. The actual work of preparing the frame for sampling, such as serializing and identifying sampling units at the various stages. g. The selection of the sample according to procedures that the statistician will prescribe, and the preparation of these units for investigation. h. The actual investigation; the training for this work, and the supervision thereof. i. The rules for coding; the coding itself. j. The processing, tabulations and computations, following procedures of estimation that the sampling plans prescribe. The client or his representative has the obligation to report at once any departure from instructions, to permit the statistician
6419
to make a decision between a fresh start or an unbiased adjustment. The client will keep a record of the actual performance. 8. Responsibilities of the statistician. The statistician owes an obligation to his own practice to forestall disappointment on the part of the client, who if he fails to understand at the start that he must exercise his own responsibilities in the planning stages, and in execution, may not realize in the end the fullest possibility of the investigation, or may discover too late that certain information that he had expected to get out of the study was not built into the design. The statistician’s responsibility may be summarized as follows: a. To formulate the client’s problem in statistical terms (probability model), subject to approval of the client, so that a proposed statistical investigation may be helpful to the purpose. b. To lay out a logical division of responsibilities for the client, and for the statistician, suitable to the investigation proposed. c. To explain to the client the advantages and disadvantages of various frames and possible choices of sampling units, and of one or more feasible statistical plans of sampling or experimentation that seem to be feasible. d. To explain to the client, in connection with the frame, that any objective inferences that one may draw by statistical theory from the results of an investigation can only refer to the material or system that the frame was drawn from, and only to the methods, levels, types, and ranges of stress presented for study. It is essential that the client understand the limitations of a proposed study, and of the statistical inferences to be drawn therefrom, so that he may have a chance to modify the content before it is too late. e. To furnish statistical procedures for the investigation—selection, computation of estimates and standard errors, tests, audits, and controls as seem warranted for detection and evaluation of important possible departures from specifications, variances between
6420
PRINCIPLES OF PROFESSIONAL STATISTICAL PRACTICE
investigators, nonresponse, and other persistent uncertainties not contained in the standard error; to follow the work and to know when to step in. f. To assist the client (on request) with statistical methods of supervision, to help him to improve uniformity of performance of investigators, gain speed, reduce errors, reduce costs, and to produce a better record of just what took place. g. To prepare a report on the statistical reliability of the results. 9. The statistician’s report or testimony. The statistician’s report or testimony will deal with the statistical reliability of the results. The usual content will cover the following points: a. A statement to explain what aspects of the study his responsibility included, and what it excluded. It will delimit the scope of the report. b. A description of the frame, the sampling unit, how defined and identified, the material covered, and a statement in respect to conditions of the survey or experiment that might throw light on the usefulness of the results. c. A statement concerning the effect of any gap between the frame and the universe for important conclusions likely to be drawn from the results. (A good rule is that the statistician should have before him a rough draft of the client’s proposed conclusions.) d. Evaluation (in an enumerative study) of the margin of uncertainty, for a specified probability level, attributable to random errors of various kinds, including the uncertainty introduced by sampling, and by small independent random variations in judgment, instruments, coding, transcription, and other processing. e. Evaluation of the possible effects of other relevant sources of variation, examples being differences between investigators, between instruments, between days, between areas.
f. Effect of persistent drift and conditioning of instruments and of investigators; changes in technique. g. Nonresponse and illegible or missing entries. h. Failure to select sampling units according to the procedure prescribed. i. Failure to reach and to cover sampling units that were designated in the sampling table. j. Inclusion of sampling units not designated for the sample but nevertheless covered and included in the results. k. Any other important slips and departures from the procedure prescribed. l. Comparisons with other studies, if any are relevant. In summary, a statement of statistical reliability attempts to present to readers all information that might help them to form their own opinions concerning the validity of conclusions likely to be drawn from the results. The aim of a statistical report is to protect clients from seeing merely what they would like to see; to protect them from losses that could come from misuse of the results. A further aim is to forestall unwarranted claims of accuracy that a client’s public might otherwise accept. Any printed description of the statistical procedures that refer to this participation, or any evaluation of the statistical reliability of the results, must be prepared by the statistician as part of the engagement. If a client prints the statistician’s report, it will be printed in full. The statistician has an obligation to institute audits and controls as a part of the statistical procedures for a survey or experiment, to discover any departure from the procedures prescribed. The statistician’s full disclosure and discussion of all the blemishes and blunders that took place in the survey or experiment will build for him a reputation of trust. In a legal case, this disclosure of blunders and blemishes and their possible effects on conclusions to be drawn from the data are a joy to the lawyer.
PRINCIPLES OF PROFESSIONAL STATISTICAL PRACTICE
A statistician does not recommend to a client any specific administrative action or policy. Use of the results that come from a survey or experiment are entirely up to the client. Statisticians, if they were to make recommendations for decision, would cease to be statisticians. Actually, ways in which the results may throw light on foreseeable problems will be settled in advance, in the design, and there should be little need for a client or for anyone else to reopen a question. However, problems sometimes change character with time (as when a competitor of the client suddenly comes out with a new model), and open up considerations of statistical precision and significance of estimates that were not initially in view. The statistician may describe in a professional or scientific meeting the statistical methods that he develops in an engagement. He will not publish actual data or substantive results or other information about a client’s business without permission. In other words, the statistical methods belong to the statistician: the data to the client. A statistician may at times perform a useful function by examining and reporting on a study in which he did not participate. A professional statistician will not write an opinion on another’s procedures or inferences without adequate time for study and evaluation. SUPPLEMENTAL REMARKS 10. Necessity for the statistician to keep in touch. A statistician, when he enters into a relationship to participate in a study, accepts certain responsibilities. He does not merely draft instructions and wait to be called. The people whom he serves are not statisticians, and thus cannot always know when they are in trouble. A statistician asks questions and will probe on his own account with the help of proper statistical design, to discover for himself whether the work is proceeding according to the intent of the instructions. He must expect to revise the instructions a number of times in the early stages. He will be accessible by mail, telephone, telegraph, or in person, to answer questions and to listen to suggestions.
6421
He may, of course, arrange consultations with other statisticians on questions of theory or procedure. He may engage another statistician to take over certain duties. He may employ other skills at suitable levels to carry out independent tests or reinvestigation of certain units, to detect difficulties and departures from the prescribed procedure, or errors in transcription or calculation. It must be firmly understood, however, that consultation or assistance is in no sense a partitioning of responsibility. The obligations of a statistician to his client, once entered into, may not be shared. 11. What is an engagement? Dangers of informal advice. It may seem at first thought that statisticians ought to be willing to give to the world informally and impromptu the benefit of their knowledge and experience, without discussion or agreement concerning participation and relationships. Anyone who has received aid from a doctor of medicine who did his best without a chance to make a more thorough examination can appreciate how important the skills of a professional man can be, even under handicap. On second thought, most statisticians can recall instances in which informal advice backfired. It is the same in any professional line. A statistician who tries to be helpful and give advice under adverse circumstances is in practice and has a client, whether he intended it so or not; and he will later find himself accountable for the advice. It is important to take special precaution under these circumstances to state the basis of understanding for any statements or recommendations, and to make clear that other conditions and circumstances could well lead to different statements. 12. When do the statistician’s obligations come to a close? A statistician should make it clear that his name may be identified with a study only as long as he is active in it and accountable for it. A statistical procedure, contrary to popular parlance, is not installed. One may install new furniture, a new carpet, or a new dean, but not a statistical procedure. Experience shows that a statistical procedure deteriorates rapidly when left completely to nonprofessional administration.
6422
PRINCIPLES OF PROFESSIONAL STATISTICAL PRACTICE
A statistician may draw up plans for a continuing study, such as for the annual inventory of materials in process in a group of manufacturing plants, or for a continuing national survey of consumers. He may nurse the job into running order, and conduct it through several performances. Experience shows, however, that if he steps out and leaves the work in nonstatistical hands, he will shortly find it to be unrecognizable. New people come on the job. They may think that they know better than their predecessor how to do the work; or they may not be aware that there ever were any rules or instructions, and make up their own. What is even worse, perhaps, is that people that have been on a job a number of years think that they know it so well that they cannot go wrong. This type of fault will be observed, for example, in a national monthly sample in which households are to be revisited a number of times; when left entirely to nonstatistical administration, it will develop faults. Some interviewers will put down their best guesses about the family, on the basis of the preceding month, without an actual interview. They will forget the exact wording of the question, or may think that they have something better. They will become lax about calling back on people not at home. Some of them may suppose that they are following literally the intent of the instructions, when in fact (as shown by a control), through a misunderstanding, they are doing something wrong. Or they may depart wilfully, thinking that they are thereby improving the design, on the supposition that the statistician did not really understand the circumstances. A common example is to substitute an averagelooking sampling unit when the sampling unit designated is obviously unusual in some respect (1). In the weighing and testing of physical product, people will in all sincerity substitute their judgment for the use of random numbers. Administration at the top will fail to rotate areas in the manner specified. Such deterioration may be predicted with confidence unless the statistician specifies statistical controls that provide detective devices and feedback. 13. The single consultation. It is wise to avoid a single consultation with a commercial
concern unless satisfactory agenda are prepared in advance and satisfactory arrangements made for absorbing advice offered. This requirement, along with an understanding that there will be a fee for a consultation, acts as a shield against a hapless conference which somebody calls in the hope that something may turn up. It also shows that statisticians, as professional people, although eager to teach and explain statistical methods, are not on the lookout for chances to demonstrate what they might be able to accomplish. Moreover, what may be intended as a single consultation often ends up with a request for a memorandum. This may be very difficult to write, especially in the absence of adequate time to study the problem. The precautions of informal advice apply here. 14. The statistician’s obligation in a muddle. Suppose that fieldwork is under way. Then the statistician discovers that the client or duly appointed representatives have disregarded the instructions for the preparation of the frame, or for the selection of the sample, or that the fieldwork seems to be falling apart. ‘‘We went ahead and did so and so before your letter came, confident that you would approve,’’ is a violation of relationship. That the statistician may approve the deed done does not mitigate the violation. If it appears to the statistician that there is no chance that the study will yield results that he could take professional responsibility for, this fact must be made clear to the client, at a sufficiently high management level. It is a good idea for the statistician to explain at the outset that such situations, while extreme, have been known. Statisticians should do all in their power to help clients to avoid such a catastrophe. There is always the possibility that a statistician may be partly to blame for not being sufficiently clear nor firm at the outset concerning his principles of participation, or for not being on hand at the right time to ask questions and to keep apprised of what is happening. Unfortunate circumstances may teach the statistician a lesson on participation. 15. Assistance in interpretation of non-probability samples. It may be humiliating, but statisticians must face the fact
PRINCIPLES OF PROFESSIONAL STATISTICAL PRACTICE
that many accepted laws of science have come from theory and experimentation without benefit of formal statistical design. Vaccination for prevention of smallpox is one; John Snow’s discovery of the source of cholera in London is another (6). So is the law F = ma in physics; also Hooke’s law, Boyle’s law, Mendel’s findings, Keppler’s laws, Darwin’s theory of evolution, the Stefan-Boltzmann law of radiation (first empirical, later established by physical theory). All this only means, as everyone knows, that there may well be a wealth of information in a nonprobability sample. Perhaps the main contribution that the statistician can make in a nonprobability sample is to advise the experimenter against conclusions based on meaningless statistical calculations. The expert in the subject matter must take the responsibility for the effects of selectivity and confounding∗ of treatments. The statistician may make a positive contribution by encouraging the expert in the subject matter to draw whatever conclusions he believes to be warranted, but to do so over his own signature, and not to attribute conclusions to statistical calculations or to the kind of help provided by a statistician. 16. A statistician will not agree to use of his name as advisor to a study, nor as a member of an advisory committee, unless this service carries with it explicit responsibilities for certain prescribed phases of the study. 17. A statistician may accept engagements from competitive firms. His aim is not to concentrate on the welfare of a particular client, but to raise the level of service of his profession. 18. A statistician will prescribe in every engagement whatever methods known to him seem to be most efficient and feasible under the circumstances. Thus, he may prescribe for firms that are competitive methods that are similar or even identical word for word in part or in entirety. Put another way, no client has a proprietary right in any procedures or techniques that a statistician prescribes. 19. A statistician will, to the best of his ability, at his request, lend technical assistance to another statistician. In rendering this assistance, he may provide copies of procedures that he has used, along with
6423
whatever modification seems advisable. He will not, in doing this, use confidential data. REFERENCES 1. Bureau of the Census (1954). Measurement of Employment and Unemployment. Report of Special Advisory Committee on Employment Statistics, Washington, D.C. 2. Bureau of the Census (1964). Evaluation and research program of the censuses of population and housing. 1960 Series ER60. No. 1. 3. Bureau of the Census (1963). The Current Population Survey and Re-interview Program. Technical Paper No. 6. 4. Center for Advanced Engineering Study. (1983). Management for Quality, Productivity, and Competitive Position. Massachusetts Institute of Technology, Cambridge, Mass. 5. Deming, W. E. (1960). Sample Design in Business Research. Wiley, New York, Chap. 3. 6. Hill, A. B. (1953). Observation and experiment. N. Engl. J. Med., 248, 995–1001. 7. Stephan, F. F. (1936). Practical problems of sampling procedure. Amer. Sociol. Rev., 1, 569–580.
FURTHER READING See the following works, as well as the references just given, for more information on the topic of principles of professional statistical practice. Brown, T. H. (1952). The statistician and his conscience. Amer. Statist., 6(1), 14–18. Burgess, R. W. (1947). Do we need a ‘‘Bureau of Standards’’ for statistics? J. Marketing, 11, 281–282. Chambers, S. P. (1965). Statistics and intellectual integrity. J. R. Statist. Soc. A, 128, 1–16. Court, A. T. (1952). Standards of statistical conduct in business and in government. Amer. Statist., 6, 6–14. Deming, W. E. (1954). On the presentation of the results of samples as legal evidence. J. Amer. Statist. Ass., 49, 814–825. Deming, W. E. (1954). On the Contributions of Standards of Sampling to Legal Evidence and Accounting. Current Business Studies, Society of Business Advisory Professions, Graduate School of Business Administration, New York University, New York. Eisenhart, C. (1947). The role of a statistical consultant in a research organization. Proc. Int. Statist. Conf., 3, 309–313.
6424
PRIOR DENSITY
Freeman, W. W. K. (1952). Discussion of Theodore Brown’s paper [see Brown, 1952]. Amer. Statist., 6(1), 18–20. Freeman, W. W. K. (1963). Training of statisticians in diplomacy to maintain their integrity. Amer. Statist., 17(5), 16–20. Gordon, R. A. et al. (1962). Measuring Employment and Unemployment Statistics. President’s Committee to Appraise Employment and Unemployment Statistics. Superintendent of Documents, Washington, D.C. Hansen, M. H. (1952). Statistical standards and the Census. Amer. Statist., 6(1), 7–10. Hotelling, H. et al. (1948). The teaching of statistics (a report of the Institute of Mathematical Statistics Committee on the teaching of statistics). Ann. Math. Statist., 19, 95–115. Jensen, A. et al. (1948). The life and works of A. K. Erlang. Trans. Danish Acad. Tech. Sci., No. 2 (Copenhagen). International Statistical Institute (1985). Declaration on Professional Ethics, website www.cbs.nl/isi/ethics.htm Molina, E. C. (1913). Computation formula for the probability of an event happening at least c times in n trials. Amer. Math. Monthly, 20, 190–192. Molina, E. C. (1925). The theory of probability and some applications to engineering problems. J. Amer. Inst. Electr. Eng., 44, 1–6. Morton, J. E. (1952). Standards of statistical conduct in business and government. Amer. Statist., 6(1), 6–7. Shewhart, W. A. (1931). Economic Control of Quality of Manufactured Product. D. Van Nostrand, New York. Shewhart, W. A. (1939). Statistical Method from the Viewpoint of Quality Control. The Graduate School, Department of Agriculture, Washington, D.C. Shewhart, W. A. (1958). Nature and origin of standards of quality. Bell Syst. Tech. J., 37, 1–22. Society of Business Advisory Professions (1957). Report of Committee on Standards of Probability Sampling for Legal Evidence. Graduate School of Business Administration, New York University, New York. Zirkle, C. (1954). Citation of fraudulent data. Science, 120, 189–190. See also CONSULTING, STATISTICAL.
W.E. DEMING
PRIOR DENSITY. See PRIOR DISTRIBUTIONS
PRIOR DISTRIBUTIONS Prior distributions are probability distributions or probability density functions (PDFs) that summarize information about a random variable or parameter known or assumed at a given point in time, prior to obtaining further information from empirical data, and so on. The use of the designation ‘‘prior distribution’’ occurs almost entirely within the context of statistical methodology generally labeled ‘‘Bayesian inference’’∗ . Bayes’ theorem∗ and a 1763 paper by Thomas Bayes [2] are generally cited as the beginnings of this methodology. However, Stigler [12] cites evidence that the littleknown English mathematician Saunderson may have been the true builder of the foundations of what is now termed Bayesian inference. Suppose that Y and X are two random variables with PDFs fY (y) and fX (x) (Y and X may be discrete or continuous). Denoting the joint distribution of Y and X by fX,Y (x, y) and conditional distributions of Y given X and X given Y by fY|X (y|x) and fX|Y (x|y), respectively, it is known by definition that fY|X (y|x)fX (x) = fX,Y (x, y) = fX|Y (x|y)fY (y). Then, for example, fX|Y (x|y) =
fY|X (y|x)fX (x) . fY (y)
This result is one representation of the socalled ‘‘Bayes’ theorem,’’ true by definition and not an object of debate among statisticians. fX (x) tells us what is known about X before or prior to the observation of Y, and is thus termed the prior distribution of X. Alternatively, fX|Y (x|y) tells us what is known about X after or posterior to the observation of Y (see POSTERIOR DISTRIBUTIONS). The representation of Bayes’ theorem that is associated with Bayesian inference and the typical reference to prior distribution is f|Y (θ |y) =
fY| (y|θ)f (θ ) , fY (y)
PRIOR DISTRIBUTIONS
where Y = (Y1 , Y2 , . . . , Yn ) is a vector of n independent Yi representing sample observations, each with probability distribution fYi | (yi |θ) dependent on the value of m parameters in the vector . For example, for normally distributed Yi , 1 1 yi − µ 2 fYi | (yi |θ) = √ , exp − 2 σ 2π σ −∞ < yi < ∞, where = (µ, σ ), the mean and standard deviation, and fY| (y|θ) = fYi | (yi |θ). The focus of this representation is inference about the typically unknown parameter vector , where the prior distribution f (θ) is modified by the empirical data y to form the posterior distribution f|Y (θ|y). The prior distribution f (θ) is an object of debate among statisticians. Classical statisticians perceive parameters as fixed, even if unknown, and reject the idea of any ‘‘distribution’’ for . They also complain that even if one accepts the concept of such a distribution, its exact nature is almost inevitably subjective, based only on some vague degree-of-belief∗ probability concept rather than the scientifically sound relative-frequency-in-repeatedexperiments probability concept. Alternatively, Bayesian statisticians are comfortable using prior distributions for unknown parameters, even if there is considerable subjectivity involved. There have been attempts to reduce subjectivity by developing procedures for estimating prior distributions. Martz and Waller [9] discuss associating distribution parameters with characteristics of the distribution that one may feel relatively well known, such as chosen percentiles, the distribution mean, and so on. O’Bryan and Walter [10] and Walter [13] discuss specific prior distribution estimation procedures. Some commonly encountered categorizations of prior distributions are: Improper Prior: Distribution does not sum or integrate to 1. Otherwise, the prior is proper. Improper priors are often used as approximations to proper prior distributions which are nearly flat in the range of indicated to be relevant by the empirical data y. Akaike [1]
6425
and Eaton [5] discuss the interpretation and evaluation of improper priors. Reference Prior: A prior distribution used as a standard in a particular area of study [3, p. 22]. Locally Uniform Prior: Typically refers to a prior density function for a continuous parameter which is at least approximately constant over a limited relevant parameter range, thus expressing equal probability of the parameter falling in any equal-length subrange. Noninformative Prior: A prior distribution which expresses little knowledge about the parameter relative to the information expected in a size n sample of empirical data. Locally uniform priors are commonly used as noninformative priors. There is a problem, however, in selecting the metric (i.e., itself, log , −1 , etc.) in which the prior f (θ ) is locally uniform to best reflect true noninformativeness. Box and Tiao [3, p. 25] discuss this in depth. Jeffrey’s rule [7] is a rule guiding the choice of noninformative prior distributions. See JEFFREYS’ PRIOR DISTRIBUTION. Conjugate Prior: A prior distribution f (θ ) which combines fY| (y|θ) to produce a posterior distribution f|Y (θ |y) which is in the same family as the prior distribution; for example, both are normal distributions. See Martz and Waller [9, p. 226], De Groot [4, p. 164], Raiffa and Schlaifer [11, p. 53], and CONJUGATE FAMILIES OF DISTRIBUTIONS. Exchangeable Prior: A prior distribution f (θ ) which is not altered by any permutation in the multiparameter . Exchangeability∗ has been examined in the context of parameters from multiple populations and multinomial parameters of a single population. See Lindley and Smith [8] and Eaves [6]. REFERENCES 1. Akaike, H. (1980). J. R. Statist. Soc. B, 42, 46–52. (On the interpretation of improper prior distributions as limits of data-dependent proper prior distributions.) 2. Bayes, T. (1958). Biometrika, 45, 298–315. (Reprint of the famous 1763 paper generally
6426
3.
4. 5.
6.
7. 8.
9.
10. 11.
12.
13.
PRIORITY QUEUE
credited with giving birth to Bayesian statistical inference.) Box, G. E. P. and Tiao, G. C. (1973). Bayesian Inference in Statistical Analysis. AddisonWesley, Reading, Mass. (An excellent general statistical reference on Bayesian inference.) DeGroot, M. H. (1970). Optimal Statistical Decisions. McGraw-Hill, New York. Eaton, M. L. (1982). In Statistical Decision Theory and Related Topics III, Vol. 1, S. Gupta and J. Berger, eds. Academic Press, New York, pp. 1329–1352. (Evaluating improper prior distributions.) Eaves, D. M. (1980). J. R. Statist. Soc. B, 42, 88–93. (Good reference list for work with exchangeable priors.) Jeffreys, H. (1961). Theory of Probability, 3rd ed. Clarendon Press, Oxford, England. Lindley, D. V. and Smith, A. F. M. (1972). J. R. Statist. Soc. B, 34, 1–41. (Exchangeable priors in the context of the usual linear statistical model.) Martz, H. F. and Waller, R. A. (1982). Bayesian Reliability Analysis. Wiley, New York. (Chapters 5 and 6 contain good general knowledge about Bayesian inference.) O’Bryan, T. and Walter, G. (1979). Sankhya A, 41, 95–108. Raiffa, H. and Schlaifer, R. (1961). Applied Statistical Decision Theory. Harvard University Press, Cambridge, Mass. Stigler, S. M. (1983). Amer. Statist., 37, 290–296. (A witty search into who really discovered Bayes’ theorem.) Walter, G. (1981). Sankhya A, 43, 228–245.
BIBLIOGRAPHY See the following works, as well as the references just given, for more information on the topic of prior distributions. Lindley, D. V. (1965). Introduction to Probability and Statistics from a Bayesian Viewpoint, Part 1: Probability; Part 2: Inference, Cambridge University Press, Cambridge, England. Savage, L. J. (1954). The Foundations of Statistics. Wiley, New York. (General reference with philosophical basis for Bayesian statistical inference.) See also BAYESIAN INFERENCE; BAYES’ THEOREM; BELIEF, DEGREES OF; CONJUGATE FAMILIES OF DISTRIBUTIONS; POSTERIOR DISTRIBUTIONS; POSTERIOR PROBABILITIES; and PRIOR PROBABILITIES.
DAVID J. PACK
PRIORITY QUEUE As a general background regarding ideas and notation, we refer the reader to QUEUEING THEORY. The theory for priority queueing systems is the branch of queueing theory dealing with systems where high-priority customers get faster through the system at the expense of others. The basic reference seems still to be ref. 4, giving a series of different models, with a variety of priority disciplines, and a solid mathematical treatment. In ref. 5, priority queueing systems of special interest in computer applications are considered. Since the applicability of queueing theory has often been questioned, we will here give reviews of three papers having specific applications in mind. We are not claiming that these are either the best or the most general in the area. One should also remember that even a complex queueing system is more sympathetic to mathematical modeling than is a series of other real-life systems. Hence queueing theory has been and will still be an experimental area to develop tools which are useful in other areas of applied probability. CLASSICAL PRIORITY QUEUEING MODELS Assume that the customers are divided into p different priority classes each having a priority index 1, 2, . . . , p. The index 1 corresponds to the highest-priority class and p corresponds to the lowest. For simplicity we assume a single server and an infinite waiting-room capacity. Within each class we have a first-in-first-out (FIFO) queueing discipline. Denoting the interarrival-time distribution and the service-time distribution within the ith class by the symbols Ai and Bi , respectively, we can use the notation A1 / · · · /Ap /B1 / · · · /Bp /1 for this class of models. However, we have to specify the queueing discipline between customers from different priority classes. Compare the ith and jth classes, where 1 i < j p. In the queue all customers from the ith class are standing in front of all customers from the jth class and will hence be served first. If, on the other hand, a customer from the jth class is being served on the arrival of a customer from the
PRIORITY QUEUE
ith class, there are different alternatives. The priority discipline is preemptive∗ if the current service is immediately interrupted and service is started for the arriving customer. It is nonpreemptive if service is continued to completion, and it is discretionary if the server may use his or her discretion to decide which of the two former strategies to use in each case. For instance, he or she may use the nonpreemptive discipline if the amount of service received exceeds a certain level [4], or if remaining service time is sufficiently small [3]. APPOINTMENT SYSTEM Consider the GI/M/1 queueing model with infinite waiting-room capacity. The customer arriving at t = 0 will find k − 1 customers waiting. The latter customers belong to the second priority class, whereas the ones arriving in [0, ∞) belong to the first. The priority discipline is nonpreemptive, whereas within each class we have a FIFO queueing discipline. As a motivation for studying the present model, consider the following specialization of the arrival pattern above which is realistic when, for example, doctors, dentists, or lawyers are consulted. Let the intervals between possible arrivals have fixed length 1/λ and let the probability of a customer not turning up be 1 − p. Customers turn up or not independently of each other. The number of intervals of length 1/λ between the arrivals of two customers are then geometrically distributed with parameter p. The k − 1 customers in the second priority class do not have an appointment but are allowed to queue up, for instance, either before the office is opened in the morning or before it is reopened after the lunch break. One is now interested in: 1. Waiting times for customers from both priority classes not to be too long. 2. The initial busy period (starting with k customers in the system) not to be too short. Small values of k will satisfy 1, whereas large values satisfy 2.
6427
In ref. 1 the transient waiting times for customers belonging to both priority classes are arrived at for the general model. Using these results on the special arrival pattern above, optimal values for k have been tabulated for various values of p, λ and service intensity. COMPUTER TIME SHARING In ref. 2 a modification of the so-called roundrobin priority discipline is treated in an M/M/ 1 queueing model. Each program receives a quantum q of service at a time from a single central processor. If this completes its service requirement, it leaves the system. If not, and there is a new arrival during the service of the quantum, it is given an additional quantum. Otherwise, the program joins the end of the queue to await its next turn. The model is analyzed under the assumption of a constant, nonzero overhead when the processor swaps one program for another. Obviously, during periods of high arrival rates, this algorithm has the effect of reducing the system’s swapping activities. On the other hand, during periods of low arrival rates the discipline is similar to the conventional round robin, which automatically gives priority to programs with lesser service time requirements. Reference 2 arrives at expressions for the mean waiting time in queue as a function of the quanta required and for the mean system cost due to waiting. Numerical comparisons with the conventional round-robin discipline are performed. TELEPHONE EXCHANGE In ref. 6 a telephone exchange is considered, handling the calls to and from a minor group of subscribers. Let the latter group belong to the second priority class, whereas people from the rest of the world belong to the first class. The Electronic Selector Bar Operator (ESBO) serves customers from both classes. The model one wished to be able to analyse is a modified version of GI/G/M/G/ 1 with a nonpreemptive priority discipline. However, customers of the higher priority are just
6428
PRIOR PROBABILITIES
allowed to wait a fixed length of time before they hear the busy signal and are lost. This is not the case for customers of lower priority. Second, whereas we have a FIFO queueing discipline within the first class, it is random within the second (i.e., all customers waiting have equal chance of being served next). Unfortunately, there seems no way of arriving at the stationary waiting-time distributions for customers of the two classes by using queueing theory of today. As a first approximation ref. 6 starts out from an M1 /G/M2 /G/1 model where Laplace transforms of the distributions above are well known. Furthermore, in this case it is possible to get a look behind the ‘‘Laplace curtain.’’ REFERENCES 1. Dalen, G. and Natvig, B. (1980). J. Appl. Prob., 17, 227–234. 2. Heacox, H. and Purdom, P. (1972). J. ACM, 19, 70–91. 3. Hokstad, P. (1978). INFOR, 16, 158–170. 4. Jaiswal, N. K. (1968). Priority Queues. Academic Press, New York. 5. Kleinrock, L. (1976). Queueing Systems, Vol. 2: Computer Applications. Wiley, New York. 6. Natvig, B. (1977). On the Waiting-Time for a Priority Queueing Model. Tech. Rep. No. 3, Dept. of Mathematical Statistics, University of Trondheim, Trondheim, Norway. Prepared for the Norwegian Telecommunications Administration Research Establishments. See also PREEMPTIVE DISCIPLINE and QUEUEING THEORY.
BENT NATVIG
PRIOR PROBABILITIES Prior probabilities are probabilities that are applicable at a point in time which is prior to one obtaining additional information provided by empirical studies, experiments, surveys, and so on, yet to be executed. In the context of a particular random variable or unknown parameter, prior probabilities follow directly from the choice of a prior distribution∗ , as the assumption of a probability distribution or probability density function in any statistical endeavor automatically leads to the ability to determine
probabilities of various events of interest. See PRIOR DISTRIBUTIONS for a full discussion and references. Suppose for the sake of illustration that you are concerned with the probability of the event E that a particular person known to you only by name is a college graduate. Knowing nothing else, you might reason that about 1 in 20 people in the general population is a college graduate, and thus your probability for E is Pr[E] = 0.05. This is a prior probability. It is based on your knowledge at this time and is not conditioned on any other event (explicitly). It is a probability that you would use prior to getting any other information about the person. If, for example, you knew the person read Newsweek or a similar news magazine, you might feel the chances of the person being a college graduate were more than the 1 in 20 in the general population. The illustration above suggests an element that is frequently present in discussions of prior probabilities—subjective probability assessment. Pr[E] = 0.05 is a statement of degree of belief∗ , not an objective statement based on repeated experiments or some scientific fact. Prior probabilities may have some objectivity, particularly when they are the posterior probabilities from previous empirical observations (see POSTERIOR DISTRIBUTIONS), but in most cases there is inevitably a significant subjective component. The discipline called applied statistical decision theory∗ as discussed by Raiffa and Schlaifer [11] and Raiffa [10] is one that particularly illustrates the utilization of subjective prior probabilities. There is a substantial literature on the quantification of degree-of-belief information in a prior probability. Martz and Waller [8] contains a good bibliographic overview. Hampton et al. [3] and Hogarth [4] are good general technical references. The issue of encoding prior probabilities from one expert source is dealt with in Lichtenstein et al. [6] and Lindley et al. [7]. One may also be concerned with forming the prior probabilities from a number of expert sources into a consensus probability, as discussed by DeGroot [2] and Morris [9]. Adams [1] and Kazakos [5] provide other perspectives on the
PROBABILISTIC NUMBER THEORY
general problem of quantifying prior probabilities. REFERENCES 1. Adams, E. W. (1976). In Foundations of Probability Theory, Statistical Inference and Statistical Theories of Science, Vol. 1, W. L. Harper and C. A. Hooker, eds. D. Reidel, Boston, pp. 1–22. 2. DeGroot, M. H. (1974). J. Amer. Statist. Ass., 69, 118–121. (Consensus prior probabilities via prior distributions which are mixtures of individual expert priors.) 3. Hampton, J. M., Moore, P. G., and Thomas, H. (1973). J. R. Statist. Soc. A, 136, 21–42. (General reference on the measurement of subjective probability.) 4. Hogarth, R. M. (1975). J. Amer. Statist. Ass., 70, 271–289. (General reference on the measurement of subjective probability.) 5. Kazakos, D. (1977). IEEE Trans. Inf. Theory, 23, 203–210. (Estimation of prior probabilities using a mixture.) 6. Lichtenstein, S., Fischhoff, B., and Phillips, L. D. (1977). In Decision Making and Change in Human Affairs, H. Jungerman and G. de Zeeuv, eds. D. Reidel, Dordrecht, The Netherlands, pp. 275–324. (Encoding prior probabilities.) 7. Lindley, D. V., Tversky, A., and Brown, R. V. (1979). J. R. Statist. Soc. A, 142, 146–180. (Reconciling inconsistent prior probability assessments.) 8. Martz, H. F. and Waller, R. A. (1982). Bayesian Reliability Analysis. Wiley, New York. [Section 6.4 gives an overview of prior probability (distribution) assessment.] 9. Morris, P. A. (1977). Manag. Sci., 23, 679–693. (Consensus prior probabilities.) 10. Raiffa, H. (1968). Decision Analysis, Introductory Lectures on Choices under Uncertainty. Addison-Wesley, Reading, Mass. (Introduction to simple statistical decision theory.) 11. Raiffa, H. and Schlaifer, R. (1961). Applied Statistical Decision Theory. Harvard University Press, Cambridge, Mass. See also BELIEF, DEGREES OF; POSTERIOR DISTRIBUTIONS; POSTERIOR PROBABILITIES; and PRIOR DISTRIBUTIONS.
DAVID J. PACK
PROBABILISTIC GRAMMAR. See LINGUISTICS, STATISTICS IN
6429
PROBABILISTIC NUMBER THEORY Some questions and results of number theory can easily be expressed in probabilistic terms; for others, proofs can be simplified by probabilistic arguments. Probabilistic number theory can therefore be defined as a branch of number theory in which questions are of interest to number theorists, but the proofs utilize techniques of probability theory. There are two major subfields of probabilistic number theory: (a) distribution of values of arithmetical functions, and (b) metric (measure theoretic) results concerning expansions of real numbers. These are discussed in detail in the first two sections. The present article is special in that it is somewhat outside the expertise of most readers. Therefore, a detailed survey of the subject is not aimed at; rather, an expository introduction is given in which the major trends, but not the fine points, are presented. DISTRIBUTION OF VALUES OF ARITHMETICAL FUNCTIONS A function f (n) on the positive integers n = 1, 2, . . ., is called an arithmetical function. In other words, an arithmetical function is a sequence in which the emphasis on the variable (index) n is its arithmetical structure rather than its magnitude. Let us start with an example. Let f (n) be the number of all (distinct) prime divisors of n. Hence, to determine the value of f (n), one first has to express n as the product of powers of primes, and then the number of factors in this product is f (n). One can hardly find any regularity of this f (n), because it is frequently equal to 1 (at prime numbers), but it also exceeds any prescribed value infinitely many times (e.g., when n is a factorial). Here is where probabilistic number theory comes in: One can prove, by an appeal to the Chebyshev inequality∗ , that, for ‘‘most numbers n,’’ f (n) is approximately loglog n. More precisely, if A(N) denotes the number of positive integers n N such that |f (n) − log log n| < (log log n)3/4 , then A(N)/N → 1 as N → +∞. To prove this claim, let us introduce the probability spaces
6430
PROBABILISTIC NUMBER THEORY
where p < q are prime numbers and [y] signifies the integer part of y, we have
that log log n ∼ log log N for all n satisfying √ N n N, say, the Chebyshev inequality now implies our claim on A(N) concerning the closeness of f (n) to log log n. This computation represents well the essence of probabilistic number theory. The question, namely the behavior of f (n), originates from number theory; then the basic idea of applying the Chebyshev inequality is a tool of probability theory, but finally, in computing E(f (n)), the application of the prime number theorem is unavoidable. This procedure is generally the case in connection with number-theoretic functions. Only the basic ideas come from probability theory; in the actual computations number-theoretic results are frequently utilized. Let us now consider a whole family of functions f (n) satisfying
PN (ep (n) = 1) = 1/p + O(1/N)
f (nm) = f (n) + f (m)
(N , AN , PN ), where N = {1, 2, . . . , N}, AN is the set of all subsets of N , and PN is the uniform distribution on N . Now the function fN (n) = f (n), 1 n N, is a random variable, which can be represented as f (n) = ep (n), where the summation is over primes p N, and ep (n) = 1 if p divides n and ep (n) = 0 otherwise. The expectation and the variance of f (n) can easily be computed by means of this representation. Namely, because PN (ep (n) = 1) = [N/p]/N and PN (ep (n) = eq (n) = 1) = [N/(pq)]/N,
and PN (ep (n) = eq (n) = 1) = 1/(pq) + O(1/N). Thus E(f (n)) =
1/p + O(π (N)/N),
pN
where π (N) is the number of primes not exceeding N. However, from number theory we know that π (N) ∼ N/ log N (the so-called prime number theorem), which yields E(f (n)) =
(1)
for all n and m which are relatively prime (i.e., they do not have any common divisor other than 1). Functions satisfying (1) are additive arithmetical functions. If, in addition, the additive function f (n) satisfies f (pk ) = f (p) for all primes p and integers k 1, then f (n) is strongly additive. In what follows there is no difference between additive and strongly additive functions. However, computations are much simpler with strongly additive functions, so we restrict ourselves to these. Now, for such functions f (n), (1) implies that
N π (m) − π (m − 1) 1 +O m log N
f (n) =
f (p)ep (n),
(2)
m=1 N
1 1 π (m) = − m m+1 m=1 1 +O log N =
N m=1
1 + O(1) (m + 1) log m
= log log N + O(1). The variance of f (n) can be computed similarly; it also equals log log N. Finally, observing that log log N is so slowly increasing
where summation is again over all primes p (when n is restricted to N , then p N). One easily gets from (2) that E(f (n)) ∼ AN and V(f (n)) ∼ B2N , where AN =
f (p) p
pN
and
B2N =
f 2 (p) . (3) p
pN
Repeating the argument with the Chebyshev inequality, we get that f (n) ∼ AN for ‘‘most n’’ whenever BN → +∞ with N. More accurate results can also be deduced from (2). Because the indicator variables ep (n)
PROBABILISTIC NUMBER THEORY
are almost independent, that is, for arbitrary primes p1 < p2 < · · · < pk , PN (ep1 (n) = ep2 (n) = · · · = epk (n) = 1) = 1/(p1 p2 · · · pk ) + O(1/N), (4) the sums in (2) and (3) suggest that the distribution of (f (n) − AN )/BN converges weakly under conditions similar to sums of independent random variables. Although most results for sums of independent random variables are indeed valid for strongly additive functions, unfortunately these cannot be deduced directly from probability theory because the error term O(1/N) in (4) is more troublesome than it looks at first. Namely, if √ only two primes in (4) are around N, the major term is already smaller than the error term, and it is not easy to show that one can limit the primes in (2) to remain below powers of N. This is why one is forced to consider f (p)ep (n) fr (n) = pr
in place of f (n), where r = r(N) is some slowly increasing function of N, and then one uses probability theory to determine the asymptotic behavior of fr (n), and number theory (so-called sieve methods) to show that the difference f (n) − fr (n) is small compared with BN . This line of attack proves very successful. A few basic results will be quoted as we now turn to the history of the subject. The references are far from complete, even when one adds all references in the survey papers and books listed later. The probabilistic theory of arithmetic functions started with the paper of ` [43], who gave a new proof, essentially Turan through the Chebyshev inequality (although without reference to probability theory), for the asymptotic relation between the number of prime divisors and log log n. The result itself had been known in number theory, but it was obtained by analytic methods. Both P. Erd¨os and M. Kac were influenced ´ by Turan’s proof. They, first independently, and later jointly, contributed at the early stages the most significant results of the distribution theory of additive functions. It was Kac who approached the subject from
6431
a purely probabilistic point of view by recognizing in the representation at (2) the sum of almost independent random variables. Kac’s early work is well set out in his popular monograph [24]. Erd¨os [7] and Erd¨os and Wintner [10] found necessary and sufficient conditions for the weak convergence of PN (f (n) < x) for arbitrary additive functions f (n). Erd¨os and Kac [9] established that if the sequence f (p) is bounded (over primes p), then, for the strongly additive function f (n), PN (f (n) < AN + BN x) is asymptotically standard normal, where AN and BN are defined at (3). It turned out that the boundedness of f (p) is not necessary for asymptotic normality∗ . In a series of papers, J. Kubilius developed a probabilistic theory of distributions for the truncated functions fr (n), in which his only requirement on r is that r → +∞ with N and log r/log N should converge to zero as N → +∞. If in addition Br /BN → 1 and BN → +∞ as N → +∞, then f (n) and fr (n) have the same asymptotic laws. Later, Kubilius published his work in a book which has had several editions; we quote here the English language edition [30]. His contribution to the theory of distributions of arithmetical functions is immeasurable because he not only published significant papers and books on the subject, but established a school at Vilnius which produced a large number of scientists working on the subject and also made a considerable influence on both individual scientists all over the world and the literature. Some of the work of this school is presented in the survey papers of Galambos [11,14], Kubilius [31], Manstavicius [34], and Schwarz [41], and in the books of Kubilius [30] and Elliott [6]. In the Kubilius model, to show that (f (n) − fr (n))/BN converges to zero (‘‘in probability’’), quite deep number theoretic results are employed. This induced investigations aimed at reobtaining at least some of the results of Kubilius by purely probabilistic arguments. Several models have been set up, which to various degrees have achieved the desired results, but none of them would have come close to the generality of the Kubilius model. However, through the models of Novoselov [35], Paul [36], Galambos [12], and Philipp [37], interesting probabilistic proofs are given for one result or another in the distribution theory of additive
6432
PROBABILISTIC NUMBER THEORY
functions. The model of Galambos has the added advantage that it presents a unified treatment of additive functions when the variable is not limited to the set of successive integers. Such investigations are extensive ` in the literature; see Kubilius [30], Katai [26,27], and Indlekofer [21]. For the truncated function fr (n) Kubilius was able to use the theory of sums of independent random variables. However, for extending his results to additive functions for which a truncation is not possible, other methods are necessary. The method of characteristic functions∗ proved fruitful, although it required new deep number-theoretic investigations. Namely, since PN is the uniform distribution over a finite set, the characteristic function of an arithmetical function f (n) is the arithmetical mean of the function g(n) = exp(itf (n)). Now, if f (n) is additive, then, in view of (1), g(n) satisfies the equation g(mn) = g(m)g(n)
(5)
for all m and n which are relatively prime. A function satisfying (5) is multiplicative. Thus asymptotic mean value theorems were needed for multiplicative functions whose absolute value is bounded by 1. Such results were established by Delange [5] and ` [18], who employed tools of analytic Halasz number theory (see also R´enyi [40]). These mean value theorems gave new life to the distribution theory of arithmetical functions. In a series of papers, Elliott, C. Ryavec, ` Delange, Halasz, and the members of the Vilnius school brought the theory of additive functions to a very advanced level. The twovolume set of books by Elliott [6] is based on this new development. The distribution theory of multiplicative functions has also been studied in detail. Because multiplicative functions which can take both positive and negative values cannot be related to real-valued additive functions, their distribution theory requires a completely different approach. The combined result of Bakstys [1] and Galambos [13] gives necessary and sufficient conditions for the weak convergence of PN (g(n) < x) for multiplicative functions g(n), and the weak convergence of the normalized multiplicative function (g(n) − CN )/DN is characterized in Levin et al. [33].
For arithmetical functions which are neither additive nor multiplicative, approximation methods have been developed: by additive functions in Galambos [15] and Erd¨os and Galambos [8], and by so-called Ramanujan sums in a series of papers by W. Schwarz and his collaborators; see the survey [41]. The distribution theory of arithmetical functions has several applications within number theory. For example, for estimating certain sums, see De Koninck and Ivic [4], and for its role in characterizing log n among all additive functions, see Indlekofer [22]. For how much joy the study of this theory can give, one should read the recollections of Kac [25]. METRIC THEORY OF EXPANSIONS OF REAL NUMBERS When a sequence of ‘‘digits’’ is assigned to real numbers by a specific algorithm, we speak of an expansion. The best known expansion is the decimal, or more generally, the q-adic expansions, in which, when q is an integer, the digits are independent and identically distributed random variables with respect to Lebesgue measure. Hence, although no regularity of the digits of irrational numbers seems to hold, the law of large numbers implies that these digits are quite regular for almost all x. Other theorems of probability theory are directly applicable as well. The situation is completely different for most other series expansions. As an example, let us look at the so-called Engel series. For a given real number 0 < x 1, the ‘‘digits’’ dj are defined by the following algorithm: Every dj is an integer such that 1/dj < xj 1/(dj − 1), where x = x1 and xj+1 = dj xj − 1. This algorithm leads to the series representation x = 1/d1 + 1/(d1 d2 ) + 1/(d1 d2 d3 ) + · · · , where 2 d1 d2 d3 · · · . The digits dj are strongly dependent in view of their monotonicity. However, the ratios
PROBABILISTIC NUMBER THEORY
dj /(dj−1 − 1) can well be approximated by independent and identically distributed random variables, which can be utilized for discovering general properties of the sequence dj , valid for almost all x. Similar situations arise in connection with most series expansions. That is, the digits are dependent but after some transformation an approximation by independent random variables becomes possible. This is not obvious, though, in most instances, and the challenge in these problems usually is probabilistic: to extend known theorems of probability theory to some weakly dependent random variables. These investigations did indeed contribute to probability theory itself. Most results on series expansions are contained in the books of Galambos [17] and Vervaat [44], although for some expansions there are newer results as well (see, e.g., Deheuvels [3] concerning the Engel series). Among the expansions which do not result in a series most attention was paid to continued fraction expansions. This is obtained by repeated application of the transformation Tx = x − [1/x], where, once again, [y] signifies the integer part of y. As far back as in the nineteenth century, Gauss recognized that, if instead of length (i.e., Lebesgue measure), an absolutely continuous measure is assigned to the unit interval whose derivative is 1/{(log 2)(1 + x)}, then the sequence an , defined by the algorithm aj = [1/xj ], where x = x1 and Txj = xj+1 , is stationary. More recent studies revealed that this sequence is stationary mixing, which fact induced intensive studies of mixing sequences of random variables. This benefited both probability theory and number theory; see Ibragimov and Linnik [20], Billingsley [2], and Philipp [37]. For a more classical approach, see Khintchine [29]. On the speed of convergence of the distribution of Txn , see Kuzmin [32] and ¨ [42]. Galambos [16], Philipp [38], and Szusz Iosifescu [23] deal with the stochastic behavior of the largest of the digits an . The work of R´enyi [39] shows the connection between series expansions and continued fractions. We conclude by noting that there are several works in probabilistic number theory which do not belong to either topic covered in the preceding two sections. These are, however, individual problems rather than a
6433
part of a systematic study. We mention two ` of these. Katai [28] is probably the purest among all papers in probabilistic number theory in that he shows that for certain sequences of integers, the number of divisors √ of n in the neighborhood of n is normally distributed, in which argument no number theory is applied. The other area we wish to mention is the proof of the existence of certain types of numbers by showing that their probability is positive. Although most of the results are due to Erd¨os, the most convenient source for consultation is Halberstam and Roth [19]. REFERENCES 1. Bakstys, A. (1968). Litovsk. Mat. Sb., 8, 5–20. 2. Billingsley, P. (1965). Ergodic Theory and Information. Wiley, New York. 3. Deheuvels, P. (1982). C. R. Acad. Sci. Paris, 295, 21–24. 4. De Koninck, J. -M. and Ivic, A. (1980). Topics in Arithmetical Functions. North-Holland, Amsterdam. ´ 5. Delange, H. (1961). Ann. Sci. Ecole Norm. Sup., 78, 273–304. 6. Elliott, P. D. T. A. (1979/1980). Probabilistic Number Theory, Vols. I and II. SpringerVerlag, Berlin. 7. Erd¨os, P. (1938). J. Lond. Math. Soc., 13, 119–127. 8. Erd¨os, P. and Galambos, J. (1974). Proc. Amer. Math. Soc., 46, 1–8. 9. Erd¨os, P. and Kac, M. (1940). Amer. J. Math., 62, 738–742. 10. Erd¨os, P. and Wintner, A. (1939). Amer. J. Math., 61, 713–721. 11. Galambos, J. (1970). Ann. Inst. Henri Poincar´e B, 6, 281–305. 12. Galambos, J. (1971). Z. Wahrscheinl. verw. Geb., 18, 261–270. 13. Galambos, J. (1971). Bull. Lond. Math. Soc., 3, 307–312. 14. Galambos, J. (1971). In The Theory of Arithmetic Functions, A. A. Gioia et al., eds. Springer-Verlag, Berlin, pp. 127–139. 15. Galambos, J. (1973). Proc. Amer. Math. Soc., 39, 19–25. 16. Galambos, J. (1974). Acta Arith., 25, 359–364. 17. Galambos, J. (1976). Representations of Real Numbers by Infinite Series. Springer-Verlag, Berlin.
6434
PROBABILISTIC VOTING MODELS
` 18. Halasz, G. (1968). Acta Math. Acad. Sci. Hung., 19, 365–403. 19. Halberstam, H. and Roth, K. F. (1966). Sequences 1. Oxford University Press, Oxford, England. 20. Ibragimov, I. A. and Linnik, Yu. V. (1971). Independent and Stationary Sequences of Random Variables. Wolters-Noordhoff, Groningen. 21. Indlekofer, K. -H. (1974). In Proc. Coll. Bolyai Math. Soc. Debrecen, B. Gyires, ed. NorthHolland, Amsterdam, pp. 111–128. 22. Indlekofer, K. -H. (1981). Ill. J. Math., 25, 251–257. 23. Iosifescu, M. (1978). Trans. 8th Prague Conf. Inf. Theory, Vol. A. D. Reidel, Dordrecht, The Netherlands, pp. 27–40. 24. Kac, M. (1959). Statistical Independence in Probability, Analysis and Number Theory. Carus Monographs. Wiley, New York. 25. Kac, M. (1979). Probability, Number Theory and Statistical Physics. MIT Press, Cambridge, Mass. ` 26. Katai, I. (1968). Compositio Math., 19, 278–289. ` 27. Katai, I. (1969). Acta Math. Acad. Sci. Hung., 20, 69–87. ` 28. Katai, I. (1977). Publ. Math. Debrecen, 24, 91–96. 29. Khintchine, A. (1964). Continued Fractions. Wolters-Noordhoff, Groningen, The Netherlands. 30. Kubilius, J. (1964). Probabilistic Methods in the Theory of Numbers. Transl. Math. Monogr., Vol. 11. American Mathematical Society, Providence, R.I. 31. Kubilius, J. (1974). In Current Problems of Analytic Number Theory, Izd. Nauka i Techn., Minsk, pp. 81–118. 32. Kuzmin, R. O. (1928). Rep. Pol. Acad. Sci. (A), 375–380. 33. Levin, B. V., Timofeev, N. M., and Tuljaganov, S. T. (1973). Litovsk. Mat. Sb., 13, 87–100. 34. Manstavicius, E. (1980). Litovsk. Mat. Sb., 20, 39–52. 35. Novoselov, E. V. (1964). Izv. Akad. Nauk SSSR, Ser. Mat., 28, 307–364. 36. Paul, E. M. (1962). Sankhya A, 24, 103–114, 209–212; ibid., 25, 273–280 (1963). 37. Philipp, W. (1971). Mem. Amer. Math. Soc., 114, 1–102. 38. Philipp, W. (1976). Acta Arith., 28, 379–386.
39. R´enyi, A. (1957). Acta Math. Acad. Sci. Hung., 8, 477–493. 40. R´enyi, A. (1965). Publ. Math. Debrecen, 12, 323–330. 41. Schwarz, W. (1976). Jb. Dtsch. Math.-Ver., 78, 147–167. ¨ 42. Szusz, P. (1961). Acta Math. Acad. Sci. Hung., 12, 447–453. ` 43. Turan, P. (1934). J. Lond. Math. Soc., 9, 274–276. 44. Vervaat, W. (1972). Success Epochs in Bernoulli Trials with Applications in Number Theory. Math. Centre Tracts No. 42, Mathematische Centrum, Amsterdam. See also FIRST-DIGIT PROBLEM; LAWS OF LARGE NUMBERS; ¨ FUNCTION; and PROBABILITY THEORY: AN MOBIUS OUTLINE).
JANOS GALAMBOS
PROBABILISTIC VOTING MODELS In many nations and at many levels of government, the question of who should hold various political offices is settled by voting. As a consequence, there are many public officials, and many politicians who would like to become public officials, who are concerned about the implications of their choices for the decisions that voters will make. In particular, they are concerned about which voters will vote in the next election and who these people will vote for. Two choices that elected officials and officeseeking politicians make that are especially important in this regard are: Their positions on the leading policy questions of the day and the way in which they allocate their campaign resources. As a consequence, researchers have developed mathematical models of the relation between these choices and the decisions that public officials and office-seeking politicians can expect the voters to make. These models can be used to address questions such as: In a given situation in which campaign resources have already been allocated, are there ‘‘best’’ policy positions that can be taken? If so, what are they? In a given situation in which policy positions have already been selected, is there a ‘‘best’’ campaign resource allocation that can be chosen? If so, what is it? In a given situation in which both policy positions
PROBABILISTIC VOTING MODELS
and an allocation of resources can be selected, is there a ‘‘best’’ possible combination? If so, what is it? The answers are useful for social scientists who want to know what policy positions and campaign resource allocations can be expected from public officials and officeseeking politicians. They are also potentially useful for campaign strategists who advise political candidates.
TWO EXAMPLES Before stating the features that differ in the two examples in this section, the characteristics that they have in common will be covered: There are two candidates for a particular political office. They will be indexed by the elements in the set C = {1, 2}. There are three voters. They will be indexed by the elements in the set N = {1, 2, 3}. Each candidate has to decide on what policy proposals he or she wants to make. The only characteristic of a candidate’s policy proposals that matters to the voters is the distribution of income that they expect the policies to lead to. For each vector of policy proposals that a candidate can make, each voter has a unique distribution of income that he or she expects the policies to lead to. These expectations are the same for all three voters. As a consequence, the candidate’s decision can be viewed as the choice of an income distribution∗ . It will be assumed that each candidate can, in particular, choose any ψc ∈ X = {(x1 , x2 , x3 ) ∈ R3 : x1 + x2 + x3 = 1, x1 0.01, x2 0.01, x3 0.01} (by a suitable choice of policy proposals). Each particular voter cares only about his or her own income. More specifically, (for any given i ∈ N) for each pair x, y ∈ X : x is at least as good as y for i if and only if xi yi . This implies that i prefers x to y (i.e., x is at least as good as y for i, but y is not at least as good as x for i) if and only if xi > yi . It also implies that i is indifferent between x and y (i.e., x is at least as good as y for i and y is at least as good as x for i) if and only if xi = yi . Thus i’s preferences on X can be represented by the utility function Ui (x) = xi ; for each pair x, y ∈ X, Ui (x) Ui (y) if and only if x is at least as good as y for i. In addition, assume
6435
that Ui (x) measures the intensity of i’s preferences. More specifically, the values assigned by Ui (x) have the following interpretations: Ui (x)/Ui (y) = 12 if and only if i likes x only half as much as he or she likes y; Ui (x)/Ui (y) = 2 if and only if i likes x twice as much as he or she likes y; Ui (x)/Ui (y) = 3 if and only if i likes x three times as much as he or she likes y; and so on. This property implies that each voter’s utility∗ function is unique up to multiplication by a positive scalar. (It is, accordingly, called a ratio-scale utility function.) In the examples, Pci (ψ1 , ψ2 ) will be used to denote the probability that the individual indexed by i will vote for candidate c when c = 1 chooses ψ1 and c = 2 chooses ψ2 . Thus [at any given (ψ1 , ψ2 ) ∈ X 2 ] the expected vote for a given c ∈ C can be written as EVc (ψ1 , ψ2 ) =
3
Pci (ψ1 , ψ2 ).
i=1
Each candidate c is concerned solely about his or her expected plurality. Plc (ψ1 , ψ2 ) = EVc (ψ1 , ψ2 ) − EVk (ψ1 , ψ2 ) (where k is the index for the other candidate), that is, with his or her expected margin of victory (or, phrased differently, how much he or she expects to win or lose by). Furthermore, each candidate wants to maximize his or her expected plurality; in a candidate’s view, the larger the expected margin of victory, the better. This implies that, for any specification of the Pci (·) functions, the decisions that the two candidates have to make can be appropriately modeled as a two-person, noncooperative game, (X, X; Pl1 , Pl2 ), in which (1) the two players are the two candidates, (2) the strategy set for each candidate is X, and (3) the payoff functions are Pl1 : X × X → R1 and Pl2 : X × X → R1 , respectively. By the definitions of Pl1 and Pl2 , Pl1 (ψ1 , ψ2 ) + Pl2 (ψ1 , ψ2 ) = 0∀(ψ1 , ψ2 ) ∈ X 2 . Hence the game is zero-sum. See GAME THEORY or NASH EQUILIBRIUM. Example 1. Consider the case in which, for each i ∈ N and each pair (ψ1 , ψ2 ) ∈ X 2 , 1 if Ui (ψ1 ) > Ui (ψ2 ) P1i (ψ1 , ψ2 ) = 12 if Ui (ψ1 ) = Ui (ψ2 ) 0 if Ui (ψ1 ) < Ui (ψ2 )
6436
PROBABILISTIC VOTING MODELS
and P2i (ψ1 , ψ2 ) = 1 − P1i (ψ1 , ψ2 ). The resulting game, (X, X; Pl1 , Pl2 ), has no Nash equilibrium. Since the game is zerosum, this can also be phrased as: The game has no saddle point. The fact that this game has no Nash equilibrium (or, equivalently, has no saddle point) can be seen quite easily. Choose any (x, y) ∈ X 2 . Since the game is zero-sum, there are three possibilities: (a)
Pl1 (x, y) = Pl2 (x, y) = 0,
(b)
1
(c)
2
Pl (x, y) > 0 > Pl (x, y), or Pl1 (x, y) < 0 < Pl2 (x, y).
Suppose that (a) or (b) holds. Identify a voter, i, who gets at least as much income at x as anyone else (i.e., for whom we have xi xj , ∀j = i). Let z be the alternative in which this individual gets zi = 0.01 and the others get zj = xj + 12 (xi − 0.01) (i.e., the others split the decrease in i’s income). Since xi > 0.01, we have Uj (z) = zj > xj = Uj (x) for each j = i and Ui (z) = zi < xi = Ui (x). Therefore, P2j (x, z) = 1 and P1j (x, z) = 0 for each j = i and, in addition, P2i (x, z) = 0 and P1i (x, z) = 1. Therefore, Pl2 (x, z) = +1 > 0 Pl2 (x, y). Therefore, if (a) or (b) holds, (x, y) is not a Nash equilibrium. Similar reasoning applies when (c) holds. Example 1 is a game-theoretic version of the voting paradox∗ , the fact that when majority rule is used to make collective decisions, a society can find itself in a situation where, for each social choice that could be made, there is a feasible alternative that a majority of the voters prefer (and, hence, any particular choice can be overturned by a majority vote). More specifically, this example illustrates how easy it is for the paradox of voting (and the corresponding absence of equilibrium policies) to occur when the issues that are to be resolved involve the distribution of income in the society (see also VOTING PARADOX, Arrow [1] and Sen [30]). Example 2. Consider the case in which the probabilities that describe the voters’ choices
satisfy the following version of Luce’s axiom of ‘‘independence from irrelevant alternatives’’ (see CHOICE AXIOM, LUCE’S): For each i ∈ N and (ψ1 , ψ2 ) ∈ X 2 , P1i (ψ1 , ψ2 ) P2i (ψ1 , ψ2 )
=
Ui (ψ1 ) ; Ui (ψ2 )
(1)
assume also (implicit in Example 1) that each voter is going to vote; that is, for each i ∈ N and (ψ1 , ψ2 ) ∈ X 2 , P1i (ψ1 , ψ2 ) + P2i (ψ1 , ψ2 ) = 1.
(2)
Equations (1) and (2) imply that for each i ∈ N and (ψ1 , ψ2 ) ∈ X 2 , P1i (ψ1 , ψ2 ) =
Ui (ψ1 ) , Ui (ψ1 ) + Ui (ψ2 )
P2i (ψ1 , ψ2 ) =
Ui (ψ2 ) . Ui (ψ1 ) + Ui (ψ2 )
This time, the resulting game (X, X; Pl1 , Pl2 ) does have a Nash equilibrium. Since the game is zero sum, this can also be phrased as: The game has a saddle point (see GAME THEORY or NASH EQUILIBRIUM). The fact that this game has a Nash equilibrium (or equivalently, has a saddle point) can be seen quite easily. For each i ∈ N and (x, y) ∈ X 2 , P1i (x, y) = xi /(xi + yi ). Therefore, each P1i (x, y) is a concave function of x = (x1 , x2 , x3 ), and each −P1i (x, y) is a concave function of y = (y1 , y2 , y3 ). This, in turn, implies that EV1 (x, y) is a concave function of x and −EV1 (x, y) is a concave function of y. Similarly, EV2 (x, y) is a concave function of y and −EV2 (x, y) is a concave function of x. Therefore, candidate 1’s payoff function, Pl1 (x, y) = EV1 (x, y) − EV2 (x, y), is a concave function of x and candidate 2’s payoff function, Pl2 (x, y) = EV2 (x, y) − EV1 (x, y), is a concave function of y. By a similar argument, Pl1 (x, y) and Pl2 (x, y) are continuous functions of (x, y). Finally, from its definition, X is a compact, convex subset of R3 . Hence all of the assumptions in the premise of one of the theorems that has been labeled ‘‘Nash’s Theorem’’ are satisfied (see NASH EQUILIBRIUM). Therefore, there is a Nash equilibrium in this example.
PROBABILISTIC VOTING MODELS
Where is (or are) the Nash equilibrium (or equilibria) located? This question can be answered by solving the problem: Find the x ∈ X that maximize(s) U1 (x) · U2 (x) · U3 (x) = x1 · x2 · x3 over the set X. There is a unique x which solves this problem: x = ( 13 , 13 , 13 ). This implies that there is a unique Nash equilibrium in the game: ψ1 = ψ2 = ( 13 , 13 , 13 ). (See Theorem 1 in Coughlin and Nitzan [9].) GENERAL MODEL Models similar to the two examples given above have been studied in the references listed at the end of this entry. They are all special cases of the following general model: There are two candidates for a particular political office. They will be indexed by the elements in the set C = {1, 2}. There is a set of possible ‘‘locations,’’ X, for the candidates. The set of possible locations is the same for both of them. A particular x ∈ X could, for instance, specify a position on the policy issues in the election, an allocation of campaign resources, or both of these. ψc will be used to denote a particular location for c. There is an index set, N, for the individuals who can vote in the election. A particular i ∈ N could be a number (e.g., if the voters are labeled as voters 1, . . . , n), a specification of various characteristics that a voter can have (such as location of residence, income, age, etc.), a vector that specifies the ‘‘ideal’’ positions on the various policy issues for a voter, or something else. For each i ∈ N and c ∈ C, there is a function Pci : X × X → [0, 1] that assigns to each (ψ1 , ψ2 ) ∈ X × X a probability for the event ‘‘a voter randomly drawn from the individuals labeled by i ∈ N will vote for c if candidate 1 is located at ψ1 , and candidate 2 is located at ψ2 .’’ For each (ψ1 , ψ2 ) ∈ X 2 , P0i (ψ1 , ψ2 ) = 1 − P1i (ψ1 , ψ2 ) − P2i (ψ1 , ψ2 ) will be the probability for the event ‘‘a voter randomly drawn from the individuals labeled by i ∈ N will not vote (i.e., will abstain from voting), if candidate 1 is located at ψ1 and candidate 2 is located at ψ2 .’’ These can be objective
6437
probabilities or they can be subjective probabilities that are believed by both of the candidates. There is a probability distribution on N which assigns to each measurable set, B ⊆ N, a probability for the event ‘‘a voter randomly drawn from the individuals who can vote in the election has an index i ∈ B.’’ These probabilities can also be either objective probabilities or subjective probabilities that are believed by both of the candidates. Each candidate is concerned solely about his or her (a) expected vote, EVc (ψ1 , ψ2 ), (b) expected plurality, Plc (ψ1 , ψ2 ), or (c) probability of winning W c (ψ1 , ψ2 ). DETERMINISTIC VOTING MODELS An important special case for the general model given above is the deterministic voting model. This terminology comes from analyses of models in which each candidate wants to maximize his or her expected plurality (as in the examples). When [for a given i ∈ N and (ψ1 , ψ2 ) ∈ X 2 ] P1i (ψ1 , ψ2 ) = P2i (ψ1 , ψ2 ), the expected vote from the individual(s) indexed by ‘‘i’’ is split evenly between the two candidates. When this occurs (since each candidate’s objective is to maximize his or her expected plurality), the expected votes corresponding to the index i cancel each other out—and, therefore, have no effect on Pl1 (ψ1 , ψ2 ) or Pl2 (ψ1 , ψ2 ). For instance, in Example 1, if U3 (ψ1 ) = U3 (ψ2 ), then P13 (ψ1 , ψ2 ) − P23 (ψ1 , ψ2 ) = 12 − 1 2 = 0. Therefore, the expected plurality for candidate 1 when only voters 1 and 2 are counted is the same as when voters 1, 2, and 3 are counted. That is, [P11 (ψ1 , ψ2 ) + P12 (ψ1 , ψ2 )] − [P21 (ψ1 , ψ2 ) + P22 (ψ1 , ψ2 )] = [P11 (ψ1 , ψ2 ) + P12 (ψ1 , ψ2 ) + P13 (ψ1 , ψ2 )] − [P21 (ψ1 , ψ2 ) + P22 (ψ1 , ψ2 ) + P23 (ψ1 , ψ2 )]. Similarly, the expected plurality for candidate 2 when only voters 1 and 2 are counted is the same as when voters 1, 2, and 3 are counted. From the preceding observations it is thus clear that at any given (ψ1 , ψ2 ), the
6438
PROBABILISTIC VOTING MODELS
only voter indices that matter (to expected plurality maximizing candidates) are ones with P1i (ψ1 , ψ2 ) = P2i (ψ1 , ψ2 ). When, in fact, P1i (ψ1 , ψ2 ) = 1 or P2i (ψ1 , ψ2 ) = 1 at a given i ∈ N and (ψ1 , ψ2 ) ∈ X 2 , one of two things must be true: (a) there is one voter with the index i and the candidates believe that his or her decision will be completely determined once they choose the strategies ψ1 and ψ2 , respectively, or (b) there is more than one voter with the index i and the decisions made by all these voters will be completely determined (and the same) once the candidates choose the strategies ψ1 and ψ2 , respectively. Because of this, any model which satisfies the assumptions of the general voting model given above and is such that, at each (ψ1 , ψ2 ) ∈ X 2 and i ∈ N where P1i (ψ1 , ψ2 ) = P2i (ψ1 , ψ2 ), either (i) P1i (ψ1 , ψ2 ) = 1 or (ii) P2i (ψ1 , ψ2 ) = 1, is called a ‘‘deterministic voting model.’’ Example 1, for instance, is such a model, whereas Example 2 is not. The defining characteristic of a deterministic voting model can be restated as: For each (ψ1 , ψ2 ) ∈ X 2 and i ∈ N, either the expected votes corresponding to the index i cancel each other out (and, therefore, have no effect on the candidates’ expected pluralities) or the decision(s) of the voter(s) corresponding to the index i will be completely determined (and identical when candidate 1 chooses ψ1 and candidate 2 chooses ψ2 . A third way of stating this characteristic is: For each (ψ1 , ψ2 ) ∈ X 2 and i ∈ N, (a) P1i (ψ1 , ψ2 ) = 1, (b) P2i (ψ1 , ψ2 ) = 1, or (c) P1i (ψ1 , ψ2 ) = P2i (ψ1 , ψ2 ) = 12 (1 − P0i (ψ1 , ψ2 )). The deterministic voting models that have received the most attention are ones in which (1) each index corresponds to one voter, (2) each voter, i, has a utility function, Ui (x), and (3a) for each voter, i, and each (ψ1 , ψ2 ) ∈ X 2 , 1 if Ui (ψ1 ) > Ui (ψ2 ), 1 Pi (ψ1 , ψ2 ) = 12 if Ui (ψ1 ) = Ui (ψ2 ), 0 if Ui (ψ1 ) < Ui (ψ2 ), P2i (ψ1 , ψ2 ) = 1 − P1i (ψ1 , ψ2 ) (as in Example 1) or (3b) for each voter, i, and each (ψ1 , ψ2 ) ∈ X 2 1 if Ui (ψ1 ) = Ui (ψ2 ), P0i (ψ1 , ψ2 ) = 0 if Ui (ψ1 ) = Ui (ψ2 ),
P1i (ψ1 , ψ2 )
1 if Ui (ψ1 ) > Ui0 (ψ2 ), = 0 if Ui (ψ1 ) Ui (ψ2 ),
P2i (ψ1 , ψ2 ) = 1 − P1i (ψ1 , ψ2 ) − P0i (ψ1 , ψ2 ). An exception is McKelvey [26], where it is assumed that (a) for each index, α, all of the voters who are labeled by this index have the same utility function, Uα (x), and (b) for each index α and each (ψ1 , ψ2 ) ∈ X 2 , (P0α (ψ1 , ψ2 ), P1α (ψ1 , ψ2 ), P2α (ψ1 , ψ2 )) ∈ {(1, 0, 0), (0, 1, 0), (0, 0, 1)}, P1α (ψ1 , ψ2 ) = 1 ⇒ Uα (ψ1 ) > Uα (ψ2 ), P2α (ψ1 , ψ2 ) = 1 ⇒ Uα (ψ1 ) < Uα (ψ2 ). This formulation provides a deterministic voting model in which abstentions can occur more frequently than just when Uα (ψ1 ) = Uα (ψ2 ). The seminal work on deterministic voting models was done by Hotelling [21], Downs [16], Black [2], and Davis and Hinich [11,12]. Surveys of results that have been derived in these and other analyses [using either Plc (ψ1 , ψ2 ) or W c (ψ1 , ψ2 ) as the objective functions for the candidates] can be found in Plott [29], Kramer [22], and Mueller [27,28]. PROBABILISTIC VOTING MODELS Any model which satisfies the assumptions of the general voting model given above, but is not a deterministic voting model, is called a probabilistic voting model. Example 2, for instance, is such a model. The terminology reflects the fact that in any such model, there is at least one pair of strategies (ψ1 , ψ2 ) ∈ X 2 at which there is at least one index that matters (to expected plurality maximizing candidates) where the candidates’ beliefs about the voter(s) corresponding to the index are probabilistic in a nontrivial way; the random variable that describes them is nondegenerate. The defining characteristic for these models can be restated as: There is at least one (ψ1 , ψ2 ) ∈ X 2 and i ∈ N where P1i (ψ1 , ψ2 ) = P2i (ψ1 , ψ2 ) and 0 < P1i (ψ1 , ψ2 ) < 1 or 0 < P2i (ψ1 , ψ2 ) < 1 (or both).
PROBABILISTIC VOTING MODELS
Any model that satisfies all the assumptions of the general voting model given above will be a probabilistic voting model if there is an index i that corresponds to two or more voters and at least one pair of possible locations (ψ1 , ψ2 ) ∈ X 2 such that the choices of the voters corresponding to index i are completely determined when candidate 1 chooses ψ1 and candidate 2 chooses ψ2 , but (a) they will not all make the same choice, and (b) those who will vote are not split evenly between the candidates. Indeed, almost any deterministic voting model can be converted into a probabilistic voting model of this sort by appropriately regrouping the indices in the deterministic voting model into indices for the probabilistic voting model. For instance, let all three voters in Example 1 have the same index i. Then, for the strategies ψ1 = ( 31 , 13 , 13 ) and ψ2 = (0, 12 , 12 ), we have P1i (ψ1 , ψ2 ) = 13 and P2i (ψ1 , ψ2 ) = 23 . The resulting model is a probabilistic voting model. This approach is the basis for the models analyzed in McKelvey [26]. A voting model that satisfies all the assumptions of the general voting model given above can, alternatively, be a probabilistic voting model if there is at least one voter whose choice of whether to vote and/or which candidate to vote for (if he or she votes) is probabilistic in nature, as in Example 2. (See CHOICE AXIOM, LUCE’S for a discussion of theories of probabilistic choice behavior.) This approach is the basis for the models analyzed in Brams [3], Brams and Davis [4], Comaner [5], Coughlin and Nitzan [9,10], Davis and Hinich [13], Hinich [17], Hinich and Ordeshook [18], Hinich et al. [19,20], and Lake [23]. A voting model that satisfies all the assumptions given above could be a probabilistic voting model because the candidates are uncertain about the choices that voters will make and use subjective probabilities to summarize their expectations about these choices (see DECISION THEORY and UTILITY THEORY). This is the basis for the model analyzed in Ledyard [24,25]. Analyses that apply to models in which any of these three interpretations can arise have been carried out in Coughlin [7,8]. The most closely scrutinized probabilistic voting models can be grouped into three basic categories. The first consists of models
6439
in which the description of the candidates’ expectations about the voters’ decisions to vote (or abstain) are probabilistic, but the description of their expectations about the choices that the voters make between the candidates are deterministic (i.e., they believe that, for each (ψ1 , ψ2 ) ∈ X × X and i ∈ N, when candidate 1 chooses ψ1 and candidate 2 chooses ψ2 , the voters corresponding to i who vote will all vote for the same candidate and the candidate who so benefits is completely determined). This category includes the models in Davis and Hinich [13], Hinich and Ordeshook [18], Hinich et al. [19,20], and McKelvey [26]. The second category consists of models in which there are no abstentions, but the description of the candidates’ expectations about the choices that the voters will make in choosing between them is probabilistic: (a) P0i (ψ1 , ψ2 ) = 0, ∀(ψ1 , ψ2 ) ∈ X × X, ∀i ∈ N and (b) ∃i ∈ N and (ψ1 , ψ2 ) ∈ X × X, where P1i (ψ1 , ψ2 ) = 12 . This category includes Brams [3], Brams and Davis [4], Hinich [17], Coughlin and Nitzan [9,10], and Coughlin [8]. The third category consists of models in which the description of the candidates’ expectations both about voter abstentions and about the choices that the voters will make in choosing between them are probabilistic. This category includes Denzau and Kats [15], Coughlin [6,7], and Ledyard [24,25]. For further discussions, see the references.
REFERENCES 1. Arrow, K. (1963). Social Choice and Individual Values, 2nd ed. Yale University Press, New Haven, Conn. 2. Black, D. (1958). The Theory of Committees and Elections. Cambridge University Press, Cambridge, England. 3. Brams, S. (1975). Game Theory and Politics. MacMillan, London. 4. Brams, S. and Davis, M. (1982). Math. Social Sci., 3, 373–388. 5. Comaner, W. 169–178.
(1976).
J. Public
6. Coughlin, P. 427–433.
(1982).
Public
Econ.,
Choice,
5, 39,
7. Coughlin, P. (1983). Math. Social Sci., 4, 275–292.
6440
PROBABILITIES, ASSESSMENT OF
8. Coughlin, P. (1984). J. Econ. Theory, 34, 1–12. 9. Coughlin, P. and Nitzan, S. (1981). J. Public Econ., 15, 113–122. 10. Coughlin, P. and Nitzan, S. (1981). J. Econ. Theory, 24, 226–240. 11. Davis, O. and Hinich, M. (1966). In Mathematical Applications in Political Science, Vol. II, J. Bernd, ed. Southern Methodist University Press, Dallas, Tex. 12. Davis, O. and Hinich, M. Choice, 5, 59–72.
(1968).
Public
13. Davis, O. and Hinich, M. (1972). In Probability Models of Collective Decision Making, R. Niemi and H. Weisberg, eds. Charles E. Merrill, Columbus, Ohio. 14. Davis, O., DeGroot, M., and Hinich, M. (1972). Econometrica, 40, 147–157. 15. Denzau, A. and Kats, A. (1977). Rev. Econ. Stud., 44, 227–233. 16. Downs, A. (1957). An Economic Theory of Democracy. Harper, New York. 17. Hinich, M. 208–219.
(1977).
J. Econ.
Theory,
16,
18. Hinich, M. and Ordeshook, P. (1969). Public Choice, 7, 81–106. 19. Hinich, M., Ledyard, J., and Ordeshook, P. (1972). J. Econ. Theory, 4, 144–153. 20. Hinich, M., Ledyard, J., and Ordeshook, P. (1973). J. Politics, 35, 154–193. 21. Hotelling, H. (1929). Econ. J., 39, 41–57. 22. Kramer, G. (1977). In Frontiers of Quantitative Economics, Vol. III, M. Intriligator, ed. North-Holland, Amsterdam. 23. Lake, M. (1979). In Applied Game Theory, S. Brams, A. Schotter, and G. Schwodiauer, eds. ¨ Physica-Verlag, Wurzburg, Federal Republic of Germany. 24. Ledyard, J. (1981). In Essays in Contemporary Fields of Economics, G. Horwich and J. Quirk, eds. Purdue University Press, West Lafayette, Ind.
30. Sen, A. (1970). Collective Choice and Social Welfare. Holden-Day, San Francisco. See also CHOICE AXIOM, LUCE’S; DECISION THEORY; GAME THEORY; NASH EQUILIBRIUM; POLITICAL SCIENCE, STATISTICS IN; UTILITY THEORY; and VOTING PARADOX.
PETER COUGHLIN
PROBABILITIES,
ASSESSMENT
OF.
See ASSESSMENT OF PROBABILITIES
PROBABILITY AND LOGIC This term usually deals with formalized theories for probability and more generally the study of relationships between logic and probability. The metamathematical structure of probability theory comes out of the necessity to study the formal linguistic structure which goes into the description of the concepts of probability. The method is a glueing of probability and logic methods, intended to give us a better understanding of the foundations of probability∗ and statistics; it may be a new starting point for incorporating theories such as those of fuzzy sets∗ , probabilistic metric spaces, Boolean and Heyting-valued models, nonstandard analysis, topos theory, and so on. As a result we may also have a general theory of uncertainty not necessarily probabilistic in nature. We review the more mathematical aspects of the subject. The reader who is interested in foundational problems and different axiomatic theories for probability should see ref. 7. The article is divided into three sections: Assigning Probabilities to Logical Formulas, Hyperfinite Model Theory, and Final Remarks.
25. Ledyard, J. (1984). Public Choice, 44, 7– 41. 26. McKelvey, R. 815–844.
(1975).
Econometrica,
43,
27. Mueller, D. (1976). J. Econ. Lit., 14, 395– 433. 28. Mueller, D. (1979). Public Choice. Cambridge University Press, Cambridge, England. 29. Plott, C. (1971). In Frontiers of Quantitative Economics, Vol. I, M. Intriligator, ed. NorthHolland, Amsterdam.
ASSIGNING PROBABILITIES TO LOGICAL FORMULAS Historically this idea goes back to the Cambridge philosopher W. E. Johnson, who influenced both Keynes and Jeffreys in their wellknown treatises on probability [10]. Also, one can find similar ideas in Koopman’s comparative logical probability [7, pp. 183–186].
PROBABILITY AND LOGIC
Carnap was the first to study probabilities on formalized sentences [7, pp. 187–204]. His study, however, was restricted to a portion of a formalized language and to symmetric probabilities. It was Gaifman [8] who first used the full power of a first-order language and arbitrary probabilities. His work was extended considerably by Scott and Krauss [19] and others [4–6], most recently Gaifman and Snir [9]. Let L be a first-order language and let C be the set of its constants with C ⊆ T. A probability model for L(T) is an ordered pair (T, µ) where µ : S0 → [0, 1] is a probability on the quantifier-free sentences S0 (T) such that, for φ, ψ ∈ S0 (T), (i) µ(φ) = 1 if φ is provable in L and µ(φ) = µ(ψ) if φ and ψ are logically equivalent in L, (ii) µ(φ ∨ ψ) + µ(φ ∧ ψ) = µ(φ) + µ(ψ), (iii) µ(φ) = 1 − µ(ϕ). Gaifman [8] first proved the following interesting extension property. Theorem. Let (T, µ) be a probability model. Then there exists a unique probability µ∗ on the set S(T), of all closed sentences of L(T), which extends µ and satisfies the Gaifman condition: φ(t) , (G)µ∗ (υφ(υ)) = sup µ∗ F∈T (ω)
t∈F
where T (ω) is the set of all finite subsets of T. It follows that Doob’s separability condition is a special case of the (G) condition. Another important aspect of assigning probabilities to logical formulas is that it is a special case of assigning truth values in a complete Boolean algebra B. More precisely, if we take as the Boolean algebra B the quotient algebra of the σ -field A of a probability space (, A, P), modulo the σ -ideal of sets of probability zero, then assigning probabilities to logical formulas is the same as assigning truth values in B. This gives us a connection to Boolean-valued models [1,23] and finally to more general Heytingvalued models [13] and topos theory. These structures are generalizations toward covering uncertainties which are not necessarily probabilistic in nature.
6441
HYPERFINITE MODEL THEORY Hyperfinite model theory was introduced by Keisler [16] and is aimed at models that arise in probability and applied mathematics, in contrast with classical model theory which is more appropriate for problems in algebra and set theory. Hyperfinite models are appropriate for the study of the limiting behavior of finite models through the use of finite combinatorial arguments in infinite settings. Keisler’s work [16], along with Hoover’s [14,15], constitutes a novel approach to the metamathematics of probability. It is based on hyperfinite probability [21], which in turn is based on Robinson’s infinitesimal analysis. A hyperfinite model is a structure A = A, µ, ai , Sj , i ∈ I, j ∈ J, where A, µ is a hyperfinite probability space, aj ∈ A, and each Sj is an internal relation on A. The logics LωP , Lω1 P involved are similar to the ordinary logics but instead of the quantifiers ∃x and ∀x, one uses the probability quantifiers (Px > r), (Px r), where (Px > r) means that the set {x, φ(x)} has probability greater than r in the Loeb measure involved. An interpretation of a hyperfinite model can be given in terms of the so-called opinion poll model [16]. We can think of A as a very large set of people, Sj as a statement about the tuple y, and Sj (a, b) is true if the person a agrees with statement Sj about the n-tuple of people b. Thus (Px > 12 )(Py > 12 )S(x, y) means that a majority of x’s agree with S about a majority of y’s. The sentence (Py > 12 )(Px > 12 )S(x, y) means that for a majority of y’s, a majority of x’s agree with S about y. Neither of the foregoing sentences implies the other. In addition to the above logics, an internal logic L is introduced with hyperfinite truth values and three quantifiers (max), (min), and . . . dx. This internal logic is needed to get deeper model-theoretic results about the probability logic Lωp, but it is also of interest in its own right. An interesting discovery is the connection between apparently unrelated theorems from model theory and
6442
PROBABILITY AND LOGIC
probability theory. Downward L¨owenheim Skolem theorems for L are closely related to the weak law of large numbers∗ , whereas the elementary chain theorem is related to the strong law of large numbers. In the opinion-poll model an elementary submodel B of A is a sample which is good in the sense that, for any statement Sj and any n-tuple b ∈ An , the percentage of a ∈ B who agree with Sj about b is infinitely close to the percentage of a ∈ A who agree with Sj about b. In a similar way one can interpret the L¨owenheim-Skolem theorem. FINAL REMARKS In addition to the above two mainstreams of metamathematics of probability there are interesting connections between logic and probability. Hyperfinite Stochastic Analysis This is based on Robinson’s infinitesimal analysis and is a formal way of reducing continuous problems to discrete ones. For example, Brownian motion∗ is treated as a hyperfinite random walk∗ , stochastic differential equations∗ as stochastic difference equations, stochastic integrals∗ as nonstandard Stieltjes integrals, etc. For details, extensions and further references, see ref. 21. Statistical Model Theory This idea starts with Suppes [22], who studies models of (statistical) theories, models of experiments, and models of data. Then we ´ ´ have the book of Hajek and Harvanek [11], ´ and Harvanek’s work [12] toward a model theory of statistical theories. These works can be considered a starting point for a metamathematical structure of statistics. From a Bayesian viewpoint, Gaifman and Snir [9] study a language L which has also an empirical part L0 , consisting of finitely many so-called empirical predicates and/or empirical function symbols. Recursion Theory and Randomness The basic issue here is what it means for a sequence to be random. R. von Mises∗ based his axiomatic treatment of probability on the notion of ‘‘kollektiv,’’ a notion of
randomness∗ for a sequence; see ref. 7, pp. 92–100. Church [2] was the first to apply recursion theory to von Mises’ concept of a kollektiv. The names of Ville, Kolmogorov, Martin-L¨of, Knuth, Schnorr, and others [17] are connected with the efforts to develop a consistent theory of random sequences, with the help of recursion theory. The most recent developments of the notion of randomness can be found in Gaifman and Snir [9]. For details and exact references that are not mentioned here, see refs. 7, 9, 12, and 17. Fuzzy Sets, Heyting-Valued Models, and Topos Theory Examining the work of Higgs [13] and other developments of topos theory one can easily convince oneself that Heyting-valued models and topos theory constitute a formalization of the theory of fuzzy sets. The logic involved is of course intuitionistic. On the other hand some of the work on elementary topoi was conducted much in the spirit of nonstandard analysis. The elementary topoi is the appropriate frame to do nonstandard mathematics. The development of nonstandard models (not only Robinson’s infinitesimal models [1,13,21,23]), has shown that they contain the concept of ‘‘randomness’’ or more generally the concept of ‘‘uncertainty’’ as a basic element. For example, in Takeuti [23, Chap. 2] the nonstandard reals are random variables, and in Solovay [20] the concept of a ‘‘random real’’ is introduced for the adoption of Cohen’s forcing into measure-theoretic problems. Furthermore, Boolean algebras and probabilities on them represent the qualitative and quantitative aspects of ‘‘randomness’’ respectively, whereas Heyting algebras represent more general aspects of uncertainty, not necessarily probabilistic in nature, e.g., fuzzy sets [3]. It is believed that fuzzy sets∗ [3] and probabilistic metric spaces [18] can be expressed formally using Heyting-valued and Booleanvalued models. REFERENCES 1. Bell, J. L. (1977). Boolean-Valued Models. Clarendon, Oxford, England. (A standard textbook on the subject. Presupposes acquaintance with mathematical logic and axiomatic set theory.)
PROBABILITY AND MATHEMATICAL STATISTICS 2. Church, A. (1940). Bull. Amer. Math. Soc., 46, 130–135. 3. Dubois, D. and Prade, H. (1980). Fuzzy Sets and Systems: Theory and Applications. Academic Press, New York. 4. Eisele, K. Th. (1976). Booleschwertige Modelle mit Wahrscheinlichkeitsmasses und Darstellungen Stochastischer Prozesse. Ph.D. Thesis, University of Heidelberg, Germany. 5. Fenstad, J. E. (1967). In Sets, Models and Recursion Theory, J. N. Crossley, ed. NorthHolland, Amsterdam, pp. 156–172. 6. Fenstad, J. E. (1968). Synth`ese, 18, 1–23. 7. Fine, T. (1973). Theories of Probability. Academic Press, New York. (Examines the foundations of probability. Perhaps the only textbook of this sort.) 8. Gaifman, H. (1964). Israel J. Math., 2, 1–18. (Original work; deals with finitary formulas.) 9. Gaifman, H. and Snir, M. (1982). J. Symbolic Logic, 47, 495–548. (A recent landmark paper which introduces new concepts; results from a Bayesian viewpoint and summarizes and extends the works in refs. 4, 5, 16, and 14.) 10. Good, I. J. (1965). The Estimation of Probabilities. MIT Press, Cambridge, MA. ´ ´ 11. Hajek, P. and Havranek, T. (1978). Mechanizing Hypothesis Formation: Mathematical Foundations for a General Theory. Universitexte, Springer-Verlag, Heidelberg. ´ 12. Havranek, T. (1977). Synth`ese, 36, 441–458. (Technical paper, worth reading by those interested in model theory of statistical theories. Also, there are further references on this subject.) 13. Higgs, D. (1973). A Category Approach to Boolean-Valued Set Theory. University of Waterloo, preprint. (Excellent technical paper for Heyting-valued models; can be considered as a formalization of the theory of fuzzy sets.) 14. Hoover, D. N. (1978). Ann. Math. Logic, 14, 287–313. (Technical paper on probability logic extending Keisler’s work in ref. 11.) 15. Hoover, D. N. (1982). J. Symbolic Logic, 47, 605–624. (Recent extensions of refs. 11 and 13.) 16. Keisler, H. J. (1977). In Logic Colloquium 76. North-Holland, Amsterdam, pp. 5–110. (Source paper; develops hyperfinite model theory, based on hyperfinite probabilities; see ref. 16.) 17. Schnorr, C. P. (1971). Zufalligkeit ¨ und Wahrscheinlichkeit. Lecture Notes in Math., No.
218, Springer, Berlin. (Studies sequences using recursion theory.)
6443 random
18. Schweizer, B. and Sklar, A. (1983). Probabilistic Metric Spaces. North-Holland, Amsterdam. (The only introductory text to this very interesting subject.) 19. Scott, D. and Krauss, P. (1966). In Aspects of Inductive Logic, J. Hintikka and P. Suppes, eds. North-Holland, Amsterdam, pp. 219–264. (Extends ref. 7 to infinitary languages.) 20. Solovay R. M. (1970). Ann. Math., 92, 1–59. (Source paper; adopts forcing to measure theoretic problems.) 21. Stroyan, K. D. and Bayod, J. M. (1983). Foundations of Infinitesimal Stochastic Analysis. North-Holland, Amsterdam. (A graduate introduction to hyperfinite probability and infinitesimal stochastic analysis.) 22. Suppes, P. (1962). In Logic, Methodology and Philosophy of Science; Proceedings of the 1960 International Congress, E. Nagel, P. Suppes, and A. Tarski, eds. Stanford University Press, Stanford, CA, pp. 252–261. 23. Takeuti, G. (1978). Two Applications of Logic to Mathematics. Iwanami Shoten, Tokyo; Princeton University Press, Princeton, NJ. (A technical book that develops Boolean analysis.) See also AXIOMS OF PROBABILITY; BELIEF FUNCTIONS; PROBABILITY, FOUNDATIONS OF —I; FUZZY SET THEORY; NONADDITIVE PROBABILITY; and SUBJECTIVE RANDOMNESS.
CONSTANTIN A. DROSSOS
PROBABILITY AND MATHEMATICAL STATISTICS This journal is published by the Institute of Mathematics, Polish Academy of Sciences, Wroclaw University, Poland. The Editor-inChief is Kazimierz Urbanik. The main purpose of the journal is to foster development in fields of probability theory of special relevance to statistics. Volume 7 appeared in 1986. Papers may be in English or French. Most papers are in English. The editorial address is Pl. Grunwaldzki 2 / 4, 50-384 Wroclaw, Poland. The website is www.math.uni.wroc.pl/mathbank/ pms.html
6444
PROBABILITY BOUNDS, WORSLEY–HUNTER
PROBABILITY BOUNDS, WORSLEY–HUNTER This is an improved Bonferroni upper bound∗ proposed by Worsley [8] in 1982. A similar result was obtained independently by Hunter [3]. A somewhat different approach for obtaining Bonferroni-type bounds was developed by Galambos [1]. In its general formulation, the Worsley bound is given by the following result: Let events A1 , . . . , An be represented as vertices υ1 , . . . , υn of a graph∗ G, where υi and υj are joined by an edge eij if and only if the events Ai and Aj are not disjoint. Let T be a spanning tree of G. Then n n Ai Pr(Ai ) − Pr(Ai ∩ Aj ). Pr i=1
i=1
(i,j):eij ∈T
(1) The sharpest bound is obtained by finding the tree for which the second term on the right hand side of (1) is maximum. [Note that the upper Bonferroni bound simply states that n n Ai Pr(Ai ).] Pr i=1
i=1
Ai = Ap1 ∪ (Ap2 \Ap2 ) ∪ · · ·
i=1
∪ (Apn \Apn ).
(2)
Applying the Bonferroni bound to (2) we obtain (1). If T ≡ G, then the events on the right-hand side of (2) are disjoint and (1) becomes equality (by the well-known properties of the union operator ∪). As a corollary, letting T = (e12 , e23 , . . . , en−1,n ), we have n n n−1 Ai Pr(Ai ) − Pr(Ai ∩ Ai+1 ). Pr i=1
i=1
i=1
n
Pr(Ai ) − (2/n)
i=1
Pr(Ai ∩ Aj )
i<j
is a particular case of the Worsley bound, as is the more general Kounias bound [4]: n Pr Ai i=1
n i=1
Pr(Ai ) −
m
Pr(Ai ∩ Ajk ),
k=1 i∈Jk \jk
where (J1 , . . . , Jm ) is a partition of {1, . . . , n} and jk ∈ Jk , k = 1, . . . , m. See Worsley [8] for more details and applications. Also compare with the Galambos [1] approach. There are enlightening discussions of the Worsley-Hunter bound in [2], [6] and, more recently [7]. In the latter paper, Seneta presents a sharpening of the bounds, and an illuminating numerical example. REFERENCES
If T is a tree it is always possible to find a permutation p1 , . . . , pn of {1, . . . , n} so that υpi is joined to some υpj = υpi say, for some j < i, i = 2, . . . , n, and ni=1 Ai can be written as n
Kwerel’s [3] refinement of the Bonferroni bound given by n Pr Ai
i=1
1. Galambos, J. (1977). Ann. Prob., 5, 577–581. 2. Hochberg, Y. and Tamhane, A. C. (1987). Multiple Comparison Procedures, Wiley, New York. 3. Hunter, D. (1976). J. Appl. Prob., 13, 597–603. 4. Kounias, E. G. (1968). Ann. Math. Statist., 39, 2154–2158. 5. Kwerel, S. M. (1975). J. Amer. Statist. Ass., 70, 472–479. ˇ aritescu, ˇ 6. Marg E. (1986). Biom. J., 28, 937–943. 7. Seneta, E. (1988). Austral. J. Statist., 30A, 27–38. 8. Worsley, K. J. (1982). Biometrika, 69, 297–302. See also BONFERRONI INEQUALITIES AND INTERVALS and GRAPH THEORY.
PROBABILITY DENSITIES, COMPATIBILITY OF This concept was introduced by Abrahams and Thomas [1]. Let p1 (x|y) and p2 (y|x) be
PROBABILITY FORECASTING
given conditional density functions. If there exists a bivariate density a (x, y) with the given conditional densities, these densities are said to be compatible. A necessary and sufficient condition for compatibility of a pair of conditional densities over the region of the (x, y) space where these densities are nonzero is that g(x) p1 (x|y) is of the form , p2 (y|x) h(y) where g and h are nonnegative (integrable) functions satisfying
g(x)dx =
h(x) dy.
Examples of beta (yielding bivariate Dirichlet density), gamma, exponential, and Gaussian conditional densities are provided in ref. [1]. This result has been used and refined later by subsequent authors for characterization of bivariate densities in terms of their conditionals. See BIVARIATE DISTRIBUTIONS, SPECIFICATION OF.
REFERENCE 1. Abrahams, J. and Thomas, J. B. (1984). A note on the characterization of bivariate densities by conditional densities. Commun. Statist. Theory Methods, 13, 395–400.
PROBABILITY FORECASTING At its simplest, probability forecasting refers to the process of attaching a numerical probability to an uncertain event. Surprisingly, this most basic requirement for any realworld application of probability theory was for long ignored by most statisticians. The motivation for its development as a discipline in its own right has largely arisen from the needs of the field of probabilistic weather forecasting∗ , and much of the theoretical work underpinning it originally appeared in the meteorological literature. This special application also forms a convenient framework on which to hang a discussion of general principles.
6445
In many countries the weather service still issues its weather forecasts in purely verbal terms, either as a categorical prediction (‘‘It will be dry and sunny’’) or with imprecise indications of uncertainty (‘‘A chance of light showers’’). But many decisions can often only be taken rationally when the uncertain nature of the weather is admitted and quantified as precisely as possible. To meet this need, the National Weather Service in the United States has, since 1965, formulated and issued probability of precipitation (PoP) forecasts. Such a forecast is a number between 0 and 1, intended to be interpreted as the probability that measurable precipitation ( 0.01 inch) will occur during a specific period (generally 12 hours) at a particular point in the area of concern. These PoP forecasts are now popularly accepted by the American public as meaningful and informative. The forecasts themselves are usually arrived at intuitively by the weather forecaster, after taking account of all the information available. This information may include, as a decision aid, an ‘‘objective’’ probability produced by a computerized statistical analysis of weather records, but the forecast issued is usually interpreted, in the last analysis, as a personal or subjective probability∗ statement for that forecaster. Questions thus arise concerning the motivation and integrity of forecasters in forming and stating their subjective probabilities, the empirical validity of the forecasts issued, the extent of the agreement between different forecasters, and the way in which the public should use the forecasts. This article will concentrate on the first two issues. A very readable guide is that of Sta¨el von Holstein [30]. The work of Savage [27] on elicitation of subjective probabilities is particularly elegant. ENCOURAGING HONESTY [28] To motivate a forecaster F to divulge honestly his personal probability p for the uncertain event A, we can present him with a decision problem having economic consequences to him which depend on his decision∗ d and the outcome a of A (where a = 1 if A occurs, a = 0 if not). These consequences may be expressed
6446
PROBABILITY FORECASTING
by a loss function L(a, d). If L is measured on a utility∗ scale, or is small overall in relation to the forecaster’s assets, the theory of coherence∗ implies that he will choose d = dp , where dp minimizes his expected loss pL(1, d) + (1 − p)L(0, d) = L(p, d), say. Suppose now that F is asked to quote a value q ∈ [0, 1], on the understanding that this will commit him to the decision dq appropriate if q were his ‘‘true’’ probability. Then he clearly has no incentive to quote any q = p, and should definitely quote q = p if dp yields the unique minimum of L(p,d). His loss, if he quotes q and A = a, is S(a, q) = L(a, dq ), so that his expected loss, if he quotes q but believes Pr(A) = p, is S(p, q) = L(p, dq ), which is minimized if q = p. Any such function S(a, q), for which S(p, ˜ q)], where a˜ = 1 with q) (defined as E[S(a, probability p, 0 with probability 1 − p) is minimized in q when q = p, is a proper scoring rule; it is strictly proper if this minimum is unique. Such a rule can always be regarded as arising, as above, from a decision problem, for example that having d ∈ [0, 1] and L(a, d) = S(a, d). Using any such scoring rule as a loss function encourages honesty. (However, difficulties in their use arise when the forecaster’s utility is nonlinear in S, or he is motivated to maximize the probability of getting a bigger score than rival forecasters.) In practice, if the elicitor of F’s forecast intends to use it as his own in a decision problem he faces, presenting F with a ‘‘share’’ in his own business will yield a scoring rule which is likely to provide the best motivation for F to concentrate his attention on good assessment where it matters most. The formula D(p, q) = S(p, q) − S(p, p)(= L(p, dq ) − L(p, dp )) defines a directed distance of q from p associated with the decision problem or scoring rule, from which S(a, q) may be effectively recovered as D(a, q). Then D(p, p) = 0, and D(p, q) is nondecreasing as q departs from p in either direction [27]. Important strictly proper scoring rules are the quadratic Brier score [3], with S(a, q) = (a − q)2 , and the logarithmic score [12] S(1, q) = − log q, S(0, q) = − log(1 − q). The scoring rule S(a, q) = |a − q| is improper, however, having optimal qp = 1 for p > 12 , 0 for p < 12 . Murphy and Epstein [23]
consider the optimal ‘‘hedges’’ qp for other scoring rules. We can introduce an operational definition of F’s personal probability p as the solution of dp = d∗ , where d∗ is F’s chosen decision in his decision problem, and the solution is supposed existent and unique. Equivalently, we take p = q, where q is F’s quote under a strictly proper scoring rule. General coherence arguments (see especially ref. 27) imply that this solution should not depend on the specific decision problem. An early approach along these lines was that of de Finetti [7,8], based on the Brier score. Multipurpose Forecasts The users of a weather forecast face many diverse decision problems. A particularly simple one, the cost-loss problem [31,32], is that of the fruit farmer, say, who can take protective measures (decision d1 ) to guard her plants against frost (event A), at a cost of C; if she does not protect (d0 ), she faces a loss L if there is frost, 0 if not. Her expected loss when Pr(A) = p is L(p, d1 ) = C, L(p, d0 ) = pL, so that dp = d1 if p > k = C/L, dp = d0 if p < k. The associated proper scoring rule S(a, q) = L(a, dq ) is not, however, strictly proper, q being an optimal quote whenever q and p both lie on the same side of k, and thus lead to the same decision. To motivate F to meet the needs of all such decision makers, one might proceed as follows [16,17]. After F states his forecast q, a value k is generated, independently of A, from a known distribution v on [0, 1], and he is faced with the cost-loss decision problem having L = 1, C = k, in which he must use his implied optimal decision (i.e., dq = d1 if k < q, d0 if k q). His overall expected loss on following this strategy for quote q when A = a is
q
S(a, q) = 0
1
kdv(k) + a
dv(k). q
This formula must therefore yield a proper scoring rule [18], and this will be strictly proper if and only if v has the whole of [0, 1] as its support (so that p and q have to be on the same side of any k ∈ [0, 1]).
PROBABILITY FORECASTING
With no essential change, 1 we can replace S(1, q) above by S(1, q) − 0 kdv(k), yielding the standard form q kdv(k), S(0, q) =
0 1
S(1, q) =
(1 − k)dv(k).
(1)
q
It can, in fact, be shown [27–29], under weak conditions, that any proper scoring rule is equivalent to one having the form (1), where v is an arbitrary measure on [0, 1]. (No additional generality is obtained by allowing L above to vary also, as in [25].) In choosing an ‘‘all-purpose’’ scoring rule to elicit a probability in practice, it makes sense to use one whose associated v gives strongest weight to values of k regarded as most typical of the decision problems faced by users. Taking v to be the uniform distribution on [0, 1], dv(k) = dk, effectively recovers the Brier score [16]. FORECASTING CONTINUOUS AND MULTISTATE QUANTITIES Many of the ideas above extend to the more general case in which the forecaster has to assess a whole distribution Q for a quantity (or quantities) X. Again we can consider a decision problem with loss function L(x, d), define dQ to minimize EQ [L(X, d)], and so construct a proper scoring rule S(x, Q) = L(x, dQ ). One important strictly proper scoring rule is the logarithmic score S(x, Q) = − log q(x), where q(·) is the density of Q with respect to a dominating measure µ. It may be shown [1,15,27] that whenever X can take more than two values, this is essentially the only proper scoring rule which is local [i.e., depends on q(·) only through its value at the point x which materializes]. The associated distance function is D(P, Q) = p(x) log[p(x)/q(x)]dµ(x), the Kullback–Leibler information distance of Q from P. In practice one often confines attention to a finite set A = (As : s = 1, 2, . . . , k) of events
6447
determined by X, and their corresponding probability forecasts q = (qs : s = 1, 2, . . . , k). Let these be judged by a strictly proper scoring rule S(a, q) when A = a. It may be that S is defined for values of q that violate the probability axioms. However, considerations of coherence imply [27] that any elicited probabilities should obey the axioms, so that if, for example, A3 is the disjoint union of A1 and A2 , then q3 = q1 + q2 (see also ref. 14 for coherence of deduced probabilities when the loss function is given by an improper scoring rule). The kBrier score2 in the context above is now s=1 (as − qs ) and is strictly proper. It is normally applied in the case that the (As ) form a partition [e.g., wind-speed rating (A9 = gale, force 9, etc.)]. Then q is a ‘‘multistate’’ forecast. However, when, as in this example, the various states have a natural ordering, the Brier score appears somewhat unsatisfactory, since no credit is given for assessing high probabilities for states close to that materializing. This end can be accomplished by an analog of the cost-loss decision problem: A suitable loss function would have the property that if ds is the optimal decision when As occurs, then the loss in using dr instead of ds increases with the distance between r and s. Suitable random selection of such a decision problem will yield a strictly proper scoring rule which is ‘‘sensitive to distance.’’ Epstein [11] used this argument to derive his ‘‘ranked probability score.’’ EMPIRICAL VALIDITY: RELIABILITY A single noncategorical probability forecast (i.e., not 0 or 1) can never be said to have been ‘‘right’’ or ‘‘wrong.’’ But when a forecaster has issued a long string of such forecasts, it becomes possible to apply checks of external validity. Suppose that of n sequential forecasts, the ith is pi and the realized outcome of the associated event Ai (‘‘rain on day i’’) is ai (= 0 or 1). Then we can compare the overall aver age forecast probability pn = n−1 ni=1 pi with the overall relative frequency of occurrence an = n−1 ni=1 ai . If pn an , the set of forecasts may be regarded as approximately valid on an overall basis. Murphy and Epstein [22]
6448
PROBABILITY FORECASTING
refer to this property as ‘‘unbiasedness in the large.’’ A more incisive test looks at that subset of occasions i for which the forecast probability pi was at, or suitably close to, some preassigned value p∗ , and compares the observed relative frequency in this subset, a(p∗ ) say, with p∗ . If a(p∗ ) p∗ for all p∗ , the forecasts have been variously termed ‘‘unbiased in the small,’’ ‘‘reliable,’’ ‘‘valid,’’ or ‘‘well calibrated.’’ [Of course, sampling variability will lead to unavoidable departures of a(p∗ ) from p∗ , particularly if based on small numbers.] The plot of a(p) against p is the forecaster’s reliability diagram or Calibration curve. Consider, for illustration, the following probability forecasts and outcomes: Outcome, ai 0 0 1 0 1 0 1 Probability, pi 0.4 0.6 0.3 0.2 0.6 0.3 0.4
Outcome, ai 1 1 0 1 0 0 1 Probability, pi 0.5 0.6 0.2 0.6 0.4 0.3 0.5 Table 1 details the calculation of a(p∗ ) for these data. For example, of the three occasions on which a probability forecast of 0.3 was given, just one of the three associated events occurred, yielding a(0.3) = 13 . The reliability diagram for these forecasts and outcomes is plotted in Fig. 1. The diagonal line corresponds, in the absence of sampling fluctuations, to perfect reliability. With such small numbers, there can be little strong evidence against this. Figure 2 displays the reliability diagram for a sequence of 2820 12-hour PoP forecasts issued by a single weather forecaster in Chicago during the period 1972–1976. These results, which are typical of experienced forecasters, clearly indicate excellent reliability.
Figure 1. Reliability diagram corresponding to Table 1.
For assessing multistate forecasts (pi ), with pi = (pi1 , · · · , pik ), one might concentrate on reliability for each state separately, or for all states jointly (by extracting the subset for which pi p∗ ). The latter (‘‘vector reliability’’) is more common, and provides a more demanding test. Another possibility, not generally recommended, is ‘‘scalar reliability,’’ in which the whole set (pis : i = 1, . . . , n; s = 1, . . . , k) is regarded as a single sequence, thus ignoring the identity of the state being forecast. For a continuous quantity X, such as maximum temperature over 12 hours, credible interval forecasting involves specifying an interval (l, u) with the interpretation that Pr(l < X < u) is some fixed preassigned value γ (e.g., 75%). Reliability of a set of such forecasts is equivalent to requiring that a proportion γ of the intervals turn out to contain their specified quantity. If one also requires Pr(X l) = Pr(X u) = 1 − γ /2, say, vector
Table 1. Calibration Calculations Probability, p∗ 0.2 0.3 0.4 0.5 0.6
Number of Instances
Number of 1’s
Relative Frequency of 1’s, a(p∗ )
2 3 3 2 4
0 1 1 2 3
0 0.33 0.33 1 0.75
PROBABILITY FORECASTING
Figure 2. Reliability diagram of a weather forecaster. Reproduced from [24] with the permission of the authors. The number by each point is the number of occasions on which the associated forecast probability was issued.
reliability can be applied to this multistate problem. EMPIRICAL VALIDITY: RESOLUTION Even when the calibration criterion above is satisfied, the forecasts can be clearly unsatisfactory. Suppose that the weather in fact alternates—dry, wet, dry, wet, . . .. Consider two forecasters assigning precipitation probabilities. F1 assigns pi = 12 always, while F2 ’s forecast probabilities alternate 0, 1, 0, 1, . . .. Then both F1 and F2 are well calibrated; but F2 ’s forecasts are perfect, whereas those of F1 are almost useless. Now the assignment of probability forecasts to a sequence of events can be considered as comprising two separate tasks, sorting and labeling [26]. In sorting, the sequence of events is divided into disjoint subsequences, such that all the events in any given subsequence are considered (approximately) equally probable. Labeling then assigns a numerical value to the common probability in each subsequence. Reliability measures how well the labeling has been carried out but does not address the sorting aspect at all. In the example above, F1 and F2 were both
6449
completely reliable, but F2 was clearly the more successful sorter. Another forecaster F3 , who assigns alternating probabilities 1, 0, 1, 0, . . . is also sorting as well as F2 , although the reliability of these forecasts is abominable. In many ways, good sorting is more important than good labeling: F3 ’s forecasts are more valuable than those of F1 once we have obtained enough data to see how to recalibrate them. In effect, ability at sorting depends on substantive (e.g., meteorological) expertise, and ability at labeling on normative expertise (i.e., skill at expressing uncertainty) [34]. The quality of the sorting process, termed resolution, might be judged by the forecaster’s ability to create a set of subsequences in which the relative frequencies are close to 0 or 1 rather than near the overall relative frequency in the whole sequence. However, it is unreasonable, in general, to expect perfect sorting, which is equivalent to absolutely correct (or absolutely incorrect!) categorical forecasting. When, then, can we say that sorting has been carried out successfully? Curtiss [4] suggests that this has been achieved when within each subsequence of events corresponding to some fixed forecast value, the sequence of actual outcomes exhibits no pattern, but is essentially a ‘‘random sequence,’’ or collective in the sense of von Mises [33]. Informally, this means that it should not be possible to extract, from the subsequence, a sub-subsequence in which the relative frequency differs from that in the subsequence. Forecaster F1 above has clearly failed, and F2 and F3 succeeded, by this criterion. Dawid [5] introduced an extended calibration criterion. Consider a subsequence Sn of (1, 2, . . . , n), extracted arbitrarily, subject to the constraint that the decision on whether or not i is to be included in Sn should be determined only by the previous outcomes (a1 , . . . , ai−1 ), not by ai or any later outcomes. Let pn be the average forecast probability, and an the empirical relative frequency, for the events in Sn . Then we require that pn an . This is justified by showing that if the (pi ) are constructed sequentially as appropriate conditional probabilities from a joint distribution for the events (A1 , A2 , . . .), so that pi = (Ai |a1 , a2 , . . . , ai−1 ), then pn − an → 0 except on a set of -probability zero.
6450
PROBABILITY FORECASTING
Consequently, if pn and an are not close enough together, there is a suggestion that, in fact, pn − an →0, which serves to discredit ’s probability assignments. This extended calibration criterion subsumes the earlier one, since the decision to include i in Sn whenever pi is close enough to some preassigned value p∗ depends only on pi , which in turn depends only on (a1 , a2 , . . . , ai−1 ). It also allows us to extract a suitable sub-subsequence of this subsequence and still require matching of forecasts and frequencies, as demanded by Curtiss. A sequence of forecasts which satisfies this criterion for every one of a large enough collection of subsequences (e.g., all computable subsequences) is called completely calibrated. Such a sequence has both perfect reliability and maximum attainable resolution. It is shown by Dawid [6] that if (p1i ) and (p2i ) are each completely calibrated computable forecast sequences, then p1i − p2i → 0(i → ∞). Consequently, this criterion of complete calibration is strong enough to exclude all but one limiting assignment of probabilities, as being empirically invalid. Unfortunately, there is no general way of constructing the asymptotically unique valid probability assignments. In practice, one must allow departures from complete calibration ascribable to sampling variation in finite sequences, so that the criterion is not overrestrictive. Refinement. DeGroot and Fienberg [9,10] introduce a partial ordering between forecasters, which is closely related to comparative resolution. Each combination of forecast p and observed outcome a can be formally regarded as one observation from a bivariate distribution for two random variables, p˜ ˜ From a sequence of such forecasts and and a. their outcomes we can build up an empirical ˜ a): ˜ Q may be bivariate distribution Q for (p, regarded as constructed by randomly selecting one of the occasions in the sequence, with equal probability for each, and setting ˜ a) ˜ to the values (pi , ai ) associated with the (p, selected occasion. Then Q serves as a partial description of the sequence. (To be usable, this empirical distribution may need smoothing or discretisation.) Nature determines the ˜ The forecaster’s empirical distribution of a. input into Q is thus contained in the con˜ given a˜ = 0 and ditional distributions of p,
given a˜ = 1. (For good resolution, these two distributions should be very different.) We can now apply concepts from the theory of comparison of experiments [2]. Forecaster A is sufficient for, or more refined than, forecaster B if B’s conditional distribution for p˜ given a˜ can be obtained by first generating p˜ ∗ from A’s conditional distribution and then generating p˜ from some fixed conditional distribution given p˜ ∗ . In other words, B’s forecast distributions can be produced by passing those of A through a noisy channel, independently of the actual outcome. Consequently, A’s forecasts might be considered more valuable than B’s. ECONOMIC EVALUATION OF FORECASTS Another approach to assessing the validity of a sequence of forecasts p = (p1 , p2 , . . . , pn ) in the light of outcomes a = (a1 , a2 , . . . , an ) is in terms of the economic consequences which would have arisen from using these forecasts to solve a sequence of decision problems. Suppose that before each event Ai is observed, a decision di must be made, leading to loss L(ai , di ) if Ai = ai (we suppose the decision space and loss function independent of i, for simplicity). Using the forecasts above corresponds to taking di = dpi , the decision optimal under Pr(Ai ) = pi , and thus leads to loss S(ai , pi ) on the ith event, where S is the proper scoring rule associated with this decision problem. Comparisons between different forecasters might thus be based on their average scores S = n−1 ni=1 S(ai , pi ). (Since any suitably regular proper scoring rule can be regarded as having arisen from choosing a random cost-loss decision problem, forecaster A will do at least as well as forecaster B for any decision problem if he does so for every simple cost-loss problem.) The average score S can be split into components of independent interest. Thus letting E denote expectation under the (perhaps discretized or smoothed) joint empirical ˜ a) ˜ based on the observed distribution Q for (p, ˜ p˜ = forecasts and outcomes, define ρ(p) = E(a| p) = Q(a˜ = 1|p˜ = p). This is the ‘‘recalibrated’’ version of p, giving the proportion of events occurring when the forecast p is quoted. We ˜ = E(a| ˜ p). ˜ Then S can be introduce r˜ = ρ(p)
PROBABILITY FORECASTING
˜ p)]. ˜ But expressed as E[S(a,
a proportion ρj results in the event occurring. Then
˜ p)] ˜ = E[E{S(a, ˜ p)| ˜ p}] ˜ E[S(a, ˜ p), ˜ p)] ˜ = E[S(˜r, p)]. ˜ = E[S(E(a| Recalling that the directed distance D(p, q) ≡ S(p, q) − S(p, p)( 0), ˜ we obtain S = S1 + S2 , where S1 = E[D(˜r, p)] and S2 = E[S(˜r, r˜ )]. Note that S1 0, with equality (if S is strictly proper) only when ρ(p) ≡ p (almost surely under Q)—that is, when the forecasts are empirically calibrated. Consequently, S1 can be regarded as a penalty score for poor calibration. On the other hand, S2 is just the average score accruing to the recalibrated forecasts. Since S(r, r) is concave in r [27], S2 will tend to be smaller when these are concentrated near the endpoints of [0, 1]. If forecaster A is more refined than forecaster B, A will have a smaller value for S2 than B for any proper scoring rule [10]. Thus S2 may be regarded as a penalty for poor resolution. Schervish [28] points out that A will be more refined than B if and only if the recalibrated forecasts of A perform at least as well as those of B in every simple cost-loss problem. ˜ = Q(a˜ = 1) denote the Now let π = E(a) empirical relative frequency of 1’s in the outcome sequence (called the climatological probability in meteorology). Then ˜ p)] ˜ = E(a) ˜ = π . So E[S(˜r, π )] = E(˜r) = E[E(a| S(E(˜r), π ) = S(π , π ). Hence S2 = E[S(˜r, r˜ )] = E[S(˜r, π ) − D(˜r, π )] = S0 − S3 , with S0 = S(π , π ), S3 = E[D(˜r, π )] 0. Thus S = S0 + S1 − S3 . In this decomposition, S0 is the score attained by the least refined wellcalibrated forecaster, who always gives probability forecast π ; S3 measures the improvement in resolution over these constant forecasts. Again, S1 penalizes poor calibration. As an example, let S be the Brier score, so that S(a, p) = (a − p)2 , D(p, q) = (p − q)2 . Suppose that the probability forecasts are confined to a set (pj : j = 1, 2, . . . , k) of values, and that forecast pj is issued on nj occasions, of which
6451
nS =
k j=1
nj {ρj (1 − pj )2 + (1 − ρj )p2j },
S0 = π (1 − π ) (with nπ =
k j=1
nS1 = nS2 = nS3 =
nj ρj ), k j=1
k j=1
k j=1
nj (ρj − pj )2 , nj ρj (1 − ρj ) nj (ρj − π )2 .
For this case, the decomposition S = S1 + S2 was first obtained by Sanders [26], and the formula S = S0 + S1 − S3 by Murphy [21]. The former decomposition was extended, as above, to general proper scoring rules by DeGroot and Fienberg [10]. The extension of the latter decomposition appears to be new. If a completely calibrated set of forecasts q exists, these will be more refined than any other set p, and lead to the minimum achievable average loss, for any decision problem. For in any subsequence for which pi and qi take on values (approximately) equal, respectively, to some preassigned p∗ and q∗ , the proportion of events occurring will be q∗ . Thus q is sufficient for the pair (p, q) [10], so is more refined than p. The average loss on applying the p’s will be L(q∗ , dp∗ ) L(q∗ , dq∗ ), the average loss for the q’s. Extensions of the decompositions above to multistate forecasts are straightforward, but can be based on either a scalar or a vector approach [20]. Indeed, even in the two-state case, scalar partitions will differ from the foregoing essentially vector partitions [19]. The vector partitions appear more relevant in general. EMPIRICAL EVIDENCE Many studies of actual forecasting performance have been carried out, investigating reliability, average scores, related ‘‘skill scores,’’ and so on. A good review is given by Lichtenstein et al. [13]. The principal finding is that naive forecasters, or experienced
6452
PROBABILITY, FOUNDATIONS OF—I
experts departing even slightly from their usual tasks, find probability assessment difficult and perform poorly, generally showing overconfidence, with ρ(p) much closer to 12 than p. Feedback sessions, in which penalties arising from applying a proper scoring rule are revealed to the forecaster (or docked from his pay!), can sharpen forecasting ability remarkably. With such practice, U.S. weather forecasters, for example, find themselves well able to discern small differences in probability and are very well calibrated. REFERENCES 1. Bernardo, J. M. (1979). Ann. Statist., 7, 686–690. 2. Blackwell, D. (1951). Proc. 2nd Berkeley Symp. Math. Statist. Prob. University of California Press, Berkeley, Calif., pp. 93–102. 3. Brier, G. W. (1950). Monthly Weather Rev., 78, 1–3. 4. Curtiss, J. H. (1968). J. Appl. Meteorol., 7, 3–17. 5. Dawid, A. P. (1982). J. Amer. Statist. Ass., 77, 605–613. 6. Dawid, A. P. (1985). Ann. Statist., 13, 1251–1274. 7. de Finetti, B. (1962). In The Scientist Speculates, I. J. Good, ed. Heinemann, London, 357–364. 8. de Finetti, B. (1975). Theory of Probability (English transl.). 2 vols. Wiley, New York. 9. DeGroot, M. H. and Fienberg, S. E. (1982). In Statistical Decision Theory and Related Topics III, Vol. 1, S. S. Gupta and J. O. Berger, eds. Academic Press, New York, pp. 291–314. 10. DeGroot, M. H. and Fienberg, S. E. (1983). Statistician, 32, 12–22. 11. Epstein, E. S. (1969). J. Appl. Meteorol., 8, 985–987. 12. Good, I. J. (1952). J. R. Statist. Soc. B, 14, 107–114. 13. Lichtenstein, S., Fischhoff, B., and Phillips, L. D. (1982). In Judgment under Uncertainty: Heuristics and Biases, D. Kahneman, P. Slovic, and A. Tversky, eds. Cambridge University Press, Cambridge, England. 14. Lindley, D. V. (1982). Int. Statist. Rev., 50, 1–26. 15. McCarthy, J. (1956). Proc. Natl. Acad. Sci. (USA), pp. 654–655. 16. Murphy, A. H. (1966). J. Appl. Meteorol., 5, 534–537.
17. Murphy, A. H. (1969). J. Appl. Meteorol., 8, 863–873. 18. Murphy, A. H. (1969). J. Appl. Meteorol., 8, 989–991. 19. Murphy, A. H. (1972). J. Appl. Meteorol., 11, 273–282. 20. Murphy, A. H. (1972). J. Appl. Meteorol., 11, 1183–1192. 21. Murphy, A. H. (1973). J. Appl. Meteorol., 12, 595–600. 22. Murphy, A. H. and Epstein, E. S. (1967). J. Appl. Meteorol., 6, 748–755. 23. Murphy, A. H. and Epstein, E. S. (1967). J. Appl. Meteorol., 6, 1002–1004. 24. Murphy, A. H. and Winkler, R. L. (1977). Appl. Statist., 26, 41–47. 25. Pearl, J. (1978). Int. J. Man-Machine Stud., 10, 175–183. 26. Sanders, F. (1963). J. Appl. Meteorol., 2, 191–201. 27. Savage, L. J. (1971). J. Amer. Statist. Ass., 66, 783–801. 28. Schervish, M. J. (1983). A General Method for Comparing Probability Assessors. Tech. Rep. No. 275, Dept. of Statistics, Carnegie-Mellon University, Pittsburgh, Pa. 29. Shuford, E. H., Albert, A., and Massengill, H. E. (1966). Psychometrika, 31, 125–145. 30. Sta¨el von Holstein, C. A. S. (1970). Assessment and Evaluation of Subjective Probability Distributions. Economic Research Institute, Stockholm School of Economics, Stockholm. 31. Thompson, J. C. (1950). Monthly Weather Rev., 78, 113–124. 32. Thompson, J. C. (1952). Bull. Amer. Meteorol. Soc., 33, 223–226. 33. von Mises, R. (1957). Probability, Statistics and Truth. Allen & Unwin, London. 34. Winkler, R. L. and Murphy, A. H. (1968). J. Appl. Meteorol., 7, 751–758. See also METEOROLOGY, STATISTICS IN; SUBJECTIVE PROBABILITIES; WEATHER FORECASTING, BRIER SCORE IN; and WEATHER FORECASTING, EPSTEIN SCORING RULE IN.
A. P. DAWID
PROBABILITY, FOUNDATIONS OF—I Foundations of Probability As A Discipline Any account of the foundations of probability (hereinafter designated F. of P.) must
PROBABILITY, FOUNDATIONS OF—I
be read skeptically. The objectives are to understand and develop probabilistic reasoning. The scale of the effort in the F. of P. is broad, usually conceptual, and generally not concerned with the more narrowly focused technical advances in the familiar numerical probability concept. Efforts in the F. of P. are likely to be critical, expository, and/or argumentative, exhibiting a tradition inherited from the philosophy of science. The F. of P. lacks widely accepted guidelines as to important problems and acceptable solutions; this lack has been hospitable to the development of several schools of thought, each housing a variety of positions. Contributions to the F. of P. come from many disciplines, including philosophy of science, economics, mathematical probability, physics, and engineering, with philosophers of science in the modal but not the majority position. There is no single journal that is a reliable source for current literature. Some journals that occasionally publish articles on the F. of P. are: Annals of Probability∗ , Annals of Statistics∗ , British Journal of the Philosophy of Science, Journal of the Royal Statistical Society∗ (Series B), Synth`ese, Theory and Decision, and Theoria. Mathematical Reviews can be profitably consulted under 60A05. Brief Historical Background While we are likely to learn little to improve the mathematics of numerical probability by consulting the past, history is of importance to the F. of P., for it provides a perspective on the conceptual assumptions that lie at the root of our present-day notions of probability. This perspective can enlighten us as to alternative assumptions, as well as enable us to better perceive the consequences of commitments to particular probability concepts and statistical methodologies. Standard historical works include David [8], Maistrov [34], Todhunter [45], and more recently the work of Hacking [21] and Shafer [43]; these latter two works are of interest for their greater concern with key conceptual issues, although their historical accuracy has been challenged. Probability ideas, as expressed by scholars and not just as implied by gambling∗ or religious practices, can be traced back to
6453
the ancient Egyptians and Greeks and then forward through the Middle Ages (e.g., the writings of Aquinas) to the appearance of recognizably modern probability in the last half of the seventeenth century. Scholarly views up to the onset of the Renaissance held that it did not admit refined degrees and that the basis for a probability assertion was the opinions of qualified individuals. Numerical probability on a scale from 0 to 1 first appears in the seventeenth century (if we omit scattered precursors who were ignored in their time), at a time when a belief in the desirability of mathematical precision is replacing the belief in the possibility of certain knowledge about the world, the latter possibility having been called into question by Cartesian skepticism. At the close of the seventeenth century we find not only numerical probability and expectation but also Bernoulli’s law of large numbers∗ (published posthumously in 1713). The latter supplied the basis for a frequentist view of probability that largely came to supplant the subjective/epistemic view predominant hitherto (see FREQUENCY INTERPRETATION IN PROBABILITY AND STATISTICAL INFERENCE). The eighteenth century saw the introduction of conditional probability∗ and Bayes’∗ formula, great advances in the calculus and limit laws, the elimination of the belief held by Bernoulli∗ that the sum of the probability of an event and the probability of its complement need not be 1, and the beginnings of the scientific use of probability. The use of probability in the physical and social sciences grew greatly in the n