Data Analysis and Graphics Using R: An Example-based Approach (Cambridge Series in Statistical and Probabilistic Mathematics)

This page intentionally left blank

Data Analysis and Graphics Using R, Third Edition

Discover what you can do with R! Introducing the R system, covering standard regression methods, then tackling more advanced topics, this book guides users through the practical, powerful tools that the R system provides. The emphasis is on hands-on analysis, graphical display, and interpretation of data. The many worked examples, from real-world research, are accompanied by commentary on what is done and why. The companion website has code and data sets, allowing readers to reproduce all analyses, along with solutions to selected exercises and updates. Assuming basic statistical knowledge and some experience with data analysis (but not R), the book is ideal for research scientists, final-year undergraduate or graduate-level students of applied statistics, and practicing statisticians. It is both for learning and for reference. This third edition takes into account recent changes in R, including advances in graphical user interfaces (GUIs) and graphics packages. The treatments of the random forests methodology and one-way analysis have been extended. Both text and code have been revised throughout, and where possible simplified. New graphs and examples have been added. john maindonald is Visiting Fellow at the Mathematical Sciences Institute at the Australian National University. He has collaborated extensively with scientists in a wide range of application areas, from medicine and public health to population genetics, machine learning, economic history, and forensic linguistics. w. john braun is Professor in the Department of Statistical and Actuarial Sciences at the University of Western Ontario. He has collaborated with biostatisticians, biologists, psychologists, and most recently has become involved with a network of forestry researchers.

Data Analysis and Graphics Using R – an Example-Based Approach Third Edition

CAMBRIDGE SERIES IN STATISTICAL AND PROBABILISTIC MATHEMATICS Editorial Board Z. Ghahramani (Department of Engineering, University of Cambridge) R. Gill (Mathematical Institute, Leiden University) F. P. Kelly (Department of Pure Mathematics and Mathematical Statistics, University of Cambridge) B. D. Ripley (Department of Statistics, University of Oxford) S. Ross (Department of Industrial and Systems Engineering, University of Southern California) B. W. Silverman (St Peter’s College, Oxford) M. Stein (Department of Statistics, University of Chicago) This series of high quality upper-division textbooks and expository monographs covers all aspects of stochastic applicable mathematics. The topics range from pure and applied statistics to probability theory, operations research, optimization, and mathematical programming. The books contain clear presentations of new developments in the field and also of the state of the art in classical methods. While emphasizing rigorous treatment of theoretical methods, the books also contain applications and discussions of new techniques made possible by advances in computational practice. A complete list of books in the series can be found at http://www.cambridge.org/uk/series/sSeries.asp?code=CSPM Recent titles include the following: 7. 8. 9. 10. 11. 12. 13. 14. 15. 16. 17. 18. 19. 20. 21. 22. 23. 24. 25. 26. 27. 28. 29. 30.

Numerical Methods of Statistics, by John F. Monahan A User’s Guide to Measure Theoretic Probability, by David Pollard The Estimation and Tracking of Frequency, by B. G. Quinn and E. J. Hannan Data Analysis and Graphics Using R, by John Maindonald and John Braun Statistical Models, by A. C. Davison Semiparametric Regression, by David Ruppert, M. P. Wand and R. J. Carroll Exercises in Probability, by Loı¨c Chaumont and Marc Yor Statistical Analysis of Stochastic Processes in Time, by J. K. Lindsey Measure Theory and Filtering, by Lakhdar Aggoun and Robert Elliott Essentials of Statistical Inference, by G. A. Young and R. L. Smith Elements of Distribution Theory, by Thomas A. Severini Statistical Mechanics of Disordered Systems, by Anton Bovier The Coordinate-Free Approach to Linear Models, by Michael J. Wichura Random Graph Dynamics, by Rick Durrett Networks, by Peter Whittle Saddlepoint Approximations with Applications, by Ronald W. Butler Applied Asymptotics, by A. R. Brazzale, A. C. Davison and N. Reid Random Networks for Communication, by Massimo Franceschetti and Ronald Meester Design of Comparative Experiments, by R. A. Bailey Symmetry Studies, by Marlos A. G. Viana Model Selection and Model Averaging, by Gerda Claeskens and Nils Lid Hjort Bayesian Nonparametrics, edited by Nils Lid Hjort et al From Finite Sample to Asymptotic Methods in Statistics, by Pranab K. Sen, Julio M. Singer and Antonio C. Pedrosa de Lima Brownian Motion, by Peter M¨orters and Yuval Peres

Data Analysis and Graphics Using R – an Example-Based Approach Third Edition John Maindonald Mathematical Sciences Institute, Australian National University

and W. John Braun Department of Statistical and Actuarial Sciences, University of Western Ontario

CAMBRIDGE UNIVERSITY PRESS

Cambridge, New York, Melbourne, Madrid, Cape Town, Singapore, São Paulo, Delhi, Dubai, Tokyo Cambridge University Press The Edinburgh Building, Cambridge CB2 8RU, UK Published in the United States of America by Cambridge University Press, New York www.cambridge.org Information on this title: www.cambridge.org/9780521762939 © Cambridge University Press 2003 Second and third editions © John Maindonald and W. John Braun 2007, 2010 This publication is in copyright. Subject to statutory exception and to the provision of relevant collective licensing agreements, no reproduction of any part may take place without the written permission of Cambridge University Press. First published in print format 2010 ISBN-13

978-0-511-71286-9

eBook (NetLibrary)

ISBN-13

978-0-521-76293-9

Hardback

Cambridge University Press has no responsibility for the persistence or accuracy of urls for external or third-party internet websites referred to in this publication, and does not guarantee that any content on such websites is, or will remain, accurate or appropriate.

For Edward, Amelia and Luke also Shireen, Peter, Lorraine, Evan and Winifred For Susan, Matthew and Phillip

Contents

Preface Content – how the chapters fit together 1

A brief introduction to R 1.1 An overview of R 1.1.1 A short R session 1.1.2 The uses of R 1.1.3 Online help 1.1.4 Input of data from a file 1.1.5 R packages 1.1.6 Further steps in learning R 1.2 Vectors, factors, and univariate time series 1.2.1 Vectors 1.2.2 Concatenation – joining vector objects 1.2.3 The use of relational operators to compare vector elements 1.2.4 The use of square brackets to extract subsets of vectors 1.2.5 Patterned data 1.2.6 Missing values 1.2.7 Factors 1.2.8 Time series 1.3 Data frames and matrices 1.3.1 Accessing the columns of data frames – with() and attach() 1.3.2 Aggregation, stacking, and unstacking 1.3.3∗ Data frames and matrices 1.4 Functions, operators, and loops 1.4.1 Common useful built-in functions 1.4.2 Generic functions, and the class of an object 1.4.3 User-written functions 1.4.4 if Statements 1.4.5 Selection and matching 1.4.6 Functions for working with missing values 1.4.7∗ Looping

page xix xxv

1 1 1 6 7 8 9 9 10 10 10 11 11 11 12 13 14 14 17 17 18 19 19 21 22 23 23 24 24

x

Contents

1.5

1.6 1.7 1.8 1.9

Graphics in R 1.5.1 The function plot( ) and allied functions 1.5.2 The use of color 1.5.3 The importance of aspect ratio 1.5.4 Dimensions and other settings for graphics devices 1.5.5 The plotting of expressions and mathematical symbols 1.5.6 Identification and location on the figure region 1.5.7 Plot methods for objects other than vectors 1.5.8 Lattice (trellis) graphics 1.5.9 Good and bad graphs 1.5.10 Further information on graphics Additional points on the use of R Recap Further reading Exercises

25 25 27 28 28 29 29 30 30 32 33 33 35 36 37

2 Styles of data analysis 2.1 Revealing views of the data 2.1.1 Views of a single sample 2.1.2 Patterns in univariate time series 2.1.3 Patterns in bivariate data 2.1.4 Patterns in grouped data – lengths of cuckoo eggs 2.1.5∗ Multiple variables and times 2.1.6 Scatterplots, broken down by multiple factors 2.1.7 What to look for in plots 2.2 Data summary 2.2.1 Counts 2.2.2 Summaries of information from data frames 2.2.3 Standard deviation and inter-quartile range 2.2.4 Correlation 2.3 Statistical analysis questions, aims, and strategies 2.3.1 How relevant and how reliable are the data? 2.3.2 How will results be used? 2.3.3 Formal and informal assessments 2.3.4 Statistical analysis strategies 2.3.5 Planning the formal analysis 2.3.6 Changes to the intended plan of analysis 2.4 Recap 2.5 Further reading 2.6 Exercises

43 43 44 47 49 52 53 56 58 59 59 63 65 67 69 70 70 71 72 72 73 73 74 74

3 Statistical models 3.1 Statistical models 3.1.1 Incorporation of an error or noise component 3.1.2 Fitting models – the model formula

77 77 78 80

Contents

3.2

3.3

3.4

3.5 3.6 3.7 4

Distributions: models for the random component 3.2.1 Discrete distributions – models for counts 3.2.2 Continuous distributions Simulation of random numbers and random samples 3.3.1 Sampling from the normal and other continuous distributions 3.3.2 Simulation of regression data 3.3.3 Simulation of the sampling distribution of the mean 3.3.4 Sampling from finite populations Model assumptions 3.4.1 Random sampling assumptions – independence 3.4.2 Checks for normality 3.4.3 Checking other model assumptions 3.4.4 Are non-parametric methods the answer? 3.4.5 Why models matter – adding across contingency tables Recap Further reading Exercises

A review of inference concepts 4.1 Basic concepts of estimation 4.1.1 Population parameters and sample statistics 4.1.2 Sampling distributions 4.1.3 Assessing accuracy – the standard error 4.1.4 The standard error for the difference of means 4.1.5∗ The standard error of the median 4.1.6 The sampling distribution of the t-statistic 4.2 Confidence intervals and tests of hypotheses 4.2.1 A summary of one- and two-sample calculations 4.2.2 Confidence intervals and tests for proportions 4.2.3 Confidence intervals for the correlation 4.2.4 Confidence intervals versus hypothesis tests 4.3 Contingency tables 4.3.1 Rare and endangered plant species 4.3.2 Additional notes 4.4 One-way unstructured comparisons 4.4.1 Multiple comparisons 4.4.2 Data with a two-way structure, i.e., two factors 4.4.3 Presentation issues 4.5 Response curves 4.6 Data with a nested variation structure 4.6.1 Degrees of freedom considerations 4.6.2 General multi-way analysis of variance designs 4.7 Resampling methods for standard errors, tests, and confidence intervals 4.7.1 The one-sample permutation test 4.7.2 The two-sample permutation test

xi

81 82 84 86 87 88 88 90 91 91 92 95 95 96 97 98 98 102 102 102 102 103 103 104 105 106 109 112 113 113 114 116 119 119 122 123 124 125 126 127 127 128 128 129

xii

Contents

4.7.3∗ Estimating the standard error of the median: bootstrapping 4.7.4 Bootstrap estimates of confidence intervals 4.8∗ Theories of inference 4.8.1 Maximum likelihood estimation 4.8.2 Bayesian estimation 4.8.3 If there is strong prior information, use it! 4.9 Recap 4.10 Further reading 4.11 Exercises

130 131 132 133 133 135 135 136 137

5 Regression with a single predictor 5.1 Fitting a line to data 5.1.1 Summary information – lawn roller example 5.1.2 Residual plots 5.1.3 Iron slag example: is there a pattern in the residuals? 5.1.4 The analysis of variance table 5.2 Outliers, influence, and robust regression 5.3 Standard errors and confidence intervals 5.3.1 Confidence intervals and tests for the slope 5.3.2 SEs and confidence intervals for predicted values 5.3.3∗ Implications for design 5.4 Assessing predictive accuracy 5.4.1 Training/test sets and cross-validation 5.4.2 Cross-validation – an example 5.4.3∗ Bootstrapping 5.5 Regression versus qualitative anova comparisons – issues of power 5.6 Logarithmic and other transformations 5.6.1∗ A note on power transformations 5.6.2 Size and shape data – allometric growth 5.7 There are two regression lines! 5.8 The model matrix in regression 5.9∗ Bayesian regression estimation using the MCMCpack package 5.10 Recap 5.11 Methodological references 5.12 Exercises

142 142 143 143 145 147 147 149 150 150 151 152 153 153 155 158 160 160 161 162 163 165 166 167 167

6 Multiple linear regression 6.1 Basic ideas: a book weight example 6.1.1 Omission of the intercept term 6.1.2 Diagnostic plots 6.2 The interpretation of model coefficients 6.2.1 Times for Northern Irish hill races 6.2.2 Plots that show the contribution of individual terms 6.2.3 Mouse brain weight example 6.2.4 Book dimensions, density, and book weight

170 170 172 173 174 174 177 179 181

Contents

6.3

6.4

6.5 6.6

6.7 6.8

6.9 6.10 6.11 7

Multiple regression assumptions, diagnostics, and efficacy measures 6.3.1 Outliers, leverage, influence, and Cook’s distance 6.3.2 Assessment and comparison of regression models 6.3.3 How accurately does the equation predict? A strategy for fitting multiple regression models 6.4.1 Suggested steps 6.4.2 Diagnostic checks 6.4.3 An example – Scottish hill race data Problems with many explanatory variables 6.5.1 Variable selection issues Multicollinearity 6.6.1 The variance inflation factor 6.6.2 Remedies for multicollinearity Errors in x Multiple regression models – additional points 6.8.1 Confusion between explanatory and response variables 6.8.2 Missing explanatory variables 6.8.3∗ The use of transformations 6.8.4∗ Non-linear methods – an alternative to transformation? Recap Further reading Exercises

Exploiting the linear model framework 7.1 Levels of a factor – using indicator variables 7.1.1 Example – sugar weight 7.1.2 Different choices for the model matrix when there are factors 7.2 Block designs and balanced incomplete block designs 7.2.1 Analysis of the rice data, allowing for block effects 7.2.2 A balanced incomplete block design 7.3 Fitting multiple lines 7.4 Polynomial regression 7.4.1 Issues in the choice of model 7.5∗ Methods for passing smooth curves through data 7.5.1 Scatterplot smoothing – regression splines 7.5.2∗ Roughness penalty methods and generalized additive models 7.5.3 Distributional assumptions for automatic choice of roughness penalty 7.5.4 Other smoothing methods 7.6 Smoothing with multiple explanatory variables 7.6.1 An additive model with two smooth terms 7.6.2∗ A smooth surface 7.7 Further reading 7.8 Exercises

xiii

183 183 186 187 189 190 191 191 196 197 199 201 203 203 208 208 208 210 210 212 212 214 217 217 217 220 222 222 223 224 228 229 231 232 235 236 236 238 238 240 240 240

xiv

Contents

8 Generalized linear models and survival analysis 8.1 Generalized linear models 8.1.1 Transformation of the expected value on the left 8.1.2 Noise terms need not be normal 8.1.3 Log odds in contingency tables 8.1.4 Logistic regression with a continuous explanatory variable 8.2 Logistic multiple regression 8.2.1 Selection of model terms, and fitting the model 8.2.2 Fitted values 8.2.3 A plot of contributions of explanatory variables 8.2.4 Cross-validation estimates of predictive accuracy 8.3 Logistic models for categorical data – an example 8.4 Poisson and quasi-Poisson regression 8.4.1 Data on aberrant crypt foci 8.4.2 Moth habitat example 8.5 Additional notes on generalized linear models 8.5.1∗ Residuals, and estimating the dispersion 8.5.2 Standard errors and z- or t-statistics for binomial models 8.5.3 Leverage for binomial models 8.6 Models with an ordered categorical or categorical response 8.6.1 Ordinal regression models 8.6.2∗ Loglinear models 8.7 Survival analysis 8.7.1 Analysis of the Aids2 data 8.7.2 Right-censoring prior to the termination of the study 8.7.3 The survival curve for male homosexuals 8.7.4 Hazard rates 8.7.5 The Cox proportional hazards model 8.8 Transformations for count data 8.9 Further reading 8.10 Exercises

244 244 244 245 245

9 Time series models 9.1 Time series – some basic ideas 9.1.1 Preliminary graphical explorations 9.1.2 The autocorrelation and partial autocorrelation function 9.1.3 Autoregressive models 9.1.4∗ Autoregressive moving average models – theory 9.1.5 Automatic model selection? 9.1.6 A time series forecast 9.2∗ Regression modeling with ARIMA errors 9.3∗ Non-linear time series 9.4 Further reading 9.5 Exercises

283 283 283 284 285 287 288 289 291 298 300 301

246 249 252 254 255 255 256 258 258 261 266 266 267 268 268 269 272 272 273 275 276 276 277 279 280 281

Contents

10

11

xv

Multi-level models and repeated measures 10.1 A one-way random effects model 10.1.1 Analysis with aov() 10.1.2 A more formal approach 10.1.3 Analysis using lmer() 10.2 Survey data, with clustering 10.2.1 Alternative models 10.2.2 Instructive, though faulty, analyses 10.2.3 Predictive accuracy 10.3 A multi-level experimental design 10.3.1 The anova table 10.3.2 Expected values of mean squares 10.3.3∗ The analysis of variance sums of squares breakdown 10.3.4 The variance components 10.3.5 The mixed model analysis 10.3.6 Predictive accuracy 10.4 Within- and between-subject effects 10.4.1 Model selection 10.4.2 Estimates of model parameters 10.5 A generalized linear mixed model 10.6 Repeated measures in time 10.6.1 Example – random variation between profiles 10.6.2 Orthodontic measurements on children 10.7 Further notes on multi-level and other models with correlated errors 10.7.1 Different sources of variance – complication or focus of interest? 10.7.2 Predictions from models with a complex error structure 10.7.3 An historical perspective on multi-level models 10.7.4 Meta-analysis 10.7.5 Functional data analysis 10.7.6 Error structure in explanatory variables 10.8 Recap 10.9 Further reading 10.10 Exercises

303 304 305 308 310 313 313 318 319 319 321 322 323 325 326 328 329 329 331 332 334 336 340 344

Tree-based classification and regression 11.1 The uses of tree-based methods 11.1.1 Problems for which tree-based regression may be used 11.2 Detecting email spam – an example 11.2.1 Choosing the number of splits 11.3 Terminology and methodology 11.3.1 Choosing the split – regression trees 11.3.2 Within and between sums of squares 11.3.3 Choosing the split – classification trees 11.3.4 Tree-based regression versus loess regression smoothing

351 352 352 353 356 356 357 357 358 359

344 345 345 347 347 347 347 348 349

xvi

Contents

11.4

11.5

11.6 11.7 11.8 11.9 11.10

Predictive accuracy and the cost–complexity trade-off 11.4.1 Cross-validation 11.4.2 The cost–complexity parameter 11.4.3 Prediction error versus tree size Data for female heart attack patients 11.5.1 The one-standard-deviation rule 11.5.2 Printed information on each split Detecting email spam – the optimal tree The randomForest package Additional notes on tree-based methods Further reading and extensions Exercises

361 361 362 363 363 365 366 366 369 372 373 374

12

Multivariate data exploration and discrimination 12.1 Multivariate exploratory data analysis 12.1.1 Scatterplot matrices 12.1.2 Principal components analysis 12.1.3 Multi-dimensional scaling 12.2 Discriminant analysis 12.2.1 Example – plant architecture 12.2.2 Logistic discriminant analysis 12.2.3 Linear discriminant analysis 12.2.4 An example with more than two groups 12.3∗ High-dimensional data, classification, and plots 12.3.1 Classifications and associated graphs 12.3.2 Flawed graphs 12.3.3 Accuracies and scores for test data 12.3.4 Graphs derived from the cross-validation process 12.4 Further reading 12.5 Exercises

377 378 378 379 383 385 386 387 388 390 392 394 394 398 404 406 407

13

Regression on principal component or discriminant scores 13.1 Principal component scores in regression 13.2∗ Propensity scores in regression comparisons – labor training data 13.2.1 Regression comparisons 13.2.2 A strategy that uses propensity scores 13.3 Further reading 13.4 Exercises

410 410 414 417 419 426 426

14

The R system – additional topics 14.1 Graphical user interfaces to R 14.1.1 The R Commander’s interface – a guide to getting started 14.1.2 The rattle GUI 14.1.3 The creation of simple GUIs – the fgui package 14.2 Working directories, workspaces, and the search list

427 427 428 429 429 430

14.3

14.4

14.5

14.6 14.7 14.8∗

14.9

14.10

14.11

14.12∗ 14.13 14.14 14.15

Contents

xvii

14.2.1∗ The search path 14.2.2 Workspace management 14.2.3 Utility functions R system configuration 14.3.1 The R Windows installation directory tree 14.3.2 The library directories 14.3.3 The startup mechanism Data input and output 14.4.1 Input of data 14.4.2 Data output 14.4.3 Database connections Functions and operators – some further details 14.5.1 Function arguments 14.5.2 Character string and vector functions 14.5.3 Anonymous functions 14.5.4 Functions for working with dates (and times) 14.5.5 Creating groups 14.5.6 Logical operators Factors Missing values Matrices and arrays 14.8.1 Matrix arithmetic 14.8.2 Outer products 14.8.3 Arrays Manipulations with lists, data frames, matrices, and time series 14.9.1 Lists – an extension of the notion of “vector” 14.9.2 Changing the shape of data frames (or matrices) 14.9.3∗ Merging data frames – merge() 14.9.4 Joining data frames, matrices, and vectors – cbind() 14.9.5 The apply family of functions 14.9.6 Splitting vectors and data frames into lists – split() 14.9.7 Multivariate time series Classes and methods 14.10.1 Printing and summarizing model objects 14.10.2 Extracting information from model objects 14.10.3 S4 classes and methods Manipulation of language constructs 14.11.1 Model and graphics formulae 14.11.2 The use of a list to pass arguments 14.11.3 Expressions 14.11.4 Environments 14.11.5 Function environments and lazy evaluation Creation of R packages Document preparation – Sweave() and xtable() Further reading Exercises

430 430 431 432 432 433 433 433 434 437 438 438 439 440 441 441 443 443 444 446 448 450 451 451 452 452 454 455 455 456 457 458 458 459 460 460 461 461 462 463 463 464 465 467 468 469

xviii

15

Contents

Graphs in R 15.1 Hardcopy graphics devices 15.2 Plotting characters, symbols, line types, and colors 15.3 Formatting and plotting of text and equations 15.3.1 Symbolic substitution of symbols in an expression 15.3.2 Plotting expressions in parallel 15.4 Multiple graphs on a single graphics page 15.5 Lattice graphics and the grid package 15.5.1 Groups within data, and/or columns in parallel 15.5.2 Lattice parameter settings 15.5.3 Panel functions, strip functions, strip labels, and other annotation 15.5.4 Interaction with lattice (and other) plots – the playwith package 15.5.5 Interaction with lattice plots – focus, interact, unfocus 15.5.6 Overlaid plots with different scales 15.6 An implementation of Wilkinson’s Grammar of Graphics 15.7 Dynamic graphics – the rgl and rggobi packages 15.8 Further reading

472 472 472 474 475 475 476 477 478 480 483 485 485 486 487 491 492

Epilogue

493

References

495

Index of R symbols and functions

507

Index of terms

514

Index of authors

523

The color plates will be found between pages 328 and 329.

Preface

This book is an exposition of statistical methodology that focuses on ideas and concepts, and makes extensive use of graphical presentation. It avoids, as much as possible, the use of mathematical symbolism. It is particularly aimed at scientists who wish to do statistical analyses on their own data, preferably with reference as necessary to professional statistical advice. It is intended to complement more mathematically oriented accounts of statistical methodology. It may be used to give students with a more specialist statistical interest exposure to practical data analysis. While no prior knowledge of specific statistical methods or theory is assumed, there is a demand that readers bring with them, or quickly acquire, some modest level of statistical sophistication. Readers should have some prior exposure to statistical methodology, some prior experience of working with real data, and be comfortable with the typing of analysis commands into the computer console. Some prior familiarity with regression and with analysis of variance will be helpful. We cover a range of topics that are important for many different areas of statistical application. As is inevitable in a book that has this broad focus, there will be investigators working in specific areas – perhaps epidemiology, or psychology, or sociology, or ecology – who will regret the omission of some methodologies that they find important. We comment extensively on analysis results, noting inferences that seem well-founded, and noting limitations on inferences that can be drawn. We emphasize the use of graphs for gaining insight into data – in advance of any formal analysis, for understanding the analysis, and for presenting analysis results. The data sets that we use as a vehicle for demonstrating statistical methodology have been generated by researchers in many different fields, and have in many cases featured in published papers. As far as possible, our account of statistical methodology comes from the coalface, where the quirks of real data must be faced and addressed. Features that may challenge the novice data analyst have been retained. The diversity of examples has benefits, even for those whose interest is in a specific application area. Ideas and applications that are useful in one area often find use elsewhere, even to the extent of stimulating new lines of investigation. We hope that our book will stimulate such cross-fertilization. To summarize: The strengths of this book include the directness of its encounter with research data, its advice on practical data analysis issues, careful critiques of analysis results, the use of modern data analysis tools and approaches, the use of simulation and other computer-intensive methods – where these provide insight or give results that are not otherwise available, attention to graphical and other presentation issues, the use of

xx

Preface

examples drawn from across the range of statistical applications, and the inclusion of code that reproduces analyses. A substantial part of the book was derived, initially, from John Maindonald’s lecture notes of courses for researchers, at the University of Newcastle (Australia) over 1996– 1997 and at The Australian National University over 1998–2001. Both of us have worked extensively over the material in these chapters. The R system

We use the R system for computations. It began in the early 1990s as a project of Ross Ihaka and Robert Gentleman, who were both at the time working at the University of Auckland (New Zealand). The R system implements a dialect of the influential S language, developed at AT&T Bell Laboratories by Rick Becker, John Chambers, and Allan Wilks, which is the basis for the commercial S-PLUS system. It follows S in its close linkage between data analysis and graphics. Versions of R are available, at no charge, for 32-bit versions of Microsoft Windows, for Linux and other Unix systems, and for the Macintosh. It is available through the Comprehensive R Archive Network (CRAN). Go to http://cran.r-project.org/, and find the nearest mirror site. The development model used for R has proved highly effective in marshalling high levels of computing expertise for continuing improvement, for identifying and fixing bugs, and for responding quickly to the evolving needs and interests of the statistical community. Oversight of “base R” is handled by the R Core Team, whose members are widely drawn internationally. Use is made of code, bug fixes, and documentation from the wider R user community. Especially important are the large number of packages that supplement base R, and that anyone is free to contribute. Once installed, these attach seamlessly into the base system. Many of the analyses offered by R’s packages were not, 20 years ago, available in any of the standard statistical packages. What did data analysts do before we had such packages? Basically, they adapted more simplistic (but not necessarily simpler) analyses as best they could. Those whose skills were unequal to the task did unsatisfactory analyses. Those with more adequate skills carried out analyses that, even if not elegant and insightful by current standards, were often adequate. Tools such as are available in R have reduced the need for the adaptations that were formerly necessary. We can often do analyses that better reflect the underlying science. There have been challenging and exciting changes from the methodology that was typically encountered in statistics courses 15 or 20 years ago. In the ongoing development of R, priorities have been: the provision of good data manipulation abilities; flexible and high-quality graphics; the provision of data analysis methods that are both insightful and adequate for the whole range of application area demands; seamless integration of the different components of R; and the provision of interfaces to other systems (editors, databases, the web, etc.) that R users may require. Ease of use is important, but not at the expense of power, flexibility, and checks against answers that are potentially misleading. Depending on the user’s level of skill with R, there will be some tasks where another system may seem simpler to use. Note however the availability of interfaces, notably John Fox’s Rcmdr, that give a graphical user interface (GUI) to a limited part of R. Such

Preface

xxi

interfaces will develop and improve as time progresses. They may in due course, for many users, be the preferred means of access to R. Be aware that the demand for simple tools will commonly place limitations on the tasks that can, without professional assistance, be satisfactorily undertaken. Primarily, R is designed for scientific computing and for graphics. Among the packages that have been added are many that are not obviously statistical – for drawing and coloring maps, for map projections, for plotting data collected by balloon-borne weather instruments, for creating color palettes, for working with bitmap images, for solving sudoko puzzles, for creating magic squares, for reading and handling shapefiles, for solving ordinary differential equations, for processing various types of genomic data, and so on. Check through the list of R packages that can be found on any of the CRAN sites, and you may be surprised at what you find! The citation for John Chambers’ 1998 Association for Computing Machinery Software award stated that S has “forever altered how people analyze, visualize and manipulate data.” The R project enlarges on the ideas and insights that generated the S language. We are grateful to the R Core Team, and to the creators of the various R packages, for bringing into being the R system – this marvellous tool for scientific and statistical computing, and for graphical presentation. We give a list at the end of the reference section that cites the authors and compilers of packages that have been used in this book. Influences on the modern practice of statistics The development of statistics has been motivated by the demands of scientists for a methodology that will extract patterns from their data. The methodology has developed in a synergy with the relevant supporting mathematical theory and, more recently, with computing. This has led to methodologies and supporting theory that are a radical departure from the methodologies of the pre-computer era. Statistics is a young discipline. Only in the 1920s and 1930s did the modern framework of statistical theory, including ideas of hypothesis testing and estimation, begin to take shape. Different areas of statistical application have taken these ideas up in different ways, some of them starting their own separate streams of statistical tradition. See, for example, the comments in Gigerenzer et al. (1989) on the manner in which differences of historical development have influenced practice in different research areas. Separation from the statistical mainstream, and an emphasis on “black-box” approaches, have contributed to a widespread exaggerated emphasis on tests of hypotheses, to a neglect of pattern, to the policy of some journal editors of publishing only those studies that show a statistically significant effect, and to an undue focus on the individual study. Anyone who joins the R community can expect to witness, and/or engage in, lively debate that addresses these and related issues. Such debate can help ensure that the demands of scientific rationality do in due course win out over influences from accidents of historical development. New computing tools We have drawn attention to advances in statistical computing methodology. These have made possible the development of new powerful tools for exploratory analysis of regression

xxii

Preface

data, for choosing between alternative models, for diagnostic checks, for handling nonlinearity, for assessing the predictive power of models, and for graphical presentation. In addition, we have new computing tools that make it straightforward to move data between different systems, to keep a record of calculations, to retrace or adapt earlier calculations, and to edit output and graphics into a form that can be incorporated into published documents. New traditions of data analysis have developed – data mining, machine learning, and analytics. These emphasize new types of data, new data analysis demands, new data analysis tools, and data sets that may be of unprecedented size. Textual data and image data offer interesting new challenges for data analysis. The traditional concerns of professional data analysts remain as important as ever. Size of data set is not a guarantee of quality and of relevance to issues that are under investigation. It does not guarantee that the source population has been adequately sampled, or that the results will generalize as required to the target population. The best any analysis can do is to highlight the information in the data. No amount of statistical or computing technology can be a substitute for good design of data collection, for understanding the context in which data are to be interpreted, or for skill in the use of statistical analysis methodology. Statistical software systems are one of several components of effective data analysis. The questions that statistical analysis is designed to answer can often be stated simply. This may encourage the layperson to believe that the answers are similarly simple. Often, they are not. Be prepared for unexpected subtleties. Effective statistical analysis requires appropriate skills, beyond those gained from taking one or two undergraduate courses in statistics. There is no good substitute for professional training in modern tools for data analysis, and experience in using those tools with a wide range of data sets. Noone should be embarrassed that they have difficulty with analyses that involve ideas that professional statisticians may take 7 or 8 years of professional training and experience to master.

Third edition changes and additions The second edition added new material on survival analysis, random coefficient models, the handling of high-dimensional data, and extended the account of regression methods. This third edition has a more adequate account of errors in predictor variables, extends the treatment and use of random forests, and adds a brief account of generalized linear mixed models. The treatment of one-way analysis of variance, and a major part of the chapter on regression, have been rewritten. Two areas of especially rapid advance have been graphical user interfaces (GUIs), and graphics. There are now brief introductions to two popular GUIs for R – the R Commander (Rcmdr) and rattle. The sections on graphics have been substantially extended. There is a brief account of the latticist and associated playwith GUIs for interfacing with R graphics. Code has again been extensively revised, simplifying it wherever possible. There are changes to some graphs, and new graphs have been added.

Preface

xxiii

Acknowledgments Many different people have helped with this project. Winfried Theis (University of Dortmund, Germany) and Detlef Steuer (University of the Federal Armed Forces, Hamburg, Germany) helped with technical LATEX issues, with a cvs archive for manuscript files, and with helpful comments. Lynne Billard (University of Georgia, USA), Murray Jorgensen (University of Waikato, NZ), and Berwin Turlach (University of Western Australia) gave highly useful comment on the manuscript. Susan Wilson (Australian National University) gave welcome encouragement. Duncan Murdoch (University of Western Ontario) helped with technical advice. Cath Lawrence (Australian National University) wrote a Python program that allowed us to extract the R code from our LATEX files; this has now at length become an R function. For the second edition, Brian Ripley (University of Oxford) made extensive comments on the manuscript, leading to important corrections and improvements. We are most grateful to him, and to others who have offered comments. Alan Welsh (Australian National University) has helped work through points where it has seemed difficult to get the emphasis right. Once again, Duncan Murdoch has given much useful technical advice. Others who made helpful comments and/or pointed out errors include Jeff Wood (Australian National University), Nader Tajvidi (University of Lund), Paul Murrell (University of Auckland, on Chapter 15), Graham Williams (http://www.togaware.com, on Chapter 1), and Yang Yang (University of Western Ontario, on Chapter 10). Comment that has contributed to this edition has come from Ray Balise (Stanford School of Medicine), Wenqing He and Lengyi Han (University of Western Ontario), Paul Murrell, Andrew Robinson (University of Melbourne, on Chapter 10), Phil Kokic (Australian National University, on Chapter 9), and Rob Hyndman (Monash University, on Chapter 9). Readers who have made relatively extensive comments include Bob Green (Queensland Health) and Zander Smith (SwissRe). Additionally, discussions on the R-help and R-devel email lists have been an important source of insight and understanding. The failings that remain are, naturally, our responsibility. A strength of this book is the extent to which it has drawn on data from many different sources. Following the references is a list of data sources (individuals and/or organizations) that we wish to thank and acknowledge. We are grateful to those who have allowed us to use their data. At least these data will not, as often happens once data have become the basis for a published paper, gather dust in a long-forgotten folder! We are grateful, also, to the many researchers who, in their discussions with us, have helped stimulate our thinking and understanding. We apologize if there is anyone that we have inadvertently failed to acknowledge. Diana Gillooly of Cambridge University Press, taking over from David Tranah for the second and third editions, has been a marvellous source of advice and encouragement.

Conventions Text that is R code, or output from R, is printed in a verbatim text style. For example, in Chapter 1 we will enter data into an R object that we call austpop. We will use the

xxiv

Preface

plot() function to plot these data. The names of R packages, including our own DAAG package, are printed in italics. Starred exercises and sections identify more technical items that can be skipped at a first reading. Solutions to exercises

Solutions to selected exercises, R scripts that have all the code from the book, and other supplementary materials are available via the link given at http://www.maths.anu. edu.au/˜johnm/r-book

Content – how the chapters fit together

Chapter 1 is a brief introduction to R. Readers who are new to R should as a minimum study Section 1.1, or an equivalent, before moving on to later chapters. In later study, refer back as needed to Chapter 1, or forward to Chapter 14. Chapters 2–4: Exploratory data analysis and review of elementary statistical ideas Chapters 2–4 cover, at greater depth and from a more advanced perspective, topics that are common in introductory courses. Different readers will use these chapters differently, depending on their statistical preparedness. Chapter 2 (Styles of data analysis) places data analysis in the wider context of the research study, commenting on some of the types of graphs that may help answer questions that are commonly of interest and that will be used throughout the remainder of the text. Subsections 2.1.7, 2.2.3 and 2.2.4 introduce terminology that will be important in later chapters. Chapter 3 (Statistical models) introduces the signal + noise form of regression model. The different models for the signal component are too varied to describe in one chapter! Coverage of models for the noise (random component) is, relative to their use in remaining chapters, more complete. Chapter 4 (A review of inference concepts) describes approaches to generalizing from data. It notes the limitations of the formal hypothesis testing methodology, arguing that a less formal approach is often adequate. It notes also that there are contexts where a Bayesian approach is essential, in order to take account of strong prior information. Chapters 5–13: Regression and related methodology Chapters 5–13 are designed to give a sense of the variety and scope of methods that come, broadly, under the heading of regression. In Chapters 5 and 6, the models are linear in the explanatory variable(s) as well as in the parameters. A wide range of issues affect the practical use of these models: influence, diagnostics, robust and resistant methods, AIC and other model comparison measures, interpretation of coefficients, variable selection, multicollinearity, and errors in x. All these issues are relevant, in one way or another, throughout later chapters. Chapters 5 and 6 provide relatively straightforward contexts in which to introduce them.

xxvi

Content – how the chapters fit together

The models of Chapters 5–13 give varying combinations of answers to the questions: 1. What is the signal term? Is it in some sense linear? Can it be described by a simple form of mathematical equation? 2. Is the noise term normal, or are there other possibilities? 3. Are the noise terms independent between observations? 4. Is the model specified in advance? Or will it be necessary to choose the model from a potentially large number of possible models? In Chapters 5–8, the models become increasingly general, but always with a model that is linear in the coefficients as a starting point. In Chapters 5–7, the noise terms are normal and independent between observations. The generalized linear models of Chapter 8 allow nonnormal noise terms. These are still assumed independent.1 Chapter 9 (Time series models) and Chapter 10 (Multilevel models and repeated measures) introduce models that allow, in their different ways, for dependence between observations. In Chapter 9 the correlation is with observations at earlier points in time, while in Chapter 10 the correlation might for example be between different students in the same class, as opposed to different students in different classes. In both types of model, the noise term is constructed from normal components – there are normality assumptions. Chapters 6–10 allowed limited opportunity for the choice of model and/or explanatory variables. Chapter 11 (Tree-based classification and regression) introduces models that are suited to a statistical learning approach, where the model is chosen from a large portfolio of possibilities. Moreover, these models do not have any simple form of equation. Note the usual implicit assumption of independence between observations – this imposes limitations that, depending on the context, may or may not be important for practical use. Chapter 12 (Multivariate data exploration and discrimination) begins with methods that may be useful for multivariate data exploration – principal components, the use of distance measures, and multi-dimensional scaling. It describes dimension reduction approaches that allow low-dimensional views of the data. Subsection 12.2 moves to discriminant methods – i.e., to regression methods in which the outcome is categorical. Subsection 12.3 identifies issues that arise when the number of variables is large relative to the number of observations. Such data is increasingly common in many different application areas. It is sometimes possible to replace a large number of explanatory variables by one, or a small number, of scoring variables that capture the relevant information in the data. Chapter 13 investigates two different ways to create scores that may be used as explanatory variables in regression. In the first example, the principal component scores are used. The second uses propensity scores to summarize information on a number of covariates that are thought to explain group differences that are, for the purposes of the investigation, nuisance variables.

1

Note, however, the extension to allow models with a variance that, relative to the binomial or Poisson, is inflated.

1

A brief introduction to R

This first chapter introduces readers to the basics of R. It provides the minimum of information that is needed for running the calculations that are described in later chapters. The first section may cover most of what is immediately necessary. The rest of the chapter may be used as a reference. Chapter 14 extends this material considerably. Most of the R commands will run without change in S-PLUS. 1.1 An overview of R 1.1.1 A short R session R must be installed!

An up-to-date version of R may be downloaded from a Comprehensive R Archive Network (CRAN) mirror site. There are links at http://cran.r-project.org/. Installation instructions are provided at the web site for installing R in Windows, Unix, Linux, and version 10 of the Macintosh operating system. For most Windows users, R can be installed by clicking on the icon that appears on the desktop once the Windows setup program has been downloaded from CRAN. An installation program will then guide the user through the process. By default, an R icon will be placed on the user’s desktop. The R system can be started by double-clicking on that icon. Various contributed packages extend the capabilities of R. A number of these are a part of the standard R distribution, but a number are not. Many data sets that are mentioned in this book have been collected into our DAAG package that is available from CRAN sites. This and other such packages can be readily installed, from an R session, via a live internet connection. Details are given below, immediately prior to Subsection 1.1.2. Using the console (or command line) window

The command line prompt (>) is an invitation to type commands or expressions. Once the command or expression is complete, and the Enter key is pressed, R evaluates and prints the result in the console window. This allows the use of R as a calculator. For example, type 2+2 and press the Enter key. Here is what appears on the screen: > 2+2 [1] 4 >

2

A brief introduction to R

The first element is labeled [1] even when, as here, there is just one element! The final > prompt indicates that R is ready for another command. In a sense this chapter, and much of the rest of the book, is a discussion of what is possible by typing in statements at the command line. Practice in the evaluation of arithmetic expressions will help develop the needed conceptual and keyboard skills. For example: > 2*3*4*5 [1] 120 > sqrt(10) [1] 3.162278 > pi [1] 3.141593 > 2*pi*6378

# * denotes ’multiply’ # the square root of 10 # R knows about pi # Circumference of earth at equator (km) # (radius at equator is 6378 km)

[1] 40074.16

Anything that follows a # on the command line is taken as comment and ignored by R. A continuation prompt, by default +, appears following a carriage return when the command is not yet complete. For example, an interruption of the calculation of 3*4ˆ2 by a carriage return could appear as > 3*4ˆ + 2 [1] 48

In this book we will omit both the command prompt (>) and the continuation prompt whenever command line statements are given separately from output. Multiple commands may appear on one line, with a semicolon (;) as the separator. For example, > 3*4ˆ2; (3*4)ˆ2 [1] 48 [1] 144

Entry of data at the command line

Figure 1.1 gives, for each of the years 1800, 1850, . . . , 2000, estimated worldwide totals of carbon emissions that resulted from fossil fuel use. To enter the columns of data from the table, and plot Carbon against Year as in Figure 1.1, proceed thus: Year <- c(1800, 1850, 1900, 1950, 2000) Carbon <- c(8, 54, 534, 1630, 6611) ## Now plot Carbon as a function of Year plot(Carbon ˜ Year, pch=16)

Note the following: r The <- is a left angle bracket (<) followed by a minus sign (-). It means “the values on the right are assigned to the name on the left”.

1.1 An overview of R

4000

Carbon

6000

●

2000 0

3

1 2 3 4 5

● ●

●

1800

1850

●

1900

1950

year

carbon

1800 1850 1900 1950 2000

8 54 534 1630 6611

2000

Year

Figure 1.1 Estimated worldwide annual totals of carbon emissions from fossil fuel use, in millions of tonnes. Data are due to Marland et al. (2003).

r The objects Year and Carbon are vectors which were each formed by joining (concatenating) separate numbers together. Thus c(8, 54, 534, 1630, 6611) joined the numbers 8, 54, 534, 1630, 6611 together to form the vector Carbon. See Subsection 1.2.2 for further details. r The construct Carbon ˜ Year is a graphics formula. The plot() function interprets this formula to mean “Plot Carbon as a function of Year” or “Plot Carbon on the y-axis against Year on the x-axis”. r The setting pch=16 (where pch is “plot character”) gives a solid black dot. r Case is significant for names of R objects or commands. Thus, Carbon is different from carbon. This basic plot could be improved by adding more informative axis labels, changing sizes of the text and/or the plotting symbol, adding a title, and so on. See Section 1.5. Once created, the objects Year and Carbon are stored in the workspace, as part of the user’s working collection of R objects. The workspace lasts only until the end of a session. In order that the session can be resumed later, a copy or “image” must be kept. Upon typing q() to quit the session, you will be asked if you wish to save the workspace.

Collection of vectors into a data frame

The two vectors Year and Carbon created earlier are matched, element for element. It is convenient to group them together into an object that has the name data frame, thus: > fossilfuel <- data.frame(year=Year, carbon=Carbon) > fossilfuel # Display the contents of the data frame. year carbon 1 1800 8 2 1850 54 3 1900 534 4 1950 1630 5 2000 6611

4

A brief introduction to R

The vector objects Year and Carbon become, respectively, the columns year and carbon in the data frame. The vector objects Year and Carbon are then redundant, and can be removed. rm(Year, Carbon)

# The rm() function removes unwanted objects

Figure 1.1 can now be reproduced, with a slight change in the x- and y-labels, using plot(carbon ˜ year, data=fossilfuel, pch=16)

The data=fossilfuel argument instructs plot() to start its search for each of carbon and year by looking among the columns of fossilfuel. There are several ways to identify columns by name. Here, note that the second column can be referred to as fossilfuel[, 2], or as fossilfuel[, "carbon"], or as fossilfuel$carbon. Data frames are the preferred way to organize data sets that are of modest size. For now, think of data frames as a rectangular row by column layout, where the rows are observations and the columns are variables. Section 1.3 has further discussion of data frames. Subsection 1.1.4 will demonstrate input of data from a file, into a data frame. The R Commander Graphical User Interface (GUI) to R

Our discussion will usually assume use of the command line. Excellent GUI interfaces, such as the R Commander, are also available. Data input is very convenient with the R Commander. When importing data, a window pops up offering a choice of common data formats. Data can be input from a text file, the clipboard, URL, an Excel spreadsheet, or one of several statistical package formats (SPSS, Stata, Minitab, SAS, . . .). Refer to Section 14.1 for more details. The working directory and the contents of the workspace

Each R session has a working directory. Within a session, the workspace is the default place where R looks for files that are read from disk, or written to disk. In between sessions, it is usual for the working directory to keep a workspace copy or “image” from which the session can be restarted. For a session that is started from a Windows icon, the initial working directory is the Start in directory that appears by right clicking on the icon and then on Properties. Users of the MacOS X GUI can change the default startup directory from within an R session by clicking on the R menu item, then on Preferences, then making the necessary change in the panel Initial working directory. On Unix or Linux sessions that are started from the command line, the working directory is the directory in which R was started. In the event of uncertainty, type getwd() to display the name of the working directory: getwd()

Objects that the user creates or copies from elsewhere go into the user workspace. To list the workspace contents, type: ls()

1.1 An overview of R

5

The only object left over from the computations above should be fossilfuel. There may additionally be objects that are left over from previous sessions (if any) in the same directory, and that were loaded when the session started.

Quitting R

Use the q() function to quit (exit) from R: q()

There will be a message asking whether to save the workspace image. Clicking Yes has the effect that, before quitting, all the objects that remain in the workspace are saved in a file that has the name .RData. Because it is a copy or “image” of the workspace, this file is known as an image file. (Note that while delaying the saving of important objects until the end of the session is acceptable when working in a learning mode, it is not in general a good strategy when using R in production mode. Section 1.6 has advice on saving and backing up through the course of a session. See also the more extended comments in Subsection 14.2.2.) Depending on the implementation, alternatives to typing q() may be to click on the File menu and then on Exit, or to click on the × in the top right-hand corner of the R window. (Under Linux, depending on the window manager that is used, clicking on × may exit from the program, but without asking whether to save the workshop image. Check the behavior on your installation.) Note: The round brackets, when using q() to quit the session, are necessary because q is a function. Typing q on its own, without the brackets, displays the text of the function on the screen. Try it!

Installation of packages

Assuming access to a live internet connection, packages can be installed pretty much automatically. Thus, for installation of the DAAG package under Windows, start R and click on the Packages menu. From that menu, choose Install packages. If a mirror site has not been set earlier, this gives a pop-up menu from which a site must be chosen. Once this choice is made, a new pop-up window appears with the entire list of available R packages. Click on DAAG to select it for installation. Control-click to select additional packages. Click on OK to start downloading and installation. For installation from the command line, enter, for example install.packages("DAAG") install.packages(c("magic", "schoolmath"), dependencies=TRUE)

A further possibility, convenient if packages are to be installed onto a number of local systems, is to download the files used for the installation onto a local directory or onto a CD or DVD, and install from there.

6

A brief introduction to R

1.1.2 The uses of R

R has extensive capabilities for statistical analysis, that will be used throughout this book. These are embedded in an interactive computing environment that is suited to many different uses, some of which we now demonstrate.

R offers an extensive collection of functions and abilities

Most calculations that users may wish to perform, beyond simple command line computations, involve explicit use of functions. There are of course functions for calculating the sum (sum()), mean (mean()), range (range()), and length of a vector (length()), for sorting values into order (sort()), and so on. For example, the following calculates the range of the values in the vector carbon: > range(fossilfuel$carbon) [1] 8 6611

Here are examples that manipulate character strings: > ## 4 cities > fourcities <- c("Toronto", "Canberra", "New York", "London") > ## display in alphabetical order > sort(fourcities) [1] "Canberra" "London" "New York" "Toronto" > ## Find the number of characters in "Toronto" > nchar("Toronto") [1] 7 > > ## Find the number of characters in all four city names at once > nchar(fourcities) [1] 7 8 8 6

R will give numerical or graphical data summaries

The data frame cars (datasets package) has columns (variables) speed and dist. Typing summary(cars) gives summary information on its columns: > summary(cars) speed Min. : 4.0 1st Qu.:12.0 Median :15.0 Mean :15.4 3rd Qu.:19.0 Max. :25.0

dist Min. : 2.00 1st Qu.: 26.00 Median : 36.00 Mean : 42.98 3rd Qu.: 56.00 Max. :120.00

Thus, the range of speeds (first column) is from 4 mph to 25 mph, while the range of distances (second column) is from 2 feet to 120 feet.

1.1 An overview of R

7

Graphical summaries, including histograms and boxplots, are discussed and demonstrated in Section 2.1. Try, for example: hist(cars$speed)

R is an interactive programming language

The following calculates the Fahrenheit temperatures that correspond to Celsius temperatures 0, 10, . . . , 40: > > > > 1 2 3 4 5

celsius <- (0:4)*10 fahrenheit <- 9/5*celsius+32 conversion <- data.frame(Celsius=celsius, Fahrenheit=fahrenheit) print(conversion) Celsius Fahrenheit 0 32 10 50 20 68 30 86 40 104

1.1.3 Online help

Familiarity with R’s help facilities will quickly pay dividends. R’s help files are comprehensive, and are frequently upgraded. Type help(help) or ?help to get information on the help features of the system that is in use. To get help on, e.g., plot(), type: ?plot

# Equivalent to help(plot)

The functions apropos() and help.search() search for functions that perform a desired task. Examples are: apropos("sort") # Try, also, apropos ("sor") # List all functions where "sort" is part of the name help.search("sort") # Note that the argument is ’sort’ # List functions with ’sort’ in the help page title or as an alias

Users are encouraged to experiment with R functions, perhaps starting by using the function example() to run the examples on the relevant help page. Be warned however that, even for basic functions, some examples may illustrate relatively advanced uses. Thus, to run the examples from the help page for the function image(), type: example(image) par(ask=FALSE)

# turn off the prompts

Press the return key to see each new plot. The par(ask=FALSE) on the second line of code stops the prompts that will otherwise continue to appear, prior to the plotting of any subsequent graph. In learning to use a new function, it may be helpful to create a simple artificial data set, or to extract a small subset from a larger data set, and use this for experimentation. For

8

A brief introduction to R

extensive experimentation, consider moving to a new working directory and working with copies of any user data sets and functions. The help pages, while not an encyclopedia on statistical methodology, have very extensive useful information. They include: insightful and helpful examples, references to related functions, and references to papers and books that give the relevant theory. Some abilities will bring pleasant surprises. It can help enormously, before launching into the use of an R function, to check the relevant help page!

Wide-ranging information access and searches

The function help.start() opens a browser interface to help information, manuals, and helpful links. It may take practice, and time, to learn to navigate the wealth of information that is on offer. The function RSiteSearch() initiates (assuming a live internet connection) a search of R manuals and help pages, and of the R-help mailing list archives, for key words or phrases. The argument restrict allows some limited targeting of the search. See help(RSiteSearch) for details.

Help in finding the right package

The CRAN Task Views can be a good place to start when looking for abilities of a particular type. The 23 Task Views that are available at the time of writing include, for example: Bayesian inference, Cluster analysis, Finance, Graphics, and Time series. Go to http://cran.r-project.org/web/views/

1.1.4 Input of data from a file

Code that will take data from the file fuel.txt that is in the working directory, entering them into the data frame fossilfuel in the workspace is: fossilfuel <- read.table("fuel.txt", header=TRUE)

Note the use of header=TRUE to ensure that R uses the first line to get header information for the columns, usually in the form of column names. Type fossilfuel at the command line prompt, and the data will be displayed almost as they appear in Figure 1.1 (the only difference is the introduction of row labels in the R output). The function read.table() has the default argument sep="", implying that the fields of the input file are separated by spaces and/or tabs. Other settings are sometimes required. In particular: fossilfuel <- read.table("fuel.csv", header=TRUE, sep=",")

reads data from a file fuel.csv where fields are separated by commas. For other options, consult the help page for read.table(). See also Subsection 14.4.1.

1.1 An overview of R

9

On Microsoft Windows systems, it is immaterial whether this file is called fuel.txt or Fuel.txt. Unix file systems may, depending on the specific file system in use, treat letters that have a different case as different.

1.1.5 R packages

This chapter and Chapter 2 will make frequent use of data from the MASS package (Venables and Ripley, 2002) and from our own DAAG package. Various further packages will be used in later chapters. The packages base, stats, datasets, and several other packages, are automatically attached at the beginning of a session. Other installed packages must be explicitly attached prior to use. Use sessionInfo() to see which packages are currently attached. To attach any further installed package, use the library() function. For example, > library(DAAG) Loading required package: MASS . . . . > sessionInfo() R version 2.9.0 (2009-04-17) i386-apple-darwin8.11.1 . . . . attached base packages: [1] stats graphics grDevices utils

datasets

methods

base

other attached packages: [1] DAAG_0.98 MASS_7.2-46

Data sets that accompany R packages

Type data() to get a list of data sets (mostly data frames) in all packages that are in the current search path. For information on the data sets in the datasets package, type data(package="datasets")

# Specify ’package’, not ’library’.

Replace "datasets" by the name of any other installed package, as required (type library() to get the names of the installed packages). In most packages, these data sets automatically become available once the package is attached. They will be brought into the workspace when and if required. (A few packages do not implement the lazy data mechanism. Explicit use of a command of the form data(airquality) is then necessary, bringing the data object into the user’s workspace.)

1.1.6 Further steps in learning R

Readers who have followed the discussion thus far and worked through the examples may at this point have learned enough to start on Chapter 2, referring as necessary to later sections of this chapter, to R’s help pages, and to Chapter 14. The remaining sections of this chapter cover the following topics:

10

r r r r r

A brief introduction to R

Numeric, character, logical, and complex vectors (Section 1.2). Factors (Subsection 1.2.7). Data frames and matrices (Section 1.3). Functions for calculating data summaries (Section 1.4). Graphics and lattice graphics (Sections 1.5 and 15.5).

1.2 Vectors, factors, and univariate time series Vectors, factors, and univariate time series are all univariate objects that can be included as columns in a data frame. The vector modes that will be noted here (there are others) are “numeric”, “logical”, and “character".

1.2.1 Vectors

Examples of vectors are > c(2, 3, 5, 2, 7, 1) [1] 2 3 5 2 7 1 > c(T, F, F, F, T, T, F) [1] TRUE FALSE FALSE FALSE

TRUE

TRUE FALSE

> c("Canberra", "Sydney", "Canberra", "Sydney") [1] "Canberra" "Sydney" "Canberra" "Sydney"

The first of these is numeric, the second is logical, and the third is a character. The global variables F (=FALSE) and T (=TRUE) can be a convenient shorthand when logical values are entered.

1.2.2 Concatenation – joining vector objects

The function c(), used in Subsection 1.1.1 to join numbers together to form a vector, is more widely useful. It may be used to concatenate any combination of vectors and vector elements. In the following, we form numeric vectors x and y, that we then concatenate to form a vector z: > x > x [1] > y > y [1] > z > z [1]

<- c(2, 3, 5, 2, 7, 1) 2 3 5 2 7 1 <- c(10, 15, 12) 10 15 12 <- c(x, y) 2 3 5 2 7 1 10 15 12

# x then holds values 2, 3, 5, 2, 7, 1

1.2 Vectors, factors, and univariate time series

11

1.2.3 The use of relational operators to compare vector elements

Relational operators are <, <=, >, >=, ==, and !=. For example, consider the carbon and year columns of the data frame fossilfuel. For example: > x > x [1] > x [1]

<- c(3, 11, 8, 15, 12) > 8 FALSE TRUE FALSE TRUE != 8 TRUE TRUE FALSE TRUE

TRUE TRUE

For further information on relational operators consult help(Comparison), help(Logic), and help(Syntax).

1.2.4 The use of square brackets to extract subsets of vectors

Note three common ways to extract elements of vectors. In each case, the identifying information (in the simplest case, a vector of subscript indices) is enclosed in square brackets. 1. Specify the indices of the elements that are to be extracted, e.g., > x <- c(3, 11, 8, 15, 12) > x[c(2,4)] [1] 11 15

# Elements in positions 2 # and 4 only

2. Use negative subscripts to omit the elements in nominated subscript positions (take care not to mix positive and negative subscripts): > x[-c(2,3)] [1] 3 15 12

# Remove the elements in positions 2 and 3

3. Specify a vector of logical values. This extracts elements for which the logical value is TRUE. The following extracts values of x that are greater than 10: > x > 10 [1] FALSE TRUE FALSE TRUE TRUE > x[x > 10] [1] 11 15 12

Elements of vectors can be given names. Elements can then be extracted by name: > heights <- c(Andreas=178, John=185, Jeff=183) > heights[c("John","Jeff")] John Jeff 185 183

1.2.5 Patterned data

Use, for example, 5:15 to generate all integers in a range, here between 5 and 15 inclusive: > 5:15 [1] 5

6

7

8

9 10 11 12 13 14 15

Conversely, 15:5 will generate the sequence in the reverse order.

12

A brief introduction to R

The function seq() allows a wider range of possibilities. For example: > seq(from=5, to=22, by=3)

# The first value is 5. # value is <= 22

The final

[1] 5 8 11 14 17 20 ## The above can be abbreviated to seq(5, 22, 3)

To repeat the sequence (2, 3, 5) four times over, enter > rep(c(2,3,5), 4) [1] 2 3 5 2 3 5 2 3 5 2 3 5

Patterned character vectors are also possible: > c(rep("female", 3), rep("male", 2)) [1] "female" "female" "female" "male" "male"

1.2.6 Missing values

The missing value symbol is NA. As an example, consider the column branch of the data set rainforest: > library(DAAG) > nbranch <- subset(rainforest, species=="Acacia mabellae")$branch > nbranch # Number of small branches (2cm or less) [1] NA 35 41 50 NA NA NA NA NA 4 30 13 10 17 46 92

Any arithmetic expression that involves an NA generates NA as its result. Functions such as mean() allow the argument na.rm=TRUE, so that NAs are omitted before proceeding with the calculation. For example: > mean(nbranch) [1] NA > mean(nbranch, na.rm=TRUE) [1] 33.8

Other functions that behave similarly are sum(), median(), range(), and sd. Arithmetic and logical expressions in which NAs appear return NA, thus: > NA == 35 [1] NA

The unknown value might just possibly equal 35. This is a matter of strict logic, not probability. Thus, the result is NA. To replace all NAs by -999 (in most circumstances a bad idea) use the function is.na(), thus: > ## Replace all NAs by -999 > nbranch[is.na(nbranch)] <- -999 > nbranch [1] -999 35 41 50 -999 -999 -999 -999 -999 4 30 13 [13] 10 17 46 92 > ## There is now no protection against use of the -999 values as

1.2 Vectors, factors, and univariate time series

13

> ## if they were legitimate numeric values > mean(nbranch) [1] -353.5 # Illegitimate calculation

Using a code such as -999 for missing values requires continual watchfulness to ensure that it is never treated as a legitimate numeric value. Missing values are discussed further in Subsection 1.4.6 and Section 14.7. For vectors of mode numeric, other legal values that may require special attention are NaN (not a number; e.g., 0/0), Inf (e.g., 1/0), and -Inf. 1.2.7 Factors

A factor is stored internally as a numeric vector with values 1, 2, 3, . . . , k. The value k is the number of levels. The levels are character strings. Consider a survey that has data on 691 females and 692 males. If the first 691 are females and the next 692 males, we can create a vector of strings that holds the values, then turning this vector into a factor, thus: > ## Create character vector > gender <- c(rep("female",691), rep("male",692)) > levels(gender) # For a character vector, this returns NULL NULL > ## From character vector, create factor > gender <- factor(gender) > levels(gender) [1] "female" "male"

Internally, the factor gender is stored as 691 1s, followed by 692 2s. It has stored with it a table that holds the information 1 2

female male

In most contexts that seem to demand a character string, the 1 is translated into female and the 2 into male. The values female and male are the levels of the factor. By default, the levels are in sorted order for the data type from which the factor was formed, so that female precedes male. Hence: > levels(gender) [1] "female" "male"

Note that if gender had been an ordinary character vector, the outcome of the above levels command would have been NULL. The order of the factor levels is used, in graphs and tables, to determine the order in which the levels will appear. To cause male to come before female, use gender <- factor(gender, levels=c("male", "female"))

This syntax is available both when the factor is first created, and later to change the order in an existing factor. Take care that the level names are correctly spelled. For example,

14

A brief introduction to R

specifying "Male" in place of "male" in the levels argument will cause all values that were "male" to be coded as missing. Note finally the function ordered(), which generates factors whose values can be compared using the relational operators <, <=, >, >=, ==, and !=. Ordered factors are appropriate for use with ordered categorical data. See Section 14.6 for further details.

1.2.8 Time series

The following are the numbers of workers (in 1000s) in the Canadian prairies for each month from January 1995 through December 1996:1 numjobs <- c(982,981,984,982,981,983,983,983,983,979,973,979, 974,981,985,987,986,980,983,983,988,994,990,999)

The function ts() converts numeric vectors into time series objects. Frequently used arguments of ts() are start, frequency, and end. The following turns numjobs into a time series, which can then be plotted: numjobs <- ts(numjobs, start=1995, frequency = 12) plot(numjobs)

Use the function window() to extract a subset of the time series. For example, the following extracts the last quarter of 1995 and the first few months of 1996: first15 <- window(numjobs, start=1995.75, end=1996.25)

Multivariate time series can also be handled. See Subsections 2.1.5 and 14.9.7.

1.3 Data frames and matrices Data frames are fundamental to the use of the R modeling and graphics functions. A data frame is a more general object than a matrix, in the sense that different columns may have different modes. All elements of any column must, however, have the same mode, i.e., all numeric, or all factor, or all character, or all logical. Included in the DAAG package is Cars93.summary, created from the Cars93 data set in the MASS package. Its contents are: > Cars93.summary Min.passengers Max.passengers No.of.cars abbrev Compact 4 6 16 C Large 6 6 11 L Midsize 4 6 22 M Small 4 5 21 Sm Sporty 2 4 14 Sp Van 7 8 9 V

1

## Alternatively, obtain from data frame jobs (DAAG) library(DAAG) numjobs <- jobs$Prairies

1.3 Data frames and matrices

15

The first three columns are numeric, and the fourth is a factor. Use the function class() to check this, e.g., enter class(Cars93.summary$abbrev). (The classification of objects into classes is discussed in Subsection 1.4.2.) On most systems, use of edit() allows access to a spreadsheet-like display of a data frame or of a vector, where entries can be edited or new data added. For example, Cars93.summary <- edit(Cars93.summary)

To close the spreadsheet, click on the File menu and then on Close. On Linux systems, click on Quit to exit. Displaying the first few, or last few, rows of a data frame

When used with a data frame (other possible arguments include vectors and functions), the head() function displays the first lines of a data frame, while tail() displays the last lines. For example, > head(Cars93.summary, n=3) Min.passengers Compact 4 Large 6 Midsize 4 > # . . .

# Display the first 3 rows (the default is 6) Max.passengers No.of.cars abbrev 6 16 C 6 11 L 6 22 M

Note also the functions str() and summary(), both of which can be used to get summary information on data frames, different in the two cases. Column and row names

The function rownames() extracts the names of rows, while colnames() extracts column names, thus: rownames(Cars93.summary) colnames(Cars93.summary)

# Extract row names # Extract column names

For use with data frames row.names() is an alternative to rownames(), while names() is an alternative to colnames(). The functions names() (or colnames()) and rownames() can also be used to assign new names. For example: names(Cars93.summary)[3] <- "numCars" names(Cars93.summary) <- c("minPass","maxPass","numCars","code")

Subsets of data frames

Data frames are indexed by row and column number. Thus to extract the element in the 4th row and 2nd column, specify Cars93.summary[4, 2]. Here are additional examples:

16

A brief introduction to R

Cars93.summary[1:3, 2:3] # Rows 1-3 and columns 2-3 Cars93.summary[, 2:3] # Columns 2-3 (all rows) Cars93.summary[, c("No.of.cars", "abbrev")] # Cols 2-3, by name Cars93.summary[, -c(2,3)] # omit columns 2 and 3

The subset() function offers an alternative way to extract rows and columns. For example, the following extracts the first two rows: subset(Cars93.summary, subset=c(TRUE, TRUE, FALSE, FALSE, FALSE, FALSE))

Use the argument select to specify a subset of columns. See help(subset) for details. Use of the subscript notation to extract a column, as in Cars93.summary[, 1], returns a vector. By contrast, extraction of the raw Cars93.summary[1, ] returns a data frame, necessary because this allows different elements (columns) to retain their existing classes. Note also r Use of unlist(Cars93.summary[1, ]) returns a vector, but with the side-effect that the factor value in the final column is coerced to numeric. Such side-effects are usually undesirable, with a result that may be meaningless. r Avoid Cars93.summary[4], at least until the subtleties of its use are understood. See Subsection 14.9.1. If used where Cars93.summary[, 4] was intended, the calculation may fail or give an erroneous result.

Data frames are a specialized type of list

A list is an arbitrary collection of R objects. Here is a simple example, containing two character vectors of differing lengths and a numeric vector: > ## Cities with more than 2.5 million inhabitants > USACanada <- list(USACities=c("NY", "LA", "Chicago"), + CanadaCities=c("Toronto", "Montreal"), + millionsPop=c(USA=305.9, Canada=31.6)) > > USACanada $USACities [1] "NY" "LA" "Chicago" $CanadaCities [1] "Toronto"

"Montreal"

$millionsPop USA Canada 305.9 31.6

Many of R’s modeling functions return their output as a list. Lists can be joined using the function c(); in this and in several other respects they are “vectors”. Important aspects of the syntax for working with data frames apply also to lists. Obviously, however,

1.3 Data frames and matrices

17

notions of “row” and “column” have no relevance to lists. See Subsection 14.9.1 for further commentary. 1.3.1 Accessing the columns of data frames – with() and attach()

In repeated computations with the same data frame, it is tiresome to keep repeating the name of the data frame. The function with() is often helpful in this connection. Thus, an alternative to c(mean(cfseal$weight), median(cfseal$weight)) is: > ## cfseal (DAAG) has data on Cape Fur seals > with(cfseal, c(mean(weight), median(weight))) [1] 54.8 46.2

Curly brackets (braces) can be used to extend the scope of with() over several lines of code: > with(pair65, # stretch of rubber bands, from DAAG + {lenchange <- heated-ambient + c(mean(lenchange), median(lenchange)) + }) [1] 6.33 6.00

An alternative is attach(). Once a data frame has been attached, its columns can be referred to by name, without further need to give the name of the data frame. For example: > year Error: Object "year" not found > attach(fossilfuel) # Attach data frame fossilfuel > year [1] 1800 1850 1900 1950 2000 > detach(fossilfuel) # Detach data frame

Be sure to detach data frames that are no longer in use. If more than one data frame is attached that has the column year, there is obvious scope for confusion. What happens if there is an object year in the workspace? References to year will then take the object that is in the workspace, ignoring the column year in the attached data frame. By contrast, use of with() ensures that the column, if present, is from the specified data frame. The attaching of a data frame extends the search list, which is the list of “databases” where R looks for objects. See Section 14.2 for more details on this and other uses of attach(). 1.3.2 Aggregation, stacking, and unstacking

The aggregate() function yields a data frame that has the mean or value of another specified function for each combination of factor levels. As an example, consider the chickwts data frame which contains observations on the weights of 71 six-week-old chicks who have been fed one of six kinds of feed. The columns of chickwts are named weight and feed. To find the average weights for the different feed groups, type

18

A brief introduction to R

> chickwtAvs <- with(chickwts, + aggregate(weight, by=list(feed), mean)) > names(chickwtAvs) <- c("Feed Group", "Mean Weight") > chickwtAvs Feed Group Mean Weight 1 casein 323.5833 2 horsebean 160.2000 3 linseed 218.7500 . . . .

See Subsection 14.9.5 for more information on the aggregate() function. For stacking columns of a data frame, i.e., placing successive columns one under the other, the function stack() is available. The following use of stack(), with data from the data frame jobs, will be required for the use of these data in Subsection 2.1.5. > library(DAAG) > head(jobs,3) BC Alberta Prairies Ontario Quebec Atlantic Date 1 1752 1366 982 5239 3196 947 95.00000 2 1737 1369 981 5233 3205 946 95.08333 3 1765 1380 984 5212 3191 954 95.16667 > # . . . > Jobs <- stack(jobs, select = 1:6) > # stack() concatenates selected data frame columns into a > # single column named "values", & adds a factor named "ind" > # that has the names of the concatenated columns as levels. > head(Jobs,3) values ind 1 1752 BC 2 1737 BC 3 1765 BC > # . . .

For a further example, see Exercise 19. The unstack() function reverses the stacking operation. For example, unstack (Jobs) (or more generally, unstack(Jobs, values ˜ ind)) recovers the original data frame.

1.3.3 ∗ Data frames and matrices

The numeric values in the data frame fossilfuel might alternatively be stored in a matrix with the same dimensions, i.e., 5 rows × 2 columns. The following enters these same data as a matrix: fossilfuelmat <- matrix(c(1800, 1850, 1900, 1950, 2000, 8, 54, 534, 1630, 6611), nrow=5) colnames(fossilfuel) <- c("year", "carbon")

1.4 Functions, operators, and loops

19

Another possibility is the use of the function cbind() to combine two or more vectors of the same length and type together into a matrix, thus: fossilfuelmat <- cbind(year=c(1800, 1850, 1900, 1950, 2000), carbon=c(8, 54, 534, 1630, 6611))

More generally, any data frame where all columns hold data that is all of the same type, i.e., all numeric or all character or all logical, can alternatively be stored as a matrix. Storage of numeric data in matrix rather than data frame format can speed up some mathematical and other manipulations when the number of elements is large, e.g., of the order of several hundreds of thousands. For further details, see Section 14.8. Note that: r Matrix elements are stored in column order in one long vector, i.e., columns are stacked one above the other, with the first column first. Section 14.8 describes how to change between a matrix with m rows and n columns, and a vector of length mn. r The extraction of submatrices has the same syntax as for data frames. Thus, fossilfuelmat[2:3,] extracts rows 2 and 3 of the matrix fossilfuelmat. (Be careful not to omit the comma, causing the matrix to be treated as one long vector.) r The names() function returns NULL when the argument is a matrix. Note however rownames() and colnames(), which can be used either with data frames or matrices. r The function nrow() (e.g. nrow(fossilfuel) or nrow(fossilfuelmat)) returns the number of rows, while ncol() returns the number of columns. 1.4 Functions, operators, and loops Functions are integral to the use of the R language. User-written functions are used in exactly the same way as built-in functions. Examples will appear from time to time through the book. An incidental advantage of putting code into functions is that the workspace is not then cluttered with objects that are local to the function. 1.4.1 Common useful built-in functions all() any() args() cat() cumprod() cumsum() diff()

# # # # # # # # history() # is.factor() # is.na() # # length() # ls() #

returns TRUE if all values are TRUE returns TRUE if any values are TRUE information on the arguments to a function prints multiple objects, one after the other cumulative product cumulative sum form vector of first differences N. B. diff(x) has one less element than x displays previous commands used returns TRUE if the argument is a factor returns TRUE if the argument is an NA NB also is.logical(), is.matrix(), etc. number of elements in a vector or of a list list names of objects in the workspace

20 mean() median() order() print() range() sort() rev() str() unique() which() which.max() which.min() with()

A brief introduction to R # # # # # # # # # # # # #

mean of the elements of a vector median of the elements of a vector x[order(x)] sorts x (by default, NAs are last) prints a single R object minimum and maximum value elements of vector sort elements into order, by default omitting NAs reverse the order of vector elements information on an R object form the vector of distinct values locates ’TRUE’ indices of logical vectors locates (first) maximum of a numeric vector locates (first) minimum of a numeric vector do computation using columns of specified data frame

Be sure to check, where this is relevant, the handling of missing values. In case of doubt, consult the relevant help page. Refer back to Subsection 1.2.6. The print() function

This is perhaps R’s most pervasive function. It is invoked whenever an object, or the result of a computation, has its value returned to the command line. For example: > x <- 2 > x [1] 2 > x*5 [1] 10

# Assign to x the value 2; nothing is printed # Equivalent to print(x) # Equivalent to print(x*5)

It can be convenient to make an assignment and print the value. For this, enclose the assignment in parentheses, i.e., in round brackets: > (x <- 2) [1] 2

# Equivalent to: x <- 2; print(x)

Calculations in parallel across all elements of a vector

Subsection 1.1.2 gave an example in which arithmetic was carried out in parallel across all elements of a vector. Many of R’s functions likewise operate in parallel on all elements of arrays, matrices, and data frames.

Data summary functions – table() and sapply()

Data summary functions that create tables of counts are: table() xtabs()

# Form a table of counts # Form a table of totals

For example, the tinting data frame in DAAG contains columns specifying the sex (sex, levels are f and m) and age group (agegp, levels are younger and older) of

1.4 Functions, operators, and loops

21

participants in a study. The table() function can be used to count up the numbers of observations in each sex–age group combination: > library(DAAG) # tinting is from DAAG > table(Sex=tinting$sex, AgeGroup=tinting$agegp) AgeGroup Sex younger older f 63 28 m 28 63

By default, table() ignores NAs. For further details of table(), and for an example of the use of xtabs(), see Subsection 2.2.1. The function sapply() applies a function to each column of a data frame, or to each element of a list. The following demonstrates its use to give the range, for all columns of the data frame jobs (DAAG): > sapply(jobs[, -7], range) BC Alberta Prairies Ontario Quebec Atlantic [1,] 1737 1366 973 5212 3167 941 [2,] 1840 1436 999 5360 3257 968

Utility functions

Type ls() (or objects()) to see the names of all objects in the workspace. One can restrict the names to those with a defined pattern, e.g., starting with the letter p:2 ls(pattern="p") ls(pattern="ˆp")

# List object names that include the letter "p" # List object names that start with "p"

Type help(ls), help(grep), and help(glob2rx) for more details. Various packages will add hidden files, whose first character is a full stop, to the workspace. To see these files, type ls(all=TRUE). To clear the workspace completely, type rm(list=ls(all=TRUE)). By default, the function dir() lists the contents of the working directory. See Subsection 14.2.3 for further details on this and other utility functions.

1.4.2 Generic functions, and the class of an object

The printing of a data frame requires steps that are different from those for the printing of a vector of numbers. Yet, in R, the same print() function handles both tasks. In order to make this possible, all objects in R have a class, which can be used to decide how the printing should be handled. The print() function does not itself attend to the printing. Instead, if print() is called with a factor argument, print.factor() is used. For a data frame print.data.frame() is used, and so on. Section 14.10 gives further details. 2

More generally, the pattern-matching conventions are the same as for grep(), which is modeled on the Unix grep command.

22

A brief introduction to R

For objects (such as numeric vectors) that do not otherwise have a print method, print.default() handles the printing. For simple objects such as numbers and text strings, the class is determined informally. More complex objects such as data frames carry a tag (an attribute) that specifies the class. In either case, the function class() can be used to determine the class. See Section 14.10 for further details.

1.4.3 User-written functions

Here is a function that returns the mean and standard deviation of a vector of numbers: mean.and.sd <- function(x){ av <- mean(x) sdev <- sd(x) c(mean=av, SD=sdev) }

Having constructed the function, we can apply it to a numeric vector, as in the following: > distance <- c(148,182,173,166,109,141,166) > mean.and.sd(distance) mean SD 155.00 24.68

The variables av and sdev are local to the function. They cannot be accessed outside of the internal function environment. Many functions have default arguments which make it possible to run them without specifying any data. We can modify the above function to have the default argument x = rnorm(10). This generates a vector of 10 random numbers to which the function is then applied. mean.and.sd <- function(x = rnorm(10)){ av <- mean(x) sdev <- sd(x) c(mean=av, SD=sdev) }

Here is the result of one execution of the modified function: > mean.and.sd() mean SD 0.6576272 0.8595572

The structure of functions

The function mean.and.sd() has the following structure:

1.4 Functions, operators, and loops

23

argument(s) function name  
 
mean.and.sd <- function(x=rnorm(10)) { function av <- mean(x) body sdev <- sd(x) return c(av = av, sd = sdev) value } If the function body consists of just one statement that gives the return value, the curly braces ({ }) are unnecessary. The return value, which must be a single object, is given by the final statement of the function body. In the example above, the return value was the vector consisting of the two named elements mean and sdev. For returning several objects that are of different types, join them into a list.3 1.4.4 if Statements

Subsection 1.2.4 introduced the use of relational operators to create particular subsets of a given vector. The R system also has the flow control capabilities of traditional programming languages, including if statements. The if function tests the truth of a given statement; if the statement is true, the succeeding expression is evaluated. An else can be added to provide an alternative expression to be evaluated in the case where the given statement is false. For example, the following checks whether the mean for the carbon emissions exceeds the median: Carbon <- fossilfuel$carbon > if (mean(Carbon) > median(Carbon)) print("Mean > Median") else + print("Median <= Mean") [1] "Mean > Median"

Here is another example: > dist <- c(148, 182, 173, 166, 109, 141, 166) > dist.sort <- if (dist[1] < 150) + sort(dist, decreasing=TRUE) else sort(dist) > dist.sort [1] 182 173 166 166 148 141 109

1.4.5 Selection and matching

A highly useful operator is %in%, used for testing set membership. For example: > x <- rep(1:5, rep(3,5)) [1] 1 1 1 2 2 2 3 3 3 4 4 4 5 5 5 > x[x %in% c(2,4)] [1] 2 2 2 4 4 4 3

## Thus, to return the mean, SD and name of the input vector ## replace c(mean=av, SD=sdev) by list(mean=av, SD=sdev, dataset = deparse(substitute(x)))

24

A brief introduction to R

We have picked out those elements of x that are either 2 or 4. To find which elements of x are 2s, which 4s, and which are neither, use match(). Thus: > match(x, c(2,4), nomatch=0) [1] 0 0 0 1 1 1 0 0 0 2 2 2 0 0 0

The nomatch argument specifies the symbol to be used for elements that do not match. Specifying nomatch=0 is often preferable to the default, which is NA. 1.4.6 Functions for working with missing values

Recall the use of the function is.na(), discussed in Subsection 1.2.6, to identify NAs. Testing for equality with NAs does not give useful information. Identification of rows that include missing values

Many of the modeling functions will fail unless action is taken to handle missing values. Two functions that are useful for identifying or handling missing values are complete.cases() and na.omit(). Applying the complete.cases() function to a data frame returns a logical vector whose length is the number of rows and whose TRUE values correspond to rows which do not contain any missing values. Thus, the following identifies rows that hold one or more missing values: > ## Which rows have missing values: data frame science (DAAG) > science[!complete.cases(science), ] State PrivPub school class sex like Class 671 ACT public 19 1 5 19.1 672 ACT public 19 1 5 19.1

The function na.omit() omits any rows that contain missing values. For example, > dim(science) [1] 1385 7 > Science <- na.omit(science) > dim(Science) [1] 1383 7

It should be noted that there may be better alternatives to omitting missing values. There is an extensive discussion in Harrell (2001, pp. 43–51). Often, the preferred approach is to estimate the values that are missing as part of any statistical analysis. It is important to consider why values are missing – is the probability of finding a missing value independent of the values of variables that appear in the analysis? 1.4.7 ∗ Looping

A simple example of a for loop is4 4

Other looping constructs are repeat <expression> # Place break somewhere inside while (x > 0) <expression> # Or (x < 0), or etc. Here <expression> is an R statement, or a sequence of statements that are enclosed within braces.

1.5 Graphics in R

25

> for (i in 1:3) print(i) [1] 1 [1] 2 [1] 3

Here is a way to estimate the increase in population for each of the Australian states and territories between 1917 and 1997, relative to 1917, using the data frame austpop. Columns are 1: census year (by decade from 1917 through 1997); 2–9: the state and territory populations that are of interest here; and 10: the national population. > > > > > + + > > >

## Relative population increase in Australian states: 1917-1997 ## Data frame austpop (DAAG) relGrowth <- numeric(8) # numeric(8) creates a numeric vector # with 8 elements, all set equal to 0 for (j in seq(from=2, to=9)) { relGrowth[j-1] <- (austpop[9, j]-austpop[1, j])/ austpop[1, j]} names(relGrowth) <- names(austpop[c(-1,-10)]) # We have used names() to name the elements of relGrowth relGrowth # Output is with options(digits=3) NSW Vic Qld SA WA Tas NT ACT 2.30 2.27 3.98 2.36 4.88 1.46 36.40 102.33

Often, there is a better alternative to the use of a loop. See Subsection 14.5.3. 1.5 Graphics in R Later chapters will make extensive use both of base graphics (using plot(), etc.) and of the more stylized graphs provided by lattice graphics. This section is a brief introduction to plot() and allied functions that are included in R’s base graphics. Subsection 1.5.8 is a brief introduction to the more stylized graphical functions in the lattice package. Note also the carefully structured abilities of the ggplot2 package, described in Section 15.2. Base graphics are provided by the graphics package that is automatically attached at startup. It includes the function plot() for creating scatterplots, and the functions points(), lines(), text(), mtext(), and axis() that add to existing plots. There is a wide range of other functions. To see some of the possibilities, enter demo(graphics)

Press the Enter key to move to each new graph. 1.5.1 The function plot( ) and allied functions

The data frame primates gives Bodywt and Brainwt, for five primate species. A basic plot of Brainwt against Bodywt can be obtained thus: plot(Brainwt ˜ Bodywt, data=primates)

# plot(y ˜ x) syntax

or with(primates, plot(Bodywt, Brainwt))

# plot(x, y) syntax

A brief introduction to R 1500

26

500

1000

Brain weight (g)

Human

Chimp

Gorilla

Brainwt

10.0 207.0 62.0 6.8 52.2

115 406 1320 179 440

Potar monkey Gorilla Human Rhesus monkey Chimp

0

Rhesus monkey Potar monkey

Bodywt

0

50

100 150 200 250 300

Body weight (kg)

Figure 1.2 Brain weight (g) versus body weight (kg). Data are from the primates data frame.

There are many possible refinements. A number of the most important are illustrated in Plate 1, which supplements the discussion that follows. Adding points, lines, text, and axis annotation

Figure 1.2 puts labels on the points. ## Place labels on points plot(Brainwt ˜ Bodywt, xlim=c(0, 300), data=primates) # Specify xlim so that there is room for the labels with(primates, text(Brainwt ˜ Bodywt, labels=row.names(primates), pos=4)) # pos=4 places text to the right of the points. Other # possibilities are: 1: below; 2: to the left; 3: above

Figure 1.2 has two further refinements. The y-axis limits were extended slightly. Small vertical offsets were incorporated that raised the label Rhesus monkey and lowered the label Potar monkey, avoiding overlap.5 Use points() to add points to a plot. Use lines() to add lines. Actually these are aliases, differing only in the default for the parameter type; points() has type = "p", while lines() has type = "l". The function mtext(text, side, line, ...) adds text in the margin of the current plot. The sides are numbered 1 (x-axis), 2 (y-axis), 3 (top), and 4 (right vertical axis). By default, adj=0.5, which centers the text at the axis midpoint. Specify adj=0 to position the left extreme of the text at the left margin, and adj=1 to position its right extreme at the right margin. The axis() function gives fine control over axis ticks and labels. To use for the x-axis, plot the initial graph with xaxt="n". Then call axis() with the argument side=1, and with other arguments as required. See help(axis) for details. 5

## Plot Brainwt vs Bodywt, primates data frame plot(Brainwt ˜ Bodywt, xlim=c(0, 300), ylim=c(0,1500), data=primates) yoff <- c(-.125,0,0,.125,0)*par()$cxy[2] with(primates, text(x=Bodywt, y=Brainwt+yoff, labels=row.names(primates), pos=4))

1.5 Graphics in R

27

Fine control – parameter settings

Here are some of the parameters that commonly require attention: r Plotting symbols: pch (choice of symbol); cex (“character expansion"); col (color). Thus par(cex=1.2) increases the plot symbol size 20% above the default. r Lines: lty (line type); lwd (line width); col (color). r Axis limits: xlim; ylim. (Assuming xaxs="r", x-axis limits are by default extended by 4% relative to the data limits. Specify xaxs="i" to make the default an exact fit to the data limits. For the y-axis, replace xaxs by yaxs.) r Axis annotation and labels: cex.axis (character expansion for axis annotation, independently of cex); cex.labels (size of the axis labels); mgp (margin line for the axis title, axis labels, and axis line; default is mgp=c(3, 1, 0)). r Graph margins: mar (inner margins, clockwise from the bottom; the out-of-the-box default is mar=c(5.1, 4.1, 4.1, 2.1), in lines out from the axis); oma (outer margins, relevant when there are multiple graphs on the one graphics page). r Plot shape: pty="s" gives a square plot (must be set using par()). r Multiple graphs on the one graphics page: Specify par(mfrow=c(m,n)) to get an m rows by n columns layout of graphs on a page. The 1 by 4 layout of plots in Figure 2.1 of Chapter 2 was obtained using par(mfrow=c(1,4)). Type help(par) to get a (very extensive) complete list. Figure 15.1 and Plate 1 demonstrate some of the possibilities. In most (not all) instances, the change can be made either in a call to a plotting function (e.g., plot(), points()), or using par(). If made in a call to a plotting function, the change applies only to that call. If made using par(), changes remain in place until changed again, or until a new device is opened. It can be helpful to store the existing settings, so that they can be restored later. For this, specify, for example: oldpar <- par(cex=1.25) # Use par(oldpar) to restore previous settings

1.5.2 The use of color

The default palette, which can be changed, has eight colors including “white”. These are a small selection from the built-in colors. The function colors() returns the 657 names of the built-in colors, some of them aliases for the same color. In the following, points are in the colors of the current palette. These are recycled as necessary. theta <- (1:50)*0.92 plot(theta, sin(theta), col=1:50, pch=16, cex=4) points(theta, cos(theta), col=51:100, pch=15, cex=4) palette() # Names of the colors in the current palette

The following repeats the plot, but now using the function colors() to supply two sets of 50 (mostly) different colors:

28

A brief introduction to R

plot(theta, sin(theta), col=colors()[1:50], pch=16, cex=4) points(theta, cos(theta), col=colors()[51:100], pch=15, cex=4)

Where data from a two-way layout are presented on the one panel, different symbols can be used for the different levels of one of the classifying factors, with different colors used for the different levels of the other classifying factor. Care may be required in the choice of colors, so that the colors show with clarity the distinctions that are required, and do not clash. Section 15.2 has further discussion of color palettes.

1.5.3 The importance of aspect ratio

Attention to aspect ratio is often crucial for creating graphs that reveal important features of the data. The following simple graphs highlight this point: ## Plot sin(theta) vs theta, at regularly spaced values of theta ## sin() expects angles to be in radians # multiply angles in degrees by pi/180 to get radians plot((0:20)*pi/10, sin((0:20)*pi/10)) plot((1:50)*0.92, sin((1:50)*0.92))

Readers might show the second of the graphs that now follows to their friends, asking them to identify the pattern! By holding with the left mouse button on the lower border until a double-sided arrow appears and dragging upwards, the vertical dimension of the graph sheet can be shortened. If sufficiently shortened, the pattern becomes obvious. The eye has difficulty in detecting slope patterns where the slope is close to the horizontal or to the vertical. Then try this: par(mfrow=c(3,1)) # Gives a 3 by 1 layout of plots plot((1:50)*0.92, sin((1:50)*0.92)) par(mfrow=c(1,1))

See Section 2.1 for further examples.

1.5.4 Dimensions and other settings for graphics devices

The shape of the graph sheet can be set when a new graphics page is started. On Microsoft Windows systems, the function windows() (or win.graph()) starts a new graphics page on the screen display. On Unix X11 systems, specify x11(). Under Macintosh OS X, use quartz(). Available arguments include height (in inches), width (in inches), and pointsize (there are 72.27 to an inch). The choice of pointsize, with a default that varies between devices, affects character heights.6 See help(Devices) for a full list of the devices, including hardcopy devices, that are available on the particular system that is in use. 6

Note that once a graph has been pasted (from the clipboard) or imported into Microsoft Word or Open Office or another similar word processor, it can be enlarged or shrunk by pointing at one corner, holding down the left mouse button, and pulling.

1.5 Graphics in R

29

1.5.5 The plotting of expressions and mathematical symbols

In commands such as text() and mtext(), character strings can be replaced by expressions. For this purpose an expression is more general than an algebraic or mathematical expression. Thus, the following code gives a grayed out circle, overlaid as in Plate 1B with the formula for the area of a circle: symbols(x=1.5, y=0, circles=1.2, xlim=c(0,3), ylim=c(-1.5,1.5), bg="gray", inches=FALSE) # inches=FALSE ensures that radius is in x-axis units text(1.5, 0.5, expression("Area" == pi*phantom("’")*italic(r)ˆ2)) # Use ’==’ to insert ’=’. # Text or symbols that appear either side of ’*’ are juxtaposed. # Notice the use of phantom("’") to insert a small space.

By default, symbols(), like plot(), starts a new graphics frame. Various alternatives to circles are available; see help(symbols) for details.7 Type help(plotmath) to get details of available forms of expression. Run demo(plotmath) to see some of the possibilities for plotting mathematical symbols. There are further brief details in Section 15.3. Figures 5.3, 10.7, and 15.2 will demonstrate the use of expressions in annotation and/or labeling.

1.5.6 Identification and location on the figure region

Following the drawing of the initial graph, the two functions that may be used are: r identify() labels points; r locator() prints the co-ordinates of points. In either case, the user positions the cursor at the location for which co-ordinates are required, and clicks the left mouse button. Depending on the platform, the identification or labeling of points may be terminated by pointing outside of the graphics area and clicking, or by clicking with a button other than the first. If continued, the process will terminate after some default number n of points, which the user can set. (For identify() the default setting is the number of data points, while for locator() the default is 500.) As an example, identify two of the plotted points on the primates scatterplot: plot(Brainwt ˜ Bodywt, data=primates) with(primates, identify(Brainwt ˜ Bodywt, labels=row.names(primates), n=2)) # Now click near 2 plotted points

7

## To add the double-headed arrow and associated label, specify: arrows(1.5, 0, 2.7, 0, length=.1, code=3) # code=3: arrows at both ends # length is the length of the arrow head (in inches!) text(2.1, -strheight("R"), expression(italic(r) == 1.2))

30

A brief introduction to R

1.5.7 Plot methods for objects other than vectors

We have seen how to plot a numeric vector y against a numeric vector x. The plot function is a generic function that also has special methods for “plotting” various different classes of object. For example, plot() accepts a data frame as argument. Try ## Use plot() with data frame trees (datasets) plot(trees) # Gives a 3 x 3 layout of pairwise # scatterplots among the three variables

This has the same effect as the function call pairs(trees). The scatterplot matrix will be used extensively in Chapter 6 for scrutiny of regression data. See, for example, Subsection 6.2.3. It will be an important tool, also, in the account of multivariate methods in Chapter 12.

1.5.8 Lattice (trellis) graphics

Many of the analyses in later chapters compare different groups within the data. Visual assessments that complement the analysis are indispensable. The lattice package has abilities that are suited to such use. The layout on the page, the choice of plotting symbols and colors, and the distinctions within panels, can be used to represent important aspects of data structure. The syntax and graphics conventions are highly consistent across all lattice functions. Lattice’s relatively automated provision of highly structured graphical layouts has a cost – changes to the basic layout and structure may be complicated.

Lattice graphics versus base graphics – xyplot() versus plot()

A Brainwt versus Bodywt scatterplot for the primates data, such as was given earlier, might alternatively have been obtained using the function xyplot() from the lattice package. The following, when typed on the command line, give a plot on the graphics device: ## Plot Brainwt vs Bodywt, data frame primates (DAAG) plot(Brainwt ˜ Bodywt, data=primates) # base graphics # ’base’ graphics use the abilities of the graphics package library(lattice) xyplot(Brainwt ˜ Bodywt, data=primates) # lattice

The mechanism that yields the plot is different in the two cases: r plot() gives a graph as a side-effect of the command. r xyplot() generates a graphics object. As this is output to the command line, the object is “printed”, i.e., a graph appears. The following illustrates the difference between the two functions: invisible(plot(Brainwt ˜ Bodywt, data=primates)) # Graph appears invisible(xyplot(Brainwt ˜ Bodywt, data=primates)) # No graph

1.5 Graphics in R 50

Row

60

70

31 80

90

Swim

ht

190

f m

180 170 160

50

60

70

80

90

wt

Figure 1.3 Height (ht) versus weight (wt), for two categories of athlete. The different plotting symbols distinguish males from females. The data relate to Telford and Cunningham (1991).

The wrapper function invisible() suppresses command line printing, so that invisible(xyplot(...)) does not yield a graph. Inside a function, xyplot(...) prints a graph only if it is the return value from the function, i.e., usually, is on the final line. In a file that is sourced (use source()), no graph will appear. Inside a function (except as mentioned), or in a file that is sourced, there must be an explicit print(), i.e., print(xyplot(ACT ˜ year, data=austpop))

Panels of scatterplots – the use of xyplot()

Graphics functions in the lattice package are designed to allow row by column layouts of panels. Different panels are for different subsets of the data. Additionally, points can be distinguished, within panels, according to some further grouping within the data. Chapter 2 will make extensive use of lattice functions. Figure 1.3 demonstrates the use of xyplot() with the ais data set (DAAG) that has data on elite Australian athletes who trained at the Australian Institute of Sport. The plot is restricted to rowers and swimmers. The two panels distinguish the two sports, while different plotting symbols (on a color device, different colors will be used) distinguish females from males. Here is suitable code: trellis.device(color=FALSE) xyplot(ht ˜ wt | sport, groups=sex, pch=c(4,1), aspect=1, data=ais, auto.key=list(columns=2), subset=sport%in%c("Row","Swim")) dev.off() # Close device trellis.device() # Start new device, by default with color=TRUE

In the graphics formula ht ˜ wt | sport, the vertical bar indicates that what follows, in this case sport, is a conditioning variable or factor. The graphical information is broken down according to the factor levels or distinct values. The parameter aspect controls the ratio of dimensions in the y and x directions. The setting auto.key=list(columns=2) generates a simple key, with the two key items side by side in two columns rather than one under another in a single column as happens with the default setting columns=1.

32

A brief introduction to R

Plotting columns in parallel

Variables and/or factors can be plotted in parallel, on the same (outer=FALSE) or different (outer=TRUE) panel(s). Separate the names with “+”. The following gives a simplified version of Figure 2.10 in Subsection 2.1.5: xyplot(Prairies+Atlantic ˜ Date, outer=TRUE, data=jobs)

The data frame jobs has changes in number of jobs in different regions of Canada over the period January 1995 to December 1996. Subsection 15.5.1 has further discussion on the plotting of columns in parallel.

Selected lattice functions dotplot(factor ˜ numeric,..) # 1-dim. Display stripplot(factor ˜ numeric,..) # 1-dim. Display barchart(character ˜ numeric,..) histogram( ˜ numeric,..) densityplot( ˜ numeric,..) # Density plot bwplot(factor ˜ numeric,..) # Box and whisker plot qqmath(factor ˜ numeric,..) # normal probability plots splom( ˜ dataframe,..) # Scatterplot matrix parallel( ˜ dataframe,..) # Parallel coordinate plots cloud(numeric ˜ numeric * numeric, ...) # 3D surface wireframe(numeric ˜ numeric * numeric, ...) # 3D scatterplot

In each instance, users can add conditioning variables. Further points to note about the lattice package are: r Because the lattice package implements the trellis style of graphics, several of the functions that control stylistic features (color, plot characters, line type, etc.) have trellis (where lattice might have seemed more natural) as part of their name. r Lattice graphics functions cannot be mixed (or not easily) with the graphics functions discussed earlier in Section 1.5. It is not possible to use points(), lines(), text(), etc., to add features to a plot that has been created using a lattice graphics function. Instead, it is necessary to use functions that are special to lattice – lpoints(), llines(), ltext(), larrows(), and lsegments(). Subsection 15.5.5 describes a mechanism for interacting with lattice plots.

1.5.9 Good and bad graphs

There is a difference! Draw graphs so that they are unlikely to mislead. Ensure that they focus the eye on features that are important, and avoid distracting features. Lines that are intended to attract attention can be thickened. In scatterplots, the intention is typically to draw attention to the points. If there are not too many of them, the use of heavy black dots or other filled symbols will focus attention

1.6 Additional points on the use of R

33

on the points, rather than on a fitted line or curve or on the axes. If they are numerous and there is substantial overlap, it then makes better sense to use open symbols. Where there is extensive overlap, ink will fill that region more densely. If there is so much overlap that the use of black symbols would merge most points into a dense black mass, use of a shade of gray may be helpful.8 Where the horizontal scale is continuous, patterns of change that are important to identify should bank at an angle of roughly 45◦ above or below the horizontal. Depending on the context, angles in the approximate range 20◦ to 70◦ may be satisfactory, and the aspect ratio should be chosen accordingly. (This was the point of the sine curve example in Subsection 1.5.3.) See Cleveland (1994) for further commentary. Colors, or gray scales, can often be used to distinguish groupings in the data. Bear in mind that the eye has difficulty in focusing simultaneously on widely separated colors that are close together on the same graph. 1.5.10 Further information on graphics

Several further graphics functions will be introduced in Section 2.1. Note especially hist() and boxplot(). See also Murrell (2005), Sarkar (2002, 2007). Note the more detailed information in Chapter 15. Section 15.5 has an extended discussion of the abilities of the lattice package. There is brief reference to the relatively specialist abilities of the grid package, or which lattice is built. Section 15.6 discusses the ggplot2 package. 1.6 Additional points on the use of R ∗

Workspace management strategies

The default choice of working directory, which may be an R installation directory, is not a good choice for long-term use, and should be changed. Subsection 1.1.1 explained how to set the startup choice of working directory. The working directory can be changed and a new workspace loaded in the course of a session, either using the menu system, if available, or using command line instructions. See Subsection 14.2.2. In a session where there are extensive calculations, cautious users will from time to time save the current workspace, perhaps first using rm() to remove objects that are no longer required. The command save.image() will save everything in the workspace, by default into the file .RData in the working directory. This can alternatively be done by clicking on the relevant menu item, where such a menu is available. (The file that is saved holds an image of the workspace at that point.) It is good practice to use a separate working directory for each different project. The ability to keep multiple image files in the one directory adds further flexibility. Use the extension .RData for such files.9 8 9

## Example of plotting with different shades of gray plot(1:4, 1:4, pch=16, col=c("gray20","gray40","gray60","gray80"), cex=3) In older versions of R for Windows, .rda was an alternative extension.

34

A brief introduction to R

Forward slashes and backslashes

Note that R syntax follows the Unix conventions and uses forward slashes, where Windows expects backslashes. Thus to read in the file fuel.txt from the directory c:\data, type fossilfuel <- read.table("c:/data/fuel.txt") # Alternative: Replace each "/" by "\\", ie, 2 backslashes Setting the number of decimal places in output

Often, calculations will, by default, give more decimal places of output than are useful. In the output that we give, we often reduce the number of decimal places below what R gives by default. The options() function can be used to make a global change to the number of significant digits that are printed. For example: > sqrt(10) [1] 3.162278 > options(digits=2)

# Change until further notice, # or until end of session.

> sqrt(10) [1] 3.2

Note that options(digits=2) expresses a wish, which R will not always obey! Rounding will sometimes introduce small inconsistencies. For example, in the calculations of Section 4.4:  372 = 5.57 12  √ 372 2× = 7.88. 12 √ Note however that 2 × 5.57 = 7.87. Other option settings

Type help(options) to get further details. We will meet another important option setting in Chapter 5. (Most of the output that we present uses the setting options(show. signif.stars=FALSE), where the default is TRUE. This affects output in Chapter 5 and later chapters.) Cosmetic issues

In our R code, we write, e.g., a <- b rather than a<-b, and y ∼ x rather than y∼x. This is intended to help readability, perhaps a small step on the way to literate programming. Such presentation details can make a large difference when others use the code. Where output is obtained with the simple use of print() or summary(), we have in general included this as the first statement in the output.

1.7 Recap

35

∗

Common sources of difficulty

r It is important to tune the parameter settings of read.table() to the input data set. See help(read.table) and Subsection 14.4.1 for further details. r Character vectors that are included as columns in data frames become, by default, factors. There are implications for the use of read.table(). See Subsection 14.4.1 and Section 14.6. r In most contexts, factors are treated as vectors of character strings, with values given by the factor levels. Use unclass() to extract the integer values if these, rather than the levels are required. See Section 14.6. r Keep in mind Section 14.7’s comments on the handling of missing values. r The syntax fossilfuel[, 2] extracts the second column from the data frame fossilfuel, yielding a numeric vector. Observe however that fossilfuel[2, ] yields a data frame, rather than the numeric vector that the user may require. Specify unlist(fossilfuel[2, ]) to obtain the vector of numeric values in the second row of the data frame. See Subsection 14.9.1. r The function sapply() is commonly used to carry out a computation across all elements of a data frame. If used with a matrix, the computation will be carried out on all matrix elements. See Subsection 14.9.5. r Once a data frame has been attached, take care with assignments to either the name of the data frame, or the name of a column. Assignment to the name of the data frame will create a new local copy, while assigment to a column name will create a new object in the workspace with that name. Later references to those names will then access the new local copies. r Data objects that individually or in combination occupy a large part of the available computer memory can slow down all memory-intensive computations. See Subsection 14.2.2 for comment on associated workspace management issues. See also the opening comments in Section 14.8. Note that most of the data objects that are used for our examples are small and thus will not, except where memory is very small, make much individual contribution to demands on memory.

Variable names in data sets

We will refer to a number of different data sets, many of them data frames in our DAAG package. When we first introduce the data set, we will give both a general description of the columns of values that we will use, and the names used in the data frame. In later discussion, we will use the name that appears in the data frame whenever the reference is to the particular values that appear in the column.

1.7 Recap r One use of R is as a calculator, to evaluate arithmetic expressions. Calculations can be carried out in parallel, across all elements of a vector at once. r Use q() to quit from R, usually taking care to save the workspace.

36

A brief introduction to R

r Help functions include help() (help page of a known function), help.search() (search for a specified word in the help page header), apropos() (search for function names that include a specified character string), and help.start (start a browser interface to help information). r The function c() (concatenate) joins vector elements into vectors. It may be used for logical and character vectors, as well as for numeric vectors. r For simple forms of scatterplot use plot(), or the lattice function xyplot(). r Important R data structures are vectors, factors, lists, and data frames. Vectors may be of mode numeric, or logical, or character. Factors have mode “numeric” and class “factor”. r Data frames use a list structure to group columns, which must all have the same length, together into a single R object. The different columns may be any mix of logical, numeric, character, or factor. r Contrast data frames with matrices. All matrix elements have the same mode. A matrix is stored as one long vector that is formatted to appear in a row by column layout. r Use is.na() to identify elements that are NAs. r The R system has extensive abilities for inputting data from rectangular files (see help(read.table)), from spreadsheets, and from a variety of statistical package formats. The R Commander GUI offers an easy means to access these abilities. r Use attach() or with() (temporary attachment) to give access to the columns of a data frame, without the need to name the data frame whenever a column is accessed. r The search path determines the order of search for objects that are accessed from the command line, or that are not found in the enclosing environment of a function that accesses them. r Factors, used for categorical data, are fundamental to the use of many of the R modeling functions. Ordered factors are appropriate for use with ordered categorical data. r Option settings, which users can change at their discretion, control such matters as the number of significant digits that will be displayed in output. r Commonly used generic functions include print(), plot(), and summary(). For such functions, the result depends on the class of object that is given as argument. r To make an assignment and print the value that is assigned, enclose the assigment statement in round brackets. For example: (x <- 2)

# Equivalent to: x <- 2; print(x)

1.8 Further reading Note that the exercises have various hints that extend the discussion in the body of the chapter. A version of An Introduction to R (R Development Core Team, 2009a), current at the time of release, is included with the R distributions. It is available from the CRAN sites as an independent document. (For a list of sites, go to http://cran.r-project.org.) Books that include an introduction to R include Dalgaard (2008), Fox (2002). At a more advanced level note Venables and Ripley (2002), which covers both S-PLUS and R. This will be an important reference throughout this book.

1.9 Exercises

37

See also documents, including Maindonald (2008), that are listed under Contributed Documentation on the CRAN sites. For careful detailed accounts of the R language, see Chambers (2007), Gentleman (2008). Books and papers that set out principles of good graphics include Cleveland (1993, 1994), Tufte (1997), Wainer (1997), and Wilkinson and Task Force on Statistical Inference (1999). See also the imaginative uses of R’s graphical abilities that are demonstrated in Murrell (2005). Maindonald (1992) comments very briefly on graphical design.

References for further reading

Chambers, J. M. 2007. Software for Data Analysis: Programming with R. Cleveland, W. S. 1993. Visualizing Data. Cleveland, W. S. 1994. The Elements of Graphing Data, revised edn. Dalgaard, P. 2008. Introductory Statistics with R. Fox, J. 2002. An R and S-PLUS Companion to Applied Regression. Gentleman, R. 2008. R Programming for Bioinformatics. Maindonald, J. H. 1992. Statistical design, analysis and presentation issues. New Zealand Journal of Agricultural Research 35: 121–41. Maindonald, J. H. 2008. Using R for Data Analysis and Graphics. Available as a pdf file at http://www.maths.anu.edu.au/˜johnm/r/usingR.pdf Murrell, P. 2005. R Graphics. http://www.stat.auckland.ac.nz/˜paul/RGraphics/rgraphics.html R Development Core Team. 2009a. An Introduction to R. Tufte, E. R. 1997. Visual Explanations. Venables, W. N. and Ripley, B. D. 2002. Modern Applied Statistics with S, 4th edn. Wainer, H. 1997. Visual Revelations. Wilkinson, L. and Task Force on Statistical Inference. 1999. Statistical methods in psychology journals: guidelines and explanation. American Psychologist 54: 594–604. See the references at the end of the book for fuller bibliographic details.

1.9 Exercises 1. The following table gives the size of the floor area (ha) and the price ($A000), for 15 houses sold in the Canberra (Australia) suburb of Aranda in 1999. area sale.price 1 694 192.0 2 905 215.0 3 802 215.0 4 1366 274.0 5 716 112.7 6 963 185.0 7 821 212.0 8 714 220.0 9 1018 276.0 10 887 260.0

38

A brief introduction to R 11 790 221.5 12 696 255.0 13 771 260.0 14 1006 293.0 15 1191 375.0 Type these data into a data frame with column names area and sale.price. (a) (b) (c) (d)

Plot sale.price versus area. Use the hist() command to plot a histogram of the sale prices. Repeat (a) and (b) after taking logarithms of sale prices. The two histograms emphasize different parts of the range of sale prices. Describe the differences.

2. The orings data frame gives data on the damage that had occurred in US space shuttle launches prior to the disastrous Challenger launch of 28 January 1986. The observations in rows 1, 2, 4, 11, 13, and 18 were included in the pre-launch charts used in deciding whether to proceed with the launch, while remaining rows were omitted. Create a new data frame by extracting these rows from orings, and plot total incidents against temperature for this new data frame. Obtain a similar plot for the full data set. 3. For the data frame possum (DAAG package) (a) Use the function str() to get information on each of the columns. (b) Using the function complete.cases(), determine the rows in which one or more values is missing. Print those rows. In which columns do the missing values appear? 4. For the data frame ais (DAAG package) (a) Use the function str() to get information on each of the columns. Determine whether any of the columns hold missing values. (b) Make a table that shows the numbers of males and females for each different sport. In which sports is there a large imbalance (e.g., by a factor of more than 2:1) in the numbers of the two sexes? 5. Create a table that gives, for each species represented in the data frame rainforest, the number of values of branch that are NAs, and the total number of cases. [Hint: Use either !is.na() or complete.cases() to identify NAs.] 6. Create a data frame called Manitoba.lakes that contains the lake’s elevation (in meters above sea level) and area (in square kilometers) as listed below. Assign the names of the lakes using the row.names() function. elevation area Winnipeg 217 24387 Winnipegosis 254 5374 Manitoba 248 4624 SouthernIndian 254 2247 Cedar 253 1353 Island 227 1223 Gods 178 1151 Cross 207 755 Playgreen 217 657

1.9 Exercises

39

(a) Use the following code to plot log2(area) versus elevation, adding labeling information (there is an extreme value of area that makes a logarithmic scale pretty much essential): attach(Manitoba.lakes) plot(log2(area) ˜ elevation, pch=16, xlim=c(170,280)) # NB: Doubling the area increases log2(area) by 1.0 text(log2(area) ˜ elevation, labels=row.names(Manitoba.lakes), pos=4) text(log2(area) ˜ elevation, labels=area, pos=2) title("Manitoba’s Largest Lakes") detach(Manitoba.lakes) Devise captions that explain the labeling on the points and on the y-axis. It will be necessary to explain how distances on the scale relate to changes in area. (b) Repeat the plot and associated labeling, now plotting area versus elevation, but specifying log="y" in order to obtain a logarithmic y-scale. [Note: The log="y" setting carries across to the subsequent text() commands. See Subsection 2.1.5 for an example.] 7. Look up the help page for the R function dotchart(). Use this function to display the areas of the Manitoba lakes (a) on a linear scale, and (b) on a logarithmic scale. Add, in each case, suitable labeling information. 8. Using the sum() function, obtain a lower bound for the area of Manitoba covered by water. 9. The second argument of the rep() function can be modified to give different patterns. For example, to get four 2s, then three 3s, then two 5s, enter rep(c(2,3,5), c(4,3,2)) (a) What is the output from the following command? rep(c(2,3,5), 4:2) (b) Obtain a vector of four 4s, four 3s, and four 2s. (c) The argument length.out can be used to create a vector whose length is length.out. Use this argument to create a vector of length 50 that repeats, as many times as necessary, the sequence: 3 1 1 5 7 (d) The argument each can be used to form a vector in which each element in the first argument is replaced by the specified number of repeats of itself. Use this to create a vector in which each of 3 1 1 5 7 is replaced by four repeats of itself. Show, also, how this can be done without use of the argument each. 10. The ˆ symbol denotes exponentiation. Consider the following: 1000*((1+0.075)ˆ5 - 1) # Interest on $1000, compounded # annually at 7.5% p.a. for five years (a) Evaluate the above expression. (b) Modify the expression to determine the amount of interest paid if the rate is 3.5% p.a. (c) Explain the result obtained when the exponent 5 is changed to seq(1, 10). 11. Run the following code: gender <- factor(c(rep("female", 91), rep("male", 92))) table(gender) gender <- factor(gender, levels=c("male", "female")) table(gender)

40

A brief introduction to R gender <- factor(gender, levels=c("Male", "female")) # Note the mistake: "Male" should be "male" table(gender) table(gender, exclude=NULL) rm(gender) # Remove gender Explain the output from the successive uses of table().

12. Write a function that calculates the proportion of values in a vector x that exceed some value cutoff. (a) Use the sequence of numbers 1, 2, . . . , 100 to check that this function gives the result that is expected. (b) Obtain the vector ex01.36 from the Devore6 (or Devore7) package. These data give the times required for individuals to escape from an oil platform during a drill. Use dotplot() to show the distribution of times. Calculate the proportion of escape times that exceed 7 minutes. 13. The following plots four different transformations of the Animals data from the MASS package. What different aspects of the data do these different graphs emphasize? Consider the effect on low values of the variables, as contrasted with the effect on high values. par(mfrow=c(2,2)) # 2 by 2 layout on the page library(MASS) # Animals is in the MASS package plot(brain ˜ body, data=Animals) plot(sqrt(brain) ˜ sqrt(body), data=Animals) plot(I(brainˆ0.1) ˜ I(bodyˆ0.1), data=Animals) # I() forces its argument to be treated "as is" plot(log(brain) ˜ log(body), data=Animals) par(mfrow=c(1,1)) # Restore to 1 figure per page 14. Use the function abbreviate() to obtain six-character abbreviations for the row names in the data frame cottonworkers (DAAG package). Plot survey1886 against census1886, and plot avwage*survey1886 against avwage*census1886, in each case using the six-letter abbreviations to label the points. How should each of these graphs be interpreted? [Hint: Be sure to specify I(avwage*survey1886) and I(avwage*census1886) when plotting the second of these graphs.] 15. The data frame socsupport (DAAG) has data from a survey on social and other kinds of support, for a group of university students. It includes Beck Depression Inventory (BDI) scores. The following are two alternative plots of BDI against age: plot(BDI ˜ age, data=socsupport) plot(BDI ˜ unclass(age), data=socsupport) For examination of cases where the score seems very high, which plot is more useful? Explain. Why is it necessary to be cautious in making anything of the plots for students in the three oldest age categories (25-30, 31-40, 40+)? 16. Functions that can be useful for labeling points on graphs are abbreviate() (create abbreviated names), and paste() (create composite labels). A composite label might, for the data from socsupport, give information about gender, country, and row number. Try the following: gender1 <- with(socsupport, abbreviate(gender, 1)) table(gender1) # Examine the result

1.9 Exercises

41

country3 <- with(socsupport, abbreviate(country, 3)) table(country3) # Examine the result Now use the following to create a label that can be used with text() or with identify(): num <- with(socsupport, seq(along=gender)) # Generate row numbers lab <- paste(gender1, country3, num, sep=":")

Use identify() to place labels on all the points that the boxplots have identified as “outliers”. 17. Given a vector x, the following demonstrates alternative ways to create a vector of numbers from 1 through n, where n is the length of the vector: x <- c(8, 54, 534, 1630, 6611) seq(1, length(x)) seq(along=x) Now set x <- NULL and repeat each of the calculations seq(1, length(x)) and seq(along=x). Which version of the calculation should be used in order to return a vector of length 0 in the event that the supplied argument is NULL. 18. The Rabbit data frame in the MASS library contains blood pressure change measurements on five rabbits (labeled as R1, R2, . . . ,R5) under various control and treatment conditions. Read the help file for more information. Use the unstack() function (three times) to convert Rabbit to the following form: Treatment Dose R1 R2 R3 R4 R5 1 Control 6.25 0.50 1.00 0.75 1.25 1.5 2 Control 12.50 4.50 1.25 3.00 1.50 1.5 . . . . 6 Control 200.00 32.00 29.00 24.00 33.00 18.0 7 MDL 6.25 1.25 1.40 0.75 2.60 2.4 8 MDL 12.50 0.75 1.70 2.30 1.20 2.5 . . . . 12 MDL 200.00 37.00 28.00 25.00 22.00 19.0 19. The data frame vlt (DAAG) consists of observations taken on a video lottery terminal during a two-day period. Eight different objects can appear in each of three windows. Here, they are coded from 0 through 7. Different combinations of the objects give prizes (although with small probability). The first four rows are: > head(vlt, 4) # first few rows of vlt window1 window2 window3 prize night 1 2 0 0 0 1 2 0 5 1 0 1 3 0 0 0 0 1 4 2 0 0 0 1 > # . . . Use stack() to convert the first three columns of this data set to a case-by-variable format, then creating a tabular summary of the results, broken down by window. vltcv <- stack(vlt[, 1:3]) head(vltcv) # first few rows of vltcv table(vltcv$values, vltcv$ind) # More cryptically, table(vltcv) gives the same result. Does any window stand out as different? 20.∗ The help page for iris (type help(iris)) gives code that converts the data in iris3 (datasets package) to case-by-variable format, with column names “Sepal.Length”,

42

A brief introduction to R “Sepal.Width”, “Petal.Length”, “Petal.Width”, and “Species”. Look up the help pages for the functions that are used, and make sure that you understand them. Then add annotation to this code that explains each step in the computation.

21.∗ The following uses the for() looping function to plot graphs that compare the relative population growth (here, by the use of a logarithmic scale) for the Australian states and territories. oldpar <- par(mfrow=c(2,4)) for (i in 2:9){ plot(austpop[, 1], log(austpop[, i]), xlab="Year", ylab=names(austpop)[i], pch=16, ylim=c(0,10))} par(oldpar) Find a way to do this without looping. [Hint: Use the function sapply(), with austpop[,2:9] as the first argument.]

2

Styles of data analysis

What is the best way to begin investigation of a new set of data? What forms of data exploration will draw attention to obvious errors or quirks in the data, or to obvious clues that the data contain? What checks are desirable before proceeding with an intended formal analysis, or to help decide what formal analysis may be appropriate? What can be learned from investigations that other researchers have done with similar data? Competent statisticians have always used graphs to check their data. Numerical summaries, such as an average, can be very useful, but important features of the data may be missed without a glance at an appropriate graph. Careful consideration may be needed to choose a graph that will be effective for the purpose in hand. We will see in Chapter 3 that an integral part of statistical analysis is the development of a model that accurately describes the data, clarifies what the data say, and makes prediction possible. Without model assumptions, there cannot be a meaningful formal analysis! As assumptions are strengthened, the chances of getting clear results improve. The price for stronger assumptions is that, if wrong, the results may be wrong. Graphical techniques have been developed for checking, to the extent possible, many of the assumptions that must be made in practice. Preliminary scrutiny of the data can readily degenerate into data snooping, so that the analysis is unduly attuned to statistical artefacts of the particular data that are to be analyzed. Under torture, the data readily yield false confessions. To avoid this, strict limits must be placed on the extent to which the data are allowed to influence the choice of model for the formal analysis. Even if data have not been collected in a way that makes them suitable for formal statistical analysis, exploratory techniques can often glean clues from them. However, it is unwise, as too often happens, to rely on this possibility! 2.1 Revealing views of the data The use of graphs to display and help understand data has a long tradition. John W. Tukey formalized and extended this tradition, giving it the name Exploratory Data Analysis (EDA). Tukey has had a huge influence on data analysis practices; see Hoaglin (2003). A key concern is that data should, as far as possible, have the opportunity to speak for themselves, prior to or as part of a formal analysis. A use of graphics that is broadly in an EDA tradition continues to develop and evolve. Statistical theory has an important role in suggesting forms of display that may be helpful

44

Styles of data analysis

and interpretable. Advances in computing have been important, facilitating the development and use of many of the graphical tools now available. The best modern statistical software makes a strong connection between data analysis and graphics, combining the computer’s ability to crunch numbers and present graphs with the ability of a trained human eye to detect pattern. Graphical exploration after the style of EDA has at least four roles: r It may suggest ideas and understandings that had not previously been contemplated. This use of EDA fits well with the view of science as inductive reasoning. r It may challenge the theoretical understanding that guided the initial collection of the data. It then acquires a more revolutionary role. It becomes the catalyst, in the language of Thomas Kuhn, for a paradigm shift. r It allows the data to criticize an intended analysis and facilitates checks on assumptions. Subsequent formal analysis can then proceed with greater confidence. r It may reveal additional information, not directly related to the research question. It may, for example, suggest fruitful new lines of research. The next several subsections will describe the histogram and density plot, the stem-andleaf display, the boxplot, the scatterplot, the lowess smoother, and the trellis-style graphics that are available in the lattice package. The lattice functions greatly extend the available styles and layouts. 2.1.1 Views of a single sample Histograms and density plots

The histogram is a basic (and over-used) EDA tool for displaying the frequency distribution of a set of data. The area of each rectangle of a histogram is proportional to the number of observations whose values lie within the width of the rectangle. A mound-shaped histogram may make it plausible that the data follow a normal distribution (the “bell curve”). In small samples, however, the shape can be highly irregular. In addition, the appearance can depend on the choice of breakpoints, which is a further reason for caution in interpreting the shape. It is often helpful to try more than one set of breakpoints. The data set possum (DAAG package) has nine morphometric measurements on each of 104 mountain brushtail possums, trapped at seven sites from southern Victoria to central Queensland (data relate to Lindenmayer et al., 1995). Attention will be limited to the measurements for 43 females, placing them in a subset data frame that will be called fossum. The following code creates this subset data frame: library(DAAG) # Ensure that the DAAG package is attached ## Form the subset of possum that holds data on females only fossum <- subset(possum, sex=="f")

Panels A and B of Figure 2.1 exhibit histogram plots of the frequency distribution of the total lengths of the female possums.1 1

## To get a 1 by 4 layout, precede with par(mfrow = c(1,4))

2.1 Revealing views of the data B

Breaks as in B

70

70

0.08

Density

10 15 20

D

Breaks as in A

0.00

5 0

Frequency

C

Breaks at 75, 80, ...

0.04

A Breaks at 72.5, 77.5, ...

45

70

80

90 100

Total length (cm)

70

80

90 100

Total length (cm)

80

90 100

Total length (cm)

80

90 100

Total length (cm)

Figure 2.1 The histograms in panels A and B show the same data, but with a different choice of breakpoints. In panels C and D, density plots are overlaid on the histograms from panels A and B, respectively. attach(fossum) hist(totlngth, breaks = xlab="Total length hist(totlngth, breaks = xlab="Total length

72.5 + (0:5) * 5, ylim = c(0, 22), (cm)", main ="A: Breaks at 72.5, 77.5, ...") 75 + (0:5) * 5, ylim = c(0, 22), (cm)", main="B: Breaks at 75, 80, ...")

The only difference in the construction of the two plots is the choice of breakpoints, but one plot suggests that the distribution is asymmetric (skewed to the left), while the other suggests symmetry. A histogram is a crude form of a density estimate. A better alternative is, often, a smooth density estimate, as in Figures 2.1C and D. Whereas the width of histogram bars must be chosen somewhat subjectively, density estimates require the choice of a bandwidth parameter that controls the amount of smoothing. In both cases, the software has default choices that can work reasonably well. dens <- density(totlngth) xlim <- range(dens$x); ylim <- range(dens$y) hist(totlngth, breaks = 72.5 + (0:5) * 5, probability = T, xlim = xlim, ylim = ylim, xlab="Total length (cm)", main=" ") lines(dens) hist(totlngth, breaks = 75 + (0:5) * 5, probability = T, xlim = xlim, ylim = ylim, xlab="Total length (cm)", main= " ") lines(dens) par(mfrow=c(1,1)); detach(fossum)

The height of the density curve at any point is an estimate of the proportion of sample values per unit interval, locally at that point. Observe that in Figures 2.1A and C, the cell of the histogram between the breakpoints (87.5, 92.5] has a frequency of 22. As the total 22 = 0.102, frequency is 43, and the width of the cell is 5, this corresponds to a density of 43×5 which is just a little smaller than the height of the density curve at its highest point or mode. Much of the methodology in this book makes assumptions that hold exactly only if the data follow a normal distribution (the “bell curve”), discussed in the next chapter. Density curves are preferable to histograms for drawing attention to particular forms of

46

Styles of data analysis The decimal point is 1 digit(s) to the right of the | 15 16 16 17 17 18 18 19 19

| | | | | | | | |

6 5 4 5678899 00000011223 55666668899 123 58

Lower quartile is 179 (10th largest) Median is 182 (19th largest)

Upper quartile is 186 (28th largest)

Figure 2.2 Stem-and-leaf display showing the heights of the 37 rowers in the ais data set. Annotation has been added that identifies the lower quartile, the median, and the upper quartile.

non-normality, such as that associated with strong skewness in the distribution, but are still not an adequate tool. A more effective way of checking for normality – the normal probability plot – is described in Subsection 3.4.2. Density curves are useful for estimating the population mode, i.e., the value that occurs most frequently. Where data values have sharp lower and/or upper cutoff limits, use the arguments from and to to specify those limits. For example, a failure time distribution may have a mode close to zero, with a sharp cutoff at zero.

The stem-and-leaf display

The stem-and-leaf display is a fine-grained alternative to a histogram, for use in displaying a single column of numbers. Figure 2.2 shows a stem-and-leaf plot of the heights of the 37 rowers in the ais data set. Code is: with(ais, stem(ht[sport=="Row"]))

The data have been rounded to the nearest centimeter. The numbers that are displayed are, in order of magnitude, 156, 165, 174, . . . . The display has broken these down as 150 + 6, 160 + 5, 170 + 4, . . . . The column of numbers on the left of the vertical bars (15, 16, . . . ) comprises the stem; these are the tens of centimeters parts of the numbers. The leaf part for that number (6, 5, 4, . . . ) is what remains after removing the stem; these are printed, in order, to the right of the relevant vertical bar. As there are 37 data values, the median or middle value is the 19th. Starting from the 156 leaf in the first line of the stem-and-leaf diagram and working down, 18 values precede the 19th largest, and 18 values follow. Thus the median (or 50th percentile) is 182. The first and third quartiles (the 25th and 75th percentiles) can be recovered in a similar way, with the exact value depending on the details of the formula used for their calculation. For present purposes the first quartile can be taken as the 10th largest value (= 179), while the third quartile is the 28th largest value (= 186), or the 10th value when starting at the largest and counting down. (The number 10 is the average of the ranks 1 and 19, while 28 is the average of 19 and 39.)2 2

## Use quantile() to obtain the quartiles of ht: data frame ais (DAAG package) quantile(ais$ht[ais$sport=="Row"], prob=c(.25,.5,.75)) # For the 50th percentile (the 2nd quartile), an alternative is median()

85.25

75

80

85

Largest value (there are no outliers)

upper quartile

median

lower quartile

Smallest value (outliers excepted)

Outlier

2.1 Revealing views of the data

90.5

90

47

Inter–quartile range = 90.5 – 85.25 = 5.25 Compare 0.75 x Inter–quartile range = 3.9 with standard deviation = 4.2

95

Total length (cm)

Figure 2.3

Boxplot, with annotation that explains boxplot features.

Boxplots

Like the histogram, the boxplot is a coarse summary. It allows a trained eye to comprehend at a glance specific important features of the data. Figure 2.3 shows a boxplot of total lengths of females in the possum data set, with annotation added that explains the interpretation of boxplot features. Code that gives the boxplot, without the annotation, is: ## Base graphics boxplot function with(fossum, boxplot(totlngth, horiz=TRUE)) ## Alternative: lattice graphics bwplot function bwplot(˜totlngth, data=fossum)

Notice that one point lies outside the “whiskers” to the left, and is thus flagged as a possible outlier. An outlier is a point that, in some sense, lies away from the main body of the data. In identifying points that are flagged as possible outliers, the normal distribution (to be discussed in Subsection 3.2.2) is taken as the standard. Using the default criterion one point in 100 will on average, for data from a normal distribution, be flagged as a possible outlier. Thus, in a boxplot display of 1000 values that are drawn at random from a normal distribution, around 10 will be plotted out beyond the boxplot whiskers and thus flagged as possible outliers. Subsection 2.1.7 has further comment on outliers. The discussion of the normal and other distributions in Chapter 3 should help clarify these ideas.

2.1.2 Patterns in univariate time series

In Figure 2.4, “measles” includes both what is nowadays called measles and the closely related rubella or German measles.3 Panel A uses a logarithmic vertical scale. Panel B uses an unlogged scale and takes advantage of the fact that deaths from measles are of the order, in any year, of one thousandth of the population. Thus, deaths in thousands and population in millions can be shown on the same scale. 3

For details of the data, and commentary, see Guy (1882), Stocks (1942), Senn (2003). (Guy’s interest was in the comparison with smallpox mortality.) The population estimates (londonpop) are from Mitchell (1988).

48

Styles of data analysis

Deaths; Population (log scale)

A (1629−1939) 5e+06 1e+06

●

●

●

●

●

●

●

●

●

●

●

●

●

1000 100 10 1 1650

Deaths; Population in 1000s

●

1700

1750

1800

1850

1900

1950

B (1841−1881) 4000

● ●

3000

● ●

2000

●

1000

1840

1850

1860

1870

1880

Figure 2.4 The two panels provide different insights into data on mortality from measles, in London over 1629–1939. Panel A shows the numbers of deaths from measles in London for the period from 1629 through 1939 (black curve) and the London population (in thousands, black dots). A log scale has been used (see Subsection 2.1.3 for details). The lower panel B shows the subset of the measles data for the period 1840 through 1882 on the linear scale (black curve), together with the London population (in thousands, black dots).

Simplified code is: ## Panel A plot(log10(measles), xlab="", ylim=log10 (c(1,5000*1000)), ylab=" Deaths; Population (log scale)", yaxt="n") ytiks <- c(1, 10, 100, 1000, 1000000, 5000000) ## London population in thousands londonpop
The function plot() recognizes that measles is a time series object, and calls the plot method plot.ts() that is used for time series. For details, see help(plot.ts). Notice the use, for panel B, of the function window() that extracts a subseries.

49

2

3

four 4 5

6

7

2.1 Revealing views of the data

2

3

4 5 one

6

7

Figure 2.5 Each of 17 panelists compared two milk samples for sweetness. One sample had one unit of additive, while the other had four units of additive.

Panel A shows broad trends over time, but is of no use for identifying changes on the time scale of a year or two. In panel B, the lines that show such changes are, mostly, at an angle that is in the approximate range of 20◦ to 70◦ from the horizontal. A sawtooth pattern, by which years in which there are many deaths are commonly followed by years in which there are fewer deaths, is thus clearly evident. (To obtain this level of detail for the whole period from 1629 until 1939, multiple panels would be necessary.) The following, with y-axis labeling in logarithms of numbers and omitting the population estimates, demonstrates the combining of the two graphs on the one page: ## Panel A: par(fig=c(0, 1, .38, 1)) # 38% to 100% of page, in y-direction plot(log10(measles), ylab="log10(Deaths)", ylim=log10(c(1,5000*1000))) mtext(side=3, line=0.5, "A (1629-1939)", adj=0) ## Panel B: window from 1840 to 1882; more complete code par(fig=c(0, 1, 0, .4), new=TRUE) # 0% to 38% of height of figure region plot(window(measles, start=1840, end=1882), ylab="Deaths") mtext(side=3, line=0.5, "B (1841-1881)", adj=0) par(fig=c(0, 1, 0, 1)) # Restore default figure region

2.1.3 Patterns in bivariate data

The scatterplot is a simple but important tool for the examination of pairwise relationships. We will illustrate with specific examples. Figure 2.5 shows data from a tasting session where each of 17 panelists assessed the sweetness of each of two milk samples, one with four units of additive, and the other with one unit of additive. The line y = x has been added. The function rug() adds a “rug”, i.e., short bars at right angles to one or other axis that show the distribution values along that axis of the plot. The code is: ## Plot four vs one: data frame milk (DAAG) xyrange <- range(milk) plot(four ˜ one, data = milk, xlim = xyrange, ylim = xyrange, pch = 16, pty="s") # pty="s": square plotting region

50

Styles of data analysis A 10 8 2

4

6

8 6 4 2

Resistance (kOhm)

10

B

10 30 50 Apparent juice content (%)

10 30 50 Apparent juice content (%)

Figure 2.6 Electrical resistance versus apparent juice content. Panel B repeats panel A, but with a smooth curve fitted to the data. rug(milk$one) rug(milk$four, side = 2) abline(0, 1)

# x-axis rug (default is side=1) # y-axis rug

There is a positive correlation between assessments for the two samples; if one was rated as sweet, by and large so was the other. The line y = x assists in comparing the two samples. Most panelists (13 out of 17) rated the sample with four units of additive as sweeter than the sample with one unit of additive.

The fitting of a smooth trend curve

Figure 2.6 shows data from a study that measured both electrical resistance and apparent juice content for a number of slabs of kiwifruit. The curve in panel B, obtained using the lowess method that is discussed further in Subsection 7.5.4, estimates the relationship between electrical resistance and apparent juice content. The code is: ## Plot ohms vs juice: data frame fruitohms (DAAG) plot(ohms ˜ juice, xlab="Apparent juice content (%)", ylab="Resistance (ohms)", data=fruitohms) ## Add a smooth curve, as in Panel B with(fruitohms, lines(lowess(juice, ohms), lwd=2)) # With lwd=2, the curve is twice the default thickness

The fitted smooth curve shows a form of response that is clearly inconsistent with a straight line. It suggests an approximate linear relationship for juice content up to somewhat over 35%. Once the juice content reaches around 45%, the curve becomes a horizontal line, and there is no evident further change in resistance. There is no obvious simple form of equation that might be used to describe the curve. A curve fitted using lowess() or another such smoothing function can provide a useful benchmark against which to compare the curve given by a theoretical or other mathematical form of equation that the data are thought to follow.

2.1 Revealing views of the data

51

log10(Body weight) A

B

−3

−1

0

1

2

3

●

0 200 600 Body weight (kg x 100)

4 3 2 0

● ● ●

●

● ●

1

1000 10 ●

● ● ●●

0.001

0.1 1

10

log10(Brain weight)

● ● ● ● ●●

●● ● ● ● ● ●● ● ●●●● ●● ●

−1

●

0.1 1

●

Brain weight (g)

4000 2000 0

Brain weight (g)

●

1000

Body weight (kg x 100)

Figure 2.7 Brain weight versus body weight, for 27 animals that vary greatly in size. Panel A uses untransformed scales, while panel B uses logarithmic scales, on both axes.

What is the appropriate scale?

Figures 2.7A and B plot brain weight (g) against body weight (kg), for a number of different animals: ## The following omits the labeling information oldpar <- par(mfrow = c(1,2), pty="s") ## Plot brain vs body: data frame Animals (MASS package) library(MASS) plot(brain ˜ body, data=Animals) # Panel A plot(log(brain) ˜ log(body), data=Animals) # Panel B par(oldpar)

Figure 2.7A is almost useless. The axes should be transformed so that the data are spread out more evenly. Here, we can do this by choosing a logarithmic scale. Multiplication by the same factor (e.g., for the tick marks in Figure 2.7B, by a factor of 10) always gives the same distance along the scale. If we marked points 1, 5, 25, 125, . . . along the vertical axis, they would also lie an equal distance apart. A logarithmic scale is appropriate for quantities that change multiplicatively. For example, if cells in a growing organism divide and produce new cells at a constant rate, then the total number of cells changes in a multiplicative manner, resulting in so-called exponential growth. Growth in the bodily measurements of organisms may similarly be multiplicative, with large organisms increasing in some time interval by the same approximate fraction as smaller organisms. Random changes in the relative growth rate will produce adult organisms whose size (e.g., height) is, on the logarithmic scale, approximately normally distributed. The reason is that growth rate on a natural logarithmic scale (loge ) equals the relative growth rate. Derivation of this result is a straightforward use of the differential calculus. The logarithmic transformation is so commonly needed that it has seemed necessary to introduce it at this point. Biologists, economists, and others should become comfortable with its use. There is a brief discussion of other transformations in Chapter 5.

52

Styles of data analysis A

B

wren tree pipit robin pied wagtail meadow pipit hedge sparrow 20

21

22

23

Length of egg (mm)

24

25

20

21

22

23

24

25

Length of egg (mm)

Figure 2.8 Strip plot (panel A) and boxplot (panel B) displays of cuckoo egg lengths. Data, from Latter (1902), are reproduced in summarized form in Tippett (1931).

2.1.4 Patterns in grouped data – lengths of cuckoo eggs

Cuckoos lay eggs in the nests of other birds. The eggs are then unwittingly adopted and hatched by the host birds. In Figure 2.8 the egg lengths are grouped by the species of the host bird, using both a strip plot display (panel A) and boxplot summaries (panel B). Strip plots and boxplots allow convenient side-by-side comparisons of different groups, here the different host species. The main part of the code used for these plots is: ## Compare stripplot() with bwplot(), both from lattice package stripplot(species ˜ length, xlab="Length of egg (mm)", data=cuckoos) bwplot(species ˜ length, xlab="Length of egg (mm)", data=cuckoos, scales=list(y=list(alternating=0))) # alternating=0; omit y-axis labels

Eggs planted in wrens’ nests appear smaller than eggs planted in other birds’ nests. Apart from several outlying egg lengths in the meadow pipit nests, the length variability within each host species’ nest is fairly uniform. Fuller details of the code are in the footnote.4 Comparing densities between groups – lattice style density plots

Lattice-style density plots can be useful for getting an indication of how distributions may differ across different groups of data. Figure 2.9 compares the ear conch measurements 4

## For tidier labels replace ".", in several of the species names, by a space specnam <- with(cuckoos, sub(pattern=".", replacement=" ", levels(species), fixed=TRUE)) # fixed=TRUE: do not interpret "." as a ‘regular expression’, ## Panel A: Strip plot: data frame cuckoos (DAAG) plt1 <- stripplot(species ˜ length, factor.levels=specnam, data=cuckoos) print(update(plt1, xlab="Length of egg (mm)"), position=c(0,0,0.55,1)) # xmin, ymin, xmax, ymax # Use print() to display lattice graphics objects ## Panel B: Box plot plt2 <- bwplot(species ˜ length, factor.levels=specnam, data=cuckoos) print(update(plt2, xlab="Length of egg (mm)", scales=list(y=list(alternating=0))), newpage=FALSE, position=c(0.55,0,1,1))

2.1 Revealing views of the data 40

f

45

50

53 55

m

0.25

Density

0.20

Vic other

0.15 0.10 0.05 0.00 40

45

50

55

earconch

Figure 2.9 Density plot that compares the ear conch measurements for each of the two “populations” of possums, for males and females separately.

of male and female possums, for each of two “populations” (Vic and other) of possums: ## Density plot for earconch: data frame possum (DAAG package) library(lattice) densityplot(˜earconch | sex, groups=Pop, data=possum, auto.key=list(space="right"))

2.1.5∗ Multiple variables and times

Overlaying plots of several time series (sequences of measurements taken at regular intervals) might seem appropriate for making direct comparisons. However, this approach will only work if the scales are similar for the different series. The data frame jobs (DAAG) gives the number of workers (in thousands) in the Canadian labor force, broken down by region (BC, Alberta, Prairies, Ontario, Quebec, Atlantic), for the 24-month period from January 1995 to December 1996. Over this time, Canada was emerging from a deep economic recession. Columns 1–6 have the respective numbers for six different regions. The ranges of values in the columns are: > ## Apply function range to columns of data frame jobs (DAAG) > sapply(jobs, range) BC Alberta Prairies Ontario Quebec Atlantic Date [1,] 1737 1366 973 5212 3167 941 95.00000 [2,] 1840 1436 999 5360 3257 968 96.91667

In order to see where the economy was taking off most rapidly, it is tempting to plot all of the series on the same graph. In order that similar changes on the scale will correspond to similar proportional changes, a logarithmic scale is used in Figure 2.10A: ## Simplified plot; all series in a single panel; use log scale (simplejobsA.xyplot <xyplot(Ontario+Quebec+BC+Alberta+Prairies+Atlantic ˜ Date, outer=FALSE, data=jobs, type="b", ylab="Number of workers", scales=list(y=list(log="e")), auto.key=list(space="right", lines=TRUE)))

54

Styles of data analysis

A Number of workers

4915 (8.5) 2981 (8)

Ontario Quebec BC Alberta Prairies Atlantic

1808 (7.5) 1097 (7) Jan95

Jul95

Jan96

B

1422 (7.26) 1394 (7.24) 1366 (7.22)

Number of jobs

Jan97

Jan95 Jul95 Jan96 Jul96 Jan97

Alberta

5432 (8.6) 5324 (8.58) 5219 (8.56) 5115 (8.54)

Jul96

1012 (6.92) 992 (6.9) 973 (6.88)

Ontario

Atlantic 973 (6.88) 953 (6.86) 934 (6.84)

Quebec 3294 (8.1) 3229 (8.08) 3165 (8.06)

Jan95 Jul95 Jan96 Jul96 Jan97

Prairies

1845 (7.52) 1808 (7.5) 1772 (7.48) 1737 (7.46)

BC

Jan95 Jul95 Jan96 Jul96 Jan97

Figure 2.10 Data are numbers in the labor force (thousands) for various regions of Canada, at quarterly intervals over 1995–1996. Panel A uses the same logarithmic scale for all regions. Panel B shows the same data as in panel A, but now with separate (“sliced”) logarithmic scales on which the same percentage increase, e.g., by 1%, corresponds to the same distance on the scale, for all plots. Distances between ticks are 0.02 on the loge scale, i.e., a change of almost exactly 2%.

The trellis object has been saved so that it can be updated, as demonstrated in the footnote, to give the graph shown in Figure 2.10A.5 The use of column names that are joined with "+" has the result that the columns are plotted in parallel. The regions have been taken in order of the number of jobs in December 1996 (or, in fact, at any other time). This ensures that the order of the labels in the key matches the positioning of the points for the different regions. Code in the footnote shows how the labeling on the x- and y-axes was obtained. 5

## Panel A: Update trellis object to improve x- and y-axis tick labels datelabpos <- seq(from=95, by=0.5, length=5) datelabs <- format(seq(from=as.Date("1Jan1995", format="%d%b%Y"), by="6 month", length=5), "%b%y") ## Now create $y$-labels that have numbers, with log values underneath ylabpos <- exp(pretty(log(unlist(jobs[,-7])), 5)) ylabels <- paste(round(ylabpos),"\n(", log(ylabpos), ")", sep="") update(simplejobsA.xyplot, xlab="", scales=list(x=list(at=datelabpos, labels=datelabs), y=list(at=ylabpos, labels=ylabels)))

2.1 Revealing views of the data 103.135

log=10 log="e"

e

7.220

10.416

log=2

2 1366

1250

103.157

55 103.244

7.270

e

e7.518

10.775

210.845 1840

e

10.488

2 1436

103.265

7.469

2 1752 1500

1800

Figure 2.11 Labeling of the values for Alberta (1366, 1436) and Ontario (1752, 1840), with alternative logarithmic scale choices of labeling.

Because the labor forces in the various regions do not have similar sizes, it is impossible to discern any differences among the regions from this plot. Plotting on the logarithmic scale did not remedy this problem.6 Figure 2.10B shows a much preferable alternative. The six different panels use different slices of the same logarithmic scale. Here is simplified code for Figure 2.9B. The regions are again taken in the order of numbers of jobs in December 1996.7 ## Simplified code for Figure 2.9B xyplot(Ontario+Quebec+BC+Alberta+Prairies+Atlantic ˜ Date, data=jobs, type="b", layout=c(3,2), ylab="Number of jobs", scales=list(y=list(relation="sliced", log=TRUE)), outer=TRUE)

Use of outer=TRUE ensures that the separate columns (regions) are plotted on separate panels. Equal distances on the scale now correspond to equal relative changes. It is now clear that Alberta and BC experienced the most rapid job growth during the period, and that there was little or no job growth in Quebec and the Atlantic region. Even better, particularly if ready comprehension is important, would be to standardize by dividing, e.g., by the respective number of persons aged 15 years and over at that time. Exercise 11 at the end of the chapter explores this approach. ∗

Small proportional changes, on a scale of natural logarithms

Tick marks have been placed a distance 0.02 apart on a scale of natural logarithms or loge . On a scale of natural logarithms a change of 0.02 is, to a close approximation, a 2% change. ∗

Tick positions and labeling, on a logarithmic scale

The following are the changes in numbers employed, in each of Alberta and Ontario, from January 1995 to December 1996. The changes are shown in actual numbers, and on scales of log2 , loge , and log10 . Figure 2.11 shows this graphically.

6 7

Figure 2.10A might alternatively be plotted as a time series. For details, see Subsection 14.9.7. Subsection 15.5.2 has code that gives the labeling shown in Figure 2.10B.

56

Styles of data analysis Increase Alberta (1366 to 1466; increase = 70) Ontario (1752 to 1840; increase = 88)

Rel. change 1.051 1.050

log2 0.072 0.070

loge 0.050 0.049

log10 0.022 0.021

From the beginning of 1995 to the end of 1996, Alberta increased by 70 from 1366 to 1436, which is a factor of 1436/1366  1.051. Ontario increased by 96 from 5239 to 5335, which is a factor of 1.050. The proper comparison is not between the absolute increases of 70 and 96, but between relative increases by factors of 1.05 and 1.018. For lattice functions, the arguments log=2 or log="e" or log=10 are available. These use the relevant logarithmic axis labeling, as in Figure 2.11, for axis labels. In base graphics, with the argument log="x", the default is to label in the original units. An alternative, both for traditional and lattice graphics, is to enter the logged values, using whatever basis is preferred (2 or "e" or 10), into the graphics formula. Unless other tick labels are provided, the tick marks will then be labeled with the logged values for the relevant basis. 2.1.6 Scatterplots, broken down by multiple factors

Data, in the data frame tinting (DAAG), are from an experiment that examined the effects of the tinting of car windows on visual performance (data relate to Burns et al., 1999). The main focus was on visual recognition tasks, where side window vision is important. Columns are: r Variables csoa (critical stimulus onset asynchrony, i.e., the time in milliseconds required to recognize an alphanumeric target), it (inspection time, i.e., the time required for a simple discrimination task), and age (age to the nearest year). r The ordered factor tint (levels no, lo, hi). r Factors target (locon, i.e., low contrast; hicon, i.e., high contrast), sex (f = female, m = male), and agegp (younger = a younger participant, in the 20s; older = an older participant, in the 70s). Each of 28 individuals was tested at each level of tint, for each of the two levels of target. In all there are four factors (tint, target, sex, and agegp) that might influence the values of csoa and it, and the relationship between them. Two of these (tint and target) take different values for the same individual, while the other two (sex and agegp) vary between individuals. A first step might be to plot csoa against it for each combination of sex and agegp. Use of the argument groups=target results in the use of different symbols (in a black and white plot) or different colors, depending on whether the target is low contrast or high contrast. Also, we can ask for a key. The code is (target.xyplot <xyplot(csoa ˜ it | sex*agegp, data=tinting, groups=target, auto.key=list(columns=2)))

2.1 Revealing views of the data locon

A

B 50

older f

hicon 100

150

57

no

lo

200

hi 50

older m

older f

100

150

200

older m 120

100

100

80

80

60

60

40

40

csoa

120

younger f

younger m

younger f

younger m

120

120

100

100

80

80

60

60

40

40

50

100

150

200

50

it

100

150

200

it

Figure 2.12 Panel A plots csoa against it, for each combination of sex and agegp. Different colors (gray and black) and symbols show different levels of target. Panel B shows the same points, but different colors (printed in grayscale) now show different levels of tint. Notice, also, the addition of smooth curves.

There are further possibilities for refinement. Figure 2.12A has used parameter settings that specify the choice of colors (here gray or black), plotting symbols, and the placement of the key.8 Observe that the longest times are for the high level of tinting. The relationship between csoa and it seems much the same for both levels of contrast. A number of older males have long response times with the low-contrast target. The analysis that will be presented later, in Chapter 10, indicates that within-subject effects – the effect of tint and target – stand up with greater clarity against the statistical noise than do effects of sex and agegp. The reason is that tint and target are effects that can be assessed within subjects, whereas the effects of sex and agegp involve a comparison across different subjects. Because there are six points for each subject, Figure 2.12A gives a visual impression that exaggerates the evidence for effects that are associated with sex and agegp. Fitting a trend curve, as in Figure 2.12B, makes the relationship clearer. The code for including the smooth curve, without the other refinements of Figure 2.12B, is: 8

## Settings used for Figure 2.12B (suitable for grayscale on a printed page) update(target.xyplot, par.settings=simpleTheme(col=c("black","gray20"), pch=c(1, 16))) # In the above, par.settings changed settings for this use of xyplot() ## Note the use of simpleTheme() for changing settings; see help(simpleTheme) ## Use trellis.par.set() to change settings while the current device is in use

58

Styles of data analysis

(tint.xyplot <xyplot(csoa ˜ it|sex*agegp, groups=tint, data=tinting, type=c("p","smooth"), span=1.25, auto.key=list(columns=3))) # "p": points; "smooth": a smooth curve # With span=1.25, the smooth curve is close to a straight line

The footnote adds the details needed to give Figure 2.12B.9 2.1.7 What to look for in plots

This is not a complete account of what plots may reveal! Its purpose is to draw attention to some of the more obvious possibilities. Outliers

Outliers are points that appear to be isolated from the main body of the data. Such points (whether errors or genuine values) are liable to distort any model that we fit. What appears as an outlier depends, inevitably, on the view that is presented. On a fairly simple level, the view is affected by whether or not, and how, the data are transformed. Boxplots, and the normal probability plot that will be discussed in Subsection 3.4.2, are useful for highlighting outliers in one dimension. Scatterplots may highlight outliers in two dimensions. Some outliers will be apparent only in three or more dimensions. The presence of outliers can indicate departure from model assumptions. Asymmetry of the distribution

Most asymmetric distributions can be characterized as either positively skewed or negatively skewed. Positive skewness is the commonest form of asymmetry. There is a long tail to the right, values near the minimum are bunched up together, and the largest values are widely dispersed. Provided that all values are greater than zero, a logarithmic transformation typically makes such a distribution more symmetric. A distribution that is skew cannot be normal. Severe skewness is typically a more serious problem for the validity of results than other types of non-normality. If values of a variable that takes positive values range by a factor of more than 10:1 then, depending on the application area context, positive skewness is to be expected. A logarithmic transformation should be considered. Changes in variability

Boxplots and histograms readily convey an impression of the extent of variability or scatter in the data. Side-by-side boxplots such as in Figure 2.8B, or strip charts such as 9

## Panel B, with refinements themeB <- simpleTheme(col=c("skyblue1", "skyblue4")[c(2,1,2)], lwd=c(1,1,2), pch=c(1,16,16)) # open, filled, filled update(tint.xyplot, par.settings=themeB, legend=NULL, auto.key=list(columns=3, points=TRUE, lines=TRUE)) # Set legend=NULL to allow new use of auto.key

2.2 Data summary

59

in Figure 2.8A, allow rough comparisons of the variability across different samples or treatment groups. They provide a visual check on the assumption, common in many uses of statistical models, that variability is constant across treatment groups. Note, however, that it is easy to over-interpret such plots. Statistical theory offers useful and necessary warnings about the potential for such over-interpretation. (The variability in a sample, typically measured by the variance, is itself highly variable under repeated sampling. Measures of variability will be discussed in Subsection 2.2.3.) When variability increases as data values increase, the logarithmic transformation will often help. If the variability is constant on a logarithmic scale, then the relative variation on the original scale is constant. Clustering

Clusters in scatterplots may suggest structure in the data that may or may not have been expected. When we proceed to a formal analysis, this structure must be taken into account. Do the clusters correspond to different values of some relevant variable? Outliers are a special form of clustering. Non-linearity

We should not fit a linear model to data where relationships are demonstrably non-linear. Often it is possible to transform variables so that terms enter into the model in a manner that is closer to linear. If not, the possibilities are wide-ranging, and we will canvass only a small number of them. See especially Chapter 7. If there is a theory that suggests the form of model, then this is a good starting point. Available theory may, however, incorporate various approximations, and the data may tell a story that does not altogether match the available theory. The data, unless they are flawed, have the final say! 2.2 Data summary Data summaries may: (1) be of interest in themselves; (2) give insight into aspects of data structure that may affect further analysis; (3) be used as data for further analysis. In case (3), it is necessary to ensure that important information, relevant to the analysis, is not lost. If no information is lost, the gain in simplicity of analysis can make the use of summary data highly worthwhile. It is important, when data are summarized, not to introduce distortions that are artefacts of the way that the data have been summarized – examples will be given. The question of whether information in the data may have been lost or obscured has especial importance for the summarizing of counts across the margins of multi-way tables, and for the use of the correlation coefficient. 2.2.1 Counts

Data in the data frame nswpsid1 (DAAG package) are derived from a study (Lalonde, 1986) that compared two groups of individuals with a history of unemployment problems – one an “untreated” control group and the other a “treatment” group whose members were exposed to a labor training program. Are the two groups genuinely comparable? This can

60

Styles of data analysis

be checked by comparing them with respect to various measures other than their exposure (or not) to the labor training program. Thus, what are the relative numbers in each of the two groups who had completed high school (nodeg = 0), as opposed to those who had not (nodeg = 1)? > > > + >

## Table of counts example: data frame nswpsid1 (DAAG) tab <- with(nswpsid1, table(trt, nodeg, useNA="ifany")) dimnames(tab) <- list(trt=c("none", "training"), educ = c("completed", "dropout")) tab educ trt completed dropout none 1730 760 training 80 217

Notice the use of the argument useNA="ifany" in the call to table(). This ensures that any NAs in either of the margins of the table will be tabulated. The training group has a much higher proportion of dropouts. Similar comparisons are required for other factors and variables, examining joint as well as individual comparisons. These data will be investigated further in Section 13.2. If x1, x2, . . . , xn are all columns (factors or vectors) of the same length and each is supplied as an argument to table(), the result is an n-way table. For example, table(x1, x2, x3) gives a three-way table. The first argument defines rows, though it is printed horizontally if there is just one column. The second argument defines columns. The table slices (rows by columns) that correspond to different values of the third argument appear in succession down the page, and so on.

Addition over one or more margins of a table

Figure 2.13 illustrates the possible hazards of adding a multi-way table over one of its margins. Data are from a study (Charig, 1986) that compared outcomes for two different types of surgery for kidney stones; A: open, which used open surgery, and B: ultrasound, which used a small incision, with the stone destroyed by ultrasound. Without additional information, the results are impossible to interpret. Different surgeons will have preferred different surgery types, and the prior condition of patients will have affected the choice of surgery type. The consequences of unsuccessful surgery may have been less serious for ultrasound than for open surgery. Code that gives the mosaic plot is: stones <- array(c(81,6,234,36,192,71,55,25), dim=c(2,2,2), dimnames=list(Success=c("yes","no"), Method=c("open","ultrasound"), Size=c("<2cm", ">=2cm"))) # NB: The margins are 1:Success, 2:Method, 3:Size library(vcd) mosaic(stones, sort=3:1) # c.f. mosaicplot() in base graphics # Re-ordering the margins gives a more interpretable plot.

2.2 Data summary

Success yes no

%Yes

Size <2cm >=2cm ultrasound <2cm >=2cm

81 192 234 55

6 71 36 25

93.1 73.0 86.7 68.8

Add over Size open ultrasound

273 289

77 61

78.0 82.6

Method ultrasound

yes no no

yes

Size >=2cm

Method open Sucess

<2cm

open

61

Figure 2.13 Mosaic plot for the kidney stone surgery data that is shown to the right of the figure. Outcomes are for two different types of surgery for kidney stones. The overall (apparent) success rates (78% for open surgery as opposed to 83% for ultrasound) favor ultrasound. The success rate for each size of stone separately favors, in each case, open surgery.

Code that tabulates the data, giving a layout similar to that on the right of the plot, is: ## Function to calculate percentage success rates roundpc <- function(x)round(100*x[1]/sum(x), 1) ## Add "%Yes" to margin 1 (Success) of the table stonesplus <- addmargins(stones, margin=1, FUN=c("%Yes"=roundpc)) ## Print table, use layout similar to that shown alongside plot ftable(stonesplus, col.vars=1) ## Get sum for each margin 1,2 combination; i.e., sum over margin 3 stones12 <- margin.table(stones, margin=c(1,2)) stones12plus <- addmargins(stones12, margin=1, FUN=c("%Yes"=roundpc)) ftable(stones12plus, col.vars=1) # Table based on sums over Size

An alternative to mosaic() in the vcd package is mosaicplot() in base graphics. The function mosaic() is more flexible and extensible. A footnote demonstrates how the size of the text in the margins can be modified.10

Tabulation that accounts for frequencies or weights – the xtabs() function

The function xtabs() will be illustrated with a further example that, again, demonstrates the hazards of summarizing tabular or other data across factors that affect the frequencies of the margins that are retained. Each year the National Highway Traffic Safety Administration in the USA uses a random sampling method, with sampling fractions that differ according to class of accident, to collect data from all police-reported crashes in which there is a harmful event (people or

10

## Add arguments that control size of textual components mosaic(aperm(stones, 3:1), main=NULL, gp_varnames=gpar(fontsize=8), labeling_args=list(gp_labels=gpar(fontsize=7), legend_args=list(fontsize=7)))

62

Styles of data analysis

property), and from which at least one vehicle is towed. The data in nassCDS (DAAG) are restricted to front-seat occupants (DAAG).11 Factors whose effect warrant investigation include, as a minimum: A: airbag (was an airbag fitted?), S: seatbelt (was a seatbelt used?), and dvcat (F: a force of impact measure). The letters A, S, and F will be used as abbreviations when tables are generated. The column weight (national inflation factor) holds the inverses of the sampling fraction estimates. The weight is designed to be the amount by which the contribution for the relevant row should be multiplied when tables of numbers of deaths and numbers of accidents are created. The following uses xtabs() to estimate numbers of front-seat passengers alive and dead, classified by airbag use: > library(DAAG) > ## NB: The parentheses generate an implicit print(abtab) > (Atab <- xtabs(weight ˜ airbag + dead, data=nassCDS)) dead airbag alive dead none 5445245.90 39676.02 airbag 6622690.98 25919.11

The function addmargins() that was introduced above can be used to add the proportion of deaths in the right margin: > roundpc2 <- function(x)round(100*x[2]/sum(x), 2) > addmargins(Atab, margin=2, FUN=c("%Dead"=roundpc2)) dead airbag alive dead %Dead none 5445245.90 39676.02 0.72 airbag 6622690.98 25919.11 0.39

The above might suggest that the deployment of an airbag substantially reduces the risk of mortality. Consider, however: > SAtab <- xtabs(weight ˜ seatbelt + airbag + dead, data=nassCDS) > ftable(addmargins(SAtab, margin=3, FUN=c("%Dead"=roundpc2)), + col.vars=3) dead alive dead %Dead seatbelt airbag none none 1342021.90 24066.65 1.76 airbag 871875.39 13759.94 1.55 belted none 4103224.00 15609.36 0.38 airbag 5750815.59 12159.17 0.21

In the earlier table (Atab), the results without airbags were mildly skewed (4.12:1.37) to those for belted. Results with airbags were strongly skewed (57.6:8.86) to those for none, that is, no seatbelt.

11

They hold a subset of the columns from a corrected version of the data analyzed in Meyer and Finney (2005). See also Farmer (2005) and Meyer (2006). More complete data are available from one of the web pages noted on the help page for nassCDS.

2.2 Data summary

63

The reader may wish to try an analysis that accounts, additionally, for estimated force of impact (dvcat): FSAtab <- xtabs(weight ˜ dvcat + seatbelt + airbag + dead, data=nassCDS) ftable(addmargins(FSAtab, margin=4, FUN=c("%Dead"=roundpc2)), col.vars=4)

The small differences that are now apparent, mostly unfavorable to airbags, are below any reasonable threshold of statistical detectability. Farmer (2005) argues that these data, tabulated as above, have too many uncertainties and potential sources of bias to give reliable results. He presents a different analysis, based on the use of front-seat passenger mortality as a standard against which to compare driver mortality. Farmer’s analysis was limited to cars without passenger airbags. In the absence of any effect from airbags, the ratio of driver mortality to passenger mortality should be the same, irrespective of whether or not there was a driver airbag. Farmer found a ratio of driver fatalities to passenger fatalities that was 11% lower in the cars with driver airbags. In addition to the functions discussed, note the function CrossTable() from the gmodels package, which offers a choice of SPSS-like and SAS-like output formats.

2.2.2 Summaries of information from data frames

For obtaining summaries at combinations of different factor levels, the aggregate() function is often a good recourse. Note also the abilities of aaply() and allied functions in Hadley Wickham’s plyr package.

Summary as a prelude to analysis – aggregate()

The data frame kiwishade (from DAAG) has yield measurements from 48 vines. Plots, made up of four vines each, were the experimental units. The analysis can be simplified by first taking means over plots that had four vines each. The 12 plots were divided into three blocks of four plots each. One block of four was north-facing, a second block west-facing, and a third block east-facing. (Because the trial was conducted in the Southern hemisphere, there is no south-facing block.) Shading treatments were applied to whole plots, i.e., to groups of four vines, with each treatment occurring once per block. The shading treatments were applied either from August to December, December to February, February to May, or not at all. For more details of the experiment, look ahead to Figure 10.4. For comparing treatments, there is no loss of information from basing analysis on the plot means. The four individual vine results that are averaged to give the plot mean are multiple measurements on the same experimental unit, here a plot. Figure 2.14 plots both the aggregated means and the individual vine results. As treatments were applied to whole plots, the graph that shows the individual vine results exaggerates the extent of information that is available, in each block, for comparing treatments. For

64

Styles of data analysis Individual vine yields

Plot means (4 vines)

85 90 95 100 105 110

east

north

west

Feb2May Dec2Feb Aug2Dec none

85 90 95 100 105 110

85 90 95 100 105 110

yield

Figure 2.14 The four panels are the four different plots. The solid gray points are plot means. The open gray circles are yields for individual vines in the plot.

gaining a correct impression of the strength of the evidence, it is best to focus the eye on the means, shown as +. The code for Figure 2.14 is given as a footnote.12 The first few rows of the data frame are: 1 2 3 4 5

yield 101.11 108.02 106.67 100.30 92.64

block shade plot north none north.none north none north.none north none north.none north none north.none west none west.none

The aggregate() function splits the data frame according to the specified combinations of factor levels, and then applies a specified function to each of the resulting subgroups. Here, it forms a data frame that has the mean for each combination of block and shading treatment. The code, with the first line of output following, is: > > + > >

## mean yield by block by shade: data frame kiwishade (DAAG) kiwimeans <- with(kiwishade, aggregate(yield, by=list(block, shade), mean)) names(kiwimeans) <- c("block","shade","meanyield") head(kiwimeans, 4) block shade meanyield 1 east none 99.0250 > # . . .

12

## Individual vine means, by block and treatment library(lattice) ## Panel function calls panel.dotplot(), then panel.average() dotplot(shade ˜ yield | block, data=kiwishade, aspect=1, panel=function(x,y,...){panel.dotplot(x, y, pch=1, col="gray40") panel.average(x, y, type="p", col="black", pch=3, cex=1.25)}, key=list(space="top", columns=2, col=c("gray40", "black"), text=list(c("Individual vine yields", "Plot means (4 vines)")), points=list(pch=c(1,3), cex=c(1,1.25))), layout=c(3,1)) # Note that parameter settings were given both in the calls to the # panel functions and in the list supplied to key.

2.2 Data summary

65

Use of the aggregated data for analysis commits us to working with plot means. What information is lost? If there were occasional highly aberrant values, use of medians might be preferable. The data should have a say in determining the form of summary.

The benefits of data summary – dengue status example

Hales et al. (2002) examined the implications of climate change projections for the worldwide distribution of dengue, a mosquito-borne disease that is a risk in hot and humid regions. Dengue status, i.e., information on whether dengue had been reported during 1965–1973, is available for 2000 administrative regions. Climate information is available on a much finer scale, on a grid of about 80 000 pixels at 0.5◦ latitude and longitude resolution. Should the analysis work with a data set that consists of 2000 administrative regions, or with the much larger data set that has one row for each of the 80 000 pixels? The following are reasons that might have argued for working with the summarized data: r Dengue status is a summary figure that is given by administrative region. An analysis that uses the separate data for the 80 000 pixels will, in effect, predict dengue status for climate variable values that are in some sense averages for the administrative region. Explicit averaging, prior to the analysis, gives the user control over the form of averaging that will be used. If, for example, values for some pixels are extreme relative to other pixels in the administrative region, medians may be more appropriate than means. In some regions, the range of climatic variation may be extreme. The mean will give the same weight to sparsely populated cold mountainous locations as to highly populated hot and humid locations on nearby plains. r Correlation between observations that are close together geographically, though still substantial, will be less of an issue for the data set in which each row is an administrative region. Points that repeat essentially identical information are a problem both for the interpretation of plots and, often, for the analysis. Regions that are geographically close will often have similar climates and the same dengue status. r Analysis is more straightforward with data sets that are of modest size. It is easier to do standard forms of data checking. The points that appear on plots are more nearly independent. Standard forms of scatterplot less readily degenerate into a dense mass of black ink. There are many possible ways to calculate a central value, of which the mean and the median are the most common. (In fact, however, the paper used the disaggregated data.) 2.2.3 Standard deviation and inter-quartile range

An important measure of variation in a population is the population standard deviation (often written σ ), which is almost always unknown. The variance σ 2 , which is the square of the standard deviation, is widely used in formal inference. The sample standard deviation, used to estimate the population standard deviation when a random sample has been taken, is  ¯ 2 (x − x) . s= n−1

66

Styles of data analysis

Table 2.1

Standard deviations for cuckoo egg data.

Hedge sparrow 1.049

Meadow pipit

Pied wagtail

Robin

Tree pipit

Wren

0.920

1.072

0.682

0.880

0.754

In words, take the difference of each data value from the mean, square, add the squared differences together, divide by n − 1, and take the square root. In R, use the function sd() to calculate the standard deviation, or var() to calculate the variance. The standard deviation is in the same units as the original measurements. For s to be an accurate estimate of σ , the sample must be large. Cuckoo eggs example

Consider again the data on cuckoo eggs that we discussed in Subsection 2.1.4. The group standard deviations are listed in Table 2.1.13 The variability in egg length is smallest when the robin is the host. Degrees of freedom

The denominator n − 1 is the number of degrees of freedom remaining after estimating the mean. With one data point, the sum of squares about the mean is zero, the degrees of freedom are zero, and no estimate of the variance is possible. The degrees of freedom are the number of data values, additional to the first data value. In later chapters, standard deviation calculations will be based on the variation that remains after fitting a model (most simply, a line) to the data. Degrees of freedom are reduced by 1 for each model parameter that is estimated. Other measures of variability

The standard deviation is similar in concept to the inter-quartile range H , which we saw in Subsection 2.1.1 is the difference between the first and third quartiles. (The region between the lower and upper quartiles takes in 50% of the data.) For data that are approximately normally distributed, note the approximate relationship s ≈ 0.75H. If data are approximately normally distributed, one standard deviation either side of the mean takes in roughly 68% of the data. Note also the median absolute deviation, calculated using the function mad(). This calculates the median of the absolute deviations from the median. By default this is multiplied by 1.4286, to ensure that in a large sample of normally distributed values the value returned should approximately equal the standard deviation. 13

## SD of length, by species: data frame cuckoos (DAAG) sapply(split(cuckoos$length, cuckoos$species), sd) # Subsection 14.9.6 has information on split()

2.2 Data summary

67

The pooled standard deviation

Consider two independent samples of sizes n1 and n2 , respectively, randomly selected from populations that have the same amount of variation but for which the means may differ. Thus, two means must be estimated. The number of degrees of freedom remaining for estimating the (common) standard deviation is n1 + n2 − 2. We compute the so-called pooled standard deviation by summing squares of differences of each data value from their respective sample mean, dividing by the degrees of freedom n1 + n2 − 2, and taking the square root:   ¯ 2 ¯ 2 + (y − y) (x − x) sp = . n1 + n2 − 2 Use of this pooled estimate of the standard deviation is appropriate if variation in the two populations is plausibly similar. The pooled standard deviation is estimated with more degrees of freedom, and therefore, more accurately, than either of the separate standard deviations. Elastic bands example

Consider data from an experiment in which 21 elastic bands were randomly divided into two groups, one of 10 and one of 11. Bands in the first group were immediately tested for the amount that they stretched under a weight of 1.35 kg. The other group were dunked in hot water at 65◦ C for four minutes, then left at air temperature for ten minutes, and then tested for the amount that they stretched under the same 1.35 kg weight as before. The results were: Ambient: 254 252 239 240 250 256 267 249 259 269 (Mean = 253.5) Heated: 233 252 237 246 255 244 248 242 217 257 254 (Mean = 244.1) The pooled standard deviation estimate is s = 10.91, with 19 (= 10 + 11 − 2) degrees of freedom. Since the separate standard deviations (s1 = 9.92; s2 = 11.73) are similar, the pooled standard deviation estimate is an acceptable summary of the variation in the data. 2.2.4 Correlation

The usual Pearson or product–moment correlation is a summary measure of linear relationship. Calculation of a correlation should always be accompanied by a check that the relevant scatterplot shows a linear relationship. Often the addition of a smooth trend line helps the assessment. Check also that the marginal distributions of the two variables are roughly normal, or at least not highly skew. If the relationship is monotonic, but is not linear and/or has asymmetric marginal distributions, it may be appropriate to use a Spearman rank correlation. Examples of the needed code are: > ## Correlation between body and brain: data frame Animals (MASS) > ## Product--moment correlation > with(Animals, cor(body, brain)) [1] -0.005341 > ## Product--moment correlation, after log transformation > with(log(Animals), cor(body, brain))

68

Styles of data analysis 0.894

0.878

0.458

0.005

Figure 2.15 Different relationships between y and x. In the second panel, the Pearson correlation is 0.878, while the Spearman rank correlation is 0.928.

[1] 0.7795 >> ## Spearman rank correlation > with(Animals, cor(body, brain, method="spearman")) [1] 0.7163

The function cor.test() returns a confidence interval, and tests for no association. Figure 2.15 gives four graphs to consider. For which does it make sense to calculate 1. A Pearson correlation coefficient? 2. A Spearman rank correlation? The figure that appears in the upper left in each panel is the Pearson correlation. For the second panel, the Pearson correlation is 0.878, while the Spearman correlation, which better captures the strength of the relationship, is 0.928. Here a linear fit clearly is inadequate. The magnitude of the correlation r, or of the squared correlation r 2 , does not of itself indicate whether the fit is adequate. Note also the Kendall correlation, obtained by specifying method="kendall" when cor.test() is called. This is often used in contexts where the same individuals are assessed by different judges. It estimates the probability that the two judges will assign the same ranking to an individual. Here are ways in which the use of correlation may mislead: r The usual interpretation of the magnitude of the coefficient assumes that sample pairs (x, y) have been taken at random from a bivariate normal distribution. Observations must be independent, and the separate marginal distributions of x and y must be approximately normal. If, for example, the marginal distributions are highly asymmetric, the correlation is likely to be smaller, with increased statistical variability. r There may be a subgroup structure in the data. If, for example, values of x and/or y are quite different for males and females, then the correlation may only reflect a difference between the sexes. Or, if random samples are taken from each of a number of villages and the data are pooled, then it will be unclear whether any correlation reflects a correlation between village averages or a correlation between individuals within villages, or a bit of each. The interpretation is confused because the two correlations may not be the same, and may even go in different directions. See Cox and Wermuth (1996). r Any correlation between a constituent and a total amount is likely to be, in part at least, a mathematical artifact. Thus, consider a study of an anti-hypertensive drug that hopes to determine whether the change y − x is larger for those with higher initial blood

2.3 Statistical analysis questions, aims, and strategies

69

pressure. If x and y have similar variances then, unfortunately, y − x will have a negative correlation with x, whatever the influence of the initial blood pressure. Note that while a correlation coefficient may sometimes be a useful single number summary of the relationship between x and y, regression methods offer a much richer framework for the examination of such relationships. 2.3 Statistical analysis questions, aims, and strategies Logically, this section might have appeared at the beginning of the chapter, prior to any discussion of analysis methods and approaches. It is here because the reader should by now have a level of familiarity with several data sets that will be used as a basis for discussion. Different questions, asked of the same data, will demand different analyses. Questions of interest may, given the available data, be unanswerable. Data on house prices in London may not have much relevance, if the interest is in house prices in New York or Paris! Questions should be structured with issues of this type in mind. Questions should be structured with a view to the intended use of results. Is the aim scientific understanding, perhaps as in the example discussed below to determine whether cuckoos do really match the eggs that they lay in the nests of other birds to the size and color of the host eggs? Or is the aim prediction, perhaps to predict, based on recent prices in the area and on house size, the price that purchasers may be willing to pay? Effective use of the information in the data

Figure 2.6 was designed to elicit the relationship between electrical resistance and apparent juice content, in kiwifruit. With the data we have, it would be bad practice to do a formal statistical test to compare, for example, juice content of less than 30% with juice content of more than 50%. Such an analysis would miss the relatively rich relationship that exists between the apparent juice content and resistance. The data are more informative than would be obtained by doing repeated trials, some with a juice content of 30% and some with a juice content of 50%. Even worse would be a study comparing resistance at 40% juice content with resistance at 50% juice content. Such a study would have very little chance of finding anything of consequence. Observational versus experimental data

Data from experiments appear throughout this book – examples are the data on the tinting of car windows that was used for Figure 2.12 in Subsection 2.1.6, and the kiwifruit shading data that were discussed in Subsection 2.2.2. Data from an experiment can, if well designed with a view to answering the questions of interest, give highly reliable results. With data from carefully designed experiments, perhaps the most serious danger is that the data will be generalized beyond the limits imposed by the experimental conditions. Observational data are another matter. Section 13.2 will discuss a comparison between results from an experimental study on the effects of a work training program (those enrolled in the study were randomly assigned to training and non-training groups), and results from various sets of matched observational data that have been used in the attempt to answer the

70

Styles of data analysis

Table 2.2 Mean lengths of cuckoo eggs, compared with mean lengths of eggs laid by the host bird species. More extensive data that bear on the comparison between cuckoo eggs and host eggs are in the data frame cuckoohosts (DAAG).

Host species

Meadow pipit

Hedge sparrow

Robin

Wagtails

Tree pipit

Wren

Yellow hammer

Length (cuckoo) Length (host)

22.3 (45) 19.7 (74)

23.1 (14) 20.0 (26)

22.5 (16) 20.2 (57)

22.6 (26) 19.9 (16)

23.1 (15) 20 (27)

21.1 (15) 17.7 (–)

22.6 (9) 21.6 (32)

(Numbers in parentheses are numbers of eggs.)

same question. In this instance results from the use of observational data can be compared with those from an experiment in which individuals were randomly assigned either to an experimental or to a control group. 2.3.1 How relevant and how reliable are the data?

Latter (1902) collected the cuckoo egg data presented in Figure 2.8 in order to investigate claims, made in Newton (1893–1896, p. 123), that the eggs that cuckoos lay in the nests of other birds tend to match the eggs of the host bird in size, shape, and color. Figure 2.8 strongly indicated differences, depending on the host bird, in length. A further step is to look for a relationship between the mean size of the cuckoo eggs for each specific host, and the mean size of the host bird’s own eggs, using data such as in Table 2.2. There are various difficulties with the data in Table 2.2. The cuckoo eggs and the host eggs are from different nests, collected in the course of different investigations. Data on the host eggs are from various sources. For the wren, the value is an indicative length from Gordon (1894). There is thus a risk of biases, different for the different sources of data, that limit the inferences that can be drawn. There is a striking difference between wrens and other species. Not only are their own eggs easily the smallest among the species considered; the eggs of the wren host are easily the smallest, among any of the hosts. Whatever biases may exist in the data, it is unlikely that they would be so large as to affect these major differences. Biases might well affect comparisons between eggs in the nests of species other than wrens. 2.3.2 How will results be used?

Studies may be designed to help scientific understanding. Consider again the data in Table 2.2. The interest of Latter’s paper is primarily in establishing whether there is a relationship, and rather less in determining the nature of the relationship. Egg size and shape is one of several pieces of evidence that Latter considers. Uniquely among the birds listed, the architecture of wren nests makes it impossible for the birds to see the eggs. In wren nests, the color of the cuckoo’s egg does not match the color of the wren’s eggs; for the other species the color does mostly match. Latter concludes that Newton is right, that the eggs that cuckoos lay tend to match the eggs of the host bird in size and shape in ways that will make it difficult for hosts to distinguish their eggs from the cuckoo eggs.

2.3 Statistical analysis questions, aims, and strategies

71

This is very different from the demands of the real-estate agent who may hope, on the basis of last year’s prices in a city location, and floor area, to predict the prices that purchasers will be willing to pay. Will a relationship that seems to work in one suburb apply also in another suburb, or in a neighboring city? Accurate prediction is crucial, with a perhaps reduced importance given to understanding the detailed reasons for any relationship that may be apparent. It is important, also, to know the intended use of the data and hence what sort of accuracy is important. Is it accuracy for purposes of making a prediction on one of the suburb(s) used in obtaining the data? Or is it accuracy for making predictions in new suburb(s)? In the latter case, data from multiple suburbs will be needed, and it must be possible to treat the sampled suburbs as a random sample from the suburbs for which predictions are required. 2.3.3 Formal and informal assessments

Statistical data analysis is, often, crucial to the answering of scientific questions. It does not however stand alone, but must be interpreted against a background of subject area knowledge and judgment. Qualitative judgments are inevitable at various points in studies that generate data sets such as are analyzed in this book. Such judgments affect the use of assumed subject area knowledge, the measurements that are taken, the design of data collection, the choice of analysis, and the interpretation of analysis results. These judgments, while they should be as informed as possible, cannot be avoided. Two examples will now be given: r In trials that evaluate therapies for conditions that commonly lead to early death, what is the relevant measure? Is it survival time from diagnosis? Or is it more appropriate to use a measure that takes account of quality of life over that time, which differs hugely between different therapies? Two such measures are “Disability Adjusted Life Years” (DALYs) and “Quality Adjusted Life Years” (QALYs). r The dataset nswpsid1 (see Subsection 2.2.1 and Section 13.2) allows comparison of two groups of individuals, both with a history of employment and related difficulties. A focus of interest was income in 1978, subsequent to the study. Because the distribution of income is highly skew, comparisons that are based directly on income will be biased towards the experience of those few individuals whose incomes were very large. This effect can be ameliorated by working with the logarithm of income. Or it might be more appropriate to compare the median salaries of the two groups, after adjusting for the effects of other variables. In neither case is the interpretation of analysis results as simple as might initially appear. There is an inevitable risk that assumed insights and judgments will carry large elements of prejudice or personal bias. A well-designed study will allow some opportunity for study results to challenge the assumed insights and understandings that have motivated the study. Questionnaires and surveys

The science and socsupport data frames (DAAG) are both from surveys. In both cases, an important question is whether the questions measured what they claimed to measure.

72

Styles of data analysis

In the science data set, a focus of interest is the variable like, which measured the students’ liking for science. What did students understand by “science”? Was science, for them, a way to gain and test knowledge of the world? Or was it a body of knowledge? Or, more likely, was it a name for their experience of science laboratory classes (smells, bangs, and sparks perhaps) and field trips? In the socsupport data set, an important variable is Beck Depression Index or BDI. The Beck Depression Index is a standard psychological measure of depression (see, e.g., Streiner and Norman, 2003), derived from a carefully designed and tested questionnaire. In either case it is impossible to escape the question: “What was measured?” This question is itself amenable to experimental investigation. For the data frame science, answers to other questions included in the survey shed some light. The Beck Depression Index is the result of an extensive process of development and testing that has seemed to validate its results, at least for populations on which it has been tested. Such background evidence helps in assessing what is measured. Finally, however, there must be a qualitative judgment that brings together subject area knowledge, background information and evidence, and the results of the immediate statistical analysis.

2.3.4 Statistical analysis strategies

We have emphasized the importance of careful initial scrutiny of the data. Techniques of a broadly EDA type have, in addition, a role in scrutinizing results from formal analysis, in checking for possible model inadequacies, and perhaps in suggesting remedies. In later chapters, we will discuss the use of diagnostic statistics and graphs in examination both of the model used and of output from the analysis. These are an “after the event” form of EDA. In the course of an analysis, the data analyst may move backwards and forwards between exploratory analysis and more formal analyses.

2.3.5 Planning the formal analysis

Planning the formal analysis is one aspect of planning a research study. Such advance planning should allow for the possibility of limited changes following preliminary investigation. An ideal is to have available data, and perhaps also analysis results, from a suitably related earlier study. Use of such information to plan the analysis in advance reduces the chance of biasing the new results in a direction that is closest to the analyst’s preference! Even so, graphical checks of the data should precede formal analysis. There may be obvious mistakes. The data may have surprises for the analyst. If available at the beginning of the study, the information from the analysis of earlier data may, additionally, be invaluable in the design of data collection for the new study. When prior data are not available, a pilot study involving a small number of experimental runs can sometimes provide this kind of information. Where it is not altogether clear what to expect, careful preliminary examination is even more necessary. In particular, the analyst should look for

2.4 Recap

r r r r r

73

outliers, clusters in the data, unexpected patterns within groups, between-group differences in the scatter of the data, whether there are unanticipated time trends associated, e.g., with order of data collection.

In all studies, it is necessary to check for obvious data errors or inconsistencies. In addition, there should be checks that the data support the intended form of analysis.

2.3.6 Changes to the intended plan of analysis

What departures from the original plan are acceptable, and what are not? If the exploratory analysis makes it clear that the data should be transformed in order to approximate normality more closely, then use the transformation. It is sometimes useful to do both analyses (with the untransformed as well as with the transformed data) and compare them. On the other hand, if there are potentially a large number of comparisons that could be made, the comparisons that will be considered should be specified in advance. Prior data, perhaps from a pilot study, can assist in this choice. Any investigation of other comparisons may be undertaken as an exploratory investigation, a preliminary to the next study. Data-based selection of one or two comparisons from a much larger number is not appropriate, since huge biases may be introduced. Alternatively, there must be allowance for such selection in the assessment of model accuracy. The issues here are non-trivial, and we defer further discussion until later.

2.4 Recap Exploratory data analysis aims to allow the data to speak for themselves, often prior to or as part of a formal analysis. It gives multiple views of the data that may provide useful insights. Histograms, density plots, stem-and-leaf displays, and boxplots are useful for examining the distributions of individual variables. Scatterplots are useful for looking at relationships two at a time. If there are several variables, the scatterplot matrix provides a compact visual summary of all two-way relationships. Before analysis, look especially for r r r r r r

outliers, skewness (e.g., a long tail) in the distribution of data values, clustering, non-linear bivariate relationships, indications of heterogeneous variability (i.e., differences in variability across samples), whether transformations seem necessary.

After analysis, check residuals for all these same features. Where relationships involve several variables, adequate checks will be possible only after analysis. Failure of the independence assumption is hard to detect, unless the likely form of dependence is known and the sample is large. Be aware of any structure in the data that may be associated with lack of independence.

74

Styles of data analysis

Do not allow the initial scrutiny of data to influence the analysis in ways that may lead to over-interpretation. 2.5 Further reading The books and papers on graphical presentation that were noted in Chapter 1 are equally relevant to this chapter. The books Cleveland (1993, 1994) are especially pertinent to the present chapter. Chatfield (2002) has a helpful and interesting discussion, drawing on consulting experience, of approaches to practical data analysis. On statistical presentation issues, and deficiencies in the published literature, see Andersen (1990), Chanter (1981), Gardner et al. (1983), Maindonald (1992), Maindonald and Cox (1984), Wilkinson and Task Force on Statistical Inference (1999). The Wilkinson et al. paper has helpful comments on the planning of data analysis, the role of exploratory data analysis, and so on. Nelder (1999) is forthright and controversial. Two helpful web pages are http://www.math.yorku.ca/SCS/friendly.html#graph and http:// www.rdg.ac.uk/ssc/publications/publications.html#under References for further reading

Andersen, B. 1990. Methodological Errors in Medical Research: an Incomplete Catalogue. Chanter, D. O. 1981. The use and misuse of regression methods in crop modelling. In: Mathematics and Plant Physiology, eds. D. A. Rose and D. A. Charles-Edwards. Chatfield, C. 2002. Confessions of a statistician. The Statistician 51: 1–20. Cleveland, W. S. 1993. Visualizing Data. Cleveland, W. S. 1994. The Elements of Graphing Data, revised edn. Gardner, M. J., Altman, D. G., Jones, D. R. and Machin, D. 1983. Is the statistical assessment of papers submitted to the British Medical Journal effective? British Medical Journal 286: 1485–8. Maindonald, J. H. 1992. Statistical design, analysis and presentation issues. New Zealand Journal of Agricultural Research 35: 121–41. Maindonald, J. H. and Cox, N. R. 1984. Use of statistical evidence in some recent issues of DSIR agricultural journals. New Zealand Journal of Agricultural Research 27: 597–610. Nelder, J. A. 1999. From statistics to statistical science. Journal of the Royal Statistical Society, Series D 48: 257–67. Wilkinson, L. and Task Force on Statistical Inference. 1999. Statistical methods in psychology journals: guidelines and explanation. American Psychologist 54: 594–604. 2.6 Exercises 1. Use the lattice function bwplot() to display, for each combination of site and sex in the data frame possum (DAAG package), the distribution of ages. Show the different sites on the same panel, with different panels for different sexes. 2. Do a stem-and-leaf display for the lengths of the female possums. On the display, identify the position of the median, either at one of the leaves or between two leaves. Explain the reasoning used to find the median, and use the function median() to check the result.

2.6 Exercises

75

3. Plot a histogram of the earconch measurements for the possum data. The distribution should appear bimodal (two peaks). This is a simple indication of clustering, possibly due to sex differences. Obtain side-by-side boxplots of the male and female earconch measurements. How do these measurement distributions differ? Can you predict what the corresponding histograms would look like? Plot them to check your answer. 4. For the data frame ais (DAAG package), draw graphs that show how the values of the hematological measures (red cell count, hemoglobin concentration, hematocrit, white cell count, and plasma ferritin concentration) vary with the sport and sex of the athlete. 5. Using the data frame cuckoohosts, plot clength against cbreadth, and hlength against hbreadth, all on the same graph and using a different color to distinguish the first set of points (for the cuckoo eggs) from the second set (for the host eggs). Join the two points that relate to the same host species with a line. What does a line that is long, relative to other lines, imply? Here is code that you may wish to use: attach(cuckoohosts) plot(c(clength, hlength), c(cbreadth, hbreadth), col=rep(1:2,c(12,12))) for(i in 1:12)lines(c(clength[i], hlength[i]), c(cbreadth[i], hbreadth[i])) text(hlength, hbreadth, abbreviate(rownames(cuckoohosts),8)) detach(cuckoohosts) 6. Enter and run the following code. Annotate it, describing each function and each function argument: deathrate <- c(40.7, 36,27,30.5,27.6,83.5) hosp <- c("Cliniques of Vienna (1834-63)\n(> 2000 cases pa)", "Enfans Trouves at Petersburg\n(1845-59, 1000-2000 cases pa)", "Pesth (500-1000 cases pa)", "Edinburgh (200-500 cases pa)", "Frankfort (100-200 cases pa)", "Lund (< 100 cases pa)") hosp <- factor(hosp, levels=hosp[order(deathrate)]) dotplot(hosp˜deathrate, xlim=c(0,95), xlab="Death rate per 1000 ", type=c("h","p")) ## Source: \cite{Nightingale}. Data are ascribed to Dr Le Fort 7. The dataset ex10.22, on tomato yields, is available from the package Devore6 (or in wide format from Devore7). Type library(Devore6) # ex10.22 is from Devore6 tomatoes <- ex10.22 This data frame gives tomato yields at four levels of salinity, as measured by electrical conductivity (EC, in nmho/cm). (a) Obtain a scatterplot of yield against EC. (b) Obtain side-by-side boxplots of yield for each level of EC. (c) The third column of the data frame is a factor representing the four different levels of EC. Comment upon whether the yield data are more effectively analyzed using EC as a quantitative or qualitative factor. 8. Examine the help for the function mean(), and use it to learn about the trimmed mean. For the total lengths of female possums, calculate the mean, the median, and the 10% trimmed

76

Styles of data analysis mean. How does the 10% trimmed mean differ from the mean for these data? Under what circumstances will the trimmed mean differ substantially from the mean?

9. Assuming that the variability in egg length for the cuckoo eggs data is the same for all host birds, obtain an estimate of the pooled standard deviation as a way of summarizing this variability. [Hint: Remember to divide the appropriate sums of squares by the number of degrees of freedom remaining after estimating the six different means.] 10. Calculate the following three correlations: with(Animals, cor(brain,body)) with(Animals, cor(log(brain),log(body))) with(Animals, cor(log(brain),log(body), method="spearman")) Comment on the different results. Which is the most appropriate measure of the relationship? 11. Figure 2.10 showed changes in labor force numbers, in six regions of Canada, in successive quarters of 1995–1996. The population (in thousands) aged 15 years and over in each of these regions was, according to the 1996 census: BC: 3955; Alberta: 2055; Prairies: 1604; Ontario: 8249; Quebec: 5673; Atlantic: 1846. Plot a version of Figure 2.10 in which the labor force numbers are standardized by division by the number in the relevant population. Compare a plot that shows all regions in the same panel, with a plot that gives each region its own panel and its own slice of a common scale, commenting on the advantages and disadvantages of each. Is there now any reason to use a logarithmic scale? 12. The following code conveys information that has points of connection with the information in Figure 2.14: bwplot(shade ˜ yield|block, data=kiwishade, layout=c(3,1)) Compare and contrast the information given by these two plots. 13. The galaxies data in the MASS library gives speeds on 82 galaxies (see the help file and the references listed there for more information). Obtain a density plot for these data. Is the distribution strongly skewed? Is there evidence of clustering? 14. The cpus data frame in the MASS library contains information on eight aspects for each of 209 different types of computers. Read the help page for more information. (a) Construct a scatterplot matrix for these data. Should any of the variables be transformed before further analysis is conducted? (b) How well does estimated performance (estperf) predict performance (perf)? Study this question by constructing a scatterplot of these two variables, after taking logarithms. Do the plotted points scatter about a straight line or is there an indication of non-linearity? Is variability in performance the same at each level of performance?

3

Statistical models

Many regularities of nature are taken for granted in daily living – the rising and setting of the sun, the effects of fire in burning anyone unfortunate enough to get too near, and so on. Experience of the world, rather than logical deductive argument, has identified these regularities. Scientific investigation, especially in the physical sciences, has greatly extended and systematized awareness of regularities. Mathematical descriptions, i.e., models, have been crucial for describing and quantifying these regularities. As when any model is pressed into service, it is important to understand which features generalize and which do not. An engineer’s scale model of a building may be helpful for checking the routing of the plumbing but may give little indication of the acoustics of seminar rooms that are included in the building. In medical research, mouse responses to disease and to therapeutic agents are widely used as models for human responses. Experimental responses in the mouse may indicate likely responses in humans. In fundamental research in the physical sciences, deterministic models are often adequate. Statistical variability may be so small that it can, for practical purposes, be ignored. In applications of the physical sciences, variability may more commonly be a serious issue. In studying how buildings respond to a demolition charge, there will be variation from one occasion to another, even for identical buildings and identically placed charges. There will be variation in which parts of the building break first, in what parts remain intact, and in the trajectories of fragments. In the natural sciences, such variability is everywhere. Statistical models rely on probabilistic forms of description that have wide application over all areas of science. They often consist of a deterministic component, with a random component added that is designed to account for residual variation.

3.1 Statistical models As we saw in Chapter 2, consideration of a model stays somewhat in the background in initial exploratory data analysis. The choice of model is crucial in formal analysis. The choice may be influenced by previous experience with comparable data, by subject area knowledge, and by cautious use of what may emerge from exploratory analysis. Models should, wherever possible, be scientifically meaningful, but not at the cost of doing violence to the data. The scientific context includes the analyses, if any, that other researchers have undertaken with related or similar data. It can be important to note and use such analyses critically. While they may give useful leads, there can be serious inadequacies

78

Statistical models

depression weight

20 10 5 0

Depression (mm)

30

weight depression (t) (mm)

0

2 4 6 8 10 Weight of roller (t)

1 2 3 4 5 6 7 8 9 10

1.9 3.1 3.3 4.8 5.3 6.1 6.4 7.6 9.8 12.4

2 1 5 5 20 20 23 10 30 25

1.1 0.3 1.5 1.0 3.8 3.3 3.6 1.3 3.1 2.0

Figure 3.1 Depression in lawn versus roller weight. Both lines were drawn by eye, with the dashed line constrained to go through the origin.

in published analyses. For further detail and discussion, see relevant articles and books in the list of references at the end of Chapter 2. 3.1.1 Incorporation of an error or noise component

Statistical models combine deterministic and random components. The random component is often called noise or error and the deterministic component is sometimes thought of as the signal. It may be helpful to think of the statistical error as the “rough”, and of the model prediction as the “smooth”. The error component models variation that cannot be accounted for by given information. The simplest models assume that the elements of the error component are uncorrelated, i.e., the size and sign (negative or positive) of one element give no information on the likely size and sign of any other element. Both noise and error are technical terms. Use of the word error does not imply that there have been mistakes in the collection of the data, though mistakes can of course contribute to making the variability unnecessarily large. Figure 3.1 shows data from an experiment where different weights of roller were rolled over different parts of a lawn, and the depression noted (data are from Stewart et al., 1988).1 The data seem broadly consistent with the assumption of a signal by which depression is proportional to roller weight, as implied by the solid line in Figure 3.1. Variation about this signal is reflected in variation in the values for depression / weight. More generally, it might be assumed that the signal is a line that does not necessarily assume strict proportionality between depression and weight. The dashed line in Figure 3.1 is an example. It allows for a systematic error in the measurement of depression. 1

## Plot depression vs weight: data frame roller (DAAG) plot(depression ˜ weight, data = roller, xlim=c(0,1.04*max(weight)), ylim=c(0,1.04*max(depression)), xaxs="i", yaxs="i", # "i"=inner: Fit axes exactly to the limits xlab = "Weight of roller (t)", ylab = "Depression(mm)", pch = 16) abline(0, 2.25) # A slope of 2.25 looks about right

3.1 Statistical models

79

The model has the form: observed value = model prediction + statistical error i.e.,

y = µ+ε

where µ = βx (no intercept), or µ = α + βx (with intercept). The model prediction (µ) is the signal component, while ε is the error component. Substituting µ = α + β × weight in the equation: depression = α + β × weight + noise. Here α and β are constants which must be estimated. (For strict proportionality between depression and weight, α will of course be zero.) Use of subscripts allows identification of the individual points. Given observations (x1 , y1 ), (x2 , y2 ), . . . , (xn , yn ), we may write yi = α + βxi + εi .

Predicting with models

The focus of interest – generally prediction and/or interpretation of model parameters – may be different for different uses of model results. For the lawn roller data of Figure 3.1, one focus of interest is the rate of increase of depression with increasing roller weight, i.e., the slope of the line. Another focus is model prediction, i.e., the fitted values. Models should as far as possible yield inferences that, for their intended use, are acceptably accurate. Model structure should reflect data structure. The model treats the pattern of change of depression with roller weight as a deterministic or fixed effect. The measured values of depression incorporate, in addition, a random effect that reflects variation from one part of the lawn to another, differences in the handling of the roller, and measurement error. Data from multiple lawns are essential, for anything more than informal judgment on how results may generalize to other lawns. Such data would allow use of a model that accounts for between-lawn variation, as well as for the within-lawn variation on which our data give information. Chapter 10 will discuss models that might be tried, if data from multiple lawns were available.

Which model is best?

Figure 3.2 shows two possible models for the lawn roller data, together with information that may be helpful in assessing the adequacy of the model. In Figure 3.2A, a line (with intercept) has been fitted, while Figure 3.2B has used lowess() to fit a smooth curve through the data. Sometimes, the fitting of a curve such as in Figure 3.2B helps indicate whether a line really is appropriate. Note that there is just one point that seems to be causing the line, and the fitted curve, to bend down. In any case, there is no statistical justification for fitting a curve rather than a line, as can be verified with a formal analysis; see Exercise 2 in Chapter 7. There is “over-fitting” in Figure 3.2B.

80

Statistical models

10 15 20 25 30

−ve residual

Data values Fitted values (smooth)

+ve residual −ve residual

0

5

Depression in lawn (mm)

+ve residual

5

10 15 20 25 30

B Data values Fitted values (smooth)

0

Depression in lawn (mm)

A

2

4 6 8 10 Roller weight (t)

12

2

4 6 8 10 Roller weight (t)

12

Figure 3.2 In panel A a line has been fitted, while panel B has a smooth curve. Residuals (the “rough”) appear as vertical lines. Positive residuals are black lines, while negative residuals are dashed. Figures 3.2A and B were created using our function g3.2(), which is available from the web page for the book. Interested readers can check the code.

3.1.2 Fitting models – the model formula

Formulae have already been used extensively to describe graphs that will be plotted using plot() or another such function. Modeling functions likewise use formulae to describe the role of variables and factors in models. Thus, the following model statement, with model formula depression ˜ weight, fits a line to the data of Figure 3.1: ## Fit line - by default, this fits intercept & slope. ## requires data frame roller (DAAG) roller.lm <- lm(depression ˜ weight, data=roller) ## Compare with the code used to plot the data plot(depression ˜ weight, data=roller) ## Add the fitted line to the plot abline(roller.lm)

The name roller.lm, used for the output object, was chosen for mnemonic reasons – the object was the result of lm calculations on the roller data set. In the formula, weight is the predictor or explanatory variable, while depression is the response.2

Fitted values, residuals, and coefficients

Residuals, which are the differences between observed values of depression, and predicted values of depression at the respective values of weight, are important for assessing the accuracy of the model fit. A large error component generates large residuals, and works against accurate prediction. Figure 3.2A exhibits the residuals for the lawn roller data after fitting a straight line, while Figure 3.2B exhibits the residuals after fitting a smooth curve. Positive residuals are represented by solid lines, while negative residuals are represented by dashed lines. 2

## For a model that omits the intercept term, specify lm(depression ˜ -1 + weight, data=roller)

3.2 Distributions: models for the random component

81

Most of the information that is commonly required from model objects can be obtained by the use of an extractor function. For example, fitted values and residuals can be calculated with functions fitted() and resid(): > round(fitted(roller.lm), 1) 1 2 3 4 5 6 7 8 9 10 3.0 6.2 6.7 10.7 12.0 14.2 15.0 18.2 24.0 31.0 > round(resid(roller.lm), 1) 1 2 3 4 5 6 7 8 9 10 -1.0 -5.2 -1.7 -5.7 8.0 5.8 8.0 -8.2 6.0 -6.0

The coef() function gives the model coefficients: > coef(roller.lm) (Intercept) weight -2.087148 2.666746

Model objects

The model object, above saved as roller.lm, is a list. Although not always recommended, we can access information in this list directly. For example, we can extract element names as follows: > names(roller.lm) # Get names of list elements [1] "coefficients" "residuals" "effects" [5] "fitted.values" "assign" "qr" [9] "xlevels" "call" "terms"

"rank" "df.residual" "model"

We can then extract information directly from a list element, such as the model coefficients: > roller.lm$coef (Intercept) weight -2.087148 2.666746

For further discussion, see Subsection 14.10.2.

Summary information about model objects

To get a summary that includes coefficients, standard errors of coefficients, t-statistics, and p-values, type summary(roller.lm)

3.2 Distributions: models for the random component In this section, we briefly review the concepts of random variables and their distributions. Our discussion will focus on the more commonly used models for count data and continuous measurement data.

82

Statistical models

3.2.1 Discrete distributions – models for counts

Counts of events or numbers of objects are examples of discrete random variables. The possible values with their associated probabilities are referred to as a distribution. We consider three important examples: Bernoulli, binomial, and Poisson distributions.

Bernoulli distribution

Successive tosses of a fair coin come up tails with probability 0.5, and heads with probability 0.5, independently between tosses. If we let X take the value 1 for a head and 0 for a tail, then X is said to have a Bernoulli distribution with parameter π = 0.5. More generally, we might consider an experiment or test with an uncertain outcome, but where the possibilities are “success” (or “1”) and “failure” (or “0”). Success may occur with probability π, where 0 ≤ π ≤ 1. Binomial distribution

The sum of a number of independent Bernoulli random variables is called a binomial random variable. The number of successes in n independent tests (where success at each trial occurs with probability π) has a binomial distribution with parameters n and π . The total number of heads in two tosses of a fair coin is a binomial random variable with n = 2 and π = 0.5. We can use the function dbinom() to determine probabilities of having 0, 1 or 2 heads in two coin tosses:3 ## To get labeled output exactly as below, see the footnote ## dbinom(0:2, size=2, prob=0.5) # Simple version 0 1 2 0.25 0.50 0.25

On average, 25% of all pairs of coin tosses will result in no heads, 50% will have one head, and 25% will have two heads. The number of heads in four coin tosses can be modeled as binomial with n = 4 and π = 0.5: ## dbinom(0:4, size=4, prob=0.5) 0 1 2 3 4 0.0625 0.2500 0.3750 0.2500 0.0625

To calculate the probability of no more than two heads, add up the probabilities of 0, 1, and 2 (0.0625 + 0.2500 + 0.3750 = 0.6875). The function pbinom() can be used to determine such cumulative probabilities, thus: pbinom(q=2, size=4, prob=0.5)

3

## To get the labeling (0, 1, 2) as in the text, specify: probs <- dbinom(0:2, size=2, prob=0.5) names(probs) <- 0:2 probs

3.2 Distributions: models for the random component

83

For another example, suppose a sample of 50 manufactured items is taken from an assembly line that produces 20% defective items, on average. To find the probability of observing no more than four defectives in a sample, use: > pbinom(q=4, size=50, prob=0.2) [1] 0.0185

The probability of observing fewer than five defectives in the sample is 0.0185. The function qbinom() goes in the other direction, from cumulative probabilities to numbers of events; it is used to compute quantiles, a generalization of the more familiar term percentiles. To calculate a 70th percentile of the distribution of the number of heads in four coin tosses, type: > qbinom(p = 0.70, size = 4, prob = 0.5) [1] 3

Means and standard deviations

In four fair coin tosses, we expect to see two heads on average. In a sample of 50 manufactured items from a population where 20% are defective, we expect to see 10 defectives on average. In general, we can compute the expected value or mean of a binomial random variable using the formula nπ . The standard deviation is one way of summarizing the spread of a probability distribution; it relates directly to the degree of uncertainty associated with predicting the value of a random variable. High values reflect more uncertainty than low values. The formula √ nπ (1 − π ) gives the standard deviation for a binomial random variable. The standard deviation of the number of heads in four coin tosses is 1, and for the number of defectives in our sample of 50 items, it is 2.83. In an absolute sense, we will be able to predict the number of heads more precisely than the number of defectives. The variance is defined as the square of the standard deviation: for the binomial, it is nπ (1 − π). Poisson distribution

The Poisson distribution is often used to model the number of events that occur in a certain time interval or for the numbers of defects that are observed in items such as manufactured products. The distribution depends on a parameter λ (the Greek letter “lambda”), which happens to coincide with the mean or expected value. As an example, consider a population of raisin buns for which there are an average of three raisins per bun, i.e., λ = 3. Because of the mixing process, the number of raisins in a particular bun is uncertain; the possible numbers of raisins are 0, 1, 2, . . . . Under the Poisson model, we have the following probabilities for 0, 1, 2, 3, or 4 raisins in a bun: ## Probabilities of 0, 1, 2, 3, 4 raisins ## mean number of raisins per bun = 3 ## dpois(x = 0:4, lambda = 3) 0 1 2 3 4 0.0498 0.1494 0.2240 0.2240 0.1680

Statistical models

0.2

0.3

pnorm(1) = 0.841

0.1

normal density

84

−3 −2 −1

0

1

2

3

Distance, in SDs, from the mean

Figure 3.3 A plot of the normal density. The horizontal axis is labeled in standard deviations (SDs) distance from the mean. The area of the shaded region is the probability that a normal random variable has a value less than one standard deviation above the mean.

The cumulative probabilities are: ## ppois(q = 0:4, lambda = 3) 0 1 2 3 4 0.0498 0.1991 0.4232 0.6472 0.8153

Thus, for example, the probability of finding two or fewer raisins in a bun is 0.4232. The variance of a Poisson random variable is equal to its mean, i.e., λ. Thus, the variance of the number of raisins in a bun is 3, and the standard deviation is the square root of λ: 1.73.

3.2.2 Continuous distributions

Models for measurement data are examples of continuous distribution. Calculations with continuous distributions are a little different from calculations with discrete distributions. While we can still speak of probabilities of measurements lying in specified intervals, it is no longer useful to consider the probability of a measurement taking on a particular value. A more useful concept is probability density. A continuous random variable is summarized by its density function or curve. The area under any density curve between x = a and x = b gives the probability that the random variable lies between those limits.

Normal distribution

The normal distribution, which has the bell-shaped density curve pictured in Figure 3.3, is often used as a model for continuous measurement data (sometimes a transformation of the data is required in order for the normal model to be useful). The height of the curve is a function of the distance from the mean. The area under the density curve is 1. The density curve plotted in Figure 3.3 corresponds to a normal distribution with a mean of 0 and standard deviation 1. A normal distribution having mean 0 and standard deviation 1 is referred to as the standard normal distribution. Multiplying a fixed value σ by a population of such normal variates changes the standard deviation to σ . By adding a fixed value µ, we can change the mean to µ, leaving the standard deviation unchanged.

3.2 Distributions: models for the random component

85

Here is code that plots the normal density function:4 ## Plot the normal density, in the range -3 to 3 z <- pretty(c(-3,3), 30) # Find ˜30 equally spaced points ht <- dnorm(z) # By default: mean=0, standard deviation=1 plot(z, ht, type="l",xlab="Normal deviate",ylab="Density",yaxs="i") # yaxs="i" locates the axes at the limits of the data

The function pnorm() calculates the cumulative probability, i.e., the area under the curve up to the specified ordinate or x-value. For example, there is a probability of 0.841 that a normal deviate is less than 1: > pnorm(1.0) [1] 0.841

# by default, mean=0 and SD=1

This corresponds to the area of the shaded region in Figure 3.3.5 The function qnorm() can be used to compute the normal quantiles. For example, the 90th percentile is 1.28: > qnorm(.9) [1] 1.28

# 90th percentile; mean=0 and SD=1

The footnote has additional examples.6 Other continuous distributions

There are many other statistical models for continuous observations. The simplest model is the uniform distribution, for which an observation is equally likely to take any value in a given interval; the probability density of values is constant on a fixed interval. Another model is the exponential distribution that gives high probability density to positive values lying near 0; the density decays exponentially as the values increase. The exponential distribution is commonly used to model times between arrivals of customers to a queue. The exponential distribution is a special case of the chi-squared distribution. The latter distribution arises, for example, when dealing with contingency tables. Details on computing probabilities for these distributions can be found in the exercises.

4

5

6

The following gives a closer approximation to Figure 3.3: ## Plot the normal density, in the range -3.25 to 3.25 z <- pretty(c(-3.25,3.25), 30) # Find ˜30 equally spaced points ht <- dnorm(z) # By default: mean=0, standard deviation=1 plot(z, ht, type="l", xlab="Normal deviate", ylab="Ordinate", yaxs="i") polygon(c(z[z <= 1.0], 1.0), c(dnorm(z[z <= 1.0]), 0), col="grey") # Around 84.1% of the total area is to the left of the vertical line. ## Additional examples: pnorm(0) # .5 (exactly half the area is to the left of the mean) pnorm(-1.96) # .025 pnorm(1.96) # .975 pnorm(1.96, mean=2) # .484 (a normal distribution with mean 2 and SD 1) pnorm(1.96, sd=2) # .836 (sd = standard deviation) ## Additional examples: qnorm(0.841) # 1.0 qnorm(0.5) # 0 qnorm(0.975) # 1.96 qnorm(c(.1,.2,.3)) # -1.282 -0.842 -0.524 (10th, 20th and 30th percentiles) qnorm(.1, mean=100, sd=10) # 87.2 (10th percentile, mean=100, SD=10)

86

Statistical models

Different ways to describe distributions

In Subsection 2.1.1 it was noted that, with the default boxplot settings, 1% of values that are drawn at random from a normal distribution will on average be flagged as possible outliers. If the distribution is not symmetric, more than 1% of points may lie outside the whiskers, mostly at the lower end if the distribution is skewed (i.e., with a long tail) to the left, and mostly at the upper end if the distribution is skewed to the right. If the distribution is symmetric, but “heavy-tailed", then a higher proportion of values are out beyond the boxplot whiskers on both sides.

3.3 Simulation of random numbers and random samples In a simulation, repeated random samples are taken from a specified distribution. Statistics, perhaps estimates that are derived from one or other model, can then be calculated for each successive sample. Information is thus obtained on variation under repeated sampling. This allows a check on results predicted by statistical theory. Or it may provide guidance when theoretical results are not available or are of uncertain relevance. This section will begin with simulation of discrete and continuous random variables and will close with a discussion of random sampling from finite populations. Ordinarily, it is undesirable to use the same random number seed in two or more successive calls to a function that uses the random number generator. However, users will sometimes, for purposes of checking a calculation, wish to repeat calculations with the same sequence of random numbers as was generated in an earlier call. The following uses set.seed() to make the call below to rbinom(10, size=1, p=.5) thus reproducible: set.seed(23286) # Use to reproduce the sample below rbinom(10, size=1, p=.5)

The seed for the random number generator is stored in the workspace in a hidden variable (.Random.seed) that changes whenever there has been a call to the random number generator. This ensures that any new simulation will be independent of earlier simulations. When the workspace is saved, .Random.seed is stored as part of the workspace. This ensures that, when the workspace is loaded again, the seed will be restored to its value when the workspace was last saved. Any new simulations will then be independent of those prior to the save. In order to take advantage of this feature, be sure to save the workspace at the end of each session.

Sampling from discrete distributions

Values can be simulated from any of many different distributions. We will offer examples of simulated binomial, Poisson, and normal samples. As a first example, simulate a random sequence of 10 binary digits (0s or 1s) from a population with a specified proportion of 1s, here 50% (i.e., a Bernoulli distribution):

3.3 Simulation of random numbers and random samples

x x x x x x x xx x xxxx xxxx xx xxxxxxxxxxx xx x xxxxxxxxxxxx xxxx 7

8

9

10

xx xxxx xxxxx x xxxxxx xx xx x xxxxxxxxx x xxxxxxxxxxxxxxxxx

x

x xx x xx x xx xx x xxxxxx xxx xxxxxxxxxxxxx xxxxxxxxxxxxxxx

x x x x x x x x x x xxxxx x xxxxxxx x xxx xxxxxxx x x xxxx xxxxxxxxx x

87

x xx x xxx x xxx x xxxxx xx xxxxxx x xx xxxxxxx x x xxxxxxxxxxx x x

12

Figure 3.4 Each panel shows a simulated distribution of 50 values from a normal distribution with mean = 10 and sd = 1. The underlying theoretical normal curve is overlaid on the leftmost panel. > rbinom(10, size=1, p=.5) 1 0 0 0 1 1 1 0 1 0

# 10 Bernoulli trials, prob=0.5

The random sample is different on each occasion, depending on the seed. To generate the numbers of daughters in a simulated sample of 25 four-child families, assuming that males and females are equally likely, use the rbinom() function thus: # For the sequence that follows, precede with set.seed(9388) > rbinom(25, size=4, prob=0.5) [1] 3 1 2 4 1 2 0 3 2 1 2 3 2 4 2 1 1 1 2 2 3 2 0 2 2

Now simulate the number of raisins in 20 raisin buns, where the expected number of raisins per bun is 3: > set.seed(9388) > rpois(20, 3) [1] 3 3 4 1 2 2 3 1 1 4 3 0 1 1 3 1 0 4 5 2

3.3.1 Sampling from the normal and other continuous distributions

The function rnorm() generates random deviates from the normal distribution. To generate 10 random values from a standard normal distribution, we type: > options(digits=2) # Suggest number of digits to display > rnorm(10) # 10 random values from the normal distribution # For our sequence, precede with set.seed(3663) [1] -0.599 -1.876 1.441 -1.025 0.612 -1.669 0.138 -0.099 1.010 0.013

Figure 3.4 demonstrates the use of simulation to indicate the extent of sample-to-sample variation in histogram summaries of the data, when five independent random samples of 50 values are taken from a normal distribution. Figure 3.4 shows histograms from five such samples.7 Histograms do not discriminate well between sample values that are consistent 7

## The following gives a rough equivalent of the figure: set.seed (21) # Use to reproduce the data in the figure par(mfrow=c(2,3)) x <- pretty(c(6.5,13.5), 40) for(i in 1:5){ y <- rnorm(50, mean=10, sd=1) hist(y, prob=TRUE, xlim=c(6.5,13.5), ylim=c(0,0.5), main="") lines(x, dnorm(x,10,1)) } par(mfrow=c(1,1))

88

Statistical models

with a normal distribution, and sample values that are not. A better tool for assessing normality, the normal probability plot, will be described in Subsection 3.4.2. Calculations for other distributions, for example runif() to generate uniform random numbers or rexp() to generate exponential random numbers, follow the same pattern. runif(n = 20, min=0, max=1) # 20 numbers, uniform distn on (0, 1) rexp(n=10, rate=3) # 10 numbers, exponential, mean 1/3. ## Exercises at the end of this chapter explore further possibilities.

3.3.2 Simulation of regression data

The following code shows how to simulate a sample of n observations from the model: y = b0 + b1 x + ε where ε is normally distributed with standard deviation σ . We take n = 8, the intercept to be 2, the slope to be 3, and σ to be 2.5 in our simulation, and we use a fixed equally spaced design for the predictor values: > > > > > >

options(digits=3) n <- 8; x <- seq(1,n); sigma <- 2.5; b0 <- 2; error <- rnorm(n, sd=sigma) y <- b0 + b1*x + error

b1 <- 3

t(data.frame(x,y)) [,1] [,2] [,3] [,4] [,5] [,6] [,7] [,8] x 1.00 2.00 3.00 4.0 5.0 6 7.0 8.0 y 6.85 8.96 9.49 12.2 19.1 20 26.2 28.5

It is often useful to repeatedly simulate data from a fitted model, then re-fitting to each new set of simulated data. This provides a check on variation under such repeated simulation. The function simulate() can be used for this purpose. Thus to do 10 simulations based on the model that was fitted to the roller data, do: roller.lm <- lm(depression ˜ weight, data=roller) roller.sim <- simulate(roller.lm, nsim=20) # 20 simulations

The object roller.sim is a data frame with 20 columns, i.e., one column for each of the 20 simulations. Each column has values of depression, simulated from the fitted model at each level of weight. To visualize this output, enter with(roller, matplot(weight, roller.sim, pch=1, ylim=range (depression)) points(roller, pch=16)

3.3.3 Simulation of the sampling distribution of the mean

The sampling distribution of the mean is the distribution of the means of repeated random samples of size n. The standard deviation of this sampling distribution has the name standard error of the mean (SEM). If the population mean is µ and the standard deviation

3.3 Simulation of random numbers and random samples

89

1.0 1.5 2.0 2.5 3.0 3.5

Sample size = 3

Sample size = 9

Sample size = 30

4

Density

3 2 1 0 1.0 1.5 2.0 2.5 3.0 3.5

1.0 1.5 2.0 2.5 3.0 3.5

Figure 3.5 The density curves are from simulations of the sampling distribution of the mean, for a distribution that is mildly skew. Each density curve is from 1000 simulations, which is large enough to give an accurate visual indication of the distribution.

is σ , then s SEM = √ . n The Central Limit Theorem will be important in the discussion of inference in Chapter 4. This theorem implies that, for large enough n, this sampling distribution will closely approximate the normal.8 The sample size n needed so that the normal is a good approximation will depend on the distribution of the population from which samples are taken. Figure 3.5 shows the simulated sampling distribution of the mean, for samples from a distribution that is mildly skew. Even for a sample size of 3, much of the skewness has gone. Code that generates samples from the sampling distribution of means of the required sample sizes is: ## Function to generate n sample values; skew population sampvals <- function(n)exp(rnorm(n, mean = 0.5, sd = 0.3)) ## Means across rows of a dimension nsamp x sampsize matrix of ## sample values gives nsamp means of samples of size sampsize. samplingDist <- function(sampsize=3, nsamp=1000, FUN=mean) apply(matrix(sampvals(sampsize*nsamp), ncol=sampsize), 1, FUN) size <- c(3,10,30) ## Simulate means of samples of 3, 9 and 30; place in dataframe df <- data.frame(y3=samplingDist(sampsize=size[1]), y9=samplingDist(sampsize=size[2]), y30=samplingDist(sampsize=size[3]))

The following then gives a slightly simplified version of Figure 3.5: ## Simulate source population (sampsize=1) y <- samplingDist(sampsize=1) densityplot(˜y3+y9+y30, data = df, outer=TRUE, layout = c(3,1), plot.points = FALSE, panel = function(x, ...) { 8

More precisely, the distribution of the sample mean approximates the normal distribution with arbitrary accuracy, for a sample that is large enough, provided the measurements are independent, and their standard deviation is finite. There are similar results for a number of other sample statistics.

90

Statistical models panel.densityplot(x, ..., col = "black") panel.densityplot(y, col = "gray40", lty = 2, ...) })

Code that will reproduce the strip panel labels is in the footnote.9 The plots can alternatively be obtained using the function sampdist() (DAAG), with default arguments. The skewness of the population can be increased by increasing sd in the call to sampvals(). 3.3.4 Sampling from finite populations

We can use the sample() function to generate a simple random sample from a given set of numbers. Suppose, for example, that names on an electoral roll are numbered from 1 to 9384. We can obtain a random sample of 15 individuals as follows: > ## For the sequence below, precede with set.seed(3676) > sample(1:9384, 15, replace=FALSE) [1] 9178 2408 8724 173 106 4664 3787 6381 5098 3228 8321 165 7332 9036 540

This gives the numerical labels for the 15 individuals that we should include in our sample. The task is then to find them! The option replace=FALSE gives a without-replacement sample, i.e., it ensures that no one is included more than once. To randomly assign 10 plants (labeled from 1 to 10, inclusive) to one of two equal-sized groups, control and treatment, the following code could be used: > ## For the sequence below, precede with set.seed(366) > split(sample(seq(1:10)), rep(c("Control","Treatment"), 5)) > # sample(1:10) gives a random re-arrangement (permutation) > # of 1, 2, ..., 10 $Control [1] 6 8 3 7 9 $Treatment [1] 5 4 2

1 10

We then assign plants 6, 8, 3, 7, and 9 to the control group. By choosing the plants in such a manner, we avoid biases that could arise, for example, due to choosing healthier-looking plants for the treatment group. With-replacement samples

We can randomly sample from the set {1, 2, . . . , 10}, allowing for repeated observations, by using: > sample(1:10, replace=TRUE) [1] 7 5 2 1 2 3 1 5 7 6 9

## Use strip.custom to customize the strip labeling doStrip <- strip.custom(strip.names = TRUE, factor.levels = as.expression(size), var.name = "Sample size", sep = expression(" = ")) ## Then include the argument ’strip=doStrip’ in the call to densityplot

3.4 Model assumptions

91

Cluster sampling

Cluster sampling is one of many different probability-based variants on simple random sampling. See Barnett (2002). In surveys of human populations cluster-based sampling, e.g., samples of households or of localities, with multiple individuals from each chosen household or locality, is likely to introduce a cluster-based form of dependence. The analysis must then take account of this clustering. Standard inferential methods require adaptation to take account of the fact that it is the clusters that are independent, not the individuals within the clusters. Donner and Klar (2000) describe methods that are designed for use in health research. Simulation in teaching and research

The R package animation (Xie and Cheng, 2008) has a number of simulations that are intended for use in teaching or self-instruction. In statistical theory and practice, simulation is widely used to determine the statistical properties of models and/or of model statistics in cases where it has not been possible to derive analytical results. 3.4 Model assumptions Common model assumptions are normality, independence of the elements of the error term, and homogeneity of variance (i.e., the standard deviations of all measurements are the same). If certain assumptions fail to hold, a statistical method may be invalid. Other assumptions may not be as important; we say that the method used is robust against those assumptions. Much of the art of applied statistics comes from knowing which assumptions are important and need careful checking. There are few hard and fast rules. 3.4.1 Random sampling assumptions – independence

Typically, the data analyst has a sample of values that will be used as a window into a wider population. Ideally, data should be gathered in such a way that the independence assumption is guaranteed. This is why randomization is so important in designed experiments, and why random sampling is so important in designed sample surveys. Elementary analysis methods can be modified or extended in various ways to handle modifications of the simple independent random sampling scheme. For example, we can modify the methodology to handle analyses of data from a random sample of clusters of individuals. In practice, analysts may make the random sampling assumption when the selection mechanism does not guarantee randomness. Inferences from data that are chosen haphazardly are inevitably less secure than when we have random samples. Random selection avoids the conscious or unconscious biases that result when survey or other samplers make their own selection, or take whatever items seem suitable. Where there has not been explicit use of a random sampling mechanism, it is necessary to consider carefully how this may have affected the data. Is some form of dependence structure likely? Temporal and spatial dependence arise because values that are close together in time

92

Statistical models

or space are relatively more similar. Is there clustering that arises because all individuals within chosen streets, or within chosen families, have been included. Two individuals in the same family or in the same street may be more similar than two individuals chosen at random from the same city suburb. Often samples are chosen haphazardly, e.g., an experimenter may pick a few plants from several different parts of a plot. Or a survey interviewer may, in a poor-quality survey, seek responses from individuals who can be found in a shopping center. Self-selected samples can be particularly unsatisfactory, e.g., those readers of a monthly magazine who are sufficiently motivated to respond to a questionnaire that is included with the magazine. Failure of the independence assumption is a common reason for wrong statistical inferences. Detecting failure of the independence assumption is often difficult. Tests for independence are at best an occasionally useful guide. They are of little use unless we have some idea how the assumption may have failed, and the sample is large! It is in general better to try to identify the nature of any possible dependence, and use a form of analysis that allows for it. Models that do not obviously reflect mechanisms that generated the data can sometimes be useful for prediction. They can also, if their deficiencies are not understood or if they are used inappropriately, be misleading. Careful checking that the model is serving its intended purpose, and caution, are necessary. 3.4.2 Checks for normality

Many data analysis methods rest on the assumption that the data are normally distributed. Real data are unlikely to be exactly normally distributed. Broadly, gross departures from normality are a cause for concern. Small departures are of no consequence. Check especially for data that are skew. Check also for data that take a small number of discrete values, perhaps as a result of excessive rounding. Whether a specific form of departure will matter depends on the use made of the data. For modest-sized samples, only gross departures will be detectable. For small samples (e.g., less than about 10), it is typically necessary to rely on sources of evidence that are external to the data, e.g., previous experience with similar data. Graphical tools for checking for normality

As noted in Subsection 2.1.1, histograms are not an effective means for assessing whether the distribution is plausibly normal. Refer back to the five histograms shown in Figure 3.4, from five independent random samples of 50 values from a normal distribution. None of these histograms showed a close resemblance to a theoretical normal distribution. The normal probability plot

A better tool for assessing normality is the normal probability (or quantile–quantile) plot. The data values are sorted, then plotted against the ordered values that might be expected if the data really were from a normal distribution. If the data are from a normal distribution, the plot should approximate a straight line. Figure 3.6 shows normal probability plots for

3.4 Model assumptions −2 −1

0

1

2

−2 −1

3

93 0

1

2

5

12 11 10 9 8 7 −2 −1

0

1

2

−2 −1

0

1

2

−2 −1

0

1

2

Figure 3.6 Normal probability plots for the same random normal data as in Figure 3.4.

the same five sets of 50 normally distributed values as were displayed in Figure 3.4. The code is: ## Use qreference() (DAAG) ## With seed=21, the random numbers are as in the previous figure qreference(m=50, seed=21, nrep=5, nrows=1) # 50 values per panel

An alternative is to use the lattice function qqmath().10 To obtain a single plot of this type, the function qqnorm(), which relies on functions from base graphics, may be used. Specify, e.g., qqnorm(rnorm(50)). Displays such as Figure 3.6 help the data analyst to calibrate the eye, to get a feel for the nature and extent of departures from linearity that are to be expected in random normal samples of the specified size, here 50. It is useful to repeat the process several times. Such plots give a standard against which to compare the normal probability plot for the sample. Note that, by plotting against the ordered values that might be expected from the relevant distribution, the methodology allows a comparison with any distribution that is of interest. The sample plot, set alongside plots for random normal data

Consider data from an experiment that tested the effect of heat on the stretchiness of elastic bands. Eighteen bands were first tested for amount of stretch under a load that stretched the bands by a small amount (the actual load was 425 g, thought small enough not to interfere with the elastic qualities of the bands). This information was used to arrange bands into nine pairs, such that the two members of a pair had similar initial stretch. One member of each pair, chosen at random, was placed in hot water (60–65 ◦ C) for four minutes. The other member of the pair remained at ambient temperature. All bands were then measured for amount of stretch under a load of 1.35 kg weight. Table 3.1 shows the results. In the next chapter, the heated–ambient differences will be the basis for various statistical calculations. Is the distribution consistent with the assumption of normality? The normal probability plot for these data is in the lower left panel of Figure 3.7. The other seven plots are for samples (all of size 9) of simulated random normal values. They give a standard against which to compare the plot for the experimental data. There is no obvious feature that distinguishes the plot in the lower left panel from the seven reference plots. The code is: 10

## Set seed to get the same data as earlier library(lattice) qqmath(˜rnorm(50*5)|rep(1:5,rep(50,5)), layout=c(5,1), aspect=1)

94

Statistical models

Table 3.1 Eighteen elastic bands were divided into nine pairs, with bands of similar stretchiness placed in the same pair. One member of each pair was placed in hot water (60–65 ◦ C) for four minutes, while the other was left at ambient temperature. After a wait of about 10 minutes, the amounts of stretch, under a 1.35 kg weight, were recorded. Pair #

Heated (mm) Ambient Difference

1

2

3

4

5

6

7

8

9

244 225 19

255 247 8

253 249 4

254 253 1

251 245 6

269 259 10

248 242 6

252 255 −3

292 286 6

0 reference 4

5

10 15

reference 5 ●

●

●

● ●

● ● ●

●

●

reference 6

5

10 15

reference 7 ●

●

● ●

0

●

●

●

● ●

● ● ● ●

● ●

●

●

●

● ●

●

●

● ●

●

●

20 15 10 5 0 −5

heated − ambient

reference 1

reference 2

reference 3

● ●

●

●

● ●

●

● ● ●

●

●

●

●

●

0

●

5

10 15

●

●

● ● ●

● ● ●

● ●

●

●

●

●

● ●

●

●

●

●

0

20 15 10 5 0 −5

5

10 15

Figure 3.7 The lower left panel is the normal probability plot for heated–ambient differences. Remaining panels show plots for samples of nine numbers from a normal distribution.

## Compare normal probability plot for normal-ambient difference ## with simulated normal values: data frame pair65 (DAAG) qreference(pair65$heated - pair65$ambient, nrep=8)

The function qreference() is from the DAAG package.

How close to normal is the sampling distribution of the mean?

Often, the interest is in the normality of the sampling distribution of a mean or other statistic. Figure 3.8 simulates repeated sampling from the same mildly skew distribution as in Figure 3.5. Instead of density curves, the normal probability plots are shown. The normal probability plot for a sample from the population is the gray dashed line. This plot can be obtained by replacing densityplot() by qqmath() in the code for Figure 3.5. Alternatively, use the function sampdist() (DAAG), with the argument plot.type="qq".

3.4 Model assumptions −2

Sample size = 3

−1

0

1

95

2

Sample size = 9

Sample size = 30

3.0 2.5 2.0 1.5 1.0 −2

−1

0

1

2

−2

−1

0

1

2

Figure 3.8 The normal probability plots are from simulations of the sampling distribution of the mean, for the same mildly skew distribution as in Figure 3.5. The plot for the population is shown in gray. The panels show the plots for samples of respective sizes 3, 9, and 30. Each is from 1000 simulations.

Notice that the plot for the sampling distribution is increasingly linear, with a reduced slope, as one goes from n = 3 to n = 9 to n = 30. The reduced slope reflects the reduced SEM, which goes from √σ3 to √σ9 to √σ30 . Formal statistical testing for normality?

Both formal statistical tests for normality and less formal graphical checks are of limited usefulness. With small or modest-sized samples, only gross departures are likely to be detected. Large samples will show departures from normality that may be too small to have any practical consequence for standard forms of statistical analysis. For a statistic such as a mean or a regression slope, the effects of averaging may give a close approximation to normality, even when the underlying population is clearly not normally distributed. (This is a consequence of the “Central Limit Theorem” that was discussed in Subsection 3.3.3.) Depending then on the specific context, normality may not be an important issue for analyses where samples are large. Tests for normality will detect non-normality in contexts where there is the least reason to be concerned about it. 3.4.3 Checking other model assumptions

In Chapter 2, we discussed a number of exploratory techniques that can aid in checking whether the standard deviation is the same for all observations in a data set. Following analysis, a plot of residuals against fitted values may give useful indications. For example, residuals may tend to fan out as fitted values increase, giving a “funnel” effect, a fairly sure sign that the standard deviation is increasing. Alternatively, or additionally, there may be evidence of outliers – one or more unusually large residuals. The major concern may however be to identify points, whether or not outliers, that have such high influence that they distort model estimates. 3.4.4 Are non-parametric methods the answer?

Non-parametric methods have been developed to handle situations where normality or other model assumptions are in question, and where it might be difficult to pose an alternative

96

Statistical models

model. These methods are only sometimes useful, and they still depend on assumptions, and we still need assurance that these assumptions are realistic. If used in a way that ignores structure in the data that we should be modeling, we risk missing insights that parametric methods may provide. Building too little structure into a model can be just as bad as building in too much structure. There is a trade-off between the strength of model assumptions and the ability to find effects. Newer methodologies such as lowess smoothing are welcome and useful additions to the statistical toolbox. However, if we assume a linear relationship, we may be able to find it, where we will find nothing if we look for a general form of smooth curve or a completely arbitrary relationship. This is why simple non-parametric approaches are often unsatisfactory – they assume too little. Often they assume much less than we know to be true. Johnson (1995) has useful comments on the role of non-parametric tests. In part, the objection is to a view of non-parametric modeling that is too limited.

3.4.5 Why models matter – adding across contingency tables

The multi-way table UCBAdmissions (datasets package) has admission frequencies, by sex, for the six largest departments at the University of California at Berkeley in 1973 (Bickel et al., 1975). Do the data provide evidence, across the University as a whole, of sex-based discrimination? Note the margins of the table: > str(UCBAdmissions) table [1:2, 1:2, 1:6] 512 313 89 19 353 207 17 8 120 205 ... - attr(*, "dimnames")=List of 3 ..$ Admit : chr [1:2] "Admitted" "Rejected" ..$ Gender: chr [1:2] "Male" "Female" ..$ Dept : chr [1:6] "A" "B" "C" "D" ...

First, calculate overall admission rates (percentages) for females and males. > ## Tabulate by Admit and Gender > byGender <- margin.table(UCBAdmissions, margin=1:2) > round(100*prop.table(byGender, margin=2)["Admitted", ], 1) Male Female 44.5 30.4

Admission rates will now be calculated for individual departments: > ## Admission rates, by department > round(100*prop.table(UCBAdmissions, + margin=2:3)["Admitted", , ], 1) Dept Gender A B C D E F Male 62.1 63 36.9 33.1 27.7 5.9 Female 82.4 68 34.1 34.9 23.9 7.0

As a fraction of those who applied, females were strongly favored in department A, and males somewhat favored in departments C and E.

3.5 Recap

97

Look now, for each department, at the numbers of males applying as a proportion of the total number of male applicants, and similarly for females: > ## Calculate totals, by department, of males & females applying > (applicants <- margin.table(UCBAdmissions, margin=2:3)) Dept Gender A B C D E F Male 825 560 325 417 191 373 Female 108 25 593 375 393 341 > ## Calculate proportions of male & female applicants > round(100*prop.table(applicants, margin=1), 1) Dept Gender A B C D E F Male 30.7 20.8 12.1 15.5 7.1 13.9 Female 5.9 1.4 32.3 20.4 21.4 18.6

Relatively few females (5.9%) applied to department A, while a high proportion (32.3% and 21.4% respectively) applied to departments C and E where admission rates were relatively low. The very high number of males applying to departments A and B has biased the male rates towards the relatively high admission rates in those departments, while the relatively high number of females applying to departments C, D and F biased the overall female rates towards the low admission rates in those departments. The overall bias arose because males favored departments where there were a relatively larger number of places. What model is in mind? Is the aim to compare the chances of admission for a randomly chosen female with the chances of admission for a randomly chosen male? The relevant figure is then the overall admission rate of 30.4% for females, as against 44.5% for males. Or, is the interest in the chances of a particular student who has decided on a department? A female had a much better chance than a male in department A, while a male had a slightly better chance in departments C and E. Here, information was available on the classifying factor on which it was necessary to condition. This will not always be the case. In any such tabulation, it is always possible that there is some further variable that, when conditioned on, can reverse or otherwise affect an observed association. The results that give the overall proportions are, for these data and depending on the intended use, an unsatisfactory and potentially misleading summary. The phenomenon that they illustrate, known as Simpson’s paradox or as the Yule–Simpson effect, is discussed in Aldrich (1995), Simpson (1951). In any overall analysis, the effect of the classifying (or conditioning) factor sex must be explicitly incorporated in the model. There are various ways to do this. Section 8.3 demonstrates one suitable approach. See also Exercise 11 in Chapter 4, and the references given there.

3.5 Recap Statistical models have both deterministic and random error components, or signal components and noise components. In simpler cases, which include most of the cases we

98

Statistical models

consider, observation = signal + noise. After fitting a model, we have observation = fitted value + residual which we can think of as observation = smooth component + rough component. The hope is that the fitted value will recapture most of the signal, and that the residual will contain mostly noise. Unfortunately, as the relative contribution of the noise increases, r it becomes harder to distinguish between signal and noise, r it becomes harder to decide between competing models. Model assumptions, such as normality, independence, and constancy of the variance, should be checked, to the extent that this is possible. 3.6 Further reading Finding the right statistical model is an important part of statistical problem-solving. Chatfield (2002, 2003b) has helpful comments. Clarke (1968) has a useful discussion of the use of models in archaeology. See also the very different points of view of Breiman and Cox (as discussant) in Breiman (2001). Our stance is much closer to Cox than to Breiman. See also our brief comments on Bayesian modeling in Section 4.10. Johnson (1995) comments critically on the limitations of widely used non-parametric methods. See Hall (2001) for an overview of non-parametrics from a modern perspective. References for further reading

Breiman, L. 2001. Statistical modeling: the two cultures. Statistical Science 16: 199– 215. Chatfield, C. 2002. Confessions of a statistician. The Statistician 51: 1–20. Chatfield, C. 2003b. Problem Solving. A Statistician’s Guide, 2nd edn. Clarke, D. 1968. Analytical Archaeology. Hall, P. 2001. Biometrika centenary: non-parametrics. Biometrika 88: 143–65. Johnson, D. H. 1995. Statistical sirens: the allure of non-parametrics. Ecology 76: 1998– 2000. 3.7 Exercises 1. The distance that a body, starting at rest, falls under gravity in t seconds is commonly given as d = 12 gt 2 , where g  9.8 msec−2 . The equation can be modified to take account of the effects of air resistance, which will vary with barometric pressure and other atmospheric conditions. Will a time–distance relationship that is obtained for a human dummy that falls from a height of some thousands of meters be useful in predicting the time–distance relationship for another dummy,

3.7 Exercises

99

or for a human, falling at another time from a similar height? Or is the situation comparable to that for the lawn roller data in Subsection 3.1.1, where the relationship is likely to be different for different lawns? [Humans have very occasionally survived falls from such heights. See http://www.greenharbor.com/fffolder/ffresearch.html] 2. Hooke’s law of elasticity is an approximation which states that the amount by which a spring or other elastic body deforms is proportional to the applied force. Data are obtained for one spring. Can those data be used to make predictions for another spring that has been manufactured in the same way? How can the accuracy of such predictions be tested? 3. An experimenter intends to arrange experimental plots in four blocks. In each block there are seven plots, one for each of seven treatments. Use the function sample() to find four random permutations of the numbers 1 to 7 that will be used, one set in each block, to make the assignments of treatments to plots. 4. Use y <- rnorm(100) to generate a random sample of size 100 from a normal distribution. (a) Calculate the mean and standard deviation of y. (b) Use a loop to repeat the above calculation 25 times. Store the 25 means in a vector named av. Calculate the standard deviation of the values in av. (c) Create a function that performs the calculations described in (b). Run the function several times, showing each of the distributions of 25 means in a density plot. 5. To simulate samples from normal populations having different means and standard deviations, the mean and sd arguments can be used in rnorm(). Simulate a random sample of size 20 from a normal population having a mean of 100 and a standard deviation of 10. 6. Use mfrow to set up the layout for a 3 by 4 array of plots. In the top 4 panels, show normal probability plots for 4 separate “random” samples of size 10, all from a normal distribution. In the middle 4 panels, display plots for samples of size 100. In the bottom 4 panels, display plots for samples of size 1000. Comment on how the appearance of the plots changes as the sample size changes. 7. The function runif()generates a sample from a uniform distribution, by default on the interval 0 to 1. Try x <- runif(10), and print out the resulting numbers. Then repeat Exercise 5 above, but taking samples from a uniform distribution rather than from a normal distribution. What shape do the plots follow? 8. The function pexp(x, rate=r) can be used to compute the probability that an exponential variable is less than x. Suppose the time between accidents at an intersection can be modeled by an exponential distribution with a rate of 0.05 per day. Find the probability that the next accident will occur during the next three weeks. 9. Use the function rexp() to simulate 100 exponential random numbers with rate 0.2. Obtain a density plot for the observations. Find the sample mean of the observations. Compare with the population mean (the mean for an exponential population is 1/rate). 10.∗ This exercise investigates simulation from other distributions. The statement x
100

Statistical models

11. The following data represent the total number of aberrant crypt foci (abnormal growths in the colon) observed in seven rats that had been administered a single dose of the carcinogen azoxymethane and sacrificed after six weeks (thanks to Ranjana Bird, Faculty of Human Ecology, University of Manitoba for the use of these data): 87 53 72 90 78 85 83 Enter these data and compute their sample mean and variance. Is the Poisson model appropriate for these data? To investigate how the sample variance and sample mean differ under the Poisson assumption, repeat the following simulation experiment several times: x <- rpois(7, 78.3) mean(x); var(x) 12.∗ A Markov chain is a data sequence which has a special kind of dependence. For example, a fair coin is tossed repetitively by a player who begins with $2. If “heads” appear, the player receives one dollar; otherwise, she pays one dollar. The game stops when the player has either $0 or $5. The amount of money that the player has before any coin flip can be recorded – this is a Markov chain. A possible sequence of plays is as follows: Player’s fortune: 2 1 2 3 4 3 2 3 2 3 2 1 0 Coin Toss result: T H H H T T H T H T T T Note that all we need to know in order to determine the player’s fortune at any time is the fortune at the previous time as well as the coin flip result at the current time. The probability of an increase in the fortune is 0.5 and the probability of a decrease in the fortune is 0.5. The transition probabilities can be summarized in a transition matrix: 

1 0.5  0  P = 0  0 0

0 0 0 0.5 0.5 0 0 0.5 0 0 0 0

0 0 0 0 0.5 0 0 0.5 0.5 0 0 0

 0 0  0   0  0.5 1

The (i, j ) entry of this matrix is the probability of making a change from the value i to the value j . Here, the possible values of i and j are 0, 1, 2, . . . , 5. According to the matrix, there is a probability of 0 of making a transition from $2 to $4 in one play, since the (2, 4) element is 0; the probability of moving from $2 to $1 in one transition is 0.5, since the (2, 1) element is 0.5. The following function can be used to simulate N values of a Markov chain sequence, with transition matrix P : Markov <- function (N=100, initial.value=1, P) { X <- numeric(N) X[1] <- initial.value + 1 # States 0:5; subscripts 1:6 n <- nrow(P) for (i in 2:N){ X[i] <- sample(1:n, size=1, prob=P[X[i-1], ])} X - 1 } Simulate 15 values of the coin flip game, starting with an initial value of $2. Repeat the simulation several times.

3.7 Exercises

101

13. A Markov chain for the weather in a particular season of the year has the transition matrix, from one day to the next:   Sun Cloud Rain  Sun 0.6 0.2 0.2    Pb =   0.4 0.4   Cloud 0.2 Rain 0.4 0.3 0.3 It can be shown, using linear algebra, that in the long run this Markov chain will visit the states according to the stationary distribution: Sun 0.641

Cloud 0.208

Rain 0.151

A result called the ergodic theorem allows us to estimate this distribution by simulating the Markov chain for a long enough time. (a) Simulate 1000 values, and calculate the proportion of times the chain visits each of the states. Compare the proportions given by the simulation with the above theoretical proportions. (b) Here is code that calculates rolling averages of the proportions over a number of simulations and plots the result. It uses the function rollmean() from the zoo package. plotmarkov
Try varying the number of simulations and the width of the window. How wide a window is needed to get a good sense of the stationary distribution? This series settles down rather quickly to its stationary distribution (it “burns in” quite quickly). A reasonable width of window is, however, needed to give an accurate indication of the stationary distribution.

4

A review of inference concepts

A random sample is a set of values drawn independently from a larger population. A (uniform) random sample has the characteristic that all members of the population have an equal chance of being drawn. In the previous chapter, we discussed the implications of drawing repeated random samples from a normally distributed population, where the probability that a value lies in a given interval is governed by the normal density. This chapter will expand upon that discussion by using the idea of a sampling distribution, with its associated standard error, to assess estimation accuracy. Confidence intervals and tests of hypotheses offer a formal basis for inference, based on the sampling distribution. We will comment on weaknesses in the hypothesis testing framework. 4.1 Basic concepts of estimation This section will introduce material that is fundamental to inference. 4.1.1 Population parameters and sample statistics

Parameters, such as the mean (µ) or standard deviation (σ ), numerically summarize various aspects of a population. Such parameters are usually unknown and are estimated using statistics calculated using a random sample taken from the population. The sample mean is an example of a statistic, and it is used to estimate the population mean. Other commonly used statistics are the proportion, standard deviation, variance, median, the quartiles, the slope of a regression line, and the correlation coefficient. Each may be used as an estimate of the corresponding population parameter. 4.1.2 Sampling distributions

Subsection 3.3.3 introduced the sampling distribution of the mean: the distribution of sample means, under repeated random sampling. The standard deviation of this sampling distribution is called the standard error of the mean (SEM). The SEM is a measure of the accuracy of the sample mean, as an estimate of the population mean. The challenge is to use the one sample that is available, together with the assumption of independent and identically distributed sample values, to infer the sampling distribution of the mean. Two approaches will be described. The first, used in the main part of this chapter, relies on statistical theory – the Central Limit Theorem. The second, relying on repeated

4.1 Basic concepts of estimation

103

resampling from the one available sample, will be the subject of Subsections 4.7.3 and 4.7.4. Reliance on the Central Limit Theorem

As a consequence of the Central Limit Theorem, the sampling distribution of the mean can, for a population with mean µ and standard deviation σ , often be well approximated by a √ normal distribution with mean µ and standard deviation σ/ n. An estimate of the SEM is thus s SEM = √ n where s is an estimator of the population standard deviation σ . Refer back to Figure 3.5 (Subsection 3.3.3). This deceivingly simple formula, relating the SEM to the standard deviation, hides quite complex mathematical ideas. Note that if the data are not independent, then the formula does not apply. Other statistics, such as the sample proportion, have their own sampling distributions. Often, these sampling distributions are also reasonably well approximated by a normal distribution. 4.1.3 Assessing accuracy – the standard error

A small SEM suggests that the sample mean is close to the population mean, while a large SEM allows for the possibility that the sample and population means may differ widely. The data frame pair65, shown earlier in Table 3.1, has information on nine sets of paired comparisons, leading to nine differences in the amount of stretch under a 1.35 kg weight. These were: Difference 19 8 4 1 6 10 6 −3 6 √ 1 The mean is 6.33, the SD is s = 6.10, and SEM = 6.10/ 9 = 2.03. We may report: “The mean change is 6.33 [SEM 2.03], based on n = 9 values”, or “The mean change is 6.10/2.03 (= 3.11) times the standard error”. 4.1.4 The standard error for the difference of means

Where there are two independent samples of size n1 and n2 , the comparison is usually in the form of a difference: x¯1 − x¯2 where x¯1 and x¯2 denote the respective sample means. If the corresponding standard errors are denoted by SEM1 and SEM2 , then the standard error of the difference (SED) is  SED = SEM21 + SEM22 . 1

## Calculate heated-ambient; take heated & ambient from columns of pair65 test <- with(pair65, heated-ambient) c(mean = mean(test), SD = sd(test), SEM = sd(test)/sqrt(length(test)))

104

A review of inference concepts

If all SEMs are the same, then for all comparisons, √ SED = 2 × SEM. It is sometimes reasonable to assume equality of the standard deviations in the populations from which the samples are drawn. Then s s SEM1 = √ , SEM2 = √ n1 n2 and the formula can be written as

 SED = s

1 1 + n1 n2

where s is the pooled standard deviation estimate described in Subsection 2.2.3. As an example, consider the unpaired elastic band experiment data of Subsection 2.2.3.  1 1 + 11 = The pooled standard deviation estimate is 10.91. Hence, the SED is 10.91 × 10 2 4.77. 4.1.5 ∗ The standard error of the median

For data from a normal distribution, there is a similarly simple formula for the standard error of the median. It is  s π s SEmedian = √ ≈ 1.25 √ . 2 n n The standard error of the median is thus about 25% greater than the standard error of the mean. For data from a normal distribution, the population mean can be estimated more precisely than can the population median. Consider again the cuckoos data. The median and standard error for the median of the egg lengths in the wrens’ nests are 21.0 and 0.244, respectively.3 A different formula for the standard error of the median, one that depends on the distribution, must be used when the data cannot reasonably be approximated by a normal model.

2

3

## Heated vs ambient; unpaired elastic band data heated <- c(254, 252, 239, 240, 250, 256, 267, 249, 259, 269) ambient <- c(233, 252, 237, 246, 255, 244, 248, 242, 217, 257, 254) v1 <- var(heated) # 10 numbers; 10-1 = 9 d.f. v2 <- var(ambient) # 11 numbers; 11-1 = 10 d.f. v <- (9*v1 + 10*v2)/(9+10) # Pooled estimate of variance # Estimate SED; variances may not be equal c(sem1 = sqrt(v1/10), sem2 = sqrt(v2/11), sed = sqrt(v1/10 + v2/11)) # Estimate SED; use pooled estimate c(sd = sqrt(v), sed = sqrt(v1/10 + v2/11)) ## median and SD for length, by species: data frame cuckoos (DAAG) wren <- split(cuckoos$length, cuckoos$species)$wren median(wren) n <- length(wren) sqrt(pi/2)*sd(wren)/sqrt(n) # this SE computation assumes normality

4.1 Basic concepts of estimation

0.4 0.3 0.2

0.2

0.0

0.1

Normal t (3 d.f.)

0.1

Probability density

Normal t (8 d.f.)

0.3

0.4

B

0.0

Probability density

A

105

0

2

4

0

No. of SEMs from mean

2

4

No. of SEMs from mean

Figure 4.1 Panel A overlays the density for a normal distribution with the density for a t-distribution with 8 d.f. Panel B overlays the density for a t-distribution with 3 d.f.

4.1.6 The sampling distribution of the t-statistic

The formula t=

x¯ − µ SEM

counts the number of standard error units between the true value µ and the sample estimate ¯ It can be thought of as a standardized distance between the true mean and the sample x. mean. The variability in t has two sources: the sampling variability of x¯ and the sampling variability of SEM. The replacing of σ by s introduces an uncertainty that is larger as the degrees of freedom n − 1 in s are smaller. Hence the use of a t-distribution with n − 1 degrees of freedom (d.f.), where if σ was known precisely a normal distribution would be used. The t-statistic becomes more and more like a standard normal random variable as the sample size increases. Figure 4.1B shows the density curve for a normal distribution overlaid with those for t-distributions with 8 and 3 d.f. respectively. The main difference, in each case, is in the tails, and much larger for the t-distribution with the smaller d.f., 3 as opposed to 8. In the terminology of Subsection 3.2.2, the t-distribution is heavy-tailed – heavier for smaller than for larger degrees of freedom. For the data in the data frame pair65, the relevant inference is suitably based on the mean d¯ of the differences that were observed when the bands were heated. In order to ¯ i.e., the standardize this mean difference, it needs to be divided by its standard error SE[d], relevant statistic is t=

d¯ d¯ 6.33 = √ = = 3.11. ¯ 2.47 SE[d] s/ 9

The mean is 3.11 times the magnitude of the standard error.

106

A review of inference concepts

−2

0 qnorm(0.975) = 1.96

0.2 0.1

0.025

0.025 0.025 + 0.95 = 0.975

0.0

0.025

t probability density (8 d.f.)

0.3 0.2 0.1

0.025 + 0.95 = 0.975

0.0

Normal probability density

0.025

0.3

B

A

−2

0 qt(0.975, 8) = 2.31

Figure 4.2 Calculation of the endpoints of the symmetrically placed region that encloses 95% of the probability: (A) for a normal distribution, and (B) for a t-distribution with 8 d.f. In each panel, the upper 2.5% of the area under the curve is shaded in gray.

4.2 Confidence intervals and tests of hypotheses Calculations with the t-distribution

Calculations for the t-distribution follow the same pattern as those shown for the normal distribution in Subsection 3.2.2, but now with a distribution whose standard deviation is the SEM, which has to be estimated. There are two sorts of calculation that may be useful, both of which can be related to Figure 4.2: r Given the distance from the mean, calculate the area under the curve. Thus, calculate the area under the density curve within some specified number of standard errors either side of the mean. For this, use functions that have p as their initial letter, here pnorm() and pt: > # Plus or minus 1.96SE normal distribution limits, e.g. > pnorm(1.96) - pnorm(-1.96) [1] 0.95 > # Plus or minus 2.31SE t distribution (8 df) limits, e.g. > pt(2.31, 8) - pt(-2.31,8) # 2.31 SEs either side [1] 0.95

r Given an area under the curve, calculate the limit or limits. Thus, what distance from the mean gives an area under the curve, up to and including that point, that takes some specified value? For this, use functions that have q as their initial letter, here qnorm() and qt(): > qnorm(0.975) [1] 1.96 > qt(0.975, 8) [1] 2.31

# normal distribution # t-distribution with 8 d.f.

Confidence intervals of 95% or 99%

The second of these statements makes it possible to say that in sampling from the sampling ¯ d−µ √ , 95% of the values of t8 will lie between −2.31 and 2.31, i.e., that distribution of t8 = s/ 9

4.2 Confidence intervals and tests of hypotheses

107

Table 4.1 Comparison of normal distribution endpoints (multipliers for the SEM) with the corresponding t-distribution endpoints on 8 d.f. Probability enclosed between limits 68.3% 95% 99% 99.9%

Number of SEMs

Cumulative probability

Normal distribution

t-Distribution (8 d.f.)

84.1% 97.5% 99.5% 99.95%

1.0 1.96 2.58 3.29

1.07 2.31 3.36 5.04

d¯ − µ will lie between −2.31s and 2.31s. In other words, in 95% of such samples d¯ will lie within a distance 2.31s of µ. Furthermore (see Table 4.1, or enter qt(0.995, 8)), in 99% of such samples d¯ will lie within a distance 3.36s of µ. This leads immediately to the following “confidence” (or coverage) interval for µ: 95% CI: (6.33 − 2.03 × 2.31, 6.33 + 2.03 × 2.31) = (1.64, 11.02) 99% CI: (6.33 − 2.03 × 3.36, 6.33 + 2.03 × 3.36) = (−0.49, 13.15) Code that may be used (here, for a 95% confidence interval) is:4 ## 95% CI for mean of heated-ambient: data frame pair65 (DAAG) with(pair65, t.test(heated, ambient, paired=TRUE, conf.level=0.95)$conf.int)

The confidence interval has been constructed so that most often, when the sample is taken in the way that the one available sample has been taken, it will include the population mean. The two common choices for the long-run proportion of similar samples for which the corresponding intervals should contain the population mean are 95% and 99%. Tests of hypotheses

If the confidence interval for the population mean does not contain zero, this is equivalent to rejection of the hypothesis that the population mean is zero. Starting from a 95% confidence interval, the “significance level” for the test is p = 1 − 0.95 = 0.05. In the example just considered, the 95% confidence interval does not contain zero, while the (wider) 99% confidence interval does contain zero. Thus the hypothesis that the population mean is zero is rejected for p = 0.05, but not for p = 0.01. The value of p that is on the borderline between rejection and non-rejection is termed the p-value. This value can be obtained by doubling the probability that the t-statistic is less than −mean/SEM = −6.33/2.03 ## Probability that t-statistic (8 d.f.) is less than -6.33/2.03 > 1-pt(6.33/2.03, 8) # Equals pt(-6.33/2.03, 8) [1] 0.00713 4

## Detailed calculations; 95% CI for mean of heated-ambient pair65.diff <- with(pair65, heated-ambient) pair65.n <- length(pair65.diff) pair65.se <- sd(pair65.diff)/sqrt(pair65.n) mean(pair65.diff) + qt(c(.025,.975),8)*pair65.se

108

A review of inference concepts

Doubling 0.00713 to determine the sum of the probabilities in the two tails yields p = 0.014. The result may be summarized in the statement: “Based on the sample mean of 6.33, the population mean is greater than zero (p = 0.014)”. Formal hypothesis testing requires the statement of null and alternative hypotheses. Taking the population mean to be µ, the null hypothesis is H0 : µ = 0 while the alternative hypothesis is µ = 0. The formal methodology of hypothesis testing may seem contorted. A small p-value makes the null hypothesis appear implausible. It is not a probability statement about the null hypothesis itself, or for that matter about its alternative. All it offers is an assessment of implications that flow from accepting the null hypothesis. A straw man is set up, the statement that µ = 0. The typical goal is to knock down this straw man. By its very nature, hypothesis testing lends itself to various abuses.

What is a small p-value?

At what point is a p-value small enough to be convincing? The conventional p = 0.05 (= 5%) cutoff is too large, if results from the experiment are to be made the basis for a recommendation for changes to farming practice or to medical treatment. It may be too small when the interest is in deciding which effects merit further experimental or other investigation. There must be a careful balancing of the likely costs and benefits of any such recommendation, having regard to the statistical evidence. In any particular case, consider carefully: r Is there other relevant evidence, additional to that summarized in a p-value or confidence interval? r What is the most helpful way to present results: a p-value, or a confidence interval, or something else again?

t-Distribution versus the normal distribution

Table 4.1 compares the normal distribution multipliers with those for a t-distribution with 8 d.f., for several different choices of area under the curve. Changing from a normal distribution to a t-distribution with 8 d.f. led to a small change, from 1.0 to 1.07, for enclosing the central 68.3% of the area. There is a substantial difference, giving an increase from 1.96 to 2.31, for enclosing 95% of the area.

How good is the normal theory approximation?

For random samples from a distribution that is close to symmetric, the approximation is often adequate, even for samples as small as 3 or 4. In practice, we may know little about the population from which we are sampling. Even if the main part of the population distribution is symmetric, occasional aberrant values are to be expected. Such aberrant

4.2 Confidence intervals and tests of hypotheses

109

Table 4.2 Formulae for confidence intervals and tests of hypothesis based on the t-distribution. Confidence interval ¯ One-sample t d¯ ± tcrit SE[d] e.g.

6.33 ± 2.306 ×

6.10 √ 9

Test statistic

d.f.

t=

d¯ ¯ SE[d]

n−1

t=

6.33√ 6.10/ 9

8

n1 + n2 − 2 t = SEx¯[2x¯ −−x¯1x¯ ] 2 1  244.1 t = 253.5 − 19 253.5 − 244.1 ± 2.09 × 10.91 101 + 111 10.91 × 101 + 111 = 253.5 − 244.12.09 × 4.77 = (−0.6, 19.4)

Two-sample t x¯2 − x¯1 ± tcrit SE[x¯2 − x¯1 ] e.g.

Here, tcrit is the 97.5th percentile of a t-statistic with 8 d.f. (one-sample example) or 19 d.f. (twosample example). (The 97.5th percentile is the same as the two-sided 5% critical value.)

values do, perhaps fortunately, work in a conservative direction – they make it more difficult to detect genuine differences. The take-home message is that, especially in small samples, the probabilities and quantiles can be quite imprecise. They are rough guides, intended to assist researchers in making a judgment.

4.2.1 A summary of one- and two-sample calculations

Confidence intervals for a mean difference, or for a difference of means, have the form difference ± t-critical value × standard error of difference. The t-statistic has the form t=

difference . standard error of difference

Given t, the p-value for a (two-sided) test is defined as P (T > t) + P (T < −t) where T has a t-distribution with the appropriate number of degrees of freedom. A small p-value corresponds to a large value of |t|, regarded as evidence that the true difference is non-zero and leading to the rejection of the null hypothesis. Table 4.2 lists confidence intervals and tests in the one- and two-sample cases.5 The single-sample example is for the paired elastic band data that we discussed at the beginning of this section. The example that we use for the two-sample calculations was discussed in Subsection 2.2.3.

5

## t-test and confidence interval calculations heated <- c(254, 252, 239, 240, 250, 256, 267, 249, 259, 269) ambient <- c(233, 252, 237, 246, 255, 244, 248, 242, 217, 257, 254) t.test(heated, ambient, var.equal=TRUE)

A review of inference concepts

SED = 2.03

230 250 270 290 Amount of stretch (ambient)

25 20

●

●●● ●

15

● ● ● ● ●

● ●

● ●●

● ● ● ●

●

●

● ●

●

SE = 0.96

●

●

SE = 4.87

●

r = −0.1002

● ●

10

290

SE = 0.64

r = 0.9276 ●

270 250 230

Amount of stretch (heated)

SE = 5.43

Height of cross-fertilized plant

110

SED = ●1.21 ●

●

10 15 20 25 Height of self-fertilized plant

Figure 4.3 Second versus first member, for each pair. The first panel is for the ambient/heated elastic band data from Subsection 4.1.6, while the second is for Darwin’s plants.

When is pairing helpful?

Figure 4.3 shows, for two different sets of paired data, a plot of the second member of the pair against the first.6 The first panel is for the paired elastic band data of Subsection 4.1.6, while the second panel (for the data set mignonette) is from the biologist Charles Darwin’s experiments that compared the heights of crossed plants with the heights of selffertilized plants (data, for the wild mignonette Reseda lutea, are from p. 118 of Darwin, 1877). Plants were paired within the pots in which they were grown, with one plant on one side and one on the other. For the paired elastic band data there is a clear correlation, and the standard error of the difference is much less than the root mean square of the two separate standard errors. For Darwin’s data there is little evidence of correlation. The standard error of differences of pairs is about equal to the root mean square of the two separate standard errors. For the elastic band data, the pairing was helpful; it led to a low SED. The pairing was not helpful for Darwin’s data (note that Darwin (cited above) gives other data sets where the pairing was helpful, in the sense of allowing a more accurate comparison). If the data are paired, then the two-sample t-test corresponds to the wrong model! It is appropriate to use the one-sample approach, whether or not there is evidence of correlation between members of the same pair. What if the standard deviations are unequal?

If variances are heterogeneous (unequal variances or standard deviations), the t-statistic based on the pooled variance estimate is inappropriate. The Welch procedure gives an adequate approximation, unless degrees of freedom are very small. The Welch statistic is 6

## heated vs ambient: pair65 (DAAG); and cross vs self: mignonette (DAAG) par(mfrow=c(1,2)) plot(heated ˜ ambient, data=pair65); abline(0, 1) # left panel with(pair65, abline(mean(heated-ambient), 1, lty=2)) plot(cross ˜ self, data=mignonette); abline(0, 1) # right panel with(mignonette, abline(mean(cross-self), 1, lty=2)) par(mfrow = c(1,1))

4.2 Confidence intervals and tests of hypotheses

111

the difference in means divided by a standard error of difference that allows for unequal variances, i.e., t= where

x¯2 − x¯1 , SED 

SED =

s22 s2 + 1. n2 n1

If the two variances are unequal this does not have a t-distribution. However, critical values are quite well approximated by the critical values of a t-distribution with degrees of freedom given by a readily calculated function of the observed sample variances and sample sizes. The most commonly used approximation is that of Welch (1949), leading to the name Welch test. For details, see Miller (1986). The function t.test() has the Welch test as its default; unequal variances are assumed unless the argument var.equal=TRUE is given. Note that if n1 = n2 then the statistic is the same as for the t-test that is based on the pooled estimate of variance. However, the degrees of freedom are likely to be reduced.

Different ways to report results

For the paired elastic band data of Table 3.1, the mean difference in amount of stretch before and after heating is 6.33,√ with a standard deviation of 6.10. The standard error of this difference (SED) is thus 6.10/ 9 = 2.03. The bare minimum of information that should be reported is: “The mean change is 6.33 [SED 2.03, n = 9]”. In engineering and physical science contexts where the aim is to accompany a report of the mean with a statement of its precision, this may be enough. It is most appropriate when differences are large, of the order of more than five times the SEM for any individual treatment or four times the SED for comparing two means. Confidence intervals and hypothesis testing give this form of report a more interpretive twist. Here are some of the various alternatives:7 1. The mean change is 6.33 [SED 2.03, n = 9]. 2. The t-statistic is t = 6.333/2.034 = 3.11, on 8 (= 9 − 1) degrees of freedom. In other words, the difference is 3.11 times the standard error. 3. A 95% confidence interval for the change is (6.33 − 2.306 × 2.034, 6.33 + 2.306 × 2.034), i.e., (1.64, 11.02). [The multiplier, equal to 2.306, is the 5% two-sided critical value for a t-statistic on 8 (= 9 − 1) d.f.] 7

## Different ways to report results: calculations pair65.diff <- with(pair65, heated-ambient) n <- length(pair65.diff) av <- mean(pair65.diff); sd <- sqrt(var(pair65.diff)); se <- sd/sqrt(n) print(c(mean=av, SED=se, "mean/SED"=av/se)) # Items 1 and 2 t.test(pair65.diff) # Items 3 and 4

112

A review of inference concepts

Table 4.3 Approximate 95% confidence interval, assuming 0.35 ≤ π ≤ 0.65. Approximate 95% confidence interval

n

p ± 20% p ± 10% p ± 5% p ± 3.1%

25 100 400 1000

4. We reject the null hypothesis that the true mean difference is 0 (p = 0.014) – see Subsection 4.2.1 for definitions. [The two-sided p-value for t = 3.11 on 8 d.f. is 0.014.] Alternative 1 is straightforward. The t-statistic (alternative 2) expresses the change as a multiple of its standard error. The conventional wisdom is that the change is worthy of note if the p-value is less than 0.05 or, equivalently, if the 95% confidence interval does not contain 0. For this, the t-statistic must be somewhat greater than 1.96, i.e., for all practical purposes >2.0. For small degrees of freedom, the t-statistic must be substantially greater than 2.0. Readers who have difficulty with alternatives 3 and 4 may find it helpful to note that these restate and interpret the information in alternatives 1 and 2. If standard errors are not enough and formal inferential information is required, confidence intervals may be preferable to formal tests of hypotheses.

4.2.2 Confidence intervals and tests for proportions

We assume that individuals are drawn independently and at random from a binomial population where individuals are in one of two categories – male as opposed to female, a favorable treatment outcome as opposed to an unfavorable outcome, survival as opposed to non-survival, defective as opposed to non-defective, Democrat as opposed to Republican, etc. Let π be the population proportion. In a sample of size n, the proportion in the category of interest is denoted by p. Then, SE[p] =



π(1 − π )/n.

√ An upper bound for SE[p] is 1/(2 n). If π is between about 0.35 and 0.65, the inaccuracy √ in taking SE[p] as 1/(2 n) is small. This approximation leads to the confidence intervals shown in Table 4.3. Note again that the approximation is poor if π is outside the range 0.35 to 0.65. An alternative is to use the estimator  p(1 − p)
. SE[p] = n

4.2 Confidence intervals and tests of hypotheses

113

An approximate 95% confidence bound for the proportion π is then  p(1 − p) p ± 1.96 . n 4.2.3 Confidence intervals for the correlation

The correlation measure that we discuss here is the Pearson or product–moment correlation, which measures linear association. The standard error of the correlation coefficient is typically not a useful statistic. The distribution of the sample correlation, under the usual assumptions (e.g., bivariate normality), is too skew. The function cor.test() may be used to test the null hypothesis that the sample has been drawn from a population in which the correlation ρ is zero. For given x, the distribution of y is assumed normal, independently for different ys and with mean given by a linear function of x. Classical methods for comparing the magnitudes of correlations, or for calculation of a confidence interval for the correlation, rely on the assumption that the joint distribution of (x, y) is bivariate normal. In addition to the assumption for the test that ρ = 0, we need to know that x is normally distributed, independently between (x, y) pairs. This assumption is required for the default confidence interval that cor.test() outputs. In practice, it may be enough to check that both x and y have normal distributions. 4.2.4 Confidence intervals versus hypothesis tests

Those who have problems with confidence intervals and (especially) tests of hypotheses (often called significance tests) are in good company. There is increasing support for the view that they should play a relatively minor role in statistical analysis or be eliminated altogether. The methodology is too often abused. Papers that present a large number of significance tests are, typically, not making good use of the data. It becomes difficult to know what to make of the results. Among a large number of tests, some will be significant as a result of chance. Misunderstandings are common in the literature, even among mature researchers. A p-value does not allow the researcher to say anything about the probability that either hypothesis, the null or its alternative, is true. Then why use them? Perhaps the best that can be said is that hypothesis tests often provide a convenient and quick answer to questions about whether effects seem to stand out above background noise. However if all that emerges from an investigation are a few p-values, we have to wonder what has been achieved. Because of these problems, there are strong moves away from hypothesis testing and towards confidence intervals. Tukey (1991) argues strongly, and cogently, that confidence intervals are more informative and more honest than p-values. He argues Statisticians classically asked the wrong question – and were willing to answer with a lie, one that was often a downright lie. They asked “Are the effects of A and B different?” and they were willing to answer “no”.

114

A review of inference concepts

All we know about the world teaches us that the effects of A and B are always different – in some decimal place – for every A and B. Thus asking “Are the effects different?” is foolish. What we should be answering first is “Can we tell the direction in which the effects of A differ from the effects of B?” In other words, can we be confident about the direction from A to B? Is it “up”, “down”, or “uncertain”? [Tukey, 1991]

Tukey argues that we should never conclude that we “accept the null hypothesis”. Rather, our uncertainty is about the direction in which A may differ from B. Confidence intervals do much better at capturing the nature of this uncertainty. Guidelines for significance testing

Few scientific papers make more than half-a-dozen points that are of consequence. Any significance tests should be closely tied to these main points, preferably with just one or two tests for each point that is made. Keep any significance tests and p-values in the background. Once it is apparent that an effect is statistically significant, the focus of interest should shift to its pattern and magnitude, and to its scientific significance. It is poor practice to perform t-tests for each comparison between treatments when the real interest is (or should be) in the overall pattern of response. Where the response depends on a continuous variable, it is often pertinent to ask whether, e.g., the response keeps on rising (falling), or whether it rises (falls) to a maximum (minimum) and then falls (rises). Significance tests should give the researcher, and the reader of the research paper, confidence that the effects that are discussed are real! The focus should then move to the substantive scientific issues. Statistical modeling can be highly helpful for this. The interest is often, finally, in eliciting the patterns of response that the data present. 4.3 Contingency tables Table 4.4 is from US data that were used in the evaluation of labor training programs, aimed at individuals who had experienced economic and social difficulties. The table shows numbers of high school graduates and dropouts who had participated in a labour training program (NSW group) and those who had not participated (PSID3 group).8 These data will be discussed further in Section 13.2. A glance at the table suggests that the proportion of high school dropouts in the NSW group is much higher than in the PSID3 group. The chi-squared test for no association is described in the next subsection; it can be used to check this formally: > # To agree with hand calculation below, specify correct=FALSE > chisq.test(with(nswpsid3, table(trt, nodeg)), correct=FALSE) . . . . X-squared = 19.9, df = 1, p-value = 8.189e-06

8

## Compare number with a high school qualification, between ‘untreated’ rows ## from data frame psid3 and ‘treated’ rows from nswdemo library(DAAG) # Data are from DAAG nswpsid3 <- rbind(psid3, subset(nswdemo, trt==1)) table(nswpsid3$trt, nswpsid3$nodeg) # PSID3 males are coded 0; NSW male trainees are coded 1.

4.3 Contingency tables

115

Table 4.4 Contingency table derived from data that relates to the Lalonde (1986) paper. High school graduate certificate

PSID3 males NSW male trainees

Yes

No

63 80

65 217

Table 4.5 The calculated expected values for the contingency table in Table 4.4. High school graduate

PSID3 NSW74 trainees Total Column proportion

Yes

No

Total

Row proportion

63 (115) 80 (267.0) 143 143/425 = 0.336

65 (12.95) 217 (30.05) 282 282/425 = 0.664

128 297 425

128/425 = 0.301 217/425 = 0.699

The small p-value confirms that high school dropouts are more strongly represented in the NSW data.

The mechanics of the chi-squared test

The null hypothesis is that the proportion of the total in each cell is, to within random error, the result of multiplying a row proportion by a column proportion. The independence assumption, i.e., the assumption of independent allocation to the cells of the table, is crucial. Assume there are I rows and J columns. The expected value in cell (i, j ) is calculated as Eij = (proportion for row i) × (proportion for column j ) × total. We can then obtain a score for each cell of the table by computing the absolute value of the difference between the expected value and the observed value (with the continuity correction that is the default, 0.5 would be subtracted at this point), squaring, dividing the result by the expected value, and replacing any negative scores by zero. Summing over all scores gives the chi-squared statistic. Under the null hypothesis the chi-squared statistic has an approximate chi-squared distribution with (I − 1)(J − 1) degrees of freedom. In Table 4.5, the values in parentheses are the expected values Eij .

116

A review of inference concepts

Table 4.6 Contingency table compiled from Hobson (1988, Table 12.1, p. 248). Object moves Dreamer moves Yes No

Yes

No

5 3

17 85

The expected values are found by multiplying the column totals by the row proportions. (Alternatively, the row totals can be multiplied by the column proportions.) Thus 117 × 0.591 = 69.15, 196 × 0.591 = 115.85, etc. An example where a chi-squared test may not be valid

In Table 4.6 we summarize information that Hobson (1988) derived from drawings of dreams, made by an unknown person that he calls “The Engine Man”. Among other information Hobson notes, for each of 110 drawings of dreams made, whether the dreamer moves, and whether an object moves. Dreamer movement may occur if an object moves, but is relatively rare if there is no object movement. (Note that Hobson does not give the form of summary that we present in Table 4.6.) It may also seem natural to do a chi-squared test for no association.9 This gives χ 2 = 7.1 (1 d.f.), p = 0.008. A reasonable rule, for the use of the chi-squared approximation, may be that all expected values should be at least 2 (Miller, 1986), a requirement that is satisfied for this application of the test. A check is to do a Fisher exact test. In this instance the Fisher exact test10 gives, surprisingly, exactly the same result as the chi-squared test, i.e., p = 0.008. A more serious concern is that there is a time sequence to the dreams. Thus, there could well be runs of dreams of the same type. Hobson gives the numbers of the dreams in sequence. Assuming these represent the sequence in time, this would allow a check of the strength of any evidence for runs. Hobson’s table has information that our tabular summary (Table 4.6) has not captured. 4.3.1 Rare and endangered plant species

The calculations for a test for no association in a two-way table can sometimes give useful insight, even where a formal test of statistical significance would be invalid. The example that now follows (Table 4.7) illustrates this point. Data are from species lists for various regions of Australia. Species were classified CC, CR, RC and RR, with C denoting common and R denoting rare. The first code letter relates to South Australia and Victoria, and the 9

10

## Engine man data engineman <- matrix(c(5,3,17,85), 2,2) chisq.test(engineman) fisher.test(engineman)

4.3 Contingency tables

117

Table 4.7 Cross-classification of species occurring in South Australia/ Victoria and in Tasmania. Habitat type

Common/rare classification

D

W

WD

CC CR RC RR

37 23 10 15

190 59 141 58

94 23 28 16

second to Tasmania. They were further classified by habitat according to the Victorian register, where D = dry only, W = wet only, and WD = wet or dry.11 We use a chi-squared calculation to check whether the classification into the different habitats is similar for the different rows. Details of the calculations are: > (x2 <- chisq.test(rareplants)) Pearson’s Chi-squared test data: rareplants X-squared = 35, df = 6, p-value = 4.336e-06

This low p-value should attract a level of skepticism. We do not have a random sample from some meaningful larger population. Suppose that there is clustering, so that species come in closely related pairs, with both members of the pair always falling into the same cell of the table. This will inflate the chi-squared statistic by a factor of 2 (the net effect of inflating the numerator by 22 , and the denominator by 2). There probably is some such clustering, though different from that of this simplistic example. Such clustering will inflate the chi-squared statistic by an amount that the available information does not allow us to estimate. The standard Pearson chi-squared tests rely on multinomial sampling assumptions, with counts entering independently into the cells. Where it is possible to form replicate tables, the assumption should be tested. Figure 4.4 shows expected number of species, by habitat.12

11

12

## Enter the data thus: rareplants <- matrix(c(37,190,94,23,59,23,10,141,28,15,58,16), ncol=3, byrow=TRUE, dimnames=list(c("CC","CR","RC","RR"), c("D","W","WD"))) ## Expected number of species, by habitat (calculate x2 as above) x2E <- stack(data.frame(t(x2$expected))) habitat <- rep(c(1,2,3), 4) plot(x2E$values ˜ habitat, axes=FALSE, xlim=c(0.5, 3.5), pch=16, xlab="habitat", ylab="Expected Number of Species") text(x2E$values ˜ habitat, labels=x2E$ind, pos=rep(c(4,4,2,2),3)) axis(1, at=seq(1,3), labels=c("D", "W", "WD")) axis(2); box()

A review of inference concepts

100

●

RC CR

● ● ●

RC

●

RR

● ●

CR

CC

●

●

CC RR

RR

D

Figure 4.4

CC

150

200

●

50

Expected number of species

118

W Habitat

● ●

RC CR

WD

Expected number of species, by habitat, for the rareplants data.

Examination of departures from a consistent overall row pattern

The investigator then needs to examine the nature of variation with the row classification. For this, it is helpful to look at the residuals; these are calculated as (observed − expected)/expected0.5 : > x2 <- chisq.test(rareplants) > ## Standardized residuals > residuals(x2) D W WD CC -0.369 -1.1960 2.263 CR 2.828 -1.0666 -0.275 RC -2.547 2.3675 -2.099 RR 1.242 0.0722 -1.023

The null hypothesis implies that the expected relative numbers in different columns are the same in every row. The chi-squared residuals show where there may be departures from this pattern. In large tables these will, under the null hypothesis, behave like random normal deviates with mean zero and variance one. The values that should be noted, if the assumptions required for a chi-squared test are satisfied, are those whose absolute value is somewhat greater than 2.0. For the present table, there are five standardized residuals whose value is substantially greater than 2.0. It is these, and especially the two that are largest, that should perhaps attract attention. Notice that the CC species are, relative to the overall average, over-represented in the WD classification, the CR species are over-represented in the D classification, while the RC species are under-represented in D and WD and over-represented in W. For reference, here is the table of expected values: > x2$expected D W CC 39.3 207.2 CR 12.9 67.8 RC 21.9 115.6 RR 10.9 57.5

WD 74.5 24.4 41.5 20.6

4.4 One-way unstructured comparisons

119

4.3.2 Additional notes Interpretation issues

Having found an association in a contingency table, what does it mean? The interpretation will differ depending on the context. The incidence of gastric cancer is relatively high in Japan and China. Do screening programs help? Here are two ways in which the problem has been studied: r In a long-term follow-up study, patients who have undergone surgery for gastric cancer may be classified into two groups – a “screened” group whose cancer was detected by mass screening, and an “unscreened” group who presented at a clinic or hospital with gastric cancer. The death rates over the subsequent five- or ten-year period are then compared. For example, the five-year mortality may be around 58% in the unscreened group, compared with 72% in the screened group, out of approximately 300 patients in each group. r In a prospective cohort study, two populations – a screened population and an unscreened population – may be compared. The death rates in the two populations over a ten-year period may then be compared. For example, the annual death rate may be of the order of 60 per 100 000 for the unscreened group, compared with 30 per 100 000 for the screened group, in populations of several thousand individuals. In the long-term follow-up study, the process that led to the detection of cancer was different between the screened and unscreened groups. The screening may lead to surgery for some cancers that would otherwise lie dormant long enough that they would never attract clinical attention. The method of detection is a confounding factor. It is necessary, as in the prospective cohort study, to compare all patients in a screened group with all patients in an unscreened group. Even so, it is necessary, in a study where assignment of participants is not random, to be sure that the two populations are comparable. Modeling approaches

Modeling approaches typically work with data that record information on each case separately. Data where there is a binary (yes/no) outcome, and where logistic regression may be appropriate, are an important special case. Chapter 8 gives further details. 4.4 One-way unstructured comparisons Figure 4.5 displays data from a one-way unstructured comparison between three treatments. The weights of the plants were measured after two months on respective treatments: water concentrated nutrient, and concentrated nutrient plus the selective herbicide 2,4-D. Data are: tomato
120

A review of inference concepts

Nutrient+24D Nutrient Water 1.0

1.5

2.0

2.5

Weight

Figure 4.5 Weights of tomato plants, after two months of the three treatments.

## Make water the first level of trt. It will then appear as ## the initial level in the graphs. In aov or lm calculations, ## it will be the baseline or reference level. tomato$trt <- relevel(tomato$trt, ref="water")

Figure 4.5 can be obtained with: stripplot(trt˜weight, aspect=0.6, data=tomato)

The strip plots display “within-group” variability, as well as giving an indication of differences among the group means. Variances seem similar for the three treatments. There is a single explanatory factor (trt), with one level for each of the different treatments that were applied. A simple-minded approach is to calculate the means for each of the three treatments, and then examine all three pairwise comparisons. With three comparisons this is, arguably, a reasonable strategy. If there were four treatments, there would be six comparisons, and it really would become desirable to do one analysis rather than six. We would certainly not want to draw six graphs, one for each pair of treatments that are compared. Hence the use of an analysis of variance (really, as noted above, the fitting of a linear model) to do an overall analysis. We will first examine an overall visual summary of the analysis results, then examine the analysis.

The tomato data – a visual summary

The function onewayPlot(), from the DAAG package, provides a convenient visual summary of results, shown in Figure 4.6. The code is: ## Do analysis of variance calculations tomato.aov <- aov(weight ˜ trt, data=tomato) ## Summarize results graphically oneway.plot(obj=tomato.aov)

Notice that the graph gives two different assessments of the least difference that should be treated as “significant”. These use different criteria: r The 5% least significant difference (LSD) is designed so that, under the null model (no differences), significant differences will be found in 5% of comparisons. r The 5% honest significant difference (HSD) is designed so that, under the null model, the maximum difference will be significant in 5% of experiments.

1.4

1.6

121

Nutrien t

Water

Nutrien t+24D

4.4 One-way unstructured comparisons

1.8

2.0

2.2

LSD Tukey HSD

Figure 4.6 Graphical presentation of results from the analysis of the tomato weight data. Means that differ by more than the LSD (least significant difference) are different, at the 5% level, in a t-test that compares the two means. Tukey’s honest significant difference (HSD) takes into account the number of means that are compared. See the text for details.

The LSD is commonly regarded as overly lax, while the HSD may be overly conservative. There are a variety of alternatives to the HSD that are less conservative; see the more detailed discussion of multiple comparisons in the next subsection. The simplicity of Figure 4.6 is appealing, but it is important to note the assumption that the standard error of difference is the same for all treatment comparisons. As all three treatments have the same number of observations, it is enough for the variance to be the same for all treatments.

The analysis of variance table

The analysis of variance table is given by the anova() function, thus: > anova(tomato.aov) Analysis of Variance Table Response: weight Df Sum Sq Mean Sq F value Pr(>F) trt 2 0.63 0.31 1.2 0.33 Residuals 15 3.91 0.26

Observe that the residual mean squared error is 3.21. Note that two degrees of freedom are associated with estimating the variance of the three group means. Each treatment contributes 6 − 1 = 5 d.f. to the pooled or residual sum of squares, giving 3 × 5 = 15 d.f. in all. Note that 2 (for trt) plus 15 (for Residuals) equals 17, which is one less than the number of observations. Estimation of the overall mean accounts for the remaining degree of freedom. The Mean Sq (“mean square”) column has estimates of between-sample (trt) and within-sample variability (Residuals). The between-sample variance can be calculated by applying the function var() to the vector of treatment means, then multiplying by the common sample size, in this case 6. The within-sample variability estimate is, effectively, a pooled variance estimate for the three treatments. Each mean square is the result from dividing the Sum Sq (“sum of squares”) column by the appropriate degrees of freedom.

122

A review of inference concepts

In the absence of systematic differences between the sample means, the two mean squares will have the same expected value, and their ratio (the F-statistic) will be near 1. Systematic differences between the sample means will add extra variation into the treatment mean square, with no effect on the residual mean square, resulting in an expected F -statistic that is larger than 1. In the output above, the F -statistic is 0.33, on 3 and 8 degrees of freedom, with p = 0.33. There is no convincing indication that there are indeed differences among the treatment means. Interest then turns to teasing out the nature of those differences. The one-way analysis of variance formally tests whether the variation among the means is greater than what might occur simply because of the natural variation within each group. This comparison is based on the F-statistic, which is given in the output column headed F value. An F-statistic that is much larger than 1 points to the conclusion that the means are different. The p-value is designed to assist this judgment. Figure 4.6 is one of a number of graphical presentation possibilities for a one-way layout. Others are (1) a side-by-side comparison of the histograms – but there are too few values for that; (2) density plots – again there are too few values; and (3) a comparison of the boxplots – this works quite well with 12 values for each treatment. 4.4.1 Multiple comparisons

In Figure 4.6 we gave two “yardsticks” – the LSD and the HSD – against which to compare differences between means. Because neither of these suggested a difference between treatments, the choice between them was not of great consequence. Also, there were just three treatment levels, so that the difference between the LSD and the HSD is not large. The 5% HSD is designed so that, under the null model (no difference between treatments), the maximum difference will be greater than the HSD in 5% of experiments. In other words, the 5% relates to an experiment-wise error rate, defined as just described. The HSD is an appropriate yardstick against which to compare treatment differences if the demand is for a 5% or other specified probability of finding a difference between the largest and smallest means when there was no difference in the populations from which they were drawn. Contrast this with the 5% least significant difference (LSD). This is designed, if used without a preliminary F -test, to give a 5% comparison-wise error rate. A reasonable practical strategy is to do a preliminary analysis of variance F -test. If that suggests differences between the means, then it is of interest to use both yardsticks in comparing them. The LSD gives an anti-conservative yardstick, i.e., one that, in the absence of the preliminary F -test, would be somewhat biased towards finding differences. Tukey’s HSD gives a stricter conservative yardstick, i.e., one that is somewhat biased against finding differences. Ignoring changes in degrees of freedom and possible associated changes in the standard error, the HSD will increase as the number of treatment groups that are to be compared increases. ∗

Microarray data – severe multiplicity

Multiple tests are a serious issue in the analysis of microarray data, where an individual slide (or sometimes, as for Plate 2, half-slide) may yield information on some thousands

4.4 One-way unstructured comparisons

123

of genes. Each slide (or, here, half-slide) is commonly used to compare, for each of a large number of genes, the gene expression in two samples of genetic material. The experiment that led to Plate 2 was designed to investigate changes in gene expression between the pre-settlement free-swimming stage of coral, and the post-settlement stage. For 3042 genes (one for each of 3042 spots), which showed an increase in gene expression and which a decrease? Note that each panel in Plate 2 has 3072 spots; this includes 30 blanks. Where there was an increase, the spot should be fairly consistently blue, or bluish, over all six panels. Where there was a decrease, the spot should be fairly consistently yellow, or yellowish. Here, all that will be attempted is to give broad indications of the experimental procedure, and subsequent processing, that led to the plots shown in Plate 2. The slides are first printed with probes, with one probe per spot. Each probe is designed to check for evidence of the expression of one gene. The two samples are separately labeled so that when later a spot “lights up” under a scanner, it will be possible to check for differences in the signal intensity. After labeling the separate samples, mixing them, and wiping the mixture over the slide or half-slide, and various laboratory processing steps, a scanner was used to determine, for each spot, the intensities generated from the two samples. Various corrections are then necessary, leading finally to the calculation of logarithms of intensity ratios. Essentially, it is logarithms of intensity ratios that are shown in Plate 2. For these data there are, potentially, 3042 t-statistics. This is small, by the standards of microarray experiments. There are severe problems of multiplicity to address. Details of a defensible approach to analyzing the data shown in Plate 2 will be posted on the web site for the book. For further information on the analysis of microarray data, see Smyth (2004). For background on the coral data, see Grasso et al. (2008).

4.4.2 Data with a two-way structure, i.e., two factors

Consider now data from an experiment that compared wild type (wt) and genetically modified rice plants (ANU843), each with three different chemical treatments. A first factor relates to whether F10 or NH4Cl or NH4NO3 is applied. A second factor relates to whether the plant is wild type (wt) or ANU843. There are 72 sets of results, i.e., two types (variety) × three chemical treatments (fert) × 6 replicates, with the experimental setup repeated across each of two blocks (Block). Figures 4.7A and B show alternative perspectives on these data.13

13

## Simplified version of code library(lattice) ## Panel A dotplot(trt ˜ ShootDryMass, data=rice, aspect=1, panel=function(x,y,...){panel.dotplot(x, y, pch=1, col="gray40") panel.average(x, y, type="p", col="black", pch=3, cex=1.25)}, xlab="Shoot dry mass (g)") ## Panel B with(rice, interaction.plot(fert, variety, ShootDryMass, xlab="Level of first factor"))

124

A review of inference concepts

● ●●● ● ●●

NH4NO3 +ANU843

F10 +ANU843

●●

● ●● ● ●● ● ●● ●●

NH4Cl +ANU843

●● ● ● ●● ● ●● ● ● ●● ●● ●●● ● ●●

NH4NO3

●● ●●● ● ● ●●

NH4Cl

●

F10 0

●

50

●

●● ● ● ●●●● ● ● ● ●

20 40 60 80

B Mean of shoot dry mass

A

100

Shoot dry mass (g)

variety wt ANU843

F10

NH4Cl

NH4NO3

Level of first factor

Figure 4.7 Both panels are for rice shoot dry mass data. Panel A shows a one-way strip plot, with different strips for different treatment regimes. Treatment means are shown with a large +. The interaction plot in panel B shows how the effect of fertilizer (the first factor) changes with variety (the second factor). Data relate to Perrine et al. (2001).

Figure 4.7B shows a large difference between ANU843 and wild type (wt) for the F10 treatment. For the other treatments, there is no detectable difference. A two-way analysis will show a large interaction. Note, finally, that the treatments were arranged in two blocks. In general, this has implications for the analysis. This example will be discussed again in Chapter 7, where block effects will be taken into account.

4.4.3 Presentation issues

The discussion so far has treated all comparisons as of equal interest. Often they are not. There are several possibilities: r Interest may be in comparing treatments with a control, with comparisons between treatments of lesser interest. r Interest may be in comparing treatments with one another, with any controls used as a check that the order of magnitude of the treatment effect is pretty much what was expected. r There may be several groups of treatments, with the chief interest in comparisons between the different groups. Any of these situations should lead to specifying in advance the specific treatment comparisons that are of interest. Often, however, scientists prefer to regard all treatments as of equal interest. Results may be presented in a graph that displays, for each factor level, the mean and its associated standard error. Alternatives to displaying bars that show the standard error may be to show a 95% confidence interval for the mean, or to show the standard deviation. Displaying or quoting the standard deviation may be appropriate when the interest is not in comparing level means, but in obtaining an idea of the extent to which the different levels are clearly separated.

4.5 Response curves

125

30

● ●

25

● ● ●

20

● ● ● ●

0

2

4

6

Starting point 3 6 9 12

● ●

15

Distance traveled (cm)

●

8

Distance traveled 31.38 26.63 18.75 13.88

30.38 25.75 22.50 11.75

33.63 27.13 21.63 14.88

10

Distance up ramp (cm)

Figure 4.8 Distance traveled (distance.traveled) by model car, as a function of starting point (starting.point), up a 20◦ ramp.

In any case: r For graphical presentation, use a layout that reflects the data structure, i.e., a one-way layout for a one-way data structure, and a two-way layout for a two-way data structure. r Explain clearly how error bars should be interpreted – ± SE limits, ± 95% confidence interval, ± SED limits, or whatever. Or if the intention is to indicate the variation in observed values, the SD (standard deviation) may be more appropriate. r Where there is more than one source of variation, explain what source(s) of “error” is/are represented. It is pointless and potentially misleading to present information on a source of error that is of little or no interest, e.g., on analytical error when the relevant “error” for the treatment comparisons that are of interest arises from fruit-to-fruit variation.

4.5 Response curves The table shown to the right of Figure 4.8 exhibits data that are strongly structured. A model car was released three times at each of four different distances (starting.point) up a 20◦ ramp. The experimenter recorded distances traveled from the bottom of the ramp across a concrete floor. Figure 4.8 shows a plot of these data.14 What is the pattern of the response? This should be handled as a regression problem rather than as an analysis of variance problem. It would be particularly nonsensical to examine all pairwise comparison, thus doing violence to the treatment structure, and confusing interpretation. Response curve analyses should be used whenever appropriate in preference to comparison of individual pairs of means. For these data, the physics can be used to suggest the likely form of response. Where no such help is available, careful examination of the graph, followed by systematic examination of plausible forms of response, may suggest a suitable form of response curve. 14

## Data frame modelcars (DAAG) plot(distance.traveled ˜ starting.point, data=modelcars, xlim=c(0,12.5), xaxs="i", xlab = "Distance up ramp (cm)", ylab="Distance traveled (cm)")

126

A review of inference concepts

Table 4.8 Each tester made two firmness tests on each of five fruit. Fruit 1 2 3 4 5 6 7 8 9 10

Tester

Firmness

Mean

1 1 1 1 1 2 2 2 2 2

6.8, 7.3 7.2, 7.3 7.4, 7.3 6.8, 7.6 7.2, 6.5 7.7, 7.7 7.4, 7.0 7.2, 7.6 6.7, 6.7 7.2, 6.8

7.05 7.25 7.35 7.2 6.85 7.7 7.2 7.4 6.7 7.0

Steps that are suitable for use with data that appear to follow a relatively simple form of response are: 1. Does a straight line explain the data better than assuming a random scatter about a horizontal line? 2. Does a quadratic response curve offer any improvement? 3. Would a cubic curve do better still? Notice that at this stage we are not concerned to say that a quadratic or cubic curve is a good description of the data. All we are examining is whether such a curve captures an important part of the pattern of change. If it does, but the curve is still not quite right, it may be worthwhile to look for a different form of curve that does fit the data adequately. A representation of the response curve in terms of coefficients of orthogonal polynomials provides information that makes it relatively easy to address questions 1–3. Consider, for example, a model that has terms in x and x 2 . Orthogonal polynomials re-express this combination of terms in such a way that the coefficient of the “linear” term is independent of the coefficient of the “quadratic” term. Higher-order (cubic, . . . ) orthogonal polynomial terms can of course be fitted, and it remains the case that the coefficients are mutually independent. There is some further limited discussion of orthogonal polynomials in Section 7.4. Steel et al. (1993) discuss the practical use of orthogonal polynomials in some detail.

4.6 Data with a nested variation structure Ten apples are taken from a box. A randomization procedure assigns five to one tester, and the other five to another tester. Each tester makes two firmness tests on each of their five fruit. Firmness is measured by the pressure needed to push the flat end of a piece of rod through the surface of the fruit. Table 4.8 gives the results, in N/m2 . For comparing the testers, we have five experimental units for each tester, not ten. One way to do a t-test is to take means for each fruit. We then have five values (means, italicized) for one treatment, that we can compare with the five values for the other treatment.

4.6 Data with a nested variation structure

127

What happens if we ignore the data structure, and compare ten values for one tester with ten values for the other tester? This pretends that we have ten experimental units for each tester. The analysis will suggest that the treatment means are more accurate than is really the case. We obtain a pretend standard error that is not the correct standard error of the mean. We are likely to under-estimate the standard error of the treatment difference. 4.6.1 Degrees of freedom considerations

For comparison of two means when the sample sizes n1 and n2 are small, it is important to have as many degrees of freedom as possible for the denominator of the t-test. It is worth tolerating possible bias in some of the calculated SEDs in order to gain extra degrees of freedom. The same considerations arise in the one-way analysis of variance, and we pursue the issue in that context. It is illuminating to plot out, side by side, say 10 SEDs based on randomly generated normal variates, first for a comparison based on 2 d.f., then 10 SEDs for a comparison based on 4 d.f., etc. A formal statistical test is thus unlikely, unless the sample is large, to detect differences in variance that may have a large effect on the result of the test. It is therefore necessary to rely on judgment. Both past experience with similar data and subject area knowledge may be important. In comparing two treatments that are qualitatively similar, differences in the population variance may be unlikely, unless the difference in means is at least of the same order of magnitude as the individual means. If the means are not much different then it is reasonable, though this is by no means inevitable, to expect that the variances will not be much different. If the treatments are qualitatively different, then differences in variance may be expected. Experiments in weed control provide an example where it would be surprising to find a common variance. There will be few weeds in all plots where there is effective weed control, and thus little variation. In control plots, or for plots given ineffective treatments, there may be huge variation. If there do seem to be differences in variance, it may be possible to model the variance as a function of the mean. It may be possible to apply a variance-stabilizing transformation. Or the variance may be a smooth function of the mean. Otherwise, if there are just one or two degrees of freedom per mean, use a pooled estimate of variance unless the assumption of equal variance seems clearly unacceptable. 4.6.2 General multi-way analysis of variance designs

Generalization to multi-way analysis of variance raises a variety of new issues. If each combination of factor levels has the same number of observations, and if there is no structure in the error (or noise), the extension is straightforward. The extension is less straightforward when one or both of these conditions are not met. For unbalanced data from designs with a simple error structure, it is necessary to use the lm() (linear model) function. The lme() function in the nlme package, or alternatively lmer() in the lme4 package, is able to handle problems where there is structure in the error term, including data from unbalanced designs. See Chapter 9 for further details.

128

A review of inference concepts

Table 4.9 These are the same data as in Table 3.1. Pair #

Heated (mm) Ambient Difference

1

2

3

4

5

6

7

8

9

244 225 19

255 247 8

253 249 4

254 253 1

251 245 6

269 259 10

248 242 6

252 255 −3

292 286 6

4.7 Resampling methods for standard errors, tests, and confidence intervals There are many different resampling methods. All rely on the selection of repeated samples from a “population” that is constructed using the sample data. In general, there are too many possible samples to take them all, and we therefore rely on repeated random samples. In this section, we demonstrate permutation and bootstrap methods. We start with permutation tests, illustrating their use for the equivalent of one-sample and two-sample t-tests. 4.7.1 The one-sample permutation test

Consider the paired elastic band data of Table 3.1 again, reproduced here as Table 4.9. If the treatment has made no difference, then an outcome of 244 for the heated band and 225 for the ambient band might equally well have been 225 for the heated band and 244 for the ambient band. A difference of 19 becomes a difference of −19. There are 29 = 512 permutations, and a mean difference associated with each permutation. We then locate the mean difference for the data that we observed within this permutation distribution. The p-value is the proportion of values that are as large in absolute value as, or larger than, the mean for the data. The absolute values of the nine differences are Difference

19

8

4

1

6

10

6

3

6

In the permutation distribution, these each have an equal probability of taking a positive ¯ or a negative sign. There are 29 possibilities, and hence 29 = 512 different values for d. The possibilities that give a mean difference that is as large as or larger than in the actual sample, where the value for pair 8 has a negative sign, are Difference

19 19 19

8 8 8

4 4 4

1 −1 1

6 6 6

10 10 10

6 6 6

3 3 −3

6 6 6

There are another three possibilities that give a mean difference that is of the same absolute value, but negative. Hence p = 6/512 = 0.0117. In general, when the number of pairs is large, it will not be feasible to use such an enumeration approach to get information on relevant parts of the upper and lower tails of the distribution. We therefore take repeated random samples from the permutation

4.7 Resampling methods for standard errors, tests, and confidence intervals x2 − x1

0.04 0.00

0.02

Density

0.06

0.08

− (x 2 − x 1)

129

−20

−10

0

10

Figure 4.9 Density curves for two samples of 2000 each from the permutation distribution of the difference in means, for the two-sample elastic band data.

distribution. The function onetPermutation() in our DAAG package may be used for this. 4.7.2 The two-sample permutation test

Suppose we have n1 values in one group and n2 in a second, with n = n1 + n2 . The permutation distribution results from taking all possible samples of n2 values from the total of n values. For each such sample, we calculate mean of the n2 values that are selected – mean of remaining n1 values. The permutation distribution is the distribution of all such differences of means. We locate the differences of means for the actual samples within this permutation distribution. The calculation of the full permutation distribution is not usually feasible. We therefore take perhaps 1000 samples from this distribution. The function twot.permutation() that is in our DAAG package may be used for this repeated sampling. Thus consider the data from Subsection 4.2.2: Ambient: 254 252 239 240 250 256 267 249 259 269 (mean = 253.5) Heated: 233 252 237 246 255 244 248 242 217 257 254 (mean = 244.1) Figure 4.9 shows two estimates of the permutation distribution that were obtained by taking, in each instance, 2000 random samples from this distribution. The point where the difference in means falls with respect to this distribution (253.5 − 244.1 = 9.4) has been marked, as has minus this difference.15 15

## Draw one curve only; permutation distribution of difference in means x1 <- two65$ambient; x2 <- two65$heated; x <- c(x1, x2) n1 <- length(x1); n2 <- length(x2); n <- n1+n2 dbar <- mean(x2) - mean(x1) z <- numeric(2000) for(i in 1:2000){ mn <- sample(n, n2, replace=FALSE) dbardash <- mean(x[mn]) - mean(x[-mn]) z[i]<- dbardash } plot(density(z), yaxs="i") abline(v = dbar) abline(v = -dbar, lty=2) signif((sum(z > abs(dbar)) + sum (z< -abs(dbar)))/length(z), 3)

130

A review of inference concepts

The density estimate corresponding to the solid line gave a p-value of 0.051. The density estimate corresponding to the dashed line gave a p-value of 0.060. Use of a larger sample size will of course lead to more accurate p-values. 4.7.3 ∗ Estimating the standard error of the median: bootstrapping

The formula given in Subsection 4.1.3 for the SEM has the same form, irrespective of the distribution, providing that the sample is chosen randomly. By contrast, the formula for the standard error of the median (Subsection 4.1.5) applies only when data are normally distributed. The bootstrap estimate of the standard error of the median avoids this requirement, and avoids also the need to look for some alternative distribution that may be a better fit to the data. A comparison between the bootstrap estimate and the normal theory estimate allows an assessment of the seriousness of any bias that may result from non-normality. We proceed to calculate the bootstrap estimate of the standard error for the median length for the eggs that were in wrens’ nests. (The boot package (Canty, 2002) is needed for all bootstrap examples.) We will use the result as a check on our earlier computation. The idea is as follows. In estimating the standard error of the median, we are seeking the standard deviation of medians that could be obtained for all possible samples of egg lengths in wrens’ nests. Of course, we have access to one sample only, but if our sample is of reasonable size and has been collected properly it should give us a good approximation to the entire population. We estimate the standard deviation of the median by computing sample medians for each of the resamples and taking the standard deviation of all of these medians. Even though the resamples are not genuine new samples, this estimate for the standard error of the median has good statistical properties, for purposes of estimating the standard error of the median. Here is the output from R for the egg lengths from the wrens’ nests: > ## bootstrap estimate of median of wren length: data frame cuckoos (DAAG) > wren <- split(cuckoos$length, cuckoos$species)$wren > library(boot) > ## First define median.fun(), with two required arguments: > ## data specifies the data vector, > ## indices selects vector elements for a each resample > median.fun <- function(data, indices){median(data[indices])} > ## Call boot(), with statistic=median.fun, R = # of resamples > (wren.boot <- boot(data = wren, statistic = median.fun, R = 999)) ORDINARY NONPARAMETRIC BOOTSTRAP Call: boot(data = wren, statistic = median.fun, R = 999) Bootstrap Statistics : original bias std. error t1* 21 0.061 0.225

4.7 Resampling methods for standard errors, tests, and confidence intervals

131

The original estimate of the median was 21. The bootstap estimate of the standard error is 0.225, based on 999 resamples. Compare this with the slightly larger standard error estimate of 0.244 given by the normal theory formula in Subsection 4.1.5. The bootstrap estimate of the standard error will of course differ somewhat between different runs of the calculation. Also given is an estimate of the bias, i.e., of the tendency to under- or over-estimate the median. 4.7.4 Bootstrap estimates of confidence intervals

The usual approach to constructing confidence intervals is based on a statistical theory that relies, in part, on the assumption of normally distributed observations. Sometimes this theory is too complicated to work out, and/or the normal assumption is not applicable. In such cases, the bootstrap may be helpful. We demonstrate the use of the methodology to calculate confidence intervals for the median and for the correlation. Several different confidence intervals can be calculated using bootstrap replicates of the data. The function boot.ci() handles five of these. The perc (percentile) type is probably the most commonly used method; it is not the most accurate. The bca type (bias corrected accelerated or BCa ) will often give a substantial improvement. Efron and Tibshirani (1993) give a clear description of these methods, together with theoretical justification for the use of the BCa method. Bootstrap 95% confidence intervals for the median

As when computing bootstrap standard errors, we calculate sample medians for a large number of resamples. The endpoints for the 95% percentile confidence interval are calculated as the 2.5 and 97.5 percentiles of the resulting distribution of medians. The endpoints for the BCa confidence interval are calculated in a more complicated way; Efron and Tibshirani (1993) can be consulted for the details. > median.fun <- function(data, indices){median(data[indices])} > ## Call the boot() function, with statistic=median.fun > wren <- cuckoos[cuckoos$species=="wren", "length"] > (wren.boot <- boot(data=wren, statistic=median.fun, R=9999)) BOOTSTRAP CONFIDENCE INTERVAL CALCULATIONS Based on 9999 bootstrap replicates CALL : boot.ci(boot.out = wren.boot, type = c("perc", "bca")) Intervals : Level Percentile BCa 95% (20.9, 22.0 ) (20.0, 21.0 ) Calculations and Intervals on Original Scale Some BCa intervals may be unstable

Interestingly, the BCa interval is slightly narrower than the cruder percentile interval in this example. The warning may be taken as an indication that the calculation should be

132

A review of inference concepts

run again, with a larger number (perhaps 99 999) of resamples. Such warnings may arise because of outliers in the data. Use qqnorm(wren) to check that this is not an issue for these data. The correlation coefficient

Bootstrap methods do not require bivariate normality. Independence between observations, i.e., between (x, y) pairs, is as important as ever. Note however that a correlation of, e.g., 0.75 for a non-normal distribution may have quite different implications from a correlation of 0.75 when normality assumptions apply. We will compute 95% confidence intervals for the correlation between chest and belly for the possum data frame:16 BOOTSTRAP CONFIDENCE INTERVAL CALCULATIONS Based on 9999 bootstrap replicates CALL : boot.ci(boot.out = possum.boot, type = c("perc", "bca")) Intervals : Level Percentile BCa 95% ( 0.476, 0.709 ) ( 0.468, 0.704 ) Calculations and Intervals on Original Scale

The bootstrap – parting comments

Bootstrap methods are not a panacea. We must respect the structure of the data; any form of dependence in the data must be taken into account. There are contexts where the bootstrap is invalid and will mislead. As a rough guideline, the bootstrap is unlikely to be satisfactory for statistics, including maximum, minimum and range, that are functions of sample extremes. The bootstrap is usually appropriate for statistics from regression analysis – means, variances, coefficient estimates, and correlation coefficients. It also works reasonably well for medians and quartiles, and other such statistics. See Davison and Hinkley (1997), Efron and Tibshirani (1993). See also the references in the help page for the boot() function in the boot package. 4.8 ∗ Theories of inference Formal statistical methodologies are of two broad types: frequentist and Bayesian. The frequentist approach is usually based on the concept of likelihood; given the model, what is the probability of obtaining a sample similar to that observed? Parameter values are 16

## Bootstrap estimate of 95% CI of cor(chest, belly): data frame possum (DAAG) possum.fun <- function(data, indices) { chest <- data$chest[indices] belly <- data$belly[indices] cor(belly, chest) } possum.boot <- boot(possum, possum.fun, R=9999) boot.ci(possum.boot, type=c("perc", "bca"))

4.8∗ Theories of inference

133

assumed to be unknown constants, and estimates are chosen to maximize this likelihood. This has been the approach that we have followed for most of this chapter. Another type of methodology, broadly known as “Bayesian” uses Bayes’ theorem. The essential idea is that we might have prior information (knowledge or belief) about the distribution of a parameter value before taking a sample of observations. This prior information can be updated using the sample and the rules of probability. 4.8.1 Maximum likelihood estimation

Consider the model yi = µ + εi ,

i = 1, 2, . . . , n

where µ is an unknown constant, and where the errors ε are assumed to be independent and normally distributed with mean 0 and variance σ 2 . The probability density for the ith y-value is normal with mean µ and variance σ 2 . Because of the independence assumption, the probability density of the entire sample of ys is simply the product of these normal densities. This product is the likelihood. The maximum likelihood estimates are the values of µ and σ which maximize this function. A √ calculus argument can be used to see that the estimates are y¯ and s (n − 1)/n. Note that the usual estimator of the standard deviation differs slightly from the maximum likelihood estimator; the denominator in the usual variance estimate is the number of degrees of freedom (n − 1 in this case), while it is n for the maximum likelihood estimate; this difference is negligible in large samples. For an example, consider the observed differences between heated and ambient, assuming an independent normal errors model: funlik <- function(mu, sigma, x=with(pair65, heated-ambient)) prod(dnorm(x, mu, sigma))

In practice, it is more convenient to work with the loglikelihood, rather than the likelihood. Maximizing on the log scale leads to exactly the same estimates as on the original scale. Try the following: > log(funlik(6.3, 6.1)) [1] -28.549 > log(funlik(6.33, 5.75)) [1] -28.520 > log(funlik(7, 5.75)) [1] -28.580

# Close to estimated mean and SD # Close to the actual mle’s

4.8.2 Bayesian estimation

As noted earlier, the Bayesian methodology provides a way to update our prior information about the model parameters using sample information. Usually, the prior information is summarized in the form of a probability law called the prior distribution of the model parameters. Interest usually centers on the posterior distribution of the parameters, which is proportional to the product of the likelihood and the prior distribution.

134

A review of inference concepts

A simple application of Bayes’ theorem is as follows. The incidence of HIV in adult Australian males (15–49 years) who do not have any known risk factor may be of the order of 1 in 100 000, i.e., the prior probability of infection is 0.00001. A person in this group has an initial test (for example, it may be required in order to obtain a US green card) that has a specificity of 0.01%, i.e., for every 10 000 people tested, there will on average be one false positive. How should such an individual interpret the result? If 100 000 individuals take the test, one will on average have AIDS and will almost certainly return a positive test. On the other hand there will, on average, be close to 10 false positives (0.1% of 99 999). Not infected

Infected

10000 × 0.001 = 10 (false) positives

1 true positive

The posterior odds that the person has AIDS are thus close to 1:10, certainly a narrowing from the prior odds of 1:99 999. Note that, as often happens when Bayesian calculations are used, the prior information is not very precise. What we can say is that the prior probability is, in the case mentioned, very low.

Bayesian estimation with normal prior and normal likelihood

A relatively simple example is that of a normal likelihood (as considered in the previous section) where the unobserved true mean is now also assumed to have a normal distribution, but this time with mean µ0 and variance σ02 . The posterior density of the mean is then normal with mean ny¯ + µ0 σ 2 /σ02 n + σ 2 /σ02 and variance σ2 . n + σ 2 /σ02 This assumes that σ 2 is actually known; an estimate can be obtained using the sample variance. Alternatively, we could put a prior distribution on this parameter as well. In problems where the model contains many parameters, each with its own prior distribution, the exact calculation of the posterior distribution for each parameter can be quite involved. Fortunately, in recent years, a simulation technique (called Markov Chain Monte Carlo, or MCMC) has been shown to give very effective approximations to these posterior distributions. Calculations must run for long enough that the posterior distribution reaches a steady state that is independent of the starting values of parameters. The steady state or stationary distribution is designed to be the posterior distribution of the parameter(s) of interest. Exercise 12 in Chapter 3, and the two following exercises, demonstrated the simulation of finite state Markov chains. Section 5.9 will demonstrate the use of Bayesian MCMC for straight line regression.

4.9 Recap

135

4.8.3 If there is strong prior information, use it!

Any methodology that ignores strong prior information is inappropriate, and may be highly misleading. Diagnostic testing (the AIDS test example mentioned above) and criminal investigations provide cogent examples. Using the hypothesis testing framework, we take the null hypothesis H0 , in the AIDS test example, to be the hypothesis that the individual does not have HIV. Given this null hypothesis, the probability of a positive result is 0.0001. Therefore the null hypothesis is rejected. As we saw, the prior information makes such a conclusion entirely inappropriate. In a serious criminal case, the police might scrutinize a large number of potential perpetrators. A figure of 10 000 or more is entirely within the range of possibility. Suppose there is a form of incriminating evidence that is found in one person in 1000. Scrutiny of 10 000 potential perpetrators will on average net 10 suspects. Suppose one of these is later charged. The probability of such incriminating evidence, assuming that the defendant is innocent, is indeed 0.001. The police screening will net around 10 innocent people along with, perhaps, the one perpetrator. The following summarizes the expected result of the police search for a suspect. It is optimistic in its assumption that the perpetrator will be among those netted.

Not the perpetrator

The perpetrator

10000 × 0.001 = 10 (false) positives

1 true positive

This evidence leads to odds of 1:10 or worse, i.e., less than 10%, that the defendant is guilty. On its own, it should be discounted. The interpretation of results from medical tests for AIDS is discussed in detail in Gigerenzer (2002). The calculations just given made an informal use of Bayesian methodology. Such an approach is essential when, as here, there is strong prior knowledge. Where there is no strong prior knowledge and the prior assessment of probabilities is little more than a guess, a Bayesian analysis may nevertheless be insightful.

4.9 Recap Dos and don’ts

r Do examine appropriate plots. r Ensure that the analysis and graphs reflect any important structure in the data. r Always present means, standard errors, and numbers for each group. Results from formal significance tests have secondary usefulness. r The use of a large number of significance tests readily leads to data summaries that lack coherence and insight. Consider whether there is an alternative and more coherent analysis that would provide better insight. r Reserve multiple range tests for unstructured data. r Think about the science behind the data. What analysis will best reflect that science?

136

A review of inference concepts

r The aim should be an insightful and coherent account of the data, placing it in the context of what is already known. Ensure that the statistical analysis assists this larger purpose. 4.10 Further reading On general issues of style and approach, see Wilkinson and Task Force on Statistical Inference (1999), Maindonald (1992), Krantz (1999) and Gigerenzer (1998, 2002). See also the statistical good practice guidelines at the web site http://www.ssc.rdg.ac.uk/ publications/publications.html. Miller (1986) has extensive comment on consequences of failure of assumptions, and on how to handle such failures. On the design of experiments, and on analysis of the resulting data, see Cox (1958), Cox and Reid (2000). We include further brief discussion of the design and analysis of experiments in Chapter 10. Formal hypothesis testing, which at one time had become almost a ritual among researchers in psychology, is now generating extensive controversy, reflected in the contributions to Harlow et al. (1997). The review of the Harlow et al. book in Krantz (1999) is a good guide to the controversy. See also Gigerenzer (1998), Wilkinson and Task Force on Statistical Inference (1999), Nicholls (2000). Gigerenzer et al. (1989) give interesting historical background to different styles and approaches to inference that have grown up in one or other area of statistical application. Wonnacott and Wonnacott (1990) have an elementary account of Bayesian methodology. See also Gelman et al. (2003), Gill (2008). There is a helpful brief summary of Bayesian methodology, including Bayesian modeling, in Chapters 4, 6 and 7 of Bolker (2008). Gigerenzer (2002) demonstrates the use of Bayesian arguments in several important practical contexts, including AIDS testing and screening for breast cancer. Chapter 4 of Senn (2003) has interesting comments on competing theories of statistical inference. Young and Smith (2005) give a terse and remarkably comprehensive exposition of competing theories, which does however make relatively severe technical demands. References for further reading

Bolker, B. M. 2008. Ecological Models and Data in R. Cox, D. R. 1958. Planning of Experiments. Cox, D. R. and Reid, N. 2000. Theory of the Design of Experiments. Gelman, A. B., Carlin, J. S., Stern, H. S. and Rubin, D. B. 2003. Bayesian Data Analysis, 2nd edn. Gigerenzer, G. 1998. We need statistical thinking, not statistical rituals. Behavioural and Brain Sciences 21: 199–200. Gigerenzer, G. 2002. Reckoning with Risk: Learning to Live with Uncertainty. Gigerenzer, G., Swijtink, Z., Porter, T., Daston, L., Beatty, J. and Kr¨uger, L. 1989. The Empire of Chance. Gill, J. 2008. Bayesian Methods: A Social and Behavioral Sciences Approach, 2nd edn. Harlow, L. L., Mulaik, S. A. and Steiger, J. H. (eds). 1997. What If There Were No Significance Tests? Krantz, D. H. 1999. The null hypothesis testing controversy in psychology. Journal of the American Statistical Association 44: 1372–81.

4.11 Exercises

137

Maindonald, J. H. 1992. Statistical design, analysis and presentation issues. New Zealand Journal of Agricultural Research 35: 121–41. Miller, R. G. 1986. Beyond ANOVA, Basics of Applied Statistics. Nicholls, N. 2000. The insignificance of significance testing. Bulletin of the American Meteorological Society 81: 981–6. Senn, S. 2003. Dicing with Death: Chance, Risk and Health. Wilkinson, L. and Task Force on Statistical Inference. 1999. Statistical methods in psychology journals: guidelines and explanation. American Psychologist 54: 594–604. Wonnacott, T. H. and Wonnacott, R. 1990. Introductory Statistics, 5th edn. Young, G. and Smith, R. L. 2005. Essentials of Statistical Inference. 4.11 Exercises 1. Using the data set nswdemo (DAAG), determine 95% confidence intervals for: (a) the 1975 mean incomes of each group; (b) the 1978 mean incomes of each group. Finally, calculate a 95% confidence interval for the difference in mean income between treated and controls in 1978. 2. Draw graphs that show, for degrees of freedom between 1 and 100, the change in the 5% critical value of the t-statistic. Compare a graph on which neither axis is transformed with a graph on which the respective axis scales are proportional to log(t-statistic) and log(degrees of freedom). Which graph gives the more useful visual indication of the change in the 5% critical value of the t-statistic with increasing degrees of freedom? 3. Generate a random sample of 10 numbers from a normal distribution with mean 0 and standard deviation 2. Use t.test() to test the null hypothesis that the mean is 0. Now generate a random sample of 10 numbers from a normal distribution with mean 1.5 and standard deviation 2. Again use t.test() to test the null hypothesis that the mean is 0. Finally write a function that generates a random sample of n numbers from a normal distribution with mean µ and standard deviation 1, and returns the p-value for the test that the mean is 0. 4. Use the function that was created in Exercise 3 to generate 50 independent p-values, all with a sample size n = 10 and with mean µ = 0. Use qqplot(), with the argument setting x = qunif(ppoints(50)), to compare the distribution of the p-values with that of a uniform random variable, on the interval [0, 1]. Comment on the plot. 5. The following code draws, in a 2 × 2 layout, 10 boxplots of random samples of 1000 from a normal distribution, 10 boxplots of random samples of 1000 from a t-distribution with 7 d.f., 10 boxplots of random samples of 200 from a normal distribution, and 10 boxplots of random samples of 200 from a t-distribution with 7 d.f.: oldpar <- par(mfrow=c(2,2)) tenfold1000 <- rep(1:10, rep(1000,10)) boxplot(split(rnorm(1000*10), tenfold1000), ylab="normal - 1000") boxplot(split(rt(1000*10, 7), tenfold1000), ylab=expression(t[7]*" - 1000")) tenfold100 <- rep(1:10, rep(100, 10)) boxplot(split(rnorm(100*10), tenfold100), ylab="normal - 100") boxplot(split(rt(100*10, 7), tenfold100), ylab=expression(t[7]*" - 100")) par(oldpar)

138

A review of inference concepts Refer back to the discussion of heavy-tailed distributions in Subsection 3.2.2, and comment on the different numbers and configurations of points that are flagged as possible outliers.

6. Here we generate random normal numbers with a sequential dependence structure: y1 <- rnorm(51) y <- y1[-1] + y1[-51] acf(y1) # acf is ‘autocorrelation function’ # (see Chapter 9) acf(y) Repeat this several times. There should be no consistent pattern in the acf plot for different random samples y1. There will be a fairly consistent pattern in the acf plot for y, a result of the correlation that is introduced by adding to each value the next value in the sequence. 7. Create a function that does the calculations in the first two lines of the previous exercise. Put the calculation in a loop that repeats 25 times. Calculate the mean and variance for each vector y that is returned. Store the 25 means in the vector av, and store the 25 variances in the vector v. Calculate the variance of av. 8. The following use the data frame nswpsid3, created as in footnote 8: (a) For each column of the data set nswpsid3 after the first, compare the control group (trt==0) with the treatment group (trt==1). Use overlaid density plots to compare the continuous variables, and two-way tables to compare the binary (0/1) variables. Where are the greatest differences? (b) Repeat the comparison, but now for the data set nswdemo. (c) Compare and contrast the two sets of results. Read carefully the help pages for psid3 and for nswdemo, and comment on why the different thrust of the two sets of results is perhaps not surprising. 9. In a study that examined the use of acupuncture to treat migraine headaches, consenting patients on a waiting list for treatment for migraine were randomly assigned in a 2:1:1 ratio to acupuncture treatment, a “sham” acupuncture treatment in which needles were inserted at non-acupuncture points, and waiting-list patients whose only treatment was self-administered (Linde et al., 2005). (The “sham” acupuncture treatment was described to trial participants as an acupuncture treatment that did not follow the principles of Chinese medicine.) Analyze the following two tables. What, in each case, are the conclusions that should be drawn from the analyses? Comment on implications for patient treatment and for further research: (a) Outcome is classified according to numbers of patients who experienced a greater than 50% reduction in headaches over a four-week period, relative to a pre-randomization baseline:

≥ 50% reduction < 50% reduction

Acupuncture

Sham acupuncture

Waiting list

74 71

43 38

11 65

(b) Patients who received the acupuncture and sham acupuncture treatments were asked to guess their treatment. Results were:

4.11 Exercises

Chinese Other Don’t know

139

Acupuncture

Sham acupuncture

82 17 30

30 26 16

10. Use mosaicplot() to display the table rareplants (Subsection 4.3.1) that was created using code in footnote 11. Annotate the mosaic plot to draw attention to the results that emerged from the analysis in Subsection 4.3.1. 11. The table UCBAdmissions was discussed in Subsection 2.2.1. The following gives a table that adds the 2 × 2 tables of admission data over all departments: ## UCBAdmissions is in the datasets package ## For each combination of margins 1 and 2, calculate the sum UCBtotal <- apply(UCBAdmissions, c(1,2), sum) What are the names of the two dimensions of this table? (a) From the table UCBAdmissions, create mosaic plots for each faculty separately. (If necessary refer to the code given in the help page for UCBAdmissions.) (b) Compare the information in the table UCBtotal with the result from applying the function mantelhaen.test() to the table UCBAdmissions. Compare the two sets of results, and comment on the difference. (c) The Mantel–Haenzel test is valid only if the male-to-female odds ratio for admission is similar across departments. The following code calculates the relevant odds ratios: apply(UCBAdmissions, 3, function(x) (x[1,1]*x[2,2])/(x[1,2]*x[2,1])) Is the odds ratio consistent across departments? Which department(s) stand(s) out as different? What is the nature of the difference? [For further information on the Mantel–Haenszel test, see the help page for mantelhaen.test and Agresti (2002, pp. 287f).] 12. Tables 4.10A and B contain fictitious data that illustrate issues that arise in combining data across tables. To enter the data for Table 4.10A, type: admissions <- array(c(30,30,10,10,15,5,30,10), dim=c(2,2,2)) and similarly for Table 4.10B. The third dimension in each table is faculty, as required for using faculty as a stratification variable for the Mantel–Haenzel test. From the help page for mantelhaen.test(), extract and enter the code for the function woolf(). Apply the function woolf(), followed by the function mantelhaen.test(), to the data of each of Tables 4.10A and B. Explain, in words, the meaning of each of the outputs. Then apply the Mantel–Haenzel test to each of these tables. 13. The function overlapDensity() in the DAAG package can be used to visualize the unpaired version of the t-test. Type in ## Compare densities for ambient & heated: list two65 (DAAG) with(two65, overlapDensity(ambient, heated)) # Do overlapDensity(ambient, heated) with ambient and heated # taken, if not found elsewhere, from the columns of two65

140

A review of inference concepts

Table 4.10 These illustrate the dangers of adding over contingency tables. In B, biases that go in different directions in the two faculties have canceled in the table of totals. A: Engineering

Admit Deny

Male

Female

30 30

10 10

30 30

20 10

Sociology Male

Female

Admit Deny

15 5

30 10

Admit Deny

10 5

20 25

Total Male

Female

Admit Deny

45 35

40 20

Admit Deny

40 35

40 35

B: Admit Deny

in order to observe estimates of the stretch distributions of the ambient (control) and heated (treatment) elastic bands. 14.∗ For constructing bootstrap confidence intervals for the correlation coefficient, it is advisable to work with the Fisher z-transformation of the correlation coefficient. The following lines of R code show how to obtain a bootstrap confidence interval for the z-transformed correlation between chest and belly in the possum data frame. The last step of the procedure is to apply the inverse of the z-transformation to the confidence interval to return it to the original scale. Run the following code and compare the resulting interval with the one computed without transformation. Is the z-transform necessary here? z.transform <- function(r) .5*log((1+r)/(1-r)) z.inverse <- function(z) (exp(2*z)-1)/(exp(2*z)+1) possum.fun <- function(data, indices) { chest <- data$chest[indices] belly <- data$belly[indices] z.transform(cor(belly, chest))} possum.boot <- boot(possum, possum.fun, R=999) z.inverse(boot.ci(possum.boot, type="perc")$percent[4:5]) # See help(bootci.object). The 4th and 5th elements of # the percent list element hold the interval endpoints. 15. The 24 paired observations in the data set mignonette were from five pots. The observations are in order of pot, with the numbers 5, 5, 5, 5, 4 in the respective pots. Plot the data in a way that shows the pot to which each point belongs. Also do a plot that shows, by pot, the differences between the two members of each pair. Do the height differences appear to be different for different pots? 16. Add code to the function mean.and.sd(), defined in Subsection 1.4.3 for calculation of a 95% confidence interval for the mean. Recall that the multiplier for the standard error is qt(0.975, nu), where nu is the number of degrees of freedom for the standard deviation estimate. 17. Use the function rexp() to simulate 100 random observations from an exponential distribution with rate 1. Use the bootstrap (with 99 999 replications) to estimate the standard error

4.11 Exercises

141

of the median. Repeat several times. Compare with the result that would be obtained using √ the normal approximation, i.e., π/(2n). 18. Low doses of the insecticide toxaphene may cause weight gain in rats (Chu et al., 1988). A sample of 20 rats are given toxaphene in their diet, while a control group of 8 rats are not given toxaphene. Assume further that weight gain among the treated rats is normally distributed with a mean of 60 g and a standard deviation of 30 g, while weight gain among the control rats is normally distributed with a mean of 10 g and a standard deviation of 50 g. Using simulation, compare confidence intervals for the difference in mean weight gain, using the pooled variance estimate and the Welch approximation. Which type of interval is correct more often? Repeat the simulation experiment under the assumption that the standard deviations are 40 g for both samples. Is there a difference between the two types of interval now? Hint: Is one of the methods giving systematically larger confidence intervals? Which type of interval do you think is best? 19. The following simulation experiment investigates least-squares estimation of the mean: set.seed(32083) x <- rnorm(100, mean=3, sd=7) Next, confirm that the sample mean of the values in x is near 4.642. Now plot the sum of squared deviations from µ: 100 

(xi − µ)2

i=1

as a function of µ, using the following code: lsfun <- function(mu) apply(outer(x, mu, "-")ˆ2, 2, sum) curve(lsfun, from=4.6, to=4.7) Repeat this experiment for different samples, noting where the minimum of the sum of the squares is located each time. 20.∗ Experiment with the pair65 example and plot various views of the likelihood function, either as a surface using the persp() function or as one-dimensional profiles using the curve() function. Is there a single maximizer? Where does it occur? 21.∗ Suppose the mean reaction time to a particular stimulus has been estimated in several previous studies, and it appears to be approximately normally distributed with mean 0.35 seconds with standard deviation 0.1 seconds. On the basis of 10 new observations, the mean reaction time is estimated to be 0.45 seconds with an estimated standard deviation of 0.15 seconds. Based on the sample information, what is the maximum likelihood estimator for the true mean reaction time? What is the Bayes’ estimate of the mean reaction time? 22.∗ Plot the likelihood function for the mean in the normal errors model for the pair65 differences. Assume a normal prior distribution with mean 6.0 and standard deviation 5.0. Plot the prior distribution as well as the posterior distribution of the mean. Is it reasonable to view the posterior distribution as a compromise between the prior distribution and the likelihood?

5

Regression with a single predictor

Data for which the models of this chapter may be appropriate can be displayed as a scatterplot. The focus will be on the straight line model, though the use of transformations makes it possible to accommodate specific forms of non-linear relationship within this framework. By convention, the x-variable, plotted on the horizontal axis, has the role of explanatory variable. The y-variable, plotted on the vertical axis, has the role of response or outcome variable. Many of the issues that arise for these simple regression models are fundamental to any study of regression methods. Various special applications of linear regression raise their own specific issues. One such special application, discussed in Subsection 5.6.2, is to size and shape data. Scrutiny of the scatterplot should precede regression calculations. Such a plot may indicate that the intended regression is plausible, or it may reveal unexpected features. If there are many observations, it is often useful to compare the fitted line with a fitted smooth curve. If this differs substantially from an intended line, then straight line regression may be inappropriate, as in Figure 2.6. The fitting of such smooth curves will be a major focus of Chapter 7.

5.1 Fitting a line to data How accurate is the line?

Application of the summary function to an lm object from a straight line regression, as in Subsection 5.1.1 following, gives a standard error for each of a and b. Standard errors of predicted values may also or alternatively be of interest; we defer discussion of these until Section 5.3. Determination of these standard errors requires the specific statistical assumptions that will now be noted. In the model yi = α + βxi + εi the assumptions are that given xi , the response yi is from a normal distribution with mean α + βxi , and that the yi are sampled independently. Equivalently, the εi are independently and identically distributed as normal variables with mean 0 and variance σ 2 . With different assumptions (e.g., a sequential correlation between successive data points), the standard errors will be different.

5.1 Fitting a line to data

143

5.1.1 Summary information – lawn roller example

As described in Subsection 3.1.2, we use the R model formula depression ˜ weight to supply the model information to the function lm(). We then use summary() to display the output:1 > > > > >

library(DAAG) ## Fit lm model: data from roller (DAAG); output in roller.lm roller.lm <- lm(depression ˜ weight, data = roller) ## Use the extractor function summary() to summarize results summary(roller.lm)

Call: lm(formula = depression ˜ weight, data = roller) Residuals: Min 1Q Median -8.18 -5.58 -1.35

3Q 5.92

Max 8.02

Coefficients: Estimate Std. Error t value Pr(>|t|) (Intercept) -2.09 4.75 -0.44 0.6723 weight 2.67 0.70 3.81 0.0052 Residual standard error: 6.74 on 8 degrees of freedom Multiple R-Squared: 0.644, Adjusted R-squared: 0.6 F-statistic: 14.5 on 1 and 8 DF, p-value: 0.00518

Following a numerical summary of the residuals, there is a table of the estimated regression coefficients and their standard errors. The intercept of the fitted line is a = −2.09 (SE = 4.75), while the estimated slope is b = 2.67 (SE = 0.70). The p-value for the slope (testing the null hypothesis that β = true slope = 0) is small, consistent with the evident linear trend. The p-value for the intercept (testing α = 0) is 0.67, i.e., the difference from zero may well be random sampling error. Thus, consistently with the intuition that depression should be proportional to weight, it would be reasonable to fit a model that lacks an intercept term. We leave this as an exercise for the reader.2 The standard deviation of the noise term, here identified as the residual standard error, is 6.735. We defer comment on R 2 and the F -statistic until Subsection 5.1.4. 5.1.2 Residual plots

The residuals provide information about the noise term in the model, and allow limited checks on model assumptions. Note however that, in such a small data set, departures from assumptions will be hard to detect. 1

2

## Global options used for this and most later output options(show.signif.stars=FALSE, digits=4) # show.signif.stars=FALSE suppresses interpretive features that, # for our use of the output, are unhelpful. ## Fit model that omits intercept term; i.e., y = bx lm(depression ˜ -1 + weight, data=roller)

144

Regression with a single predictor

B

Residuals 0 5

● ● ●

●

●

● ●

●

● ●

5

15 25 Fitted values

Ordered data values 0 5

A

●

●

● ●

●

●

● ● ● ●

0.5 1.5 Quantiles of normal distribution

Figure 5.1 Diagnostic plots for the regression of Figure 3.2A. Panel A plots residuals against fitted values. Panel B is a normal probability plot of residuals, giving a visual check whether data are consistent with a normal error distribution.

Two common checks, both available by using the plot() function with an lm object, are: r A plot of residuals versus fitted values, as in Figure 5.1A. This allows a visual check for any pattern in the residuals that might suggest a curve rather than a line. r A normal probability plot of residuals, as in Figure 5.1B. If residuals are from a normal distribution points should lie, to within statistical error, close to a line. Code for these plots is:3 ## A: Plot residuals vs fitted values; B: normal probability plot plot(roller.lm, which = 1:2)

Note that if the argument which is left at its default, i.e., type plot(roller.lm), there are two further plots. These further default plots will be demonstrated later. In Figure 5.1A, there is a suggestion of clustering in the residuals, but no clear indication that there should be a curve rather than a line. For interpreting the normal probability plot in Figure 5.1B, the eye needs a reference standard. It is useful to compare a plot such as Figure 5.1B against a number of independent plots from computer-generated normal data with the same number of observations, as in Figure 5.2, thus in effect calibrating the eye. The function qreference() (DAAG) may be used to generate suitable plots:4 ## Normal probability plot, plus 7 reference plots qreference(residuals(roller.lm), nrep=8, nrows=2)

3 4

For side-by-side plots, precede with par(mfrow=c(1,2)). ## Alternatively, use the following code: test <- residuals(roller.lm); n <- length(test) av <- mean(test); sdev <- sd(test) y <- c(test, rnorm(7*n, av, sdev)) fac <- c(rep("residuals(roller.lm)",n), paste("reference", rep(1:7, rep(n,7)))) fac <- factor(fac, levels=unique(fac)) library(lattice) qqmath(˜ y|fac, aspect=1, layout=c(4,2))

5.1 Fitting a line to data −10 −5 reference 4

0

5

145

10

−10 −5

reference 5

reference 6

●

●

● ● ●

●

5

10

reference 7 ●

●

● ●

●

●

●

● ●

●

10

●

● ● ●

● ● ● ●

●

●

●

● ● ●

●

●

●

0

●

● ●

● ●

0 −10

● ●

residuals(roller.lm)

reference 1

reference 2

reference 3

●

10 ● ●

0 −10

●

●

● ● ●

●

●● ●

●

● ●

●

●● ●

●

●

● ● ● ●

● ● ●

●

●

●

●

● ●

● ● ●

●●

●

−10 −5

0

5

10

−10 −5

0

5

10

Figure 5.2 The normal probability plot for the regression of Figure 3.2A is shown in the lower left panel. Other panels show normal probability plots for computer-generated normal data.

5.1.3 Iron slag example: is there a pattern in the residuals?

Now consider an example where there is an evident pattern in residuals from a straight line regression. The data compare two methods for measuring the iron content in slag – a magnetic method and a chemical method (data are from Roberts, 1974, p. 126). The chemical method requires greater effort and is presumably expensive, while the magnetic method is quicker and easier. Figure 5.3A suggests that the straight line model is wrong.5 The smooth curve (shown with a dashed line) gives a better indication of the pattern in the data. Panel B shows the residuals from the straight line fit.6 The non-linearity is now more evident. Panel C plots observed values against predicted values.7 Panel D allows a visual check on whether the error variance is constant.8 There are theoretical reasons for plotting the square root of absolute values of the residuals on the vertical axis in panel D. Taken at face value, Figure 5.3D might seem to indicate that the variance decreases with increasing value of magnetic. Note, however, that the residuals are from the straight line model, which Figures 5.3A and B suggested was inappropriate. Thus the plot in panel D 5

6

7

8

## Panel A: chemical vs magnetic (Data frame ironslag from DAAG) plot(chemical ˜ magnetic, data=ironslag) ironslag.lm <- lm(chemical ˜ magnetic, data=ironslag) abline(ironslag.lm) with(ironslag, lines(lowess(chemical ˜ magnetic, f=.9), lty=2)) ## Panel B: Residuals from straight line fit, vs magnetic res <- residuals(ironslag.lm) plot(res ˜ magnetic, xlab="Residual", data=ironslag) with(ironslag, lines(lowess(res ˜ magnetic, f=.9), lty=2)) ## Panel C: Observed vs predicted yhat <- fitted(ironslag.lm) plot(chemical ˜ yhat, data=ironslag, xlab="Predicted chemical", ylab="Chemical") with(ironslag, lines(lowess(chemical ˜ yhat, f=.9), lty=2)) ## Panel D: Check whether error variance seems constant sqrtabs <- sqrt(abs(res)) plot(sqrtabs ˜ yhat, data=ironslag, xlab = "Predicted chemical", ylab = expression(sqrt(abs(residual))), type = "n") panel.smooth(yhat, sqrtabs, span = 0.95)

Regression with a single predictor

●

● ● ● ●● ● ● ● ● ● ● ● ● ●● ●● ● ●● ● ● ●● ● ● ● ●● ● ● ●●● ● ●●● ● ●● ● ● ●

Residual

25 20 15

B

●

10

Chemical

30

A Fitted line and lowess curve

2 4 6

146

Residuals from line, with smooth ●

●

● ● ● ● ● ●● ● ● ● ● ●●● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ●● ● ● ● ● ●

●

10

●

20

30

40

10

20

Magnetic

2.5 1.5

Residual

●

0.5

30 25 20

Chemical

15 10

● ● ● ● ● ●● ● ● ● ● ● ● ● ●● ●● ● ●● ● ● ●● ● ● ● ●● ● ● ●●● ● ●●● ● ●● ● ● ● ●

25

30

40

D Is variance constant about line?

●

20

●

Magnetic

C Observed, vs prediction from line

15

● ●●

●

30

● ● ● ● ●● ● ● ● ● ●● ● ● ●● ● ● ● ● ● ●● ● ● ●● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ●

15

Predicted chemical

20

25

●

30

Predicted chemical

Figure 5.3 Chemical test of iron content in slag versus magnetic test. The fitted curves used the lowess smooth. In panel D, the downward slope suggests lower variance for larger fitted values. See however Figure 5.4.

B

0.5

1.5

residual

2 −2 −6

Residual

Is variance about curve constant?

2.5

Residuals from fitted lowess curve.

6

A

10

20 30 Magnetic

40

15

20 25 30 Predicted chemical

Figure 5.4 Residuals (panel A), and square root of absolute values of residuals (panel B), for the lowess smooth for the data of Figure 5.3. (Exercise 8 at the end of the chapter has the R code.)

may be misleading. For an accurate assessment, it is necessary to examine the equivalent plot for residuals from the smooth curve. Figure 5.4 shows the plot of residuals from the smooth curve versus magnetic, together with the plot of square root of absolute values of residuals against predicted chemical test result. Any suggestion of heterogeneity is now of small consequence. Where there is genuine heterogeneity of variance, and an accurate estimate of the variance at each data point is available, data points should be weighted proportionately to the reciprocal of the variance. Getting an estimate to within some constant of proportionality is enough. It may be possible to guess at a suitable functional form for the change in variance with x or (equivalently, since y = a + bx) with y. For example, the variance may be proportional to y.

5.2 Outliers, influence, and robust regression

147

5.1.4 The analysis of variance table

The analysis of variance table breaks the sum of squares about the mean, for the y-variable, into two parts: a part that is accounted for by the deterministic component of the model, i.e., by a linear function of weight, and a part attributed to the noise component or residual. For the lawn roller example, the analysis of variance table is: > anova(roller.lm) Analysis of Variance Table Response: depression Df Sum Sq Mean Sq F value Pr(>F) weight 1 657.97 657.97 14.503 0.005175 Residuals 8 362.93 45.37

The total sum of squares (about the mean) for the 10 observations is 1020.9 (= 658.0 + 362.9; we round to one decimal place). Including weight reduced this by 658.0, giving a residual sum of squares equal to 362.9. For comparing the reduction with the residual, it is best to look at the column headed Mean Sq, i.e., mean square. The mean square for weight was 658.0; this compares with a mean square of 45.4 for the residual. The degrees of freedom can be understood thus: Two points determine a line. With just two observations, both residuals would be zero, yielding no information about the noise. Every additional observation beyond two yields one additional degree of freedom for estimating the noise variance. Thus with 10 points, 10 − 2 (= 8) degrees of freedom are available (in the residuals) for estimating the noise variance. (Where a line is constrained to pass through the origin, one point is enough to determine the line, and with 10 points the variance would be estimated with 9 degrees of freedom.) This table has the information needed for calculating R 2 (also known as the “coefficient of determination”) and adjusted R 2 . The R 2 statistic is the square of the correlation coefficient, and is the sum of squares due to weight divided by the total sum of squares: R2 =

658.0 = 0.64, 1020.9

while adjusted R 2 = 1 −

362.9/8 = 0.60. 1020.9/9

Adjusted R 2 takes into account the number of degrees of freedom, and is in general preferable to R 2 . Neither statistic gives any direct indication of how well the regression equation will predict when applied to a new data set. Subsection 6.3.2 will argue against use of these statistics, and suggest alternatives. See also Section 5.4.

5.2 Outliers, influence, and robust regression The data displayed in Figure 5.5, with data shown on the right, are for a collection of eight softback books. Additionally, the figure shows the fitted regression line, with information on the residuals from the line.

148

Regression with a single predictor

1000

●

●

volume

800

● ●●

● ● ● 5 ●

400

●

● ● ●21.3 ●

a = 41.4 SE = 97.6 b = 0.686 SE = 0.106

●

400

800 1200 Volume (cc)

weight

412 953 929 1492 419 1010 595 1034

215.9

600

Weight (g)

7

●

1600

250 700 650 975 350 950 425 725

Figure 5.5 Volumes (≤m3 ) and weights (g), for eight softback books. The figure also shows the fitted regression line, with information on the residuals from the line.

400

600

800

Fitted values

1000

●1 ●4

−1.5

● ●

●

−0.5

0.5

1.5

1.2

4

0.8

●

Cook's distance

●1

D: Cook's Distance

6

0.4

1.5 1.0

4●

0.5

2 1

●

6●

1

●

0.0

4●

●

Standardized residuals

−100

●1

● ●

●

C: Scale−Location

0.0

● ● ●

●

6●

0

●

B: Normal Q−Q

−1

100 0

Residuals

200

6●

Standardized residuals

A: Residuals vs Fitted

400

Theoretical quantiles

600

800

1000

Fitted values

1

2

3

4

5

6

7

8

Obs. number

Figure 5.6 Diagnostic plots for Figure 5.5.

Here is the output from the regression calculations: ## Fit lm model: data frame softbacks (DAAG) > softbacks.lm <- lm(weight ˜ volume, data=softbacks) > summary(softbacks.lm) . . . . Coefficients: Estimate Std. Error t value Pr(>|t|) (Intercept) 41.372 97.559 0.42 0.68629 volume 0.686 0.106 6.47 0.00064 . . . .

Figure 5.6 shows regression diagnostics. Suitable code is: par(mfrow=c(2,2)) # Use par(mfrow=c(1,4)) for layout in figure plot(softbacks.lm, which=1:4) # By default, plots 1:3 and 5 [which=c(1:3,5)] are given par(mfrow=c(1,1))

We should already be familiar with plots A and B from Figure 5.6. For regression with one explanatory variable, plot A is equivalent to a plot of residuals against the explanatory variable. Plot C, which is not of much interest for the present data, is designed for examining the constancy of the variance. Plot D identifies residuals that are influential in determining

5.3 Standard errors and confidence intervals

149

the form of the regression line. Points with Cook’s distances that are greater than one, or whose Cook’s distances are substantially larger than for other points, may warrant investigation. The largest two residuals in Figure 5.6D, for observations 4 and 6, are identified with their row labels. For the softbacks data, the row labels are the observation numbers. Cook’s distance is a measure of influence; it measures the extent to which the line would change if the point were omitted. Although observation 6 has the largest residual, its Cook’s distance is relatively small. Observation 4 has the largest Cook’s distance. In part, the value is large because this point is at the extreme end of the range of x-values. It is a high leverage point that exerts a greater pull on the regression line than observations closer to the center of the range. Since its y-value is lower than would be predicted by the line, it pulls the line downward. Diagnostic plots, such as Figure 5.6, are not definitive. Rather, they draw attention to points that require further investigation. Here, with only eight points, it would not make sense to omit any of them, especially as points 4 and 6 are both, for different reasons, candidates for omission. It may, on checking, turn out that an outlier has arisen from a recording or similar error. Where an outlier seems a genuine data value, it is good practice to do the analysis both with and without the outlier. If retention of an apparent outlier makes little difference to the practical use and interpretation of the results, it is usually best to retain it in the main analysis. If an outlier that seems a genuine data value is omitted from the main analysis, it should be reported along with the main analysis, and included in graphs.

Robust regression

Robust regression offers a half-way house between including outliers and omitting them entirely. Rather than omitting outliers, it downweights them, reducing their influence on the fitted regression line. This has the additional advantage of making outliers stand out more strongly against the line. The MASS package has the robust regression function rlm(). Note also the resistant regression function lqs(), also in the MASS package. Resistant regression methods aim to ensure that outliers do not contribute to the regression fit. For both robust and resistant methods, it is important that residuals have an approximately symmetric distribution. See further, Section 6.3 and Exercise 14 at the end of Chapter 6.

5.3 Standard errors and confidence intervals Recall that since two parameters (the slope and intercept) have been estimated, the error mean square is calculated with n − 2 degrees of freedom. As a consequence, the standard errors for the slope and for predicted values are calculated with n − 2 degrees of freedom. Both involve the use of the square root of the error mean square. Additionally, as will be demonstrated in Subsection 5.3.2, the predict() function can be used to obtain standard errors and/or confidence intervals for predicted values. A wide confidence interval for the regression slope implies that intervals for predicted values will likewise be wide.

150

Regression with a single predictor

5.3.1 Confidence intervals and tests for the slope

A 95% confidence interval for the regression slope is b ± t.975 SEb where t.975 is the 97.5% point of the t-distribution with n − 2 degrees of freedom, and SEb is the standard error of b. We demonstrate the calculation for the roller data. From the second row of the Coefficients table in the summary output of Subsection 5.1.1, the slope estimate is 2.67, with SE = 0.70. The t-critical value for a 95% confidence interval on 10 − 2 = 8 degrees of freedom is t.975 = 2.30. Therefore, the 95% confidence interval is:9 2.67 ± 2.3 × 0.7 = (1.1, 4.3). If the 95% confidence interval for b contains 0, the null hypothesis that the slope is zero will be rejected at the 95% significance level. (For the testing of such hypotheses, refer back to Subsection 5.1.1.)

5.3.2 SEs and confidence intervals for predicted values

There are two types of predictions: prediction of points on the line, and prediction of a new data value. The SE estimates of predictions for new data values take account both of uncertainty in the line and of the variation of individual points about the line. Thus the SE for prediction of a new data value is larger than that for prediction of points on the line. Table 5.1 shows expected values of the depression, with SEs, for various roller weights.10 The column headed SE indicates the precision of points on the line. The column headed SE.OBS indicates the precision with which new observations can be predicted. For determining SE.OBS, there are two sources of uncertainty: the standard error for the fitted value (estimated at 3.6 in row 1) and the noise standard error (estimated at 6.74) associated with a new observation. Figure 5.7 shows 95% pointwise confidence bounds for the fitted line.11 It bears emphasizing that the validity of these calculations depends crucially on the appropriateness of the fitted model for the given data.

9

10

11

## Code for confidence interval calculations SEb <- summary(roller.lm)$coefficients[2, 2] coef(roller.lm)[2] + qt(c(0.025,.975), 8)*SEb ## Code to obtain fitted values and standard errors (SE, then SE.OBS) fit.with.se <- predict(roller.lm, se.fit=TRUE) fit.with.se$se.fit # SE sqrt(fit.with.se$se.fitˆ2+fit.with.se$residual.scaleˆ2) # SE.OBS ## Plot depression vs weight, with 95\% pointwise bounds for the fitted line plot(depression ˜ weight, data=roller, xlab = "Weight of Roller (tonnes)", ylab = "Depression in Lawn (mm)", pch = 16) roller.lm <- lm(depression ˜ weight, data = roller) abline(roller.lm$coef, lty = 1) xy <- data.frame(weight = pretty(roller$weight, 20)) yhat <- predict(roller.lm, newdata = xy, interval="confidence") ci <- data.frame(lower=yhat[, "lwr"], upper=yhat[, "upr"]) lines(xy$weight, ci$lower, lty = 2, lwd=2, col="grey") lines(xy$weight, ci$upper, lty = 2, lwd=2, col="grey")

5.3 Standard errors and confidence intervals

151

Table 5.1 Observed and fitted values of depression at the given weight values, together with two different types of SE. The column headed SE gives the precision of the predicted value. The column headed SE.OBS gives the precision of a new observation.

SE

SE.OBS

1.9 3.1 3.3 12.4

2 1 5 25

3.0 6.2 6.7 31.0

3.6 3.0 2.9 4.9

7.6 7.4 7.3 8.3

30

Fitted

√ 3.62 + 6.742 √ 3.02 + 6.742

● ●

20

● ●●

5 10

Depression in lawn (mm)

Observed depression

● ● ●

0

1 2 3 . . . 10

Predictor weight

●

●

0 2 4 6 8 10 Weight of roller (tonnes)

Figure 5.7 Lawn depression, for various weights of roller, with fitted line and showing 95% pointwise confidence bounds for points on the fitted line.

5.3.3∗ Implications for design

An emphasis of this subsection is that the choice of location of the x-values, which is a design issue, is closely connected with sample size considerations. Increasing the sample size is not the only, or necessarily the best, way to improve precision. The estimated variance of the slope estimate is SE2b = where we define

 sx2 =

s2 , nsx2 ¯ 2 − x) . n

i (xi

Here s 2 is the error mean square, i.e., s is the estimated SD for the population from which the residuals are taken. The expected value of SE2b is E[SE2b ] =

σ2 . ns2x

Now consider two alternative ways to reduce SEb by a factor of 2:

152

Regression with a single predictor 50

60

200

300

40

100

Distance moved (cm)

30

30

40

50 60 Amount of stretch (mm)

Figure 5.8 Two rubber band experiments, with different ranges of x-values. The dashed curves are pointwise 95% confidence bounds for points on the fitted line. Note that, for the panel on the right, the axis labels appear above the panel, as is done for lattice plots. Data are from elastic1 (left panel, 7 points) and elastic2 (right panel, 9 points), both from DAAG. Even with 9 points as against 7, the right panel has much wider pointwise bounds.

r By fixing the configuration of x-values, but multiplying by 4 the number of values at each discrete x-value, sx is unchanged. As n increases by a factor of 4, the expected value of SE2b reduces by a factor of 4, and SEb by a factor of 2. r Alternatively, increasing the average separation between x-values by a factor of 2 will reduce SEb by a factor of 2. Spreading out the x-values achieves the same reduction in SEb , as increasing the number of points. Checking for linearity over the extended range of x-values is, however, important. Reducing SEb reduces the standard error of the fitted values as well. Figure 5.8 shows the effect of increasing the range of x-values (the code for both panels is a ready adaptation of the code for Figure 5.7). Both experiments used the same rubber band. The first experiment used a much wider range of values of x (= amount by which the rubber band was stretched). For the left panel of Figure 5.8, sx = 10.8, while for the right panel, sx = 4.3.

5.4 Assessing predictive accuracy The training data estimate of predictive accuracy, derived by direct application of model predictions to the data from which the regression relationship was derived, gives in general an optimistic assessment. There is a mutual dependence between the model prediction and the data used to derive that prediction. It is because of this dependence that degrees of freedom for the variance are adjusted to take account of the number of parameters estimated. The issue becomes more important in contexts, such as the classification models that will be discussed in Chapter 11 and Section 12.2, where no satisfactory theoretical adjustment for the dependence is available. The simple models discussed in the present chapter are a good context in which to demonstrate general approaches that address this issue.

5.4 Assessing predictive accuracy

153

Table 5.2 Floor area and sale price, for 15 houses in Aranda, a suburb of Canberra, Australia. Row number 1 2 15

area

bedrooms

sale.price

694 905 ··· 1191

4 4 ··· 6

192.0 215.0 ··· 375.0

5.4.1 Training/test sets and cross-validation

An ideal is to assess the performance of the model on a new data set. It is good practice to split the data into two sets: the training set is for developing the model, and the test set is for testing predictions. This is a valid procedure, if the test set can be regarded as a random sample of the population to which predictions will be applied. If there are too few data to make it reasonable to divide data into training and test sets, then the method of cross-validation can be used. Cross-validation extends the training/test set approach. As with that approach, it estimates predictive accuracy for data that are sampled from the population in the same way as the existing data. The data are divided into k sets (or folds), where k is typically in the range 3 to 10. Each of the k sets becomes in turn the test set, with the remaining data forming the training set. The predictive accuracy assessments from the k folds are combined to give a measure of the predictive performance of the model. This may be done for several different measures of predictive performance. 5.4.2 Cross-validation – an example

We present an example of the use of cross-validation with a small data set. In order to simplify the discussion, we will use threefold validation only. Table 5.2 shows data on floor area and sale price for 15 houses in a suburb of Canberra, in 1999. Rows of data have been numbered from 1 to 15. For demonstrating cross-validation, we use a random number sampling system to divide the data up into three equal groups.12 The observation numbers for the three groups we obtain are: 2 3 12 13 15 1 5 7 8 14 4 6 9 10 11

Rerunning the calculations will of course lead to a different division into three groups. At the first pass (fold 1) the first set of rows will be set aside as the test data, with remaining rows making up the training data. Each such division between training data and 12

## Split row numbers randomly into 3 groups rand <- sample(1:15)%%3 + 1 # a%%3 is the remainder of a, modulo 3 # Subtract from a the largest multiple of 3 that (1:15)[rand == 1] # Observation numbers for the (1:15)[rand == 2] # Observation numbers for the (1:15)[rand == 3] # Observation numbers for the

is <= a; take remainder first group second group third group.

154

Regression with a single predictor Fold 2

Fold 3

250 150

Sale price

350

Fold 1

700

900

1100

1300

Floor area

Figure 5.9 Graphical summary of threefold cross-validation for the house sale data (Table 5.2). The line is fitted leaving out the corresponding “test” set of points. Predictions for these omitted points are used to assess predictive accuracy.

test data is known as a fold. At the second pass (fold 2) the second set of rows will be set aside as the test data, while at the third pass (fold 3) the third set of rows will be set aside as the test data. A crucial point is that at each pass the data that are used for testing are separate from the data used for prediction. Figure 5.9 is a visual summary, obtained by using the function CVlm() (DAAG) with the default setting plotit=TRUE. The following summary of the cross-validation results includes, for each fold, estimates of the mean square error. > ## Cross-validate lm calculations: data frame houseprices (DAAG) > houseprices.lm <- lm(sale.price ˜ area, data=houseprices) > CVlm(houseprices, houseprices.lm, plotit=TRUE) fold 1 Observations in Floor area Predicted price Observed price Residual

test set: 2 3 12 13 15 905.0 802.00 696.0 771.0 1191 225.9 208.63 190.9 203.4 274 215.0 215.00 255.0 260.0 375 -10.9 6.37 64.1 56.6 101

Sum of squares = 17719

Mean square = 3544

n = 5

fold 2 Observations in test set: 1 5 7 8 14 Floor area Predicted price Observed price Residual

694.0 716 222.4 225 192.0 113 -30.4 -113

Sum of squares = 15269 fold 3

821.0 714.00 1006.0 238.6 224.97 262.2 212.0 220.00 293.0 -26.6 -4.97 30.8

Mean square = 3054

n = 5

5.4 Assessing predictive accuracy

155

Observations in test set: 4 6 9 10 11 Floor area 1366 963.0 1018.0 887.00 790.00 Predicted price 412 278.4 296.6 253.28 221.22 Observed price 274 185.0 276.0 260.00 221.50 Residual -138 -93.4 -20.6 6.72 0.28 Sum of squares = 28127 Overall ms 4074

Mean square = 5625

n = 5

At each fold, the training set consists of the remaining rows of data. To obtain the estimate of the error mean square, take the total of the sums of squares and divide by 15. This gives s 2 = (17 719 + 15 269 + 28 127)/15 = 4074. Actually, what we have is an estimate of the error mean square when we use only two-thirds of the data. Thus we expect the cross-validated error to be larger than the error if all the data could be used. We can reduce the error by doing 10-fold rather than threefold crossvalidation. Or we can do leave-one-out cross-validation, which for these data is 15-fold cross-validation. Contrast s 2 = 4074 with the estimate s 2 = 2323 that we obtained from the model-based estimate in the regression output for the total data.13 5.4.3 ∗ Bootstrapping

We first indicate how resampling methods can be used to estimate the standard error of slope of a regression line. Recalling that the standard error of the slope is the standard deviation of the sampling distribution of the slope, we need a way of approximating this sampling distribution. One way of obtaining such an approximation is to resample the observations or cases directly. For example, suppose five observations have been taken on a predictor x and response y: (x1 , y1 ), (x2 , y2 ), (x3 , y3 ), (x4 , y4 ), (x5 , y5 ). Generate five random numbers with replacement from the set {1, 2, 3, 4, 5}: 3, 5, 5, 1, 2, say. The corresponding resample is then (x3 , y3 ), (x5 , y5 ), (x5 , y5 ), (x1 , y1 ), (x2 , y2 ). Note we are only demonstrating the so-called case-resampling approach. Another approach involves fitting a model and resampling the residuals. Details for both methods are in Davison and Hinkley (1997, Chapter 6). A regression line can be fit to the resampled observations, yielding a slope estimate. Repeatedly taking such resamples, we obtain a distribution of slope estimates, the bootstrap distribution. 13

## Estimate of sigmaˆ2 from regression output summary(houseprices.lm)$sigmaˆ2

156

Regression with a single predictor

As an example, consider the regression relating sale.price to area in the houseprices data. We will compute a bootstrap estimate of the standard error of the slope. For comparison purposes, note first the estimate given by lm(): 0.0664. > houseprices.lm <- lm(sale.price ˜ area, data=houseprices) > summary(houseprices.lm)$coef

(Intercept) area

Estimate Std. Error t value Pr(>|t|) 70.750 60.3477 1.17 0.2621 0.188 0.0664 2.83 0.0142

In order to use the boot() function, we need a function that will evaluate the slope for each of the bootstrap resamples: houseprices.fn <- function (houseprices, index){ house.resample <- houseprices[index, ] house.lm <- lm(sale.price ˜ area, data=house.resample) coef(house.lm)[2] # slope estimate for resampled data }

We then use the boot() function to make 999 calls to the houseprices.fn() function with different randomly generated resamples from the data frame houseprices. > set.seed(1028)

# use to replicate the exact results below

> library(boot) # ensure that the boot package is loaded > ## requires the data frame houseprices (DAAG) > (houseprices.boot <- boot(houseprices, R=999, statistic=houseprices.fn)) . . . . Bootstrap Statistics : original bias std. error t1*

0.188

0.0169

0.0916

The output shows us the original slope estimate, a bootstrap estimate of the bias of this estimate, and the standard error estimate: 0.0916. This standard error was computed from the standard deviation of the 999 resampled slope estimates. By changing the statistic argument in the boot() function appropriately, we can compute standard errors and confidence intervals for fitted values. Here we use the predict() function to obtain predictions for the given area: housepred.fn <- function(houseprices, index){ house.resample <- houseprices[index, ] house.lm <- lm(sale.price ˜ area, data=house.resample) predict(house.lm, newdata=data.frame(area=1200)) }

For example, a 95% confidence interval for the expected sale price of a house (in Aranda) having an area of 1200 square feet is (249 000, 363 000).14 14

## 95% CI for predicted price of 1200 square foot house housepred.boot <- boot(houseprices, R=999, statistic=housepred.fn) boot.ci(housepred.boot, type="perc") # "basic" is an alternative to "perc"

● ● ● ●

● ● ●●

● ●

● ● ●

● ●● ●

● ● ●● ●●

●● ● ●

4

7

10 13

House

0.9 1.0 1.1 1.2 1.3

●

1

157

B ●

Ratio of SEs bootstrap to model-based

100

A

0

Prediction errors

200

5.4 Assessing predictive accuracy

●

●

●

●

● ● ● ●

●

●

●

●

●●

2 4 6 8

●

12

House

Figure 5.10 (A) Plot of bootstrap distributions of prediction errors for regression relating sale.price to area, each based on 200 bootstrap estimates of the prediction error. (B) Ratios of bootstrap prediction standard errors to model-based prediction standard errors.

The bootstrap procedure can be used to gain additional insight into how well a regression model is making predictions. Regression estimates for each resample are used to compute predicted values at all of the original values of the predictor. The differences (i.e., the prediction errors) between the observed responses and these resampled predictions can be plotted against observation number. Repeating this procedure a number of times gives a distribution of the prediction errors at each observation. Figure 5.10 A displays a prediction error plot for the houseprices data.15 Note the large variability in the prediction error associated with observation 4. We can use the same bootstrap output to estimate the standard errors. These can be compared with the usual estimates obtained by lm. Figure 5.10B displays ratios of the bootstrap standard errors to the model-based standard errors.16 In this case, the model-based standard errors are generally smaller than the bootstrap standard errors. A cautious data analyst might prefer the bootstrap standard errors. We can also compute an estimate of the aggregate prediction error, as an alternative to the cross-validation estimate obtained in the previous subsection. There are a number of ways to do this, and some care should be taken. We refer the interested reader to Davison and Hinkley (1997, Section 6.4).

15

16

## Bootstrap estimates of prediction errors of house prices houseprices2.fn <- function (houseprices, index) { house.resample <- houseprices[index, ] house.lm <- lm(sale.price ˜ area, data=house.resample) houseprices$sale.price - predict(house.lm, houseprices) # resampled prediction # errors } n <- length(houseprices$area); R <- 200 houseprices2.boot <- boot(houseprices, R=R, statistic=houseprices2.fn) house.fac <- factor(rep(1:n, rep(R, n))) plot(house.fac, as.vector(houseprices2.boot$t), ylab="Prediction Errors", xlab="House") ## Ratios of bootstrap to model-based standard errors bootse <- apply(houseprices2.boot$t, 2, sd) usualse <- predict.lm(houseprices.lm, se.fit=TRUE)$se.fit plot(bootse/usualse, ylab="Ratio of Bootstrap SE’s to Model-Based SE’s", xlab="House", pch=16) abline(1, 0)

158

Regression with a single predictor

Commentary

The cross-validation and bootstrap estimates of mean square error are valid, provided we can assume a homogeneous variance. This is true even if data values are not independent. However, the estimate of predictive error applies only to data that have been sampled in the same way as the data that are used as the basis for the calculations. They assume that the target population will be highly comparable to the source population that generated the data. In the present instance, the estimate of predictive accuracy applies only to 1999 house prices in the same city suburb. Such standard errors may have little relevance to the prediction of house prices in another suburb, even if thought to be comparable, or to prediction for more than a very short period of time into the future. This point has relevance to the use of regression methods in business “data mining” applications. A prediction that a change will make cost savings of $500 000 in the current year may have little relevance to subsequent years. The point has special force if changes will take years rather than months to implement. A realistic, though still not very adequate, assessment of accuracy may be derived by testing a model that is based on data from previous years on a test set that is formed from the current year’s data. Predictions based on the current year’s data may, if other features of the business environment do not change, have a roughly comparable accuracy for prediction a year into the future. If the data series is long enough, we might, starting at a point part-way through the series, compare predictions one year into the future with data for that year. The estimated predictive accuracy would be the average accuracy for all such predictions. A more sophisticated approach might involve incorporation of temporal components into the model, i.e., use of a time series model. See Maindonald (2003) for more extended commentary on such issues.

5.5 Regression versus qualitative anova comparisons – issues of power An analysis that fails to take advantage of structure in the data may fail to find what is there. Figure 5.11 shows six sets of data that have been simulated to follow a linear trend. Simulation of regression models was discussed in Subsection 3.3.2. The first p-value tests for linear trend, while the second p-value tests for qualitative differences between treatment effects, ignoring the fact that the levels are quantitative (note that the test for linear trend is equivalent to the test for a linear contrast from the aov()function that is available when the explanatory term is an ordered factor). A test for linear trend is more powerful than an analysis of variance that treats the five levels as qualitatively different levels. In repeated simulations of Figure 5.11, the p-values in the test for linear trend will on average be smaller than in the analysis of variance that makes qualitative comparisons between the five levels. To get a clear indication of the effect, we need a more extensive simulation. Figure 5.12 plots results from 200 simulations. Both axes use a scale of log(p/(1 − p)). On the vertical axis are the p-values for a test for linear trend, while the horizontal axis plots p-values for an aov test for qualitative differences. The majority of points (for this simulation, 91%) lie below the line y = x.

5.5 Regression versus qualitative anova comparisons – issues of power

102

102 106 1

2

3

4

5

98 1

2

3

4

5

4

5

3

4

5

105

Linear trend:0.11 aov: 0.057

99

98 3

2

102

102

102

Response

98

2

Treatment level

1

Linear trend:0.0021 aov: 0.039 106

Linear trend:0.0018 aov: 0.062

1

Linear trend:0.13 aov: 0.17 106

Linear trend:0.0014 aov: 0.013

98

102 98

Response

106

p-values are – Linear trend:0.096 aov: 0.3

159

1

2

3

4

5

Treatment level

1

2

3

4

5

Treatment level

Figure 5.11 Test for linear trend, versus analysis of variance comparison that treats the levels as qualitatively different. The six panels are six different simulations from the straight line model with slope 0.8, SD = 2, and 4 replications per level.

Figure 5.12 This plot compares p-values from a test for linear trend with p-values from an analysis of variance test for qualitative differences, in each of 200 sets of simulated results. The line y = x is superimposed.

The function simulateLinear() in our DAAG package allows readers to experiment with such simulations. Write the p-values for a test for linear trend as pl , and the p-values for the analysis of variance test for qualitative differences as pa . Specifying type="density" gives overlaid plots of the densities for the two sets of p-values, both on a scale of log(p/(1 − p)), together with a plot of the density of log(pl /(1 − pl )) − log(pa /(1 − pa )). As the data are paired, this last plot is the preferred way to make the comparison.

The pattern of change

There are other reasons for fitting a line or curve, where this is possible, rather than fitting an analysis of variance model that has a separate parameter for each separate level of

Regression with a single predictor

0

2

4

6

8

10

0

2

4

2 6

8

10

Replace y by y~ = y 0.25

6000

60 40

y

og

l y=

(x)

0

2

4

6

8

10

Replace y by y~ = log(x) 15000

80

100

Replace y by y~ = y

y

=x

1

x

25 0.

Replace y by y~ = exp( y )

0

y=

Replace y by y~ = y4

0.6 0.8 1.0 1.2 1.4 1.6 1.8

0.5 1.0 1.5 2.0 2.5 3.0

y

Replace y by y~ = y 2

10000

160

2000

4

0 0

2

4

6

x

8

10

y = exp(x)

0

y=x

0

20

5000

2

x y=

0

2

4

6

8

10

0

2

x

4

6

8

10

x

Figure 5.13 The above panels show some alternative response curves. The formula for y˜ gives the power family transformation of y that will make y˜ a linear function of x. Thus, if y = log(x), then the transformation y˜ = exp(y) will make y˜ a linear function of x.

the explanatory variable. Fitting a line (or a curve) allows interpolation between successive levels of the explanatory variable. It may be reasonable to hazard prediction a small distance beyond the range of the data. The pattern of response may give scientific insight.

5.6 Logarithmic and other transformations 5.6.1∗ A note on power transformations

Among the more common transformations for continuous data are: r Logarithmic. This is often the right transformation for size measurements (linear, surface, volume or weight) of biological organisms. Some data may be too skewed even for a logarithmic transformation. For example, counts of insects on leaves may have this character. r Square root or cube root. These are milder than the logarithmic transformation. If linear measurements on insects are normally distributed, then we might expect the cube root of weight to be approximately normally distributed. The square root is useful for data for counts of “rare events”. The power transformation generalizes the transformations that we have just discussed. Examples of power transformations are y 2 , y 0.5 , y 3 , etc. Figure 5.13 shows a number of response curves, and describes the particular power transformation that would make the relationship linear. If the ratio of largest to smallest data value is greater than 10, and especially if it is more than 100, then the logarithmic transformation should be tried as a matter of course. Check this advice against the response curves shown in Figure 5.13.

5.6 Logarithmic and other transformations

161

log y = 1.2 [SE 0.211 ] + 1.126 [ 0.05 ] log x

400 700

Heart weight (g, log scale)

● ● ● ●● ● ● ● ● ●

200 100

●

●

● ●

●

●● ● ● ● ● ● ● ● ●

● ● ●

●

●

20 40 60 100 160 Body weight (kg, log scale)

Figure 5.14

Heart weight versus body weight, for 30 Cape fur seals.

We have so far mentioned only transformation of y. We might alternatively transform x, or transform both x and y. ∗

General power transformations

For λ = 0, the power transformation replaces a value y by y λ . The logarithmic transformation corresponds to λ = 0. In order to make this connection, a location and scale correction is needed. The transformation is then yλ − 1 , λ y(λ) = log(y),

y(λ) =

if λ = 0, if λ = 0.

r If the small values of a variable need to be spread, make λ smaller. r If the large values of a variable need to be spread, make λ larger. This is called the Box–Cox transformation, as proposed in Box and Cox (1964). The function boxcox() (MASS), whose syntax is similar to that of lm(), can be used to obtain data-driven estimates of λ. An exercise at the end of the chapter pursues investigation of boxcox().

5.6.2 Size and shape data – allometric growth

The logarithmic transformation is commonly important for morphometric data, i.e., for data on the size and shape of organisms. Figure 5.14 uses logarithmic scales to plot heart weight against body weight, for 30 seals that had been snared in trawl nets as an unintended consequence of commercial fishing (Stewardson et al., 1999). For each animal, the data provide information at just one point in time, when they died. The data thus have limited usefulness for the study of growth profiles through time. At best, if conditions have not changed too much over the lifetimes of the animals in the sample, the data may provide an indication of the average of the population growth profiles. If, e.g., sample ages range from 1 to 10 years, it is pertinent to ask how food availability may have

162

Regression with a single predictor

changed over the past 10 years, and whether this may have had differential effects on the different ages of animal in the sample.

The allometric growth equation

The allometric growth equation is y = ax b where x may, e.g., be body weight and y heart weight. It may alternatively be written log y = log a + b log x, i.e., Y = A + bX, where Y = log y, A = log a,

and X = log x.

Thus, we have an equation that can be fitted by linear regression methods, allowing prediction of values of Y given a value for X. If b = 1, then the two organs (e.g., heart and body weight) grow at the same rate. Here is the R output for the calculations that fit the regression line in Figure 5.14: > summary(cfseal.lm <- lm(log(heart) ˜ log(weight), data=cfseal)) . . . . Coefficients: Estimate Std. Error t value Pr(>|t|) (Intercept) 1.2043 0.2113 5.7 4.1e-06 log(weight) 1.1261 0.0547 20.6 < 2e-16 Residual standard error: 0.18 on 28 degrees of freedom Multiple R-Squared: 0.938, Adjusted R-squared: 0.936 F-statistic: 424 on 1 and 28 DF, p-value: <2e-16

Note that the estimate of the exponent b (= 1.126) differs from 1.0 by 2.3 (= 0.126/0.0547) times its standard error. Thus for these data, the relative rate of increase seems slightly greater for heart weight than for body weight. We have no interest in the comparison between b and zero, for which the t-statistic and p-value in the regression output are appropriate (authors sometimes present p-values that focus on the comparison with zero, even though their interest is in the comparison with 1.0. See Table 10 and other similar tables in Gihr and Pilleri (1969, p. 43)). For an elementary discussion of allometric growth, see Schmidt-Nielsen (1984). 5.7 There are two regression lines! At this point, we note that there are two regression lines – a regression line for y on x, and a regression line for x on y. It makes a difference which is the explanatory variable, and

5.8 The model matrix in regression

●

●

● ● ● ● ●

60

●

● ● ● ● ● ● ● ● ● ● ●●●● ● ● ● ●● ●●● ● ●● ● ●● ●●● ●● ●● ● ●● ●● ●● ● ● ●●●● ● ● ● ●● ●● ● ● ●●●●● ● ●●● ● ● ● ● ● ●● ● ● ● ● ●● ●● ● ● ●●●●● ● ● ● ●● ●● ● ● ● ● ● ● ●● ● ●● ● ● ●● ● ● ● ● ● ● ●● ●●●● ● ● ● ● ●● ● ● ● ●● ● ●● ● ● ● ●● ● ● ●● ●● ● ● ● ● ●●● ● ●● ●● ● ●●● ●● ● ● ● ●● ● ● ● ● ● ●●● ● ●● ● ●● ● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ● ●●●● ● ● ● ●●● ● ●● ●●● ●●● ● ● ● ●●● ● ●● ●● ● ●● ●

●

● ● ● ● ● ●

● ● ● ●●

●

230

r = 0.69

20

Leaf length (mm)

270

290

B ●

r = 0.93

250

Stretch (heated band)

A

163

●

250

270

0.5

Stretch (band held at ambient)

10

40

Petiole length (mm)

Figure 5.15 Each plot shows both the regression line for y on x (solid line), and the regression line for x on y (dotted line). In panel A the lines are quite similar, while in panel B where the correlation is smaller, the lines are quite different. Plot A is for the data of Table 3.1, while B is for a leaf data set.

which the dependent variable. The two lines are quite different if the correlation is small. Figure 5.15 illustrates the point for two other data sets.

An alternative to a regression line

There are yet other possibilities. A perspective that makes good sense for the seal organ growth data is that there is an underlying linear functional relationship. The analysis assumes that observed values of log(organ weight) and log(body weight) differ from the values for this underlying functional relationship by independent random amounts. The line that is obtained will lie between the regression line for y on x and the line for x on y. See Sprent (1966). Exercise 12 demonstrates a method for finding such a line.

5.8 The model matrix in regression For many of the uses of the lm() function in later chapters, it will be important to understand the use of the model matrix to structure calculations for practical computation. This is especially true for Chapter 7. Straight line regression is a simple context in which to introduce these ideas. In straight line regression, the model or X matrix has two columns – a column of 1s and a column that holds values of the explanatory variable x. As fitted, the straight line model is  y = a + bx which we can write as  y = 1 × a + x × b. For an example, we return to the lawn roller data. The model matrix, with the y-vector alongside, is given in Table 5.3. To obtain the model matrix, specify: model.matrix(roller.lm)

164

Regression with a single predictor

Table 5.3 The model matrix, for the lawn roller data, with the vector of observed values in the column to the right. X

y

weight (t)

depression (mm)

1 1 1 1 1 1 1 1 1 1

1.9 3.1 3.3 4.8 5.3 6.1 6.4 7.6 9.8 12.4

2 1 5 5 20 20 23 10 30 25

Table 5.4 The use of the model matrix for calculation of fitted values and residuals, in fitting a straight line to the lawn roller data. Model matrix × 2.67 × −2.09

weight

Multiply and add to yield fitted value  y

Compare with observed y

Residual = y − y

1 1 1 1 1 1 1 1 1 1

1.9 3.1 3.3 4.8 5.3 6.1 6.4 7.6 9.8 12.4

−2.1 + 2.67 × 1.9 = 2.98 −2.1 + 2.67 × 3.1 = 6.18 −2.1 + 2.67 × 3.3 = 6.71 −2.1 + 2.67 × 4.8 = 10.71 −2.1 + 2.67 × 5.3 = 12.05 −2.1 + 2.67 × 6.1 = 14.18 −2.1 + 2.67 × 6.4 = 14.98 −2.1 + 2.67 × 7.6 = 18.18 −2.1 + 2.67 × 9.8 = 24.05 −2.1 + 2.67 × 12.4 = 30.98

2 1 5 5 20 20 23 10 30 25

2 − 2.98 1 − 6.18 5 − 6.71 5 − 10.71 20 − 12.05 20 − 14.18 23 − 14.98 10 − 18.18 30 − 24.05 25 − 30.98

For each row, we take some multiple of the value in the first column, another multiple of the value in the second column, and add them. Table 5.4 shows how calculations proceed given the estimates of a and b obtained earlier. Note also the simpler (no intercept) model. For this:  y = bx. In this case the model matrix has only a single column, containing the values of x.

5.9∗ Bayesian regression estimation using the MCMCpack package

165

5.9∗ Bayesian regression estimation using the MCMCpack package Subsection 4.8.2 discussed ideas of Bayesian estimation, drawing attention to the use of the Markov Chain Monte Carlo (MCMC) simulation technique to generate successive parameter estimates. The simulation process must be allowed to burn in, i.e., run for long enough that the posterior distribution reaches a steady state that is independent of the starting values of parameters. The MCMCpack package has the function MCMCregress(), with a similar syntax to lm(), that can be used for regression calculations. The following is intended as a straightforward demonstration of the methodology, albeit for an example where use of the function lm() might in practice be preferable. The default is to assume independent uniform priors for the regression coefficients, to allow the simulation to run for 10 000 iterations, and to take the first 1000 iterations as burn-in. Here is the code and accompanying output, for the roller data: > library(MCMCpack) > roller.mcmc <- MCMCregress(depression ˜ weight, data=roller) > summary(roller.mcmc) Iterations = 1001:11000 Thinning interval = 1 Number of chains = 1 Sample size per chain = 10000 1. Empirical mean and standard deviation for each variable, plus standard error of the mean: Mean SD Naive SE Time-series SE (Intercept) -2.00 5.486 0.05486 0.05422 weight 2.65 0.812 0.00812 0.00843 sigma2 60.47 40.610 0.40610 0.57218 2. Quantiles for each variable: 2.5% 25% 50% 75% 97.5% (Intercept) -12.80 -5.38 -1.99 1.29 9.25 weight 1.01 2.17 2.66 3.16 4.26 sigma2 21.01 35.40 49.39 71.42 166.23

Because estimates from the previous iteration are the starting values for the current iteration, the sequence of estimates is Markovian and there is a lag 1 partial autocorrelation. The time series SE in the final column is designed to adjust for this autocorrelation. (See Section 9.1 for the relevant time series concepts. Specifically, it is assumed that the sequence of estimates follows an autoregressive process of order 1.) The standard error is inflated to take account of the correlation between successive estimates. Notice that the coefficient estimates are very similar to those obtained in Subsection 5.1.1 using lm(), while the SEs (both sets) are slightly larger.

166

Regression with a single predictor

Figure 5.16

Diagnostic plots for the Bayesian analysis that used MCMCregress().

There is a plot method for objects of class mcmc that allows a check on whether an adequate number of iterations were allowed for burn-in. Figure 5.16 shows this information. The layout has been changed somewhat from the default. The code is: mat <- matrix(c(1:6), byrow=TRUE, ncol=2) # panels are 1, then 2, ... 6. Layout=dim(mat), i.e., 3 by 2 layout(mat, widths=rep(c(2,1),3), heights=rep(1,6)) # NB: widths & heights are relative plot(roller.mcmc, auto.layout=FALSE, ask=FALSE, col="gray40") # The method is plot.mcmc()

These plots are unremarkable. For this very simple model, burn-in occurs quickly, and none of the plots show any indication of a trend. The posterior distributions of the model coefficients all look plausibly normal. The coda package, on which MCMCpack depends, has several other functions that give diagnostic information that may be helpful in interpreting the MCMC results. See help(package="coda"). 5.10 Recap In exploring the relationships in bivariate data, the correlation coefficient can be a crude and unduly simplistic summary measure. Keep in mind that it measures linear association. Wherever possible, use the richer and more insightful regression framework.

5.12 Exercises

167

The model matrix, together with the coefficients, allows calculation of predicted values. The coefficients give the values by which the values in the respective columns must be multiplied. These are then summed over all columns. In later chapters, we will use the model matrix formulation to fit models that are not inherently linear. In the study of regression relationships, there are many more possibilities than regression lines. If a line is adequate, use that. It is in any case useful to fit a smooth curve, to see whether it suggests systematic departure from a line. Simple alternatives to straight line regression using the original data are: r Transform x and/or y. r Use polynomial regression. r Fit a smoothing curve. Following the calculations: r Plot residuals against fitted values. r If it seems necessary, do a plot that checks homogeneity of variance. Use of the function plot(), with an lm model, gives both these plots, by default as the first and third plots (cf. panels A and C in Figure 5.6.) For size and shape data, the equation that assumes allometric variation is a good starting point. Relationships between the logarithms of the size variables are linear. The line for the regression of y on x is different from the line for the regression of x on y. The difference between the two lines is most marked when the correlation is small.

5.11 Methodological references We refer the reader to the suggestions for further reading at the end of Chapter 6.

5.12 Exercises 1. The data sets elastic1 and elastic2 were obtained using the same apparatus, including the same rubber band, as the data frame elasticband. Using a different symbol and/or a different color, plot the data from the two data frames elastic1 and elastic2 on the same graph. Do the two sets of results appear consistent? 2. For each of the data sets elastic1 and elastic2, determine the regression of stretch on distance. In each case determine (i) fitted values and standard errors of fitted values and (ii) the R 2 statistic. Compare the two sets of results. What is the key difference between the two sets of data? Use the robust regression function rlm() from the MASS package to fit lines to the data in elastic1 and elastic2. Compare the results with those from use of lm(). Compare regression coefficients, standard errors of coefficients, and plots of residuals against fitted values. 3. Using the data frame cars (datasets), plot distance (i.e., stopping distance) versus speed. Fit a line to this relationship, and plot the line. Then try fitting and plotting a quadratic curve. Does the quadratic curve give a useful improvement to the fit? [Readers who have studied the

168

Regression with a single predictor relevant physics might develop a model for the change in stopping distance with speed, and check the data against this model.]

4. Calculate volumes (volume) and page areas (area) for the books on which information is given in the data frame oddbooks (DAAG). (a) (b) (c) (d)

Plot log(weight) against log(volume), and fit a regression line. Plot log(weight) against log(area), and again fit a regression line. Which of the lines (a) and (b) gives the better fit? Repeat (a) and (b), now with log(density) in place of log(weight) as the dependent variable. Comment on how results from these regressions may help explain the results obtained in (a) and (b).

5. In the data set pressure (datasets), examine the dependence of pressure on temperature. [The relevant theory is that associated with the Claudius–Clapeyron equation, by which the logarithm of the vapor pressure is approximately inversely proportional to the absolute temperature. For further details of the Claudius–Clapeyron equation, search on the internet, or look in a suitable reference text.] 6.∗ Look up the help page for the function boxcox() from the MASS package, and use this function to determine a transformation for use in connection with Exercise 5. Examine diagnostics for the regression fit that results following this transformation. In particular, examine the plot of residuals against temperature. Comment on the plot. What are its implications for further investigation of these data? 7. Annotate the code that gives panels B and D of Figure 5.3, explaining what each function does, and what the function arguments are. 8. The following is a simplified version of the code used for the two panels of Figure 5.4: ## requires the data frame ironslag (DAAG) xy <- with(ironslag, lowess(chemical ˜ magnetic)) chemfit <- approx(xy$x, xy$y, xout=ironslag$magnetic, ties= "ordered")$y res2 <- with(ironslag, chemical - chemfit) plot(res2 ˜ magnetic, data=ironslag) # Panel A abline(v=0, lty=2) sqrtabs2 <- sqrt(abs(res2)) plot(sqrtabs2 ˜ chemfit, type="n") # Panel B panel.smooth(chemfit, sqrtabs2) Examine the help page for lowess(), and explain why the call to approx() is needed. 9. In the data frame nswdemo (DAAG), plot 1978 income (re78) against 1975 income (re75). What features of the plot make the fitting of a regression relationship a challenge? (a) Restricting attention to observations for which both re78 and re75 are non-zero, plot log(re78) against log(re75), and fit a trend curve. Additionally, fit a regression line to the plot. Does the regression line accurately describe the relationship. In what respects is it deficient? (b) Now examine the diagnostic plot that is obtained by using plot() with the regression object as parameter. What further light does this shed on the regression line model? 10. Write a function which simulates simple linear regression data from the model y = 2 + 3x + ε where the noise terms are independent normal random variables with mean 0 and variance 1.

5.12 Exercises

169

Using the function, simulate two samples of size 10. Consider two designs: first, assume that the x-values are independent uniform variates on the interval [−1, 1]; second, assume that half of the x-values are −1s, and the remainder are 1s. In each case, compute slope estimates, standard error estimates, and estimates of the noise standard deviation. What are the advantages and disadvantages of each type of design? 11. For each of the data sets elastic1 and elastic2, simulate artificial data from the model that was fitted. (a) Thus, after fitting the elastic1 data using e1.lm <- lm(distance ˜ stretch, data=elastic1) simulate artificial data from this model (conditional on the stretch measurements) using elastic1$newdistance
# slope - functional eqn line

} ## Try the following: funRel(scale=c(1,1))

# Take x and y errors as equally important

funRel(scale=c(1,10))

# Error is mostly in y; structural relation

funRel(scale=c(10,1))

# line is close to regression line of y on x # Error is mostly in x ...

## Note that all lines pass through (xbar, ybar)

(a) Note where, for each of the three settings of the argument scale, the values of the −1 , where by.x is the slope of the functional coefficient lie in the range between by.x and bx.y regression line of y on x and bx.y is the slope of the regression line of x on y. (b) Repeat this for each of the data frames softbacks and elastic2 and (with the variables logpet and loglen) leafshape17. (c) Explain the effect of changing the settings of the argument scale.

6

Multiple linear regression

In straight line regression, a response variable y is regressed on a single explanatory variable x. Multiple linear regression generalizes this methodology to allow multiple explanatory or predictor variables. The focus may be on accurate prediction. Or it may, alternatively or additionally, be on the regression coefficients themselves. Be warned that interpreting the regression coefficients is not as straightforward as it might appear. The discussion will emphasize the use of model diagnostics, and of graphs that give insight into the model fit. Diagnostic checks are intended to assist in the tuning of models so that they perform well when used for their intended purpose, and do not give unexpected and perhaps unpleasant surprises. For example, a model fit that is unduly affected by influential outliers may give results that are less than satisfactory with the main body of the data. This is one of several common types of departure from model assumptions that diagnostic checks may bring to attention.

6.1 Basic ideas: a book weight example The book weight example has two x-variables in the regression equation. In the data shown in Figure 6.1 and printed to the right of the figure, seven books with hardback covers have been added to the eight softbacks. Code for the figure is: ## Plot weight vs volume: data frame allbacks (DAAG) plot(weight ˜ volume, data=allbacks, pch=c(16,1)[unclass(cover)]) # unclass(cover) gives the integer codes that identify levels with(allbacks, text(weight ˜ volume, labels=paste(1:15), pos=c(2,4)[unclass(cover)]))

Explanatory variables are the volume of the book ignoring the covers, and the total area of the front and back covers. We might expect that weight of book = b0 + b1 × volume + b2 × area of covers. The intercept, b0 , may not be needed. However, we will retain it for the moment. Later, we can decide whether to set it to zero. Here is the regression output: > summary(allbacks.lm <- lm(weight ˜ volume+area, data=allbacks)) . . . . Coefficients: Estimate Std. Error t value Pr(>|t|)

6.1 Basic ideas: a book weight example

2 ● 13

5●

400

600

Weight

800 1000

3 ●7 ●

6●

● 11

● 15 ●9 ● 10

● 12 ●8

500

volume (cm3 )

area (cm2 )

weight (g)

cover

885

382

800

hb

1228 412

396 0

1075 250

hb pb

1034

0

725

pb

1●

● 14

4●

171

1000

1500

1 ... 7 8 ... 15

Volume

Figure 6.1 Weight versus volume, for seven hardback and eight softback books. Filled dots are hardbacks, while open dots are softbacks. Selected data are shown to the right of the graph.

(Intercept) volume area

22.4134 0.7082 0.4684

58.4025 0.0611 0.1019

0.38 11.60 4.59

0.70786 7e-08 0.00062

Residual standard error: 77.7 on 12 degrees of freedom Multiple R-Squared: 0.928, Adjusted R-squared: 0.917 F-statistic: 77.9 on 2 and 12 DF, p-value: 1.34e-007 > ## coefficient estimates and SEs only: summary(allbacks.lm)$coef

The coefficient estimates are b0 = 22.4, b1 = 0.708, and b2 = 0.468. Standard errors and p-values are provided for each estimate. Note that the p-value for the intercept suggests that it cannot be distinguished from 0, as we guessed earlier. The p-value for volume tests b1 = 0, in the equation that has both volume and area as explanatory variables. The estimate of the noise standard deviation (the residual standard error) is 77.7. There are now 15 − 3 = 12 degrees of freedom for the residual; we start with 15 observations and estimate three parameters. In addition, there are two versions of R 2 . The output is geared towards various tests of hypotheses. The F -statistic allows an overall test of significance of the regression. The null hypothesis for this test is that all coefficients (other than the intercept) are 0. Here, we obviously reject this hypothesis and conclude that the equation does have explanatory power. The t-statistics and associated p-values should however be used informally, rather than as a basis for formal tests of significance. Even in this simple example, the output has four p-values. This may not be too bad, but what if there are six or eight p-values? There are severe problems in interpreting results from such a multiplicity of formal tests, with varying amounts of dependence between the various tests. The information on individual regression coefficients can readily be adapted to obtain a confidence interval for the coefficient. The 5% critical value for a t-statistic with 12 degrees of freedom is 2.18.1 Thus, a 95% confidence interval for volume is 0.708 ± 2.18 × 0.0611,

1

## 5% critical value; t-statistic with 12 d.f. qt(0.975, 12)

172

Multiple linear regression

i.e., it ranges from 0.575 to 0.841. As for the tests of hypotheses that were noted, these confidence intervals are not independent between parameters. A sequential analysis of variance table assesses the contribution of each predictor variable to the model in turn, assuming inclusion of previously assessed variables. It can be obtained using the anova() function. > anova(allbacks.lm) Analysis of Variance Table Response: weight Df Sum Sq Mean Sq F value Pr(>F) volume 1 812132 812132 134.7 7e-08 area 1 127328 127328 21.1 0.00062 Residuals 12 72373 6031

This table gives the contribution of volume after fitting the overall mean, then the contribution of area after fitting both the overall mean and volume. The p-value for area in the anova table must agree with that in the main regression output, since both these p-values test the contribution of area after including volume in the model. The p-values for volume will differ if there is a correlation between volume and area. Here, the correlation of volume with area is 0.0015, i.e., small.2 As a consequence, the p-values for volume are very nearly equal. Finally, note the model matrix that has been used in the least square calculations: > model.matrix(allbacks.lm) (Intercept) volume area 1 1 885 382 . . . . 7 1 1228 396 8 1 412 0 . . . . 15 1 1034 0

Predicted values are given by multiplying the first column by b0 (=22.4), the second by b1 (=0.708), the third by b2 (=0.468), and adding. 6.1.1 Omission of the intercept term

We now investigate the effect of leaving out the intercept. Here is the regression output: > allbacks.lm0 <- lm(weight ˜ -1+volume+area, data=allbacks) > summary(allbacks.lm0) . . . Coefficients: 2

## Correlation of volume with area with(allbacks, cor(volume,area))

6.1 Basic ideas: a book weight example

400

600

800 1000

Fitted values

● ● ● ● ●

●

●

●

● 11

−1

0

1

Theoretical quantiles

400

600

800 1000

3 2

1 0.5

1

1.5

●

●

● 13

●5 ●

●

0

●●

● ●

Standardized residuals

●●●

●5

−2 −1

● ●●

11 ●

1.0

3 2 1 −2

11 ●

● ●

● 13

0.5

●

Standardized residuals

●

●●

C: Scale−Location

0.0

●

5●

0

● ● ● ● ●

● ●

0

●

13 ●

−1

●5

B: Normal Q−Q Standardized residuals

100

● 13

●

−100

Residuals

200

A: Resids vs Fitted

173

● ●

● ● ● ● ●● ●

●

Cook's distance11 ● 0.00

Fitted values

0.10

0.20

0.5 1

0.30

Leverage

Figure 6.2 Diagnostic plots for the model that fits weight as a function of volume and area, omitting the intercept.

volume area

Estimate Std. Error t value Pr(>|t|) 0.7289 0.0277 26.34 1.1e-12 0.4809 0.0934 5.15 0.00019

Residual standard error: 75.1 on 13 degrees of freedom Multiple R-Squared: 0.991, Adjusted R-squared: 0.99 F-statistic: 748 on 2 and 13 DF, p-value: 3.8e-014

The regression coefficients now have smaller standard errors. The reason is that, in the model that included the intercept, there was a substantial negative correlation between the estimate of the intercept and the coefficient estimates. The reduction in standard error is greater for the coefficient of volume, where the correlation was −0.88, than for area, where the correlation was −0.32. Correlations between estimates can be obtained by setting corr=TRUE in the call to summary(): ## Display correlations between estimates of model coefficients summary(allbacks.lm, corr=TRUE)

6.1.2 Diagnostic plots

Figure 6.2 displays useful information for checking on the adequacy of the model fit to the allbacks data. It used the code: par(mfrow=c(2,2)) plot(allbacks.lm0) par(mfrow=c(1,1))

# Get all 4 plots on one page

Figure 6.2 gives the default set of diagnostic plots. By comparison with Figure 5.6D in Section 5.2, it disentangles the contributions that the residual and the leverage make to the influence. Note the large residual (panel A) for observation 13. Note also that observation 13 lies outside the 0.5 contour of Cook’s distance, well out towards the contour for a Cook’s distance of 1. Thus it is a (somewhat) influential point. The Cook’s distance measure, which was mentioned in Section 5.2, will be discussed in more detail in Subsection 6.3.1. Should we omit observation 13? The first task is to check the data, which are however correct. The book is a computing book, with a smaller height to width ratio than any of the other books. It has heavier paper, though differences in the paper may not be immediately obvious. It may be legitimate to omit it from the main analysis, but noting the omission

174

Multiple linear regression

Table 6.1 Distance (dist), height climbed (climb), and record times (time), for four of the 23 Northern Irish hill races. Name 1 2 3 ··· 23

Binevenagh Slieve Gullion Glenariff Mountain ··· BARF Turkey Trot

dist (mi)

climb (ft)

time (h)

timef (h)

7.5 4.2 5.9 ··· 5.7

1740 1110 1210 ··· 1430

0.86 0.47 0.70 ··· 0.71

1.06 0.62 0.89 ··· 0.94

of this one book that had a much higher weight-to-volume ratio than other books. The following omits observation 13: > allbacks.lm13 <- lm(weight ˜ -1+volume+area, data=allbacks[-13, ]) > summary(allbacks.lm13) . . . Coefficients: Estimate Std. Error t value Pr(>|t|) volume 0.6949 0.0163 42.6 1.8e-14 area 0.5539 0.0527 10.5 2.1e-07 Residual standard error: 41 on 12 degrees of freedom Multiple R-Squared: 0.997, Adjusted R-squared: 0.997 F-statistic: 2.25e+003 on 2 and 12 DF, p-value: 3.33e-016

The residual standard error is substantially smaller (41 instead of 75.1) in the absence of observation 13. Observation 11 now has a Cook’s distance that is close to 1, but does not stand out in the plot of residuals. This is about as far as it is reasonable to go in the investigation of diagnostic plots. With just 15 points, there is a risk of over-interpretation. 6.2 The interpretation of model coefficients If an aim is a scientific understanding that involves interpretation of model coefficients, then it is important to fit a model whose coefficients are open to the relevant interpretations. Different formulations of the regression model, or different models, may serve different explanatory purposes. Predictive accuracy is in any case a consideration, and is often the main interest. Three examples will demonstrate issues for the interpretation of model coefficients. The first is a data set on record times, distances, and amounts of climb for Northern Irish hill races. The second has data on mouse brain weight, litter size, and body weight. The third has data on book dimensions and weight, from a highly biased sample of books. 6.2.1 Times for Northern Irish hill races

The data set nihills (DAAG), from which Table 6.1 has selected observations, gives distances (dist), heights climbed (climb), male record times (time), and female record times (timef), for 23 Northern Irish hill races.

6.2 The interpretation of model coefficients A: Untransformed scales

B: Logarithmic scales

●

●

4

●

3

●

●

●

● ● ● ● ●● ● ● ● ● ● ● ● ●● ● ● ●

1

2

6000

● ●

climb (feet) 4000

2000 4000

20 15

5

●

10

5

●

●

● ● ●●

2000

●●● ● ● ● ● ● ●● ● ● ●● ●

●

●● ● ●●●● ●●● ● ● ● ● ● ●

●

●

● ● ● ● ●● ● ● ● ●● ● ● ● ● ● ● ●●

●

●

●

−0.5 −1.0−0.5 0.0

−1.0

●

●

●

9.0 8.0 8.5 9.0

8.5

●

8.0

●

climb (log feet)7.5

●

● ●

●

● ● ● ● ●●● ●● ● ● ●● ●● ●

7.0

6 .5 7.0 7.5

●

65

3.0 2.5

●

●

time (log hours)0.0

●

● ● ● ● ● ● ● ● ● ●● ● ● ● ● ●

2

dist (miles) 10

● ●●● ● ● ● ●●● ●●● ● ●

●

●

15

● ●

6000 8000

●

●● ● ●● ● ●● ●●● ● ●●

● ● ●●● ● ●●●●● ●● ● ● ●● ●

8000

●

2

0.5

●

●

●

●

●

●

1

0.5 1.0 1.

1.0 ●

time (hours)

1.5

●

4

3

● ●● ● ● ● ●●● ● ● ● ● ● ●●●● ●

175

●

2.0 2.5 3.0 ●

dist 2.0 (log miles)

2.0

1.5 1.0 1.5 2.0 1.0

● ●● ● ●● ●●● ●● ● ● ● ● ●

● ● ●

●

●

●

●

● ● ● ● ● ● ●● ● ●● ●● ●● ● ● ● ● ●

Figure 6.3 Scatterplot matrix for the nihills data (Table 6.1). Panel A uses the untransformed scales, while panel B uses logarithmic scales.

We begin with scatterplot matrices, both for the untransformed data (Figure 6.3A), and for the log transformed data (Figure 6.3B). Attention is limited to the male results.3 The diagonal panels give the x-variable names for all plots in the column above or below, and the y-variable names for all plots in the row to the left or right. Note that the vertical axis labels alternate between the axis on the extreme left and the axis on the extreme right, and similarly for the horizontal axis labels. This avoids a proliferation of axis labels on the extreme left and lower axes. Apart from a possible outlier, the relationship between dist and climb seems approximately linear. The same is true when logarithmic scales are used, and the outlier is now much less evident. Nonlinear relationships are undesirable, in part, because they create problems for the interpretation of diagnostic plots. The following are reasons for investigating the taking of logarithms: r The range of values of time is large (3.9:0.32, i.e., >10:1), and similarly for dist and climb. The times are bunched up towards zero, with a long tail. In such instances, use of a logarithmic transformation is likely to lead to a more symmetric distribution. r One point in particular has a time that is more than twice that of the next largest time, as is evident in Figure 6.3A. The values of dist and climb similarly stand out as much larger than for other points. In a regression that uses the untransformed variables, this point will have a much greater say in determining the regression equation than any

3

## : data frame nihills (DAAG) ## Panel A: Scatterplot matrix, untransformed data, data frame nihills (DAAG) library(lattice); library(DAAG) splom(˜ nihills[, c("dist","climb","time")], cex.labels=1.2, varnames=c("dist\n(miles)","climb\n(feet)", "time\n(hours)")) ## Panel B: log transformed data splom(˜ log(nihills[, c("dist","climb","time")]), cex.labels=1.2, varnames=c("dist\n(log miles)", "climb\n(log feet)", "time\n(log hours)"))

Multiple linear regression

● Meelbeg

−1.0

0.0 0.5 1.0

−2

Fitted values

Figure 6.4

−1

Meelmore

0

●

● ●

1

2

Theoretical quantiles

−1.0

0.0 0.5 1.0

2

● Hen & Cock 1 Seven Sevens● 0.5

0

1

● ● ● ●●● ● ●● ● ●

−1

1.5 1.0

2

●● ●●● ●●● ●

Standardized residuals

−0.20

Meelbeg Meelmore ●

●

●

Meelmore

& Cock

●● ● ● ● ● ●● ● ● ● ● ● ● ● ● ●

● ●● ● ● ● ●● ●

● Meelbeg

−3

●

● Meelbeg ● Hen

Seven Sevens●

0.5

●

●

C: Scale−Location Standardized residuals

●

●● ●● ● ● ● ● Forest ●

● ● ●● ●● ●●

1

●

Hen & Cock ● Seven Sevens●

0.0

Donard

● ● ●

● ● ● ●

B: Normal Q−Q

0

●

& Cock

−1

● Hen

Standardized residuals

A: Resids vs Fitted

−2

Residuals

−0.05 0.05

0.15

176

0.5 1 Meelmore

Cook's distance 0.0

0.1

Fitted values

0.2

0.3

0.4

Leverage

Diagnostic plots for the regression of log(time) on log(dist) and log(climb).

other point. In the terminology of Subsection 6.3.1, it has large leverage. Even after taking logarithms (Figure 6.3B), its leverage remains large, but not quite so dominating. r It can be expected that time will increase more than linearly at very long times, and similarly for climb, as physiological demands on the human athlete move closer to limits of human endurance. r Such relationship as is evident between the explanatory variables (dist and climb) is more nearly linear on the logarithmic scale of Figure 6.3B. Additionally, use of a logarithmic scale may help stabilize the variance. These considerations suggest fitting the equation log(time) = a + b1 log(dist) + b2 log(climb). This is equivalent to the power relationship time = A(dist)b1 (climb)b2 , where a = log(A). Now fit the model and examine the diagnostic plots (shown in Figure 6.4): nihills.lm <- lm(log(time) ˜ log(dist) + log(climb), data = nihills) plot(nihills.lm)

The diagnostic plots do not indicate problems, apart from the moderately large residual associated with the Meelbeg Meelmore race. The estimates of the coefficients (a, b1 and b2 ) are: > summary(nihills.lm)$coef Estimate Std. Error t value Pr(>|t|) (Intercept) -4.961 0.2739 -18.1 7.09e-14 log(dist) 0.681 0.0552 12.3 8.19e-11 log(climb) 0.466 0.0453 10.3 1.98e-09 Estimate Std. Error t value Pr(>|t|)

Interpreting the coefficients

The estimated equation is log(time) =

−4.96 +0.68 × log(dist)+ 0.47 × log(climb). [SE = 0.27] [SE = 0.055] [SE = 0.045]

6.2 The interpretation of model coefficients

177

Exponentiating both sides of this equation, and noting exp(−4.96) = 0.0070, gives time = 0.00070 × dist0.68 × climb0.47 . This equation implies that for a given height of climb, the time taken is smaller for the second three miles than for the first three miles. The relative rate of increase in time is 68% of the relative rate of increase in distance. Is this plausible? The answer comes from examination of the implications of holding climb constant. For a given value of climb, short races will be steep while for long races the slope will be relatively gentle. Thus a coefficient that is less than 1.0 is unsurprising.

A meaningful coefficient for logdist

The coefficient for logdist will be more meaningful if we regress on logdist and log(climb/dist). Then we find: > > > > >

lognihills <- log(nihills) names(lognihills) <- paste("log", names(nihills), sep="") lognihills$logGrad <- with(nihills, log(climb/dist)) nihillsG.lm <- lm(logtime ˜ logdist + logGrad, data=lognihills) summary(nihillsG.lm)$coef Estimate Std. Error t value Pr(>|t|) Estimate Std. Error t value Pr(>|t|) (Intercept) -4.961 0.2739 -18.1 7.09e-14 logdist 1.147 0.0346 33.2 5.90e-19 logGrad 0.466 0.0453 10.3 1.98e-09

The coefficient of logdist is now, reassuringly, greater than 1. The coefficient of logdist depends, critically, on what other explanatory variables are used! There is another, related, benefit. The correlation between logdist and logGradient is −0.065, negligible relative to the correlation of 0.78 between logdist and logclimb.4 Because the correlation between logdist and logGradient is so small, the coefficient of logdist (=1.124) in the regression on logdist alone is almost identical to the coefficient of logdist (=1.147) in the regression on logdist and logGradient. The standard error of the coefficient of logdist is smaller – 0.035 as against 0.055 – when the second explanatory variable is logGradient rather than logclimb. Note that the predicted values do not change. The models nihills.lm nihillsG.lm are different mathematical formulations of the same underlying model.

6.2.2 Plots that show the contribution of individual terms

For simplicity, the discussion will assume just two explanatory variables, x1 and x2 , with the intention of showing the contribution of each in turn to the model. 4

## Correlations of logGrad and logclimb with logdist with(lognihills, cor(cbind(logGrad, logclimb), logdist))

● ● ● ● ● ● ● ●● ● ● ●● ●● ● ●●

1.0

●

1.5

2.0

2.5

0.5

●

●

0.0

●●

−0.5

Partial for logclimb

0.5

●

0.0 −0.5

Partial for logdist

1.0

Multiple linear regression 1.0

178

●

3.0

●● ● ●●●●●● ● ●●● ● ●●

7.0

logdist

●● ● ●

8.0

9.0

logclimb

Figure 6.5 The solid lines show the respective contributions of the two model terms, in the regression of logtime on logdist and logclimb. Partial residuals (specify partial.resid=TRUE) and an associated smooth curve (specify smooth=panel.smooth) have been added.

The fitting of a regression model makes it possible to write: y = b0 + b1 x1 + b2 x2 + e = y + e,

(6.1) (6.2)

where  y = b0 + b1 x1 + b2 x2 . Another way to write the model that is to be fitted is: y − y¯ = a + b1 (x1 − x¯1 ) + b2 (x2 − x¯2 ) + e. It is a fairly straightforward algebraic exercise to show that a = 0. Notice that, for fitting the model in this form: r The observations are y − y, ¯ with mean zero. r The first explanatory variable is x − x¯ , with mean zero, and the first term in the model 1 1 is b1 (x1 − x¯1 ), with mean zero. r The second explanatory variable is x − x¯ , with mean zero, and the first term in the 2 2 model is b1 (x1 − x¯1 ), with mean zero. The residuals e are exactly the same as before, and have mean zero. The fitted model can then be written: y = y¯ + b1 (x1 − x¯1 ) + b2 (x2 − x¯2 ) + e = y¯ + t1 + t2 + e. ¯ a term that is This neatly splits the response value y into three parts – an overall mean y, due to x1 , a term that is due to x2 , and a residual e. Moreover, the values of t1 and t2 sum, in each case, to zero. The predict() function has an option (type="terms") that gives t1 and t2 . yterms <- predict(nihills.lm, type="terms")

The first column of yterms has the values of t1 = b1 (x1 − x¯1 ), while the second has the values of t2 . Values in both these columns sum to zero. The solid lines of the component plus residual plot in Figure 6.5 show the contributions of the individual terms to the model.

6.2 The interpretation of model coefficients

179

The solid line in the left panel shows a plot of b1 (x1 − x¯1 ) against x1 , while the solid line in the right panel shows a plot of b2 (x2 − x¯2 ) against x2 . The lines can be obtained directly with the termplot() command. As the contributions of the terms are linear on a logarithmic scale, a plot in which the x-axis variables are the log transformed variables will serve the purpose best. We therefore refit the equation before using termplot(), thus: lognihills <- log(nihills) names(lognihills) <- paste("log", names(nihills), sep="") nihills.lm <- lm(logtime ˜ logdist + logclimb, data = lognihills) ## To show points, specify partial.resid=TRUE ## For a smooth curve, specify smooth=panel.smooth termplot(nihills.lm, partial.resid=TRUE, smooth=panel.smooth, col.res="gray30")

The plotted points are the partial residuals, for the respective term. r The vector t + e =  y − t2 holds the partial residuals for x1 given x2 , i.e., they account 1 for that part of the response that is not explained by the term in x2 . r The vector t + e holds the partial residuals for x given x . 2 2 1 The smooth curve that has been passed through the partial residuals is designed to help in assessing any departure from the lines. Both plots show a hint of curvature, more pronounced for logdist than logclimb. The difference from a line is however small, and may not be of much practical consequence. Each plot indicates the pattern in the residuals when there is no change to the linear pattern of response that is assumed for the other variable. Both plots suggest a slight but noticeable departure from linearity, though in the right panel largely due to a single point. Further investigation, if it should seem warranted, can best proceed using methods for fitting smooth curves that will be described in Chapter 7.

6.2.3 Mouse brain weight example

The litters data frame (DAAG library) has observations on brain weight, body weight, and litter size of 20 mice. As Figure 6.6 makes clear, the explanatory variables lsize and bodywt are strongly correlated. Code for obtaining a simplified version of Figure 6.6 is:5 pairs(litters)

# For improved labeling, see the footnote.

Observe now that, in a regression with brainwt as the dependent variable, the coefficient for lsize has a different sign (−ve versus +ve) depending on whether or not bodywt

5

## Scatterplot matrix for data frame litters (DAAG); labels as in figure library(lattice) splom(˜litters, varnames=c("lsize\n\n(litter size)", "bodywt\n\n(Body Weight)", "brainwt\n\n(Brain Weight)"))

180

Multiple linear regression

0.44 0.42

0.42

0.44

brainwt 0.40

(Brain Weight) 0.38

0.40

0.38

10 8

9 8

9

bodywt (Body Weight)

6

7

8

10

8 7 6

12 8

10

12

10

lsize 8

8

(litter size) 4

6

8

6 4

Scatter Plot Matrix

Figure 6.6

Scatterplot matrix for the litters data set. Data relate to Wainright et al. (1989).

also appears as an explanatory variable. Both coefficients are significant (p < 0.05). Here are the calculations: > ## Regression of brainwt on lsize > summary(lm(brainwt ˜ lsize, data = litters))$coef Estimate Std. Error t value Pr(>|t|) (Intercept) 0.44700 0.00962 46.44 3.39e-20 lsize -0.00403 0.00120 -3.37 3.44e-03 > ## Regression of brainwt on lsize and bodywt > summary(lm(brainwt ˜ lsize + bodywt, data = litters))$coef Estimate Std. Error t value Pr(>|t|) (Intercept) 0.17825 0.07532 2.37 0.03010 lsize 0.00669 0.00313 2.14 0.04751 bodywt 0.02431 0.00678 3.59 0.00228

The coefficients have different interpretations in the two cases: r In the first regression, variation in brainwt is being explained only with lsize, regardless of bodywt. No adjustment has been made for the fact that bodywt increases as lsize decreases: individuals having small values of lsize have brainwt values corresponding to large values of bodywt, while individuals with large values of lsize have brainwt values corresponding to low bodywt values. r In the multiple regression, the coefficient for lsize is a measure of the change in brainwt with lsize, when bodywt is held constant. For any particular value of bodywt, brainwt increases with lsize. This was a noteworthy finding for the purposes of the study. The results are consistent with the biological concept of brain sparing, whereby the nutritional deprivation that results from large litter sizes has a proportionately smaller effect on brain weight than on body weight.

6.2 The interpretation of model coefficients A

B

● ●

● ● ●

● ● ●

● ●

●

● ● ●

● ● ●

●

●

●

● ●

● ●

●

● ● ● ●

●

● ● ● ●

●

●● ● ● ●

● ●

181

●

●

●

● ● ●

●

●

log(height)

●●

log(breadth)

●●

●

●

●

●

●

● ● ●

● ● ● ●

● ● ● ● ●

●

● ●

●

●

● ●

●

● ●

●

●

●

● ● ●

● ●

●

●

●

●

●

● ●● ●

● ●

●

● ●● ● ● ●

● ● ● ●

● ● ● ● ● ●

●

● ● ●● ● ● ●

● ●

●

● ●

●● ●

● ●

● ● ●●

●● ●

● ●

● ●

log(density) ●

●

● ●

log(area)

● ●

●

log(thick) ●

● ●●

●

●

● ● ● ● ●●

●

●

log(weight) ●

●

● ● ● ●●

● ●

●

● ●

● ● ●

● ● ● ●

●

●● ●

●

●

● ●

● ● ●

● ● ● ●

●

● ● ● ● ● ●

● ● ●

● ● ● ● ●●

●

●● ●

●

● ● ●

● ●

● ●

log(thick)

log(weight)

●

● ● ● ●

● ●● ● ●

●

●

● ●

●

● ●● ● ● ● ● ●

●

●● ● ● ● ●● ● ● ●

Figure 6.7 Panel A is the scatterplot matrix for the logarithms of the variables in the oddbooks data frame. Books were selected in such a way that weight increased with decreasing thickness. Panel B has the scatterplot matrix for the logarithms of the derived variables density and area, together with log(thick) and log(weight).

6.2.4 Book dimensions, density, and book weight

The way that data are sampled can affect the coefficients. This section will examine data, sampled in a deliberately biased way, on the effect of book dimensions (thickness, height, and width) on book weight. Figure 6.7A shows a scatterplot matrix for logged measurements, from the data frame oddbooks, on 12 soft-cover books. Figure 6.7B is for later reference.6 Books were selected in such a way that weight increased with decreasing thickness, reflected in the strong negative correlation between log(weight) and log(thick). It might be expected that weight would be proportional to volume, i.e., w = tbh and
= log(t) + log(b) + log(h) log(w)

(6.3)

where w = weight, t = thick, b = breadth, and h = height. Although equation (6.3) seems plausible, there can be no guarantee that it will give a result that makes sense for data where there have been strong constraints on the choice of books. We will fit models in which the fitted values take the following forms: 1 : log(w) = a0 + a1 (log(t) + log(b) + log(h)) 2 : log(w) = a0 + a1 log(t) + a2 (log(b) + log(h)) 3 : log(w) = a0 + a1 log(t) + a2 log(b) + a3 log(h) 6

## Code for Panel A splom(˜log(oddbooks), varnames=c("thick\n\nlog(mm)", "breadth\n\nlog(cm)", "height\n\nlog(cm)", "weight\n\nlog(g)"), pscales=0) ## Code for Panel B oddothers <with(oddbooks, data.frame(density = weight/(breadth*height*thick), area = breadth*height, thick=thick, weight=weight)) splom(˜log(oddothers), pscales=0, varnames=c("log(density)", "log(area)", "log(thick)", "log(weight)"))

182

Multiple linear regression

●

●

●

● ●

●

6.5

log(weight)

● ●

●

● ●

●

8.8

9.0

9.2

●

5.6

log(thick)+log(breadth)+log(height)

●

●

5.5

5.5

●

●

●

●

8.6

●

●

6.0

●

6.5

log(weight)

●

● ●

●

5.5

●

●

6.0

6.5

● ● ●

6.0

log(weight)

●

●

7.0

C

7.0

B

7.0

A

6.0

6.4

6.8

●

5.6

Predicted log(weight)

6.0

6.4

6.8

Predicted log(weight)

Figure 6.8 Panel A plots log(weight) against log(thick) + log(breadth) + log(height). The dashed line shows the fitted values for model 1. Panel B plots observed values against fitted values for model 2, with the line y = x superposed. Panel C plots observed values against fitted values for model 3 and again shows the line y = x.

Figure 6.8 assists comparison of these models. The coefficients in the fitted equations, with SEs in square brackets underneath, are:7 1 : log(w) = −8.9 + 1.7 × (log(t) + log(b) + log(h)) [SE=2.7]

[0.31]

2 : log(w) = −1.6 + 0.48 log(t) + 1.10(log(b) + log(h)) [2.9]

[0.42]

[0.28]

3 : log(w) = −0.72 + 0.46 log(t) + 1.88 log(b) + 0.15 log(h)) [3.2]

[0.43]

[1.07]

[1.27]

Transforming back to a relationship between weight and volume, the result for model 1 is that weight ∝ volume1.7 , implying that weight increases faster than volume. As the volume of books increases, their density increases. Almost certainly, the effect has arisen because the books with larger page sizes are printed on heavier paper. Note that the predicted values in model 2 are very similar to those for model 3. The coefficient of area in model 2 indicates that, for a given value of thick, weight is very nearly proportional to page area. Note finally that the regression of log(weight) on log(thick) yields: > coef(summary(lm(log(weight) ˜ log(thick), data=oddbooks))) Estimate Std. Error t value Pr(>|t|) (Intercept) 9.69 0.708 13.7 8.35e-08 log(thick) -1.07 0.219 -4.9 6.26e-04

The implication is that weight decreases as thick increases. For these data, it does! The oddbooks data were contrived to give a skewed picture of the way that book weight varies with dimensions. Correlations between area and thick, and between both area and thick and density of paper, make it impossible to use multiple regression to separate the effects of these different variables. The one fairly solid piece of information 7

## Details of calculations volume <- apply(oddbooks[, 1:3], 1, prod) area <- apply(oddbooks[, 2:3], 1, prod) lob1.lm <- lm(log(weight) ˜ log(volume), data=oddbooks) lob2.lm <- lm(log(weight) ˜ log(thick)+log(area), data=oddbooks) lob3.lm <- lm(log(weight) ˜ log(thick)+log(breadth)+log(height), data=oddbooks) coef(summary(lob1.lm)) ## Similarly for coefficients and SEs for other models

6.3 Multiple regression assumptions, diagnostics, and efficacy measures

183

that is available from these data is obtained by using our knowledge of what the relationship should be, to indicate how density changes with area or thick, thus: > book.density <- oddbooks$weight/volume > bookDensity.lm <- lm(log(book.density) ˜ log(area), data=oddbooks) > coef(summary(bookDensity.lm)) Estimate Std. Error t value Pr(>|t|) (Intercept) -5.109 0.5514 -9.27 3.18e-06 log(area) 0.419 0.0958 4.37 1.39e-03

Observational data is very susceptible to such bias. For example solar radiation, windspeed, temperature, and rainfall may change systematically with distance up a hillside, and it may be impossible to distinguish the effects of the different factors on plant growth or on the ecology. Worse, effects may be at work that will be discerned only from substantial understanding of the physical processes and which are not obvious from the measured data. 6.3 Multiple regression assumptions, diagnostics, and efficacy measures At this point, as a preliminary to setting out a general strategy for fitting multiple regression models, we describe the assumptions that underpin multiple regression modeling. Given the explanatory variables x1 , x2 , . . . , xp , the assumptions are that: r The expectation E[y] is some linear combination of x , x ,. . . , x : 1 2 p E[y] = α + β1 x1 + β2 x2 + · · · + βp xp . r The distribution of y is normal with mean E[y] and constant variance, independently between observations. In general, the assumption that E[y] is a linear combination of the xs is likely to be false. It may, however, be a good approximation. In addition, there are simple checks that, if the assumption fails, may indicate the nature of the failure. The assumption may be hard to fault when the range of variation of each explanatory variable is small relative to the noise component of the variation in y, so that any non-linearity in the effects of explanatory variables is unlikely to show up. Even where there are indications that it is not entirely adequate, a simple form of multiple regression model may be a reasonable starting point for analysis, on which we can then try to improve. 6.3.1 Outliers, leverage, influence, and Cook’s distance

This extends earlier discussions of regression diagnostics in Section 5.2 and Subsection 6.1.2. Detection of outliers

Outliers can be hard to detect. Two (or more) outliers that are influential may mask each other. If this seems a possible issue, it is best to work with residuals from a resistant fit. Resistant fits aim to completely ignore the effect of outliers. Use of this methodology will be demonstrated in Subsection 6.4.3. See also Exercise 14 at the end of the chapter.

184

Multiple linear regression Residuals vs Leverage 2

● Hen & Cock

1

log (clim mb) e Annalong Horseshoe

1

● ●

● ●●

●

0.5

● ●

0

Seven S Sevens

Standardized residuals

Seven Sevens ● ●

● ●

● ● ● ● ● ●

● ●

●

0.5 1 ● Meelbeg Meelmore

log (dist)

Cook's distance log (time)

0.0

0.1

0.2

0.3

0.4

Leverage

Figure 6.9 The left panel is a snapshot of a three-dimensional dynamic graphic plot. The two points that have the largest leverage in the regression of logtime on logdist and logclimb have been labeled. In the right panel, standardized residuals are plotted against leverages. Contours are shown for Cook’s distances of 0.5 and 1.0.

This demonstrates the type of aberrant results that may result from resistant regression, if residuals do not have an approximately symmetric distribution. Dynamic graphic exploration can be helpful. Abilities in the rgl package for dynamic three-dimensional plots can be accessed conveniently from the graphics menu of the R Commander GUI. The rgobi package (Cook and Weisberg, 1999) has very extensive abilities for dynamic graphic exploration. ∗

Leverage and the hat matrix

What difference does replacing of yi by yi + i , while leaving other y-values unchanged, make to the fitted surface. There is a straightforward answer; the fitted value changes from yi to yi + hii i , where hii is the leverage for that point. The leverage values hii are the diagonal elements of the so-called hat matrix that can be derived from the model matrix. (The name of the hat matrix H is due to the fact that the vector of fitted values (“hat” values) is related to the observed response vector by  y = Hy.) Large values represent high leverage. Use the function hatvalues() to obtain the leverages, thus: > as.vector(hatvalues(nihills.lm)) # as.vector() strips off names [1] 0.119 0.074 0.134 0.109 0.085 0.146 0.052 0.146 0.247 0.218 [11] 0.090 0.057 0.056 0.127 0.049 0.103 0.175 0.214 0.444 0.090 [21] 0.139 0.048 0.077

The largest leverage, for observation 19, is 0.44. As this is more than three times the average value of 0.13, it warrants attention. (There are p = 3 coefficients and n = 23 observations, so that the average is p/n = 0.13.) The left panel of Figure 6.9 is a snapshot of a three-dimensional dynamic graphic plot that shows the regression plane in the regression of logtime on logdist and logclimb.

6.3 Multiple regression assumptions, diagnostics, and efficacy measures

185

The two points that have the largest leverage have been labeled. The right panel is a plot of residuals against leverage values, in which the Cook’s distances are shown as contours. The snapshot in the left panel can be obtained from the 3D graph submenu on the R Commander’s Graphs pulldown menu, then rotating the graph to give the view shown. Alternatively, enter from the command line the code that is given in Section 15.7. To rotate the display, hold down the left mouse button and move the mouse. Try it! The snapshot is a poor substitute for the experience of rotating the display on a computer screen! The right panel can be obtained thus: ## Residuals versus leverages plot(nihills.lm, which=5, add.smooth=FALSE) ## The points can alternatively be plotted using ## plot(hatvalues(model.matrix(nihills.lm)), residuals(nihills.lm))

Influential points and Cook’s distance

Data points that distort the fitted response are “influential”. Such distortion is a combined effect of the size of the residual, and its leverage. The Cook’s distance statistic is a commonly used measure of “influence”. It measures, for each observation, the change in model estimates when that observation is omitted. It measures the combined effect of leverage and of the magnitude of the residual. Recall the guideline that was given in Section 5.2, that any Cook’s distance of 1.0 or more, or that is substantially larger than other Cook’s distances, should be noted. Any serious distortion of the fitted response may lead to residuals that are hard to interpret or even misleading. Thus it is wise to check the effect of removing any highly influential data points before proceeding far with the analysis. Influence on the regression coefficients

In addition to the diagnostic plots, it is useful to investigate the effect of each observation on the estimated regression coefficients. One approach is to calculate the difference in the coefficient estimates obtained with and without each observation. The function dfbetas() does these calculations and standardizes the resulting differences, i.e., they are divided by a standard error estimate. These are more readily interpretable than absolute differences. For the regression model fit to the allbacks data set, without the intercept term, the values are shown in Figure 6.10. If the distributional assumptions are satisfied, standardized changes that are larger than 2 can be expected, for a specified coefficient, in about 1 observation in 20. Here, the only change that seems worthy of note is for volume in observation 13. For absolute changes, should they be required, use the function lm.influence().

Outliers, influential or not, should be taken seriously

Outliers, influential or not, should never be disregarded. Careful scrutiny of the original data may reveal an error in data entry that can be corrected. Alternatively, their exclusion may be a result of use of the wrong model, so that with the right model they can be re-incorporated.

186

Multiple linear regression area

|

| |||||| ||| | |

|

13

volume

|

||| |||| ||

| 13

0

1

2

Standardized change in coefficient dfbetas(allbacks.lm0)

Figure 6.10 Standardized changes in regression coefficients, for the model that was fitted to the allbacks data set. The points for the one row (row 13) where the change for one of the coefficients was greater than 2 in absolute value are labeled with the row number.

If apparently genuine outliers remain excluded from the final fitted model, they must be noted in the eventual report or paper. They should be included, separately identified, in graphs. ∗

Additional diagnostic plots

The functions in the car package, designed to accompany Fox (2002), greatly extend the range of diagnostic plots. See the examples and references included on the help pages for this package. As an indication of what is available, try library(car) leverage.plots(allbacks.lm, term.name="volume", identify.points=FALSE)

6.3.2 Assessment and comparison of regression models

The measures that will be discussed all focus on predictive accuracy, assuming that data used for fitting the model can be treated as a random sample from the data that will be used in making predictions. This assumption may be incorrect. Also, as pointed out earlier, predictive accuracy may not be the only or the most important consideration. It is nonetheless always an important consideration, even where interpretation of model parameters is the primary focus of interest. R 2 and adjusted R 2

The R 2 and adjusted R 2 that are included in the default output from summary.lm() (cf. Section 6.1) are commonly used to give a rough sense of the adequacy of a model. R 2 is the proportion of the sum of squares about the mean that is explained by the model. Adjusted R 2 , which is the proportion of the mean sum of squares that is explained, is preferable to R 2 . Neither is appropriate for comparisons between different studies, where the ranges of values of the explanatory variables may be different. Both are likely to be largest in those studies where the range of values of the explanatory variables is greatest. The R 2 statistic (whether or not adjusted) has a more legitimate use for comparing different models for the same data. For this purpose, however, AIC and related statistics are much preferable. The next subsection will discuss these “information measure” statistics.

6.3 Multiple regression assumptions, diagnostics, and efficacy measures

187

AIC and related statistics

These statistics are designed to choose, from among a small number of alternatives, the model with the best predictive power. Statistics that are in use include the Akaike Information Criterion (AIC), Mallows’ Cp , and various elaborations of the AIC. The AIC criterion and related statistics can be used for more general models than multiple regression. We give the version that is used in R. The model that gives the smallest value is preferred, whether for the AIC or for the Cp statistic. The variance estimate  σ 2 is often determined from the full model. Another statistic is the Bayesian Information Criterion (BIC), which is claimed as an improvement on the AIC. The AIC is inclined to over-fit, i.e., to choose too complex a model.

Calculation of R’s version of the AIC

Let n be the number of observations, let p be the number of parameters that have been fitted, and let RSS denote the residual sum of squares. Then, if the variance is known, R takes the AIC statistic as AIC =

RSS + 2p + const. σ2

where, here, the constant term arises from the assumption of an i.i.d. normal distribution for the errors. In the more usual situation where the variance is not known, R takes the AIC statistic as   RSS + 2p + const. AIC = n log n The BIC (Schwarz’s Bayesian criterion) statistic, which replaces 2p by log(n) × p, penalizes models with many parameters more strongly. The Cp statistic differs from the AIC statistic only by subtraction of n, and by omission of the constant term. It is   RSS Cp = n log + 2p − n. n 6.3.3 How accurately does the equation predict?

The best test of how well an equation predicts is the accuracy of its predictions. We can measure this theoretically, by examining, e.g., 95% confidence intervals for predicted values. Alternatively we can use cross-validation, or another resampling procedure, to assess predictive accuracy. Both methods (theoretical and cross-validation) for assessing predictive accuracy assume that the sample is randomly drawn from the population that is of interest, and that any new sample for which predictions are formed is from that same population. Earlier, we noted that a quadratic term in ldist would probably improve the model slightly. We can accommodate this by replacing logdist by poly(logdist,2), or

188

Multiple linear regression

●

● ● ● ● ● ● ● ●●

● ● ● ● ● ● ●

●

● ●

●

●

0.5

0.4 Residual

●

●

●

● ●

●

● ● ● ● ●● ● ● ●●● ● ● ● ● ●● ●● ● ● ● ● ● ● ● ● ●

● ●

●

● ●

●

1

0.0

1.1

●

1

B: Untransformed scales (time)

● ● ● ● ● ● ● ● ●

●

0.9

Residual/(fitted value)

A: Logarithmic scales

1.5

Observed time

2.5

4

0.5

1.5

2.5

3.5

Observed time

Figure 6.11 Residuals versus predicted values for the hill race data. The vertical limits of the bounding curves are 95% pointwise confidence limits, with the predicted value subtracted off. Black points and 95% curves are for the model that is linear in logdist, Gray points and 95% curves are for the model that is quadratic in logdist. Panel A uses logarithmic scales, with the points labeled on the untransformed scale.

by poly(logdist,2,raw=TRUE) if coefficients are required that are immediately interpretable. Orthogonal polynomials will be discussed in Section 7.4. The following table gives 95% coverage (confidence) intervals for predicted times in the regression of log(time) on log(climb) and log(dist), for the first few observations from the data set nihills: > lognihills <- log(nihills) > names(lognihills) <- paste("log", names(nihills), sep="") > nihills.lm <- lm(logtime ˜ logdist + logclimb, data = lognihills) ## Coverage intervals; use exp() to undo the log transformation > exp(predict(nihills.lm, interval="confidence"))[1:5, ] fit lwr upr Binevenagh 0.893 0.845 0.944 Slieve Gullion 0.488 0.467 0.510 Glenariff Mountain 0.640 0.604 0.679 Donard & Commedagh 1.126 1.068 1.187 McVeigh Classic 0.570 0.544 0.597

Prediction intervals from the model that replaces logGrad by logclimb will be identical. Figure 6.11 shows these prediction intervals graphically. Panel A plots the points on a scale of time, while panel B plots the points on a scale of log(time). The 95% pointwise intervals apply to the fitted values, that is, to the values on the vertical axis. Fitted values and 95% confidence bounds are shown both for the model that is linear in log(dist) and for the model that is quadratic in log(dist).8 Notice that the bounds for the quadratic model are narrower within the main body of the data, but fan out as time increases. The quadratic bounds better indicate the uncertainty in the predictions for large times. Note the trade-off between bias for the model that lacks the quadratic term, and increased variance for the model that includes the quadratic term. 8

## Model that is quadratic in logdist nihills2.lm <- lm(logtime ˜ poly(logdist,2) + logclimb, data = lognihills) citimes2 <- exp(predict(nihills2.lm, interval="confidence"))

6.4 A strategy for fitting multiple regression models

189

This assessment of predictive accuracy has important limitations: 1. It is crucially dependent on the model assumptions – independence of the data points, homogeneity of variance, and normality. If the theoretical assumptions are wrong, perhaps because of clustering or other forms of dependence, then these prediction intervals will also be wrong. Departures from normality are commonly of less consequence than the other assumptions. 2. It applies only to the population from which these data have been sampled. If the sample for which predictions are made is really from a different population, or is drawn with a different sampling bias, the assessments of predictive accuracy will be wrong. Thus it might be hazardous to use the above model to predict winning times for hill races in England or Mexico or Tasmania. Point 2 can be addressed only by testing the model against data from these other locations. Thus, it is interesting to compare the results from the Scottish hills2000 data with results from the Northern Irish nihills data. The results are broadly comparable. We leave further investigation of this comparison to the exercises. Better still, where this is possible, is to use data from multiple locations to develop a model that allows for variation between locations as well as variation within locations. Chapter 10 discusses models that are, in principle, suitable for such applications. We might consider cross-validation, or the bootstrap, or another resampling method. Cross-validation and other resampling methods can cope, to a limited extent and depending on how they are used, with lack of independence. Note that neither this nor any other such internal method can address sampling bias. Heterogeneity is just as much an issue as for model-based assessments of accuracy. Thus for the hill race data, if there is no adjustment for changing variability with increasing length of race: 1. If we use the untransformed data, cross-validation will exaggerate the accuracy for long races and be slightly too pessimistic for short races. 2. If we use the log transformed data, cross-validation will exaggerate the accuracy for short races and under-rate the accuracy for long races. 6.4 A strategy for fitting multiple regression models Careful graphical scrutiny of the explanatory variables is an essential first step. This may lead to any or all of: r Transformation of some or all variables. r Replacement of existing variables by newly constructed variables that are a better summary of the information. For example, we might want to replace variables x1 and x2 by the new variables x1 + x2 and x1 − x2 . r Omission of some variables. Why are linear relationships between explanatory variables preferable?

Where there are many explanatory variables, the class of models that would result from allowing all possible transformations of explanatory variables is too wide to be a satisfactory starting point for investigation. The following are reasons for restricting attention to

190

Multiple linear regression

transformations, where available, that lead to scatterplots in which relationships between explanatory variables are approximately linear. r If relationships between explanatory variables are non-linear, diagnostic plots may be misleading. See Cook and Weisberg (1999). r Approximately linear relationships ensure that all explanatory variables have similar distributions, preferably distributions that are not asymmetric to an extent that gives the smallest or largest points undue leverage. r If relationships are linear, it is useful to check the plots of explanatory variables against the response variable for indications of the relationship with the dependent variable. Contrary to what might be expected, this is more helpful than looking at the plots of the response variable against explanatory variables. Surprisingly often, logarithmic or other standard forms of transformation give more symmetric distributions, lead to scatterplots where the relationships appear more nearly linear, and make it straightforward to identify a regression equation that has good predictive power.

6.4.1 Suggested steps

Here are steps that are reasonable to follow. They involve examination of the distributions of values of explanatory variables, and of the pairwise scatterplots. r Examine the distribution of each of the explanatory variables, and of the dependent variable. Look for any instances where distributions are highly skew, or where there are outlying values. Check whether any outlying values may be mistakes. r Examine the scatterplot matrix involving all the explanatory variables. (Including the dependent variable is, at this point, optional.) Look first for evidence of non-linearity in the plots of explanatory variables against each other. Look for values that appear as outliers in any of the pairwise scatterplots. r Note the ranges of each of the explanatory variables. Do they vary sufficiently to affect values of the dependent variable? r How accurately are each of the explanatory variables measured? At worst, the inaccuracy may be so serious that it is unlikely that any effect can be detected, and/or that coefficients of other explanatory variables are seriously in error. Section 6.7 has further details. r If some pairwise plots show evidence of non-linearity, consider use of transformation(s) to give more nearly linear relationships. r Where the distribution is skew, consider transformations that may lead to a more symmetric distribution. r Look for pairs of explanatory variables that are so highly correlated that they appear to give the same information. Do scientific considerations help in judging whether both variables should be retained? For example, the two members of the pair may measure what is essentially the same quantity. Note however that there will be instances where the difference, although small, is important. Section 8.2 has an example.

6.4 A strategy for fitting multiple regression models

Figure 6.12

191

Scatterplot matrix for the hills2000 data (Table 6.1).

6.4.2 Diagnostic checks

Checks should include: r Plot residuals against fitted values. For initial checks, consider the use of residuals from a resistant regression model. Check for patterns in the residuals, and for the fanning out (or in) of residuals as the fitted values change. (Do not plot residuals against observed values. This is potentially deceptive; there is an inevitable positive correlation.) r Examine the Cook’s distance statistics. If it seems helpful, examine standardized versions of the drop-1 coefficients directly, using dfbetas(). It may be necessary to delete influential data points and refit the model. r For each explanatory variable, construct a component plus residual plot, to check whether any of the explanatory variables require transformation. A further question is whether multiple regression methodology is adequate, or whether a non-linear form of equation may be required. Ideas of “structural dimension”, which are beyond the scope of the present text, become important. The package dr (dimension reduction) addresses such issues.

6.4.3 An example – Scottish hill race data

The data in hills2000 is comparable to nihills, but for Scotland rather than Northern Ireland. Again, we will work on the logarithmic scale and limit attention to the male results. Figure 6.12 shows the scatterplot matrix.9 Apart from a possible outlier, the relationship between dist and climb seems approximately linear on the log scale.

9

## Scatterplot matrix: data frame hills2000 (DAAG), log scales splom(˜ log(hills2000[, c("dist","climb","time")]), cex.labels=1.2, varnames=c("dist\n(log miles)", "climb\n(log feet)", "time\n(log hours)"))

192

Multiple linear regression

Figure 6.13 Plots of residuals against fitted values from the regression of log(time) on log(climb) and log(dist). Panel A is from the least squares (lm) fit, while panel B is for a resistant fit that uses lqs from the MASS package. Note that the resistant fit relies on repeated sampling of the data, and will differ slightly from one run to the next.

The help page for races2000 (hills2000 is a subset of races2000) suggests uncertainty about the distance for the Caerketton race in row 42. We will include Caerketton during our initial analysis and check whether it appears to be an outlier. Figure 6.13 shows residuals (A) from a least squares (lm) fit and (B) from a resistant lqs fit, in both cases plotted against fitted values. Resistant fits completely ignore the effects of large residuals. By default, even if almost half the observations are outliers, the effect on the fitted model would be small. See the help page for lqs for more information. The code is: ## Panel A lhills2k.lm <- lm(log(time) ˜ log(climb) + log(dist), data = hills2000) plot(lhills2k.lm, caption="", which=1) ## Panel B library(MASS) # lqs() is in the MASS package lhills2k.lqs <- lqs(log(time) ˜ log(climb) + log(dist), data = hills2000) reres <- residuals(lhills2k.lqs) refit <- fitted(lhills2k.lqs) big3 <- which(abs(reres) >= sort(abs(reres), decreasing=TRUE)[3]) plot(reres ˜ refit, xlab="Fitted values (resistant fit)", ylab="Residuals (resistant fit)") lines(lowess(reres ˜ refit), col=2) text(reres[big3] ˜ refit[big3], labels=rownames(hills2000)[big3], pos=4-2*(refit[big3] > mean(refit)), cex=0.8)

Caerketton shows up rather clearly as an outlier, in both panels. Its residual in panel 1 is −0.356 (visual inspection might suggest −0.35). The predicted log(time) for this race is  Then conveniently written as log(time).  = −0.356. log(time) − log(time)

0.5

1.5 2.5 logdist

3.5

2.0

193

1.0

●● ●●●

0.0

Partial for logclimb

● ● ●● ●

● ●●●● ●●●● ●●● ● ● ● ●● ● ●● ● ● ● ●● ●●●●

−1.0

1.0

●

0.0 −1.0

Partial for logdist

2.0

6.4 A strategy for fitting multiple regression models

● ● ●●● ● ● ●●● ● ●● ●● ●● ●●●● ● ● ● ● ● ●●●● ●● ● ●● ● ● ● ● ● ● ● ●

6.5

● ● ● ● ●●

7.5 8.5 logclimb

Figure 6.14 The solid lines show the respective contributions of the two model terms, in the regression of logtime on logdist and logclimb. Partial residuals, and an associated smooth curve, have been added.

Thus time time ) = −0.356, i.e.,  exp −0.356 = 0.7.   time time Thus the time given for this race is 70% of that predicted by the regression equation, a very large difference indeed. The standardized difference is −3; this can be seen by use of log(

plot(lhills2k.lm, which=2)

If the model is correct and residuals are approximately normally distributed, a residual of this magnitude will occur about 2.6 times in 1000 residuals.10 The resistant fit in panel B suggests that Beinn Lee and 12 Trig Trog are also outliers. These outliers may be a result of non-linearity. We next use termplot() to check whether log(dist) and/or log(climb) should enter the model non-linearly. The contribution of the separate terms

In order to get a plot that is easy to interpret, it is best to transform the variables before entering them into the model, thus: ## Create data frame that has logs of time, dist & climb lhills2k <- log(hills2000[, c("dist", "climb", "time")]) names(lhills2k) <- paste("log", names(lhills2k), sep="") lhills2k.lm <- lm(logtime ˜ logdist+logclimb, data=lhills2k) termplot(lhills2k.lm, partial.resid=TRUE, smooth=panel.smooth, col.res="gray30")

Figure 6.14 shows the result. There is a small but clear departure from linearity, most evident for logdist. The linear model does however give a good general summary of the indications in the data. If the shortest and longest of the races are left out, it appears entirely adequate.

10

> ## Probability of a value <= -3 or >= 3 > 2 * pnorm(-3) [1] 0.0027

194

Multiple linear regression

It is necessary to model the departure from linearity before proceeding further with checking residuals. A quadratic term in logdist will work well here. A resistant fit that has a polynomial term in logdist

The quadratic term needs to be entered as I(logdistˆ2); this ensures that I(logdistˆ2) is taken as the square, rather than as an interaction of logdist with itself. It is however better, as will be explained in Section 7.4, to combine logdist and I(logdistˆ2) together into a single orthogonal polynomial “term”. Subsequent use of the termplot() function then shows the combined effects from logdist and I(logdistˆ2). Here is the code: > reres2 <- residuals(lqs(logtime ˜ poly(logdist,2)+logclimb, data=lhills2k)) > reres2[order(abs(reres2), decreasing=TRUE)[1:4]] Caerketton Beinn Lee Yetholm Ardoch Rig -0.417 0.299 0.200 0.188

As might have been expected, Caerketton is now the only large residual. In the sequel, we therefore omit Caerketton. Refining the model

The component plus residual plot suggested taking a quadratic function of distance. Another possibility might be to add the interaction term logdist:logclimb or, equivalently (as these are continuous variables), I(logdist*logclimb). The following compares these possibilities, using AIC as a criterion. (Lower AIC is better.) > add1(lhills2k.lm, ˜ logdist+I(logdistˆ2)+logclimb+logdist:logclimb, + test="F") Single term additions Model: logtime ˜ logdist + logclimb Df Sum of Sq <none> I(logdistˆ2) 1 0.17 logdist:logclimb 1 0.06

RSS AIC F value Pr(F) 0.77 -233.89 0.61 -245.52 14.331 0.000399 0.71 -236.69 4.651 0.035682

On its own, the AIC value is pretty meaningless. What matters is the comparison between the values of the statistic for the different models. A smaller AIC is preferred; it indicates that the model has better predictive power. Clearly the quadratic term in logdist is doing the job much better. We now examine the diagnostic plots for the model that includes poly(logdist,2). The code is: lhills2k.lm2 <- lm(logtime ˜ poly(logdist,2)+logclimb, data=lhills2k[-42, ]) plot(lhills2k.lm2)

Figure 6.15 shows the result.

6.4 A strategy for fitting multiple regression models

−1.0

0.0

1.0

2.0

−2

Fitted values

−1

0

1

Theoretical quantiles

2

●

−1.0

0.0

1.0

2.0

● Beinn

2 1 0

1.0 0.5

−1

● ● ●● ●● ● ●● ● ● ●● ● ● ●● ● ●●● ● ●● ● ● ● ● ●●● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ●

Standardized residuals

1.5

● Beinn Lee ● Dollar Ben Lomond ●

● ●

−2

● Ben Lomond ● Dollar

Standardized residuals

2 1

● ●●● ●●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●●●● ●

C: Scale−Location

0.0

●

Beinn Lee ●

0

● ● ● ● ●● ● ● ●●● ●● ● ● ● ● ● ● ● ● ● ● ● ●● ● ●● ● ● ● ● ●●●● ● ●●● ● ● ● ● ● ● ● ● ● Ben Lomond ● Dollar ●

B: Normal Q−Q

−1

Lee

Standardized residuals

● Beinn

−2

0.1 0.0 −0.2

Residuals

0.2

A: Resids vs Fitted

195

Lee

● ● ● ●● Chapelgill ● ● ●● ●● ● ●● ● ●●● ● ●● ● ●● ●● ●● ● ●● ● ●● ● ● ● ● ● ● ● ● ● ● ●● ● ● Oban ● ● Cook's

0.0

Fitted values

0.2

1 0.5 ●

0.5 1

distance

0.4

0.6

Leverage

Figure 6.15 Diagnostic plots from the fit of logtime to a polynomial of degree 2 in logdist and logclimb.

Note that poly(logdist,2) generates orthogonal polynomial contrasts. To get a result in which the parameters are coefficients of logdist and logdistˆ2, repeat the fit specifying the quadratic term as poly(logdist,2, raw=TRUE). Additionally, the reader may wish to check that adding the interaction term to a model that already has the quadratic term gives no useful improvement.11 It will be found that use of the interaction term increases the AIC slightly.

The model without the interaction term

As noted earlier, the model that is linear in logdist and logclimb would be adequate for many practical purposes. The coefficients are: > lhills2k.lm <- lm(logtime ˜ logdist+logclimb, data=lhills2k[-42, ]) > summary(lhills2k.lm)$coef Estimate Std. Error t value Pr(>|t|) (Intercept) -4.023 0.1865 -21.6 1.33e-27 logdist 0.778 0.0335 23.2 3.89e-29 logclimb 0.317 0.0302 10.5 1.90e-14

The estimated equation is: log(time) = −4.02 + 0.78 log(dist) + 0.32 log(climb). By noting that exp(−4.02) = 0.018, this can be rewritten as time = 0.018 × dist0.78 × climb0.32 . Are “errors in x” an issue?

Most distances are given to the nearest half mile, and may in any case not be known at all accurately. The error is, however, likely to be small relative to the range of values of distances, so that the attenuation effects that will be discussed in Section 6.7 are likely to be small and of little consequence. See Section 6.7 for the theory that is relevant to the assessment of likely attenuation effects.

11

## Check addition of interaction term add1(lhills2k.lm2, ˜ poly(logdist,2)+logclimb+logdist:logclimb)

196

Multiple linear regression

What happens if we do not transform?

If we avoid transformation and do not allow for increasing variability for the longer races (see Exercise 6 at the end of the chapter for further development), we find several outlying observations, with the race that has the longest time highly influential. Venables and Ripley (2002, p. 154) point out that it is reasonable to expect that variances will be larger for longer races. Using dist as a surrogate for time, they give observations weights of 1/distˆ2. This is roughly equivalent, in its effect on the variance, to our use of log(time) as the dependent variable.

6.5 Problems with many explanatory variables Variable selection is an issue when the aim is to obtain the best prediction possible. Be sure to distinguish the variable selection problem from that of determining which variables have greatest explanatory power. If the interest is in which variables have useful explanatory power, then the choice of variables will depend on which quantities are to be held constant when the effects of other quantities are estimated. There should in any case be an initial exploratory investigation of explanatory variables, as described in Section 6.4, leading perhaps to transformation of one or more of the variables. One suggested rule is that there should be at least 10 times as many observations as variables, before any use of variable selection takes place. For any qualitative factor, subtract one from the number of levels, and count this as the number of variables contributed by that factor. This may be a reasonable working rule when working with relatively noisy data where none of the variables have a dominant effect. There are important contexts where it is clearly inapplicable. For an extended discussion of state-of-the-art approaches to variable selection, we refer the reader to Harrell (2001), Hastie et al. (2009, Sections 4.3 and 4.7). The technically demanding paper by Rao and Wu (2001) was, at the time of publication, a good summary of the literature on model selection. We begin by noting strategies that are designed, broadly, to keep to a minimum the number of different models that will be compared. The following strategies may be used individually, or in combination. 1. A first step may be an informed guess as to what variables/factors are likely to be important. An extension of this approach classifies explanatory variables into a small number of groups according to an assessment of scientific “importance”. Fit the most important variables first, then add the next set of variables as a group, checking whether the fit improves from the inclusion of all variables in the new group. 2. Interaction effects are sometimes modeled by including pairwise multiples of explanatory variables, e.g., x1 × x2 as well as x1 and x2 . Use is made of an omnibus check for all interaction terms, rather than checking for interaction effects one at a time. 3. Principal components analysis is one of several methods that may be able to identify a small number of components, i.e., combinations of the explanatory variables, that together account for most of the variation in the explanatory variables. In favorable circumstances, one or more of the first few principal components will prove to be useful explanatory variables, or may suggest useful simple forms of summary of the

6.5 Problems with many explanatory variables

197

original variables. In unfavorable circumstances, the components will prove irrelevant! See Section 13.1 and Harrell (2001, Sections 4.7 and 8.6) for further commentary and examples. See Chapter 13 for examples. 4. Discriminant analysis can sometimes be used to identify a summary variable. There is an example in Chapter 13. 6.5.1 Variable selection issues

We caution against giving much credence to output from conventional automatic variable selection techniques – various forms of stepwise regression, and best subsets regression. The resulting regression equation may have poorer genuine predictive power than the regression that includes all explanatory variables. The standard errors and t-statistics typically ignore the effects of the selection process; estimates of standard errors, p-values, and F -statistics will be optimistic. Estimates of regression coefficients are biased upwards in absolute value – positive coefficients will be larger than they should be, and negative coefficients will be smaller than they should be. See Harrell (2001) for further discussion. Variable selection – a simulation with random data

Repeated simulation of a regression problem where the data consist entirely of noise will demonstrate the extent of the problem. In each regression there are 41 vectors of 100 numbers that have been generated independently and at random from a normal distribution. In these data:12 1. The first vector is the response variable y. 2. The remaining 40 vectors are the variables x1 , x2 , . . . , x40 . If we find any regression relationships in these data, this will indicate faults with our methodology. (In computer simulation, we should not however totally discount the possibility that a quirk of the random number generator will affect results. We do not consider this an issue for the present simulation!) We perform a best subsets regression in which we look for the three x-variables that best explain y. (This subsection has an example that requires access to the leaps package, implemented by Thomas Lumley using Fortran code by Alan Miller. Before running the code, be sure that leaps is installed.)13 Call: lm(formula = y ˜ -1 + xx[, subvar]) . . . . 12

13

## Generate a 100 by 40 matrix of random normal data y <- rnorm(100) xx <- matrix(rnorm(4000), ncol = 40) dimnames(xx)<- list(NULL, paste("X",1:40, sep="")) ## Find the best fitting model library(leaps) xx.subsets <- regsubsets(xx, y, method = "exhaustive", nvmax = 3, nbest = 1) subvar <- summary(xx.subsets)$which[3,-1] best3.lm <- lm(y ˜ -1+xx[, subvar]) print(summary(best3.lm, corr = FALSE)) ## The following call to bestsetNoise() (from DAAG) achieves the same purpose bestsetNoise(m=100, n=40)

198

Multiple linear regression

Coefficients: xx[, subvar]X12 xx[, subvar]X23 xx[, subvar]X28

Estimate Std. Error t 0.2204 0.0896 -0.1431 0.0750 0.2529 0.0927

value Pr(>|t|) 2.46 0.0156 -1.91 0.0593 2.73 0.0076

Residual standard error: 0.892 on 97 degrees of freedom Multiple R-Squared: 0.132, Adjusted R-squared: 0.105 F-statistic: 4.93 on 3 and 97 DF, p-value: 0.00314

Note that two of the three variables selected have p-values less than 0.05. When we repeated this experiment 10 times, the outcomes were as follows. Categories are exclusive:

Instances All three variables selected were significant at p < 0.01 All three variables had p < 0.05 Two out of three variables had p < 0.05 One variable with p < 0.05 Total

1 3 3 3 10

In the modeling process there are two steps: 1. Select variables. 2. Do a regression and determine SEs and p-values, etc. The p-value calculations have taken no account of step 1. Our ability to find “significance” in data sets that consist only of noise is evidence of a large bias. Cross-validation that accounts for the variable selection process

Cross-validation is one way to determine realistic standard errors and p-values. At each cross-validation step, we repeat both of steps 1 and 2 above, i.e., both the variable selection step and the comparison of predictions from the regression equation with data different from that used in forming the regression equation. Estimates of regression coefficients and standard errors should similarly be based on fitting regressions, at each fold, to the test data for that fold. The issue here is that the equation must be fitted to data that were not used for variable selection. The estimates from the separate folds must then be combined.

Regression on principal components

Regression on principal components, which we discussed briefly in the preamble to this section and will demonstrate in Section 13.1, may sometimes be a useful recourse. Hastie et al. (2009) discuss principal components regression alongside a number of other methods that are available. Note especially the “shrinkage” methods, which directly shrink the

6.6 Multicollinearity

199

coefficients. The function lars(), in the package with the same name, implements a class of methods of this type that has the name least angle regression, allowing also for variable selection. See Efron et al. (2003).

6.6 Multicollinearity Some explanatory variables may be linearly related to combinations of one or more of the other explanatory variables. Technically, this is known as multicollinearity. For each multicollinear relationship, there is one redundant variable. The approaches that we have advocated – careful thinking about the background science, careful initial scrutiny of the data, and removal of variables whose effect is already accounted for by other variables – will generally avoid the more extreme effects of multicollinearity that we will illustrate. Milder consequences are pervasive, especially for observational data.

An example – compositional data

The data set Coxite, in the compositions package, has the mineral compositions of 25 rock specimens of coxite type. Each composition consists of the percentage by weight of five minerals, the depth of location, and porosity. The names of the minerals are abbreviated to A = albite, B = blandite, C = cornite, D = daubite, and E = endite. The analysis that follows is a relatively crude use of these data. For an analysis that uses a method that is designed for compositional data, see Aitchison (2003). Figure 6.16 shows the scatterplot matrix. Notice that the relationship between D and E is close to linear. Also, the percentages of the five minerals sum, in each row, to 100. The code for Figure 6.16 is: library(compositions) data(Coxite) # For pkg compositions, needed to access data coxite <- as.data.frame(Coxite) # From matrix, create data frame ## Scatterplot matrix for data frame coxite pairs(coxite)

We will look for a model that explains porosity as a function of mineral composition. Here is a model that tries to use all six explanatory variables: > coxiteAll.lm <- lm(porosity ˜ A+B+C+D+E+depth, data=coxite) > summary(coxiteAll.lm) . . . . Coefficients: (1 not defined because of singularities) Estimate Std. Error t value Pr(>|t|) (Intercept) -217.7466 253.4439 -0.86 0.40 A 2.6486 2.4825 1.07 0.30 B 2.1915 2.6015 0.84 0.41 C 0.2113 2.2271 0.09 0.93 D 4.9492 4.6720 1.06 0.30 E NA NA NA NA depth 0.0145 0.0333 0.44 0.67

Multiple linear regression

● ● ●

● ● ●● ●

●

●

●● ●

●

● ●● ●

32

●

●

28

●

24

●● ● ● ● ● ● ●

●

●

●

12.5

●

● ● ●

●

●●● ● ●

●

●●

●

● ●● ● ● ●

●● ● ●● ● ●

● ●

●

25

●●

● ● ● ●● ● ●● ● ● ● ● ●● ● ● ●● ●

15 5

● ● ●● ● ● ●● ● ● ●● ● ● ● ● ●● ● ● ● ● ● ●

46

50

Figure 6.16

●

● ● ●

● ● ●● ● ● ●●●

●

●

● ● ● ● ● ●● ●● ● ● ●

●●

● ●

● ● ●

●

●

●

● ●

● ●

● ●●● ●● ● ●● ● ● ●

● ●● ●● ●●● ● ●

D

●

●● ● ● ●

● ●

●

● ● ● ●●● ●● ● ●● ●● ● ● ● ● ● ●●

●

●

●

● ●● ●● ● ● ● ● ● ●

●

● ● ● ●

● ●● ● ●● ● ●

● ● ● ●

●

●●

●

●● ● ● ● ● ● ● ● ● ● ●

● ● ●

5.0

6.5

8.0

● ●

●

●

●● ● ● ●

●

●

● ●

● ● ● ● ●● ●● ●●● ● ● ● ●● ●

●

●

● ●

●

●●● ●

E

●

●

●●

●

●

●

●

●

● ●●●

● ● ●● ●

●

●

● ● ● ● ● ● ● ● ● ●●● ●●● ● ● ● ●

● ● ● ● ● ● ●

●

● ●● ●● ● ● ●

● ●●

● ●

●

●

● ● ●● ● ● ● ● ●● ● ● ● ● ● ● ● ●● ●

●●

●

●

depth

● ●

8.0

●

●

● ● ● ● ●● ● ● ● ● ● ●●● ●

● ●

●

●● ●

● ● ● ● ● ●● ● ● ●●● ●● ● ● ● ●

7.0

●

●

● ● ●● ● ● ●●

●

●● ● ●

9.0

●

● ● ●

● ●●

●●

● ● ● ●● ● ● ● ●● ● ● ● ● ● ● ●

●

●

●● ● ● ● ●● ● ● ● ● ● ● ● ●●● ● ●

●

● ●

●● ● ● ●

●

● ●

●

● ● ● ● ●

●

●

●

50

●

●●

● ●

●

● ●● ●

● ●● ●

● ● ● ● ● ● ● ● ● ●● ●●

●

●

●

●

●

● ● ● ●●● ●

● ● ●● ● ● ● ● ● ● ● ●

●

●

●

●

●●

●

● ● ●● ● ● ● ● ● ● ● ●

● ● ●● ●

●

●● ● ●

● ● ●

● ●

●

●●

●

●● ● ●● ●●●●

● ● ●● ●●●● ● ●● ●●●● ● ●● ● ●

●●

●● ●

● ● ●

●

●

● ● ●

● ●●

●

●

● ●

●

● ●

● ●

● ● ●● ●● ● ●● ●● ● ● ● ● ● ● ● ●● ●

● ●

● ● ●

●

● ● ● ●● ●●

●

●

● ●●

● ●● ● ● ● ● ●

●

●

●

● ●

●

●

● ●

●

●● ●● ●

●

●

● ●

●

●

●

●

●

●

●

●

●

●●●

●●● ●● ● ● ●

● ● ● ●● ● ●

● ●●● ●

●

● ●

●● ●

●

●

●● ●● ●● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ●●

●

●● ●●●● ●

● ● ● ● ●

●

●●

●

●

●● ●

● ●

●

●

●

●

●

C

●

●

● ●● ● ●● ● ●● ● ●

●

● ●

● ●

●

●

●● ● ●

● ●●

●●● ●

●

●

●

●

●

●

● ●

● ● ● ● ●● ● ● ●● ●● ● ●● ● ● ● ● ●

●

●

●

● ●

●

● ●

● ● ●

●

●

● ●

42

●● ● ●

● ●

●

●

● ●● ● ●● ●

●

●

● ●● ●

●

● ● ●

● ●

●● ● ●● ● ● ● ● ●●● ● ● ● ● ● ●● ●

●

●

●

●

●

9.5

11.0

●

●

●● ●

● ●● ● ●

● ●

●

●

●

●

● ● ●● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ●

● ●●

B

●

● ●●

●

●●

●●

● ●

● ●

●●● ● ● ●●

●● ●

●

●

●

●● ● ● ●

●

●

●

● ●

●

46

●

● ●

● ●●

25 ●

42

●

● ●● ●● ●

●

● ●

15

8.0

●

●

5 ●

6.5

A

● ● ●

12.5

5.0

● ● ● ●●

11.0

●

9.0

9.5 ●

8.0

32

7.0

28 ●

porosity

24

24

20

200

20

24

Scatterplot matrix for the variables in the constructed Coxite data set.

Residual standard error: 0.649 on 19 degrees of freedom Multiple R-squared: 0.936, Adjusted R-squared: 0.919 F-statistic: 55.1 on 5 and 19 DF, p-value: 1.18e-10

Notice that: r The variable E, because it is a linear combination of earlier variables, adds no information additional to those variables. Effectively, its coefficient has been set to zero. r None of the individual coefficients comes anywhere near the usual standards of statistical significance. r The overall regression fit, with a p-value of 1.18 × 10−10 , is highly significant. The overall regression fit has good predictive power, notwithstanding the inability to tease out the contributions of the individual coefficients. Figure 6.17 shows 95% pointwise confidence intervals for fitted values at several points within the range. Pointwise confidence bounds can be obtained thus: hat <- predict(coxiteAll.lm, interval="confidence", level=0.95)

The object that is returned is a matrix, with columns fit (fitted values), lwr (lower confidence limits), and upr (upper confidence limits).

6.6 Multicollinearity

201

Figure 6.17 Line y = x, with 95% pointwise confidence bounds for fitted values shown at several locations along the range of fitted values. The points show the observed porosities at each of the fitted values.

6.6.1 The variance inflation factor

The variance inflation factor (VIF) measures the effect of correlation with other variables in increasing the standard error of a regression coefficient. If xj , with values xij (i = 1, . . . , n) is the only variable in a straight line regression model, and bj is the estimated coefficient, then: var[bj ] =

σ2 , sjj

where sjj =

n 

(xij − x¯j )2

i=1

and σ 2 is the variance of the error term in the model. When further terms are included in the regression model, this variance is inflated, as a multiple of σ 2 , by the variance inflation factor. Notice that the VIF depends only on the model matrix. It does not reflect changes in the residual variance. In order to obtain VIFs, we need to explicitly omit E (or one of A, B, C, or D) from the model, thus: > vif(lm(porosity ˜ A+B+C+D+depth, data=coxite)) A B C D depth 2717.82 2484.98 192.59 566.14 3.42

Given the size of these factors, it is unsurprising that none of the individual coefficients can be estimated meaningfully. It is reasonable to try a model that uses those variables that, individually, correlate most strongly with porosity. Here are the correlations: > cor(coxite$porosity, coxite) A B C D E depth porosity [1,] 0.869 -0.551 -0.723 -0.32 -0.408 -0.147 1

Thus we try: > summary(coxiteABC.lm <- lm(porosity ˜ A+B+C, data=coxite)) . . . .

202

Multiple linear regression

Coefficients: Estimate Std. Error t value Pr(>|t|) (Intercept) 53.3046 13.2219 4.03 0.00060 A -0.0125 0.1558 -0.08 0.93700 B -0.5867 0.1513 -3.88 0.00087 C -2.2188 0.3389 -6.55 1.7e-06 . . . . > vif(coxiteABC.lm) A B C 10.94 8.59 4.56

It is then reasonable to simplify this to: > summary(coxiteBC.lm <- lm(porosity ˜ B+C, data=coxite)) . . . . Coefficients: Estimate Std. Error t value Pr(>|t|) (Intercept) 52.2571 1.7831 29.3 < 2e-16 B -0.5753 0.0508 -11.3 1.2e-10 C -2.1949 0.1562 -14.1 1.8e-12 > vif(coxiteBC.lm) B C 1.01 1.01

Using the AIC statistic to compare this model with the model that used all six explanatory variables, we have: > AIC(coxiteAll.lm, coxiteBC.lm) df AIC coxiteAll.lm 7 56.5 coxiteBC.lm 4 52.5

The simpler model wins. Predictions from this simpler model should have slightly narrower confidence bounds than those shown in Figure 6.17. Verification of this is left as an exercise.

Numbers that do not quite add up

Now round all the percentages to whole numbers, and repeat the analysis that uses all six available explanatory variables. > coxiteR <- coxite > coxiteR[, 1:5] <- round(coxiteR[, 1:5]) > summary(lm(porosity ˜ ., data=coxiteR)) . . . . Coefficients: Estimate Std. Error t value Pr(>|t|) (Intercept) -1.4251 23.4043 -0.06 0.95212 A 0.5597 0.2457 2.28 0.03515 B 0.0157 0.2403 0.07 0.94865 C -1.3180 0.2940 -4.48 0.00029

6.7 Errors in x D E depth

0.9748 -0.5530 -0.0149

0.4222 0.5242 0.0223

2.31 -1.05 -0.67

203 0.03305 0.30543 0.51310

Residual standard error: 0.708 on 18 degrees of freedom Multiple R-squared: 0.927, Adjusted R-squared: 0.903 F-statistic: 38.3 on 6 and 18 DF, p-value: 2.69e-09 > vif(lm(porosity ˜ .-E, data=coxiteR)) A B C D depth 16.96 16.49 3.28 4.66 1.25

This result may seem surprising. Noise has been introduced that has removed some of the correlation, so that C now appears significant even when all other explanatory variables are included. Perhaps more surprising is the p = 0.033 for D. While this is contrived, we have from time to time seen comparable effects in computer output that researchers have brought us for scrutiny.

6.6.2 Remedies for multicollinearity

As noted at the beginning of the section, careful initial choice of variables, based on scientific knowledge and careful scrutiny of relevant exploratory plots of explanatory variables, will often avert the problem. Occasionally, it may be possible to find or collect additional data that will reduce correlations among the explanatory variables. Ridge regression is one of several approaches that may be used to alleviate the effects of multicollinearity, in inflating coefficients and their standard errors. We refer the reader to the help page for the function lm.ridge() in the MASS package and to the discussions in Berk (2008), Myers (1990), Harrell (2001). For a less elementary and more comprehensive account, see Hastie et al. (2009). Use of the function lm.ridge() requires the user to choose the tuning parameter lambda. Typically, the analyst tries several settings and chooses the value that, according to one of several available criteria, is optimal. Principal components regression is another possible remedy.

6.7 Errors in x The issues that are canvassed here have large practical importance. It can be difficult to assess the implications of the theoretical results for particular practical circumstances. The discussion so far has been assumed: either that the explanatory variables are measured with negligible error, or that the interest is in the regression relationship given the observed values of explanatory variables. This subsection draws attention to the effect that errors in the explanatory variables can have on regression slope. Discussion is mainly on the relatively simple “classical” errors in x model. With a single explanatory variable, the effect under the classical “errors in x” model is to reduce the expected magnitude of the slope, that is, the slope is attenuated. Furthermore, the estimated slope is less likely to be distinguishable from statistical noise. For estimating the magnitude of the error and consequent attenuation of the slope, there must be information

204

Multiple linear regression

additional to that shown in a scatterplot of y versus x. There must be a direct comparison with values that are measured with negligible error. The study of the measurement of dietary intake that is reported in Schatzkin et al. (2003) illustrates some of the key points. The error in the explanatory variable, as commonly measured, was shown to be larger and of greater consequence than most researchers had been willing to contemplate. Measurement of dietary intake

The 36-page Diet History Questionnaire is a Food Frequency Questionnaire (FFQ) that was developed and evaluated at the US National Cancer Institute. In large-scale trials that look for dietary effects on cancer and on other diseases, it has been important to have an instrument for measuring food intake that is relatively cheap and convenient. (Some trials have cost US$100 000 000 or more.) The FFQ asks for details of food intake over the previous year for 124 food items. It queries frequency of intake and, for most items, portion sizes. Supplementary questions query such matters as seasonal intake and food type. More detailed food records may be collected at specific times, which can then be used to calibrate the FFQ results. One such instrument is a 24-hour dietary recall, which questions participants on their dietary intake in the previous 24 hours. Schatzkin et al. (2003) compared FFQ measurements with those from Doubly Labeled Water, used as an accurate but highly expensive biomarker. They conclude that the FFQ is too inaccurate for its intended purpose. The 24-hour dietary recall, although better, was still seriously inaccurate. In some instances, the standard deviation for estimated energy intake was seven times the standard deviation, between different individuals, of the reference. There was a bias in the relationship between FFQ and reference that further reduced the attenuation factor, to 0.04 for women and to 0.08 for men. For the relationship between the 24-hour recalls and the reference, the attenuation factors were 0.1 for women and 0.18 for men, though these can be improved by use of repeated 24-hour recalls. These results raise serious questions about what such studies can achieve, using presently available instruments that are sufficiently cheap and convenient that they can be used in large studies. Carroll (2004) gives an accessible summary of the issues. Subsection 10.7.6 has further brief comment on the modeling issues. Simulations of the effect of measurement error

Suppose that the underlying regression relationship that is of interest is: yi = α + βxi + εi , where var[εi ] = σ 2 (i = 1, . . . , n).

 n ¯ 2 /(n − 1) be the standard deviation of the values that are measured Let sx = i=1 (xi − x) without error. Take the measured values as wi = xi + ηi , where var[ηi ] = s2x τ 2 . The ηi are assumed independent of the εi .

6.7 Errors in x

205

Figure 6.18 The fitted solid lines show the change in the regression line as the error in x changes. The underlying relationship, shown with the dashed line, is in each instance y = 15 + 1.5x. For the definition of τ , see the text. This figure was obtained by use of the DAAG function errorsINx(), with default arguments.

Figure 6.18 shows results from a number of simulations that use the wi as the explanatory values. If τ = 0.4 (the added error has a variance that is 40% of sx ), the effect on the slope is modest. If τ = 2, the attenuation is severe. The function errorsINx() (DAAG) can be used for additional simulations such as are shown in Figure 6.18. An estimate of the attenuation in the slope is, to a close approximation: λ=

1 . 1 + τ2

Here, λ has the name reliability ratio. If, for example, τ = 0.4, then λ  0.86. Whether a reduction in the estimated slope by a factor of 0.86 is of consequence will depend on the application. Often there will be more important concerns. Very small attenuation factors (large attenuations), e.g., less than 0.1 such as were found in the Schatzkin et al. (2003) study, are likely to have serious consequences for the use of analysis results. Points to note are: r From the data used in the panels of Figure 6.18, it is impossible to estimate τ , or to know the underlying xi values. This can be determined only from an investigation that compares the wi with an accurate, i.e., for all practical purposes error-free, determination of the xi . r A test for β = 0 can be undertaken in the usual way, but with reduced power to detect an effect that may be of interest.

206

Multiple linear regression

r The t-statistic for testing β = 0 is affected in two ways; the numerator is reduced by an expected factor of λ, while the standard error that appears in the numerator increases. Thus if λ = 0.1, the sample size required to detect a non-zero slope is inflated by more than the factor of 100 that is suggested by the effect on the slope alone. ∗

Two explanatory variables

Consider first the case where one predictor is measured with error, and others without error. The coefficient of the variable that is measured with error is attenuated, as in the single variable case. The coefficients of other variables may be reversed in sign, or show an effect when there is none. See Carroll et al. (2006, pp. 52–55) for summary comment. Suppose that y = β1 x1 + β2 x2 + ε. If w1 is unbiased for x1 and the measurement error η is independent of x1 and x2 , then least squares regression with explanatory variables w1 and x2 yields an estimate of λβ1 , where if ρ is the correlation between x1 and x2 : λ=

1 − ρ2 . 1 − ρ2 + τ 2

A new feature is the bias in the least squares estimate of β2 . The naive least squares estimator estimates s1 (6.4) β2 + β1 (1 − λ) γ12 , where γ12 = ρ . s2 Here, γ12 is the coefficient of x2 in the least squares regression of x1 on x2 , s1 = SD[x1 ] and s2 = SD[x2 ]. The estimate of β2 may be substantially different from zero, even though β2 = 0. Where β2 = 0, the least squares estimate can be reversed in sign from β2 . Some of the effect of x1 is transferred to the estimate of the effect of x2 . Two explanatory variables, one measured without error – a simulation

The function errorsINx() (DAAG), when supplied with a non-zero value for the argument gpdiff, simulates the effect when the variable that is measured without error codes for a categorical effect. Figure 6.19 had gpdiff=1.5. Two lines appear, suggesting a “treatment” effect where there was none. The function errorsINseveral() simulates a model where there are two continuous variables x1 and x2 . The default choice of arguments has β1 = 1.5, β2 = 0, ρ = −0.5, s1 = s2 = 2, τ = 1.5, var[ε] = 0.25. Measurement error variances are x1 : s12 τ 2 , x2 : 0. Then λ = 0.25, γ12 = −0.5, and the expected value for the naive least squares estimator of β2 is β2 + β1 (1 − λ)γ12 = 0 + 1.5 × 0.75 × (−0.5) = −0.5675. Here is a simulation result, with default arguments as noted:

6.7 Errors in x

207

Figure 6.19 In the simulations whose results are shown here, y is a linear function of x. The mean value of x is 12.5 for the first level (“ctl”) of a grouping variable, and 14.0 for the second level (“trt”) of the grouping variable. In the panel on the left, the values of x are measured without error. In the middle and right panels, independent errors have been added to x from distributions with SDs that are 0.8 and 1.6 times that of the within-group standard deviation of x. The SEDs are conditional on w = xWITHerr.

> errorsINseveral() Values for simulation Estimates: no error in x1 LS Estimates: error in x1

Intercept b1 b2 2.482 1.500 0.000 2.554 1.494 0.009 35.448 0.400 -0.597

An arbitrary number of variables

Where two or more variables are measured with substantial error, there is an increased range of possibilities for transferring some part or parts of effects between variables. By specifying the arguments V and xerrV, the function errorsINseveral() can be used for simulations with an arbitrary correlation structure for the explanatory variables, and with an arbitrary variance–covariance matrix for the added errors.14 ∗

The classical error model versus the Berkson error model

In the classical model, E[wi |xi ] = xi . In the Berkson model, E[xi |wi ] = wi . This may be a realistic model in an experiment where wi is an instrument setting, but the true value varies randomly about the instrument setting. For example, the temperature in an oven or kiln may be set to wi , but the resulting (and unknown) actual temperature is xi . In straight line regression, the coefficient is then unbiased, but the variance of the estimate of the coefficient increases.

14

If β is the vector of coefficients in the model without errors in the measured values, V corresponds in the obvious way to V, and U to xerrV, then an estimate for the resulting least squares estimates for regression on the values that are measured with error is β V (V + U )−1 . Note that Zeger et al. (2000, p. 421) have an initial T (our U ), where V is required.

208

Multiple linear regression

Zeger et al. (2000) discuss the practical consequences of both types of error, though giving most of their attention to the classical model. In their context, realistic models may have elements of both the classical and Berkson models.

6.8 Multiple regression models – additional points The following notes should help dispel any residual notion that this chapter’s account of multiple regression models has covered everything of importance.

6.8.1 Confusion between explanatory and response variables

As an example, we return to the allbacks data. We compare the coefficients in the equation for predicting area given volume and weight with the rearranged coefficients from the equation that predicts weight given volume and area: > coef(lm(area ˜ volume + weight, data=allbacks)) (Intercept) volume weight 35.459 -0.964 1.361 > b <- as.vector(coef(lm(weight ˜ volume + area, data=allbacks))) > c("_Intercept_"=-b[1]/b[3], volume=-b[2]/b[3], weight=1/b[3]) _Intercept_ volume weight -47.85 -1.51 2.13

Only if the relationship is exact, so that predicted time is the same as observed time, will the equations be exactly the same. For examples from the earth sciences literature, see Williams (1983).

Unintended correlations

Suppose that xi (i = 1, 2, . . . , n) are results from a series of controls, while yi (i = 1, 2, . . . , n) are results from the corresponding treated group. It is tempting to plot y − x versus x. Unfortunately, there is likely to be a negative correlation between y − x and x, though this is not inevitable. This emphasizes the desirability of maintaining a clear distinction between explanatory and response variables. See the example in Sharp et al. (1996).

6.8.2 Missing explanatory variables

Here the issue is use of the wrong model for the expected value. With the right “balance” in the data, the expected values are unbiased or nearly unbiased. Where there is serious imbalance, the bias may be huge. Figure 6.20 relates to data collected in an experiment on the use of painkillers (Gordon et al., 1995). Pain was measured as a VAS (Visual-Analogue Scale) score. Researchers were investigating differences in the pain score between the two analgesic treatments, without and with baclofen.

6.8 Multiple regression models – additional points

209

Figure 6.20 Does baclofen, following operation (additional to earlier painkiller), reduce pain? Subgroup numbers, shown below each point in the graph, weight the overall averages when sex is ignored.

Notice that the overall comparison (average for baclofen versus average for no baclofen) goes in a different direction from the comparison for the two sexes separately. As the two treatment groups had very different numbers of men and women, and as there was a strong sex effect, an analysis that does not account for the sex effect gives an incorrect estimate of the treatment effect (Cohen, 1996). The overall averages in Figure 6.20 reflect the following subgroup weighting effects (f is shorthand for female and m for male): 3 to 18 (a little less than f average) Baclofen: 15f to 3m, i.e., 15 18 7 9 No baclofen: 7f to 9m, i.e., 16 to 16 (≈ 12 -way between m & f)

There is a sequel. More careful investigation revealed that the response to pain has a different pattern over time. For males, the sensation of pain declined more rapidly over time.

Strategies

(i) Simple approach. Calculate means for each subgroup separately. Overall treatment effect is average of subgroup differences. Effect of baclofen (reduction in pain score from time 0) is: Females: 3.479 − 4.151 = −0.672 (−ve, therefore an increase) Males: 1.311 − 1.647 = −0.336 Average over male and female = −0.5 × (0.672 + 0.336) = −0.504 (ii) Fit a model that accounts for sex and baclofen effects. y = overall mean + sex effect + baclofen effect + interaction. (At this point, we are not including an error term.) When variables or factors are omitted from models, values of the outcome variable are as far as possible accounted for using those that remain. The mouse brain weight example in Subsection 6.2.3 can be understood in this way. Bland and Altman (2005) give several examples of published results where conclusions have been vitiated by effects of this type.

210

Multiple linear regression

Another example of this same type, albeit in the context of contingency tables, was discussed in Subsection 3.4.5. The analysis of the UCB admissions data in Section 8.3 formulates the analysis of contingency table data as a regression problem. 6.8.3∗ The use of transformations

Often there are scientific reasons for transformations. Thus, suppose we have weights w of individual apples, but the effects under study are more likely to be related to surface 2 area. We should consider using x = w 3 as the explanatory variable. If the interest is in studying relative, rather than absolute, changes, it may be best to work with the logarithms of measurements. Statisticians use transformations for one or more of the following reasons: 1. To form a straight line or other simple relationship. 2. To ensure that the “scatter” of the data is similar for all categories, i.e., to ensure that the boxplots all have a similar shape. In more technical language, the aim is to achieve homogeneity of variance. 3. To make the distribution of data more symmetric and closer to normal. If there is a transformation that deals with all these issues at once, we are fortunate. It may greatly simplify the analysis. A log transformation may both remove an interaction and give more nearly normal data. It may, on the other hand, introduce an interaction where there was none before. Or a transformation may reduce skewness while increasing heterogeneity. The availability of direct methods for fitting special classes of model with non-normal errors, for example the generalized linear models that we will discuss in Chapter 8, has reduced the need for transformations.

6.8.4∗ Non-linear methods – an alternative to transformation?

This is a highly important area for which, apart from the present brief discussion, we have not found room in the present book. We will investigate the use of the R nls() function (stats package) to shed light on the loglinear model that we used for the hill race data in Subsection 6.2.1. The analysis of Subsection 6.2.1 assumed additive errors on the transformed logarithmic scale. This implies, on the untransformed scale, multiplicative errors. We noted that the assumption of homogeneity of errors was in doubt. One might alternatively assume that the noise term is additive on the untransformed scale, leading to the non-linear model β

y = x1α x2 + ε where y = time, x1 = dist, and x2 = climb. We will use the nls() non-linear least squares function to estimate α and β. The procedure used to solve the resulting non-linear equations is iterative, requiring starting values for α and β. We use the estimates from the earlier loglinear regression as starting

6.8 Multiple regression models – additional points

211

values. Because we could be taking a square or cube of the climb term, we prefer to work with the variable climb.mi that is obtained by dividing climb by 5280, so that numbers are of modest size: nihills$climb.mi <- nihills$climb/5280 nihills.nls0 <- nls(time ˜ (distˆalpha)*(climb.miˆbeta), start = c(alpha = 0.68, beta = 0.465), data = nihills) plot(residuals(nihills.nls0) ˜ log(predict(nihills.nls0)))

Output from the summary() function includes the following: Parameters: Estimate Std. Error t value Pr(>|t|) alpha 0.31516 0.00806 39.1 <2e-16 beta 0.81429 0.02949 27.6 <2e-16

These parameter estimates differ substantially from those obtained under the assumption of multiplicative errors. This is not an unusual occurrence; the non-linear least squares problem has an error structure that is different from the linearized least-squares problem solved earlier. Residuals suggest a non-linear pattern. Another possibility, that allows time to increase non-linearly with climb.mi, is y = α + βx1 + γ x2δ + ε. We then fit the model, using an arbitrary starting guess: nihills.nls <- nls(time ˜ gamma + delta1*distˆalpha + delta2*climb.miˆbeta, start=c(gamma = .045, delta1 = .09, alpha = 1, delta2=.9, beta = 1.65), data=nihills) plot(residuals(nihills.nls) ˜ log(predict(nihills.nls)))

The starting values were obtained by fitting an initial model in which alpha was constrained to equal 1. The result is: Parameters: Estimate Std. Error t value Pr(>|t|) gamma 0.1522 0.0714 2.13 0.04708 delta1 0.0399 0.0284 1.41 0.17694 alpha 1.3102 0.2709 4.84 0.00013 delta2 0.8657 0.0922 9.39 2.3e-08 beta 1.5066 0.1810 8.32 1.4e-07 Estimate Std. Error t value Pr(>|t|)

There are no obvious outliers in the residual plot. In addition, there is no indication of an increase in the variance as the fitted values increase. Thus, a variance-stabilizing transformation or the use of weighted least squares is unnecessary.

212

Multiple linear regression

6.9 Recap A coefficient in a multiple regression equation predicts the effect of a variable when other variables are held constant. Coefficients can thus be different, sometimes dramatically, for a different choice of explanatory variables. Regression equation predictions are for the data used to derive the equation, and reflect any sampling biases that affect that data. Biases that arise because data were not randomly sampled from the population can lead to predictions that, for the population as a whole, are seriously astray. Plots that can be useful for checking regression assumptions and/or for checking whether results may be unduly influenced by individual data points include: scatterplot matrices of the variables in the regression equation, plots of residuals against fitted values, partial residual plots such as are provided by the termplot() function (these are a better guide than plots of residuals against individual explanatory variables), normal probability plots of residuals, scale–location plots, and Cook’s distance and related plots. Robust methods downweight points that may be outliers. Resistant regression methods are designed to completely remove the contribution of outliers to the regression fit. 6.10 Further reading Faraway (2004) is a wide-ranging account that covers many of the important practical issues. Harrell (2001) is likewise wide-ranging, with an emphasis on biostatistical applications. Again it has a great deal of useful practical advice. Weisberg (1985) offers a relatively conventional approach. Cook and Weisberg (1999) rely heavily on graphical explorations to uncover regression relationships. Venables and Ripley (2002) is a basic reference for using R for regression calculations. See also Fox (2002). Hastie et al. (2009) offer wideranging challenges for the reader who would like to explore beyond the forms of analysis that we have described. On variable selection, which warrants more attention than we have given it, see Bolker (2008), Harrell (2001), Hastie et al. (2009), Venables (1998). There is certain to be, in the next several years, substantial enhancement to what R packages offer in this area. Bolker’s account extends (p. 215) to approaches that weight and average models. Rosenbaum (2002) is required reading for anyone who wishes to engage seriously with the analysis of data from observational studies. Results can rarely be interpreted with the same confidence as for a carefully designed experimental study. There are however checks and approaches that, depending on the context, can be helpful in assessing the credence that should be given one or other interpretation of analysis results. On errors in variables, see Carroll et al. (2006). Linear models are a special case of non-linear models. Several of the studies that are discussed in Leavitt and Dubner (2005), some with major public policy relevance, relied to a greater or lesser extent on regression methods. References in the notes at the end of their book allow interested readers to pursue technical details of the statistical and other methodology. The conflation of multiple sources of insight and evidence is invariably necessary, in such studies, if conclusions are to carry conviction. Especially hazardous is the use of analyses where there are multiple potential confounding variables, i.e., variables whose effects must be accounted for if coefficients for remaining

6.10 Further reading

213

variables are to be genuinely suggestive of a causal link. Not only must confounders be included; their effects, including possible interaction effects, must be correctly modeled. Controversy over studies on the health effects of moderate alcohol consumption provide a good example; see, for example, Jackson et al. (2005). For commentary on the use of regression and other models for predictive purposes, see Maindonald (2003). Structural equation models allow, in addition to explanatory variables and dependent variables, intermediate variables that are dependent with respect to one or more of the explanatory variables, and explanatory with respect to one or more of the dependent variables. Cox and Wermuth (1996) and Edwards (2000) describe approaches that use regression methods to elucidate the relationships. Cox and Wermuth is useful for the large number of examples and for its illuminating comments on practical issues, while Edwards is more up-to-date in its account of the methodology. Bates and Watts (1988) discuss non-linear models in detail. A more elementary presentation is given in one of the chapters of Myers (1990).

References for further reading

Bates, D. M. and Watts, D. G. 1988. Non-linear Regression Analysis and Its Applications. Bolker, B. M. 2008. Ecological Models and Data in R. Carroll, R. J., Ruppert, D. and Stefanski, L. A. 2006. Measurement Error in Nonlinear Models: A Modern Perspective, 2nd edn. Cook, R. D. and Weisberg, S. 1999. Applied Regression Including Computing and Graphics. Cox, D. R. and Wermuth, N. 1996. Multivariate Dependencies: Models, Analysis and Interpretation. Edwards, D. 2000. Introduction to Graphical Modelling, 2nd edn. Faraway, J. J. 2004. Linear Models with R. Fox, J. 2002. An R and S-PLUS Companion to Applied Regression. Harrell, F. E. 2001. Regression Modelling Strategies, with Applications to Linear Models, Logistic Regression and Survival Analysis. Hastie, T., Tibshirani, R. and Friedman, J. 2009. The Elements of Statistical Learning. Data Mining, Inference and Prediction, 2nd edn. Jackson, R., Broad, J., Connor, J. and Wells, S. 2005. Alcohol and ischaemic heart disease: probably no free lunch. The Lancet 366: 1911–12. Leavitt, S. D. and Dubner, S. J. 2005. Freakonomics. A Rogue Economist Explores the Hidden Side of Everything. Maindonald, J. H. 2003. The role of models in predictive validation. Invited Paper, ISI Meeting, Berlin. Myers, R. H. 1990. Classical and Modern Regression with Applications, 2nd edn. Rosenbaum, P. R. 2002. Observational Studies, 2nd edn. Venables, W. N. 1998. Exegeses on linear models. Proceedings of the 1998 International S-PLUS User Conference. Venables, W. N. and Ripley, B. D. 2002. Modern Applied Statistics with S, 4th edn. Weisberg, S. 1985. Applied Linear Regression, 2nd edn.

214

Multiple linear regression

6.11 Exercises 1. The data set cities lists the populations (in thousands) of Canada’s largest cities over 1992 to 1996. There is a division between Ontario and the West (the so-called “have” regions) and other regions of the country (the “have-not” regions) that show less rapid growth. To identify the “have” cities we can specify ## Set up factor that identifies the ‘have’ cities ## Data frame cities (DAAG) cities$have <- factor((cities$REGION=="ON")| (cities$REGION=="WEST")) Plot the 1996 population against the 1992 population, using different colors to distinguish the two categories of city, both using the raw data and taking logarithms of data values, thus: plot(POP1996 ˜ POP1992, data=cities, col=as.integer(cities$have)) plot(log(POP1996) ˜ log(POP1992), data=cities, col=as.integer(cities$have)) Which of these plots is preferable? Explain. Now carry out the regressions cities.lm1 <- lm(POP1996 ˜ have+POP1992, data=cities) cities.lm2 <- lm(log(POP1996) ˜ have+log(POP1992), data=cities) and examine diagnostic plots. Which of these seems preferable? Interpret the results. 2. In the data set cement (MASS package), examine the dependence of y (amount of heat produced) on x1, x2, x3 and x4 (which are proportions of four constituents). Begin by examining the scatterplot matrix. As the explanatory variables are proportions, do they require transformation, perhaps by taking log(x/(100 − x))? What alternative strategies might be useful for finding an equation for predicting heat? 3. Use the model that was fitted to the data in nihills to give predicted values for the data in hills2000. Plot these against predicted values from the model fitted to hills2000, and use differences from observed values of log(time) to estimate a prediction variance that is relevant when Northern Irish data are used to predict Scottish times. Would you expect this variance to be larger or smaller than the estimated error variance from the hills2000 model fit? Is this expectation born out? 4. The data frame hills2000 (DAAG) updates the 1984 information in the data set hills. Fit regression models, for men and women separately, based on the data in hills. Check whether they fit satisfactorily over the whole range of race times. Compare the two equations. 5. Section 6.1 used lm() to analyze the allbacks data that are presented in Figure 6.1. Repeat the analysis using (1) the function rlm() in the MASS package, and (2) the function lqs() in the MASS package. Compare the two sets of results with the results in Section 6.1. 6. The following investigates the consequences of not using a logarithmic transformation for the nihills data analysis. The second differs from the first in having a dist × climb interaction term, additional to linear terms in dist and climb. (a) Fit the two models: nihills.lm <- lm(time ˜ dist+climb, data=nihills) nihills2.lm <- lm(time ˜ dist+climb+dist:climb, data=nihills) anova(nihills.lm, nihills2.lm)

6.11 Exercises

215

(b) Using the F -test result, make a tentative choice of model, and proceed to examine diagnostic plots. Are there any problematic observations? What happens if these points are removed? Refit both of the above models, and check the diagnostics again. 7. Check the variance inflation factors for bodywt and lsize for the model brainwt ˜ bodywt + lsize, fitted to the litters data set. Comment. 8. Apply the lm.ridge() function to the litters data, using the generalized cross-validation (GCV) criterion to choose the tuning parameter. (GCV is an approximation to cross-validation.) (a) In particular, estimate the coefficients of the model relating brainwt to bodywt and lsize and compare with the results obtained using lm(). (b) Using both ridge and ordinary regression, estimate the mean brain weight when litter size is 10 and body weight is 7. Use the bootstrap, with case-resampling, to compute approximate 95% percentile confidence intervals using each method. Compare with the interval obtained using predict.lm(). 9.∗ Compare the ranges of dist and climb in the data frames nihills and hills2000. In which case would you expect it to be more difficult to find a model that fits well? For each of these data frames, fit both the model based on the formula log(time) ˜ log(dist) + log(climb) and the model based on the formula time ˜ alpha*dist + beta*I(climbˆ2) Is there one model that gives the best fit in both cases? 10. The data frame table.b3 in the MPV package contains data on gas mileage and 11 other variables for a sample of 32 automobiles. (a) Construct a scatterplot of y (mpg) versus x1 (displacement). Is the relationship between these variables non-linear? (b) Use the xyplot() function, and x11 (type of transmission) as a group variable. Is a linear model reasonable for these data? (c) Fit the model relating y to x1 and x11 which gives two lines having possibly different slopes and intercepts. Check the diagnostics. Are there any influential observations? Are there any influential outliers? (d) Plot the residuals against the variable x7 (number of transmission speeds), again using x11 as a group variable. Is there anything striking about this plot? 11. The following code is designed to explore effects that can result from the omission of explanatory variables: > x1 <- runif(10) # predictor which will be missing > x2 <- rbinom(10, 1, 1-x1) # observed predictor which depends > # on missing predictor > y <- 5*x1 + x2 + rnorm(10,sd=.1) # simulated model; coef > # of x2 is positive > y.lm <- lm(y ˜ factor(x2)) # model fitted to observed data > coef(y.lm) (Intercept) factor(x2)1 2.8224119 -0.6808925 # effect of missing variable: # coefficient of x2 has wrong sign > y.lm2 <- lm(y ˜ x1 + factor(x2)) # correct model > coef(y.lm2)

216

Multiple linear regression (Intercept) x1 factor(x2)1 0.06654892 4.91216206 0.92489061 # coef estimates are now OK What happens if x2 is generated according to x2 <- rbinom(10, 1, x1)? x2 <- rbinom(10, 1, .5)?

12. Fit the model investigated in Subsection 6.8.4, omitting the parameter α. Investigate and comment on changes in the fitted coefficients, standard errors, and fitted values. 13. Figure 6.19 used the function errorsINx(), with the argument gpdiff=1.5, to simulate data in which the regression relationship y = 15 + 1.5x is the same in each of two groups (called ctl and trt). The left panel identifies the two fitted lines when the explanatory variable is measured without error. These are, to within statistical error, identical. The right panel shows the fitted regression lines when random error of the same order of magnitude as the within-groups variation in x is added to x, giving the column of values zWITHerr. (a) Run the function for several different values of gpdiff in the interval (0, 1.5), and plot the estimate of the treatment effect against gpdiff. (b) Run the function for several different values of timesSDz in the interval (0, 1.5), and plot the estimate of the treatment effect against gpdiff. (c) Run the function with beta = c(-1.5,0). How does the estimate of the treatment effect change, as compared with b = c(1.5,0)? Explain the change. 14. Fit the following two resistant regressions, in each case plotting the residuals against Year. library(MASS} nraw.lqs <- lqs(northRain ˜ SOI + CO2, data=bomregions) north.lqs <- lqs(I(northRainˆ(1/3)) ˜ SOI + CO2, data=bomregions) par(mfrow=c(2,1)) plot(residuals(nraw.lqs) ˜ Year, data=bomregions) plot(residuals(north.lqs) ˜ Year, data=bomregions) par(mfrow=c(1,1)) Compare, also, normal probably plots for the two sets of residuals. (a) Repeat the calculations several times. Comment on the extent of variation, from one run to the next, in the regression coefficients. (b) Based on examination of the residuals, which regression model seems more acceptable: nraw.lqs or north.lqs? (c) Compare the two sets of regression coefficients. Can you explain why they are so very different? (More careful modeling will take into account the temporal sequence in the observations. See Section 9.2 for an analysis that does this.)

7

Exploiting the linear model framework

The model matrix X is fundamental to all calculations for a linear model. The model matrix carries the information needed to calculate the fitted values that correspond to any particular choice of coefficients. There is a one-to-one correspondence between columns of X and regression coefficients. In Chapter 6, the columns of the model matrix contained the observed values of the explanatory variables, perhaps after transformation. Fitted values were obtained by multiplying the first column by the first coefficient (usually the intercept), the second column by the second coefficient, and so on across all columns. The sum of the products in any row is the fitted value for that row. This chapter will explore new ways to relate the columns of the model matrix to the explanatory variables, where a variable may be either a vector of numeric values, or a factor. Vectors of zeros and ones (columns of “dummy” variables) can be used to handle factor levels, but as noted below there are other possibilities. For modeling a quadratic form of response, we take values of x as one of the columns and values of x 2 as another. The model matrix framework also allows the modeling of many other forms of non-linear response. As before, the regression calculations find the set of coefficients that best predicts the observed responses, in the sense of minimizing the sum of squares of the residuals. The latter part of this chapter will describe the fitting of smooth curves and surfaces that do not necessarily have a simple parametric form of mathematical description. Although not as convenient as linear or other simple parametric relationships, such curves and surfaces do allow the calculation of predicted values and associated standard error estimates. For many purposes, this is all that is needed. 7.1 Levels of a factor – using indicator variables 7.1.1 Example – sugar weight

Figure 7.1 displays data from an experiment that compared an unmodified wild-type plant with three different genetically modified (GM) forms (data are in the data set sugar in our DAAG package). The measurements are weights (mg) of sugar that were obtained by breaking down the cellulose. There is a single explanatory factor (treatment), with one level for each of the different control agents. For convenience, we will call the factor levels Control, A (GM1), B (GM2), and C (GM3).1 1

stripplot(trt ˜ weight, pch=0, xlab="Weight (mg)", data=sugar, aspect=0.5))

218

Exploiting the linear model framework

Control (WTa )

C B A Control 50

60

70

80

Weight (mg)

90

100

A (GM1)

B (GM2)

82.0 97.8 69.9

58.3 67.9 59.3

68.1 70.8 63.6

50.7 47.1 48.9

Mean = 83.2

61.8

67.5

48.9

a

C (GM3)

WT = Wild Type; GM = Genetically Modified

Figure 7.1 Weights (weight) of sugar extracted from a control (wild-type) plant, and from three different genetically modified plant types.

We could reduce the apparent difference in variability between treatments by working with log(weight). For present illustrative purposes, we will however work with the variable weight, leaving as an exercise for the reader the analysis that works with log(weight). The model can be fitted either using the function lm() or using the function aov(). The two functions give different default output. For any problem that involves factor(s), there are several different ways to set up the model matrix. A common strategy (on a vanilla setup, the default) is to set up one of the treatment levels as a baseline or reference, with the effects of other treatment levels then measured from the baseline. Here it makes sense to set Control (Wild) as the baseline. Before proceeding, it may pay to do the following check: > options()$contrasts # Check the default factor contrasts unordered ordered "contr.treatment" "contr.poly" > ## If your output does not agree with the above, then enter > options(contrasts=c("contr.treatment", "contr.poly"))

The reason for this check will become apparent shortly. With Control as baseline, a one-way analysis of variance of the data in Figure 7.1 has the model matrix that is shown in Table 7.1; values of the response (sugar$weight) have been added in the final column. The following checks the order of the levels in the column trt and prints the model matrix: > levels(sugar$trt) # Note the order of the levels [1] "Control" "A" "B" "C" > sugar.aov <- aov(weight ˜ trt, data=sugar) > model.matrix(sugar.aov)

Here are the results from the least squares calculations: > summary.lm(sugar.aov) +

# NB: summary.lm(), # not summary() or summary.aov()

Call: aov(formula = weight ˜ trt, data = sugar) Residuals:

7.1 Levels of a factor – using indicator variables

219

Table 7.1 Model matrix for the analysis of variance calculation for the data in Figure 7.1. The values of the response are in the final column. Control (baseline)

A

B

C

weight

1 1 1 1 1 1 1 1 1 1 1 1

0 0 0 1 1 1 0 0 0 0 0 0

0 0 0 0 0 0 1 1 1 0 0 0

0 0 0 0 0 0 0 0 0 1 1 1

82.0 97.8 69.9 58.3 67.9 59.3 68.1 70.8 63.6 50.7 47.1 48.9

Min 1Q Median 3Q Max -13.333 -2.783 -0.617 2.175 14.567 Coefficients: Estimate Std. Error t value Pr(>|t|) (Intercept) 83.23 4.47 18.61 7.2e-08 trtA -21.40 6.33 -3.38 0.00960 trtB -15.73 6.33 -2.49 0.03768 trtC -34.33 6.33 -5.43 0.00062 Residual standard error: 7.75 on 8 degrees of freedom Multiple R-Squared: 0.791, Adjusted R-squared: 0.713 F-statistic: 10.1 on 3 and 8 degrees of freedom, p-value: 0.00425

The row labeled (Intercept) gives the estimate (= 83.23) for the baseline, i.e., Control. The remaining coefficients (differences from the baseline) are: A: weight differs by −21.40. B: weight differs by −15.73. C: weight differs by −34.33. All three differences from the control are significant at the conventional 5% level. In Table 7.2, the multiples determined by the least squares calculations are shown above each column. Also shown is  y , which is the fitted or predicted value, calculated either as fitted(sugar.aov) or as predict(sugar.aov). Residuals can be obtained by subtracting the predicted values ( y ) in Table 7.2 from the observed values (y) in Table 7.1. In this example, the estimate for each treatment is the treatment mean. Regression calculations have given us a complicated way to compute averages! The methodology

220

Exploiting the linear model framework

Table 7.2 At the head of each column is the multiple, as determined by least squares, that is taken in forming the fitted values. Control: 83.2 1 1 1 1 1 1 1 1 1 1 1 1

A: −21.4

B: −15.7

C: −34.3

Fitted value

0 0 0 1 1 1 0 0 0 0 0 0

0 0 0 0 0 0 1 1 1 0 0 0

0 0 0 0 0 0 0 0 0 1 1 1

83.2 83.2 83.2 61.8 61.8 61.8 67.5 67.5 67.5 48.9 48.9 48.9

shows its power to better effect in more complex models, where there is no such simple alternative. Use of the overall analysis of variance F -test, prior to these individual comparisons, is often a sufficient safeguard against over-interpretation of the results of such comparisons. Nevertheless, insight may be gained from assessing differences against Tukey’s experimentwise HSD criterion that was discussed in Subsection 4.4.1. > sem <- summary.lm(sugar.aov)$sigma/sqrt(3) > # Alternatively, sem <- 6.33/sqrt(2) > qtukey(p=.95, nmeans=4, df=8) * sem 1] 20.26

# 3 results/trt

Using this stricter criterion, B cannot be distinguished from the control, and A, B and C cannot be distinguished from each other.

7.1.2 Different choices for the model matrix when there are factors

In the language used in the R help pages, different choices of contrasts are available, with each different choice leading to a different model matrix. The different choices thus give different mathematical descriptions for the same model. The coefficients (parameters) change and must be interpreted differently. The fitted values and the analysis of variance table do not change. The choice of contrasts may call for careful consideration, in order to obtain coefficient estimates with the interpretation that is most helpful for the problem in hand. Or, more than one run of the analysis may be necessary, to gain information on all effects of interest. The default (treatment) choice of contrasts uses the initial factor level as baseline, as we have noted. Different choices of the baseline or reference level lead to different versions of the model matrix. The other common choice, i.e., sum contrasts, uses the average of treatment effects as the baseline.

7.1 Levels of a factor – using indicator variables

221

In order to use sum contrasts in place of treatment contrasts, specify options (contrasts=c("contr.sum", "contr.poly")). It is also possible to set contrasts separately for each factor. See help(C). Here is the output when the baseline is the average of the treatment effects, i.e., from using the sum contrasts:2 > oldoptions <- options(contrasts=c("contr.sum", "contr.poly")) > # The mean over all treatment levels is now the baseline. > # (The second setting ("contr.poly") is for ordered factors.) > summary.lm(aov(weight ˜ trt, data = sugar)) . . . . Coefficients: Estimate Std. Error t value Pr(>|t|) (Intercept) 65.37 2.24 29.23 2.0e-09 trt1 17.87 3.87 4.61 0.0017 trt2 -3.53 3.87 -0.91 0.3883 trt3 2.13 3.87 0.55 0.5968 Residual standard error: 7.75 on 8 degrees of freedom Multiple R-Squared: 0.791, Adjusted R-squared: 0.713 F-statistic: 10.1 on 3 and 8 degrees of freedom, p-value: 0.00425 > options(oldoptions)

# Restore default treatment contrasts

Note the differences from the output from the default choice of contrasts. The baseline, labeled (Intercept), is now the treatment mean. This equals 65.37. Remaining coefficients are differences, for Control and for treatment levels A and B, from this mean. The sum of the differences for all three treatments is zero. Thus the difference for C is (rounding up) −(17.87 − 3.53 + 2.13) = −16.5. The estimates (means) are: Control: 65.37 + 17.87 = 83.2. A: 65.37 − 3.53 = 61.8. B: 65.37 + 2.13 = 67.5. C: 65.37 − 16.5 = 48.9. Note also the possibility of using helmert contrasts. For a factor that has two levels, helmert contrasts lead to a parameter estimate that is just half that for treatment contrasts. For factors with more than two levels, the parameter estimates that are associated with helmert contrasts rarely correspond to the scientific questions that are of interest. For ordered factors, polynomial contrasts are the default. There is a brief discussion of contrasts in Section 14.6. 2

The first vector element specifies the choice of contrasts for factors (i.e., unordered factors), while the second specifies the choice for ordered factors.

222

Exploiting the linear model framework

7.2 Block designs and balanced incomplete block designs Data in the data frame rice (DAAG) were displayed in Figure 4.7. They were from an experiment where the plants were laid out in blocks, with each treatment combination occurring once in each block. As all combinations of factors occur equally often in each block, the experimental design is a complete block design. The data in appletaste are from a balanced incomplete block design (BIBD). In this particular BIBD, one treatment is left out of each block, but in such a way that the number of blocks in which a treatment is left out is the same for all treatments. (More generally, the requirement for a BIBD is that all treatments must occur together equally often in the same block.) Blocks should be chosen so that conditions are as uniform as possible within each block. In a glasshouse (or greenhouse) experiment all plants in a single block should be in a similar position in the glasshouse, with a similar exposure to light.

7.2.1 Analysis of the rice data, allowing for block effects

In general, there should be allowance for block differences when data from block designs are analyzed. Otherwise, if there are substantial differences between blocks, treatment effects are likely to be masked by these substantial block differences. The interest is in knowing the extent to which treatment differences are consistent across blocks, irrespective of block-to-block differences that affect all plants in a block pretty much equally. The analysis of variance table is a useful first point of reference, for examining results: > ricebl.aov <- aov(ShootDryMass ˜ Block + variety * fert, data=rice) > summary(ricebl.aov) Df Sum Sq Mean Sq F value Pr(>F) Block 1 3528 3528 10.9 0.0016 variety 1 22685 22685 70.1 6.4e-12 fert 2 7019 3509 10.8 8.6e-05 variety:fert 2 38622 19311 59.7 1.9e-15 Residuals 65 21034 324

This makes it clear that there are substantial differences between blocks. Use summary.lm() to obtain details of the effects: > summary.lm(ricebl.aov) . . . . Coefficients: (Intercept) Block2 varietyANU843 fertNH4Cl fertNH4NO3 varietyANU843:fertNH4Cl varietyANU843:fertNH4NO3

Estimate Std. Error t value Pr(>|t|) 115.33 5.61 20.56 < 2e-16 -14.00 4.24 -3.30 0.0016 -101.00 7.34 -13.75 < 2e-16 -58.08 7.34 -7.91 4.2e-11 -35.00 7.34 -4.77 1.1e-05 97.33 10.39 9.37 1.1e-13 99.17 10.39 9.55 5.4e-14

7.2 Block designs and balanced incomplete block designs

223

Residual standard error: 18 on 65 degrees of freedom Multiple R-Squared: 0.774, Adjusted R-squared: 0.753 F-statistic: 37 on 6 and 65 DF, p-value: <2e-16

The above residual standard error, i.e., 18.0 on 65 degrees of freedom, may be compared with a standard error of 19.3 on 66 degrees of freedom when there is no allowance for block effects.3 Because this was a complete balanced design, the function model.tables() can be used to obtain a summary of treatment effects in a pleasantly laid out form. Any visual summary of results should, at a minimum, include the information given in Figure 4.7. (Do not try to use model.tables() for anything other than complete balanced designs. Even for balanced “incomplete” results such as will now be discussed, results will be incorrect.) 7.2.2 A balanced incomplete block design

In tasting experiments, a number of different products, e.g., wines, are to be compared. If presented with too many different specimens to test, tasters can become confused, even when precautions are taken (including washing the palette) to minimize carry-over effects from one product to another. Hence it is usual to limit the number of products given to any one taster. In the example that will now be given, the products were different varieties of apple, identified by the numerical codes 298, 493, 649, and 937. The 20 tasters were divided into four groups of five. For each group of five tasters, a different product was omitted. Panelists made a mark on a line that gave their rating of aftertaste (0 for extreme dislike; 150 for extreme approval). The following is a summary of the experimental design: > table(appletaste$product, appletaste$panelist)

298 493 649 937

a 1 1 0 1

b 1 1 0 1

c 1 1 0 1

d 1 1 0 1

e 1 1 0 1

f 0 1 1 1

g 0 1 1 1

h 0 1 1 1

i 0 1 1 1

j 0 1 1 1

k 1 0 1 1

l 1 0 1 1

m 1 0 1 1

n 1 0 1 1

o 1 0 1 1

p 1 1 1 0

q 1 1 1 0

r 1 1 1 0

s 1 1 1 0

t 1 1 1 0

The tasters play the role that blocks would play in a field design. In spite of differences in the way that different raters use the scale (some will tend to score low and some high), there may be acceptable consistency in their comparative ratings of the products. For analysis, it is necessary only to specify factors panelist and product as explanatory factors. > sapply(appletaste, is.factor) # panelist & product are factors aftertaste panelist product FALSE TRUE TRUE > summary(appletaste.aov <- aov(aftertaste ˜ panelist + product, + data=appletaste)) 3

## AOV calculations, ignoring block effects rice.aov <- aov(ShootDryMass ˜ variety * fert, data=rice) summary.lm(rice.aov)$sigma

Exploiting the linear model framework

● ●

● ● ● ●● ●●● ● ● ● ●● ●●●● ● ● ● ●● ● ● ● ● ● ●● ● ● ●● ● ● ●● ● ● ●● ● ●● ● ●

a d g

j m

q

t

●

50

●● ●

● ● ● ● ● ● ● ● ● ● ●

0

●

● ● ●

−50

50 0 −50

Partial for panelist

●

Partial for product

224

● ● ● ● ● ● ● ● ● ● ● ● ● ●

● ● ● ● ● ● ● ● ● ●

● ● ● ● ● ● ● ● ● ● ● ●

298

Panelist

649 Product

Figure 7.2 These plots show the respective contributions of the factors panelist and product to aftertaste scores, in an apple tasting experiment.

panelist product Residuals

Df Sum Sq Mean Sq F value Pr(>F) 19 30461 1603 2.21 0.019 3 34014 11338 15.60 1.0e-06 37 26892 727

There are differences between the ratings of different panelists, but they are of minor consequence relative to differences between products. Figure 7.2 shows the partial residual plot, obtained using the function termplot() with the argument partial=TRUE; this gives a useful summary of the results. Notice that ratings seem generally lower for the final few raters. Had product quality deteriorated over time? In its present form, this function is useful only for displaying the effects of factors for which no interaction terms are present. As noted above, do not try to use model.tables() to obtain estimates of effects; the results, for this incomplete block design, will be incorrect. 7.3 Fitting multiple lines Multiple regression can be used to fit multiple lines. In the example that follows (Table 7.3), there are measurements of vapor pressure (vapPress) and of the difference between leaf and air temperature (tempDiff), for three different levels of carbon dioxide. Possibilities we may want to consider are: r Model 1 (constant response): y = a. r Model 2 (a single line): y = a + bx. r Model 3 (three parallel lines): y = a + a z + a z + bx. 1 2 2 3 3 (For the low CO2 group (z2 = 0 and z3 = 0) the constant term is a1 ; for the medium CO2 group (z2 = 1 and z3 = 0) the constant term is a1 + a2 ; while for the high CO2 group (z2 = 0 and z3 = 1) the constant term is a1 + a3 .) r Model 4 (three separate lines): y = a + a z + a z + b x + b z x + b z x. 1 2 2 3 3 1 2 2 3 3 (Here, z2 and z3 are as in model 3 (panel B). For the low CO2 group (z2 = 0 and z3 = 0) the slope is b1 ; for the medium CO2 group (z2 = 1 and z3 = 0) the slope is b1 + b2 ; while for the high CO2 group (z2 = 0 and z3 = 1) the slope is b1 + b3 .)

7.3 Fitting multiple lines

225

Table 7.3 Selected rows, showing values of CO2level, vapPress and tempDiff, from the data set leaftemp.

Table 7.4 Model matrix for fitting three parallel lines (model 3) to the data of Table 7.3. The y-values are in the separate column to the right.

CO2level vapPress tempDiff

(Intercept) Medium High vapPress tempDiff

low low ... medium medium ... high high ...

1.88 2.20 ... 2.38 2.72 ... 2.56 2.55 ...

1.36 0.60 ... 1.94 0.83 ... 1.50 0.85 ...

1 1 ... 1 1 ... 1 1 ...

0 0 ... 1 1 ... 0 0 ...

0 0 ... 0 0 ... 1 1 ...

1.88 2.2 ... 2.38 2.72 ... 2.56 2.55 ...

1.36 0.6 ... 1.94 0.83 ... 1.5 0.85 ...

Table 7.5 Model matrix for fitting three separate lines (model 4), with y-values in the separate column to the right.

(Intercept) 1 1 ... 1 1 ... 1 1 ...

Medium

High

vapPress

Medium: vapPress

High: vapPress

0 0 ... 1 1 ... 0 0 ...

0 0 ... 0 0 ... 1 1 ...

1.88 2.2 ... 2.38 2.72 ... 2.56 2.55 ...

0 0 ... 2.38 2.72 ... 0 0 ...

0 0 ... 0 0 ... 2.56 2.55 ...

tempDiff 1.36 0.6 ... 1.94 0.83 ... 1.5 0.85 ...

Selected rows from the model matrices for model 3 and model 4 are displayed in Tables 7.4 and 7.5, respectively. The statements used to fit the four models are: ## Fit various models to columns of data frame leaftemp (DAAG) leaf.lm1 <- lm(tempDiff ˜ 1 , data = leaftemp) leaf.lm2 <- lm(tempDiff ˜ vapPress, data = leaftemp) leaf.lm3 <- lm(tempDiff ˜ CO2level + vapPress, data = leaftemp) leaf.lm4 <- lm(tempDiff ˜ CO2level + vapPress + vapPress:CO2level, data = leaftemp)

Recall that CO2level is a factor and vapPress is a variable. Technically, vapPress:CO2level is an interaction. The effect of an interaction between a factor and a variable is to allow different slopes for different levels of the factor. The analysis of variance table is helpful in making a choice between these models:

226

Exploiting the linear model framework

Table 7.6 Analysis of variance information. The starting point is a model that has only an intercept or “constant" term. The entries in rows 1–3 of the Df column and of the Sum of Sq column are then sequential decreases from fitting, in turn, vapPress, then three parallel lines, and then finally three separate lines.

vapPress (variable) Three parallel lines Three different lines Residuals

Df

Sum of Sq

Mean square

1

5.272

5.272

11.3

0.0014

2

6.544

3.272

7.0

0.0019

2

2.126

1.063

2.3

0.1112

61

40.000

0.656

F

Pr(
> anova(leaf.lm1, leaf.lm2, leaf.lm3, leaf.lm4) Analysis of Variance Table Model 1: Model 2: Model 3: Model 4: Res.Df 1 61 2 60 3 58 4 56

tempDiff ˜ 1 tempDiff ˜ vapPress tempDiff ˜ CO2level + vapPress tempDiff ˜ CO2level + vapPress + CO2level:vapPress RSS Df Sum of Sq F Pr(>F) 40.00 34.73 1 5.272 11.33 0.0014 28.18 2 6.544 7.03 0.0019 26.06 2 2.126 2.28 0.1112

This is a sequential analysis of variance table. Thus, the quantity in the sum of squares column (Sum of Sq) is the reduction in the residual sum of squares due to the inclusion of that term, given that earlier terms had already been included. The Df (degrees of freedom) column gives the change in the degrees of freedom due to the addition of that term. Table 7.6 explains this in detail. The analysis of variance table suggests use of the parallel line model, shown in panel B of Figure 7.3. The reduction in the mean square from model 3 (panel B in Figure 7.3) to model 4 (panel C) in the analysis of variance table has a p-value equal to 0.1112. The coefficients and standard errors for model 3 are: > summary(leaf.lm3) Call: lm(formula = tempDiff ˜ CO2level + vapPress, data = leaftemp) . . . .

7.3 Fitting multiple lines low

medium

C: Separate lines

Intercepts are: 2.68, 3, 3.48 Slope is −0.84

Intercepts are: 1, 2.85, 4.69 Slopes are −0.02, −0.76, −1.43

●

●

●

1.0

●

0.0

●

●

1.8

●

●

●

●

2.2

2.0 ●

●

2.6

1.4

●

●

●

2.2

●

2.6

●

● ●

● ●

●

● ●

1.8

●

●

●

●●

Vapor pressure

● ●

●

● ●

● ●

●●

1.4

●

●

●

1.0

●

● ●

● ●

●

0.0

●

●

●

●●

0.0

● ●

●

2.0

2.0

●

3.0

3.0

B: Parallel lines

tempDiff = 3.1−0.86 x vapPress

●

1.0

high

A: Single line

3.0

Temperature difference

●

227

1.4

Vapor pressure

●

●

1.8

●

2.2

2.6

Vapor pressure

Figure 7.3 A sequence of models fitted to the plot of tempDiff versus vapPress, for low, medium and high levels of CO2level. Panel A relates to model 2, panel B to model 3, and panel C to model 4.

0.5

1.0

1.5

−2

−1

0

1

2

0.5

1.0

Theoretical quantiles

1.5

2.0

−1

0

1

2

● ● ● ● 43 ● ●● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ●● ● ●● ● ● ● ●● ● ●●● ● ●●● ● ● ●● ●

−2

8 ● ●● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ●● ●● ●● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ● ●● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ●

Standardized residuals

24 ●

0.12 ● ● ●

45 ●

0.12

●7 Cook's distance

−3

1.5 1.0

Resid vs Leverage

7● ●

0.5

2 1 0 ●7

2.0

Fitted values

Standardized residuals

7●

8 ●●

Scale−Location

0.0

8

24 ●

●● ●● ●●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ●●●

−1

● ●● ● ● ●● ● ● ● ●●● ● ● ● ● ● ●●● ● ● ● ●●● ● ● ● ● ●● ● ●● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ●●

Normal Q−Q Standardized residuals

●

−2

0.0 −1.0

●

24 ●

●

−2.0

Residuals

1.0

Resids vs Fitted

0.00

Fitted values

0.04

0.08

0.12

Leverage

Figure 7.4 Diagnostic plots for the parallel line model of Figure 7.3.

Coefficients: Estimate Std. Error t value (Intercept) 2.685 0.560 4.80 CO2levelmedium 0.320 0.219 1.46 CO2levelhigh 0.793 0.218 3.64 vapPress -0.839 0.261 -3.22

Pr(>|t|) 1.16e-05 0.14861 0.00058 0.00213

Residual standard error: 0.69707 on 58 degrees of freedom Multiple R-Squared: 0.295, Adjusted R-squared: 0.259 F-statistic: 8.106 on 3 and 58 degrees of freedom, p-value: 0.000135

The coefficients in the equations for this parallel line model are given in the annotation for Figure 7.3B. For the first equation (low CO2 ), the constant term is 2.685; for the second equation (medium CO2 ), the constant term is 2.685 + 0.320 = 3.005; while for the third equation, the constant term is 2.685 + 0.793 = 3.478. In addition, we examine a plot of residuals against fitted values, and a normal probability plot of residuals (Figure 7.4). These plots seem unexceptional.

Exploiting the linear model framework

20.0

●

●

19.0

●

● ●

50

Model matrix – quadratic fit

Data

18.0

Grains per head

21.0

228

75

100

125

1 2 3 4 5

rate

grain

50 75 100 125 150

21.2 19.9 19.2 18.4 17.9

(Intercept) rate rate2 1 1 1 1 1

50 75 100 125 150

2 500 5 625 10 000 15 625 22 500

150

Seeding rate

Figure 7.5 Plot of number of grains per head versus seeding rate, for the barley seeding rate data shown to the right of the figure, with fitted quadratic curve. The model matrix for fitting a quadratic curve is shown on the far right. Data relate to McLeod (1982).

7.4 Polynomial regression Polynomial regression provides a straightforward way to model simple forms of departure from linearity. The simplest case is where the response curve has a simple cup-up or cupdown shape. For a cup-down shape, the curve has some part of the profile of a path that follows the steepest slope up a rounded hilltop towards the summit and down over the other side. For a cup-up shape the curve passes through a valley. Such cup-down or cup-up shapes can often be modeled quite well using quadratic, i.e., polynomial with degree 2, regression. For this the analyst uses x 2 as well as x as explanatory variables. If a straight line is not adequate, and the departure from linearity suggests a simple cup-up or cup-down form of response, then it is reasonable to try a quadratic regression. The calculations are formally identical to those for multiple regression. To avoid numerical problems, it is often preferable to use orthogonal polynomial regression. Interested readers may wish to pursue for themselves the use of orthogonal polynomial regression, perhaps using as a starting point Exercise 18 at the end of the chapter. Orthogonal polynomials have the advantage that the coefficient(s) of lower-order terms (linear, . . .) do(es) not change when higher-order terms are added. One model fit, with the highestorder term present that we wish to consider, provides the information needed to make an assessment about the order of polynomial that is required. The orthogonal polynomial coefficients must be translated back into coefficients of powers of x (these are not of course independent), if these are required. Figure 7.5 shows number of grains per head (averaged over eight replicates), for different seeding rates of barley. A quadratic curve has been fitted. The code is: ## Fit quadratic curve: data frame seedrates (DAAG) seedrates.lm2 <- lm(grain ˜ rate + I(rateˆ2), data = seedrates) # The wrapper function I() ensures that the result from # calculating rateˆ2 is treated as a variable in its own right. plot(grain ˜ rate, data = seedrates, pch = 16, xlim = c(50, 160), cex=1.4) new.df <- data.frame(rate = (1:14) * 12.5) # for plotting the fitted curve hat2 <- predict(seedrates.lm2, newdata = new.df, interval="predict", coverage = 0.95) lines(new.df$rate, hat2[, "fit"], lty = 2, lwd=2)

7.4 Polynomial regression

229

The quadratic regression appears, from visual inspection, a good fit to the data. The fitted model may be written  y = a + b1 x1 + b2 x2 where x1 = x, and x2 = x 2 . Thus, the model matrix has a column of 1s, a column of values of x, and a column that has values of x 2 . Here is the output from R: > summary(seedrates.lm2, corr=TRUE) Call: lm(formula = grain ˜ rate + I(rateˆ2), data = seedrates) Coefficients: (Intercept) rate I(rateˆ2)

Value Std. Error t value Pr(>|t|) 24.060 0.456 52.799 0.000 -0.067 0.010 -6.728 0.021 0.000 0.000 3.497 0.073

Residual standard error: 0.115 on 2 degrees of freedom Multiple R-Squared: 0.996 F-statistic: 256 on 2 and 2 degrees of freedom, the p-value is 0.0039 Correlation of Coefficients: (Intercept) rate rate -0.978 I(rateˆ2) 0.941 -0.989

(In a model formula, rateˆ2 will be interpreted as rate:rate = rate. Hence the use of I(rateˆ2) to denote the square of rate.) Observe the high correlations between the coefficients. Note in particular the large negative correlation between the coefficients for rate and I(rateˆ2). Forcing the coefficient for rate to be high would lead to a low coefficient for I(rateˆ2), and so on. In orthogonal polynomial regression, the separate terms rate and I(rateˆ2) are replaced by the single term poly(rate,2), i.e., an orthogonal polynomial of degree 2 in rate. The fitted values will be identical, but the coefficients are then coefficients of the orthogonal polynomials, not coefficients of rate and I(rateˆ2).

7.4.1 Issues in the choice of model

The coefficient of the x 2 term in the quadratic model fell short of statistical significance at the 5% level. Fitting the x 2 leaves only two degrees of freedom for error. For prediction, our interest is likely to be in choosing the model that is on balance likely to give the more accurate predictions; for this, use of the model that includes the quadratic term may be preferred.

Exploiting the linear model framework

20

●

●

19

●

18

●

17

●

16

Grains per head

21

22

230

60

80

100

120

140

160

180

Seeding rate

Figure 7.6 Number of grains per head versus seeding rate, with fitted line (solid) and fitted quadratic curve (dashed). Also shown are 95% pointwise confidence bounds.

Figure 7.6 shows both a fitted line and a fitted curve, in both cases with 95% confidence bounds.4 It shows a quadratic curve (dashed line) as well as a line (solid line). In addition, the graph shows 95% pointwise confidence bounds for the expected number of grains per head, both about the line and about the curve. The curve is a better fit to the data than the line. For a short distance beyond the final data point, it almost certainly gives a better estimate than does the line. Notice that the confidence bounds for the curve are much wider, beyond a rate of about 160, than the line. Not only does the line almost certainly give a biased estimate, it also gives unrealistically narrow bounds for that estimate. If the model is wrong, it will give wrong estimates of predictive accuracy. This is especially serious for extrapolation beyond the limits of the data. Beyond the limits of the data, it would be unwise to put much trust in either the line or the curve. Our point is that the bounds for the quadratic curve do better reflect uncertainty in the curve that ought to be fitted. We could try other, single-parameter, models. Selecting a model from a number of choices that allow for the curvature may not however be much different, in its effect on the effective degrees of freedom, from adding an x 2 term. The wider confidence bounds for the quadratic model reflect this uncertainty in choice of model, better than results from any individual model that has one parameter additional to the intercept. We can in fact fit the data well by modeling grain as a linear function of log(rate). This model seems intuitively more acceptable; the fitted value of grain continues to 4

## Fit line, fit curve, determine pointwise bounds, and create the plots CIcurves
7.5 ∗ Methods for passing smooth curves through data

231

decrease as the rate increases beyond the highest rate used in the experiment. This is perhaps the model that we should have chosen initially on scientific grounds. 7.5∗ Methods for passing smooth curves through data In the previous section, we used the linear model framework to fit a curve that had x and x 2 , etc. terms. This framework can be adapted to fit higher-order polynomial curves to regression data. For a polynomial of degree m, the model matrix must have, in addition to a column of 1s, columns that hold values of x, x 2 , . . . , x m . Polynomials can be effective when a curve of degree m equal to 2 or 3 is appropriate. Polynomial curves where m is greater than 3 can be problematic. High-degree polynomials tend to move up and down between the data values in a snake-like manner. Splines, or piecewise polynomials, which we now consider, are usually preferable to polynomials of degree greater than 3. A spline curve joins two or more polynomial curves, and is sometimes called a piecewise polynomial curve. The locations of the joints are called knots. The following is a simple piecewise linear spline function, with a knot at x = 2: y = 3 + 4x − 5(x − 2)I{x≥2} .

(7.1)

The indicator I takes the value 1, when x ≥ 2 and the value 0, when x < 2. For values of x less than 2, this spline function behaves as a straight line with slope 4, but for x greater than 2, the slope switches to 4 − 5 = −1. More generally, spline functions take the form y = b0 P0 (x) + b1 P1 (x) + · · · + bk Pk (x) where the bi s are constant coefficients, and the functions Pi (x) are either polynomial functions like the first two terms of equation (7.1), or polynomial functions multiplied by indicators like the last term in equation (7.1). Given a set of knots, there are many different ways to choose the Pi (x). Some choices that are useful in practice may involve relatively complicated forms of algebraic expression that are different from those used in our definition. The Pi (x) are known as basis functions. A particularly convenient set of basis functions for splines is referred to as B-splines. Given data of the form (x1 , y1 ), (x2 , y2 ), . . . , (xn , yn ), we can write a regression model in terms of these B-spline functions as: y = b0 B0 (x) + b1 B1 (x) + · · · + bk Bk (x) + ε where ε represents an error term as in the previous chapter. Written in this form, it is clear that a spline regression model can be fit using the same methods as for fitting multiple regression models. The columns of the model matrix are constructed from evaluating each of the B-splines at each of the values of x. Once the basis functions have been evaluated, the lm() function can be used to carry out the estimation of the regression coefficients. Natural splines, implemented using the function ns() from the splines package, are an alternative to B-splines. For natural splines, the slope of the curve is constrained to be constant at and beyond the boundary knots. For B-splines, there are no boundary constraints, though there are boundary knots that anchor the spline basis. By default, these are placed at the limits of the data.

232

Exploiting the linear model framework

Table 7.7 Resistance (ohms) versus apparent juice content. The table shows a selection of the data.

1 2 3 4 5 ...

Juice (%)

Ohms

4 5 5.5 7.5 8.5 ...

4860 5860 6650 7050 5960 ...

33 34 35 36 37 ...

Juice (%)

Ohms

20 20.5 21.5 21.5 21.5 ...

7500 8500 5600 6950 7200 ...

65 66 67 68 69 ...

Juice (%)

Ohms

41.5 42.5 43 43 43 ...

3350 2700 2750 3150 3250 ...

123 124 125 126 127 128

Juice (%)

Ohms

58.5 58.5 58.5 59.5 60 9

3650 3750 4550 3300 3600 9850

The number of degrees of freedom (df) additional to the intercept will be specified for the spline curve, with the software then allowed to determine the number and location of internal join points (=knots). For B-splines, the number of internal knots is df - degree, where degree is the degree (by default 3) of the piecewise polynomial. For natural splines, the number of internal knots is df - degree + 2, where degree must be 3. (The difference in df is a result of the boundary constraints.) In either case, internal knots are by default placed at equally spaced quantiles of the data. The use of regression splines, using B-spline or N-spline bases, will be demonstrated in the next subsection. The discussion will then move to more general types of smoothing terms. We will take a simple example where there is just one explanatory variable, and try several different methods on it.

7.5.1 Scatterplot smoothing – regression splines

We have (in Table 7.7) the apparent juice content and resistance (in ohms) for 128 slabs of fruit (these data relate to Harker and Maindonald, 1994). Figure 7.7 shows four different curves fitted to these data:5 Figures 7.7A and B show spline curves, the first with one knot and the second with two knots. Figures 7.7C and D show, for comparison, third- and fourth-degree polynomials. The polynomials do quite well here relative to the splines. Also shown are 95% pointwise confidence intervals for the fitted curves. Diagnostic plots can be used, just as for the models considered in earlier chapters, to highlight points that are associated with large residuals, or that are having a strong influence

5

## Fit various models to columns of data frame fruitohms (DAAG) library(splines) ## Panel A plot(ohms ˜ juice, cex=0.8, xlab="Apparent juice content (%)", ylab="Resistance (ohms)", data=fruitohms) CIcurves(form=ohms ˜ ns(juice, 2), data=fruitohms, newdata=data.frame(juice=pretty(fruitohms$juice,20))) ## For panels B, C, D replace form = ohms ˜ ns(juice,2) by: ## form = ohms ˜ ns(juice,3) # panel B: nspline, df = 4 ## ohms ˜ poly(juice,2) # panel C: polynomial, df = 3 ## ohms ˜ poly(juice,3) # panel D: polynomial, df = 4 # For more information on poly(), see help(poly) and Exercise 15.

10000

A: N−spline, 1 internal knots (d.f. = 2+1) ● ●

● ●

● ●

●

2000

●

● ● ● ● ● ● ●●● ● ● ● ● ● ● ●● ● ●● ●● ●●● ● ● ●● ● ●● ●● ● ● ● ● ● ● ● ●● ● ● ●● ● ● ● ●● ● ● ● ● ● ● ● ●● ●● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ●

● ●

●

●● ● ●● ● ● ●●● ●● ● ● ● ● ● ●● ● ● ●● ● ● ● ● ● ●● ● ● ●● ●

● ●

● ●

●

●

●

10000

●

●

6000

●

● ● ●

● ●

2000

●

●

● ● ● ● ● ●●● ● ● ● ● ● ●● ●● ● ●● ●●● ● ● ● ●● ●● ● ● ● ● ● ●● ● ● ● ●● ●● ● ● ● ● ●● ● ● ● ● ● ● ● ●● ●● ● ● ● ●● ● ● ● ● ● ● ● ●● ● ● ●

D: Polynomial (d.f. = 3+1) ● ●

●

●● ● ●● ● ● ●●● ●● ● ● ● ● ● ●● ● ● ●● ● ● ● ● ● ●● ● ●● ● ●

●

● ●

●

2000

10000 6000

Resistance (ohms)

10 20 30 40 50 60

C: Polynomial (d.f. = 2+1)

●● ● ●● ● ● ●●● ●● ● ● ● ● ● ●● ● ● ●● ● ● ● ● ● ●● ● ●● ●

● ● ● ● ● ● ●●● ● ● ● ● ● ● ●● ● ●● ●● ●●● ● ● ●● ● ●● ●● ● ● ● ● ● ● ● ●● ● ● ●● ● ● ● ●● ● ● ● ● ● ● ● ●● ●● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ●

10 20 30 40 50 60

●

233

B: N−spline, 2 internal knots (d.f. = 3 + 1)

2000

●

6000

●

●● ● ●● ● ● ●●● ●● ● ● ● ● ● ●● ● ● ●● ● ● ● ● ● ●● ● ● ●●

6000

Resistance (ohms)

10000

7.5 ∗ Methods for passing smooth curves through data

●

●

● ● ● ● ● ● ● ●●● ● ● ● ● ● ●● ●● ● ●● ●●● ● ● ● ●● ●● ● ● ● ● ● ●● ● ● ● ●● ●● ● ● ● ● ●● ● ● ● ● ● ● ● ● ●● ●● ● ● ●● ● ● ● ● ● ● ● ●● ● ● ● ●

10 20 30 40 50 60

10 20 30 40 50 60

Figure 7.7 Different smooth curves fitted to the data of Table 7.7. The dashed lines show 95% pointwise confidence bounds for the fitted curve. In panels A and B, vertical lines show the locations of the knots. The degrees of freedom (“df” or “degree”) shown are those supplied to ns() or poly(). These must in each case be increased by one to allow for the intercept.

3000

5000

−2 −1

Fitted values

Figure 7.8

0

1

2

Theoretical quantiles

3000

5000

7000

● ● 128 ●● ●● ● ●● ● ● ● ●●●● ● ● ● ● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ● ● ● ● ●● ● ●● ● ● ● ● ● ●● ●●● ● ●●●●● ● ● ●● ● ● ● ●● ● ● ● ● ● ● ● ●●●● ●● ●● ●● ●● ● ● ●● ● ● ●● ●● ● ●● ● ● ● ● ●● ● ●

2

● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ●●● ● ●● ●● ● ● ● ● ●● ●● ●● ●● ● ● ● ●●● ● ● ● ● ●● ● ● ● ● ● ● ●● ●●● ● ●●●● ● ● ● ● ● ● ● ● ●●● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ●● ● ● ●

Resids vs Leverage

0

●

−2

1●

● 34

Standardized residuals

1.0

1.5

● 52

● Cook's 52

−4

●1 ● 52

Scale−Location

0.5

1 2 3

7000

● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ●

Standardized residuals

1●

● 52

34 ●

0.0

0

● ● ●● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ●●●● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●●● ● ●● ●●● ● ● ● ●● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●

Normal Q−Q

−1

●

Standardized residuals

●

−3

2000

● 34

−4000

Residuals

Resids vs Fitted

0.00

0.02

Fitted values

0.12

0.12

●

1 distance

0.04

0.06

Leverage

Diagnostic plots for the fitted model given in Figure 7.7A.

on the curve. Figure 7.8 shows the default diagnostic plots for the fitted model shown in Figure 7.7A.6 Apart from the large residual associated with point 52 (at 32.5% apparent juice content), these plots show nothing of note. The curves have bent to accommodate points near the extremes that might otherwise have appeared as outliers. Here is the summary information: > summary(fruit.lm2) . . . . 6

## Fit degree 2 normal spline, plot diagnostics par(mfrow = c(2,2)) fruit.lm2 <- lm(ohms ˜ ns(juice,2), data=fruitohms) # for panel B: ns(juice,3) plot(fruit.lm2) par(mfrow = c(1,1))

234

Exploiting the linear model framework A: Degree 2 N−spline

B: Degree 3 N−spline

● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ●● ●● ● ●●● ● ●● ● ●● ● ● ● ● ●● ● ● ● ● ● ● ●● ●● ● ● ● ●● ● ●● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ●● ● ● ● ● ● ●● ● ● ● ● ● ● ●● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ●● ● ●● ●● ●● ●●● ● ● ●● ● ● ● ●●

0.4 0.2 0.0

−8596

● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ●● ● ● ● ●● ● ● ● ● ● ● ● ● ● ●● ●●● ● ● ●● ● ●● ●● ●● ● ●● ●● ●● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ●●● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ●● ● ●● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ●● ● ●● ●● ●● ● ●● ● ● ● ● ●● ● ● ●●

Intercept = 7569

0.6 0.4 0.2 0.0 −0.2 −0.4

−2569 −6329

−4535

3000

5000

5000

7000

−0.2

−2105

7000

0.6

3000

Fitted curve (ohms)

Spline basis functions

Intercept = 8041

10

20

30

40

50

Apparent juice content

60

10

20

30

40

50

60

Apparent juice content

Figure 7.9 Panel A shows the N-spline basis curves (one knot) fitted in Figure 7.7A. The fitted curve, obtained by multiplying the values by −8596 and −2108, respectively, summing, and adding an intercept of 8041, is shown below. Panel B (two knots) relates, similarly, to Figure 7.7B.

Coefficients: (Intercept) ns(juice, 2)1 ns(juice, 2)2

Estimate Std. Error t value Pr(>|t|) 8041 260 30.9 < 2e-16 -8596 578 -14.9 < 2e-16 -2105 357 -5.9 3.3e-08

Residual standard error: 1030 on 125 degrees of freedom Multiple R-Squared: 0.701, Adjusted R-squared: 0.696 F-statistic: 146 on 2 and 125 DF, p-value: <2e-16

Attention is best focused on the fitted curve, ignoring the fact that the curve can be constructed by the smooth joining of separate cubic curves, or as a linear combination of basis functions. Nevertheless, comments that will help make sense of the coefficients and standard errors in the R output may be helpful. To help understand how the curve has been formed as a linear combination of basis functions, we plot graphs that show the curves for which these are the coefficients. For this, we plot the relevant column of the X-matrix against x, and join up the points, as in Figure 7.9.7 7

## Display the basis curves as in panel A mm2 <- model.matrix(fruit.lm2) ylim <- range(mm2[, -1]) plot(mm2[, 2] ˜ juice, ylim=ylim, xlab="Apparent Juice Content (%)", ylab="Spline basis functions", type="l", data=fruitohms) lines(fruitohms$juice, mm2[, 3], col="gray") # NB: Values of juice are already ordered ## For panel B basis curves: mm3 <- model.matrix(fruit.lm3), etc.

7.5 ∗ Methods for passing smooth curves through data

235

Looking back again at the coefficients, basis curve 2 (with a coefficient of −8596) seems more strongly represented than the other basis curve. Compare sets of N-spline basis functions with sets of polynomial basis functions. An orthogonal polynomial basis gives successively more accurate approximations to the data. A set of polynomial basis functions of degree 3 is the set of degree 2 with another basis function added. For the splines this is not the case. All the basis functions change if another degree of freedom is added. For given degrees of freedom, the prediction that they combine to give is accurate. If one basis function is dropped from the set of basis functions, predictions will be disastrously inaccurate. Instead, it is necessary to find a complete new set of basis functions for a spline curve with the reduced degrees of freedom. The regression splines that we described above are attractive because they fit easily within a linear model framework, i.e., we can fit them by specifying an appropriate Xmatrix. There are a wide variety of other methods, most of which do not fit within the linear model framework required for use of lm(). 7.5.2∗ Roughness penalty methods and generalized additive models

In the use of regression splines in the previous subsection, the approach was to specify the degrees of freedom for the spline curve; knots were then located at equal quantiles of the data. This is unlikely to be optimal. Use of too few knots can lead to a curve that fails to capture all of the nuances of the regression function while choosing too many can result in excessive bumpiness, i.e., the details of the curve capture noise. The function gam() in the mgcv package implements generalized additive models (GAMs). The function s() is used to generate smoothing terms. A brief explanation will now be provided of some of the methods that are available, in the mgcv package, for generating smoothing terms. For more extended information, attach the mgcv package, and type help(smooth.terms). The roughness penalty approach is designed to reduce or remove arbitrariness that can result from the choice of the number and placement of knots. Cubic smoothing spline methods assign a knot to each predictor value, while applying a roughness penalty that constrains the fitted spline to smoothly pass through the cloud of observations. Note that without such a constraint, the spline would interpolate the observations, usually rendering a rough curve. In order to reduce the time and memory requirements, knots can be placed at a subset of the predictor values. Even so, there will be many more knots than would be used in the absence of a roughness penalty. The “cubic regression splines” that are provided in the mgcv package place knots at equally spaced quantiles of the data, as does the function smooth.spline() (stats package). Penalized splines, as implemented by pspline() in the survival package, are another approach that places knots at values that are evenly spaced through the data. Thin plate splines circumvent the overt choice of knots. The roughness penalty is, on its own, enough to determine a set of basis functions. The basis functions do, moreover, plausibly give successively more accurate approximations. They generalize in a natural manner to the fitting of smooth surfaces, in an arbitrary number of dimensions. In practice, in order to keep computational demands within reason, a low rank approximation is used,

236

Exploiting the linear model framework

resulting in a much reduced number of basis functions. Wood (2006) calls these thin plate regression splines. The fruitohms data set furnishes examples: library(mgcv) ## Thin plate regression splines (bs="tp") fruit.tp <- gam(ohms ˜ s(juice, bs="tp"), data=fruitohms) ## Plot points, fitted curve, and +/- 1SE limits plot(fruit.tp, residuals=TRUE) ## Cubic regression splines (bs="cr") fruit.cr <- gam(ohms ˜ s(juice, bs="cr"), data=fruitohms) ## Plot points, fitted curve, and +/- 1SE limits plot(fruit.cr, residuals=TRUE)

Note also the gam package, which ports to R the code used for the gam() function in S-PLUS.

7.5.3 Distributional assumptions for automatic choice of roughness penalty

The residual sum of squares is increased by a roughness (or wiggliness) penalty that is a multiple λ of the integral of the squared second derivative, to give a penalized sum of squares. Parameter estimates (multiples of the basis terms) are chosen to minimize this penalized residual sum of squares. The gam() function’s default method for choosing λ is a variant of generalized crossvalidation. This relies on an analytical approximation to the expected value of the crossvalidation estimate of the residual sum of squares. All automatic methods for choosing the penalty assume that errors are independently and identically distributed (i.i.d.), with implications that users will do well to keep in mind. Sequential correlation structures, which are the major focus of Chapter 9, are a common type of departure from i.i.d. errors. Where there is a sequential correlation structure in the data, the methodology will – if possible – use a smooth curve to account for it. The pattern that is thus extracted will not be reproducible under a re-run of the process. Exercise 19 at the end of the chapter is designed to illustrate this point.

7.5.4 Other smoothing methods

Lowess curves are a popular alternative to spline curves. Figure 2.6 showed a curve that was fitted to the fruitohms data using the function lowess(), which always uses a resistant form of smoothing. The curve is thus relatively insensitive to large residuals. Special steps are taken to avoid distortions due to end effects. As implemented in R, lowess() is not available for use when there are multiple explanatory variables, and there is no mechanism (or theory) for calculating pointwise confidence bounds. The loess() function is an alternative to lowess() that is able to handle multidimensional smoothing. The default for loess() is a non-resistant smooth; for a resistant smooth, specify family=symmetric.

6000

237

● ● ● ●● ● ●● ● ● ● ●●●●● ● ● ● ● ● ●●●● ●● ● ● ● ● ● ●● ● ● ● ● ●

● ●

2000

Resistance (ohms)

10000

7.5 ∗ Methods for passing smooth curves through data

● ● ● ● ● ●● ● ● ● ● ●●● ●● ● ● ● ●● ● ● ● ● ●●●● ●● ●● ●● ●● ● ● ● ● ● ●● ● ●● ● ●● ●● ●● ●●● ● ● ● ● ● ●● ●●● ● ●●● ● ● ● ●● ● ●● ● ● ● ●

10 20 30 40 50 60 Apparent juice content (%)

Figure 7.10 and 7.9.

A monotonic decreasing spline curve has been fitted to the points shown in Figures 7.7

Both lowess() and loess() implement a locally weighted regression methodology. The following description is directly relevant to lowess(); calculations for loess() follow the same general pattern. The method is said to be local, because the fitted value m(x) at a point x uses only the data within a specified neighborhood of x. Points nearest to x are given highest weight. Those farther away are given little or no weight. Outlier resistance is achieved by assigning low weight to observations which generate large residuals; this allows for curves which are relatively unaffected by the presence of outliers. An iterative method is used, with the residual at the previous iteration determining the weight at the current iteration. Kernel smoothing methods further widen the range of possibilities. For example, see the documentation for the function locpoly() in the KernSmooth package. On lowess smoothing, see Cleveland (1981). There is a useful brief discussion of smoothing methods in Venables and Ripley (2002) and a fuller discussion of kernel smoothing, splines, and lowess in Fan and Gijbels (1996). See also Hall (2001). ∗

Monotone curves

Constraints can be included that force curves to be monotone increasing or monotone decreasing. The function monoproc() in the monoProc package can be used, starting with a fit using loess() or another function whose output follows the same conventions, to create a monotone fit, as in Figure 7.10. The code is: library(monoProc) fit.mono <- monoproc(loess(ohms˜juice, data=fruitohms), bandwidth=0.1, mono1="decreasing", gridsize=30) plot(ohms ˜ juice, data=fruitohms, xlab="Apparent juice content (%)", ylab="Resistance (ohms)") lines(fit.mono)

238

Exploiting the linear model framework

Table 7.8 Average dewpoint (dewpt), for available combinations of monthly averages of minimum temperature (mintemp) and maximum temperature (maxtemp). The table shows a selection of the data. maxtemp

mintemp

dewpt

18 18 20 20 20 ...

8 10 6 8 10 ...

7 10 5 7 9 ...

mintemp

dewpt

38 40 40 40 40 40

26 18 20 22 24 26

20 5 8 11 14 17

0 −5 −15

−5

0

5

5

10

10

67 68 69 70 71 72

−15

Contribution to predicted dewpoint

1 2 3 4 5 ...

maxtemp

10

15 20 mintemp

25

20

25 30 35 maxtemp

40

Figure 7.11 Representation of average dewpoint (dewpt) as the sum of an effect due to minimum temperature (mintemp), and an effect due to maximum temperature (maxtemp). (Data are from Table 7.8.) The dashed lines are 95% pointwise confidence bounds.

7.6 Smoothing with multiple explanatory variables Attention will now move to models with multiple explanatory variables. Table 7.8 has data on monthly averages of minimum temperature, maximum temperature, and dewpoint. For the background to these data, see Linacre (1992), Linacre and Geerts (1997). The dewpoint is the maximum temperature at which the relative humidity reaches 100%. Monthly data were obtained for a large number of sites worldwide. For each combination of minimum and maximum temperature, the average dewpoint was then determined.

7.6.1 An additive model with two smooth terms

Figure 7.11 shows a representation of these data using an additive model with two splinesmoothing terms. A simplified version of the code used for the fit and for the graph is: ## Regression of dewpt vs maxtemp: data frame dewpoint (DAAG) library(mgcv) ds.gam <- lm(dewpt ˜ s(mintemp) + s(maxtemp), data=dewpoint) oldpar <- par(mfrow = c(1,2), pty="s")

7.6 Smoothing with multiple explanatory variables 20

25

mintempRange

30

35

239

40

mintempRange

mintempRange ●

1.0

Residual

● ●

●

0.5

● ● ●

0.0 ●

● ●

● ●

● ●

●

● ● ●

● ● ● ●

● ●

● ● ●

●

●

●

● ● ●

●

−0.5

●

●

●

● ●

● ●

●

●

● ●

●

●

● ●

●

● ●

● ● ● ●

● ● ● ●

● ● ●

● ● ● ● ●

● ● ●

● ● ● ●

● ●

● ● ● ●

●

● ● ●

●

● ●

● ● ●

●

● ● ● ●

● ● ●

● ● ●

● ● ● ●

● ● ● ●

● ● ● ●

● ● ●

●

20

25

30

35

40

20

25

30

35

40

Maximum temperature

Figure 7.12 Plot of residuals against maximum temperature, for three different ranges of values of minimum temperature. Panel strips are shaded to show the range of values of the conditioning variable. plot(ds.gam, se=2) # se=2: Show 2SE limits ## Try also: plot(ds.gam, se=2, residuals=TRUE, pch=1, cex=0.4) par(oldpar)

We can write the model as y = µ + f1 (x1 ) + f2 (x2 ) + ε where y = dewpt, x1 = maxtemp, and x2 = mintemp. Here µ is estimated by the mean of y, so that the estimates of f1 (x1 ) and f2 (x2 ) give differences from this overall mean. In Figure 7.11, both f1 (x1 ) and f2 (x2 ) are modeled by spline functions with five degrees of freedom. The left panel is a plot of the estimate of f1 (x1 ) against x1 , while the right panel plots f2 (x2 ) against x2 . There is no obvious reason why the additive model should work so well. In general, we might expect an interaction term, i.e., we might expect that f1 (x1 ) would be different for different values of x2 , or equivalently that f2 (x2 ) would be different for different values of x1 . Even where the effects are not additive, an additive model is often a good starting approximation. We can fit the additive model, and then check whether there are departures from it that require examination of the dependence of y upon x1 and x2 jointly. One check is to take, e.g., for x1 = mintemp, three perhaps overlapping ranges of values, which we might call “low”, “medium”, and “high”. For this purpose we are then treating mintemp as a conditioning variable. We then plot residuals against 5u = maxtemp for each range of values of x1 , as in Figure 7.12. If there is a pattern in these plots that changes with the range of the conditioning variable, this is an indication that there are non-additive effects that require attention.8 8

## Residuals vs maxtemp, for different mintemp ranges library(lattice) mintempRange <- equal.count(dewpoint$mintemp, number=3) xyplot(residuals(ds.lm) ˜ maxtemp | mintempRange, data=dewpoint, aspect=1, layout=c(3,1), type=c("p","smooth"), xlab="Maximum temperature", ylab="Residual")

240

Exploiting the linear model framework

7.6.2∗ A smooth surface

If it is suspected that the additive model is inappropriate, an alternative is to use a thin plate regression spline basis for mintemp and maxtemp jointly. For this, specify: ds.tp <- gam(dewpt ˜ s(mintemp, maxtemp), data=dewpoint) vis.gam(ds.tp, plot.type="contour") # gives a contour plot of the # fitted regression surface vis.gam(ds.gam, plot.type="contour") # cf. model with 2 smooth terms

Three-dimensional perspective plots can also be obtained with the argument plot.type="persp".

7.7 Further reading There is a review of the methodologies we have described, and of extensions, in Venables and Ripley (2002). See also references on the help pages for functions in the locfit and mgcv packages. Eubank (1999) gives a comprehensive and readable introduction to the use of splines in non-parametric regression. Maindonald (1984) has an elementary introduction to B-splines, starting with piecewise linear functions. Faraway (2006) is a wide-ranging and practically oriented account that starts with Generalized Linear Models. Wood (2006) is even more wide-ranging, with stronger technical demands. It has extensive coverage of Linear Models and Generalized Linear Models, then proceeding to an account of Generalized Additive Models and Generalized Additive Mixed Models that makes heavy use of various forms of spline bases.

References for further reading

Eubank, R. L. 1999. Nonparametric Regression and Spline Smoothing, 2nd edn. Faraway, J. J. 2006. Extending the Linear Model with R. Generalized Linear, Mixed Effects and Nonparametric Regression Models. Hastie, T., Tibshirani, R. and Friedman, J. 2009. The Elements of Statistical Learning. Data Mining, Inference and Prediction, 2nd edn. Venables, W. N. and Ripley, B. D. 2002. Modern Applied Statistics with S, 4th edn. Wood, S. N. 2006. Generalized Additive Models. An Introduction with R.

7.8 Exercises 1. Reanalyze the sugar weight data of Subsection 7.1.1 using log(weight) in place of weight. 2. Use anova() to compare the two models: roller.lm <- lm(depression˜weight, data=roller) roller.lm2 <- lm(depression˜weight+I(weightˆ2), data=roller) Is there any justification for including the squared term? 3. Use the method of Section 7.3 to compare, formally, the regression lines for the two data frames elastic1 and elastic2 from Exercise 1 in Chapter 5.

7.8 Exercises

241

4. The data frame toycars consists of 27 observations on the distance (in meters) traveled by one of three different toy cars on a smooth surface, starting from rest at the top of a 16-inch-long ramp tilted at varying angles (measured in degrees). Because of differing frictional effects for the three different cars, we seek three regression lines that relate distance traveled to angle. (a) As a first try, fit three lines that have the same slope but different intercepts. (b) Note the value of R 2 from the summary table. Examine the diagnostic plots carefully. Is there an influential outlier? How should it be treated? (c) The physics of the problem actually suggests that the three lines should have the same intercept (very close to 0, in fact), and possibly differing slopes, where the slopes are inversely related to the coefficient of dynamic friction for each car. Fit the model, and note that the value of R 2 is slightly lower than that for the previously fitted model. Examine the diagnostic plots. What has happened to the influential outlier? In fact, this is an example where it is inadvisable to take R 2 too seriously; in this case, a more carefully considered model can accommodate all of the data satisfactorily. Maximizing R 2 does not necessarily give the best model! 5. The data frame cuckoos holds data on the lengths and breadths of eggs of cuckoos, found in the nests of six different species of host birds. Fit models for the regression of length on breadth that have: A: a single line for all six species. B: different parallel lines for the different host species. C: separate lines for the separate host species. Use the anova() function to print out the sequential analysis of variance table. Which of the three models is preferred? Print out the diagnostic plots for this model. Do they show anything worthy of note? Examine the output coefficients from this model carefully, and decide whether the results seem grouped by host species. How might the results be summarized for reporting purposes? 6. Fit the three models A, B and C from the previous exercise, but now using the robust regression function rlm() from the MASS package. Do the diagnostic plots look any different from those from the output from lm()? Is there any substantial change in the regression coefficients? 7. Apply polynomial regression to the seismic timing data in the data frame geophones. Specifically, check the fits of linear, quadratic, cubic, and quartic (degree = 4) polynomial estimates of the expected thickness as a function of distance. What do you observe about the fitted quartic curve? Do any of the fitted curves capture the curvature of the data in the region where distance is large? 8. Apply spline regression to the geophones data frame. Specifically, regress thickness against distance, and check the fits of 4-, 5- and 6-degree-of-freedom cases. Which case gives the best fit to the data? How does this fitted curve compare with the polynomial curves obtained in the previous exercise? Calculate pointwise confidence bounds for the 5-degree-of-freedom case. 9. In the data frame worldRecords (DAAG): (i) fit log(Time) as a linear function of log(Distance); (ii) fit log(Time) as a polynomial of degree 4 in log(Distance); (iii) fit log(Time) as a natural spline function of degree 4 in log(Distance). (a) Use anova() to compare the fits (i) and (ii). (b) Compare the R 2 statistics from the fits (i), (ii) and (iii). Do they convey useful information about the adequacy of the models?

242

Exploiting the linear model framework (c) For each of (i), (ii) and (iii), plot residuals against log(Distance). Which model best accounts for the pattern of change of time with log(Distance)? For what range(s) of distances does there seem, for all three models, to be some apparent residual bias?

10. Apply lowess() to the geophones data as in the previous two exercises. You will need to experiment with the f argument, since the default value oversmooths this data. Small values of f (less than 0.2) give a very rough plot, while larger values give a smoother plot. A value of about 0.25 seems a good compromise. 11. Check the diagnostic plots for the results of Exercise 8 for the 5-degree-of-freedom case. Are there any influential outliers? 12. Continuing to refer to Exercise 8, obtain plots of the spline basis curves for the 5-degree-offreedom case. That is, plot the relevant column of the model matrix against y. 13. Apply the penalized spline to the geophones data using the default arguments in mgcv’s s() function. Is this a satisfactory fit to the data? The argument k controls the number of knots. Try setting k to 20, and examine the fit. How might an “optimal” value of k be selected? 14. The ozone data frame holds data, for nine months only, on ozone levels at the Halley Bay station between 1956 and 2000. (See Christie (2000), Shanklin (2001) for the scientific background.) Up-to-date data are available from the web site given under help(ozone, package="DAAG"). Replace zeros by missing values. Determine, for each month, the number of missing values. Plot the October levels against Year, and fit a smooth curve. At what point does there seem to be clear evidence of a decline? Plot the data for other months also. Do other months show a similar pattern of decline? 15. The wages1833 data frame holds data on the wages of Lancashire cotton factory workers in 1833. Plot male wages against age and fit a smooth curve. Repeat using the numbers of male workers as weights. Do the two curves seem noticeably different? Repeat the exercise for female workers. [See Boot and Maindonald (2008) for background information on these data.] 16. Clutton-Brock et al. (1999) studied how the time that adult meerkats spent on guarding varied with the size of the group. (Studies were conducted in the Kalahari Gemsbok Park in South Africa.) Approximate percentages of time were: Group size 1: 50,47; 2: 26; 3: 26; 4: 24,23; 5: 19; 6: 13; 7: 3. (NB: These numbers were read off from a graph.) Model the percentage of time as a function of group size. 17. From the data set cricketers extract the subset for which year (year of birth) is in the range 1840 to 1960, inclusive. Fit the following: (a) A polynomial of degree 2 in year. (b) A polynomial of degree 3 in year. Plot the fitted curves on a graph of proportion left-handed versus year of birth. Does the polynomial of degree 3 give any worthwhile improvement over a polynomial of degree 2? Compare also with a regression B-spline of degree 3; i.e., bs(year, 3). 18.∗ Compare the two results: seedrates.lm <- lm(grain ˜ rate + I(rateˆ2), data=seedrates) seedrates.pol <- lm(grain ˜ poly(rate,2), data=seedrates) Check that the fitted values and residuals from the two calculations are the same, and that the t-statistic and p-value are the same for the coefficient labeled poly(rate, 2)2 in the

7.8 Exercises

243

polynomial regression as for the coefficient labeled I(rateˆ2) in the regression on rate and rateˆ2. Check that the coefficients remain the same if, in the calculation of seedrates.lm, rate + I(rateˆ2) is replaced by poly(rate, 2, raw=TRUE). Regress the second column of model.matrix(seedrates.pol) on rate and I(rateˆ2), and similarly for the third column of model.matrix(seedrates.pol). Hence, express the first and second orthogonal polynomial terms as functions of rate and rateˆ2. 19.∗ The following fits a gam model to data that have a strong sequential correlation (see Section 9.1 for the basic time series concepts assumed in this exercise): library(mgcv) xy <- data.frame(x=1:200, y=arima.sim(list(ar=0.75), n=200)) df.gam <- gam(y ˜ s(x), data=xy) plot(df.gam, residuals=TRUE) (a) Run the code several times. (Be sure, on each occasion, to simulate a new data frame xy.) Is the function gam() overfitting? What is overfitting in this context? Compare with the result from re-running the code with ar=0. (b) Repeat, now with ar=-0.75 in the code that generates the sequentially correlated series. Why is the result so very different?

8

Generalized linear models and survival analysis

The straight line regression model we considered in Chapter 5 had the form y = α + βx + ε where, if we were especially careful, we would add a subscript i to each of y, x, and ε. In this chapter, we will resume with models where there is just one x, as in Chapter 5, in order to keep the initial discussion simple. Later, we will add more predictor variables, as required. The regression model can be written E[y] = α + βx where E is expectation. This form of the equation is a convenient point of departure for moving to generalized linear models, abbreviated to GLMs. This class of models, first introduced in the 1970s, gives a unified theoretical and computational approach to models that had previously been treated as distinct. They have been a powerful addition to the data analyst’s armory of statistical tools. The present chapter will limit attention to a few important special cases. The chapter will end with a discussion of survival methods. Survival methods, while having important theoretical connections with generalized linear models, require a distinct theoretical and computational treatment.

8.1 Generalized linear models Generalized linear models (GLMs) differ in two ways from the models used in earlier chapters. They allow a more general form of expression for the expectation, and they allow various types of non-normal error terms. Logistic regression models are perhaps the most widely used GLM.

8.1.1 Transformation of the expected value on the left

GLMs allow a transformation f () to the left-hand side of the regression equation, i.e., to E[y]. The result specifies a linear relation with x. In other words, f (E[y]) = α + βx

logit(Proportion) = log(Odds)

8.1 Generalized linear models

245 0.999

6

0.99

4 2

0.9

0

0.5

−2

0.1

−4

0.01

−6

0.001 0.0

0.2

0.4

0.6

0.8

1.0

Proportion

Figure 8.1 The logit or log(odds) transformation. Shown here is a plot of log(odds) versus proportion. Notice how the range is stretched out at both ends.

where f () is a function, which is usually called the link function. In the fitted model, we call α + βx the linear predictor, while E[y] is the expected value of the response. The function f () transforms from the scale of the response to the scale of the linear predictor. Some common examples of link functions are: f (x) = x, f (x) = 1/x, f (x) = log(x), and f (x) = log(x/(1 − x)). The last, shown in Figure 8.1, is the logit link that is the link function for logistic regression.1 Observe that these functions are all monotonic, i.e., they increase or (in the case of 1/x) decrease with increasing values of x. 8.1.2 Noise terms need not be normal

We may write y = E[y] + ε. Here, the elements of y may have a distribution different from the normal. Common distributions are the binomial where y is the number responding out of a given total n, and the Poisson where y is a count. Even more common may be models where the random component differs from the binomial or Poisson by having a variance that is larger than the mean. The analysis proceeds as though the distribution were binomial or Poisson, but the theoretical binomial or Poisson variance estimates are replaced by a variance that is estimated from the data. Such models are called, respectively, quasi-binomial models and quasi-Poisson models. 8.1.3 Log odds in contingency tables

With proportions that range from less than 0.1 to greater than 0.9, it is not reasonable to expect that the expected proportion will be a linear function of x. A transformation (link function) such as the logit is required. A good way to think about logit models is that they 1

## Simplified plot showing the logit link function p <- (1:999)/1000 gitp <- log(p/(1 - p)) plot(p, gitp, xlab = "Proportion", ylab = "", type = "l", pch = 1)

246

Generalized linear models and survival analysis

Table 8.1 Terminology used for logistic regression (or more generally for generalized linear models), compared with multiple regression terminology. Regression

Logistic regression

Degrees of freedom Sum of squares (SS) Mean sum of squares (divide by degrees of freedom) Fit models by minimizing the residual sum of squares.

Degrees of freedom Deviance (D) Mean deviance (divide by degrees of freedom) Fit models by minimizing the deviance.

work on a log(odds) scale. If p is a probability (e.g., that horse A will win the race), then the corresponding odds are p/(1 − p), and log(odds) = log(p/(1 − p)) = log(p) − log(1 − p). Logistic regression provides a framework for analyzing contingency table data. Let us now recall the fictitious admissions data presented in Table 4.10. The observed proportion of students (male and female) admitted into Engineering is 40/80 = 0.5. For Sociology, the admission proportion is 15/60 = 0.25. Thus, we have log(odds) = log(0.5/0.5) = 0 for Engineering, log(odds) = log(0.75/0.25) = 1.0986 for Sociology. What determines whether a student will be admitted to Engineering? What determines whether a student will be admitted to Sociology? Is age a factor? Logistic regression allows us to model log(odds of admission) as a function of age, or as a function of any other predictor that we may wish to investigate. For such data, we may write log(odds) = constant + effect due to faculty + effect due to gender. This now has the form of a linear model. 8.1.4 Logistic regression with a continuous explanatory variable

The likelihood is the joint probability of the observed data values, given the model parameters. It is thus a function of the model parameters. The deviance is minus twice the logarithm of the likelihood. Maximizing the likelihood is equivalent to minimizing the deviance. The fitting of the logistic model is accomplished by minimizing deviances. A deviance has a role very similar to a sum of squares in regression (in fact, if the data are normally distributed, the two quantities are equivalent). This aspect of the analogy between regression and logistic regression is furnished by Table 8.1. Data for the example that now follows are in the data frame anesthetic (DAAG). Thirty patients were given an anesthetic agent that was maintained at a predetermined (alveolar) concentration for 15 minutes before making an incision. It was then noted whether the patient moved, i.e., jerked or twisted. The interest is in estimating how the

0.75

4●

2●

0.5 0.25

7●

5●

0.5

1.0

1.5

2.0

Concentration

2.5

247

Value of nomove conc

6● 6●

0.0

Proportion

1.0

8.1 Generalized linear models

0.8 1 1.2 1.4 1.6 2.5

0 (move)

1 (no move)

prop

6 4 2 2 0 0

1 1 4 4 4 2

0.17 0.20 0.67 0.67 1.00 1.00

Figure 8.2 Plot, versus concentration, of proportion of patients not moving, for each of six different alveolar concentrations. The horizontal line is the proportion of no-moves over the data as a whole. Data are displayed to the right of the plot.

probability of jerking or twisting varies with increasing concentration of the anesthetic agent. We take the response as nomove, because the proportion then increases with increasing concentration. The totals and proportions, for each of the six concentrations, can be calculated thus: library(DAAG) anestot <- aggregate(anesthetic[, c("move","nomove")], by=list(conc=anesthetic$conc), FUN=sum) ## The column ’conc’, because from the ’by’ list, is then a factor. ## The next line recovers the numeric values anestot$conc <- as.numeric(as.character(anestot$conc)) anestot$total <- apply(anestot[, c("move","nomove")], 1 , sum) anestot$prop <- anestot$nomove/anestot$total

Figure 8.2 plots the proportions. The table that is shown to the right of Figure 8.2 gives the information in the data frame anestot that has just been calculated. It gives, for each concentration, the respective numbers with nomove equal to 0 (i.e., movement) and nomove equal to 1 (i.e., no movement).2 We can fit the logit model either directly to the 0/1 data, or to the proportions in the table that appears to the right of Figure 8.2: ## Fit model directly to the 0/1 data in nomove anes.logit <- glm(nomove ˜ conc, family=binomial(link="logit"), data=anesthetic) ## Fit model to the proportions; supply total numbers as weights anes1.logit <- glm(prop ˜ conc, family=binomial(link="logit"), weights=total, data=anestot) 2

## Plot proportion moving vs conc: data frame anesthetic (DAAG) plot(prop ˜ conc, data=anestot, xlab = "Concentration", ylab = "Proportion", xlim = c(.5,2.5), ylim = c(0, 1), pch = 16) with(anestot, {text(conc, prop, paste(total), pos=2) abline(h=sum(nomove)/sum(total), lty=2)})

Generalized linear models and survival analysis ●1

1

●1 ● ● 0.67

0

0.67

−1

Empirical logit

2

248

●

● 0.2

−2

0.14

0.0

0.5

1.0

1.5

2.0

2.5

Concentration

Figure 8.3 Plot, on an empirical logit scale, of log(odds) = logit(proportion) of patients not moving, versus concentration. If x is the number of patients not moving, and n is the total, then the empirical x+0.5 ). Notice that the order changes from the order of the untransformed proportions. logit is log( n−x+0.5 Labels on the points show the observed proportions. The line gives the estimate of the proportion of moves, based on the fitted logit model.

The analysis assumes that individuals respond independently with a probability, estimated from the data, that on a logistic scale is a linear function of the concentration. For any fixed concentration, the assumption is that we have Bernoulli trials, i.e., that individual responses are drawn at random from the same population. Output from the summary() function (see also Figure 8.33 ) is: > summary(anes.logit) . . . . Coefficients: Estimate Std. Error z value Pr(>|z|) (Intercept) -6.47 2.42 -2.67 0.0075 conc 5.57 2.04 2.72 0.0064 (Dispersion parameter for binomial family taken to be 1) Null deviance: 41.455 Residual deviance: 27.754 AIC: 31.75

on 29 on 28

degrees of freedom degrees of freedom

Number of Fisher Scoring iterations: 5

With such a small sample size, a convincing check of the adequacy of the model is impossible.

3

## Graphical summary of log