Data Analysis and Graphics Using R: An Example-Based Approach (Cambridge Series in Statistical and Probabilistic Mathematics)

This page intentionally left blank

Data Analysis and Graphics Using R, Third Edition

Discover what you can do with R! Introducing the R system, covering standard regression methods, then tackling more advanced topics, this book guides users through the practical, powerful tools that the R system provides. The emphasis is on hands-on analysis, graphical display, and interpretation of data. The many worked examples, from real-world research, are accompanied by commentary on what is done and why. The companion website has code and data sets, allowing readers to reproduce all analyses, along with solutions to selected exercises and updates. Assuming basic statistical knowledge and some experience with data analysis (but not R), the book is ideal for research scientists, final-year undergraduate or graduate-level students of applied statistics, and practicing statisticians. It is both for learning and for reference. This third edition takes into account recent changes in R, including advances in graphical user interfaces (GUIs) and graphics packages. The treatments of the random forests methodology and one-way analysis have been extended. Both text and code have been revised throughout, and where possible simplified. New graphs and examples have been added. john maindonald is Visiting Fellow at the Mathematical Sciences Institute at the Australian National University. He has collaborated extensively with scientists in a wide range of application areas, from medicine and public health to population genetics, machine learning, economic history, and forensic linguistics. w. john braun is Professor in the Department of Statistical and Actuarial Sciences at the University of Western Ontario. He has collaborated with biostatisticians, biologists, psychologists, and most recently has become involved with a network of forestry researchers.

Data Analysis and Graphics Using R – an Example-Based Approach Third Edition

CAMBRIDGE SERIES IN STATISTICAL AND PROBABILISTIC MATHEMATICS Editorial Board Z. Ghahramani (Department of Engineering, University of Cambridge) R. Gill (Mathematical Institute, Leiden University) F. P. Kelly (Department of Pure Mathematics and Mathematical Statistics, University of Cambridge) B. D. Ripley (Department of Statistics, University of Oxford) S. Ross (Department of Industrial and Systems Engineering, University of Southern California) B. W. Silverman (St Peter’s College, Oxford) M. Stein (Department of Statistics, University of Chicago) This series of high quality upper-division textbooks and expository monographs covers all aspects of stochastic applicable mathematics. The topics range from pure and applied statistics to probability theory, operations research, optimization, and mathematical programming. The books contain clear presentations of new developments in the field and also of the state of the art in classical methods. While emphasizing rigorous treatment of theoretical methods, the books also contain applications and discussions of new techniques made possible by advances in computational practice. A complete list of books in the series can be found at http://www.cambridge.org/uk/series/sSeries.asp?code=CSPM Recent titles include the following: 7. 8. 9. 10. 11. 12. 13. 14. 15. 16. 17. 18. 19. 20. 21. 22. 23. 24. 25. 26. 27. 28. 29. 30.

Numerical Methods of Statistics, by John F. Monahan A User’s Guide to Measure Theoretic Probability, by David Pollard The Estimation and Tracking of Frequency, by B. G. Quinn and E. J. Hannan Data Analysis and Graphics Using R, by John Maindonald and John Braun Statistical Models, by A. C. Davison Semiparametric Regression, by David Ruppert, M. P. Wand and R. J. Carroll Exercises in Probability, by Loı¨c Chaumont and Marc Yor Statistical Analysis of Stochastic Processes in Time, by J. K. Lindsey Measure Theory and Filtering, by Lakhdar Aggoun and Robert Elliott Essentials of Statistical Inference, by G. A. Young and R. L. Smith Elements of Distribution Theory, by Thomas A. Severini Statistical Mechanics of Disordered Systems, by Anton Bovier The Coordinate-Free Approach to Linear Models, by Michael J. Wichura Random Graph Dynamics, by Rick Durrett Networks, by Peter Whittle Saddlepoint Approximations with Applications, by Ronald W. Butler Applied Asymptotics, by A. R. Brazzale, A. C. Davison and N. Reid Random Networks for Communication, by Massimo Franceschetti and Ronald Meester Design of Comparative Experiments, by R. A. Bailey Symmetry Studies, by Marlos A. G. Viana Model Selection and Model Averaging, by Gerda Claeskens and Nils Lid Hjort Bayesian Nonparametrics, edited by Nils Lid Hjort et al From Finite Sample to Asymptotic Methods in Statistics, by Pranab K. Sen, Julio M. Singer and Antonio C. Pedrosa de Lima Brownian Motion, by Peter M¨orters and Yuval Peres

Data Analysis and Graphics Using R – an Example-Based Approach Third Edition John Maindonald Mathematical Sciences Institute, Australian National University

and W. John Braun Department of Statistical and Actuarial Sciences, University of Western Ontario

CAMBRIDGE UNIVERSITY PRESS

Cambridge, New York, Melbourne, Madrid, Cape Town, Singapore, São Paulo, Delhi, Dubai, Tokyo Cambridge University Press The Edinburgh Building, Cambridge CB2 8RU, UK Published in the United States of America by Cambridge University Press, New York www.cambridge.org Information on this title: www.cambridge.org/9780521762939 © Cambridge University Press 2003 Second and third editions © John Maindonald and W. John Braun 2007, 2010 This publication is in copyright. Subject to statutory exception and to the provision of relevant collective licensing agreements, no reproduction of any part may take place without the written permission of Cambridge University Press. First published in print format 2010 ISBN-13

978-0-511-71286-9

eBook (NetLibrary)

ISBN-13

978-0-521-76293-9

Hardback

Cambridge University Press has no responsibility for the persistence or accuracy of urls for external or third-party internet websites referred to in this publication, and does not guarantee that any content on such websites is, or will remain, accurate or appropriate.

For Edward, Amelia and Luke also Shireen, Peter, Lorraine, Evan and Winifred For Susan, Matthew and Phillip

Contents

Preface Content – how the chapters fit together 1

A brief introduction to R 1.1 An overview of R 1.1.1 A short R session 1.1.2 The uses of R 1.1.3 Online help 1.1.4 Input of data from a file 1.1.5 R packages 1.1.6 Further steps in learning R 1.2 Vectors, factors, and univariate time series 1.2.1 Vectors 1.2.2 Concatenation – joining vector objects 1.2.3 The use of relational operators to compare vector elements 1.2.4 The use of square brackets to extract subsets of vectors 1.2.5 Patterned data 1.2.6 Missing values 1.2.7 Factors 1.2.8 Time series 1.3 Data frames and matrices 1.3.1 Accessing the columns of data frames – with() and attach() 1.3.2 Aggregation, stacking, and unstacking 1.3.3∗ Data frames and matrices 1.4 Functions, operators, and loops 1.4.1 Common useful built-in functions 1.4.2 Generic functions, and the class of an object 1.4.3 User-written functions 1.4.4 if Statements 1.4.5 Selection and matching 1.4.6 Functions for working with missing values 1.4.7∗ Looping

page xix xxv

1 1 1 6 7 8 9 9 10 10 10 11 11 11 12 13 14 14 17 17 18 19 19 21 22 23 23 24 24

x

Contents

1.5

1.6 1.7 1.8 1.9

Graphics in R 1.5.1 The function plot( ) and allied functions 1.5.2 The use of color 1.5.3 The importance of aspect ratio 1.5.4 Dimensions and other settings for graphics devices 1.5.5 The plotting of expressions and mathematical symbols 1.5.6 Identification and location on the figure region 1.5.7 Plot methods for objects other than vectors 1.5.8 Lattice (trellis) graphics 1.5.9 Good and bad graphs 1.5.10 Further information on graphics Additional points on the use of R Recap Further reading Exercises

25 25 27 28 28 29 29 30 30 32 33 33 35 36 37

2 Styles of data analysis 2.1 Revealing views of the data 2.1.1 Views of a single sample 2.1.2 Patterns in univariate time series 2.1.3 Patterns in bivariate data 2.1.4 Patterns in grouped data – lengths of cuckoo eggs 2.1.5∗ Multiple variables and times 2.1.6 Scatterplots, broken down by multiple factors 2.1.7 What to look for in plots 2.2 Data summary 2.2.1 Counts 2.2.2 Summaries of information from data frames 2.2.3 Standard deviation and inter-quartile range 2.2.4 Correlation 2.3 Statistical analysis questions, aims, and strategies 2.3.1 How relevant and how reliable are the data? 2.3.2 How will results be used? 2.3.3 Formal and informal assessments 2.3.4 Statistical analysis strategies 2.3.5 Planning the formal analysis 2.3.6 Changes to the intended plan of analysis 2.4 Recap 2.5 Further reading 2.6 Exercises

43 43 44 47 49 52 53 56 58 59 59 63 65 67 69 70 70 71 72 72 73 73 74 74

3 Statistical models 3.1 Statistical models 3.1.1 Incorporation of an error or noise component 3.1.2 Fitting models – the model formula

77 77 78 80

Contents

3.2

3.3

3.4

3.5 3.6 3.7 4

Distributions: models for the random component 3.2.1 Discrete distributions – models for counts 3.2.2 Continuous distributions Simulation of random numbers and random samples 3.3.1 Sampling from the normal and other continuous distributions 3.3.2 Simulation of regression data 3.3.3 Simulation of the sampling distribution of the mean 3.3.4 Sampling from finite populations Model assumptions 3.4.1 Random sampling assumptions – independence 3.4.2 Checks for normality 3.4.3 Checking other model assumptions 3.4.4 Are non-parametric methods the answer? 3.4.5 Why models matter – adding across contingency tables Recap Further reading Exercises

A review of inference concepts 4.1 Basic concepts of estimation 4.1.1 Population parameters and sample statistics 4.1.2 Sampling distributions 4.1.3 Assessing accuracy – the standard error 4.1.4 The standard error for the difference of means 4.1.5∗ The standard error of the median 4.1.6 The sampling distribution of the t-statistic 4.2 Confidence intervals and tests of hypotheses 4.2.1 A summary of one- and two-sample calculations 4.2.2 Confidence intervals and tests for proportions 4.2.3 Confidence intervals for the correlation 4.2.4 Confidence intervals versus hypothesis tests 4.3 Contingency tables 4.3.1 Rare and endangered plant species 4.3.2 Additional notes 4.4 One-way unstructured comparisons 4.4.1 Multiple comparisons 4.4.2 Data with a two-way structure, i.e., two factors 4.4.3 Presentation issues 4.5 Response curves 4.6 Data with a nested variation structure 4.6.1 Degrees of freedom considerations 4.6.2 General multi-way analysis of variance designs 4.7 Resampling methods for standard errors, tests, and confidence intervals 4.7.1 The one-sample permutation test 4.7.2 The two-sample permutation test

xi

81 82 84 86 87 88 88 90 91 91 92 95 95 96 97 98 98 102 102 102 102 103 103 104 105 106 109 112 113 113 114 116 119 119 122 123 124 125 126 127 127 128 128 129

xii

Contents

4.7.3∗ Estimating the standard error of the median: bootstrapping 4.7.4 Bootstrap estimates of confidence intervals 4.8∗ Theories of inference 4.8.1 Maximum likelihood estimation 4.8.2 Bayesian estimation 4.8.3 If there is strong prior information, use it! 4.9 Recap 4.10 Further reading 4.11 Exercises

130 131 132 133 133 135 135 136 137

5 Regression with a single predictor 5.1 Fitting a line to data 5.1.1 Summary information – lawn roller example 5.1.2 Residual plots 5.1.3 Iron slag example: is there a pattern in the residuals? 5.1.4 The analysis of variance table 5.2 Outliers, influence, and robust regression 5.3 Standard errors and confidence intervals 5.3.1 Confidence intervals and tests for the slope 5.3.2 SEs and confidence intervals for predicted values 5.3.3∗ Implications for design 5.4 Assessing predictive accuracy 5.4.1 Training/test sets and cross-validation 5.4.2 Cross-validation – an example 5.4.3∗ Bootstrapping 5.5 Regression versus qualitative anova comparisons – issues of power 5.6 Logarithmic and other transformations 5.6.1∗ A note on power transformations 5.6.2 Size and shape data – allometric growth 5.7 There are two regression lines! 5.8 The model matrix in regression 5.9∗ Bayesian regression estimation using the MCMCpack package 5.10 Recap 5.11 Methodological references 5.12 Exercises

142 142 143 143 145 147 147 149 150 150 151 152 153 153 155 158 160 160 161 162 163 165 166 167 167

6 Multiple linear regression 6.1 Basic ideas: a book weight example 6.1.1 Omission of the intercept term 6.1.2 Diagnostic plots 6.2 The interpretation of model coefficients 6.2.1 Times for Northern Irish hill races 6.2.2 Plots that show the contribution of individual terms 6.2.3 Mouse brain weight example 6.2.4 Book dimensions, density, and book weight

170 170 172 173 174 174 177 179 181

Contents

6.3

6.4

6.5 6.6

6.7 6.8

6.9 6.10 6.11 7

Multiple regression assumptions, diagnostics, and efficacy measures 6.3.1 Outliers, leverage, influence, and Cook’s distance 6.3.2 Assessment and comparison of regression models 6.3.3 How accurately does the equation predict? A strategy for fitting multiple regression models 6.4.1 Suggested steps 6.4.2 Diagnostic checks 6.4.3 An example – Scottish hill race data Problems with many explanatory variables 6.5.1 Variable selection issues Multicollinearity 6.6.1 The variance inflation factor 6.6.2 Remedies for multicollinearity Errors in x Multiple regression models – additional points 6.8.1 Confusion between explanatory and response variables 6.8.2 Missing explanatory variables 6.8.3∗ The use of transformations 6.8.4∗ Non-linear methods – an alternative to transformation? Recap Further reading Exercises

Exploiting the linear model framework 7.1 Levels of a factor – using indicator variables 7.1.1 Example – sugar weight 7.1.2 Different choices for the model matrix when there are factors 7.2 Block designs and balanced incomplete block designs 7.2.1 Analysis of the rice data, allowing for block effects 7.2.2 A balanced incomplete block design 7.3 Fitting multiple lines 7.4 Polynomial regression 7.4.1 Issues in the choice of model 7.5∗ Methods for passing smooth curves through data 7.5.1 Scatterplot smoothing – regression splines 7.5.2∗ Roughness penalty methods and generalized additive models 7.5.3 Distributional assumptions for automatic choice of roughness penalty 7.5.4 Other smoothing methods 7.6 Smoothing with multiple explanatory variables 7.6.1 An additive model with two smooth terms 7.6.2∗ A smooth surface 7.7 Further reading 7.8 Exercises

xiii

183 183 186 187 189 190 191 191 196 197 199 201 203 203 208 208 208 210 210 212 212 214 217 217 217 220 222 222 223 224 228 229 231 232 235 236 236 238 238 240 240 240

xiv

Contents

8 Generalized linear models and survival analysis 8.1 Generalized linear models 8.1.1 Transformation of the expected value on the left 8.1.2 Noise terms need not be normal 8.1.3 Log odds in contingency tables 8.1.4 Logistic regression with a continuous explanatory variable 8.2 Logistic multiple regression 8.2.1 Selection of model terms, and fitting the model 8.2.2 Fitted values 8.2.3 A plot of contributions of explanatory variables 8.2.4 Cross-validation estimates of predictive accuracy 8.3 Logistic models for categorical data – an example 8.4 Poisson and quasi-Poisson regression 8.4.1 Data on aberrant crypt foci 8.4.2 Moth habitat example 8.5 Additional notes on generalized linear models 8.5.1∗ Residuals, and estimating the dispersion 8.5.2 Standard errors and z- or t-statistics for binomial models 8.5.3 Leverage for binomial models 8.6 Models with an ordered categorical or categorical response 8.6.1 Ordinal regression models 8.6.2∗ Loglinear models 8.7 Survival analysis 8.7.1 Analysis of the Aids2 data 8.7.2 Right-censoring prior to the termination of the study 8.7.3 The survival curve for male homosexuals 8.7.4 Hazard rates 8.7.5 The Cox proportional hazards model 8.8 Transformations for count data 8.9 Further reading 8.10 Exercises

244 244 244 245 245

9 Time series models 9.1 Time series – some basic ideas 9.1.1 Preliminary graphical explorations 9.1.2 The autocorrelation and partial autocorrelation function 9.1.3 Autoregressive models 9.1.4∗ Autoregressive moving average models – theory 9.1.5 Automatic model selection? 9.1.6 A time series forecast 9.2∗ Regression modeling with ARIMA errors 9.3∗ Non-linear time series 9.4 Further reading 9.5 Exercises

283 283 283 284 285 287 288 289 291 298 300 301

246 249 252 254 255 255 256 258 258 261 266 266 267 268 268 269 272 272 273 275 276 276 277 279 280 281

Contents

10

11

xv

Multi-level models and repeated measures 10.1 A one-way random effects model 10.1.1 Analysis with aov() 10.1.2 A more formal approach 10.1.3 Analysis using lmer() 10.2 Survey data, with clustering 10.2.1 Alternative models 10.2.2 Instructive, though faulty, analyses 10.2.3 Predictive accuracy 10.3 A multi-level experimental design 10.3.1 The anova table 10.3.2 Expected values of mean squares 10.3.3∗ The analysis of variance sums of squares breakdown 10.3.4 The variance components 10.3.5 The mixed model analysis 10.3.6 Predictive accuracy 10.4 Within- and between-subject effects 10.4.1 Model selection 10.4.2 Estimates of model parameters 10.5 A generalized linear mixed model 10.6 Repeated measures in time 10.6.1 Example – random variation between profiles 10.6.2 Orthodontic measurements on children 10.7 Further notes on multi-level and other models with correlated errors 10.7.1 Different sources of variance – complication or focus of interest? 10.7.2 Predictions from models with a complex error structure 10.7.3 An historical perspective on multi-level models 10.7.4 Meta-analysis 10.7.5 Functional data analysis 10.7.6 Error structure in explanatory variables 10.8 Recap 10.9 Further reading 10.10 Exercises

303 304 305 308 310 313 313 318 319 319 321 322 323 325 326 328 329 329 331 332 334 336 340 344

Tree-based classification and regression 11.1 The uses of tree-based methods 11.1.1 Problems for which tree-based regression may be used 11.2 Detecting email spam – an example 11.2.1 Choosing the number of splits 11.3 Terminology and methodology 11.3.1 Choosing the split – regression trees 11.3.2 Within and between sums of squares 11.3.3 Choosing the split – classification trees 11.3.4 Tree-based regression versus loess regression smoothing

351 352 352 353 356 356 357 357 358 359

344 345 345 347 347 347 347 348 349

xvi

Contents

11.4

11.5

11.6 11.7 11.8 11.9 11.10

Predictive accuracy and the cost–complexity trade-off 11.4.1 Cross-validation 11.4.2 The cost–complexity parameter 11.4.3 Prediction error versus tree size Data for female heart attack patients 11.5.1 The one-standard-deviation rule 11.5.2 Printed information on each split Detecting email spam – the optimal tree The randomForest package Additional notes on tree-based methods Further reading and extensions Exercises

361 361 362 363 363 365 366 366 369 372 373 374

12

Multivariate data exploration and discrimination 12.1 Multivariate exploratory data analysis 12.1.1 Scatterplot matrices 12.1.2 Principal components analysis 12.1.3 Multi-dimensional scaling 12.2 Discriminant analysis 12.2.1 Example – plant architecture 12.2.2 Logistic discriminant analysis 12.2.3 Linear discriminant analysis 12.2.4 An example with more than two groups 12.3∗ High-dimensional data, classification, and plots 12.3.1 Classifications and associated graphs 12.3.2 Flawed graphs 12.3.3 Accuracies and scores for test data 12.3.4 Graphs derived from the cross-validation process 12.4 Further reading 12.5 Exercises

377 378 378 379 383 385 386 387 388 390 392 394 394 398 404 406 407

13

Regression on principal component or discriminant scores 13.1 Principal component scores in regression 13.2∗ Propensity scores in regression comparisons – labor training data 13.2.1 Regression comparisons 13.2.2 A strategy that uses propensity scores 13.3 Further reading 13.4 Exercises

410 410 414 417 419 426 426

14

The R system – additional topics 14.1 Graphical user interfaces to R 14.1.1 The R Commander’s interface – a guide to getting started 14.1.2 The rattle GUI 14.1.3 The creation of simple GUIs – the fgui package 14.2 Working directories, workspaces, and the search list

427 427 428 429 429 430

14.3

14.4

14.5

14.6 14.7 14.8∗

14.9

14.10

14.11

14.12∗ 14.13 14.14 14.15

Contents

xvii

14.2.1∗ The search path 14.2.2 Workspace management 14.2.3 Utility functions R system configuration 14.3.1 The R Windows installation directory tree 14.3.2 The library directories 14.3.3 The startup mechanism Data input and output 14.4.1 Input of data 14.4.2 Data output 14.4.3 Database connections Functions and operators – some further details 14.5.1 Function arguments 14.5.2 Character string and vector functions 14.5.3 Anonymous functions 14.5.4 Functions for working with dates (and times) 14.5.5 Creating groups 14.5.6 Logical operators Factors Missing values Matrices and arrays 14.8.1 Matrix arithmetic 14.8.2 Outer products 14.8.3 Arrays Manipulations with lists, data frames, matrices, and time series 14.9.1 Lists – an extension of the notion of “vector” 14.9.2 Changing the shape of data frames (or matrices) 14.9.3∗ Merging data frames – merge() 14.9.4 Joining data frames, matrices, and vectors – cbind() 14.9.5 The apply family of functions 14.9.6 Splitting vectors and data frames into lists – split() 14.9.7 Multivariate time series Classes and methods 14.10.1 Printing and summarizing model objects 14.10.2 Extracting information from model objects 14.10.3 S4 classes and methods Manipulation of language constructs 14.11.1 Model and graphics formulae 14.11.2 The use of a list to pass arguments 14.11.3 Expressions 14.11.4 Environments 14.11.5 Function environments and lazy evaluation Creation of R packages Document preparation – Sweave() and xtable() Further reading Exercises

430 430 431 432 432 433 433 433 434 437 438 438 439 440 441 441 443 443 444 446 448 450 451 451 452 452 454 455 455 456 457 458 458 459 460 460 461 461 462 463 463 464 465 467 468 469

xviii

15

Contents

Graphs in R 15.1 Hardcopy graphics devices 15.2 Plotting characters, symbols, line types, and colors 15.3 Formatting and plotting of text and equations 15.3.1 Symbolic substitution of symbols in an expression 15.3.2 Plotting expressions in parallel 15.4 Multiple graphs on a single graphics page 15.5 Lattice graphics and the grid package 15.5.1 Groups within data, and/or columns in parallel 15.5.2 Lattice parameter settings 15.5.3 Panel functions, strip functions, strip labels, and other annotation 15.5.4 Interaction with lattice (and other) plots – the playwith package 15.5.5 Interaction with lattice plots – focus, interact, unfocus 15.5.6 Overlaid plots with different scales 15.6 An implementation of Wilkinson’s Grammar of Graphics 15.7 Dynamic graphics – the rgl and rggobi packages 15.8 Further reading

472 472 472 474 475 475 476 477 478 480 483 485 485 486 487 491 492

Epilogue

493

References

495

Index of R symbols and functions

507

Index of terms

514

Index of authors

523

The color plates will be found between pages 328 and 329.

Preface

This book is an exposition of statistical methodology that focuses on ideas and concepts, and makes extensive use of graphical presentation. It avoids, as much as possible, the use of mathematical symbolism. It is particularly aimed at scientists who wish to do statistical analyses on their own data, preferably with reference as necessary to professional statistical advice. It is intended to complement more mathematically oriented accounts of statistical methodology. It may be used to give students with a more specialist statistical interest exposure to practical data analysis. While no prior knowledge of specific statistical methods or theory is assumed, there is a demand that readers bring with them, or quickly acquire, some modest level of statistical sophistication. Readers should have some prior exposure to statistical methodology, some prior experience of working with real data, and be comfortable with the typing of analysis commands into the computer console. Some prior familiarity with regression and with analysis of variance will be helpful. We cover a range of topics that are important for many different areas of statistical application. As is inevitable in a book that has this broad focus, there will be investigators working in specific areas – perhaps epidemiology, or psychology, or sociology, or ecology – who will regret the omission of some methodologies that they find important. We comment extensively on analysis results, noting inferences that seem well-founded, and noting limitations on inferences that can be drawn. We emphasize the use of graphs for gaining insight into data – in advance of any formal analysis, for understanding the analysis, and for presenting analysis results. The data sets that we use as a vehicle for demonstrating statistical methodology have been generated by researchers in many different fields, and have in many cases featured in published papers. As far as possible, our account of statistical methodology comes from the coalface, where the quirks of real data must be faced and addressed. Features that may challenge the novice data analyst have been retained. The diversity of examples has benefits, even for those whose interest is in a specific application area. Ideas and applications that are useful in one area often find use elsewhere, even to the extent of stimulating new lines of investigation. We hope that our book will stimulate such cross-fertilization. To summarize: The strengths of this book include the directness of its encounter with research data, its advice on practical data analysis issues, careful critiques of analysis results, the use of modern data analysis tools and approaches, the use of simulation and other computer-intensive methods – where these provide insight or give results that are not otherwise available, attention to graphical and other presentation issues, the use of

xx

Preface

examples drawn from across the range of statistical applications, and the inclusion of code that reproduces analyses. A substantial part of the book was derived, initially, from John Maindonald’s lecture notes of courses for researchers, at the University of Newcastle (Australia) over 1996– 1997 and at The Australian National University over 1998–2001. Both of us have worked extensively over the material in these chapters. The R system

We use the R system for computations. It began in the early 1990s as a project of Ross Ihaka and Robert Gentleman, who were both at the time working at the University of Auckland (New Zealand). The R system implements a dialect of the influential S language, developed at AT&T Bell Laboratories by Rick Becker, John Chambers, and Allan Wilks, which is the basis for the commercial S-PLUS system. It follows S in its close linkage between data analysis and graphics. Versions of R are available, at no charge, for 32-bit versions of Microsoft Windows, for Linux and other Unix systems, and for the Macintosh. It is available through the Comprehensive R Archive Network (CRAN). Go to http://cran.r-project.org/, and find the nearest mirror site. The development model used for R has proved highly effective in marshalling high levels of computing expertise for continuing improvement, for identifying and fixing bugs, and for responding quickly to the evolving needs and interests of the statistical community. Oversight of “base R” is handled by the R Core Team, whose members are widely drawn internationally. Use is made of code, bug fixes, and documentation from the wider R user community. Especially important are the large number of packages that supplement base R, and that anyone is free to contribute. Once installed, these attach seamlessly into the base system. Many of the analyses offered by R’s packages were not, 20 years ago, available in any of the standard statistical packages. What did data analysts do before we had such packages? Basically, they adapted more simplistic (but not necessarily simpler) analyses as best they could. Those whose skills were unequal to the task did unsatisfactory analyses. Those with more adequate skills carried out analyses that, even if not elegant and insightful by current standards, were often adequate. Tools such as are available in R have reduced the need for the adaptations that were formerly necessary. We can often do analyses that better reflect the underlying science. There have been challenging and exciting changes from the methodology that was typically encountered in statistics courses 15 or 20 years ago. In the ongoing development of R, priorities have been: the provision of good data manipulation abilities; flexible and high-quality graphics; the provision of data analysis methods that are both insightful and adequate for the whole range of application area demands; seamless integration of the different components of R; and the provision of interfaces to other systems (editors, databases, the web, etc.) that R users may require. Ease of use is important, but not at the expense of power, flexibility, and checks against answers that are potentially misleading. Depending on the user’s level of skill with R, there will be some tasks where another system may seem simpler to use. Note however the availability of interfaces, notably John Fox’s Rcmdr, that give a graphical user interface (GUI) to a limited part of R. Such

Preface

xxi

interfaces will develop and improve as time progresses. They may in due course, for many users, be the preferred means of access to R. Be aware that the demand for simple tools will commonly place limitations on the tasks that can, without professional assistance, be satisfactorily undertaken. Primarily, R is designed for scientific computing and for graphics. Among the packages that have been added are many that are not obviously statistical – for drawing and coloring maps, for map projections, for plotting data collected by balloon-borne weather instruments, for creating color palettes, for working with bitmap images, for solving sudoko puzzles, for creating magic squares, for reading and handling shapefiles, for solving ordinary differential equations, for processing various types of genomic data, and so on. Check through the list of R packages that can be found on any of the CRAN sites, and you may be surprised at what you find! The citation for John Chambers’ 1998 Association for Computing Machinery Software award stated that S has “forever altered how people analyze, visualize and manipulate data.” The R project enlarges on the ideas and insights that generated the S language. We are grateful to the R Core Team, and to the creators of the various R packages, for bringing into being the R system – this marvellous tool for scientific and statistical computing, and for graphical presentation. We give a list at the end of the reference section that cites the authors and compilers of packages that have been used in this book. Influences on the modern practice of statistics The development of statistics has been motivated by the demands of scientists for a methodology that will extract patterns from their data. The methodology has developed in a synergy with the relevant supporting mathematical theory and, more recently, with computing. This has led to methodologies and supporting theory that are a radical departure from the methodologies of the pre-computer era. Statistics is a young discipline. Only in the 1920s and 1930s did the modern framework of statistical theory, including ideas of hypothesis testing and estimation, begin to take shape. Different areas of statistical application have taken these ideas up in different ways, some of them starting their own separate streams of statistical tradition. See, for example, the comments in Gigerenzer et al. (1989) on the manner in which differences of historical development have influenced practice in different research areas. Separation from the statistical mainstream, and an emphasis on “black-box” approaches, have contributed to a widespread exaggerated emphasis on tests of hypotheses, to a neglect of pattern, to the policy of some journal editors of publishing only those studies that show a statistically significant effect, and to an undue focus on the individual study. Anyone who joins the R community can expect to witness, and/or engage in, lively debate that addresses these and related issues. Such debate can help ensure that the demands of scientific rationality do in due course win out over influences from accidents of historical development. New computing tools We have drawn attention to advances in statistical computing methodology. These have made possible the development of new powerful tools for exploratory analysis of regression

xxii

Preface

data, for choosing between alternative models, for diagnostic checks, for handling nonlinearity, for assessing the predictive power of models, and for graphical presentation. In addition, we have new computing tools that make it straightforward to move data between different systems, to keep a record of calculations, to retrace or adapt earlier calculations, and to edit output and graphics into a form that can be incorporated into published documents. New traditions of data analysis have developed – data mining, machine learning, and analytics. These emphasize new types of data, new data analysis demands, new data analysis tools, and data sets that may be of unprecedented size. Textual data and image data offer interesting new challenges for data analysis. The traditional concerns of professional data analysts remain as important as ever. Size of data set is not a guarantee of quality and of relevance to issues that are under investigation. It does not guarantee that the source population has been adequately sampled, or that the results will generalize as required to the target population. The best any analysis can do is to highlight the information in the data. No amount of statistical or computing technology can be a substitute for good design of data collection, for understanding the context in which data are to be interpreted, or for skill in the use of statistical analysis methodology. Statistical software systems are one of several components of effective data analysis. The questions that statistical analysis is designed to answer can often be stated simply. This may encourage the layperson to believe that the answers are similarly simple. Often, they are not. Be prepared for unexpected subtleties. Effective statistical analysis requires appropriate skills, beyond those gained from taking one or two undergraduate courses in statistics. There is no good substitute for professional training in modern tools for data analysis, and experience in using those tools with a wide range of data sets. Noone should be embarrassed that they have difficulty with analyses that involve ideas that professional statisticians may take 7 or 8 years of professional training and experience to master.

Third edition changes and additions The second edition added new material on survival analysis, random coefficient models, the handling of high-dimensional data, and extended the account of regression methods. This third edition has a more adequate account of errors in predictor variables, extends the treatment and use of random forests, and adds a brief account of generalized linear mixed models. The treatment of one-way analysis of variance, and a major part of the chapter on regression, have been rewritten. Two areas of especially rapid advance have been graphical user interfaces (GUIs), and graphics. There are now brief introductions to two popular GUIs for R – the R Commander (Rcmdr) and rattle. The sections on graphics have been substantially extended. There is a brief account of the latticist and associated playwith GUIs for interfacing with R graphics. Code has again been extensively revised, simplifying it wherever possible. There are changes to some graphs, and new graphs have been added.

Preface

xxiii

Acknowledgments Many different people have helped with this project. Winfried Theis (University of Dortmund, Germany) and Detlef Steuer (University of the Federal Armed Forces, Hamburg, Germany) helped with technical LATEX issues, with a cvs archive for manuscript files, and with helpful comments. Lynne Billard (University of Georgia, USA), Murray Jorgensen (University of Waikato, NZ), and Berwin Turlach (University of Western Australia) gave highly useful comment on the manuscript. Susan Wilson (Australian National University) gave welcome encouragement. Duncan Murdoch (University of Western Ontario) helped with technical advice. Cath Lawrence (Australian National University) wrote a Python program that allowed us to extract the R code from our LATEX files; this has now at length become an R function. For the second edition, Brian Ripley (University of Oxford) made extensive comments on the manuscript, leading to important corrections and improvements. We are most grateful to him, and to others who have offered comments. Alan Welsh (Australian National University) has helped work through points where it has seemed difficult to get the emphasis right. Once again, Duncan Murdoch has given much useful technical advice. Others who made helpful comments and/or pointed out errors include Jeff Wood (Australian National University), Nader Tajvidi (University of Lund), Paul Murrell (University of Auckland, on Chapter 15), Graham Williams (http://www.togaware.com, on Chapter 1), and Yang Yang (University of Western Ontario, on Chapter 10). Comment that has contributed to this edition has come from Ray Balise (Stanford School of Medicine), Wenqing He and Lengyi Han (University of Western Ontario), Paul Murrell, Andrew Robinson (University of Melbourne, on Chapter 10), Phil Kokic (Australian National University, on Chapter 9), and Rob Hyndman (Monash University, on Chapter 9). Readers who have made relatively extensive comments include Bob Green (Queensland Health) and Zander Smith (SwissRe). Additionally, discussions on the R-help and R-devel email lists have been an important source of insight and understanding. The failings that remain are, naturally, our responsibility. A strength of this book is the extent to which it has drawn on data from many different sources. Following the references is a list of data sources (individuals and/or organizations) that we wish to thank and acknowledge. We are grateful to those who have allowed us to use their data. At least these data will not, as often happens once data have become the basis for a published paper, gather dust in a long-forgotten folder! We are grateful, also, to the many researchers who, in their discussions with us, have helped stimulate our thinking and understanding. We apologize if there is anyone that we have inadvertently failed to acknowledge. Diana Gillooly of Cambridge University Press, taking over from David Tranah for the second and third editions, has been a marvellous source of advice and encouragement.

Conventions Text that is R code, or output from R, is printed in a verbatim text style. For example, in Chapter 1 we will enter data into an R object that we call austpop. We will use the

xxiv

Preface

plot() function to plot these data. The names of R packages, including our own DAAG package, are printed in italics. Starred exercises and sections identify more technical items that can be skipped at a first reading. Solutions to exercises

Solutions to selected exercises, R scripts that have all the code from the book, and other supplementary materials are available via the link given at http://www.maths.anu. edu.au/˜johnm/r-book

Content – how the chapters fit together

Chapter 1 is a brief introduction to R. Readers who are new to R should as a minimum study Section 1.1, or an equivalent, before moving on to later chapters. In later study, refer back as needed to Chapter 1, or forward to Chapter 14. Chapters 2–4: Exploratory data analysis and review of elementary statistical ideas Chapters 2–4 cover, at greater depth and from a more advanced perspective, topics that are common in introductory courses. Different readers will use these chapters differently, depending on their statistical preparedness. Chapter 2 (Styles of data analysis) places data analysis in the wider context of the research study, commenting on some of the types of graphs that may help answer questions that are commonly of interest and that will be used throughout the remainder of the text. Subsections 2.1.7, 2.2.3 and 2.2.4 introduce terminology that will be important in later chapters. Chapter 3 (Statistical models) introduces the signal + noise form of regression model. The different models for the signal component are too varied to describe in one chapter! Coverage of models for the noise (random component) is, relative to their use in remaining chapters, more complete. Chapter 4 (A review of inference concepts) describes approaches to generalizing from data. It notes the limitations of the formal hypothesis testing methodology, arguing that a less formal approach is often adequate. It notes also that there are contexts where a Bayesian approach is essential, in order to take account of strong prior information. Chapters 5–13: Regression and related methodology Chapters 5–13 are designed to give a sense of the variety and scope of methods that come, broadly, under the heading of regression. In Chapters 5 and 6, the models are linear in the explanatory variable(s) as well as in the parameters. A wide range of issues affect the practical use of these models: influence, diagnostics, robust and resistant methods, AIC and other model comparison measures, interpretation of coefficients, variable selection, multicollinearity, and errors in x. All these issues are relevant, in one way or another, throughout later chapters. Chapters 5 and 6 provide relatively straightforward contexts in which to introduce them.

xxvi

Content – how the chapters fit together

The models of Chapters 5–13 give varying combinations of answers to the questions: 1. What is the signal term? Is it in some sense linear? Can it be described by a simple form of mathematical equation? 2. Is the noise term normal, or are there other possibilities? 3. Are the noise terms independent between observations? 4. Is the model specified in advance? Or will it be necessary to choose the model from a potentially large number of possible models? In Chapters 5–8, the models become increasingly general, but always with a model that is linear in the coefficients as a starting point. In Chapters 5–7, the noise terms are normal and independent between observations. The generalized linear models of Chapter 8 allow nonnormal noise terms. These are still assumed independent.1 Chapter 9 (Time series models) and Chapter 10 (Multilevel models and repeated measures) introduce models that allow, in their different ways, for dependence between observations. In Chapter 9 the correlation is with observations at earlier points in time, while in Chapter 10 the correlation might for example be between different students in the same class, as opposed to different students in different classes. In both types of model, the noise term is constructed from normal components – there are normality assumptions. Chapters 6–10 allowed limited opportunity for the choice of model and/or explanatory variables. Chapter 11 (Tree-based classification and regression) introduces models that are suited to a statistical learning approach, where the model is chosen from a large portfolio of possibilities. Moreover, these models do not have any simple form of equation. Note the usual implicit assumption of independence between observations – this imposes limitations that, depending on the context, may or may not be important for practical use. Chapter 12 (Multivariate data exploration and discrimination) begins with methods that may be useful for multivariate data exploration – principal components, the use of distance measures, and multi-dimensional scaling. It describes dimension reduction approaches that allow low-dimensional views of the data. Subsection 12.2 moves to discriminant methods – i.e., to regression methods in which the outcome is categorical. Subsection 12.3 identifies issues that arise when the number of variables is large relative to the number of observations. Such data is increasingly common in many different application areas. It is sometimes possible to replace a large number of explanatory variables by one, or a small number, of scoring variables that capture the relevant information in the data. Chapter 13 investigates two different ways to create scores that may be used as explanatory variables in regression. In the first example, the principal component scores are used. The second uses propensity scores to summarize information on a number of covariates that are thought to explain group differences that are, for the purposes of the investigation, nuisance variables.

1

Note, however, the extension to allow models with a variance that, relative to the binomial or Poisson, is inflated.

1

A brief introduction to R

This first chapter introduces readers to the basics of R. It provides the minimum of information that is needed for running the calculations that are described in later chapters. The first section may cover most of what is immediately necessary. The rest of the chapter may be used as a reference. Chapter 14 extends this material considerably. Most of the R commands will run without change in S-PLUS. 1.1 An overview of R 1.1.1 A short R session R must be installed!

An up-to-date version of R may be downloaded from a Comprehensive R Archive Network (CRAN) mirror site. There are links at http://cran.r-project.org/. Installation instructions are provided at the web site for installing R in Windows, Unix, Linux, and version 10 of the Macintosh operating system. For most Windows users, R can be installed by clicking on the icon that appears on the desktop once the Windows setup program has been downloaded from CRAN. An installation program will then guide the user through the process. By default, an R icon will be placed on the user’s desktop. The R system can be started by double-clicking on that icon. Various contributed packages extend the capabilities of R. A number of these are a part of the standard R distribution, but a number are not. Many data sets that are mentioned in this book have been collected into our DAAG package that is available from CRAN sites. This and other such packages can be readily installed, from an R session, via a live internet connection. Details are given below, immediately prior to Subsection 1.1.2. Using the console (or command line) window

The command line prompt (>) is an invitation to type commands or expressions. Once the command or expression is complete, and the Enter key is pressed, R evaluates and prints the result in the console window. This allows the use of R as a calculator. For example, type 2+2 and press the Enter key. Here is what appears on the screen: > 2+2 [1] 4 >

2

A brief introduction to R

The first element is labeled [1] even when, as here, there is just one element! The final > prompt indicates that R is ready for another command. In a sense this chapter, and much of the rest of the book, is a discussion of what is possible by typing in statements at the command line. Practice in the evaluation of arithmetic expressions will help develop the needed conceptual and keyboard skills. For example: > 2*3*4*5 [1] 120 > sqrt(10) [1] 3.162278 > pi [1] 3.141593 > 2*pi*6378

# * denotes ’multiply’ # the square root of 10 # R knows about pi # Circumference of earth at equator (km) # (radius at equator is 6378 km)

[1] 40074.16

Anything that follows a # on the command line is taken as comment and ignored by R. A continuation prompt, by default +, appears following a carriage return when the command is not yet complete. For example, an interruption of the calculation of 3*4ˆ2 by a carriage return could appear as > 3*4ˆ + 2 [1] 48

In this book we will omit both the command prompt (>) and the continuation prompt whenever command line statements are given separately from output. Multiple commands may appear on one line, with a semicolon (;) as the separator. For example, > 3*4ˆ2; (3*4)ˆ2 [1] 48 [1] 144

Entry of data at the command line

Figure 1.1 gives, for each of the years 1800, 1850, . . . , 2000, estimated worldwide totals of carbon emissions that resulted from fossil fuel use. To enter the columns of data from the table, and plot Carbon against Year as in Figure 1.1, proceed thus: Year <- c(1800, 1850, 1900, 1950, 2000) Carbon <- c(8, 54, 534, 1630, 6611) ## Now plot Carbon as a function of Year plot(Carbon ˜ Year, pch=16)

Note the following: r The <- is a left angle bracket (<) followed by a minus sign (-). It means “the values on the right are assigned to the name on the left”.

1.1 An overview of R

4000

Carbon

6000

●

2000 0

3

1 2 3 4 5

● ●

●

1800

1850

●

1900

1950

year

carbon

1800 1850 1900 1950 2000

8 54 534 1630 6611

2000

Year

Figure 1.1 Estimated worldwide annual totals of carbon emissions from fossil fuel use, in millions of tonnes. Data are due to Marland et al. (2003).

r The objects Year and Carbon are vectors which were each formed by joining (concatenating) separate numbers together. Thus c(8, 54, 534, 1630, 6611) joined the numbers 8, 54, 534, 1630, 6611 together to form the vector Carbon. See Subsection 1.2.2 for further details. r The construct Carbon ˜ Year is a graphics formula. The plot() function interprets this formula to mean “Plot Carbon as a function of Year” or “Plot Carbon on the y-axis against Year on the x-axis”. r The setting pch=16 (where pch is “plot character”) gives a solid black dot. r Case is significant for names of R objects or commands. Thus, Carbon is different from carbon. This basic plot could be improved by adding more informative axis labels, changing sizes of the text and/or the plotting symbol, adding a title, and so on. See Section 1.5. Once created, the objects Year and Carbon are stored in the workspace, as part of the user’s working collection of R objects. The workspace lasts only until the end of a session. In order that the session can be resumed later, a copy or “image” must be kept. Upon typing q() to quit the session, you will be asked if you wish to save the workspace.

Collection of vectors into a data frame

The two vectors Year and Carbon created earlier are matched, element for element. It is convenient to group them together into an object that has the name data frame, thus: > fossilfuel <- data.frame(year=Year, carbon=Carbon) > fossilfuel # Display the contents of the data frame. year carbon 1 1800 8 2 1850 54 3 1900 534 4 1950 1630 5 2000 6611

4

A brief introduction to R

The vector objects Year and Carbon become, respectively, the columns year and carbon in the data frame. The vector objects Year and Carbon are then redundant, and can be removed. rm(Year, Carbon)

# The rm() function removes unwanted objects

Figure 1.1 can now be reproduced, with a slight change in the x- and y-labels, using plot(carbon ˜ year, data=fossilfuel, pch=16)

The data=fossilfuel argument instructs plot() to start its search for each of carbon and year by looking among the columns of fossilfuel. There are several ways to identify columns by name. Here, note that the second column can be referred to as fossilfuel[, 2], or as fossilfuel[, "carbon"], or as fossilfuel$carbon. Data frames are the preferred way to organize data sets that are of modest size. For now, think of data frames as a rectangular row by column layout, where the rows are observations and the columns are variables. Section 1.3 has further discussion of data frames. Subsection 1.1.4 will demonstrate input of data from a file, into a data frame. The R Commander Graphical User Interface (GUI) to R

Our discussion will usually assume use of the command line. Excellent GUI interfaces, such as the R Commander, are also available. Data input is very convenient with the R Commander. When importing data, a window pops up offering a choice of common data formats. Data can be input from a text file, the clipboard, URL, an Excel spreadsheet, or one of several statistical package formats (SPSS, Stata, Minitab, SAS, . . .). Refer to Section 14.1 for more details. The working directory and the contents of the workspace

Each R session has a working directory. Within a session, the workspace is the default place where R looks for files that are read from disk, or written to disk. In between sessions, it is usual for the working directory to keep a workspace copy or “image” from which the session can be restarted. For a session that is started from a Windows icon, the initial working directory is the Start in directory that appears by right clicking on the icon and then on Properties. Users of the MacOS X GUI can change the default startup directory from within an R session by clicking on the R menu item, then on Preferences, then making the necessary change in the panel Initial working directory. On Unix or Linux sessions that are started from the command line, the working directory is the directory in which R was started. In the event of uncertainty, type getwd() to display the name of the working directory: getwd()

Objects that the user creates or copies from elsewhere go into the user workspace. To list the workspace contents, type: ls()

1.1 An overview of R

5

The only object left over from the computations above should be fossilfuel. There may additionally be objects that are left over from previous sessions (if any) in the same directory, and that were loaded when the session started.

Quitting R

Use the q() function to quit (exit) from R: q()

There will be a message asking whether to save the workspace image. Clicking Yes has the effect that, before quitting, all the objects that remain in the workspace are saved in a file that has the name .RData. Because it is a copy or “image” of the workspace, this file is known as an image file. (Note that while delaying the saving of important objects until the end of the session is acceptable when working in a learning mode, it is not in general a good strategy when using R in production mode. Section 1.6 has advice on saving and backing up through the course of a session. See also the more extended comments in Subsection 14.2.2.) Depending on the implementation, alternatives to typing q() may be to click on the File menu and then on Exit, or to click on the × in the top right-hand corner of the R window. (Under Linux, depending on the window manager that is used, clicking on × may exit from the program, but without asking whether to save the workshop image. Check the behavior on your installation.) Note: The round brackets, when using q() to quit the session, are necessary because q is a function. Typing q on its own, without the brackets, displays the text of the function on the screen. Try it!

Installation of packages

Assuming access to a live internet connection, packages can be installed pretty much automatically. Thus, for installation of the DAAG package under Windows, start R and click on the Packages menu. From that menu, choose Install packages. If a mirror site has not been set earlier, this gives a pop-up menu from which a site must be chosen. Once this choice is made, a new pop-up window appears with the entire list of available R packages. Click on DAAG to select it for installation. Control-click to select additional packages. Click on OK to start downloading and installation. For installation from the command line, enter, for example install.packages("DAAG") install.packages(c("magic", "schoolmath"), dependencies=TRUE)

A further possibility, convenient if packages are to be installed onto a number of local systems, is to download the files used for the installation onto a local directory or onto a CD or DVD, and install from there.

6

A brief introduction to R

1.1.2 The uses of R

R has extensive capabilities for statistical analysis, that will be used throughout this book. These are embedded in an interactive computing environment that is suited to many different uses, some of which we now demonstrate.

R offers an extensive collection of functions and abilities

Most calculations that users may wish to perform, beyond simple command line computations, involve explicit use of functions. There are of course functions for calculating the sum (sum()), mean (mean()), range (range()), and length of a vector (length()), for sorting values into order (sort()), and so on. For example, the following calculates the range of the values in the vector carbon: > range(fossilfuel$carbon) [1] 8 6611

Here are examples that manipulate character strings: > ## 4 cities > fourcities <- c("Toronto", "Canberra", "New York", "London") > ## display in alphabetical order > sort(fourcities) [1] "Canberra" "London" "New York" "Toronto" > ## Find the number of characters in "Toronto" > nchar("Toronto") [1] 7 > > ## Find the number of characters in all four city names at once > nchar(fourcities) [1] 7 8 8 6

R will give numerical or graphical data summaries

The data frame cars (datasets package) has columns (variables) speed and dist. Typing summary(cars) gives summary information on its columns: > summary(cars) speed Min. : 4.0 1st Qu.:12.0 Median :15.0 Mean :15.4 3rd Qu.:19.0 Max. :25.0

dist Min. : 2.00 1st Qu.: 26.00 Median : 36.00 Mean : 42.98 3rd Qu.: 56.00 Max. :120.00

Thus, the range of speeds (first column) is from 4 mph to 25 mph, while the range of distances (second column) is from 2 feet to 120 feet.

1.1 An overview of R

7

Graphical summaries, including histograms and boxplots, are discussed and demonstrated in Section 2.1. Try, for example: hist(cars$speed)

R is an interactive programming language

The following calculates the Fahrenheit temperatures that correspond to Celsius temperatures 0, 10, . . . , 40: > > > > 1 2 3 4 5

celsius <- (0:4)*10 fahrenheit <- 9/5*celsius+32 conversion <- data.frame(Celsius=celsius, Fahrenheit=fahrenheit) print(conversion) Celsius Fahrenheit 0 32 10 50 20 68 30 86 40 104

1.1.3 Online help

Familiarity with R’s help facilities will quickly pay dividends. R’s help files are comprehensive, and are frequently upgraded. Type help(help) or ?help to get information on the help features of the system that is in use. To get help on, e.g., plot(), type: ?plot

# Equivalent to help(plot)

The functions apropos() and help.search() search for functions that perform a desired task. Examples are: apropos("sort") # Try, also, apropos ("sor") # List all functions where "sort" is part of the name help.search("sort") # Note that the argument is ’sort’ # List functions with ’sort’ in the help page title or as an alias

Users are encouraged to experiment with R functions, perhaps starting by using the function example() to run the examples on the relevant help page. Be warned however that, even for basic functions, some examples may illustrate relatively advanced uses. Thus, to run the examples from the help page for the function image(), type: example(image) par(ask=FALSE)

# turn off the prompts

Press the return key to see each new plot. The par(ask=FALSE) on the second line of code stops the prompts that will otherwise continue to appear, prior to the plotting of any subsequent graph. In learning to use a new function, it may be helpful to create a simple artificial data set, or to extract a small subset from a larger data set, and use this for experimentation. For

8

A brief introduction to R

extensive experimentation, consider moving to a new working directory and working with copies of any user data sets and functions. The help pages, while not an encyclopedia on statistical methodology, have very extensive useful information. They include: insightful and helpful examples, references to related functions, and references to papers and books that give the relevant theory. Some abilities will bring pleasant surprises. It can help enormously, before launching into the use of an R function, to check the relevant help page!

Wide-ranging information access and searches

The function help.start() opens a browser interface to help information, manuals, and helpful links. It may take practice, and time, to learn to navigate the wealth of information that is on offer. The function RSiteSearch() initiates (assuming a live internet connection) a search of R manuals and help pages, and of the R-help mailing list archives, for key words or phrases. The argument restrict allows some limited targeting of the search. See help(RSiteSearch) for details.

Help in finding the right package

The CRAN Task Views can be a good place to start when looking for abilities of a particular type. The 23 Task Views that are available at the time of writing include, for example: Bayesian inference, Cluster analysis, Finance, Graphics, and Time series. Go to http://cran.r-project.org/web/views/

1.1.4 Input of data from a file

Code that will take data from the file fuel.txt that is in the working directory, entering them into the data frame fossilfuel in the workspace is: fossilfuel <- read.table("fuel.txt", header=TRUE)

Note the use of header=TRUE to ensure that R uses the first line to get header information for the columns, usually in the form of column names. Type fossilfuel at the command line prompt, and the data will be displayed almost as they appear in Figure 1.1 (the only difference is the introduction of row labels in the R output). The function read.table() has the default argument sep="", implying that the fields of the input file are separated by spaces and/or tabs. Other settings are sometimes required. In particular: fossilfuel <- read.table("fuel.csv", header=TRUE, sep=",")

reads data from a file fuel.csv where fields are separated by commas. For other options, consult the help page for read.table(). See also Subsection 14.4.1.

1.1 An overview of R

9

On Microsoft Windows systems, it is immaterial whether this file is called fuel.txt or Fuel.txt. Unix file systems may, depending on the specific file system in use, treat letters that have a different case as different.

1.1.5 R packages

This chapter and Chapter 2 will make frequent use of data from the MASS package (Venables and Ripley, 2002) and from our own DAAG package. Various further packages will be used in later chapters. The packages base, stats, datasets, and several other packages, are automatically attached at the beginning of a session. Other installed packages must be explicitly attached prior to use. Use sessionInfo() to see which packages are currently attached. To attach any further installed package, use the library() function. For example, > library(DAAG) Loading required package: MASS . . . . > sessionInfo() R version 2.9.0 (2009-04-17) i386-apple-darwin8.11.1 . . . . attached base packages: [1] stats graphics grDevices utils

datasets

methods

base

other attached packages: [1] DAAG_0.98 MASS_7.2-46

Data sets that accompany R packages

Type data() to get a list of data sets (mostly data frames) in all packages that are in the current search path. For information on the data sets in the datasets package, type data(package="datasets")

# Specify ’package’, not ’library’.

Replace "datasets" by the name of any other installed package, as required (type library() to get the names of the installed packages). In most packages, these data sets automatically become available once the package is attached. They will be brought into the workspace when and if required. (A few packages do not implement the lazy data mechanism. Explicit use of a command of the form data(airquality) is then necessary, bringing the data object into the user’s workspace.)

1.1.6 Further steps in learning R

Readers who have followed the discussion thus far and worked through the examples may at this point have learned enough to start on Chapter 2, referring as necessary to later sections of this chapter, to R’s help pages, and to Chapter 14. The remaining sections of this chapter cover the following topics:

10

r r r r r

A brief introduction to R

Numeric, character, logical, and complex vectors (Section 1.2). Factors (Subsection 1.2.7). Data frames and matrices (Section 1.3). Functions for calculating data summaries (Section 1.4). Graphics and lattice graphics (Sections 1.5 and 15.5).

1.2 Vectors, factors, and univariate time series Vectors, factors, and univariate time series are all univariate objects that can be included as columns in a data frame. The vector modes that will be noted here (there are others) are “numeric”, “logical”, and “character".

1.2.1 Vectors

Examples of vectors are > c(2, 3, 5, 2, 7, 1) [1] 2 3 5 2 7 1 > c(T, F, F, F, T, T, F) [1] TRUE FALSE FALSE FALSE

TRUE

TRUE FALSE

> c("Canberra", "Sydney", "Canberra", "Sydney") [1] "Canberra" "Sydney" "Canberra" "Sydney"

The first of these is numeric, the second is logical, and the third is a character. The global variables F (=FALSE) and T (=TRUE) can be a convenient shorthand when logical values are entered.

1.2.2 Concatenation – joining vector objects

The function c(), used in Subsection 1.1.1 to join numbers together to form a vector, is more widely useful. It may be used to concatenate any combination of vectors and vector elements. In the following, we form numeric vectors x and y, that we then concatenate to form a vector z: > x > x [1] > y > y [1] > z > z [1]

<- c(2, 3, 5, 2, 7, 1) 2 3 5 2 7 1 <- c(10, 15, 12) 10 15 12 <- c(x, y) 2 3 5 2 7 1 10 15 12

# x then holds values 2, 3, 5, 2, 7, 1

1.2 Vectors, factors, and univariate time series

11

1.2.3 The use of relational operators to compare vector elements

Relational operators are <, <=, >, >=, ==, and !=. For example, consider the carbon and year columns of the data frame fossilfuel. For example: > x > x [1] > x [1]

<- c(3, 11, 8, 15, 12) > 8 FALSE TRUE FALSE TRUE != 8 TRUE TRUE FALSE TRUE

TRUE TRUE

For further information on relational operators consult help(Comparison), help(Logic), and help(Syntax).

1.2.4 The use of square brackets to extract subsets of vectors

Note three common ways to extract elements of vectors. In each case, the identifying information (in the simplest case, a vector of subscript indices) is enclosed in square brackets. 1. Specify the indices of the elements that are to be extracted, e.g., > x <- c(3, 11, 8, 15, 12) > x[c(2,4)] [1] 11 15

# Elements in positions 2 # and 4 only

2. Use negative subscripts to omit the elements in nominated subscript positions (take care not to mix positive and negative subscripts): > x[-c(2,3)] [1] 3 15 12

# Remove the elements in positions 2 and 3

3. Specify a vector of logical values. This extracts elements for which the logical value is TRUE. The following extracts values of x that are greater than 10: > x > 10 [1] FALSE TRUE FALSE TRUE TRUE > x[x > 10] [1] 11 15 12

Elements of vectors can be given names. Elements can then be extracted by name: > heights <- c(Andreas=178, John=185, Jeff=183) > heights[c("John","Jeff")] John Jeff 185 183

1.2.5 Patterned data

Use, for example, 5:15 to generate all integers in a range, here between 5 and 15 inclusive: > 5:15 [1] 5

6

7

8

9 10 11 12 13 14 15

Conversely, 15:5 will generate the sequence in the reverse order.

12

A brief introduction to R

The function seq() allows a wider range of possibilities. For example: > seq(from=5, to=22, by=3)

# The first value is 5. # value is <= 22

The final

[1] 5 8 11 14 17 20 ## The above can be abbreviated to seq(5, 22, 3)

To repeat the sequence (2, 3, 5) four times over, enter > rep(c(2,3,5), 4) [1] 2 3 5 2 3 5 2 3 5 2 3 5

Patterned character vectors are also possible: > c(rep("female", 3), rep("male", 2)) [1] "female" "female" "female" "male" "male"

1.2.6 Missing values

The missing value symbol is NA. As an example, consider the column branch of the data set rainforest: > library(DAAG) > nbranch <- subset(rainforest, species=="Acacia mabellae")$branch > nbranch # Number of small branches (2cm or less) [1] NA 35 41 50 NA NA NA NA NA 4 30 13 10 17 46 92

Any arithmetic expression that involves an NA generates NA as its result. Functions such as mean() allow the argument na.rm=TRUE, so that NAs are omitted before proceeding with the calculation. For example: > mean(nbranch) [1] NA > mean(nbranch, na.rm=TRUE) [1] 33.8

Other functions that behave similarly are sum(), median(), range(), and sd. Arithmetic and logical expressions in which NAs appear return NA, thus: > NA == 35 [1] NA

The unknown value might just possibly equal 35. This is a matter of strict logic, not probability. Thus, the result is NA. To replace all NAs by -999 (in most circumstances a bad idea) use the function is.na(), thus: > ## Replace all NAs by -999 > nbranch[is.na(nbranch)] <- -999 > nbranch [1] -999 35 41 50 -999 -999 -999 -999 -999 4 30 13 [13] 10 17 46 92 > ## There is now no protection against use of the -999 values as

1.2 Vectors, factors, and univariate time series

13

> ## if they were legitimate numeric values > mean(nbranch) [1] -353.5 # Illegitimate calculation

Using a code such as -999 for missing values requires continual watchfulness to ensure that it is never treated as a legitimate numeric value. Missing values are discussed further in Subsection 1.4.6 and Section 14.7. For vectors of mode numeric, other legal values that may require special attention are NaN (not a number; e.g., 0/0), Inf (e.g., 1/0), and -Inf. 1.2.7 Factors

A factor is stored internally as a numeric vector with values 1, 2, 3, . . . , k. The value k is the number of levels. The levels are character strings. Consider a survey that has data on 691 females and 692 males. If the first 691 are females and the next 692 males, we can create a vector of strings that holds the values, then turning this vector into a factor, thus: > ## Create character vector > gender <- c(rep("female",691), rep("male",692)) > levels(gender) # For a character vector, this returns NULL NULL > ## From character vector, create factor > gender <- factor(gender) > levels(gender) [1] "female" "male"

Internally, the factor gender is stored as 691 1s, followed by 692 2s. It has stored with it a table that holds the information 1 2

female male

In most contexts that seem to demand a character string, the 1 is translated into female and the 2 into male. The values female and male are the levels of the factor. By default, the levels are in sorted order for the data type from which the factor was formed, so that female precedes male. Hence: > levels(gender) [1] "female" "male"

Note that if gender had been an ordinary character vector, the outcome of the above levels command would have been NULL. The order of the factor levels is used, in graphs and tables, to determine the order in which the levels will appear. To cause male to come before female, use gender <- factor(gender, levels=c("male", "female"))

This syntax is available both when the factor is first created, and later to change the order in an existing factor. Take care that the level names are correctly spelled. For example,

14

A brief introduction to R

specifying "Male" in place of "male" in the levels argument will cause all values that were "male" to be coded as missing. Note finally the function ordered(), which generates factors whose values can be compared using the relational operators <, <=, >, >=, ==, and !=. Ordered factors are appropriate for use with ordered categorical data. See Section 14.6 for further details.

1.2.8 Time series

The following are the numbers of workers (in 1000s) in the Canadian prairies for each month from January 1995 through December 1996:1 numjobs <- c(982,981,984,982,981,983,983,983,983,979,973,979, 974,981,985,987,986,980,983,983,988,994,990,999)

The function ts() converts numeric vectors into time series objects. Frequently used arguments of ts() are start, frequency, and end. The following turns numjobs into a time series, which can then be plotted: numjobs <- ts(numjobs, start=1995, frequency = 12) plot(numjobs)

Use the function window() to extract a subset of the time series. For example, the following extracts the last quarter of 1995 and the first few months of 1996: first15 <- window(numjobs, start=1995.75, end=1996.25)

Multivariate time series can also be handled. See Subsections 2.1.5 and 14.9.7.

1.3 Data frames and matrices Data frames are fundamental to the use of the R modeling and graphics functions. A data frame is a more general object than a matrix, in the sense that different columns may have different modes. All elements of any column must, however, have the same mode, i.e., all numeric, or all factor, or all character, or all logical. Included in the DAAG package is Cars93.summary, created from the Cars93 data set in the MASS package. Its contents are: > Cars93.summary Min.passengers Max.passengers No.of.cars abbrev Compact 4 6 16 C Large 6 6 11 L Midsize 4 6 22 M Small 4 5 21 Sm Sporty 2 4 14 Sp Van 7 8 9 V

1

## Alternatively, obtain from data frame jobs (DAAG) library(DAAG) numjobs <- jobs$Prairies

1.3 Data frames and matrices

15

The first three columns are numeric, and the fourth is a factor. Use the function class() to check this, e.g., enter class(Cars93.summary$abbrev). (The classification of objects into classes is discussed in Subsection 1.4.2.) On most systems, use of edit() allows access to a spreadsheet-like display of a data frame or of a vector, where entries can be edited or new data added. For example, Cars93.summary <- edit(Cars93.summary)

To close the spreadsheet, click on the File menu and then on Close. On Linux systems, click on Quit to exit. Displaying the first few, or last few, rows of a data frame

When used with a data frame (other possible arguments include vectors and functions), the head() function displays the first lines of a data frame, while tail() displays the last lines. For example, > head(Cars93.summary, n=3) Min.passengers Compact 4 Large 6 Midsize 4 > # . . .

# Display the first 3 rows (the default is 6) Max.passengers No.of.cars abbrev 6 16 C 6 11 L 6 22 M

Note also the functions str() and summary(), both of which can be used to get summary information on data frames, different in the two cases. Column and row names

The function rownames() extracts the names of rows, while colnames() extracts column names, thus: rownames(Cars93.summary) colnames(Cars93.summary)

# Extract row names # Extract column names

For use with data frames row.names() is an alternative to rownames(), while names() is an alternative to colnames(). The functions names() (or colnames()) and rownames() can also be used to assign new names. For example: names(Cars93.summary)[3] <- "numCars" names(Cars93.summary) <- c("minPass","maxPass","numCars","code")

Subsets of data frames

Data frames are indexed by row and column number. Thus to extract the element in the 4th row and 2nd column, specify Cars93.summary[4, 2]. Here are additional examples:

16

A brief introduction to R

Cars93.summary[1:3, 2:3] # Rows 1-3 and columns 2-3 Cars93.summary[, 2:3] # Columns 2-3 (all rows) Cars93.summary[, c("No.of.cars", "abbrev")] # Cols 2-3, by name Cars93.summary[, -c(2,3)] # omit columns 2 and 3

The subset() function offers an alternative way to extract rows and columns. For example, the following extracts the first two rows: subset(Cars93.summary, subset=c(TRUE, TRUE, FALSE, FALSE, FALSE, FALSE))

Use the argument select to specify a subset of columns. See help(subset) for details. Use of the subscript notation to extract a column, as in Cars93.summary[, 1], returns a vector. By contrast, extraction of the raw Cars93.summary[1, ] returns a data frame, necessary because this allows different elements (columns) to retain their existing classes. Note also r Use of unlist(Cars93.summary[1, ]) returns a vector, but with the side-effect that the factor value in the final column is coerced to numeric. Such side-effects are usually undesirable, with a result that may be meaningless. r Avoid Cars93.summary[4], at least until the subtleties of its use are understood. See Subsection 14.9.1. If used where Cars93.summary[, 4] was intended, the calculation may fail or give an erroneous result.

Data frames are a specialized type of list

A list is an arbitrary collection of R objects. Here is a simple example, containing two character vectors of differing lengths and a numeric vector: > ## Cities with more than 2.5 million inhabitants > USACanada <- list(USACities=c("NY", "LA", "Chicago"), + CanadaCities=c("Toronto", "Montreal"), + millionsPop=c(USA=305.9, Canada=31.6)) > > USACanada $USACities [1] "NY" "LA" "Chicago" $CanadaCities [1] "Toronto"

"Montreal"

$millionsPop USA Canada 305.9 31.6

Many of R’s modeling functions return their output as a list. Lists can be joined using the function c(); in this and in several other respects they are “vectors”. Important aspects of the syntax for working with data frames apply also to lists. Obviously, however,

1.3 Data frames and matrices

17

notions of “row” and “column” have no relevance to lists. See Subsection 14.9.1 for further commentary. 1.3.1 Accessing the columns of data frames – with() and attach()

In repeated computations with the same data frame, it is tiresome to keep repeating the name of the data frame. The function with() is often helpful in this connection. Thus, an alternative to c(mean(cfseal$weight), median(cfseal$weight)) is: > ## cfseal (DAAG) has data on Cape Fur seals > with(cfseal, c(mean(weight), median(weight))) [1] 54.8 46.2

Curly brackets (braces) can be used to extend the scope of with() over several lines of code: > with(pair65, # stretch of rubber bands, from DAAG + {lenchange <- heated-ambient + c(mean(lenchange), median(lenchange)) + }) [1] 6.33 6.00

An alternative is attach(). Once a data frame has been attached, its columns can be referred to by name, without further need to give the name of the data frame. For example: > year Error: Object "year" not found > attach(fossilfuel) # Attach data frame fossilfuel > year [1] 1800 1850 1900 1950 2000 > detach(fossilfuel) # Detach data frame

Be sure to detach data frames that are no longer in use. If more than one data frame is attached that has the column year, there is obvious scope for confusion. What happens if there is an object year in the workspace? References to year will then take the object that is in the workspace, ignoring the column year in the attached data frame. By contrast, use of with() ensures that the column, if present, is from the specified data frame. The attaching of a data frame extends the search list, which is the list of “databases” where R looks for objects. See Section 14.2 for more details on this and other uses of attach(). 1.3.2 Aggregation, stacking, and unstacking

The aggregate() function yields a data frame that has the mean or value of another specified function for each combination of factor levels. As an example, consider the chickwts data frame which contains observations on the weights of 71 six-week-old chicks who have been fed one of six kinds of feed. The columns of chickwts are named weight and feed. To find the average weights for the different feed groups, type

18

A brief introduction to R

> chickwtAvs <- with(chickwts, + aggregate(weight, by=list(feed), mean)) > names(chickwtAvs) <- c("Feed Group", "Mean Weight") > chickwtAvs Feed Group Mean Weight 1 casein 323.5833 2 horsebean 160.2000 3 linseed 218.7500 . . . .

See Subsection 14.9.5 for more information on the aggregate() function. For stacking columns of a data frame, i.e., placing successive columns one under the other, the function stack() is available. The following use of stack(), with data from the data frame jobs, will be required for the use of these data in Subsection 2.1.5. > library(DAAG) > head(jobs,3) BC Alberta Prairies Ontario Quebec Atlantic Date 1 1752 1366 982 5239 3196 947 95.00000 2 1737 1369 981 5233 3205 946 95.08333 3 1765 1380 984 5212 3191 954 95.16667 > # . . . > Jobs <- stack(jobs, select = 1:6) > # stack() concatenates selected data frame columns into a > # single column named "values", & adds a factor named "ind" > # that has the names of the concatenated columns as levels. > head(Jobs,3) values ind 1 1752 BC 2 1737 BC 3 1765 BC > # . . .

For a further example, see Exercise 19. The unstack() function reverses the stacking operation. For example, unstack (Jobs) (or more generally, unstack(Jobs, values ˜ ind)) recovers the original data frame.

1.3.3 ∗ Data frames and matrices

The numeric values in the data frame fossilfuel might alternatively be stored in a matrix with the same dimensions, i.e., 5 rows × 2 columns. The following enters these same data as a matrix: fossilfuelmat <- matrix(c(1800, 1850, 1900, 1950, 2000, 8, 54, 534, 1630, 6611), nrow=5) colnames(fossilfuel) <- c("year", "carbon")

1.4 Functions, operators, and loops

19

Another possibility is the use of the function cbind() to combine two or more vectors of the same length and type together into a matrix, thus: fossilfuelmat <- cbind(year=c(1800, 1850, 1900, 1950, 2000), carbon=c(8, 54, 534, 1630, 6611))

More generally, any data frame where all columns hold data that is all of the same type, i.e., all numeric or all character or all logical, can alternatively be stored as a matrix. Storage of numeric data in matrix rather than data frame format can speed up some mathematical and other manipulations when the number of elements is large, e.g., of the order of several hundreds of thousands. For further details, see Section 14.8. Note that: r Matrix elements are stored in column order in one long vector, i.e., columns are stacked one above the other, with the first column first. Section 14.8 describes how to change between a matrix with m rows and n columns, and a vector of length mn. r The extraction of submatrices has the same syntax as for data frames. Thus, fossilfuelmat[2:3,] extracts rows 2 and 3 of the matrix fossilfuelmat. (Be careful not to omit the comma, causing the matrix to be treated as one long vector.) r The names() function returns NULL when the argument is a matrix. Note however rownames() and colnames(), which can be used either with data frames or matrices. r The function nrow() (e.g. nrow(fossilfuel) or nrow(fossilfuelmat)) returns the number of rows, while ncol() returns the number of columns. 1.4 Functions, operators, and loops Functions are integral to the use of the R language. User-written functions are used in exactly the same way as built-in functions. Examples will appear from time to time through the book. An incidental advantage of putting code into functions is that the workspace is not then cluttered with objects that are local to the function. 1.4.1 Common useful built-in functions all() any() args() cat() cumprod() cumsum() diff()

# # # # # # # # history() # is.factor() # is.na() # # length() # ls() #

returns TRUE if all values are TRUE returns TRUE if any values are TRUE information on the arguments to a function prints multiple objects, one after the other cumulative product cumulative sum form vector of first differences N. B. diff(x) has one less element than x displays previous commands used returns TRUE if the argument is a factor returns TRUE if the argument is an NA NB also is.logical(), is.matrix(), etc. number of elements in a vector or of a list list names of objects in the workspace

20 mean() median() order() print() range() sort() rev() str() unique() which() which.max() which.min() with()

A brief introduction to R # # # # # # # # # # # # #

mean of the elements of a vector median of the elements of a vector x[order(x)] sorts x (by default, NAs are last) prints a single R object minimum and maximum value elements of vector sort elements into order, by default omitting NAs reverse the order of vector elements information on an R object form the vector of distinct values locates ’TRUE’ indices of logical vectors locates (first) maximum of a numeric vector locates (first) minimum of a numeric vector do computation using columns of specified data frame

Be sure to check, where this is relevant, the handling of missing values. In case of doubt, consult the relevant help page. Refer back to Subsection 1.2.6. The print() function

This is perhaps R’s most pervasive function. It is invoked whenever an object, or the result of a computation, has its value returned to the command line. For example: > x <- 2 > x [1] 2 > x*5 [1] 10

# Assign to x the value 2; nothing is printed # Equivalent to print(x) # Equivalent to print(x*5)

It can be convenient to make an assignment and print the value. For this, enclose the assignment in parentheses, i.e., in round brackets: > (x <- 2) [1] 2

# Equivalent to: x <- 2; print(x)

Calculations in parallel across all elements of a vector

Subsection 1.1.2 gave an example in which arithmetic was carried out in parallel across all elements of a vector. Many of R’s functions likewise operate in parallel on all elements of arrays, matrices, and data frames.

Data summary functions – table() and sapply()

Data summary functions that create tables of counts are: table() xtabs()

# Form a table of counts # Form a table of totals

For example, the tinting data frame in DAAG contains columns specifying the sex (sex, levels are f and m) and age group (agegp, levels are younger and older) of

1.4 Functions, operators, and loops

21

participants in a study. The table() function can be used to count up the numbers of observations in each sex–age group combination: > library(DAAG) # tinting is from DAAG > table(Sex=tinting$sex, AgeGroup=tinting$agegp) AgeGroup Sex younger older f 63 28 m 28 63

By default, table() ignores NAs. For further details of table(), and for an example of the use of xtabs(), see Subsection 2.2.1. The function sapply() applies a function to each column of a data frame, or to each element of a list. The following demonstrates its use to give the range, for all columns of the data frame jobs (DAAG): > sapply(jobs[, -7], range) BC Alberta Prairies Ontario Quebec Atlantic [1,] 1737 1366 973 5212 3167 941 [2,] 1840 1436 999 5360 3257 968

Utility functions

Type ls() (or objects()) to see the names of all objects in the workspace. One can restrict the names to those with a defined pattern, e.g., starting with the letter p:2 ls(pattern="p") ls(pattern="ˆp")

# List object names that include the letter "p" # List object names that start with "p"

Type help(ls), help(grep), and help(glob2rx) for more details. Various packages will add hidden files, whose first character is a full stop, to the workspace. To see these files, type ls(all=TRUE). To clear the workspace completely, type rm(list=ls(all=TRUE)). By default, the function dir() lists the contents of the working directory. See Subsection 14.2.3 for further details on this and other utility functions.

1.4.2 Generic functions, and the class of an object

The printing of a data frame requires steps that are different from those for the printing of a vector of numbers. Yet, in R, the same print() function handles both tasks. In order to make this possible, all objects in R have a class, which can be used to decide how the printing should be handled. The print() function does not itself attend to the printing. Instead, if print() is called with a factor argument, print.factor() is used. For a data frame print.data.frame() is used, and so on. Section 14.10 gives further details. 2

More generally, the pattern-matching conventions are the same as for grep(), which is modeled on the Unix grep command.

22

A brief introduction to R

For objects (such as numeric vectors) that do not otherwise have a print method, print.default() handles the printing. For simple objects such as numbers and text strings, the class is determined informally. More complex objects such as data frames carry a tag (an attribute) that specifies the class. In either case, the function class() can be used to determine the class. See Section 14.10 for further details.

1.4.3 User-written functions

Here is a function that returns the mean and standard deviation of a vector of numbers: mean.and.sd <- function(x){ av <- mean(x) sdev <- sd(x) c(mean=av, SD=sdev) }

Having constructed the function, we can apply it to a numeric vector, as in the following: > distance <- c(148,182,173,166,109,141,166) > mean.and.sd(distance) mean SD 155.00 24.68

The variables av and sdev are local to the function. They cannot be accessed outside of the internal function environment. Many functions have default arguments which make it possible to run them without specifying any data. We can modify the above function to have the default argument x = rnorm(10). This generates a vector of 10 random numbers to which the function is then applied. mean.and.sd <- function(x = rnorm(10)){ av <- mean(x) sdev <- sd(x) c(mean=av, SD=sdev) }

Here is the result of one execution of the modified function: > mean.and.sd() mean SD 0.6576272 0.8595572

The structure of functions

The function mean.and.sd() has the following structure:

1.4 Functions, operators, and loops

23

argument(s) function name  
 
mean.and.sd <- function(x=rnorm(10)) { function av <- mean(x) body sdev <- sd(x) return c(av = av, sd = sdev) value } If the function body consists of just one statement that gives the return value, the curly braces ({ }) are unnecessary. The return value, which must be a single object, is given by the final statement of the function body. In the example above, the return value was the vector consisting of the two named elements mean and sdev. For returning several objects that are of different types, join them into a list.3 1.4.4 if Statements

Subsection 1.2.4 introduced the use of relational operators to create particular subsets of a given vector. The R system also has the flow control capabilities of traditional programming languages, including if statements. The if function tests the truth of a given statement; if the statement is true, the succeeding expression is evaluated. An else can be added to provide an alternative expression to be evaluated in the case where the given statement is false. For example, the following checks whether the mean for the carbon emissions exceeds the median: Carbon <- fossilfuel$carbon > if (mean(Carbon) > median(Carbon)) print("Mean > Median") else + print("Median <= Mean") [1] "Mean > Median"

Here is another example: > dist <- c(148, 182, 173, 166, 109, 141, 166) > dist.sort <- if (dist[1] < 150) + sort(dist, decreasing=TRUE) else sort(dist) > dist.sort [1] 182 173 166 166 148 141 109

1.4.5 Selection and matching

A highly useful operator is %in%, used for testing set membership. For example: > x <- rep(1:5, rep(3,5)) [1] 1 1 1 2 2 2 3 3 3 4 4 4 5 5 5 > x[x %in% c(2,4)] [1] 2 2 2 4 4 4 3

## Thus, to return the mean, SD and name of the input vector ## replace c(mean=av, SD=sdev) by list(mean=av, SD=sdev, dataset = deparse(substitute(x)))

24

A brief introduction to R

We have picked out those elements of x that are either 2 or 4. To find which elements of x are 2s, which 4s, and which are neither, use match(). Thus: > match(x, c(2,4), nomatch=0) [1] 0 0 0 1 1 1 0 0 0 2 2 2 0 0 0

The nomatch argument specifies the symbol to be used for elements that do not match. Specifying nomatch=0 is often preferable to the default, which is NA. 1.4.6 Functions for working with missing values

Recall the use of the function is.na(), discussed in Subsection 1.2.6, to identify NAs. Testing for equality with NAs does not give useful information. Identification of rows that include missing values

Many of the modeling functions will fail unless action is taken to handle missing values. Two functions that are useful for identifying or handling missing values are complete.cases() and na.omit(). Applying the complete.cases() function to a data frame returns a logical vector whose length is the number of rows and whose TRUE values correspond to rows which do not contain any missing values. Thus, the following identifies rows that hold one or more missing values: > ## Which rows have missing values: data frame science (DAAG) > science[!complete.cases(science), ] State PrivPub school class sex like Class 671 ACT public 19 1 5 19.1 672 ACT public 19 1 5 19.1

The function na.omit() omits any rows that contain missing values. For example, > dim(science) [1] 1385 7 > Science <- na.omit(science) > dim(Science) [1] 1383 7

It should be noted that there may be better alternatives to omitting missing values. There is an extensive discussion in Harrell (2001, pp. 43–51). Often, the preferred approach is to estimate the values that are missing as part of any statistical analysis. It is important to consider why values are missing – is the probability of finding a missing value independent of the values of variables that appear in the analysis? 1.4.7 ∗ Looping

A simple example of a for loop is4 4

Other looping constructs are repeat <expression> # Place break somewhere inside while (x > 0) <expression> # Or (x < 0), or etc. Here <expression> is an R statement, or a sequence of statements that are enclosed within braces.

1.5 Graphics in R

25

> for (i in 1:3) print(i) [1] 1 [1] 2 [1] 3

Here is a way to estimate the increase in population for each of the Australian states and territories between 1917 and 1997, relative to 1917, using the data frame austpop. Columns are 1: census year (by decade from 1917 through 1997); 2–9: the state and territory populations that are of interest here; and 10: the national population. > > > > > + + > > >

## Relative population increase in Australian states: 1917-1997 ## Data frame austpop (DAAG) relGrowth <- numeric(8) # numeric(8) creates a numeric vector # with 8 elements, all set equal to 0 for (j in seq(from=2, to=9)) { relGrowth[j-1] <- (austpop[9, j]-austpop[1, j])/ austpop[1, j]} names(relGrowth) <- names(austpop[c(-1,-10)]) # We have used names() to name the elements of relGrowth relGrowth # Output is with options(digits=3) NSW Vic Qld SA WA Tas NT ACT 2.30 2.27 3.98 2.36 4.88 1.46 36.40 102.33

Often, there is a better alternative to the use of a loop. See Subsection 14.5.3. 1.5 Graphics in R Later chapters will make extensive use both of base graphics (using plot(), etc.) and of the more stylized graphs provided by lattice graphics. This section is a brief introduction to plot() and allied functions that are included in R’s base graphics. Subsection 1.5.8 is a brief introduction to the more stylized graphical functions in the lattice package. Note also the carefully structured abilities of the ggplot2 package, described in Section 15.2. Base graphics are provided by the graphics package that is automatically attached at startup. It includes the function plot() for creating scatterplots, and the functions points(), lines(), text(), mtext(), and axis() that add to existing plots. There is a wide range of other functions. To see some of the possibilities, enter demo(graphics)

Press the Enter key to move to each new graph. 1.5.1 The function plot( ) and allied functions

The data frame primates gives Bodywt and Brainwt, for five primate species. A basic plot of Brainwt against Bodywt can be obtained thus: plot(Brainwt ˜ Bodywt, data=primates)

# plot(y ˜ x) syntax

or with(primates, plot(Bodywt, Brainwt))

# plot(x, y) syntax

A brief introduction to R 1500

26

500

1000

Brain weight (g)

Human

Chimp

Gorilla

Brainwt

10.0 207.0 62.0 6.8 52.2

115 406 1320 179 440

Potar monkey Gorilla Human Rhesus monkey Chimp

0

Rhesus monkey Potar monkey

Bodywt

0

50

100 150 200 250 300

Body weight (kg)

Figure 1.2 Brain weight (g) versus body weight (kg). Data are from the primates data frame.

There are many possible refinements. A number of the most important are illustrated in Plate 1, which supplements the discussion that follows. Adding points, lines, text, and axis annotation

Figure 1.2 puts labels on the points. ## Place labels on points plot(Brainwt ˜ Bodywt, xlim=c(0, 300), data=primates) # Specify xlim so that there is room for the labels with(primates, text(Brainwt ˜ Bodywt, labels=row.names(primates), pos=4)) # pos=4 places text to the right of the points. Other # possibilities are: 1: below; 2: to the left; 3: above

Figure 1.2 has two further refinements. The y-axis limits were extended slightly. Small vertical offsets were incorporated that raised the label Rhesus monkey and lowered the label Potar monkey, avoiding overlap.5 Use points() to add points to a plot. Use lines() to add lines. Actually these are aliases, differing only in the default for the parameter type; points() has type = "p", while lines() has type = "l". The function mtext(text, side, line, ...) adds text in the margin of the current plot. The sides are numbered 1 (x-axis), 2 (y-axis), 3 (top), and 4 (right vertical axis). By default, adj=0.5, which centers the text at the axis midpoint. Specify adj=0 to position the left extreme of the text at the left margin, and adj=1 to position its right extreme at the right margin. The axis() function gives fine control over axis ticks and labels. To use for the x-axis, plot the initial graph with xaxt="n". Then call axis() with the argument side=1, and with other arguments as required. See help(axis) for details. 5

## Plot Brainwt vs Bodywt, primates data frame plot(Brainwt ˜ Bodywt, xlim=c(0, 300), ylim=c(0,1500), data=primates) yoff <- c(-.125,0,0,.125,0)*par()$cxy[2] with(primates, text(x=Bodywt, y=Brainwt+yoff, labels=row.names(primates), pos=4))

1.5 Graphics in R

27

Fine control – parameter settings

Here are some of the parameters that commonly require attention: r Plotting symbols: pch (choice of symbol); cex (“character expansion"); col (color). Thus par(cex=1.2) increases the plot symbol size 20% above the default. r Lines: lty (line type); lwd (line width); col (color). r Axis limits: xlim; ylim. (Assuming xaxs="r", x-axis limits are by default extended by 4% relative to the data limits. Specify xaxs="i" to make the default an exact fit to the data limits. For the y-axis, replace xaxs by yaxs.) r Axis annotation and labels: cex.axis (character expansion for axis annotation, independently of cex); cex.labels (size of the axis labels); mgp (margin line for the axis title, axis labels, and axis line; default is mgp=c(3, 1, 0)). r Graph margins: mar (inner margins, clockwise from the bottom; the out-of-the-box default is mar=c(5.1, 4.1, 4.1, 2.1), in lines out from the axis); oma (outer margins, relevant when there are multiple graphs on the one graphics page). r Plot shape: pty="s" gives a square plot (must be set using par()). r Multiple graphs on the one graphics page: Specify par(mfrow=c(m,n)) to get an m rows by n columns layout of graphs on a page. The 1 by 4 layout of plots in Figure 2.1 of Chapter 2 was obtained using par(mfrow=c(1,4)). Type help(par) to get a (very extensive) complete list. Figure 15.1 and Plate 1 demonstrate some of the possibilities. In most (not all) instances, the change can be made either in a call to a plotting function (e.g., plot(), points()), or using par(). If made in a call to a plotting function, the change applies only to that call. If made using par(), changes remain in place until changed again, or until a new device is opened. It can be helpful to store the existing settings, so that they can be restored later. For this, specify, for example: oldpar <- par(cex=1.25) # Use par(oldpar) to restore previous settings

1.5.2 The use of color

The default palette, which can be changed, has eight colors including “white”. These are a small selection from the built-in colors. The function colors() returns the 657 names of the built-in colors, some of them aliases for the same color. In the following, points are in the colors of the current palette. These are recycled as necessary. theta <- (1:50)*0.92 plot(theta, sin(theta), col=1:50, pch=16, cex=4) points(theta, cos(theta), col=51:100, pch=15, cex=4) palette() # Names of the colors in the current palette

The following repeats the plot, but now using the function colors() to supply two sets of 50 (mostly) different colors:

28

A brief introduction to R

plot(theta, sin(theta), col=colors()[1:50], pch=16, cex=4) points(theta, cos(theta), col=colors()[51:100], pch=15, cex=4)

Where data from a two-way layout are presented on the one panel, different symbols can be used for the different levels of one of the classifying factors, with different colors used for the different levels of the other classifying factor. Care may be required in the choice of colors, so that the colors show with clarity the distinctions that are required, and do not clash. Section 15.2 has further discussion of color palettes.

1.5.3 The importance of aspect ratio

Attention to aspect ratio is often crucial for creating graphs that reveal important features of the data. The following simple graphs highlight this point: ## Plot sin(theta) vs theta, at regularly spaced values of theta ## sin() expects angles to be in radians # multiply angles in degrees by pi/180 to get radians plot((0:20)*pi/10, sin((0:20)*pi/10)) plot((1:50)*0.92, sin((1:50)*0.92))

Readers might show the second of the graphs that now follows to their friends, asking them to identify the pattern! By holding with the left mouse button on the lower border until a double-sided arrow appears and dragging upwards, the vertical dimension of the graph sheet can be shortened. If sufficiently shortened, the pattern becomes obvious. The eye has difficulty in detecting slope patterns where the slope is close to the horizontal or to the vertical. Then try this: par(mfrow=c(3,1)) # Gives a 3 by 1 layout of plots plot((1:50)*0.92, sin((1:50)*0.92)) par(mfrow=c(1,1))

See Section 2.1 for further examples.

1.5.4 Dimensions and other settings for graphics devices

The shape of the graph sheet can be set when a new graphics page is started. On Microsoft Windows systems, the function windows() (or win.graph()) starts a new graphics page on the screen display. On Unix X11 systems, specify x11(). Under Macintosh OS X, use quartz(). Available arguments include height (in inches), width (in inches), and pointsize (there are 72.27 to an inch). The choice of pointsize, with a default that varies between devices, affects character heights.6 See help(Devices) for a full list of the devices, including hardcopy devices, that are available on the particular system that is in use. 6

Note that once a graph has been pasted (from the clipboard) or imported into Microsoft Word or Open Office or another similar word processor, it can be enlarged or shrunk by pointing at one corner, holding down the left mouse button, and pulling.

1.5 Graphics in R

29

1.5.5 The plotting of expressions and mathematical symbols

In commands such as text() and mtext(), character strings can be replaced by expressions. For this purpose an expression is more general than an algebraic or mathematical expression. Thus, the following code gives a grayed out circle, overlaid as in Plate 1B with the formula for the area of a circle: symbols(x=1.5, y=0, circles=1.2, xlim=c(0,3), ylim=c(-1.5,1.5), bg="gray", inches=FALSE) # inches=FALSE ensures that radius is in x-axis units text(1.5, 0.5, expression("Area" == pi*phantom("’")*italic(r)ˆ2)) # Use ’==’ to insert ’=’. # Text or symbols that appear either side of ’*’ are juxtaposed. # Notice the use of phantom("’") to insert a small space.

By default, symbols(), like plot(), starts a new graphics frame. Various alternatives to circles are available; see help(symbols) for details.7 Type help(plotmath) to get details of available forms of expression. Run demo(plotmath) to see some of the possibilities for plotting mathematical symbols. There are further brief details in Section 15.3. Figures 5.3, 10.7, and 15.2 will demonstrate the use of expressions in annotation and/or labeling.

1.5.6 Identification and location on the figure region

Following the drawing of the initial graph, the two functions that may be used are: r identify() labels points; r locator() prints the co-ordinates of points. In either case, the user positions the cursor at the location for which co-ordinates are required, and clicks the left mouse button. Depending on the platform, the identification or labeling of points may be terminated by pointing outside of the graphics area and clicking, or by clicking with a button other than the first. If continued, the process will terminate after some default number n of points, which the user can set. (For identify() the default setting is the number of data points, while for locator() the default is 500.) As an example, identify two of the plotted points on the primates scatterplot: plot(Brainwt ˜ Bodywt, data=primates) with(primates, identify(Brainwt ˜ Bodywt, labels=row.names(primates), n=2)) # Now click near 2 plotted points

7

## To add the double-headed arrow and associated label, specify: arrows(1.5, 0, 2.7, 0, length=.1, code=3) # code=3: arrows at both ends # length is the length of the arrow head (in inches!) text(2.1, -strheight("R"), expression(italic(r) == 1.2))

30

A brief introduction to R

1.5.7 Plot methods for objects other than vectors

We have seen how to plot a numeric vector y against a numeric vector x. The plot function is a generic function that also has special methods for “plotting” various different classes of object. For example, plot() accepts a data frame as argument. Try ## Use plot() with data frame trees (datasets) plot(trees) # Gives a 3 x 3 layout of pairwise # scatterplots among the three variables

This has the same effect as the function call pairs(trees). The scatterplot matrix will be used extensively in Chapter 6 for scrutiny of regression data. See, for example, Subsection 6.2.3. It will be an important tool, also, in the account of multivariate methods in Chapter 12.

1.5.8 Lattice (trellis) graphics

Many of the analyses in later chapters compare different groups within the data. Visual assessments that complement the analysis are indispensable. The lattice package has abilities that are suited to such use. The layout on the page, the choice of plotting symbols and colors, and the distinctions within panels, can be used to represent important aspects of data structure. The syntax and graphics conventions are highly consistent across all lattice functions. Lattice’s relatively automated provision of highly structured graphical layouts has a cost – changes to the basic layout and structure may be complicated.

Lattice graphics versus base graphics – xyplot() versus plot()

A Brainwt versus Bodywt scatterplot for the primates data, such as was given earlier, might alternatively have been obtained using the function xyplot() from the lattice package. The following, when typed on the command line, give a plot on the graphics device: ## Plot Brainwt vs Bodywt, data frame primates (DAAG) plot(Brainwt ˜ Bodywt, data=primates) # base graphics # ’base’ graphics use the abilities of the graphics package library(lattice) xyplot(Brainwt ˜ Bodywt, data=primates) # lattice

The mechanism that yields the plot is different in the two cases: r plot() gives a graph as a side-effect of the command. r xyplot() generates a graphics object. As this is output to the command line, the object is “printed”, i.e., a graph appears. The following illustrates the difference between the two functions: invisible(plot(Brainwt ˜ Bodywt, data=primates)) # Graph appears invisible(xyplot(Brainwt ˜ Bodywt, data=primates)) # No graph

1.5 Graphics in R 50

Row

60

70

31 80

90

Swim

ht

190

f m

180 170 160

50

60

70

80

90

wt

Figure 1.3 Height (ht) versus weight (wt), for two categories of athlete. The different plotting symbols distinguish males from females. The data relate to Telford and Cunningham (1991).

The wrapper function invisible() suppresses command line printing, so that invisible(xyplot(...)) does not yield a graph. Inside a function, xyplot(...) prints a graph only if it is the return value from the function, i.e., usually, is on the final line. In a file that is sourced (use source()), no graph will appear. Inside a function (except as mentioned), or in a file that is sourced, there must be an explicit print(), i.e., print(xyplot(ACT ˜ year, data=austpop))

Panels of scatterplots – the use of xyplot()

Graphics functions in the lattice package are designed to allow row by column layouts of panels. Different panels are for different subsets of the data. Additionally, points can be distinguished, within panels, according to some further grouping within the data. Chapter 2 will make extensive use of lattice functions. Figure 1.3 demonstrates the use of xyplot() with the ais data set (DAAG) that has data on elite Australian athletes who trained at the Australian Institute of Sport. The plot is restricted to rowers and swimmers. The two panels distinguish the two sports, while different plotting symbols (on a color device, different colors will be used) distinguish females from males. Here is suitable code: trellis.device(color=FALSE) xyplot(ht ˜ wt | sport, groups=sex, pch=c(4,1), aspect=1, data=ais, auto.key=list(columns=2), subset=sport%in%c("Row","Swim")) dev.off() # Close device trellis.device() # Start new device, by default with color=TRUE

In the graphics formula ht ˜ wt | sport, the vertical bar indicates that what follows, in this case sport, is a conditioning variable or factor. The graphical information is broken down according to the factor levels or distinct values. The parameter aspect controls the ratio of dimensions in the y and x directions. The setting auto.key=list(columns=2) generates a simple key, with the two key items side by side in two columns rather than one under another in a single column as happens with the default setting columns=1.

32

A brief introduction to R

Plotting columns in parallel

Variables and/or factors can be plotted in parallel, on the same (outer=FALSE) or different (outer=TRUE) panel(s). Separate the names with “+”. The following gives a simplified version of Figure 2.10 in Subsection 2.1.5: xyplot(Prairies+Atlantic ˜ Date, outer=TRUE, data=jobs)

The data frame jobs has changes in number of jobs in different regions of Canada over the period January 1995 to December 1996. Subsection 15.5.1 has further discussion on the plotting of columns in parallel.

Selected lattice functions dotplot(factor ˜ numeric,..) # 1-dim. Display stripplot(factor ˜ numeric,..) # 1-dim. Display barchart(character ˜ numeric,..) histogram( ˜ numeric,..) densityplot( ˜ numeric,..) # Density plot bwplot(factor ˜ numeric,..) # Box and whisker plot qqmath(factor ˜ numeric,..) # normal probability plots splom( ˜ dataframe,..) # Scatterplot matrix parallel( ˜ dataframe,..) # Parallel coordinate plots cloud(numeric ˜ numeric * numeric, ...) # 3D surface wireframe(numeric ˜ numeric * numeric, ...) # 3D scatterplot

In each instance, users can add conditioning variables. Further points to note about the lattice package are: r Because the lattice package implements the trellis style of graphics, several of the functions that control stylistic features (color, plot characters, line type, etc.) have trellis (where lattice might have seemed more natural) as part of their name. r Lattice graphics functions cannot be mixed (or not easily) with the graphics functions discussed earlier in Section 1.5. It is not possible to use points(), lines(), text(), etc., to add features to a plot that has been created using a lattice graphics function. Instead, it is necessary to use functions that are special to lattice – lpoints(), llines(), ltext(), larrows(), and lsegments(). Subsection 15.5.5 describes a mechanism for interacting with lattice plots.

1.5.9 Good and bad graphs

There is a difference! Draw graphs so that they are unlikely to mislead. Ensure that they focus the eye on features that are important, and avoid distracting features. Lines that are intended to attract attention can be thickened. In scatterplots, the intention is typically to draw attention to the points. If there are not too many of them, the use of heavy black dots or other filled symbols will focus attention

1.6 Additional points on the use of R

33

on the points, rather than on a fitted line or curve or on the axes. If they are numerous and there is substantial overlap, it then makes better sense to use open symbols. Where there is extensive overlap, ink will fill that region more densely. If there is so much overlap that the use of black symbols would merge most points into a dense black mass, use of a shade of gray may be helpful.8 Where the horizontal scale is continuous, patterns of change that are important to identify should bank at an angle of roughly 45◦ above or below the horizontal. Depending on the context, angles in the approximate range 20◦ to 70◦ may be satisfactory, and the aspect ratio should be chosen accordingly. (This was the point of the sine curve example in Subsection 1.5.3.) See Cleveland (1994) for further commentary. Colors, or gray scales, can often be used to distinguish groupings in the data. Bear in mind that the eye has difficulty in focusing simultaneously on widely separated colors that are close together on the same graph. 1.5.10 Further information on graphics

Several further graphics functions will be introduced in Section 2.1. Note especially hist() and boxplot(). See also Murrell (2005), Sarkar (2002, 2007). Note the more detailed information in Chapter 15. Section 15.5 has an extended discussion of the abilities of the lattice package. There is brief reference to the relatively specialist abilities of the grid package, or which lattice is built. Section 15.6 discusses the ggplot2 package. 1.6 Additional points on the use of R ∗

Workspace management strategies

The default choice of working directory, which may be an R installation directory, is not a good choice for long-term use, and should be changed. Subsection 1.1.1 explained how to set the startup choice of working directory. The working directory can be changed and a new workspace loaded in the course of a session, either using the menu system, if available, or using command line instructions. See Subsection 14.2.2. In a session where there are extensive calculations, cautious users will from time to time save the current workspace, perhaps first using rm() to remove objects that are no longer required. The command save.image() will save everything in the workspace, by default into the file .RData in the working directory. This can alternatively be done by clicking on the relevant menu item, where such a menu is available. (The file that is saved holds an image of the workspace at that point.) It is good practice to use a separate working directory for each different project. The ability to keep multiple image files in the one directory adds further flexibility. Use the extension .RData for such files.9 8 9

## Example of plotting with different shades of gray plot(1:4, 1:4, pch=16, col=c("gray20","gray40","gray60","gray80"), cex=3) In older versions of R for Windows, .rda was an alternative extension.

34

A brief introduction to R

Forward slashes and backslashes

Note that R syntax follows the Unix conventions and uses forward slashes, where Windows expects backslashes. Thus to read in the file fuel.txt from the directory c:\data, type fossilfuel <- read.table("c:/data/fuel.txt") # Alternative: Replace each "/" by "\\", ie, 2 backslashes Setting the number of decimal places in output

Often, calculations will, by default, give more decimal places of output than are useful. In the output that we give, we often reduce the number of decimal places below what R gives by default. The options() function can be used to make a global change to the number of significant digits that are printed. For example: > sqrt(10) [1] 3.162278 > options(digits=2)

# Change until further notice, # or until end of session.

> sqrt(10) [1] 3.2

Note that options(digits=2) expresses a wish, which R will not always obey! Rounding will sometimes introduce small inconsistencies. For example, in the calculations of Section 4.4:  372 = 5.57 12  √ 372 2× = 7.88. 12 √ Note however that 2 × 5.57 = 7.87. Other option settings

Type help(options) to get further details. We will meet another important option setting in Chapter 5. (Most of the output that we present uses the setting options(show. signif.stars=FALSE), where the default is TRUE. This affects output in Chapter 5 and later chapters.) Cosmetic issues

In our R code, we write, e.g., a <- b rather than a<-b, and y ∼ x rather than y∼x. This is intended to help readability, perhaps a small step on the way to literate programming. Such presentation details can make a large difference when others use the code. Where output is obtained with the simple use of print() or summary(), we have in general included this as the first statement in the output.

1.7 Recap

35

∗

Common sources of difficulty

r It is important to tune the parameter settings of read.table() to the input data set. See help(read.table) and Subsection 14.4.1 for further details. r Character vectors that are included as columns in data frames become, by default, factors. There are implications for the use of read.table(). See Subsection 14.4.1 and Section 14.6. r In most contexts, factors are treated as vectors of character strings, with values given by the factor levels. Use unclass() to extract the integer values if these, rather than the levels are required. See Section 14.6. r Keep in mind Section 14.7’s comments on the handling of missing values. r The syntax fossilfuel[, 2] extracts the second column from the data frame fossilfuel, yielding a numeric vector. Observe however that fossilfuel[2, ] yields a data frame, rather than the numeric vector that the user may require. Specify unlist(fossilfuel[2, ]) to obtain the vector of numeric values in the second row of the data frame. See Subsection 14.9.1. r The function sapply() is commonly used to carry out a computation across all elements of a data frame. If used with a matrix, the computation will be carried out on all matrix elements. See Subsection 14.9.5. r Once a data frame has been attached, take care with assignments to either the name of the data frame, or the name of a column. Assignment to the name of the data frame will create a new local copy, while assigment to a column name will create a new object in the workspace with that name. Later references to those names will then access the new local copies. r Data objects that individually or in combination occupy a large part of the available computer memory can slow down all memory-intensive computations. See Subsection 14.2.2 for comment on associated workspace management issues. See also the opening comments in Section 14.8. Note that most of the data objects that are used for our examples are small and thus will not, except where memory is very small, make much individual contribution to demands on memory.

Variable names in data sets

We will refer to a number of different data sets, many of them data frames in our DAAG package. When we first introduce the data set, we will give both a general description of the columns of values that we will use, and the names used in the data frame. In later discussion, we will use the name that appears in the data frame whenever the reference is to the particular values that appear in the column.

1.7 Recap r One use of R is as a calculator, to evaluate arithmetic expressions. Calculations can be carried out in parallel, across all elements of a vector at once. r Use q() to quit from R, usually taking care to save the workspace.

36

A brief introduction to R

r Help functions include help() (help page of a known function), help.search() (search for a specified word in the help page header), apropos() (search for function names that include a specified character string), and help.start (start a browser interface to help information). r The function c() (concatenate) joins vector elements into vectors. It may be used for logical and character vectors, as well as for numeric vectors. r For simple forms of scatterplot use plot(), or the lattice function xyplot(). r Important R data structures are vectors, factors, lists, and data frames. Vectors may be of mode numeric, or logical, or character. Factors have mode “numeric” and class “factor”. r Data frames use a list structure to group columns, which must all have the same length, together into a single R object. The different columns may be any mix of logical, numeric, character, or factor. r Contrast data frames with matrices. All matrix elements have the same mode. A matrix is stored as one long vector that is formatted to appear in a row by column layout. r Use is.na() to identify elements that are NAs. r The R system has extensive abilities for inputting data from rectangular files (see help(read.table)), from spreadsheets, and from a variety of statistical package formats. The R Commander GUI offers an easy means to access these abilities. r Use attach() or with() (temporary attachment) to give access to the columns of a data frame, without the need to name the data frame whenever a column is accessed. r The search path determines the order of search for objects that are accessed from the command line, or that are not found in the enclosing environment of a function that accesses them. r Factors, used for categorical data, are fundamental to the use of many of the R modeling functions. Ordered factors are appropriate for use with ordered categorical data. r Option settings, which users can change at their discretion, control such matters as the number of significant digits that will be displayed in output. r Commonly used generic functions include print(), plot(), and summary(). For such functions, the result depends on the class of object that is given as argument. r To make an assignment and print the value that is assigned, enclose the assigment statement in round brackets. For example: (x <- 2)

# Equivalent to: x <- 2; print(x)

1.8 Further reading Note that the exercises have various hints that extend the discussion in the body of the chapter. A version of An Introduction to R (R Development Core Team, 2009a), current at the time of release, is included with the R distributions. It is available from the CRAN sites as an independent document. (For a list of sites, go to http://cran.r-project.org.) Books that include an introduction to R include Dalgaard (2008), Fox (2002). At a more advanced level note Venables and Ripley (2002), which covers both S-PLUS and R. This will be an important reference throughout this book.

1.9 Exercises

37

See also documents, including Maindonald (2008), that are listed under Contributed Documentation on the CRAN sites. For careful detailed accounts of the R language, see Chambers (2007), Gentleman (2008). Books and papers that set out principles of good graphics include Cleveland (1993, 1994), Tufte (1997), Wainer (1997), and Wilkinson and Task Force on Statistical Inference (1999). See also the imaginative uses of R’s graphical abilities that are demonstrated in Murrell (2005). Maindonald (1992) comments very briefly on graphical design.

References for further reading

Chambers, J. M. 2007. Software for Data Analysis: Programming with R. Cleveland, W. S. 1993. Visualizing Data. Cleveland, W. S. 1994. The Elements of Graphing Data, revised edn. Dalgaard, P. 2008. Introductory Statistics with R. Fox, J. 2002. An R and S-PLUS Companion to Applied Regression. Gentleman, R. 2008. R Programming for Bioinformatics. Maindonald, J. H. 1992. Statistical design, analysis and presentation issues. New Zealand Journal of Agricultural Research 35: 121–41. Maindonald, J. H. 2008. Using R for Data Analysis and Graphics. Available as a pdf file at http://www.maths.anu.edu.au/˜johnm/r/usingR.pdf Murrell, P. 2005. R Graphics. http://www.stat.auckland.ac.nz/˜paul/RGraphics/rgraphics.html R Development Core Team. 2009a. An Introduction to R. Tufte, E. R. 1997. Visual Explanations. Venables, W. N. and Ripley, B. D. 2002. Modern Applied Statistics with S, 4th edn. Wainer, H. 1997. Visual Revelations. Wilkinson, L. and Task Force on Statistical Inference. 1999. Statistical methods in psychology journals: guidelines and explanation. American Psychologist 54: 594–604. See the references at the end of the book for fuller bibliographic details.

1.9 Exercises 1. The following table gives the size of the floor area (ha) and the price ($A000), for 15 houses sold in the Canberra (Australia) suburb of Aranda in 1999. area sale.price 1 694 192.0 2 905 215.0 3 802 215.0 4 1366 274.0 5 716 112.7 6 963 185.0 7 821 212.0 8 714 220.0 9 1018 276.0 10 887 260.0

38

A brief introduction to R 11 790 221.5 12 696 255.0 13 771 260.0 14 1006 293.0 15 1191 375.0 Type these data into a data frame with column names area and sale.price. (a) (b) (c) (d)

Plot sale.price versus area. Use the hist() command to plot a histogram of the sale prices. Repeat (a) and (b) after taking logarithms of sale prices. The two histograms emphasize different parts of the range of sale prices. Describe the differences.

2. The orings data frame gives data on the damage that had occurred in US space shuttle launches prior to the disastrous Challenger launch of 28 January 1986. The observations in rows 1, 2, 4, 11, 13, and 18 were included in the pre-launch charts used in deciding whether to proceed with the launch, while remaining rows were omitted. Create a new data frame by extracting these rows from orings, and plot total incidents against temperature for this new data frame. Obtain a similar plot for the full data set. 3. For the data frame possum (DAAG package) (a) Use the function str() to get information on each of the columns. (b) Using the function complete.cases(), determine the rows in which one or more values is missing. Print those rows. In which columns do the missing values appear? 4. For the data frame ais (DAAG package) (a) Use the function str() to get information on each of the columns. Determine whether any of the columns hold missing values. (b) Make a table that shows the numbers of males and females for each different sport. In which sports is there a large imbalance (e.g., by a factor of more than 2:1) in the numbers of the two sexes? 5. Create a table that gives, for each species represented in the data frame rainforest, the number of values of branch that are NAs, and the total number of cases. [Hint: Use either !is.na() or complete.cases() to identify NAs.] 6. Create a data frame called Manitoba.lakes that contains the lake’s elevation (in meters above sea level) and area (in square kilometers) as listed below. Assign the names of the lakes using the row.names() function. elevation area Winnipeg 217 24387 Winnipegosis 254 5374 Manitoba 248 4624 SouthernIndian 254 2247 Cedar 253 1353 Island 227 1223 Gods 178 1151 Cross 207 755 Playgreen 217 657

1.9 Exercises

39

(a) Use the following code to plot log2(area) versus elevation, adding labeling information (there is an extreme value of area that makes a logarithmic scale pretty much essential): attach(Manitoba.lakes) plot(log2(area) ˜ elevation, pch=16, xlim=c(170,280)) # NB: Doubling the area increases log2(area) by 1.0 text(log2(area) ˜ elevation, labels=row.names(Manitoba.lakes), pos=4) text(log2(area) ˜ elevation, labels=area, pos=2) title("Manitoba’s Largest Lakes") detach(Manitoba.lakes) Devise captions that explain the labeling on the points and on the y-axis. It will be necessary to explain how distances on the scale relate to changes in area. (b) Repeat the plot and associated labeling, now plotting area versus elevation, but specifying log="y" in order to obtain a logarithmic y-scale. [Note: The log="y" setting carries across to the subsequent text() commands. See Subsection 2.1.5 for an example.] 7. Look up the help page for the R function dotchart(). Use this function to display the areas of the Manitoba lakes (a) on a linear scale, and (b) on a logarithmic scale. Add, in each case, suitable labeling information. 8. Using the sum() function, obtain a lower bound for the area of Manitoba covered by water. 9. The second argument of the rep() function can be modified to give different patterns. For example, to get four 2s, then three 3s, then two 5s, enter rep(c(2,3,5), c(4,3,2)) (a) What is the output from the following command? rep(c(2,3,5), 4:2) (b) Obtain a vector of four 4s, four 3s, and four 2s. (c) The argument length.out can be used to create a vector whose length is length.out. Use this argument to create a vector of length 50 that repeats, as many times as necessary, the sequence: 3 1 1 5 7 (d) The argument each can be used to form a vector in which each element in the first argument is replaced by the specified number of repeats of itself. Use this to create a vector in which each of 3 1 1 5 7 is replaced by four repeats of itself. Show, also, how this can be done without use of the argument each. 10. The ˆ symbol denotes exponentiation. Consider the following: 1000*((1+0.075)ˆ5 - 1) # Interest on $1000, compounded # annually at 7.5% p.a. for five years (a) Evaluate the above expression. (b) Modify the expression to determine the amount of interest paid if the rate is 3.5% p.a. (c) Explain the result obtained when the exponent 5 is changed to seq(1, 10). 11. Run the following code: gender <- factor(c(rep("female", 91), rep("male", 92))) table(gender) gender <- factor(gender, levels=c("male", "female")) table(gender)

40

A brief introduction to R gender <- factor(gender, levels=c("Male", "female")) # Note the mistake: "Male" should be "male" table(gender) table(gender, exclude=NULL) rm(gender) # Remove gender Explain the output from the successive uses of table().

12. Write a function that calculates the proportion of values in a vector x that exceed some value cutoff. (a) Use the sequence of numbers 1, 2, . . . , 100 to check that this function gives the result that is expected. (b) Obtain the vector ex01.36 from the Devore6 (or Devore7) package. These data give the times required for individuals to escape from an oil platform during a drill. Use dotplot() to show the distribution of times. Calculate the proportion of escape times that exceed 7 minutes. 13. The following plots four different transformations of the Animals data from the MASS package. What different aspects of the data do these different graphs emphasize? Consider the effect on low values of the variables, as contrasted with the effect on high values. par(mfrow=c(2,2)) # 2 by 2 layout on the page library(MASS) # Animals is in the MASS package plot(brain ˜ body, data=Animals) plot(sqrt(brain) ˜ sqrt(body), data=Animals) plot(I(brainˆ0.1) ˜ I(bodyˆ0.1), data=Animals) # I() forces its argument to be treated "as is" plot(log(brain) ˜ log(body), data=Animals) par(mfrow=c(1,1)) # Restore to 1 figure per page 14. Use the function abbreviate() to obtain six-character abbreviations for the row names in the data frame cottonworkers (DAAG package). Plot survey1886 against census1886, and plot avwage*survey1886 against avwage*census1886, in each case using the six-letter abbreviations to label the points. How should each of these graphs be interpreted? [Hint: Be sure to specify I(avwage*survey1886) and I(avwage*census1886) when plotting the second of these graphs.] 15. The data frame socsupport (DAAG) has data from a survey on social and other kinds of support, for a group of university students. It includes Beck Depression Inventory (BDI) scores. The following are two alternative plots of BDI against age: plot(BDI ˜ age, data=socsupport) plot(BDI ˜ unclass(age), data=socsupport) For examination of cases where the score seems very high, which plot is more useful? Explain. Why is it necessary to be cautious in making anything of the plots for students in the three oldest age categories (25-30, 31-40, 40+)? 16. Functions that can be useful for labeling points on graphs are abbreviate() (create abbreviated names), and paste() (create composite labels). A composite label might, for the data from socsupport, give information about gender, country, and row number. Try the following: gender1 <- with(socsupport, abbreviate(gender, 1)) table(gender1) # Examine the result

1.9 Exercises

41

country3 <- with(socsupport, abbreviate(country, 3)) table(country3) # Examine the result Now use the following to create a label that can be used with text() or with identify(): num <- with(socsupport, seq(along=gender)) # Generate row numbers lab <- paste(gender1, country3, num, sep=":")

Use identify() to place labels on all the points that the boxplots have identified as “outliers”. 17. Given a vector x, the following demonstrates alternative ways to create a vector of numbers from 1 through n, where n is the length of the vector: x <- c(8, 54, 534, 1630, 6611) seq(1, length(x)) seq(along=x) Now set x <- NULL and repeat each of the calculations seq(1, length(x)) and seq(along=x). Which version of the calculation should be used in order to return a vector of length 0 in the event that the supplied argument is NULL. 18. The Rabbit data frame in the MASS library contains blood pressure change measurements on five rabbits (labeled as R1, R2, . . . ,R5) under various control and treatment conditions. Read the help file for more information. Use the unstack() function (three times) to convert Rabbit to the following form: Treatment Dose R1 R2 R3 R4 R5 1 Control 6.25 0.50 1.00 0.75 1.25 1.5 2 Control 12.50 4.50 1.25 3.00 1.50 1.5 . . . . 6 Control 200.00 32.00 29.00 24.00 33.00 18.0 7 MDL 6.25 1.25 1.40 0.75 2.60 2.4 8 MDL 12.50 0.75 1.70 2.30 1.20 2.5 . . . . 12 MDL 200.00 37.00 28.00 25.00 22.00 19.0 19. The data frame vlt (DAAG) consists of observations taken on a video lottery terminal during a two-day period. Eight different objects can appear in each of three windows. Here, they are coded from 0 through 7. Different combinations of the objects give prizes (although with small probability). The first four rows are: > head(vlt, 4) # first few rows of vlt window1 window2 window3 prize night 1 2 0 0 0 1 2 0 5 1 0 1 3 0 0 0 0 1 4 2 0 0 0 1 > # . . . Use stack() to convert the first three columns of this data set to a case-by-variable format, then creating a tabular summary of the results, broken down by window. vltcv <- stack(vlt[, 1:3]) head(vltcv) # first few rows of vltcv table(vltcv$values, vltcv$ind) # More cryptically, table(vltcv) gives the same result. Does any window stand out as different? 20.∗ The help page for iris (type help(iris)) gives code that converts the data in iris3 (datasets package) to case-by-variable format, with column names “Sepal.Length”,

42

A brief introduction to R “Sepal.Width”, “Petal.Length”, “Petal.Width”, and “Species”. Look up the help pages for the functions that are used, and make sure that you understand them. Then add annotation to this code that explains each step in the computation.

21.∗ The following uses the for() looping function to plot graphs that compare the relative population growth (here, by the use of a logarithmic scale) for the Australian states and territories. oldpar <- par(mfrow=c(2,4)) for (i in 2:9){ plot(austpop[, 1], log(austpop[, i]), xlab="Year", ylab=names(austpop)[i], pch=16, ylim=c(0,10))} par(oldpar) Find a way to do this without looping. [Hint: Use the function sapply(), with austpop[,2:9] as the first argument.]

2

Styles of data analysis

What is the best way to begin investigation of a new set of data? What forms of data exploration will draw attention to obvious errors or quirks in the data, or to obvious clues that the data contain? What checks are desirable before proceeding with an intended formal analysis, or to help decide what formal analysis may be appropriate? What can be learned from investigations that other researchers have done with similar data? Competent statisticians have always used graphs to check their data. Numerical summaries, such as an average, can be very useful, but important features of the data may be missed without a glance at an appropriate graph. Careful consideration may be needed to choose a graph that will be effective for the purpose in hand. We will see in Chapter 3 that an integral part of statistical analysis is the development of a model that accurately describes the data, clarifies what the data say, and makes prediction possible. Without model assumptions, there cannot be a meaningful formal analysis! As assumptions are strengthened, the chances of getting clear results improve. The price for stronger assumptions is that, if wrong, the results may be wrong. Graphical techniques have been developed for checking, to the extent possible, many of the assumptions that must be made in practice. Preliminary scrutiny of the data can readily degenerate into data snooping, so that the analysis is unduly attuned to statistical artefacts of the particular data that are to be analyzed. Under torture, the data readily yield false confessions. To avoid this, strict limits must be placed on the extent to which the data are allowed to influence the choice of model for the formal analysis. Even if data have not been collected in a way that makes them suitable for formal statistical analysis, exploratory techniques can often glean clues from them. However, it is unwise, as too often happens, to rely on this possibility! 2.1 Revealing views of the data The use of graphs to display and help understand data has a long tradition. John W. Tukey formalized and extended this tradition, giving it the name Exploratory Data Analysis (EDA). Tukey has had a huge influence on data analysis practices; see Hoaglin (2003). A key concern is that data should, as far as possible, have the opportunity to speak for themselves, prior to or as part of a formal analysis. A use of graphics that is broadly in an EDA tradition continues to develop and evolve. Statistical theory has an important role in suggesting forms of display that may be helpful

44

Styles of data analysis

and interpretable. Advances in computing have been important, facilitating the development and use of many of the graphical tools now available. The best modern statistical software makes a strong connection between data analysis and graphics, combining the computer’s ability to crunch numbers and present graphs with the ability of a trained human eye to detect pattern. Graphical exploration after the style of EDA has at least four roles: r It may suggest ideas and understandings that had not previously been contemplated. This use of EDA fits well with the view of science as inductive reasoning. r It may challenge the theoretical understanding that guided the initial collection of the data. It then acquires a more revolutionary role. It becomes the catalyst, in the language of Thomas Kuhn, for a paradigm shift. r It allows the data to criticize an intended analysis and facilitates checks on assumptions. Subsequent formal analysis can then proceed with greater confidence. r It may reveal additional information, not directly related to the research question. It may, for example, suggest fruitful new lines of research. The next several subsections will describe the histogram and density plot, the stem-andleaf display, the boxplot, the scatterplot, the lowess smoother, and the trellis-style graphics that are available in the lattice package. The lattice functions greatly extend the available styles and layouts. 2.1.1 Views of a single sample Histograms and density plots

The histogram is a basic (and over-used) EDA tool for displaying the frequency distribution of a set of data. The area of each rectangle of a histogram is proportional to the number of observations whose values lie within the width of the rectangle. A mound-shaped histogram may make it plausible that the data follow a normal distribution (the “bell curve”). In small samples, however, the shape can be highly irregular. In addition, the appearance can depend on the choice of breakpoints, which is a further reason for caution in interpreting the shape. It is often helpful to try more than one set of breakpoints. The data set possum (DAAG package) has nine morphometric measurements on each of 104 mountain brushtail possums, trapped at seven sites from southern Victoria to central Queensland (data relate to Lindenmayer et al., 1995). Attention will be limited to the measurements for 43 females, placing them in a subset data frame that will be called fossum. The following code creates this subset data frame: library(DAAG) # Ensure that the DAAG package is attached ## Form the subset of possum that holds data on females only fossum <- subset(possum, sex=="f")

Panels A and B of Figure 2.1 exhibit histogram plots of the frequency distribution of the total lengths of the female possums.1 1

## To get a 1 by 4 layout, precede with par(mfrow = c(1,4))

2.1 Revealing views of the data B

Breaks as in B

70

70

0 08

Density

10 15 20

D

Breaks as in A

0 00

5 0

Frequency

C

Breaks at 75, 80, ...

0 04

A Breaks at 72.5, 77.5, ...

45

70

80

90 100

Total length (cm)

70

80

90 100

Total length (cm)

80

90 100

Total length (cm)

80

90 100

Total length (cm)

Figure 2.1 The histograms in panels A and B show the same data, but with a different choice of breakpoints. In panels C and D, density plots are overlaid on the histograms from panels A and B, respectively. attach(fossum) hist(totlngth, breaks = xlab="Total length hist(totlngth, breaks = xlab="Total length

72.5 + (0:5) * 5, ylim = c(0, 22), (cm)", main ="A: Breaks at 72.5, 77.5, ...") 75 + (0:5) * 5, ylim = c(0, 22), (cm)", main="B: Breaks at 75, 80, ...")

The only difference in the construction of the two plots is the choice of breakpoints, but one plot suggests that the distribution is asymmetric (skewed to the left), while the other suggests symmetry. A histogram is a crude form of a density estimate. A better alternative is, often, a smooth density estimate, as in Figures 2.1C and D. Whereas the width of histogram bars must be chosen somewhat subjectively, density estimates require the choice of a bandwidth parameter that controls the amount of smoothing. In both cases, the software has default choices that can work reasonably well. dens <- density(totlngth) xlim <- range(dens$x); ylim <- range(dens$y) hist(totlngth, breaks = 72.5 + (0:5) * 5, probability = T, xlim = xlim, ylim = ylim, xlab="Total length (cm)", main=" ") lines(dens) hist(totlngth, breaks = 75 + (0:5) * 5, probability = T, xlim = xlim, ylim = ylim, xlab="Total length (cm)", main= " ") lines(dens) par(mfrow=c(1,1)); detach(fossum)

The height of the density curve at any point is an estimate of the proportion of sample values per unit interval, locally at that point. Observe that in Figures 2.1A and C, the cell of the histogram between the breakpoints (87.5, 92.5] has a frequency of 22. As the total 22 = 0.102, frequency is 43, and the width of the cell is 5, this corresponds to a density of 43×5 which is just a little smaller than the height of the density curve at its highest point or mode. Much of the methodology in this book makes assumptions that hold exactly only if the data follow a normal distribution (the “bell curve”), discussed in the next chapter. Density curves are preferable to histograms for drawing attention to particular forms of

46

Styles of data analysis The decimal point is 1 digit(s) to the right of the | 15 16 16 17 17 18 18 19 19

| | | | | | | | |

6 5 4 5678899 00000011223 55666668899 123 58

Lower quartile is 179 (10th largest) Median is 182 (19th largest)

Upper quartile is 186 (28th largest)

Figure 2.2 Stem-and-leaf display showing the heights of the 37 rowers in the ais data set. Annotation has been added that identifies the lower quartile, the median, and the upper quartile.

non-normality, such as that associated with strong skewness in the distribution, but are still not an adequate tool. A more effective way of checking for normality – the normal probability plot – is described in Subsection 3.4.2. Density curves are useful for estimating the population mode, i.e., the value that occurs most frequently. Where data values have sharp lower and/or upper cutoff limits, use the arguments from and to to specify those limits. For example, a failure time distribution may have a mode close to zero, with a sharp cutoff at zero.

The stem-and-leaf display

The stem-and-leaf display is a fine-grained alternative to a histogram, for use in displaying a single column of numbers. Figure 2.2 shows a stem-and-leaf plot of the heights of the 37 rowers in the ais data set. Code is: with(ais, stem(ht[sport=="Row"]))

The data have been rounded to the nearest centimeter. The numbers that are displayed are, in order of magnitude, 156, 165, 174, . . . . The display has broken these down as 150 + 6, 160 + 5, 170 + 4, . . . . The column of numbers on the left of the vertical bars (15, 16, . . . ) comprises the stem; these are the tens of centimeters parts of the numbers. The leaf part for that number (6, 5, 4, . . . ) is what remains after removing the stem; these are printed, in order, to the right of the relevant vertical bar. As there are 37 data values, the median or middle value is the 19th. Starting from the 156 leaf in the first line of the stem-and-leaf diagram and working down, 18 values precede the 19th largest, and 18 values follow. Thus the median (or 50th percentile) is 182. The first and third quartiles (the 25th and 75th percentiles) can be recovered in a similar way, with the exact value depending on the details of the formula used for their calculation. For present purposes the first quartile can be taken as the 10th largest value (= 179), while the third quartile is the 28th largest value (= 186), or the 10th value when starting at the largest and counting down. (The number 10 is the average of the ranks 1 and 19, while 28 is the average of 19 and 39.)2 2

## Use quantile() to obtain the quartiles of ht: data frame ais (DAAG package) quantile(ais$ht[ais$sport=="Row"], prob=c(.25,.5,.75)) # For the 50th percentile (the 2nd quartile), an alternative is median()

85.25

75

80

85

Largest value (there are no outliers)

upper quartile

median

lower quartile

Smallest value (outliers excepted)

Outlier

2.1 Revealing views of the data

90.5

90

47

Inter–quartile range = 90.5 – 85.25 = 5.25 Compare 0.75 x Inter–quartile range = 3.9 with standard deviation = 4.2

95

Total length (cm)

Figure 2.3

Boxplot, with annotation that explains boxplot features.

Boxplots

Like the histogram, the boxplot is a coarse summary. It allows a trained eye to comprehend at a glance specific important features of the data. Figure 2.3 shows a boxplot of total lengths of females in the possum data set, with annotation added that explains the interpretation of boxplot features. Code that gives the boxplot, without the annotation, is: ## Base graphics boxplot function with(fossum, boxplot(totlngth, horiz=TRUE)) ## Alternative: lattice graphics bwplot function bwplot(˜totlngth, data=fossum)

Notice that one point lies outside the “whiskers” to the left, and is thus flagged as a possible outlier. An outlier is a point that, in some sense, lies away from the main body of the data. In identifying points that are flagged as possible outliers, the normal distribution (to be discussed in Subsection 3.2.2) is taken as the standard. Using the default criterion one point in 100 will on average, for data from a normal distribution, be flagged as a possible outlier. Thus, in a boxplot display of 1000 values that are drawn at random from a normal distribution, around 10 will be plotted out beyond the boxplot whiskers and thus flagged as possible outliers. Subsection 2.1.7 has further comment on outliers. The discussion of the normal and other distributions in Chapter 3 should help clarify these ideas.

2.1.2 Patterns in univariate time series

In Figure 2.4, “measles” includes both what is nowadays called measles and the closely related rubella or German measles.3 Panel A uses a logarithmic vertical scale. Panel B uses an unlogged scale and takes advantage of the fact that deaths from measles are of the order, in any year, of one thousandth of the population. Thus, deaths in thousands and population in millions can be shown on the same scale. 3

For details of the data, and commentary, see Guy (1882), Stocks (1942), Senn (2003). (Guy’s interest was in the comparison with smallpox mortality.) The population estimates (londonpop) are from Mitchell (1988).

48

Styles of data analysis

Deaths; Population (log scale)

A (1629−1939) 5e+06 1e+06

●

●

●

●

●

●

●

●

●

●

●

●

●

1000 100 10 1 1650

Deaths; Population in 1000s

●

1700

1750

1800

1850

1900

1950

B (1841−1881) 4000

● ●

3000

● ●

2000

●

1000

1840

1850

1860

1870

1880

Figure 2.4 The two panels provide different insights into data on mortality from measles, in London over 1629–1939. Panel A shows the numbers of deaths from measles in London for the period from 1629 through 1939 (black curve) and the London population (in thousands, black dots). A log scale has been used (see Subsection 2.1.3 for details). The lower panel B shows the subset of the measles data for the period 1840 through 1882 on the linear scale (black curve), together with the London population (in thousands, black dots).

Simplified code is: ## Panel A plot(log10(measles), xlab="", ylim=log10 (c(1,5000*1000)), ylab=" Deaths; Population (log scale)", yaxt="n") ytiks <- c(1, 10, 100, 1000, 1000000, 5000000) ## London population in thousands londonpop
The function plot() recognizes that measles is a time series object, and calls the plot method plot.ts() that is used for time series. For details, see help(plot.ts). Notice the use, for panel B, of the function window() that extracts a subseries.

49

2

3

four 4 5

6

7

2.1 Revealing views of the data

2

3

4 5 one

6

7

Figure 2.5 Each of 17 panelists compared two milk samples for sweetness. One sample had one unit of additive, while the other had four units of additive.

Panel A shows broad trends over time, but is of no use for identifying changes on the time scale of a year or two. In panel B, the lines that show such changes are, mostly, at an angle that is in the approximate range of 20◦ to 70◦ from the horizontal. A sawtooth pattern, by which years in which there are many deaths are commonly followed by years in which there are fewer deaths, is thus clearly evident. (To obtain this level of detail for the whole period from 1629 until 1939, multiple panels would be necessary.) The following, with y-axis labeling in logarithms of numbers and omitting the population estimates, demonstrates the combining of the two graphs on the one page: ## Panel A: par(fig=c(0, 1, .38, 1)) # 38% to 100% of page, in y-direction plot(log10(measles), ylab="log10(Deaths)", ylim=log10(c(1,5000*1000))) mtext(side=3, line=0.5, "A (1629-1939)", adj=0) ## Panel B: window from 1840 to 1882; more complete code par(fig=c(0, 1, 0, .4), new=TRUE) # 0% to 38% of height of figure region plot(window(measles, start=1840, end=1882), ylab="Deaths") mtext(side=3, line=0.5, "B (1841-1881)", adj=0) par(fig=c(0, 1, 0, 1)) # Restore default figure region

2.1.3 Patterns in bivariate data

The scatterplot is a simple but important tool for the examination of pairwise relationships. We will illustrate with specific examples. Figure 2.5 shows data from a tasting session where each of 17 panelists assessed the sweetness of each of two milk samples, one with four units of additive, and the other with one unit of additive. The line y = x has been added. The function rug() adds a “rug”, i.e., short bars at right angles to one or other axis that show the distribution values along that axis of the plot. The code is: ## Plot four vs one: data frame milk (DAAG) xyrange <- range(milk) plot(four ˜ one, data = milk, xlim = xyrange, ylim = xyrange, pch = 16, pty="s") # pty="s": square plotting region

50

Styles of data analysis A 10 8 2

4

6

8 6 4 2

Resistance (kOhm)

10

B

10 30 50 Apparent juice content (%)

10 30 50 Apparent juice content (%)

Figure 2.6 Electrical resistance versus apparent juice content. Panel B repeats panel A, but with a smooth curve fitted to the data. rug(milk$one) rug(milk$four, side = 2) abline(0, 1)

# x-axis rug (default is side=1) # y-axis rug

There is a positive correlation between assessments for the two samples; if one was rated as sweet, by and large so was the other. The line y = x assists in comparing the two samples. Most panelists (13 out of 17) rated the sample with four units of additive as sweeter than the sample with one unit of additive.

The fitting of a smooth trend curve

Figure 2.6 shows data from a study that measured both electrical resistance and apparent juice content for a number of slabs of kiwifruit. The curve in panel B, obtained using the lowess method that is discussed further in Subsection 7.5.4, estimates the relationship between electrical resistance and apparent juice content. The code is: ## Plot ohms vs juice: data frame fruitohms (DAAG) plot(ohms ˜ juice, xlab="Apparent juice content (%)", ylab="Resistance (ohms)", data=fruitohms) ## Add a smooth curve, as in Panel B with(fruitohms, lines(lowess(juice, ohms), lwd=2)) # With lwd=2, the curve is twice the default thickness

The fitted smooth curve shows a form of response that is clearly inconsistent with a straight line. It suggests an approximate linear relationship for juice content up to somewhat over 35%. Once the juice content reaches around 45%, the curve becomes a horizontal line, and there is no evident further change in resistance. There is no obvious simple form of equation that might be used to describe the curve. A curve fitted using lowess() or another such smoothing function can provide a useful benchmark against which to compare the curve given by a theoretical or other mathematical form of equation that the data are thought to follow.

2.1 Revealing views of the data

51

log10(Body weight) A

B

−3

−1

0

1

2

3

●

0 200 600 Body weight (kg x 100)

4 3 2 0

● ● ●

●

● ●

1

1000 10 ●

● ● ●●

0.001

0.1 1

10

log10(Brain weight)

● ● ● ● ●●

●● ● ● ● ● ●● ● ●●●● ●● ●

−1

●

0.1 1

●

Brain weight (g)

4000 2000 0

Brain weight (g)

●

1000

Body weight (kg x 100)

Figure 2.7 Brain weight versus body weight, for 27 animals that vary greatly in size. Panel A uses untransformed scales, while panel B uses logarithmic scales, on both axes.

What is the appropriate scale?

Figures 2.7A and B plot brain weight (g) against body weight (kg), for a number of different animals: ## The following omits the labeling information oldpar <- par(mfrow = c(1,2), pty="s") ## Plot brain vs body: data frame Animals (MASS package) library(MASS) plot(brain ˜ body, data=Animals) # Panel A plot(log(brain) ˜ log(body), data=Animals) # Panel B par(oldpar)

Figure 2.7A is almost useless. The axes should be transformed so that the data are spread out more evenly. Here, we can do this by choosing a logarithmic scale. Multiplication by the same factor (e.g., for the tick marks in Figure 2.7B, by a factor of 10) always gives the same distance along the scale. If we marked points 1, 5, 25, 125, . . . along the vertical axis, they would also lie an equal distance apart. A logarithmic scale is appropriate for quantities that change multiplicatively. For example, if cells in a growing organism divide and produce new cells at a constant rate, then the total number of cells changes in a multiplicative manner, resulting in so-called exponential growth. Growth in the bodily measurements of organisms may similarly be multiplicative, with large organisms increasing in some time interval by the same approximate fraction as smaller organisms. Random changes in the relative growth rate will produce adult organisms whose size (e.g., height) is, on the logarithmic scale, approximately normally distributed. The reason is that growth rate on a natural logarithmic scale (loge ) equals the relative growth rate. Derivation of this result is a straightforward use of the differential calculus. The logarithmic transformation is so commonly needed that it has seemed necessary to introduce it at this point. Biologists, economists, and others should become comfortable with its use. There is a brief discussion of other transformations in Chapter 5.

52

Styles of data analysis A

B

wren tree pipit robin pied wagtail meadow pipit hedge sparrow 20

21

22

23

Length of egg (mm)

24

25

20

21

22

23

24

25

Length of egg (mm)

Figure 2.8 Strip plot (panel A) and boxplot (panel B) displays of cuckoo egg lengths. Data, from Latter (1902), are reproduced in summarized form in Tippett (1931).

2.1.4 Patterns in grouped data – lengths of cuckoo eggs

Cuckoos lay eggs in the nests of other birds. The eggs are then unwittingly adopted and hatched by the host birds. In Figure 2.8 the egg lengths are grouped by the species of the host bird, using both a strip plot display (panel A) and boxplot summaries (panel B). Strip plots and boxplots allow convenient side-by-side comparisons of different groups, here the different host species. The main part of the code used for these plots is: ## Compare stripplot() with bwplot(), both from lattice package stripplot(species ˜ length, xlab="Length of egg (mm)", data=cuckoos) bwplot(species ˜ length, xlab="Length of egg (mm)", data=cuckoos, scales=list(y=list(alternating=0))) # alternating=0; omit y-axis labels

Eggs planted in wrens’ nests appear smaller than eggs planted in other birds’ nests. Apart from several outlying egg lengths in the meadow pipit nests, the length variability within each host species’ nest is fairly uniform. Fuller details of the code are in the footnote.4 Comparing densities between groups – lattice style density plots

Lattice-style density plots can be useful for getting an indication of how distributions may differ across different groups of data. Figure 2.9 compares the ear conch measurements 4

## For tidier labels replace ".", in several of the species names, by a space specnam <- with(cuckoos, sub(pattern=".", replacement=" ", levels(species), fixed=TRUE)) # fixed=TRUE: do not interpret "." as a ‘regular expression’, ## Panel A: Strip plot: data frame cuckoos (DAAG) plt1 <- stripplot(species ˜ length, factor.levels=specnam, data=cuckoos) print(update(plt1, xlab="Length of egg (mm)"), position=c(0,0,0.55,1)) # xmin, ymin, xmax, ymax # Use print() to display lattice graphics objects ## Panel B: Box plot plt2 <- bwplot(species ˜ length, factor.levels=specnam, data=cuckoos) print(update(plt2, xlab="Length of egg (mm)", scales=list(y=list(alternating=0))), newpage=FALSE, position=c(0.55,0,1,1))

2.1 Revealing views of the data 40

f

45

50

53 55

m

0.25

Density

0.20

Vic other

0.15 0.10 0.05 0.00 40

45

50

55

earconch

Figure 2.9 Density plot that compares the ear conch measurements for each of the two “populations” of possums, for males and females separately.

of male and female possums, for each of two “populations” (Vic and other) of possums: ## Density plot for earconch: data frame possum (DAAG package) library(lattice) densityplot(˜earconch | sex, groups=Pop, data=possum, auto.key=list(space="right"))

2.1.5∗ Multiple variables and times

Overlaying plots of several time series (sequences of measurements taken at regular intervals) might seem appropriate for making direct comparisons. However, this approach will only work if the scales are similar for the different series. The data frame jobs (DAAG) gives the number of workers (in thousands) in the Canadian labor force, broken down by region (BC, Alberta, Prairies, Ontario, Quebec, Atlantic), for the 24-month period from January 1995 to December 1996. Over this time, Canada was emerging from a deep economic recession. Columns 1–6 have the respective numbers for six different regions. The ranges of values in the columns are: > ## Apply function range to columns of data frame jobs (DAAG) > sapply(jobs, range) BC Alberta Prairies Ontario Quebec Atlantic Date [1,] 1737 1366 973 5212 3167 941 95.00000 [2,] 1840 1436 999 5360 3257 968 96.91667

In order to see where the economy was taking off most rapidly, it is tempting to plot all of the series on the same graph. In order that similar changes on the scale will correspond to similar proportional changes, a logarithmic scale is used in Figure 2.10A: ## Simplified plot; all series in a single panel; use log scale (simplejobsA.xyplot <xyplot(Ontario+Quebec+BC+Alberta+Prairies+Atlantic ˜ Date, outer=FALSE, data=jobs, type="b", ylab="Number of workers", scales=list(y=list(log="e")), auto.key=list(space="right", lines=TRUE)))

54

Styles of data analysis

A Number of workers

4915 (8.5) 2981 (8)

Ontario Quebec BC Alberta Prairies Atlantic

1808 (7.5) 1097 (7) Jan95

Jul95

Jan96

B

1422 (7.26) 1394 (7.24) 1366 (7.22)

Number of jobs

Jan97

Jan95 Jul95 Jan96 Jul96 Jan97

Alberta

5432 (8.6) 5324 (8.58) 5219 (8.56) 5115 (8.54)

Jul96

1012 (6.92) 992 (6.9) 973 (6.88)

Ontario

Atlantic 973 (6.88) 953 (6.86) 934 (6.84)

Quebec 3294 (8.1) 3229 (8.08) 3165 (8.06)

Jan95 Jul95 Jan96 Jul96 Jan97

Prairies

1845 (7.52) 1808 (7.5) 1772 (7.48) 1737 (7.46)

BC

Jan95 Jul95 Jan96 Jul96 Jan97

Figure 2.10 Data are numbers in the labor force (thousands) for various regions of Canada, at quarterly intervals over 1995–1996. Panel A uses the same logarithmic scale for all regions. Panel B shows the same data as in panel A, but now with separate (“sliced”) logarithmic scales on which the same percentage increase, e.g., by 1%, corresponds to the same distance on the scale, for all plots. Distances between ticks are 0.02 on the loge scale, i.e., a change of almost exactly 2%.

The trellis object has been saved so that it can be updated, as demonstrated in the footnote, to give the graph shown in Figure 2.10A.5 The use of column names that are joined with "+" has the result that the columns are plotted in parallel. The regions have been taken in order of the number of jobs in December 1996 (or, in fact, at any other time). This ensures that the order of the labels in the key matches the positioning of the points for the different regions. Code in the footnote shows how the labeling on the x- and y-axes was obtained. 5

## Panel A: Update trellis object to improve x- and y-axis tick labels datelabpos <- seq(from=95, by=0.5, length=5) datelabs <- format(seq(from=as.Date("1Jan1995", format="%d%b%Y"), by="6 month", length=5), "%b%y") ## Now create $y$-labels that have numbers, with log values underneath ylabpos <- exp(pretty(log(unlist(jobs[,-7])), 5)) ylabels <- paste(round(ylabpos),"\n(", log(ylabpos), ")", sep="") update(simplejobsA.xyplot, xlab="", scales=list(x=list(at=datelabpos, labels=datelabs), y=list(at=ylabpos, labels=ylabels)))

2.1 Revealing views of the data 103.135

log=10 log="e"

e

7.220

10.416

log=2

2 1366

1250

103.157

55 103.244

7.270

e

e7.518

10.775

210.845 1840

e

10.488

2 1436

103.265

7.469

2 1752 1500

1800

Figure 2.11 Labeling of the values for Alberta (1366, 1436) and Ontario (1752, 1840), with alternative logarithmic scale choices of labeling.

Because the labor forces in the various regions do not have similar sizes, it is impossible to discern any differences among the regions from this plot. Plotting on the logarithmic scale did not remedy this problem.6 Figure 2.10B shows a much preferable alternative. The six different panels use different slices of the same logarithmic scale. Here is simplified code for Figure 2.9B. The regions are again taken in the order of numbers of jobs in December 1996.7 ## Simplified code for Figure 2.9B xyplot(Ontario+Quebec+BC+Alberta+Prairies+Atlantic ˜ Date, data=jobs, type="b", layout=c(3,2), ylab="Number of jobs", scales=list(y=list(relation="sliced", log=TRUE)), outer=TRUE)

Use of outer=TRUE ensures that the separate columns (regions) are plotted on separate panels. Equal distances on the scale now correspond to equal relative changes. It is now clear that Alberta and BC experienced the most rapid job growth during the period, and that there was little or no job growth in Quebec and the Atlantic region. Even better, particularly if ready comprehension is important, would be to standardize by dividing, e.g., by the respective number of persons aged 15 years and over at that time. Exercise 11 at the end of the chapter explores this approach. ∗

Small proportional changes, on a scale of natural logarithms

Tick marks have been placed a distance 0.02 apart on a scale of natural logarithms or loge . On a scale of natural logarithms a change of 0.02 is, to a close approximation, a 2% change. ∗

Tick positions and labeling, on a logarithmic scale

The following are the changes in numbers employed, in each of Alberta and Ontario, from January 1995 to December 1996. The changes are shown in actual numbers, and on scales of log2 , loge , and log10 . Figure 2.11 shows this graphically.

6 7

Figure 2.10A might alternatively be plotted as a time series. For details, see Subsection 14.9.7. Subsection 15.5.2 has code that gives the labeling shown in Figure 2.10B.

56

Styles of data analysis Increase Alberta (1366 to 1466; increase = 70) Ontario (1752 to 1840; increase = 88)

Rel. change 1.051 1.050

log2 0.072 0.070

loge 0.050 0.049

log10 0.022 0.021

From the beginning of 1995 to the end of 1996, Alberta increased by 70 from 1366 to 1436, which is a factor of 1436/1366  1.051. Ontario increased by 96 from 5239 to 5335, which is a factor of 1.050. The proper comparison is not between the absolute increases of 70 and 96, but between relative increases by factors of 1.05 and 1.018. For lattice functions, the arguments log=2 or log="e" or log=10 are available. These use the relevant logarithmic axis labeling, as in Figure 2.11, for axis labels. In base graphics, with the argument log="x", the default is to label in the original units. An alternative, both for traditional and lattice graphics, is to enter the logged values, using whatever basis is preferred (2 or "e" or 10), into the graphics formula. Unless other tick labels are provided, the tick marks will then be labeled with the logged values for the relevant basis. 2.1.6 Scatterplots, broken down by multiple factors

Data, in the data frame tinting (DAAG), are from an experiment that examined the effects of the tinting of car windows on visual performance (data relate to Burns et al., 1999). The main focus was on visual recognition tasks, where side window vision is important. Columns are: r Variables csoa (critical stimulus onset asynchrony, i.e., the time in milliseconds required to recognize an alphanumeric target), it (inspection time, i.e., the time required for a simple discrimination task), and age (age to the nearest year). r The ordered factor tint (levels no, lo, hi). r Factors target (locon, i.e., low contrast; hicon, i.e., high contrast), sex (f = female, m = male), and agegp (younger = a younger participant, in the 20s; older = an older participant, in the 70s). Each of 28 individuals was tested at each level of tint, for each of the two levels of target. In all there are four factors (tint, target, sex, and agegp) that might influence the values of csoa and it, and the relationship between them. Two of these (tint and target) take different values for the same individual, while the other two (sex and agegp) vary between individuals. A first step might be to plot csoa against it for each combination of sex and agegp. Use of the argument groups=target results in the use of different symbols (in a black and white plot) or different colors, depending on whether the target is low contrast or high contrast. Also, we can ask for a key. The code is (target.xyplot <xyplot(csoa ˜ it | sex*agegp, data=tinting, groups=target, auto.key=list(columns=2)))

2.1 Revealing views of the data locon

A

B 50

older f

hicon 100

150

57

no

lo

200

hi 50

older m

older f

100

150

200

older m 120

100

100

80

80

60

60

40

40

csoa

120

younger f

younger m

younger f

younger m

120

120

100

100

80

80

60

60

40

40

50

100

150

200

50

it

100

150

200

it

Figure 2.12 Panel A plots csoa against it, for each combination of sex and agegp. Different colors (gray and black) and symbols show different levels of target. Panel B shows the same points, but different colors (printed in grayscale) now show different levels of tint. Notice, also, the addition of smooth curves.

There are further possibilities for refinement. Figure 2.12A has used parameter settings that specify the choice of colors (here gray or black), plotting symbols, and the placement of the key.8 Observe that the longest times are for the high level of tinting. The relationship between csoa and it seems much the same for both levels of contrast. A number of older males have long response times with the low-contrast target. The analysis that will be presented later, in Chapter 10, indicates that within-subject effects – the effect of tint and target – stand up with greater clarity against the statistical noise than do effects of sex and agegp. The reason is that tint and target are effects that can be assessed within subjects, whereas the effects of sex and agegp involve a comparison across different subjects. Because there are six points for each subject, Figure 2.12A gives a visual impression that exaggerates the evidence for effects that are associated with sex and agegp. Fitting a trend curve, as in Figure 2.12B, makes the relationship clearer. The code for including the smooth curve, without the other refinements of Figure 2.12B, is: 8

## Settings used for Figure 2.12B (suitable for grayscale on a printed page) update(target.xyplot, par.settings=simpleTheme(col=c("black","gray20"), pch=c(1, 16))) # In the above, par.settings changed settings for this use of xyplot() ## Note the use of simpleTheme() for changing settings; see help(simpleTheme) ## Use trellis.par.set() to change settings while the current device is in use

58

Styles of data analysis

(tint.xyplot <xyplot(csoa ˜ it|sex*agegp, groups=tint, data=tinting, type=c("p","smooth"), span=1.25, auto.key=list(columns=3))) # "p": points; "smooth": a smooth curve # With span=1.25, the smooth curve is close to a straight line

The footnote adds the details needed to give Figure 2.12B.9 2.1.7 What to look for in plots

This is not a complete account of what plots may reveal! Its purpose is to draw attention to some of the more obvious possibilities. Outliers

Outliers are points that appear to be isolated from the main body of the data. Such points (whether errors or genuine values) are liable to distort any model that we fit. What appears as an outlier depends, inevitably, on the view that is presented. On a fairly simple level, the view is affected by whether or not, and how, the data are transformed. Boxplots, and the normal probability plot that will be discussed in Subsection 3.4.2, are useful for highlighting outliers in one dimension. Scatterplots may highlight outliers in two dimensions. Some outliers will be apparent only in three or more dimensions. The presence of outliers can indicate departure from model assumptions. Asymmetry of the distribution

Most asymmetric distributions can be characterized as either positively skewed or negatively skewed. Positive skewness is the commonest form of asymmetry. There is a long tail to the right, values near the minimum are bunched up together, and the largest values are widely dispersed. Provided that all values are greater than zero, a logarithmic transformation typically makes such a distribution more symmetric. A distribution that is skew cannot be normal. Severe skewness is typically a more serious problem for the validity of results than other types of non-normality. If values of a variable that takes positive values range by a factor of more than 10:1 then, depending on the application area context, positive skewness is to be expected. A logarithmic transformation should be considered. Changes in variability

Boxplots and histograms readily convey an impression of the extent of variability or scatter in the data. Side-by-side boxplots such as in Figure 2.8B, or strip charts such as 9

## Panel B, with refinements themeB <- simpleTheme(col=c("skyblue1", "skyblue4")[c(2,1,2)], lwd=c(1,1,2), pch=c(1,16,16)) # open, filled, filled update(tint.xyplot, par.settings=themeB, legend=NULL, auto.key=list(columns=3, points=TRUE, lines=TRUE)) # Set legend=NULL to allow new use of auto.key

2.2 Data summary

59

in Figure 2.8A, allow rough comparisons of the variability across different samples or treatment groups. They provide a visual check on the assumption, common in many uses of statistical models, that variability is constant across treatment groups. Note, however, that it is easy to over-interpret such plots. Statistical theory offers useful and necessary warnings about the potential for such over-interpretation. (The variability in a sample, typically measured by the variance, is itself highly variable under repeated sampling. Measures of variability will be discussed in Subsection 2.2.3.) When variability increases as data values increase, the logarithmic transformation will often help. If the variability is constant on a logarithmic scale, then the relative variation on the original scale is constant. Clustering

Clusters in scatterplots may suggest structure in the data that may or may not have been expected. When we proceed to a formal analysis, this structure must be taken into account. Do the clusters correspond to different values of some relevant variable? Outliers are a special form of clustering. Non-linearity

We should not fit a linear model to data where relationships are demonstrably non-linear. Often it is possible to transform variables so that terms enter into the model in a manner that is closer to linear. If not, the possibilities are wide-ranging, and we will canvass only a small number of them. See especially Chapter 7. If there is a theory that suggests the form of model, then this is a good starting point. Available theory may, however, incorporate various approximations, and the data may tell a story that does not altogether match the available theory. The data, unless they are flawed, have the final say! 2.2 Data summary Data summaries may: (1) be of interest in themselves; (2) give insight into aspects of data structure that may affect further analysis; (3) be used as data for further analysis. In case (3), it is necessary to ensure that important information, relevant to the analysis, is not lost. If no information is lost, the gain in simplicity of analysis can make the use of summary data highly worthwhile. It is important, when data are summarized, not to introduce distortions that are artefacts of the way that the data have been summarized – examples will be given. The question of whether information in the data may have been lost or obscured has especial importance for the summarizing of counts across the margins of multi-way tables, and for the use of the correlation coefficient. 2.2.1 Counts

Data in the data frame nswpsid1 (DAAG package) are derived from a study (Lalonde, 1986) that compared two groups of individuals with a history of unemployment problems – one an “untreated” control group and the other a “treatment” group whose members were exposed to a labor training program. Are the two groups genuinely comparable? This can

60

Styles of data analysis

be checked by comparing them with respect to various measures other than their exposure (or not) to the labor training program. Thus, what are the relative numbers in each of the two groups who had completed high school (nodeg = 0), as opposed to those who had not (nodeg = 1)? > > > + >

## Table of counts example: data frame nswpsid1 (DAAG) tab <- with(nswpsid1, table(trt, nodeg, useNA="ifany")) dimnames(tab) <- list(trt=c("none", "training"), educ = c("completed", "dropout")) tab educ trt completed dropout none 1730 760 training 80 217

Notice the use of the argument useNA="ifany" in the call to table(). This ensures that any NAs in either of the margins of the table will be tabulated. The training group has a much higher proportion of dropouts. Similar comparisons are required for other factors and variables, examining joint as well as individual comparisons. These data will be investigated further in Section 13.2. If x1, x2, . . . , xn are all columns (factors or vectors) of the same length and each is supplied as an argument to table(), the result is an n-way table. For example, table(x1, x2, x3) gives a three-way table. The first argument defines rows, though it is printed horizontally if there is just one column. The second argument defines columns. The table slices (rows by columns) that correspond to different values of the third argument appear in succession down the page, and so on.

Addition over one or more margins of a table

Figure 2.13 illustrates the possible hazards of adding a multi-way table over one of its margins. Data are from a study (Charig, 1986) that compared outcomes for two different types of surgery for kidney stones; A: open, which used open surgery, and B: ultrasound, which used a small incision, with the stone destroyed by ultrasound. Without additional information, the results are impossible to interpret. Different surgeons will have preferred different surgery types, and the prior condition of patients will have affected the choice of surgery type. The consequences of unsuccessful surgery may have been less serious for ultrasound than for open surgery. Code that gives the mosaic plot is: stones <- array(c(81,6,234,36,192,71,55,25), dim=c(2,2,2), dimnames=list(Success=c("yes","no"), Method=c("open","ultrasound"), Size=c("<2cm", ">=2cm"))) # NB: The margins are 1:Success, 2:Method, 3:Size library(vcd) mosaic(stones, sort=3:1) # c.f. mosaicplot() in base graphics # Re-ordering the margins gives a more interpretable plot.

2.2 Data summary

Success yes no

%Yes

Size <2cm >=2cm ultrasound <2cm >=2cm

81 192 234 55

6 71 36 25

93.1 73.0 86.7 68.8

Add over Size open ultrasound

273 289

77 61

78.0 82.6

Method ultrasound

yes no no

yes

Size >=2cm

Method open Sucess

<2cm

open

61

Figure 2.13 Mosaic plot for the kidney stone surgery data that is shown to the right of the figure. Outcomes are for two different types of surgery for kidney stones. The overall (apparent) success rates (78% for open surgery as opposed to 83% for ultrasound) favor ultrasound. The success rate for each size of stone separately favors, in each case, open surgery.

Code that tabulates the data, giving a layout similar to that on the right of the plot, is: ## Function to calculate percentage success rates roundpc <- function(x)round(100*x[1]/sum(x), 1) ## Add "%Yes" to margin 1 (Success) of the table stonesplus <- addmargins(stones, margin=1, FUN=c("%Yes"=roundpc)) ## Print table, use layout similar to that shown alongside plot ftable(stonesplus, col.vars=1) ## Get sum for each margin 1,2 combination; i.e., sum over margin 3 stones12 <- margin.table(stones, margin=c(1,2)) stones12plus <- addmargins(stones12, margin=1, FUN=c("%Yes"=roundpc)) ftable(stones12plus, col.vars=1) # Table based on sums over Size

An alternative to mosaic() in the vcd package is mosaicplot() in base graphics. The function mosaic() is more flexible and extensible. A footnote demonstrates how the size of the text in the margins can be modified.10

Tabulation that accounts for frequencies or weights – the xtabs() function

The function xtabs() will be illustrated with a further example that, again, demonstrates the hazards of summarizing tabular or other data across factors that affect the frequencies of the margins that are retained. Each year the National Highway Traffic Safety Administration in the USA uses a random sampling method, with sampling fractions that differ according to class of accident, to collect data from all police-reported crashes in which there is a harmful event (people or

10

## Add arguments that control size of textual components mosaic(aperm(stones, 3:1), main=NULL, gp_varnames=gpar(fontsize=8), labeling_args=list(gp_labels=gpar(fontsize=7), legend_args=list(fontsize=7)))

62

Styles of data analysis

property), and from which at least one vehicle is towed. The data in nassCDS (DAAG) are restricted to front-seat occupants (DAAG).11 Factors whose effect warrant investigation include, as a minimum: A: airbag (was an airbag fitted?), S: seatbelt (was a seatbelt used?), and dvcat (F: a force of impact measure). The letters A, S, and F will be used as abbreviations when tables are generated. The column weight (national inflation factor) holds the inverses of the sampling fraction estimates. The weight is designed to be the amount by which the contribution for the relevant row should be multiplied when tables of numbers of deaths and numbers of accidents are created. The following uses xtabs() to estimate numbers of front-seat passengers alive and dead, classified by airbag use: > library(DAAG) > ## NB: The parentheses generate an implicit print(abtab) > (Atab <- xtabs(weight ˜ airbag + dead, data=nassCDS)) dead airbag alive dead none 5445245.90 39676.02 airbag 6622690.98 25919.11

The function addmargins() that was introduced above can be used to add the proportion of deaths in the right margin: > roundpc2 <- function(x)round(100*x[2]/sum(x), 2) > addmargins(Atab, margin=2, FUN=c("%Dead"=roundpc2)) dead airbag alive dead %Dead none 5445245.90 39676.02 0.72 airbag 6622690.98 25919.11 0.39

The above might suggest that the deployment of an airbag substantially reduces the risk of mortality. Consider, however: > SAtab <- xtabs(weight ˜ seatbelt + airbag + dead, data=nassCDS) > ftable(addmargins(SAtab, margin=3, FUN=c("%Dead"=roundpc2)), + col.vars=3) dead alive dead %Dead seatbelt airbag none none 1342021.90 24066.65 1.76 airbag 871875.39 13759.94 1.55 belted none 4103224.00 15609.36 0.38 airbag 5750815.59 12159.17 0.21

In the earlier table (Atab), the results without airbags were mildly skewed (4.12:1.37) to those for belted. Results with airbags were strongly skewed (57.6:8.86) to those for none, that is, no seatbelt.

11

They hold a subset of the columns from a corrected version of the data analyzed in Meyer and Finney (2005). See also Farmer (2005) and Meyer (2006). More complete data are available from one of the web pages noted on the help page for nassCDS.

2.2 Data summary

63

The reader may wish to try an analysis that accounts, additionally, for estimated force of impact (dvcat): FSAtab <- xtabs(weight ˜ dvcat + seatbelt + airbag + dead, data=nassCDS) ftable(addmargins(FSAtab, margin=4, FUN=c("%Dead"=roundpc2)), col.vars=4)

The small differences that are now apparent, mostly unfavorable to airbags, are below any reasonable threshold of statistical detectability. Farmer (2005) argues that these data, tabulated as above, have too many uncertainties and potential sources of bias to give reliable results. He presents a different analysis, based on the use of front-seat passenger mortality as a standard against which to compare driver mortality. Farmer’s analysis was limited to cars without passenger airbags. In the absence of any effect from airbags, the ratio of driver mortality to passenger mortality should be the same, irrespective of whether or not there was a driver airbag. Farmer found a ratio of driver fatalities to passenger fatalities that was 11% lower in the cars with driver airbags. In addition to the functions discussed, note the function CrossTable() from the gmodels package, which offers a choice of SPSS-like and SAS-like output formats.

2.2.2 Summaries of information from data frames

For obtaining summaries at combinations of different factor levels, the aggregate() function is often a good recourse. Note also the abilities of aaply() and allied functions in Hadley Wickham’s plyr package.

Summary as a prelude to analysis – aggregate()

The data frame kiwishade (from DAAG) has yield measurements from 48 vines. Plots, made up of four vines each, were the experimental units. The analysis can be simplified by first taking means over plots that had four vines each. The 12 plots were divided into three blocks of four plots each. One block of four was north-facing, a second block west-facing, and a third block east-facing. (Because the trial was conducted in the Southern hemisphere, there is no south-facing block.) Shading treatments were applied to whole plots, i.e., to groups of four vines, with each treatment occurring once per block. The shading treatments were applied either from August to December, December to February, February to May, or not at all. For more details of the experiment, look ahead to Figure 10.4. For comparing treatments, there is no loss of information from basing analysis on the plot means. The four individual vine results that are averaged to give the plot mean are multiple measurements on the same experimental unit, here a plot. Figure 2.14 plots both the aggregated means and the individual vine results. As treatments were applied to whole plots, the graph that shows the individual vine results exaggerates the extent of information that is available, in each block, for comparing treatments. For

64

Styles of data analysis Individual vine yields

Plot means (4 vines)

85 90 95 100 105 110

east

north

west

Feb2May Dec2Feb Aug2Dec none

85 90 95 100 105 110

85 90 95 100 105 110

yield

Figure 2.14 The four panels are the four different plots. The solid gray points are plot means. The open gray circles are yields for individual vines in the plot.

gaining a correct impression of the strength of the evidence, it is best to focus the eye on the means, shown as +. The code for Figure 2.14 is given as a footnote.12 The first few rows of the data frame are: 1 2 3 4 5

yield 101.11 108.02 106.67 100.30 92.64

block shade plot north none north.none north none north.none north none north.none north none north.none west none west.none

The aggregate() function splits the data frame according to the specified combinations of factor levels, and then applies a specified function to each of the resulting subgroups. Here, it forms a data frame that has the mean for each combination of block and shading treatment. The code, with the first line of output following, is: > > + > >

## mean yield by block by shade: data frame kiwishade (DAAG) kiwimeans <- with(kiwishade, aggregate(yield, by=list(block, shade), mean)) names(kiwimeans) <- c("block","shade","meanyield") head(kiwimeans, 4) block shade meanyield 1 east none 99.0250 > # . . .

12

## Individual vine means, by block and treatment library(lattice) ## Panel function calls panel.dotplot(), then panel.average() dotplot(shade ˜ yield | block, data=kiwishade, aspect=1, panel=function(x,y,...){panel.dotplot(x, y, pch=1, col="gray40") panel.average(x, y, type="p", col="black", pch=3, cex=1.25)}, key=list(space="top", columns=2, col=c("gray40", "black"), text=list(c("Individual vine yields", "Plot means (4 vines)")), points=list(pch=c(1,3), cex=c(1,1.25))), layout=c(3,1)) # Note that parameter settings were given both in the calls to the # panel functions and in the list supplied to key.

2.2 Data summary

65

Use of the aggregated data for analysis commits us to working with plot means. What information is lost? If there were occasional highly aberrant values, use of medians might be preferable. The data should have a say in determining the form of summary.

The benefits of data summary – dengue status example

Hales et al. (2002) examined the implications of climate change projections for the worldwide distribution of dengue, a mosquito-borne disease that is a risk in hot and humid regions. Dengue status, i.e., information on whether dengue had been reported during 1965–1973, is available for 2000 administrative regions. Climate information is available on a much finer scale, on a grid of about 80 000 pixels at 0.5◦ latitude and longitude resolution. Should the analysis work with a data set that consists of 2000 administrative regions, or with the much larger data set that has one row for each of the 80 000 pixels? The following are reasons that might have argued for working with the summarized data: r Dengue status is a summary figure that is given by administrative region. An analysis that uses the separate data for the 80 000 pixels will, in effect, predict dengue status for climate variable values that are in some sense averages for the administrative region. Explicit averaging, prior to the analysis, gives the user control over the form of averaging that will be used. If, for example, values for some pixels are extreme relative to other pixels in the administrative region, medians may be more appropriate than means. In some regions, the range of climatic variation may be extreme. The mean will give the same weight to sparsely populated cold mountainous locations as to highly populated hot and humid locations on nearby plains. r Correlation between observations that are close together geographically, though still substantial, will be less of an issue for the data set in which each row is an administrative region. Points that repeat essentially identical information are a problem both for the interpretation of plots and, often, for the analysis. Regions that are geographically close will often have similar climates and the same dengue status. r Analysis is more straightforward with data sets that are of modest size. It is easier to do standard forms of data checking. The points that appear on plots are more nearly independent. Standard forms of scatterplot less readily degenerate into a dense mass of black ink. There are many possible ways to calculate a central value, of which the mean and the median are the most common. (In fact, however, the paper used the disaggregated data.) 2.2.3 Standard deviation and inter-quartile range

An important measure of variation in a population is the population standard deviation (often written σ ), which is almost always unknown. The variance σ 2 , which is the square of the standard deviation, is widely used in formal inference. The sample standard deviation, used to estimate the population standard deviation when a random sample has been taken, is  ¯ 2 (x − x) . s= n−1

66

Styles of data analysis

Table 2.1 Standard deviations for cuckoo egg data. Hedge sparrow 1.049

Meadow pipit

Pied wagtail

Robin

Tree pipit

Wren

0.920

1.072

0.682

0.880

0.754

In words, take the difference of each data value from the mean, square, add the squared differences together, divide by n − 1, and take the square root. In R, use the function sd() to calculate the standard deviation, or var() to calculate the variance. The standard deviation is in the same units as the original measurements. For s to be an accurate estimate of σ , the sample must be large. Cuckoo eggs example

Consider again the data on cuckoo eggs that we discussed in Subsection 2.1.4. The group standard deviations are listed in Table 2.1.13 The variability in egg length is smallest when the robin is the host. Degrees of freedom

The denominator n − 1 is the number of degrees of freedom remaining after estimating the mean. With one data point, the sum of squares about the mean is zero, the degrees of freedom are zero, and no estimate of the variance is possible. The degrees of freedom are the number of data values, additional to the first data value. In later chapters, standard deviation calculations will be based on the variation that remains after fitting a model (most simply, a line) to the data. Degrees of freedom are reduced by 1 for each model parameter that is estimated. Other measures of variability

The standard deviation is similar in concept to the inter-quartile range H , which we saw in Subsection 2.1.1 is the difference between the first and third quartiles. (The region between the lower and upper quartiles takes in 50% of the data.) For data that are approximately normally distributed, note the approximate relationship s ≈ 0.75H. If data are approximately normally distributed, one standard deviation either side of the mean takes in roughly 68% of the data. Note also the median absolute deviation, calculated using the function mad(). This calculates the median of the absolute deviations from the median. By default this is multiplied by 1.4286, to ensure that in a large sample of normally distributed values the value returned should approximately equal the standard deviation. 13

## SD of length, by species: data frame cuckoos (DAAG) sapply(split(cuckoos$length, cuckoos$species), sd) # Subsection 14.9.6 has information on split()

2.2 Data summary

67

The pooled standard deviation

Consider two independent samples of sizes n1 and n2 , respectively, randomly selected from populations that have the same amount of variation but for which the means may differ. Thus, two means must be estimated. The number of degrees of freedom remaining for estimating the (common) standard deviation is n1 + n2 − 2. We compute the so-called pooled standard deviation by summing squares of differences of each data value from their respective sample mean, dividing by the degrees of freedom n1 + n2 − 2, and taking the square root:   ¯ 2 ¯ 2 + (y − y) (x − x) sp = . n1 + n2 − 2 Use of this pooled estimate of the standard deviation is appropriate if variation in the two populations is plausibly similar. The pooled standard deviation is estimated with more degrees of freedom, and therefore, more accurately, than either of the separate standard deviations. Elastic bands example

Consider data from an experiment in which 21 elastic bands were randomly divided into two groups, one of 10 and one of 11. Bands in the first group were immediately tested for the amount that they stretched under a weight of 1.35 kg. The other group were dunked in hot water at 65◦ C for four minutes, then left at air temperature for ten minutes, and then tested for the amount that they stretched under the same 1.35 kg weight as before. The results were: Ambient: 254 252 239 240 250 256 267 249 259 269 (Mean = 253.5) Heated: 233 252 237 246 255 244 248 242 217 257 254 (Mean = 244.1) The pooled standard deviation estimate is s = 10.91, with 19 (= 10 + 11 − 2) degrees of freedom. Since the separate standard deviations (s1 = 9.92; s2 = 11.73) are similar, the pooled standard deviation estimate is an acceptable summary of the variation in the data. 2.2.4 Correlation

The usual Pearson or product–moment correlation is a summary measure of linear relationship. Calculation of a correlation should always be accompanied by a check that the relevant scatterplot shows a linear relationship. Often the addition of a smooth trend line helps the assessment. Check also that the marginal distributions of the two variables are roughly normal, or at least not highly skew. If the relationship is monotonic, but is not linear and/or has asymmetric marginal distributions, it may be appropriate to use a Spearman rank correlation. Examples of the needed code are: > ## Correlation between body and brain: data frame Animals (MASS) > ## Product--moment correlation > with(Animals, cor(body, brain)) [1] -0.005341 > ## Product--moment correlation, after log transformation > with(log(Animals), cor(body, brain))

68

Styles of data analysis 0 894

0 878

0 458

0 005

Figure 2.15 Different relationships between y and x. In the second panel, the Pearson correlation is 0.878, while the Spearman rank correlation is 0.928.

[1] 0.7795 >> ## Spearman rank correlation > with(Animals, cor(body, brain, method="spearman")) [1] 0.7163

The function cor.test() returns a confidence interval, and tests for no association. Figure 2.15 gives four graphs to consider. For which does it make sense to calculate 1. A Pearson correlation coefficient? 2. A Spearman rank correlation? The figure that appears in the upper left in each panel is the Pearson correlation. For the second panel, the Pearson correlation is 0.878, while the Spearman correlation, which better captures the strength of the relationship, is 0.928. Here a linear fit clearly is inadequate. The magnitude of the correlation r, or of the squared correlation r 2 , does not of itself indicate whether the fit is adequate. Note also the Kendall correlation, obtained by specifying method="kendall" when cor.test() is called. This is often used in contexts where the same individuals are assessed by different judges. It estimates the probability that the two judges will assign the same ranking to an individual. Here are ways in which the use of correlation may mislead: r The usual interpretation of the magnitude of the coefficient assumes that sample pairs (x, y) have been taken at random from a bivariate normal distribution. Observations must be independent, and the separate marginal distributions of x and y must be approximately normal. If, for example, the marginal distributions are highly asymmetric, the correlation is likely to be smaller, with increased statistical variability. r There may be a subgroup structure in the data. If, for example, values of x and/or y are quite different for males and females, then the correlation may only reflect a difference between the sexes. Or, if random samples are taken from each of a number of villages and the data are pooled, then it will be unclear whether any correlation reflects a correlation between village averages or a correlation between individuals within villages, or a bit of each. The interpretation is confused because the two correlations may not be the same, and may even go in different directions. See Cox and Wermuth (1996). r Any correlation between a constituent and a total amount is likely to be, in part at least, a mathematical artifact. Thus, consider a study of an anti-hypertensive drug that hopes to determine whether the change y − x is larger for those with higher initial blood

2.3 Statistical analysis questions, aims, and strategies

69

pressure. If x and y have similar variances then, unfortunately, y − x will have a negative correlation with x, whatever the influence of the initial blood pressure. Note that while a correlation coefficient may sometimes be a useful single number summary of the relationship between x and y, regression methods offer a much richer framework for the examination of such relationships. 2.3 Statistical analysis questions, aims, and strategies Logically, this section might have appeared at the beginning of the chapter, prior to any discussion of analysis methods and approaches. It is here because the reader should by now have a level of familiarity with several data sets that will be used as a basis for discussion. Different questions, asked of the same data, will demand different analyses. Questions of interest may, given the available data, be unanswerable. Data on house prices in London may not have much relevance, if the interest is in house prices in New York or Paris! Questions should be structured with issues of this type in mind. Questions should be structured with a view to the intended use of results. Is the aim scientific understanding, perhaps as in the example discussed below to determine whether cuckoos do really match the eggs that they lay in the nests of other birds to the size and color of the host eggs? Or is the aim prediction, perhaps to predict, based on recent prices in the area and on house size, the price that purchasers may be willing to pay? Effective use of the information in the data

Figure 2.6 was designed to elicit the relationship between electrical resistance and apparent juice content, in kiwifruit. With the data we have, it would be bad practice to do a formal statistical test to compare, for example, juice content of less than 30% with juice content of more than 50%. Such an analysis would miss the relatively rich relationship that exists between the apparent juice content and resistance. The data are more informative than would be obtained by doing repeated trials, some with a juice content of 30% and some with a juice content of 50%. Even worse would be a study comparing resistance at 40% juice content with resistance at 50% juice content. Such a study would have very little chance of finding anything of consequence. Observational versus experimental data

Data from experiments appear throughout this book – examples are the data on the tinting of car windows that was used for Figure 2.12 in Subsection 2.1.6, and the kiwifruit shading data that were discussed in Subsection 2.2.2. Data from an experiment can, if well designed with a view to answering the questions of interest, give highly reliable results. With data from carefully designed experiments, perhaps the most serious danger is that the data will be generalized beyond the limits imposed by the experimental conditions. Observational data are another matter. Section 13.2 will discuss a comparison between results from an experimental study on the effects of a work training program (those enrolled in the study were randomly assigned to training and non-training groups), and results from various sets of matched observational data that have been used in the attempt to answer the

70

Styles of data analysis

Table 2.2 Mean lengths of cuckoo eggs, compared with mean lengths of eggs laid by the host bird species. More extensive data that bear on the comparison between cuckoo eggs and host eggs are in the data frame cuckoohosts (DAAG).

Host species

Meadow pipit

Hedge sparrow

Robin

Wagtails

Tree pipit

Wren

Yellow hammer

Length (cuckoo) Length (host)

22.3 (45) 19.7 (74)

23.1 (14) 20.0 (26)

22.5 (16) 20.2 (57)

22.6 (26) 19.9 (16)

23.1 (15) 20 (27)

21.1 (15) 17.7 (–)

22.6 (9) 21.6 (32)

(Numbers in parentheses are numbers of eggs.)

same question. In this instance results from the use of observational data can be compared with those from an experiment in which individuals were randomly assigned either to an experimental or to a control group. 2.3.1 How relevant and how reliable are the data?

Latter (1902) collected the cuckoo egg data presented in Figure 2.8 in order to investigate claims, made in Newton (1893–1896, p. 123), that the eggs that cuckoos lay in the nests of other birds tend to match the eggs of the host bird in size, shape, and color. Figure 2.8 strongly indicated differences, depending on the host bird, in length. A further step is to look for a relationship between the mean size of the cuckoo eggs for each specific host, and the mean size of the host bird’s own eggs, using data such as in Table 2.2. There are various difficulties with the data in Table 2.2. The cuckoo eggs and the host eggs are from different nests, collected in the course of different investigations. Data on the host eggs are from various sources. For the wren, the value is an indicative length from Gordon (1894). There is thus a risk of biases, different for the different sources of data, that limit the inferences that can be drawn. There is a striking difference between wrens and other species. Not only are their own eggs easily the smallest among the species considered; the eggs of the wren host are easily the smallest, among any of the hosts. Whatever biases may exist in the data, it is unlikely that they would be so large as to affect these major differences. Biases might well affect comparisons between eggs in the nests of species other than wrens. 2.3.2 How will results be used?

Studies may be designed to help scientific understanding. Consider again the data in Table 2.2. The interest of Latter’s paper is primarily in establishing whether there is a relationship, and rather less in determining the nature of the relationship. Egg size and shape is one of several pieces of evidence that Latter considers. Uniquely among the birds listed, the architecture of wren nests makes it impossible for the birds to see the eggs. In wren nests, the color of the cuckoo’s egg does not match the color of the wren’s eggs; for the other species the color does mostly match. Latter concludes that Newton is right, that the eggs that cuckoos lay tend to match the eggs of the host bird in size and shape in ways that will make it difficult for hosts to distinguish their eggs from the cuckoo eggs.

2.3 Statistical analysis questions, aims, and strategies

71

This is very different from the demands of the real-estate agent who may hope, on the basis of last year’s prices in a city location, and floor area, to predict the prices that purchasers will be willing to pay. Will a relationship that seems to work in one suburb apply also in another suburb, or in a neighboring city? Accurate prediction is crucial, with a perhaps reduced importance given to understanding the detailed reasons for any relationship that may be apparent. It is important, also, to know the intended use of the data and hence what sort of accuracy is important. Is it accuracy for purposes of making a prediction on one of the suburb(s) used in obtaining the data? Or is it accuracy for making predictions in new suburb(s)? In the latter case, data from multiple suburbs will be needed, and it must be possible to treat the sampled suburbs as a random sample from the suburbs for which predictions are required. 2.3.3 Formal and informal assessments

Statistical data analysis is, often, crucial to the answering of scientific questions. It does not however stand alone, but must be interpreted against a background of subject area knowledge and judgment. Qualitative judgments are inevitable at various points in studies that generate data sets such as are analyzed in this book. Such judgments affect the use of assumed subject area knowledge, the measurements that are taken, the design of data collection, the choice of analysis, and the interpretation of analysis results. These judgments, while they should be as informed as possible, cannot be avoided. Two examples will now be given: r In trials that evaluate therapies for conditions that commonly lead to early death, what is the relevant measure? Is it survival time from diagnosis? Or is it more appropriate to use a measure that takes account of quality of life over that time, which differs hugely between different therapies? Two such measures are “Disability Adjusted Life Years” (DALYs) and “Quality Adjusted Life Years” (QALYs). r The dataset nswpsid1 (see Subsection 2.2.1 and Section 13.2) allows comparison of two groups of individuals, both with a history of employment and related difficulties. A focus of interest was income in 1978, subsequent to the study. Because the distribution of income is highly skew, comparisons that are based directly on income will be biased towards the experience of those few individuals whose incomes were very large. This effect can be ameliorated by working with the logarithm of income. Or it might be more appropriate to compare the median salaries of the two groups, after adjusting for the effects of other variables. In neither case is the interpretation of analysis results as simple as might initially appear. There is an inevitable risk that assumed insights and judgments will carry large elements of prejudice or personal bias. A well-designed study will allow some opportunity for study results to challenge the assumed insights and understandings that have motivated the study. Questionnaires and surveys

The science and socsupport data frames (DAAG) are both from surveys. In both cases, an important question is whether the questions measured what they claimed to measure.

72

Styles of data analysis

In the science data set, a focus of interest is the variable like, which measured the students’ liking for science. What did students understand by “science”? Was science, for them, a way to gain and test knowledge of the world? Or was it a body of knowledge? Or, more likely, was it a name for their experience of science laboratory classes (smells, bangs, and sparks perhaps) and field trips? In the socsupport data set, an important variable is Beck Depression Index or BDI. The Beck Depression Index is a standard psychological measure of depression (see, e.g., Streiner and Norman, 2003), derived from a carefully designed and tested questionnaire. In either case it is impossible to escape the question: “What was measured?” This question is itself amenable to experimental investigation. For the data frame science, answers to other questions included in the survey shed some light. The Beck Depression Index is the result of an extensive process of development and testing that has seemed to validate its results, at least for populations on which it has been tested. Such background evidence helps in assessing what is measured. Finally, however, there must be a qualitative judgment that brings together subject area knowledge, background information and evidence, and the results of the immediate statistical analysis.

2.3.4 Statistical analysis strategies

We have emphasized the importance of careful initial scrutiny of the data. Techniques of a broadly EDA type have, in addition, a role in scrutinizing results from formal analysis, in checking for possible model inadequacies, and perhaps in suggesting remedies. In later chapters, we will discuss the use of diagnostic statistics and graphs in examination both of the model used and of output from the analysis. These are an “after the event” form of EDA. In the course of an analysis, the data analyst may move backwards and forwards between exploratory analysis and more formal analyses.

2.3.5 Planning the formal analysis

Planning the formal analysis is one aspect of planning a research study. Such advance planning should allow for the possibility of limited changes following preliminary investigation. An ideal is to have available data, and perhaps also analysis results, from a suitably related earlier study. Use of such information to plan the analysis in advance reduces the chance of biasing the new results in a direction that is closest to the analyst’s preference! Even so, graphical checks of the data should precede formal analysis. There may be obvious mistakes. The data may have surprises for the analyst. If available at the beginning of the study, the information from the analysis of earlier data may, additionally, be invaluable in the design of data collection for the new study. When prior data are not available, a pilot study involving a small number of experimental runs can sometimes provide this kind of information. Where it is not altogether clear what to expect, careful preliminary examination is even more necessary. In particular, the analyst should look for

2.4 Recap

r r r r r

73

outliers, clusters in the data, unexpected patterns within groups, between-group differences in the scatter of the data, whether there are unanticipated time trends associated, e.g., with order of data collection.

In all studies, it is necessary to check for obvious data errors or inconsistencies. In addition, there should be checks that the data support the intended form of analysis.

2.3.6 Changes to the intended plan of analysis

What departures from the original plan are acceptable, and what are not? If the exploratory analysis makes it clear that the data should be transformed in order to approximate normality more closely, then use the transformation. It is sometimes useful to do both analyses (with the untransformed as well as with the transformed data) and compare them. On the other hand, if there are potentially a large number of comparisons that could be made, the comparisons that will be considered should be specified in advance. Prior data, perhaps from a pilot study, can assist in this choice. Any investigation of other comparisons may be undertaken as an exploratory investigation, a preliminary to the next study. Data-based selection of one or two comparisons from a much larger number is not appropriate, since huge biases may be introduced. Alternatively, there must be allowance for such selection in the assessment of model accuracy. The issues here are non-trivial, and we defer further discussion until later.

2.4 Recap Exploratory data analysis aims to allow the data to speak for themselves, often prior to or as part of a formal analysis. It gives multiple views of the data that may provide useful insights. Histograms, density plots, stem-and-leaf displays, and boxplots are useful for examining the distributions of individual variables. Scatterplots are useful for looking at relationships two at a time. If there are several variables, the scatterplot matrix provides a compact visual summary of all two-way relationships. Before analysis, look especially for r r r r r r

outliers, skewness (e.g., a long tail) in the distribution of data values, clustering, non-linear bivariate relationships, indications of heterogeneous variability (i.e., differences in variability across samples), whether transformations seem necessary.

After analysis, check residuals for all these same features. Where relationships involve several variables, adequate checks will be possible only after analysis. Failure of the independence assumption is hard to detect, unless the likely form of dependence is known and the sample is large. Be aware of any structure in the data that may be associated with lack of independence.

74

Styles of data analysis

Do not allow the initial scrutiny of data to influence the analysis in ways that may lead to over-interpretation. 2.5 Further reading The books and papers on graphical presentation that were noted in Chapter 1 are equally relevant to this chapter. The books Cleveland (1993, 1994) are especially pertinent to the present chapter. Chatfield (2002) has a helpful and interesting discussion, drawing on consulting experience, of approaches to practical data analysis. On statistical presentation issues, and deficiencies in the published literature, see Andersen (1990), Chanter (1981), Gardner et al. (1983), Maindonald (1992), Maindonald and Cox (1984), Wilkinson and Task Force on Statistical Inference (1999). The Wilkinson et al. paper has helpful comments on the planning of data analysis, the role of exploratory data analysis, and so on. Nelder (1999) is forthright and controversial. Two helpful web pages are http://www.math.yorku.ca/SCS/friendly.html#graph and http:// www.rdg.ac.uk/ssc/publications/publications.html#under References for further reading

Andersen, B. 1990. Methodological Errors in Medical Research: an Incomplete Catalogue. Chanter, D. O. 1981. The use and misuse of regression methods in crop modelling. In: Mathematics and Plant Physiology, eds. D. A. Rose and D. A. Charles-Edwards. Chatfield, C. 2002. Confessions of a statistician. The Statistician 51: 1–20. Cleveland, W. S. 1993. Visualizing Data. Cleveland, W. S. 1994. The Elements of Graphing Data, revised edn. Gardner, M. J., Altman, D. G., Jones, D. R. and Machin, D. 1983. Is the statistical assessment of papers submitted to the British Medical Journal effective? British Medical Journal 286: 1485–8. Maindonald, J. H. 1992. Statistical design, analysis and presentation issues. New Zealand Journal of Agricultural Research 35: 121–41. Maindonald, J. H. and Cox, N. R. 1984. Use of statistical evidence in some recent issues of DSIR agricultural journals. New Zealand Journal of Agricultural Research 27: 597–610. Nelder, J. A. 1999. From statistics to statistical science. Journal of the Royal Statistical Society, Series D 48: 257–67. Wilkinson, L. and Task Force on Statistical Inference. 1999. Statistical methods in psychology journals: guidelines and explanation. American Psychologist 54: 594–604. 2.6 Exercises 1. Use the lattice function bwplot() to display, for each combination of site and sex in the data frame possum (DAAG package), the distribution of ages. Show the different sites on the same panel, with different panels for different sexes. 2. Do a stem-and-leaf display for the lengths of the female possums. On the display, identify the position of the median, either at one of the leaves or between two leaves. Explain the reasoning used to find the median, and use the function median() to check the result.

2.6 Exercises

75

3. Plot a histogram of the earconch measurements for the possum data. The distribution should appear bimodal (two peaks). This is a simple indication of clustering, possibly due to sex differences. Obtain side-by-side boxplots of the male and female earconch measurements. How do these measurement distributions differ? Can you predict what the corresponding histograms would look like? Plot them to check your answer. 4. For the data frame ais (DAAG package), draw graphs that show how the values of the hematological measures (red cell count, hemoglobin concentration, hematocrit, white cell count, and plasma ferritin concentration) vary with the sport and sex of the athlete. 5. Using the data frame cuckoohosts, plot clength against cbreadth, and hlength against hbreadth, all on the same graph and using a different color to distinguish the first set of points (for the cuckoo eggs) from the second set (for the host eggs). Join the two points that relate to the same host species with a line. What does a line that is long, relative to other lines, imply? Here is code that you may wish to use: attach(cuckoohosts) plot(c(clength, hlength), c(cbreadth, hbreadth), col=rep(1:2,c(12,12))) for(i in 1:12)lines(c(clength[i], hlength[i]), c(cbreadth[i], hbreadth[i])) text(hlength, hbreadth, abbreviate(rownames(cuckoohosts),8)) detach(cuckoohosts) 6. Enter and run the following code. Annotate it, describing each function and each function argument: deathrate <- c(40.7, 36,27,30.5,27.6,83.5) hosp <- c("Cliniques of Vienna (1834-63)\n(> 2000 cases pa)", "Enfans Trouves at Petersburg\n(1845-59, 1000-2000 cases pa)", "Pesth (500-1000 cases pa)", "Edinburgh (200-500 cases pa)", "Frankfort (100-200 cases pa)", "Lund (< 100 cases pa)") hosp <- factor(hosp, levels=hosp[order(deathrate)]) dotplot(hosp˜deathrate, xlim=c(0,95), xlab="Death rate per 1000 ", type=c("h","p")) ## Source: \cite{Nightingale}. Data are ascribed to Dr Le Fort 7. The dataset ex10.22, on tomato yields, is available from the package Devore6 (or in wide format from Devore7). Type library(Devore6) # ex10.22 is from Devore6 tomatoes <- ex10.22 This data frame gives tomato yields at four levels of salinity, as measured by electrical conductivity (EC, in nmho/cm). (a) Obtain a scatterplot of yield against EC. (b) Obtain side-by-side boxplots of yield for each level of EC. (c) The third column of the data frame is a factor representing the four different levels of EC. Comment upon whether the yield data are more effectively analyzed using EC as a quantitative or qualitative factor. 8. Examine the help for the function mean(), and use it to learn about the trimmed mean. For the total lengths of female possums, calculate the mean, the median, and the 10% trimmed

76

Styles of data analysis mean. How does the 10% trimmed mean differ from the mean for these data? Under what circumstances will the trimmed mean differ substantially from the mean?

9. Assuming that the variability in egg length for the cuckoo eggs data is the same for all host birds, obtain an estimate of the pooled standard deviation as a way of summarizing this variability. [Hint: Remember to divide the appropriate sums of squares by the number of degrees of freedom remaining after estimating the six different means.] 10. Calculate the following three correlations: with(Animals, cor(brain,body)) with(Animals, cor(log(brain),log(body))) with(Animals, cor(log(brain),log(body), method="spearman")) Comment on the different results. Which is the most appropriate measure of the relationship? 11. Figure 2.10 showed changes in labor force numbers, in six regions of Canada, in successive quarters of 1995–1996. The population (in thousands) aged 15 years and over in each of these regions was, according to the 1996 census: BC: 3955; Alberta: 2055; Prairies: 1604; Ontario: 8249; Quebec: 5673; Atlantic: 1846. Plot a version of Figure 2.10 in which the labor force numbers are standardized by division by the number in the relevant population. Compare a plot that shows all regions in the same panel, with a plot that gives each region its own panel and its own slice of a common scale, commenting on the advantages and disadvantages of each. Is there now any reason to use a logarithmic scale? 12. The following code conveys information that has points of connection with the information in Figure 2.14: bwplot(shade ˜ yield|block, data=kiwishade, layout=c(3,1)) Compare and contrast the information given by these two plots. 13. The galaxies data in the MASS library gives speeds on 82 galaxies (see the help file and the references listed there for more information). Obtain a density plot for these data. Is the distribution strongly skewed? Is there evidence of clustering? 14. The cpus data frame in the MASS library contains information on eight aspects for each of 209 different types of computers. Read the help page for more information. (a) Construct a scatterplot matrix for these data. Should any of the variables be transformed before further analysis is conducted? (b) How well does estimated performance (estperf) predict performance (perf)? Study this question by constructing a scatterplot of these two variables, after taking logarithms. Do the plotted points scatter about a straight line or is there an indication of non-linearity? Is variability in performance the same at each level of performance?

3

Statistical models

Many regularities of nature are taken for granted in daily living – the rising and setting of the sun, the effects of fire in burning anyone unfortunate enough to get too near, and so on. Experience of the world, rather than logical deductive argument, has identified these regularities. Scientific investigation, especially in the physical sciences, has greatly extended and systematized awareness of regularities. Mathematical descriptions, i.e., models, have been crucial for describing and quantifying these regularities. As when any model is pressed into service, it is important to understand which features generalize and which do not. An engineer’s scale model of a building may be helpful for checking the routing of the plumbing but may give little indication of the acoustics of seminar rooms that are included in the building. In medical research, mouse responses to disease and to therapeutic agents are widely used as models for human responses. Experimental responses in the mouse may indicate likely responses in humans. In fundamental research in the physical sciences, deterministic models are often adequate. Statistical variability may be so small that it can, for practical purposes, be ignored. In applications of the physical sciences, variability may more commonly be a serious issue. In studying how buildings respond to a demolition charge, there will be variation from one occasion to another, even for identical buildings and identically placed charges. There will be variation in which parts of the building break first, in what parts remain intact, and in the trajectories of fragments. In the natural sciences, such variability is everywhere. Statistical models rely on probabilistic forms of description that have wide application over all areas of science. They often consist of a deterministic component, with a random component added that is designed to account for residual variation.

3.1 Statistical models As we saw in Chapter 2, consideration of a model stays somewhat in the background in initial exploratory data analysis. The choice of model is crucial in formal analysis. The choice may be influenced by previous experience with comparable data, by subject area knowledge, and by cautious use of what may emerge from exploratory analysis. Models should, wherever possible, be scientifically meaningful, but not at the cost of doing violence to the data. The scientific context includes the analyses, if any, that other researchers have undertaken with related or similar data. It can be important to note and use such analyses critically. While they may give useful leads, there can be serious inadequacies

78

Statistical models

depression weight

20 10 5 0

Depression (mm)

30

weight depression (t) (mm)

0

2 4 6 8 10 Weight of roller (t)

1 2 3 4 5 6 7 8 9 10

1.9 3.1 3.3 4.8 5.3 6.1 6.4 7.6 9.8 12.4

2 1 5 5 20 20 23 10 30 25

1.1 0.3 1.5 1.0 3.8 3.3 3.6 1.3 3.1 2.0

Figure 3.1 Depression in lawn versus roller weight. Both lines were drawn by eye, with the dashed line constrained to go through the origin.

in published analyses. For further detail and discussion, see relevant articles and books in the list of references at the end of Chapter 2. 3.1.1 Incorporation of an error or noise component

Statistical models combine deterministic and random components. The random component is often called noise or error and the deterministic component is sometimes thought of as the signal. It may be helpful to think of the statistical error as the “rough”, and of the model prediction as the “smooth”. The error component models variation that cannot be accounted for by given information. The simplest models assume that the elements of the error component are uncorrelated, i.e., the size and sign (negative or positive) of one element give no information on the likely size and sign of any other element. Both noise and error are technical terms. Use of the word error does not imply that there have been mistakes in the collection of the data, though mistakes can of course contribute to making the variability unnecessarily large. Figure 3.1 shows data from an experiment where different weights of roller were rolled over different parts of a lawn, and the depression noted (data are from Stewart et al., 1988).1 The data seem broadly consistent with the assumption of a signal by which depression is proportional to roller weight, as implied by the solid line in Figure 3.1. Variation about this signal is reflected in variation in the values for depression / weight. More generally, it might be assumed that the signal is a line that does not necessarily assume strict proportionality between depression and weight. The dashed line in Figure 3.1 is an example. It allows for a systematic error in the measurement of depression. 1

## Plot depression vs weight: data frame roller (DAAG) plot(depression ˜ weight, data = roller, xlim=c(0,1.04*max(weight)), ylim=c(0,1.04*max(depression)), xaxs="i", yaxs="i", # "i"=inner: Fit axes exactly to the limits xlab = "Weight of roller (t)", ylab = "Depression(mm)", pch = 16) abline(0, 2.25) # A slope of 2.25 looks about right

3.1 Statistical models

79

The model has the form: observed value = model prediction + statistical error i.e.,

y = µ+ε

where µ = βx (no intercept), or µ = α + βx (with intercept). The model prediction (µ) is the signal component, while ε is the error component. Substituting µ = α + β × weight in the equation: depression = α + β × weight + noise. Here α and β are constants which must be estimated. (For strict proportionality between depression and weight, α will of course be zero.) Use of subscripts allows identification of the individual points. Given observations (x1 , y1 ), (x2 , y2 ), . . . , (xn , yn ), we may write yi = α + βxi + εi .

Predicting with models

The focus of interest – generally prediction and/or interpretation of model parameters – may be different for different uses of model results. For the lawn roller data of Figure 3.1, one focus of interest is the rate of increase of depression with increasing roller weight, i.e., the slope of the line. Another focus is model prediction, i.e., the fitted values. Models should as far as possible yield inferences that, for their intended use, are acceptably accurate. Model structure should reflect data structure. The model treats the pattern of change of depression with roller weight as a deterministic or fixed effect. The measured values of depression incorporate, in addition, a random effect that reflects variation from one part of the lawn to another, differences in the handling of the roller, and measurement error. Data from multiple lawns are essential, for anything more than informal judgment on how results may generalize to other lawns. Such data would allow use of a model that accounts for between-lawn variation, as well as for the within-lawn variation on which our data give information. Chapter 10 will discuss models that might be tried, if data from multiple lawns were available.

Which model is best?

Figure 3.2 shows two possible models for the lawn roller data, together with information that may be helpful in assessing the adequacy of the model. In Figure 3.2A, a line (with intercept) has been fitted, while Figure 3.2B has used lowess() to fit a smooth curve through the data. Sometimes, the fitting of a curve such as in Figure 3.2B helps indicate whether a line really is appropriate. Note that there is just one point that seems to be causing the line, and the fitted curve, to bend down. In any case, there is no statistical justification for fitting a curve rather than a line, as can be verified with a formal analysis; see Exercise 2 in Chapter 7. There is “over-fitting” in Figure 3.2B.

80

Statistical models

10 15 20 25 30

−ve residual

Data values Fitted values (smooth)

+ve residual −ve residual

0

5

Depression in lawn (mm)

+ve residual

5

10 15 20 25 30

B Data values Fitted values (smooth)

0

Depression in lawn (mm)

A

2

4 6 8 10 Roller weight (t)

12

2

4 6 8 10 Roller weight (t)

12

Figure 3.2 In panel A a line has been fitted, while panel B has a smooth curve. Residuals (the “rough”) appear as vertical lines. Positive residuals are black lines, while negative residuals are dashed. Figures 3.2A and B were created using our function g3.2(), which is available from the web page for the book. Interested readers can check the code.

3.1.2 Fitting models – the model formula

Formulae have already been used extensively to describe graphs that will be plotted using plot() or another such function. Modeling functions likewise use formulae to describe the role of variables and factors in models. Thus, the following model statement, with model formula depression ˜ weight, fits a line to the data of Figure 3.1: ## Fit line - by default, this fits intercept & slope. ## requires data frame roller (DAAG) roller.lm <- lm(depression ˜ weight, data=roller) ## Compare with the code used to plot the data plot(depression ˜ weight, data=roller) ## Add the fitted line to the plot abline(roller.lm)

The name roller.lm, used for the output object, was chosen for mnemonic reasons – the object was the result of lm calculations on the roller data set. In the formula, weight is the predictor or explanatory variable, while depression is the response.2

Fitted values, residuals, and coefficients

Residuals, which are the differences between observed values of depression, and predicted values of depression at the respective values of weight, are important for assessing the accuracy of the model fit. A large error component generates large residuals, and works against accurate prediction. Figure 3.2A exhibits the residuals for the lawn roller data after fitting a straight line, while Figure 3.2B exhibits the residuals after fitting a smooth curve. Positive residuals are represented by solid lines, while negative residuals are represented by dashed lines. 2

## For a model that omits the intercept term, specify lm(depression ˜ -1 + weight, data=roller)

3.2 Distributions: models for the random component

81

Most of the information that is commonly required from model objects can be obtained by the use of an extractor function. For example, fitted values and residuals can be calculated with functions fitted() and resid(): > round(fitted(roller.lm), 1) 1 2 3 4 5 6 7 8 9 10 3.0 6.2 6.7 10.7 12.0 14.2 15.0 18.2 24.0 31.0 > round(resid(roller.lm), 1) 1 2 3 4 5 6 7 8 9 10 -1.0 -5.2 -1.7 -5.7 8.0 5.8 8.0 -8.2 6.0 -6.0

The coef() function gives the model coefficients: > coef(roller.lm) (Intercept) weight -2.087148 2.666746

Model objects

The model object, above saved as roller.lm, is a list. Although not always recommended, we can access information in this list directly. For example, we can extract element names as follows: > names(roller.lm) # Get names of list elements [1] "coefficients" "residuals" "effects" [5] "fitted.values" "assign" "qr" [9] "xlevels" "call" "terms"

"rank" "df.residual" "model"

We can then extract information directly from a list element, such as the model coefficients: > roller.lm$coef (Intercept) weight -2.087148 2.666746

For further discussion, see Subsection 14.10.2.

Summary information about model objects

To get a summary that includes coefficients, standard errors of coefficients, t-statistics, and p-values, type summary(roller.lm)

3.2 Distributions: models for the random component In this section, we briefly review the concepts of random variables and their distributions. Our discussion will focus on the more commonly used models for count data and continuous measurement data.

82

Statistical models

3.2.1 Discrete distributions – models for counts

Counts of events or numbers of objects are examples of discrete random variables. The possible values with their associated probabilities are referred to as a distribution. We consider three important examples: Bernoulli, binomial, and Poisson distributions.

Bernoulli distribution

Successive tosses of a fair coin come up tails with probability 0.5, and heads with probability 0.5, independently between tosses. If we let X take the value 1 for a head and 0 for a tail, then X is said to have a Bernoulli distribution with parameter π = 0.5. More generally, we might consider an experiment or test with an uncertain outcome, but where the possibilities are “success” (or “1”) and “failure” (or “0”). Success may occur with probability π, where 0 ≤ π ≤ 1. Binomial distribution

The sum of a number of independent Bernoulli random variables is called a binomial random variable. The number of successes in n independent tests (where success at each trial occurs with probability π) has a binomial distribution with parameters n and π . The total number of heads in two tosses of a fair coin is a binomial random variable with n = 2 and π = 0.5. We can use the function dbinom() to determine probabilities of having 0, 1 or 2 heads in two coin tosses:3 ## To get labeled output exactly as below, see the footnote ## dbinom(0:2, size=2, prob=0.5) # Simple version 0 1 2 0.25 0.50 0.25

On average, 25% of all pairs of coin tosses will result in no heads, 50% will have one head, and 25% will have two heads. The number of heads in four coin tosses can be modeled as binomial with n = 4 and π = 0.5: ## dbinom(0:4, size=4, prob=0.5) 0 1 2 3 4 0.0625 0.2500 0.3750 0.2500 0.0625

To calculate the probability of no more than two heads, add up the probabilities of 0, 1, and 2 (0.0625 + 0.2500 + 0.3750 = 0.6875). The function pbinom() can be used to determine such cumulative probabilities, thus: pbinom(q=2, size=4, prob=0.5)

3

## To get the labeling (0, 1, 2) as in the text, specify: probs <- dbinom(0:2, size=2, prob=0.5) names(probs) <- 0:2 probs

3.2 Distributions: models for the random component

83

For another example, suppose a sample of 50 manufactured items is taken from an assembly line that produces 20% defective items, on average. To find the probability of observing no more than four defectives in a sample, use: > pbinom(q=4, size=50, prob=0.2) [1] 0.0185

The probability of observing fewer than five defectives in the sample is 0.0185. The function qbinom() goes in the other direction, from cumulative probabilities to numbers of events; it is used to compute quantiles, a generalization of the more familiar term percentiles. To calculate a 70th percentile of the distribution of the number of heads in four coin tosses, type: > qbinom(p = 0.70, size = 4, prob = 0.5) [1] 3

Means and standard deviations

In four fair coin tosses, we expect to see two heads on average. In a sample of 50 manufactured items from a population where 20% are defective, we expect to see 10 defectives on average. In general, we can compute the expected value or mean of a binomial random variable using the formula nπ . The standard deviation is one way of summarizing the spread of a probability distribution; it relates directly to the degree of uncertainty associated with predicting the value of a random variable. High values reflect more uncertainty than low values. The formula √ nπ (1 − π ) gives the standard deviation for a binomial random variable. The standard deviation of the number of heads in four coin tosses is 1, and for the number of defectives in our sample of 50 items, it is 2.83. In an absolute sense, we will be able to predict the number of heads more precisely than the number of defectives. The variance is defined as the square of the standard deviation: for the binomial, it is nπ (1 − π). Poisson distribution

The Poisson distribution is often used to model the number of events that occur in a certain time interval or for the numbers of defects that are observed in items such as manufactured products. The distribution depends on a parameter λ (the Greek letter “lambda”), which happens to coincide with the mean or expected value. As an example, consider a population of raisin buns for which there are an average of three raisins per bun, i.e., λ = 3. Because of the mixing process, the number of raisins in a particular bun is uncertain; the possible numbers of raisins are 0, 1, 2, . . . . Under the Poisson model, we have the following probabilities for 0, 1, 2, 3, or 4 raisins in a bun: ## Probabilities of 0, 1, 2, 3, 4 raisins ## mean number of raisins per bun = 3 ## dpois(x = 0:4, lambda = 3) 0 1 2 3 4 0.0498 0.1494 0.2240 0.2240 0.1680

Statistical models

0.2

0.3

pnorm(1) = 0.841

0.1

normal density

84

−3 −2 −1

0

1

2

3

Distance, in SDs, from the mean

Figure 3.3 A plot of the normal density. The horizontal axis is labeled in standard deviations (SDs) distance from the mean. The area of the shaded region is the probability that a normal random variable has a value less than one standard deviation above the mean.

The cumulative probabilities are: ## ppois(q = 0:4, lambda = 3) 0 1 2 3 4 0.0498 0.1991 0.4232 0.6472 0.8153

Thus, for example, the probability of finding two or fewer raisins in a bun is 0.4232. The variance of a Poisson random variable is equal to its mean, i.e., λ. Thus, the variance of the number of raisins in a bun is 3, and the standard deviation is the square root of λ: 1.73.

3.2.2 Continuous distributions

Models for measurement data are examples of continuous distribution. Calculations with continuous distributions are a little different from calculations with discrete distributions. While we can still speak of probabilities of measurements lying in specified intervals, it is no longer useful to consider the probability of a measurement taking on a particular value. A more useful concept is probability density. A continuous random variable is summarized by its density function or curve. The area under any density curve between x = a and x = b gives the probability that the random variable lies between those limits.

Normal distribution

The normal distribution, which has the bell-shaped density curve pictured in Figure 3.3, is often used as a model for continuous measurement data (sometimes a transformation of the data is required in order for the normal model to be useful). The height of the curve is a function of the distance from the mean. The area under the density curve is 1. The density curve plotted in Figure 3.3 corresponds to a normal distribution with a mean of 0 and standard deviation 1. A normal distribution having mean 0 and standard deviation 1 is referred to as the standard normal distribution. Multiplying a fixed value σ by a population of such normal variates changes the standard deviation to σ . By adding a fixed value µ, we can change the mean to µ, leaving the standard deviation unchanged.

3.2 Distributions: models for the random component

85

Here is code that plots the normal density function:4 ## Plot the normal density, in the range -3 to 3 z <- pretty(c(-3,3), 30) # Find ˜30 equally spaced points ht <- dnorm(z) # By default: mean=0, standard deviation=1 plot(z, ht, type="l",xlab="Normal deviate",ylab="Density",yaxs="i") # yaxs="i" locates the axes at the limits of the data

The function pnorm() calculates the cumulative probability, i.e., the area under the curve up to the specified ordinate or x-value. For example, there is a probability of 0.841 that a normal deviate is less than 1: > pnorm(1.0) [1] 0.841

# by default, mean=0 and SD=1

This corresponds to the area of the shaded region in Figure 3.3.5 The function qnorm() can be used to compute the normal quantiles. For example, the 90th percentile is 1.28: > qnorm(.9) [1] 1.28

# 90th percentile; mean=0 and SD=1

The footnote has additional examples.6 Other continuous distributions

There are many other statistical models for continuous observations. The simplest model is the uniform distribution, for which an observation is equally likely to take any value in a given interval; the probability density of values is constant on a fixed interval. Another model is the exponential distribution that gives high probability density to positive values lying near 0; the density decays exponentially as the values increase. The exponential distribution is commonly used to model times between arrivals of customers to a queue. The exponential distribution is a special case of the chi-squared distribution. The latter distribution arises, for example, when dealing with contingency tables. Details on computing probabilities for these distributions can be found in the exercises.

4

5

6

The following gives a closer approximation to Figure 3.3: ## Plot the normal density, in the range -3.25 to 3.25 z <- pretty(c(-3.25,3.25), 30) # Find ˜30 equally spaced points ht <- dnorm(z) # By default: mean=0, standard deviation=1 plot(z, ht, type="l", xlab="Normal deviate", ylab="Ordinate", yaxs="i") polygon(c(z[z <= 1.0], 1.0), c(dnorm(z[z <= 1.0]), 0), col="grey") # Around 84.1% of the total area is to the left of the vertical line. ## Additional examples: pnorm(0) # .5 (exactly half the area is to the left of the mean) pnorm(-1.96) # .025 pnorm(1.96) # .975 pnorm(1.96, mean=2) # .484 (a normal distribution with mean 2 and SD 1) pnorm(1.96, sd=2) # .836 (sd = standard deviation) ## Additional examples: qnorm(0.841) # 1.0 qnorm(0.5) # 0 qnorm(0.975) # 1.96 qnorm(c(.1,.2,.3)) # -1.282 -0.842 -0.524 (10th, 20th and 30th percentiles) qnorm(.1, mean=100, sd=10) # 87.2 (10th percentile, mean=100, SD=10)

86

Statistical models

Different ways to describe distributions

In Subsection 2.1.1 it was noted that, with the default boxplot settings, 1% of values that are drawn at random from a normal distribution will on average be flagged as possible outliers. If the distribution is not symmetric, more than 1% of points may lie outside the whiskers, mostly at the lower end if the distribution is skewed (i.e., with a long tail) to the left, and mostly at the upper end if the distribution is skewed to the right. If the distribution is symmetric, but “heavy-tailed", then a higher proportion of values are out beyond the boxplot whiskers on both sides.

3.3 Simulation of random numbers and random samples In a simulation, repeated random samples are taken from a specified distribution. Statistics, perhaps estimates that are derived from one or other model, can then be calculated for each successive sample. Information is thus obtained on variation under repeated sampling. This allows a check on results predicted by statistical theory. Or it may provide guidance when theoretical results are not available or are of uncertain relevance. This section will begin with simulation of discrete and continuous random variables and will close with a discussion of random sampling from finite populations. Ordinarily, it is undesirable to use the same random number seed in two or more successive calls to a function that uses the random number generator. However, users will sometimes, for purposes of checking a calculation, wish to repeat calculations with the same sequence of random numbers as was generated in an earlier call. The following uses set.seed() to make the call below to rbinom(10, size=1, p=.5) thus reproducible: set.seed(23286) # Use to reproduce the sample below rbinom(10, size=1, p=.5)

The seed for the random number generator is stored in the workspace in a hidden variable (.Random.seed) that changes whenever there has been a call to the random number generator. This ensures that any new simulation will be independent of earlier simulations. When the workspace is saved, .Random.seed is stored as part of the workspace. This ensures that, when the workspace is loaded again, the seed will be restored to its value when the workspace was last saved. Any new simulations will then be independent of those prior to the save. In order to take advantage of this feature, be sure to save the workspace at the end of each session.

Sampling from discrete distributions

Values can be simulated from any of many different distributions. We will offer examples of simulated binomial, Poisson, and normal samples. As a first example, simulate a random sequence of 10 binary digits (0s or 1s) from a population with a specified proportion of 1s, here 50% (i.e., a Bernoulli distribution):

3.3 Simulation of random numbers and random samples

x x x x x x x xx x xxxx xxxx xx xxxxxxxxxxx xx x xxxxxxxxxxxx x xx 7

8

9

10

xx xxxx xxxxx x xxxxxx xx xx x xxxxxxxxx x xxxxxxxxxxxxxxxxx

x

x xx x xx x xx xx x xxxxxx xxx xxxxxxxxxxxxx xxxxxxxxxxxxxxx

x x x x x x x x x x xxxxx x xxxxxxx x xxx xxxxxxx x x xxxx xxxxxxxxx x

87

x xx x xxx x xxx x xxxxx xx xxxxxx x xx xxxxxxx x x xxxxxxxxxxx x x

12

Figure 3.4 Each panel shows a simulated distribution of 50 values from a normal distribution with mean = 10 and sd = 1. The underlying theoretical normal curve is overlaid on the leftmost panel. > rbinom(10, size=1, p=.5) 1 0 0 0 1 1 1 0 1 0

# 10 Bernoulli trials, prob=0.5

The random sample is different on each occasion, depending on the seed. To generate the numbers of daughters in a simulated sample of 25 four-child families, assuming that males and females are equally likely, use the rbinom() function thus: # For the sequence that follows, precede with set.seed(9388) > rbinom(25, size=4, prob=0.5) [1] 3 1 2 4 1 2 0 3 2 1 2 3 2 4 2 1 1 1 2 2 3 2 0 2 2

Now simulate the number of raisins in 20 raisin buns, where the expected number of raisins per bun is 3: > set.seed(9388) > rpois(20, 3) [1] 3 3 4 1 2 2 3 1 1 4 3 0 1 1 3 1 0 4 5 2

3.3.1 Sampling from the normal and other continuous distributions

The function rnorm() generates random deviates from the normal distribution. To generate 10 random values from a standard normal distribution, we type: > options(digits=2) # Suggest number of digits to display > rnorm(10) # 10 random values from the normal distribution # For our sequence, precede with set.seed(3663) [1] -0.599 -1.876 1.441 -1.025 0.612 -1.669 0.138 -0.099 1.010 0.013

Figure 3.4 demonstrates the use of simulation to indicate the extent of sample-to-sample variation in histogram summaries of the data, when five independent random samples of 50 values are taken from a normal distribution. Figure 3.4 shows histograms from five such samples.7 Histograms do not discriminate well between sample values that are consistent 7

## The following gives a rough equivalent of the figure: set.seed (21) # Use to reproduce the data in the figure par(mfrow=c(2,3)) x <- pretty(c(6.5,13.5), 40) for(i in 1:5){ y <- rnorm(50, mean=10, sd=1) hist(y, prob=TRUE, xlim=c(6.5,13.5), ylim=c(0,0.5), main="") lines(x, dnorm(x,10,1)) } par(mfrow=c(1,1))

88

Statistical models

with a normal distribution, and sample values that are not. A better tool for assessing normality, the normal probability plot, will be described in Subsection 3.4.2. Calculations for other distributions, for example runif() to generate uniform random numbers or rexp() to generate exponential random numbers, follow the same pattern. runif(n = 20, min=0, max=1) # 20 numbers, uniform distn on (0, 1) rexp(n=10, rate=3) # 10 numbers, exponential, mean 1/3. ## Exercises at the end of this chapter explore further possibilities.

3.3.2 Simulation of regression data

The following code shows how to simulate a sample of n observations from the model: y = b0 + b1 x + ε where ε is normally distributed with standard deviation σ . We take n = 8, the intercept to be 2, the slope to be 3, and σ to be 2.5 in our simulation, and we use a fixed equally spaced design for the predictor values: > > > > > >

options(digits=3) n <- 8; x <- seq(1,n); sigma <- 2.5; b0 <- 2; error <- rnorm(n, sd=sigma) y <- b0 + b1*x + error

b1 <- 3

t(data.frame(x,y)) [,1] [,2] [,3] [,4] [,5] [,6] [,7] [,8] x 1.00 2.00 3.00 4.0 5.0 6 7.0 8.0 y 6.85 8.96 9.49 12.2 19.1 20 26.2 28.5

It is often useful to repeatedly simulate data from a fitted model, then re-fitting to each new set of simulated data. This provides a check on variation under such repeated simulation. The function simulate() can be used for this purpose. Thus to do 10 simulations based on the model that was fitted to the roller data, do: roller.lm <- lm(depression ˜ weight, data=roller) roller.sim <- simulate(roller.lm, nsim=20) # 20 simulations

The object roller.sim is a data frame with 20 columns, i.e., one column for each of the 20 simulations. Each column has values of depression, simulated from the fitted model at each level of weight. To visualize this output, enter with(roller, matplot(weight, roller.sim, pch=1, ylim=range (depression)) points(roller, pch=16)

3.3.3 Simulation of the sampling distribution of the mean

The sampling distribution of the mean is the distribution of the means of repeated random samples of size n. The standard deviation of this sampling distribution has the name standard error of the mean (SEM). If the population mean is µ and the standard deviation

3.3 Simulation of random numbers and random samples

89

1.0 1.5 2.0 2.5 3.0 3.5

Sample size = 3

Sample size = 9

Sample size = 30

4

Density

3 2 1 0 1.0 1.5 2.0 2.5 3.0 3.5

1.0 1.5 2.0 2.5 3.0 3.5

Figure 3.5 The density curves are from simulations of the sampling distribution of the mean, for a distribution that is mildly skew. Each density curve is from 1000 simulations, which is large enough to give an accurate visual indication of the distribution.

is σ , then s SEM = √ . n The Central Limit Theorem will be important in the discussion of inference in Chapter 4. This theorem implies that, for large enough n, this sampling distribution will closely approximate the normal.8 The sample size n needed so that the normal is a good approximation will depend on the distribution of the population from which samples are taken. Figure 3.5 shows the simulated sampling distribution of the mean, for samples from a distribution that is mildly skew. Even for a sample size of 3, much of the skewness has gone. Code that generates samples from the sampling distribution of means of the required sample sizes is: ## Function to generate n sample values; skew population sampvals <- function(n)exp(rnorm(n, mean = 0.5, sd = 0.3)) ## Means across rows of a dimension nsamp x sampsize matrix of ## sample values gives nsamp means of samples of size sampsize. samplingDist <- function(sampsize=3, nsamp=1000, FUN=mean) apply(matrix(sampvals(sampsize*nsamp), ncol=sampsize), 1, FUN) size <- c(3,10,30) ## Simulate means of samples of 3, 9 and 30; place in dataframe df <- data.frame(y3=samplingDist(sampsize=size[1]), y9=samplingDist(sampsize=size[2]), y30=samplingDist(sampsize=size[3]))

The following then gives a slightly simplified version of Figure 3.5: ## Simulate source population (sampsize=1) y <- samplingDist(sampsize=1) densityplot(˜y3+y9+y30, data = df, outer=TRUE, layout = c(3,1), plot.points = FALSE, panel = function(x, ...) { 8

More precisely, the distribution of the sample mean approximates the normal distribution with arbitrary accuracy, for a sample that is large enough, provided the measurements are independent, and their standard deviation is finite. There are similar results for a number of other sample statistics.

90

Statistical models panel.densityplot(x, ..., col = "black") panel.densityplot(y, col = "gray40", lty = 2, ...) })

Code that will reproduce the strip panel labels is in the footnote.9 The plots can alternatively be obtained using the function sampdist() (DAAG), with default arguments. The skewness of the population can be increased by increasing sd in the call to sampvals(). 3.3.4 Sampling from finite populations

We can use the sample() function to generate a simple random sample from a given set of numbers. Suppose, for example, that names on an electoral roll are numbered from 1 to 9384. We can obtain a random sample of 15 individuals as follows: > ## For the sequence below, precede with set.seed(3676) > sample(1:9384, 15, replace=FALSE) [1] 9178 2408 8724 173 106 4664 3787 6381 5098 3228 8321 165 7332 9036 540

This gives the numerical labels for the 15 individuals that we should include in our sample. The task is then to find them! The option replace=FALSE gives a without-replacement sample, i.e., it ensures that no one is included more than once. To randomly assign 10 plants (labeled from 1 to 10, inclusive) to one of two equal-sized groups, control and treatment, the following code could be used: > ## For the sequence below, precede with set.seed(366) > split(sample(seq(1:10)), rep(c("Control","Treatment"), 5)) > # sample(1:10) gives a random re-arrangement (permutation) > # of 1, 2, ..., 10 $Control [1] 6 8 3 7 9 $Treatment [1] 5 4 2

1 10

We then assign plants 6, 8, 3, 7, and 9 to the control group. By choosing the plants in such a manner, we avoid biases that could arise, for example, due to choosing healthier-looking plants for the treatment group. With-replacement samples

We can randomly sample from the set {1, 2, . . . , 10}, allowing for repeated observations, by using: > sample(1:10, replace=TRUE) [1] 7 5 2 1 2 3 1 5 7 6 9

## Use strip.custom to customize the strip labeling doStrip <- strip.custom(strip.names = TRUE, factor.levels = as.expression(size), var.name = "Sample size", sep = expression(" = ")) ## Then include the argument ’strip=doStrip’ in the call to densityplot

3.4 Model assumptions

91

Cluster sampling

Cluster sampling is one of many different probability-based variants on simple random sampling. See Barnett (2002). In surveys of human populations cluster-based sampling, e.g., samples of households or of localities, with multiple individuals from each chosen household or locality, is likely to introduce a cluster-based form of dependence. The analysis must then take account of this clustering. Standard inferential methods require adaptation to take account of the fact that it is the clusters that are independent, not the individuals within the clusters. Donner and Klar (2000) describe methods that are designed for use in health research. Simulation in teaching and research

The R package animation (Xie and Cheng, 2008) has a number of simulations that are intended for use in teaching or self-instruction. In statistical theory and practice, simulation is widely used to determine the statistical properties of models and/or of model statistics in cases where it has not been possible to derive analytical results. 3.4 Model assumptions Common model assumptions are normality, independence of the elements of the error term, and homogeneity of variance (i.e., the standard deviations of all measurements are the same). If certain assumptions fail to hold, a statistical method may be invalid. Other assumptions may not be as important; we say that the method used is robust against those assumptions. Much of the art of applied statistics comes from knowing which assumptions are important and need careful checking. There are few hard and fast rules. 3.4.1 Random sampling assumptions – independence

Typically, the data analyst has a sample of values that will be used as a window into a wider population. Ideally, data should be gathered in such a way that the independence assumption is guaranteed. This is why randomization is so important in designed experiments, and why random sampling is so important in designed sample surveys. Elementary analysis methods can be modified or extended in various ways to handle modifications of the simple independent random sampling scheme. For example, we can modify the methodology to handle analyses of data from a random sample of clusters of individuals. In practice, analysts may make the random sampling assumption when the selection mechanism does not guarantee randomness. Inferences from data that are chosen haphazardly are inevitably less secure than when we have random samples. Random selection avoids the conscious or unconscious biases that result when survey or other samplers make their own selection, or take whatever items seem suitable. Where there has not been explicit use of a random sampling mechanism, it is necessary to consider carefully how this may have affected the data. Is some form of dependence structure likely? Temporal and spatial dependence arise because values that are close together in time

92

Statistical models

or space are relatively more similar. Is there clustering that arises because all individuals within chosen streets, or within chosen families, have been included. Two individuals in the same family or in the same street may be more similar than two individuals chosen at random from the same city suburb. Often samples are chosen haphazardly, e.g., an experimenter may pick a few plants from several different parts of a plot. Or a survey interviewer may, in a poor-quality survey, seek responses from individuals who can be found in a shopping center. Self-selected samples can be particularly unsatisfactory, e.g., those readers of a monthly magazine who are sufficiently motivated to respond to a questionnaire that is included with the magazine. Failure of the independence assumption is a common reason for wrong statistical inferences. Detecting failure of the independence assumption is often difficult. Tests for independence are at best an occasionally useful guide. They are of little use unless we have some idea how the assumption may have failed, and the sample is large! It is in general better to try to identify the nature of any possible dependence, and use a form of analysis that allows for it. Models that do not obviously reflect mechanisms that generated the data can sometimes be useful for prediction. They can also, if their deficiencies are not understood or if they are used inappropriately, be misleading. Careful checking that the model is serving its intended purpose, and caution, are necessary. 3.4.2 Checks for normality

Many data analysis methods rest on the assumption that the data are normally distributed. Real data are unlikely to be exactly normally distributed. Broadly, gross departures from normality are a cause for concern. Small departures are of no consequence. Check especially for data that are skew. Check also for data that take a small number of discrete values, perhaps as a result of excessive rounding. Whether a specific form of departure will matter depends on the use made of the data. For modest-sized samples, only gross departures will be detectable. For small samples (e.g., less than about 10), it is typically necessary to rely on sources of evidence that are external to the data, e.g., previous experience with similar data. Graphical too