Statistics
121
The book provides unified frameworks for estimation, inference, and software implementation by using the general forms of nonparametric/ semiparametric, linear/nonlinear, and fixed/mixed smoothing spline models. The theory of reproducing kernel Hilbert space (RKHS) is used to present various smoothing spline models in a unified fashion. Although this approach can be technical and difficult, the author makes the advanced smoothing spline methodology based on RKHS accessible to practitioners and students. He offers a gentle introduction to RKHS, keeps theory at a minimum level, and explains how RKHS can be used to construct spline models. Smoothing Splines offers a balanced mix of methodology, computation, implementation, software, and applications. It uses R to perform all data analyses and includes a host of real data examples from astronomy, economics, medicine, and meteorology. The codes for all examples, along with related developments, can be found on the book’s web page.
Smoothing Splines
A general class of powerful and flexible modeling techniques, spline smoothing has attracted a great deal of research attention in recent years and has been widely used in many application areas, from medicine to economics. Smoothing Splines: Methods and Applications covers basic smoothing spline models, including polynomial, periodic, spherical, thin-plate, L-, and partial splines, as well as more advanced models, such as smoothing spline ANOVA, extended and generalized smoothing spline ANOVA, vector spline, nonparametric nonlinear regression, semiparametric regression, and semiparametric mixed-effects models. It also presents methods for model selection and inference.
Monographs on Statistics and Applied Probability 121
Smoothing Splines Methods and Applications
Wang
Yuedong Wang
C7755
C7755_Cover.indd 1
4/25/11 9:28 AM
Smoothing Splines Methods and Applications
C7755_FM.indd 1
4/29/11 2:13 PM
MONOGRAPHS ON STATISTICS AND APPLIED PROBABILITY General Editors F. Bunea, V. Isham, N. Keiding, T. Louis, R. L. Smith, and H. Tong 1 Stochastic Population Models in Ecology and Epidemiology M.S. Barlett (1960) 2 Queues D.R. Cox and W.L. Smith (1961) 3 Monte Carlo Methods J.M. Hammersley and D.C. Handscomb (1964) 4 The Statistical Analysis of Series of Events D.R. Cox and P.A.W. Lewis (1966) 5 Population Genetics W.J. Ewens (1969) 6 Probability, Statistics and Time M.S. Barlett (1975) 7 Statistical Inference S.D. Silvey (1975) 8 The Analysis of Contingency Tables B.S. Everitt (1977) 9 Multivariate Analysis in Behavioural Research A.E. Maxwell (1977) 10 Stochastic Abundance Models S. Engen (1978) 11 Some Basic Theory for Statistical Inference E.J.G. Pitman (1979) 12 Point Processes D.R. Cox and V. Isham (1980) 13 Identification of Outliers D.M. Hawkins (1980) 14 Optimal Design S.D. Silvey (1980) 15 Finite Mixture Distributions B.S. Everitt and D.J. Hand (1981) 16 Classification A.D. Gordon (1981) 17 Distribution-Free Statistical Methods, 2nd edition J.S. Maritz (1995) 18 Residuals and Influence in Regression R.D. Cook and S. Weisberg (1982) 19 Applications of Queueing Theory, 2nd edition G.F. Newell (1982) 20 Risk Theory, 3rd edition R.E. Beard, T. Pentikäinen and E. Pesonen (1984) 21 Analysis of Survival Data D.R. Cox and D. Oakes (1984) 22 An Introduction to Latent Variable Models B.S. Everitt (1984) 23 Bandit Problems D.A. Berry and B. Fristedt (1985) 24 Stochastic Modelling and Control M.H.A. Davis and R. Vinter (1985) 25 The Statistical Analysis of Composition Data J. Aitchison (1986) 26 Density Estimation for Statistics and Data Analysis B.W. Silverman (1986) 27 Regression Analysis with Applications G.B. Wetherill (1986) 28 Sequential Methods in Statistics, 3rd edition G.B. Wetherill and K.D. Glazebrook (1986) 29 Tensor Methods in Statistics P. McCullagh (1987) 30 Transformation and Weighting in Regression R.J. Carroll and D. Ruppert (1988) 31 Asymptotic Techniques for Use in Statistics O.E. Bandorff-Nielsen and D.R. Cox (1989) 32 Analysis of Binary Data, 2nd edition D.R. Cox and E.J. Snell (1989) 33 Analysis of Infectious Disease Data N.G. Becker (1989) 34 Design and Analysis of Cross-Over Trials B. Jones and M.G. Kenward (1989) 35 Empirical Bayes Methods, 2nd edition J.S. Maritz and T. Lwin (1989) 36 Symmetric Multivariate and Related Distributions K.T. Fang, S. Kotz and K.W. Ng (1990) 37 Generalized Linear Models, 2nd edition P. McCullagh and J.A. Nelder (1989) 38 Cyclic and Computer Generated Designs, 2nd edition J.A. John and E.R. Williams (1995) 39 Analog Estimation Methods in Econometrics C.F. Manski (1988) 40 Subset Selection in Regression A.J. Miller (1990) 41 Analysis of Repeated Measures M.J. Crowder and D.J. Hand (1990) 42 Statistical Reasoning with Imprecise Probabilities P. Walley (1991) 43 Generalized Additive Models T.J. Hastie and R.J. Tibshirani (1990) 44 Inspection Errors for Attributes in Quality Control N.L. Johnson, S. Kotz and X. Wu (1991) 45 The Analysis of Contingency Tables, 2nd edition B.S. Everitt (1992)
C7755_FM.indd 2
4/29/11 2:13 PM
46 The Analysis of Quantal Response Data B.J.T. Morgan (1992) 47 Longitudinal Data with Serial Correlation—A State-Space Approach R.H. Jones (1993) 48 Differential Geometry and Statistics M.K. Murray and J.W. Rice (1993) 49 Markov Models and Optimization M.H.A. Davis (1993) 50 Networks and Chaos—Statistical and Probabilistic Aspects O.E. Barndorff-Nielsen, J.L. Jensen and W.S. Kendall (1993) 51 Number-Theoretic Methods in Statistics K.-T. Fang and Y. Wang (1994) 52 Inference and Asymptotics O.E. Barndorff-Nielsen and D.R. Cox (1994) 53 Practical Risk Theory for Actuaries C.D. Daykin, T. Pentikäinen and M. Pesonen (1994) 54 Biplots J.C. Gower and D.J. Hand (1996) 55 Predictive Inference—An Introduction S. Geisser (1993) 56 Model-Free Curve Estimation M.E. Tarter and M.D. Lock (1993) 57 An Introduction to the Bootstrap B. Efron and R.J. Tibshirani (1993) 58 Nonparametric Regression and Generalized Linear Models P.J. Green and B.W. Silverman (1994) 59 Multidimensional Scaling T.F. Cox and M.A.A. Cox (1994) 60 Kernel Smoothing M.P. Wand and M.C. Jones (1995) 61 Statistics for Long Memory Processes J. Beran (1995) 62 Nonlinear Models for Repeated Measurement Data M. Davidian and D.M. Giltinan (1995) 63 Measurement Error in Nonlinear Models R.J. Carroll, D. Rupert and L.A. Stefanski (1995) 64 Analyzing and Modeling Rank Data J.J. Marden (1995) 65 Time Series Models—In Econometrics, Finance and Other Fields D.R. Cox, D.V. Hinkley and O.E. Barndorff-Nielsen (1996) 66 Local Polynomial Modeling and its Applications J. Fan and I. Gijbels (1996) 67 Multivariate Dependencies—Models, Analysis and Interpretation D.R. Cox and N. Wermuth (1996) 68 Statistical Inference—Based on the Likelihood A. Azzalini (1996) 69 Bayes and Empirical Bayes Methods for Data Analysis B.P. Carlin and T.A Louis (1996) 70 Hidden Markov and Other Models for Discrete-Valued Time Series I.L. MacDonald and W. Zucchini (1997) 71 Statistical Evidence—A Likelihood Paradigm R. Royall (1997) 72 Analysis of Incomplete Multivariate Data J.L. Schafer (1997) 73 Multivariate Models and Dependence Concepts H. Joe (1997) 74 Theory of Sample Surveys M.E. Thompson (1997) 75 Retrial Queues G. Falin and J.G.C. Templeton (1997) 76 Theory of Dispersion Models B. Jørgensen (1997) 77 Mixed Poisson Processes J. Grandell (1997) 78 Variance Components Estimation—Mixed Models, Methodologies and Applications P.S.R.S. Rao (1997) 79 Bayesian Methods for Finite Population Sampling G. Meeden and M. Ghosh (1997) 80 Stochastic Geometry—Likelihood and computation O.E. Barndorff-Nielsen, W.S. Kendall and M.N.M. van Lieshout (1998) 81 Computer-Assisted Analysis of Mixtures and Applications— Meta-analysis, Disease Mapping and Others D. Böhning (1999) 82 Classification, 2nd edition A.D. Gordon (1999) 83 Semimartingales and their Statistical Inference B.L.S. Prakasa Rao (1999) 84 Statistical Aspects of BSE and vCJD—Models for Epidemics C.A. Donnelly and N.M. Ferguson (1999) 85 Set-Indexed Martingales G. Ivanoff and E. Merzbach (2000) 86 The Theory of the Design of Experiments D.R. Cox and N. Reid (2000) 87 Complex Stochastic Systems O.E. Barndorff-Nielsen, D.R. Cox and C. Klüppelberg (2001) 88 Multidimensional Scaling, 2nd edition T.F. Cox and M.A.A. Cox (2001)
C7755_FM.indd 3
4/29/11 2:13 PM
89 Algebraic Statistics—Computational Commutative Algebra in Statistics G. Pistone, E. Riccomagno and H.P. Wynn (2001) 90 Analysis of Time Series Structure—SSA and Related Techniques N. Golyandina, V. Nekrutkin and A.A. Zhigljavsky (2001) 91 Subjective Probability Models for Lifetimes Fabio Spizzichino (2001) 92 Empirical Likelihood Art B. Owen (2001) 93 Statistics in the 21st Century Adrian E. Raftery, Martin A. Tanner, and Martin T. Wells (2001) 94 Accelerated Life Models: Modeling and Statistical Analysis Vilijandas Bagdonavicius and Mikhail Nikulin (2001) 95 Subset Selection in Regression, Second Edition Alan Miller (2002) 96 Topics in Modelling of Clustered Data Marc Aerts, Helena Geys, Geert Molenberghs, and Louise M. Ryan (2002) 97 Components of Variance D.R. Cox and P.J. Solomon (2002) 98 Design and Analysis of Cross-Over Trials, 2nd Edition Byron Jones and Michael G. Kenward (2003) 99 Extreme Values in Finance, Telecommunications, and the Environment Bärbel Finkenstädt and Holger Rootzén (2003) 100 Statistical Inference and Simulation for Spatial Point Processes Jesper Møller and Rasmus Plenge Waagepetersen (2004) 101 Hierarchical Modeling and Analysis for Spatial Data Sudipto Banerjee, Bradley P. Carlin, and Alan E. Gelfand (2004) 102 Diagnostic Checks in Time Series Wai Keung Li (2004) 103 Stereology for Statisticians Adrian Baddeley and Eva B. Vedel Jensen (2004) 104 Gaussian Markov Random Fields: Theory and Applications H˚avard Rue and Leonhard Held (2005) 105 Measurement Error in Nonlinear Models: A Modern Perspective, Second Edition Raymond J. Carroll, David Ruppert, Leonard A. Stefanski, and Ciprian M. Crainiceanu (2006) 106 Generalized Linear Models with Random Effects: Unified Analysis via H-likelihood Youngjo Lee, John A. Nelder, and Yudi Pawitan (2006) 107 Statistical Methods for Spatio-Temporal Systems Bärbel Finkenstädt, Leonhard Held, and Valerie Isham (2007) 108 Nonlinear Time Series: Semiparametric and Nonparametric Methods Jiti Gao (2007) 109 Missing Data in Longitudinal Studies: Strategies for Bayesian Modeling and Sensitivity Analysis Michael J. Daniels and Joseph W. Hogan (2008) 110 Hidden Markov Models for Time Series: An Introduction Using R Walter Zucchini and Iain L. MacDonald (2009) 111 ROC Curves for Continuous Data Wojtek J. Krzanowski and David J. Hand (2009) 112 Antedependence Models for Longitudinal Data Dale L. Zimmerman and Vicente A. Núñez-Antón (2009) 113 Mixed Effects Models for Complex Data Lang Wu (2010) 114 Intoduction to Time Series Modeling Genshiro Kitagawa (2010) 115 Expansions and Asymptotics for Statistics Christopher G. Small (2010) 116 Statistical Inference: An Integrated Bayesian/Likelihood Approach Murray Aitkin (2010) 117 Circular and Linear Regression: Fitting Circles and Lines by Least Squares Nikolai Chernov (2010) 118 Simultaneous Inference in Regression Wei Liu (2010) 119 Robust Nonparametric Statistical Methods, Second Edition Thomas P. Hettmansperger and Joseph W. McKean (2011) 120 Statistical Inference: The Minimum Distance Approach Ayanendranath Basu, Hiroyuki Shioya, and Chanseok Park (2011) 121 Smoothing Splines : Methods and Applications Yuedong Wang (2011)
Monographs on Statistics and Applied Probability 121
Smoothing Splines Methods and Applications
Yuedong Wang University of California Santa Barbara, California, USA
CRC Press Taylor & Francis Group 6000 Broken Sound Parkway NW, Suite 300 Boca Raton, FL 33487-2742 © 2011 by Taylor & Francis Group, LLC CRC Press is an imprint of Taylor & Francis Group, an Informa business No claim to original U.S. Government works Version Date: 20110429 International Standard Book Number-13: 978-1-4200-7756-8 (eBook - PDF) This book contains information obtained from authentic and highly regarded sources. Reasonable efforts have been made to publish reliable data and information, but the author and publisher cannot assume responsibility for the validity of all materials or the consequences of their use. The authors and publishers have attempted to trace the copyright holders of all material reproduced in this publication and apologize to copyright holders if permission to publish in this form has not been obtained. If any copyright material has not been acknowledged please write and let us know so we may rectify in any future reprint. Except as permitted under U.S. Copyright Law, no part of this book may be reprinted, reproduced, transmitted, or utilized in any form by any electronic, mechanical, or other means, now known or hereafter invented, including photocopying, microfilming, and recording, or in any information storage or retrieval system, without written permission from the publishers. For permission to photocopy or use material electronically from this work, please access www.copyright.com (http://www.copyright.com/) or contact the Copyright Clearance Center, Inc. (CCC), 222 Rosewood Drive, Danvers, MA 01923, 978-750-8400. CCC is a not-for-profit organization that provides licenses and registration for a variety of users. For organizations that have been granted a photocopy license by the CCC, a separate system of payment has been arranged. Trademark Notice: Product or corporate names may be trademarks or registered trademarks, and are used only for identification and explanation without intent to infringe. Visit the Taylor & Francis Web site at http://www.taylorandfrancis.com and the CRC Press Web site at http://www.crcpress.com
TO YAN, CATHERINE, AND KEVIN
This page intentionally left blank
Contents
1 Introduction 1.1 Parametric and Nonparametric Regression . . 1.2 Polynomial Splines . . . . . . . . . . . . . . . . 1.3 Scope of This Book . . . . . . . . . . . . . . . 1.4 The assist Package . . . . . . . . . . . . . . .
. . . .
. . . .
. . . .
. . . .
. . . .
1 1 4 7 9
2 Smoothing Spline Regression 2.1 Reproducing Kernel Hilbert Space . . . . . . 2.2 Model Space for Polynomial Splines . . . . . 2.3 General Smoothing Spline Regression Models 2.4 Penalized Least Squares Estimation . . . . . 2.5 The ssr Function . . . . . . . . . . . . . . . 2.6 Another Construction for Polynomial Splines 2.7 Periodic Splines . . . . . . . . . . . . . . . . 2.8 Thin-Plate Splines . . . . . . . . . . . . . . . 2.9 Spherical Splines . . . . . . . . . . . . . . . . 2.10 Partial Splines . . . . . . . . . . . . . . . . . 2.11 L-splines . . . . . . . . . . . . . . . . . . . . 2.11.1 Motivation . . . . . . . . . . . . . . . 2.11.2 Exponential Spline . . . . . . . . . . . 2.11.3 Logistic Spline . . . . . . . . . . . . . 2.11.4 Linear-Periodic Spline . . . . . . . . . 2.11.5 Trigonometric Spline . . . . . . . . . .
. . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . .
11 11 14 16 17 20 22 24 26 29 30 39 39 41 44 46 48
3 Smoothing Parameter Selection and Inference 3.1 Impact of the Smoothing Parameter . . . . . . . . 3.2 Trade-Offs . . . . . . . . . . . . . . . . . . . . . . 3.3 Unbiased Risk . . . . . . . . . . . . . . . . . . . . 3.4 Cross-Validation and Generalized Cross-Validation 3.5 Bayes and Linear Mixed-Effects Models . . . . . . 3.6 Generalized Maximum Likelihood . . . . . . . . . 3.7 Comparison and Implementation . . . . . . . . . . 3.8 Confidence Intervals . . . . . . . . . . . . . . . . . 3.8.1 Bayesian Confidence Intervals . . . . . . . . 3.8.2 Bootstrap Confidence Intervals . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
53 53 57 62 64 67 71 72 75 75 81
. . . . . . . . . . . . . . . .
ix
x 3.9
Hypothesis Tests . . . . . . . . . . . . . . . . 3.9.1 The Hypothesis . . . . . . . . . . . . . 3.9.2 Locally Most Powerful Test . . . . . . 3.9.3 Generalized Maximum Likelihood Test 3.9.4 Generalized Cross-Validation Test . . 3.9.5 Comparison and Implementation . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
84 84 85 86 87 87
4 Smoothing Spline ANOVA 4.1 Multiple Regression . . . . . . . . . . . . . . . . . . . 4.2 Tensor Product Reproducing Kernel Hilbert Spaces . 4.3 One-Way SS ANOVA Decomposition . . . . . . . . . 4.3.1 Decomposition of Ra : One-Way ANOVA . . . 4.3.2 Decomposition of W2m [a, b] . . . . . . . . . . . 4.3.3 Decomposition of W2m (per) . . . . . . . . . . . 4.3.4 Decomposition of W2m (Rd ) . . . . . . . . . . . 4.4 Two-Way SS ANOVA Decomposition . . . . . . . . . 4.4.1 Decomposition of Ra ⊗ Rb : Two-Way ANOVA 4.4.2 Decomposition of Ra ⊗ W2m [0, 1] . . . . . . . . 4.4.3 Decomposition of W2m1 [0, 1] ⊗ W2m2 [0, 1] . . . . 4.4.4 Decomposition of Ra ⊗ W2m (per) . . . . . . . . 4.4.5 Decomposition of W2m1 (per) ⊗ W2m2 [0, 1] . . . . 4.4.6 Decomposition of W22 (R2 ) ⊗ W2m (per) . . . . . 4.5 General SS ANOVA Decomposition . . . . . . . . . . 4.6 SS ANOVA Models and Estimation . . . . . . . . . . 4.7 Selection of Smoothing Parameters . . . . . . . . . . 4.8 Confidence Intervals . . . . . . . . . . . . . . . . . . . 4.9 Examples . . . . . . . . . . . . . . . . . . . . . . . . . 4.9.1 Tongue Shapes . . . . . . . . . . . . . . . . . . 4.9.2 Ozone in Arosa — Revisit . . . . . . . . . . . . 4.9.3 Canadian Weather — Revisit . . . . . . . . . . 4.9.4 Texas Weather . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . .
91 91 92 93 95 96 97 97 98 99 100 103 106 107 108 110 111 114 116 117 117 126 131 133
5 Spline Smoothing with Heteroscedastic and/or Correlated Errors 139 5.1 Problems with Heteroscedasticity and Correlation . . . 139 5.2 Extended SS ANOVA Models . . . . . . . . . . . . . . 142 5.2.1 Penalized Weighted Least Squares . . . . . . . . 142 5.2.2 UBR, GCV and GML Criteria . . . . . . . . . . 144 5.2.3 Known Covariance . . . . . . . . . . . . . . . . . 147 5.2.4 Unknown Covariance . . . . . . . . . . . . . . . . 148 5.2.5 Confidence Intervals . . . . . . . . . . . . . . . . 150 5.3 Variance and Correlation Structures . . . . . . . . . . . 150 5.4 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . 153
xi 5.4.1 5.4.2 5.4.3 5.4.4
Simulated Motorcycle Accident — Revisit Ozone in Arosa — Revisit . . . . . . . . . Beveridge Wheat Price Index . . . . . . . Lake Acidity . . . . . . . . . . . . . . . .
. . . .
. . . .
. . . .
. . . .
153 154 157 158
6 Generalized Smoothing Spline ANOVA 163 6.1 Generalized SS ANOVA Models . . . . . . . . . . . . . 163 6.2 Estimation and Inference . . . . . . . . . . . . . . . . . 164 6.2.1 Penalized Likelihood Estimation . . . . . . . . . 164 6.2.2 Selection of Smoothing Parameters . . . . . . . . 167 6.2.3 Algorithm and Implementation . . . . . . . . . . 168 6.2.4 Bayes Model, Direct GML and Approximate Bayesian Confidence Intervals . . . . . . . . . . . 170 6.3 Wisconsin Epidemiological Study of Diabetic Retinopathy . . . . . . . . . . . . . . . . . . . . . . . . 172 6.4 Smoothing Spline Estimation of Variance Functions . . 176 6.5 Smoothing Spline Spectral Analysis . . . . . . . . . . . 182 6.5.1 Spectrum Estimation of a Stationary Process . . 182 6.5.2 Time-Varying Spectrum Estimation of a Locally Stationary Process . . . . . . . . . . . . . . . . . 183 6.5.3 Epileptic EEG . . . . . . . . . . . . . . . . . . . 185 7 Smoothing Spline Nonlinear Regression 195 7.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . 195 7.2 Nonparametric Nonlinear Regression Models . . . . . . 196 7.3 Estimation with a Single Function . . . . . . . . . . . . 197 7.3.1 Gauss–Newton and Newton–Raphson Methods . 197 7.3.2 Extended Gauss–Newton Method . . . . . . . . . 199 7.3.3 Smoothing Parameter Selection and Inference . . 201 7.4 Estimation with Multiple Functions . . . . . . . . . . . 204 7.5 The nnr Function . . . . . . . . . . . . . . . . . . . . . 205 7.6 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . 206 7.6.1 Nonparametric Regression Subject to Positive Constraint . . . . . . . . . . . . . . . . . . . . . . 206 7.6.2 Nonparametric Regression Subject to Monotone Constraint . . . . . . . . . . . . . . . . . . . . . . 207 7.6.3 Term Structure of Interest Rates . . . . . . . . . 212 7.6.4 A Multiplicative Model for Chickenpox Epidemic 218 7.6.5 A Multiplicative Model for Texas Weather . . . . 223
xii 8 Semiparametric Regression 8.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . 8.2 Semiparametric Linear Regression Models . . . . . . 8.2.1 The Model . . . . . . . . . . . . . . . . . . . 8.2.2 Estimation and Inference . . . . . . . . . . . 8.2.3 Vector Spline . . . . . . . . . . . . . . . . . . 8.3 Semiparametric Nonlinear Regression Models . . . . 8.3.1 The Model . . . . . . . . . . . . . . . . . . . 8.3.2 SNR Models for Clustered Data . . . . . . . 8.3.3 Estimation and Inference . . . . . . . . . . . 8.3.4 The snr Function . . . . . . . . . . . . . . . 8.4 Examples . . . . . . . . . . . . . . . . . . . . . . . . 8.4.1 Canadian Weather — Revisit . . . . . . . . . 8.4.2 Superconductivity Magnetization Modeling . 8.4.3 Oil-Bearing Rocks . . . . . . . . . . . . . . . 8.4.4 Air Quality . . . . . . . . . . . . . . . . . . . 8.4.5 The Evolution of the Mira Variable R Hydrae 8.4.6 Circadian Rhythm . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . .
227 227 228 228 229 233 240 240 241 242 245 247 247 254 257 259 262 267
9 Semiparametric Mixed-Effects Models 9.1 Linear Mixed-Effects Models . . . . . . . . . . . 9.2 Semiparametric Linear Mixed-Effects Models . . 9.2.1 The Model . . . . . . . . . . . . . . . . . 9.2.2 Estimation and Inference . . . . . . . . . 9.2.3 The slm Function . . . . . . . . . . . . . 9.2.4 SS ANOVA Decomposition . . . . . . . . 9.3 Semiparametric Nonlinear Mixed-Effects Models 9.3.1 The Model . . . . . . . . . . . . . . . . . 9.3.2 Estimation and Inference . . . . . . . . . 9.3.3 Implementation and the snm Function . . 9.4 Examples . . . . . . . . . . . . . . . . . . . . . . 9.4.1 Ozone in Arosa — Revisit . . . . . . . . . 9.4.2 Lake Acidity — Revisit . . . . . . . . . . 9.4.3 Coronary Sinus Potassium in Dogs . . . . 9.4.4 Carbon Dioxide Uptake . . . . . . . . . . 9.4.5 Circadian Rhythm — Revisit . . . . . . .
. . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . .
273 273 274 274 275 279 280 283 283 284 286 288 288 291 294 305 310
A Data Sets A.1 Air Quality Data . . . . . . . . . . A.2 Arosa Ozone Data . . . . . . . . . A.3 Beveridge Wheat Price Index Data A.4 Bond Data . . . . . . . . . . . . . A.5 Canadian Weather Data . . . . .
. . . . .
. . . . .
. . . . .
. . . . .
323 324 324 324 324 325
. . . . . . . . . . . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
xiii A.6 Carbon Dioxide Data . . . A.7 Chickenpox Data . . . . . A.8 Child Growth Data . . . . A.9 Dog Data . . . . . . . . . . A.10 Geyser Data . . . . . . . . A.11 Hormone Data . . . . . . . A.12 Lake Acidity Data . . . . . A.13 Melanoma Data . . . . . . A.14 Motorcycle Data . . . . . . A.15 Paramecium caudatum Data A.16 Rock Data . . . . . . . . . A.17 Seizure Data . . . . . . . . A.18 Star Data . . . . . . . . . . A.19 Stratford Weather Data . . A.20 Superconductivity Data . . A.21 Texas Weather Data . . . . A.22 Ultrasound Data . . . . . . A.23 USA Climate Data . . . . A.24 Weight Loss Data . . . . . A.25 WESDR Data . . . . . . . A.26 World Climate Data . . . .
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . .
325 325 326 326 326 327 327 327 328 328 328 328 329 329 329 330 330 331 331 331 332
B Codes for Fitting Strictly Increasing Functions 333 B.1 C and R Codes for Computing Integrals . . . . . . . . . 333 B.2 R Function inc . . . . . . . . . . . . . . . . . . . . . . 336 C Codes for Term Structure of Interest Rates 339 C.1 C and R Codes for Computing Integrals . . . . . . . . . 339 C.2 R Function for One Bond . . . . . . . . . . . . . . . . . 341 C.3 R Function for Two Bonds . . . . . . . . . . . . . . . . 342 References
347
Author Index
355
Subject Index
359
This page intentionally left blank
List of Tables
2.1 2.2 5.1 5.2
Bases of null spaces and RKs for linear and cubic splines under the construction in Section 2.2 with X = [0, b] . . Bases of null spaces and RKs for linear and cubic splines under the construction in Section 2.6 with X = [0, 1] . .
21 23
Standard varFunc classes . . . . . . . . . . . . . . . . . Standard corStruct classes for serial correlation structures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Standard corStruct classes for spatial correlation structures . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
151
A.1 List of all data sets . . . . . . . . . . . . . . . . . . . . .
323
5.3
152 153
xv
This page intentionally left blank
List of Figures
1.1 1.2 1.3 1.4 1.5
Geyser data, observations, the straight line fit, and residuals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Motorcycle data, observations, and a polynomial fit . . . Geyser data, residuals, and the cubic spline fits . . . . . Motorcycle data, observations, and the cubic spline fit . Relationship between functions in the assist package and some of the existing R functions . . . . . . . . . . . . .
2.1 2.2 2.3 2.4 2.5
Motorcycle data, the linear, and cubic spline fits . . . . Arosa data, observations, and the periodic spline fits . . USA climate data, the thin-plate spline fit . . . . . . . . World climate data, the spherical spline fit . . . . . . . . Geyser data, the partial spline fit, residuals, and the AIC and GCV scores . . . . . . . . . . . . . . . . . . . . . . 2.6 Motorcycle data, the partial spline fit, and the AIC and GCV scores . . . . . . . . . . . . . . . . . . . . . . . . . 2.7 Arosa data, the partial spline estimates of the month and year effects . . . . . . . . . . . . . . . . . . . . . . . . . 2.8 Canadian weather data, estimate of the weight function, and confidence intervals . . . . . . . . . . . . . . . . . . 2.9 Weight loss data, observations and the nonlinear regression, cubic spline, and exponential spline fits . . . . . . 2.10 Paramecium caudatum data, observations and the nonlinear regression, cubic spline, and logistic spline fits . . 2.11 Melanoma data, observations, and the cubic spline and linear-periodic spline fits . . . . . . . . . . . . . . . . . . 2.12 Arosa data, the overall fits and their projections . . . . 3.1 3.2 3.3 3.4 3.5
Stratford weather data, observations, and the periodic spline fits with different smoothing parameters . . . . . Weights of the periodic spline filter . . . . . . . . . . . . Stratford data, degrees of freedom, and residual sum of squares . . . . . . . . . . . . . . . . . . . . . . . . . . . Squared bias, variance, and MSE from a simulation . . . PSE and UBR functions . . . . . . . . . . . . . . . . . .
2 2 3 4 10 22 25 28 31 33 34 36 38 43 45 48 51 54 57 59 61 65
xvii
xviii 3.6 3.7 3.8 3.9 4.1 4.2 4.3 4.4 4.5 4.6 4.7 4.8 4.9 4.10 4.11 4.12 4.13 4.14 4.15 4.16 5.1 5.2 5.3 5.4 5.5 5.6 5.7
PSE, CV, and GCV functions . . . . . . . . . . . . . . . Geyser data, estimates of the smooth components in the cubic and partial spline models . . . . . . . . . . . . . . Motorcycle data, partial spline fit, and t-statistics . . . . Pointwise coverages and across-the-function coverages . Ultrasound data, 3-d plots of observations . . . . . . . . Ultrasound data, observations, fits, confidence intervals, and the mean curves among three environments . . . . . Ultrasound data, the overall interaction . . . . . . . . . Ultrasound data, effects of environment . . . . . . . . . Ultrasound data, estimated tongue shapes as functions of length and time . . . . . . . . . . . . . . . . . . . . . . Ultrasound data, the estimated time effect . . . . . . . . Ultrasound data, estimated tongue shape as a function of environment, length and time . . . . . . . . . . . . . . Ultrasound data, the estimated environment effect . . . Arosa data, estimates of the interactions and smooth component . . . . . . . . . . . . . . . . . . . . . . . . . . . . Arosa data, estimates of the main effects . . . . . . . . . Canadian weather data, temperature profiles of stations in four regions and the estimated profiles . . . . . . . . Canadian weather data, the estimated region effects to temperature . . . . . . . . . . . . . . . . . . . . . . . . . Texas weather data, observations as curves . . . . . . . Texas weather data, observations as surfaces . . . . . . . Texas weather data, the location effects for four selected stations . . . . . . . . . . . . . . . . . . . . . . . . . . . Texas weather data, the month effects for January, April, July, and October . . . . . . . . . . . . . . . . . . . . . WMSEs and coverages of Bayesian confidence intervals with the presence of heteroscedasticity . . . . . . . . . . Cubic spline fits when data are correlated . . . . . . . . Cubic spline fits and estimated autocorrelation functions for two simulations . . . . . . . . . . . . . . . . . . . . . Motorcycle data, estimates of the mean and variance functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Arosa data, residuals variances and PWLS fit . . . . . . Beveridge data, time series and cubic spline fits . . . . . Lake acidity data, effects of calcium and geological location . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
68 79 80 83 93 118 119 120 122 123 125 126 128 129 132 134 135 135 137 138 140 141 149 154 155 157 160
xix 6.1 6.2 6.3 6.4 6.5 6.6 6.7
WESDR data, the estimated probability functions . . . Motorcycle data, estimates of the variance function based on three procedures . . . . . . . . . . . . . . . . . . . . Motorcycle data, DGML function, and estimate of the variance and mean functions . . . . . . . . . . . . . . . . Seizure data, the baseline and preseizure IEEG segments Seizure data, periodograms, estimates of the spectra based on the iterative UBR method and confidence intervals . Seizure data, estimates of the time-varying spectra based on the iterative UBR method . . . . . . . . . . . . . . . Seizure data, estimates of the time-varying spectra based on the DGML method . . . . . . . . . . . . . . . . . . .
7.1 7.2 7.3
Nonparametric regression under positivity constraint . . Nonparametric regression under monotonicity constraint Child growth data, cubic spline fit, fit under monotonicity constraint and estimate of the velocity . . . . . . . . . . 7.4 Bond data, unconstrained and constrained estimates of the discount functions, forward rates and credit spread . 7.5 Chickenpox data, time series plot and the fits by multiplicative and SS ANOVA models . . . . . . . . . . . . . 7.6 Chickenpox data, estimates of the mean and amplitude functions in the multiplicative model . . . . . . . . . . . 7.7 Chickenpox data, estimates of the shape function in the multiplicative model and its projections . . . . . . . . . 7.8 Texas weather data, estimates of the mean and amplitude functions in the multiplicative model . . . . . . . . . . . 7.9 Texas weather data, temperature profiles . . . . . . . . . 7.10 Texas weather data, the estimated interaction against the estimated main effect for two stations . . . . . . . . . . 8.1 8.2 8.3 8.4 8.5 8.6
Separate and joint fits from a simulation . . . . . . . . . Estimates of the differences . . . . . . . . . . . . . . . . Canadian weather data, estimated region effects to precipitation . . . . . . . . . . . . . . . . . . . . . . . . . . Canadian weather data, estimate of the coefficient function for the temperature effect . . . . . . . . . . . . . . Canadian weather data, estimates of the intercept and weight functions . . . . . . . . . . . . . . . . . . . . . . Superconductivity data, observations, and the fits by nonlinear regression, cubic spline, nonlinear partial spline, and L-spline . . . . . . . . . . . . . . . . . . . . . . . . .
176 178 182 186 187 189 192 207 210 210 214 219 222 222 224 225 226 236 238 249 249 253
254
xx 8.7 8.8 8.9 8.10 8.11 8.12 8.13 8.14 9.1 9.2 9.3 9.4 9.5 9.6 9.7 9.8 9.9 9.10 9.11 9.12
9.13 9.14
Superconductivity data, estimates of departures from the straight line model and the “interpolation formula” . . . 256 Rock data, estimates of functions in the projection pursuit regression model . . . . . . . . . . . . . . . . . . . . . . 259 Air quality data, estimates of functions in SNR models . 261 Star data, observations, and the overall fit . . . . . . . . 262 Star data, folded observations, estimates of the common shape function and its projection . . . . . . . . . . . . . 264 Star data, estimates of the amplitude and period functions 266 Hormone data, cortisol concentrations for normal subjects and the fits based on an SIM . . . . . . . . . . . . . . . 268 Hormone data, estimate of the common shape function in an SIM and its projection for normal subjects . . . . . . 270 Arosa data, the overall fit, seasonal trend, and long-term trend . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Arosa data, the overall fit, seasonal trend, long-term trend and local stochastic trend . . . . . . . . . . . . . . . . . Lake acidity data, effects of calcium and geological location . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Dog data, coronary sinus potassium concentrations over time . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Dog data, estimates of the group mean response curves . Dog data, estimates of the group mean response curves under new penalty . . . . . . . . . . . . . . . . . . . . . Dog data, predictions for four dogs . . . . . . . . . . . . Carbon dioxide data, observations and fits by the NLME and SNM models . . . . . . . . . . . . . . . . . . . . . . Carbon dioxide data, overall estimate and projections of the nonparametric shape function . . . . . . . . . . . . . Hormone data, cortisol concentrations for normal subjects, and the fits based on a mixed-effects SIM . . . . . Hormone data, cortisol concentrations for depressed subjects, and the fits based on a mixed-effects SIM . . . . . Hormone data, cortisol concentrations for subjects with Cushing’s disease, and the fits based on a mixed-effects SIM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Hormone data, estimates of the common shape functions in the mixed-effect SIM . . . . . . . . . . . . . . . . . . Hormone data, plot of the estimated 24-hour mean levels against amplitudes . . . . . . . . . . . . . . . . . . . . .
290 292 294 295 302 304 305 307 309 312 313
314 315 321
xxi
Symbol Description (x)+ x∧z x∨z det+ kr (x) (·, ·) k · k: X S H L N P A M R(x, z) Rd N S 2m (t1 , · · · , tk ) W2m [a, b] W2m (per) W2m (Rd ) W2m (S) ⊕ ⊗
max{x, 0} min{x, z} max{x, z} Product of the nonzero eigenvalues Scaled Bernoulli polynomials Inner product Norm Domain of a function Unit sphere Function space Linear functional Nonlinear functional Projection Averaging operator Model space Reproducing kernel Euclidean d-space Natural polynomial spline space Sobolev space on [a, b] Sobolev space on unit circle Thin-plate spline model space Sobolev space on unit sphere Direct sum of function spaces Tensor product of function spaces
This page intentionally left blank
Preface
Statistical analysis often involves building mathematical models that examine the relationship between dependent and independent variables. This book is about a general class of powerful and flexible modeling techniques, namely, spline smoothing. Research on smoothing spline models has attracted a great deal of attention in recent years, and the methodology has been widely used in many areas. This book provides an introduction to some basic smoothing spline models, including polynomial, periodic, spherical, thin-plate, L-, and partial splines, as well as an overview of more advanced models, including smoothing spline ANOVA, extended and generalized smoothing spline ANOVA, vector spline, nonparametric nonlinear regression, semiparametric regression, and semiparametric mixed-effects models. Methods for model selection and inference are also presented. The general forms of nonparametric/semiparametric linear/nonlinear fixed/mixed smoothing spline models in this book provide unified frameworks for estimation, inference, and software implementation. This book draws on the theory of reproducing kernel Hilbert space (RKHS) to present various smoothing spline models in a unified fashion. On the other hand, the subject of smoothing spline in the context of RKHS and regularization is often regarded as technical and difficult. One of my main goals is to make the advanced smoothing spline methodology based on RKHS more accessible to practitioners and students. With this in mind, the book focuses on methodology, computation, implementation, software, and application. It provides a gentle introduction to the RKHS, keeps theory at the minimum level, and provides details on how the RKHS can be used to construct spline models. User-friendly software is key to the routine use of any statistical method. The assist library in R implements methods presented in this book for fitting various nonparametric/semiparametric linear/nonlinear fixed/mixed smoothing spline models. The assist library can be obtained at http://www.r-project.org Much of the exposition is based on the analysis of real examples. Rather than formal analysis, these examples are intended to illustrate the power and versatility of the spline smoothing methodology. All data analyses are performed in R, and most of them use functions in the
xxiii
xxiv assist library. Codes for all examples and further developments related to this book will be posted on the web page http://www.pstat.ucsb.edu/faculty/yuedong/book.html This book is intended for those wanting to learn about smoothing splines. It can be a reference book for statisticians and scientists who need advanced and flexible modeling techniques. It can also serve as a text for an advanced-level graduate course on the subject. In fact, topics in Chapters 1–4 were covered in a quarter class at the University of California — Santa Barbara, and the University of Science and Technology of China. I was fortunate indeed to have learned the smoothing spline from Grace Wahba, whose pioneering work has paved the way for much ongoing research and made this book possible. I am grateful to Chunlei Ke, my former student and collaborator, for developing the assist package. Special thanks goes to Anna Liu for reading the draft carefully and correcting many mistakes. Several people have helped me over various phases of writing this book: Chong Gu, Wensheng Guo, David Hinkley, Ping Ma, and Wendy Meiring. I must thank my editor, David Grubbes, for his patience and encouragement. Finally, I would like to thank several researchers who kindly shared their data sets for inclusion in this book; they are cited where their data are introduced. Yuedong Wang Santa Barbara December 2010
Chapter 1 Introduction
1.1
Parametric and Nonparametric Regression
Regression analysis builds mathematical models that examine the relationship of a dependent variable to one or more independent variables. These models may be used to predict responses at unobserved and/or future values of the independent variables. In the simple case when both the dependent variable y and the independent variable x are scalar variables, given observations (xi , yi ) for i = 1, . . . , n, a regression model relates dependent and independent variables as follows: yi = f (xi ) + ǫi ,
i = 1, . . . , n,
(1.1)
where f is the regression function and ǫi are zero-mean independent random errors with a common variance σ 2 . The goal of regression analysis is to construct a model for f and estimate it based on noisy data. For example, for the Old Faithful geyser in Yellowstone National Park, consider the problem of predicting the waiting time to the next eruption using the length of the previous eruption. Figure 1.1(a) shows the scatter plot of waiting time to the next eruption (y = waiting) against duration of the previous eruption (x = duration) for 272 observations from the Old Faithful geyser. The goal is to build a mathematical model that relates the waiting time to the duration of the previous eruption. A first attempt might be to approximate the regression function f by a straight line f (x) = β0 + β1 x. (1.2) The least squares straight line fit is shown in Figure 1.1(a). There is no apparent sign of lack-of-fit. Furthermore, there is no clear visible trend in the plot of residuals in Figure 1.1(b). Often f is nonlinear in x. A common approach to dealing with nonlinear relationship is to approximate f by a polynomial of order m f (x) = β0 + β1 x + · · · + βm−1 xm−1 .
(1.3)
1
2
Smoothing Splines: Methods and Applications (b)
−10
50
waiting (min) 60 70 80
residuals (min) 0 5 10
90
15
(a)
1.5
1.5
2.5 3.5 4.5 duration (min)
2.5 3.5 4.5 duration (min)
FIGURE 1.1 Geyser data, plots of (a) observations and the least squares straight line fit, and (b) residuals.
Figure 1.2 shows the scatter plot of acceleration (y = acceleration) against time after impact (x = time) from a simulated motorcycle crash experiment on the efficacy of crash helmets. It is clear that a straight line cannot explain the relationship between acceleration and time. Polynomials with m = 1, . . . , 20 are fitted to the data, and Figure 1.2 shows the best fit selected by Akaike’s information criterion (AIC). There are waves in the fitted curve at both ends of the range. The fit is still not completely satisfactory even when polynomials up to order 20 are considered. Unlike the linear regression model (1.2), except for small m, coefficients in model (1.3) no longer have nice interpretations.
acceleration (g) −100 −50 0 50
o
ooooo
ooo ooo oooooo ooo oo o o o o o ooooo o
oo o oooo o o o o oo o o o o oo o o o o o o o ooo o o o o
0
10
o o
o oo o o o o o o o oo o o o o o o o o oo o o o o o oo o o oo o o o o o o o o o o oo o o o o o o o oo o o o o o
20
o
o
30 time (ms)
40
50
60
FIGURE 1.2 Motorcycle data, plot of observations, and a polynomial fit.
Introduction
3
In general, a parametric regression model assumes that the form of f is known except for finitely many unknown parameters. The specific form of f may come from scientific theories and/or approximations to mechanics under some simplified assumptions. The assumptions may be too restrictive and the approximations may be too crude for some applications. An inappropriate model can lead to systematic bias and misleading conclusions. In practice, one should always check the assumed form for the function f . It is often difficult, if not impossible, to obtain a specific functional form for f . A nonparametric regression model does not assume a predetermined form. Instead, it makes assumptions on qualitative properties of f . For example, one may be willing to assume that f is “smooth”, which does not reduce to a specific form with finite number of parameters. Rather, it usually leads to some infinite dimensional collections of functions. The basic idea of nonparametric regression is to let the data speak for themselves. That is to let the data decide which function fits the best without imposing any specific form on f . Consequently, nonparametric methods are in general more flexible. They can uncover structure in the data that might otherwise be missed.
(b)
−10
50
waiting (min) 60 70 80
residuals (min) 0 5 10
90
15
(a)
1.5
2.5 3.5 4.5 duration (min)
1.5
2.5 3.5 4.5 duration (min)
FIGURE 1.3 Geyser data, plots of (a) residuals from the straight line fit and the cubic spline fit to the residuals, and (b) the cubic spline fit to the original data.
For illustration, we fit cubic splines to the geyser data. The cubic spline is a special nonparametric regression model that will be introduced in Section 1.2. A cubic spline fit to residuals from the linear model (1.2) reveals a nonzero trend in Figure 1.3(a). This raises the question of
4
Smoothing Splines: Methods and Applications
whether a simple linear regression model is appropriate for the geyser data. A cubic spline fit to the original data is shown in Figure 1.3(b). It reveals that there are two clusters in the independent variable, and a different linear model may be required for each cluster. Sections 2.10, 3.8, and 3.9 contain more analysis of the geyser data. A cubic spline fit to the motorcycle data is shown in Figure 1.4. It fits data much better than the polynomial model. Sections 2.10, 3.8, 5.4.1, and 6.4 contain more analysis of the motorcycle data.
acceleration (g) −100 −50 0 50
o
ooooo
ooo ooo oooooo ooo oo o o o o o ooooo o
oo o oooo o o o o oo o o o o oo o o o o o o o ooo o o o o
0
10
o o
o oo o o o o o o o oo o o o o o o o o oo o o o o o oo o o oo o o o o oo o o o o o o o o o o o o o oo o o o o o
20
o
o
30 time (ms)
40
50
60
FIGURE 1.4 Motorcycle data, plot of observations, and the cubic spline fit.
The above simple exposition indicates that the nonparametric regression technique can be applied to different steps in regression analysis: data exploration, model building, testing parametric models, and diagnosis. In fact, as illustrated throughout the book, spline smoothing is a powerful and versatile tool for building statistical models to exploit structures in data.
1.2
Polynomial Splines
The polynomial (1.3) is a global model which makes it less adaptive to local variations. Individual observations can have undue influence on the fit in remote regions. For example, in the motorcycle data, the behavior of the mean function varies drastically from one region to another.
Introduction
5
These local variations led to oscillations at both ends of the range in the polynomial fit. A natural solution to overcome this limitation is to use piecewise polynomials, the basic idea behind polynomial splines. Let a < t1 < · · · < tk < b be fixed points called knots. Let t0 = a and tk+1 = b. Roughly speaking, polynomial splines are piecewise polynomials joined together smoothly at knots. Formally, a polynomial spline of order r is a real-valued function on [a, b], f (t), such that (i) f is a piecewise polynomial of order r on [ti , ti+1 ), i = 0, 1, . . . , k; (ii) f has r − 2 continuous derivatives and the (r − 1)st derivative is a step function with jumps at knots. Now consider even orders represented as r = 2m. The function f is a natural polynomial spline of order 2m if, in addition to (i) and (ii), it satisfies the natural boundary conditions (iii) f (j) (a) = f (j) (b) = 0, j = m, . . . , 2m − 1. The natural boundary conditions imply that f is a polynomial of order m on the two outside subintervals [a, t1 ] and [tk , b]. Denote the function space of natural polynomial splines of order 2m with knots t1 , . . . , tk as N S 2m (t1 , . . . , tk ). One approach, known as regression spline, is to approximate f using a polynomial spline or natural polynomial spline. To get a good approximation, one needs to decide the number and locations of knots. This book covers a different approach known as smoothing spline. It starts with a well-defined model space for f and introduces a penalty to prevent overfitting. We now describe this approach for polynomial splines. Consider the regression model (1.1). Suppose f is “smooth”. Specifically, assume that f ∈ W2m [a, b] where the Sobolev space W2m [a, b] = f : f, f ′ , . . . , f (m−1) are absolutely continuous, Z b (f (m) )2 dx < ∞ . (1.4) a
For any a ≤ x ≤ b, Taylor’s theorem states that f (x) =
m−1 X ν=0
|
Z x f (ν) (a) (x − u)m−1 (m) (x − a)ν + f (u)du . ν! (m − 1)! a | {z } {z }
polynomial of order m
(1.5)
Rem(x)
It is clear that the polynomial regression model (1.3) ignores the remainder term Rem(x) in the hope that it is negligible. It is often difficult
6
Smoothing Splines: Methods and Applications
to verify this assumption in practice. The idea behind the spline smoothing is to let data decide how large Rem(x) should be. Since W2m [a, b] is an infinite dimensional space, a direct fit to f by minimizing the least squares (LS) n 1X (yi − f (xi ))2 (1.6) n i=1
leads to interpolation. Therefore, certain control over Rem(x) is necessary. One natural approach is to control how far f is allowed to depart from the polynomial model. Under appropriate norms defined later in Sections 2.2 and 2.6, one measure of distance between f and polynomials Rb is a (f (m) )2 dx. It is then reasonable to estimate f by minimizing the LS (1.6) under the constraint Z b (f (m) )2 dx ≤ ρ (1.7) a
for a constant ρ. By introducing a Lagrange multiplier, the constrained minimization problem (1.6) and (1.7) is equivalent to minimizing the penalized least squares (PLS): Z b n 1X (yi − f (xi ))2 + λ (f (m) )2 dx. (1.8) n i=1 a In the remainder of this book, a polynomial spline refers to the solution of the PLS (1.8) in the model space W2m [a, b]. A cubic spline is a special case of the polynomial spline with m = 2. Since it measures Rb the roughness of the function f , a (f (m) )2 dx is often referred to as a roughness penalty. It is obvious that there is no penalty for polynomials of order less than or equal to m. The smoothing parameter λ balances the trade-off between goodness-of-fit measured by the LS and roughness Rb of the estimate measured by a (f (m) )2 dx. Suppose that n ≥ m and a ≤ x1 < x2 < · · · < xn ≤ b. Then, for fixed 0 < λ < ∞, (1.8) has a unique minimizer fˆ and fˆ ∈ N S 2m (x1 , . . . , xn ) (Eubank 1988). This result indicates that even though we started with the infinite dimensional space W2m [a, b] as the model space for f , the solution to the PLS (1.8) belongs to a finite dimensional space. Specifically, the solution is a natural polynomial spline with knots at distinct design points. One approach to computing the polynomial spline estimate is to represent fˆ as a linear combination of a basis of N S 2m (x1 , . . . , xn ). Several basis constructions were provided in Section 3.3.3 of Eubank (1988). In particular, the R function smooth.spline implements this approach for the cubic spline using the B-spline basis. For example, the cubic spline fit in Figure 1.4 is derived by the following statements:
Introduction
7
> library(MASS); attach(mcycle) > smooth.spline(times, accel, all.knots=T) This book presents a different approach. Instead of basis functions, representers of reproducing kernel Hilbert spaces will be used to represent the spline estimate. This approach allows us to deal with many different spline models in a unified fashion. Details of this approach for polynomial splines will be presented in Sections 2.2 and 2.6. When λ = 0, there is no penalty, and the natural spline that interpolates observations is the unique minimizer. When λ = ∞, the unique minimizer is the mth order polynomial. As λ varies from ∞ to 0, we have a family of models ranging from the parametric polynomial model to interpolation. The value of λ decides how far f is allowed to depart from the polynomial model. Thus the choice of λ holds the key to the success of a spline estimate. We discuss how to choose λ based on data in Chapter 3.
1.3
Scope of This Book
Driven by many sophisticated applications and fueled by modern computing power, many flexible nonparametric and semiparametric modeling techniques have been developed to relax parametric assumptions and to exploit possible hidden structure. There are many different nonparametric methods. This book concentrates on one of them, smoothing spline. Existing books on this topic include Eubank (1988), Wahba (1990), Green and Silverman (1994), Eubank (1999), Gu (2002), and Ruppert, Wand and Carroll (2003). The goals of this book are to (a) make the advanced smoothing spline methodology based on reproducing kernel Hilbert spaces more accessible to practitioners and students; (b) provide software and examples so that the spline smoothing methods can be routinely used in practice; and (c) provide a comprehensive coverage of recently developed smoothing spline nonparametric/semiparametric linear/nonlinear fixed/mixed models. We concentrate on the methodology, implementation, software, and application. Theoretical results are stated without proofs. All methods will be demonstrated using real data sets and R functions. The polynomial spline in Section 1.2 concerns the functions defined on the domain [a, b]. In many applications, the domain of the regression function is not a continuous interval. Furthermore, the regression function may only be observed indirectly. Chapter 2 introduces gen-
8
Smoothing Splines: Methods and Applications
eral smoothing spline regression models with reproducing kernel Hilbert spaces on general domains as model spaces. Penalized LS estimation, Kimeldorf–Wahba representer theorem, computation, and the R function ssr will be covered. Explicit constructions of model spaces will be discussed in detail for some popular smoothing spline models including polynomial, periodic, thin-plate, spherical, and L-splines. Chapter 3 introduces methods for selecting the smoothing parameter and making inferences about the regression function. The impact of the smoothing parameter and basic concepts for model selection will be discussed and illustrated using an example. Connections between smoothing spline models and Bayes/mixed-effects models will be established. The unbiased risk, generalized cross-validation, and generalized maximum likelihood methods will be introduced for selecting the smoothing parameter. Bayesian and bootstrap confidence intervals will be introduced for the regression function and its components. The locally most powerful, generalized maximum likelihood and generalized cross-validation tests will also be introduced to test the hypothesis of a parametric model versus a nonparametric alternative. Analogous to multiple regression, Chapter 4 constructs models for multivariate regression functions based on smoothing spline analysis of variance (ANOVA) decompositions. The resulting models have hierarchical structures that facilitate model selection and interpretation. Smoothing spline ANOVA decompositions for tensor products of some commonly used smoothing spline models will be illustrated. Penalized LS estimation involving multiple smoothing parameters and componentwise Bayesian confidence intervals will be covered. Chapter 5 presents spline smoothing methods for heterogeneous and correlated observations. Presence of heterogeneity and correlation may lead to wrong choice of the smoothing parameters and erroneous inference. Penalized weighted LS will be used for estimation. Unbiased risk, generalized cross-validation, and generalized maximum likelihood methods will be extended for selecting the smoothing parameters. Variance and correlation structures will also be discussed. Analogous to generalized linear models, Chapter 6 introduces smoothing spline ANOVA models for observations generated from a particular distribution in the exponential family including binomial, Poisson, and gamma distributions. Penalized likelihood will be used for estimation, and methods for selecting the smoothing parameters will be discussed. Nonparametric estimation of variance and spectral density functions will be presented. Analogous to nonlinear regression, Chapter 7 introduces spline smoothing methods for nonparametric nonlinear regression models where some unknown functions are observed indirectly through nonlinear function-
Introduction
9
als. In addition to fitting theoretical and empirical nonlinear nonparametric regression models, methods in this chapter may also be used to deal with constraints on the nonparametric function such as positivity or monotonicity. Several algorithms based on Gauss–Newton, Newton– Raphson, extended Gauss–Newton and Gauss–Seidel methods will be presented for different situations. Computation and the R function nnr will be covered. Chapter 8 introduces semiparametric regression models that involve both parameters and nonparametric functions. The mean function may depend on the parameters and the nonparametric functions linearly or nonlinearly. The semiparametric regression models include many wellknown models such as the partial spline, varying coefficients, projection pursuit, single index, multiple index, functional linear, and shape invariant models as special cases. Estimation, inference, computation, and the R function snr will also be covered. Chapter 9 introduces semiparametric linear and nonlinear mixedeffects models. Smoothing spline ANOVA decompositions are extended for the construction of semiparametric mixed-effects models that parallel the classical mixed models. Estimation and inference methods, computation, and the R functions slm and snm will be covered as well.
1.4
The assist Package
The assist package was developed for fitting various smoothing spline models covered in this book. It contains five main functions, ssr, nnr, snr, slm, and snm for fitting various smoothing spline models. The function ssr fits smoothing spline regression models in Chapter 2, smoothing spline ANOVA models in Chapter 4, extended smoothing spline ANOVA models with heterogeneous and correlated observations in Chapter 5, generalized smoothing spline ANOVA models in Chapter 6, and semiparametric linear regression models in Chapter 8, Section 8.2. The function nnr fits nonparametric nonlinear regression models in Chapter 7. The function snr fits semiparametric nonlinear regression models in Chapter 8, Section 8.3. The functions slm and snm fit semiparametric linear and nonlinear mixed-effects models in Chapter 9. The assist package is available at http://cran.r-project.org Figure 1.5 shows how the functions in assist generalize some of the existing R functions for regression analysis.
10
Smoothing Splines: Methods and Applications
glm XXXX XXX
z gam X
: HH
X smooth.spline H XXX
3 Hj z H
: nnr lm Q JQ Q nls XXXX J s XXX J X z : nlme XX J XX z X ^ J X lme 1 XX XXX X z slm X
ssr snr
snm
FIGURE 1.5 Functions in assist (dashed boxes) and some existing R functions (solid boxes). An arrow represents an extension to a more general model. lm: linear models. glm: generalized linear models. smooth.spline: cubic spline models. nls: nonlinear regression models. lme: linear mixed-effects models. gam: generalized additive models. nlme: nonlinear mixed-effects models. ssr: smoothing spline regression models. nnr: nonparametric nonlinear regression models. snr: semiparametric nonlinear regression models. slm: semiparametric linear mixed-effects models. snm: semiparametric nonlinear mixed-effects models.
Chapter 2 Smoothing Spline Regression
2.1
Reproducing Kernel Hilbert Space
Polynomial splines concern functions defined on a continuous interval. This is the most common situation in practice. Nevertheless, many applications require modeling functions defined on domains other than a continuous interval. For example, for spatial data with measurements on latitude and longitude, the domain of the function is the Euclidean space R2 . Specific spline models were developed for different applications. It is desirable to develop methodology and software on a general platform such that special cases are dealt with in a unified fashion. Reproducing Kernel Hilbert Space (RKHS) provides such a general platform. This section provides a very brief review of RKHS. Throughout this book, important theoretical results are presented in italic without proofs. Details and proofs related to RKHS can be found in Aronszajn (1950), Wahba (1990), Gu (2002), and Berlinet and Thomas-Agnan (2004). A nonempty set E of elements f, g, h, . . . forms a linear space if there are two operations: (1) addition: a mapping (f, g) → f + g from E × E into E; and (2) multiplication: a mapping (α, f ) → αf from R × E into E, such that for any α, β ∈ R, the following conditions are satisfied: (a) f + g = g + f ; (b) (f + g) + h = f + (g + h); (c) for every f, g ∈ E, there exists h ∈ E such that f + h = g; (d) α(βf ) = (αβ)f ; (e) (α + β)f = αf + βf ; (f) α(f + g) = αf + αg; and (g) 1f = f . Property (c) implies that there exists a zero element, denoted as 0, such that f + 0 = f for all f ∈ E. A finite collection of elements f1 , . . . , fk in E is called linearly independent if the relation α1 f1 + · · ·+ αk fk = 0 holds only in the trivial case with α1 = · · · = αk = 0. An arbitrary collection of elements A is called linearly independent if every finite subcollection is linearly independent. Let A be a subset of a linear space E. Define spanA , {α1 f1 + · · · + αk fk : f1 , . . . , fk ∈ A, α1 , . . . , αk ∈ R, k = 1, 2, . . . }.
11
12
Smoothing Splines: Methods and Applications
A set B ⊂ E is called a basis of E if B is linearly independent and spanB = E. A nonnegative function || · || on a linear space E is called a norm if (a) ||f || = 0 if and only if f = 0; (b) ||αf || = |α|||f ||; and (c) ||f + g|| ≤ ||f || + ||g||. If the function || · || satisfies (b) and (c) only, then it is called a seminorm. A linear space with a norm is called a normed linear space. Let E be a linear space. A mapping (·, ·) : E × E → R is called an inner product in E if it satisfies (a) (f, g) = (g, f ); (b) (αf + βg, h) = α(f, h) + β(g, h); and (c) (f, f ) ≥ 0, and (f, pf ) = 0 if and only if f = 0. An inner product defines a norm: ||f || , (f, f ). A linear space with an inner product is called an inner product space. Let E be a normed linear space and fn be a sequence in E. The sequence fn is said to converge to f ∈ E if limn→∞ ||fn − f || = 0, and f is called the limit point. The sequence fn is called a Cauchy sequence if liml,n→∞ ||fl − fn || = 0. The space E is complete if every Cauchy sequence converges to an element in E. A complete inner product space is called a Hilbert space. A functional L on a Hilbert space H is a mapping from H to R. L is a linear functional if it satisfies L(αf + βg) = αLf + βLg. L is said to be continuous if limn→∞ Lfn = Lf when limn→∞ fn = f . L is said to be bounded if there exists a constant M such that |Lf | ≤ M ||f || for all f ∈ H. L is continuous if and only if L is bounded. For every fixed h ∈ H, Lh f , (h, f ) defines a continuous linear functional. Conversely, every continuous linear functional L can be represented as an inner product with a representer. Riesz representation theorem Let L be a continuous linear functional on a Hilbert space H. There exists a unique hL such that Lf = (hL , f ) for all f ∈ H. The element hL is called the representer of L. Let H be a Hilbert space of real-valued functions from X to R where X is an arbitrary set. For a fixed x ∈ X , the evaluational functional Lx : H → R is defined as Lx f , f (x). Note that the evaluational functional Lx maps a function to a real value while the function f maps a point x to a real value. Lx applies to all functions in H with a fixed x. Evaluational functionals are linear since Lx (αf + βg) = αf (x) + βg(x) = αLx f + βLx g. Definition A Hilbert space of real-valued functions H is an RKHS if every evaluational functional is continuous.
Smoothing Spline Regression
13
Let H be an RKHS. Then, for each x ∈ X , the evaluational functional Lx f = f (x) is continuous. By the Riesz representation theorem, there exists an element Rx in H such that Lx f = f (x) = (Rx , f ), where the dependence of the representer on x is expressed explicitly as Rx . Consider Rx (z) as a bivariate function of x and z and let R(x, z) , Rx (z). The bivariate function R(x, z) is called the reproducing kernel (RK) of an RKHS H. The term reproducing kernel comes from the fact that (Rx , Rz ) = R(x, z). It is easy to check that an RK is nonnegative definite. That is, R is symmetric R(x, z) = R(z, x), and for any α1 , . . . , αn ∈ R and x1 , . . . , xn ∈ X , n X
i,j=1
αi αj R(xi , xj ) ≥ 0.
Therefore, every RKHS has a unique RK that is nonnegative definite. Conversely, an RKHS can be constructed based on a nonnegative definite function. Moore–Aronszajn theorem For every nonnegative definite function R on X ×X , there exists a unique RKHS on X with R as its RK. The above results indicate that there exists an one-to-one correspondence between RKHS’s and nonnegative definite functions. For a finite dimensional space H with an orthonormal basis φ1 (x), . . . , φp (x), it is easy to see that p X R(x, z) , φi (x)φi (z) i=1
is the RK of H. The following definitions and results are useful for the construction and decomposition of model spaces. S is called a subspace of a Hilbert space H if S ⊂ H and αf + βg ∈ S for every α, β ∈ R and f, g ∈ S. A closed subspace S is a Hilbert space. The orthogonal complement of S is defined as S ⊥ , {f ∈ H : (f, g) = 0 for all g ∈ S}.
S ⊥ is a closed subspace of H. If S is a closed subspace of a Hilbert space H, then every element f ∈ H has a unique decomposition in the form f = g + h, where g ∈ S and h ∈ S ⊥ . Equivalently, H is decomposed
14
Smoothing Splines: Methods and Applications
into two subspaces H = S ⊕ S ⊥ . This decomposition is called a tensor sum decomposition, and elements g and h are called projections onto S and S ⊥ , respectively. Sometimes the notation H ⊖ S will be used to denote the subspaces S ⊥ . Tensor sum decomposition with more than two subspaces can be defined recursively. All closed subspaces of an RKHS are RKHS’s. If H = H0 ⊕ H1 and R, R0 , and R1 are RKs of H, H0 , and H1 respectively, then R = R0 + R1 . Suppose H is a Hilbert space and H = H0 ⊕ H1 . If H0 and H1 are RKHS’s with RKs R0 and R1 , respectively, then H is an RKHS with RK R = R0 + R1 .
2.2
Model Space for Polynomial Splines
Before introducing the general smoothing spline models, it is instructive to see how the polynomial splines introduced in Section 1.2 can be derived under the RKHS setup. Again, consider the regression model yi = f (xi ) + ǫi ,
i = 1, . . . , n,
(2.1)
where the domain of the function f is X = [a, b] and the model space for f is the Sobolev space W2m [a, b] defined in (1.4). The smoothing spline estimate fˆ is the solution to the PLS (1.8). Model space construction and decomposition of W2m [a, b] The Sobolev space W2m [a, b] is an RKHS with the inner product (f, g) =
m−1 X
f
(ν)
(a)g
(ν)
ν=0
(a) +
Z
b
f (m) g (m) dx.
(2.2)
a
Furthermore, W2m [a, b] = H0 ⊕ H1 , where H0 = span 1, (x − a), . . . , (x − a)m−1 /(m − 1)! , Z b (2.3) (ν) H1 = f : f (a) = 0, ν = 0, . . . , m − 1, (f (m) )2 dx < ∞ , a
are RKHS’s with corresponding RKs
m X (x − a)ν−1 (z − a)ν−1 , (ν − 1)! (ν − 1)! ν=1 Z b (z − u)m−1 (x − u)m−1 + + du. R1 (x, z) = (m − 1)! (m − 1)! a
R0 (x, z) =
(2.4)
Smoothing Spline Regression
15
The function (x)+ = max{x, 0}. Details about the foregoing construction can be found in Schumaker (2007). It is clear that H0 contains the polynomial of order m in the Taylor expansion. Note that the basis listed in (2.3), φν (x) = (x − a)ν−1 /(ν − 1)! for ν = 1, . . . , m, is an orthonormal basis of H0 . For any f ∈ H1 , it is easy to check that Z x Z x Z x1 Z xm−1 ′ f (x) = f (u)du = · · · = dx1 dx2 · · · f (m) (u)du a Z ax Z x1 Z xam−2 a (m) = dx1 dx2 · · · (xm−2 − u)f (u)du = · · · a a a Z x (x − u)m−1 (m) = f (u)du. (m − 1)! a Thus the subspace H1 contains the remainder term in the Taylor expansion. Denote P1 as the orthogonal projection operator onto H1 . From the definition of the inner product, the roughness penalty Z
a
b
(f (m) )2 dx = ||P1 f ||2 .
(2.5)
Rb Therefore, a (f (m) )2 dx measures the distance between f and the parametric polynomial space H0 . There is no penalty to functions in H0 . The PLS (1.8) can be rewritten as n
1X (yi − f (xi ))2 + λ||P1 f ||2 . n i=1
(2.6)
The solution to (2.6) will be given for the general case in Section 2.4. The above setup for polynomial splines suggests the following ingredients for the construction of a general smoothing spline model: 1. An RKHS H as the model space for f 2. A decomposition of the model space into two subspaces, H = H0 ⊕ H1 , where H0 consists of functions that are not penalized 3. A penalty ||P1 f ||2 Based on prior knowledge and purpose of the study, different choices can be made on the model space, its decomposition, and the penalty. These options make the spline smoothing method flexible and versatile. Choices of these options will be illustrated throughout the book.
16
2.3
Smoothing Splines: Methods and Applications
General Smoothing Spline Regression Models
A general smoothing spline regression (SSR) model assumes that yi = f (xi ) + ǫi ,
i = 1, . . . , n,
(2.7)
where yi are observations of the function f evaluated at design points xi , and ǫi are zero-mean independent random errors with a common variance σ 2 . To deal with different situations in a unified fashion, let the domain of the function f be an arbitrary set X , and the model space be an RKHS H on X with RK R(x, z). The choice of H depends on several factors including the domain X and prior knowledge about the function f . Suppose H can be decomposed into two subspaces, H = H0 ⊕ H1 ,
(2.8)
where H0 is a finite dimensional space with basis functions φ1 (x), . . . , φp (x), and H1 is an RKHS with RK R1 (x, z). H0 , often referred to as the null space, consists of functions that are not penalized. In addition to the construction for polynomial splines in Section 2.2, specific constructions of commonly used model spaces will be discussed in Sections 2.6–2.11. The decomposition (2.8) is equivalent to decomposing the function f = f0 + f1 ,
(2.9)
where f0 and f1 are projections onto H0 and H1 , respectively. The component f0 represents a linear regression model in space H0 , and the component f1 represents systematic variation not explained by f0 . Therefore, the magnitude of f1 can be used to check or test if the parametric model is appropriate. Projections f0 and f1 will be referred to as the “parametric” and “smooth” components, respectively. Sometimes observations of f are made indirectly through linear funcRb tionals. For example, f may be observed in the form a wi (x)f (x)dx where wi are known functions. Another example is that observations are taken on the derivatives f ′ (xi ). Therefore, it is useful to consider an even more general SSR model yi = Li f + ǫi ,
i = 1, . . . , n,
(2.10)
where Li are bounded linear functionals on H. Model (2.7) is a special case of (2.10) with Li being evaluational functionals at design points defined as Li f = f (xi ). By the definition of an RKHS, these evaluational functionals are bounded.
Smoothing Spline Regression
2.4
17
Penalized Least Squares Estimation
The estimation method will be presented for the general model (2.10). The smoothing spline estimate of f , fˆ, is the minimizer of the PLS n
1X (yi − Li f )2 + λ||P1 f ||2 , n i=1
(2.11)
where λ is a smoothing parameter controlling the balance between the goodness-of-fit measured by the least squares and departure from the null space H0 measured by ||P1 f ||2 . Functions in H0 are not penalized since ||P1 f ||2 = 0 when f ∈ H0 . Note that fˆ depends on λ even though the dependence is not expressed explicitly. Estimation procedures presented in this chapter assume that the λ has been fixed. The impact of the smoothing parameter and methods of selecting it will be discussed in Chapter 3. Since Li are bounded linear functionals, by the Riesz representation theorem, there exists a representer ηi ∈ H such that Li f = (ηi , f ). For a fixed x, consider Rx (z) , R(x, z) as a univariate function of z. Then, by properties of the reproducing kernel, we have ηi (x) = (ηi , Rx ) = Li Rx = Li(z) R(x, z),
(2.12)
where Li(z) indicates that Li is applied to what follows as a function of z. Equation (2.12) implies that the representer ηi can be obtained by applying the operator to the RK R. Let ξi = P1 ηi be the projection of ηi onto H1 . Since R(x, z) = R0 (x, z) + R1 (x, z), where R0 and R1 are RKs of H0 and H1 , respectively, and P1 is self-adjoint such that (P1 g, h) = (g, P1 h) for any g, h ∈ H, we have ξi (x) = (ξi , Rx ) = (P1 ηi , Rx ) = (ηi , P1 Rx ) = Li(z) R1 (x, z).
(2.13)
Equation (2.13) implies that the representer ξi can be obtained by applying the operator to the RK R1 . Furthermore, (ξi , ξj ) = Li(x) ξj (x) = Li(x) Lj(z) R1 (x, z). Denote p
T = {Li φν }ni=1 ν=1 , Σ = {Li(x) Lj(z) R1 (x, z)}ni,j=1 ,
(2.14)
where T is an n × p matrix, and Σ is an n × n matrix. For the special case of evaluational functionals Li f = f (xi ), we have ξi (x) = R1 (x, xi ), p T = {φν (xi )}ni=1 ν=1 , and Σ = {R1 (xi , xj )}ni,j=1 .
18
Smoothing Splines: Methods and Applications
Write the estimate fˆ as fˆ(x) =
p X
dν φν (x) +
ν=1
n X
ci ξi (x) + ρ,
i=1
where ρ ∈ H1 and (ρ, ξi ) = 0 for i = 1, . . . , n. Since ξi = P1 ηi , then ηi can be written as ηi = ζi + ξi , where ζi ∈ H0 . Therefore, Li ρ = (ηi , ρ) = (ζi , ρ) + (ξi , ρ) = 0.
(2.15)
Let y = (y1 , . . . , yn )T and fˆ = (L1 fˆ, . . . , Ln fˆ)T be the vectors of observations and fitted values, respectively. Let d = (d1 , . . . , dp )T and c = (c1 , . . . , cn )T . From (2.15), we have fˆ = T d + Σc. (2.16) P Furthermore, ||P1 f ||2 = || ni=1 ci ξi + ρ||2 = cT Σc + ||ρ||2 . Then the PLS (2.11) becomes 1 ||y − T d − Σc||2 + λcT Σc + ||ρ||2 . n
(2.17)
It is obvious that (2.17) is minimized when ρ = 0, which leads to the following result in Kimeldorf and Wahba (1971). Kimeldorf–Wahba representer theorem Suppose T is of full column rank. Then the PLS (2.11) has a unique minimizer given by fˆ(x) =
p X
ν=1
dν φν (x) +
n X
ci ξi (x).
(2.18)
i=1
The above theorem indicates that the smoothing spline estimate fˆ falls in a finite dimensional space. Equation (2.18) represents the smoothing spline estimate fˆ as a linear combination of basis of H0 and representers in H1 . Coefficients c and d need to be estimated from data. Based on (2.18), the PLS (2.17) reduces to 1 ||y − T d − Σc||2 + λcT Σc. n
(2.19)
Taking the first derivatives leads to the following equations for c and d: (Σ + nλI)Σc + ΣT d = Σy, T T Σc + T T T d = T T y,
(2.20)
Smoothing Spline Regression
19
where I is the identity matrix. Equations in (2.20) are equivalent to Σ + nλI ΣT Σc Σy = . TT TTT d TTy There may be multiple sets of solutions for c when Σ is singular. Nevertheless, all sets of solutions lead to the same estimate of the function fˆ (Gu 2002). Therefore, it is only necessary to derive one set of solutions. Consider the following equations (Σ + nλI)c + T d = y, T T c = 0.
(2.21)
It is easy to see that a set of solutions to (2.21) is also a set of solutions to (2.20). The solutions to (2.21) are d = (T T M −1 T )−1 T T M −1 y, c = M −1 {I − T (T T M −1 T )−1 T T M −1 }y,
(2.22)
where M = Σ + nλI. To compute the coefficients c and d, consider the QR decomposition of T , R T = (Q1 Q2 ) , 0 where Q1 , Q2 , and R are n × p, n × (n − p), and p × p matrices; Q = (Q1 Q2 ) is an orthogonal matrix; and R is upper triangular and invertible. Since T T c = RT QT1 c = 0, we have QT1 c = 0 and c = QQT c = (Q1 QT1 + Q2 QT2 )c = Q2 QT2 c. Multiplying the first equation in (2.21) by QT2 and using the fact that QT2 T = 0, we have QT2 M Q2 QT2 c = QT2 y. Therefore, c = Q2 (QT2 M Q2 )−1 QT2 y. (2.23) Multiplying the first equation in (2.21) by QT1 , we have Rd = QT1 (y − M c). Thus, d = R−1 QT1 (y − M c). (2.24) Equations (2.23) and (2.24) will be used to compute coefficients c and d. Based on (2.16), the first equation in (2.21) and equation (2.23), the fitted values fˆ = T d + Σc = y − nλc = H(λ)y, (2.25) where H(λ) , I − nλQ2 (QT2 M Q2 )−1 QT2
(2.26)
20
Smoothing Splines: Methods and Applications
is the so-called hat (influence, smoothing) matrix. The dependence of the hat matrix on the smoothing parameter λ is expressed explicitly. Note that equation (2.25) provides the fitted values while equation (2.18) can be used to compute estimates at any values of x.
2.5
The ssr Function
The R function ssr in the assist package is designed to fit SSR models. After deciding the model space and the penalty, the estimate fˆ is completely decided by y, T , and Σ. Therefore, these terms need to be specified in the ssr function. A typical call is ssr(formula, rk) where formula and rk are required arguments. Together they specify y, T , and Σ. Suppose the vector y and matrices T and Σ have been created in R. Then, formula lists y on the left-hand side, and T matrix on the right-hand side of an operator ~. The argument rk specifies the matrix Σ. In the most common situation where Li are evaluational functionals, the fitting can be greatly simplified since Li are decided by design points xi . There is no need to compute T and Σ matrices before calling the ssr function. Instead, they can be computed internally. Specifically, a direct approach to fit the standard SSR model (2.7) is to list y on the left-hand side and φ1 (x), . . . , φp (x) on the right-hand side of an operator ~ in the formula, and to specify a function for computing R1 in the rk argument. Functions for computing the RKs of some commonly used RKHS’s are available in the assist package. Users can easily write their own functions for computing RKs. There are several optional arguments for the ssr function, some of which will be discussed in the following chapters. In particular, methods for selecting the smoothing parameter λ will be discussed in Chapter 3. For simplicity, unless explicitly specified, all examples in this chapter use the default method that selects λ using the generalized cross-validation criterion. Bayesian and bootstrap confidence intervals for fitted functions are constructed based on the methods in Section 3.8. We now show how to fit polynomial splines to the motorcycle data. Consider the construction of polynomial splines in Section 2.2. For simplicity, we first consider the special cases of polynomial splines with m = 1 and m = 2, which are called linear and cubic splines, respectively. Denote x ∧ z = min{x, z} and x ∨ z = max{x, z}. Based on (2.3)
Smoothing Spline Regression
21
and (2.4), Table 2.1 lists bases for null spaces and RKs of linear and cubic splines for the special domain X = [0, b].
TABLE 2.1
Bases of null spaces and RKs of linear and cubic splines under the construction in Section 2.2 with X = [0, b] m Spline φν R0 R1 1 Linear 1 1 x∧z 2 Cubic 1, x 1 + xz (x ∧ z)2 {3(x ∨ z) − x ∧ z}/6
Functions linear2 and cubic2 in the assist package compute evaluations of R1 in Table 2.1 for linear and cubic splines, respectively. Functions for higher-order polynomial splines are also available. Note that the domain for functions linear2 and cubic2 is X = [0, b] for any fixed b > 0. The RK on the general domain X = [a, b] can be calculated by a translation, for example, cubic2(x-a). To fit a cubic spline to the motorcycle data, one may create matrices T and Σ first and then call the ssr function: > T <- cbind(1, times) > Sigma <- cubic2(times) > ssr(accel~T-1, rk=Sigma) The intercept is automatically included in the formula statement. Therefore, T-1 is used to exclude the intercept since it is already included in the T matrix. Since Li are evaluational functionals for the motorcycle example, the ssr function can be called directly: > ssr(accel~times, rk=cubic2(times)) The inputs for formula and rk can be modified for fitting polynomial splines of different orders. For example, the following statements fit linear, quintic (m = 3), and septic (m = 4) splines: > ssr(accel~1, rk=linear2(times)) > ssr(accel~times+I(times^2), rk=quintic2(times)) > ssr(accel~times+I(times^2)+I(times^3), rk=septic2(times)) The linear and cubic spline fits are shown in Figure 2.1.
22
Smoothing Splines: Methods and Applications Cubic spline fit
Linear spline fit
acceleration (g) −100 −50 0 50
o o oo
o ooo
oo o o oooo oooooooooo oooo oo o o o o o o o o o o ooooo oo o oo o oo ooo o oo oo oo o o o oo o oo o o oo o oo o o o o o ooo oo o o
0
10
o
o o
o
o oo
o o o o
oo
o oo o o o o o o o o o oo oo o o o oo o o o o o
oo o o ooooo oooooooooo oooo o oo o o o o o o o o o oooooo oo o oo o oo ooo o oo oo oo o o o oo o oo o o oo o oo o o o o o ooo oo o o
o
20 30 40 time (ms)
50
60
0
10
o o
o ooo
o o o o
o
oo o
o o
o oo o o o o o o o oo oo o o o oo o o o
o o
20 30 40 time (ms)
50
60
FIGURE 2.1 Motorcycle data, plots of observations (circles), the linear spline fit (left), and the cubic spline fit (right) as solid lines, and 95% Bayesian confidence intervals (shaded regions).
2.6
Another Construction for Polynomial Splines
One construction of RKHS for the polynomial spline was presented in Section 2.2. This section presents an alternative construction for W2m [0, 1] on the domain X = [0, 1]. In practice, without loss of generality, a continuous interval [a, b] can always be transformed into [0, 1]. Let kr (x) = Br (x)/r! be scaled Bernoulli polynomials where Br are R1 defined recursively by B0 (x) = 1, Br′ (x) = rBr−1 (x) and 0 Br (x)dx = 0 for r = 1, 2, . . . (Abramowitz and Stegun 1964). The first four scaled Bernoulli polynomials are k0 (x) = 1, k1 (x) = x − 0.5, 1 1 k12 (x) − , k2 (x) = 2 12 1 1 7 k4 (x) = k14 (x) − k12 (x) + . 24 2 240
(2.27)
Alternative model space construction and decomposition of W2m [0, 1] The Sobolev space W2m [0, 1] is an RKHS with the inner product Z 1 Z 1 m−1 X Z 1 (f, g) = f (ν) dx g (ν) dx + f (m) g (m) dx. (2.28) ν=0
0
0
0
Smoothing Spline Regression
23
Furthermore, W2m [0, 1] = H0 ⊕ H1 , where H0 = span k0 (x), k1 (x), . . . , km−1 (x) , Z 1 H1 = f : f (ν) dx = 0, ν = 0, . . . , m − 1, Z
0 1
0
(2.29)
(f (m) )2 dx < ∞ ,
are RKHS’s with corresponding RKs R0 (x, z) =
m−1 X
kν (x)kν (z),
ν=0
(2.30)
R1 (x, z) = km (x)km (z) + (−1)m−1 k2m (|x − z|).
The foregoing alternative construction was derived by Craven and Wahba (1979). Note that the inner product (2.28) is different from (2.2). Again, H0 contains polynomials, and the basis listed in (2.29), φν (x) = kν−1 (x) for ν = 1, . . . , m, is an orthonormal basis of H0 . Denote P1 as the orthogonal projection operator onto . From the definition R 1 H1(m) of the inner product, the roughness penalty 0 (f )2 dx = ||P1 f ||2 . Based on (2.29) and (2.30), Table 2.2 lists bases for the null spaces and RKs of linear and cubic splines under the alternative construction in this section.
TABLE 2.2 cubic m 1 2
splines Spline Linear Cubic
Bases of the null spaces and RKs for linear and under the construction in Section 2.6 with X = [0, 1] φν R0 R1 1 1 k1 (x)k1 (z) + k2 (|x − z|) 1, k1 (x) 1 + k1 (x)k1 (z) k2 (x)k2 (z) − k4 (|x − z|)
Functions linear and cubic in the assist package compute evaluations of R1 in Table 2.2 for linear and cubic splines respectively. Functions for higher-order polynomial splines are also available. Note that the domain under construction in this section is restricted to [0, 1]. Thus the scale option is needed when the domain is not [0, 1]. For example, the following statements fit linear and cubic splines to the motorcycle data:
24
Smoothing Splines: Methods and Applications
> ssr(accel~1, rk=linear(times), scale=T) > ssr(accel~times, rk=cubic(times), scale=T) The scale option scales the independent variable times into the interval [0, 1]. It is a good practice to scale a variable first before fitting. For example, the following statements lead to the same cubic spline fit: > x <- (times-min(times))/(max(times)-min(times)) > ssr(accel~x, rk=cubic(x))
2.7
Periodic Splines
Many natural phenomena follow a cyclic pattern. For example, many biochemical, physiological, or behavioral processes in living beings follow a daily cycle called circadian rhythm, and many Earth processes follow an annual cycle. In these cases the mean function f is known to be a smooth periodic function. Without loss of generality, assume that the domain of the function X = [0, 1] and f is a periodic function on [0, 1]. Since periodic functions can be regarded as functions defined on the unit circle, periodic splines are often referred to as splines on the circle. The model space for periodic spline of order m is W2m (per) = f : f (j) are absolutely continuous, f (j) (0) = f (j) (1), Z 1 j = 0, . . . , m − 1, (f (m) )2 dx < ∞ . (2.31) 0
Craven and Wahba (1979) derived the following construction. Model space construction and decomposition of W2m (per) The space W2m (per) is an RKHS with inner product (f, g) =
Z
0
1
Z f dx
0
1
Z gdx +
1
f (m) g (m) dx.
0
Furthermore, W2m (per) = H0 ⊕ H1 , where H0 = span 1 , Z 1 H1 = f ∈ W2m (per) : f dx = 0 , 0
(2.32)
Smoothing Spline Regression
25
are RKHS’s with corresponding RKs R0 (x, z) = 1, R1 (x, z) = (−1)m−1 k2m (|x − z|).
(2.33)
R1 Again, the roughness penalty 0 (f (m) )2 dx = ||P1 f ||2 . The function periodic in the assist library calculates R1 in (2.33). The order m is specified by the argument order. The default is a cubic periodic spline with order=2. We now illustrate how to fit a periodic spline using the Arosa data, which contain monthly mean ozone thickness (Dobson units) in Arosa, Switzerland, from 1926 to 1971. Suppose we want to investigate how ozone thickness changes over months in a year. It is reasonable to assume that the mean ozone thickness is a periodic function of month. Let thick be the dependent variable and x be the independent variable month scaled into the interval [0, 1]. The following statements fit a cubic periodic spline: > data(Arosa); Arosa$x <- (Arosa$month-0.5)/12 > ssr(thick~1, rk=periodic(x), data=Arosa)
300
thickness 350
400
The fit of the periodic spline is shown in Figure 2.2.
1
2
3
4
5
6 7 month
8
9
10 11 12
FIGURE 2.2 Arosa data, plot of observations (points), and the periodic spline fits (solid line). The shaded region represents 95% Bayesian confidence intervals.
26
2.8
Smoothing Splines: Methods and Applications
Thin-Plate Splines
Suppose f is a function of a multivariate independent variable x = (x1 , . . . , xd ) ∈ Rd , where Rd is the Euclidean d-space. Assume the regression model yi = f (xi ) + ǫi , i = 1, . . . , n, (2.34) where xi = (xi1 , . . . , xid ) and ǫi are zero-mean independent random errors with a common variance σ 2 . Define the model space for a thin-plate spline as d W2m (Rd ) = f : Jm (f ) < ∞ , (2.35) where
d (f ) = Jm
X
α1 +···+αd
m! α ! . . . αd ! =m 1
Z
∞
−∞
···
Z
∞
−∞
∂mf α1 d ∂x1 . . . ∂xα d
2 Y d
dxj .
j=1
(2.36) d Since Jm (f ) is invariant under a rotation of the coordinates, the thinplate spline is especially well suited for spatial data (Wahba 1990, Gu 2002). Define an inner product as Z ∞ Z ∞ X m! ··· (f, g) = α ! . . . αd ! −∞ −∞ α1 +···+αd =m 1 Y d ∂ mg ∂ mf dxj . (2.37) αd αd 1 1 ∂xα ∂xα 1 . . . ∂xd 1 . . . ∂xd j=1 Model space construction of W2m (Rd ) With the inner product (2.37), W2m (Rd ) is an RKHS if and only if 2m − d > 0. Details can be found in Duchon (1977) and Meinguet (1979). A thinplate spline estimate is the minimizer to the PLS n
1X d (yi − f (xi ))2 + λJm (f ) n i=1
(2.38)
d in W2m (Rd ). The null space H0 of the penalty functional Jm (f ) is the space spanned by polynomials in d variables of total degree up to m − 1.
Smoothing Spline Regression 27 d+m−1 Thus the dimension of the null space p = . For example, d when d = 2 and m = 2, 2 2 2 2 ) Z ∞ Z ∞ ( 2 2 ∂ f ∂ f ∂ f J22 (f ) = +2 + dx1 dx2 , 2 ∂x ∂x ∂x ∂x22 1 2 −∞ −∞ 1 and the null space is spanned by φ1 (x) = 1, φ2 (x) = x1 , and φ3 (x) = x2 . In general, denote φ1 , . . . , φp as the p polynomials of total degree up to m−1 that span H0 . Denote Em as the Green function for the m-iterated Laplacian Em (x, z) = E(||x − z||), where ||x − z|| is the Euclidean distance and ( d (−1) 2 +1+m |u|2m−d log |u|, d even, E(u) = |u|2m−d , d odd. p Let T = {φν (xi )}ni=1 ν=1 and K = {Em (xi , xj )}ni,j=1 . The bivariate function Em is not the RK of W2m (Rd ) since it is not nonnegative definite. Nevertheless, it is conditionally nonnegative definite in the sense that T T c = 0 implies that cT Kc ≥ 0. Referred to as a semi-kernel, the function Em is sufficient for the purpose of estimation. Assume that T is of full column rank. It can be shown that the unique minimizer of the PLS (2.38) is given by (Wahba 1990, Gu 2002)
fˆ(x) =
p X
ν=1
dν φν (x) +
n X
ci ξi (x),
(2.39)
i=1
where ξi (x) = Em (xi , x). Therefore, Em plays the same role as the RK R1 . The coefficients c and d are solutions to (K + nλI)c + T d = y, T T c = 0.
(2.40)
The above equations have the same form as those in (2.21). Therefore, computations in Section 2.4 carry over with Σ being replaced by K. The semi-kernel Em is calculated by the function tp.pseudo in the assist package. The order m is specified by the order argument with default as order=2. The USA climate data contain average winter temperatures in 1981 from 1214 stations in USA. To investigate how average winter temperature (temp) depends on geological locations (long and lat), we fit a thin-plate spline as follows: > attach(USAtemp) > ssr(temp~long+lat, rk=tp.pseudo(list(long,lat)))
28
Smoothing Splines: Methods and Applications 10 15 20 25 30 35 40 45 50 55 60 65 70
FIGURE 2.3 USA climate data, contour plot of the thin-plate spline fit. The contour plot of the fit is shown in Figure 2.3. A genuine RK for W2m (Rd ) is needed later in the computation of posterior variances in Chapter 3 and the construction of tensor product splines in Chapter 4. We now discuss briefly how to derive the genuine RK. Define inner product (f, g)0 =
J X
wj f (uj )g(uj ),
(2.41)
j=1
where uj are fixed points in Rd , and wj are fixed positive weights such P that Jj=1 wj = 1. Points uj and weights wj are selected in such a way that the matrix {(φν , φµ )0 }pν,µ=1 is nonsingular. Let φ˜ν , ν = 1, . . . , p, be an orthonormal basis derived from φν with P φ˜1 (x) = 1. Let P0 be the p projection operator onto H0 defined as P0 f = ν=1 (f, φν )0 φν . Then it can be shown that (Gu 2002) R0 (x, z) =
p X
φ˜ν (x)φ˜ν (z),
ν=1
R1 (x, z) = (I − P0(x) )(I − P0(z ) )E(||x − z||)
(2.42)
Smoothing Spline Regression
29
are RKs of H0 and H1 , W2m (Rd ) ⊖ H0 , where P0(x) and P0(z ) are projections applied to the arguments x and z, respectively. p Assume that T = {φν (xi )}ni=1 ν=1 is of full column rank. Let Σ = {R1 (xi , xj )}ni,j=1 . One relatively simple approach to compute φ˜ν and Σ is to let J = n, uj = xj , and wj = n−1 . It is easy to see that {(φν , φµ )0 }pν,µ=1 = n−1 T T T , which is nonsingular. Let R T = (Q1 Q2 ) 0 be the QR decomposition of T . Then √ (φ˜1 (x), . . . , φ˜p (x)) = n(φ1 (x), . . . , φp (x))R−1 and Σ = Q2 QT2 KQ2 QT2 . The function tp computes evaluations of R1 in (2.42) with J = n, uj = xj and wj = n−1 .
2.9
Spherical Splines
Spherical spline, also called spline on the sphere, is an extension of both the periodic spline defined on the unit circle and the thin-plate spline defined on R2 . Let the domain be X = S, where S is the unit sphere. Any point x on S can be represented as x = (θ, φ), where θ (0 ≤ θ ≤ 2π) is the longitude and φ (−π/2 ≤ φ ≤ π/2) is the latitude. Define R 2π R π m 2 2 f )2 cos φdφdθ, m even, 0 − π2 (∆ ( m−1 ) J(f ) = R 2π R π m−1 (∆ 2 f )2θ 2 + (∆ 2 f )2φ cos φdφdθ, m odd, 2 0 −π 2 cos φ
where the notation (g)z represents the partial derivative of g with respect to z, ∆f represents the surface Laplacian on the unit sphere defined as ∆f =
1 1 fθθ + (cos φfφ )φ . 2 cos φ cos φ
Consider the model space m W2 (S) = f :
Z f dx < ∞, J(f ) < ∞ . S
30
Smoothing Splines: Methods and Applications
Model space construction and decomposition of W2m (S) W2m (S) is an RKHS when m > 1. Furthermore, W2m (S) = H0 ⊕ H1 , where H0 = span{1}, Z m H1 = f ∈ W2 (S) : f dx = 0 , S
are RKHS’s with corresponding RKs
R0 (x, z) = 1, ∞ X 2i + 1 1 R1 (x, z) = Gi (cos γ(x, z)), 4π {i(i + 1)}m i=1 where γ(x, z) is the angle between x and z, and Gi are the Legendre polynomials. Details of the above construction can be found in Wahba (1981). The penalty ||P1 f ||2 = J(f ). The RK R1 is in the form of an infinite series, which is inconvenient to compute. Closed-form expressions are available only when m = 2 and m = 3. Wahba (1981) proposed replacing J by a topologically equivalent seminorm Q under which closed-form RKs can be derived. The function sphere in the assist package calculates R1 under the seminorm Q for 2 ≤ m ≤ 6. The argument order specifies m with default as order=2. The world climate data contain average winter temperatures in 1981 from 725 stations around the globe. To investigate how average winter temperature (temp) depends on geological locations (long and lat), we fit a spline on the sphere: > data(climate) > ssr(temp~1, rk=sphere(cbind(long,lat)), data=climate) The contour plot of the spherical spline fit is shown in Figure 2.4.
2.10
Partial Splines
A partial spline model assumes that yi = sTi β + Li f + ǫi ,
i = 1, . . . , n,
(2.43)
Smoothing Spline Regression
31
−30
−25
−35
−20 −5
−40
50
−15
0
5
−10
15
20
10
latitude 0
25
25
25 20
−50
15
10
5
−150
−100
−50
0 longitude
50
100
150
FIGURE 2.4 World climate data, contour plot of the spherical spline fit. where s is a q-dimensional vector of independent variables, β is a vector of parameters, Li are bounded linear functionals, and ǫi are zero-mean independent random errors with a common variance σ 2 . We assume that f ∈ H, where H is an RKHS on an arbitrary domain X . Model (2.43) contains two components: a parametric linear model and a nonparametric function f . The partial spline model is a special case of the semiparametric linear regression model discussed in Chapter 8. Suppose H = H0 ⊕ H1 , where H0 = span{φ1 , . . . , φp } and H1 is an RKHS with RK R1 . Denote P1 as the projection onto H1 . The function f and parameters β are estimated as minimizers to the following PLS: n
1X (yi − sTi β − Li f )2 + λ||P1 f ||2 . n i=1
(2.44)
p Let S = (s1 , . . . , sn )T , T = {Li φν }ni=1 ν=1 , X = (S T ), and Σ = n {Li(x) Lj(z) R1 (x, z)}i,j=1 . Assume that X is of full column rank. Following similar arguments as in Section 2.4, it can be shown that the PLS (2.44) has a unique minimizer, and the solution of f is given in (2.18).
32
Smoothing Splines: Methods and Applications
Therefore, the PLS (2.44) reduces to 1 ||y − Xα − Σc||2 + λcT Σc, n where α = (β T , dT )T . As in Section 2.4, we can solve α and c from the following equations: (Σ + nλI)c + Xα = y, X T c = 0.
(2.45)
The above equations have the same form as those in (2.21). Thus, computations in Section 2.4 carry over with T and d being replaced by X and α, respectively. The ssr function can be used to fit partial splines. When Li are evaluational functionals, the partial spline model (2.43) can be fitted by adding s variables at the right-hand side of the formula argument. When Li are not evaluational functionals, matrices X and Σ need to be created and supplied in the formula and rk arguments. One interesting application of the partial spline model is to fit a nonparametric regression model with potential change-points. A changepoint is defined as a discontinuity in the mean function or one of its derivatives. Note that the function g(x) = (x − t)k+ has a jump in its kth derivative at location t. Therefore, it can be used to model changepoints. Specifically, consider the following model yi =
J X j=1
k
βj (xi − tj )+j + f (xi ) + ǫi ,
i = 1, . . . , n,
(2.46)
where xi ∈ [a, b] are design points, tj ∈ [a, b] are change-points, f is a smooth function, and ǫi are zero-mean independent random errors with a common variance σ 2 . The mean function in model (2.46) has a jump at tj in its kj th derivative with magnitude βj . The choice of model space for f depends on the application. For example, the polynomial or periodic spline space may be used. When tj and kj are known, model k (2.46) is a special case of partial spline with q = J, sij = (xi − tj )+j , and T si = (si1 , . . . , siJ ) . We now use the geyser data and motorcycle data to illustrate changepoints detection using partial splines. For the geyser data, Figure 1.3(b) indicates that there may be a jump in the mean function between 2.5 and 3.5 minutes. Therefore, we consider the model yi = β(xi − t)0+ + f (xi ) + ǫi ,
i = 1, . . . , n,
(2.47)
where xi are the duration variable scaled into [0, 1], t is a change-point, and (x − t)0+ = 0 when x ≤ t and 1 otherwise. We assume that f ∈
Smoothing Spline Regression
33
W22 [0, 1]. For a fixed t, say t = 0.397, we can fit the partial spline as follows: > attach(faithful) > x <- (eruptions-min(eruptions))/diff(range(eruptions)) > ssr(waiting~x+(x>.397), rk=cubic(x)) The partial spline fit is shown in Figure 2.5(a). No trend is shown in the residual plot in Figure 2.5(b). The change-point is fixed at t = 0.397 in the above fit. Often it is unknown in practice. To search for the location of the change-point t, we compute AIC and GCV (generalized cross-validation) criteria on a grid points between 0.25 and 0.55: > aic <> for (t fit
gcv <- NULL in seq(.25,.55,by=.001)) { ssr(waiting~x+(x>t), rk=cubic(x)) c(aic, length(x)*log(sum(fit$resi**2))+2*fit$df) c(gcv, fit$rkpk.obj$score)
The vector fit$resi contains residuals. The value fit$df represents the degrees of freedom (trH(λ)) defined later in Chapter 3, Section 3.2. The GCV criterion is defined in (3.24). Figure 2.5(c) shows the AIC and GCV scores scaled into [0, 1]. The scaled scores are identical and reach the minimum in the same region with the middle point at t = 0.397.
15
(b)
1.5
2.5 3.5 4.5 duration (min)
scaled AIC/GCV 0.4 0.8
(c)
0.0
50
waiting (min) 60 70 80
residuals (min) −10 −5 0 5 10
90
(a)
1.5
2.5 3.5 4.5 duration (min)
0.25
0.35
0.45
0.55
t
FIGURE 2.5 Geyser data, plots of (a) observations (points), the partial spline fit (solid line) with 95% Bayesian confidence intervals (shaded region), (b) residuals (points) from the partial spline fit and the cubic spline fit (solid line) to the residuals, and (c) the AIC (dashed line) and GCV scores (solid line) scaled into [0, 1].
34
Smoothing Splines: Methods and Applications
For the motorcycle data, it is apparent that the mean curve is flat on the left and there is a sharp corner around 15 ms. The linear and cubic splines fit this region with round corners (Figure 2.1). The sharp corner suggests that there may be a change-point in the first derivative of the mean function. Therefore, we consider the model yi = β(xi − t)+ + f (xi ) + ǫi ,
i = 1, . . . , n,
(2.48)
where x is the time variable scaled into [0, 1] and t is the change-point in the first derivative. We assume that f ∈ W22 [0, 1]. Again, we use the AIC and GCV criteria to search for the location of the change-point t. Figure 2.6(b) shows the scaled AIC and GCV scores. They both reach the minimum at t = 0.214. For the fixed t = 0.214, the model (2.48) can be fitted as follows: > t <- .214; s <- (x-t)*(x>t) > ssr(accel~x+s, rk=cubic(x)) The partial spline fit is shown in Figure 2.6(a). The sharp corner around 15 ms is preserved. Chapter 3 contains more analysis of potential changepoints for the motorcycle data.
(a)
(b) o o
o o ooo o oo o o oo o o o oo o o o o oo o o o o o oo o ooo o ooo oo o o o o o o o oo o o oo oo o oo o o
0
10
20 30 40 time (ms)
0.0
oooo ooooooooo oooo oo o o o o o ooooo o oo o ooo o oo o o oo oo o o oo o oo o o o o o ooo oo o o
scaled AIC/GCV 0.4 0.8
acceleration (g) −100 −50 0 50
o
50
60
0.20
0.22
0.24 t
FIGURE 2.6 Motorcycle data, plots of (a) observations (circles), the partial spline fit (solid line) with 95% Bayesian confidence intervals (shaded region), and (b) the AIC (dashed line) and GCV (solid line) scores scaled into [0, 1].
Often, in practice, there is enough knowledge to model some components in the regression function parametrically. For other uncertain
Smoothing Spline Regression
35
components, it may be desirable to leave them unspecified. Combining parametric and nonparametric components, the partial spline (semiparametric) models are well suited to these situations. As an illustration, consider the Arosa data. We have investigated how ozone thickness changes over months in a year by fitting a periodic spline (Figure 2.2). Suppose now we want to investigate how ozone thickness changes over time. That is, we need to consider both month (seasonal) effect and year (long-term) effect. Let x1 and x2 be the month and year variables scaled into the interval [0, 1]. For the purpose of illustration, suppose the seasonal trend can be well approximated by a simple sinusoidal function. The form of long-term trend will be left unspecified. Therefore, we consider the following partial spline model yi = β1 + β2 sin(2πxi1 ) + β3 cos(2πxi1 ) + f (xi2 ) + ǫi , i = 1, . . . , 518,
(2.49)
where yi represents the average ozone thickness in month xi1 of year xi2 , and f ∈ W22 [0, 1] ⊖ {1}. Note that the constant functions are removed from the model space for f such that f is identifiable with the constant β1 . The partial spline model (2.49) can be fitted as follows: > x1 <- (Arosa$month-0.5)/12; x2 <- (Arosa$year-1)/45 > ssr(thick~sin(2*pi*x1)+cos(2*pi*x1)+x2, rk=cubic(x2)) Estimates of the main effect of month, βˆ2 sin(2πx1 ) + βˆ3 cos(2πx1 ), and the main effect of year, fˆ(x2 ), are shown in Figure 2.7. The simple sinusoidal model for the seasonal trend is too restrictive. More general models for the Arosa data can be found in Chapter 4, Section 4.9.2. Functional data are observations in a form of functions. The most common forms of functional data are curves defined on a continuous interval and surfaces defined on R2 . A functional linear model (FLM) is a linear model that involves functional data as (i) the independent variable, (ii) the dependent variable, or (iii) both independent and dependent variables. Many methods have been developed for fitting FLMs where functional data are curves or surfaces (Ramsay and Silverman 2005). We will use the Canadian weather data to illustrate how to fit FLMs using methods in this book. An FLM corresponding to situation (i) is discussed in this section. FLMs corresponding to situations (ii) and (iii) will be introduced in Chapter 4, Sections 4.9.3, and Chapter 8, Section 8.4.1, respectively. Note that methods illustrated in these sections for functional data apply to general functions defined on arbitrary domains. In particular, they can be used to fit surfaces defined on R2 . Now consider the Canadian weather data. To investigate the relationship between total annual precipitation and the temperature function,
36
Smoothing Splines: Methods and Applications
Main effect of month
o
o
−40
o
ooo
thickness 0
o
20
o
o
oo oo o o
o
o
o o
o
o
o o ooo o o
o
oo o
o
o oo
o
o
o o o o o o
o o
−20
o
oo
o o
o
−40
thickness −20 0 20
40
40
Main effect of year o
o
o o
1
2
3
4
5
6 7 8 month
9
o
11
1926
1935
1944 1953 year
1962
1971
FIGURE 2.7 Arosa data, plots of estimated main effects with 95% Bayesian confidence intervals. A circle in the left panel represents the average thickness for a particular month minus the overall mean. A circle in the right panel represents the average thickness for a particular year minus the overall mean.
consider the following FLM y i = β1 +
Z
1
wi (x)f (x)dx + ǫi ,
i = 1, . . . , 35,
(2.50)
0
where yi is the logarithm of total annual precipitation at station i, β1 is a constant parameter, x is the variable month transformed into [0, 1], wi (x) is the temperature function at station i, f (x) is an unknown weight function, and ǫi are random errors. Model (2.50) is the same as model (15.2) in Ramsay and Silverman (2005). It is an example when the independent variable is a curve. The goal is to estimate the weight function f . It is reasonable to assume that f is a smooth periodic function. Specifically, we model f using cubic periodic spline space W22 (per). Write f (x) = β2 + f1 (x) where f1 ∈ W22 (per) ⊖ {1}. Then model (2.50) can be rewritten as a partial spline model y i = β1 + β2
Z
1
wi (x)dx + 0
, sTi β + Li f1 + ǫi ,
Z
1
wi (x)f1 (x)dx + ǫi
0
(2.51)
R1 R1 where si = (1, 0 wi (x)dx)T , β = (β1 , β2 )T , and Li f1 = 0 wi (x)f1 (x)dx. Assume that wi are square integrable. Then Li are bounded linear functionals. Let R1 be the RK of W22 (per) ⊖ {1}. From (2.14), the (i, j)th
Smoothing Spline Regression
37
element of Σ equals
Li(x) Lj(z) R1 (x, z) =
Z
1 0
Z
1
wi (s)wj (t)R1 (s, t)dsdt 0 12 X 12 X
≈
1 144
=
1 T w R1 (x, x)w j , 144 i
wi (xk )wj (xl )R1 (xk , xl )
k=1 l=1
(2.52)
where xk represents the middle point of month k, x = (x1 , . . . , x12 )T , R1 (x, x) = {R1 (xk , xl )}12 k,l=1 , wi (xk ) is the temperature of month k at station i, and wi = (wi (x1 ), . . . , wi (x12 ))T . The rectangle rule is used to approximate the integrals. More accurate approximations may be used. Let W = (w 1 , . . . , w35 ). Then Σ ≈ W T R1 (x, x)W/144. The following statements fit the partial spline model (2.51):
> > > > > > >
library(fda); attach(CanadianWeather) y <- log(apply(monthlyPrecip,2,sum)) W <- monthlyTemp s <- apply(W,2,mean) x <- seq(0.5,11.5,1)/12 Sigma <- t(W)%*%periodic(x)%*%W/144 canada.fit1 <- ssr(y~s, rk=Sigma, spar=‘‘m’’)
P12 where the vector s contains elements j=1 wi (xj )/12, which are approxR1 imations of 0 wi (x)dx. The generalized maximum likelihood (GML) method in Chapter 3, Section 3.6, is used to select the smoothing parameter since the GCV estimate is too small due to small sample size. This is accomplished by setting the option spar=‘‘m’’. From equation (2.18), the estimate of the weight function is
fˆ(x) = d2 +
35 X i=1
ci Li(z) R1 (x, z),
where d2 is an estimate of β2 . To compute fˆ at a set of points, say,
38
Smoothing Splines: Methods and Applications
x0 = (x01 , . . . , x0n0 )T , we have fˆ(x0 ) = d2 1n0 +
35 X
ci
= d2 1n0 +
35 X
ci
R1 (x0 , x)wi (x)dx
12 X
R1 (x0 , xj )wi (xj )/12
j=1
i=1
35 X
1
0
i=1
≈ d2 1n0 +
Z
ci R1 (x0 , x)w i /12
i=1
= d2 1n0 + Sc, where fˆ(x0 ) = (fˆ(x01 ), . . . , fˆ(x0n0 ))T , 1m represents a m-vector of all ones, R1 (x0 , x) = (R1 (x01 , x), . . . , R1 (x0n0 , x))T , R1 (x0 , x) = T 0 12 {R1 (x0i , xj )}ni=1 j=1 , S = R1 (x0 , x)W/12, and c = (c1 , . . . , cn ) . We compute fˆ(x0 ) at 50 equally spaced points in [0, 1] as follows: > S <- periodic(seq(0,1,len=50),x)%*%W/12 > fhat <- canada.fit1$coef$d[2]+S%*%canada.fit1$coef$c
−0.4
weight function f(x) 0.0 0.2 0.4 0.6
Figure 2.8 displays the estimated weight function and 95% bootstrap confidence intervals. The shape of the weight function is similar to that in Figure 15.5 of Ramsay and Silverman (2005). Note that monthly data are used in this book while daily data were used in Ramsay and Silverman (2005).
J
F M A M
J J A month
S O N D
FIGURE 2.8 Canadian weather data, plot of the estimated weight function (solid line), and 95% bootstrap confidence intervals (shaded region).
Smoothing Spline Regression
2.11 2.11.1
39
L-splines
Motivation
In the construction of a smoothing spline model, one needs to decide the penalty functional or, equivalently, the null space H0 consisting of functions that are not penalized. The squared bias of the spline estimate satisfies the following inequality (Wahba, 1990, p. 59) n
1X {E(Li fˆ) − Li f }2 ≤ λ||P1 f ||2 . n i=1 That is, the squared bias is bounded by the distance of f to the null space H0 . Therefore, selecting a penalty such that ||P1 f ||2 is small can lead to low bias in the spline estimate of the function. This is equivalent to selecting a null space H0 such that it is close to the true function. Ideally, one wants to choose H0 such that ||P1 f ||2 = 0, that is f ∈ H0 . However, it is usually difficult, if not impossible, to specify such a parametric space in practice. Nevertheless, often there is prior information suggesting that f can be well approximated by a parametric model. That is, f is close to, but not necessarily in, the space H0 . Heckman and Ramsay (2000) called such a space a favored parametric model. L-splines allow us to incorporate this kind of indefinite information. L-splines can also be used to check or test parametric models (Chapter 3). Consider functions on the domain X = [a, b]. Let L be the linear differential operator L = Dm +
m−1 X
ωj (x)Dj ,
(2.53)
j=0
where m ≥ 1, Dj denotes the jth derivative operator, and ωi are continuous real-valued functions. The minimizer of the following PLS n
1X (yi − Li f )2 + λ n i=1
Z
b
(Lf )2 dx
a
is called an L-spline. Note that the penalty is in general different from that of a polynomial or periodic spline. To utilize the general estimation procedure developed in Section 2.4, we need to construct an RKHS such Rb that the penalty a (Lf )2 dx = ||P1 f ||2 with an appropriately defined inner product.
40
Smoothing Splines: Methods and Applications
Suppose f ∈ W2m [a, b]. Then Lf exists and is square integrable. Let H0 = {f : Lf = 0} be the kernel of L. Based on results from differential equations (Coddington 1961), there exist real-valued functions u1 , . . . , um ∈ W2m [a, b] such that u1 , . . . , um form a basis of H0 . Furthermore, the Wronskian matrix associated with u1 , . . . , um , u1 (x) u2 (x) · · · um (x) u′1 (x) u′2 (x) · · · u′m (x) W (x) = .. .. .. , . . . (m−1) (m−1) (m−1) u1 (x) u2 (x) · · · um (x)
is invertible for all x. Define inner product (f, g) =
m−1 X
f (ν) (a)g (ν) (a) +
Z
b
(Lf )(Lg)dx.
(2.54)
a
ν=0
Equation (2.54) defines a proper inner product since f (ν) (a) = 0, ν = 0, . . . , m−1, and Lf = 0 leads to f = 0. Denote u(x) = (u1 (x), . . . , um (x))T . Let u∗ (x) = (u∗1 (x), . . . , u∗m (x))T be the last column of W −1 (x) and ( uT (x)u∗ (s), s ≤ x, G(x, s) = 0, s > x, be the Green function associated with L. Model space construction and decomposition of W2m [a, b] The space W2m [a, b] is an RKHS with inner product (2.54). Furthermore, W2m [a, b] = H0 ⊕ H1 , where H0 = span{u1 , . . . , um }, H1 = f ∈ W2m [a, b] : f (ν) (a) = 0, ν = 0, . . . , m − 1 ,
(2.55)
are RKHS’s with corresponding RKs
R0 (x, z) = uT (x){W T (a)W (a)}−1 u(z), Z b R1 (x, z) = G(x, s)G(z, s)ds.
(2.56)
a
The above construction of an L-spline is based on a given differential operator L. In practice, rather than a differential operator, prior knowledge may be in a form of basis functions for the null space H0 .
Smoothing Spline Regression
41
Specifically, suppose prior knowledge suggests that f can be well approximated by a parametric space spanned by a basis u1 , . . . , um . Then one can solve the following equations to derive coefficients ωj in (2.53): (Luν )(x) = u(m) ν (x) +
m−1 X
ωj (x)u(j) ν (x) = 0,
ν = 1, . . . , m.
j=0
Let W (x) be the Wronskian matrix, ω(x) = (ω0 (x), . . . , ωm−1 (x))T , and (m) (m) u(m) (x) = (u1 (x), . . . , um (x))T . Then the above equations can be written in a matrix form: W T (x)ω(x) = −u(m) (x). Assume that W (x) is invertible. Then ω(x) = −W −T (x)u(m) (x). It is clear that the polynomial spline is a special case with L = Dm . Some special L-splines are discussed in the following subsections. More details can be found in Schumaker (2007), Wahba (1990), Dalzell and Ramsay (1993), Heckman (1997), Gu (2002), and Ramsay and Silverman (2005).
2.11.2
Exponential Spline
Assume that f ∈ W22 [a, b]. Suppose prior knowledge suggests that f can be well approximated by a linear combination of 1 and exp(−γx) for a fixed γ 6= 0. Consider the parametric model space H0 = span{1, exp(−γx)}. It is easy to see that H0 is the kernel of the differential operator L = D2 + γD. Nevertheless, we derive this operator following the procedure discussed in Section 2.11.1. Let u1 (x) = 1 and u2 (x) = exp(−γx). The Wronskian matrix 1 exp(−γx) W (x) = . 0 −γ exp(−γx) Then
ω0 (x) ω1 (x)
=−
1 1 γ
− γ1
0 exp(γx)
0 γ 2 exp(−γx)
=
0 . γ
Thus we have L = D2 + γD. Since T
{W (a) W (a)}
−1
=
− γ12 exp(γa) 1 + γ12 1 − γ 2 exp(γa) γ12 exp(2γa)
,
42
Smoothing Splines: Methods and Applications
the RK of H0
1 1 1 − 2 exp(−γx∗ ) − 2 exp(−γz ∗ ) γ2 γ γ 1 + 2 exp{−γ(x∗ + z ∗ )}, γ
R0 (x, z) = 1 +
(2.57)
where x∗ = x − a and z ∗ = z − a. The Green function ( 1 [1 − exp{−γ(x − s)}], s ≤ x, G(x, s) = γ 0, s > x. Thus the RK of H1 R1 (x, z) Z b = G(x, s)G(z, s)ds a
=
Z
0
x∗ ∧z ∗
1 [1 − exp{−γ(x∗ − s∗ )}] [1 − exp{−γ(z ∗ − s∗ )}] ds∗ γ2
1 = 3 γ(x∗ ∧ z ∗ ) + exp(−γx∗ ) + exp(−γz ∗ ) γ − exp{γ(x∗ ∧ z ∗ − x∗ )} − exp{γ(x∗ ∧ z ∗ − z ∗ )} 1 1 − exp{−γ(x∗ + z ∗ )} + exp[γ{2(x∗ ∧ z ∗ ) − x∗ − z ∗ }] . (2.58) 2 2 Evaluations of the RK R1 for some simple L-splines can be calculated using the lspline function in the assist package. The argument type specifies the type of an L-spline. The option type=‘‘exp’’ computes R1 in (2.58) for the special case when a = 0 and γ = 1. For general a and γ 6= 0, the RK R1 can be computed using a simple transformation x ˘ = γ(x − a). The weight loss data contain weight measurements of a male obese patient since the start of a weight rehabilitation program. We now illustrate how to fit an exponential spline to the weight loss data. Observations are shown in Figure 2.9(a). Let y = Weight and x = Days. We first fit a nonlinear regression model as in Venables and Ripley (2002): yi = β1 + β2 exp(−β3 xi ) + ǫi ,
i = 1, . . . , 51.
The following statements fit model (2.59): > library(MASS); attach(wtloss) > y <- Weight; x <- Days > weight.nls <- nls(y~b1+b2*exp(-b3*x), start=list(b1=81,b2=102,b3=.005))
(2.59)
Smoothing Spline Regression
43
The fit is shown in Figure 2.9(a).
Weight (kg) 130 150 170
o
(b)
1.5
(a) o o o o
o oo
oo oo o
−0.5
o o
o ooo oo o oo oo oo oo o oo
110
o
0
50
100 150 Days
o ooo o o oo
200
250
−1.5
o oo oo o
0.5
o o
0
50
100 150 Days
200
250
FIGURE 2.9 Weight loss data, plots of (a) observations (circles), the nonlinear regression fit (dotted line), the cubic spline fit (dashed line), and the exponential spline fit (solid line); and (b) the cubic spline fit and 95% Bayesian confidence intervals minus the nonlinear regression fit as dashed lines, and the exponential spline fit and 95% Bayesian confidence intervals minus the nonlinear regression fit as solid lines.
Next we consider the nonparametric regression model (1.1). It is reasonable to assume that the regression function can be well approximated by H0 = span{1, exp(−γx)}, where γ = βˆ3 = 0.0048 is the LS estimate of β3 in the model (2.59). We now fit the exponential spline: > r <- coef(weight.nls)[3] > ssr(y~exp(-r*x), rk=lspline(r*x,type=‘‘exp’’)) The exponential spline fit in Figure 2.9(a) is essentially the same as that from nonlinear regression model. The parameter β3 is fixed as βˆ3 in the above construction of the exponential spline. One may treat β3 as a parameter and estimate it using the GCV criterion as in Gu (2002): > gcv.fun <- function(r) ssr(y~exp(-r*x), rk=lspline(r*x,type=‘‘exp’’))$rkpk.obj$score > nlm(gcv.fun,.001)$estimate 0.004884513 For comparison, the cubic spline fit is also shown in Figure 2.9(a). To look at the difference between cubic and exponential splines more closely, we plot their fits and 95% Bayesian confidence intervals minus
44
Smoothing Splines: Methods and Applications
the fit from nonlinear regression in Figure 2.9(b). It is clear that the confidence intervals for the exponential spline are narrower. Figure 2.9 is essentially the same as Figure 4.3 in Gu (2002). A different approach was used to fit the exponential spline in Gu (2002): it is shown that fitting the exponential spline is equivalent to fitting a cubic spline to the transformed variable x ˜ = 1 − exp(−γx). Thus the following statements lead to the same fit to the exponential spline: > tx <- 1-exp(-r*x) > ssr(y~tx, rk=cubic2(tx))
2.11.3
Logistic Spline
Assume that f ∈ W22 [a, b]. Suppose prior knowledge suggests that f can be well approximated by a logistic model H0 = span{1/(1+δ exp(−γx))} for some fixed δ > 0 and γ > 0. It is easy to see that H0 is the kernel of the differential operator L=D−
δγ exp(−γx) . 1 + δ exp(−γx)
The Wronskian is an 1 × 1 matrix W (x) = {1 + δ exp(−γx)}−1 . Since {W T (a)W (a)}−1 = {1 + δ exp(−γa)}2 , then the RK of H0 R0 (x, z) =
{1 + δ exp(−γa)}2 . {1 + δ exp(−γx)}{1 + δ exp(−γz)}
(2.60)
The Green function 1 + δ exp(−γs) , s ≤ x, G(x, s) = 1 + δ exp(−γx) 0, s > x.
Thus the RK of H1
R1 (x, z) = {1 + δ exp(−γx)}−1 {1 + δ exp(−γz)}−1 x ∧ z − a + 2δγ −1 [exp(−γa) − exp{−γ(x ∧ z)}] +δ 2 (2γ)−1 [exp(−2γa) − exp{−2γ(x ∧ z)}] . (2.61)
The paramecium caudatum data consist of growth of paramecium caudatum population in the medium of Osterhout. We now illustrate how to fit a logistic spline to the paramecium caudatum data. Observations are shown in Figure 2.10. Let y = density and x = days. We first fit the following logistic growth model yi =
β1 + ǫi , 1 + β2 exp(−β3 x)
i = 1, . . . , 25,
(2.62)
Smoothing Spline Regression
45
using the statements > data(paramecium); attach(paramecium) > para.nls <- nls(density~b1/(1 + b2*exp(-b3*day)), start=list(b1=202,b2=164,b3=0.74)) Initial values are the estimates in Neal (2004). The fit is shown in Figure 2.10.
o
200
o o
o
o
o o o
o
o
o o
o
o o
o
density 100 150
o
o
o
50
o
o
0
o
o
o
o
0
5
10
15
20
25
day
FIGURE 2.10 Paramecium caudatum data, observations (circles), the nonlinear regression fit (dotted line), the cubic spline fit (dashed line), and the logistic spline fit (solid line).
Now we consider the nonparametric regression model (2.7). It is reasonable to assume that the regression function can be well approximated by H0 = span{1/(1 + δ exp(−γx)}, where δ = βˆ2 = 705.9496 and γ = βˆ3 = 0.9319 are the LS estimates in model (2.62). The option type=‘‘logit’’ in the lspline function computes R1 in (2.61) for the special case when a = 0, δ = 1, and γ = 1. It cannot be adapted to compute R1 for the general situation. We take this opportunity to show how to write a function for computing an RK. > logit.rk tmp1
<- function(x,a,d,r) { x%o%rep(1,length(x)) (tmp1+t(tmp1)-abs(tmp1-t(tmp1)))/2 exp(-r*a)-exp(-r*tmp2) exp(-2*r*a)-exp(-2*r*tmp2)
46
Smoothing Splines: Methods and Applications
tmp5 <- 1/((1+d*exp(-r*x))%o%(1+d*exp(-r*x))) (tmp2-a+2*d*tmp3/r+d**2*tmp4/(2*r))*tmp5 } > bh <- coef(para.nls) > ssr(density~I(1/(1+bh[2]*exp(-bh[3]*day)))-1, rk=logit.rk(day,0,bh[2],bh[3]),spar=‘‘m’’) The function logit.rk computes R1 in (2.61). Since the sample size is small, the GML method is used to select the smoothing parameter. The logistic spline fit is shown in Figure 2.10. The fit is essentially the same as that from the logistic growth model (2.62). For comparison, the cubic spline fit is also shown in Figure 2.10. The logistic spline fit smooths out oscillations after 10 days while the cubic spline fit preserves them. To include the constant function in H0 , one may consider the operator δγ exp(−γx) L=D D− . 1 + δ exp(−γx) Details of this situation can be found in Gu (2002).
2.11.4
Linear-Periodic Spline
Assume that f ∈ W24 [a, b]. Suppose prior knowledge suggests that f can be well approximated by a parametric model H0 = span{1, x, cos x, sin x}. It is easy to check that H0 is the kernel of the differential operator L = D4 + D2 .
The Wronskian matrix and its inverse are, respectively, 1 x cos x sin x 0 1 − sin x cos x W (x) = 0 0 − cos x − sin x 0 0 sin x − cos x and
1 −x 1 −x 0 1 0 1 . W −1 (x) = 0 0 − cos x sin x 0 0 − sin x − cos x For simplicity, suppose a = 0. Then 2 0 −1 0 0 2 0 −1 {W T (0)W (0)}−1 = −1 0 1 0 . 0 −1 0 1
Smoothing Spline Regression
47
Therefore, the RK of H0 R0 (x, z) = 2 − cos x − cos z + 2xz − x sin z − z sin x + cos x cos z + sin x sin z.
(2.63)
The Green function ( x − s − sin(x − s), s ≤ x, G(x, s) = 0, s > x. Thus the RK of H1 1 1 R1 (x, z) = − (x ∧ z)3 + xz(x ∧ z) − |x − z| − sin x − sin z 6 2 1 + x cos z + z cos x + (x ∧ z) cos(z − x) 2 5 1 + sin |x − z| − sin(x + z). (2.64) 4 4 The option type=‘‘linSinCos’’ in the lspline function computes R1 in (2.64) for the special case when a = 0. The translation x−a may be used when a 6= 0. When the null space H0 = span{1, x, cos τ x, sin τ x} for a fixed τ 6= 0, the corresponding differential operator is L = D4 + τ 2 D2 . The linear-periodic spline with a general τ can be fitted using the transformation x ˜ = τ x. The melanoma data contain numbers of melanoma cases per 100,000 in Connecticut from 1936 to 1972. We now illustrate how to fit a linearperiodic spline to the melanoma data. The observations are shown in Figure 2.11. There are two apparent trends: a nearly linear long-term trend over the years, and a cycle of around 10 years corresponding to the sunspot cycle. Let y = cases and x = year. As in Heckman and Ramsay (2000), we fit a linear-periodic spline with L = D4 + τ 2 D2 , where τ = 0.58. > > > >
library(fda); attach(melanoma) x <- year-1936; y <- incidence tau <- .58; tx <- tau*x ssr(y~tx+cos(tx)+sin(tx), rk=lspline(tx,type=‘‘linSinCos’’), spar=‘‘m’’)
Again, since the sample size is small, the GML method is used to select the smoothing parameter. For comparison, the cubic spline fit with smoothing parameter selected by the GML method is also shown in Figure 2.11.
48
Smoothing Splines: Methods and Applications o o o
o o
cases per 100,000 2 3 4
o o
o
o
o
o
o o
o
o
o
o o o o
1
o o
o
o o o
o o
o
o
o o
o o
o
o o
1940
1950
1960
1970
year
FIGURE 2.11 Melanoma data, observations (circles), the cubic spline fit (dashed line), and the linear-periodic spline fit (solid line).
2.11.5
Trigonometric Spline
Suppose f is a periodic function on X = [0, 1]. Assume that f ∈ W2m (per). Then we have the Fourier expansion f (x) = a0 +
∞ X
aν cos 2πνx +
ν=1
= a0 +
m−1 X ν=1
∞ X
bν sin 2πνx
ν=1
aν cos 2πνx +
m−1 X
bν sin 2πνx + Rem(x),
(2.65)
ν=1
where the first three elements in (2.65) a trigonometric polyPrepresent ∞ nomial of degree m − 1 and Rem(x) , ν=m (aν cos 2πνx + bν sin 2πνx). The penalty used in Section 2.7 for a periodic spline corresponds to L = Dm with kernel H0 = span{1}. That is, all nonconstant functions including those in the trigonometric polynomial are penalized. Analogous to the Taylor expansion and polynomial splines, one may want to include lower-degree trigonometric polynomials in the null space. The operator L = Dm does not decompose W2m (per) into lower-degree trigonometric polynomials plus the remainder terms. Now consider the null space H0 = span 1, sin 2πνx, cos 2πνx, ν = 1, . . . , m − 1 , (2.66) which includes trigonometric polynomials with degree up to m − 1. Assume that f ∈ W2m+1 (per). It is easy to see that H0 is the kernel of the
Smoothing Spline Regression
49
differential operator L=D
m−1 Y
ν=1
D2 + (2πν)2 .
(2.67)
Model space construction and decomposition of W2m+1 (per) The space W2m+1 (per) is an RKHS with the inner product (f, g) = +
Z
1
0 m−1 X Z 1 ν=1
+
0
m−1 X Z 1 ν=1 1
+
Z f dx
Z
0
1
0
gdx
Z f cos 2πνxdx
1
g cos 2πνxdx
Z f sin 2πνxdx
1
g sin 2πνxdx
0
0
(Lf )(Lg)dx,
0
where L is defined in (2.67). Furthermore, W2m+1 (per) = H0 ⊕ H1 , where H0 is given in (2.66) and H1 = W2m+1 (per) ⊖ H0 . H0 and H1 are RKHS’s with corresponding RKs R0 (x, z) = 1 +
m−1 X ν=1
cos{2π(x − z)},
m−1 Y 2 2 2 −2 R1 (x, z) = (j − ν ) cos 2πν(x − z). (2π)4m+2 j=0 ν=m ∞ X
(2.68)
R1 Sometimes f satisfies the constraint 0 f dx = 0. This constraint can be handled easily by removing constant functions in the above construction. Specifically, let H0 = span sin 2πνx, cos 2πνx, ν = 1, . . . , m − 1 . (2.69)
Assume that f ∈ W2m (per)⊖{1}. Then H0 is the kernel of the differential operator L=
m−1 Y ν=1
2 D + (2πν)2 .
(2.70)
50
Smoothing Splines: Methods and Applications
Model space construction and decomposition of W2m (per) ⊖ {1} The space W2m (per) ⊖ {1} is an RKHS with the inner product (f, g) = +
m−1 X Z 1
0 ν=1 Z m−1 1 X 0
ν=1 1
+
Z
Z f cos 2πxdx
1
0
Z f sin 2πxdx
0
1
g cos 2πxdx
g sin 2πxdx
(Lf )(Lg)dx,
0
where L is defined in (2.70). Furthermore, W2m (per) = H0 ⊕ H1 , where H0 is given in (2.69) and H1 = W2m (per) ⊖ {1} ⊖ H0 . H0 and H1 are RKHS’s with RKs R0 (x, z) =
m−1 X ν=1
R1 (x, z) =
∞ X
ν=m
cos{2π(x − z)}, 2 (2π)4(m−1)
(2.71) m−1 Y 2 2 −2 (j − ν ) cos 2πν(x − z). j=1
In the lspline function, the options type=‘‘sine1’’ and type= ‘‘sine0’’ compute RKs R1 in (2.68) and (2.71), respectively. We now use the Arosa data to show how to fit a trigonometric spline and illustrate its difference from periodic and partial splines. Suppose we want to investigate how ozone thickness changes over months in a year. We have fitted a cubic periodic spline with f ∈ W22 (per) in Section 2.7. Note that W22 (per) = H10 ⊕ H11 , where H10 = {1} and H11 = W22 (per) ⊖ {1}. Therefore, we can rewrite the periodic spline model as yi = f10 (xi ) + f11 (xi ) + ǫi ,
i = 1, . . . , n,
(2.72)
where f10 and f11 are projections onto H10 and H11 , respectively. The estimates of the overall function, fˆ10 + fˆ11 , and its projections fˆ10 (parametric) and fˆ11 (smooth) are shown in the first row of Figure 2.12. It is apparent that the monthly pattern can be well approximated by a simple sinusoidal function in the model space P = span 1, sin 2πx, cos 2πx . (2.73) Suppose we want to check the departure from the parametric model P. One approach is to add the sine and cosine functions to the null space
Smoothing Spline Regression
2
4
6
51
8 10 12
periodic spline overall
periodic spline parametric
periodic spline smooth
partial spline overall
partial spline parametric
partial spline smooth
400 300 200 100 0
400
thickness
300 200 100 0 L−spline overall
L−spline parametric
L−spline smooth
400 300 200 100 0 2
4
6
8 10 12
2
4
6
8 10 12
month FIGURE 2.12 Arosa data, plots the overall fits (left), the parametric components (middle), and the smooth components (right) of the periodic spline (top), the partial spline (middle), and the L-spline (bottom). Parametric components represent components in the spaces H10 , P, and P for periodic spline, partial spline, and L-spline, respectively. Smooth components represent components in the spaces H11 , H11 , and W23 (per)⊖P for periodic spline, partial spline, and L-spline, respectively. Dotted lines are 95% Bayesian confidence intervals.
52
Smoothing Splines: Methods and Applications
H10 of the periodic spline. This leads to the following partial spline model yi = f20 (xi ) + f21 (xi ) + ǫi , i = 1, . . . , n, (2.74) where f20 ∈ P and f21 ∈ H11 . The model (2.74) is fitted as follows: > ssr(thick~sin(2*pi*x)+cos(2*pi*x), rk=periodic(x)) The estimates of the overall function and its two components fˆ20 (parametric) and fˆ21 (smooth) are shown in the second row of Figure 2.12. Another approach is to fit a trigonometric spline with m = 2: yi = f30 (xi ) + f31 (xi ) + ǫi ,
i = 1, . . . , n,
(2.75)
where f30 ∈ P and f31 ∈ W23 (per) ⊖ P. The model (2.75) is fitted as follows: > ssr(thick~sin(2*pi*x)+cos(2*pi*x), rk=lspline(x,type=‘‘sine1’’)) The estimates of the overall function and its two components fˆ30 (parametric) and fˆ31 (smooth) are shown in the third row of Figure 2.12. All three models have similar overall fits. However, their components are quite different. The smooth component of the periodic spline reveals the departure from a constant function. To check the departure from the simple sinusoidal model space P, we can look at the smooth components from the partial and L-splines. Estimates of the smooth components from the partial and L-splines are similar. However, the confidence intervals based on the L-spline are narrower than those based on the partial spline. This is due to the fact that the two components f30 and f31 in the L-spline are orthogonal, while the two components f20 and f21 in the partial spline are not necessarily orthogonal. Therefore, we can expect the inference based on the L-spline to be more efficient in general.
Chapter 3 Smoothing Parameter Selection and Inference
3.1
Impact of the Smoothing Parameter
The penalized least squares (2.11) represents a compromise between the goodness-of-fit and a penalty to the departure from the null space H0 . The balance is controlled by the smoothing parameter λ. As λ varies from 0 to ∞, we have a family of estimates with fˆ ∈ H0 when λ = ∞. To illustrate the impact of the smoothing parameter, consider the Stratford weather data consisting of daily maximum temperatures in Stratford, Texas, during 1990. Observations are shown in Figure 3.1. Consider the regression model (1.1) where n = 73 and f represents expected maximum temperature as a function of time in a year. Denote x as the time variable scaled into [0, 1]. It is reasonable to assume that f is a smooth periodic function. In particular, we assume that f ∈ W22 (per). For a fixed λ, say 0.001, one can fit the cubic periodic spline as follows: > data(Stratford); attach(Stratford) > ssr(y~1, rk=periodic(x), limnla=log10(73*.001)) where the argument limnla specifies a search range for log10 (nλ). To see how a spline fit is affected by the choice of λ, periodic spline fits with six different values of λ are shown in Figure 3.1. It is obvious that the fit with λ = ∞ is a constant, that is, f∞ ∈ H0 . The fit with λ = 0 interpolates data. A larger λ leads to a smoother fit. Both λ = 0.0001 and λ = 0.00001 lead to visually reasonable fits. In practice it is desirable to select the smoothing parameter using an objective method rather than visual inspection. In a sense, a data-driven choice of λ allows data to speak for themselves. Thus, it is not exaggerating to say that the choice of λ is the spirit and soul of nonparametric regression. We now inspect how λ controls the fit. Again, consider model (1.1) for Stratford weather data. Let us first consider a parametric approach that approximates f using a trigonometric polynomial up to a certain
53
54
Smoothing Splines: Methods and Applications
0.0
λ=∞
0.2
0.4
0.6
0.8
1.0
λ = 0.001
λ = 1e−04 100 80
temperature (Fahrenheit)
60 40 20
λ = 1e−05
λ = 1e−06
λ=0
100 80 60 40 20
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
time
FIGURE 3.1 Stratford weather data, plot of observations, and the periodic spline fits with different smoothing parameters. degree, say k, where 0 ≤ k ≤ K and K = (n − 1)/2 = 36. Denote the corresponding parametric model space for f as √ √ (3.1) Mk = span{1, 2 sin 2πνx, 2 cos 2πνx, ν = 1, . . . , k}, where M0 = span{1}. For a fixed k, write the regression model based on Mk in a matrix form as y = Xk βk + ǫ, where √ √ 1 √2 sin 2πx1 √2 cos 2πx1 1 2 sin 2πx2 2 cos 2πx2 Xk = . .. .. .. . . √ √ 1 2 sin 2πxn 2 cos 2πxn
√ √ · · · √2 sin 2πkx1 √2 cos 2πkx1 · · · 2 sin 2πkx2 2 cos 2πkx2 .. .. . . ··· √ √ · · · 2 sin 2πkxn 2 cos 2πkxn
is the design matrix, xi = i/n, β k = (β1 , . . . , β2k+1 )T , and ǫ = (ǫ1 , . . . , ǫn )T . Since design points are equally spaced, we have the following
Smoothing Parameter Selection and Inference
55
orthogonality relations: n
2X cos 2πνxi cos 2πµxi = δν,µ , 1 ≤ ν, µ ≤ K, n i=1 n
2X sin 2πνxi sin 2πµxi = δν,µ , 1 ≤ ν, µ ≤ K, n i=1 n
2X cos 2πνxi sin 2πµxi = 0, n i=1
1 ≤ ν, µ ≤ K,
where δν,µ is the Kronecker delta. Therefore, XkT Xk = √ nI2k+1 , where I2k+1 is an identity matrix of size 2k+1. Note that XK / n is an orthogT ˜ = XK onal matrix. Define the discrete Fourier transformation y y/n. −1 T T T ˆ ˜k, Then the LS estimate of β k is βk = (Xk Xk ) Xk y = Xk y/n = y ˜ k consists of the first 2k + 1 elements of the discrete Fourier where y ˜ . More explicitly, transformation y n
1X βˆ1 = yi = y˜1 , n i=1 √ n 2X yi sin 2πνxi = y˜2ν , 1 ≤ ν ≤ k, βˆ2ν = n i=1 √ n 2X yi cos 2πνxi = y˜2ν+1 , 1 ≤ ν ≤ k. βˆ2ν+1 = n i=1
(3.2)
Now consider modeling f using the cubic periodic spline space W22 (per). The exact solution was given in Chapter 2. To simplify the argument, let us consider the following PLS min
f ∈MK
(
n
1X (yi − f (xi ))2 + λ n i=1
Z
0
1
′′ 2
)
(f ) dx ,
(3.3)
where the model space W22 (per) is approximated by MK . The following discussion holds true for the exact solution in W22 (per) (Gu 2002). However, the approximation makes the following argument simpler and transparent. Let fˆ(x) = α ˆ1 +
K X
ν=1
√ √ ˆ 2ν+1 2 cos 2πνx) (ˆ α2ν 2 sin 2πνx + α
56
Smoothing Splines: Methods and Applications
ˆ where be the solution to (3.3). Then fˆ , (fˆ(x1 ), . . . , fˆ(xn ))T = XK α, ˆ = (ˆ α α1 , . . . , α ˆ 2K+1 )T . The LS 1 1 1 T ||y − fˆ ||2 = || √ XK (y − fˆ )||2 n n n 1 T 1 T ˆ 2 = || XK y − XK XK α|| n n ˆ 2. = ||˜ y − α|| Thus (3.3) reduces to the following ridge regression problem (ˆ α1 − y˜1 )2 + +λ
K X
K X (ˆ α2ν − y˜2ν )2 + (ˆ α2ν+1 − y˜2ν+1 )2 ν=1
(2πν)4 (ˆ α22ν + α ˆ 22ν+1 ).
(3.4)
ν=1
The solutions to (3.4) are α ˆ 1 = y˜1 , y˜2ν , 1 + λ(2πν)4 y˜2ν+1 , = 1 + λ(2πν)4
α ˆ2ν = α ˆ2ν+1
ν = 1, . . . , K,
(3.5)
ν = 1, . . . , K.
Thus the periodic spline is essentially a low-pass filter: components at frequency ν are downweighted by a factor of 1 + λ(2πν)4 . Figure 3.2 shows how λ controls the nature of the filter: more high frequencies are filtered out as λ increases. Comparing (3.2) and (3.5), it is clear that selecting an order k for the trigonometric polynomial model may be viewed as hard thresholding, and selecting the smoothing parameter λ for the periodic spline may be viewed as soft thresholding. Now consider the general spline model (2.10). From (2.26), the hat matrix H(λ) = I − nλQ2 (QT2 M Q2 )−1 QT2 . Let U EU T be the eigendecomposition of QT2 ΣQ2 , where U(n−p)×(n−p) is an orthogonal matrix and E = diag(e1 , . . . , en−p ). The projection onto the space spanned by T PT , T (T T T )−1 T T = Q1 R(RT R)−1 RT QT1 = Q1 QT1 . Then H(λ) = I − nλQ2 U (E + nλI)−1 U T QT2 =
Q1 QT1
= PT +
+ Q2 QT2 − nλQ2 U (E Q2 U V U T QT2 ,
(3.6) −1
+ nλI)
U
T
QT2 (3.7)
57
0.0
0.2
weights 0.4 0.6
0.8
1.0
Smoothing Parameter Selection and Inference
0
5
10
15 20 frequency
25
30
35
FIGURE 3.2 Weights of the periodic spline filter, 1/(1 + λ(2πν)4 ), plotted as a function of frequency ν. Six curves from top down correspond to six different λ: 0, 10−6 , 10−5 , 10−4 , 10−3 , and ∞. where V = diag(e1 /(e1 + nλ), . . . , en−p /(en−p + nλ)). The hat matrix is divided into two mutually orthogonal matrices: one is the projection onto the space spanned by T , and the other is responsible for shrinking part of the signal that is orthogonal to T . The smoothing parameter shrinks eigenvalues in the form eν /(eν + nλ). The choices λ = ∞ and λ = 0 lead to the parametric model H0 and interpolation, respectively. Equation (3.7) also indicates that the hat matrix H(λ) is nonnegative definite. However, unlike the projection matrix for a parametric model, H(λ) is usually not idempotent. H(λ) has p eigenvalues equal to one and the remaining eigenvalues less than one when λ > 0.
3.2
Trade-Offs
Before introducing methods for selecting the smoothing parameter, it is helpful to discuss some basic concepts and principles for model selection. In general, model selection boils down to compromises between different aspects of a model. Occam’s razor has been the guiding principle for the compromises: the model that fits observations sufficiently well in the least complex way should be preferred. To be precise on fits observations sufficiently well, one needs a quantity that measures how well a model fits the data. One such measure is the LS in (1.6). To be precise on the least complex way, one needs a quantity that measures the complexity of a
58
Smoothing Splines: Methods and Applications
model. For a parametric model, a common measure of model complexity is the number of parameters in the model, often called the degrees of freedom (df). For example, the df of model Mk in (3.1) equals 2k + 1. What would be a good measure of model complexity for a nonparametric regression procedure? Consider the general nonparametric regression model (2.10). Let fi = Li f , and f = (f1 , . . . , fn ). Let fˆ be an estimate of f based on a modeling procedure M, and fˆi = Li fˆ. Ye (1998) defined generalized degrees of freedom (gdf) of M as gdf(M) ,
n X ∂Ef (fˆi ) i=1
∂fi
.
(3.8)
The gdf is an extension of the standard degrees of freedom for general modeling procedures. It can be viewed as the sum of the average sensitivities of the fitted values fˆi to a small change in the response. It is easy to check that (Efron 2004) n 1 X gdf(M) = 2 Cov(fˆi , yi ), σ i=1
Pn ˆ where i=1 Cov(fi , yi ) is the so-called covariance penalty (Tibshirani and Knight 1999). For spline estimate with a fixed λ, we have fˆ = H(λ)y based on (2.25). Denote the modeling procedure leading to fˆ as Mλ and H(λ) = {hij }ni,j=1 . Then Pn Pn n n X ∂ j=1 hij fj ∂Ef ( j=1 hij yj ) X = = trH(λ), gdf(Mλ ) = ∂fi ∂fi i=1 i=1
where tr represents the trace of a matrix. Even though λ does not have a physical interpretation as k, trH(λ) is a useful measure of model complexity and will be simply referred to as the degrees of freedom. For Stratford weather data, Figure 3.3(a) depicts how trH(λ) for the cubic periodic spline depends on the smoothing parameter λ. It is clear that the degrees of freedom decrease as λ increases. To illustrate the interplay between the LS and model complexity, we fit trigonometric polynomial models from the smallest model with k = 0 to the largest model with k = K. The square root of residual sum of squares (RSS) are plotted against the degrees of freedom (2k + 1) as circles in Figure 3.3(b). Similarly, we fit the periodic spline with a wide range of values for the smoothing parameter λ. Again, we plot the square root of RSS against the degrees of freedom (trH(λ)) as the solid line in Figure 3.3(b). Obviously, RSS decreases to zero (interpolation) as the degrees of freedom increase to n. The square root of RSS keeps
Smoothing Parameter Selection and Inference (b) o
square root of RSS 5 10 15
degrees of freedom 10 20 30 40 50 60 70
20
(a)
ooo oooo ooo oo ooo ooo o ooooo oo oo oo oooo o o
0
0 −12
−10
−8
−6 −4 log10(λ)
59
−2
0
0
10
20 30 40 50 60 degrees of freedom
70
FIGURE 3.3 Stratford data, plots of (a) degrees of freedom of the periodic spline against the smoothing parameter on the logarithm base 10 scale, and (b) square root of RSS from the trigonometric polynomial model (circles) and periodic spline (line) against the degrees of freedom. declining almost linearly after the initial big drop. It is quite clear that the constant model does not fit data well. However, it is unclear which model fits observations sufficiently well. Figure 3.3(b) shows that the LS and model complexity are two opposite aspects of a model: the approximation error decreases as the model complexity increases. Our goal is to find the “best” model that strikes a balance between these two conflicting aspects. To make the term “best” meaningful, we need a target criterion that quantifies a model’s performance. It is clear that the LS cannot be used as the target because it will lead to the most complex model. Even though there is no universally accepted measure, some criteria are widely accepted and used in practice. We now introduce a criterion that is commonly used for regression models. Consider the loss function L(λ) =
1 ˆ ||f − f ||2 . n
(3.9)
Define the risk function, also called mean squared error (MSE), as 1 ˆ ||f − f ||2 . (3.10) MSE(λ) , EL(λ) = E n We want the estimate fˆ to be as close to the true function f as possible. Obviously, MSE is the expectation of the Euclidean distance between the estimates and the true values. It can be decomposed into two com-
60
Smoothing Splines: Methods and Applications
ponents:
= = = = ,
MSE(λ) 1 E||(Efˆ − f ) + (fˆ − Efˆ )||2 n 1 2 1 E||Efˆ − f ||2 + E(Efˆ − f )T (fˆ − Efˆ ) + E||fˆ − Efˆ ||2 n n n 1 ˆ 1 ||Ef − f ||2 + E||fˆ − Efˆ ||2 n n 1 σ2 ||(I − H(λ))f ||2 + trH 2 (λ) n n b2 (λ) + v(λ), (3.11)
where b2 and v represent squared bias and variance, respectively. Note that bias depends on the true function, while the variance does not. Based on notations introduced in Section 3.2, let h = (h1 , . . . , hn−p )T , U T QT2 f . From (3.6), we have 1 ||(I − H(λ))f ||2 n ( 2 2 ) 1 T nλ nλ = f Q2 U diag ,..., U T QT2 f n e1 + nλ en−p + nλ 2 n−p 1X nλhν = . n ν=1 eν + nλ
b2 (λ) =
From (3.7), we have σ2 σ2 trH 2 (λ) = tr(PT + Q2 U V 2 U T QT2 ) n n ( ) n−p X eν 2 σ2 = p+ . n eν + nλ ν=1
v(λ) =
The squared bias measures how well fˆ approximates the true function f , and the variance measures how well the function can P be estimated. n−p As λ increases from 0 to ∞, b2 (λ) increases from 0 to ν=1 h2ν /n = T 2 2 2 ||Q2 f || /n, while v(λ) decreases from σ to pσ /n. Therefore, the MSE represents a trade-off between bias and variance. Note that QT2 f represents the signal that is orthogonal to T . It is easy to check that db2 (λ)/dλ|λ=0 = 0 and dv(λ)/dλ|λ=0 < 0. Thus, dMSE(λ)/dλ|λ=0 < 0, and MSE(λ) has at least one minimizer λ∗ > 0. Therefore, when MSE(0) ≤ MSE(∞), there exists at least one λ∗ such that the corresponding PLS estimate performs better in
Smoothing Parameter Selection and Inference
61
terms of MSE than the LS estimate in H0 and the interpolation. When MSE(0) > MSE(∞), considering the MSE as a function of δ = 1/λ, we have db2 (δ)/dδ|δ=0 < 0 and dv(δ)/dδ|δ=0 = 0. Then, again, there exists at least one δ ∗ such that the corresponding PLS estimate performs better in terms of MSE than the LS estimate in H0 and the interpolation. To calculate MSE, one needs to know the true function f . The following simulation illustrates the bias-variance trade-off. Observations are generated from model (1.1) with f (x) = sin(4πx2 ) and σ = 0.5. The same design points as in the Stratford weather data are used: xi = i/n for i = 1, . . . , n and n = 73. The true function and one realization of observations are shown in Figure 3.4(a). For a fixed λ, the bias, variance, and MSE can be calculated since the true function is known in the simulation. For the cubic periodic spline, Figure 3.4(b) shows b2 (λ), v(λ), and MSE(λ) as functions of log10 (nλ). Obviously, as λ increases, the squared bias increases and the variance decreases. The MSE represents a compromise between bias and variance.
1.5 0.5
y −0.5 −1.5 0.0
0.2
0.4
0.6
x
(b)
squared bias,variance, and MSE 0.00 0.10 0.20 0.30
(a)
0.8
1.0
−8
−7
−6 −5 −4 log10(nλ)
−3
−2
FIGURE 3.4 Plots of (a) true function (line) and observations (circles), and (b) squared bias b2 (λ) (dashed), variance v(λ) (dotted line), and MSE (solid line) for the cubic periodic spline.
Another closely related target criterion is the average predictive squared error (PSE) 1 + ˆ 2 (3.12) ||y − f || , PSE(λ) = E n + T where y + = f + ǫ+ are new observations of f , ǫ+ = (ǫ+ 1 , . . . , ǫn ) are + independent of ǫ, and ǫi are independent and identically distributed
62
Smoothing Splines: Methods and Applications
with mean zero and variance σ 2 . PSE measures the performance of a model’s prediction for new observations. We have 1 + 2 ˆ PSE(λ) = E ||(y − f ) + (f − f )|| = σ 2 + MSE(λ). n Thus PSE differs from MSE only by a constant σ 2 . Ideally, one would want to select λ that minimizes the MSE (PSE). This is, however, not practical because MSE (PSE) depends on the unknown true function f that one wants to estimate in the first place. Instead, one may estimate MSE (PSE) from the data and then minimize the estimated criterion. We discuss unbiased and cross-validation estimates of PSE (MSE) in Sections 3.3 and 3.4, respectively.
3.3
Unbiased Risk
First consider the case when the error variance σ 2 is known. Since 1 2 ˆ ||y − f || E n 1 2 1 2 T 2 ˆ ˆ =E ||y − f || + (y − f ) (f − f ) + ||f − f || n n n 2 2σ trH(λ) + MSE(λ), (3.13) = σ2 − n then, UBR(λ) ,
1 2σ 2 ||(I − H(λ))y||2 + trH(λ) n n
(3.14)
is an unbiased estimate of PSE(λ). Since PSE differs from MSE only by a constant σ 2 , one may expect the minimizer of UBR(λ) to be close to the minimizer of the risk function MSE(λ). In fact, a stronger result holds: under certain regularity conditions, UBR(λ) is a consistent estimate of the relative loss function L(λ) + n−1 ǫT ǫ (Gu 2002). The function UBR(λ) is referred to as the unbiased risk (UBR) criterion, and the minimizer of UBR(λ) is referred to as the UBR estimate of λ. It is obvious that UBR(λ) is an extension of the Mallow’s Cp criterion. The error variance σ 2 is usually unknown in practice. In general, there are two classes of estimators for σ 2 : residual-based and differencebased estimators. The first class of estimators is based on residuals from an estimate of f . For example, analogous to parametric regression, an
Smoothing Parameter Selection and Inference
63
estimator of σ 2 based on fit fˆ = H(λ)y, is σ ˆ2 ,
||(I − H(λ))y||2 . n − trH(λ)
(3.15)
The estimator σ ˆ 2 is consistent under certain regularity conditions (Gu 2002). However, it depends critically on the smoothing parameter λ. Thus, it cannot be used in the UBR criterion since the purpose of this criterion is to select λ. For choosing the amount of smoothing, it is desirable to have an estimator of σ 2 without needing to fit the function f first. The difference-based estimators of σ 2 do not require an estimate of the mean function f . The basic idea is to remove the mean function f by taking differences based on some well-chosen subsets of data. Consider the general SSR model (2.10). Let Ij = {i(j, 1), . . . , i(j, Kj )} ⊂ {1, . . . , n} be a subset of index and d(j, k) be some fixed coefficients such that Kj X
d2 (j, k) = 1,
k=1
Since
E then
k=1
Kj X
Kj X
d(j, k)yi(j,k)
k=1
d(j, k)Li(j,k) f ≈ 0,
2
≈E
Kj X
j = 1, . . . , J.
d(j, k)ǫi(j,k)
k=1
2 Kj J X X 1 d(j, k)yi(j,k) σ ˜2 ≈ J j=1
2
= σ2 ,
(3.16)
k=1
provides an approximately unbiased estimator of σ 2 . The estimator σ ˜2 is referred to as a differenced-based estimator since d(j, k) are usually PKj chosen to be contrasts such that k=1 d(j, k) = 0. The specific choices of subsets and coefficients depend on factors including prior knowledge about f and the domain X . Several methods have been proposed for the common situation when x is a univariate continuous variable, f is a smooth function, and Li are evaluational functionals. Suppose design points are ordered such that x1 ≤ x2 ≤ · · · ≤ xn . Since f is smooth, then f (xj+1 ) − f (xj ) ≈ 0 when neighboring design points are √ close to each other. Setting Ij = {j, j + 1} and d(j, 1) = −d(j, 2) = 1/ 2 for j = 1, . . . , n−1, we have the first-order difference-based estimator proposed by Rice (1984): n
2 σ ˜R =
X 1 (yi − yi−1 )2 . 2(n − 1) i=2
(3.17)
64
Smoothing Splines: Methods and Applications
Hall, Kay and Titterington (1990) proposed the mth order differencebased estimator (m )2 n−m X X 1 2 σ ˜HKT = δk yj+k , (3.18) n − m j=1 k=1
Pm Pm where coefficients δk satisfy k=1 δk = 0, k=1 δk2 = 1, and δ1 δm 6= 0. Optimal choices of δk are studied in Hall et al. (1990). It is easy to 2 see that σ ˜HKT corresponds to Ij = {j, . . . , j + m} and d(j, k) = δk for j = 1, . . . , n − m. 2 2 Both σ ˜R and σ ˜HKT require an ordering of design points that could be problematic for multivariate independent variables. Tong and Wang (2005) proposed a different method for a general domain X . Suppose X is equipped with a norm. Collect squared distances, dij = ||xi − xj ||2 , for all pairs {xi , xj }, and half squared differences, sij = (yi − yj )2 /2, for all pairs {yi , yj }. Then E(sij ) = {f (xi ) − f (xj )}2 /2 + σ 2 . Suppose {f (xi )−f (xj )}2 /2 can be approximated by βdij when dij is small. Then the LS estimate of the intercept in the simple linear model sij = α + βdij + ǫij ,
dij ≤ D,
(3.19)
2 provides an estimate of σ 2 . Denote such an estimator as σ ˜TW . Theoretical properties and the choice of bandwidth D were studied in Tong and Wang (2005). To illustrate the UBR criterion as an estimate of PSE, we generate responses from model (1.1) with f (x) = sin(4πx2 ), σ = 0.5, xi = i/n for i = 1, . . . , n, and n = 73. For the cubic periodic spline, the UBR functions based on 50 replications of simulation data are shown in Figure 3.5 where the true variance is used in (a) and the Rice estimator is used in (b).
3.4
Cross-Validation and Generalized Cross-Validation
Equation (3.13) shows that the RSS underestimates the PSE by the amount of 2σ 2 trH(λ)/n. The second term in the UBR criterion corrects this bias. The bias in RSS is a consequence of using the same data for model fitting and model evaluation. Ideally, these two tasks should be separated using independent samples. This can be achieved by splitting the whole data into two subsamples: a training (calibration) sample for
Smoothing Parameter Selection and Inference UBR with Rice estimator
UBR with true variance
0.1
PSE and UBR 0.2 0.3 0.4 0.5
0.6
65
−8
−7
−6
−5 −4 log10(nλ)
−3
−2
−8
−7
−6
−5 −4 log10(nλ)
−3
−2
FIGURE 3.5 Plots of the PSE function as solid lines, the UBR functions with true σ 2 as dashed lines (left), and the UBR functions with 2 Rice estimator σ ˜R . The minimum point of the PSE is marked as long bars at the bottom. The UBR estimates of log10 (nλ) are marked as short bars. model fitting, and a test (validation) sample for model evaluation. This approach, however, is not efficient unless the sample size is large. The idea behind cross-validation is to recycle data by switching the roles of training and test samples. For simplicity, we present leaving-out-one cross-validation only. That is, each time one observation will be left out as the test sample, and the remaining n − 1 samples will be used as the training sample. Let fˆ[i] be the minimizer of the PLS based on all observations except yi : 1X (yj − Lj f )2 + λ||P1 f ||2 . (3.20) n j6=i
The cross-validation estimate of PSE is n
CV(λ) ,
2 1 X ˆ[i] Li f − yi . n i=1
(3.21)
CV(λ) is referred to as the cross-validation criterion, and the minimizer of CV(λ) is called the cross-validation estimate of the smoothing parameter. Computation of fˆ[i] based on (3.21) for each i = 1, . . . , n would be costly. Fortunately, this is unnecessary due to the following lemma.
66
Smoothing Splines: Methods and Applications
Leaving-out-one Lemma For any fixed i, fˆ[i] is the minimizer of 2 1 X 1 ˆ[i] Li f − Li f + (yj − Lj f )2 + λ||P1 f ||2 . n n
(3.22)
j6=i
[Proof] For any function f , we have 2 1 X 1 ˆ[i] Li f − Li f + (yj − Li f )2 + λ||P1 f ||2 n n j6=i
1X (yj − Li f )2 + λ||P1 f ||2 ≥ n j6=i 2 1 X ≥ yj − Lj fˆ[i] + λ||P1 fˆ[i] ||2 n j6=i 2 1 X 2 1 ˆ[i] = Li f − Li fˆ[i] + yj − Lj fˆ[i] + λ||P1 fˆ[i] ||2 , n n j6=i
where the second inequality holds since fˆ[i] is the minimizer of (3.20). The above lemma indicates that the solution to the PLS (3.20) without the ith observation, fˆ[i] , is also the solution to the PLS (2.11) with the ith observation yi being replaced by the fitted value Li fˆ[i] . Note that the hat matrix H(λ) depends on the model space and operators Li only. It does not depend on observations of the dependent variable. Therefore, both fits based on (2.11) and (3.22) have the same hat matrix. That is, [i] [i] fˆ = H(λ)y and fˆ = H(λ)y [i] , where fˆ = (L1 fˆ[i] , . . . , Ln fˆ[i] )T and y [i] is the same as y except that the ith element is replaced by Li fˆ[i] . Denote H(λ) = {hij }ni,j=1 . Then Li fˆ = Li fˆ[i] =
n X
hij yj ,
j=1
X j6=i
hij yj + hii Li fˆ[i] .
Solving for Li fˆ[i] , we have Li fˆ − hii yi . Li fˆ[i] = 1 − hii Then
Li fˆ − hii yi Li fˆ − yi Li fˆ[i] − yi = − yi = . 1 − hii 1 − hii
Smoothing Parameter Selection and Inference
67
Plugging into (3.21), we have CV(λ) =
n 1 X (Li fˆ − yi )2 . n i=1 (1 − hii )2
(3.23)
Therefore, the cross-validation criterion can be calculated using the fit based on the whole sample and the diagonal elements of the hat matrix. Replacing hii by the average of diagonal elements, trH(λ), we have the generalized cross-validation (GCV) criterion Pn 1 (Li fˆ − yi )2 GCV(λ) , n i=1 (3.24) 2 . 1 tr(I − H(λ)) n
The GCV estimate of λ is the minimizer of GCV(λ). Since trH(λ)/n is usually small in the neighborhood of the optimal λ, we have σ2 2σ 2 1 2 2 2 ||(I − H(λ))f || + trH (λ) + σ − trH(λ) E{GCV(λ)} ≈ n n n 2 1 + trH(λ) n = PSE(λ){1 + o(1)}.
The above approximation provides a very crude argument supporting the GCV criterion as a proxy for the PSE. More formally, under certain regularity conditions, GCV(λ) is a consistent estimate of the relative loss function. Furthermore, GCV(λ) is invariant to an orthogonal transformation of y. See Wahba (1990) and Gu (2002) for details. One distinctive advantage of the GCV criterion over the UBR criterion is that the former does not require an estimate of σ 2 . To illustrate the CV(λ) and GCV(λ) criteria as estimates of the PSE, we generate responses from model (1.1) with f (x) = sin(4πx2 ), σ = 0.5, xi = i/n for i = 1, . . . , n, and n = 73. For cubic periodic spline, the CV and GCV scores for 50 replications of simulation data are shown in Figure 3.6.
3.5
Bayes and Linear Mixed-Effects Models
Assume a prior for f as F (x) =
p X
ν=1
1
ζν φν (x) + δ 2 U (x),
(3.25)
68
Smoothing Splines: Methods and Applications 0.6
GCV
0.1
0.1
0.2
PSE and CV 0.3 0.4 0.5
PSE and GCV 0.2 0.3 0.4 0.5
0.6
CV
−8
−7
−6 −5 −4 log10(nλ)
−3
−2
−8
−7
−6 −5 −4 log10(nλ)
−3
−2
FIGURE 3.6 Plots of the PSE function as solid lines, the CV functions as dashed lines (left), and the GCV functions as dashed lines (right). The minimum point of the PSE is marked as long bars at the bottom. CV and GCV estimates of log10 (nλ) are marked as short bars. iid
where ζ1 , . . . , ζp ∼ N(0, κ), U (x) is a zero-mean Gaussian stochastic process with covariance function R1 (x, z), ζν and U (x) are independent, and κ and δ are positive constants. Note that the bounded linear functionals Li are defined for elements in H. Its application to the random process F (x) is yet to be defined. For simplicity, the subscript i in Li is ignored in the following definition. Define L(ζν φν ) = ζν Lφν . The definition of LU requires the duality between the Hilbert space spanned by a family of random variables and its associated RKHS. Consider the linear space n o X U= W : W = αj U (xj ), xj ∈ X , αj ∈ R
with inner product (W1 , W2 ) = E(W1 W2 ). Let L2 (U ) be the Hilbert space that is the completion of U. Note that the RK R1 of H1 coincides with the covariance function of U (x). Consider a linear map Ψ : H1 → L2 (U ) such that Ψ{R1 (xj , ·)} = U (xj ). Since (R1 (x, ·), R1 (z, ·)) = R1 (x, z) = E{U (x)U (z)} = (U (x), U (z)),
the map Ψ is inner product preserving. In fact, H1 is isometrically isomorphic to L2 (U ). See Parzen (1961) for details. Since L is a bounded linear functional in H1 , by the Riesz representation theorem, there exists a representer h such that Lf = (h, f ). Finally we define LU , Ψh.
Smoothing Parameter Selection and Inference
69
Note that LU is a random variable in L2 (U ). The application of L to F Pp 1 is defined as LF = ν=1 ζν Lφν + δ 2 LU . When L is an evaluational functional, say Lf = f (x0 ) for a fixed x0 , we have h(·) = R1 (x0 , ·). Consequently, LU = Ψh = Ψ{R1 (x0 , ·)} = U (x0 ), the evaluation of U at x0 . Therefore, as expected, LF = F (x0 ) when L is an evaluational functional. Suppose observations are generated by yi = Li F + ǫi ,
i = 1, . . . , n,
(3.26)
iid
where the prior F is defined in (3.25) and ǫi ∼ N(0, σ 2 ). Note that the normality assumption has been made for random errors. We now compute the posterior mean E(L0 F |y) for a bounded linear functional L0 on H. Note that L0 is arbitrary, which could be quite different from Li . For example, suppose f ∈ W2m [a, b] and Li are evaluational functionals. Setting L0 f = f ′ (x0 ) leads to an estimate of f ′ . Using the correspondence between H and L2 (U ), we have E(Li U Lj U ) = (Li U, Lj U ) = (Li(x) R1 (x, ·), Lj(z) R1 (z, ·)) = Li(x) Lj(z) R1 (x, z),
E(L0 U Li U ) = (L0 U, Lj U ) = (L0(x) R1 (x, ·), Lj(z) R1 (z, ·)) = L0(x) Lj(z) R1 (x, z). Let ζ = (ζ1 , . . . , ζp )T , φ = (φ1 , . . . , φp )T and L0 φ = (L0 φ1 , . . . , L0 φp )T . 1 Then F (x) = φT (x)ζ + δ 2 U (x). It is easy to check that 1
y = T ζ + δ 2 (L1 U, . . . , Ln U )T + ǫ ∼ N(0, κT T T + δΣ + σ 2 I), (3.27) and 1
L0 F = (L0 φ)T ζ + δ 2 L0 U
∼ N(0, κ(L0 φ)T L0 φ + δL0(x) L0(z) R1 (x, z)).
(3.28)
E{(L0 F )y} = κT L0 φ + δL0 ξ,
(3.29)
Furthermore, where ξ(x) = (L1(z) R1 (x, z), . . . , Ln(z) R1 (x, z))T ,
L0 ξ = (L0(x) L1(z) R1 (x, z), . . . , L0(x) Ln(z) R1 (x, z))T .
(3.30)
70
Smoothing Splines: Methods and Applications
Let λ = σ 2 /nδ and η = κ/δ. Using properties of multivariate normal random variables and equations (3.27), (3.28), and (3.29), we have E(L0 F |y) = (L0 φ)T ηT T (ηT T T + M )−1 y + (L0 ξ)T (ηT T T + M )−1 y. (3.31) It can be shown (Wahba 1990, Gu 2002) that for any full-column rank matrix T and symmetric and nonsingular matrix M , lim (ηT T T + M )−1 = M −1 − M −1 T (T T M −1 T )−1 T T M −1 ,
η→∞
lim ηT T (ηT T T + M )−1 = T (T T M −1 T )−1 T T M −1 .
η→∞
(3.32)
Combining results in (3.31), (3.32), and (2.22), we have lim E(L0 F |y) = (L0 φ)T T (T T M −1 T )−1 T T M −1 y
κ→∞
+ (L0 ξ)T {M −1 − M −1 T (T T M −1 T )−1 T T M −1 }y
= (L0 φ)T d + (L0 ξ)T c = L0 fˆ.
The above result indicates that the smoothing spline estimate fˆ is a Bayes estimator with a diffuse prior for ζ. From a frequentist perspective, the smoothing spline estimate may be regarded as the best linear unbiased prediction (BLUP) estimate of a linear mixed-effects (LME) model. We now present three corresponding LME models. The first LME model assumes that y = T ζ + u + ǫ,
(3.33)
where ζ = (ζ1 , . . . , ζp )T are deterministic parameters, u = (u1 , . . . , un )T are random effects with distribution u ∼ N(0, σ 2 Σ/nλ), ǫ = (ǫ1 , . . . , ǫn )T are random errors with distribution ǫ ∼ N(0, σ 2 I), and u and ǫ are independent. The second LME model assumes that y = T ζ + Σu + ǫ,
(3.34)
where ζ are deterministic parameters, u are random effects with distribution u ∼ N(0, σ 2 Σ+ /nλ), Σ+ is the Moore–Penrose inverse of Σ, ǫ are random errors with distribution ǫ ∼ N(0, σ 2 I), and u and ǫ are independent. It is inconvenient to use the above two LME models for computation since Σ may be singular. Write Σ = ZZ T , where Z is a n × m matrix with m = rank(Σ). The third LME model assumes that y = T ζ + Zu + ǫ,
(3.35)
Smoothing Parameter Selection and Inference
71
where ζ are deterministic parameters, u are random effects with distribution u ∼ N(0, σ 2 I/nλ), ǫ are random errors with distribution ǫ ∼ N(0, σ 2 I), and u and ǫ are independent. It can be shown that the BLUP estimates for each of the three LME models (3.33), (3.34), and (3.35) are the same as the smoothing spline estimate. See Wang (1998b) and Chapter 9 for more details.
3.6
Generalized Maximum Likelihood
The connection between smoothing spline models and Bayes models can be exploited to develop a likelihood-based estimate for the smoothing parameter. From (3.27), the marginal distribution of y is N(0, δ(ηT T T + M )). Consider the following transformation QT2 w1 (3.36) = √1 T T y. w2 η It is easy to check that w1 = QT2 y ∼ N(0, δQT2 M Q2 ), δ Cov(w1 , w2 ) = √ QT2 (ηT T T + M )T → 0, η → ∞, η δ T Var(w 2 ) = T (ηT T T + M )T → δ(T T T )(T T T ), η → ∞. η Note that the distribution of w2 is independent of λ. Therefore, we consider the negative marginal log-likelihood of w 1 l(λ, δ|w1 ) =
1 1 log |δQT2 M Q2 | + w T1 (QT2 M Q2 )−1 w 1 + C1 , 2 2δ
(3.37)
where C1 is a constant. Minimizing l(λ, δ|w 1 ) with respect to δ, we have wT (QT M Q2 )−1 w 1 . δˆ = 1 2 n−p
(3.38)
The profile negative log-likelihood ˆ 1 ) = 1 log |QT M Q2 | + n − p log δˆ + C2 lp (λ, δ|w 2 2 2 n−p w T1 (QT2 M Q2 )−1 w1 = + C2 , log 1 2 {det(QT M Q2 )−1 } n−p 2
(3.39)
72
Smoothing Splines: Methods and Applications
where C2 is another constant. The foregoing profile negative log-likelihood is equivalent to GML(λ) ,
w T1 (QT2 M Q2 )−1 w 1 {det(QT2 M Q2 )−1 }
1 n−p
=
y T (I − H(λ))y
+
1
[det {(I − H(λ))}] n−p
, (3.40)
where the second equality is based on (2.26), and det+ represents the product of the nonzero eigenvalues. The function GML(λ) is referred to as the generalized maximum likelihood (GML) criterion, and the minimizer of GML(λ) is called the GML estimate of the smoothing parameter. From (3.38), a likelihood-based estimate of σ 2 is σ ˆ2 ,
ˆ T (QT M Q2 )−1 w1 nλw y T (I − H(λ))y 1 2 = . n−p n−p
(3.41)
The GML criterion may also be derived from the connection between smoothing spline models and LME models. Consider any one of the three corresponding LME models (3.33), (3.34), and (3.35). The smoothing parameter λ is part of the variance component for the random effects. It is common practice in the mixed-effects literature to estimate the variance components using restricted likelihood based on an orthogonal contrast of original observations where the orthogonal contrast is used to eliminate the fixed effects. Note that w 1 is one such orthogonal contrast since Q2 is orthogonal to T . Therefore, l(λ, δ|w 1 ) in (3.37) is the negative log restricted likelihood, and the GML estimate of the smoothing parameter is the restricted maximum likelihood (REML) estimate. Furthermore, the estimate of error variance in (3.41) is the REML estimate of σ 2 . The connection between a smoothing spline estimate with GML estimate of the smoothing parameter and a BLUP estimate with REML estimate of the variance component in a corresponding LME model may be utilized to fit a smoothing spline model using software for LME models. This approach will be adopted in Chapters 5, 8, and 9 to fit smoothing spline models for correlated observations.
3.7
Comparison and Implementation
Theoretical properties of the UBR, GCV, and GML criteria can be found in Wahba (1990) and Gu (2002). The UBR criterion requires an estimate of the variance σ 2 . No distributional assumptions are required for the UBR and GCV criteria, while the normality assumption is required in
Smoothing Parameter Selection and Inference
73
the derivation of the GML criterion. Nevertheless, limited simulations suggest that the GML method is quite robust to the departure from the normality assumption. Theoretical comparisons between UBR, GCV, and GML criteria have been studied using large-sample asymptotics (Wahba 1985, Li 1986, Stein 1990) and finite sample arguments (Efron 2001). Conclusions based on different perspectives do not always agree with each other. In practice, all three criteria usually perform well and lead to similar estimates. Each method has its own strengths and weaknesses. The UBR and GCV criteria occasionally lead to gross undersmooth (interpolation) when sample size is small. Fortunately, this problem diminishes quickly when sample size increases (Wahba and Wang 1995). The argument spar in the ssr function specifies which method should be used for selecting the smoothing parameter λ. The options spar=‘‘v’’, spar=‘‘m’’, and spar=‘‘u’’ correspond to the GCV, GML, and UBR methods, respectively. The default choice is the GCV method. We now use the motorcycle data to illustrate how to specify the spar option. For simplicity, the variable times is first scaled into [0, 1]. We first use the Rice method to estimate error variance and use the estimated variance in the UBR criterion: > x <- (times-min(times))/(max(times)-min(times)) > vrice <- mean((diff(accel))**2)/2 > mcycle.ubr.1 <- ssr(accel~x, rk=cubic(x), spar=‘‘u’’, varht=vrice) > summary(mcycle.ubr.1) Smoothing spline regression fit by UBR method ... UBR estimate(s) of smoothing parameter(s) : 8.60384e-07 Equivalent Degrees of Freedom (DF): 12.1624 Estimate of sigma: 23.09297 The option varht specifies the parameter σ 2 required for the UBR method. The summary function provides a synopsis including the estimate of the smoothing parameter, the degrees of freedom trH(λ), and the estimate of standard deviation σ ˆ. Instead of the Rice estimator, we can estimate the error variance using 2 Tong and Wang’s estimator σ ˜TW . Note that there are multiple observations at some time points. We use these replicates to estimate the error variance. That is, we select all pairs with zero distances: > d <- s <- NULL > for (i in 1:132) {
74
Smoothing Splines: Methods and Applications
for (j in (i+1):133) { d <- c(d, (x[i]-x[j])**2) s <- c(s, (accel[i]-accel[j])**2/2) }} > vtw <- coef(lm(s~d,subset=d==0))[1] > mcycle.ubr.2 <- ssr(accel~x, rk=cubic(x), spar=‘‘u’’, varht=vtw) > summary(mcycle.ubr.2) Smoothing spline regression fit by UBR method ... UBR estimate(s) of smoothing parameter(s) : 8.35209e-07 Equivalent Degrees of Freedom (DF): 12.24413 Estimate of sigma: 22.69913 Next we use the GCV method to select the smoothing parameter:
> mcycle.gcv <- ssr(accel~x, rk=cubic(x), spar=‘‘v’’) > summary(mcycle.gcv) Smoothing spline regression fit by GCV method ... GCV estimate(s) of smoothing parameter(s) : 8.325815e-07 Equivalent Degrees of Freedom (DF): 12.25284 Estimate of sigma: 22.65806 Finally, we use the GML method to select the smoothing parameter:
> mcycle.gml <- ssr(accel~x, rk=cubic(x), spar=‘‘m’’) > summary(mcycle.gml) Smoothing spline regression fit by GML method ... GML estimate(s) of smoothing parameter(s) : 4.729876e-07 Equivalent Degrees of Freedom (DF): 13.92711 Estimate of sigma: 22.57701 For the motorcycle data, all three methods lead to similar estimates of the smoothing parameter and the function f .
Smoothing Parameter Selection and Inference
3.8 3.8.1
75
Confidence Intervals Bayesian Confidence Intervals
Consider the Bayes model (3.25) and (3.26). The computation in Section 3.5 can be carried out one step further to derive posterior distributions. In the following arguments, as in Section 3.5, a diffuse prior is assumed for ζ with κ → ∞. For simplicity of notation, the limit is not expressed explicitly. 1 Let F0ν = ζν φν for ν = 1, . . . , p, and F1 = δ 2 U . Let L0 , L01 , and L02 be bounded linear functionals. Since F0ν , F1 , and ǫi are all normal random variables, then the posterior distributions of L0 F0ν and L0 F1 are normal with the following mean and covariances. Posterior means and covariances For ν, µ = 1, . . . , p, the posterior means are E(L0 F0ν |y) = (L0 φν )eTν d,
(3.42)
E(L0 F1 |y) = (L0 ξ)T c,
and the posterior covariances are δ −1 Cov(L01 F0ν , L02 F0µ |y) = (L01 φν )(L02 φµ )eTν Aeµ , δ −1 Cov(L01 F0ν , L02 F1 |y) = −(L01 φν )eTν B(L02 ξ), δ
−1
(3.43)
T
Cov(L01 F1 , L02 F1 |y) = L01 L02 R1 − (L01 ξ) C(L02 ξ),
where eν is a vector of dimension p with the νth element being one and all other elements being zero, the vectors c and d are given in (2.22), and the matrices A = (T T M −1 T )−1 , B = AT T M −1 , and C = M −1 (I − B). The vectors ξ and Lξ are defined in (3.30). Proofs can be found in Wahba (1990) and Gu (2002). Note that H = span{φ1 , . . . , φp } ⊕ H1 . Then any f ∈ H can be represented as f = f01 + · · · + f0p + f1 ,
(3.44)
where f0ν ∈ span{φν } for ν = 1, . . . , p, and f1 ∈ H1 . The estimate fˆ can also be decomposed similarly fˆ = fˆ01 + · · · + fˆ0p + fˆ1 , where fˆ0ν = φν dν for ν = 1, . . . , p, and fˆ1 = ξ T c.
(3.45)
76
Smoothing Splines: Methods and Applications
The functionals L0 , L01 , and L02 are arbitrary as long as they are well defined. Equations in (3.42) indicate that the posterior means of components in F equal their corresponding components in the spline estimate fˆ. Equations (3.42) and (3.43) can be used to compute posterior means and variances for any combinations of components of F . Specifically, consider the linear combination Fγ (x) =
p X
γν F0ν (x) + γp+1 F1 (x),
(3.46)
ν=1
where γν equals 1 when the corresponding component in F is to be included and 0 otherwise, and γ = (γ1 , . . . , γp+1 )T . Then, for any linear functional L0 , E(L0 Fγ |y) = Var(L0 Fγ |y) =
p X
γν (L0 φν )dν + γp+1 (L0 ξ)T c,
ν=1 p X p X
ν=1 µ=1 p X
+
ν=1
γν γµ Cov(L0 F0ν , L0 F0µ |y)
(3.47)
γν γp+1 Cov(L0 F0ν , L0 F1 |y)
+ γp+1 Cov(L0 F1 , L0 F1 |y). For various reasons it is often desirable to have interpretable confidence intervals for the function f and its components. For example, one may want to decide whether a nonparametric model is more suitable than a particular parametric model. A parametric regression model may be considered not suitable if a larger portion of its estimate is outside the confidence intervals of a smoothing spline estimate. Consider a collection of points x0j ∈ X , j = 1, . . . , J. For each j, posterior mean E{Fγ (x0j )|y} and variance Var{Fγ (x0j )|y} can be calculated using equations in (3.47) by setting L0 F = F (x0j ). Then, 100(1 − α)% Bayesian confidence intervals for fγ (x0j ) =
p X
γν f0ν (x0j ) + γp+1 f1 (x0j ),
j = 1, . . . , J
(3.48)
j = 1, . . . , J,
(3.49)
ν=1
are E{Fγ (x0j )|y} ± z α2
q Var{Fγ (x0j )|y},
where z α2 is the 1 − α/2 percentile of a standard normal distribution.
Smoothing Parameter Selection and Inference
77
In particular, let F = (L1 F, . . . , Ln F )T . Applying (3.42) and (3.43), we have E(F |y) = H(λ)y,
Cov(F |y) = σ 2 H(λ).
(3.50)
Therefore, the posterior variances of the fitted values Var(Li F |y) = σ 2 hii , where hii are diagonal elements of the matrix H(λ). When Li are evaluational functionals Li f = f (xi ), Wahba (1983) proposed the following 100(1 − α)% confidence intervals p fˆ(xi ) ± z α2 σ (3.51) ˆ hii , where σ ˆ is an estimates of σ. Note that confidence intervals for a linear combination of components of f can be constructed similarly. Though based on the Bayesian argument, it has been found that the Bayesian confidence intervals have good frequentist properties provided that the smoothing parameter has been estimated properly. They must be interpreted as “across-the-function” rather than pointwise. More precisely, define the average coverage probability (ACP) as n
ACP =
1X P {f (xi ) ∈ C(α, xi )} n i=1
for some (1 − α)100% confidence intervals {C(α, xi ), i = 1, . . . , n}. Rather than considering a confidence interval for f (τ ), where f (·) is the realization of a stochastic process and τ is fixed, one may consider confidence intervals for f (τn ), where f is now a fixed function and τn is a point randomly selected from {xi , i = 1, . . . , n}. Then ACP = P {f (τn ) ∈ C(α, τn )}. Note that the ACP coverage property is weaker than the pointwise coverage property. For polynomial splines and C(α, xi ) being the Bayesian confidence intervals defined in (3.51), under certain regularity conditions, Nychka (1988) showed that ACP ≈ 1 − α. The predict function in the assist package computes the posterior mean and standard deviation of Fγ (x) in (3.46). The option terms specifies the coefficients γ, and the option newdata specifies a data frame consisting of the values at which predictions are required. We now use the geyser, motorcycle, and Arosa data to illustrate how to use the predict function to compute the posterior means and standard deviations. For the geyser data, we have fitted a cubic spline in Chapter 1, Section 1.1 and a partial spline in Chapter 2, Section 2.10. In the following we fit a cubic spline using the ssr function, compute posterior means and standard deviations for the estimate of the smooth component P1 f using the predict function, and plot the estimate of the smooth components and 95% Bayesian confidence intervals:
78
Smoothing Splines: Methods and Applications
> geyser.cub.fit <- ssr(waiting~x, rk=cubic(x)) > grid <- seq(0,1,len=200) > geyser.cub.pred <- predict(geyser.cub.fit, pstd=T, terms=c(0,0,1), newdata=data.frame(x=grid)) > grid1 <- grid*diff(range(eruptions))+min(eruptions) > plot(eruptions, waiting, xlab=‘‘duration (mins)’’, ylim=c(-6,6), ylab=‘‘smooth component (mins)’’, type=‘‘n’’) > polygon(c(grid1,rev(grid1)), c(geyser.cub.pred$fit-1.96*geyser.cub.pred$pstd, rev(geyser.cub.pred$fit+1.96*geyser.cub.pred$pstd)), col=gray(0:8/8)[8], border=NA) > lines(grid1, geyser.cub.pred$fit) > abline(0,0,lty=2) where the option pstd specifies whether the posterior standard deviations should be calculated. Note that the option pstd=T can be dropped in the above statement since it is the default. There are in total three components in the cubic spline fit: two basis functions φ1 (constant) and φ2 (linear) for the null space, and the smooth component in the space H1 . In the order in which they appear in the ssr function, these three components correspond to the intercept (~1, which is automatically included), the linear basis specified by ~x, and the smooth component specified by rk=cubic(x). Therefore, the option terms=c(0,0,1) was used to compute the posterior means and standard deviations for the smooth component f1 in the space H1 . The estimate of the smooth components and 95% Bayesian confidence intervals are shown in Figure 3.7(a). A large portion of the zero constant line is outside the confidence intervals, indicating the lack-of-fit of a linear model (the null space of the cubic spline). For the partial spline model (2.43), consider the following Bayes model yi = sTi β + Li F + ǫi ,
i = 1, . . . , n,
(3.52)
where the prior for β is assumed to be N(0, κIq ), the prior F is defined iid
in (3.25), and ǫi ∼ N(0, σ 2 ). Again, it can be shown that the PLS estimates of components in β and f based on (2.44) equal the posterior means of their corresponding components in the Bayes model as κ → ∞. Posterior covariances and Bayesian confidence intervals for β and fγ can be calculated similarly. We now refit the partial spline model (2.47) and compute posterior means and standard deviations for the estimate of the smooth component P1 f :
Smoothing Parameter Selection and Inference (b)
smooth component (min) −0.04 0.00 0.04
smooth component (min) −6 −4 −2 0 2 4 6
(a)
79
1.5
2.5 3.5 4.5 duration (min)
1.5
2.5 3.5 4.5 duration (min)
FIGURE 3.7 Geyser data, plots of estimates of the smooth components, and 95% Bayesian confidence intervals for (a) the cubic spline and (b) the partial spline models. The constant zero is marked as the dotted line in each plot. > t <- .397; s <- 1*(x>t) > geyser.ps.fit <- ssr(waiting~x+s, rk=cubic(x)) > geyser.ps.pred <- predict(geyser.ps.fit, terms=c(0,0,0,1), newdata=data.frame(x=grid,s=1*(grid>t))) Since there are in total four components in the partial spline fit, the option terms=c(0,0,0,1) was used to compute the posterior means and standard deviations for the smooth component f1 . Figure 3.7(b) shows the estimate of the smooth components and 95% Bayesian confidence intervals for the partial spline. The zero constant line is well inside the confidence intervals, indicating that this smooth component may be dropped in the partial spline model. That is, a simple linear change-point model may be appropriate for this data. For the motorcycle data, based on visual inspection, we have searched a potential change-point t to the first derivative in the interval [0.2, 0.25] for the variable x in Section 2.10. To search for all possible change-points to the first derivative, we fit the partial spline model (2.48) repeatedly with t taking values on a grid point in the interval [0.1, 0.9]. We then calculate the posterior mean and p standard deviation for β. Define a t-statistic at point t as E(β|y)/ Var(β|y). The t-statistics were calculated as follows: > tgrid <- seq(0.05,.95,len=200); tstat <- NULL > for (t in tgrid) { s <- (x-t)*(x>t)
80
Smoothing Splines: Methods and Applications tmp <- ssr(accel~x+s, rk=cubic(x)) tmppred <- predict(tmp, terms=c(0,0,1,0), newdata=data.frame(x=.5,s=1)) tstat <- c(tstat, tmppred$fit/tmppred$pstd) }
Note that term=c(0,0,1,0) requests the posterior mean and standard deviation for the component β × s, where s = (x − t)+ , and s is set to one in the newdata argument. The t-statistics are shown in the bottom panel of Figure 3.8.
o
o
50
o o o o o
o o
o
o
o o
o o
acceleration (g) −50 0
o
oo o o o
o ooo oo o o oo o o o
o oo o o o o o
o
o o o
o o
o
o
o
o oo o
o o
o o
o
o
o
o
o
o
oo
o
o o o
o
oo o
o o
o
o o
o o
o o
o
o
o
o
o oo
o o
o
o o
o
o
o
o
o
o
o
o
o o o
o
o
o
o
o
o oooo o o
−100
o
o o o
o o o o
o
o o o
t−statistics −2 0 2
o
0
10
20
30 time (ms)
40
50
60
FIGURE 3.8 Motorcycle data, plots of t-statistics (bottom), and the partial spline fit with 95% Bayesian confidence intervals (top).
There are three clusters of large t-statistics that suggests three potential change points in the first derivative at t1 = 0.2128, t2 = 0.3666, and t3 = 0.5113, respectively (see also Speckman (1995)). So we fit the
Smoothing Parameter Selection and Inference
81
following partial spline model yi =
3 X j=1
βj (xi − tj )+ + f (xi ) + ǫi ,
i = 1, . . . , n,
(3.53)
where x is the variable times scaled into [0, 1], tj are the change-points in the first derivative, and f ∈ W22 [0, 1]. We fit model (3.53) and compute posterior means and standard deviations as follows: > > > > > >
t1 <- .2128; t2 <- .3666; t3 <- .5113 s1 <- (x-t1)*(x>t1); s2 <- (x-t2)*(x>t2) s3 <- (x-t3)*(x>t3) mcycle.ps.fit2 <- ssr(accel~x+s1+s2+s3, rk=cubic(x)) grid <- seq(0,1,len=100) mcycle.ps.pred2 <- predict(mcycle.ps.fit2, newdata=data.frame(x=grid, s1=(grid-t1)*(grid>t1), s2=(grid-t2)*(grid>t2), s3=(grid-t3)*(grid>t3)))
The fit and Bayesian confidence intervals are shown in Figure 3.8. For the Arosa data, Figure 2.12 in Chapter 2 shows the estimates and 95% confidence intervals for the overall function and its decomposition. In particular, the posterior means and standard deviations for the periodic spline were calculated as follows: > arosa.per.fit <- ssr(thick~1, rk=periodic(x)) > grid <- data.frame(x=seq(.5/12,11.5/12,length=50)) > arosa.per.pred <- predict(arosa.per.fit,grid, terms=matrix(c(1,0,0,1,1,1),nrow=3,byrow=T)) where the input for the term argument is a 3 × 2 matrix with the first row (1,0) specifying the parametric component, the second row (0,1) specifying the smooth component, and the third row (1,1) specifying the overall function.
3.8.2
Bootstrap Confidence Intervals
Consider the general SSR model (2.10). Let fˆ and σ ˆ 2 be the estimates 2 of f and σ , respectively. Let ∗ yi,b = Li fˆ + ǫ∗i,b ,
i = 1, . . . , n; b = 1, . . . , B iid
(3.54)
be B bootstrap samples where ǫ∗i,b ∼ N(0, σ ˆ 2 ). The random errors ǫ∗i,b may also be drawn from residuals with replacement when the normality assumption is undesirable. Let fˆγ∗ ,b be the estimate of fγ in (3.48) based
82
Smoothing Splines: Methods and Applications
∗ on the bth bootstrap sample {yi,b , i = 1, . . . , n}. For any well-defined functional L0 , there are B bootstrap estimates L0 fˆγ∗ ,b for b = 1, . . . , B. Then the 100(1 − α)% percentile bootstrap confidence interval of L0 fγ is
(L0 fˆγ ,L , L0 fˆγ ,U ),
(3.55)
where L0 fˆγ ,L and L0 fˆγ ,U are the lower and upper α/2 quantiles of {L0 fˆγ∗ ,b , b = 1, . . . , B}. The percentile-t bootstrap confidence intervals can be constructed as follows. Let τ = (L0 fˆγ − L0 fγ )/ˆ σ , where division by σ ˆ is introduced ∗ ∗ ˆ to reduce the dependence on σ. Let τb = (L0 fγ ,b − L0 fˆγ )/ˆ σb∗ be the ∗ bootstrap estimates of τ for b = 1, . . . , B, where σ ˆb is the estimate of σ based on the bth bootstrap sample. Then the 100(1 − α)% percentile-t bootstrap confidence interval of L0 fγ is (L0 fˆγ − q1− α2 σ ˆ , L0 fˆγ − q α2 σ ˆ ),
(3.56)
where q α2 and q1− α2 are the lower and upper α/2 quantiles of {τb∗ , b = 1, . . . , B}. Note that the bounded linear condition is not required for L0 in the above construction of bootstrap confidence intervals. Other forms of bootstrap confidence intervals and comparison between Bayesian and bootstrap approaches can be found in Wang and Wahba (1995). For example, the 95% percentile bootstrap confidence intervals in Figure 2.8 in Chapter 2 were computed as follows: > nboot <- 9999 > fb <- NULL > for (i in 1:nboot) { yb <- canada.fit1$fit + sample(canada.fit1$resi, 35, replace=T) bfit <- ssr(yb~s, rk=Sigma, spar=‘‘m’’) fb <- cbind(fb, bfit$coef$d[2]+S%*%bfit$coef$c) } > lb <- apply(fb, 1, quantile, prob=.025) > ub <- apply(fb, 1, quantile, prob=.975) where random errors for bootstrap samples were drawn from residuals with replacement, and lb and ub represent lower and upper bounds. We use the following simulation to show the performance of Bayesian and bootstrap confidence intervals. Observations are generated from model (1.1) with f (x) = exp{−64(x − 0.5)2 }, σ = 0.1, xi = i/n for i = 1, . . . , n, and n = 100. We fit a cubic spline and construct 95% Bayesian, percentile bootstrap (denoted as Per), and percentile-t bootstrap (denoted as T) confidence intervals. We set B = 1000 and repeat
Smoothing Parameter Selection and Inference
83
the simulation 100 times. Figures 3.9(a)(b)(c) show average pointwise coverages for the Bayesian, percentile bootstrap, and percentile-t bootstrap confidence intervals, respectively. The absolute value of f ′′ is also plotted to show the curvature of the function. The pointwise coverage is usually smaller than the nominal value at high curvature points. Boxplots of the across-the-function coverages for these three methods are shown in Figure 3.9(d). The average and median ACPs are close to the nominal value.
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
x
x
(c)
(d)
+
0.8
+
+
Per
T
1.0
0.65
average coverage 0.75 0.85 0.95
pointwise coverage 0.0 0.2 0.4 0.6 0.8 1.0
0.0
(b)
pointwise coverage 0.0 0.2 0.4 0.6 0.8 1.0
pointwise coverage 0.0 0.2 0.4 0.6 0.8 1.0
(a)
0.0
0.2
0.4
0.6
0.8
1.0
Bayes
x
FIGURE 3.9 Plots of pointwise coverages (solid line) and nominal value (dotted line) for (a) the Bayesian confidence intervals, (b) the percentile bootstrap confidence intervals, and (c) the percentile-t bootstrap confidence intervals. Dashed lines in (a), (b), and (c) represent a scaled version of |f ′′ |. Boxplots of ACPs are shown in (d) with mean coverages marked as pluses and nominal value plotted as a dotted line.
84
3.9 3.9.1
Smoothing Splines: Methods and Applications
Hypothesis Tests The Hypothesis
One of the most useful applications of the nonparametric regression models is to check or suggest a parametric model. When appropriate, parametric models, especially linear models, are preferred in practice because of their simplicity and interpretability. One important step in building a parametric model is to investigate potential departure from a specified model. Tests with specific alternatives are often performed in practice. This kind of tests would not perform well for different forms of departure from the parametric model, especially those orthogonal to the specific alternative. For example, to detect departure from a straight line model, one may consider a quadratic polynomial as the alternative. Then departure in the form of higher-order polynomials may be missed. It is desirable to have tools that can detect general departures from a specific parametric model. One approach to check a parametric model is to construct confidence intervals for f or its smooth component in an SSR model using methods in Section 3.8. The parametric model may be deemed unsuitable if a larger portion of its estimate is outside the confidence intervals of f . When the null space H0 corresponds to the parametric model under consideration, one may check the magnitude of the estimate of the smooth component P1 f since it represents the remaining systematic variation not explained by the parametric model under consideration. If a larger portion of the confidence intervals for P1 f does not contain zero, the parametric model may be deemed unsuitable. This approach was illustrated using the geyser data in Section 3.8.1. Often the confidence intervals are all one needs in practice. Nevertheless, sometimes it may be desirable to conduct a formal test on the departure from a parametric model. In this section we consider the following hypothesis H0 : f ∈ H0 ,
H1 : f ∈ H and f ∈ / H0 ,
(3.57)
where H0 corresponds to the parametric model. Note that when the parametric model satisfies Lf = 0 with a differential operator L given in (2.53), the L-spline may be used to check or test the parametric model. The alternative is equivalent to ||P1 f || > 0. Note that λ = ∞ in (2.11), or equivalently, δ = 0 in the corresponding Bayes model (3.25) leads to f ∈ H0 . Thus the hypothesis (3.57) can be reexpressed as H0 : λ = ∞,
H1 : λ < ∞,
(3.58)
Smoothing Parameter Selection and Inference
85
or H0 : δ = 0,
3.9.2
H1 : δ > 0.
(3.59)
Locally Most Powerful Test
Consider the hypothesis (3.59). As in Section 3.8, we use the marginal log-likelihood of w1 , where w 1 = QT2 y ∼ N(0, δQT2 M Q2 ). Since Q2 is orthogonal to T , it is clear that the transformation QT2 y eliminates contribution from the model under the null hypothesis. Thus w1 reflects signals, if any, from H1 . Note that M = Σ + nλI and QT2 ΣQ2 = U EU T , where notations Q2 , U , and E were defined in Section 3.2. Let z , U T w1 and denote z = (z1 , . . . , zn−p )T . Then z ∼ N(0, δE + σ 2 I). First assume that σ 2 is known. Note that λ = σ 2 /nδ. The negative loglikelihood of z l(δ|z) =
n−p n−p 1X 1X zν2 log(δeν + σ 2 ) + + C1 , 2 ν=1 2 ν=1 δeν + σ 2
(3.60)
where C1 is a constant. Let Uδ (δ) and Iδδ (δ) be the score and Fisher information of δ. It is not difficult to check that the score test statistic (Cox and Hinkley 1974) ! n−p X Uδ (0) 2 = C2 eν zν + C3 , tscore , p Iδδ (0) ν=1
where C2 and C3 are constants. Therefore, the score test is equivalent to the following test statistic tLMP =
n−p X
eν zν2 .
(3.61)
ν=1
For polynomial splines, Cox, Koh, Wahba and Yandell (1988) showed that there is no uniformly most powerful test and that tLMP is the locally most powerful test. The variance is usually unknown in practice. σ 2 by its Pn−pReplacing 2 2 MLE under the null hypothesis (3.59), σ ˆ0 = ν=1 zν /(n − p), leads to the approximate LMP test statistic Pn−p eν zν2 tappLMP = Pν=1 (3.62) n−p 2 . ν=1 zν
The null hypothesis is rejected for large values of tappLMP . The test statistic tappLMP does not follow a simple distribution under H0 . Nevertheless, it is straightforward to simulate the null distribution. Under
86
Smoothing Splines: Methods and Applications iid
H0 , zν ∼ N(0, σ 2 ). Without loss of generality, σ 2 can be set to one in the simulation for the null distribution since both the numerator and iid the denominator depend on zν2 . Specifically, samples zν,j ∼ N(0, 1) for ν = 1, . . . , nP− p and j = 1, . . . , N are generated, and the statistics Pn−p n−p 2 2 tappLMP,j = ν=1 eν zν,j / ν=1 zν,j are computed. Note that tappLMP,j are N realizations of the statistic under H0 . Then the proportion that tappLMP,j is greater than tappLMP provides an estimate of the p-value. This approach usually requires a very large N . The p-value can also be calculated numerically using the algorithm in Davies (1980). The approximation method is very fast and agrees with the results from the Monte Carlo method (Liu and Wang 2004).
3.9.3
Generalized Maximum Likelihood Test
Consider the hypothesis (3.58). Since z ∼ N(0, δ(E + nλI)), the MLE of δ is n−p 1 X zν2 δˆ = . n − p ν=1 eν + nλ From (3.39), the profile likelihood of λ is ˆ =C Lp (λ, δ|z)
(P n−p
2 ν=1 zν /(eν + nλ) Qn−p 1 − n−p ν=1 (eν + nλ)
)− n−p 2
,
(3.63)
ˆGML , is the maximizer where C is a constant. The GML estimate of λ, λ of (3.63). The GML test statistic for the hypothesis (3.58) is tGML
2 )− n−p ˆ GML |z) Lp (λ , Lp (∞|z) Pn−p 2 ˆ 1 ν=1 zν /(eν + nλGML ) = Qn−p Pn−p 2 . 1 − n−p ˆ (eν + nλGML ) ν=1 zν
(
(3.64)
ν=1
It is clear that the GML test is equivalent to the ratio of restricted likelihoods. The null hypothesis is rejected when tGML is too small. The standard theory for likelihood ratio tests does not apply because the parameter λ locates on the boundary of the parameter space under the null hypothesis. Thus, it is difficult to derive the null distribution for tGML . The Monte Carlo method described for the LMP test can be adapted to compute an estimate of the p-value. Note that tGML involves ˆGML needs the GML estimate of the smoothing parameter. Therefore, λ to be estimated for each simulation sample, which makes this approach computationally intensive.
Smoothing Parameter Selection and Inference
87
The null distribution of −(n−p) log tGML can be well approximated by a mixture of χ21 and χ20 , denoted by rχ20 + (1 − r)χ21 . However, the ratio r is not fixed. It is difficult to derive a formula for r since it depends on many factors. One can approximate the ratio r first and then calculate the p-value based on the mixture of χ21 and χ20 with the approximated r. A relatively small sample size is required to approximate r. See Liu and Wang (2004) for details.
3.9.4
Generalized Cross-Validation Test
ˆ GCV be the GCV estimate of λ. Consider the hypothesis (3.58). Let λ Similar to the GML test statistic, the GCV test statistic is defined as the ratio between GCV scores ˆ GCV ) GCV(λ GCV(∞) Pn−p 2 ˆ GCV )2 zν /(1 + eν /nλ 1 2 = (n − p) Pν=1 Pn−p 2 . n−p 2 ˆ { ν=1 1/(1 + eν /nλGCV )} ν=1 zν
tGCV ,
(3.65)
H0 is rejected when tGCV is too small. Again, similar to the GML test, the Monte Carlo method can be used to compute an estimate of the p-value.
3.9.5
Comparison and Implementation
The LMP and GML tests derived based on Bayesian arguments perform well under deterministic models. In terms of eigenvectors of QT2 ΣQ2 , the LMP test is more powerful in detecting departure in the direction of the first eigenvector, the GML test is more powerful in detecting departure in low-frequencies, and the GCV test is more powerful in detecting departure in high frequencies. See Liu and Wang (2004) for more details. These tests can be carried out using the anova function in the assist package. We now use the geyser and Arosa data to illustrate how to use this function. For the geyser data, first consider the following hypotheses H0 : f ∈ span{1, x},
/ span{1, x}. H1 : f ∈ W22 [0, 1] and f ∈
We have fitted cubic spline model where the null space corresponds to the model under H0 . Therefore, we can test the hypothesis as follows: > anova(geyser.cub.fit, simu.size=500)
88
Smoothing Splines: Methods and Applications
Testing H_0: f in the NULL space
LMP GCV
test.value simu.size simu.p-value 0.02288057 500 0 0.00335470 500 0
where the option simu.size specifies the Monte Carlo sample size N . The simple linear model is rejected. Next we consider the hypothesis H0 : f ∈ span{1, x, s},
H1 : f ∈ W22 [0, 1] and f ∈ / span{1, x, s},
where s = (x − 0.397)0+ . We have fitted a partial spline model where the null space corresponds to the model under H0 . Therefore, we can test the hypothesis as follows: > anova(geyser.ps.fit, simu.size=500) Testing H_0: f in the NULL space
LMP GCV
test.value simu.size simu.p-value 0.000351964 500 0.602 0.003717477 500 0.602
The null hypothesis is not rejected. The conclusions are the same as those based on Bayesian confidence intervals. To apply the GML test, we need to first fit using the GML method to select the smoothing parameter: > geyser.ps.fit.m <- ssr(waiting~x+s, rk=cubic(x), spar=‘‘m’’) > anova(geyser.ps.fit.m, simu.size=500) Testing H_0: f in the NULL space
LMP GML
test.value simu.size simu.p-value approximate.p-value 0.000352 500 0.634 1.000001 500 0.634 0.5
where the approximate.p-value was computed using the mixture of two Chi-square distributions. For the Arosa data, consider the hypothesis H0 : f ∈ P,
H1 : f ∈ W22 (per) and f ∈ / P,
where P = span{1, sin 2πx, cos 2πx} is the model space for the sinusoidal model. Two approaches can be used to test the above hypothesis: fit a partial spline and fit an L-spline:
Smoothing Parameter Selection and Inference
89
> arosa.ps.fit <- ssr(thick~sin(2*pi*x)+cos(2*pi*x), rk=periodic(x), data=Arosa) > anova(arosa.ps.fit,simu.size=500) Testing H_0: f in the NULL space test.value simu.size simu.p-value LMP 0.001262064 500 0 GCV 0.001832394 500 0 > arosa.ls.fit <- ssr(thick~sin(2*pi*x)+cos(2*pi*x), rk=lspline(x,type=‘‘sine1’’)) > anova(arosa.ls.fit,simu.size=500) Testing H_0: f in the NULL space test.value simu.size simu.p-value LMP 2.539163e-06 500 0 GCV 0.001828071 500 0 The test based on the L-spline is usually more powerful since the parametric and the smooth components are orthogonal.
This page intentionally left blank
Chapter 4 Smoothing Spline ANOVA
4.1
Multiple Regression
Consider the problem of building regression models that examine the relationship between a dependent variable y and multiple independent variables x1 , . . . , xd . For generality, let the domain of each xk be an arbitrary set Xk . Denote x = (x1 , . . . , xd ). Given observations (xi , yi ) for i = 1, . . . , n, where xi = (xi1 , . . . , xid ), a multiple regression model relates the dependent variable and independent variables as follows: yi = f (xi ) + ǫi ,
i = 1, . . . , n,
(4.1)
where f is a multivariate regression function, and ǫi are zero-mean independent random errors with a common variance σ 2 . The goal is to construct a model for f and estimate it based on noisy data. There exist many different methods to construct a model space for f , parametrically, semiparametrically, or nonparametrically. For example, a thin-plate spline model may be used when all xk are univariate continuous variables, and partial spline models may be used if a linear parametric model can be assumed for all but one variable. This chapter introduces a nonparametric approach called smoothing spline analysis of variance (smoothing spline ANOVA or SS ANOVA) decomposition for constructing model spaces for the multivariate function f . The multivariate function f is defined on the product domain X = X1 ×X2 ×· · ·×Xd . Note that each Xk is arbitrary: it may be a continuous interval, a discrete set, a unit circle, a unit sphere, or Rd . Construction of model spaces for a single variable was introduced in Chapter 2. Let H(k) be an RKHS on Xk . The choice of the marginal space H(k) depends on the domain Xk and prior knowledge about f as a function of xk . To model the joint function f , we start with the tensor product of these marginal spaces defined in the following section.
91
92
4.2
Smoothing Splines: Methods and Applications
Tensor Product Reproducing Kernel Hilbert Spaces
First consider the simple case when d = 2. Denote RKs for H(1) and H(2) as R(1) and R(2) , respectively. It is known that the product of nonnegative definite functions is nonnegative definite (Gu 2002). As RKs, both R(1) and R(2) are nonnegative definite. Therefore, the bivariate function on X = X1 × X2 R((x1 , x2 ), (z1 , z2 )) , R(1) (x1 , z1 )R(2) (x2 , z2 ) is nonnegative definite. By Moore–Aronszajn theorem, there exists a unique RKHS H on X = X1 × X2 such that R is its RK. The resulting RKHS H is called the tensor product RKHS and is denoted as H(1) ⊗H(2) . For d > 2, the tensor product RKHS of H(1) , . . . , H(d) on the product domain X = X1 × X2 × · · · × Xd , H(1) ⊗ H(2) ⊗ · · · ⊗ H(d) , is defined recursively. Note that the RK for a tensor product space equals the product of RKs of marginal spaces. That is, the RK of H(1) ⊗ H(2) ⊗ · · · ⊗ H(d) equals R(x, z) = R(1) (x1 , z1 )R(2) (x2 , z2 ) · · · R(d) (xd , zd ), where x ∈ X , z = (z1 , . . . , zd ) ∈ X , X = X1 × X2 × · · · × Xd , and R(k) is the RK of H(k) for k = 1, . . . , d. For illustration, consider the ultrasound data consisting of tongue shape measurements over time from ultrasound imaging. The data set contains observations on the response variable height (y) and three independent variables: environment (x1 ), length (x2 ), and time (x3 ). The variable x1 is a factor with three levels: x1 = 1, 2, 3 corresponding to 2words, cluster, and schwa, respectively. Both continuous variables x2 and x3 are scaled into [0, 1]. Interpolations of the raw data are shown in Figure 4.1. In linguistic studies, researchers want to determine (1) how tongue shapes for an articulation differ under different environments, (2) how the tongue shape changes as a function of time, and (3) how changes over time differ under different environments. To address the first question at a fixed time point, we need to model a bivariate regression function f (x1 , x2 ). Assume marginal spaces R3 and W2m [0, 1] for variables x1 and x2 , respectively. Then we may consider the tensor product space R3 ⊗ W2m [0, 1] for the bivariate function f . To address the second question, for a fixed environment, we need to model a bivariate regression function f (x2 , x3 ). Assume marginal spaces W2m1 [0, 1] and W2m2 [0, 1] for variables
Smoothing Spline ANOVA 70
65 60 55 50 45 40
60 55 50 45 40
2words 120
140
50 45
0
80
100
120
140
(mm) length
150 100 50
s) (m
le
m) ngth (m
55
e tim
100
60
schwa200
150 100 50
s) (m
s) (m
80
e tim
e tim
0
65
40
cluster 200
200 150 100 50
93
70
65
height (mm)
height (mm)
height (mm)
70
0
80
100
FIGURE 4.1 Ultrasound data, 3-d plots of observations.
x2 and x3 , respectively. Then we may consider the tensor product space W2m1 [0, 1] ⊗ W2m2 [0, 1] for the bivariate function. To address the third question, we need to model a trivariate regression function f (x1 , x2 , x3 ). We may consider the tensor product space R3 ⊗ W2m1 [0, 1] ⊗ W2m2 [0, 1] for the trivariate function. Analysis of the ultrasound data are given in Section 4.9.1. The SS ANOVA decomposition decomposes a tensor product space into subspaces with a hierarchical structure similar to the main effects and interactions in the classical ANOVA. The resulting hierarchical structure facilitates model selection and interpretation. Sections 4.3, 4.4, and 4.5 present SS ANOVA decompositions for a single space, tensor product of two spaces, and tensor product of d spaces. More SS ANOVA decompositions can be found in Sections 4.9, 5.4.4, 6.3, and 9.2.4.
4.3
120
140
)
(mm length
One-Way SS ANOVA Decomposition
SS ANOVA decompositions of tensor product RKHS’s are based on decompositions of marginal spaces for each independent variable. Therefore, decompositions for a single space are introduced first in this section. Spline models for a single independent variable have been introduced in Chapter 2. Denote the independent variable as x and the regression
94
Smoothing Splines: Methods and Applications
function as f . It was assumed that f belongs to an RKHS H. The function f was decomposed into a parametric and a smooth component, f = f0 + f1 , or in terms of model space, H = H0 ⊕ H1 . This can be regarded as one form of the SS ANOVA decomposition. We now introduce general decompositions based on averaging operators. Consider a function space H on the domain X . An operator A is called an averaging operator if A = A2 . Instead of the more appropriate term as an idempotent operator, the term averaging operator is used since it is motivated by averaging in the classical ANOVA decomposition. Note that an averaging operator does not necessarily involve averaging. As we will see in the following subsections, the commonly used averaging operators are projection operators. Thus they are idempotent. Suppose model space H = H0 ⊕ H1 , where H0 is a finite dimensional space with orthogonal basis φ1 (x), . . . , φp (x). Let Aν be the projection operator onto the subspace {φν (x)} for ν = 1, . . . , p, and Ap+1 be the projection operator onto H1 . Then the function can be decomposed as f = (A1 + · · · + Ap + Ap+1 )f , f01 + · · · + f0p + f1 .
(4.2)
Correspondingly, the model space is decomposed into H = {φ1 (x)} ⊕ · · · ⊕ {φp (x)} ⊕ H1 . For simplicity, {·} represents the space spanned by the basis functions inside the bracket. Some averaging operators Aν (subspaces) can be combined. For example, combining A1 , . . . , Ap leads to the decomposition f = f0 + f1 , a parametric component plus a smooth component. When φ1 (x) = 1, combining A2 , . . . , Ap+1 leads to the decomposition f = f01 + f˜1 , where f01 is a constant independent of x, and f˜1 = f02 + · · · + f0p + f1 collects all components that depend on x. Therefore, f = f01 + f˜1 decomposes the function into a constant plus a nonconstant function. For the same model space, different SS ANOVA decompositions may be constructed for different purposes. In general, we denote the one-way SS ANOVA decomposition as f = A1 f + · · · + Ar f,
(4.3)
where A1 + · · · + Ar = I, and I is the identity operator. The above equality always holds since f = If . Equivalently, in terms of the model space, the one-way SS ANOVA decomposition is denoted as H = H(1) ⊕ · · · ⊕ H(r) . The following subsections provide one-way SS ANOVA decompositions for some special model spaces.
Smoothing Spline ANOVA
4.3.1
95
Decomposition of Ra : One-Way ANOVA
Suppose x is a discrete variable with a levels. The classical one-way mean model assumes that yik = µi + ǫik ,
i = 1, . . . , a; k = 1, . . . , ni ,
(4.4)
where yik represents the observation of the kth replication at level i of x, µi represents the mean at level i, and ǫik represent random errors. Regarding µi as a function of i and expressing explicitly as f (i) , µi , then f is a function defined on the discrete domain X = {1, . . . , a}. It is easy to see that the model space for f is the Euclidean a-space Ra . Model space construction and decomposition of Ra The space Ra is an RKHS with the inner product (f, g) = f T g. Furthermore, Ra = H0 ⊕ H1 , where H0 = {f : f (1) = · · · = f (a)}, a X H1 = {f : f (i) = 0},
(4.5)
i=1
are RKHS’s with corresponding RKs R0 (i, j) =
1 , a
1 R1 (i, j) = δi,j − , a
(4.6)
and δi,j is the Kronecker delta. Details about the above construction can be found in Gu (2002). Define an averaging operator A1 : Ra → Ra such that a
1X A1 f = f (i). a i=1 The operator A1 maps f to the constant function that equals the average over all indices. Let A2 = I − A1 . The one-way ANOVA effect model is based on the following decomposition of the function f : f = A1 f + A2 f , µ + αi ,
(4.7)
where µ is the overall mean, and αi is the effect at level i. From the definition Pa of the averaging operator, αi satisfy the sum-to-zero side condition i=1 αi = 0. It is clear that A1 and A2 are the projection operators
96
Smoothing Splines: Methods and Applications
from Ra onto H0 and H1 defined in (4.5). Thus, they divide Ra into H0 and H1 . Pa Under the foregoing construction, ||P1 f ||2 = i=1 {f (i) − f¯}2 , where P a f¯ = i=1 f (i)/a. For balanced designs with ni = n, the solution to the PLS (2.11) is aλ fˆ(i) = y¯i· − (¯ yi· − y¯·· ), (4.8) 1 + aλ Pn Pa where y¯i· = ¯·· = ¯i· /a. k=1 yik /n, and y i=1 y PaIt is easy to check that when n = 1, σ 2 = 1 and λ = (a − 3)/a( i=1 y¯i·2 − a + 3), the spline estimate fˆ is the James–Stein estimator (shrink toward mean). Therefore, in a sense, the Stein’s shrinkage estimator can be regarded as spline estimate on a discrete domain. There exist other ways to decompose Ra . For example, the averaging operator A1 f = f (1) leads to the same decomposition as (4.7) with the set-to-zero side condition: α1 = 0 (Gu 2002).
4.3.2
Decomposition of W2m [a, b]
Under the construction in Section 2.2, let Aν f (x) = f (ν−1) (a)
(x − a)ν−1 , (ν − 1)!
ν = 1, . . . , m.
(4.9)
It is easy to see that A2v = Av . Thus, they are averaging operators. In fact, Aν is the projection operator onto {(x − a)ν−1 /(ν − 1)!}. Let Am+1 = I − A1 − · · · − Am . The decomposition f = A1 f + · · · + Am f + Am+1 f corresponds to the Taylor expansion (1.5). It decomposes the model space W2m [a, b] into W2m [a, b] = {1} ⊕ {x − a} ⊕ · · · ⊕ {(x − a)m−1 /(m − 1)!} ⊕ H1 , where H1 is given in (2.3). For f ∈ H1 , conditions f (ν) (a) = 0 for ν = 0, . . . , m−1 are analogous to the set-to-zero condition in the classical one-way ANOVA model. Under the construction in Section 2.6 for W2m [0, 1], let Z 1 (ν−1) Aν f (x) = f (u)du kν (x), ν = 1, . . . , m. 0
Again, Aν is an averaging (projection) operator extracting the polynoR1 mial of order ν. In particular, A1 f = 0 f (u)du is a natural extension
Smoothing Spline ANOVA
97
of the averaging in the discrete domain. Let Am+1 = I − A1 − · · · − Am . The decomposition f = A1 f + · · · + Am f + Am+1 f decomposes the model space W2m [0, 1] into W2m [0, 1] = {1} ⊕ {k1 (x)} ⊕ · · · ⊕ {km−1 (x)} ⊕ H1 , R1 where H1 is given in (2.29). For f ∈ H1 , conditions 0 f (ν) dx = 0 for ν = 0, . . . , m − 1 are analogous to the sum-to-zero condition in the classical one-way ANOVA model.
4.3.3
Decomposition of W2m (per)
Under the construction in Section 2.7, let Z 1 A1 f = f du 0
be an averaging (projection) operator. Let A2 = I − A1 . The decomposition f = A1 f + A2 f decomposes the model space W2m (per) = 1 ⊕ f ∈ W2m (per) :
4.3.4
Z
0
1
f du = 0 .
Decomposition of W2m (Rd )
For simplicity, consider the special case with d = 2 and m = 2. Decompositions for general d and m can be derived similarly. Let φ1 (x) = 1, φ2 (x) = x1 , and φ3 (x) = x2 be polynomials of total degree less than m = 2. Let φ˜1 = 1, and φ˜2 and φ˜3 be an orthonormal basis such that (φ˜ν , φ˜µ )0 = δν,µ based on the norm (2.41). Define two averaging operators A1 f (x) = A2 f (x) =
J X
wj f (uj ),
j=1
J X j=1
(4.10) wj f (uj ) φ˜2 (uj )φ˜2 (x) + φ˜3 (uj )φ˜3 (x) ,
98
Smoothing Splines: Methods and Applications
where uj are fixed points in R2 , and wj are fixed positive weights such PJ that j=1 wj = 1. It is clear that A1 and A2 are projection operators onto spaces {φ˜1 } and {φ˜2 , φ˜3 }, respectively. To see how they generalize averaging operators, define a probability measure µ on X = R2 by assigning probability wj to the point uj , j = 1, . . . , J. Then A1 f (x) = R R R f φ˜1 dµ and A2 f (x) = ( R2 f φ˜2 dµ)φ˜2 (x) + ( R2 f φ˜3 dµ)φ˜3 (x). ThereR2 fore, A1 and A2 take averages with respect to the discrete probability measure µ. In particular, µ puts mass 1/n on design points xj when J = n, wj = 1/n and uj = xj . A continuous density on R2 may be used. However, the resulting integrals usually do not have closed forms, and approximations such as a quadrature formula would have to be used. It is then essentially equivalent to using an approximate discrete probability measure. Let A3 = I − A1 − A2 . The decomposition f = A1 f + A2 f + A3 f divides the model space W2m (R2 ) = 1 ⊕ φ˜2 , φ˜3 ⊕ f ∈ W2m (R2 ) : J22 (f ) = 0 , d where Jm (f ) is defined in (2.36).
4.4
Two-Way SS ANOVA Decomposition
Suppose there are two independent variables, x1 ∈ X1 and x2 ∈ X2 . Consider the tensor product space H(1) ⊗ H(2) on the product domain X1 × X2 . For f as a marginal function of xk , assume the following one-way decomposition based on Section 4.3, (k)
f = A1 f + · · · + A(k) rk f, (k)
k = 1, 2,
where Aj are averaging operators on H(k) and for the joint function, we have
P rk
j=1
(4.11) (k)
Aj
= I. Then
r1 X r2 X (1) (2) (1) (2) (2) f = A1 +· · ·+A(1) A +· · ·+A f = Aj1 Aj2 f. (4.12) r1 r2 1 j1 =1 j2 =1
The above decomposition of the bivariate function f is referred to as the two-way SS ANOVA decomposition.
Smoothing Spline ANOVA
99
Denote (k)
(k)
H(k) = H(1) ⊕ · · · ⊕ H(rk ) ,
k = 1, 2,
as the one-way decomposition to H(k) associated with (4.11). Then, (4.12) decomposes the tensor product space n o n o (1) (1) (2) (2) H(1) ⊗ H(2) = H(1) ⊕ · · · ⊕ H(r1 ) ⊗ H(1) ⊕ · · · ⊕ H(r2 ) =
r1 X r2 X
j1 =1 j2 =1
(1)
(2)
H(j1 ) ⊗ H(j2 ) .
Consider the special case when rk = 2 for k = 1, 2. Assume that (k) is independent of xk , or equivalently, H0 = {1}. Then the decomposition (4.12) can be written as (k) A1 f
(1)
(2)
(1)
(2)
(1)
(2)
(1)
(2)
f = A1 A1 f + A2 A1 f + A1 A2 f + A2 A2 f , µ + f1 (x1 ) + f2 (x2 ) + f12 (x1 , x2 ),
(4.13)
where µ represents the grand mean, f1 (x1 ) and f2 (x2 ) represent the main effects of x1 and x2 , respectively, and f12 (x1 , x2 ) represents the interaction between x1 and x2 . (k) For general rk , assuming A1 f is independent of xk , the decomposi(k) (k) tion (4.13) can be derived by combining operators A2 , . . . , Ark into (k) (k) (k) one averaging operator A˜2 = A2 + · · · + Ark . Therefore, decomposition (4.13) combines components in (4.12) and reorganizes them into the overall main effects and interactions. The following subsections provide two-way SS ANOVA decompositions for combinations of some special model spaces.
4.4.1
Decomposition of Ra ⊗ Rb : Two-Way ANOVA
Suppose both x1 and x2 are discrete variables with a and b levels, respectively. The classical two-way mean model assumes that yijk = µij + ǫijk ,
i = 1, . . . , a; j = 1, . . . , b; k = 1, . . . , nij ,
where yijk represents the observation of the kth replication at level i of x1 , and level j of x2 , µij represents the mean at level i of x1 and level j of x2 , and ǫijk represent random errors. Regarding µij as a bivariate function of (i, j) and letting f (i, j) , µij , then f is a bivariate function defined on the product domain X = {1, . . . , a}×{1, . . . , b}. The model space for f is the tensor product space
100
Smoothing Splines: Methods and Applications (1)
Ra ⊗ Rb . Define two averaging operators A1
(2)
and A2
such that
a
(1)
A1 f =
1X f (i, j), a i=1 b
(2)
A1 f = (1)
1X f (i, j). b j=1
(2)
A1 and A1 map f to univariate functions by averaging over all levels (1) (1) (2) (2) of x1 and x2 , respectively. Let A2 = I −A1 and A2 = I −A1 . Then the classical two-way ANOVA effect model is based on the following SS ANOVA decomposition of the function f : (1)
(1)
(2)
(2)
f = (A1 + A2 )(A1 + A2 )f (1)
(2)
(1)
(2)
(1)
(2)
(1)
(2)
= A1 A1 f + A2 A1 f + A1 A2 f + A2 A2 f , µ + αi + βj + (αβ)ij ,
(4.14)
where µ represents the overall mean, αi represents the main effect of x1 , βj represents the main effect of x2 , and (αβ)ij represents the interaction between x1 and x2 . The sum-to-zero side conditions are satisfied from the definition of the averaging operators. Based on one-way ANOVA decomposition in Section 4.3.1, we have (1) (1) (2) (2) (1) (2) Ra = H0 ⊕ H1 and Rb = H0 ⊕ H1 , where H0 and H0 are (1) (2) subspaces containing constant functions, and H1 and H1 are orthog(1) (2) onal complements of H0 and H0 , respectively. The classical two-way ANOVA model decomposes the tensor product space n o n o (1) (1) (2) (2) Ra ⊗ Rb = H0 ⊕ H1 ⊗ H0 ⊕ H1 n o n o (1) (2) (1) (2) = H0 ⊗ H0 ⊕ H1 ⊗ H0 n o n o (1) (2) (1) (2) ⊕ H0 ⊗ H1 ⊕ H1 ⊗ H1 . (4.15)
The four subspaces in (4.15) contain components µ, αi , βj , and (αβ)ij in (4.14), respectively.
4.4.2
Decomposition of Ra ⊗ W2m [0, 1]
Suppose x1 is a discrete variable with a levels, and x2 is a continuous variable in [0, 1]. A natural model space for x1 is Ra and a natural model space for x2 is W2m [0, 1]. Therefore, we consider the tensor product space Ra ⊗ W2m [0, 1] for the bivariate regression function f (x1 , x2 ). For
Smoothing Spline ANOVA
101
simplicity, we derive SS ANOVA decompositions for m = 1 and m = 2 only. SS ANOVA decompositions for higher-order m can be derived similarly. In this and the remaining sections, the construction in Section 2.6 for the marginal space W2m [0, 1] will be used. Similar SS ANOVA decompositions can be derived under the construction in Section 2.2. Consider the tensor product space Ra ⊗ W21 [0, 1] first. Define two (1) (2) averaging operators A1 and A1 as a 1 X f, a x =1 1 Z 1 (2) A1 f = f dx2 , (1)
A1 f =
0
(1)
(2)
where A1 and A1 extract the constant term out of all possible func(1) (1) (2) (2) tions for each variable. Let A2 = I − A1 and A2 = I − A1 . Then (1) (1) (2) (2) f = A1 + A2 A1 + A2 f (1)
(2)
(1)
(2)
(1)
(2)
(1)
(2)
= A1 A1 f + A2 A1 f + A1 A2 f + A2 A2 f , µ + f1 (x1 ) + f2 (x2 ) + f12 (x1 , x2 ).
(4.16)
Obviously, (4.16) is a natural extension of the classical two-way ANOVA decomposition (4.14) from the product of two discrete domains to the product of one discrete and one continuous domain. As in the classical ANOVA model, components in (4.16) have nice interpretations: µ represents the overall mean, f1 (x1 ) represents the main effect of x1 , f2 (x2 ) represents the main effect of x2 , and f12 (x1 , x2 ) represents the interaction. Components also have nice interpretation collectively: µ + f2 (x2 ) represents the mean curve among all levels of x1 , and f1 (x1 )+f12 (x1 , x2 ) represents the departure from the mean curve at level x1 . Write Ra = (1) (1) (2) (2) (1) (1) H0 ⊕ H1 and W21 [0, 1] = H0 ⊕ H1 , where H0 and H1 are given R 1 (2) (2) in (4.5), H0 = {1} and H1 = {f ∈ W21 [0, 1] : 0 f du = 0}. Then, in terms of the model space, (4.16) decomposes Ra ⊗ W21 [0, 1] n o n o (1) (1) (2) (2) = H0 ⊕ H1 ⊗ H0 ⊕ H1 n o n o n o n o (1) (2) (1) (2) (1) (2) (1) (2) = H0 ⊗ H0 ⊕ H1 ⊗ H0 ⊕ H0 ⊗ H1 ⊕ H1 ⊗ H1
, H0 ⊕ H1 ⊕ H2 ⊕ H3 .
(4.17)
To fit model (4.16), we need to find basis for H0 and RKs for H1 , H2 , and H3 . It is clear that H0 contains all constant functions. Thus, H0
102
Smoothing Splines: Methods and Applications (1)
is an one-dimensional space with the basis φ(x) = 1. The RKs of H0 (1) (2) (2) and H1 are given in (4.6), and the RKs of H0 and H1 are given in 1 2 3 Table 2.2. The RKs of H , H , and H can be calculated using the fact that the RK of a tensor product space equals the product of RKs of the (1) (2) involved marginal spaces. For example, the RK of H3 = H1 ⊗ H1 equals (δx1 ,z1 − 1/a){k1 (x2 )k1 (z2 ) + k2 (|x2 − z2 |)}. Now suppose we want to model the effect of x2 using the cubic spline space W22 [0, 1]. Consider the tensor product space Ra ⊗ W22 [0, 1]. Define (1) (2) (2) three averaging operators A1 , A1 , and A2 as a 1 X f, a x =1 1 Z 1 (2) A1 f = f dx2 , 0 Z 1 (2) ′ A2 f = f dx2 (x2 − 0.5), (1)
A1 f =
0
(1)
(2)
where A1 and A1 extract the constant function out of all possible (2) functions for each variable, and A2 extracts the linear function for x2 . (1) (1) (2) (2) (2) Let A2 = I − A1 and A3 = I − A1 − A2 . Then (1) (1) (2) (2) (2) f = A1 + A2 A1 + A2 + A3 f (1)
(2)
(1)
(2)
(1)
(2)
= A1 A1 f + A1 A2 f + A1 A3 f (1)
(2)
(1)
(2)
(1)
(2)
+ A2 A1 f + A2 A2 f + A2 A3 f
, µ + β × (x2 − 0.5) + f2s (x2 )
ss + f1 (x1 ) + γx1 × (x2 − 0.5) + f12 (x1 , x2 ),
(4.18)
where µ represents the overall mean, f1 (x1 ) represents the main effect of x1 , β ×(x2 −0.5) represents the linear main effect of x2 , f2s (x2 ) represents the smooth main effect of x2 , γx1 ×(x2 −0.5) represents the smooth–linear ss interaction, and f12 (x1 , x2 ) represents the smooth–smooth interaction. The overall main effect of x2 f2 (x2 ) = β × (x2 − 0.5) + f2s (x2 ), and the overall interaction between x1 and x2 ss f12 (x1 , x2 ) = γx1 × (x2 − 0.5) + f12 (x1 , x2 ).
It is obvious that f2 and f12 are the results of combining averaging (2) (3) operators A2 and A2 . One may look at the components in the overall
Smoothing Spline ANOVA
103
main effects and interactions to decide whether to include them in the model. The first three terms in (4.18) represent the mean curve among all levels of x1 , and the last three terms represent the departure from the mean curve. The simple ANCOVA (analysis of covariance) model with x2 being modeled by a straight line is a special case of (4.18) with ss ss f2s = f12 = 0. Thus, checking whether f2s and f12 are negligible provides (1) (1) a diagnostic tool for the ANCOVA model. Write Ra = H0 ⊕ H1 and (2) (2) (2) (1) (1) W22 [0, 1] = H0 ⊕ H1 ⊕ H2 , where H0 and H1 are given in (4.5), R1 (2) (2) (2) H0 = {1}, H1 = {x2 − 0.5}, and H2 = {f ∈ W22 [0, 1], 0 f du = R1 ′ f du = 0}. Then, in terms of the model space, (4.18) decomposes 0 Ra ⊗ W22 [0, 1] o n o n (2) (2) (2) (1) (1) ⊗ H0 ⊕ H1 ⊕ H2 = H0 ⊕ H1 n o n o n o (1) (2) (1) (2) (1) (2) = H0 ⊗ H0 ⊕ H0 ⊗ H1 ⊕ H0 ⊗ H2 n o n o n o (1) (2) (1) (2) (1) (2) ⊕ H1 ⊗ H0 ⊕ H1 ⊗ H1 ⊕ H1 ⊗ H2 , H0 ⊕ H1 ⊕ H2 ⊕ H3 ⊕ H4 , (1)
(2)
(1)
(2)
(1)
(4.19)
(2)
where H0 = {H0 ⊗ H0 } ⊕ {H0 ⊗ H1 }, H1 = H0 ⊗ H2 , H2 = (1) (2) (1) (2) (1) (2) H1 ⊗ H0 , H3 = H1 ⊗ H1 , and H4 = H1 ⊗ H2 . It is easy to see 0 that H is a two-dimensional space with basis functions φ1 (x) = 1, and φ2 (x) = x2 − 0.5. RKs of H1 , H2 , H3 , and H4 can be calculated from (1) (1) (2) (2) (2) RKs of H0 and H1 given in (4.6) and the RKs of H0 , H1 , and H2 given in Table 2.2.
4.4.3
Decomposition of W2m1 [0, 1] ⊗ W2m2 [0, 1]
Suppose both x1 and x2 are continuous variables in [0, 1]. W2m [0, 1] is a natural model space for both effects of x1 and x2 . Therefore, we consider the tensor product space W2m1 [0, 1] ⊗ W2m2 [0, 1]. For simplicity, we will derive SS ANOVA decompositions for combinations m1 = m2 = 1 and m1 = m2 = 2 only. SS ANOVA decompositions for other combinations of m1 and m2 can be derived similarly. Consider the tensor product space W21 [0, 1] ⊗ W21 [0, 1] first. Define two averaging operators as (k)
A1 f = (k)
where A1
Z
1
f dxk ,
k = 1, 2,
0
extracts the constant term out of all possible functions of
104
Smoothing Splines: Methods and Applications (k)
(k)
xk . Let A2 = I − A1
for k = 1, 2. Then
(1) (1) (2) (2) f = A1 + A2 A1 + A2 f (1)
(2)
(1)
(2)
(1)
(2)
(1)
(2)
= A1 A1 f + A2 A1 f + A1 A2 f + A2 A2 f , µ + f1 (x1 ) + f2 (x2 ) + f12 (x1 , x2 ).
(4.20)
Obviously, (4.20) is a natural extension of the classical two-way ANOVA decomposition (4.14) from the product of two discrete domains to the product of two continuous domains. Components µ, f1 (x1 ), f2 (x2 ), and f12 (x1 , x2 ) represent the overall mean, the main effect of x1 , the main effect of x2 , and the interaction between x1 and x2 , respectively. In terms of the model space, (4.20) decomposes W21 [0, 1] ⊗ W21 [0, 1] o n o n (1) (1) (2) (2) = H0 ⊕ H1 ⊗ H0 ⊕ H1 n o n o n o n o (1) (2) (1) (2) (1) (2) (1) (2) = H0 ⊗ H0 ⊕ H1 ⊗ H0 ⊕ H0 ⊗ H1 ⊕ H1 ⊗ H1
, H0 ⊕ H1 ⊕ H2 ⊕ H3 ,
R1 (k) (k) where H0 = {1} and H1 = {f ∈ W21 [0, 1] : 0 f dxk = 0} for (1) (2) (1) (2) (1) (2) 0 1 2 k = 1, 2, H = H0 ⊗ H0 , H = H1 ⊗ H0 , H = H0 ⊗ H1 , and (1) (2) H3 = H1 ⊗ H1 . H0 is an one-dimensional space with basis φ(x) = 1. (k) The RKs of H1 , H2 , and H3 can be calculated from RKs of H0 and (k) H1 given in Table 2.2. Now suppose we want to model both x1 and x2 using cubic splines. That is, we consider the tensor product space W22 [0, 1]⊗W22 [0, 1]. Define four averaging operators
(k)
A1 f = (k)
A2 f = (1)
(2)
Z
1
f dxk , 0
Z
0
1
f ′ dxk (xk − 0.5),
k = 1, 2,
where A1 and A1 extract the constant function out of all possible (1) (2) functions for each variable, and A2 and A2 extract the linear function
Smoothing Spline ANOVA (k)
(k)
(k)
for each variable. Let A3 = I − A1 − A2
105
for k = 1, 2. Then
(1) (1) (1) (2) (2) (2) f = A1 + A2 + A3 A1 + A2 + A3 f (1)
(2)
(1)
(2)
(1)
(2)
= A1 A1 f + A1 A2 f + A1 A3 f (1)
(2)
(1)
(2)
(1)
(2)
(1)
(2)
(1)
(2)
(1)
(2)
+ A2 A1 f + A2 A2 f + A2 A3 f + A3 A1 f + A3 A2 f + A3 A3 f
, µ + β2 × (x2 − 0.5) + f2s (x2 )
ls + β1 × (x1 − 0.5) + β3 × (x1 − 0.5) × (x2 − 0.5) + f12 (x1 , x2 )
sl ss + f1s (x1 ) + f12 (x1 , x2 ) + f12 (x1 , x2 ),
(4.21)
where µ represents the overall mean; β1 × (x1 − 0.5) and β2 × (x2 − 0.5) represent the linear main effects of x1 and x2 ; f1s (x1 ) and f2s (x2 ) represent the smooth main effect of x1 and x2 ; β3 × (x1 − 0.5) × (x2 − ls sl ss 0.5), f12 (x1 , x2 ), f12 (x1 , x2 ) and f12 (x1 , x2 ) represent the linear–linear, linear–smooth, smooth–linear, and smooth–smooth interactions between x1 and x2 . The overall main effect of xk fk (xk ) = βk × (xk − 0.5) + fks (xk ),
k = 1, 2,
and the overall interaction between x1 and x2 ls sl f12 (x1 , x2 ) = β3 × (x1 − 0.5) × (x2 − 0.5) + f12 (x1 , x2 ) + f12 (x1 , x2 ) ss + f12 (x1 , x2 ).
The simple regression model with both x1 and x2 being modeled by ls sl straight lines is a special case of (4.21) with f1s = f2s = f12 = f12 = ss f12 = 0. In terms of the model space, (4.21) decomposes W22 [0, 1] ⊗ W22 [0, 1] n o n o (1) (1) (1) (2) (2) (2) = H0 ⊕ H1 ⊕ H2 ⊗ H0 ⊕ H1 ⊕ H2 n o n o n o (1) (2) (1) (2) (1) (2) = H0 ⊗ H0 ⊕ H1 ⊗ H0 ⊕ H2 ⊗ H0 n o n o n o (1) (2) (1) (2) (1) (2) ⊕ H0 ⊗ H1 ⊕ H1 ⊗ H1 ⊕ H2 ⊗ H1 n o n o n o (1) (2) (1) (2) (1) (2) ⊕ H0 ⊗ H2 ⊕ H1 ⊗ H2 ⊕ H2 ⊗ H2 , H0 ⊕ H1 ⊕ H2 ⊕ H3 ⊕ H4 ⊕ H5 ,
R1 (k) (k) (k) where H0 = {1}, H1 = {xk − 0.5}, and H2 = {f ∈ W22 [0, 1] : 0 f R1 (1) (2) (1) (2) dxk = 0 f ′ dxk = 0} for k = 1, 2, H0 = {H0 ⊗ H0 } ⊕ {H1 ⊗ H0 } ⊕
106
Smoothing Splines: Methods and Applications
(1)
(2)
(1)
(2)
(1)
(2)
(0)
(2)
{H0 ⊗ H1 } ⊕ {H1 ⊗ H1 }, H1 = H2 ⊗ H0 , H2 = H2 ⊗ H1 , (1) (2) (1) (2) (1) (2) H3 = H0 ⊗ H2 , H4 = H1 ⊗ H2 , and H5 = H2 ⊗ H2 . H0 is a four-dimensional space with basis functions φ1 (x) = 1, φ2 (x) = x1 − 0.5, φ3 (x) = x2 − 0.5, and φ4 (x) = (x1 − 0.5) × (x2 − 0.5). The RKs of H1 , (k) (k) (k) H2 , H3 , H4 and H5 can be calculated from RKs of H0 , H1 , and H2 given in Table 2.2.
4.4.4
Decomposition of Ra ⊗ W2m (per)
Suppose x1 is a discrete variable with a levels, and x2 is a continuous variable in [0, 1]. In addition, suppose that f is a periodic function of x2 . A natural model space for x1 is Ra , and a natural model space for x2 is W2m (per). Therefore, we consider the tensor product space Ra ⊗ W2m (per). (1) (2) Define two averaging operators A1 and A1 as a 1 X f, a x =1 1 Z 1 (2) f dx2 . A1 f = (1)
A1 f =
0
(1)
(1)
Let A2 = I − A1
(2)
(2)
and A2 = I − A1 . Then
(1) (1) (2) (2) f = A1 + A2 A1 + A2 f (1)
(2)
(1)
(2)
(1)
(2)
(1)
(2)
= A1 A1 f + A2 A1 f + A1 A2 f + A2 A2 f , µ + f1 (x1 ) + f2 (x2 ) + f12 (x1 , x2 ),
(4.22)
where µ represents the overall mean, f1 (x1 ) represents the main effect of x1 , f2 (x2 ) represents the main effect of x2 , and f12 (x1 , x2 ) represents the (1) (1) interaction between x1 and x2 . Write Ra = H0 ⊕ H1 and W2m (per) = (2) (2) (1) (1) (2) H0 ⊕ H1 , where H0 and H1 are given in (4.5), H0 = {1}, and R 1 (2) H1 = {f ∈ W2m (per) : 0 f du = 0}. Then, in terms of the model space, (4.22) decomposes Ra ⊗ W2m (per) n o n o (1) (1) (2) (2) = H0 ⊕ H1 ⊗ H0 ⊕ H1 n o n o n o n o (1) (2) (1) (2) (1) (2) (1) (2) = H0 ⊗ H0 ⊕ H1 ⊗ H0 ⊕ H0 ⊗ H1 ⊕ H1 ⊗ H1
, H0 ⊕ H1 ⊕ H2 ⊕ H3 ,
Smoothing Spline ANOVA (1)
(2)
(1)
107
(2)
(1)
(2)
where H0 = H0 ⊗ H0 , H1 = H1 ⊗ H0 , H2 = H0 ⊗ H1 , and (1) (2) H3 = H1 ⊗ H1 . H0 is an one-dimensional space with basis function φ(x) = 1. The RKs of H1 , H2 , and H3 can be calculated from RKs of (1) (1) (2) (2) H0 and H1 given in (4.6) and RKs of H0 and H1 given in (2.33).
4.4.5
Decomposition of W2m1 (per) ⊗ W2m2 [0, 1]
Suppose both x1 and x2 are continuous variables in [0, 1]. In addition, suppose f is a periodic function of x1 . A natural model space for x1 is W2m1 (per), and a natural model space for x2 is W2m2 [0, 1]. Therefore, we consider the tensor product space W2m1 (per)⊗ W2m2 [0, 1]. For simplicity, we derive the SS ANOVA decomposition for m2 = 2 only. Define three averaging operators (1)
A1 f = (2)
A1 f = (2) A2 f (1)
(1)
Let A2 = I − A1
=
Z Z
1
f dx1 , 0 1
f dx2 , 0
Z
0
1
′
f dx2 (x2 − 0.5).
(2)
(2)
(2)
and A3 = I − A1 − A2 . Then
n on o (1) (1) (2) (2) (2) f = A1 + A2 A1 + A2 + A3 f (1)
(2)
(1)
(2)
(1)
(2)
= A1 A1 f + A1 A2 f + A1 A3 f (1)
(2)
(1)
(2)
(1)
(2)
+ A2 A1 f + A2 A2 f + A2 A3 f
, µ + β × (x2 − 0.5) + f2s (x2 )
sl ss + f1 (x1 ) + f12 (x1 , x2 ) + f12 (x1 , x2 ),
(4.23)
where µ represents the overall mean, f1 (x1 ) represents the main effect of x1 , β × (x2 − 0.5) and f2s (x2 ) represent the linear and smooth main sl ss effects of x2 , f12 (x1 , x2 ) and f12 (x1 , x2 ) represent the smooth–linear and smooth–smooth interactions. The overall main effect of x2 f2 (x2 ) = β × (x2 − 0.5) + f2s (x2 ), and the overall interaction between x1 and x2 sl ss f12 (x1 , x2 ) = f12 (x1 , x2 ) + f12 (x1 , x2 ).
108
Smoothing Splines: Methods and Applications (1)
(1)
(2)
(2)
(2)
Write W2m1 (per) = H0 ⊕ H1 and W22 [0, 1] = H0 ⊕ H1 ⊕ H2 , R1 (1) (1) (2) where H0 = {1}, H1 = {f ∈ W2m1 (per) : 0 f du = 0}, H0 = {1}, R R 1 1 (2) (2) H1 = {x2 − 0.5}, and H2 = {f ∈ W22 [0, 1] : 0 f du = 0 f ′ du = 0}. Then, in terms of the model space, (4.23) decomposes W2m1 (per) ⊗ W22 [0, 1] n o n o (1) (1) (2) (2) (2) = H0 ⊕ H1 ⊗ H0 ⊕ H1 ⊕ H2 n o n o n o (1) (2) (1) (2) (1) (2) = H0 ⊗ H0 ⊕ H0 ⊗ H1 ⊕ H0 ⊗ H2 n o n o n o (1) (2) (1) (2) (1) (2) ⊕ H1 ⊗ H0 ⊕ H1 ⊗ H1 ⊕ H1 ⊗ H2 , H0 ⊕ H1 ⊕ H2 ⊕ H3 ⊕ H4 , (1)
(2)
(1)
(2)
(1)
(2)
where H0 = {H0 ⊗ H0 } ⊕ {H0 ⊗ H1 }, H1 = H0 ⊗ H2 , H2 = (1) (2) (1) (2) (1) (2) H1 ⊗ H0 , H3 = H1 ⊗ H1 , and H4 = H1 ⊗ H2 . H0 is a twodimensional space with basis functions φ(x) = 1 and φ(x) = x2 − 0.5. (1) The RKs of H1 , H2 , H3 , and H4 can be calculated from the RKs H0 (1) (2) (2) (2) and H1 given in (2.33) and the RKs of H0 , H1 , and H2 given in Table 2.2.
4.4.6
Decomposition of W22 (R2 ) ⊗ W2m (per)
Suppose x1 = (x11 , x12 ) is a bivariate continuous variable in R2 , and x2 is a continuous variable in [0, 1]. In addition, suppose that f is a periodic function of x2 . We consider the tensor product space W22 (R2 )⊗ W22 (per) for the joint regression function f (x1 , x2 ). Let φ1 (x1 ) = 1, φ2 (x1 ) = x11 , and φ3 (x1 ) = x12 be polynomials of total degree less than 2. Define three averaging operators (1)
A1 f = (1)
A2 f = (2)
A1 f =
J X
wj f (uj ),
j=1
J X j=1
Z
wj f (uj ){φ˜2 (uj )φ˜2 + φ˜3 (uj )φ˜3 },
1
f dx2 ,
0
where uj are fixed points in R2 , wj are fixed positive weights such that PJ ˜ ˜ ˜ j=1 wj = 1, φ1 = 1, and φ2 and φ3 are orthonormal bases based on
Smoothing Spline ANOVA (1)
(1)
(1)
the norm (2.41). Let A3 = I − A1 − A2
109 (2)
(2)
and A2 = I − A1 . Then
n on o (1) (1) (1) (2) (2) f = A1 + A2 + A3 A1 + A2 f (1)
(2)
(1)
(2)
(1)
(2)
= A1 A1 f + A2 A1 f + A3 A1 f (1)
(2)
(1)
(2)
(1)
(2)
+ A1 A2 f + A2 A2 f + A3 A2 f = µ + β1 φ˜2 (x1 ) + β2 φ˜3 (x1 ) ls ss + f1s (x1 ) + f2 (x2 ) + f12 (x1 , x2 ) + f12 (x1 , x2 ),
(4.24)
where µ is the overall mean, β1 φ˜2 (x1 ) + β2 φ˜3 (x1 ) is the linear main effect of x1 , f1s (x1 ) is the smooth main effect of x1 , f2 (x2 ) is the main ls ss effect of x2 , f12 (x1 , x2 ) is the linear–smooth interaction, and f12 (x1 , x2 ) is smooth–smooth interaction. The overall main effect of x1 f1 (x1 ) = β1 φ˜2 (x1 ) + β2 φ˜3 (x1 ) + f1s (x1 ), and the overall interaction ls ss f12 (x1 , x2 ) = f12 (x1 , x2 ) + f12 (x1 , x2 ). (1)
(1)
(1)
(2)
(2)
Write W22 (R2 ) = H0 ⊕ H1 ⊕ H2 and W2m (per) = H0 ⊕ H1 , where (1) (1) (1) H0 = {1}, H1 = {φ˜2 , φ˜3 }, H2 = {f ∈ W22 (R2 ) : J22 (f ) = 0}, R1 (2) (2) m1 H0 = {1}, and H1 = {f ∈ W2 (per) : 0 f du = 0}. Then, in terms of the model space, (4.24) decomposes W22 (R2 ) ⊗ W2m (per) n o n o (1) (1) (1) (2) (2) = H0 ⊕ H1 ⊕ H2 ⊗ H0 ⊕ H1 n o n o n o (1) (2) (1) (2) (1) (2) = H0 ⊗ H0 ⊕ H1 ⊗ H0 ⊕ H2 ⊗ H0 n o n o n o (1) (2) (1) (2) (1) (2) ⊕ H0 ⊗ H1 ⊕ H1 ⊗ H1 ⊕ H2 ⊗ H1 , H0 ⊕ H1 ⊕ H2 ⊕ H3 ⊕ H4 , (1)
(2)
(1)
(2)
(1)
(2)
(4.25)
where H0 = {H0 ⊗ H0 } ⊕ {H1 ⊗ H0 }, H1 = {H2 ⊗ H0 }, H2 = (1) (2) (1) (2) (1) (2) {H0 ⊗ H1 }, H3 = {H1 ⊗ H1 }, and H4 = {H2 ⊗ H1 }. The basis (1) (1) functions of H0 are 1, φ˜2 , and φ˜3 . The RKs of H0 and H1 are 1 and (1) φ˜2 (x1 )φ˜2 (z 1 ) + φ˜3 (x1 )φ˜3 (z 1 ), respectively. The RK of H2 is given in (2) (2) (2.42). The RKs of H0 and H1 are given in (2.33). The RKs of H1 , 2 3 4 H , H , and H can be calculated as products of the RKs of the involved marginal spaces.
110
Smoothing Splines: Methods and Applications
4.5
General SS ANOVA Decomposition
Consider the general case with d independent variables x1 ∈ X1 , x2 ∈ X2 , . . . , xd ∈ Xd , and the tensor product space H(1) ⊗H(2) ⊗· · ·⊗H(d) on X = X1 × X2 × · · · × Xd . For f as a function of xk , assume the following one-way decomposition as in (4.3), (k)
f = A1 f + · · · + A(k) rk f, (k)
1 ≤ k ≤ d,
(4.26)
(k)
where A1 + · · · + Ark = I. Then, for the joint function, (1) (d) f = A1 + · · · + A(1) . . . A1 + · · · + A(d) r1 rd f rd r1 X X (1) (d) = ... Aj1 . . . Ajd f. j1 =1
(4.27)
jd =1
The above decomposition of the function f is referred to as the SS ANOVA decomposition. Denote (k) (k) H(k) = H(1) ⊕ · · · ⊕ H(rk ) , k = 1, 2, . . . , d as the one-way decomposition to H(k) associated with (4.26). Then, (4.27) decomposes the tensor product space H(1) ⊗ H(2) ⊗ · · · ⊗ H(d) n o n o (1) (1) (d) (d) = H(1) ⊕ · · · ⊕ H(r1 ) ⊗ · · · ⊗ H(1) ⊕ · · · ⊕ H(rd ) =
r1 X
j1 =1
...
rd X
jd =1
(1)
(1)
(d)
H(j1 ) ⊗ . . . ⊗ H(jd ) . (d)
The RK of H(j1 ) ⊗ · · · ⊗ H(jd ) equals (k)
Qd
k=1
(4.28) (k)
(k)
R(jk ) , where R(jk ) is the RK
of H(jk ) for k = 1, . . . , d. Consider the special case when rk = 2 for all k = 1, . . . , d. Assume (k) (k) that A1 f is independent of xk , or equivalently, H(1) = {1}. Then the decomposition (4.27) can be written as ( ) X Y (k) Y (k) f= A2 A1 f B⊆{1,...,d}
=µ+
d X
k=1
k∈B c
k∈B
fk (xk ) +
X k
fkl (xk , xl ) + · · · + f1...d (x1 , . . . , xd ),
(4.29)
Smoothing Spline ANOVA
111
where µ represents the grand mean, fk (xk ) represents the main effect of xk , fkl (xk , xl ) represents the two-way interaction between xk and xl , and the remaining terms represent higher-order interactions. (k) For general rk , assuming A1 f is independent of xk , the decomposi(k) (k) tion (4.29) can be derived by combining operators A2 , . . . , Ark into (k) (k) (k) one averaging operator A˜2 = A2 + · · · + Ark . Therefore, decomposition (4.29) combines components in (4.27) and reorganizes them into overall main effects and interactions. When all xk are discrete variables, the SS ANOVA decomposition (4.27) leads to the classical d-way ANOVA model. Therefore, SS ANOVA decompositions are natural extensions of classical ANOVA decompositions from discrete domains to general domains and from finite dimensional spaces to infinite dimensional spaces. They decompose tensor product RKHS’s into meaningful subspaces. As the classical ANOVA decompositions, SS ANOVA decompositions lead to hierarchical structures that are useful for model selection and interpretation. Different SS ANOVA decompositions can be derived based on different averaging operators (or equivalently different decompositions of marginal spaces). Therefore, the SS ANOVA decomposition should be regarded as a general prescription for building multivariate nonparametric models rather than some fixed models. They can also be used to construct submodels for components in more complicated models.
4.6
SS ANOVA Models and Estimation
The curse of dimensionality is a major problem in dealing with multivariate functions. From the SS ANOVA decomposition in (4.27), it is clear that the number of components in the decomposition increases exponentially as the dimension d increases. To overcome this problem, as in classical ANOVA, high-order interactions are often deleted from the model space. A model containing any subset of components in the SS ANOVA decomposition (4.27) is referred to as an SS ANOVA model. The wellknown additive model is a special case that contains main effects only (Hastie and Tibshirani 1990). Given an SS ANOVA model, we can regroup and write the model space as M = H0 ⊕ H1 ⊕ · · · ⊕ Hq ,
(4.30)
where H0 is a finite dimensional space collecting all functions that are
112
Smoothing Splines: Methods and Applications
not going to be penalized, and H1 , . . . , Hq are orthogonal RKHS’s with RKs Rj for j = 1, . . . , q. The RK Rj equals the product of RKs of the subspaces involved in the tensor product space Hj . The norms on the composite Hj are the tensor product norms induced by the norms on the component subspaces. Details about the induced norm can be found in Aronszajn (1950), and an illustrative example can P be found in Chapter q 10 of Wahba (1990). Note that kf k2 = kP0 f k2 + j=1 kPj f k2 , where Pj is the orthogonal projector in M onto Hj . For generality, suppose observations are generated by yi = Li f + ǫi ,
i = 1, . . . , n,
(4.31)
where f is a multivariate function in the model space M defined in (4.30), Li are bounded linear functionals, and ǫi are zero-mean independent random errors with a common variance σ 2 . The PLS estimate of f is the solution to q n 1 X X min (yi − Li f )2 + λj kPj f k2 . (4.32) f ∈M n i=1
j=1
Different smoothing parameters λj allow different penalties for each component. For fixed smoothing parameters, the following rescaling allows us to derive and compute the solution to (4.32) using results in Chapter 2. Let H1∗ = ⊕qj=1 Hj . Then, for any f ∈ H1∗ , f (x) = f1 (x) + · · · + fq (x),
fj ∈ Hj , j = 1, . . . , q.
Write λj , λ/θj . The set of parameters λ and θj are overparameterized. The penalty is controlled by the ratio λ/θj , that is, λj . Define the inner product in H1∗ as q X (f, g)∗ = θj−1 (fj , gj ). (4.33) j=1
Then,
||f ||2∗
=
Pq
−1 2 j=1 θj ||fj || .
(R1∗ (x, ·), f (·))∗ =
q X j=1
Let R1∗ =
Pq
j=1 θj R
θj−1 (θj Rj (x, ·), fj (·)) =
j
. Since
q X
fj (x) = f (x),
j=1
∗ then R1∗ is the PqRK of H1 with the inner product (4.33). ∗ Let P1 = j=1 Pj be the orthogonal projection in M onto H1∗ . Then the minimization problem (4.32) is reduced to ( n ) 1X 2 ∗ 2 min (4.34) (yi − Li f ) + λkP1 f k∗ . f ∈M n i=1
Smoothing Spline ANOVA
113
The PLS (4.34) has the same form as (2.11), with H1 and P1 being replaced by H1∗ and P1∗ , respectively. Therefore, results in Section 2.4 apply. Specifically, let θ = (θ1 , . . . , θq ), φ1 , . . . , φp be basis functions of H0 , and p T = {Li φν }ni=1 ν=1 ,
Σk = {Li Lj Rk }ni,j=1 ,
k = 1, . . . , q,
(4.35)
Σθ = θ1 Σ1 + · · · + θq Σq . Applying the Kimeldorf–Wahba representer theorem and noting that ξi (x) =
Li(z ) R1∗ (x, z)
=
q X j=1
θj Li(z ) Rj (x, z),
the solution can be represented as fˆ(x) =
p X
dν φν (x) +
ν=1
n X i=1
ci
q X j=1
θj Li(z ) Rj (x, z),
(4.36)
where d = (d1 , . . . , dp )T and c = (c1 , . . . , cn )T are solutions to (Σθ + nλI)c + T d = y, T T c = 0.
(4.37)
Equations in (4.37) have the same form as those in (2.21), with Σ being replaced by Σθ . Therefore, the coefficients c and d can be computed similarly. Let fˆ = (L1 fˆ, . . . , Ln fˆ)T be the vector of fitted values. Let the QR decomposition of T be R T = (Q1 Q2 ) 0 and M = Σθ + nλI. Then fˆ = H(λ, θ)y where H(λ, θ) = I − nλQ2 (QT2 M Q2 )−1 QT2
(4.38)
is the hat matrix. The ssr function in the assist package can be used to fit the SS ANOVA model (4.31). As in Chapter 2, observations y and T matrix can be specified using the argument formula. Instead of a single matrix
114
Smoothing Splines: Methods and Applications
Σ, we now have multiple matrices Σj for j = 1, . . . , q. They are specified as elements of a list for the argument rk. Examples are given in Section 4.9. Sometimes it may be desirable to use the same smoothing parameter for a subset of penalties in the PLS (4.32). For illustration, suppose we want to solve the PLS (4.32) with λq−1 = λq . This can be achieved ˜ q−1 , and fit the SS by combining Hq−1 and Hq into one space, say H 0 1 ˜ q−1 . The ANOVA model with model space M = H ⊕ H ⊕ · · · ⊕ H q−1 q−1 q−1 q ˜ ˜ RK of H is R =R + R . Then the model can be fitted by a call to the ssr function with the combined RK. The same approach can be applied to multiple subsets such that penalties in each subset share the same smoothing parameter. When appropriate, this approach can greatly reduce the computation time when q is large. An example will be given in Section 4.9.1.
4.7
Selection of Smoothing Parameters
The set of parameters λ and θ are overparameterized. Therefore, even though the criteria in this section are presented as functions of λ and θ, they should be understood as functions of (λ1 , . . . , λq ), where λj = λ/θj . Define mean squared error (MSE) as MSE(λ, θ) = E
1 ˆ 2 ||f − f || , n
(4.39)
where f = (L1 f, . . . , Ln f )T . Following the same arguments as in Section 3.3, it is easy to check that the function UBR(λ, θ) ,
1 2σ 2 ||(I − H(λ, θ))y||2 + trH(λ, θ) n n
(4.40)
is an unbiased estimate of MSE(λ, θ) + σ 2 . The function UBR(λ, θ) is referred to as the unbiased risk (UBR) criterion and the minimizer of UBR(λ, θ) is referred to as the UBR estimate of (λ, θ). The UBR method requires an estimate of error variance σ 2 . Few methods are available for estimating σ 2 in a multivariate nonparametric model without needing to estimate the function f first. When the product domain X is equipped with a norm, the method in Tong and Wang (2005) may be used to estimate σ 2 .
Smoothing Spline ANOVA
115
A parallel derivation as in Section 3.4 leads to the following extension of the GCV criterion 1 Pn 2 ˆ i=1 (Li f − yi ) n GCV(λ, θ) , (4.41) 2. 1 n tr(I − H(λ, θ)) The GCV estimate of (λ, θ) is the minimizer of GCV(λ, θ). We now construct a Bayes model for the SS ANOVA model (4.31). Assume a prior for f as F (x) =
p X
1
ζν φν (x) + δ 2
ν=1
q X p θj Uj (x),
(4.42)
j=1
iid
where ζν ∼ N(0, κ); Uj (x) are independent, zero-mean Gaussian stochastic processes with covariance function Rj (x, z); ζν and Uj (x) are mutually independent; and κ and δ are positive constants. Suppose observations are generated from yi = Li F + ǫi ,
i = 1, . . . , n,
(4.43)
iid
where ǫi ∼ N(0, σ 2 ). Let L0 be a bounded linear functional on M. Let λ = σ 2 /nδ. The same arguments in Section 3.6 hold when M = Σ + nλI is replaced by M = Σθ + nλI in this chapter. Therefore, lim E(L0 F |y) = L0 fˆ.
κ→∞
That is, the PLS estimate fˆ is a Bayes estimate. Furthermore, we have the following extension of the GML criterion: T
GML(λ, θ) ,
y (I − H(λ, θ))y
1
[det+ ((I − H(λ, θ)))] n−p
.
(4.44)
All three forms of LME models for the SSR model in Section 3.5 can be extended for the SS ANOVA model. We present the extension of (3.35) only. Let Σk = Zk ZkT , where Zk is an n × mk matrix with mk = rank(Σk ). It is not difficult to see that the GML criterion is the REML criterion based on the following linear mixed-effects model y = Tζ +
q X
Zk uk + ǫ,
(4.45)
k=1
where ζ = (ζ1 , . . . , ζp )T are deterministic parameters, uk are mutually independent random effects, uk ∼ N(0, σ 2 θk Imk /nλ), ǫ ∼ N(0, σ 2 I), and uk are independent of ǫ. Details can be found in Chapter 9.
116
4.8
Smoothing Splines: Methods and Applications
Confidence Intervals
Any function f ∈ M can be represented as f=
p X
ν=1
f0ν +
q X
f1j ,
(4.46)
j=1
where f0ν ∈ span{φν } for ν = 1, . . . , p, and f1j ∈ Hj for j = 1, . . . , q. Our goal is to construct Bayesian confidence intervals for L0 fγ =
p X
ν=1
γν L0 f0ν +
q X j=1
γp+j L0 f1j
(4.47)
for any bounded linear functional L0 and any γ = (γ1 , . . . , γp+q )T , where γk = 1 when the corresponding component in (4.46) is to be included and 0 otherwise. p Let F0ν = ζν φν for ν = 1, . . . , p, and F1j = δθj Uj for j = 1, . . . , q. Let L0 , L01 , and L02 be bounded linear functionals. Posterior means and covariances For ν, µ = 1, . . . , p and j, k = 1, . . . , q, the posterior means are E(L0 F0ν |y) = (L0 φν )eTν d,
(4.48)
E(L0 F1j |y) = θj (L0 ξ j )T c,
and the posterior covariances are δ −1 Cov(L01 F0ν , L02 F0µ |y) = (L01 φν )(L02 φµ )eTν Aeµ ,
δ −1 Cov(L01 F0ν , L02 F1j |y) = −θj (L01 φν )eTν B(L02 ξ j ), δ
−1
j
(4.49) T
Cov(L01 F1j , L02 F1k |y) = δj,k θj L01 L02 R − θj θk (L01 ξj ) C(L02 ξ k ),
where eν is a vector of dimension p with the νth element being one and all other elements being zero, δj,k is the Kronecker delta, c and d are solutions to (4.37), Lξ j = (LL1 Rj , . . . , LLn Rj )T for any welldefined L, M = Σθ + nλI, A = (T T M −1 T )−1 , B = AT T M −1 , and C = M −1 (I − B). As in Section 3.8.1, even though not explicitly expressed, a diffuse prior is assumed for ζ with κ → ∞. The first two equations in (4.48) state that the projections of fˆ on subspaces are the posterior means of the corresponding components in the Bayes model (4.42). The next
Smoothing Spline ANOVA
117
three equations in (4.49) can be used to compute posterior covariances of the spline estimates and their projections. Based on these posterior covariances, we construct Bayesian confidence intervals for the overall function f and its components in (4.46). Specifically, posterior mean and variance for L0 fγ in (4.47) can be calculated using the formulae in (4.48) and (4.49). Then 100(1 − α)% Bayesian confidence interval for L0 fγ is q E{L0 Fγ |y} ± z α2 Var{L0 Fγ |y}
where
Fγ (x) =
p X
ν=1
γν F0ν (x) +
q X
γp+j F1j (x).
j=1
Confidence intervals for a collection of points can be constructed similarly. The same approach in Section 3.8.2 can be used to construct bootstrap confidence intervals. The extension is straightforward.
4.9 4.9.1
Examples Tongue Shapes
Consider the ultrasound data. Let y be the response variable height; x1 be the index of environment with x1 = 1, 2, 3 corresponding to 2words, cluster, and schwa respectively; x2 be the variable length scaled into [0, 1]; and x3 be the variable time scaled into [0, 1]. We first investigate how tongue shapes for an articulation differ under different environments at a particular time, say, at time 60 ms. Observations are shown in Figure 4.2. Consider a bivariate regression function f (x1 , x2 ) where x1 (environment) is a discrete variable with three levels, and x2 (length) is a continuous variable in [0, 1]. Therefore, we model the joint function using the tensor product space R3 ⊗ W2m [0, 1]. The SS ANOVA decompositions of Ra ⊗ W2m [0, 1] are given in (4.16) for m = 1 and (4.18) for m = 2. The following statements fit the SS ANOVA model (4.16): > > > > >
data(ultrasound) ultrasound$y <- ultrasound$height ultrasound$x1 <- ultrasound$env ultrasound$x2 <- ident(ultrasound$length) ssr(y~1, data=ultrasound, subset=ultrasound$time==60,
Smoothing Splines: Methods and Applications
height (mm) 45 50 55 60 65 70 75 80
118
cluster
2words
schwa o
oo
o o
o
o
o o
oo
o
o
oo o
o o oo o o
o o o
o o
o
o oo
o o
o
o
o o o
oo o
80
o
o o o oo o
oo o
o o o
oo
o
o
oo
o o
o o
ooo
100 120 140 length (mm)
o
o
o oo
o oo
o o o oo
o
o
o ooo
o o
o o
o o o
o o oo
80
o o
o
o o
o o
o
o
o o o
o
o
o o o o
o o o
o
o o o o o oo o o oo oo
o
100 120 140 length (mm)
80
100 120 140 length (mm)
FIGURE 4.2 Ultrasound data, plots of observations (circles), fits (thin lines), and 95% Bayesian confidence intervals (shaded regions), and the mean curves among three environments (thicker lines). The tongue root is on the left, and the tongue tip is on the right in each plot.
rk=list(shrink1(x1), linear(x2), rk.prod(shrink1(x1),linear(x2)))) where the ident function transforms a variable into [0, 1], the shrink1 P3 (1) function computes the RK of H1 = {f : i=1 f (i) = 0} using the formula in (4.6), and the rk.prod function computes the product of RKs. The following statements fit the SS ANOVA model (4.18) and summarize the fit: > ultra.el.c.fit <- ssr(y~x2, data=ultrasound, subset=ultrasound$time==60, rk=list(shrink1(x1), cubic(x2), rk.prod(shrink1(x1),kron(x2-.5)), rk.prod(shrink1(x1),cubic(x2)))) > summary(ultra.el.c.fit) ... GCV estimate(s) of smoothing parameter(s) : 6.061108e-02 2.650966e-05 1.557379e-03 3.177218e-05 Equivalent Degrees of Freedom (DF): 13.57631 Estimate of sigma: 2.654545
Smoothing Spline ANOVA
119
where the function kron computes the RK, k1 (x2 )k1 (z2 ), of the space (2) H1 = {k1 (x2 )}. Estimates of the smoothing parameters λ3 and λ4 are small, which ss indicates that both interactions γx1 × (x2 − 0.5) and f12 (x1 , x2 ) may not be negligible. We compute the posterior mean and standard deviation ss for the overall interaction f12 (x1 , x2 ) = γx1 × (x2 − 0.5) + f12 (x1 , x2 ) on grid points as follows: > grid <- seq(0,1,len=40) > predict(ultra.el.c.fit, terms=c(0,0,0,0,1,1), newdata=expand.grid(x2=grid,x1=as.factor(1:3))) The overall interaction and its 95% Bayesian confidence intervals are shown in Figure 4.3. It is clear that the interaction is nonzero.
cluster
schwa
−5
height (mm) 0 5
2words
80
100 120 140 length (mm)
80
100 120 140 length (mm)
80
100 120 140 length (mm)
FIGURE 4.3 Ultrasound data, plots of the overall interaction (solid lines), and 95% Bayesian confidence intervals (shaded regions). Dashed line in each plot represents the constant function zero.
The posterior mean and standard deviation of the function f (x1 , x2 ) can be calculated as follows: > predict(ultra.el.c.fit, terms=c(1,1,1,1,1,1), newdata=expand.grid(x2=grid,x1=as.factor(1:3))) The default for the option terms is a vector of all 1’s. Therefore, this option can be dropped in the above statement. The fits and 95% Bayesian confidence intervals are shown in Figure 4.2.
120
Smoothing Splines: Methods and Applications
Note that in model (4.18) the first three terms represent the mean curve among three environments and the last three terms represent the departure of a particular environment from the mean curve. For comparison, we compute the estimate of the mean curve among three environments: > predict(ultra.el.c.fit, terms=c(1,1,0,1,0,0), newdata=expand.grid(x2=grid,x1=as.factor(1))) The estimate of the mean curve is also displayed in Figure 4.2. The difference between the tongue shape under a particular environment and the average tongue shape can be made by comparing two lines in each plot. To look at the effect of each environment more closely, we calculate the estimate of the departure from the mean curve for each environment: > predict(ultra.el.c.fit, terms=c(0,0,1,0,1,1), newdata=expand.grid(x2=grid,x1=as.factor(1:3))) The estimates of environment effects are shown in Figure 4.4. We can see that, comparing to the average shape, the tongue shape for 2words is front-raising, and the tongue shape for cluster is back-raising. The tongue shape for schwa is close to the average shape.
2words
schwa
−5
height (mm) 0 5
cluster
80
100 120 140 length (mm)
80
100 120 140 length (mm)
80
100 120 140 length (mm)
FIGURE 4.4 Ultrasound data, plots of effects of environment, and 95% Bayesian confidence intervals. The dashed line in each plot represents the constant function zero. The model space of the SS ANOVA model (4.18) is M = H0 ⊕ H1 ⊕ H ⊕ H3 ⊕ H4 , where Hj for j = 0, . . . , 4 are defined in (4.19). In 2
Smoothing Spline ANOVA
121
particular, the spaces H3 and H4 contain smooth–linear and smooth– smooth interactions between x1 and x2 . For illustration, suppose now that we want to fit model (4.18) with the same smoothing parameter for penalties to functions in H3 and H4 . That is to set λ3 = λ4 in the PLS. As discussed in Section 4.6, this can be achieved by combining H3 and H4 into one space. The following statements fit the SS ANOVA model (4.18) with λ3 = λ4 : > ultra.el.c.fit1 <- ssr(y~x2, data=ultra, subset=ultra$time==60, rk=list(shrink1(x1), cubic(x2), rk.prod(shrink1(x1),kron(x2-.5))+ rk.prod(shrink1(x1),cubic(x2)))) > summary(ultra.el.c.fit1) ... GCV estimate(s) of smoothing parameter(s) : 5.648863e-02 2.739598e-05 3.766634e-05 Equivalent Degrees of Freedom (DF): 13.52603 Estimate of sigma: 2.65806 Next we investigate how the tongue shapes change over time for each environment. Figure 4.1 shows 3-d plots of observations. For a fixed environment, consider a bivariate regression function f (x2 , x3 ) where both x2 and x3 are continuous variables. Therefore, we model the joint function using the tensor product space W2m1 [0, 1] ⊗ W2m2 [0, 1]. The SS ANOVA decompositions of W2m1 [0, 1] ⊗ W2m2 [0, 1] were presented in Section 4.4.3. Note that variables x2 and x3 in this section correspond to x1 and x2 in Section 4.4.3. The following statements fit the tensor product of linear splines with m1 = m2 = 1, that is, the SS ANOVA model (4.20), under environment 2words (x1 = 1): > ultrasound$x3 <- ident(ultrasound$time) > ssr(height~1, data=ultrasound, subset=ultrasound$env==1, rk=list(linear(x2), linear(x3), rk.prod(linear(x2),linear(x3)))) The following statements fit the tensor product of cubic splines with m1 = m2 = 2, that is, the SS ANOVA model (4.21), under environment 2words and calculate estimates at grid points: > ultra.lt.c.fit[[1]] <- ssr(height~x2+x3+x2*x3, data=ultrasound, subset=ultrasound$env==1,
122
Smoothing Splines: Methods and Applications
70 60 50
2words
70 60 50
cluster 40
40
100
(mm)
50 40 200 150 100 50 0
80
s)
length
60
(m
80
140
70
e
0
120
80
tim
s)
s)
(mm)
200 150 100 50
(m
100
length
140
schwa
e
(m
80
120
tim
e
tim
200 150 100 50 0
80
height (mm)
80
height (mm)
height (mm)
rk=list(cubic(x2), cubic(x3), rk.prod(kron(x2),cubic(x3)), rk.prod(cubic(x2),kron(x3)), rk.prod(cubic(x2),cubic(x3)))) > grid <- seq(0,1,len=20) > ultra.lt.c.pred <- predict(ultra.lt.c.fit[[1]], newdata=expand.grid(x2=grid,x3=grid))
FIGURE 4.5 Ultrasound data, 3-d plots of the estimated tongue shapes as functions of length and time based on the SS ANOVA model (4.21).
The SS AONVA model (4.21) for environments cluster and schwa can be fitted similarly. The estimates of all three environments are shown in Figure 4.5. These surfaces show how the tongue shapes change over time. Note that µ + β1 × (x2 − 0.5) + f1s(x2 ) represents the mean tongue shape over the time period [0, 140], and the rest in (4.21), β2 ×(x3 −0.5)+ ls sl ss f2s (x3 )+β3 ×(x2 −0.5)×(x3 −0.5)+f12 (x2 , x3 )+f12 (x2 , x3 )+f12 (x2 , x3 ), represents the departure from the mean shape at time x3 . To look at the time effect, we compute posterior means and standard deviations of the departure on grid points: > predict(ultra.lt.c.fit[[1]], term=c(0,0,1,1,0,1,1,1,1), newdata=expand.grid(x2=grid,x3=grid))
100
120
length
140
(mm)
Smoothing Spline ANOVA
123
Figure 4.6 shows the contour plots of the estimated time effect for three environments. Regions where the lower bounds of the 95% Bayesian confidence intervals are positive are shaded in dark grey, while regions where the upper bounds of the 95% Bayesian confidence intervals are negative are shaded in light grey.
2words
cluster
200
10
4
2
schwa
5
0
6
0
−5
6
2
4
time (ms) 100 150
0 −4
−2
−3 4
−2
0
0
5
0
50
6
2
10
2
2
6
4
0
0
4
80
100 120 140 length (mm)
80
100 120 140 length (mm)
80
100 120 140 length (mm)
FIGURE 4.6 Ultrasound data, contour plots of the estimated time effect for three environments based on the SS ANOVA model (4.21). Regions where the lower bounds of the 95% Bayesian confidence intervals are positive are shaded in dark grey. Regions where the upper bounds of the 95% Bayesian confidence intervals are negative are shaded in light grey.
Finally we investigate how the changes of tongue shapes over time differ among different environments. Consider a trivariate regression function f (x1 , x2 , x3 ) in tensor product space R3 ⊗W2m1 [0, 1]⊗W2m2 [0, 1]. For simplicity, we derive the SS ANOVA decomposition for m1 = m2 = 2 only. Define averaging operators: 3 1 X f, 3 x =1 1 Z 1 (k) A1 f = f dxk , 0 Z 1 (k) A2 f = f ′ dxk (xk − 0.5), (1)
A1 f =
k = 2, 3,
0
(1)
(2)
(3)
where A1 , A1 , and A1
extract the constant function out of all pos-
124
Smoothing Splines: Methods and Applications (2)
(3)
sible functions for each variable, and A2 and A2 extract the linear (1) (1) (k) (k) (k) function for x2 and x3 . Let A2 = I − A1 and A3 = I − A1 − A2 for k = 2, 3. Then (1) (1) (2) (2) (2) (3) (3) (3) f = A1 + A2 A1 + A2 + A3 A1 + A2 + A3 f (1)
(2)
(3)
(1)
(2)
(1)
(1)
(2)
(3)
(3)
(1)
(2)
(2)
(3)
(1)
(1)
(2)
(3)
(1)
(2)
(1)
(2)
(1)
(2)
(3)
(3)
(1)
(2)
(3)
(2)
(3)
(1)
(2)
(3)
(1)
(2)
(3)
(1)
(2)
(3)
(3)
(1)
(2)
(3)
(1)
(2)
(3)
(3)
(1)
(2)
(3)
(1)
(2)
(3)
= A1 A1 A1 f + A1 A1 A2 f + A1 A1 A3 f
+ A1 A2 A1 f + A1 A2 A2 f + A1 A2 A3 f + A1 A3 A1 f + A1 A3 A2 f + A1 A3 A3 f + A2 A1 A1 f + A2 A1 A2 f + A2 A1 A3 f + A2 A2 A1 f + A2 A2 A2 f + A2 A2 A3 f + A2 A3 A1 f + A2 A3 A2 f + A2 A3 A3 f
, µ + β2 × (x3 − 0.5) + f3s (x3 )
ls + β1 × (x2 − 0.5) + β3 × (x2 − 0.5)(x3 − 0.5) + f23 (x2 , x3 )
ss sl (x2 , x3 ) + f23 (x2 , x3 ) + f2s (x2 ) + f23 ss sl + f1 (x1 ) + f13 (x1 , x3 ) + f13 (x1 , x3 )
sl sll sls + f12 (x1 , x2 ) + f123 (x1 , x2 , x3 ) + f123 (x1 , x2 , x3 ) ss ssl sss + f12 (x1 , x2 ) + f123 (x1 , x2 , x3 ) + f123 (x1 , x2 , x3 ),
(4.50)
where µ represents the overall mean; f1 (x1 ) represents the main effect of x1 ; β1 × (x2 − 0.5) and β2 × (x3 − 0.5) represent the linear main effects of x2 and x3 ; f2s (x2 ) and f3s (x3 ) represent the smooth main effects of x2 and sl sl x3 ; f12 (x1 , x2 ) (f13 (x1 , x3 )) represents the smooth–linear interaction bels sl tween x1 and x2 (x3 ); β3 × (x2 − 0.5) × (x3 − 0.5), f23 (x2 , x3 ), f23 (x2 , x3 ) ss and f23 (x2 , x3 ) represent linear–linear, linear–smooth, smooth–linear sll and smooth–smooth interactions between x2 and x3 ; and f123 (x1 , x2 , x3 ), sls ssl sss f123 (x1 , x2 , x3 ), f123 (x1 , x2 , x3 ), and f123 (x1 , x2 , x3 ) represent three-way interactions between x1 , x2 , and x3 . The overall main effect of xk , fk (xk ), equals βk−1 × (xk − 0.5) + fks (xk ) for k = 2, 3. The overall intersl ss action between x1 and xk , f1k (x1 , xk ), equals f12 (x1 , xk )+f1k (x1 , xk ) for k = 2, 3. The overall interaction between x2 and x3 , f23 (x2 , x3 ), equals ls sl ss β3 × (x2 − 0.5) × (x3 − 0.5) + f23 (x2 , x3 ) + f23 (x2 , x3 ) + f23 (x2 , x3 ). The sll overall three-way interaction, f123 (x1 , x2 , x3 ), equals f123 (x1 , x2 , x3 ) + sls ssl sss f123 (x1 , x2 , x3 ) + f123 (x1 , x2 , x3 ) + f123 (x1 , x2 , x3 ). We fit model (4.50) as follows: > ssr(height~I(x2-.5)+I(x3-.5)+I((x2-.5)*(x3-.5)), data=ultrasound, rk=list(shrink1(x1),
80
height (mm)
height (mm)
height (mm)
80
Smoothing Spline ANOVA
70 60 50
70
60 50 40
1.0 0.8 0.6 0.4 0.2 0.0 0.0
1.0 0.8 0.6 0.4 0.2 0.0 0.0
le
m) ngth (m
1.0
50
cluster
0.2
0.4
0.6
0.8
len
) gth (mm
schwa
1.0
1.0 0.8 0.6 0.4 0.2 0.0 0.0
s) (m
0.4
0.8
s) (m
s) (m
0.2
0.6
60
e tim
2words
125
70
40
e tim
e tim
40
80
0.2
FIGURE 4.7 Ultrasound data, 3-d plots of the estimated tongue shape as a function of environment, length and time based on the SS ANOVA model (4.50). cubic(x2), cubic(x3), rk.prod(shrink1(x1),kron(x2-.5)), rk.prod(shrink1(x1),cubic(x2)), rk.prod(shrink1(x1),kron(x3-.5)), rk.prod(shrink1(x1),cubic(x3)), rk.prod(cubic(x2),kron(x3-.5)), rk.prod(kron(x2-.5),cubic(x3)), rk.prod(cubic(x2),cubic(x3)), rk.prod(shrink1(x1),kron(x2-.5),kron(x3-.5)), rk.prod(shrink1(x1),kron(x2-.5),cubic(x3)), rk.prod(shrink1(x1),cubic(x2),kron(x3-.5)), rk.prod(shrink1(x1),cubic(x2),cubic(x3)))) The estimates of all three environments are shown in Figure 4.7. Note that the first nine terms in (4.50) represent the mean tongue shape surface over time, and the last nine terms in (4.50) represent the departure of an environment from this mean surface. To look at the environment effect on the tongue shape surface over time, we calculate the posterior mean and standard deviation of the departure for each environment: > pred <- predict(ultra.elt.c.fit, newdata=expand.grid(x1=as.factor(1:3),x2=grid,x3=grid), terms=c(0,0,0,0,1,0,0,1,1,1,1,0,0,0,1,1,1,1))
0.4
0.6
0.8
len
) gth (mm
1.0
126
Smoothing Splines: Methods and Applications
The contour plots of the estimated departures for three environments are shown in Figure 4.8. Note that the significant regions at time 60 ms are similar to those in Figure 4.4.
cluster
2words 3
6
−4
−3
3
−2
−1
−1
1
−1
0
0 −1
0
4
−2
2
0
−1
−2
−3
1
0
50
schwa
−1
0
−5
time (ms) 100 150
200
2
4
−1
−1 0
−1
1 0
0
1
80
100 120 140 length (mm)
80
100 120 140 length (mm)
80
100 120 140 length (mm)
FIGURE 4.8 Ultrasound data, contour plots of estimated environment effect for three environments in the SS ANOVA model (4.50). Regions where the lower bounds of the 95% Bayesian confidence intervals are positive are shaded in dark grey. Regions where the upper bounds of the 95% Bayesian confidence intervals are negative are shaded in light grey.
4.9.2
Ozone in Arosa — Revisit
Suppose we want to investigate how ozone thickness changes over time by considering the effects of both month and year. In Section 2.10 we fitted an additive model (2.49) with the month effect modeled parametrically by a simple sinusoidal function, and the year effect modeled nonparametrically by a cubic spline. The following analyses show how to model both effects nonparametrically and investigate their interaction using SS ANOVA decompositions. We also show how to check the partial spline model (2.49). Let y be the response variable thick, x1 be the independent variable month scaled into [0, 1], and x2 be the independent variable year scaled into [0, 1]. It is reasonable to assume that the mean ozone thickness is a periodic function of x1 . We model the effect of x1 using a periodic spline space W22 (per) and the effect of x2 using a cubic spline space W22 [0, 1].
Smoothing Spline ANOVA
127
Therefore, we consider the SS ANOVA decomposition (4.23) for the tensor product space W2m1 (per) ⊗ W22 [0, 1]. The following statements fit model (4.23) with m1 = 2: > Arosa$x1 <- (Arosa$month-0.5)/12 > Arosa$x2 <- (Arosa$year-1)/45 > arosa.ssanova.fit1 <- ssr(thick~I(x2-0.5), data=Arosa, rk=list(periodic(x1), cubic(x2), rk.prod(periodic(x1),kron(x2-.5)), rk.prod(periodic(x1),cubic(x2)))) > summary(arosa.ssanova.fit1) ... GCV estimate(s) of smoothing parameter(s) : 5.442106e-06 2.154531e-09 3.387917e-06 2.961559e-02 Equivalent Degrees of Freedom (DF): 50.88469 Estimate of sigma: 14.7569 The mean function f (x) in model (4.23) evaluated at design points f = (f (x1 ), . . . , f (xn ))T can be represented as f = µ1 + f 1 + f 2 + f 12 , where 1 is a vector of all ones, f 1 = (f1 (x1 ), . . . , f1 (xn ))T , f 2 = (f2 (x1 ), . . . , f2 (xn ))T , f 12 = (f12 (x1 ), . . . , f12 (xn ))T , and f1 (x), f2 (x) and f12 (x1 ) are the main effect of x1 , the main effect of x2 , and the interaction between x1 and x2 . Eliminating the constant term, we have f ∗ = f ∗1 + f ∗2 + f ∗12 , Pn ∗ ∗ ∗ ∗ where a∗ = a − a ¯1, and a ¯ = i=1 ai /n. Let fˆ , fˆ 1 , fˆ 2 , and fˆ 12 be the ∗ ∗ ∗ ∗ estimates of f , f 1 , f 2 , and f 12 , respectively. To check the contributions of the main effects and interaction, we compute the quantities πk = ∗ ∗ ∗ ∗ ∗ ∗ (fˆ k )T fˆ /||fˆ ||2 for k = 1, 2, 12, and the Euclidean norms of fˆ , fˆ 1 , fˆ 2 , ∗ and fˆ 12 : > > > > > > > > >
f1 <- predict(arosa.ssanova.fit1, terms=c(0,0,1,0,0,0)) f2 <- predict(arosa.ssanova.fit1, terms=c(0,1,0,1,0,0)) f12 <- predict(arosa.ssanova.fit1, terms=c(0,0,0,0,1,1)) fs1 <- scale(f1$fit, scale=F) fs2 <- scale(f2$fit, scale=F) fs12 <- scale(f12$fit, scale=F) ys <- fs1+fs2+fs12 pi1 <- sum(fs1*ys)/sum(ys**2) pi2 <- sum(fs2*ys)/sum(ys**2)
128
Smoothing Splines: Methods and Applications
> pi12 <- sum(fs12*ys)/sum(ys**2) > print(round(c(pi1,pi2,pi12),4)) 0.9375 0.0592 0.0033 > print(round(sqrt(c(sum(fs1**2),sum(fs2**2), sum(fs12**2))),2)) 768.31 186.21 30.25 See Gu (2002) for more details about the cosine diagnostics. It is clear that the contribution from the interaction to the total variation is negligible. We also compute the posterior mean and standard deviation of the interaction f12 : > grid <- seq(0,1,len=50) > predict(arosa.ssanova.fit1, terms=c(0,0,0,0,1,1), expand.grid(x1=grid,x2=grid)) The estimate of the interaction is shown in Figure 4.9(a). Except for a narrow region, the zero function is contained in the 95% Bayesian confidence intervals. Thus the interaction is negligible.
6 8 month
10
−1
10 5
3
0
0
0
1 2
−2
−1
−1
−2
3
4
0
1
thickness 0
1
−1
2
0
−1
−5
0
−2
1
(c)
3
−10
0
2
0
1960
−1
year 1950
1
1
(b) −3
1940
−2
2
1930
1970
−3 2
1930
1940
year 1950
1960
1970
(a)
2
4
6 8 month
10
2
4
6 8 10 month
FIGURE 4.9 Arosa data, (a) plot of estimate of the interaction in model (4.23), (b) plot of estimate of the interaction in model (4.52), and (c) plot of estimate of the smooth component f1s in model (4.53). For plots (a) and (b), regions where the lower bounds of the confidence intervals are positive are shaded in dark grey, while regions where the upper bounds of the confidence intervals are negative are shaded in light grey. For plot (c), the solid line represents the estimate of f1s , the shaded region represents 95% Bayesian confidence intervals, and the dashed line represents the zero function.
Smoothing Spline ANOVA
129
We drop the interaction term and fit an additive model f (x1 , x2 ) = µ + β × (x2 − 0.5) + f2s (x2 ) + f1 (x1 )
(4.51)
as follows:
> update(arosa.ssanova.fit1, rk=list(periodic(x1),cubic(x2)))
The estimates of two main effects are shown in Figure 4.10.
Main effect of month
Main effect of year 40
o
o
thickness 0
o
o o
20
o
o
−40
oo oo oo o o
o
o
o o
o
o
o o ooo o o
o
oo o
o
o oo
−20
o
ooo o
o
o o o o o o
o o o
o
o
−40
thickness −20 0 20
40
o
o o
1
2
3
4
5
6 7 8 month
9
11
o
1926
1935
1944 1953 year
1962
1971
FIGURE 4.10 Arosa data, plots of the estimates of the main effects in the SS ANOVA model (4.51). Circles in the left panel represent monthly average thickness minus the overall mean. Circles in the right panel represent yearly average thickness minus the overall mean. Shaded regions are 95% Bayesian confidence intervals.
Now suppose we want to check if the partial spline model (2.49) is appropriate for the Arosa data. For the month effect, consider the trigonometric spline model W23 (per) with L = D{D2 +(2π)2 } defined in Section 2.11.5. The null space span{1, sin(2πx), cos(2πx)} corresponds to the sinusoidal model space assumed for the partial spline model (2.49). Consider the tensor product space W23 (per) ⊗ W22 [0, 1]. Define averaging
130
Smoothing Splines: Methods and Applications
operators: (1)
A1 f = (1)
A2 f = (1)
A3 f = (2)
A1 f = (2) A2 f (1)
(1)
=
Z Z
Z
Z
1
f dx1 , 0 1
f sin(2πx1 )dx1 , 0 1
f cos(2πx1 )dx1 , 0 1
f dx2 , 0
Z
′
0
(1)
1
f dx2 (x2 − 0.5).
(1)
Let A4 = I − A1 − A2 − A3
(2)
(2)
(2)
and A3 = I − A1 − A2 . Then
(1) (1) (1) (1) (2) (2) (2) f = A1 + A2 + A3 + A4 A1 + A2 + A3 f (1)
(2)
(1)
(2)
(1)
(2)
(1)
(2)
= A1 A1 f + A2 A1 f + A3 A1 f + A4 A1 f (1)
(2)
(1)
(2)
(1)
(2)
(1)
(2)
(1)
(2)
(1)
(2)
(1)
(2)
(1)
(2)
+ A1 A2 f + A2 A2 f + A3 A2 f + A4 A2 f + A1 A3 f + A2 A3 f + A3 A3 f + A4 A3 f
, µ + β1 × sin(2πx1 ) + β2 × cos(2πx1 ) + f1s (x1 )
+ β3 × (x2 − 0.5) + β4 × sin(2πx1 ) × (x2 − 0.5)
sl + β5 × cos(2πx1 ) × (x2 − 0.5) + f12 (x1 , x2 )
1s 2s ss + f2s (x2 ) + f12 (x1 , x2 ) + f12 (x1 , x2 ) + f12 (x1 , x2 ),
(4.52)
where µ represents the overall mean; β1 × sin(2πx1 ) and β2 × cos(2πx1 ) represent the parametric main effects of x1 ; f1s (x1 ) represents the smooth main effect of x1 ; β3 × (x2 − 0.5) represents the linear main effect of x2 ; f2s (x2 ) represents the smooth main effects of x2 ; and β4 × sin(2πx1 ) × sl 1s 2s (x2 −0.5), β5 ×cos(2πx1 )×(x2 −0.5), f12 (x1 , x2 ), f12 (x1 , x2 ), f12 (x1 , x2 ), ss and f12 (x1 , x2 ) represent interactions. We fit model (4.52) and compute the posterior mean and standard deviations for the overall interaction as follows: > arosa.ssanova.fit3 <- ssr(thick~sin(2*pi*x1)+cos(2*pi*x1) +I(x2-0.5)+I(sin(2*pi*x1)*(x2-0.5)) +I(cos(2*pi*x1)*(x2-0.5)), data=Arosa, rk=list(lspline(x1,type=‘‘sine1’’), cubic(x2), rk.prod(kron(sin(2*pi*x1)),cubic(x2)), rk.prod(kron(cos(2*pi*x1)),cubic(x2)),
Smoothing Spline ANOVA
131
rk.prod(lspline(x1,type=‘‘sine1’’), kron(x2-.5)), rk.prod(lspline(x1,type=‘‘sine1’’), cubic(x2)))) > > > >
ngrid <- 50 grid1 <- seq(.5/12,11.5/12,length=ngrid) grid2 <- seq(0,1,length=ngrid) predict(arosa.ssanova.fit3, expand.grid(x1=grid1,x2=grid2), terms=c(0,0,0,0,1,1,0,0,1,1,1,1))
The estimate of the interaction is shown in Figure 4.9(b). Except for a narrow region, the zero function is contained in the 95% Bayesian confidence intervals. Therefore, we drop interaction terms and consider the following additive model: f (x1 , x2 ) = µ + β1 × sin(2πx1 ) + β2 × cos(2πx1 ) + f1s (x1 ) (4.53) β3 × (x2 − 0.5) + f2s (x2 ). Note that the partial spline model (2.49) is a special case of model (4.53) with f1s (x1 ) = 0. We fit model (4.53) and compute posterior means and standard deviations for f1s (x1 ): > arosa.ssanova.fit4 <- ssr(thick~sin(2*pi*x1) +cos(2*pi*x1)+I(x2-0.5), data=Aros, rk=list(lspline(x1,type=‘‘sine1’’), cubic(x2))) > predict(arosa.ssanova.fit4, expand.grid(x1=grid1,x2=0), terms=c(0,0,0,0,1,0)) The estimate of f1s (x1 ) is shown in Figure 4.9(c) with 95% Bayesian confidence intervals. It is clear that f1s (x1 ) is nonzero, which indicates that the simple sinusoidal function is inadequate for modeling the month effect.
4.9.3
Canadian Weather — Revisit
Consider the Canadian weather data with annual temperature profiles from all 35 stations as functional data. To investigate how the weather patterns differ, Ramsay and Silverman (2005) divided Canada into four climatic regions: Atlantic, Continental, Pacific, and Arctic. Let y be the response variable temp, x1 be the independent variable region, and x2 be the independent variable month scaled into [0, 1]. The functional ANOVA (FANOVA) model (13.1) in Ramsay and Silverman (2005) assumes that yk,x1 (x2 ) = η(x2 ) + αx1 (x2 ) + ǫk,x1 (x2 ), (4.54)
132
Smoothing Splines: Methods and Applications
where yk,x1 (x2 ) is the temperature profile of station k in climate region x1 , η(x2 ) is the average temperature profile across all of Canada, αx1 (x2 ) is the departure of the region x1 profile from the population average profile η(x2 ), and ǫk,x1 (x2 ) are random errors. It is clear that the FANOVA can be derived from the SS ANOVA decomposition (4.22) by letting η(x2 ) = µ + f2 (x2 ) and αx1 (x2 ) = f1 (x1 ) + f12 (x1 , x2 ). The P4 side condition for the FANOVA model (4.54), x1 =1 αx1 (x2 ) = 0 for all x2 , is satisfied from the construction of the SS ANOVA decomposition. Model (4.54) is an example of situation (ii) in Section 2.10 where the dependent variable involves functional data.
2
temperature (C)
Atlantic
4
6
8
10 12
Pacific 20 10 0 −10 −20 −30
Continental
Arctic
20 10 0 −10 −20 −30 2
4
6
8
10 12 month
FIGURE 4.11 Canadian weather data, plots of temperature profiles of stations in four regions (thin lines), and the estimated profiles (thick lines).
Observed temperature profiles are shown in Figure 4.11. The following statements fit model (2.10) and compute posterior means and standard deviations for four regions: > x1 <- rep(as.factor(region),rep(12,35))
Smoothing Spline ANOVA
133
> x2 <- (rep(1:12,35)-.5)/12 > y <- as.vector(monthlyTemp) > canada.fit2 <- ssr(y~1, rk=list(shrink1(x1), periodic(x2), rk.prod(shrink1(x1),periodic(x2)))) > xgrid <- seq(.5/12,11.5/12,len=50) > zone <- c(‘‘Atlantic’’,‘‘Pacific’’, ‘‘Continental’’,‘‘Arctic’’) > grid <- data.frame(x1=rep(zone,rep(50,4)), x2=rep(xgrid,4)) > canada.fit2.p1 <- predict(canada.fit2, newdata=grid) Estimates of mean temperature functions for four regions are shown in Figure 4.11. To look at the region effects αx1 more closely, we compute their posterior means and standard deviations: > canada.fit2.p2 <- predict(canada.fit2, newdata=grid, terms=c(0,1,0,1)) Estimates of region effects and 95% Bayesian confidence intervals are shown in Figure 4.12. These estimates are similar to those in Ramsay and Silverman (2005).
4.9.4
Texas Weather
Instead of dividing stations into geological regions as in model (4.54), suppose we want to investigate how the weather patterns depend on geographical locations in terms of latitude and longitude. For illustration, consider the Texas weather data consisting of average monthly temperatures during 1961–1990 from 48 weather stations in Texas. Denote x1 = (lat, long) as the geological location of a station, and x2 as the month variable scaled into [0, 1]. We want to investigate how the expected temperature, f (x1 , x2 ), depends on both location and month variables. Average monthly temperatures are computed using monthly temperatures during 1986–1990 for all 48 stations and are used as observations of f (x1 , x2 ). For each fixed station, the annual temperature profile can be regarded as functional data on a continuous interval. Figure 4.13 shows these curves for all 48 stations. For each fixed month, the temperature surface as a function of latitude and longitude can be regarded as functional data on R2 . Figure 4.14 shows contour plots of observed surfaces for January, April, July, and October. A natural model space for the location variable is the thin-plate spline W22 (R2 ), and a natural model space for the month variable is W22 (per). Therefore, we fit the SS ANOVA model (4.24):
134
Smoothing Splines: Methods and Applications
2 Atlantic
4
6
8
10 12
Pacific 10
region effect
0 −10 Continental
Arctic
10 0 −10
2
4
6
8
10 12 month
FIGURE 4.12 Canadian weather data, plots of the estimated region effects to temperature, and 95% Bayesian confidence intervals.
> data(TXtemp); TXtemp1 <- TXtemp[TXtemp$year>1985,] > y <- gapply(TXtemp1, which=5, FUN=function(x) mean(x[x!=-99.99]), group=TXtemp1$stacod*TXtemp1$month) > tx.dat <- data.frame(y=as.vector(t(matrix(y,48,12, byrow=F)))) > tx.dat$x2 <- rep((1:12-0.5)/12, 48) > lat <- TXtemp$lat[seq(1, nrow(TXtemp),by=360)] > long <- TXtemp$long[seq(1, nrow(TXtemp),by=360)] > tx.dat$x11 <- rep(scale(lat), rep(12,48)) > tx.dat$x12 <- rep(scale(long), rep(12,48)) > tx.dat$stacod <- rep(TXtemp$stacod[seq(1,nrow(TXtemp), by=360)],rep(12,48)) > tx.ssanova <- ssr(y~x11+x12, data=tx.dat, rk=list(tp(list(x11,x12)), periodic(x2), rk.prod(tp.linear(list(x11,x12)),periodic(x2)), rk.prod(tp(list(x11,x12)), periodic(x2))))
135
temperature (F) 50 60 70
80
90
Smoothing Spline ANOVA
40
x x x x x x x x x x x x x x x x x x x x x x x x
x x x x x x x x x x x x x x x x x x x x x x x x
x x x x x x x x x x x x x x x x x x
x x x x x x x x x x x x x x
x x x x x x x x x x x x x x x x
x x x x x x x x x x x x x x x x
x x x x x x x x x x x x x x x x x x x x x x x
x x x x x x x x
10
11
x
x x x x x x x x x x x x x x x
x x x x x x x x x x x x x x x x x x x x x x
30
x x x
x x x x x x x x x x x x x x x x x x x x
x x x x x x x x x x x x x x x x x x
x x x x x x x x x x x x x x x
1
2
3
4
5
6 7 months
8
9
12
FIGURE 4.13 Texas weather data, plot of temperature profiles for all 48 stations. 36
38
56
January 40
April
58
42
62 60
46 44
48
64
50 54
52
79
82
78
July
58
60
October
62
80
81
70
72
56
78
66
68
64
64
66
82
83
78
64
68
84
80
70 85
68 72
FIGURE 4.14 Texas weather data, contour plots of observations for January, April, July, and October. Dots represent station locations.
136
Smoothing Splines: Methods and Applications
where tp.linear computes the RK, φ˜2 (x)φ˜2 (z) + φ˜3 (x)φ˜3 (z), of the subspace {φ˜2 , φ˜3 }. ls The location effect equals β1 φ˜1 (x1 )+β2 φ˜2 (x1 )+f1s (x1 )+f12 (x1 , x2 )+ ss f12 (x1 , x2 ). We compute posterior means and standard deviations of the location effect for the southmost (Rio Grande City 3W), northmost (Stratford), westmost (El Paso WSO AP), and eastmost (Marshall) stations: > selsta <- c(tx.dat[tx.dat$x11==min(tx.dat$x11),7][1], tx.dat[tx.dat$x11==max(tx.dat$x11),7][1], tx.dat[tx.dat$x12==min(tx.dat$x12),7][1], tx.dat[tx.dat$x12==max(tx.dat$x12),7][1]) > sellat <- sellong <- NULL > for (i in 1:4) { sellat <- c(sellat, tx.dat$x11[tx.dat$stacod==selsta[i]][1]) sellong <- c(sellong, tx.dat$x12[tx.dat$stacod==selsta[i]][1]) } > grid <- data.frame(x11=rep(sellat,rep(40,4)), x12=rep(sellong ,rep(40,4)), x2=rep(seq(0,1,len=40), 4)) > tx.pred1 <- predict(tx.ssanova, grid, terms=c(0,1,1,1,0,1,1)) The estimates of these effects and 95% Bayesian confidence intervals are shown in Figure 4.15. The curve in each plot shows how temperature profile of that particular station differ from the average profile among the 48 stations. It is clear that the temperature in Rio Grande City is higher than average, and the temperature in Stratford is lower than average, especially in the winter. The temperature in El Paso is close to average in the first half year, and lower than average in the second half year. The temperature in Marshall is slightly above average. ls ss The month effect equals f2 (x2 )+f12 (x1 , x2 )+f12 (x1 , x2 ). We compute posterior means and standard deviations of the month effect: > tx.pred2 <- predict(tx.ssanova, terms=c(0,0,0,0,1,1,1)) The estimates of these effects for January, April, July, and October are shown in Figure 4.16. Each plot in Figure 4.16 shows how temperature pattern for that particular month differ from the average pattern among all 12 months. As expected, the temperature in January is colder than average, while the temperature in July is warmer than average. In general, the difference becomes smaller from north to south. The temperatures in April and October are close to average.
Smoothing Spline ANOVA
0
temperature (F)
Rio Grande City 3W
2
4
137
6
8 10 12
Stratford 10 5 0 −5 −10 −15
El Paso WSO AP
Marshall
10 5 0 −5 −10 −15 0
2
4
6
8 10 12 month
FIGURE 4.15 Texas weather data, plots of the location effect (solid lines) for four selected stations with 95% Bayesian confidence intervals (dashed lines).
138
Smoothing Splines: Methods and Applications
January
April
0.3
0
−1
9
−20
0.9
−2
−18
−19
−16
−16
5 −1
1.2
0
0.6
0.9
−17
July
21
October
0.8
0.8
19
0 0.4
17 15
15
13
0.8
1.2
1.2
FIGURE 4.16 Texas weather data, plots of the month effect for January, April, July, and October.
Chapter 5 Spline Smoothing with Heteroscedastic and/or Correlated Errors
5.1
Problems with Heteroscedasticity and Correlation
In the previous chapters we have assumed that observations are independent with equal variances. These assumptions may not be appropriate for many applications. This chapter presents spline smoothing methods for heteroscedastic and/or correlated observations. Before introducing these methods, we first illustrate potential problems associated with the presence of heteroscedasticity and correlation in spline smoothing. We use the following simulation to illustrate potential problems with heteroscedasticity. Observations are generated from model (1.1) with f (x) = sin(4πx2 ), xi = i/n for i = 1, . . . , n and n = 100. Random errors are generated independently from the Gaussian distribution with mean zero and variance σ 2 exp{α|f (x)|}. Therefore, we have unequal variances when α 6= 0. We set σ = 0.05 and α = 4. For each simulated data, we first fit the cubic spline directly using PLS with GCV and GML choices of smoothing parameters. Note that these direct fits ignore heteroscedasticity. We then fit the cubic spline using the penalized weighted LS (PWLS) introduced in Section 5.2.1 with known weights W = diag(exp{−4|f (x1 )|}, . . . , exp{−4|f (xn )|}). For each fit, we compute weighted MSE (WMSE) n
WMSE =
1X wi (fˆ(xi ) − f (xi ))2 , n i=1
where wi = exp{−4|f (xi )|}. We also construct 95% Bayesian confidence intervals for each fit. The simulation is repeated 100 times. Figure 5.1 shows the performances of unweighted and weighted methods in terms of WMSE and coverages of Bayesian confidence intervals. Figure 5.1(a)
139
140
Smoothing Splines: Methods and Applications (b)
+ +
GCV
GCVW
GML
average coverage 0.85 0.95
0.0005
+
+ GMLW
0.2
0.4
0.6
x
unweighted
0.8
1.0
weighted
(d)
pointwise coverage 0.0 0.2 0.4 0.6 0.8 1.0
pointwise coverage 0.0 0.2 0.4 0.6 0.8 1.0
(c)
0.0
+
+
0.75
WMSE
0.0015
(a)
0.0
0.2
0.4
0.6
0.8
1.0
x
FIGURE 5.1 (a) Boxplots of WMSEs based on PLS with GCV and GML choices of the smoothing parameter and PWLS with GCV (labeled as GCVW) and GML (labeled as GMLW) choices of the smoothing parameter. Average WMSEs are marked as pluses; (b) boxplots of across-the-function coverages of 95% Bayesian confidence intervals for the PLS fits with GML choice of the smoothing parameter (labeled as unweighted) and PWLS fits with GML choice of the smoothing parameter (labeled as weighted). Average coverages are marked as pluses; (c) plot of pointwise coverages (solid line) of 95% Bayesian confidence intervals for the PLS fits with GML choice of the smoothing parameter; (d) plot of pointwise coverages (solid line) of 95% Bayesian confidence intervals for the PWLS fits with GML choice of the smoothing parameter. Dotted lines in (b), (c), and (d) represent the nominal value. Dashed lines in (c) and (d) represent a scaled version of the variance function exp{α|f (x)|}.
indicates that, even though ignoring heteroscedasticity, the unweighted methods provide good fits to the function. The weighted methods lead to better fits. Bayesian confidence intervals based on both methods provide the intended across-the-function coverages (Figure 5.1(b)). However, Figure 5.1(c) reveals the problem with heteroscedasticity: point-
Spline Smoothing with Heteroscedastic and/or Correlated Errors 141 wise coverages in regions with larger variances are smaller than the nominal value, while pointwise coverages in other regions are larger than the nominal value. Obviously, this is caused by ignoring heteroscedasticity. Bayesian confidence intervals based on the PWLS method overcomes this problem (Figure 5.1(d)).
0.0
1.5 1.0
y
0.5 0.0 −0.5 −1.0 −1.5
0.2
0.4
0.6
0.8
1.0
UBR
GCV
GML
oo o o oo o o o o o oo oo ooo o o o o o o o o oo o oo oo oo o ooo ooo oo o o oo o o oo o o o o o o o ooo o o o oo o o o ooo ooo o oo oo oo o o ooo o oo o o o o o o oo o
oo o o oo o o o o o oo oo ooo o o o o o o o o oo o oo oo oo o ooo ooo oo o o oo o o oo o o o o o o o ooo o o o oo o o o ooo ooo o oo oo oo o o ooo o oo o o o o o o oo o
oo o o oo o o o o o oo oo ooo o o o o o o o o oo o oo oo oo o ooo ooo oo o o oo o o oo o o o o o o o ooo o o o oo o o o ooo ooo o oo oo oo o o ooo o oo o o o o o o oo o
0.0
0.2
0.4
0.6
0.8
0.0
1.0
0.2
0.4
0.6
0.8
1.0
x
FIGURE 5.2 Plots of the true function (dashed lines), observations (circles), and the cubic spline fits (solid lines) with UBR, GCV, and GML choices of smoothing parameters. The true variance is used in the calculation of the UBR criterion. Comparing with heteroscedasticity, the potential problems associated with correlation are more fundamental and difficult to deal with. For illustration, again we simulate data from model (1.1) with f (x) = sin(4πx2 ), xi = i/n for i = 1, . . . , n and n = 100. Random errors ǫi are generated by a first-order autoregressive model (AR(1)) with mean zero, standard deviation 0.2, and first-order correlation 0.6. Figure 5.2 shows the simulated data, the true function, and three cubic spline fits with smoothing parameters chosen by the UBR, GCV, and GML methods, respectively. The true variance is used in the calculation of the UBR criterion. All fits are wiggly, which indicates that the estimated smoothing parameters are too small. This undersmoothing phenomenon is common in the presence of positive correlation. Ignoring correlation, the UBR, GCV, and GML methods perceive that all the trend (signal) in the data is due to the mean function f and attempt to incorporate that trend into the estimate. Correlated random errors may induce local trend and thus fool these methods to select smaller smoothing parameters such that the local trend can be picked up.
142
5.2
Smoothing Splines: Methods and Applications
Extended SS ANOVA Models
Suppose observations are generated by yi = Li f + ǫi ,
i = 1, . . . , n,
(5.1)
where f is a multivariate function in a model space M, and Li are bounded linear functionals. Let ǫ = (ǫ1 , . . . , ǫn )T . We assume that E(ǫ) = 0 and Cov(ǫ) = σ 2 W −1 . In this chapter we consider the SS ANOVA model space M = H0 ⊕ H1 ⊕ · · · ⊕ Hq ,
(5.2)
where H0 is a finite dimensional space collecting all functions that are not going to be penalized, and H1 , . . . , Hq are orthogonal RKHS’s with RKs Rj for j = 1, . . . , q. Model (5.1) is an extension of the SS ANOVA model (4.31) with non-iid random errors. See Section 8.2 for a more general model involving linear functionals and non-iid errors.
5.2.1
Penalized Weighted Least Squares
Our goal is to estimate f as well as W when it is unknown. We first assume that W is fixed and consider the estimation of the nonparametric function f . A direct generalization of the PLS (4.32) is the following PWLS: q 1 X (y − f )T W (y − f ) + λ θj−1 kPj f k2 , (5.3) min f ∈M n j=1
where y = (y1 , . . . , yn )T , f = (L1 f, . . . , Ln f )T , and Pj is the orthogonal projection in M onto Hj . Let θ = (θ1 , . . . , θq ). Denote φ1 , . . . , φp as basis functions of H0 , p T = {Li φν }ni=1 ν=1 , Σk = {Li Lj Rk }ni,j=1 for k = 1, . . . , q, and Σθ = Pq j=1 θj Σj . As in the previous chapters, we assume that T is of full column rank. The same arguments in Section 2.4 apply to the PWLS. Therefore, by the Kimeldorf–Wahba representer theorem, the solution to (5.3) exists and is unique, and the solution can be represented as fˆ(x) =
p X
ν=1
dν φν (x) +
n X i=1
ci
q X j=1
θj Li(z ) Rj (x, z).
Spline Smoothing with Heteroscedastic and/or Correlated Errors 143 Let c = (c1 , . . . , cn )T and d = (d1 , . . . , dp )T . Let fˆ = (L1 fˆ, . . . , Ln fˆ)T be the vector of fitted values. It is easy to check that fˆ = T d + Σθ c and kP1∗ fˆk2∗ = cT Σθ c. Then the PWLS (5.3) reduces to
1 (y − T d − Σθ c)T W (y − T d − Σθ c) + λcT Σθ c. (5.4) n Taking the first derivatives leads to the following equations for c and d: (Σθ W Σθ + nλΣθ )c + Σθ W T d = Σθ W y, T T W Σθ c + T T W T d = T T W y.
(5.5)
It is easy to check that a solution to (Σθ + nλW −1 )c + T d = y, T T c = 0,
(5.6)
is also a solution to (5.5). Let M = Σθ + nλW −1 and R T = (Q1 Q2 ) 0 be the QR decomposition of T . Then the solutions to (5.6) are c = Q2 (QT2 M Q2 )−1 QT2 y, d = R−1 QT1 (y − M c).
(5.7)
Based on the first equation in (5.6) and the fact that fˆ = T d + Σθ c, we have fˆ = y − nλW −1 c = H(λ, θ)y
where
H(λ, θ) = I − nλW −1 Q2 (QT2 M Q2 )−1 QT2
(5.8)
is the hat matrix. The dependence of H on the smoothing parameters is expressed explicitly in (5.8). In the reminder of this chapter, for simplicity, the notation H will be used. Note that, different from the independent case, H may be asymmetric. To solve (5.6), for fixed W and smoothing parameters, consider trans˜ = W 1/2 Σ W 1/2 , c ˜ = W 1/2 y, T˜ = W 1/2 T , Σ ˜ = W −1/2 c, formations y θ θ ˜ and d = d. Then equations in (5.6) are equivalent to the following equations ˜=y ˜ + nλI)˜ ˜, (Σ c + T˜d θ (5.9) ˜ = 0. T˜ T c Note that equations in (5.9) have the same form as those in (2.21); thus ˜ Transforming ˜ and d. methods in Section 2.4 can be used to compute c back, we have the solutions of c and d.
144
Smoothing Splines: Methods and Applications
5.2.2
UBR, GCV and GML Criteria
We now extend the UBR, GCV, and GML criteria for the estimation of smoothing parameters λ and θ as well as W when it is unknown. Define the weighted version of the loss function L(λ, θ) =
1 ˆ (f − f )T W (fˆ − f ) n
(5.10)
and weighted MSE as WMSE(λ, θ) , EL(λ, θ). Then WMSE(λ, θ) =
1 T σ2 f (I − H T )W (I − H)f + trH T W HW −1 . (5.11) n n
It is easy to check that an unbiased estimate of WMSE(λ, θ) + σ 2 is UBR(λ, θ) =
2σ 2 1 T y (I − H T )W (I − H)y + trH. n n
(5.12)
The UBR estimates of smoothing parameters and W when it is unknown are the minimizers of UBR(λ, θ). To extend the CV and GCV methods, we first consider the special case when W is diagonal: W = diag(w1 , . . . , wn ). Denote fˆ[i] as the minimizer of the PWLS (5.3) based on all observations except yi . It is easy to check that the leaving-out-one lemma in Section 3.4 still holds and Li fˆ[i] −yi = (Li fˆ−yi )/(1−hii ), where hii are the diagonal elements of H. Then the cross-validation criterion is CV(λ, θ) ,
n n 2 1 X ˆ[i] 1 X (Li fˆ − yi )2 wi Li f − yi = wi . n i=1 n i=1 (1 − hii )2
(5.13)
Replacing hii by the average of diagonal elements leads to the GCV criterion 1 ||W 1/2 (I − H)y||2 (5.14) GCV(λ, θ) , n 2 . 1 n tr(I − H) Next consider the following model for clustered data yij = Lij f + ǫij ,
i = 1, . . . , m; j = 1, . . . , ni ,
(5.15)
where yij is the jth observation in cluster i, and Lij are bounded linear Pm functionals. Let n = i=1 ni , y i = (yi1 , . . . , yini )T , y = (y T1 , . . . , yTm )T , ǫi = (ǫi1 , . . . , ǫini )T , and ǫ = (ǫT1 , . . . , ǫTm )T . We assume that Cov(ǫi ) = σ 2 Wi−1 and observations between clusters are independent. Consequently, Cov(ǫ) = σ 2 W −1 , where W = diag(W1 , . . . , Wm ) is a block diagonal matrix. Assume that f ∈ M, where M is the model space in
Spline Smoothing with Heteroscedastic and/or Correlated Errors 145 (5.2). We estimate f using the PWLS (5.3). Let fˆ be the PWLS estimate based on all observations, and fˆ[i] be the PWLS estimate based on all observations except those from the ith cluster y i . Let f i = (Li1 f, . . . , Lini f )T , f = (f T1 , . . . , f Tm )T , fˆ i = (Li1 fˆ, . . . , Lini fˆ)T , fˆ = T T [i] [i] [i] (fˆ , . . . , fˆ )T , fˆ = (Lj1 fˆ[i] , . . . , Ljnj fˆ[i] )T , and fˆ = ((fˆ )T , . . . , 1 [i]
m
j
1
(fˆ m )T )T .
Leaving-out-one-cluster Lemma For any fixed i, fˆ[i] is the minimizer of T [i] 1X T 1 ˆ [i] f i − f i Wi fˆ i − f i + y j − f j Wj y j − f j n n j6=i
+λ
q X j=1
θj−1 kPj f k2 .
(5.16)
[Proof] For any function f ∈ M, we have [i] 1X T T 1 ˆ [i] f i − f i Wi fˆ i − f i + y j − f j Wj y j − f j n n j6=i
+λ
q X j=1
θj−1 kPj f k2
q X T 1X y j − f j Wj y j − f j + λ θj−1 kPj f k2 ≥ n j=1 j6=i
≥
q X [i] T [i] 1 X y j − fˆ j Wj y j − fˆ j + λ θj−1 kPj fˆ[i] k2 n j=1 j6=i
1 X [i] [i] [i] [i] T 1 ˆ [i] ˆ [i] T Wi fˆ i − fˆ i + Wj y j − fˆ j fi − fi y j − fˆ j = n n j6=i
+λ
q X j=1
θj−1 kPj fˆ[i] k2 .
[i] The leaving-out-one-cluster lemma implies that fˆ = Hy and fˆ = Hy [i] , where H is the hat matrix and y [i] is the same as y except that y i [i] is replaced by fˆ . Divide the hat matrix H according to clusters such i
146
Smoothing Splines: Methods and Applications
that H = {Hik }m i,k=1 , where Hik is an ni × nk matrix. Then we have fˆ i =
m X
Hij y j ,
j=1
[i] fˆ i =
X
[i] Hij y j + Hii fˆ i .
j6=i
Assume that I − Hii is invertible. Then
[i] fˆ i − y i = (I − Hii )−1 (fˆ i − y i ).
The cross-validation criterion CV(λ, θ) m [i] T 1 X ˆ [i] , f i − y i Wi fˆ i − y i n i=1 m
T 1 X ˆ f i − yi (I − Hii )−T Wi (I − Hii )−1 fˆ i − yi . (5.17) = n i=1
Replacing I −Hii by its generalized average Gi (Ma, Dai, Klein, Klein, Lee and Wahba 2010), we have m T 1 X ˆ GCV(λ, θ) , (5.18) f i − y i Gi Wi Gi fˆ i − y i , n i=1 where Gi = ai Ini − bi 1ni 1Tni , ai = 1/(δi − γi ), bi = γi /[(δi − γi ){δi + (nP i − 1)γ i }], δi = (n − trH)/mni , γi = 0 when ni = 1 and γi = m P − i=1 s6=t hist /mni (ni − 1) when ni > 1, hist is the element of the sth row and tth column of the matrix Hii , Ini is an identity matrix of size ni , and 1ni is an ni -vector of all ones. The CV and GCV estimates of smoothing parameters and W when it is unknown are the minimizers of the CV and GCV criteria. To derive the GML criterion, we first construct a Bayes model for the extended SS ANOVA model (5.1). Assume the same prior for f as in (4.42): F (x) =
p X
ν=1
q X p ζν φν (x) + δ θj Uj (x), 1 2
(5.19)
j=1
iid
where ζν ∼ N(0, κ), Uj (x) are independent, zero-mean Gaussian stochastic processes with covariance function Rj (x, z), ζν and Uj (x) are mutually independent, and κ and δ are positive constants. Suppose observations are generated from yi = Li F + ǫi ,
i = 1, . . . , n,
(5.20)
Spline Smoothing with Heteroscedastic and/or Correlated Errors 147 where ǫ = (ǫ1 , . . . , ǫn )T ∼ N(0, σ 2 W −1 ). Let L0 be a bounded linear functional on M. Let λ = σ 2 /nδ. The same arguments in Section 3.6 hold when M = Σ + nλI is replaced by M = Σθ + nλW −1 in this chapter (Wang 1998b). Therefore, lim E(L0 F |y) = L0 fˆ,
κ→∞
and an extension of the GML criterion is GML(λ, θ) =
y T W (I − H)y
1
{det+ (W (I − H))} n−p
,
(5.21)
where det+ is the product of the nonzero eigenvalues. The GML estimates of smoothing parameters and W when it is unknown are the minimizers of GML(λ, θ). The GML estimator of the variance σ 2 is (Wang 1998b) σ ˆ2 =
5.2.3
y T W (I − H)y . n−p
(5.22)
Known Covariance
In this section we discuss the implementation of the UBR, GCV, and GML criteria in Section 5.2.2 when W is known. In this situation we only need to estimate the smoothing parameters λ and θ. Consider transformations discussed at the end of Section 5.2.1. Let f˜ = W 1/2 fˆ ˜ be the hat matrix associated be the fits to the transformed data, and H ˜y ˜ = with the transformed data. Then Hy = fˆ = W −1/2 f˜ = W −1/2 H ˜ 1/2 y for any y. Therefore, W −1/2 HW ˜ 1/2 . H = W −1/2 HW
(5.23)
From (5.23), the UBR, GCV, and GML in (5.12), (5.14), and (5.21) can be rewritten based on the transformed data as 2 1 ˜ y ||2 + 2σ trH, ˜ ||(I − H)˜ n n P n 1 ˜ y ||2 ||(I − H)˜ GCV(λ, θ) = n n i=1 o2 , 1 ˜ tr(I − H)
UBR(λ, θ) =
(5.24) (5.25)
n
GML(λ, θ) =
˜ y ˜ T (I − H)˜ Cy , 1 + ˜ n−p {det (I − H)}
(5.26)
where C in GML(λ, θ) is a constant independent of λ and θ. Equations (5.24), (5.25), and (5.26) indicate that, when W is known, the UBR,
148
Smoothing Splines: Methods and Applications
GCV, and GML estimates of smoothing parameters can be calculated based on the transformed data using the method described in Section 4.7.
5.2.4
Unknown Covariance
We now consider the case when W is unknown. When a separate method is available for estimating W , one approach is to estimate the function f and covariance W iteratively. For example, the following two-step procedure is simple and easy to implement: (1) Estimate the function with a “sensible” choice of the smoothing parameter; (2) Estimate the covariance using residuals; and (3) Estimate the function again using the estimated covariance. However, it may not work in certain situations due to the interplay between the smoothing parameter and correlation. We use the following two simple simulations to illustrate the potential problem associated with the above iterative approach. In the first simulation, n = 100 observations are generated according to model (1.1) iid
with f (x) = sin(4πx2 ), xi = i/n for i = 1, . . . , n, and ǫi ∼ N(0, 0.22 ). We fit a cubic spline with a fixed smoothing parameter λ such that log10 (nλ) = −3.5. Figure 5.3(a) shows the fit, which is slightly oversmoothed. The estimated autocorrelation function (ACF) of residuals (Figure 5.3(b)) suggests an autoregressive structure even though the true random errors are independent. It is clear that the leftover signal due to oversmoothing shows up in the residuals. In the second simulation, n = 100 observations are generated according to model (1.1) with f (x) ≡ 0, xi = i/n for i = 1, . . . , n, and ǫi are generated by an AR(1) model with mean zero, standard deviation 0.2, and first-order correlation 0.6. Figure 5.3(c) shows the cubic spline fit with GCV choice of the smoothing parameter. The fit picks up local trend in the AR(1) process, and the estimated ACF of residuals (Figure 5.3(d)) does not reveal any autoregressive structure. In both cases, the mean functions are incorrectly estimated, and the conclusions about the error structures are erroneous. These two simulations indicate that a wrong choice of the smoothing parameter in the first step will lead to a deceptive serial correlation in the second step. In the most general setting where no parametric shape is assumed for the mean or the correlation function, the model is essentially unidentifiable. In the following we will model the correlation structure parametrically. Specifically we assume that W depends on an unknown vector of parameters τ . Models for W −1 will be discussed in Section 5.3. When there is no strong connection between the smoothing and correlation parameters, an iterative procedure as described earlier may be
Spline Smoothing with Heteroscedastic and/or Correlated Errors 149
oo o oo o o ooo o o o oo o oo o oo o o oo o oo oo oooo oo o o o o o o o o o o o o o oooo o
o
o
o
o
oo
oo o oo
o
o
−1.0
o
o
o oo oo o
o
o o o
o o oooo
o
0.0
0.2
0.4
0.6
(b)
1.0
oo oo o o o o o o o
ACF 0.2 0.6
o o
−0.2
y 0.0 0.5 1.0
(a) o
0.8
1.0
0
5
x
o
o
−0.6
−0.2
o
0.0
o
o
o o
o
o oo o o oo oo o oo oo o o ooo oo o o o o oo o o o ooo o o o o o o o oo oo o oo o o oo o o o o o oo o o o o ooo ooo o o o o o oo o o o
oo o o
o
0.2
0.4
0.6
x
0.8
1.0
15
20
ACF
y 0.2
oo
0.5
o
0.0
o oo
o
20
−0.5
0.6
o o
15
(d)
1.0
(c)
10 Lag
0
5
10 Lag
FIGURE 5.3 For the first simulation, plots of (a) the true function (dashed lines), observations (circles), and the cubic spline fits (solid line) with log10 (nλ) = −3.5, and (b) estimated ACF of residuals. For the second simulation, plots of (c) the true function (dashed lines), observations (circles), and the cubic spline fits (solid line) with GCV choice of the smoothing parameter, and (d) estimated ACF of residuals.
used to estimate f and W . See Sections 5.4.1 and 6.4 for examples. When there is a strong connection, it is helpful to estimate the smoothing and correlation parameters simultaneously. One may use the UBR, GCV, or GML criterion to estimate the smoothing and correlation parameters simultaneously. It was found that the GML method performs better than the UBR and GCV methods (Wang 1998b). Therefore, in the following, we present the implementation of the GML method only. To compute the minimizers of the GML criterion (5.21), we now construct a corresponding LME model for the extended SS ANOVA model (5.1). Let Σk = Zk ZkT , where Zk is an n × mk matrix with
150
Smoothing Splines: Methods and Applications
mk = rank(Σk ). Consider the following LME model y = Tζ +
q X
Zk uk + ǫ,
(5.27)
k=1
where ζ is a p-dimensional vector of deterministic parameters, uk are mutually independent random effects, uk ∼ N(0, σ 2 θk Imk /nλ), Imk is the identity matrix of order mk , ǫ ∼ N(0, σ 2 W −1 ), and uk are independent of ǫ. It is not difficult to show that the spline estimate based on the PWLS is the BLUP estimate, and the GML criterion (5.21) is the REML criterion based on the LME model (5.27) (Opsomer, Wang and Yang 2001). Details for the more complicated semiparametric mixedeffects models will be given in Chapter 9. In the ssr function, we first calculate Zk through Cholesky decomposition. Then we calculate the REML (GML) estimates of λ, θ and τ using the function lme in the nlme library by Pinheiro and Bates (2000).
5.2.5
Confidence Intervals
Consider the Bayes model defined in (5.19) and (5.20). Conditional on W , it is not difficult to show that formulae for posterior means and covariances in Section 4.8 hold when M = Σθ + nλI is replaced by M = Σθ +nλW −1 . Bayesian confidence intervals can be constructed similarly. When the matrix W involves unknown parameters τ , it is replaced by W (ˆ τ ) in the construction of Bayesian confidence intervals. This simple plug-in approach does not account for the variation in estimating the covariance parameters τ . The bootstrap approach may also be used to construct confidence intervals. More research is necessary for inferences on the nonparametric function f as well as on parameters τ .
5.3
Variance and Correlation Structures
In this section we discuss commonly used models for the variance– covariance matrix W −1 . The matrix W −1 can be decomposed as W −1 = V CV, where V is a diagonal matrix with positive diagonal elements, and C is a correlation matrix with all diagonal elements equal to one. The matrices V and C describe the variance and correlation, respectively.
Spline Smoothing with Heteroscedastic and/or Correlated Errors 151 The above decomposition allows us to develop separate models for the variance structure and correlation structure. We now describe structures available for the assist package. Details about these structures can be found in Pinheiro and Bates (2000). Consider the variance structure first. Define general variance function as Var(ǫi ) = σ 2 v 2 (µi , z i ; ζ), (5.28) where v is a known variance function of the mean µi = E(yi ) and a vector of covariates z i associated with the variance, and ζ is a vector of variance parameters. As in Pinheiro and Bates (2000), when v depends on µi , the pseudo-likelihood method will be used to estimate all parameters and the function f . In the ssr function, a variance matrix (vector) or a model for the variance function is specified using the weights argument. The input to the weights argument may be the matrix W when it is known. Furthermore, when W is a known diagonal matrix, the diagonal elements may be specified as a vector by the weights argument. The input to the weights argument may also be a varFunc structure specifying a model for variance function v in (5.28). All varFunc objects available in the nlme package are available for the assist package. The varFunc structure is defined through the function v(s; ζ), where s can be either a variance covariate or the fitted value. Standard varFunc classes and their corresponding variance functions v are listed in Table 5.1. Two or more variance models may be combined using the varComb constructor.
TABLE 5.1
Standard varFunc classes Class v(s; p ζ) varFixed |s| varIdent ζs , s is a stratification variable |s|ζ varPower exp(ζs) varExp varConstPower ζ1 + |s|ζ2
Next consider the correlation structure. Assume the general isotropic correlation structure cor(ǫi , ǫj ) = h(d(pi , pj ); ρ),
(5.29)
where pi and pj are position vectors associated with observations i and j respectively; h is a known correlation function of the distance d(pi , pj )
152
Smoothing Splines: Methods and Applications
such that h(0; ρ) = 1; and ρ is a vector of the correlation parameters. In the ssr function, correlation structures are specified as corStruct objects through the correlation argument. There are two common families of correlation structures: serial correlation structures for time series and spatial correlation structures for spatial data. For time series data, observations are indexed by an one-dimensional position vector, and d(pi , pj ) represents lags that take nonnegative integers. Therefore, serial correlation is determined by the function h(k; ρ) for k = 1, 2, . . . . Standard corStruct classes for serial correlation structures and their corresponding correlation functions h are listed in Table 5.2.
TABLE 5.2
Standard corStruct classes for serial correlation structures Class h(k; ρ) corCompSymm ρ corSymm ρk ρk corAR1 correlation function given in (5.30) corARMA
The classes corCompSymm and corSymm correspond to the compound symmetry and general correlation structures. An autoregressive-moving average (ARMA(p,q)) model assumes that ǫt =
p X
φi ǫt−i +
i=1
q X
θj at−j + at ,
j=1
where at are iid random variables with mean zero and a constant variance, φ1 , . . . , φp are autoregressive parameters, and θ1 , . . . , θq are moving average parameters. The correlation function is defined recursively as φ1 h(|k − 1|; ρ) + · · · + φp h(|k − p|; ρ)+ h(k; ρ) = θ1 ψ(k − 1; ρ) + · · · + θq ψ(k − q; ρ), k ≤ q, (5.30) φ1 h(|k − 1|; ρ) + · · · + φp h(|k − p|; ρ), k > q, where ρ = (φ1 , . . . , φp , θ1 , . . . , θq ) and ψ(k; ρ) = E(ǫt−k at )/Var(ǫt ). The continuous AR(1) correlation function is defined as h(s; ρ) = ρs ,
s ≥ 0,
ρ ≥ 0.
The corCAR1 constructor specifies the continuous AR(1) structure. For spatial data, locations pi ∈ Rr . Any well-defined distance such as
Spline Smoothing with Heteroscedastic and/or Correlated Errors 153 P Euclidean d(pi , pj ) = { rk=1 (pik − pjk )2 }1/2 , Manhattan d(pi , pj ) = P r k=1 |pik −pjk |, and maximum distance d(pi , pj ) = max1≤k≤r |pik −pjk | may be used. The correlation function structure is defined through the function h(s; ρ), where s ≥ 0. Standard corStruct classes for spatial correlation structures and their corresponding correlation functions h are listed in Table 5.3. The function I(s < ρ) equals 1 when s < ρ and 0 otherwise. When desirable, the following correlation function allows a nugget effect: (1 − c0 )hcont (s; ρ), s > 0, hnugg (s, c0 ; ρ) = 1, s = 0, where hcont is any standard correlation function that is continuous in s and 0 < c0 < 1 is a nugget effect.
TABLE 5.3
Standard corStruct classes for spatial correlation structures Class h(s; ρ) corExp exp(−s/ρ) corGaus exp{−(s/ρ)2 } corLin (1 − s/ρ)I(s < ρ) corRatio 1/{1 + (s/ρ)2 } corSpher {1 − 1.5(s/ρ) + 0.5(s/ρ)3 }I(s < ρ)
5.4 5.4.1
Examples Simulated Motorcycle Accident — Revisit
Plot of observations in Figure 3.8 suggests that variances may change over time. For the partial spline fit to model (3.53), we compute squared residuals and plot the logarithm of squared residuals over time in Figure 5.4(a). Cubic spline fit to the logarithm of squared residuals in Figure 5.4(a) indicates that the constant variance assumption may be violated. We then fit model (3.53) again using PWLS with weights fixed as the estimated variances: > r <- residuals(mcycle.ps.fit2); y <- log(r**2)
154
Smoothing Splines: Methods and Applications
> mcycle.v <- ssr(y~x, cubic(x), spar=‘‘m’’) > update(mcycle.ps.fit2, weights=exp(mcycle.v$fit)) > predict(mcycle.ps.fit3, newdata=data.frame(x=grid, s1=(grid-t1)*(grid>t1), s2=(grid-t2)*(grid>t2), s3=(grid-t3)*(grid>t3)))
(a)
(b)
o
0
10
acceleration (g) −50 0 50
o
20 30 40 time (ms)
50
60
o o
o o ooo o oo o o oo o o o oo o o o o oo o o o o oooo ooooooooo oooo o oo o ooo oo o o o o o ooo oo o o o o o o o o ooooo oo o o o oo o oo ooo o oo o oo oo o o o oo o oo o o oo o oo o o o o o ooo oo o o
−150
log(squared residual) −5 0 5
o o ooo oo o oo o o o oo o o oooo ooo ooo o o oo o o o o o ooooooo oo o oo o ooo oo o o oo o oo ooo o o o ooooo o o o o o o o o o oo o o o o o o o o o o o o o o o oo o o o oo o oo o oo o o o o o o o o oo o o o o o o o o
0
10
20 30 40 time (ms)
50
60
FIGURE 5.4 Motorcycle data, plots of (a) logarithm of the squared residuals (circles) based on model (3.53), and the cubic spline fit (line) to logarithm of the squared residuals; and (b) observations (circles), new PWLS fit (line), and 95% Bayesian confidence intervals (shaded region).
The PWLS fit and 95% Bayesian confidence intervals are shown in 5.4(b). The impact of the unequal variances is reflected in the widths of confidence intervals. The two-step approach adapted here is crude, and the variation in the estimation of the variance function has been ignored in the construction of the confidence intervals. Additional methods for estimating the mean and variance functions will be discussed in Section 6.4.
5.4.2
Ozone in Arosa — Revisit
Figure 2.2 suggests that the variances may not be a constant. Based on fit to the trigonometric spline model (2.75), we calculate residual variances for each month and plot them on the logarithm scale in Figure 5.5(a). It is obvious that variations depend on the time of the year. It seems that a simple sinusoidal function can be used to model the variance function.
Spline Smoothing with Heteroscedastic and/or Correlated Errors 155 > v <- sapply(split(arosa.ls.fit$resi,Arosa$month),var) > a <- sort(unique(Arosa$x)) > b <- lm(log(v)~sin(2*pi*a)+cos(2*pi*a)) > summary(b) ... Coefficients: Estimate Std. Error t value Pr(>|t|) (Intercept) 5.43715 0.05763 94.341 8.57e-15 *** sin(2 * pi * a) 0.71786 0.08151 8.807 1.02e-05 *** cos(2 * pi * a) 0.49854 0.08151 6.117 0.000176 *** --Residual standard error: 0.1996 on 9 degrees of freedom Multiple R-squared: 0.9274, Adjusted R-squared: 0.9113 F-statistic: 57.49 on 2 and 9 DF, p-value: 7.48e-06
log(variance) 5.5 6.5
(a)
(b) o
o
o
o o
o o o
o
o
thickness 300 350 400
o
4.5
o
1
3
5
7 month
9
11
1
3
5
7 month
9
11
FIGURE 5.5 Arosa data, plots of (a) logarithm of residual variances (circles) based on the periodic spline fit in Section 2.7, the sinusoidal fit to logarithm of squared residuals (dashed line), and the fit from model (5.31) 