COMPREHENSIVE CHEMOMETRICS Chemical and Biochemical Data Analysis
Related Titles of Interest JOURNALS Chemometrics and Intelligent Laboratory Systems Analytica Chimica Acta Talanta BOOKS AND SERIES Chemometrics in Spectroscopy Chemometrics Tutorials Statistics in Spectroscopy Data Handling in Science and Technology Series Chemometrics: A Textbook Design and Optimization in Organic Synthesis Data Analysis and Signal Processing in Chromatography The Data Analysis Handbook Experimental Design: A Chemometric Approach Handbook of Chemometrics and Qualimetrics Multivariate Analysis of Data in Sensory Science Nature-inspired Methods in Chemometrics: Genetic Algorithms and Artificial Neural Networks Scientific Data Ranking Methods Statistical Design – Chemometrics Wavelets in Chemistry
COMPREHENSIVE CHEMOMETRICS Chemical and Biochemical Data Analysis
Editors-in-Chief
Steven D. Brown University of Delaware, Newark, DE, USA
` Tauler Roma Institute of Environmental Assessment and Water Research (IDÆA), Spanish Council of Scientific Research (CSIC), Barcelona, Spain
Beata Walczak University of Silesia, Katowice, Poland
AMSTERDAM BOSTON HEIDELBERG LONDON NEW YORK OXFORD PARIS SAN DIEGO SAN FRANCISCO SINGAPORE SYDNEY TOKYO
Elsevier Radarweg 29, PO Box 211, 1000 AE Amsterdam, The Netherlands Linacre House, Jordan Hill, Oxford OX2 8DP, UK First edition 2009 Copyright ª 2009 Elsevier B.V. All rights reserved The following article is a US Government work in the public domain and is not subject to copyright: CHAPTER 1.16 CONSTRAINED AND UNCONSTRAINED OPTIMIZATION No part of this publication may be reproduced, stored in a retrieval system or transmitted in any form or by any means electronic, mechanical, photocopying, recording or otherwise without the prior written permission of the publisher Permissions may be sought directly from Elsevier’s Science & Technology Rights Department in Oxford, UK: phone (+44) (0) 1865 843830; fax (+44) (0) 1865 853333; email:
[email protected]. Alternatively you can submit your request online by visiting the Elsevier web site at http://elsevier.com/locate/permissions, and selecting Obtaining permission to use Elsevier material Notice No responsibility is assumed by the publisher for any injury and/or damage to persons or property as a matter of products liability, negligence or otherwise, or from any use or operation of any methods, products, instructions or ideas contained in the material herein. Because of rapid advances in the medical sciences, in particular, independent verfication of diagnoses and drug dosages should be made British Library Cataloguing in Publication Data A catalogue record for this book is available from the British Library Library of Congress Catalog Number: 2008943480 ISBN: 978-0-444-52702-8 For information on all Elsevier publications visit our website at books.elsevier.com Printed and bound in Slovenia 09 10 11 12 11 10 9 8 7 6 5 4 3 2 1
Working together to grow libraries in developing countries www.elsevier.com | www.bookaid.org | www.sabre.org
Contents of Volume 1 Contributors to Volume 1
vii
Preface
ix
Editors in Chief
x
Contents of All Volumes
xi
Section Editors
xv
Statistics 1.01
An Introduction to the Theory of Sampling: An Essential Part of Total Quality Management F. F. Pitard, Francis Pitard Sampling Consultants, Broomfield, CO, USA
1
1.02
Quality of Analytical Measurements: Statistical Methods for Internal Validation M. C. Ortiz, L. A. Sarabia, M. S. Sa´nchez, and A. Herrero, University of Burgos, Burgos, Spain
17
1.03
Proficiency Testing in Analytical Chemistry M. Thompson, University of London, London, UK
77
1.04
Statistical Control of Measures and Processes A. J. Ferrer-Riquelme, Technical University of Valencia, Valencia, Spain
97
1.05
Quality of Analytical Measurements: Univariate Regression M. C. Ortiz, M. S. Sa´nchez, and L. A. Sarabia, University of Burgos, Burgos, Spain
127
1.06
Resampling and Testing in Regression Models with Environmetrical Applications J. Roca-Pardin˜as, University of Vigo, Vigo, Spain C. Cadarso-Sua´rez and W. Gonza´lez-Manteiga, University of Santiago de Compostela, Santiago de Compostela, Spain
171
1.07
Robust and Nonparametric Statistical Methods I. Gijbels and M. Hubert, Katholieke Universiteit Leuven, Leuven, Belgium
189
1.08
Bayesian Methodology in Statistics J. M. Bernardo, Universitat de Vale`ncia, Valencia, Spain
213
Experimental Design 1.09
Experimental Design: Introduction R. Phan-Tan-Luu, University Paul Cezanne, Marseille, France R. Cela, Universidad de Santiago de Compostela, Santiago, Spain
247
1.10
Screening Strategies R. Cela, Universidad de Santiago de Compostela, Santiago, Spain M. Claeys-Bruno and R. Phan-Tan-Luu, University Paul Cezanne, Marseille, France
251
v
vi Contents of Volume 1
1.11
The Study of Experimental Factors R. Carlson, University of Tromsø, Tromsø, Norway J. E. Carlson, Lulea˚ University of Technology, Lulea˚, Sweden
301
1.12
Response Surface Methodology L. A. Sarabia and M. C. Ortiz, University of Burgos, Burgos, Spain
345
1.13
Experimental Design for Mixture Studies D. Voinovich and B. Campisi, University of Trieste, Trieste, Italy R. Phan-Tan-Luu, University Paul Cezanne, Marseille, France
391
1.14
Nonclassical Experimental Designs R. Phan-Tan-Luu and M. Sergent, University Paul Cezanne, Marseille, France
453
1.15
Experimental Designs: Conclusions, Terminology, and Symbols R. Phan-Tan-Luu, University Paul Cezanne, Marseille, France R. Cela, Universidad de Santiago de Compostela, Santiago, Spain
501
Optimization 1.16
Constrained and Unconstrained Optimization B. W. Bader, Sandia National Laboratories, Albuquerque, NM, USA
507
1.17
Sequential Optimization Methods B. Dejaegher and Y. V. Heyden, Vrije Universiteit Brussel, Laarbeeklaan, Brussels, Belgium
547
1.18
Steepest Ascent, Steepest Descent, and Gradient Methods R. G. Brereton, University of Bristol, Bristol, UK
577
1.19
Multicriteria Decision-Making Methods M. Pavan, Joint Research Centre, European Commission, Ispra, Italy R. Todeschini, University of Milano–Bicocca, Milan, Italy
591
1.20
Genetic Algorithms R. Leardi, University of Genoa, Genoa, Italy
631
Index to Volume 1
655
Contributors to Volume 1 B. W. Bader Sandia National Laboratories, Albuquerque, NM, USA
Chapter 1.16 p. 507
J. M. Bernardo Universitat de Vale`ncia, Valencia, Spain
Chapter 1.08 p. 213
R. G. Brereton University of Bristol, Bristol, UK
Chapter 1.18 p. 577
C. Cadarso-Sua´rez University of Santiago de Compostela, Santiago de Compostela, Spain
Chapter 1.06 p. 171
B. Campisi University of Trieste, Trieste, Italy
Chapter 1.13 p. 391
J. E. Carlson Lulea˚ University of Technology, Lulea˚, Sweden
Chapter 1.11 p. 301
R. Carlson University of Tromsø, Tromsø, Norway
Chapter 1.11 p. 301
R. Cela Universidad de Santiago de Compostela, Santiago, Spain
Chapter 1.09 p. 247 Chapter 1.10 p. 251 Chapter 1.15 p. 501
M. Claeys-Bruno University Paul Cezanne, Marseille, France
Chapter 1.10 p. 251
B. Dejaegher Vrije Universiteit Brussel, Laarbeeklaan, Brussels, Belgium
Chapter 1.17 p. 547
A. J. Ferrer-Riquelme Technical University of Valencia, Valencia, Spain
Chapter 1.04 p. 97
I. Gijbels Katholieke Universiteit Leuven, Leuven, Belgium
Chapter 1.07 p. 189
W. Gonza´lez-Manteiga University of Santiago de Compostela, Santiago de Compostela, Spain
Chapter 1.06 p. 171
A. Herrero University of Burgos, Burgos, Spain
Chapter 1.02 p. 17
Y. V. Heyden Vrije Universiteit Brussel, Laarbeeklaan, Brussels, Belgium
Chapter 1.17 p. 547
M. Hubert Katholieke Universiteit Leuven, Leuven, Belgium
Chapter 1.07 p. 189
vii
viii Contributors to Volume 1
R. Leardi University of Genoa, Genoa, Italy
Chapter 1.20 p. 631
M. C. Ortiz University of Burgos, Burgos, Spain
Chapter 1.02 p. 17 Chapter 1.05 p. 127 Chapter 1.12 p. 345
M. Pavan Joint Research Centre, European Commission, Ispra, Italy
Chapter 1.19 p. 591
R. Phan-Tan-Luu University Paul Cezanne, Marseille, France
Chapter 1.09 p. 247 Chapter 1.10 p. 251 Chapter 1.13 p. 391 Chapter 1.14 p. 453 Chapter 1.15 p. 501
F. F. Pitard Francis Pitard Sampling Consultants, Broomfield, CO, USA
Chapter 1.01 p. 1
J. Roca-Pardin˜as University of Vigo, Vigo, Spain
Chapter 1.06 p. 171
M. S. Sa´nchez University of Burgos, Burgos, Spain
Chapter 1.02 p. 17 Chapter 1.05 p. 127
L. A. Sarabia University of Burgos, Burgos, Spain
Chapter 1.02 p. 17 Chapter 1.05 p. 127 Chapter 1.12 p. 345
M. Sergent University Paul Cezanne, Marseille, France
Chapter 1.14 p. 453
M. Thompson University of London, London, UK
Chapter 1.03 p. 77
R. Todeschini University of Milano–Bicocca, Milan, Italy
Chapter 1.19 p. 591
D. Voinovich University of Trieste, Trieste, Italy
Chapter 1.13 p. 391
Preface Some 40 years ago, the first publications appeared on the use of computer-aided mathematics to analyze chemical data. With those publications, the modern field of chemometrics was launched. Both the speed and power of computers and the sophistication of analytical instrumentation have made great leaps in the intervening time. The ready availability of chemometric software, coupled with the increasing need for rigorous, systematic examination of ever-larger and more sophisticated sets of measurements from instrumentation, has generated great interest in reliable methods for converting the mountains of measurements into more manageable piles of results, and for converting those results into nuggets of useful information. Interest in application of chemometrics has spread well beyond chemists with a need to understand and interpret their measurements; now chemometrics is helping to make important contributions in process engineering, systems biology, environmental science, and other disciplines that rely on chemical instrumentation, to name only a few areas. As applications of chemometrics continue to grow, so too does the methodology of chemometrics itself. At 40, chemometrics is a scientific field with many mature areas, but it is also a field where change continues to occur at a rapid pace, driven both by advances in chemical instrumentation and measurement and by close connection of chemometrics with the machine learning, statistics, and signal processing research communities. The interfacial location of chemometrics, falling between measurements on the one side and statistical and computational theory and methods on the other, poses a challenge to the new practitioner: gaining sufficient breadth and depth to use chemometrics effectively. The four volumes of Comprehensive Chemometrics are the result of a meeting in Oxford in September 2005, where the editors planned a work that would cover all of the major areas of chemometric research and a wide sample of current applications. Our goal was to produce a reference work that would serve both the new and the experienced practitioner. We divided the coverage of methodology into sections: Statistics, edited by L. Sarabia; Experimental Design, edited by R. Phan-Tan-Luu; Optimization, edited by R. Leardi; Data Preprocessing, edited by J. Trygg; Linear Soft-Modeling, edited by A. de Juan; Unsupervised Data Mining, edited by D. Coomans; Linear Regression Modeling, edited by J. Kalivas; Non-Linear Regression, edited by L. Buydens; Classification Feature Selection, edited by B. Lavine; and Multivariate Robust Techniques, edited by P. van Espen. The Editors-in-Chief oversaw a section on applications, where several of the newer directions in chemometrics are explored in depth. What has resulted from this collaboration is a resource that captures the practice of chemometrics in the early twenty-first century. The four volumes in this work include about 100 chapters, making this the most wide-reaching and detailed overview of the field of chemometrics ever published. Comprehensive Chemometrics offers depth and rigor to the new practitioner entering the field, and breadth and varied perspectives on current literature to more experienced practitioners aiming to expand their horizons. Software and data sets, both of which are especially valuable to those learning the methods, are provided in many chapters. The coverage is not only comprehensive, but it is also authoritative; authors contributing to Comprehensive Chemometrics are among the most distinguished practitioners of the field. Comprehensive Chemometrics would not have been possible without the work of the Section Editors named above, who helped us identify authors and who were tireless in reviewing submissions. We also owe special thanks to Adrian Shell, Senior Acquisitions Editor at Elsevier, for supporting the project and seeing the project off, to Claire Byrne, our Developmental Editor, for her patience and persistence in seeing this project to completion, and to Hazel Harris, our Project Manager, for keeping the production schedule. Finally, we extend special thanks to all of our authors whose efforts have made the work the valuable reference that it is. Steven D. Brown, Roma´ Tauler and Beata Walczak September 2008
ix
Editors in Chief Steven D. Brown obtained the Ph.D. degree in analytical chemistry in 1978 from the University of Washington, working with Bruce Kowalski. The same year he was appointed Assistant Professor at the University of California, Berkeley, and he held a joint appointment at Lawrence Berkeley Laboratory. In 1981, he moved to Washington State University and, in 1986, to the Department of Chemistry and Biochemistry at the University of Delaware, where he is presently Willis F. Harrington Professor. He has served as Chair of the Department of Chemistry and Biochemistry for 5 years, as a Section president of the American Chemical Society, and as President of the North American Chapter of the International Chemometrics Society. He is one of the three Founding Editors of the Journal of Chemometrics and has served for 20 years, first as its North American Editor and then for 12 years as its Editor-inChief. He was winner of the first EAS Award in Chemometrics in 1986. His research interests concern a wide range of problems in chemometrics. The focus of his research has been the development of new instrumental methods through use of multivariate mathematical methods for multicomponent analysis, including calibration transfer, and the novel use of data fusion methods. Roma` Tauler Ferre´ obtained his PhD in Chemistry at the University of Barcelona in 1984. He is Research Professor at the Institute of Environmental Assessment and Water Research (IDÆA), CSIC, in Barcelona (Spain), since July 2003. Previously he was Associate Professor of Analytical Chemistry at the Analytical Chemistry Department of the University of Barcelona during 1987– 2003. He performed postdoct stays at Institut fu¨r Anorg. u. Anal. Chemie, Univ. of Innsbruck, Innsbruck (Austria) in 1985 and 1989 and a year’s sabbatical as a research scientist at the Center for Process Analytical Chemistry (CPAC), Department of Chemistry, University of Washington, Seattle, USA, in 1992. At present, he is the Editor in Chief of the journal Chemometrics and Intelligent Laboratory Systems and of this Major Reference Work on Chemometrics. He has published more than 200 research papers, most of them in the field of chemometrics and its applications, and in particular in the development and applications of new multivariate resolution methods. In the recent years he has focused more on the investigation of environmental problems. Beata Walczak graduated in chemistry from the Faculty of Mathematics, Physics and Chemistry, Silesian University, Katowice, Poland, in 1979. Since then, she has been working in the Institute of Chemistry, Silesian University, where now she is the head of the Department of Chemometrics. She has been involved in chemometrics from the early 1990s and her main scientific interest is in all aspects of data exploration and modeling (dealing with missing and censored data, dealing with outliers, data representativity, enhancement of instrumental signals, signal warping, data compression, linear and nonlinear projections, development of modeling approaches, feature selection techniques, etc.). She has authored and co-authored around 140 scientific papers and 250 conference papers, and has delivered many invited lectures at numerous international chemistry meetings. She acts as Editor of Chemometrics and Intelligent Laboratory Systems and ‘Data Handling in Chemistry and Technology’ (the Elsevier book series), and also as a member of the editorial boards of Talanta, Analytical Letters, Journal of Chemometrics and Acta Chromatographica. x
Contents of All Volumes Volume 1 Statistics Edited by L. Sarabia, University of Burgos, Burgos, Spain 1.01 1.02 1.03 1.04 1.05 1.06 1.07 1.08
An Introduction to the Theory of Sampling: An Essential Part of Total Quality Management Quality of Analytical Measurements: Statistical Methods for Internal Validation Proficiency Testing in Analytical Chemistry Statistical Control of Measures and Processes Quality of Analytical Measurements: Univariate Regression Resampling and Testing in Regression Models with Environmetrical Applications Robust and Nonparametric Statistical Methods Bayesian Methodology in Statistics
Experimental Design Edited by R. Phan-Tan-Luu, University Paul Cezanne, Marseille, France 1.09 1.10 1.11 1.12 1.13 1.14 1.15
Experimental Design: Introduction Screening Strategies The Study of Experimental Factors Response Surface Methodology Experimental Design for Mixture Studies Nonclassical Experimental Designs Experimental Designs: Conclusions, Terminology, and Symbols
Optimization Edited by R. Leardi, University of Genoa, Genoa, Italy 1.16 1.17 1.18 1.19 1.20
Constrained and Unconstrained Optimization Sequential Optimization Methods Steepest Ascent, Steepest Descent, and Gradient Methods Multicriteria Decision-Making Methods Genetic Algorithms
Volume 2 Data Preprocessing Edited by J. Trygg, Umea˚ University, Umea˚, Sweden 2.01 2.02
Background Estimation, Denoising, and Preprocessing Denoising and Signal-to-Noise Ratio Enhancement: Classical Filtering xi
xii Contents of All Volumes
2.03 2.04 2.05 2.06 2.07 2.08 2.09 2.10 2.11
Denoising and Signal-to-Noise Ratio Enhancement: Wavelet Transform and Fourier Transform Denoising and Signal-to-Noise Ratio Enhancement: Derivatives Denoising and Signal-to-Noise Ratio Enhancement: Splines Variable Shift and Alignment Normalization and Closure Model Based Preprocessing and Background Elimination: OSC, OPLS, and O2PLS Standard Normal Variate, Multiplicative Signal Correction and Extended Multiplicative Signal Correction Preprocessing in Biospectroscopy Batch Process Modeling and MSPC Evaluation of Preprocessing Methods
Linear Soft-Modeling Edited by A. de Juan, University of Barcelona, Barcelona, Spain 2.12 2.13 2.14 2.15 2.16 2.17 2.18 2.19 2.20 2.21 2.22 2.23 2.24 2.25
Linear Soft-Modeling: Introduction Principal Component Analysis: Concept, Geometrical Interpretation, Mathematical Background, Algorithms, History, Practice Independent Component Analysis Introduction to Multivariate Curve Resolution Two-Way Data Analysis: Evolving Factor Analysis Two-Way Data Analysis: Detection of Purest Variables Two-Way Data Analysis: Multivariate Curve Resolution – Noniterative Resolution Methods Two-Way Data Analysis: Multivariate Curve Resolution – Iterative Resolution Methods Two-Way Data Analysis: Multivariate Curve Resolution – Error in Curve Resolution Multiway Data Analysis: Eigenvector-Based Methods Multilinear Models: Iterative Methods Multiset Data Analysis: ANOVA Simultaneous Component Analysis and Related Methods Multiset Data Analysis: Extended Multivariate Curve Resolution Other Topics in Soft-Modeling: Maximum Likelihood-Based Soft-Modeling Methods
Unsupervised Data Mining Edited by D. Coomans, James Cook University, Townsville, QLD, Australia 2.26 2.27 2.28 2.29 2.30 2.31
Unsupervised Data Mining: Introduction Common Clustering Algorithms Data Mapping: Linear Methods versus Nonlinear Techniques Density-Based Clustering Methods Model-Based Clustering Tree-Based Clustering and Extensions
Volume 3 Linear Regression Modeling Edited by J. Kalivas, Idaho State University, Pocatello, ID, USA 3.01 3.02 3.03 3.04 3.05 3.06 3.07 3.08 3.09
Calibration Methodologies Regression Diagnostics Validation and Error Preprocessing Methods Variable Selection Missing Data Robust Calibration Transfer of Multivariate Calibration Models Three-Way Calibration
Contents of All Volumes xiii
Non-Linear Regression Edited by L. Buydens, Institute for Molecules and Materials, Radboud University of Nijmegen, Nijmegen, The Netherlands 3.10 3.11 3.12 3.13 3.14
Model-Based Data Fitting Kernel Methods Linear Approaches for Nonlinear Modeling Other Methods in Nonlinear Regression Neural Networks
Classification Edited by B. Lavine, Oklahoma State University, Stillwater, OK, USA 3.15 3.16 3.17 3.18 3.19
Classification: Basic Concepts Statistical Discriminant Analysis Decision Tree Modeling in Classification Feed-Forward Neural Networks Validation of Classifiers
Feature Selection Edited by B. Lavine, Oklahoma State University, Stillwater, OK, USA 3.20 3.21 3.22 3.23
Feature Selection: Introduction Multivariate Approaches: UVE-PLS Multivariate Approaches to Classification using Genetic Algorithms Feature Selection in the Wavelet Domain: Adaptive Wavelets
Multivariate Robust Techniques Edited by P. van Espen, University of Antwerp, Antwerp, Belgium 3.24
Robust Multivariate Methods in Chemometrics
Volume 4 Applications Edited by S. D. Brown, University of Delaware, Newark, DE, USA; R. Tauler, Institute of Environmental Assessment and Water Research (IDÆA), Spanish Council of Scientific Research (CSIC), Barcelona, Spain and B. Walczak, University of Silesia, Katowice, Poland 4.01 4.02 4.03 4.04 4.05 4.06 4.07 4.08 4.09 4.10 4.11 4.12 4.13 4.14 4.15
Representative Sampling, Data Quality, Validation – A Necessary Trinity in Chemometrics Multivariate Statistical Process Control and Process Control, Using Latent Variables Environmental Chemometrics Application of Chemometrics to Food Chemistry Chemometrics in QSAR Spectroscopic Imaging Spectral Map Analysis of Microarray Data Analysis of Megavariate Data in Functional Genomics Systems Biology Chemometrics Role within the PAT Context: Examples from Primary Pharmaceutical Manufacturing Smart Sensors Chemometric Analysis of Sensory Data Chemometrics in Electrochemistry Chemoinformatics High-Performance GRID Computing in Chemoinformatics
This page intentionally left blank
Section Editors Lutgarde Buydens is head of the Chemometrics Department at the Institute for Molecules and Materials of the Radboud University of Nijmegen, the Netherlands. She originally studied pharmacy at the Brussels free University and did her Ph.D. with professor Massart on a QSAR subject. After a postdoc position at the University of Illinois in Chicago with the group of Professor William Dunn III, she started at the University of Nijmegen where she became a full professor in 1995. She received the first Elsevier Chemometrics award in 1992. Her research interests include the (further) development of chemometrical techniques within several application areas. Chemometrical techniques for combining different (spectroscopic imaging) data is her major research area. Professor Buydens is author of more than 200 publications in international scientific journals and co-author of 4 scientific books, including the 2 volumes of the Handbook of Chemometrics and Qualimetrics. Since 1998, she has been the editor of the journal Analytica Chimica Acta and has also been the co-editor of various books and special issues of international scientific journals. Danny Coomans was the first Ph.D. student of chemometrics of the late Professor D. L. Massart at the Vrije Universiteit Brussel, one of the founders of the field of chemometrics. He obtained the degree in 1982. He is still parttime associated with the department. He is full-time Professor in Statistics and Intelligent Data Analysis in the School of Mathematics, Physics and IT at the James Cook University, Townsville, Australia. His research interests are in multivariate statistics, statistical pattern recognition and data mining, waveletbased feature extraction, computer intensive methods, and distributed computing. From an applied point of view, his areas of expertise include chemometrics, environmetrics, and bioinformatics with special interest in the analysis of spectral databases, microarrays, QSAR data, and aspects of industrial laboratory quality control. He has published over 250 research papers and has contributed to monographs related to chemometrics.
Anna de Juan has been an Associate Professor in the Department of Analytical Chemistry at the University of Barcelona since 2003, teaching chemometrics at undergraduate and graduate levels. She holds a degree and Ph.D. in chemistry from the University of Barcelona and her expertise is in multivariate curve resolution (MCR) methods: theoretical development and application to bioanalytical and analytical problems. Since 2002 she has been a member of the Editorial Advisory Board of Chemometrics and Intelligent Laboratory Systems and since 2006 of Analytica Chimica Acta. In 2004, she received the 4th Chemometrics Elsevier Award together with Karl Booksh. She has published around 60 papers in international journals and books, especially on multivariate curve resolution developments and related methods and on applications to the study of bioanalytical processes, image analysis, and general analytical applications.
xv
xvi
Section Editors
John H. Kalivas is a Professor in the Department of Chemistry at Idaho State University, a primary undergraduate institution in Pocatello. He is author and co-author of over 80 professional papers, book chapters, and books. He is a member of Sigma Xi, the Society for Applied Spectroscopy, and the Council on Undergraduate Research and he serves on the Editorial Board of the Journal of Chemometrics, Applied Spectroscopy, and Analytical Letters. Dr. Kalivas received the B.S. degree (1978) in chemistry from California Polytechnic State University, San Luis Obispo, and the Ph.D. degree (1982) in chemistry from the University of Washington, Seattle.
Barry K. Lavine is an Associate Professor of Chemistry at Oklahoma State University in Stillwater, OK. He has published around 90 papers in chemometrics and is on the editorial board of several journals including the Journal of Chemometrics, Microchemical Journal, and Chemoinformatics. He is the Assistant Editor of Chemometrics for Analytical Letters. Lavine’s research interests encompass many aspects of the applications of computers in chemical analysis including pattern recognition, multivariate curve resolution, and multivariate calibration using genetic algorithms and other evolutionary techniques.
Riccardo Leardi graduated in pharmaceutical chemistry and technology in 1983. Since then, he has been working in the section of Analytic Chemistry of the Department of Pharmaceutical and Food Chemistry and Technologies of the Faculty of Pharmacy of the University of Genova, and his research field is chemometrics. His interests are mainly devoted to problems related to food, environmental and clinical data, and to experimental design and process optimization. In the last years, his research focused mainly on genetic algorithms and on three-way methods. He is author of around 80 papers and around 80 communications in national and international meetings; he has been an invited speaker in 13 international meetings and in several industries and research centers. He is Review Editor of Journal of Chemometrics and Editorial Adviser of Analytica Chimica Acta. In November 2002, he started his activity of chemometric consultancy.
Roger Phan-Tan-Luu was Professor at the University Paul Cezanne of Marseille, France. In 1970, he founded the Laboratory of Methodology of Experimental Research in which he developed both the methodological approach and the algorithms that enable construction of efficient designs. He was at the start of the establishment of trainings about the Methodology of Experimental Research in several universities and industries in France and abroad. He is author and co-author of numerous scientific articles dealing with experimental designs. He belongs to the scientific committees of several international congresses. In 1992, Roger Phan-Tan-Luu was awarded Doctor Honoris Causa of the University of Ume˚a (Sweden).
Section Editors
xvii
Luis A. Sarabia received his Ph.D. in Statistics from the University of Valladolid (Spain) in 1979. Since 1974, he is teaching Statistics and Mathematics mostly to graduate and postgraduate students of Chemistry. At present, his research is centred on Chemometrics as a member of the Chemometrics and Qualimetrics group of the University of Burgos. His research activities include development of software and implementation of nonparametric and robust statistical methods, genetic algorithms, neural networks, etc. He is also involved in multivariate/multiway regression methods, methodology of the experimental design, quality assurance, and validation. He is the author of about a hundred papers on these matters.
Johan Trygg obtained his Ph.D in Organic Chemistry from Ume˚a University in 2001 (S. Wold). After receiving his Ph.D. at Ume˚a University he was awarded the prestigious Knut & Alice Wallenberg scholarship, and spent 2 years abroad as postDoc, first at the Institute for Molecular Bioscience (IMB), Brisbane, Australia, and later at Imperial College, London, both positions in the field of metabonomics. Johan Trygg brings with him years of analysis of high-complexity systems using an ever-broadening adaptation of chemometrics analysis, and provides expertise on the analysis and interpretation of highly complex data sets. Johan Trygg is an Associate Professor of Chemometrics at Ume˚a University, Sweden. He is acting group leader of Chemometrics & Bioinformatics at Ume˚a Plant Science Centre (UPSC) at Ume˚a University.
Pierre van Espen was one of the first Ph.D. students at the University of Antwerp. He is currently Professor in the Department of Chemistry at the University of Antwerp and Profesor Invitado at the University of Havana, Cuba. He accomplished some pioneering work on the analysis of X-ray spectra by means of the development of software for the PDP 11/45 computer. At a later stage, this resulted in the AXIL (analysis of X-ray spectra by iterative least squares regression) package, which gained international acceptance and widespread use through the International Atomic Energy Agency (IAEA). Apart from the AXIL package, his work also comprises application and extension of chemometric methods for the analysis of data from X-ray spectrometry.
This page intentionally left blank
1.01 An Introduction to the Theory of Sampling: An Essential Part of Total Quality Management F. F. Pitard, Francis Pitard Sampling Consultants, Broomfield, CO, USA ª 2009 Elsevier B.V. All rights reserved.
1.01.1 1.01.2 1.01.3 1.01.4 1.01.4.1 1.01.4.2 1.01.4.2.1 1.01.4.2.2 1.01.4.2.3 1.01.4.3 1.01.4.3.1 1.01.4.3.2 1.01.4.3.3 1.01.4.4 1.01.4.4.1 1.01.5 1.01.5.1 1.01.5.2 1.01.5.3 1.01.5.4 1.01.6 1.01.6.1 1.01.6.2 1.01.7 1.01.7.1 1.01.8 1.01.8.1 1.01.8.1.1 1.01.8.1.2 1.01.8.1.3 1.01.8.2 1.01.8.3 1.01.8.3.1 1.01.8.3.2 1.01.8.3.3 1.01.8.3.4 1.01.9 1.01.10 References
Introduction Scope Definitions and Notations Dividing a Complex Problem into its Basic Components Dividing the Small-Scale Variability Optimization of Sampling Protocols The in situ nugget effect The fundamental sampling error The grouping and segregation error The Practical Implementation of the Sampling Protocol The increment delimitation error The increment extraction error The weighting error The Preservation of Samples Integrity Sample preparation errors Exercises Challenging the Reader Exercise #1: A Worldwide Problem for Ore Grade Control in Open Pits: Blasthole Sampling Exercise #2: Correctness of Straight-Path Cross-Stream Sampling Systems Exercise #3: Correctness of Rotating Vezin Cross-Stream Sampling Systems Exercise #4: Correctness of Hammer Cross-Belt Samplers The Critical Importance of Sampling Courses Case #1: A Bad Protocol for Blastholes Followed by an Incorrect Implementation Case #2: An Incorrect Sampling System for the Final Tail of a Flotation Plant The Enemies and Their Link to Geostatistics Important Remark Large-Scale Variability Definition of the Variogram Selection of a given process parameter of interest Heterogeneity affecting the given process parameter of interest Measuring heterogeneity variability with the variogram Extrapolation of the Variogram to Time or Distance Zero The Importance of Large-Scale Variability Variability issues during exploration Variability issues during mining Variability issues within a processing plant Variability issues during trade with customers Conclusions Recommendations
2 2 3 3 3 3 3 4 4 5 5 5 5 5 5 6 6 7 7 7 9 9 9 10 10 10 10 11 11 11 12 12 12 13 13 13 13 13 15
1
2 An Introduction to the Theory of Sampling: An Essential Part of Total Quality Management
1.01.1 Introduction Considering the example of the mining industry, out of many other possible examples, the key to total quality management (TQM) is to optimize the recovery of natural resources and make quality products to satisfy users on several fronts such as reliability, effectiveness, and minimal cost: it cannot and will not be done without a thorough understanding of theory of sampling (TOS). As TOS is not taught at universities, the resulting economic losses are plaguing the industry. Management decisions are based on precise and accurate sampling. It is of utmost importance to communicate the benefits of correct sampling to management, to a board of directors, to shareholders, to geologists, to drillers, to miners, to metallurgists, to chemists, to sales people, environmentalists, geostatisticians, and to statisticians. It is the only way to get necessary cash flow, more profit, and added share value. If stakeholders cannot see the value of correct sampling, it is the company’s responsibility to show them through education of the management team. Training of key personnel such as geologists, drillers, miners, metallurgists, chemists, and environmentalists is essential to obtain results. They all must monitor and verify the quality of data, so that geostatisticians and even conventional statisticians can perform reliable, believable risk assessments, to enable the management to make ultimately crucial financial decisions.
1.01.2 Scope The acquisition of a reliable database is always a valuable asset to a company. However, for such a success to happen, a correct and balanced strategy is needed, which is illustrated as a three-legged table in Figure 1. If one leg of the table is weak the entire table may collapse. Usually, the weakest leg is a style of management that does not sufficiently support good sampling practices, though it is the only way to allow statisticians to do their work in understanding variability in the deposit or in processes, and subsequently advise management for wise courses of action. However, implementing correct sampling is easier said than done. Similar to safety issues for which companies spend millions of dollars, it must be internally standardized through correctness, internal guidelines, sustained training, and enforcement auditing. It must also be monitored for its added value through improved natural resources recovery, improved conciliation between economic units (e.g., mine, mill, smelter, and refinery), and added stakeholder value. The collection of samples has only one objective in mind: understanding variability on a large scale. Unfortunately, for practical reasons a sample is often made of a relatively small amount of material. Therefore, as it is collected, another type of unwanted variability is always introduced, which is a
Company $ benefits, added stakeholder value, and market perception
Emphasis on causes of problems by proactive management
A strong commitment to good sampling and good laboratory practices
Figure 1 The concept of the three-legged table.
Capability to understand variability and to perform reliable statistical studies
An Introduction to the Theory of Sampling: An Essential Part of Total Quality Management
3
small-scale variability that may often overwhelm the large-scale variability that is important to measure. Many people often do not realize that the variability they are looking at has little to do with the variability they want to measure; it is essential to make the difference as clear as possible, and this is the true, indepth objective of TOS.
1.01.3 Definitions and Notations The length of this paper being limited, for definitions and notations, the reader is referred to textbooks from various authors listed in References.
1.01.4 Dividing a Complex Problem into its Basic Components The strength of the TOS is the use of a logical Cartesian approach, which divides small- and large-scale variability into basic components that can be further analyzed one by one. If everyone understood that basic concept they would not struggle to understand TOS. It does not matter how deep anyone goes into the mathematics of TOS; if this basic Cartesian concept is not understood, it will result in confusion, ineffectiveness, and failure to reach economic objectives, which is a common observation in many industries around the world today. 1.01.4.1
Dividing the Small-Scale Variability
The small-scale variability can be divided into four categories: 1. 2. 3. 4.
the optimization of sampling protocols, the practical implementation of sampling protocols, the preservation of samples integrity, and the minimization of analytical errors.
Analytical errors are not the subject of this paper, but can be explored in guidelines for good laboratory practices. 1.01.4.2
Optimization of Sampling Protocols
The optimization of sampling protocols involves a good understanding of small-scale heterogeneity, which affects any component of interest in the lot to be sampled. 1.01.4.2.1
The in situ nugget effect The TOS addresses sampling difficulties when sampling particulate materials. However, before the material is broken up, as distance between samples tends toward zero, a small-scale Constitution Heterogeneity may greatly affect the repeatability of a sample, with a given selected mass, drilled at any given place. The preliminary knowledge of local mineralogy may help in the selection of a minimum mass for the basic module of observation (e.g., selection of a drilling diameter and sample length) during exploration, ore reserves estimations, and for the capability to be selective at a certain cutoff grade and at any given place during the mining operation. Failure to address this problem early enough can result in stunning estimation failures. The mean m[NE] of the in situ Nugget effect (NE) is zero; however, its large variance s2[NE] may lead to severe data skewness and therefore unwanted local illusions followed by stunning economic consequences. It is suggested to make a great difference between m[NE] ¼ 0 when there is free access to an entire population of possible skewed data points in a given area, and m[NE] 6¼ 0 when too few skewed data points affected by a large variance
4 An Introduction to the Theory of Sampling: An Essential Part of Total Quality Management
are available in the same given area. Too often, large mistakes are repeated over and over because too few data points belonging to a skewed distribution are available in a given area.
1.01.4.2.2
The fundamental sampling error As soon as the in situ material is broken up into fragments, the size distribution, shape, and density of fragments begin to play an important role in sampling. Typical differences between individual fragments lead to the concept of constitution heterogeneity in TOS. A good understanding of constitution heterogeneity is the key to optimize sample and subsample masses. Heterogeneity tests and sampling nomographs are used to make such optimization.1–4 It is critically important to understand the various approaches that were used in TOS to perform this task. Indeed, it would be a great mistake, as frequently happens, to believe that Dr. Pierre M. Gy created one universal formula to calculate the variance of the fundamental sampling error (s2[FSE]), which is the result of constitution heterogeneity for broken up materials.5 There are many different approaches depending on the heterogeneity of the constituents of interest. The main approaches are:
1. the general case where the constituent of interest is not liberated from its gangue, 2. the special case where the constituent of interest is liberated from its gangue and may show delayed comminution properties, 3. the case where the constituent of interest is finely disseminated inside another major constituent, and 4. the case where the main objective of sampling is to obtain a sample representative of the particle size distribution. Failure to distinguish these different cases leads to a total misunderstanding of TOS. However, there are more subtleties to understand: suggested formulas have their limitations well addressed in TOS. For example, if the variance of the FSE becomes very large, formulas become meaningless and a massive misuse of TOS may take place. It was never the objective of TOS to address out-of-control variances generated by unwise protocols for which skewness introduces unfortunate illusions and misconceptions. Let us be clear on this issue: TOS cannot help people who are not preventive in understanding the constitution heterogeneity of their materials. The mean m[FSE] of the FSE is zero; however, its large variance may lead to severe data skewness and therefore unwanted local illusions. Wisdom in sampling starts with a good understanding of FSE so that simplistic Gaussian statistics may indeed reasonably apply. The last thing anyone wants to deal with is a FSE introducing skewness in a generated database as it would most certainly result in estimation errors very few people understand.
1.01.4.2.3
The grouping and segregation error Gravity is the mother of segregation. Materials segregate as soon as they are crushed, transported, and stored in stockpiles, silos, bins, or tanks throughout processes. Increments collected at any given place within a lot of particulate material to be sampled (i.e., small-scale heterogeneity) can be very different, leading to a certain amount of distribution heterogeneity. Materials segregate because fragments have a different size, shape, and density. Because segregation is a transient phenomenon changing all the time, so does distribution heterogeneity. The most effective action to minimize the negative effect of distribution heterogeneity in sampling protocols is to collect as many and as small correct increments to make up a sample as practically possible. If problems generated by small-scale distribution heterogeneity are not carefully addressed, a substantial grouping and segregation error (GSE) may result. The mean m[GSE] of the GSE is zero; however, its large variance may lead to catastrophic local illusions. Experience proves that GSE is far more difficult to minimize than what many people may think. Actually, many standard practices suggested for homogenizing materials do the opposite and segregate materials even further. In sampling, beware of old traditions and ceremonial misconceptions.
An Introduction to the Theory of Sampling: An Essential Part of Total Quality Management
1.01.4.3
5
The Practical Implementation of the Sampling Protocol
Now that nearly everything is known to minimize the irrelevant, harmful variability introduced by a sampling protocol, it is still necessary to find out what happens during the practical implementation of that protocol. During this critically important phase, new sources of annoying and harmful variability take place, perhaps the most dangerous ones. There are three independent components that should be carefully reviewed, which are defined in the TOS with no possible ambiguity. 1.01.4.3.1
The increment delimitation error For an increment making up a sample to be correct, every part of the lot to be sampled must have exactly the same chance of becoming part of the sample. Therefore, the boundaries of the selected increment must coincide with an isotropic volume of observation which is:
A sphere if the lot is considered three-dimensional. Nobody knows how to do this, and this is the reason why
grab sampling or in-stream stationary probe sampling leads to unsolvable sampling problems. They are nothing more than a form of gambling. A cylinder if the lot can be considered two-dimensional. This is possible when drilling ore bodies or drilling a copper cathode. A uniform slice across a flowing stream if the stream is considered one-dimensional. We know how to solve these problems under the discharge of a stream. We do not know how to solve these problems when taking a cut on the top of a conveyor belt using a cross-belt hammer sampler.
The mean m[DE] of delimitation error (DE) is not zero; therefore, this error is a dangerous bias generator. The variance of DE can be quite large as it depends on how severe the deviation from the ideal isotropic volume of observation is and how severe transient segregation taking place at the sampling point is; the combination of these two problems can be devastating. Rules of increment delimitation correctness listed in the TOS are not negotiable. 1.01.4.3.2
The increment extraction error The increment extraction error (EE) could have been called the increment recovery error, which is well known by geologists drilling a deposit. This error takes place when the sampling tool is selective on what it is taking. The mean m[EE] of EE is not zero; therefore, this error is a dangerous bias generator. The variance of EE can be quite large as it depends on how severe the recovery problem is. Sampling systems must be carefully designed to minimize this error. Rules of increment extraction correctness listed in TOS are not negotiable. 1.01.4.3.3
The weighting error A good sampling system must be reasonably proportional. For example, when the cutter of a cross-stream sampler crosses a falling stream, it must take an increment the mass of which is proportional to the stream flow rate at this instant. Excessive deviation from proportionality can lead to a bias. Proportional sampling is probably the future of sampling. The sum of the variances of DE, EE, and weighting error (WE) is part of the variance of the materialization error (ME). Readers familiar with TOS may notice that the author chose to include the WE in the family making up the ME. The reason is that WE is the direct effect of how the sampling system collects increments. If the sampling system is not proportional, WE is likely to be significant; whereas if the sampling system is proportional, WE may be negligible. However, the flow rate fluctuations of a stream, which is part of the largescale variability, may also greatly affect WE. It is important to regulate flow rates as much as possible so that WE can be kept negligible during the sample materialization phase.
1.01.4.4
The Preservation of Samples Integrity
1.01.4.4.1
Sample preparation errors The variance of the preparation errors (PE) is a separate component introduced between sampling stages. It is convenient to include PE in the ME, grouping together all the errors generated by the practical implementation
6 An Introduction to the Theory of Sampling: An Essential Part of Total Quality Management
2
S ME =
ΣS n
2 DEn
+
ΣS
2 EEn
+
n
ΣS
2 WEn
+
n
ΣS
2 PEn
n
Selective process during sampling stages
Non-selective process during preparation stages
Figure 2 Expanding the materialization error to include all errors introduced by the practical implementation of the sampling protocol.
of a sampling protocol. PE can be the result of contamination, losses, alteration of physical or chemical composition, human errors, ignorance, carelessness, fraud, or sabotage. The mean of PE is not zero; therefore, these errors are dangerous bias generators. The variance of PE can be 2 quite large as it depends on many nonselective factors. Finally, the variance SME of the ME can be written as illustrated in Figure 2. The components of the ME are not well known by manufacturers of sampling equipment, or perhaps, a more accurate statement would be to say they do not make a clear difference between these four independent errors. This problem is also perpetuated by standards on sampling, most of which are obsolete, and reluctant to endorse the TOS. Furthermore, these standards usually do not distinctly address the difference between errors generated by the selective process (i.e., DE, EE, and WE) and those generated by the nonselective process (i.e., PE). The key to understanding problems created by these four errors can be summarized by the following statement: all the constituents of the lot to be sampled must be given an equal probability p of being selected and preserved as part of the sample, which leads to the notion of equiprobabilistic sampling, which in turn leads to the notion of correct sampling systems. Not respecting this cardinal rule almost always results in the presence of sampling biases that cannot be accounted for and totally ruins the previous efforts made to optimize the sampling protocol. The correctness or incorrectness of a sampling system is a structural property. If the sampling system is not correct by design (i.e., structure), devastating effects will result, regardless of results from bias tests that would at times tend to prove otherwise.
1.01.5 Exercises Challenging the Reader Solutions for the following exercises are beyond the scope of this paper, but they are well addressed in other documents from the author listed in References. 1.01.5.1 Exercise #1: A Worldwide Problem for Ore Grade Control in Open Pits: Blasthole Sampling Figure 3 illustrates seven areas in sampling blastholes. Each area can be the object of major deviations from sampling correctness. At each of these areas name the possible sampling error that may take place (e.g., DE, EE, WE, or PE). The reader has 20 min to sort them by name and to provide a solution. If it cannot be done, further training in sampling methodology is highly recommended.6 Biases taking place from blasthole drilling cost mining companies a fortune every year. Usually and unfortunately it is a well-hidden cost. Assuming DE, EE, WE, and PE are well taken care of, which is a huge assumption, geostatistical simulations using results from duplicate field samples can have access to some extent to such worrisome costs, and actually it would be advisable for geostatisticians to address such potential financial losses in feasibility studies if they had a deep knowledge of TOS. Someone may ask why do this if DE, EE, WE, and PE have been taken care of. Make no mistake, conciliation problems are not only the result of sampling errors introducing biases, they are also the result of unreasonable variances affecting the in situ Nugget effect, FSE, and GSE, and these indeed can have their effects simulated. But, nobody can make a simulation of economic losses generated by the bias generators DE, EE, WE, and PE.
An Introduction to the Theory of Sampling: An Essential Part of Total Quality Management
7
F
E Segregation
D G
Ideal sample Actual sample
A
Former sub-drill
C
Current sub-drill
B
Figure 3 The nightmare of ore-grade control.
1.01.5.2
Exercise #2: Correctness of Straight-Path Cross-Stream Sampling Systems
Straight-path cross-stream sampling systems are critically important to assess the mineral content of the feed to a plant, concentrates, and tailings. Yet, most of these sampling systems are flawed by design and will never fulfill their mission. The sampler illustrated in Figure 4 is a typical bottom-dump straight-path cross-stream sampler and as drawn it is apparently correct. Eleven important points are illustrated where other sampling systems may not be built correctly and generate difficulties and sampling biases at these points. At each of these areas name the possible sampling error that may take place (e.g., DE, EE, WE, or PE). The reader has 20 min to sort them by name and to provide a solution. If it cannot be done, further training in sampling methodology is highly recommended.7 Biases taking place at such sampling points can make it impossible to perform correct metallurgical accounting, and therefore it is very unlikely that the subsequent process can be optimized. Nobody can optimize an operation with a false database. 1.01.5.3
Exercise #3: Correctness of Rotating Vezin Cross-Stream Sampling Systems
Rotating Vezin cross-stream sampling systems are critically important for sampling small streams, or performing secondary or tertiary sampling in large sampling stations. Yet, most of these sampling systems are flawed by design and will never fulfill their mission. The sampler illustrated in Figure 5 is a well-designed Vezin sampler. Eleven important points are illustrated where other sampling systems may not be built correctly and generate difficulties and sampling biases at these points. At each of these areas, name the possible sampling error that may take place (e.g., DE, EE, WE, or PE). The reader has 20 min to sort them by name and to provide a solution. If it cannot be done, further training in sampling methodology is highly recommended.7 Biases taking place at such sampling points can make it impossible to perform correct metallurgical accounting, and therefore it is very unlikely that the subsequent process can be optimized. Nobody can optimize an operation with a false database. 1.01.5.4
Exercise #4: Correctness of Hammer Cross-Belt Samplers
Hammer cross-belt samplers are very popular, but they are not as correct as manufacturers may claim. The sampler illustrated in Figure 6 is a typical design. Seven important points are illustrated where severe
8 An Introduction to the Theory of Sampling: An Essential Part of Total Quality Management
1
11
3 6 7
5
4
2
9 10
8
Figure 4 A primary sampler for the feed of a plant.
2
3
6 9
Falling stream
4
5 7
10
11
8 1
Figure 5 A rotating Vezin sampler.
An Introduction to the Theory of Sampling: An Essential Part of Total Quality Management
9
7 6
4
1 2 3
5 Figure 6 A typical hammer cross-stream sampler.
difficulties may arise and generate sampling biases. At each of these areas name the possible sampling error that may take place (e.g., DE, EE, WE, or PE). The reader has 20 min to sort them by name and to provide a solution. If it cannot be done, further training in sampling methodology is highly recommended.7 Biases taking place at these sampling points can be financially devastating.
1.01.6 The Critical Importance of Sampling Courses All possible problems created by each point addressed in the above four exercises should be solved within minutes. Yet, because of a massive ignorance of TOS, these problems are the object of unnecessary doubts and arguments, time-consuming meetings, endless debates with manufacturers and engineering firms, and very expensive bias tests followed by doubtful statistics. Someone may wonder why doubtful statistics are used to interpret bias tests. The reason has something to do with a subtle property of segregation. Biases in sampling are the result of one form or another of segregation. However, the prime property of segregation is being a transient phenomenon changing all the time. Too many people think of a sampling bias as a constant bias as often observed for analytical biases. There is no such thing as a constant bias in sampling. A sample can be biased one way today, another way tomorrow, and can remain unbiased for a while. Therefore, a bias test for a sampling system can only declare there is indeed a bias; however, it cannot state that there is no bias. Furthermore, it is almost common practice to make a sampling bias test by comparing two bad samples. Each point addressed in the above exercises can lead to devastating money losses for the unaware company. Let us review a few well-documented cases that were presented at WCSB1 (World Conference on Sampling and Blending1) in Denmark in 2003.
1.01.6.1
Case #1: A Bad Protocol for Blastholes Followed by an Incorrect Implementation
Over a 10-year period a loss of US$134 000 000 was the result of a bad sampling and subsampling protocol used in a copper mine in Chile.8 The increase in the recovery of natural resources could be measured only after a better protocol was implemented for several consecutive months.
1.01.6.2
Case #2: An Incorrect Sampling System for the Final Tail of a Flotation Plant
Over a period of 20 years at a large copper mine in Chile, a stunning loss of US$2 000 000 000 was the result of using a flawed sampling system for the Final Tail of a copper–molybdenum flotation plant.8 Another company that treated the tailings for that period of time was quite prosperous. The first reaction of management after a flawless sampling station was installed was denial.
10 An Introduction to the Theory of Sampling: An Essential Part of Total Quality Management
A book could be written about many other examples around the world, but usually companies are very silent about such a catastrophic economic outcome. This is where confidentiality agreements become convenient. There is no doubt denial can become a form of fraud.
1.01.7 The Enemies and Their Link to Geostatistics Franc¸ois-Bongarc¸on rightly said in 2003 that ‘‘TOS and geostatistics are inseparable’’. So far, we made a list of the enemies responsible for slowly inflating one of the most annoying sources of variability plaguing geostatistics. When calculating a variogram, it is often observed that as the distance between samples becomes practically nil a certain amount of variability remains, which is random and discontinuous in nature; it is called V[j ¼ 0], j being the lag between samples. For simplicity, we chose to call it V[0]. Figure 7 makes a summary of 2 refers to the partial variance introduced by the protocol. It them, expressed as variances, and the variance SHE 1 is indeed annoying when an exploration or ore-grade control database is affected by a certain amount of variability that cannot be explained by the structural properties of the deposit. It is indeed annoying when a process control database is affected by a certain amount of variability that cannot be explained by what is happening in the process. It is indeed annoying when an expensive environmental assessment database is affected by a certain amount of variability that cannot be explained by what is happening in the environment. 1.01.7.1
Important Remark
V[0] is a variance. Some sampling errors such as DE, EE, WE, and PE are bias generators. A variogram does not see a bias, though the mean of DE, EE, WE, and PE is not zero. However, because these sampling biases fluctuate considerably, the variogram may show an increase for V[0], which cannot be explained by the sampling protocol or the analytical variance.
1.01.8 Large-Scale Variability The study of the total variability in any lot to be sampled can be broken up into several components, and the variogram helps us take a close look at each of these components. This is well done in geostatistics9 for the study of mineral deposits, and in chronostatistics10,11 for the study of variability in a process. 1.01.8.1
Definition of the Variogram
The scientifically sound definition of the variogram as well as its limitations is beyond the scope of this paper. For these issues the reader is referred to the References section. Nevertheless, we shall proceed with a simplistic definition. Also, in what follows, the variability of differences between samples with a small lag must remain
ΣS n
ΣS n
ΣS
2
FSEn
n
2
S HE1
2
DEn
n
2
WEn
2
GSEn
ΣS
V[0]
2
S NE
ΣS
ΣS n
2
n
ΣS n
2
AEn
2
PEn
EEn
Figure 7 V[0] can be inflated by many factors; they are the enemies in sampling and they quickly add up.
An Introduction to the Theory of Sampling: An Essential Part of Total Quality Management
11
reasonably constant through time if the variogram is used to study a parameter within a process. This limitation suggests taking precautions when selecting a series of chronological data, such as making reasonably sure along the selected chronology that no important changes have been applied to the process in the way it works. As such, it is likely that the suggested applications lose power when the basic sampling interval in the selected chronology is too long (e.g., days, weeks, or months). Aware of this limitation, the practitioner can use common sense not to mix data that do not belong to each other. 1.01.8.1.1
Selection of a given process parameter of interest The variability of parameters controlling or making up a process stream is most of the time a stochastic phenomenon. Such phenomenon is a hybrid in which we can find random and chaotic elements taking place at very short time intervals, but in which we can also find functional and continuous elements. Therefore, a certain parameter of interest (e.g., the proportion of a given size fraction in a stream feeding a SAG mill) characterizing a process stream can be described as follows: f ½t ¼ fL þ fA ½t þ fB ½t þ fC ½t
ð1Þ
where: fL is a constant term, such as the average proportion of a given size fraction of a stream feeding a SAG mill over f [t]. fA[t] is a random component characterized by neighboring fragments that are different and the resulting proportion of a given size fraction changing in a chaotic and unpredictable way at very short time intervals, regardless of the fact that these fragments may originate at the same place earlier in the process. fB[t] is a nonrandom, nonperiodic component, essentially continuous, which is an inherent property of the feed heading to the plant, for example, the slowly changing hardness of the ore. fC[t] is a periodic component characterized by the fact that people and machinery often work in a cyclic way, for example, a segregating stockpile introducing a cycle in the proportion of coarse fragments and fines feeding a SAG mill. 1.01.8.1.2
Heterogeneity affecting the given process parameter of interest Taking into account the fact that heterogeneity of a constant, such as fL, is obviously zero, the total heterogeneity affecting the parameter of interest in the stream is fully characterized by fA[t], fB[t], and fC[t]. 1.01.8.1.3
Measuring heterogeneity variability with the variogram The most natural way to compare two values such as f1 (e.g., the proportion of a given size fraction in the stream at time t1) and f2 (e.g., the proportion of a given size fraction in the stream at time t2), t1 and t2 being separated by a time lag called j, is to measure their difference d: d ¼ f1 – f2
ð2Þ
However, what is mostly relevant is the average difference d between many measurements N a given interval j apart: d¼
1 X fmþj – fm N –j m
ð3Þ
Such average difference converges toward zero; therefore, the squared difference should be used instead. Furthermore, differences account for variability twice; therefore, the convention has been taken to calculate the semi-variogram, leading to the following formula used for an absolute variogram: V ½j ¼
2 1 X fmþj – fm 2½N – j m
ð4Þ
A relative, dimensionless variogram could be calculated as well, making it easier to compare variograms from different experiments: V ½j ¼
X 2 1 fmþj – fm 2 2½N – j fL m
ð5Þ
12 An Introduction to the Theory of Sampling: An Essential Part of Total Quality Management
The study of many variograms shows that they are usually made of four major components: V ½j ¼ VA ½j þ VB ½j þ VC ½j þ VD ½j
ð6Þ
where: VA[j] is a very short-range term, random, and discontinuous. At the limit, when j ¼ 0, this leads to a very important term in chronostatistics called V[j ¼ 0], simplified in further discussions to V[0], which is the variability from sampling, subsampling, and measurement that does not exist in the process. VB[j] is a long-range term, usually nonrandom, and continuous. This variability is the one that needs to be controlled or tamed in a process. VC[j] is a periodic term, continuous, tied to ways people work or to ways some process machines work. This is a source of variability usually poorly understood, leading to process overcorrections, and therefore losses of process efficiency. The economic impact of this source of variability, if misunderstood, is enormous. VD[j] is a random residual variability tied to the variogram precision when the variance V[j] is calculated with too few pairs N–j. VD[j] tends toward zero when the number of pairs increases. It is not a good practice to calculate any point on a variogram with less than 20 pairs. Actually, 30 pairs or more is strongly recommended.
1.01.8.2
Extrapolation of the Variogram to Time or Distance Zero
An accurate extrapolation of the variogram to time or distance zero is required for many of its applications. The most effective solution is to extrapolate either the first- or the second-order integral of the variogram4. Indeed, W[j] and W9[j] smooth out residual noise due to VD[j], cycles, and other features interfering with good extrapolation. However, as demonstrated in the sampling theory,1–4 the variogram and its integrals all intercept the variance axis at the same place V[0].
1.01.8.3
The Importance of Large-Scale Variability
The ultimate goal of sampling is to understand the variability on a large scale. It is extremely important in a very large number of areas. Let us give a few examples:
To find new natural resources To quantify natural resources To show reasonable continuity of natural resources To understand conciliation problems during mining To optimize processes at mines and plants To improve the quality of products To secure fair money return from products To diminish penalties To curtail fraud To minimize environmental liability To lift the market perception of the company’s fiscal health To improve profitability trends, etc.
The list could go on and on, but let us be more specific in some well-known areas and see what could interfere in a negative way to understand large-scale variability. 1.01.8.3.1
Variability issues during exploration During exploration it is important to measure the anisotropy of a geological unit. We need to find out whether grade trends may be different: North–South than East–West or than along a vertical axis. Variogram ranges need to be defined in different directions. Density of drilling needs to be optimized in different directions. Ore continuity and zone of influence need to be defined in different directions. Ultimately, a reliable geological model must be created. With a large V[0] damaging the database, these critically important tasks leading to a reliable feasibility study become weak.
An Introduction to the Theory of Sampling: An Essential Part of Total Quality Management
13
1.01.8.3.2
Variability issues during mining During mining it is important to select a logical grade control drilling pattern and drilling density. The selection of a logical Kriging technique depends on quality data,8 and so does the selection of realistic and economic cutoff grades. All this may affect the selection of a long-term pit design. A reasonable acceptance level for conciliation differences between ore-grade control and the prediction from the geological model is crucial. With a large V[0] damaging the database, these critically important tasks leading to a reasonable and economic recovery of natural resources become weak, which necessarily result in financial losses.
1.01.8.3.3
Variability issues within a processing plant Within a processing plant, believable metallurgical accounting needs to monitor performance. Metallurgists need to control key process-operating parameters. Process trends need to be tamed in due time. Process cycles, always very costly, need to be identified and either eliminated or minimized. Reliable variographic control charts must be constantly updated at many places. Overcorrections of the process must be prevented. With a large V[0] damaging the database, these critically important tasks leading to a reasonable and economic recovery of natural resources become weak, which necessarily result in financial losses.
1.01.8.3.4
Variability issues during trade with customers Customers like a fair price, but hate bad surprises on product quality. Penalty application is a common way of doing business: ‘what costs me must cost you!’ The quality of a product cannot be controlled after the fact, and the key for success is to implement the many things that lead to a good product, all the way from geology, mining, and the processing plants. With a large V[0] damaging the database, these critically important tasks to keep quality of products within client specifications become weak, which necessarily result in further financial losses.
1.01.9 Conclusions There is no such a thing as continuous improvement without a clear understanding of TOS followed by a strong commitment to correct sampling practices. Many statisticians believe that interleaved sampling can solve everything if followed by a careful statistical analysis of the results. There is nothing wrong in their approach; however, there are two possible outcomes: 1. The variance analysis shows that there is no problem and all parties involved may feel comfortable enough. 2. The variance analysis shows that there is a problem. Now, without a clear understanding of TOS it is very unlikely they will find the causes and cure them effectively in the most economic way. The strategy that is suggested in TOS by dividing the complex problem into its basic components is the inescapable way to reach solutions quickly and effectively.
1.01.10 Recommendations The objective of this paper is only to help people looking at TOS for the first time to get organized. It is well known that newcomers often get discouraged as the way TOS is frequently presented can be intimidating. If the reader gets well organized, looking at the challenge with a clear Cartesian spirit, TOS is actually a lot simpler than what people may think.12,13 Follow the strategy illustrated in Figure 8 and quick progress is likely to occur. With emphasis on causes of sampling problems, the difficult climb to continuous improvement may take place. With emphasis on effects of sampling problems, as it is often the case, there is no progress possible. After a commitment has been taken for continuous improvement, it is still necessary to create a road map so that economic benefits can quickly be measured. A typical road map is illustrated in Figure 9.
14 An Introduction to the Theory of Sampling: An Essential Part of Total Quality Management
If cause eliminated:
standardize If cause not eliminated:
re-analyze
Compulsory action on causes of sampling problems
Short courses, workshops, and training
Analyze existing data and find structural problems
Lost opportunities with emphasis on effects of problems Continuous improvement of mining process with emphasis on causes of problems Figure 8 Make a commitment for continuous improvement.
Director of standards of mining process: The synergy necessary for mining process efficiency
Selection of standards useful to a mining company Implementation of company's guidelines
Selection and offering of short courses, workshops, and training
Guidelines of best practices Selection of world experts
Identification of structural sampling problems and continuous improvement of mining process
Compulsory actions
Communication with QA/QC and Laboratory Managers
Figure 9 A typical road map suggested for the mining industry.
Accountability
Communication with top management at a company’s operations
An Introduction to the Theory of Sampling: An Essential Part of Total Quality Management
15
References 1. Gy, P. Sampling of Particulate Materials, Theory and Practice. Developments in Geomathematics; Elsevier Scientific Publishing: Amsterdam, 1979 and 1983; Vol. 4. 2. Gy, P. Heterogeneite – Echantillonnage – Homogeneisation: Ensemble coherent de theories; Masson Editeur: Paris; 1988, ISBN: 2-225-81313-2. ISSN: 0765-0221. 3. Gy, P. Sampling of Heterogeneous and Dynamic Material Systems: Theories of Heterogeneity, Sampling and Homogenizing; Elsevier: Amsterdam, 1992. 4. Pitard, F. Pierre Gy’s Sampling Theory and Sampling Practice, 2nd ed.; CRC Press: Boca Raton, FL, 1993. 5. Pitard, F. Effects of Residual Variances on the Estimation of the Variance of the Fundamental Error. Chemometr. Intell. Lab. 2004, 74(1), 149–164. In the text, specifically referred to equations (2), (3), (4), (5). 6. Pitard, F. Blasthole Sampling for Grade Control: The Many Problems and Solutions. Sampling 2008, The Australasian Institute of Mining and Metallurgy, Perth, Australia, 2008; May 27–28, 2008. 7. Pitard, F. Sampling Correctness – A Comprehensive Guideline. Second World Conference on Sampling and Blending 2005, Publication Series No 4/2005; The Australian Institute of Mining and Metallurgy, 2005; ISBN: 1-920-80628-8. 8. Carrasco, P.; Carrasco, P.; Jara, E.; The Economic Impact of Correct Sampling and Analysis Practices in the Copper Mining Industry. Chemometr. Intell. Lab. 2004, 74(1), 209–231. 9. Franc¸ois-Bongarc¸on, D.; Theory of Sampling and Geostatistics: An Intimate Link. Chemometr. Intell. Lab. 2004, 74(1), 143–148. 10. Pitard, F. Practical Statistical Tools for Managers, Metallurgical Plant Design and Operating Strategies Conference,Sydney April 15–16, The Australasian Institute of Mining and Metallurgy, Sydney, April 15–16, 2002. 11. Pitard, F. Chronostatistics – A Powerful, Pragmatic, New Science for Metallurgists. Metallurgical Plant Design and Operating Strategies (MetPlant 2006), The Australasian Institute of Mining and Metallurgy, Perth, W.A. September 18–19, 2006. 12. Esbensen, K. H.; Minkkinen, P. Guest Editors. In Chemometrics and Intelligent Laboratory Systems, Vol. 74, 1, 2004, Special issue: 50 years of Pierre Gy’s Theory of Sampling, Proceedings First World Conference on Sampling and Blending (WCSB1). Tutorials on Sampling: Theory and Practice. 13. Lars, P. Pierre Gy’s Theory of Sampling (TOS) – In Practice: Laboratory and Industrial Didactics Including a First Foray into Image Analytical Sampling, Phd Thesis, ACABS Research Group, Aalborg University Esbjerg, Niels Bohrs Vej 8, Dk-67000 Esbjerg, Denmark, 2005.
16 An Introduction to the Theory of Sampling: An Essential Part of Total Quality Management
Biographical Sketch
Mr. Francis F. Pitard is a consulting expert in Sampling, Statistical Process Control, and Total Quality Management for 23 years. He is President of Francis Pitard Sampling Consultants (www.fpscsampling.com) and Technical Director of Mineral Stats Inc. (www.mineralstats.com) in Broomfield, CO, USA. He provides consulting services in many countries. Mr. Pitard has 6 years of experience with the French Atomic Energy Commission and 15 years with Amax Extractive R&D. He has taught Sampling Theory, SPC, and TQM to approximately 6000 people and to more than 275 companies around the world through Continuing Education Offices of the Colorado School of Mines, for the Australian Mineral Foundation, for the Mining Department of the Universidad de Chile, and the University of Witwatersrand. He has degrees in chemistry from the Gay-Lussac Institute in Paris and from the Academy of Paris.
1.02 Quality of Analytical Measurements: Statistical Methods for Internal Validation M. C. Ortiz, L. A. Sarabia, M. S. Sa´nchez, and A. Herrero, University of Burgos, Burgos, Spain ª 2009 Elsevier B.V. All rights reserved.
1.02.1 1.02.2 1.02.2.1 1.02.2.2 1.02.2.3 1.02.2.4 1.02.2.5 1.02.2.6 1.02.2.7 1.02.2.8 1.02.3 1.02.3.1 1.02.3.2 1.02.3.3 1.02.3.4 1.02.3.5 1.02.3.6 1.02.3.7 1.02.3.8 1.02.4 1.02.4.1 1.02.4.2 1.02.4.3 1.02.4.4 1.02.4.5 1.02.4.6 1.02.5 1.02.5.1 1.02.5.2 1.02.5.3 1.02.5.4 1.02.5.5 1.02.5.6 References
Introduction Confidence and Tolerance Intervals Confidence Interval Confidence Interval on the Mean of a Normal Distribution Confidence Interval on the Variance of a Normal Distribution Confidence Interval on the Difference in Two Means Confidence Interval on the Ratio of Variances of Two Normal Distributions Confidence Interval on the Median Joint Confidence Intervals Tolerance Intervals Hypothesis Test Elements of a Hypothesis Test Hypothesis Test on the Mean of a Normal Distribution Hypothesis Test on the Variance of a Normal Distribution Hypothesis Test on the Difference in Two Means Test Based on Intervals Hypothesis Test on the Variances of Two Normal Distributions Hypothesis Test on the Comparison of Several Independent Variances Goodness-of-Fit Tests: Normality Tests One-Way Analysis of Variance The Fixed Effects Model Power of the ANOVA for the Fixed Effects Model Uncertainty and Testing of the Estimated Parameters for the Fixed Effects Model The Random Effects Model Power of the ANOVA for the Random Effects Model Confidence Intervals of the Estimated Parameters for the Random Effects Model Statistical Inference and Validation Trueness Precision Statistical Aspects of the Experiments to Determine Precision Consistency Analysis and Incompatibility of Data Accuracy Ruggedness
Symbols 1– 1– CC CC
F1 ;2 H0
confidence level power limit of decision capability of detection F distribution with v1 and v2 degrees of freedom null hypothesis
H1 N(,) NID(,2)
s s2
18 23 23 23 25 26 27 28 28 29 31 31 34 36 36 38 39 41 43 45 46 48 49 53 53 54 54 54 55 58 58 63 63 72
alternative hypothesis normal distribution with mean and standard deviation independent random variables equally distributed as normal with mean and variance 2 sample standard deviation sample variance
17
18 Quality of Analytical Measurements: Statistical Methods for Internal Validation
t X
"
2 R r
Student’s t distribution with v degrees of freedom sample mean significance level, probability of type I error probability of type II error bias (systematic error) random error
w 2 ;w2
mean degree(s) of freedom, d.f. standard deviation variance reproducibility (as standard deviation) repeatability (as standard deviation) (chi-squared) distribution with degrees of freedom
1.02.1 Introduction The set of operations to determine the value of an amount (measurand) suitably defined is called the measurement. The method of measurement is the sequence of operations that is used in the performance of measuremens. It is documented with enough details so that the measurement may be done without additional information. Once a method is designed or selected, it is necessary to evaluate its characteristics of operation and to identify the factors that can change these characteristics and to what extent they can change. If, in addition, the method is developed to solve a particular analytical problem, it is necessary to verify that the method is fit for purpose.1 This process of evaluation is called validation of the method. It implies the determination of several parameters that characterize the performance of the method: capability of detection, selectivity, specificity, ruggedness, and accuracy (trueness and precision). In any case, they are the measurements themselves which allow evaluation of the performance characteristics of the method and its fit for purpose. In addition, in the use stage of the method, they turn out to be the obtained measurements that will be used to make decisions on the analyzed sample, for example whether the amount of an analyte fulfills a legal specification and consequently whether the material from which the sample is taken is valid. Therefore, it is necessary to suitably model the data that a method provides. In what follows we will consider that the data provided by the analytical method are real numbers; other possibilities exist, for example, the count of bacteria or impacts in a detector take only (discrete) natural values. In addition, sometimes, the data resulting from an analysis are qualitative, for example, detection of the presence of an analyte in a sample. With regard to the analytical measurement, it is admitted that the value (measurment), x, provided by the method of analysis consists of three terms, the true value of the parameter , a systematic error (bias) , and a random error " with a zero mean, in an additive way: x ¼þþ"
ð1Þ
All the possible measurements that a method can provide when analyzing a specific sample constitute the population of the measurements. Ideally, this supposes to admit that it has limitless samples and that the method of analysis remains unalterable. In these conditions, the model of the analytical method, Equation (1), is mathematically a random variable, X, with mathematical expectation þ and variance equal to the variance of ", that is, a random variable X of mean E(X) ¼ þ and variance V(X) ¼ V("). A random variable, and thus the analytical method, is described by its probability distribution FX(x), that is, the probability that the method provides measurements less than or equal to x for any value x. Symbolically this is written as FX(x) ¼ pr{X x} for any real value x. In most of the applications, it is assumed that FX(x) is differentiable, which implies, among other things, that none of the possible results of the method has positive probability, that is, the probability of obtaining exactly a specific value is zero. In the case of a differentiable probability distribution, the derivative of FX(x) is the probability density function R (pdf) fX(x). Any function f (x) such that (1) it is positive, f (x) 0, and (2) the area under the function is 1, R f ðxÞ dx ¼ 1, is the pdf of a random variable. The probability that the random variable X takes values in the interval [a, b] is the area under the pdf over the interval [a, b], that is Z
b
prfX P½a; b g ¼ a
f ðxÞ dx
ð2Þ
Quality of Analytical Measurements: Statistical Methods for Internal Validation
19
and the mean and variance of X are written as Z E ðX Þ ¼
xf ðxÞ dx
ð3Þ
ðx – E ðX ÞÞ2 f ðxÞ dx
ð4Þ
R
Z V ðX Þ ¼ R
In general, the mean and the variance do not characterize in a unique way a random variable and therefore the method of analysis. Figure 1 shows the pdf of four random variables with the same mean 6.00 and standard deviation 0.61. These four distributions, uniform or rectangular (Figure 1(a)), triangular (Figure 1(b)), normal (Figure 1(c)), and Weibull (Figure 1(d)), are rather frequent in the scope of analytical determinations, and they appear in the EURACHEM/CITAC Guide1 (Appendix E) and also they are used in metrology.2 If the only available information regarding a quantity X is the lower limit, l, and the upper limit, u, but the quantity could be anywhere in between, with no idea of whether any part of the range is more likely than another part, then a rectangular distribution in the interval [l,u] would be assigned to X. This is so because this is the pdf that maximizes the ‘information entropy’ of Shannon, in other words the pdf that adequately characterizes the incomplete knowledge about X. Frequently, in reference material, the certified concentration is expressed in terms of a number and unqualified limits (e.g., 1000 2 mg l1). In this case, a rectangular distribution should be used (Figure 1(a)). When the available information concerning X includes the knowledge that values close to c (between l and u) are more likely than those near the bounds, the adequate distribution is a triangular one (Figure 1(b)), with the maximum of its pdf in c. If a good estimate, m, and associated standard uncertainty, s, are the only information available regarding X, then, according to the principle of maximum entropy, a normal probability distribution N(m,s) (Figure 1(c)) would be assigned to X (remember that m and s may have been obtained from repeated applications of a measurement method). Finally, the Weibull distribution (Figure 1(d)) is very versatile; it can mimic the behavior of other distributions such as the normal or exponential. It is adequate for the analysis of reliability of processes, and in chemical analysis it is useful in describing the behavior of the figures of merit of a long-term procedure, for
(a) 1.2 1.0 0.8 0.6 0.4 0.2 0.0
(b) 1.2 1.0 0.8 0.6 0.4 0.2 0.0 4
5
6
7
8
(c) 1.2 1.0 0.8 0.6 0.4 0.2 0.0
4
5
6
7
8
4
5
6
7
8
(d) 1.2 1.0 0.8 0.6 0.4 0.2 0.0 4
5
6
7
8
Figure 1 Probability density functions of four random variables with mean 6 and variance 0.375. (a) Uniform in [4.94, 7.06]. (b) Symmetric triangular in [4.5, 7.5]. (c) Normal (6, 0.61). (d) Weibull with shape 1.103 and scale 0.7 shifted to give a mean of 6.
20 Quality of Analytical Measurements: Statistical Methods for Internal Validation
example, the distribution of the capability of detection CCb3 or in the determination of ammonia in water by UV–vis spectroscopy during 350 different days.4 In the four cases given in Figure 1, the probability of obtaining values between 5 and 7 has been computed from Equation (2). For the uniform distribution (Figure 1(a)) this probability is 0.94, whereas for the triangular distribution (Figure 1(b)) it is 0.88, for the normal distribution (Figure 1(c)) 0.90, and for the Weibull distribution (Figure 1(d)) 0.93. Therefore, the proportion of the values that each distribution accumulates in the interval [5,7] orders the distributions as uniform, Weibull, normal, and triangular, although the triangular and normal distributions tend to give values symmetrically around the mean and the Weibull distribution does not. If another interval is considered, say [5.4,6.6], the distributions accumulate probabilities of 0.57, 0.64, 0.67, and 0.54, respectively, in which the difference between the values is more than before and, in addition, order the distributions as normal, triangular, uniform, and Weibull. If for each of those variables, value b is determined so that there is a fixed probability, p, of obtaining values below b (i.e., the value b such that p ¼ pr{X < b} for each distribution X), the results of Table 1 are obtained. For example, 5% of the times, a uniform distribution gives values less than b ¼ 5.05, less than 4.97 if it is a triangular distribution, and so on. In the table, the extreme values among the four distributions for each value of p have been marked, and great differences are observed caused as much by the form in which the values far from 6 are distributed (i.e., very different for the normal, the triangular, or the uniform distribution) as by the asymmetry of the Weibull distribution. Therefore, the mean and variance of a random variable constitute very limited information on the values provided by the random variable, unless additional information is at hand about the form of its density (pdf). For example, if one knows that the distribution is uniform or symmetrical triangular or normal, the random variable is completely characterized by its mean and variance. In practice, the pdf of a method of analysis is unknown. We only have a finite number, n, of these measurements, which are the results obtained when applying repeatedly (n times) the same method to the same sample. These n measurements constitute a statistical sample of the random variable X defined by the method of analysis. Figure 2 shows 100 results obtained when applying four methods of analysis, named A,B,C, and D, to aliquot parts of a sample to determine an amount whose true value (unknown) is ¼ 6. Clearly, the four methods behave differently. From the experimental data, the (sample) mean and variance are computed as Pn
i¼1 xi
x ¼ Pn s2 ¼
ð5Þ
n – xÞ2 n–1
i¼1 ðxi
ð6Þ
x and s2 are estimates of the mean and variance of the distribution of X. These estimates with the data in Figure 2 are shown in Table 2. Table 1 Values of b such that p ¼ pr{X < b} where X is one of the random variables in Figure 1 Random variable p
Uniform
Triangular
Normal
Weibull
0.01 0.05 0.50 0.95 0.99
4.96 5.05 6.00b 6.95a 7.04a
4.71 4.97a 6.00b 7.03 7.29
4.58a 4.99 6.00b 7.01 7.42
5.34b 5.37b 5.83a 7.22b 8.12b
a b
Minimum b among the four distributions. Maximum b among the four distributions.
Quality of Analytical Measurements: Statistical Methods for Internal Validation
(a) 40
(b) 40
30
30
20
20
10
10
0
21
0 3
4
5
6
7
8
9
10
(c) 40
(d) 40
30
30
20
20
10
10
0
3
4
5
6
7
8
9
10
3
4
5
6
7
8
9
10
0 3
4
5
6
7
8
9
10
Figure 2 Histograms of 100 measurements obtained with four different analytical methods on aliquot parts of a sample. (a) method A; (b) method B; (c) method C; (d) method D.
Table 2 Some characteristics of the distributions in Figure 2 Method
Mean, x Variance, s2 pr{5 < X < 7} pr{X < 6} pr{5 < N(x,s) < 7} pr{N(x,s) < 6}
A
B
C
D
6.66 0.25 0.76 0.09 0.75 0.09
6.66 1.26 0.54 0.24 0.55 0.27
6.16 1.26 0.63 0.46 0.62 0.44
6.16 0.25 0.94 0.35 0.94 0.37
According to the model of Equation (1), E ðX Þ ¼ þ . x , that is, the mean estimates the true value plus the bias . The bias estimated for methods A and B is 0.66 whereas for methods C and D it is 0.16. The bias of a method is one of its performance characteristics and must be evaluated during the validation of the method. In fact, technical guides, for example, that by the International Organization for Standardization (ISO), define that a method better fulfills the trueness if it has less bias. To estimate the bias, it is necessary to have samples with known concentration (e.g., certified material, spiked samples). The value of the variance is independent of the true content, , of the sample. For this reason, to estimate the variance, it is only necessary to have replicated measurements on aliquot parts of the same sample. Table 2 shows that methods B and C have the same variance, 1.26, which is 5 times greater than that of methods A and D, 0.25. The dispersion of the data that provides a method is the precision of the method and constitutes another performance characteristic to be determined in the validation of the method. In agreement with model (1), a measure of the dispersion is the variance V(X), which is estimated by means of s2. In some occasions, for evaluating trueness and precision, it is more descriptive to use statistics other than mean and variance. For example, when the distribution is rather asymmetric, as in Figure 1(d), it is more reasonable to use the median than the mean. The median is the value in which the distribution accumulates 50% of the probability, 5.83 for the pdf in Figure 1(d) and 6.00 for the other three distributions, which are symmetric around their mean. In practice, it is frequent to see the presence of anomalous (outliers) data that influence the mean and above all the variance, which is improperly increased; in these cases, it is advisable to use robust estimates of the centrality and dispersion values.5–7 In Chapter 1.07 of the present book, there is a detailed description of these robust procedures.
22 Quality of Analytical Measurements: Statistical Methods for Internal Validation
Figure 2 and Table 2 show that the two characteristics, trueness and precision, are independent in the sense that a method with better trueness (small bias), cases C and D, can be more, case D, or less, case C, precise. Analogously, A and B have an appreciable bias but A is more precise than B. A method is said to be accurate when it is precise and fulfills trueness. The histograms are an estimate of the pdf and allow evaluation of the performance of each method in a more detailed way than when considering only trueness and precision. For example, the probability of obtaining values in any interval can be estimated with the histogram. The third row in Table 2 shows the estimated frequencies for the interval [5,7]. Method D (better trueness, better precision) provides 94% of the values in the interval, whereas method B (worse trueness and precision) provides only 54% of the values in the interval. Trueness and precision interact, according to data in Table 2: The effect of increasing the precision, from B to A, when the bias is ‘small’ is an increase of 22% of results of a measurement to be in the interval [5,7], whereas when the bias is high, the increase is about 31%. This interaction should be taken into account when optimizing a process and also in the ruggedness analysis, which is another performance characteristic to be validated according to most of the guides. As can be seen in the last row of Table 2, if the method that provides more results below 6 is needed, C would be the method selected. The previous analysis shows the usefulness of knowing the pdf of the results of a method of analysis. As in practice we have only a limited number of results, two basic strategies are possible to estimate it: (1) to demonstrate that the experimental data are compatible with a known distribution (e.g., normal) and use the corresponding pdf; (2) to estimate the pdf by a data-driven technique based on a computer-intensive method such as the kernel method8 described in Section 1.03.2.4 of the present book, or by using other methods such as adaptive or penalized likelihood.9,10 The data of Figure 2 can be adequately modeled by a normal distribution, according to the results of four tests (chi-square, Shapiro–Wilks, skewness, kurtosis) (see Section 1.02.3.8), and the procedure will be detailed in the following. The last two rows in Table 2 show the probabilities of obtaining values in the interval [5,7] or less than 6 with the fitted normal distribution. When comparing these values with those computed with the empirical histograms (compare rows 3 and 5, and rows 4 and 6), evidently, there are no appreciable differences. In the validation of an analytical method and during its later use, methodological strategies of statistical type are needed to make decisions from the available experimental data. The knowledge of these strategies supposes a way to think and to act that, subordinated to the chemical knowledge, objectives both the analytical results and their comparison with those of other researchers and/or other analytical methods. Ultimately, a good method of analysis is a serious attempt to come close to the true value of the measurement. The difference between the true value, always unknown, and the one experimentally obtained can be estimated. For this reason, the result of a measurement has to be accompanied by an evaluation of (un)certainty or its degree of reliability. This is done by means of a confidence interval. When it is required to establish the quality of an analytical method, its capability of detection, precision, etc. have to be compared with those corresponding to other methods. This is formalized with a hypothesis test. Confidence intervals and test of hypothesis are the basic tools in the validation of analytical methods. In this introduction, the word sample has been used with two different meanings. Usually, there is no confusion because the context allows one to distinguish whether it is a sample in the statistical or chemical sense. In chemistry, according to the International Union for Pure and Applied Chemistry (IUPAC) (Section 18.3.2 of Incze´dy et al.11), ‘sample’ should be used only when it is a part of a selected material of a great amount of material. This meaning coincides with that of a statistical sample and implies the existence of sampling error, that is, error caused by the fact that the sample can be more or less representative of the amount in the material. For example, suppose that we want to measure the amount of plaguicide that remains in the ground of an arable land after a certain time. For this, we take several samples ‘representative’ of the ground of the parcel (statistical sampling) and this introduces an uncertainty in the results characterized by a variance (theoretical) 2s . Afterward, the quantity of plaguicide in each chemical sample is determined by an analytical method, which has its own uncertainty, characterized by 2m , in such a way that the uncertainty in the quantity of plaguicide in the parcel is 2s þ 2m provided that the method gives results independent of the location of the sample. Sometimes, when evaluating whether a method is adequate for a task, the sampling error can be an important part of the uncertainty in the result and, of course, should be taken into account to plan the experimentation. The modeling of sampling errors is studied in Chapter 1.01.
Quality of Analytical Measurements: Statistical Methods for Internal Validation
23
When the sampling error is negligible, for example when a portion is taken from a homogeneous solution, the IUPAC recommends to use words such as test portion, aliquot, or specimen.
1.02.2 Confidence and Tolerance Intervals There are important questions when evaluating a method, for example, what is the maximum value that it provides? In fact, given the random character of the results, the question cannot be answered with just a number. In order to include the degree of certainty in the answer, the question should be reformulated: What is the maximum value, U, that will be obtained 95% of the times that the method is used? The answer is the tolerance interval and to build it, the probability distribution must be known. Let us suppose that we know that it is a N(,), then a possible answer is U ¼ þ z0.05 because then the probability that the analytical method gives values greater than U is pr{method > U} ¼ pr{N(,) > þ z0.05}, which, according to the result in Appendix, is equal to pr{N(0,1) > z0.05} ¼ 0.05. For any 100 (1 )%, the value U ¼ þ z
ð7Þ
is the maximum value provided by the method, with a probability that the aforementioned assertion is false. Analogously, the results that will be obtained the 100 (1 )% of times will be above the following value, L L ¼ – z
ð8Þ
Then, the statement ‘the method provides values greater than L’ will be false 100 % of the times. Finally, the interval [L,U] contains 100 (1 )% of the values proportioned by the method ½L; U ¼ – z=2 ; þ z=2
ð9Þ
or, in other words, the statement that the method gives values between L and U is false 100 % of the times. 1.02.2.1
Confidence Interval
We have already noted that estimation of solely the mean, x, and the variance, s2, from n independent results provides very limited information on the performance of the method. The objective now is to make affirmations of the type ‘in the sample, the amount of the analyte, (unknown), is between L and U ( P [L,U]) with a certain probability that the statement is true’. In general, to obtain a confidence interval for a random variable, X, from a sample x1,x2,. . .,xn consists of obtaining two functions l(x1,x2,. . .,xn) and u(x1,x2,. . .,xn) such that prfX P½l; ug ¼ prfl X ug ¼ 1 –
ð10Þ
1 is the confidence level and is the significance level, that is to say, the statement that the value of X is between l and u will be false 100 % of the times. In the following, this idea will be specified in some interesting cases. 1.02.2.2
Confidence Interval on the Mean of a Normal Distribution
Case 1: Suppose that we have a random variable that follows a normal distribution with known variance. This will be the case, for example, of using an already validated method of analysis. In this case, we know that " in Equation (1) is normally distributed and its variance. Taking into account the pffiffiffiproperties of the normal distribution (see Appendix), the sample mean, X, is a random variable N ð; = nÞ; thus, by Equation (10), the following holds: pr – z=2 pffiffiffi X þ z=2 pffiffiffi ¼ 1 – n n
ð11Þ
24 Quality of Analytical Measurements: Statistical Methods for Internal Validation
that is, 100 (1 )% of the values of the sample mean are in the interval in Equation (11). A simple algebraic manipulation (subtract and X, multiply by –1) gives p ffiffi ffi p ffiffi ffi pr X – z=2 X þ z=2 ¼ 1– n n
ð12Þ
Therefore, according to Equation (10), the confidence interval on the mean that is obtained from Equation (12) is X – z=2 pffiffiffi ; X þ z=2 pffiffiffi n n
ð13Þ
Analogously, the confidence intervals at level 100 (1 )% for the maximum and minimum values of the mean are computed from pr X þ z pffiffiffi ¼ 1 – n pr X – z pffiffiffi ¼ 1 – n
ð14Þ ð15Þ
When measuring n aliquot parts of a test sample, we obtain n values x1,x2,. . .,xn; the sample mean x is the value that the random variable X takes and an estimate of the quantity . Example 1: Suppose that the analytical method follows a N(,4) and we have a sample of size 10 made up of values 98.87, 92.54, 99.42, 105.66, 98.70, 97.23, 98.44, 103.73, 94.45, and 101.08. pffiffiffiffiffi
The mean is 99.01 and, pffiffiffiffiffi
thus, the interval at 95% confidence level for this sample is [99.01 1.96 4= 10 , 99.01 þ 1.96 4= 10 ] equal to [96.54,101.48]. When considering different samples of size 10, different intervals will be obtained at the same 95% confidence level. The endpoints of these intervals are nonrandom values, and the unknown mean value, which is also a specific value, will or will not belong to the interval. Therefore, the affirmation ‘the interval contains the mean’ is a deterministic assertion that is true or false for each of the intervals. What one knows is that it is true for 100 (1 )% of those intervals. In our case, as 95% of the constructed intervals will contain the true value, we say, with a confidence level of 95%, that the interval [96.54,101.48] contains . This is the interpretation with the frequentist approach adopted in this chapter, that is to say that the information on random variables is obtained by means of samples of them and that the parameters to be estimated are not known but are fixed amounts (e.g., the amount of analyte in a sample, , is estimated by the measurement results obtained by analyzing it n times). With a Bayesian approach to the problem, a probability distribution is attributed to the amount of analyte and once fixed an interval of interest [a,b], the ‘a priori’ distribution of , the experimental results, and the Bayes’ theorem are used to calculate the probability ‘a posteriori’ that belongs to the interval [a,b]. It is shown that, although in most practical cases the uncertainty intervals obtained from repeated measurements using either theory may be similar, their interpretation is completely different. In Chapter 1.08 of this book, Bayesian statistical techniques are described. The works by Lira and Wo¨ger 12 and Zech13 are devoted to compare both approaches from the point of view of the experimental data and their uncertainty. Case 2: Suppose a normally distributed random variable with unknown variance that must be estimated together with the mean from n experimental data. The confidence interval is computed as in Case 1, but now the random variable X follows a Student’s t distribution with n 1 degrees of freedom (d.f.) (see Appendix); thus, the interval at the 100 (1 )% confidence level is given from s s pr X – t =2; pffiffiffi X þ t =2; pffiffiffi ¼ 1 – n n
ð16Þ
Quality of Analytical Measurements: Statistical Methods for Internal Validation
25
where t/2, is the upper percentage point of the Student t distribution with ¼ n 1 d.f. and s is the standard deviation computed with the sample. Analogously, the one-sided intervals at the 100 (1 )% confidence level come from s pr X þ t ; pffiffiffi ¼ 1 – n s pr X – t ; pffiffiffi ¼ 1 – n
ð17Þ ð18Þ
Example 2: Suppose that the distribution of the analytical method is a normal but its standard deviation is not known. With the data of Example 1, the sample standard deviation, s, is computed as 3.91. As t0.025,9 ¼ 2.262 (see Appendix), the confidence interval at 95% level is [99.01 2.26 1.24, 99.01 þ 2.26 1.24] ¼ [96.21,101.81]. The 95% confidence interval on the minimum of the mean is made, according to Equation (18), by all the values greater than 96.74 ¼ 99.01 1.83 1.24. The corresponding interval on the maximum will be made by the values less than 101.28. The length of the confidence intervals from Equations (12)–(15) tends toward zero when the sample size tends to infinity. This permits the computation
2 d. pffiffiffi of the sample size needed to obtain an interval of given length, It will suffice to consider d =2 ¼ z=2 = n, and take as n the nearest integer greater than 2z=2 =d . For example, if we want a 95% confidence interval with length, d, less than 2, in the hypothesis of Example 1, we will need a sample size greater than or equal to 62. The same argument can
2 be applied when the standard deviation is unknown. However, in this case, to compute n by 2t=2; s=d it is necessary to have an initial estimation of s, which, in general, is obtained in a pilot test with size n9, in such a way that in the previous expression the d.f., , are n9 1. An alternative is to enunciate the length of the interval in standard deviation units (remember that the standard deviation is unknown). For instance, in Example 2, if we want d ¼ 0.5s, we will need a sample size greater than (4z/2)2 ¼ 61.5; note the substitution of t/2, by z/2, which is mandatory because we do not have the sample size needed to compute t/2,, which is precisely what we want to estimate.
1.02.2.3
Confidence Interval on the Variance of a Normal Distribution
In this case, the data come from a N(,) distribution with and unknown, and we have a sample with values x1,x2,. . .,xn. The distribution of the random variable ‘sample variance’ S2 is related to the chi-square distribution, w2 (see Appendix). As a consequence, the confidence interval at 100 (1 )% level for the variance 2 is obtained from pr
( ) ðn – 1ÞS 2 ðn – 1ÞS 2 2 ¼ 1– w2=2; w21 – =2;
ð19Þ
where w2=2; is the critical value of a chi-square distribution with ¼ n 1 d.f. at level /2. As in the previous case for the sample mean, we should distinguish between the random variable sample variance S2 and one of its concrete values, s2 computed with Equation (6), that takes this variable when we have the sample x1,x2,. . .,xn. The intervals on the maximum and minimum of the variance at 100 (1 )% confidence level are obtained from Equations (20) and (21), respectively. (
ðn – 1ÞS 2 pr 2 w1 – ;
)
2
(
ðn – 1ÞS 2 pr 2 w2;
¼ 1–
ð20Þ
¼ 1–
ð21Þ
)
26 Quality of Analytical Measurements: Statistical Methods for Internal Validation
Example 3: Knowing that the n ¼ 10 data of Example 2 come from a normal distribution with both mean and variance unknown, the 95% confidence interval on 2 is found from Equation (19) as [7.22,50.83] because s2 ¼ 15.25, w20:025;9 ¼ 19:02, and w20:975;9 ¼ 2:70. If the analyst is interested in obtaining a confidence interval for the maximum variance, the 95% upper confidence interval is found from Equation (20) as [0,41.22] because w20:95;9 ¼ 3:33, that is, the upper bound for the variance is 41.22 with a probability of error equal to 0.05. To obtain confidence intervals for the standard deviation, it suffices to take the square root of the aforementioned intervals because this operation is a monotonically increasing transformation of the values; therefore, the intervals at 95% confidence level for the standard deviation are [2.69,7.13] and [0,6.42]. 2 2 The size, n, of thesample needed pffiffiffiffiffiffiffiffiffiffi ffi so that 2 s / is between 1 k and 1 þ k is given by the nearest integer greater than 1 þ 1=2 z=2 1 þ k þ 1 =k . For example, for k ¼ 0.5, such that the length of the confidence interval verifies 0.5 < s2/2 < 1.5, we need n ¼ 40 data (at least). Just for comparative purposes, we will admit that with the sample of size 40 we obtain the same variance s2 ¼ 15.25, w20:025;39 ¼ 58:12, and w20:975;39 ¼ 23:65; hence, the interval at 95% confidence level is [10.23,25.15], which verifies the required specifications.
1.02.2.4
Confidence Interval on the Difference in Two Means
Case 1. Known variances: Consider two independent random variables distributed as N1(1,1) and N2(2,2) with unknown means and known variances 21 and 22. We wish to find a 100 (1 )% confidence interval on the difference in means 1 2. Let x11,x12,. . ., x1n1 be a random sample of n1 observations from N1 and x21,x22,. . ., x2n2 be a random sample of n2 observations from N2. The 100 (1 )% confidence interval on 1 2 is obtained from the equation 8 sffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi sffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi9 < 2 2 21 22 = 1 2 ¼ 1– pr X 1 – X2 – z=2 þ 1 – 2 X1 – X 2 þ z=2 þ : n1 n2 n1 n2 ;
ð22Þ
where X1 and X 2 are the random variables of the sample mean, which take the values x1 and x2 . The expressions analogous to Equations (14) and (15) for the one-sided intervals are obvious. Case 2. Unknown variances: The approach to this question is similar to the previous case, but here even the variances 21 and 22 are unknown. However, it can be reasonable to assume that both variances are equal, 21 ¼ 22 ¼ 2, and that the differences observed in their estimates with the data of both samples are not significant. Later, in the section dedicated to hypothesis tests, the methodology to decide about this question will be explained. An estimate of the common variance 2 is given by the pooled sample variance, which is an arithmetic average of both variances weighted by the corresponding d.f., sp2 ¼
ðn1 – 1Þs12 þ ðn2 – 1Þs22 n 1 þ n2 – 2
ð23Þ
The 100 (1 )% confidence interval is obtained from the following equation: rffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi rffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi 1 1 1 1 ¼ 1– pr X1 – X2 – t =2; s p þ 1 – 2 X1 – X 2 þ t =2; s p þ n1 n2 n1 n2
ð24Þ
where ¼ n1 þ n2 2 are the d.f. of the Student’s t distribution. The one-sided intervals at 100 (1 )% confidence level have an obvious expression deduced from Equation (24) by substituting t/2, for t, . If a fixed length is desired for the confidence interval, the computation explained in Section 1.02.2.2 can be immediately adapted to obtain the needed sample size. Example 4: We want to verify that a substance is sufficiently stable to remain unchanged in composition when it is stored for a month. Two series of measurements (n1 ¼ n2 ¼ 8) were made before and after a storage period. The results were x1 ¼ 90:8, s12 ¼ 3:89 and x2 ¼ 92:7, s22 ¼ 4:02, respectively. Compute a 95% confidence interval on the difference of means.
Quality of Analytical Measurements: Statistical Methods for Internal Validation
27
The two-sided interval when assuming equal variances (sp2 ¼ 3:96) (Equation (24)) is (90.8 92.7) 2.1448 pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi 1.99 1=8 þ 1=8 ¼ ½ – 4:034; 0:234. As zero belongs to the interval, we can conclude that the substance is stable at 95% confidence level. When the assumption 21 ¼ 22 is not reasonable, we can still obtain an interval on the difference 1 2 by pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi using the fact that the statistic ½X1 – X 2 – ð1 – 2 Þ s12 =n1 þs22 =n2 is distributed approximately as a t with d.f. given by ¼
2
2 s1 =n1 þ s22 =n2 2 2 ðs12 =n1 Þ ðs22 =n2 Þ n1 – 1 þ n2 – 1
ð25Þ
The 100 (1 )% confidence interval is obtained from the following equation: 8 sffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi sffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi9 < 2 2 s s s2 s2 = 1 2 pr X 1 – X2 – t =2; þ 1 – 2 X1 – X 2 þ t =2; 1 þ 2 ¼ 1 – : n1 n2 n1 n2 ;
ð26Þ
Example 5: We want to compute a confidence interval on the difference of two means with unknown and nonequal variances, with the results that come from an experiment carried out with four aliquot samples by two different analysts. The first analyst obtains x1 ¼ 3:285, whereas the second one obtains x2 ¼ 3:257. The variances were s12 ¼ 3:3310 – 5 and s22 ¼ 9:1710 – 5 respectively. Assuming that 21 6¼ 22 , Equation (25) gives ¼ 4.9, so the d.f. to apply Equation (26) are 5 and t0.025,5 ¼ 2.571. Thus, the 95% confidence interval is pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi ð3:285 – 3:257Þ 2:571 ð3:3310 – 5 =4Þ þ ð9:1710 – 5 =4Þ ¼ ½0:014; 0:042. That is, at 95% of confidence, the two analysts provide unequal measurements because zero is not in the interval. The confidence intervals for the maximum and the minimum are obtained by considering the last or the first term respectively in Equation (26) and replacing t/2, by t, . Case 3. Confidence interval for paired samples: Sometimes we are interested in evaluating an effect (e.g., the reduction of a polluting agent in an industrial spill by means of a catalyst) but it is impossible to have two homogeneous populations of samples without and with treatment to obtain the two means of the recoveries, because the amount of polluting agent is not controllable. In these cases, the solution is to determine the polluting agent before and after applying the procedure to the same spill. The difference between both determinations is a measure of the effect of the catalyst. The (statistical) samples obtained in this way are known as paired samples. Formally, the problem consists of having two paired samples of size n, x11,x12,. . ., x1n and x21,x22,. . ., x2n , and compute the differences between any pair of data, di ¼ x1i– x2i, i ¼ 1,2,. . .,n. If these differences follow a normal distribution, the 100 (1 )% confidence interval is obtained from sd sd pr d – t =2; pffiffiffi d þ t =2; pffiffiffi ¼ 1 – n n
ð27Þ
where d and sd are the mean and standard deviation of the differences di and ¼ n 1 are the d.f.
1.02.2.5
Confidence Interval on the Ratio of Variances of Two Normal Distributions
This section approaches the question of giving a confidence interval on the ratio 21/22 of the variances of two distributions N1(1,1) and N2(2,2) with unknown means and variances. Let x11,x12,. . ., x1n1 be a random sample of n1 observations from N1 and x21,x22,. . ., x2n2 be a random sample of n2 observations from N2. The 100 (1 )% confidence interval on the ratio of variances is given by the following equation: S12 21 S12 pr F1 – =2;1 ;2 2 2 F=2;1 ;2 2 ¼ 1 – S2 2 S2
ð28Þ
28 Quality of Analytical Measurements: Statistical Methods for Internal Validation
where F1 – =2;1 ;2 and F=2;1 ;2 are the critical values of an F distribution with 1 ¼ n2 1 d.f. in the numerator and 2 ¼ n1 1 d.f. in the denominator. Appendix contains a description of the F distribution. We can also compute one-sided confidence intervals. The 100 (1 )% upper or lower confidence limit on 21/22 is obtained from Equations (29) and (30), respectively: 2 S2 pr 12 F;1 ;2 12 ¼ 1 – 2 S2 S12 21 pr F1 – ;1 ;2 2 2 ¼ 1 – S2 2
ð29Þ ð30Þ
Example 6: In this example, we compute a two-sided 95% confidence interval for the ratio of the variances in Example 4. The resulting interval is [0.20 (3.89/4.02), 4.99 (3.89/4.02)] ¼ [0.19,4.83]. As 1 belongs to this interval, we can admit that both variances are equal. 1.02.2.6
Confidence Interval on the Median
This case is different from the previous cases, because the confidence interval is a ‘distribution-free’ interval, that is, there is no distribution assumed for the data. As it is known, a percentile (pct) is the value xpct such that 100 pct% of the values are less than or equal to xpct. It is possible to compute confidence intervals on any pct, but for values of pct near one or zero we need sample sizes, n, that are very large because the values n pct and n (1 pct) must be greater than 5. For the median (pct ¼ 0.5), it suffices to consider samples of size 10 or more. The fundamentals of these confidence intervals are based on the binomial distribution whose details are outside the scope of this chapter and can be found in Sprent.14 We use the data of Example 1 to show step by step how a 100 (1 )% confidence interval on the median is computed. The procedure consists of three steps: 1. To sort the data in ascending order. In our case, 92.54, 94.45, 97.23, 98.44, 98.70, 98.87, 99.42, 101.08, 103.73, and 105.66. The rank of each datum is the position that it occupies in the sorted list, for example, the rank of 98.44 is four. 2. To calculate the rank,prlffiffi,ffi of the the interval. It is the nearest integer
value that will be the lower endpoint of pffiffiffiffiffi less than 1=2 n – z=2 n þ 1 . In our case, this value is 0.5(10 1.96 10 þ 1Þ ¼ 2:40, thus rl ¼ 2. the value that will be the upper endpoint of the interval. It is the nearest integer 3. To calculate the rank, ru, of pffiffiffiffiffi pffiffiffi
greater than 1=2 n þ z=2 n – 1 . In our case, this value is 0.5(10 þ 1.96 10 – 1Þ ¼ 7:60, then ru ¼ 8. Hence, the 95% confidence interval on the median is made by the values that are between positions 2 and 8, that is, [94.45,101.08]. 1.02.2.7
Joint Confidence Intervals
Sometimes it is necessary to compute confidence intervals for several parameters but maintaining a 100 (1 )% confidence that all of them contain the true value of the corresponding parameter. For example, if two parameters are statistically independent, we can assure a 95% joint level of confidence by taking separately the corresponding 100 (1 )1/2% confidence intervals because (1 )1/2 (1 )1/2 ¼ (1 ). In general, if there are k parameters, we will compute the 100 (1 )1/k% confidence interval for any of them. However, if the sample statistics used are not independent, the above computation is not valid. The Bonferroni inequality states that the probability P that all the affirmations are true at 100 (1 – )% confidence k , where 1 i is the confidence level considered for the ith level is greater than or equal to 1 – i i¼1 interval (usually i ¼ /k). For example, if a joint 90% confidence interval is needed for the mean of two distributions, according to Bonferroni inequality i ¼ /2 ¼ 0.10/2 ¼ 0.05; thus, each should be the corresponding 95% confidence interval.
Quality of Analytical Measurements: Statistical Methods for Internal Validation
1.02.2.8
29
Tolerance Intervals
In the introduction to Section 1.02.2, the tolerance intervals of a normal distribution have been calculated knowing its mean and variance. Remember that the confidence interval [l,u] contains 100 (1 )% of the values of the distribution or, equivalently, pr{X ˇ [l,u]} ¼ . Actually, the values of the parameters that define the probability distribution are unknown; this uncertainty should be transferred into the endpoints of the interval. There are several types of tolerance regions, but in this chapter we will restrict ourselves to two common cases. Case 1. -content tolerance interval: Given a random variable X, an interval [l,u] is a -content tolerance interval at the confidence level if the following is fulfilled: prfprfX P½l; ug g
ð31Þ
To put in words, [l,u] contains at least 100 % of the X values with confidence level. For the case of an analytical method, this is to say that we have to determine, based on a sample of size n, the interval that will contain 95% ( ¼ 0.95) of the results and this assertion must be true 90% of the times ( ¼ 0.90). Evidently, the -content tolerance intervals can be one-sided, which means that the procedure will provide 95% of its results above l (respectively, below u) 90% of the times. We leave to the reader the corresponding formal definitions. One-sided and two-sided -content tolerance intervals can be computed either by controlling the center or by controlling the tails, and for both continuous and discrete random variables (a review can be seen in Patel15). Here we will limit to the case of a normally distributed X with unknown mean and variance, of which we have a sample of size n, with which the mean x and standard deviation s are estimated. We want to obtain a two-sided -content tolerance interval controlling the center, that is, an interval such that prfprfX P½x – ks; x þ ks g g
ð32Þ
To determine k, several approximations have been reported; consult Patel15 for a discussion on them. The approach by Wald and Wolfowitz16 is based on determining k1 such that 1 1 pr N ð0; 1Þ pffiffiffi þ k1 – pr N ð0; 1Þ pffiffiffi – k1 ¼ n n
ð33Þ
sffiffiffiffiffiffiffiffiffiffiffiffi n–1 k ¼ k1 w2;n – 1
ð34Þ
Therefore
w2,n1 is the point exceeded with probability when using the chi-square distribution with n 1 d.f. Example 7: With the data in Example 1, x ¼ 99:01, s ¼ 3.91, k1 ¼ 2.054, and w20:95;9 ¼ 3:33; thus, according to Equation (34), k ¼ 3.377 and, as a consequence, the interval [99.01 3.38 3.91, 99.01 þ 3.38 3.91] ¼ [85.81,112.23] contains 95% of the results of the method 95% of the times that the procedure is repeated with a sample of size 10. Remember the following points: (1) This tolerance interval is for individual results and pffiffinot ffi for mean values as the confidence intervals. (2) The standard deviation of the mean is estimated as s= n ¼ 1:24, whereas the standard deviation of the individual results of the method is estimated to be 3.91. (3) The length of the -content tolerance interval does not tend toward zero when increasing the sample size, as does with the confidence intervals. Now, the value of k in Equation (34) tends to z(1–)/2, which is the length of the theoretical interval that, in our example, would be [91.35,106.67] with z0.025 ¼ 1.96. Case 2. The interval [l,u] is called a -expectation tolerance interval if E fprfX P½l; ugg ¼
ð35Þ
Unlike the -content tolerance interval, condition (35) only demands that, on average, the probability that the random variable takes values between l and u is .
30 Quality of Analytical Measurements: Statistical Methods for Internal Validation
As in the previous case, we limit ourselves to obtain intervals of the form [ x – ks; x þ ks]. When the distribution of the random variable is normal and we have a sample of size n, the solution was obtained for the first time by Wilks17 and is k ¼ tð1 – Þ=2;
rffiffiffiffiffiffiffiffiffiffiffi nþ1 n
ð36Þ
where tð1 – Þ=2; is the upper (1-)/2 percentage point of the t distribution with ¼ n 1 d.f. For the same data as before, the expectation tolerance interval at 95% is [99.01 2.37 3.91, 99.01 þ 2.37 3.91] ¼ [89.73,108.29] because t0:025;9 ¼ 2:262. This interval is shorter than the -content tolerance interval because it only assures the mean of the probabilities that the interval contains the value of the random variable X. In fact, the interval [89.73,108.29] contains 95% of the values of X only 64% of the times, conclusion drawn by applying Equation (32) with k ¼ 2.37. Also, note that when the sample size tends to infinity, the value of k in Equation (36) tends toward z(1–)/2. Case 3. It is also possible to obtain tolerance intervals independent of the distribution (provided it is continuous) of variable X. These intervals are based on the rank of the observations, but they demand very large sample sizes, which makes them quite useless in practice. For example, to guarantee that the -content tolerance interval [l,u] is made as l ¼ x(1) and u ¼ x(n) (i.e., the endpoints are the smallest and the greatest values in the sample), it is necessary that n fulfills approximately the equation logðnÞ þ ðn – 1Þlogð Þ ¼ logð1 – Þ – logð1 – Þ.18 If we need, as in Example 7, ¼ ¼ 0.95, the value of n has to be 89. Nevertheless, Willinks19 has used the Monte Carlo method to compute shorter ‘distribution-free’ tolerance intervals (-content and -expectation); this is of utility in the calculation of the uncertainty proposed in Draft Supplement2 but it still requires sample sizes that are rather large in the scope of the chemical analysis. A complete theoretical development on the tolerance intervals (including their estimation by means of Bayesian methods) is in the book by Guttman.20 The tolerance intervals are of interest to show that a method is fit for purpose because when establishing that the interval [ x – ks; x þ ks] will contain in average 100 % of the values provided by the method (or 100 % of the values with confidence level), we are including precision and trueness. It suffices that the tolerance interval [ x – ks; x þ ks] is included in the specifications that the method should fulfill, so that it is valid for that purpose. Note that a method with high precision (small value of s) but with a significant bias can get to fulfill the specifications in the sense that a high proportion of its values are within the specifications. In addition in the estimation of s, the repeatability can be introduced as the intermediate precision or the reproducibility to consider the scope of application of the method. The use of the tolerance interval solves the problem of introducing the bias as a component of the uncertainty, in clear contradiction with the model of Equation (1). With the aim of developing analytical fit for purpose methods, the Societe´ Franc¸aise des Sciences et Techniques Pharmaceutiques (SFSTP) has proposed21–24 the use of the -expectation tolerance intervals in the validation of quantitative methods. In four case studies, it has shown the validity of -expectation tolerance intervals as an adequate way to conciliate both the objectives of the analytical method in routine analysis and those of the validation step, and it proposes them as a criterion to select the calibration curve.25 Also, it has analyzed their adequacy to the guides26 that establish the performance criteria that should be validated and their usefulness in the problem of the transference of an analytical method.27 Gonza´lez and Herrador28 have proposed their computation from the estimation of uncertainty of the analytical assay. In all these cases, the expectation tolerance intervals based on the normality of data are used, that is, using Equation (36). To avoid dependence on the underlying distribution and the use of the classic distribution-free methods, Rebafka et al.29 have proposed the use of a bootstrap technique to calculate -expectation tolerance intervals, whereas Fernholz and Gillespie30 have studied the estimation by means of bootstrap of the -content tolerance intervals. Nevertheless, the confidence intervals will have to be used when evaluating the trueness and precision of a method with no need to have as objective the fulfillment of external requirements but their possible comparison with other methods and the quantification of the uncertainty and the bias of the results that it provides. There are other aspects of the determination of the uncertainty that are of practical interest, for example the problem that raises the fact that any uncertainty interval, particularly an expanded uncertainty interval, should
Quality of Analytical Measurements: Statistical Methods for Internal Validation
31
be restricted to the range of feasible values of the measurand. Cowen and Ellison31 analyzed how to modify the interval when the data are close to a natural limit in a feasible range such as 0 or 100% mass or mole fraction.
1.02.3 Hypothesis Test This section is dedicated to the introduction of a statistical methodology to decide whether an affirmation is false, for example, the affirmation ‘this method of analysis applied to this sample of reference provides the certified value’. If on the basis of the experimental results it is decided that it is false, we will conclude that the method has bias. The affirmation is habitually called hypothesis and the procedure of decision making is called hypothesis testing. A statistical hypothesis is an asseveration on the probability distribution that follows a random variable. Sometimes one has to decide on a parameter, for example, whether the mean of a normal distribution is a specific value. In other occasions it may be required to decide on other characteristics of the distribution, for example, whether the experimental data are compatible with the hypothesis that they come from a normal or uniform distribution. 1.02.3.1
Elements of a Hypothesis Test
As the results that provide the analytical methods are modeled by means of a probability distribution, it is evident that both, the validation of a method and its routine use, imply to make decisions that are formulated in a natural way as problems of hypothesis testing. In order to describe the elements of a hypothesis test, we will use a concrete case. Example 8: For an experimental procedure, we need solutions with pH values less than 2. The preparation of these solutions provides pH values that follow a normal distribution with ¼ 0.55. pH values obtained from 10 measurements were 2.09, 1.53, 1.70, 1.65, 2.00, 1.68, 1.52, 1.71, 1.62, and 1.58. Is the pH of the resulting solution adequate to proceed with the experiment? We may express this formally as H0 : ¼ 2:00 ðinadequate solutionÞ H1 : < 2:00 ðvalid solutionÞ
ð37Þ
The statement ‘ ¼ 2.00’ in Equation (37) is called the null hypothesis, denoted as H0, and the statement ‘ < 2.00’ is called the alternative hypothesis, H1. As the alternative hypothesis specifies values of that are less than 2.00 it is called one-sided alternative. In some situations, we may wish to formulate a two-sided alternative hypothesis to specify values of that could be either greater or less than 2.00 as in H0 : ¼ 2:00 H1 : 6¼ 2:00
ð38Þ
The hypotheses are not affirmations about the sample but about the distribution from which those values come, that is to say, is the value, unknown, of the pH of the solution that will be the same as the value provided by the procedure if the bias is zero (see the model of Equation (1)). In general, to test a hypothesis, the analyst must consider the experimental objective and decide upon a null hypothesis for the test, as in Equation (37). Hypothesistesting procedures rely on using the information in a random sample; if this information is inconsistent with the null hypothesis, we would conclude that the hypothesis is false. If sufficient evidence does not exist to prove falseness, the test defaults to the conclusion of the test is not to reject the null hypothesis but this does not actually prove that it is correct. It is therefore critical to choose carefully the null hypothesis in each problem. In practice, to test a hypothesis, we must take a random sample, compute an appropriate test statistic from the sample data, and then use the information contained in this statistic to make a decision. However, as the
32 Quality of Analytical Measurements: Statistical Methods for Internal Validation Table 3 Decisions in hypothesis testing The unknown truth Researcher’s decision
H0 is true
H0 is false
Accept H0 Reject H0
No error Type I error
Type II error No error
decision is based on a random sample, it is subject to error. Two kinds of potential errors may be made when testing hypothesis. If the null hypothesis is rejected when it is true, then a type I error has been made. A type II error occurs when the researcher fails to reject the null hypothesis when it is false. The situation is described in Table 3. In Example 8, if the experimental data lead to rejection of the null hypothesis H0 being true, our (wrong) conclusion is that the pH of the solution is less than 2. A type I error has been made and the analyst will use the solution in the procedure when in fact it is not chemically valid. If, on the contrary, the experimental data lead to acceptance of the null hypothesis when it is false, the analyst will not use the solution when in fact the pH is less than 2 and a type II error has been made. Note that both types of error have to be considered because their consequences are very different. In the case of type I error, a nonsuitable solution is accepted, the procedure will be inadequate, and the analytical result will be wrong with the later damages that it may cause (e.g., the loss of a client, or a mistaken environmental diagnosis). On the contrary, the type II error implies that a valid solution is not used with the corresponding extra cost of the analysis. It is clear that the analyst has to specify the risk that assumes to make these errors, and this is done in terms of the probability that they occur. The probabilities of occurrence of type I and II errors are denoted by specific symbols:
¼ prftype I errorg ¼ pr reject H0 =H0 is true
¼ prftype II errorg ¼ pr accept H0 =H0 is false
ð39Þ
In Equation (39), the symbol ‘/’ indicates that the probability is calculated under that condition. In the example we are following, will be calculated by means of a normal distribution of mean 2 and standard deviation 0.55. The probability of the test is called the significance level, and the power of the test is 1 , which measures the probability of correctly rejecting the null hypothesis. Statistically, in the example, one wants to decide about the value of the mean of a normal distribution with known variance and one-sided alternative test). p ffiffiffiffiffi With these data, the statistic is pffiffiffi hypothesis (a one-tail (Table 4, second row) Zcalc ¼ ðx – Þ=ð= nÞ ¼ ð1:708 – 2:0Þ= 0:55= 10 ¼ – 1:679: In addition, the analyst must assume the risk , say 0.05, which means that the decision rule that is going to apply to its experimental results will accept an inadequate solution 5% of the times. Therefore, the critical or rejection region is defined (Table 4, second row) as the set CR ¼ {Zcalc < 1.645}, that is, the null hypothesis will be rejected for the samples of size 10 that provide values of the statistic less than –1.645. In the example, the value Zcalc belongs to the critical region; thus, the decision is to reject the null hypothesis (i.e., the prepared solution is adequate) at 5% significance level. Given the present facilities of computation, instead of the CR, the statistical packages calculate the so-called p-value, which is the probability of obtaining the computed value of the statistic under the null hypothesis H0. In our case, p-value ¼ pr{Z 1.679} ¼ 0.0466. When the p-value is less than the significance level , the null hypothesis is rejected because this is equivalent to saying that the value of the statistic belongs to the critical region. However, which is the power of the decision rule (statistic and critical region) that has been used? Equation (39) implies that to calculate it is necessary to specify what is exactly understood by the alternative hypothesis. In our case, what is understood by pH smaller than 2. From a mathematical point of view, the answer is clear: any number smaller than 2, for example 1.999 9; from the point of view of the analyst, this mathematical answer does not have sense. Sometimes on the basis of the previous knowledge, in other cases because of the regulatory stipulations or simply by the detail of the working standardized procedure, the analyst
Quality of Analytical Measurements: Statistical Methods for Internal Validation
33
Table 4 Some parametric hypothesis tests Null hypothesis
Alternative hypothesis
Statistic
Critical region – 0 pffiffiffi = n
1 2 3
¼ 0 ¼ 0 ¼ 0
6¼ 0 < 0 > 0
Zcalc ¼
4 5 6
¼ 0 ¼ 0 ¼ 0
6¼ 0 < 0 > 0
tcalc ¼
7 8
1 ¼ 2 1 ¼ 2
1 6¼ 2 1 > 2
ð1 – 2 Þ ffi Zcalc ¼ qffiffiffiffiffiffiffiffiffiffiffiffi 21 22 n1 þ n2
9
1 ¼ 2
1 6¼ 2
ð1 – 2 Þ tcalc ¼ qffiffiffiffiffiffiffiffiffiffiffiffiffi ð Þ sp n11 þ n12
10
1¼ 2
1 > 2
11
d ¼ 0
d 6¼ 0
12
d ¼ 0
d > 0
13
2 ¼ 20
2 6¼ 20
14
2 ¼ 20
2 > 20
15
21 ¼ 22
21 6¼ 22
16
21 ¼ 22
21 > 22
– 0 pffiffiffi s= n
Zcalc < – Z=2 or Zcalc >Z=2 fZcalc < – Z g fZcalc >Z g
t < – t=2;n – 1 or tcalc >t=2;n – 1
calc t < – t;n – 1
calc tcalc > t;n – 1
Zcalc < – z=2 or Zcalc > z=2 fZcalc > – z g
tcalc < – t=2;n1 þn2 – 2 or tcalc > t=2;n1 þn2 – 2 tcalc >t;n1 þn2 – 2
n
o
tcalc ¼
d¯ pffiffiffi ð Þ sd = n
w2calc ¼
ðn – 1Þs12 20
n o w2calc < w21 – =2;n – 1 or w2calc >w2=2;n – 1 n o w2calc > w2;n – 1
Fcalc ¼
s12 s22
n o Fcalc < F1 – =2;n1 – 1;n2 – 1 or Fcalc >F=2;n1 – 1;n2 – 1
Fcalc > F;n1 – 1;n2 – 1
tcalc < – t=2;n – 1 or tcalc >t=2;n – 1
tcal > t;n – 1
The values z are the percentiles of a standard normal distribution such that ¼ pr{N(0,1) > z}. The values t, are the percentiles of a Student’s t distribution with degrees of freedom such that ¼ pr{t > t,}. ðn1 – 1Þs12 þ ðn2 – 1Þs22 is the pooled variance. ( ) sp2 ¼ n1 þ n2 – 2 ¯ ( ) d is the mean of the differences di ¼ x1i – x2i between the paired samples; sd is its standard deviation. n o The values w2; are the percentiles of a w2 distribution with degrees of freedom such that ¼ pr w2 > w2; . The values F, 1,2 are the percentiles of an F distribution with 1 degrees of freedom for the numerator and 2 degrees of freedom for the denominator, such that ¼ pr{F > F,1,2}.
can decide the value of pH that is considered less than 2.00, for example, a pH less than 1.60. This is the same as assuming as ‘pH equal to 2’ any smaller value such that its distance to 2 is less than 0.40. In these conditions jj pffiffiffi ¼ pr N ð0; 1Þ < z – n
ð40Þ
Where jj ¼ 0.40 in our problem and when replacing it in Equation (40), we have ¼ 0.26 (calculations can be seen in Example A9 of Appendix). That is to say, whatever the decision taken, the decision rule leads to throw a valid solution away 26% of the times. Evidently, this is an inadequate rule. A simple analysis of Equation (40) explains the situation. To decrease , we should decrease the value pffiffiffi z – ðjj=Þ n. This may be done by decreasing z (i.e., increasing the significance level ) or increasing pffiffiffi ðjj=Þ n. As both the procedure precision, , and the difference of pH that we wish to detect are fixed, the only possibility left is to increase the sample size. Solving Equation (40) in n we have n.
2 z þ z ðjj=Þ2
ð41Þ
The values of and for sample sizes of 10, 15, 20, and 25, maintaining and fixed, are given in Figure 3. As can be seen, and exhibit opposite behavior and, unless the sample size is increased, it is not
34 Quality of Analytical Measurements: Statistical Methods for Internal Validation
0.50 0.45
β = pr{Type II error}
0.40 0.35
n = 10
0.30 n =15
0.25
n = 20
0.20
n = 25
0.15 0.10 0.05 0
0
0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40 0.45 0.50 α = pr{Type I error}
Figure 3 Simultaneous (opposite) behavior of and for different sample sizes, n ¼ 10, 15, 20, and 25.
possible to simultaneously decrease the probability of both errors. In our case, Equation (41) gives n ¼ 20.5 for ¼ ¼ 0.05, thus n ¼ 21 because the sample size must be an integer. The dotted lines in Figure 3 intersect in values of 0.263, 0.126, 0.058, and 0.025, which correspond to the sample sizes considered while maintaining the significance level ¼ 0.05. Again, we see that for a given , the risk decreases with the increase in n. Equation (40) also allows the analyst to decide the standard deviation (precision) necessary to obtain a decision rule according to the risks and that he/she wishes to admit. For example, if one has to make a decision on the validity of the prepared solution with 10 results and the analyst wishes ¼ ¼ 0.05, the only pffiffiffiffiffi option according to Equation (40) is to increase jj=. Solving 0.05 ¼ pr{N(0,1) < 1.645 ð0:40=Þ 10}, one obtains ¼ 0.3845. This means that the procedure should be improved from the current value of 0.55 until 0.38. If only five results were allowed, the standard deviation would have to decrease to 0.27 for maintaining both the significance level and the power of the test. Finally, there is an aspect in Equation (40) that does not have to pass unnoticed. Maintaining , , and n fixed, it is possible to modify the pH value that can be distinguished from 2.00 if the analyst simultaneously increases the precision of the method so that the ratio jj= stays constant. Said otherwise, without changing any of the specifications of the hypothesis test, by diminishing we can discriminate a value of pH nearer to 2. Qualitatively this argument is obvious – if a procedure is more precise, more similar results are easier to distinguish, so that with a more precise procedure different values will appear that would be considered equal with a less precise procedure. Equation (40) quantifies this relation for the hypothesis test we are doing. In summary, a hypothesis test includes the following steps: (1) defining the null, H0, and the alternative, H1, hypothesis according to the purpose of the test and the properties of the distribution of the random variable, which, according to Equation (1), is the distribution of the values provided by a method of measurement; (2) deciding on the probabilities and , that is, the risk for the two types of error that will be assumed for the decision; (3) computing the needed sample size; (4) obtaining the results, computing the corresponding test statistic and evaluating whether it belongs to the critical region CR; and (5) writing the analytical conclusion, which should entail more than reporting the pure statistical test decision. The conclusion should include the elements of the statistical test, the assumed distribution, , , and n. Care must be taken in writing the conclusion; for example, it is more adequate to say ‘there is no experimental evidence to reject the null hypothesis’ than ‘the null hypothesis is accepted’. Table 4 summarizes the tests most frequently used in the validation of analytical procedures and in the analysis of their results. 1.02.3.2
Hypothesis Test on the Mean of a Normal Distribution
Case 1. Known variance: Admit that the data follow a normal N(,) distribution with unknown . The corresponding tests are in rows 1–3 in Table 4. The test statistic is always the same, but, depending on
Quality of Analytical Measurements: Statistical Methods for Internal Validation
35
whether the alternative hypothesis is two-sided (row 1 in Table 4) or one-sided (rows 2 and 3), the critical region is different. The value z/2 verifies /2 ¼ pr{N(0,1) > z/2} or the analogous result for z. For the twotail test, the relation among n, , and is given by n.
2 z=2 þ z
ð42Þ
ðjj=Þ2
whereas for the one-tail tests, Equation (41) must be used. Case 2. Unknown variance: In this case, both the mean, , and the standard deviation, , of the normal distribution are unknown. The hypothesis tests are in row 4 of Table 4 for the two-tail case and in rows 5 and 6 for the one-tail tests. The statistic tcalc should be compared to the values t,n1 and t/2,n1 of the Student’s t distribution with n 1 d.f. The equation that relates , , and n is
¼ pr – t=2;n – 1 tn – 1 ðÞ t=2;n – 1
ð43Þ
pffiffiffi where ¼ ðjj=Þ n is the noncentrality parameter of a noncentral t() distribution, which in Equation (43) has n 1 d.f. Note the analogy with the ‘shift’ of the N(0,1) in Equation (40). The discussion about the relative effect of the sample size and the precision is similar to the case in which the variance is t ð Þ if H1: < 0 and known. The corresponding equations for one-tail tests are ¼ pr – t ;n – 1 n – 1 ¼ pr tn – 1 ðÞ t;n – 1 if H1: > 0. To compute n from Equation (43), the standard deviation is needed. To solve this additional difficulty, the comments in Case 2 of Section 1.02.2.2 are valid and can be applied here also. Usually, ¼ 2 or 3. Let us compare the solutions with known and unknown variance by supposing the data of Example 8, but considering that the variance is unknown. We wish to detect differences in pH of 0.73 (the same / as in Example 8). By using a sample of size 10, the probability is 0.32 instead of the previous 0.26 (calculations can be seen in Example A10 of Appendix). This increase in the probability of type II error is due to the less information we have about the problem; now the standard deviation is unknown. Case 3. The paired t-test: In Case 3 of Section 1.02.2.4 the experimental procedure and the reasons for considering paired samples have been already explained. To decide on the effect of a treatment, the null hypothesis is that the mean of the differences is zero, that is, H0: d ¼ 0 and the two-sided alternative H1: d 6¼ 0. This is the test shown in row 11 of Table 4, where there is only a one-tail test (row 12) because, if needed, it suffices to consider the opposite differences di ¼ x2i x1i instead of di ¼ x1i x2i, i ¼ 1,. . .,n. The statistic and the critical region are analogous to those of Case 2 (test on the mean with unknown variance). Example 9: Table 5 shows the results of the recoveries obtained in 10 different places with two solid-phase extraction (SPE) cartridges after fortification of wastewater with a sulfonamide. We want to decide whether cartridge A is more efficient than cartridge B and to compute the risk of the text. For answer these questions, it is important to define that we consider ‘different’ those differences between the means of recoveries that are greater than 2%. We use a paired t-test on the mean of the differences between the recoveries obtained with the two cartridges on the same sample (those of cartridge A minus those of cartridge B). By considering these differences, we eliminate the effect that the wastewater could have on the performance of the two cartridges. The test is carried out as follows: H0 : d ¼ 0 ðthe differences of recoveries are nullÞ
ð44Þ
H1 : d > 0 ðcartridge A provides recoveries greater than cartridge BÞ Table 5 Recovery of sulfonamides spiked in wastewater obtained by using two different extraction cartridges Place
1
2
3
4
5
6
7
8
9
10
Cartridge A (%) Cartridge B (%)
77.2 74.4
74.0 70.0
75.6 70.2
80.0 77.2
75.2 75.9
69.2 60.0
75.4 77.0
74.0 76.0
71.6 70.0
60.4 55.0
See Example 9 for more details.
36 Quality of Analytical Measurements: Statistical Methods for Internal Validation
pffiffiffiffiffi
pffiffiffi Following row 12 of Table 4, the statistic is tcalc ¼ d=ðsd = nÞ ¼ 2:69= 3:526= 10 ¼ 2:412 and the critical region CR ¼ {tcalc > t,n – 1}. The critical value t0.05,9 is equal to 1.833; thus the statistic belongs to the critical region. Therefore, the null hypothesis is rejected for ¼ 0.05 and we can conclude that cartridge A is more efficient than cartridge B, because the mean of the differences is positive.
To evaluate the power (1 ) of the test, the equation ¼ pr tn – 1 ðÞ t;n – 1 with d ¼j – 0 j= ¼ jj= ¼ 2=3:53 ¼ 0:57 provides 1 ¼ 0.496 6 for ¼ 0.05 and n ¼ 10 (calculations can be seen in Example A11 of Appendix). Hence, 50% of the times the conclusion of accepting that there is no difference between recoveries is wrong. In this case, the risk of a type II error is very large; in other words, the power is very poor when we want to discriminate differences of 2% in recovery because the ratio d is small. 1.02.3.3
Hypothesis Test on the Variance of a Normal Distribution
The variance is a measure of the dispersion of the data used to evaluate the precision of a procedure of analysis; thus, decisions have to be made on this parameter frequently. The corresponding hypothesis tests are in rows 13 and 14 of Table 4. Example 10: A validated procedure has a repeatability of 0 ¼ 1.40 mg l–1 when measuring concentrations around 400 mg l–1. After a technical revision of the instrument, the laboratory is interested in testing the hypothesis H0 : 2 ¼ 20 ¼ 1:96 ðthe repeatability has not changedÞ H1 : 2 > 1:96 ðthe repeatability got worseÞ
ð45Þ
The analyst decides that a repeatability is admissible until 2.0 times the initial one, 1.40 mg l–1, and assumes the risks ¼ ¼ 0.05. The sample size needed to guarantee the requirements of the analyst is formally obtained from a one-tail hypothesis test on the variance where k ¼ pr w2n – 1 < 2
ð46Þ
k is the value such that ¼ pr w2n – 1 > k and ¼ =0 . As ¼ 2.0, Equation (46) gives that for n ¼ 14, ¼ 0.040 2, whereas for n ¼ 13, ¼ 0.051 1 (calculations can be seen in Example A12 of Appendix). Therefore, he/she decides to do 14 determinations on aliquot parts of a sample with 400 mg l–1 obtaining a variance of 3.1 (s ¼ 1.76). The statistic is w2calc ¼ ð14 – 1Þ3:10=1:96 ¼ 20:56. As the critical region is made by the values CR ¼ fw2calc >w20:05;13 ¼ 22:36g, he/she concludes that there is no sufficient experimental evidence to say that the precision has worsened. In this case, the acceptance of the null hypothesis, that is, to maintain the repeatability below 2.0 times the initial one, will be erroneous 5% of the times because was fixed at 5%. The decision rule is equally protected against type I and II errors. 1.02.3.4
Hypothesis Test on the Difference in Two Means
Case 1. Known variances: We assume that X1 is normal and has unknown mean 1 and known variance 21, and that X2 is also normal with unknown mean 2 and known variance 22. We will be concerned with testing the hypothesis that the means 1 and 2 are equal. The two-sided alternative hypothesis is in line 7 of Table 4 and the one-sided in line 8 when we have a random sample of size n1 of X1 and another random sample of size n2 of X2. Example 11: A solid-phase microextraction (SPME) procedure to extract triazines from wastewater has been carried out. The results must be compared with previous ones where extraction was made by means of SPE. The repeatability of both procedures is known to be 5.36% for SPME procedure and 3.12% for SPE. The mean for 10 samples (Table 6) is 85.9% for recovery obtained with SPME and 81.8% for SPE. At a 0.05 significance level, is the recovery of SPME procedure greater than that of SPE?
Quality of Analytical Measurements: Statistical Methods for Internal Validation
37
Table 6 Recoveries of triazines from wastewater using solid-phase microextraction (SPME) and solid-phase extraction (SPE) Recovery (%) SPME SPE
91 86
85 82
90 85
81 86
79 79
78 82
84 80
87 77
93 79
91 82
See Example 11 for more details.
As the standard repeatability of both procedures is known, a test to compare two means with normal distribution and known variances is adequate. H0 : SPME ¼ SPE ðthe recovery is the same for both proceduresÞ H1 : SPME > SPE ðthe recovery using SPME procedure is greater than that using SPEÞ
ð47Þ
pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi The statistic is Zcalc ¼ ðxSPME – xSPE Þ= 2SPME =n1 þ 2SPE =n2 ¼ ð85:9 – 81:8Þ= 28:73=10 þ 9:73=10 ¼ 2:091 following line 8 of Table 4. For a significance level ¼ 0.05, CR ¼ {Zcalc > z ¼ 1.645}. As the statistic 2.091PCR, the null hypothesis is rejected and we conclude that the mean of the recovery with SPME is greater than that with SPE. If a difference in recovery of 3% is sufficient in the analysis, what is the risk for this hypothesis test? A simple modification of Equation (40) shows that 8 9 > > < j j = ffi ¼ pr N ð0; 1Þ > z – qffiffiffiffiffiffiffiffiffiffiffiffi > 21 22 > : ; n1 þ n2
ð48Þ
By substituting our data in Equation (48), one obtains ¼ 0.55. That means that in 55% of the cases, we will incorrectly accept that the recovery is the same for both procedures. It is also possible to derive formulas to estimate the sample size required to obtain a specified for given and . For the one-sided alternative, the sample size n ¼ n1 ¼ n2 is n.
2 z þ z 2 21 þ22
ð49Þ
Again, with the data of the problem at hand and ¼ 0.05, Equation (49) gives the number 46.30, that is, 47 aliquot samples should be analyzed for each procedure so that ¼ ¼ 0.05. For the two-sided alternative, the sample size n ¼ n1 ¼ n2 is n.
2 z=2 þ z 2 21 þ22
ð50Þ
Case 2. Unknown variances: As in Section 1.02.2.4 (Case 2), there are two possibilities: (1) the unknown variances are equal 21 ¼ 22 ¼ 2, although for chance reasons the numerical values differ, and (2) both variances are different 21 6¼ 22. The question of deciding between (1) and (2) will be approached in Section 1.02.3.6. Let X1 and X2 be two independent normal random variables with unknown but equal variances. The statistics and the critical region for the two-tail test are in line 9 of Table 4 and in line 10 we can see the one-tail case. For the two-sided alternative, with risks and , we consider further that the two means are different if their difference is at least a quantity ¼ j1 2j. As the variances are unknown, the comments in Case 2 of Section 1.02.2.2 are also applicable. If we have samples from a pilot experiment with respective sizes n19 and n29, and sp2 is the pooled variance computed with them (see the statistic in lines 9 and 10 of Table 4), the sample size needed n ¼ n1 ¼ n2 is n.
2 t=2; þ t; 2 =2sp2
ð51Þ
38 Quality of Analytical Measurements: Statistical Methods for Internal Validation
where ¼ n19 þ n29 2 are the d.f. of the Student’s t distribution. If the aforementioned is not possible, the difference to be detected should be expressed in standard deviation units, then ¼ j1 2j ¼ k and the following expression applies:
2 z=2 þ z n. k2 =2
ð52Þ
where z/2 and z are the corresponding upper percentage points of the standard normal distribution. Example 12: An experimenter wishes to compare the mean of two procedures, stating that they are to be considered different if they differ in 2 or more standard deviations (k ¼ 2). In addition, he/she wants to assume ¼ 0.05 and ¼ 0.10. As z0.025 ¼ 1.96 and z0.10 ¼ 1.282, Equation (52) gives n ¼ 5.26, then six samples must be considered for each procedure. If he/she wishes to distinguish 1 standard deviation (k ¼ 1), then n ¼ 21.02, that is, he/she should have 22 data from each procedure. Although it is preferable to always take equal sample sizes, it can happen that it is more expensive or laborious to collect the data of X1 than that of X2. In this case, there are weighted sample sizes to be considered.32 In the case where equality of variances 21 and 22 cannot be admitted, there is no completely justified solution for the test. However, approximations exist with good power and easy to use tests, such as the Welch test. This method consists of substituting the known variances in the expression of Zcalc (Case 1 of this section) by their sample estimates, in such a way that the statistic becomes x1 – x2 t calc ¼ qffiffiffiffiffiffiffiffiffiffiffiffi s12 s22 n1 þ n2
ð53Þ
which follows a Student’s t with the d.f. in Equation (25). The critical region for the two-tail test is CR ¼ {tcalc < –t/2, or tcalc > t/2,}. The critical region for the one-tail test is CR ¼ {tcalc < –t, } if H1: < 0 and CR ¼ {tcalc > t,} if H1: > 0. As the variances are different, it seems reasonable to take the sample sizes, n1 and n2, also different. If 2 ¼ r 1, similar to Equation (52), one obtains n1 .
z=2 þ z
2
k2 ðrþ1Þ
ð54Þ
Once n1 is determined, n2 is obtained as n2 ¼ r n1. The computation of the sample sizes with different variances when pilot samples are at hand can be found in Schouten.32
1.02.3.5
Test Based on Intervals
The problem of deciding ‘the equality’ of the means of two distributions discussed in the previous section shows the fact that the result of interest (the two means are equal) is obtained by accepting the null hypothesis. Hence, the type II error becomes very important. To compute it, it is necessary to decide which is the least difference between the means that is to be detected, ¼ j1 2j. A more natural framework is to define the null and alternative hypotheses in such a way that the decision of accepting the equality of means is made by rejecting the null hypothesis, that is, the test should be posed as H0 : j1 – 2 j ðthe means are differentÞ H1 : j1 – 2 j < ðthe means are equalÞ
Quality of Analytical Measurements: Statistical Methods for Internal Validation
39
Contrary to the tests so far, the hypothesis of this test, called interval hypothesis, is not made by one point but an interval. The two one-sided tests (TOST) procedure consists of decomposing the interval hypotheses H0 and H1 into two sets of one-sided hypothesis: H01 : 1 – 2 – H11 : 1 – 2 > –
and H02 : 1 – 2 H12 : 1 – 2 <
The TOST procedure consists of rejecting the interval hypothesis H0 (and thus concluding equality of 1 and 2) if and only if both H01 and H02 are rejected at a chosen level of significance . If two normal distributions with the same unknown variance, 2, are supposed and two samples of size n1 and n2 are taken from each one, respectively, the two sets of one-sided hypothesis will be tested with ordinary onetail test (row 10 of Table 4). Thus, the critical region is 8 9 > > <ðx – x Þ – = – ð x – x Þ 1 2 1 2 qffiffiffiffiffiffiffiffiffiffiffiffi t; and qffiffiffiffiffiffiffiffiffiffiffiffi t; CR ¼ > > : sp n1 þ n1 ; sp n11 þ n12 1 2
ð55Þ
where sp2 is the pooled variance and ¼ n1 þ n2 2 its d.f. The TOST procedure turns out to be operationally identical to the procedure of declaring equality only if the usual confidence interval at 100 (1 2)% on 1 2 is completely contained in the interval [,]. The expression that relates the sample sizes with and is n1 .
z þ z=2
2 f 2
ð56Þ
where f ¼ (r þ 1)/r and n2 ¼ r n1. When the sample sizes are equal, f ¼ 2. As is unknown in Equation (56), it should be adapted as in Case 2 of Section 1.02.3.4. When comparing Equation (56) with those corresponding to the two-tail t-test on the difference of means, one observes that it is completely analogous by exchanging the two risks (see Equations (50) and (52)). That is, the significance level and the power of the t-test become the power and significance level, respectively, of the TOST procedure, which completely agrees with the exchange of the hypotheses. The tests based on intervals have a great tradition in statistics, see for example the book (very technical) by Lehmann33. The TOST procedure is a particular case that has also been used under the name bioequivalence test.34 Mehring35 has proposed some technical improvements to obtain optimal interval hypothesis tests, including equivalence testing. It is shown that TOST is always biased, in particular, the power tends to zero for increasing variances independently of the difference of means. As a result, an unbiased test36 and a suitable compromise between the most powerful test and the shape of its critical region37 have been proposed. In chemistry the use of TOST has been proposed to verify the equality of two procedures.38,39 Kuttatharmmakull et al.40 provide a detailed analysis of the sample sizes necessary in a TOST procedure to compare methods of measurement. There are different versions of TOST for ratio of variables and for proportions; the details of the equations for these cases can be consulted in Section 8.13 of the book by Martin Andre´s and Luna del Castillo41 and in the book by Wellek.42 The latter is a comprehensive review of inferential procedures that enable one to ‘prove the null hypothesis’ for many areas of applied statistical data analysis. 1.02.3.6
Hypothesis Test on the Variances of Two Normal Distributions
Suppose that two procedures follow normal distributions X1 and X2 with unknown means and variances. We wish to test the hypothesis on the equality of the two variances, that is, H0: 21 ¼ 22. In practice, this is a relevant problem because this hypothesis is related to the equality of the precision of the two procedures, and also as a
40 Quality of Analytical Measurements: Statistical Methods for Internal Validation
previous check to decide about the equality of variances before applying the test on the equality of means (Case 2 of Section 1.02.3.4) or to compute a confidence interval on the difference of means (Case 2 of Section 1.02.2.4). Assume that two random samples of size n1 of X1 and of size n2 of X2 are available and let s21 and s22 be the sample variances. To test the two-sided alternative, we use the statistic and CR of line 15 of Table 4. The probability can be computed as a function of the ratio of variances l 2 ¼ 21 =22 that is to be detected, by the equation ¼ pr
F=2;n1 – 1;n2 – 1 F1 – =2;n1 – 1;n2 – 1 < F < n – 1;n – 1 1 2 l2 l2
ð57Þ
where with n1 1 and n2 1 d.f. and F=2;n1 –1;n2 –1 its upper /2 point, so that
Fn1 –1;n2 –1 denotes an F-distribution pr Fn1 –1;n2 –1 > F=2;n1 –1;n2 –1 ¼ =2. Similarly, F1–=2;n1 –1;n2 –1 is the upper 1 – ð=2Þ point. Example 13: Aliquot samples have been analyzed in random order under the same experimental conditions to carry out a stability test. The results are given in Table 7 and must be compared for assessing test material stability. (1) Is there experimental evidence of instability in the material? (2) What is the probability of accepting the null hypothesis when it is in fact wrong? To answer these questions, the analyst considers that the material is not stable if the mean of the test sample differs from the mean of the control sample in 2 standard deviations. (3) What should be the sample size if 1 standard deviation is needed for fit for purpose of this analysis (and ¼ ¼ 0.05)? 1. As we only know the estimates of the variance, a t-test to compare means before and after the storage of the analyte has been carried out. The first step is to test if the variances can be considered equal by using a two-tail F-test: H0: 21 ¼ 22 H1: 21 6¼ 22
Following row 15 in Table 4, Fcalc ¼ s12 =s22 ¼ 50:76=26:22 ¼ 1:93 and CR ¼ Fcalc < F1 – =2;n1 – 1;n2 – 1 or Fcalc > F=2;n1 – 1;n2 – 1 g with F0.025,8,8 ¼ 4.43 and F1– 0:025;8;8 ¼ 1=F0:025;8;8 ¼ 1=4:43 ¼ 0:225 7. Hence, there is no experimental evidence to say that the variances differ. Therefore, a hypothesis t-test on the difference of the two means with equal variances is being formulated (Case 2 of Section 1.02.3.4). The statistic and the CR are given in line 9 of Table 4. H0: 1 ¼ 2 ðthe test material is stableÞ H1: 1 6¼ 2 ðthe material is not stableÞ
pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi 2 The ‘pooled’ variance, s , is 38.49, so s ¼ 6:20, with 9 þ 9 2 ¼ 16 d.f. t ¼ ð x – x Þ= sp 1=n1 þ 1=n2 p calc 1 2 p pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi ¼ ð45:41 – 43:46Þ= 6:20 1=9 þ 1=9 ¼ 0:67 and t0:025;16 ¼ 2:120. Therefore, the critical region is CR ¼ftcalc < – 2:120 or tcalc > 2:120g. Hence, there is no evidence to reject the null hypothesis, that is, with these data there is no experimental evidence of instability. 2. Power of the test: With the condition imposed by the analyst, Equation (52) with k ¼ 2 provides 0.05. 3. In this case, the analyst is interested in computing the sample size under the assumption that only 1 standard deviation is admissible for fit for purpose of this analysis. Therefore, k ¼ 1 and from Equation (52) n ¼ 25.99, so n1 ¼ n2 ¼ 26. The sample size is greater than the former because he/she is interested in distinguishing a quantity that is less than this in previous point 2. Table 7 Data for analysis of stability (arbitrary units) Control sample Test sample
46.31 43.12
See Example 13 for more details.
44.90 43.00
44.12 44.75
36.07 39.66
39.20 37.74
36.39 37.50
50.71 54.79
47.85 53.08
45.60 55.07
Quality of Analytical Measurements: Statistical Methods for Internal Validation
41
Incidentally, the null hypothesis of the F-test has been accepted. When a standard deviation twice that of the control samples is to be detected, Equation (57) gives a probability for this test of 0.56. That means that 56% of the times the null hypothesis will be wrongly accepted, and in this case, we have accepted the null hypothesis. When the F-test is used as a previous step to the one of equality of means, and this one will be used with ¼ 0.05, it is common to fix ¼ 0.10 for the F-test. Equation (57) would provide n ¼ n1 ¼ n2 ¼ 24 with ¼ 0.11, and by maintaining that a change of 2 times the standard deviation of the control samples is to be detected (all calculations of can be seen in Example A13 of Appendix). In general, the F-tests on the equality of variances are very conservative and large sample sizes are needed to assure an adequately small probability of type II error.
1.02.3.7
Hypothesis Test on the Comparison of Several Independent Variances
When the hypothesis of equality of variances of several groups of data coming from normal and independent distributions is to be tested, a good practice is to draw the data for a visual inspection of their dispersion. Example 14: Table 8 shows the results of the determination of the acetic degree by means of an acid–base titrimetry, employing sodium hydroxide as the titrant. These data are adapted from the practice ‘Analysis and comparison of the acetic grade of a vinegar’ included in Ortiz et al.,43 and each series is a replicated determination carried out by a group of students over the same vinegar sample. The means and variances obtained by each group are also included in Table 8. Figure 4 shows the plot of the results obtained by different groups of students. The most commonly used tests to compare several variances are the Cochran’s, the Bartlett’s, and the Levene’s tests. In all the cases, we wish testing the hypothesis H0 : 21 ¼ ¼ 2i ¼ ¼ 2k
ð58Þ
Ha : At least one 2i is different
P The sample size of each group is denoted as ni, i ¼ 1,2,. . .,k, and N ¼ ki¼1 ni . Case 1. Testing the equality of several variances according to Cochran’s test: The null hypothesis is that the variances within each of the k groups of data are the same. This test detects if one variance is greater than the rest. The statistic is max s 2 Gcalc ¼ Pk i 2 i¼1 si
ð59Þ
The critical region at significance level is given by
CR ¼ Gcalc > G;k;
ð60Þ
where G;k; is the value tabulated in Table 9 for d.f. In the case ni ¼ n for all i, is ¼ n1. Table 8 Determination of acetic degree of a vinegar by means of an acid–base titration
Mean xi Variance si2
Group 1
Group 2
Group 3
Group 4
Group 5
6.028 6.028 5.998 6.089 6.059
5.974 6.004 6.005 5.852 5.944
5.886 5.970 5.880 5.910 5.910
6.132 6.120 6.131 6.072 6.071
5.916 6.123 6.034 6.004 6.152
6.040 4 1.203 10–3
5.955 8 3.997 10–3
5.911 2 1.267 10–3
6.105 2 0.969 10–3
6.045 8 8.993 10–3
See Example 14 for more details.
42 Quality of Analytical Measurements: Statistical Methods for Internal Validation
Acetic degree
6.2 6.1 6.0 5.9 5.8 1
2
3 Group
4
5
Figure 4 Data for testing the equality of variances.
Table 9 Critical values for Cochran’s test for testing homogeneity of several variances at 5% significance level , degrees of freedom k
1
2
3
4
5
6
7
8
9
10
2 3 4 5 6 7 8 9 10
0.9985 0.9669 0.9065 0.8412 0.7808 0.7271 0.6798 0.6385 0.6020
0.9750 0.8709 0.7679 0.6838 0.6161 0.5612 0.5157 0.4775 0.4450
0.9392 0.7977 0.6841 0.5981 0.5321 0.4800 0.4377 0.4027 0.3733
0.9057 0.7457 0.6287 0.5441 0.4803 0.4307 0.3910 0.3584 0.3311
0.8772 0.7071 0.5895 0.5065 0.4447 0.3974 0.3595 0.3286 0.3029
0.8534 0.6771 0.5598 0.4783 0.4184 0.3726 0.3362 0.3067 0.2823
0.8332 0.6530 0.5365 0.4564 0.3980 0.3939 0.3185 0.2901 0.2666
0.8159 0.6333 0.5175 0.4387 0.3817 0.3384 0.3043 0.2768 0.2541
0.8010 0.6167 0.5017 0.4241 0.3682 0.3259 0.2926 0.2659 0.2439
0.7880 0.6025 0.4884 0.4118 0.3568 0.3154 0.2820 0.2568 0.2353
k, number of groups; , degrees of freedom. Adapted from Sachs, L. Applied Statistics. A Handbook of Techniques; New York: Springer-Verlag, 1982.
With the data of Example 14, Gcalc ¼ 8.993 10–3/16.429 10–3 ¼ 0.5474 and G0:05;5;5 – 1 ¼ 0:5441; thus, at 0.05 significance level, the null hypothesis should be rejected and the variance of group 5 should be considered different from the rest. Case 2. Bartlett’s test is appropriate to detect groups of similar variance within each of the k groups of data but that differs from one group to another. The statistic is defined using the following equations: w2calc ¼ 2:3026
q c
ð61Þ
k X
ðni – 1Þlog si2 q ¼ ðN – kÞlog sp2 –
ð62Þ
i¼1
hP c ¼1þ
k 1 i¼1 ni – 1
i
–
1 N –k
3ðk – 1Þ
ð63Þ
In Equation (62), ‘log’ meansh the decimal logarithm and sp2 is the pooled variance that, analogous to Equation i P k 2 (23), for k variances is sp2 ¼ i¼1 ðni – 1Þsi =ðN – k Þ. The critical region is n o CR ¼ w2calc > w2;k – 1
ð64Þ
In Example 14, c ¼ 1.10, q ¼ 3.43, and w2calc ¼ 7:19, which does not belong to the critical region defined in Equation (64) because w20:05;4 ¼ 9:49. Cochran’s and Bartlett’s tests are very sensitive to the normality assumption. The Levene’s test, particularly when it is based on the medians of each group, is more robust to the lack of normality of data.
Quality of Analytical Measurements: Statistical Methods for Internal Validation
43
Case 3. Levene’s test: Consider, in the jth group of replicates, the absolute deviations of the values xij from the mean of their repeats groups. lij ¼ xij – xi ; i ¼ 1; 2; . . . ; ni
ð65Þ
Consider the data arranged as in Table 8 and compute the usual F statistics for the deviations lij Pk Fcalc ¼
i¼1
Pk
j ¼1
2 ni ðli – lÞ k–1
Pni ðlij – li Þ2 i¼1
ð66Þ
N –k
where li is the mean of the ith group and lis the global mean. The critical region at 100 (1 )% confidence level is
CR ¼ Fcalc > F;k – 1;N – k
ð67Þ
Note that the numerator of Equation (66) is the pooled variance of the deviations and the denominator is the global variance of these deviations. Computing the differences in Equation (65) with the data of Table 8, Fcalc ¼ (2.205 10–3)/(0.905 10–3) ¼ 2.44. As F0.05,4,20 ¼ 2.866, there is no evidence to reject the null hypothesis (the variances are equal). The Levene’s test using group medians instead of means is more recommendable. The adaptation is simple; one has to consider the absolute value of the differences but to the median, x˜i , of each group lij ¼ xij ˜– x i ; i ¼ 1; 2; . . . ; ni
ð68Þ
The statistics is again that of Equation (66) and it is tested in the same way as before. With the same data of Table 8, but applying the transformation in Equation (68), one obtains Fcalc ¼ (2.146 10–3)/(1.360 10–3) ¼ 1.58, and the conclusion is the same. The variance of the five groups should be considered as equal. Frequently it happens that the three tests do not agree in the result, as is the case here. But the joint interpretation clarifies the situation: In the data of Example 14 , the variance of group 5 is greater than the variance of other groups. When the Levene’s test is applied, a large difference between both statistics is observed in the statistics when using the median. This makes one think that the increase in the variance of the last group is caused by some data being different from the others; this is graphically seen in Figure 4.
1.02.3.8
Goodness-of-Fit Tests: Normality Tests
The test on distribution or goodness-of-fit tests are designed to decide whether the experimental data are compatible with a predetermined probability distribution, generally characterized by one or several parameters, such as the normal, the Student’s t, or the F distribution. Almost all the inferential procedures proposed in this chapter are based on normality; thus, in most of the cases, it is necessary to verify whether the data are compatible with this assumption. In this section, we will show the chi-square test that is used for any distribution and the D’Agostino test that is recommendable for evaluating the normality of a data set. Case 1. Chi-square test: The test is designed to detect frequencies inadequate for a specified probability distribution F0. Given a sample x1,x2,. . .,xn from a random variable, one is interested in testing the hypothesis H0 : The distribution of the random variable is F0 H1 : This is not the case
ð69Þ
To compute the statistics, the n sample values are grouped into k classes (intervals). Denote by Oi, i ¼ 1,. . .,k, the frequency observed in each class and by Ei the expected frequency for the same class provided the distribution is exactly F0. Then, w2calc ¼
k X ðOi – Ei Þ2 i¼1
Ei
ð70Þ
44 Quality of Analytical Measurements: Statistical Methods for Internal Validation
The critical region at (1 ) 100% of confidence is n o CR ¼ w2calc > w2;k – p – 1
ð71Þ
n o where w2;k – p – 1 is the value such that pr w2k – p – 1 > w2;k – p – 1 ¼ , where p is a number that depends on the distribution F0, for instance p ¼ 2 for a normal, p ¼ 1 for a Poisson, and p ¼ 0 for a uniform distribution. The test requires that the expected frequencies are not too small. Therefore, the data are regrouped into bigger classes. In the practice of chemical analysis, the sample sizes are not large and when grouping the data the d.f. of the chisquare statistics are few, the critical value of Equation (71) becomes large, and it is necessary to have a large discrepancy between the estimated and observed frequencies to reject the null hypothesis. That means that the test is very conservative. Example 15: To show the validity of the use of the crystal violet (CV) as an internal standard in the determination by HPLC-MS-MS of malachite green (MG) in trout, a sample of trout was spiked with increasing concentrations of MG between 0.5 and 5.0 mg kg–1 and in all of them with 1 mg kg–1 of CV. The area of the CV-specific peak (transition 372 > 356) resulted: 1326, 1384, 1419, 1464, 1425, 1409, 1387, 1449, 1311, 1338, 1350, and 1345 abundance per count. To verify whether the CV is constant and independent of the concentration of MG, we can test the hypothesis H0 : The distribution of the random variable is uniform H1 : This is not the case
Table 10 shows the calculation of both observed and expected frequencies under the uniform distribution in the interval [1311,1464], the endpoints being respectively the minimum and maximum values in the sample. By summing up the values of the last column of Table 10, the statistics is w2calc ¼ 0:51, which does not belong to the critical region because it is not greater than w20:05;5–0–1 ¼ 9:49. Therefore, there is no evidence to reject the hypothesis that the data come from a uniform distribution. Case 2. Normality D’Agostino: The problem of verifying the normality of a set of data has been extensively treated. When the empirical and theoretical histograms are compared, the most commonly used tests are those of the chi-square and the Kolmogoroff–Smirnov. However, there are many characteristics that are specific for the pdf of a normal; for example, the skewness and the kurtosis, which are statistics related to the higher than two order moments for the normal pdf. A very powerful test is that of D’Agostino. H0 : The distribution of the random variable is normal H1 : This is not the case
To apply the test, the data are sorted in increasing order, so that x1 x2 xn. The statistic is n þ 1 Xn ixi – x i¼1 i 2 ffi ¼ rffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi P
Pn
2 n 2 – n n x x =n i i¼1 i i¼1 Pn
Dcalc
i¼1
ð72Þ
Table 10 w2 test for uniform distribution applied to asses the validity of crystal violet as internal standard; data of Example 15 Class
Observed frequency (Oi)
Expected frequency (Ei)
ðOi – Ei Þ2 Ei
[1311, 1341.6) [1341.6, 1372.2) [1372.2, 1402.8) [1402.8, 1433.4) [1433.4, 1464)
3 2 2 3 2
2.40 2.40 2.40 2.40 2.40
0.15 0.07 0.07 0.15 0.07
Quality of Analytical Measurements: Statistical Methods for Internal Validation
45
Table 11 Significance limits for the D’Agostino normality test Significance level Sample size
¼ 0.05
10 12 14 16 18 20 22 24 26 28 30
DL 0.2513 0.2544 0.2568 0.2587 0.2603 0.2617 0.2629 0.2639 0.2647 0.2655 0.2662
¼ 0.01 DU 0.2849 0.2854 0.2858 0.2860 0.2862 0.2863 0.2864 0.2865 0.2866 0.2866 0.2866
DL 0.2379 0.2420 0.2455 0.2482 0.2505 0.2525 0.2542 0.2557 0.2570 0.2581 0.2592
DU 0.2857 0.2862 0.2865 0.2867 0.2868 0.2869 0.2870 0.2871 0.2872 0.2873 0.2873
Adapted from Martı´n Andre´s, A.; Luna del Castillo, J. D. Bioestadı´stica para las ciencias de la salud; Spain: Norma Capitel Madrid, 2004.
Index i in Equation (72) refers to the ordered data. Table 11 shows some of the critical values of the statistics D,n with the two values, DL,n and DU,n, for each sample size n and significance level . The critical region of the test is
CR ¼ Dcalc < DL;n or Dcalc > DU;n
ð73Þ
For further details, consult the work by D’Agostino and Stephens.44 As with the confidence intervals, a Bayesian approach exists for the construction of the hypothesis tests that many statisticians prefer because of its internal coherence. Consult Chapter 1.08 and the basic references on this subject; for a recent comparative analysis of both approaches, see Moreno and Giro´n.45
1.02.4 One-Way Analysis of Variance Sometimes, more than two means must be compared. One can think in comparing, say, five means, by applying the test of comparison of two means of Section 1.02.3.4 to each of the 10 pairs of means that can be formed taking them from two in two. This option has the serious problem of the requirement of enormous sample sizes, because to test the null hypothesis ‘the five means are equal’ with ¼ 0.05 and assuming that the 10 tests are independent, each one of the hypothesis ‘the means xi and xj are equal’ should be tested with a significance level of 0.0051 to obtain (1 0.0051)10 0.95. The appropriate procedure for testing the equality of several means is the analysis of variance (ANOVA). The ANOVA has many more applications; it is particularly useful in the validation of a model fit to some experimental data and, hence, in an analytical calibration (Chapter 1.05), or in the analysis of response surfaces (Chapter 1.12). Table 12 shows how the data are usually arranged in a general case, with a factor to k levels (e.g., five different extraction cartridges) and in each one n data (e.g., four determinations with each cartridge). Each of the N ¼ k n values xij (i ¼ 1,2,. . .,k, j ¼ 1,2,. . .,n) is the result obtained when using the ith cartridge withPthe jth aliquot sample. In general, in each level i, a different number of replicates are available ni, with N ¼ ki¼1 ni . To make the notation easier, we will suppose that all ni are equal, that is, ni ¼ n for each level. Suppose that the data in Table 12 can be described by the model. xij ¼ þ i þ "ij
with i¼ 1; 2; . . . ;k; j ¼ 1; 2; . . . ;n
ð74Þ
46 Quality of Analytical Measurements: Statistical Methods for Internal Validation Table 12 Arrangement of data for an ANOVA Factor Level 1
Level 2
Level 3
x11 x12 x13 .. . x1n
x21 x22 x23 .. . x2n
x31 x32 x33 .. . x3n
Level k .. .
xk1 xk2 xk3 .. . xkn
where is a parameter common to all treatments, called the overall mean, i is a parameter associated with the ith level, called the factor effect, and "ij is the random error component. In our example, is the content of the sample and i is the variation in this quantity caused by the use of the ith cartridge. Note that in the model of Equation (74), the factor effect is additive; this is an assumption that may be unacceptable in some practical situations. We would like to test certain hypotheses about the treatment effects and to estimate them. For the hypothesis testing the model errors are assumed to be normal and independently distributed random variables, with mean zero and variance 2, NID(0,2). The variance 2 is assumed to be constant for all levels of the factor. The model of Equation (74) is called the one-way ANOVA, because only one factor is studied. The analysis for two or more factors can be seen in the Chapter 1.12 about factorial techniques. Furthermore, the data of Table 12 are required to be obtained in random order so that the environment in which the factor varies is as uniform as possible. There are two ways for choosing the k factor levels in the experiment. In the first case, the k levels are specifically chosen by the researcher, as the cartridges in our example. In this case, we wish to test the hypothesis about the size of i, and conclusions will apply only to the factor levels considered in the analysis and they cannot be extended to similar levels that were not considered. This is called the ‘fixed effects model’. Alternatively, the k levels could be a random sample from a larger population of levels. In this case, we would like to be able to extend the conclusions based in the sample to all levels in the population, independent of whether they have been explicitly considered in the analysis or not. Here i are random variables and information about the specific values included in the analysis is useless. Instead, we test the hypothesis about the variability. This is called the ‘random effects model’. This model is used to evaluate the repeatability and reproducibility of a method and also the laboratory bias when the method of analysis is being tested by a proficiency test. In the same experiment, and provided there are at least two factors, fixed and random effects can simultaneously appear.46,47 1.02.4.1
The Fixed Effects Model
In this model, the effect of the factors is defined as a deviation with respect to the overall mean, so that k X
i ¼ 0
ð75Þ
i¼1
From the individual data, the mean value per level is defined as Pn xi ¼
j ¼1 xij
n
; i ¼ 1; 2; . . . ; k
ð76Þ
and the overall mean is Pk x ¼
Pn
j ¼1 xij
i¼1
N
ð77Þ
Quality of Analytical Measurements: Statistical Methods for Internal Validation
47
A simple calculation gives k X n X
k k X n X X
2
2 xij – x ¼ n ðxi – xÞ2 þ xij – xi
i¼1 j ¼1
ð78Þ
i¼1 j ¼1
i¼1
Equation (78) shows that the total variability of the data, measured by the sum of squares of the difference of each datum and the overall mean, can be partitioned into a sum of squares of differences between level means and the overall mean and a sum of squares of differences of individual values and their level mean. The term
2 P P P n ki¼1 ðxi – xÞ2 measures the differences between levels, whereas ki¼1 nj¼1 xij – xi can be due to random error alone. It is common to write Equation (78) as SST ¼ SSF þ SSE
ð79Þ
where SST is the total sum of squares, SSF is the sum of squares due to change levels of factor, which is called sum of squares between levels, and SSE is the sum of squares due to error, which is called sum of squares within levels. There are N individual values, thus SST has N 1 d.f. Similarly, as there are k factor levels, SSF has k 1 d.f. Finally, SSE has N k d.f. We are interested in testing H0 : 1 ¼ 2 ¼ 3 ¼ ¼ k ¼ 0 ðthere is no effect due to the factorÞ H1: i 6¼ 0 for at least one i
Because of the assumption that the errors "ij are NID(0,2), the values xij are NID( þ i,2), and therefore SST/2 is distributed as a w2N – 1 . The Cochran’s theorem guarantees that, under the null hypothesis, SSF/2 and SSE/2 are independent chi-square distributions with k 1 and N k d.f., respectively. Therefore, under the null hypothesis, the statistic Fcalc ¼
SSF k–1 SSE N –k
¼
MSF MSE
ð80Þ
46 follows an Fk–1,N–k distribution, whereas under the alternative hypothesis, it follows aP noncentral F. The k 2 quantities MSF and MSE are called mean squares. Their means are E ðMSF Þ ¼ 2 þ n i¼1 i =ðk – 1Þ and 2 E ðMSE Þ ¼ , respectively. Therefore, under the null hypothesis, both are unbiased estimates of the residual variance, 2, whereas under the alternative hypothesis, the expected value of MSF is greater than 2. Therefore, if the null hypothesis is false, the numerator is significantly greater than the denominator of Equation (80) and the critical region at significance level is
CR ¼ Fcalc > F;k – 1;N – k
ð81Þ
Usually, the test procedure is summarized in a table, called ANOVA table (Table 13). Example 16: To investigate the influence of fibers composition on an SPME procedure, an experiment was performed using five different fibers. The data shown in Table 14 are the results (after extraction) of four replicated analyses, with each fiber, of a sample spiked with 1000 ppb of triazine. All the analyses were carried out in random order and maintaining the rest of experimental conditions controlled. Table 13 Skeleton of an ANOVA of fixed effects Source of variation
Sum of squares
d.f.
Mean squares
E(MS)
Fcalc
Factor (between levels)
SSF
k–1
MSF
2
MSF MSE
Error (within levels)
SSE
N–k
MSE
Total
SST
N–1
2 þ
n
Pk
2 i¼1 i
k–1
48 Quality of Analytical Measurements: Statistical Methods for Internal Validation Table 14 Effect of the type of fiber in a solid-phase microextraction of triazine (ppb) Type of fiber
Replicates
Means xi Variances si2
Fiber 1
Fiber 2
Fiber 3
Fiber 4
Fiber 5
490 478 492 499
612 609 599 589
509 496 489 500
620 601 580 603
490 502 495 479
489.75 76.25
602.25 108.92
498.50 69.67
601.00 268.67
491.50 93.67
In the last two rows of Table 14, the means and variances for each fiber are given. Before doing the ANOVA, the hypothesis of equality of variances should be tested: H0: 21 ¼ 22 ¼ ¼ 2k H1 : At least one 2i is different
With the variances in Table 14, the statistics of the Cochran’s test (Equation (59)) is Gcalc ¼ 268.67/ 617.168 ¼ 0.435. As G0.05,k,n1 ¼ 0.5981 (see Table 9), the statistic does not belong to the critical region (Equation (60)) and there is no evidence to reject the null hypothesis at 5% significance level. The statistics of the Bartlett’s test is w2calc¼1.792 (Equation (61)) and the critical value is w20.05,4¼9.488, so there is no evidence to reject the null hypothesis either (Equation (64)). The same happens with the Levene’s test; computing the absolute values, according to Equation (65), of the data of Table 14, Fcalc ¼ 14.70/44.01 ¼ 0.33. As F0.05,4,15 ¼ 3.06, there is no evidence to reject the null hypothesis on the equality of variances. By using the median instead of the mean (Equation (68)), Fcalc ¼ 15.13/44.23 ¼ 0.34, and the conclusion is the same. From the analysis of the equality of variances, we can conclude that the variances of the five levels should be considered as equal. The ANOVA of the experimental data gives the results in Table 15. Considering the critical region defined in Equation (81), as Fcalc ¼ 114.54 is greater than the critical value F0.05,4,15 ¼ 3.06, we reject the null hypothesis and hence the conclusion is that the effect of the factor ‘fiber composition over extraction’ is significant at 0.05 level. 1.02.4.2
Power of the ANOVA for the Fixed Effects Model
The power of the ANOVA is computed by the following expression: n o 1 – ¼ pr Fk – 1;N – k; > F;k – 1;N – k
ð82Þ
where F;k – 1;N – k is the critical value of Equation (81), Fk – 1;N – k; is a noncentral F distribution with k 1 and N k d.f. of the numerator and denominator, respectively, and is the noncentrality parameter, whose value is given by Pk ¼n
2 i¼1 i 2
ð83Þ
Table 15 Results of the ANOVA for data of Table 14 Source
Sum of squares
d.f.
Mean squares
Fcalc
Between fibers Error (within fibers) Total
56 551.3 1 851.5 58 402.8
4 15 19
14 137.8 123.4
114.54
Quality of Analytical Measurements: Statistical Methods for Internal Validation
49
The noncentrality parameter P depends on the number of replicates and also on the difference in means that we wish to detectP in terms of ki¼1 i2 . Furthermore, the error variance is usually unknown. In such cases, we must choose ratios ki¼1 i2 =2 that we wish to detect. As the power, 1 , of the test increases with , we next ask what is the minimum subject to the condition that two of the i differ by or more. The minimum is obtained if two of the i differ by and the remaining k 2 equal the mean of these two;46 therefore, k X
i2 ¼
i¼1
2 2
ð84Þ
For example, with the data of Example 16 (Table 14), we are now interested in the risk to affirm that the type of composition fiber is not significant for the recovery. The answer consists of evaluating the probability by Equation (82). Suppose that we want to discriminate P effects greater than twice the MSE, that is, ki¼1 i2 =2 2, so ¼ n 2 ¼ 8, F0:05;4;15 ¼ 3:06 and ¼ 0.54 (calculations can be seen in Example A14 of Appendix). To put in words, 54 out of 100 times we will accept the null hypothesis (there is no effect of fiber) when it is wrong. This is not good enough for a useful decision rule. According to Equation (84), our value means that we want to discriminate a difference at least equal to 2 between the two types of fiber. Another question related to the above equation is to know the sample size before starting an experiment. In many situations, we would like to know what the sample size should be to assess whether both risks and will be good enough. Also, the following question can be answered: How P many replicates should be carried out in the experiment for ¼ ¼ 0.05 maintaining the approximation ki¼1 i2 =2 3? Note that, in this case, the analyst considers ‘effect of fiber type’ if it is greater than 3 times p MS ffiffiffi E, which is equivalent, using Equation (84), to detect a difference between two fibers at least equal to ¼ 6 2:5. To calculate the sample size, a table must be made to write as a function of n in Equation (82) with k, , and fixed at 5, 0.05, and 3 n, respectively. Following the results shown in Table 16, we need n ¼ 8 replicates with each fiber to achieve 0.05, but in practice n ¼ 7 suffices. 1.02.4.3
Uncertainty and Testing of the Estimated Parameters for the Fixed Effects Model
It is possible to derive estimators for the parameters and i (i ¼ 1,. . .,k) in the one-way ANOVA modeled by Equation (74). The normality assumption on the errors is not needed to obtain an estimate by least squares; however, the solution is not unique, so the constraint of Equation (75) is imposed. Using this constraint we obtain the estimates ˆ ¼ x and ˆi ¼ xi – x; i ¼ 1; . . . ; k
ð85Þ
where x and xi have been defined in Equations (77) and (76), respectively. If the number of replicates, nP i, in each level is not equal (unbalanced ANOVA), then the constraint of Equation (75) should be changed by ki¼1 ni i ¼ 0 and use the weighted average of xi instead of the unweighted average in Equation (85). Now, if we assume that the errors are NID(0,2) and ni ¼ n, i ¼ 1,. . .,k, the estimates of Equation (85) are also the maximum likelihood ones. For unbalanced designs, the least squares solution is biased and the maximum likelihood is better. The reader interested in this subject should consult statistical monographs that describe this matter at a high level, such as Milliken and Johnson48 and Searle.49 The mean of the ith level is i ¼ þ i, i ¼ 1,. . .,k. In our case, with a balanced design an estimator of i would be ˆ i ¼ ˆ þ ˆi ¼ xi and, as errors are NID(0,2), then xi is NID(i,2/n). Using MSE as an estimator of 2, Equation (16) gives the confidence interval at (1 ) 100% level: "
rffiffiffiffiffiffiffiffiffiffi rffiffiffiffiffiffiffiffiffiffi# MSE MSE ; xi þ t =2;N – k xi – t =2;N – k n n
ð86Þ
Table 16 Probability of type II error, , as a function of the number n of replicates in the ANOVA of the fiber types n
4 0.347
5 0.203
6 0.111
7 0.058
8 0.029
9 0.014
50 Quality of Analytical Measurements: Statistical Methods for Internal Validation
A (1 ) 100% confidence interval on the difference in any two level means, say i – j , would be "
xi – xj – t=2;N – k
rffiffiffiffiffiffiffiffiffiffiffiffi rffiffiffiffiffiffiffiffiffiffiffiffi#
2MSE 2MSE ; xi – xj þ t=2;N – k n n
ð87Þ
With the data in Example 16 (Tablep 14), a 95% confidence interval on the difference between fibers 1 and 2 is ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi ffi given by [(489.75 602.25) 2.131 2 123:43=4] ¼ [–129.24,–95.76]. Finally, the (1 ) 100% confidence interval on the global mean is "
rffiffiffiffiffiffiffiffiffiffiffi rffiffiffiffiffiffiffiffiffiffiffi# MSE MSE ; x þ t =2;N – k x – t =2;N – k nk nk
ð88Þ
Rejecting the null hypothesis in the fixed effect model of the ANOVA implies that there are differences between the k levels, but the exact nature of the differences is not specified. To treat this question, two procedures are used: the orthogonal contrasts and the multiple test. Case 1. Orthogonal contrasts: For example, with the data of Table 14, we would like to test the hypothesis H0: 4 ¼ 5. The linear relation related with this hypothesis is x4 – x5 ¼ 0; this linear combination is called a contrast. A contrast is tested by comparing its sum of squares to the mean square error. The resulting statistic would be distributed as an F, with 1 and N k d.f. Each contrast is defined by the coefficients of the linear combination, P in the previous case (0,0,0,1,–1). Two contrasts C ¼ (c1,c2,. . .,ck) and D ¼ (d1,d2,. . .,dk) are orthogonal if ki¼1 ci di ¼ 0. There are many ways to choose the orthogonal contrast coefficients for a set of levels. Usually, something in the experiment should suggest which comparison will be of interest. In order to show the procedure, we are going to raise with the data of Table 14 a fictitious case of orthogonal contrasts with purely didactic purpose; in each problem, its peculiarities and the previous knowledge of the analyst will suggest the contrast to be studied. The comparisons between the means per fiber and the associated orthogonal contrasts proposed are H0 : 4 ¼ 5
C1 ¼ – x4 þ x5
H0 : 1 þ 3 ¼ 4 þ 5
C2 ¼ x1 þ x3 – x4 – x5
H0 : 1 ¼ 3
C3 ¼ x1 – x3
H0 : 42 ¼ 1 þ 3 þ 4 þ 5
C4 ¼ – x1 þ 4 x 2 – x3 – x4 – x5
The sum of squares associated with each contrast C is SSC ¼
n
P
k i i¼1 ci x
2 ð89Þ
Pk
2 i¼1 ci
For example, SSC1 ¼ 4 (–1 601.00 þ 1 491.50)2/2 ¼ 23 981 with 1 d.f. These sums of squares are incorporated into the ANOVA table as shown in Table 17. Now, to test each of the hypotheses written in the table, it suffices to compare the corresponding Fcalc with the critical value F0.05,1,15 ¼ 4.54. Except for C3, the rest of the contrasts are significant according to Equation (81). Thus, we should reject the hypothesis that fibers 4 and 5 give the same recovery. The hypothesis that the mean of the Table 17 ANOVA table with orthogonal contrasts for composition of fibers for SPME Source
Sum of squares
d.f.
Mean squares
Fcalc
Between fibers C1: 4 ¼ 5 C2: 1 þ 3 ¼ 4 þ 5 C3: 1 ¼ 3 C4: 42 ¼ 1 þ 3 þ 4 þ 5 Error (within fibers) Total
56 551.3 23 981.0 10 668.0 153.1 21 550.0 1 851.5 58 402.8
4 1 1 1 1 15 19
14 137.8 23 981.0 10 668.0 153.1 21 550.0 123.4
114.54 194.28 86.43 1.24 174.59
Quality of Analytical Measurements: Statistical Methods for Internal Validation
51
recovery for fibers 1 and 3 is the same as for fibers 4 and 5 is also rejected. Also, fiber 2 differs significantly from the mean of the other four, whereas there is no experimental evidence to reject that fibers 1 and 3 provide the same recovery. It is evident that the possibilities of the analysis of the experimental results are very ample. Case 2. Comparison of several means: Many different methods have been described that were specifically designed for the comparison of several means. Here, we will describe the method of Newman–Keuls. The hypothesis test is the following: H0 : All the differences two by two are equal to zero H1 : At least one difference is nonnull
The procedure consists of the following steps: 1. To sort the means per level, xi , i ¼ 1,2,. . .,k, in decreasing order, xr ð1Þ xr ð2Þ xr ðkÞ . The subindex r(i) refers to the rank of the corresponding mean, that is, the position that it occupies in the ordered list. For example, the means of Table 14 have the following ranks: r(1) ¼ 2, r(2) ¼ 4, r(3) ¼ 3, r(4) ¼ 5, and r(5) ¼ 1, respectively. That means that the first one, which is 489.75, has rank 5, that is, is in the fifth position in the decreasing ordered list. Table 18 shows the ordered means and the ranks in the second column. 2. To create a table with the differences between the means from greatest to lowest being t equal to the difference of ranks plus one. Table 19 contains all the possible contrasts two by two of the means, grouped in each column by the number, t (number of means that separates them plus one), in the ordered list. We wish to test each of the following hypotheses: H0 : xrðiÞ – xrðiþk – 1Þ ¼ 0 H1 : xrðiÞ – xrðiþk – 1Þ > 0
The statistic is Rt ¼ q ðt; kðn – 1ÞÞ
rffiffiffiffiffiffiffiffiffiffi MSE n
ð90Þ
The value of the statistic depends, as always, on the significance level , on t, and on the d.f. N-k of MSE. Further, the first term in Rt changes with the difference of ranks, t. The corresponding values are written in the last row in Table 19 and some of their tabulated values are in Table 20. Table 18 Newman–Keuls procedure for multiple comparison test; data of SPME fibers Levels
Rank
Mean
Homogeneous groups
2 4 3 5 1
1 2 3 4 5
602.25 601.00 498.50 491.50 489.75
Table 19 Skeleton for using the corresponding tabulated values for the Newman– Keuls procedure t: difference of ranks plus one k
k–1
...
2
xr ð1Þ – xr ðkÞ
xr ð1Þ – xr ðk – 1Þ xr ð2Þ – xr ðkÞ
... ... .. .
xr ð1Þ – xr ð2Þ xr ð2Þ – xr ð3Þ ... xr ðk – 1Þ – xr ðkÞ
q ðk; kðn – 1ÞÞ
q ðk – 1; kðn – 1ÞÞ
...
q ð2; kðn – 1ÞÞ
The subscript r(i) indicates the ith rank. k is the number of levels in the ANOVA and q are the tabulated values at significance level .
52 Quality of Analytical Measurements: Statistical Methods for Internal Validation Table 20 Values of q(t,), the upper percentage points of the Studentized range for ¼ 0.05 t
2
3
4
5
6
7
8
9
10
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
17.969 6.085 4.501 3.926 3.635 3.460 3.344 3.261 3.199 3.151 3.113 3.081 3.055 3.033 3.014 2.998 2.984 2.971 2.960 2.950
26.98 8.33 5.91 5.04 4.60 4.34 4.16 4.04 3.95 3.88 3.82 3.77 3.73 3.70 3.67 3.65 3.63 3.61 3.59 3.58
32.82 9.80 6.82 5.76 5.22 4.90 4.68 4.53 4.41 4.33 4.26 4.20 4.15 4.11 4.08 4.05 4.02 4.00 3.98 3.96
37.08 10.88 7.50 6.29 5.67 5.30 5.06 4.89 4.76 4.66 4.57 4.51 4.45 4.41 4.37 4.33 4.30 4.28 4.25 4.23
40.41 11.74 8.04 6.71 6.03 5.63 5.36 5.17 5.02 4.91 4.82 4.75 4.69 4.64 4.59 4.56 4.52 4.49 4.47 4.45
43.12 12.44 8.48 7.05 6.33 5.90 5.61 5.40 5.24 5.12 5.03 4.95 4.88 4.83 4.78 4.74 4.70 4.67 4.65 4.62
45.50 13.03 8.85 7.35 6.58 6.12 5.82 5.60 5.43 5.30 5.20 5.12 5.05 4.99 4.94 4.90 4.86 4.82 4.79 4.77
47.36 13.54 9.18 7.60 6.80 6.32 6.00 5.77 5.59 5.46 5.35 5.27 5.19 5.13 5.08 5.03 4.99 4.96 4.92 4.90
49.07 13.99 9.46 7.83 6.99 6.49 6.16 5.92 5.74 5.60 5.49 5.39 5.32 5.25 5.20 5.15 5.11 5.07 5.04 5.01
t is the difference of ranks plus one. is the degrees of freedom of MSE. Adapted from Sachs, L. Applied Statistics. A Handbook of Techniques; New York: Springer-Verlag, 1982.
The values of q ðt ; kðn – 1ÞÞ are tabulated. Table 20 shows some of them. The critical region is:
CR ¼ xrðiÞ – xrðiþk – 1Þ Rt
ð91Þ
The results obtained when applying the method of Newman–Keuls to the data of Example 16 are given in Table 21. The first column contains the means to be compared, for example, 1–2 indicates that the comparison is between x1 and x2 ; the second column contains the differences (without sign) between them; the values of t (difference of ranks plus one) are in the third column, as x1 has rank 5 and the rank of x2 is 1 (the ranks are in Table 18), the value of t is 5. With the value of q (Table 20) and Equation (90), the critical value (Rt) is computed for each of the tests. The critical value Rt allows the analyst to decide whether the estimated difference is significant or not. The decision of rejecting or not rejecting the null hypothesis is shown in the last column of Table 21.
Table 21 Results of the Newman–Keuls test to data of SPME fibers Contrast levels
Differences xi – xj
t
q0.05,t,15
Critical values, Rt
CR ¼ xr ðiÞ – xr ðiþk – 1Þ Rt
1–2 1–3 1–4 1–5 2–3 2–4 2–5 3–4 3–5 4–5
112.5 8.75 111.25 1.75 103.75 1.25 110.75 102.5 7.0 109.5
5 3 4 2 3 2 4 2 2 3
4.37 3.67 4.08 3.01 3.67 3.01 4.08 3.01 3.01 3.67
4.37 5.555 ¼ 24.27 3.67 5.555 ¼ 20.39 4.08 5.555 ¼ 22.66 3.01 5.555 ¼ 16.72 3.67 5.555 ¼ 20.39 3.01 5.555 ¼ 16.72 4.08 5.555 ¼ 22.66 3.01 5.555 ¼ 16.72 3.01 5.555 ¼ 16.72 3.67 5.555 ¼ 20.39
Reject H0 No evidence to reject H0 Reject H0 No evidence to reject H0 Reject H0 No evidence to reject H0 Reject H0 Reject H0 No evidence to reject H0 Reject H0
H0: the difference is null, x¯ix¯j ¼ 0; H1: x¯ix¯j 6¼ 0.
Quality of Analytical Measurements: Statistical Methods for Internal Validation
53
Usually, the result of this multiple comparison is presented as in the last column of Table 18. The symbols ‘’ aligned indicate that the corresponding means are all equal two by two. For example, the means x2 and x4 by one side and any other pair among x1 , x3 , and x5 on the other side, as can be seen in Table 21. It is possible to conclude that there are two fiber groups; as far as the recovery is concerned, fibers 2 and 4 provide results that are significantly equal and greater than the others. The other group is made by the other three that are similar to each other but different from the two previous ones. 1.02.4.4
The Random Effects Model
In many cases, the factor of interest is a random variable as well, so that the chosen levels are in fact a sample of this random variable and we want to extract conclusions about the population from which the sample comes. For example, in the case of validating an analytical method, several laboratories will apply it to aliquot samples so that it is possible to decide what part of the variability of the results is attributable to the change of laboratory and what to the repetition of the procedure inside the same laboratory. These are the concepts of reproducibility and repeatability. The same happens in the analytical control of processes: It is necessary to distribute the variability observed between the one due to the measurement procedure and the one assignable to the process. The linear statistical model is xij ¼ þ i þ "ij
with i ¼ 1; 2; . . . ; k; j ¼ 1; 2; . . . ; n
ð92Þ
where i and "ij are independent random variables. Note that the model is identical in structure to the fixed effect case (Equation (74)), but the parameters have a different interpretation. If the V( i) ¼ 2 , then the variance of any observation is
V xij ¼ 2 þ 2
ð93Þ
The variances of Equation (93) are called variance components, and the model, Equation (92), is called components of variance or the random effects model. To test hypotheses in this model, we require that the "ij are NID(0,2), all of the i are NID(0,2 ), and i and "ij are independent. The sum of squares equality SST ¼ SSF þ SSE still holds. However, instead of testing the hypothesis about individual levels effects, we test the hypothesis H0 : 2 ¼ 0 H1 : 2 > 0
If 2 ¼ 0, all levels are identical; if 2 > 0, then there is variability between levels. Thus, under the null hypothesis, the ratio Fcalc
SSF MSF k ¼ –1 ¼ SSE MSE N –k
ð94Þ
is distributed as an F with k 1 and N k d.f. The expected means of MSF and MSE are E ðMSF Þ ¼ 2 þ n2
ð95Þ
E ðMSE Þ ¼ 2
ð96Þ
CR ¼ Fcalc > F;k – 1;N – k
ð97Þ
and
Therefore, the critical region is
1.02.4.5
Power of the ANOVA for the Random Effects Model
The power of the random effects ANOVA is obtained from F;k – 1;N – k 1 – ¼ pr Fk – 1;N – k > 2
ð98Þ
54 Quality of Analytical Measurements: Statistical Methods for Internal Validation 2
where 2 ¼ 1 þ n 2 . As 2 is usually unknown, we may either use a prior estimate or define the value of 2 that we are interested in detecting in terms of the ratio 2 =2 . An application to determine the number of replicates in a proficiency test can be seen in Example A15 of Appendix. 1.02.4.6
Confidence Intervals of the Estimated Parameters for the Random Effects Model
In general, the mean value per level xi does not have more statistical meaning than being a sample of the random factor. But sometimes, as in the case of the proficiency tests, this mean value is of interest for each laboratory participant. The variance of the mean value per level is theoretically equal to V ðxi Þ ¼ 2 þ 2 =n. From Equations (95) and (96), MSF =n (with k – 1 d.f.) estimates the variance of the mean per level. As a consequence, the 100 (1 )% confidence interval is "
rffiffiffiffiffiffiffiffiffiffi rffiffiffiffiffiffiffiffiffiffi# MSF MSF ; xi þ t =2;k – 1 xi – t =2;k – 1 n n
ð99Þ
When calculating the variance for the global mean, it is necessary to consider the variability contributed by the factor, as the factor always acts/. For example, when evaluating an analytical method, the results without the variability attributable to the factor laboratory are not conceivable. The variance of the overall mean is P V ðxÞ ¼ ki¼1 V ðxi Þ=k2 , which is estimated by MSF =nk, with k 1 d.f., in such a way that the 100 (1 )% confidence interval is " x – t =2;k – 1
rffiffiffiffiffiffiffiffiffiffi rffiffiffiffiffiffiffiffiffiffi# MSF MSF ; x þ t =2;k – 1 nk nk
ð100Þ
The ANOVA of random effects is a model of practical interest because it allows attributing real meaning to many statements that seem evident. For example, when in a proficiency test the samples are distributed to laboratories, it is insisted that they must be homogeneous. Strictly speaking, in most of the occasions, it is impossible to assure homogeneity, but it is enough that the variability attributable to the change of sample is significantly smaller than the one attributable to the procedure of analysis. This can be guaranteed by means of an ANOVA of random effects.
1.02.5 Statistical Inference and Validation 1.02.5.1
Trueness
The trueness is a key concept; several international organizations are unifying its definition. For example, the definition ‘‘The closeness of agreement between the average value obtained from a large series of test results and an accepted reference value’’ has been adopted by the IUPAC (Incze´dy et al.,11 Chapter 18). The definition of the ISO7 exactly coincides with it, and it is the definition accepted by the European Union in the Decision 2002/657/EC3 as far as the operation of the analytical methods and the interpretation of results are concerned. The trueness is usually expressed in terms of bias, which combines all the components of the systematic error, denoted by in Equation (1). The decisions on the trueness of a method are problems of hypothesis testing on the central value of a distribution; in case the random error can be assumed to have a zero mean, they are in fact tests on the mean because, according to Equation (1), the expected mean value for a series of measurements will be þ and the problem is reduced to test whether is zero or not (equivalently, to test whether x is significantly equal to ). To use one or another test depends only on the information available about the distribution of the random error – its type (normal, parametric, or unknown) and, in the case of normality, whether the variance is known or not. Some common cases are given below: 1. To decide whether an analytical procedure fulfills trueness by the use of a reference sample whose value is assumed to be true. If normal data with known variance 2 are supposed, then the tests of Section 1.02.3.2 will be used.
Quality of Analytical Measurements: Statistical Methods for Internal Validation
55
2. To decide whether an analytical procedure has bias specifically positive (or negative) by the use of a reference sample whose value is assumed to be true. If normal data are assumed, the one-tail test versions of Cases 1 and 2 of Section 1.02.3.2 will be of use. When a hypothesis test is to be posed, one can think about omitting some known data, for example the variance. The effect is a loss of power, that is, with the same value of and the same sample size, the probability of type II error will be greater. Said otherwise, if it is desired to have the same power, larger sample sizes are needed to acquire the same experimental evidence; a calculation on this matter is in Case 2 of Section 1.02.3.2. The same applies about the use of one-tail tests (2, 3, 5, and 6 in Table 4) respect to the respective two-tail tests (1 and 4); or about the use of nonparametric tests that do not impose any type of distribution a priori. 3. In other occasions, the question of trueness is considered comparatively between two methods: ‘To decide if the difference in means between both, when they are applied to the same sample of reference, is significant or not’. It is the two-tail test. The one-tail case is ‘to decide whether one has bias of specific sign against the other’. In these tests (Section 1.02.3.4), two experimental means are compared, one coming from applying n1 times the first method to the reference sample (aliquot parts) and the other one coming from applying n2 times the second method. Under the normality assumption, we will have to know whether the variances of both methods are known (tests 7 and 8 of Table 4) or it is necessary to estimate them from the sample. In addition, in the second case we have to decide whether they are equal (tests 9 and 10 of Table 4) or different (Equation (53)). 4. Sometimes it is impossible to use reference samples similar enough; a solution is the use of the ‘test on the difference of means with paired data’ (Case 3 of Section 1.02.3.2). For example, suppose that we wish to introduce a new method of analysis on line, faster, to indirectly determine the content of an analyte in wastewater of a company, and we need to decide whether it maintains the trueness at the same level as the previous method, which is sufficiently proved. Once the new method is ready and validated with reference samples, real samples have to be measured. The difficulty is that we cannot be sure about the amount that is to be found because this may vary from day to day. In order to eliminate the ‘factor sample’, paired designs are used: Both methods are applied to aliquot parts of the same sample, and two series of paired results x1i and x2i are obtained when applying the old and new methods, respectively. To compute the mean here has no sense because in that case we would be introducing variability due to the change of sample in each series. The correct procedure is to subtract both values di ¼ x1i x2i; now the differences are caused exclusively by the change of method and its mean will estimate the bias attributable to the new method. It will be enough to apply a test on the mean; thus, the normality and independence hypotheses must be evaluated on the differences, as well as the standard deviation, which is also estimated with them. This test for paired data is frequently used to evaluate the improvement achieved in a procedure by a technical variation, as is the case of Example 9. The effect of the change on the trueness must always be evaluated in the range of concentrations in which one is going to use the procedure. An alternative to the use of this test is the analysis of the pairs of data by a linear regression; in this case, the regression method used should consider the existence of error in both axes (see Section 1.05.10). The normality (Section 1.02.3.8), independence of the determinations, and, if it is necessary the equality of variances hypotheses will have to be tested with the appropriate tests (Section 1.02.3.7). Also, it is important to remember that the presence of outlier data tends to greatly increase the variance so that the tests become insensitive, that is to say, a larger experimental evidence is needed to reject the null hypothesis. The nonparametric alternative has, in general, a high cost in terms of power for the same significance level (or in terms of sample size). For this reason, it will not have to be used unless it is strictly necessary. In addition, some nonparametric tests also assume hypothesis on the distribution of the values, for example, to be symmetric or unimodal.
1.02.5.2
Precision
The other very important criterion in the validation of a method is the precision. In the ISO 5725,7 the IUPAC (Incze´dy et al.,11 Sections 2 and 3), and the European norm,3 we can read ‘‘Precision, the closeness of agreement between independent test results obtained under stipulated conditions’’.
56 Quality of Analytical Measurements: Statistical Methods for Internal Validation
The precision usually is expressed as imprecision. The lesser the dispersion of the random component in Equation (1), the more precise the procedure. It must be remembered that the precision depends solely on the distribution of the random errors and is not related to the value of reference or that assigned to the sample. In a first approach, it is computed as a standard deviation of the results; nevertheless, even the ISO 5725-5 recommends the use of a robust estimation. Two measures, limits in a certain sense, of the precision of an analytical method are the reproducibility and the repeatability. Repeatability is defined as ‘‘precision under repeatability conditions. Repeatability conditions means conditions where independent test results are obtained with the same method on identical test items in the same laboratory by the same operator using the same equipment with short intervals of time’’. The repeatability limit, r, is the value below which is, with a probability of 95%, the absolute value of the difference between two results of a test, obtained under repeatability conditions. The repeatability limit is given by r ¼ z=2
pffiffiffi 2 r
ð101Þ
where z/2 is the critical value of the standard normal distribution. r is the repeatability expressed as standard deviation. Reproducibility is defined as ‘‘precision under reproducibility conditions. Reproducibility conditions means conditions where test results are obtained with the same method on identical test items in different laboratories with different operators using different equipment’’. The reproducibility limit, R, defined in Equation (102), is the value below which is, with a probability of 95%, the absolute value of the difference between two results of a test, obtained under reproducibility conditions. R ¼ z=2
pffiffiffi 2 R
ð102Þ
where R is the standard deviation computed under reproducibility conditions. If n < 10, a correction factor7 should be applied to Equation (6) when r (or R) is estimated. The ISO introduces the concept of intermediate precision when only some of the factors described in the reproducibility conditions are varied. A particular interesting case is when the ‘internal’ factors of the laboratory (analyst, instrument, day) are varied, which in Commission Decision3 is called intralaboratory reproducibility. One of the causes of the ambiguity when defining precision is the laboratory bias. When the method is applied only in a laboratory, the laboratory bias is a systematic error of that laboratory. If the analytical method is evaluated in general, the laboratory bias becomes a part of the random error: to change the laboratory contributes to the variance, which is expected for a determination done with that method in any laboratory. The most eclectic position is the one described in the ISO 5725 that declares ‘‘The laboratory bias is considered constant when the method is used under repeatability conditions, but is considered as a random variable if series of applications of the method are made under reproducibility conditions.’’ On the basis of these points, we can deduce that to evaluate the precision of an analytical method is equivalent to estimating the variance of the random error in the results and that the discrepancies that an appear when establishing the sources of variability must be explicitly identified, for example the laboratory bias. The precision of two methods can be compared by a hypothesis test on the equality of variances, under the normality assumption that it is a (Snedecor) F-test (Section 1.02.3.6). Another usual problem is to decide whether the variance observed can be considered significantly equal or not to an external value, using a w2 test (Section 1.02.3.3). It is common that the lack of control on a concrete aspect of an analytical procedure is the origin of a great variability. If the experimental conditions are not stable, we will have an additional variability in the determinations. The F-test permits one to decide whether the precision improves significantly when an optimization is carried out. In fact, many improvements in the procedures are due to the identification of the causes of the variability in the results and their quantification. This aspect of control and improvement of the precision is dealt with some detail in the section dedicated to the ruggedness of the chemical analyses.
Quality of Analytical Measurements: Statistical Methods for Internal Validation
57
The technique used to construct an ANOVA with random effects (Section 1.02.4.4) is also the adequate technique to split the variance that presents each experimental data in addends that are specially adapted to estimate the repeatability and the reproducibility of an analytical method when interlaboratory test comparison has been carried out. In the following, the use of an ANOVA to estimate reproducibility and repeatability in a proficiency test is briefly explained. There is no doubt that a good analytical procedure has to be insensitive to the change of laboratory. To decide about this question, k laboratories apply a procedure to aliquot samples; each laboratory makes n determinations. In the terminology of the ANOVA, we have a random factor (the laboratory) at k levels and n replicates in each level. In general, it is not necessary to have the same number of replicates in all the levels. We denote by xij the experimental results, where i ¼ 1,. . .,k identifies the laboratory and j ¼1,. . .,n the replicate. Figure 5 is a skeleton of Equations (93)–(96) and shows how to compute an estimate of the variance of the random variable " in Equation (93). If the analytical procedure is well defined, the k estimates si2 are expected to be approximately equal and to gather the variability due to the use of the analytical method by only one laboratory. In these conditions, the pooled variance sp2 is a joint (‘pooled’) estimate of the same variance, that is, by definition, the repeatability of the method expressed as standard deviation (ISO 5725) sr ¼
pffiffiffiffiffiffiffiffiffiffi V ð"Þ sp
ð103Þ
From the same data we can obtain k estimates of the bias i (Figure 5, top) and then the variance of the laboratory bias, considering this bias as a random variable. Taking into account the quantities estimated by the variances described in Figure 5, one obtains the following expression for the interlaboratory variance: V ðÞ sx2 –
Variance n s x2 with k –1 degrees of freedom It estimates V(ε) + n V(Δ), and thus the reproducibility is sR = V(ε) + V(Δ)
sp2
ð104Þ
n
If the variability due to the change of laboratory is not significant, it also estimates the intralaboratory variance ⇑
Variance of the means, V(Δ)
∑ i =1( xi − x ) 2 k
s x2 =
k−1 ⇑
Mean estimated per laboratory
Determinations made by each laboratory xij = μ + Δi + εij
Variance estimated per laboratory
Pooled variance
x1
x2
x3
...
xk
⇑ x11 x12 x13 . . . x1n
⇑ x21 x22 x23 . . . x2n
⇑ x31 x32 x33 . . . x3n
... ... ... ... ... ... ...
⇑ xk1 xk2 xk3 . . . xkn
⇓
⇓
⇓
s 12
s 22
s 32
⇓ ...
s k2
⇓ 2 + (n – 1) s 2 + … + (n – 1) 2 – 1) (n s sk 2 1 ≈ V(ε) s p2 = k(n – 1) ⇓
Intralaboratory variance = s p2 k(n–1) degrees of freedom It estimates V(ε) Figure 5 ANOVA of random effects for a interlaboratory study.
sp estimates the repeatability, sr
58 Quality of Analytical Measurements: Statistical Methods for Internal Validation
which, linked to Equation (1), provides the following estimate of the reproducibility as standard deviation (ISO 5725): sR
pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi V ðÞ þ V ð"Þ
ð105Þ
In the ANOVA, the null hypothesis is that 1 ¼ 2 ¼ ¼ k ¼ 0 (i.e., there is no effect of the factor), and the alternative is that at least one laboratory has nonnull bias (there is effect of the factor). The conclusion of the ANOVA is obtained by deciding whether both variances, n sx2 and sp2 , can be considered significantly equal. To decide it, an F-test is applied. The logic is clear: If there is no effect (laboratory effect), V() should be significantly zero and, thus, both variances are equal, or in other words, they estimate the same quantity. In practice, the expression for the computation of the power of the ANOVA with random effects (Equation (98)) is useful in deciding the number of laboratories that should participate, k, and the number of replicated determinations, n, that each one must do. It is essential to remember that an ANOVA requires normal distribution of the residuals and equality of the variances s21,s22,. . .,sk2. When the number of replicates is two (n ¼ 2), a common way of the interlaboratory analysis is the use of the graph of Youden50 to show the trueness and precision of each laboratory. Actually, the Youden’s graph is none other than the graphical representation of an ANOVA shown in Kateman and Pijpers51 (p 112). In addition to being used for comparing the quality of the laboratories, the graph of Youden can be used to compare two methods of analysis in terms of the laboratory bias they have. An approach for the comparison of two methods in the intralaboratory situation has been proposed by Kuttatharmmakull et al.52 Instead of the reproducibility, as included in Figure 5 and ISO guidelines, the (operator þ instrument þ time) different intermediate precision is considered in the comparison. In the case of precision, the effect of outlier data is really devastating; hence, a very delicate task of analysis to detect those outlier data is essential. In general, more than one test is needed (usual ones are those of Dixon, and Grubbs and Cochran), especially to accept the hypotheses of the ANOVA made for the determination of repeatability and reproducibility. In view of the difficulties, the AMC5,6 advises the use of robust methods to evaluate the precision and trueness and for proficiency testing. This path is also followed in the new ISO norm about reproducibility and repeatability. 1.02.5.3
Statistical Aspects of the Experiments to Determine Precision
The analysis of data implies three steps: 1. Critical examination of the data, in order to identify outliers or other irregularities, and to verify the suitability of the model. 2. To compute for each level of concentration the preliminary values of precision and the mean values. 3. To establish the final values of precision and means, including the establishment of a relation between precision and the level of concentration when the analysis indicates that such relation can exist. The analysis includes a systematic application of statistical tests for detecting outliers, and a great variety of such tests are available from the literature and could be used for this task.
1.02.5.4
Consistency Analysis and Incompatibility of Data
From the data collected in a specific number of levels, a decision must be taken about certain individual results or values that seem to be ‘different’ from those of the rest of laboratories or that can vary the estimations. Specific tests are used for the detection of these outlier numerical results. Case 1. Elimination of data: It is the classic procedure based on detecting and, if this is the case, eliminating the outlier data. The tests are of two types. The test of Cochran is related to the interlevels variability of the factor
Quality of Analytical Measurements: Statistical Methods for Internal Validation
59
and should be applied first; its objective is to detect an anomalous variance in one or several of the levels of the factor. The test of Cochran has already been described in Section 1.02.3.7. Later the test of Grubbs will be applied. It is basically a test on the intralevel variability to discover possible outlier individual data. It can be used (if ni > 2) for those levels in which the test of Cochran has led to the suspicion that the interlevel variation is attributable to an individual result. It is applied in two stages: 1. Detection of a unique outlying observation (single Grubbs’ test) In a data set xi (i ¼ 1,2,. . .,n) sorted in increasing order, to prove whether the greatest observation, xn, is incompatible with the rest, the following statistic is computed: Gn;calc ¼
xn – x s
ð106Þ
On the contrary, to verify whether the smallest observation, x1, is significantly different from the rest, the statistic G1 is computed as G1;calc ¼
x – x 1 s
ð107Þ
In Equations (106) and (107), x and s are, respectively, the mean and standard deviation of xi. To decide whether the greatest or smallest value is significantly different from the rest at % significance level, the values obtained in Equations (106) and (107) are compared to the corresponding critical values written down in Table 22. The decision includes two ‘anomaly levels’: (a) If Gi,calc < G0.05,i, with i ¼ 1 or i ¼ n, accept that the corresponding x1 or xn is similar to the rest. (b) If G0.05,i < Gi,calc < G0.01,i, with i ¼ 1 or i ¼ n, the corresponding x1 or xn is considered an straggler. (c) If G0.01,i < Gi,calc, with i ¼ 1 or i ¼ n, the corresponding x1 or xn is incompatible with the rest of data of the same level (statistical outlier). Table 22 Critical values for Grubbs’ test One largest or one smallest
Two largest or two smallest
n
¼ 0.05
¼ 0.01
¼ 0.05
¼ 0.01
4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
1.481 1.715 1.887 2.020 2.126 2.215 2.290 2.355 2.412 2.462 2.507 2.549 2.585 2.620 2.651 2.681 2.709
1.496 1.764 1.973 2.139 2.274 2.387 2.482 2.564 2.636 2.699 2.755 2.806 2.852 2.894 2.932 2.968 3.001
0.0002 0.0090 0.0349 0.0708 0.1101 0.1492 0.1864 0.2213 0.2537 0.2836 0.3112 0.3367 0.3603 0.3822 0.4025 0.4214 0.4391
0.0000 0.0018 0.0116 0.0308 0.0563 0.0851 0.1150 0.1448 0.1738 0.2016 0.2280 0.2530 0.2767 0.2990 0.3200 0.3398 0.3585
Adapted with permission from ISO-5725–2. Accuracy, Trueness and Precision of Measurement Methods and Results; Ge`neve, 1994; pp. 22, Table 5.
60 Quality of Analytical Measurements: Statistical Methods for Internal Validation
2. Detection of two outlying observations (double Grubbs’ test) Sometimes it is necessary to verify that two extreme data (very large or very small) incompatible with the others do not exist. In the case of the two greatest observations, xn and xn–1, the statistic G is computed as G¼
sn2 – 1;n s02
ð108Þ
2 s1;2 s02
ð109Þ
Pn – 2 2 P P 2 1 where s02 ¼ ni¼1 ðxi – xÞ2 and sn1;n ¼ n2 i¼1 xi – n – 2 i¼1 xi . Similarly, it is possible to decide jointly on the two smallest observations, x1 and x2, by means of the following statistic: G¼
2 P P 2 where s1;2 ¼ ni¼3 xi – n –1 2 ni¼3 xi . The decision rule is analogous to that of the case of an extreme value but with the corresponding critical values in Table 22. In general, the norms, for example ISO 5725,7 propose to inspect the origin of the anomalous results and, if assignable cause does not exist eliminate the incompatible ones and leave the straggler ones indicating their condition with an asterisk. Example 17: With didactic purpose to apply the test of Grubbs and to verify the effect of outliers, the data of Table 23 have been considered as a unique series of 20 results. The greatest value is 15.93 and the lowest is 13.03, as s ¼ 0.60 and x ¼ 13:60, Equation (106) gives G20,calc ¼ 3.889 and Equation (107) gives G1,20 ¼ 0.942. By consulting the critical values in Table 22, G0.05,20 ¼ 2.709 and G0.01,20 ¼ 3.001; therefore, according to the decision rule in Case 1 (single Grubbs’ test), the value 15.93 should be considered different from the rest. Applying again the test, with 19 data, the greatest value now is 13.92 and the lowest is still 13.03, with G19,calc ¼ 1.804 and G1,calc ¼ 1.785. As the tabulated values are G0.05,19 ¼ 2.681 and G0.01,19 ¼ 2.968, there is no evidence to say that any of the extreme values is different from the rest. Table 24 contains the mean and standard deviation, with and without the value 15.93. A large effect is observed on the standard deviation, which is reduced in more than 50%. Table 23 Data of Example 17 Series 1
Series 2
Series 3
Series 4
Series 5
13.50 13.40 13.47 13.49
13.50 13.51 13.35 13.35
13.70 13.71 13.76 13.80
13.04 13.03 15.93 13.04
13.48 13.47 13.92 13.46
Table 24 Robust and nonrobust estimates of the centrality and dispersion parameters (data of Table 23)
Nonrobust procedures Mean, x Standard deviation, s
With all data (n ¼ 20)
Without 15.93 (n ¼ 19)
13.60 0.60
13.47 0.25
13.49
13.48
13.50 0.26 0.27
13.48 0.21 0.24
Robust procedures Median H15, centrality parameter MAD H15, dispersion
Quality of Analytical Measurements: Statistical Methods for Internal Validation
61
The test of Grubbs can also be applied to the mean values by level. In practice, the test of Grubbs is also used to restore the equality of variances in the ANOVA when the homogeneity of variances is rejected (Section 1.02.3.7). The work by Ortiz et al.43 contains a complete analysis with sequential application of the tests of Cochran, Bartlett, and Grubbs. Case 2. Robust methods: The procedure described in the previous section is focused on the detection of anomalous data within a set of results. Nevertheless, the elimination of these data is not recommendable when the variability of the analytical procedure is to be evaluated because the procedure is sensitive to the present values, that is to say, it depends on the data that have been eliminated (Equations (106)–(109) can lead to elimination of data in successive stages because of the reduction of the variance), and because the attainable real variance is underestimated. As previously indicated, the values of repeatability (r) and reproducibility (R) are determined by means of an ANOVA whose validity depends on whether the hypotheses of normality and homogeneity of variances are fulfilled. The robust methodology proposed in this section avoids these limitations. Its technical details can be found in Hampel et al.53 and Huber.54 An alternative to the procedures based on the elimination of outlier data, as exposed in the ISO 5725-5 norm, consists in using the H15 estimator proposed by Huber (c ¼ 1.5 and ‘Proposal 2 Scale’, Huber54), recommended by the Analytical Methods Committee5,6 and accepted in the Harmonized Protocol.55 It is an estimator whose influence function is a monotone function that limits the influence of the anomalous data by ‘moving them’ toward the position of the majority, but maintaining for them the maximum influence. This is carried out by transforming the original data by means of the function m:s:c: ðx Þ ¼ max½m – cs; minðm þ cs; x Þ
ð110Þ
where m and s are the central and dispersion parameters, which must be iteratively estimated. The function in Equation (110) is represented in Figure 6. The estimate is exactly the generalization of the maximum-likelihood estimate. It is asymptotically optimum for high-quality data, that is, data with little contamination and not very different from data following a Student’s t distribution with three d.f. Remember that Hampel et al.53 have shown that the Student’s t distributions with between 3 and 9 d.f. reproduce high-quality experimental data, and that for t3 the efficiency of the mean and standard deviation is 50 and 0%, respectively. Therefore, in practice there is the need for a robust estimate of high-quality empirical data (as those provided for the present analytical methods). The H15 estimator provides enough protection against high concentration of data that are abnormally large but near to the correct data. Nevertheless, the clearly anomalous data are not rejected with the H15 estimator, and they maintain the maximum influence but bounded. This produces an avoidable loss of the efficiency of the H15 estimator between 5 and 15% when the proportion of anomalous data present is also between 5 and 15% Ψ(x) m+cs
m
m–cs
m–cs
Figure 6 Function m.s.c.
m
m+cs
62 Quality of Analytical Measurements: Statistical Methods for Internal Validation
(rather usual percentages in routine analyses). In order to avoid this limited weakness, robust estimators as the median and the median of absolute deviation (MAD) (Equation (111)) are necessary at least in the first step of the calculation, to identify surely most of the ‘suitable’ data. MAD ¼ median
jxi – medianfxi gj 0:6745
ð111Þ
The robust procedure obtained when adapting the H15 estimator to the problem of the estimation of repeatability and reproducibility as posed in ISO norm consists of two stages and it has been followed in an identical way to the proposal in Sanz et al.56 As in the parametric procedure, it uses the mean of the table of data and the standard deviation. Therefore, once the robust procedure is applied, the data necessary to estimate the reproducibility or the intermediate precision are at hand. In order to verify the utility of these robust procedures, with the same data of Table 23 considered as a unique series of 20 values, the median and the centrality parameter of the H15 estimator have been written down in Table 24. These are very similar to the nonrobust estimates, as much with 20 as with 19 data. Nevertheless, when comparing the robust parameters of dispersion, MAD and H15, they do not differ when considering 20 or 19 data and are similar to the standard deviation obtained after applying the method of Grubbs and repeating the calculations without the outlier. For this reason, it is a good strategy to apply systematically robust procedures together with the classic ones. The difference in the results is an indication of the presence of outlier data, in which case the robust estimations will have to be used. The effect, and therefore the advantage, of the robust procedures is much more remarkable when an ANOVA of random effects is evaluated, for example, to estimate the reproducibility and repeatability of a method by means of an interlaboratory test as the one described in Figure 5. To show this, we will use the data of Table 23, this time considering its structure of levels of the factor (k ¼ 5) and replicates (n ¼ 4). The values of reproducibility and repeatability should not be accepted if the homogeneity of variances in the ANOVA assumption is not fulfilled. In this case, it is necessary to verify whether some of the levels have outlier data. The first column of Table 25 shows that the ANOVA with all the data is not acceptable because of the lack of equality of variances (rejection in the tests of variance homogeneity). In addition, it is observed that the anomaly in the data causes that the estimates sR and sr are equal and very different from the robust estimates. Once the value 15.93 of series 5 is removed, the ANOVA (column 2 of Table 25) is significant but still there is lack of variance homogeneity. Nevertheless, the new estimates of sR and sr are more similar to those obtained with the robust procedure. The lack of equality of variances forces one to eliminate series 4, which has a very different variance (smaller than the others), and later the value 13.92 of series 5. The final result of this sequential process is the third column of Table 25. The ANOVA is significant, the homogeneity of variances can be accepted, and the estimates of the reproducibility and the repeatability are 0.153 and 0.058, similar to the ones that would be obtained with the robust procedure without series 4. The values sR and sr can be too small due to the elimination of data, with the risk of having underestimations that are not realistic and thus the laboratories cannot fulfill them. For this reason, it is advised to avoid reduction of the sample and to maintain the initial robust estimates.5–7 Table 25 Robust and nonrobust estimates of the repeatability and reproducibility with data of Table 23 ANOVA
With all data (n ¼ 20)
Without 15.93 (n ¼ 19)
Without series 4 and 13.92 (n ¼ 15)
Nonrobust procedure Fcalc (p-value) SSF (d.o.f.) SSE (d.o.f.) sR sr p-value, Cochran’s test p-value, Bartlett’s test p-value, Levene’s test
0.22 (0.92) 0.094 (4) 0.431 (15) 0.657 0.657 8.9 10–9 3.9 10–8 0.53
17.02 (<5 10–5) 0.229 (4) 0.013 (14) 0.260 0.116 0.001 0.001 0.61
24.91 (<5 10–5) 0.083 (3) 0.003 (11) 0.153 0.058 0.093 0.100 0.005
Robust procedures Robust sR Robust sr
0.281 0.072
0.172 0.072
Quality of Analytical Measurements: Statistical Methods for Internal Validation
63
As the presence of outliers in the experimental work is unavoidable, the robust statistical methodology has been consolidated as an essential tool in the chemical analysis. Also, Chapter 1.07 of this book is dedicated to robust statistical techniques. 1.02.5.5
Accuracy
According to the IUPAC (Incze´dy et al.,11 Sections 2–3), the ISO,7 and the Directive of the European Union (Commission Decision,3 Definition 1.1), the accuracy is a concept defined as ‘‘Closeness of agreement between a test result and the accepted reference value’’. It is estimated by determining trueness and precision. Evidently, this definition collects together the systematic and random errors, because for an individual determination it is xi ¼ (xi x) þ ( x ) ¼ " þ . In practice, it is unreasonable to think that an analytical procedure has no bias; experimentally we can decide about the hypothesis of null bias. If it is significant, it is possible to correct the measurement by subtracting the value . However, this implies an increase in the variance of the final result because is estimated by experimental replicates and therefore it has uncertainty. For this reason, when the uncertainty of a measurement is expressed, it is usual to include a term to consider the bias in a form similar to Equation (105). For a detailed treatment of this question, consult the guide of the EURACHEM/CITAC.1 1.02.5.6
Ruggedness
The ruggedness of a method is defined as its capacity to maintain trueness and precision throughout the time. The same applies for the robustness of a material of reference or any other reagent. Ruggedness means susceptibility of an analytical method to changes in experimental conditions, which can be expressed as a list of sample materials, analytes, storage conditions, environmental conditions, and/or sample preparation conditions under which the method can be applied as presented or with specified minor modifications.3 The study on ruggedness can be approached using two different statistical methodologies, one of which consists of using the well-known control charts (they are confidence intervals on the mean, the variance, or the range of the measured parameter) and continuously writing down the results obtained on known samples throughout the time. This type of ‘a posteriori’ control is essential to maintain the quality (precision, trueness, capability of detection, etc.) of a measurement method and to establish alarm mechanisms when the observed drift can alter the quality of the procedure affecting the value of the analytical results. Chapter 1.04 of this book deals specifically with control charts. The other approach to the problem of ruggedness involves evaluating ‘a priori’ the variability expected in the analytical procedure and identifying the sources of that variability. Before routinely using a procedure, the effect of small changes in the reagents, in the conditions of work, or in the specifications of its protocol would have been verified. It can happen that small changes in the volume of extracting reagent do not lead to great variations in the response, whereas a small variation in, say, pH does. One way of knowing and controlling this quality criterion is by making small changes in the potentially influential factors and observing the effect on the response. The influence of each factor should not be separately analyzed because methodologically it is not adequate and in addition it is not realistic, because in practice unforeseeable combinations of all the factors will occur that can affect the results. Instead, the methodology of the design of experiments should be used, and details about this are in Chapters 1.09–1.15 of this collection. As the number of factors that potentially affect the response is large, highly fractional factorial designs for two levels have to be used (to decrease for example the needed 27 ¼ 128 different experiments that contain a complete factorial design for seven factors); Chapter 1.10 dedicated to these strategies. Plackett–Burman designs and D-optimal designs have proven to be useful tools in ruggedness analysis.57–59 Example 18: An analysis of ruggedness of an extraction procedure of three sulfonamides is carried out. The seven factors considered are buffer solution, pH, methanol as extracting agent, extraction cycles, petroleum benzin, volume
64 Quality of Analytical Measurements: Statistical Methods for Internal Validation Table 26 Experimental factors with nominal (þ) and extreme () levels selected for a Plackett–Burman design for seven factors Level Factor (units)
þ
x1: buffer solution (ml) x2: pH x3: extracting agent (ml) x4: extraction cycles x5: petroleum benzin (ml) x6: volume of elution (ml) x7: evaporation
1.0 4.5 20 One 20 7 To dryness
1.5 4.8 25 Two 25 8 Evaporated until 1 ml
of elution, and evaporation. A Plackett–Burman design has been proposed to estimate the effects of the factors by fitting a linear model for each sulfonamide: y ¼ b0 þ b1 x1 þ b2 x2 þ þ b7 x7
ð112Þ
where y represents the response to be modeled, which is the area under the corresponding chromatographic peak for each of the three sulfonamides. The details about the experimental domain can be seen in Table 26, where the nominal level is codified as ‘—’ and the extreme level as ‘þ’. Table 27 shows the experimental runs and the two values (replicates) of the three responses. Finally, Table 28 contains the estimated coefficients of model (112) and its p-values. The conclusion is that only the extracting agent (x3), the times that the extraction is carried out (x4), and the volume of petroleum benzin (x5) are significant at 5% level. Hence, special care should be taken with these factors because small changes in any of them can cause large changes in the response. Table 27 Plackett–Burman design for the seven factors of Table 26 Factor Run
x1
Response (y) x2
x3
x4
x5
x6
x7
1
1
1
1
1
1
1
1
2
1
1
1
1
1
1
1
3
1
1
1
1
1
1
1
4
1
1
1
1
1
1
1
5
1
1
1
1
1
1
1
6
1
1
1
1
1
1
1
7
1
1
1
1
1
1
1
8
1
1
1
1
1
1
1
SDZ
SMT
SMP
10.50 10.30 5.87 7.56 7.55 8.08 3.82 5.61 6.74 8.19 6.89 7.63 8.21 8.70 8.56 5.85
10.50 10.30 6.31 7.74 8.88 11.01 6.58 7.20 10.00 10.63 5.35 6.99 7.86 8.56 6.95 8.11
10.50 10.30 8.91 8.72 7.06 8.23 6.11 7.53 9.61 10.38 4.29 6.40 10.24 9.94 7.02 7.96
The three responses are the areas by duplicate under the chromatographic peak (in a.u.) of sulfadiazine (SDZ), sulfamethazine (SMT), and sulfamethoxypyridazine (SMP).
Quality of Analytical Measurements: Statistical Methods for Internal Validation
65
Table 28 Estimated coefficients of the linear model (Equation (112)) fit for each sulfonamide by means of a Plackett–Burman design Sulfadiazine
b0 b1 b2 b3 b4 b5 b6 b7 a
Sulfamethazine
Sulfamethoxypyridazine
Coefficient
p-value
Coefficient
p-value
Coefficient
p-value
7.504 –0.093 0.456 1.030 0.690 0.666 –0.057 0.204
<0.000 1a 0.726 0.111 0.004a 0.020a 0.039a 0.820 0.447
8.309 0.252 0.165 1.409 –0.021 0.203 0.475 –0.391
<0.000 1a 0.274 0.465 0.000a 0.924 0.374 0.058 0.106
8.325 0.095 0.314 1.208 0.874 –0.605 0.351 –0.161
<0.000 1a 0.635 0.142 0.000a 0.002a 0.014a 0.105 0.426
Significant factor at a 0.05 significance level.
Appendix 1. Some Basic Elements of Statistics A distribution function (cumulative distribution function (cdf)) in R is any function F, such that 1. F is an application from R to the interval [0,1] lim F ðx Þ ¼ 0 2. x! –1 lim F ðx Þ ¼ 1 3. x!þ1 4. F is a monotonously increasing function, that is, a b implies F(a) F(b). 5. F is continuous on the left or the right. For example, F is continuous on the left if lim F ðx Þ ¼ F ðaÞ for x!a;x
a
p½a; bÞ ¼ pfa x < b g ¼ F ðb Þ – F ða Þ pða; b Þ ¼ pfa < x < b g ¼ F ðb Þ – lim F ðx Þ x!a;x>a
If the distribution function is continuous, then the above limits coincide with the value of the function in the corresponding point. The probability density function f(x), abbreviated pdf, if it exists, is the derivative of the cdf. Each random variable X is characterized by a distribution function FX(x). When several random variables are handled, it is necessary to define the joint distribution function. FX1 ;X2 ;...;Xk ða1 ; a2 ; . . . ; ak Þ ¼ prfX1 a1 and X2 a2 and . . . and Xk ak g
ðA1Þ
If the previous joint probability is equal to the product of the individual probabilities, it is said that the random variables are independent: FX1 ;X2 ;:::Xk ða1 ; a2 ; :::; ak Þ ¼ prfX1 a1 g prfX2 a2 g prfXk ak g
ðA2Þ
66 Quality of Analytical Measurements: Statistical Methods for Internal Validation
Equations (3) and (4) define the mean and variance of a random variable. Some basic properties are E ðaX þ bY Þ ¼ aE ðX Þ þ bE ðY Þ for any X and Y
ðA3Þ
V ðaX Þ ¼ a 2 V ðX Þ for any random variable X
ðA4Þ
Given a random variable, X, the standardized pffiffiffiffiffiffiffiffiffiffiffivariable is obtained by subtracting the mean and dividing by the standard deviation, Y ¼ ðX – E ðX ÞÞ= V ðX Þ. The standardized variable has E ðY Þ ¼ 0 and V ðY Þ ¼ 1. For any two random variables, the variance is V ðX þ Y Þ ¼ V ðX Þ þ V ðY Þ þ 2CovðX ; Y Þ
ðA5Þ
and the covariance is defined as ZZ CovðX ; Y Þ ¼
ðx – E ðX ÞÞðy – E ðY ÞÞfX ;Y ðx; y Þ dx dy
ðA6Þ
In the definition of the covariance (Equation (A6)), fX ;Y ðx; y Þ is the joint pdf of the random variables. In the case where they are independent, the joint pdf is equal to the product fX ðx ÞfY ðy Þ and the covariance is zero. In general, E ðXY Þ 6¼ E ðX ÞE ðY Þ, except where the variables are independent, in which case the equality holds. In the applications in Analytical Chemistry, it is very frequent to use formulas to obtain the final measurement from other intermediate ones that had experimental variability. A strategy for the calculation of the uncertainty (variance) in the final result under two basic hypotheses has been developed. The strategy is to make a linear approach to the formula and then to assimilate the quadratic terms to the variance of the implied random variable (see for example the ‘Guide to the Expression of Uncertainty in Measurement’).2 This procedure, called in many texts the method of transmission of errors, can lead to unacceptable results. Hence, an improvement based on Monte Carlo simulation has been suggested for the calculation of the compound uncertainty (see the Supplement 1 to the aforementioned guide). A useful representation of the data is the so-called box and whisker plot (or simply box plot). To explain its procedure of construction, we will use the 100 values of the method A of Figure 2. These data have the following characteristics (summary of statistics): Minimum: 5.23 Maximum: 7.86 First or lower quartile, Q1 ¼ 6.39. It is the value below which lie 25% of the data. Second quartile (median), Q2 ¼ 6.66. It is the value below which lie 50% of the data. Third or upper quartile, Q3 ¼ 6.98. It is the value below which lie 75% of the data. Interquartile range, IR ¼ Q3 – Q1 ¼ 0.59 in our case. With these quartiles, the central rectangle (the box) is drawn that contains 50% of the data around the median. The lower and upper limits are computed as LL ¼ Q1 1.5IR and UL ¼ Q3 þ 1.5IR. In the example, LL ¼ 6.39 1.5 0.59 ¼ 5.505 and UL ¼ 7.865. Then, the ‘whiskers’ are made by joining the inferior side of the rectangle with the data immediately greater than or equal to LL, and the superior side of the rectangle with the greatest value in the data that is immediately less than UL. The three smallest values in our case are 5.396, 5.233, and 5.507; thus the whisker will go until 5.507 and the other two values are left ‘disconnected’. The other whisker reaches the maximum 7.86 because it is less than UL. The box and whisker plot is the first one in Figure A1. The advantage of using box plots is that the quartiles are practically insensitive to outliers. For example, suppose that value 7.86 is changed by 8.86; this change does not affect the median or the quartiles, the box plot continues being similar but with a datum outside the upper whisker, as can be seen in the second box plot in Figure A1.
Quality of Analytical Measurements: Statistical Methods for Internal Validation
67
9.2 8.2 7.2 6.2 5.2 A
B
Figure A1 Box and whisker plot. (A) data of method A in Figure 2. (B) data of method A with an outlier.
2. The Normal Distribution A normal distribution with mean and standard deviation , N(,), has as pdf the following function defined for all real numbers: 1 x– 2 1 f ðx Þ ¼ pffiffiffiffiffi e – 2ð Þ 2
ðA7Þ
The normal distribution is a continuous random variable with E ðN ð; ÞÞ ¼ and V ðN ð;ÞÞ ¼ 2 , and these two parameters completely define the distribution. Particularly interesting is the N(0,1), usually called Z, because any other normal distribution N(,) is transformed into a Z when standardizing it, that is, Z ¼ ðN ð; Þ – Þ=. The distribution function of a normal random variable does not have an analytical expression; hence it is necessary to use tables or somewhat complex formulas to calculate the probabilities. As any normal distribution can be transformed into a N(0,1), it is customary to use only the table of this distribution. Table A1 contains some of its values that, in any case, cover the cases used in this chapter. For example, if z ¼ 1.83, from the reading in row 1.8 and column 0.03, p ¼ prfN ð0;1Þ > 1:83gP ¼ 0:0336. n The sum of normal and independent random variables, i¼1 N ði ;i Þ, also follows a normal distribution P
P n n 2 1=2 N . i¼1 i ; i¼1 i
3. Student’s t Distribution If X is a random variable N ð; Þ and X1,X2,. . .,Xn arepnffiffiffirandom variables independent and with the same distribution as X, then the random variable ðX – Þ=ð= nÞ is a N(0,1), where X denotes the random variable Pn i¼1 Xi =n. pffiffiffi Let S2 be the sample variance. The statistics t ¼ ðX – Þ=ðS= nÞ follows a t distribution with ¼ n 1 d.f. The mean and variance are respectively E ðt Þ ¼ 0 and V ðt Þ ¼ =ð – 2Þ, for > 2. The general appearance of its pdf is similar to that of the standard normal distribution, both are symmetrical around zero, unimodal, and vary in Table A1 Selected point of the Z ¼ N(0,1) distribution z
0.00
0.01
0.02
0.03
0.04
0.05
0.06
0.07
0.08
0.09
1.5 1.6 1.7 1.8 1.9 2.0
0.0668 0.0548 0.0446 0.0359 0.0287 0.0227 5
0.0655 0.0537 0.0436 0.0351 0.0281 0.0222 2
0.0643 0.0526 0.0427 0.0344 0.0274 0.0216 9
0.0630 0.0516 0.0418 0.0336 0.0268 0.0211 8
0.0618 0.0505 0.0409 0.0329 0.0282 0.0208 8
0.0606 0.0495 0.0401 0.0322 0.0258 0.0201 8
0.0594 0.0485 0.0392 0.0314 0.0250 0.0197 0
0.0582 0.0475 0.0384 0.0307 0.0244 0.0192 3
0.0571 0.0465 0.0375 0.0301 0.0239 0.0187 8
0.0559 0.0455 0.0367 0.0294 0.0233 0.0183 1
Values of p such that p ¼ pr{N(0,1) > z}.
68 Quality of Analytical Measurements: Statistical Methods for Internal Validation Table A2 Selected point of the t distribution with degrees of freedom
0.25
0.10
0.05
0.025
0.01
0.005
0.0025
0.001
0.0005
1 2 3 4 5 6 7 8 9 10 20 100 1000 1
1.000 0.816 0.765 0.741 0.721 0.718 0.711 0.716 0.713 0.711 0.687 0.677 0.675 0.675
3.078 1.886 1.638 1.533 1.476 1.440 1.415 1.397 1.383 1.372 1.325 1.290 1.282 1.282
6.314 2.920 2.353 2.132 2.015 1.943 1.895 1.860 1.833 1.812 1.725 1.660 1.645 1.645
12.706 4.303 3.182 2.776 2.571 2.447 2.365 2.365 2.262 2.228 2.086 1.984 1.962 1.960
31.821 6.965 4.541 3.747 3.365 3.143 2.998 2.896 2.821 2.764 2.528 2.364 2.330 2.326
63.657 9.925 5.841 4.604 4.032 3.707 3.499 3.355 3.250 3.169 2.845 2.626 2.581 2.576
127.321 14.089 7.453 5.598 4.773 4.317 4.929 3.833 3.690 3.581 3.497 2.871 2.813 2.807
318.309 22.327 10.214 7.173 5.893 5.208 4.785 4.501 4.297 4.144 3.552 3.174 3.098 3.090
636.619 31.598 12.924 8.610 6.869 5.959 5.408 5.041 4.781 4.587 3.850 3.390 3.300 3.290
Values of t such that ¼ pr{t > t}.
(–1,1). However, the t distribution has heavier tails than the normal; that is, exhibits greater variability. As the number of d.f. tends to infinity, the limiting distribution is the standard normal one. The family of t distributions depend only on one parameter, its degrees of freedom. Table A2 contains some values of the t distribution. For example, if ¼ 5, for ¼ 0.025, the value t ¼ 2.571 in the table is the one for which 0.025 ¼ pr{t5 > 2.571} holds. Compare with the value 1.96 in Table A1 that would correspond, in the same conditions, to a N(0,1), that is, 0.025 ¼ pr{N(0,1) > 1.96}. Because of the symmetry, 0.025 ¼ pr{t5 < –2.571} also holds and, consequently, 0.95 ¼ pr{–2.571 < t5 < 2.571}. 4. The w2 (Chi-Square) Distribution Under the same conditions as the previous section for the variables X1,X2,. . .,Xn, the random variable w2 ¼ ½ðn – 1ÞS 2 =2 follows a chi-square distribution with ¼ n 1 d.f.
The mean and variance of a w2 distribution with d.f. are E w2 ¼ and V w2 ¼ 2. The chi-square distribution is nonnegative and the pdf is skewed to the right. However, as increases, the distribution becomes more symmetric. Some percentage points of the chi-square distribution are given in Table A3. Table A3 Selected percentage points of the w2 distribution with degrees of freedom 1 2 3 4 5 6 7 8 9 10 100
0.990 0.000 16 0.020 1 0.115 0.297 0.554 0.872 1.24 1.65 2.09 2.56 70.06
0.975
0.000 98 0.050 6 0.216 0.484 0.831 1.24 1.69 2.18 2.70 3.25 74.22
Values of x such that ¼ pr w2 > x .
0.950
0.900
0.100
0.050
0.025
0.010
0.003 9 0.102 8 0.352 0.711 1.15 1.64 2.17 2.73 3.33 3.94 77.93
0.015 8 0.210 7 0.584 1.064 1.61 2.20 2.83 3.49 4.17 4.87 82.36
2.71 4.61 6.25 7.78 9.24 10.64 12.02 13.36 14.68 15.99 118.50
3.84 5.99 7.81 9.49 11.07 12.69 14.07 15.51 16.92 18.31 124.34
5.02 7.38 9.35 11.14 12.83 14.45 16.01 17.53 19.02 20.48 129.56
6.63 9.21 11.34 13.28 15.09 16.81 18.48 20.09 21.67 23.21 135.91
Quality of Analytical Measurements: Statistical Methods for Internal Validation
69
For example, if ¼ 5 and ¼ 0.025, the value of the chi-square distribution with 5 d.f., w20:025;5 , that leaves to
2 its right the probability 0.025 is 12.83. That is, 0:025 ¼ pr w > 12:83 . Analogously, 0:975 ¼ pr w25 > 0:83 , 5 and consequently, 0:95 ¼ pr 0:83 < w25 < 12:83 . The chi-square distribution has an important property: Let w21 ; w22 ; :::; w2n be independent chi-square random ., n d.f., respectively. Then the random variable w21 þ w22 þ þ w2n is a chi-square variables with 1, 2,. .P distribution with ¼ ni¼1 i d.f. This property becomes apparent if we note Pthat if Z1,Z2,. . .,Zn are normally and independently distributed random variables, then the random variable ni¼1 Zi2 is a chi-square distribution with n d.f. 5. The F Distribution Let V and W be independent chi-square random variables with 1 and 2 d.f., respectively. Then, the ratio F ¼ ðV =1 Þ=ðW =2 Þ is an F distribution with 1 d.f. in the numerator and 2 d.f.
in the denominator. It is usually abbreviated as F . The mean and variance of F are E F ; ; ; 1 2 1 2 1 2 ¼ 2 =ð2 – 2Þ for
2 > 2 and V F1 ;2 ¼ 22 ð1 þ 2 – 2Þ=1 ð2 – 2Þ2 ð2 – 4Þ with 2 > 4. The F distribution is nonnegative and skewed to the right. Some percentage points of the F distribution are given in Table A4 for ¼ 0.05. For, say, 1 ¼ 5 and 2 ¼ 10, F0:05;5;10 ¼ 3:33 is the value such that 0:05 ¼ pr F5;10 > 3:33 . The lower percentage points can be found taking into account that F1 – ;1 ;2 ¼ 1=F
;2 ;1 . For example, to find F0:95;5;10 is F0:95;5;10 ¼ 1=F0:05;10;5 ¼ 1=4:74 ¼ 0:21. Therefore, we have 0:90 ¼ pr 0:21 < F5;10 < 3:33 .
6. Convergence of Random Variables Sometimes, it is useful to think how a succession of random variables converges to another random variable. Let X1,X2,. . . be a sequence of random variables and let F1(x),F2(x),. . . be the corresponding sequence of distribution functions. If the distribution functions are more and more similar to the distribution function F of a random variable X when n ! 1, then we say that they converge to X in ‘distribution’. Formally, this means that lim Fn ðx Þ ¼ F ðx Þ for each x
n!1
Table A4 Selected percentage points of the F 1,2 distribution for ¼ 0.05 1 2
1
2
3
4
5
6
7
8
9
10
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
161.45 18.51 10.13 7.71 6.61 5.99 5.59 5.32 5.12 4.96 4.84 4.75 4.67 4.60 4.54
99.50 19.00 9.55 6.94 5.79 5.14 4.74 4.46 4.26 4.10 3.98 3.89 3.81 3.74 3.68
215.71 19.16 9.28 6.59 5.41 4.76 4.35 4.07 3.86 3.71 3.59 3.49 3.41 3.34 3.29
224.58 19.25 9.12 6.39 5.19 4.53 4.12 3.84 3.63 3.48 3.36 3.26 3.18 3.11 3.06
230.16 19.30 9.01 6.26 5.05 4.39 3.97 3.69 3.48 3.33 3.20 3.11 3.03 2.96 2.90
233.99 19.33 8.94 6.16 4.95 4.28 3.87 3.58 3.37 3.22 3.09 3.00 2.92 2.85 2.79
236.77 19.35 8.89 6.09 4.88 4.21 3.79 3.50 3.29 3.14 3.01 2.91 2.83 2.76 2.71
238.88 19.37 8.85 6.04 4.82 4.15 3.73 3.44 3.23 3.07 2.95 2.85 2.77 2.70 2.64
240.54 19.38 8.81 6.00 4.77 4.10 3.68 3.39 3.18 3.02 2.90 2.80 2.71 2.65 2.59
241.88 19.40 8.79 5.96 4.74 4.06 3.64 3.35 3.14 2.98 2.85 2.75 2.67 2.60 2.54
Values of x such that 0.05 ¼ pr {F 1, 2>x}.
70 Quality of Analytical Measurements: Statistical Methods for Internal Validation
We say that Xn converges to X in ‘probability’ if 8 " > 0 lim prfjXn – X j > "g ¼ 0
n!1
This means that the probability of the set where Xn differs from X is getting smaller and smaller. Furthermore, we say that Xn converges to X ‘almost surely’ if n o pr lim jXn – X j ¼ 0 ¼ 1 n!1
Almost sure convergence implies that the set of ideal measurements such that the outcomes (real values) of Xn are getting closer and closer to X has probability one. It can be proven that almost sure convergence implies convergence in probability, which then implies convergence in distribution. The following three fundamental results for convergence are the most widely used in practice. The ‘weak law of large numbers’ states that if X1,X2,. . .,Xn,. . . are independent and identically distributed random variables with finite mean , then X1 þ þ Xn ! in probability: n
If the random variables also have a finite variance (a weaker condition is also possible), then we have the ‘strong law of large numbers’, X1 þ þ Xn ! almost surely: n
The ‘central limit theorem’ says that for independent (or weakly correlated) random variables X1,X2,. . .,Xn, with the same distribution, pffiffiffi nðX – Þ ! N ð0; 1Þin distribution
where and 2 are the mean and variance of the random variables Xn. This means that the distributional shape of X is more and more like that of a normal standard random variable as n increases. 7. Some Computational Aspects The accessibility of the personal computers allows doing the statistical calculations without tables. It is advisable to use software specially dedicated to Statistics, but in the initial stages of the learning it is worthwhile to make the calculations manually, so that the intuition necessary to avoid the errors derived from a nonreflexive and automatic use of the software is acquired. The basic distributions can be programmed with the algorithms in Abramowitz and Stegun,60 normal distributions are in Chapter 3.12, the Student’s t and F are in Chapter 3.12, and finally, the chi-square distribution is in Section Chapter 3.12. Chapter 5 (appendices) of the book by Meier and Zu¨nd61 shows the necessary numerical approximations and programs in BASIC for these distributions. To compute the noncentral F, the needed numerical approximation can be consulted in Johnson and Kotz,62 and Evans et al.63 All the calculations in this chapter have been made with the Statistics Toolbox for MATLAB.64 What follows is a list of basic command instructions used. Note that all the commands referring to cumulative distribution functions, equations (A8–A11), compute the cumulative probability until the corresponding value of the distribution. However, along the text and in tables A1–A4, the calculated probability is always the upper percentage point, that is, the cumulative probability above the corresponding value.
Quality of Analytical Measurements: Statistical Methods for Internal Validation
71
Normal distribution ¼ prfN ð;Þ < zg
ðA8Þ
– z ¼ norminv(, , ) Example A1: ¼ 0.05, ¼ 0, ¼ 1; then norminv(0.05, 0, 1) gives z ¼ –1.645. – ¼ normcdf(z, , ) Example A2: z ¼ 1.645, ¼ 0, ¼ 1; then normcdf(1.645, 0, 1) gives ¼ 0.95 Student’s t distribution with degrees of freedom
¼ pr t < t;
ðA9Þ
– t ¼ tinv(,v) Example A3: ¼ 0.05, ¼ 5; then tinv(0.05,5) gives t ¼ –2.015. – ¼ tcdf(t,v) Example A4: t ¼ 1.645, ¼ 5; then tcdf(1.645,5) gives ¼ 0.9196. n o ¼ pr w2 < w2;
ðA10Þ
w2 distribution with degrees of freedom
– x ¼ chi2inv(,) Example A5: ¼ 0.05, ¼ 5; then chi2inv(0.05,5) gives x ¼ 1.1455. – ¼ chi2cdf(x,) Example A6: x ¼ 9.24, ¼ 5; then chi2cdf(9.24,5) gives ¼ 0.9001. F1 ;2 distribution 1 and 2 degrees of freedom
¼ pr F1 ;2 < F;1 ;2
ðA11Þ
– x ¼ finv(,1,2) Example A7: ¼ 0.95, 1 ¼ 5, 2 ¼ 15; then finv(0.95,5,15) gives x ¼ 2.9013. – ¼ fcdf(x,1,2) Example A8: x ¼ 2.90, 1 ¼ 5, 2 ¼ 15; then fcdf(2.90,5,15) gives ¼ 0.9499. Power for the z-test, Equation (40)
Example A9. With the data of Example 8, jj ¼ 0.40, ¼ 0.55, n ¼ 10, ¼ 0.05, normcdf(norminv(0.95,0,1) – 0.40 sqrt(10)/0.55) gives 0.2562. Power for the t-test, Equation (43)
Example A10. With the same data as for the z-test, the sentence includes the noncentral t-distributionp‘nctcdf’ ffiffiffiffiffi and the t distribution ‘tinv’, both of them with n – 1 ¼ 9 d.f. and noncentrality parameter ð0:40=0:55Þ 10. nctcdf(tinv(0.95,9),9, 0.40 sqrt(10)/0.55) gives ¼ 0.3165. pffiffiffiffiffi Example A11. With data of Example 9, ¼ 0.05, n ¼ 10, and the noncentrality parameter is 0:57 10. nctcdf(tinv(0.95,9),9, 0.57 sqrt(10)) gives ¼ 0.5034.
72 Quality of Analytical Measurements: Statistical Methods for Internal Validation
Power for the chi-square test, Equation (46)
Example A12. We have ¼ 2, ¼ 0.05, and n ¼ 14 or n ¼ 13 to obtain a value of 0.05. chi2cdf(chi2inv(0.95,13)/(2 2),13) gives ¼ 0.0402. chi2cdf(chi2inv(0.95,12)/(2 2),12) gives ¼ 0.0511. (note that the d.f. equal 14 – 1 ¼ 13 or 13 – 1 ¼ 12). Power for the F-test, Equation (57)
Example A13. Data: ¼ 0.05, n1 ¼ n2 ¼ 9, l ¼ 1/2 ¼ 2 are those of question (3) in Example 13. fcdf(finv(0.975,8,8)/(2 2),8,8)-fcdf(finv(0.025,8,8)/(2 2),8,8) gives ¼ 0.5558. If we look for the sample size n ¼ n1 ¼ n2 so that 0.10, trying some values, we get fcdf(finv(0.975,22,22)/(2 2),22,22)-fcdf(finv(0.025,22,22)/(2 2),22,22) that gives ¼ 0.1115 and fcdf(finv(0.975,23,23)/(2 2),23,23)-fcdf(finv(0.025,23,23)/(2 2),23,23) that gives ¼ 0.0981. Consequently, n ¼ 24. Power for the ANOVA with fixed effects, Equation (82)
Example A14. ¼ 0.05, 1 ¼ 4, 2 ¼ 15, noncentrality parameter ¼ n
P
i2 =2 ¼ 4 2 ¼ 8.
ncfcdf(finv((0.95),4,15),4,15,8) gives ¼ 0.5364 Note that in the ANOVA the levels of factor are k ¼ 5 and n ¼ 4 replicates per level. Power for the ANOVA with random effects, Equation (98)
Example A15. Suppose that 10 laboratories participate in a proficiency test to evaluate a method. Assumed risks are ¼ ¼ 0.05 and it is desirable to detect at least an interlaboratory variance equal to the intralaboratory variance, that is, 2 ¼ 1 þ n2 =2 ¼ 1 þ n 1. With these data k ¼ 10;n ¼ 4;fcdf(finv(0.95,k – 1,k (n – 1))/(1 þ 1 n),k – 1,n (k – 1)) gives ¼ 0.0973 k ¼ 10;n ¼ 5;fcdf(finv(0.95,k – 1,k (n – 1))/(1 þ 1 n),k – 1,n (k – 1)) gives ¼ 0.0494 Thus, each laboratory must do five determinations.
References 1. EURACHEM/CITAC. Guide CG4 In Quantifying Uncertainty in Analytical Measurement, 2nd ed.; Ellison, S. L. R., Rosslein, M., Williams, A., Ed.; 2000. ISBN 0–948926–15-5. Available from the Europchem Secretariat. (see http://www.eurochem.org). 2. Draft Supplement 1 to the ‘Guide to the Expression of Uncertainty in Measurement’. Evaluation of measurement data. Propagation of distributions using a Monte Carlo method (2004). 3. Commission Decision 12 August 2002, Brussels. Off. J. Eur. Commun. L 221 (17 August 2002) 8-36. Implementing Council Directive 96/23/EC concerning the performance of analytical methods and the interpretation of results. 4. Aldama, J. M. Practicum of Master in Advanced Chemistry; University of Burgos: Burgos, Spain, 2007. 5. Analytical Methods Committee. Robust Statistics – How Not to Reject Outliers, Part 1. Basic Concepts. Analyst 1989, 114, 1693–1697. 6. Analytical Methods Committee. Robust Statistics – How Not to Reject Outliers, Part 2. Inter-laboratory Trials. Analyst 1989, 114, 1699–1702. 7. ISO 5725. Accuracy Trueness and Precision of Measurement Methods and Results, Part 1. General Principles and Definitions, Part 2. Basic Method for the Determination of Repeatability and Reproducibility of a Standard Measurement Method, Part 3. Intermediate Measures of the Precision of a Standard Measurement Method, Part 4. Basic Methods for the Determination of the Trueness of a Standard Measurement Method, Part 5. Alternative Methods for the Determination of the Precision of a Standard Measurement Method, Part 6. Use in Practice of Accuracy Values. Gene`ve, 1994. 8. Analytical Methods Committee. Technical brief No. 4. Ed. M. Thompson. 2006. www.rsc.org/amc/. 9. Silverman, B. W. Density Estimation for Statistics and Data Analysis; Chapman and Hall: London, Great Britain, 1986. 10. Wand, M. P.; Jones, M. C. Kernel Smoothing; Chapman and Hall: London, Great Britain, 1995. 11. Incze´dy, J.; Lengyel, T.; Ure, A. M.; Gelencse´r, A.; Hulanicki, A. Compendium of Analytical Nomenclature IUPAC, 3rd ed.; Pot City Press Inc.: Baltimore 2nd printing, 2000; p 50. 12. Lira, I.; Wo¨ger, W. Comparison Between the Conventional and Bayesian Approaches to Evaluate Measurement Data. Metrologia 2006, 43, S249–S259. 13. Zech, G. Frequentist and Bayesian Confidence Intervals. Eur. Phys. J. Direct 2002, C12, 1–81. 14. Sprent, P. Applied Nonparametric Statistical Methods; Chapman and Hall, Ltd: New York, 1989. 15. Patel, J. K. Tolerance Limits. A Review. Commun. Stat. Theory Methods 1986, 15 (9), 2716–2762. 16. Wald, A.; Wolfowitz, J. Tolerance Limits for a Normal Distribution. Ann. Math. Stat. 1946, 17, 208–215.
Quality of Analytical Measurements: Statistical Methods for Internal Validation
73
17. Wilks, S. S. Determination of Sample Sizes for Setting Tolerance Limits. Ann. Math. Stat. 1941, 12, 91–96. 18. Kendall, M.; Stuart, A. The Advanced Theory of Statistics, Inference and Relationship. Charles Griffin & Company Ltd: London, 1979; pp 547–548; Section 32.11; Vol. 2. 19. Willink, R. On Using the Monte Carlo Method to Calculate Uncertainty Intervals. Metrologia 2006, 43, L39–L42. 20. Guttman, I. Statistical Tolerance Regions; Charles Griffin and Company: London, 1970. 21. Huber, Ph.; Nguyen-Huu, J. J.; Boulanger, B.; Chapuzet, E.; Chiap, P.; Cohen, N.; Compagnon, P. A.; Dewe´, W.; Feinberg, M.; Lallier, M.; Laurentie, M.; Mercier, N.; Muzard, G.; Nivet, C.; Valat, L. Harmonization of Strategies for the Validation of Quantitative Analytical Procedures. A SFSTP Proposal – Part I. J. Pharm. Biomed. Anal. 2004, 36, 579–586. 22. Huber, Ph.; Nguyen-Huu, J. J.; Boulanger, B.; Chapuzet, E.; Chiap, P.; Cohen, N.; Compagnon, P. A.; Dewe´, W.; Feinberg, M.; Lallier, M.; Laurentie, M.; Mercier, N.; Muzard, G.; Nivet, C.; Valat, L.; Rozet, E. Harmonization of Strategies for the Validation of Quantitative Analytical Procedures. A SFSTP Proposal – Part II. J. Pharm. Biomed. Anal. 2007, 45, 70–81. 23. Huber, Ph.; Nguyen-Huu, J. J.; Boulanger, B.; Chapuzet, E.; Cohen, N.; Compagnon, P. A.; Dewe´, W.; Feinberg, M.; Laurentie, M.; Mercier, N.; Muzard, G.; Valat, L.; Rozet, E. Harmonization of Strategies for the Validation of Quantitative Analytical Procedures. A SFSTP Proposal – Part III. J. Pharm. Biomed. Anal. 2007, 45, 82–86. 24. Feinberg, M. Validation of Analytical Methods Based on Accuracy Profiles. J. Chromatogr. A 2007, 1158, 174–183. 25. Rozet, E.; Hubert, C.; Ceccato, A.; Dewe´, W.; Ziemons, E.; Moonen, F.; Michail, K.; Wintersteiger, R.; Streel, B.; Boulanger, B.; Hubert, Ph. Using Tolerance Intervals in Pre-Study Validation of Analytical Methods to Predict In-Study Results. The Fit-forFuture-Purpose Concept. J. Chromatogr. A 2007, 1158, 126–137. 26. Rozet, E.; Ceccato, A.; Hubert, C.; Ziemons, E.; Oprean, R.; Rudaz, S.; Boulanger, B.; Hubert, Ph. Analysis of Recent Pharmaceutical Regulatory Documents on Analytical Method Validation. J. Chromatogr. A 2007, 1158, 111–125. 27. Dewe´, W.; Govaerts, B.; Boulanger, B.; Rozet, E.; Chiap, P.; Hubert, Ph. Using Total Error as Decision Criterion in Analytical Method Transfer. Chemom. Intell. Lab. Syst. 2007, 85, 262–268. 28. Gonza´lez, A. G.; Herrador, M. A. Accuracy Profiles from Uncertainty Measurements. Talanta 2006, 70, 896–901. 29. Rebafka, T.; Cle´menc¸on, S.; Feinberg, M. Bootstrap-Based Tolerance Intervals for Application to Method Validation. Chemom. Intell. Lab. Syst. 2007, 89, 69–81. 30. Fernholz, L. T.; Gillespie, J. A. Content-Correct Tolerance Limits Based on the Bootstrap. Technometrics 2001, 43 (2), 147–155. 31. Cowen, S.; Ellison, S. L. R. Reporting Measurement Uncertainty and Coverage Intervals Near Natural Limits. Analyst 2006, 131, 710–717. 32. Schouten, H. J. A. Sample Size Formulae with a Continuous Outcome for Unequal Group Sizes and Unequal Variances. Stat. Med. 1999, 18, 87–91. 33. Lehmann, E. L. Testing Statistical Hypothesis; Wiley & Sons: New York, 1959. 34. Schuirmann, D. J. A Comparison of the Two One-Sided Tests Procedure and the Power Approach for Assessing the Equivalence of Average Bioavailability. J. Pharmacokinet. Biopharm. 1987, 15, 657–680. 35. Mehring, G. H. On Optimal Tests for General Interval Hypothesis. Commun. Stat. Theory Methods 1993, 22 (5), 1257–1297. 36. Brown, L. D.; Hwang, J. T. G.; Munk, A. An Unbiased Test the Bioequivalence Problem. Ann. Stat. 1998, 25, 2345–2367. 37. Munk, A.; Hwang, J. T. G.; Brown, L. D. Testing Average Equivalence. Finding a Compromise Between Theory and Practice. Biom. J. 2000, 42 (5), 531–552. 38. Hartmann, C.; Smeyers-Verbeke, J.; Penninckx, W.; Vander Heyden, Y.; Vankeerberghen, P.; Massart, D. L. Reappraisal of Hypothesis Testing for Method Validation: Detection of Systematic Error by Comparing the Means of Two Methods or of Two Laboratories. Anal. Chem. 1995, 67, 4491–4499. 39. Limentani, G. B.; Ringo, M. C.; Ye, F.; Bergquist, M. L.; McSorley, E. O. Beyond the t-Test. Statistical Equivalence Testing. Anal. Chem. 2005, 77, 221A–226A. 40. Kuttatharmmakull, S.; Massart, D. L.; Smeyers-Verbeke, J. Comparison of Alternative Measurement Methods: Determination of the Minimal Number of Measurements Required for the Evaluation of the Bias by Means of Interval Hypothesis Testing. Chemom. Intell. Lab. Syst. 2000, 52, 61–73. 41. Andre´s Martin, A.; Luna del Castillo, J. D. Bioestadı´stica para las ciencias de la salud; Norma-Capitel: Madrid, 2004. 42. Wellek, S. Testing Statistical Hypotheses of Equivalence; Chapman & May/CRC Press LLC: Boca Raton, FL, 2003. 43. Ortiz, M. C.; Herrero, A.; Sanllorente, S.; Reguera, C. The Quality of the Information Contained in Chemical Measures (electronic book); . Servicio de Publicaciones Universidad de Burgos: Burgos, 2005 Available on. http://web.ubu.es/investig/grupos/ cien_biotec/QA4/index.htm. 44. D’Agostino, R. B.; Stephens, M. A.; Eds.; Goodness-of-Fit Techniques. Marcel Dekker Inc.: New York, 1986. 45. Moreno, E.; Giro´n, F. J. On the Frequentist and Bayesian Approaches to Hypothesis Testing (with discussion). Stat. Oper. Res. Trans. 2006, 30 (1), 3–28. 46. Scheffe´, H. The Analysis of Variance; Wiley & Sons: New York, 1959. 47. Anderson, V. L.; MacLean, R. A. Design of Experiments. A Realistic Approach; Marcel Dekker Inc.: New York, 1974. 48. Milliken, G. A.; Johnson, D. E. Analysis of Messy Data: Designed Experiments; Wadsworth Publishing Co, Belmont, NJ, 1984; Vol. I. 49. Searle, S. R. Linear Models; Wiley & Sons, Inc: New York, 1971. 50. Youden, W. J. Statistical Techniques for Collaborative Tests; Association of Official Analytical Chemists: Washington, DC, 1972. 51. Kateman, G.; Pijpers, F. W. Quality Control in Analytical Chemistry; Wiley & Sons: New York, 1981. 52. Kuttatharmmakull, S.; Massart, D. L.; Smeyers-Verbeke, J. Comparison of Alternative Measurement Methods. Anal. Chim. Acta 1999, 391, 203–225. 53. Hampel, F. R.; Ronchetti, E. M.; Rousseeuw, P. J.; Stahel, W. A. Robust Statistics. The Approach Based on Influence Functions; Wiley-Interscience: Zurich, 1985. 54. Huber, P. J. Robust Statistics; Wiley & Sons: New York, 1981. 55. Thompson, M.; Wood, R. J. Assoc. Off. Anal. Chem. Int. 1993, 76, 926–940.
74 Quality of Analytical Measurements: Statistical Methods for Internal Validation 56. Sanz, M. B.; Ortiz, M. C.; Herrero, A.; Sarabia, L. A. Robust and Non Parametric Statistic in the Validation of Chemical Analysis Methods. Quı´m. Anal. 1999, 18, 91–97. 57. Garcı´a, I.; Sarabia, L.; Ortiz, M. C.; Aldama, J. M. Usefulness of D-optimal Designs and Multicriteria Optimization in Laborious Analytical Procedures. Application to the Extraction of Quinolones from Eggs. J. Chromatogr. A 2005, 1085, 190–198. 58. Garcı´a, I.; Sarabia, L.; Ortiz, M. C.; Aldama, J. M. Robustness of the Extraction Step When Parallel Factor Analysis (PARAFAC) is Used to Quantify Sufonamides in Kidney by High Performance Liquid Chromatography-Diode Array Detection (HPLC-DAD). Analyst 2004, 129 (8), 766–771. 59. Massart, D. L.; Vandeginste, B. G. M.; Buydens, L. M. C.; de Jong, S.; Lewi, P. J.; Smeyers-Verbeke, J. Handbook of Chemometrics and Qualimetrics: Part A; Elsevier: Amsterdam, 1997. 60. Abramowitz, M.; Stegun, I. A. Handbook of Mathematical Functions; Government Printing Office, 1964. 61. Meier, P. C.; Zu¨nd, R. E. Statistical Methods in Analytical Chemistry, 2nd ed.; Wiley & Sons: New York, 2000. 62. Johnson, N.; Kotz, S. Distributions in Statistics: Continuous Univariate Distributions – 2; Wiley & Sons: New York, 1970; p 191 (Equation (5)). 63. Evans, M.; Hastings, N.; Peacock, B. Statistical Distributions, 2nd ed.; Wiley & Sons: New York, 1993; pp 73–74. 64. Statistics Toolbox for use with Matlab, version 5.3, The MathWorks, Inc., 2006.
Quality of Analytical Measurements: Statistical Methods for Internal Validation
Biographical Sketches
Marı´a Cruz Ortiz received her Ph.D. in Chemistry from the University of Valladolid (Spain) in 1988. She is a member of the staff of Analytical Chemistry in the University of Burgos (Faculty of Sciences) since 1989. She does research and teaching on Analytical Chemistry and Chemometrics. Her research activity has focused on experimental design, optimization, pattern recognition, quality assurance, and validation of analytical methods according to official regulations and multivariate and/or multiway regression models. All of these applied to problems in Food Chemistry, typification, etc. has resulted about 100 papers. She is at present the head of an active research group, the Chemometrics and Qualimetrics group of the University of Burgos.
Luis A. Sarabia received his Ph.D. in Statistics from the University of Valladolid (Spain) in 1979. Since 1974, he is teaching Statistics and Mathematics mostly to graduate and postgraduate students of Chemistry. At present, his research is centred on Chemometrics as a member of the Chemometrics and Qualimetrics group of the University of Burgos. His research activities include development of software and implementation of nonparametric and robust statistical methods, genetic algorithms, neural networks, etc. He is also involved in multivariate/multiway regression methods, methodology of the experimental design, quality assurance, and validation. He is the author of about a hundred papers on these matters.
75
76 Quality of Analytical Measurements: Statistical Methods for Internal Validation
Ma Sagrario Sa´nchez received her Ph.D. in Mathematics from the University of Valladolid in 1997. She is working at the University of Burgos since 1991 and as a member of the permanent staff since 2002. The teaching activities are mostly directed to students of the degree in Chemistry and Science and Food Technology, and postgraduate courses. She is also a permanent member of the Chemometrics and Qualimetrics Group since its foundation. Her main research activities are developed into the areas of interest of the group, which include modeling and analysis of n-way data, modeling of categories, design of experiments, optimization, etc. by using classical methods as well as computationally intensive methods (such as neural networks or evolutionary algorithms).
Ana Herrero, after completing her undergraduate studies in Chemistry from the University of Valladolid (Spain) in 1991, received her Ph.D. at the University of Burgos (Spain) in 1996. She is working at the University of Burgos since 1992, teaching and researching in Analytical Chemistry. Her research involves Chemometrics and she is a member of the Chemometrics and Qualimetrics group of the University of Burgos. Experimental design, optimization, pattern recognition, multivariate regression analysis, quality assurance, and validation of analytical methods are among the fields she is working on.
1.03
Proficiency Testing in Analytical Chemistry
M. Thompson, University of London, London, UK ª 2009 Elsevier B.V. All rights reserved.
1.03.1 1.03.1.1 1.03.1.2 1.03.1.3 1.03.2 1.03.2.1 1.03.2.2 1.03.2.3 1.03.2.4 1.03.2.5 1.03.2.6 1.03.2.7 1.03.2.8 1.03.2.9 1.03.2.10 1.03.3 1.03.3.1 1.03.3.2 1.03.3.3 1.03.4 1.03.4.1 1.03.4.2 References
Overview of Proficiency Testing in Analytical Chemistry Introduction: The Purpose and Organization of Proficiency Testing Proficiency Testing in the Context of Analytical Quality Assurance The Development of Proficiency Testing and Normative Documents z-Scoring Requirements for Scoring Assigned Values and Their Uncertainties Robust Statistics Kernel Densities Mixture Models Choice of p in Calculating z-Scores Interpretation and Use of z-Scores Scope of Proficiency Testing and Problems with z-Scoring Skewed Data Sets Lognormal Data Sets Validation of Test Materials Introduction Testing for Homogeneity Testing for Stability Further Information from Proficiency Testing Results Review of the Operation of a Scheme Information about Analytical Methods
Symbols Fi; j h HA H0 H15
k l
m n N(0, 1) pi
the F statistic with i and j degrees of freedom. smoothing parameter in estimation of a kernel density. an alternative hypothesis. a null hypothesis. the ‘Huber H15 proposal’ for estimating simultaneously a robust mean and variance. multiplier for a standard deviation to define various confidence intervals. correction factor for a variance that accommodates downweighting of some observations. index for an iterative procedure. number of observations in a dataset. the standard normal deviate. the proportion of the ith component in a mixture model.
Pˆ i xj uA var( ) x x˜ i
xA xL y z zL 2f ˆ i
78 78 79 79 79 79 80 80 81 84 85 87 88 89 90 90 90 90 93 93 93 94 94
the likelihood of datum j belonging to component i of a mixture model. uncertainty on an assigned value. the variance of a variable. result of an analytical measurement. ‘pseudo-value’ used in the calculation of a robust statistic. The index i refers to a stage in an iterative procedure. assigned value a limiting value for arbitrarily dichotomizing a metric. a probability density. z-score defined by z ¼ ðx – xA Þ=p a z-score based on a customised target value. chi-squared distribution with f degrees of freedom. estimate, usually robust, of a sample mean. The index i refers to a stage in an iterative procedure.
77
78 Proficiency Testing in Analytical Chemistry
all
an
ˆ i
a standard deviation regarded as the maximum allowable between-sample component in a test for sufficient homogeneity in a proficiency test material. within-sample (repeatability) component of standard deviation in a randomized replicated experiment. estimate, usually robust, of a sample standard deviation. The index i refers to a stage in an iterative procedure.
f p ˆ rob sam
ðÞ
a target value based on fitness for purpose. standard deviation for proficiency (informally, ‘target value’). a robust estimate of a sample standard deviation. between-sample component of standard deviation in a randomized replicated experiment. the standard normal density.
1.03.1 Overview of Proficiency Testing in Analytical Chemistry 1.03.1.1
Introduction: The Purpose and Organization of Proficiency Testing
Since 1980, proficiency testing has grown from a small collection of schemes organized on a self-help basis to a minor industry, for the most part with third-party accreditation, in which nearly all analytical laboratories participate. As a result, proficiency testing practice has been subjected to a deep scrutiny, and procedures have been studied, refined, and documented. Protocols relevant to the conduct of schemes have been issued by ISO (International Organization for Standardization), IUPAC (International Union of Pure and Applied Chemistry), ILAC (International Laboratory Accreditation Cooperation), and AOAC International (Association of Analytical Communities International). Regardless of all this activity, the basic plan for proficiency testing has remained unchanged. Proficiency testing in analytical chemistry laboratories comprises the regular distribution, by the scheme provider, of portions of an effectively homogeneous material for analysis in the participating laboratories by an unspecified method. The concentration of the analyte is unknown to the participant at the time of analysis. The participant reports the result to the scheme provider by a deadline. The scheme provider then issues a report containing the preferred value of the analyte concentration, a summary of the results of all the participants, and a score that gives an indication of the accuracy achieved by each individual result. This sequence of events, known as a ‘round’, is repeated at regular intervals several times per year. A round can comprise several test materials and call for the determination of a number of analytes per material. The participant usually pays for this service. The primary purpose of proficiency testing is to check the accuracy of the results obtained by a laboratory and thereby allow the participant to detect and subsequently remedy faulty procedures. The scoring should be designed solely to support that requirement. The basis of proficiency testing is therefore the distribution of materials typically encountered, and the use by the participant of routine methods, so that the outcome is, as closely as possible, representative of normal working practice. Ideally, the proficiency test material should be analyzed as part of a routine run. (A ‘run’ is a set of analyses carried out under repeatability conditions, i.e., with the same analyte, same method, same equipment and analyst, and in a short period of time, so that factors affecting accuracy remain constant.) There are, however, secondary uses of proficiency testing scores that tend to undermine this ethos. Accreditation agencies require laboratories to participate in a proficiency test if a relevant one is available, and may take an interest in the scores obtained and the documented efforts of the laboratory to reduce notable inaccuracies. Moreover, laboratories are often asked for an account of their recent scores, so that potential customers can gage their efficiency. As a result, there is an undue pressure on participants to excel, and this may result in proficiency test materials receiving special (rather than routine) treatment. Such practice is contrary to the ethos of proficiency testing and must be avoided by participants. Moreover, scheme providers and accreditation agencies should take steps they can to discourage it. It is easy to imagine that these problems could be circumvented by the distribution of proficiency test materials that were disguised as routine samples.
Proficiency Testing in Analytical Chemistry
79
But it only takes a moment’s thought to see that organizing such a scheme would present considerable difficulty and be prohibitively expensive on the scale required for analytical chemistry.
1.03.1.2
Proficiency Testing in the Context of Analytical Quality Assurance
Aspects of quality assurance bearing directly on the accuracy of analytical results can be considered under four principal headings that should be considered as a sequence: 1. Fitness for purpose – specifying appropriate accuracy. 2. Method validation – determining the accuracy that can be obtained in a laboratory and checking that it fulfills the fitness-for-purpose specification. 3. Internal quality control – ensuring that the accuracy obtained at validation time is maintained in every successive run of analysis. 4. Proficiency testing – regular checking that the previous steps are fulfilling their purpose. Proficiency testing is essentially an occasional check on the whole analytical system, where it has the special advantage of allowing participants to compare their results with those of peer laboratories and with an external reference. It should not be seen as an alternative to the internal quality control, which should be carried out in every run of analysis. There is an important corollary here: an unexpected inaccuracy in a proficiency test means that the participant’s validation procedure and/or internal quality control are faulty as well as the analytical method used. Proficiency testing is not a method for training analysts, for certifying reference materials, for validating analytical methods, nor for diagnosing problems, although it can sometimes throw some light on these aspects. 1.03.1.3
The Development of Proficiency Testing and Normative Documents
Proficiency testing developed independently in various application sectors of analytical chemistry, initially in biomedical analysis, then in food analysis and later in environmental analysis. (Schemes in the industrial sector undoubtedly existed from an early date, but are usually not openly documented.) The first attempt to produce a universally applicable format was the International Harmonized Protocol1 recognized by IUPAC, ISO, and AOAC International. This was followed by an ISO Guide to proficiency testing generally (i.e., not just chemical testing)2 and a book dealing with putting proficiency testing into practice in analytical chemistry.3 ILAC Guidelines for assessing the competence of proficiency test providers appeared in 2000 and is currently (2008) under revision.4 An ISO Standard on statistical methods for proficiency testing appeared in 20055 and a revised International Harmonized Protocol for analytical chemistry in 2006.6 There are now a large number of schemes available, many of them accredited.7
1.03.2 z-Scoring 1.03.2.1
Requirements for Scoring
Most schemes convert a participant’s result x into a score reflecting its accuracy. The ideal score would be universal: a particular score of (say) 2.5 should always convey the same information about accuracy. A score that approaches the ideal is the z-score, given by z¼
x – xA p
ð1Þ
where p is the ‘standard deviation for proficiency’ (known informally as the ‘target value’), which is provided in advance by the scheme provider. The ‘assigned value’ xA is the best available estimate of the true concentration of the analyte. The provider releases the value of xA only after the deadline for submission of results. This assigned value (with a suitably small uncertainty) could conceivably be found by a procedure comparable in effort and cost to the certification of a reference material. Alternatively, it could be a value
80 Proficiency Testing in Analytical Chemistry
determined by a national metrological reference laboratory by a procedure such as isotope dilution mass spectrometry. Both these options are usually too costly for use in routine proficiency testing. Therefore, the assigned value is, at present, nearly always identified as the consensus of the participants’ results, although this may be inadequate if there is a general bias in the results because of the widespread use of a biased analytical method. Consequently, there is considerable emphasis on the identification of a ‘consensus’. The value of p should represent fitness for purpose and should be determined so that z-scores within the range 2 are regarded as fit, whereas scores outside the range 3 suggest that some investigation is called for, where necessary followed by remedial action. More detail on selecting a suitable value for p is given in Section 1.03.2.6. Some metrologists advocate a scoring system in which the score is based not only on the participant’s result, but also on the estimated uncertainty of the individual result. The premise of the idea is that analysts should always report (or, at least, be able to report) their uncertainty. Such a scoring system suffers from a number of problems and is not recommended by the IUPAC Harmonized Protocol. In most schemes, the provider specifies the uncertainty that is broadly fit for purpose in the application sector, and it is the task of the participants to provide a result falling within that uncertainty. The required accuracy is explicit in the scheme, and participants’ estimates of uncertainty are irrelevant. If a different criterion of uncertainty is needed of a participant by a particular customer, the analyst should calculate a special score for that specific purpose, based on the customer’s criterion, as discussed in Section 1.03.2.5: however, that is not a legitimate concern of a proficiency testing scheme.
1.03.2.2
Assigned Values and Their Uncertainties
In the determination of an assigned value, attention is usually focused on the exact meaning of ‘consensus’ and how best to estimate it from the participants’ results. The usual estimates of central tendency (mean, median, and mode) are all potential candidates for consensus, but each presents some interesting problems. The arithmetic mean is clearly unsatisfactory because results of a round nearly always contain outliers, often extreme outliers. A robust mean (including the median) is often suitable, but its utility depends on the distribution of results (outliers aside) being largely symmetric. In practice, results from a round of a proficiency test are often somewhat skewed and sometimes multimodal. The dominant mode is an attractive choice intuitively, but it requires a complicated procedure to quantify the mode from the data of limited digit resolution sampled from an underlying continuous distribution. Clearly, the choice of estimator calls for inspection of the data as a graphical presentation and the exercise of judgment. (Accreditation agencies are apt to question this call for judgment: they prefer to see a rigidly applied procedure. However, the possibilities inherent in data from a round of a proficiency test are so large that it is impracticable to lay down adequately a fixed protocol for determining a consensus. Hence, this is the appropriate time to emphasize that statistics is a tool designed to assist judgment, not to replace it. Accordingly, all proficiency schemes must have access to a statistical expert.) The uncertainty on the consensus uA is also of interest: if the uncertainty is too large, the z-scores produced will reflect this variation to an undue extent. Thus, uA < p =3 is preferred, so that the z-scores are affected to a rffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi pffiffiffiffiffiffiffiffiffiffi maximum extent of 2p þ uA2 =2p ¼ 10=9, about 5% relative. This criterion can usually be achieved if there are more than about 15 participants reporting valid results. Some metrologists advocate the use of a qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi modified score defined by ðx – xA Þ= 2p þ uA2 , which allows scoring to proceed even when uA is high. This strategy has the defect that it allows the calculation of apparently meaningful scores when an unduly large part of the variation is derived from uA and not from the variation in the accuracy of the participants: there is no indication in the modified score that suggests the possibility of such a problem.
1.03.2.3
Robust Statistics
The Harmonized Protocol recommends the use of a robust mean as consensus when the results appear to be largely symmetric, outliers aside. This is the most frequent outcome of a round of a test. An example is shown in
Proficiency Testing in Analytical Chemistry
81
Frequency
15
10
5
0 20
24 21 22 23 Result for alumina (% mass fraction)
25
Figure 1 Outcome of a round of a proficiency test (Al2O3 in a rock) where the participants’ results (histogram) approximate to a slightly contaminated sample from a normal distribution. The parameters of the fitted normal distribution (red line) were estimated by the H15 procedure.
Figure 1. The H15 estimates of Huber are recommended but other robust estimates may well be suitable.8 Robust statistics accommodate extreme results by downweighting them. They are calculated by a procedure rather than a formula. The H15 procedure is as follows: 1. Obtain initial estimates ˆ 0 ; ˆ 0 2 of the mean and variance. These could be simple mean and variance or (better) estimates based on the median and 1.5 MAD (median absolute difference from the median). 2. Select a value 1 k 2 depending on the expected proportion of outliers. The usual value of k ¼ 1.5 is suitable for 5–10% of outliers. 3. Transform the raw results xi , i ¼ 1, . . ., n, to ‘pseudo-values’ x˜i as follows, where m ¼ 0 initially: 8 xi þ kˆ m ; xi > ˆ m þ kˆ m > > < x˜i ¼ xi – kˆ m ; xi < ˆ m – kˆ m > > : xi ; ˆ m – kˆ m < x i < ˆ m þ kˆ m :
ð2Þ
4. Calculate ˆ mþ1 ¼ meanðx˜Þ; ˆ 2mþ1 ¼ l varðx˜Þ, where l is a factor determined by the value of k. 5. Increment m in the formula for x˜ and repeat the iteration until the statistics have converged to a required degree of accuracy. Robust estimation also providesp a ffiffimethod of estimating uncertainty on the consensus. The standard error of an ffi arithmetic mean is given by = n, so it is convenient to regard the standard uncertainty of the consensus as pffiffiffi uA ¼ ˆ rob = n. This value is usually a slight underestimate, as robustification is equivalent to downweighting pffiffiffi some of the data, so that n is too great a denominator. However, a more accurate value can be obtained by the bootstrap. In practice, there is no consequential difference between the two estimates.9 The main purpose of uA is the comparison with p =3, so a rough estimate will suffice. An Excel add-in and a Minitab macro for calculating H15 robust statistics can be found in the AMC web site.10
1.03.2.4
Kernel Densities
Kernel densities provide a useful means of judging whether a set of results from a round of a proficiency test is unimodal or multimodal and, in either case, estimating the values of the identified mode, modes, or shoulders. The judgment of whether a data set is multimodal depends on having a good graphical representation of the distribution of the data: the number of modes apparent depends on aspects of the presentation. A dotplot will show many modes. A histogram will show a number depending on the width of the bins used. The automatic choice of width provided by statistical packages may prove misleading in this context. In a sense, there is no
82 Proficiency Testing in Analytical Chemistry
‘true’ number of modes – it depends on the degree of smoothing, which in turn depends on what the scientist expects the data to be. However, the kernel density has been found to be serviceable in proficiency testing because, in that instance, there is a rational approach to the choice of smoothing parameter.11 A kernel density is produced by replacing each data point xi on the line by a probability density function (the kernel), usually the normal distribution Nðxi ; h2 Þ, where h is a parameter that controls the degree of smoothing. The densities are added together at each point on the line and normalized to unit area to give the kernel density, n x – x 1 X i
fˆ¼ h nh i¼1
ð3Þ
Density
where ðÞ is the standard normal density.12 The procedure is illustrated in Figures 2 and 3. The kernel density is a smooth representation of the data, but we can estimate it only at discrete values of x. The density can be calculated directly by adding the individually calculated kernels at each point. However, as the kernel density is a convolution of the data with a normal distribution, it can also be obtained via the Fourier transform.10 The number of modes shown depends on the selected value of h. The kernel density is more disperse than the original data (as to be expected with smoothing) with a variance varðx Þ þ h2 . The inflation is not at all serious if h is less than about 0.75 of the original standard deviation of x, when it amounts to 25%. This points to a suitable benchmark for h in proficiency testing: because p is set to represent a dispersion that the data should conform to, we can use h ¼ 0.75p as a starting value to investigate the parameter space.
0.0 Measurement axis
Density
Figure 2 A kernel density (red line) derived from four data points by the addition of normal kernels (black lines) with standard deviation h.
0 Measurement axis Figure 3 A kernel density (red line) derived from the same data as shown in Figure 2, but with kernels of standard deviation 2h.
Proficiency Testing in Analytical Chemistry
83
An Excel add-in and a Minitab macro for producing kernel densities can be found in the AMC Website.10 Figures 4–6 illustrate an example of this procedure applied to a proficiency test for polyunsaturated fatty acids in cooking oil. The data can be downloaded from the AMC Website.10 The original data displayed as a dotplot (Figure 4) are difficult to interpret in terms of modality, but show an unexpected large robust standard deviation of 0.96% mass fraction. A plot of the number of modes observed versus the value of h used to make the kernel density (Figure 5) shows that a two-mode model has a good range of stability. Normal kernel densities with 0.2 h 0.4 (Figure 6) show a marked tendency for two modes, at 39.3 and 40.6. As will be seen (Section 1.03.2.5), this is a useful model of the data. The uncertainties of the modes can be quantified as their standard errors, which can be estimated by the bootstrap.11 However, the latter is a somewhat lengthy process, likely to take several minutes on modern personal computers. The bootstrap standard errors corresponding to the modes of the example data are 0.09 and 0.21, respectively.
39
40 Analytical result (% mass fraction)
41
Figure 4 Dotplot of data from a proficiency test round: polyunsaturated fatty acids in cooking oil.
8
Number of modes
7 6 5 4 3 2 1 0 0.0
0.1
0.2
0.3 0.4 0.5 Value of h
0.6
0.7
0.8
Figure 5 Plot of number of modes vs. h. Data from Figure 4.
0.6
Density
0.5 0.4 0.3 0.2 0.1 0.0 35
36
37
38 39 40 41 Result (% by mass)
42
43
44
Figure 6 Kernel densities obtained by using h values of 0.2 (black line), 0.3 (green line), and 0.4 (red line). Data from Figure 4.
84 Proficiency Testing in Analytical Chemistry
When we have additional information about the various analytical methods used by the participants, it is sometimes possible to identify one of several modes as a consensus. In that circumstance, we need to estimate the analyte concentration where the mode is situated. This can be done by using quadratic interpolation between (say) the six discrete points surrounding the mode on the concentration axis of the kernel density. In the instance illustrated, we know that the two modes originated from the use of two different calibration strategies by participants, only one of which corresponds with the proper definition of the measurand. As a consequence, one can ascertain that the major mode at 39.30 0.09 is the correct consensus. A novel mathematical procedure akin to kernel density estimation, ‘Cofino statistics’, has been advocated for use in proficiency tests.13 Although it provides a good estimate of a consensus, expert opinion is that it suffers from certain difficulties.14 First, it is couched in terms of an unusual notation and, second, the variance associated with the consensus is ambiguous.
1.03.2.5
Mixture Models
Another method of modeling a multimodal data set is to regard it as a mixture of two or more component normal distributions with unknown parameters and in unknown proportions. The ‘EM’ (expectation maximization) algorithm is a rapid and easily coded procedure for estimating the unknown parameters.15–17 The procedure starts with initial estimates of the means (or of component membership), but quite rough estimates are usually good enough to lead to convergence in a small number of iterations. It is often convenient to start with the position of modes (or shoulders) estimated visually from a kernel distribution. The normal mixture model has a density y given by the sum of m separate normal components in proportions pi,i ¼ 1, . . .,m, so that we have y¼
m X
m X
pi ¼ 1
ð4Þ
exp – ðx – i Þ2 =22i pffiffiffiffiffi fi ðxÞ ¼ 2i
ð5Þ
i¼1
pi fi ðxÞ;
i¼1
where fi(x) is the normal distribution
We estimate the parameters in the ‘M-step’ from the data xj , j = 1, . . . ,n, where the weights Pˆi ðx j Þ are likelihoods of datum j belonging to component i: Pn pˆi ¼
j ¼1
Pˆi xj
n
ˆ j ¼1 xj Pi xj
ˆ i ¼ Pn ˆ j ¼1 Pi xj
2 Pn xj – i Pˆi xj j ¼1 for individual variances ˆ i 2 ¼ Pn ˆ j ¼1 Pi ðx j Þ
2 Pm Pn x – Pˆi xj j i i¼1 j ¼1 ˆ 2 ¼ for a pooled variance n Pn
ð6Þ
The parameter estimates for each component are simply the mean ˆ i and variance ˆ i 2 weighted by the likelihood of each data point belonging to that component. We now estimate the weights in the ‘E-step’ as
pˆi f i xj
Pˆi xj ¼ Pm i¼1 pˆi f i xj
ð7Þ
Proficiency Testing in Analytical Chemistry
85
The steps are used alternately until a sufficient degree of convergence is obtained. The algorithm can start at either step. A simple and effective way of starting is to cut the data at the midpoints between the modes of a kernel density. Then for component i, set Pˆi xj ¼ 1 if xj falls in the appropriate range and Pˆi xj ¼ 0 otherwise. The algorithm is then started at the M-step. Alternatively, the algorithm can be started at the Estep using ‘guestimated’ parameters for each component. Goodness of fit of the data to models with different numbers of components can be investigated until a satisfactory but parsimonious model is found. Adding an extra component will always improve the fit but, unfortunately, there is no simple method of determining whether one model is significantly better than another. The likelihood ratio method is regarded as unreliable in this context. A Minitab macro for producing one-dimensional mixture models can be found in the AMC Website.10 We can apply the method to the example data shown in Figures 4–6. This gives us a solution (Figure 7) with two components with the following statistics:
Mean
Pooled standard deviation
Proportion
Standard error
39.29 40.58
0.34 0.34
0.65 0.35
0.07 0.09
The statistics are close to those derived from the kernel density. However, no judgment about the degree of smoothing is required, and the computation is much quicker. The kernel density is by no means superseded, however: it provides a useful first look at the data. That the two-component mixture model comprises a good fit can be judged from a quantile–quantile plot of the data versus the model (Figure 8). If one component is identified (from additional evidence such as methods used by each participant) as comprising ‘correct’ results, the mean ˆ i of that component can be taken as the consensus and its standard error pffiffiffiffiffiffi with the uncertainty. The standard error is given by ˆ i = npi , where ˆ i is the estimated standard deviation, pi the estimated proportion of the ith component, and n the total number of results.
1.03.2.6
Choice of p in Calculating z-Scores
Ideally a specific z-score, say –1.7, should convey the same information about accuracy, irrespective of the identity or concentration of the analyte, the composition of the test material, the method of analysis, or the laboratory in which it is conducted. The ‘target value’ p can therefore be regarded as a scaling (for the ‘error’ (xxA)) that is designed to approach the ideal. Early thoughts on the subject were that p should be identified 0.9 0.8 0.7 Density
0.6 0.5 0.4 0.3 0.2 0.1 0.0 36.6
37.6
40.6 38.6 39.6 Analytical result
41.6
42.6
Figure 7 Two-component normal mixture model (red line), showing separate components (blue lines), from the results of a proficiency test (data from Figure 4).
86 Proficiency Testing in Analytical Chemistry
Quantile of data
1.0
0.5
0.0 0.0
0.5 Quantile of model
1.0
Figure 8 Quantile–quantile plot of proficiency test results and the two-component model (Figure 7).
with a robust standard deviation of the results of a round. For the usual slightly contaminated normal distribution of results, this would provide a set of z-scores that would resemble the standard normal deviate N(0,1), but with somewhat more than 5% of the z-scores falling outside the range 2. That choice of p certainly harmonizes the outcomes of diverse proficiency tests, but provides the wrong information. It has the unfortunate effect of causing a false sense of security: up to 95% of participants will receive an apparently satisfactory score, within the range 2, regardless of whether or not the uncertainties of the results are appropriate for the application. A much better strategy is to equate the value of p with the standard uncertainty that is regarded as fit for purpose. All of the examples in this chapter are based on this principle. The strategy demands a clear definition of fitness for purpose and a methodology for locating the corresponding uncertainty. We can start by noting that an analytical method is not fit for purpose if it is too expensive to use or if its use results in an unduly high proportion of incorrect (and therefore costly) decisions. Experienced analysts are accustomed to balancing these losses as a matter of professional judgment and indeed, in the long term, methods that fall short in either of these aspects will be replaced with more cost-effective methods. We have recently seen that it is possible to employ decision theoretic methods18 to make the judgment more rigorous. To carry out this, we need to find how each cost affecting the customer is determined by the uncertainty of the measurement.19–22 We have to consider both the cost of measurement and the extra cost of an incorrect decision. There is an inverse relationship between the cost of analysis and the accuracy achieved. Indeed, in the absence of specific information to the contrary, it is a good rule of thumb to assume an inverse square relationship. For example, if we need to reduce the uncertainty by a factor of 2, the cost will increase by a factor of 4. The probability of a mistaken decision (for instance, whether the concentration of a toxin in a food exceeds the legal limit) also depends on the uncertainty – the greater the uncertainty, the greater the chance of an incorrect decision. The consequence cost of an incorrect decision may be twofold. Incorrect rejection of a batch of product implies a loss amounting to the cost of production and disposal, or the cost of reprocessing, an already compliant material. Incorrect acceptance of noncompliant material gives rise to the cost of compensation to down-line stakeholders multiplied by the probability of detection. The total long-term loss to the customer is the sum of these two losses, measurement costs and consequence costs, one decreasing and the other increasing in relation to rising uncertainty. The sum must therefore have a minimum cost, and the uncertainty that is fit for purpose (Figure 9). Although some of these contributions may be difficult to cost exactly, a workable value for consequence cost might be the cost of insuring against the occurrence of an incorrect decision. Even when the necessary data are unavailable, the idea provides a useful conceptual framework for fitness for purpose considerations. All of the above presupposes that the scheme provider can specify a value of p that is suitable for a whole application sector, that is, for all of the participants on all occasions. That is not invariably the case. Sometimes, an analyst’s customer may have a fitness criterion that differs from that of the closest relevant scheme. In such circumstances, it would be correct for the participant to calculate modified z-scores zL ¼ ðx – xA Þ=f , using the
87
Cost
Proficiency Testing in Analytical Chemistry
0 0
uf Uncertainty
Figure 9 Cost of measurement (black line) and long-term average cost of incorrect decisions (green line) as a function of uncertainty of measurement. The expectation of total cost (red line) shows a minimum cost defining an uncertainty uf that is ‘fit for purpose’.
scheme’s assigned value xA but a customer-specific fitness criterion f. Such modified z-scores can be interpreted in exactly the same way as normal z-scores.
1.03.2.7
Interpretation and Use of z-Scores
If a participant was performing exactly according to the specified p, the z-scores in the long term would resemble the standard normal deviate z N(0,1).23,24 About 95% of the results would fall in the range 2 and about 0.3% would fall outside 3. The latter outcome would be so unusual that it could safely be regarded as indicating a change, which called for investigation, in the performance of the system. It is therefore sensible to interpret z-scores in terms of N(0,1). Needless to say, z-scores, either from different participants in a round or from a single participant over time, never exactly fulfill N(0,1). Individual participants may operate with any combination of (1) a reproducibility variance greater or less than p, (2) a bias that shifts the center of the distribution, and (3) a heavy tailed distribution. Those participants with a variance greater than p, and/or those with a bias, will produce in the long term more than 5% of scores outside the range 2 and more than 0.3% of scores outside the range 3. (Of course, many of the results from maladapted participants will fall inside these ranges in individual rounds: proficiency testing cannot guarantee to detect questionable performance.) Those with variance less than p and no bias will, of course, produce less than the expected number of suspect scores. This is exactly the desired outcome, so it is correct to interpret scores on the basis of N(0,1), even though we know that that is not a realistic expectation. The point is emphasized here because it is widely misunderstood among analytical chemists. The target value p is meant to prescribe the uncertainty that is fit for purpose, not to describe the results that are produced by the participants, and the normal distribution is nearly always appropriate for this purpose. There are a number of statistically sound methods of combining z-scores to obtain a broader perspective on accuracy, but all require caution in their interpretation. P pffiffiffiffi For our hypothetical ideal participant, the rescaled sum of m normally distributed z-scores z= m is distributed as N(0,1) and can be interpreted like z. It has the disadvantage that large scores of opposite sign could compensate and create (for the unwary) a false impression of satisfactory performance. The sum of the squared z-scores P z2 interpreted as 2m would avoid this problem, but is unduly sensitive to extreme results. If used at all, such combination scores should be restricted to successive scores for the same analyte. Combining scores from several different analytes may enable a persistently poor outcome for a particular analyte to escape detection. A visual presentation of the scores is far more informative and less likely to be misinterpreted. Successive scores from the same analyte can be plotted on a Shewhart control chart with control lines at 2 and 3 and interpreted by using the usual Westgard rules25,26 (Figure 10). Alternatively, a cusum chart or a zone chart27
88 Proficiency Testing in Analytical Chemistry
3 2
z-Score
1 0 –1 –2 –3 0
10 Round number
20
Figure 10 Shewhart control chart applied to successive z-scores produced by a participant. All of the scores are ‘in control’ and apparently call for no action.
Weights 8 3
z-Score
2 1 0 –1 –2 –3
4 2
2
1
9
6 3 4
1
7
1
1 2
1
4 5 6
1 6
2
2
3
4
4 8 2
7 12 Round number
17
Figure 11 Zone chart (or ‘J-chart’) applied to successive z-scores produced by a participant (same data as Figure 10).
could be used to pick up a persistent bias. The useful but relatively little known zone chart (otherwise known as a ‘J-chart’) combines the capabilities of the Shewhart and cusum charts. It is constructed by dividing a standard control chart into zones with widths of one standard deviation each side of the mean (Figure 11). Results as they occur are assigned to the appropriate zone and given a corresponding weight as shown in Figure 11. While successive results fall on the same side of the mean, the weights are cumulated. If the nth result falls on the opposite side of the mean to its predecessor, the cumulated weight is reset to zero before cumulation resumes with the nth weight. When the total weight amounts to 8 or greater, the system is deemed to be out of control. In Figure 11, the chart flags an ‘out-of-control’ condition at observation 7, by virtue of the accumulated J-score of 9 produced by six successive positive z-scores. There is a strong temptation among non-scientists to classify or rank laboratories on the basis of proficiency test scores. This tendency should be resisted, because it can lead to serious misunderstanding. Arbitrarily chosen decision limits such as 3 should not be regarded as class boundaries. Ranking is even more invidious. The rank of a laboratory consistently performing according to fitness for purpose could vary greatly from round to round.
1.03.2.8
Scope of Proficiency Testing and Problems with z-Scoring
Proficiency testing and scoring as described above is limited to measurements on interval or ratio scales. It therefore cannot accommodate results on nominal or ordinal scales, results such as ‘less than xL’ (a limit value), ‘not detected’, or ‘absent’. Although some results of that kind may be clearly fit for purpose, they cannot be
Proficiency Testing in Analytical Chemistry
89
converted into z-scores. Some providers have adopted the expedient of replacing such results with numerical results such as zero, or one-half of the detection limit, but this practice has the danger of biasing the data set. Attempts at a Bayesian interpretation of these low results have not been satisfactory so far because the outcome is very sensitive to any realistic prior distribution. Other schemes simply reject such results and do not convert them into scores. Attempts to devise a hybrid scoring system that can accommodate both types of data have not been successful so far. An extreme version of this problem occurs in proficiency tests where the result is qualitative, that is, the result is either ‘present’ or ‘absent’. Scoring in such tests comprises little more than tabulating the proportions of true negative, false negative, true positive, and false positive. Needless to say, the information content of a single result is small compared with that of a result on an interval or ratio scale, and many results have to be accumulated for a useful conclusion to be drawn.
1.03.2.9
Skewed Data Sets
Sometimes in a round of a proficiency test, the results are unimodal but display a strong positive skew. To ensure the correct treatment of such results, it is essential to understand the underlying causes of the skew. One commonly encountered cause occurs when some or all of the participants are reporting results near the detection limit of their analytical method. Results less than the detection limit are often not reported or, if they are reported, not displayed by the scheme provider. As the participants’ methods will have a variety of detection limits, and the results are low-censored, the outcome is a positively skewed distribution. An example is shown in Figure 12. Another contribution to skewed results near the detection limit is encountered when the analytical result is the quotient of two intermediate results, each with a substantial relative standard deviation (RSD). This sometimes happens when the results are corrected for recovery. Even if the two parent distributions are normal, the quotient will have a positive skew. With parent RSDs up to about 15%, the skew is mild and can be safely ignored. With parent RSDs of 20%, however, the skew is pronounced and, as resulting RSD will be around 30%, the final results will be at the detection limit and the data set will be censored to some degree. Scheme providers often circumvent these problems by ensuring that the concentration of the analyte is more than about 5 times the detection limit expected from the majority of participants. However, measuring nearlimit concentrations (even those of exactly zero) is a legitimate concern for many analysts – a substantial proportion of the work involves just such materials – so this solution is hardly supportive of participants needs. Unfortunately, no simple general solution has emerged. Under these circumstances, it is tempting for providers to log-transform positively skewed data sets and then interpret the outcome on the basis of a normal distribution. That action is inappropriate in most instances, because the distributions are not lognormal and, indeed, may contain negative values before they are censored. In addition, action limits based on the lognormal distribution will be asymmetric in the original (untransformed) domain of the results, and impute a different importance to low and high errors of equal magnitude. High errors would seem to be diminished in importance, whereas low errors would tend to attract unduly high scores. It has to be mentioned that a satisfactory method of dealing with low results and the resulting skews has yet to be devised. Although research in this area is being conducted, it is not clear that an acceptable solution will emerge. If participants reported the uncensored result and its true uncertainty, all would be clear. But this is unlikely to happen. Even the definition of uncertainty in this area is ambiguous. As in many other instances, a judgment call is required from the scheme provider. Such judgments are credible only if both analytical and statistical expertise are available.
0
1 2 Result for niobium (ppm, mass fraction)
Figure 12 Dotplot of the results from a round of a proficiency test (niobium in a rock) showing a marked skew. In this instance, the skew probably resulted from the use of methods with poor detection by some participants.
90 Proficiency Testing in Analytical Chemistry
1.03.2.10
Lognormal Data Sets
We must also consider those unusual instances (in chemical measurements at least) where the measurement process is such that the errors are predominantly multiplicative. The quantitative measurement of genetically modified food constituents via the polymerase chain reaction is one such process. This process naturally gives rise to a lognormal distribution of results (although the binomial and normal distributions also make a (probably minor) contribution to the final result). A study of two such schemes has concluded that log-transformation before the formation of z-scores is the correct approach in that case.28
1.03.3 Validation of Test Materials 1.03.3.1
Introduction
Test materials distributed by the scheme should be of a matrix typically encountered in the relevant analytical sector and, in the long term, should contain analyte concentrations representing both acceptable and unacceptable levels. Materials must be sufficiently stable and homogeneous, so that differences in composition among the units received by the participants are insignificantly small in comparison with the uncertainty of the measurement process. Such a requirement calls for regular (or at least occasional) testing by the scheme provider. Because of this restriction, it is usually impossible to distribute materials in their native state. For example, rock samples have to be distributed as a fine powder to ensure a sufficient degree of homogeneity, and vegetable samples as a puree. 1.03.3.2
Testing for Homogeneity
To ensure that participants receive portions of test material with only inconsequential differences in composition (a condition known as ‘sufficient homogeneity’), it is necessary to test the material after it has been divided into containers for distribution. As it is universally recognized that it is the analyst’s responsibility to take a representative test portion from the laboratory sample, there is no need to test for heterogeneity within individual containers, only between containers. The test comprises the selection of 10 or more distribution units at random from the complete set and the analysis of each in duplicate, in a random sequence, under repeatability conditions, by a method with appropriate precision (i.e., with a standard deviation an < 0:3p ; the results must be recorded with sufficient digit resolution to support the statistical analysis). The between-unit variance sam is then found by analysis of variance and compared with a maximum-allowable variance all to test whether it is significantly greater. Two precautions, however, are needed to make this test meaningful. First, the randomization has to be strictly conducted, so that the data contain no systematic features that could falsely enhance or reduce the appearance of heterogeneity. It is worthwhile to give the analysts explicit instructions on how to do that: experience has shown that implicit instructions are often misunderstood or ignored. It is also necessary to check every data set for systematic effects by inspection of a simple display of result versus sample number. Undesirable features that may invalidate the test are a trend or discontinuity (Figure 13), bias between the first and second results (Figure 14), or insufficient digit resolution of data (Figure 15). Second, a statistical approach more sophisticated than the simple F-test of the hypothesis H0 : sam ¼ 0 versus HA : sam > 0
ð8Þ
is required to avoid an undue probability of rejection of good test material or rejection of the whole data set as incapable of supporting the required decision.29 The problem here is that, given an analytical method of sufficiently high precision, almost any material can be shown to be significantly heterogeneous. This does not necessarily mean, however, that the material is unsuitable for use in the proficiency test. In addition, as homogeneity testing is expensive (often requiring as much as one-third of the whole cost of the proficiency test), it is desirable to minimize the false rejection rate. For example, if sam is acceptable but only slightly less
91
Result
Proficiency Testing in Analytical Chemistry
1
2
3
4 5 6 7 8 9 Sample identity number
10 11 12
Result
Figure 13 Data from a homogeneity test showing the first (red) and second (green) results from each distribution unit. The data show a trend, probably because the data were analyzed in the order of their code numbers rather than in a random order. This feature would tend to suggest falsely that the material was significantly heterogeneous.
1
2
3
4 5 6 7 8 9 Sample identity number
10 11 12
Result
Figure 14 Data from a homogeneity test showing the first (red) and second (green) results from each distribution unit. The data show a bias between the first and second results, indicating that the analysis was conducted in two sequences, with each unit analyzed once in each sequence. This feature would tend to disguise significant heterogeneity.
1
2
3
4 5 6 7 8 9 Sample identity number
10 11 12
Figure 15 Data from a homogeneity test showing a single outlying result, normally attributable to an analytical mistake.
92 Proficiency Testing in Analytical Chemistry
than all, about one-half of all sufficiently homogeneous materials would be rejected. To deal with these problems, it is necessary to test a different hypothesis, namely H0 : sam all versus HA : sam > all
ð9Þ
where all is the maximum-allowable value, determined independently by the practical requirements of the proficiency test (i.e., not derived from the data). To carry out this test, we calculate ˆ an and ˆ sam in the normal way from the mean squares in the analysis of variance, and reject the material when ˆ 2sam >
2all 2n – 1 ˆ 2an Fn – 1;n – 1 þ 2 n–1
ð10Þ
1. 2n1 is the value exceeded with a probability of 0.05 by a chi-squared random variable with n 1 degrees of freedom. 2. Fn1, n is the value exceeded with a probability of 0.05 by a random variable with the F-distribution with n 1 and n degrees of freedom. 3. n is the number of units analyzed in duplicate. (Duplicate analysis is assumed in Equation (10).) 4. A suitable value for the allowable variance is 2all 2p =9 (i.e., all p =3 where p is the target value), which ensures that any heterogeneity between distribution units has a negligible effect on the z-scores.
Result
As each homogeneity test requires at least 20 replicate analyses, the occurrence of an analytical outlier in a data set is not rare. The first edition1 of the Harmonized Protocol stated that no outlying data should be excluded from the analysis of variance, because outliers comprised valid evidence of heterogeneity. However, a single analytical outlier (Figure 15) has the (perhaps) unexpected effect of making rejection of the null hypothesis less probable, because it has a bigger effect on the within-sample mean square than the between-sample mean square. Moreover, a single outlier must have occurred because of an analytical lapse, as the analyst should have ensured that both test portions were representative of the distribution unit. More recent opinion, therefore, is that a single outlier should be rejected before the calculation of the mean squares. Cochran’s test for unequal variances at 95% confidence is suitable for this purpose. (There is a potential exception to this rule. Some trace analytes occur only as rare discrete particles containing a high concentration of the analyte, even in ‘homogenized’ material. This could give rise to a valid solitary outlier. An example would be the determination of platinum in an ore.) However, a pair of apparently outlying results from the same distribution unit (Figure 16) must never be rejected: it is a true indication of heterogeneity. The occurrence of two outlying results from different distribution units suggests that the analysis was unreliable and that the whole data set should be discarded.
1
2
3
4 5 6 7 8 9 Sample identity number
10 11 12
Figure 16 Data from a homogeneity test showing a pair of outlying results from a single unit. This is a strong indication of heterogeneity.
Proficiency Testing in Analytical Chemistry
93
It should be noted that homogeneity tests have a low statistical power unless an unrealistically large number of distribution units are tested. The test should therefore be regarded as a screen to ensure that serious problems are avoided. 1.03.3.3
Testing for Stability
Materials distributed in proficiency tests should be sufficiently stable over the period during which the assigned value is designed to be definitive. The term ‘sufficiently stable’ implies that any changes that occur during the relevant period have an inconsequential effect on the interpretation of the results of a round. Normally, the period in question is the interval between preparation of the material and the deadline for return of the results, although the period may be longer if the provider intends subsequently to offer unused distribution units for sale. Before the stability test, the material should be packed in containers in which it is to be distributed. It is not usually necessary or even feasible that the material for every round in a series should be tested. However, each newly considered material/analyte combination should be tested once before it is used in a proficiency test and occasionally thereafter. Stability tests comprise a comparison of the material before and after the ‘appropriate’ treatment. This usually requires a batch of the distribution units to be randomly divided into two equal subsets. The ‘experimental’ subset is subjected to the appropriate treatment, whereas the ‘control’ subset is kept under conditions of maximum stability, for example, low temperatures and low oxygen tension. Ideally, the stability test should involve exposing the experimental subset to the most extreme conditions likely to be encountered during the distribution and storage. More realistically, because of the low power of the statistical test, the experimental subset should be kept under conditions of accelerated decomposition, for example, higher temperatures than normal. After the treatment, the two subsets are analyzed by a method of high precision. Although simple in principle, stability tests must be carefully designed to avoid confounding any change in the material with variation in the efficacy of the analytical method used. Analysis of the control material at the beginning of the test period and experimental material at the end automatically includes any run-to-run analytical difference in the results and may well lead to an incorrect conclusion. The recommended approach is, if at all possible, to analyze the experimental and control subsets together in a random order, within a single run of analysis, that is, under repeatability conditions. Any highly significant difference between the mean results of the two subsets can then safely be regarded as evidence of instability. However, it must be emphasized that such tests are of low statistical power unless inordinately large numbers of test units are involved or highprecision analytical methods are employed. Although, in principle, a distinction must be drawn between statistically significant instability and consequential instability, because of the low statistical power of the test given practicable conditions of testing, a significant result is unlikely. For instance, given an analytical method with repeatability standard deviation an and two subsets of 10 distribution units each, an instability change of a magnitude of an would be found significant in only 56% of the experiments by using the two-sample t-test at 95% confidence. Thus, if it were deemed that a change in the z-score of 0.05 would be inconsequential, then an instability amounting to a change in analyte concentration of 0.05p could be tolerated. However, an analytical method with an 0:05p is unlikely to be available.
1.03.4 Further Information from Proficiency Testing Results Although proficiency tests are designed solely for the immediate benefit of individual participants, there are certain further uses of the resultant data that can be used for more general purposes, without requiring changes that would compromise the primary purpose. 1.03.4.1
Review of the Operation of a Scheme
The scheme provider can examine several aspects of the scheme that reflect its efficacy. First, on the assumption that participants are trying to achieve fitness for purpose, we should see the robust standard
94 Proficiency Testing in Analytical Chemistry
deviation of the results ˆ rob p or, in immature schemes, ˆ rob approaching p in successive rounds. Efficacy can be quantified as the proportion of participants that obtain a ‘satisfactory’ score, say within the range 2, or the ratio ˆ rob =p , which should be approximately 1. Providers should review the operation of a scheme that seems to be inefficacious, although they usually have no powers to compel participants to compliance. They should, however, advise participants if the lack of compliance seems to be the result of factors such as inept choice of the analytical method. Providers should also occasionally compare for significant bias a consensustype assigned value with an independent result from a national reference laboratory. 1.03.4.2
Information about Analytical Methods
If participants report the methods they have used to obtain the results, some performance indicators of the methods can sometimes be deduced. For example, if many participants followed closely a particular fully documented analytical procedure, a robust standard deviation of the results from a round would be akin to a reproducibility standard deviation obtained from a collaborative trial. This in turn can be regarded as a guide to the uncertainty that might pertain to results from individual laboratories, although, of course, results from some laboratories may not reach the implied accuracy. Over the course of several (say 5–10) rounds of the scheme, a good picture of the capability of the method would be assembled, at little cost. In some instances where there are several distinct methods available (or clearly defined variants of a single method), and enough participants use each of them, it is possible to compare the mean result of each method in a round and search for significant bias among them. It must be stressed that the within-method variability is of such magnitude that it is difficult to achieve a worthwhile result with this kind of comparison, unless (1) the methods are well defined, (2) large numbers of results are available for each method, and (3) the bias is substantial. As a consequence, few satisfactory studies of this kind have been reported.30,31
References 1. Thompson, M.; Wood, R. The International Harmonized Protocol for the Proficiency Testing of (Chemical) Analytical Laboratories. Pure Appl. Chem. 1993, 65, 926–940. 2. International Organisation for Standardisation. ISO Guide 43: Proficiency Testing by Interlaboratory Comparisons – Part 1: Development and Operation of Proficiency Testing Schemes; International Organisation for Standardisation: Geneva, Switzerland, 1994. 3. Lawn, R. E.; Thompson, M.; Walker, R. F. Proficiency Testing in Analytical Chemistry; The Royal Society of Chemistry: Cambridge, 1997. 4. International Laboratory Accreditation Cooperation. ILAC-G13:2000. Guidelines for the Requirements for the Competence of Providers of Proficiency Testing Schemes. Available at www.ilac.org. 5. International Organisation for Standardisation. ISO 13528: Statistical Methods for Use in Proficiency Testing by Interlaboratory Testing; International Organisation for Standardisation: Geneva, Switzerland, 2005. 6. Thompson, M.; Ellison, S. L. R.; Wood, R. The International Harmonized Protocol for the Proficiency Testing of Analytical Chemistry Laboratories. Pure Appl. Chem. 2006, 78, 145–196. 7. Nogueira, J. M. F.; Nieto-de-Castro, C. A.; Cortez, L. EPTIS: The New European Database of Proficiency Testing Schemes for Analytical Laboratories. Trends Anal. Chem. 2001, 20, 457–461 (see also www.eptis.bam.de). 8. Analytical Methods Committee. Robust Statistics – How Not to Reject Outliers – Part 1: Basic Concepts. Analyst 1989, 114, 1693–1697. 9. Thompson, M. The Variance of a Consensus. Accred. Qual. Assur. 2006, 10, 574–575. 10. AMC Software and AMC Datasets. Available at www.rsc.org. 11. Lowthian, P. J.; Thompson, M. Bump-Hunting for the Proficiency Tester: Searching for Multimodality. Analyst 2002, 127, 1359–1364. 12. Silverman, B. W. Density Estimation for Statistics and Data Analysis; Chapman and Hall: London, 1986. 13. Cofino, W. P.; van Stokkum, I. H. M.; Wells, D. E.; Ariese, F.; Wegener, J. W. M.; Peerboom, R. A. L. A new Model for the Inference of Population Characteristics from Experimental Data Using Uncertainties Application to Interlaboratory Studies. Chemom. Intell. Lab. Syst. 2000, 53, 37–55 14. Fearn, T. Comments on ‘Cofino Statistics’. Accred. Qual. Assur. 2004, 9, 441–444. 15. Dempster, A. P.; Laird, N. M.; Rubin, D. B. Maximum Likelihood from Incomplete Data via the EM Algorithm. J. R. Stat. Soc. Ser. B 1977, 39, 1–38. 16. Aitkin, M.; Wilson, G. T. Mixture Models, Outliers and the EM Algorithm. Technometrics 1980, 22, 325–331. 17. Thompson, M. Using Mixture Models for Bump-Hunting in the Results of Proficiency Tests. Accred. Qual. Assur. 2006, 10, 501–505. 18. Lindley, D. V. Making Decisions, 2nd ed.; Wiley: London, 1984. 19. Thompson, M.; Fearn, T. What Exactly Is Fitness for Purpose in Analytical Measurement? Analyst 1996, 121, 275–278.
Proficiency Testing in Analytical Chemistry
95
20. Fearn, T.; Fisher, S.; Thompson, M.; Ellison, S. R. L. A Decision Theory Approach to Fitness for Purpose in Analytical Measurement. Analyst 2002, 127, 818–824. 21. Ramsey, M. H.; Lyn, J. A.; Wood, R. Optimised Uncertainty at Minimal Overall Cost to Achieve Fitness for Purpose in Food Analysis. Analyst 2001, 126, 1777–1783. 22. Ramsey, M. H.; Taylor, P. D.; Lee, J. C. Optimised Contaminated Land Investigation at Minimal Overall Cost to Achieve Fitness for Purpose. J. Environ. Monit. 2002, 4, 809–814. 23. Analytical Methods Committee. Understanding and Acting Upon Scores Obtained in Proficiency Testing Schemes. AMC Technical Briefs 2002, No. 11. Available at www.rsc.org. 24. Analytical Methods Committee. Proficiency Testing: Assessing z-Scores in the Longer Term. AMC Technical Briefs 2004, No. 16. Available at www.rsc.org. 25. Thompson, M. Harmonised Guidelines for Internal Quality Control in Analytical Chemistry Laboratories. Pure Appl. Chem. 1995, 67, 649–666. 26. Montgomery, D. C. Introduction to Statistical Quality Control; Wiley: Hoboken, NJ, 2005. 27. Analytical Methods Committee. The J-Chart: A Simple Plot that Combines the Capabilities of Shewhart and Cusum Charts, for Use in Analytical Quality Control. AMC Technical Briefs 2003, No. 12. Available at www.rsc.org. 28. Thompson, M.; Ellison, S. L. R.; Owen, L.; Mathieson, K.; Powell, J.; Key, P.; Wood, R.; Damant, A. P. Scoring in Genetically Modified Organism Proficiency Tests Based on Log-Transformed Results. J. AOAC Int. 2006, 89, 232–239. 29. Fearn, T.; Thompson, M. A New Test for ‘Sufficient Homogeneity’. Analyst 2001, 126, 1414–1417. 30. Thompson, M.; Owen, L.; Wilkinson, K.; Wood, R.; Damant, A. A Comparison of the Kjeldahl and Dumas Methods for the Determination of Protein in Foods, Using Data from a Proficiency Testing Scheme. Analyst 2000, 127, 1666–1668. 31. Lowthian, P. J.; Thompson, M.; Wood, R. The Use of Proficiency Tests to Assess the Comparative Performance of Analytical Methods: The Determination of Fat in Foodstuffs. Analyst 1996, 121, 977–982.
96 Proficiency Testing in Analytical Chemistry
Biographical Sketch
Michael Thompson is Emeritus Professor of Analytical Chemistry in the University of London. He is the author of a book on proficiency testing (Proficiency Testing in Analytical Chemistry) and the International Harmonized Protocol on Proficiency Testing in Analytical Chemistry. He has also published about 30 research papers on proficiency testing and is statistical adviser to five international proficiency testing schemes. He is the Editor of AMC Technical Briefs. He is the recipient of the SAC Gold Medal (1995), the L. S. Theobald Lectureship (2006) (both from the Royal Society of Chemistry), the Harvey W. Wiley Award (2007) (from AOAC International), and Honorary Life Membership of the International Association of Geoanalysts (2008).
1.04
Statistical Control of Measures and Processes
A. J. Ferrer-Riquelme, Technical University of Valencia, Valencia, Spain ª 2009 Elsevier B.V. All rights reserved.
1.04.1 1.04.2 1.04.3 1.04.4 1.04.5 1.04.6 1.04.6.1 1.04.7 1.04.7.1 1.04.8 1.04.9 1.04.9.1 1.04.9.1.1 1.04.9.1.2 1.04.9.1.3 1.04.9.1.4 1.04.9.2 1.04.9.2.1 1.04.9.2.2 1.04.9.3 1.04.9.3.1 1.04.9.4 1.04.10 References
Introduction: Basics of Process Monitoring Phases in Process Monitoring Shewhart Control Charts CUSUM Control Charts EWMA Control Charts Performance Measures of Control Charts Example 1 Control Charts for Autocorrelated Processes Example 2 Integration of SPC and Engineering Process Control Multivariate Control Charts Original Variables-Based MSPC Schemes Hotelling T 2 control charts for monitoring process mean Multivariate Shewhart-type control charts for monitoring process variability MEWMA control charts Fault diagnosis Latent Variables-Based MSPC Schemes PCA-based MSPC: exploratory data analysis and offline process monitoring (Phase I) PCA-based MSPC: online process monitoring (Phase II) Potential of Latent Variables-Based MSPC in Industry Example 3 Monitoring Measurement Systems: Applications of SPC in Analytical Laboratory Methods Software
Symbols A
A3 ,B3 , B4 , and c4
A2 ,D3 ,D4 , and d2
b dimension of the principal component subspace coefficients for the calculation of the control limits of the
BðK=2;ðm – K – 1Þ=2Þ;
x – s control chart
c
coefficients for the calculation of the control limits of the
Ct
x – R control chart at Nð0;2 Þ
series of independent and identically normally distributed (iind) random disturbances with zero mean and variance
ContðDModX;xnew;k Þ
100 102 102 104 105 106 106 108 108 111 112 112 113 114 114 115 115 116 119 120 121 121 123 124
sample mean of the SPE sample 100(1–)% percentile of the beta distribution with K/2 and (m–K–1)/2 degrees of freedom correction factor for DModX CUSUM control statistic at stage t (definition 1) number of nonconformities in samples of size n at stage t (definition 2) contribution of kth variable to the
2
97
98 Statistical Control of Measures and Processes
ContðSPE;xnew;k Þ
Contðtnew;a ;xnew;k Þ
D
Dt
E enew,k
Et ei et
FðK;ðmn – m – Kþ1ÞÞ;
f(x; ) H
K
L
m
MR ¼
m P t¼2
n P
MRð2Þt =ðm – 1Þ
DModX in a new observation contribution of kth variable to the SPE in a new observation contribution of kth variable to the ath score at a new observation magnitude of the shift in the process parameter number of nonconforming units in samples of size n at stage t (m K) residual matrix residual corresponding to the kth variable in a new observation EWMA control statistic at stage t residual vector (ith row of matrix E) residual at stage t 100(1–)% percentile of the F distribution with K and mn–m–Kþ1 degrees of freedom probability distribution of x decision value (threshold) in the CUSUM control chart reference value in the CUSUM control chart distance of the control limits from the centerline number of samples average of the moving ranges sample size (K A) loading matrix
p
¼ p
m P
Dt =nm ¼
t¼1
m P
pˆ t =m
t¼1
pak
pˆ t
m ¼ P Rt =m R
population fraction nonconforming average sample fraction nonconforming loading of the kth variable at the ath principal component sample fraction nonconforming at stage t average range
t¼1
rk S
jSj s ¼
m P
st =m qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi ffi Pn 2 st ¼ i¼1 ðxti – xt Þ =n – 1 t¼1
s0 T tnew,a
tr S Tt2
tTi ¼ fti1 ;ti2 ;:::;tiA g
¼ u
m P t¼1
ut
v
Ct =nm ¼
m P t¼1
ut =m
sample autocorrelation function (K K) sample variance– covariance matrix sample generalized variance average standard deviation sample standard deviation at stage t pooled residual standard deviation (m A) score matrix score of a new observation at the ath principal component trace of the covariance matrix Hotelling T2 control statistic at stage t A-dimensional score vector (ith row of matrix T) average sample number of nonconformities per unit sample average number of nonconformities per unit at stage t sample variance of the SPE sample
Statistical Control of Measures and Processes
vt ¼ v(zt)
wk
wt Wt
x X X
x Nð; SÞ
x¯
x¯ t
Xt ¼ fx1t ;x2t ;:::;xKt g
x¯ ¼
m P
xt =m
control statistic to be plotted in the control chart square root of the explained sum of squares for the kth variable MEWMA control statistics at stage t multivariate control statistic for monitoring process variability (a direct extension of the univariate s2 control chart) key characteristic to be measured preprocessed Z (m K) data matrix (m K) prediction matrix x follows a multivariate normal distribution with mean vector m and covariance matrix S (K1) overall sample mean vector. (K 1) vector of sample means at stage t (n K) observations matrix of K characteristics measured on n units sampled at stage t grand average
t¼1
xTkt ¼ fxk1t ; xk2t ;:::; xknt g
xTnt ¼ fxn1t ; xn2t ;:::; xnKt g
x new;k
n-dimensional vector of sample values of kth variable (kth column of matrix X) K-dimensional observation vector of nth unit (nth row of matrix X) prediction of the kth variable in a new observation
99
xnew;k from the PCA model xTt ¼ fxt1 ; xt2 ;:::; xtn g
xˆ tþ1=t
Z
zt ¼ – ln½fðxt ; 0 Þ=fðxt ; 1 Þ zt ¼ MRð2Þt ¼ jxt – xt – 1 j
zt ¼ Rt ¼ maxðxt Þ – minðxt Þ P zt ¼ s2t ¼ ni¼1 ðxti – xt Þ2 =n – 1 Pn zt ¼ xt ¼ i¼1 xti =n zt ¼ z(xTt ;0 ) Y jS j
0 ˆt
observations vector of the key characteristic on n units sampled randomly from the production or measurement process at stage t one-step ahead forecast of x made at stage t (m K) data matrix likelihood ratio statistic moving range of two consecutive observations at stage t sample range sample variance sample mean suitable statistic (or seed) to build the chart on. (A A) covariance matrix of T generalized variance Type I risk number of standard deviations of the shift in the process parameter process parameter to be monitored in-control (target or reference) value of minimum variance unbiased estimator of based on the sample values xTt ¼
fxt1 ;xt2 ;:::;xtn g
smoothing constant of the EWMA control statistic (definition 1) population average number of
100
a
Statistical Control of Measures and Processes
nonconformities in samples of size n (definition 2) smoothing constant of the MEWMA control statistic (definition 3) eigenvalue of the ath principal component
2K;
process mean autoregressive coefficient of AR(1) model 100(1–)% percentile of the
2 distribution with K degrees of freedom
1.04.1 Introduction: Basics of Process Monitoring Processes, whether chemical, biochemical, or analytical in nature, are subject to certain variability. You will not always get the same result each time no matter how tightly you operate a process. This is exactly what any customer perceives from the output (i.e., product, service, or information) of any process. If this output is to meet or exceed customer expectations, it should be produced by a process that is stable or repeatable.1 Variation remaining in a stable process reflects ‘common causes’ that are an inherent part of the process and which cannot be removed easily without fundamental changes in the process itself: for example, the design and operating settings of a polymerization reactor contribute to the common-cause variation of the viscosity of a polymer. As long as you keep the process the same, common-cause variation is the same from day to day, yielding a process consistent over time. A process affected by only common (random) causes of variation is said to be ‘in statistical control’. This means that the process is predictable, that is, you can predict what the process will make in the near future, for example, how many batches will meet the customer specifications for a particular key property. In addition to the common-cause system, other types of variability may occasionally affect the stability of a process: for example, operator errors, defective raw material, improperly adjusted or controlled equipment, sudden change in environmental conditions, and so on. These sources of variability are not part of the commoncause system and, thus, they are usually referred to as ‘assignable’ or ‘special causes’. They are sporadic in nature and cannot be predicted. A process also affected by this type of special causes of variation is said to be ‘out of control’ in the sense that it is unstable and, thus, unpredictable. It is crucial to distinguish between the two sources of variation because sharing the responsibility for process improvement depends on what type of variation is present: Frontline personnel are responsible for finding and taking decisions on assignable causes; common-cause variation, on the contrary, is management’s responsibility. Statistical process control (SPC) is a powerful collection of problem-solving tools useful in understanding process variability, achieving process stability, and improving process performance through the reduction of avoidable variability (i.e., special causes). The goal of any SPC scheme is to monitor the performance of a process along time to check whether the process behaves as it is expected to do (i.e., the predicted behavior from the common-cause system), and to detect any unusual (special) event that may occur. By finding assignable causes for them, significant improvements in the process performance can be achieved by eliminating these causes (or implementing them if they are beneficial). Control charts are the essential tools for pursuing this goal. A control chart is a picture of a process over time that helps to identify the magnitude and the type of variation present. To implement a control chart, one must register data from a process over time. Decisions are required about the variables to measure, the sample statistics to be monitorized, the sample size, the time between samples, the type of control chart, the control limits, and the decision rules. All these choices
Statistical Control of Measures and Processes
101
determine the cost of the monitoring scheme and its ability to detect certain out-of-control conditions. The design of a control chart involves the following steps: 1. Identifying the key characteristic x to be measured (e.g., a quality property, a key process variable that in some way affects the quality of a product, or an analytical measurement of a control or reference sample), the process parameter to be monitored (e.g., process mean, variance, proportion of defectives or nonconforming items, rate of defects or nonconformities), and the probability distribution of x, f (x; ) (e.g., normal, binomial, or Poisson). In this context, a process is said to be in statistical control (i.e., in control) if the underlying statistical model representing the key characteristic is stable over time. In this case, the process is said to be operated under normal operating conditions (NOCs). If there is some change over time in this statistical model, the process is said to be out of control.2 2. Selecting a suitable statistic (or seed) zt ¼ zðxT t ; 0 Þ to build the chart, where 0 is the in-control (target or reference) value of the monitorized process parameter and xT t ¼ fxt 1 ;xt 2 ; . . .; xtn g is the observations vector of the key characteristic on n units sampled randomly from the production or measurement process at stage t (t ¼ 1, 2, . . .). In general, seeds of the form zt ¼ ˆt , where ˆt is the minimum variance unbiased estimator of based on the sample values xT t ¼ fxt1 ; xt 2 ; . . .; xtn g, are often considered. Another choice is the likelihood ratio statistic, zt ¼ – ln½f ðxt ; 0 Þ=f ðxt ; 1 Þ, motivated by the Neyman–Pearson theory of testing statistical hypotheses (H0: ¼ 0 versus H1: ¼ 1).3 The observations in xT t can be measurements (e.g., filling volumes, weights, viscosities, particle sizes, and analyte sample concentrations) or counts (e.g., defective/nondefective item or number of defects) yielding the so-called control charts for variables or attributes, respectively. When observations are measurements, sample size n is small with values ranging from 1 to 5; when dealing with counts, sample size n is usually large (e.g., 50 or even more). Standard control charts for variables assume that the key characteristic x at stage t is a normally distributed random variable that can be expressed as xt ¼ þ at , where is the process mean and at follows a stochastic ‘white noise’ process, that is, a series of independent and identically normally distributed (iind) random in xt disturbances with zero mean and variance 2 (at N(0, 2)). Therefore, in control, the observations Pn are assumed to be iind with mean 0 and variance 20 . The sample mean zt ¼ xt ¼ i¼1 xti =n is the seed usually chosen to monitor the process mean ¼ . To monitor the process variance ¼ 2, several seeds have been proposed: for example, the sample range zt ¼ Rt ¼ maxðxt Þ – minðxt Þ or the sample variance P zt ¼ st2 ¼ ni¼1 ðxti – xt Þ2 =n – 1 Control charts for attributes are based on counts (attributes or discrete data). There are two kinds of attributes data: (i) yes/no and (ii) number of defects. In the yes/no case, each unit (item or sample) inspected is either defective (i.e., it does not meet some preset criterion) or not defective (i.e., it meets the criterion). The key characteristic x is a dichotomous random variable with only two possible outcomes: It either passes or fails the preset criterion. Examples of this kind of yes/no data include the following: catalyst has been correctly activated or not; polymer reaction has suffered from fouling problems or not; batch meets the requirements on one or more product properties or not; and so on. The statistical principles underlying this attributes control chart are based on the binomial distribution. Suppose the production process is operating in a stable manner (i.e., in control), such that the probability that any unit will not meet the preset criterion (i.e., the unit is defective or nonconforming) is p, and that successive units produced are independent, then each unit produced is a realization of a Bernouilli random variable with parameter p. If at stage t a random sample of n units is selected and if Dt is the number of nonconforming units, then Dt has a binomial distribution with parameters n and p (Dt binomial (n; p)), with mean Dt ¼ np and variance 2Dt ¼ npð1 – pÞ. In this case, ¼ p is the only parameter to be monitorized and the seed usually chosen is the sample fraction nonconforming zt ¼ pˆt ¼ Dt =n. A nonconforming unit will contain at least one defect or nonconformity. If it is feasible to count these defects, the key characteristic x is the number of defects or nonconformities in each inspected unit. The statistical assumption in this case is that at stage t the occurrence of nonconformities in samples of size n, Ct , is well modeled by the Poisson distribution with parameter (Ct Poisson ( )), with mean and variance
102
Statistical Control of Measures and Processes
Ct ¼ 2Ct ¼ . This requires that the number of opportunities for defects be infinitely large, the probability of occurrence of a nonconformity be small and constant, and that successive units produced are independent. This process depends only on parameter ¼ and usually the seed used is the sample average number of nonconformities per unit zt ¼ ut ¼ Ct =n. 3. Selecting the control statistic vt ¼ v(zt) most appropriate to detect certain out-of-control situations (e.g., changes in the parameter or other kind of abnormalities), the control limits of the chart, and the decision rules. The role of a control chart is to provide a procedure for a timely detection of a change in the value of . Depending on the particular goal, to detect a pulse, exponential decrease or increase, rectangular bump or sustained change in the parameter , a Shewhart, EWMA (exponentially weighted moving average), MA (moving average), or CUSUM (cumulative sum) type of control chart may be constructed on the selected seed zt, respectively.4 These various types of charts differ in the way present and past seeds zt are weighted, leading to different control statistics vt to be plotted in a time sequence on the control chart.
1.04.2 Phases in Process Monitoring Standard control chart usage involves two distinct phases, with two different goals: Phase I (model building) and Phase II (model exploitation). In Phase I, a set of process data is gathered and analyzed all at once in a retrospective way, estimating the parameters of the control statistics used and calculating trial control limits to determine if the process has been in control over the period of time where the data were collected. This is an iterative process where at each step the out-of-control observations are detected, carefully analyzed, and root causes are investigated. Those out-of-control observations considered as real anomalies are omitted from the data set and new trial control limits are recalculated from the resulting data. This phase ends when there is enough evidence that the data gathered come from an in-control process and hence the control limits can be used for Phase II. This means that the performance of the process has been understood and modeled, and the assumptions of its behavior and process stability are checked. No matter whether the collected data come from a historical database or they have been recently sampled, a wise use of Phase I control charts can increase process understanding and process improvement, assisting operating personnel and process managers in bringing the process into a state of statistical control. In Phase II, real online process monitoring is carried out by plotting the control statistics from new samples on the control charts defined in Phase I. As in Phase I, once an out-of-control sample is detected, this is carefully studied trying to find out the nature of the assignable causes affecting the process and implementing countermeasures to eliminate them if they deteriorate the process performance (or incorporate them in case they improve the process performance). Process knowledge and experience from operators and process managers are critical inputs for a successful diagnostic process. In simplex processes (e.g., univariate), the out-of-control action plans (OCAPs) as discussed by Montgomery1 can also be very useful in expediting this diagnostic process. The OCAPs are based on the analysis of known failure modes and provide the manual for identification and elimination (or incorporation if beneficial) of the assignable causes in an expert system-like fashion. In complex processes (e.g., multivariate), other diagnostic tools are needed as explained later on. As a result of this diagnostic process, process understanding and process improvement can also be obtained.
1.04.3 Shewhart Control Charts This type of monitoring schemes refers to any control chart developed according to the statistical principles proposed in the 1920s by Shewhart.5 In this case, the control statistic to be charted along stages t is the selected seed: vt ¼ zt ¼ zðxT t ; 0 Þ
ð1Þ
Statistical Control of Measures and Processes
103
The chart contains a centerline that represents the in-control average value of the selected seed zt and two horizontal lines, called upper control limit (UCL) and lower control limit (LCL). These control limits are chosen so that if the process is in control, the expected fraction of seeds beyond the control limits takes a prespecified value . Choosing the control limits is equivalent to setting up the critical region for testing the hypothesis H0: ¼ 0 versus H1: 6¼ 0, being the Type I risk of the test (i.e., the chance of wrongly rejecting the null hypothesis when it is true; for more information on hypothesis testing, see Chapter 1.02). Under the assumption that zt ðiindÞ N ðz ;2z Þ, the control limits are set at z Lz , where L is the distance of the control limits from the centerline, expressed in standard deviation units, and this depends on . Usually, ¼ 0.0027, leading to the classical ‘three-sigma’ control limits (L ¼ 3). One of the most common Shewhart charts Pn for variables is the xR control chart. This is made of two charts: The x chart uses the sample mean ðzt ¼ xt ¼ i¼1 xti =nÞ to monitor the process mean ¼ , whereas the R chart uses the sample range ðzt ¼ Rt ¼ maxðxt Þ – minðxt ÞÞ to monitor the process variance ¼ 2. Control limits on both charts are estimated from preliminary samples taken when the process is thought to be in control. To obtain good where estimates, at least m ¼ 20–25 samples are required. The control limits on the x chart are set at x A2 R, Pm x ¼ t ¼2 xt =m is the grand average. The control limits on the R chart are plotted at D3 R (LCL) and D4 R (UCL), P where the average range R ¼ mt ¼1Rt =m is the centerline. Assuming the process is in control, the process standard 2 . The values for A2, D3, D4, and d2 are tabulated for various sample sizes n.1 deviation can be estimated by R=d The sample range R, although easy to calculate by hand and easy to understand for most people, is not the best pP ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi ffi n t Þ2 =n – 1 give better estimates way to monitor process variability. Sample standard deviations st ¼ i¼1 ðxti – x of the process standard deviation, particularly for moderate to large sample sizes. The current availability of handheld calculators and personal computers for online implementation of control charts has eliminated any computational difficulty. If st replaces Rt as the seed, this leads to the x – s control chart. The control limits Pmon the x The centerline of the s chart is plotted at the average standard deviation s ¼ t¼1 chart are set at x A3 R. st =m, and the control limits at B3 s (LCL) and B4 s (UCL). Assuming the process is in control, the process standard deviation can be estimated by s =c 4 . The values for A3, B3, B4, and c4 are tabulated for various sample sizes n.1 There are many situations in which the sample size used for process monitoring is 1, that is, n ¼ 1. Some examples are (i) every unit manufactured is analyzed (e.g., 100% inspection in highly automated parts industry or batch processes), (ii) the process generates data on a very limited frequency (e.g., the test method used to analyze the process is very expensive to run or takes a long time, or when dealing with accumulated quantities such as daily energy consumption), and (iii) repeated measurements on the process differ only because of laboratory or analysis error, as in many chemical continuous processes. In such situations, a control chart for individual measurements is useful to monitor the process. The most common individuals control chart is the x – MRð2Þ individuals control chart (moving range (MR)). This actually includes two charts: The x chart displays the value of the key characteristic at stage t ðzt ¼ xt Þ to monitor the process mean ¼ , whereas the MR(2) chart uses the moving range of two consecutive observations ðzt ¼ MRð2Þt ¼ jxt – xt –1 jÞ to monitorPthe process variance ¼ 2. The MR(2) chart has centerline at the m average of the moving ranges MR ¼ t ¼2 MRð2Þt =ðm – 1Þ and control limits Pm at D3 MR (LCL) and D4 MR (UCL). The control limits on the x chart are set at x 3MR=d2 , where x ¼ t ¼1 xt =m is the sample average of all the observations. Assuming the process is in control, the process standard deviation can be estimated by MR=d2 . The values for D3, D4, and d2 are tabulated for various sample sizes n.1 Caution should be taken when normality assumption cannot be assumed. In this case, the control limits may be based on resampling methods.6 Regarding control charts for attributes, the most frequently used are the p chart and the u chart. The p chart displays the sample fraction nonconforming zt ¼ pˆt ¼ Dt =n to monitor the population fraction nonconforming p, where Dt is the number of nonconforming units in the sample. Assuming that the binomial probability model pffiffiffiffiffiffiffiffiffiffiffiffiffiffi ffi is adequate and can be approximated by a normal distribution, the controlPlimits are set at p 3 p ð1 – pÞ=n, P m m where the centerline is the average sample fraction nonconforming p ¼ t ¼1 Dt =nm ¼ t ¼1 pˆ t =m. In those cases where the normality assumption is not appropriate (i.e., when n pð1 – pÞ < 9), the control limits have to be set based on /2 and 1/2 percentiles from the binomial probability model.7 On the contrary, the u chart displays the sample average number of nonconformities per unit zt ¼ ut ¼ Ct =n to monitor the population average number of nonconformities per unit /n, where Ct is the occurrence of nonconformities in samples of size n. Assuming Ct is well modeled by the Poisson distribution with parameter and can be approximated by a
104
Statistical Control of Measures and Processes
Table 1 Formulas for Shewhart control charts Chart
Centerline
Control limits
x (using R) R x (using s) s x (using MR) MR p u
x R
x A2 R UCL ¼ D4 R LCL ¼ D3 R; x A3 R LCL ¼ B3 s ; UCL ¼ B4 s x 3MR=d2 LCL ¼ D3 MR; UCL ¼ D4 MR pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi p 3 pð1 – pÞ=n pffiffiffiffiffiffiffi u 3 u=n
x s x MR p u
pffiffiffiffiffiffiffi normal distribution, the control limits 3 u=n, where the centerline is the average sample number Pm are set at u P m of nonconformities per unit u ¼ t¼1 Ct =nm ¼ t ¼1 ut =m. In those cases where the normality assumption is not appropriate (i.e., when < 9), the control limits have to be set based on /2 and 1/2 percentiles from the Poisson probability model.7 Table 1 summarizes the formulas to calculate the centerline and control limits for the different Shewhart control charts introduced. To determine whether a process is out of control, several rules have to be defined. These rules refer to extremely low probability events when the process is in control. So, when they occur, you are fairly certain that something has changed in the process and it is worth looking for an assignable cause. The main rule used on Shewhart control charts is one point exceeding the control limits. Using three-sigma limits for the control charts, this rule has an in-control probability of 0.0027. This is called the false alarm probability for this out-of-control rule. Other rules, called supplementary rules, have been suggested for detecting any nonrandom pattern of behavior. Some examples are as follows: a run of eight (or seven) consecutive points on one side of the centerline; eight (or seven) points in a row trending up (or down); and two out of three consecutive points beyond the twosigma warning limits ðz 2z Þ. Although the application of these supplementary rules increases the sensitivity of the Shewhart control charts to the detection of small process shifts, care should be exercised when using several decision rules simultaneously because of the increase in the overall false alarm probability.8 Shewhart control charts are extremely useful in Phase I implementation of SPC (model building), where the process is likely to be out of control and experiencing assignable causes resulting in large shifts in the monitored parameters. They are also very useful in the diagnostic aspects of bringing a ‘wild’ process into statistical control, because the patterns on these charts often provide guidance regarding the nature of the assignable cause.1 This advantage comes from the fact that they are plots of the actual data, providing a picture of what the process is actually doing, which makes the interpretation easy. Because they are not specialized, these charts act as global radars, being potentially capable of drawing attention to unusual kinds of behavior and hence to possible signals and causes previously unsuspected.4 The major disadvantage of Shewhart control charts is that they are relatively insensitive to small process shifts, quite often in Phase II monitoring (model exploitation). This drawback is due to the fact that at each sampling time t it uses only the contemporary information, ignoring any potential information contained in past samples (t1, t2, . . .). Time weighted control charts can be designed for this purpose (i.e., detecting small shifts). Out of the different proposals, the CUSUM control charts and the EWMA control charts are the best choices.
1.04.4 CUSUM Control Charts The CUSUM control chart was first designed by Page during the 1950s.9,10 In this case, the control statistic to be charted at stage t is the CUSUM of past and present deviations of the selected seed zt ¼ zðxT t ;0 Þ from a target (in-control) value 0: vt ¼ Ct ¼
t X k¼1
ðzk – 0 Þ
ð2Þ
Statistical Control of Measures and Processes
105
This feature makes this scheme very powerful for detecting persistent changes in the process parameter , particularly a shift from the in-control value 0 to another (out-of-control) value 1. CUSUMs can be represented in two ways: the tabular (or algorithmic) CUSUM and the V-mask CUSUM. The tabular CUSUM, which is the preferred one, involves two statistics, Ctþ and Ct– . Ctþ is the sum of deviations above the target and is referred to as the one-sided upper CUSUM. Ct– is the sum of deviations below the target and is referred to as the one-sided lower CUSUM:
Ctþ ¼ max 0;zt – ð0 þ K Þ þ Ctþ– 1
Ct– ¼ max 0;ð0 – K Þ – zt þ Ct–– 1
ð3Þ ð4Þ
where the starting values are Ctþ ¼ Ct– ¼ 0. The constant K is referred to as the reference value, and it is often chosen as K ¼ j1 – 0 j=2, that is, halfway between the target value 0 and the out-of-control value 1 we are aiming to detect quickly. If the shift is expressed in standard deviation units as 1 ¼ 0 þ , then K ¼ ð=2Þ, that is, one-half of the magnitude of the shift. However, other values of K can be used. The two-sided CUSUM control chart plots the values of Ctþ and Ct– for each sample. If either statistics lies beyond a stated decision value (or threshold) H, the process is considered to be out of control. The choice of H is not arbitrary and should be chosen after careful consideration. A reasonable value is H ¼ 5. Actually, the proper selection of both parameters K and H is crucial for the good performance of the CUSUM control chart.3 CUSUMs have been developed for a large variety of sample statistics (monitoring seeds) such as individual observations, sample averages, ranges, and standard deviations of rational subgroups, sample fractions nonconforming, and average numbers of nonconformities per inspection unit.
1.04.5 EWMA Control Charts The EWMA control chart is also a good alternative to the Shewhart control chart for detecting small shifts. The EWMA control chart was first introduced by Roberts.11 In this case, the control statistic to be charted is an EWMA of present and past values of the selected seed zt : vt ¼ Et ¼ zt þ ð1 – ÞEt – 1
ð5Þ
where is a smoothing constant (0 < 1). The EWMA can also be expressed as Et ¼ ð1 – Þt E0 þ
t X
ð1 – Þt – k zk
ð6Þ
k¼1
where the starting value E0 is the process target 0. From this expression, it is clear how the weights ð1 – Þt –k decrease geometrically with the age of the sample statistic zt. The parameter is a tuning parameter. If is small, the EWMA performs like a CUSUM. On the contrary, if is close to one, the EWMA is quite similar to the Shewhart chart. Several EWMA charts have been developed for different goals using a large variety of sample statistics (monitoring seeds). The most used are the EWMA with individual observations ðzt ¼ xt Þ or sample averages qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi ffi ðzt ¼ xt Þ for monitoring the process mean. The control limits are set at 0 Lz ð =ð2 – ÞÞ 1 – ð1 – Þ2t , where L is the width of the control limits and z is the standard deviation of the sample statistic z. Once the chart is started, after several time periods these control limits will approach steady-state values given by pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi 0 Lz =ð2 – Þ. The process is considered out of control if one or more zt values exceed the control limits. For a given smoothing constant , the value L is chosen to give the desired performance monitoring of the EWMA chart. Usual values for both parameters are 0.05 0.25 and 2.6 L 3 (see Hunter12 for details). In practice, the EWMA chart is used more than the CUSUM chart because it is easier to set up and operate. For monitoring the variability of normally distributed processes, several EWMA-based statistics have been proposed. One example is the exponentially weighted mean square error (EWMS) chart. This chart uses the sample statistic zt ¼ ðxt – 0 Þ2 . To avoid the sensitivity of this chart to shifts in process mean, the exponentially
106
Statistical Control of Measures and Processes
weighted moving variance (EWMV) chart has been proposed by replacing 0 in the sample statistic of EWMS with an estimate of the process mean obtained from the classical EWMA statistic for the process mean. EWMA charts can also be used for monitoring the average numbers of nonconformities per inspection unit. This is called the Poisson EWMA, and uses the sample average number of nonconformities per unit zt ¼ ut ¼ Ct =n as the sample statistic to be weighted. For technical details on control limits of these charts see Montgomery.1
1.04.6 Performance Measures of Control Charts The performance of a control chart is usually evaluated based on two criteria: how often the control chart signals for an in-control process (false alarm rate) and how slow the control chart is in detecting a specific change in the process (excessive delay in detection). Unfortunately, both criteria cannot be optimized at the same time. For example, to reduce the false alarm rate, the control limits can be moved farther from the centerline. However, widening the control limits will increase the risk of a point falling between the control limits when the process is really out of control. Therefore, it will take more time to detect a change in the process, hence causing excessive delay in detection. This concept is closely related to Type I error (rejecting the null hypothesis when it is true) and Type II error (accepting the null hypothesis when it is false) in hypothesis testing. Two different performance measures, in-control average run length (ARL0) and out-of-control average run length (ARLD), are defined for evaluating the false alarm rate and the excessive delay in detection, respectively. ARLD depends on the magnitude of the shift in the process parameter to be monitored, D ¼ j1 – 0 j ¼ . The average run length (ARL) is defined as the expected (average) number of samples that must be plotted before an out-of-control signal is observed. For an in-control process ARL0 should be as high as possible. For example, if the process observations are uncorrelated, then for any Shewhart control chart with no supplementary out-of-control rules ARL0 ¼ 1/, where is the probability that any point exceeds the control limits when the process is in control. Under normality and three-sigma control limits, ARL0 ¼ 1/0.0027 ¼ 370. This means that, even if the process remains in control, a point exceeding the control limits will appear every 370 samples, on the average. For a control chart with given control limits, ARL0 is unique. On the contrary, if the process is out of control owing to a shift of magnitude D, ARLD is a function of D and it should be as small as possible, meaning that the control chart signals the change as soon as possible right after the shift occurs. The performance of a particular control chart can be described by plotting the ARLD values against different suspected shifts D (note that ARLD¼0 ¼ ARL0). This is also a useful tool to compare the performance of different control charts. When comparing the performance of different control charts, ARL0 should be equal for all the schemes. This can be done by adjusting the control limits of the control charts: For example, in a CUSUM control chart, for a given particular K value (chosen to efficiently detect a particular shift in the process parameter D ¼ j1 – 0 j), the value of threshold H is selected to give the desired ARL0. On the contrary, in an EWMA control chart, for a given smoothing constant , the value L is chosen to give the desired ARL0. 1.04.6.1
Example 1
Let us consider the following example elaborated from a data set available in Montgomery,1 page 387: the first 20 observations were drawn at random from a normal distribution z0 N ð0 ¼ 10; 20 ¼ 1Þ; the last 10 observations were drawn at random from a normal distribution z1 N ð1 ¼ 11; 20 ¼ 1Þ. This data set could be considered, for example, as reaction run times for 30 batches; the last 10 as corresponding to batches produced with a different catalyst. Figures 1–3 display the x individuals Shewhart, CUSUM, and EWMA control charts, respectively. All the charts have been adjusted to yield the same in-control ARL, ARL0 ¼ 370, by setting L ¼ 3 in the Shewhart chart, H ¼ 4.773 and K ¼ 0.5 in the CUSUM chart, and L ¼ 2.8 and ¼ 0.15 in the EWMA chart. The control limits of the charts have been worked out based on the parameters of the normal distribution z0. The CUSUM and EWMA charts signal an out-of-control point: Sample 29 is exceeding the UCL. See the cross points in Figures 2 and 3. Both charts also show a stable upward trend in the last 10 observations, indicating that a shift in the process mean likely occurred around samples 21 and 22. This matches the time
Statistical Control of Measures and Processes
107
14 13 12 x
11 10 9 8 7 6 0
5
10
15 20 Observation
25
30
Figure 1 x individuals Shewhart control chart for the reaction run time data (L ¼ 3). Cross points indicate out-of-control observations. Dotted lines indicate control limits.
7
CUSUM
5 3 1 –1 –3 –5 0
5
10
15 20 Observation
25
30
Figure 2 CUSUM individuals control chart for the simulated reaction run time data (H ¼ 4.773; K ¼ 0.5). Cross points indicate out-of-control observations. Dotted lines indicate control limits.
11
EWMA
10.5 10 9.5 9 0
5
10
15 20 Observation
25
30
Figure 3 EWMA individuals control chart for the simulated reaction run time data (L ¼ 2.8; ¼ 0.15). Cross points indicate out-of-control observations. Dotted lines indicate control limits.
when a new catalyst was used. The slope of the trend estimates the magnitude of the shift, D ¼ j1 – 0 j. In this case, D is approximately 1, corresponding to one standard deviation of the run time, D ¼ ¼ 1. Figure 1 shows that the Shewhart chart apparently looks in control (there is no point exceeding the control limits). Only if the supplementary rules are considered, a run of seven consecutive points above the centerline occurs at sample 29, suggesting the process is out of control. This example illustrates the benefits of using supplementary rules: increasing the sensitivity of the Shewhart chart to detect small shifts in the process. However, the shift in the process is much more evident from CUSUM and EWMA control charts.
108
Statistical Control of Measures and Processes
1.04.7 Control Charts for Autocorrelated Processes As commented in Section 1.04.1, the standard model assumed in most SPC applications is that when the process is in control the key characteristic x to be measured at stage t can be expressed as xt ¼ þ at
ð7Þ
where at is normally and independently distributed with mean zero and standard deviation (i.e., at follows a white noise stochastic process). In many cases, the monitoring schemes are robust to slight deviations from normality, and then the statistical properties of the control charts are not compromised. The independence assumption, however, is much more critical and has a profound impact on the performance of the control charts if indeed violated. For example, if the data are positively correlated (i.e., values above the mean tend to be followed by other values above the mean and vice versa), control charts will give misleading results in the form of too many false alarms and they will not signal real out-of-control situations. Lack of independence between successive observations (i.e., data correlation over time) shows up whenever the interval between samples becomes small relative to the process dynamics (i.e., inertial elements as raw materials flow, storage tanks, reactors’ residence times, or environmental conditions, defining the settling time of the process). This being more the rule than the exception in modern process environments because of the information technologies that allow registering data from every part produced (e.g., in computer integrated manufacturing (CIM) environments) or at high sampling rates (e.g., in continuous and batch process industries). Most of the present-day analytical systems contain mechanical, electrical, and/or optical parts, in which the correlation over time can be of great importance. Some examples are lamp aging of the photometric detector, a decrease in the separation capacity of the chromatographic column, changes in the detector response with use, the influence of points of contamination on subsequent responses in continuous flow analytical systems, sensor deterioration, or changes in the environmental conditions (temperature, humidity, etc.), causing drifts in the results.13
1.04.7.1
Example 2
As an example, consider the data in Figure 4. This graph plots the x–MR(2) individuals control chart from the results of 56 consecutive sulfate determinations in a control sample using a sequential injection analysis (SIA) system with UV/visible detection for determining sulfates in water.14 Note that mainly the x individuals control chart signals many out-of-control measurements (cross points in Figure 4): for example, measurement 8 completes a run of seven consecutive points below the centerline; or measurements 11–18 are above the UCL. The sulfate determinations of the reference sample are drifting or wandering over time, indicating that the analytical process is apparently out of control. Before taking any decision, it would be of great interest to know whether the out-of-control signals are indicating real problems or whether they are false alarms due to autocorrelation (correlation over time). 240
24
230
20
x
MR(2)
220 210 200
16 12 8 4
190
0 0
10
20 30 40 Observation
50
60
0
10
20 30 40 Observation
50
Figure 4 x–MR(2) individuals control chart for sulfate determinations in a control sample. Cross points indicate out-of-control measurements. Dotted lines indicate control limits.
60
Statistical Control of Measures and Processes
109
The correlation over a series of n time-oriented observations can be estimated by the sample autocorrelation function (ACF): rk ¼
Pn – k ðx – xÞðx t þk – xÞ t ¼1 Pn t Þ2 t ¼1 ðxt – x
ð8Þ
This assumes that the mean and the variance of the process are constant and that correlation between successive observations depends only on the lag k (number of time periods between observations). The sample ACF for the sulfate determinations is shown in Figure 5. Note that there is a statistically significant (p-value < 0.05) strong positive correlation till lag k ¼ 3. This means that measurements that are one, two, and three periods apart are positively correlated. This level of autocorrelation is sufficient to distort the performance of a Shewhart control chart and the interpretation rules. For example, supplementary run rules should never be used because they can be generated from autocorrelation (i.e., they are part of the common-cause system) and not from assignable causes of variability. On the contrary, estimating the variability of a positive autocorrelated process from moving ranges (as done in Figure 4) may lead to a considerably high underestimation of the real process variability, yielding control limits too much narrower than should be and, hence, increasing the frequency of measurements exceeding them. It is important to highlight that autocorrelation must be considered as part of the common-cause system affecting the process, and any attempt to get rid of it (e.g., by reducing the sampling rate) will lead to a very inefficient use of available data, degrading the performance of the resulting monitoring scheme. Therefore, we should model the autocorrelation. For this purpose, autoregressive integrated moving average (ARIMA) models15 can be used. In addition to the sample ACF, the so-called partial ACF15 is a useful tool to give more information about the correlation structure of the data. The partial ACF measures the correlation between measurements that are shifted k lags without the influence of the intermediate values. Figure 6 shows the partial ACF for the sulfate determinations. In this case, only the partial ACF at lag 1 is statistically significant (p-value < 0.05). Figures 5 and 6 suggest a first-order autoregressive model, AR(1), to describe this analytical process xt – ¼ ðxt –1 – Þ þ at
ð9Þ
where is the process mean, is the autoregressive coefficient ðj j < 1Þ, and at follows a white noise stochastic process, at N(0, ). The standard deviation of xt is given by x ¼ pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi ð1 – 2 Þ
ð10Þ
1
r(k)
0.6 0.2 –0.2 –0.6 –1 0
3
6
9
12
15
Lag Figure 5 Sample ACF for the sulfate determinations in a control sample. Dotted lines indicate limits for statistical significance ( ¼ 0.05).
110
Statistical Control of Measures and Processes
1
r(k)
0.6 0.2 –0.2 –0.6 –1 0
3
6
9
12
15
Lag Figure 6 Partial ACF for the sulfate determinations in a control sample. Dotted lines indicate limits for statistical significance ( ¼ 0.05).
The estimated AR(1) model for the sulfate measurements is xt – 215:70 ¼ 0:85ðxt –1 – 215:70Þ þ et
ð11Þ
where the residuals et ¼ xt – xˆt are approximately normally and independently distributed with mean zero and constant variance (et N(0, s ¼ 4.85)). This is validated by computing the simple and partial ACF of the residuals (not shown) where no coefficient are statistically significant (p-value > 0.05). There are several approaches to monitor this analytical process. One is to estimate the process variability qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi ˆ 2 . No supplementary from Equation (10) and set the control limits on the x individuals chart at ˆ 3s= ð1 – Þ rules should be used when interpreting the chart. Note that in the sulfate data the process standard deviation estimated from the average moving range ðMR=d2 ¼ 2:99Þ is approximately one-third of the one estimated (i.e., 9.21) from Equation (10). This explains the large number of measurements exceeding the control limits in Figure 4. Another approach is to apply classical control charts to the residuals. Points out of control or unusual patterns on such charts would indicate that the parameters , , or had changed, implying that the analytical process was out of control. Figure 7 is an x–MR(2) individuals control chart of the residuals from the fitted AR(1) model. Most of the out-of-control points in the x–MR(2) individuals control chart of the original data (Figure 4) have disappeared. This confirms the hypothesis that they were real false alarms due to autocorrelation. As shown in Figure 7, there is only one out-of-control point (measurement 36) in the x chart that generates two out-of-control points (measurements 36 and 37) in the MR(2) chart. By looking at the original data (Figure 4) at measurement 35, there is a drastic jump to measurement 36 that is signaled by the MR(2) chart. This momentary dysfunction of the analytical systems can be caused by bubbles that temporarily produce some anomalous data but, once they have disappeared, the systems begin to function correctly again.16 27
24 20 MR(2)
x
17 7 –3
16 12 8 4 0
–13 0
10
20 30 40 Observation
50
60
0
10
20 30 40 Observation
50
60
Figure 7 x–MR(2) individuals control chart for the residuals of AR(1) model from sulfate determinations in a control sample. Cross points indicate out-of-control observations. Dotted lines indicate control limits.
Statistical Control of Measures and Processes
111
Sulfate (mg l–1)
240 230 220 210 200 190 0
20
40
60
Observation Figure 8 One-step ahead forecast sulfate determinations from AR(1) model.
To exploit the inertial properties of the process, in addition to the residual control charts, one-step ahead forecasts, xˆt þ1=t (i.e., prediction for the measurement at stage t þ 1 made at stage t), can also be plotted in a run chart: xˆt þ1=t ¼ þ ðx t – Þ
ð12Þ
Figure 8 displays the run chart of the one-step ahead forecasts, xˆtþ1=t , for the sulfate measurements. Note the big gap (error) between the real measurement 36 and its prediction when measurement 35 was obtained, xˆ36=35 . If this monitoring scheme was implemented online, this kind of anomalies could be detected in real time. Note that for estimating this model, outliers should be eliminated. Several methods can be used for this purpose. See for example Box et al.15 Finally, it is important to note that in correlated processes, if there is any easily manipulated variable related to the key characteristic of interest, process dynamics can be exploited to design control rules to force the key characteristic to track a desired trajectory. By integrating control theory with SPC ideas, process improvement can be achieved as commented in the following section.
1.04.8 Integration of SPC and Engineering Process Control The traditional control charting discipline of ‘monitor, then adjust when out of control’ is not the best strategy for process improvement in processes with some kind of dynamics. This is especially true in processes having a tendency to drift or wander away from the target. In these contexts, another approach called stochastic control, also known as engineering process control (EPC), is needed. This approach is based on process compensation and regulation (feedback and/or feedforward control), in which some easily manipulable process variables are adjusted following some control rules with the goal of keeping the key characteristic (controlled variable) close to the desired target. This requires good knowledge of the relationship between the controlled variable and the manipulated variables, as well as an understanding of process dynamics. If the cost of making adjustments (control actions) in the manipulable process variables is negligible, then the variability of the controlled variable is minimized by taking control actions in every sample. This is in sharp contrast to SPC.1 SPC and EPC originated in different industries – the parts industry and the process industry – and have been developed independently with some controversy in the past because of the different meaning of ‘control’ in both approaches: process monitoring (SPC) versus process regulation (EPC). It is true that a dynamic process should not be adjusted using SPC, but it is also true that assignable causes cannot be detected with EPC. In fact, regulation schemes react to process upsets; they do not make any effort to remove the root causes. For example, let us consider a control loop that regulates the inside reactor temperature through a thermostat (feedback controller) by making adjustments in the heating flow. If there appears a leaking problem in the heating jacket, the reactor temperature decreases and the thermostat tries to compensate for the deviation by increasing the heating flow. By continuous adjustments (provided that the heating system can keep up with the temperature
112
Statistical Control of Measures and Processes
differential), the reactor temperature is kept around the target but the assignable cause (leaking) goes undetected. The EPC scheme masks the presence of the assignable cause keeping the process at the desired target at all costs. In the above example, this will result in high energy and repair costs (at the very least the heater will eventually breakdown). By monitoring, for example, the adjustments in the heating flow, the leaking problem would have been detected, saving energy and avoiding the breakdown of the heating system. Therefore, EPC and SPC should be considered as two complementary (not alternative) strategies for quality improvement. SPC monitoring procedures seek to reduce output variability by detecting and eliminating assignable causes of variation. On the contrary, EPC tries to minimize output variability by making regular adjustments exploiting the process dynamics (common-cause system). Hence, ideas from both fields can be used together in an integrated EPC/SPC system called to secure both optimization and improvement. The notion of superimposing SPC on a closed-loop system has opened new lines of research in the area of quality improvement. This new strategy called engineering statistical process control (ESPC), also known as algorithmic statistical process control (ASPC),17 is a proactive approach to quality improvement that reduces predictable variations in quality characteristics using feedback and feedforward techniques and then monitors the entire system (by plotting deviations from the target, prediction errors, adjustments, etc.) to detect and help remove unpredictable process upsets. ESPC integrates two complementary approaches: process adjustment and process monitoring. As such, ESPC is a marriage of control theory and SPC that aims to reduce both short- and long-term variability by replacing the traditional control charting discipline of ‘monitor, then adjust when out of control’ with ‘adjust optimally and monitor’.18 Successful applications and thorough research of this methodology can be found in the literature.4,19–24
1.04.9 Multivariate Control Charts Most SPC schemes currently in practice are based on charting one or a small number of product quality variables in a univariate way. These approaches, although effective in the past where data were scarce, are totally inadequate for modern continuous and batch processes and manufacturing industries where massive amounts of highly correlated variables are being collected. As commented by Professor John F. MacGregor ‘‘the presence of variable interactions in experimental designs leads to the same difficulties in interpreting the results of one factor at a time experimentation as does the presence of correlation among variables in interpreting univariate SPC charts’’.25 Applying univariate SPC charts to each individual variable separately will force the operator to inspect a large number of control charts. When special events occur in a process, they affect not only the magnitude of the variables but also their relationship to each other. These events are often difficult (or even impossible) to detect by charting one variable at a time in the same way as the human eye cannot perceive perspective by looking at an object with one eye at a time.26 Multivariate statistical process control (MSPC) schemes that treat all the variables simultaneously are required in these new data-rich environments. Several approaches have been proposed. In this chapter, they are going to be classified as original variables-based MSPC and latent variables-based MSPC.
1.04.9.1
Original Variables-Based MSPC Schemes
Let us consider Xt ¼ fx1t ; x2t ; . . .; xKt g, the n K observations matrix of K characteristics measured on n units sampled randomly from the production or measurement process at stage t (t ¼ 1, 2, . . .), where xT kt ¼ fxk1t ; xk2t ; . . .; xknt g is the n-dimensional vector of sample values of the kth variable (the kth column of matrix X) and xT nt ¼ fxn1t ; xn2t ; . . .; xnKt g is the K-dimensional observation vector of the nth unit (the nth row of matrix X). Original variables-based MSPC schemes are based on multivariate statistics built from measured variables Xk (k ¼ 1, . . ., K). They can be thought of as the multivariate counterpart to the Shewhart, CUSUM, and EWMA control charts for monitoring the process mean and variability. The Hotelling T 2 and multivariate EWMA (MEWMA) control charts are the most used in practice.
Statistical Control of Measures and Processes
1.04.9.1.1
113
Hotelling T 2 control charts for monitoring process mean
The Hotelling T 2 control chart is the multivariate extension of the univariate Shewhart x control chart for monitoring the process mean. This approach assumes that the (K 1) vector of measured variables x is not time dependent and follows a K-dimensional normal distribution, x NK ðm; SÞ. The chart checks if the (K 1) mean vector of the process m remains stable, assuming a constant (K K) covariance matrix S. This can be used for either individual (n ¼ 1) or subgroup data (n > 1).
1.04.9.1.1(i) Subgroup data Assume that there are m samples of K measured variables each of size n > 1 available from the process. The Hotelling T 2 control chart is based on monitoring at each time t the Hotelling T 2 statistic:1,27–30 Tt2 ¼ nðx t – xÞ S – 1 ðx t – xÞ T
ð13Þ
where x t is the (K 1) vector of sample means at stage t, x the (K 1) overall sample mean vector (estimate of the in-control (K 1) mean vector m), and S is the (K K) pooled sample variance–covariance matrix (estimate of the in-control (K K) covariance matrix S) from the m samples. The Tt2 statistic represents the estimated weighted distance (Mahalanobis distance) of any observation from the target m. Therefore, only UCLs are defined for this chart. The Phase I and II control limits at significance level (Type I risk) are given by UCLðT 2 ÞPhase I; ¼
K ðm – 1Þðn – 1Þ FðK ;ðmn – m – K þ1ÞÞ; mn – m – K þ 1
ð14Þ
UCLðT 2 ÞPhase II; ¼
K ðm þ 1Þðn – 1Þ FðK ;ðmn – m – K þ1ÞÞ; mn – m – K þ 1
ð15Þ
where FðK ;ðmn – m – K þ1ÞÞ; is the 100(1)% percentile of the corresponding F distribution. Thus, if the value of the Tt2 statistic plots above the UCL, the chart signals a potential out-of-control process. The difference in both control limits comes from the fact that in Phase II, when the chart is used for monitoring new observations, x t is independent of x and S sample estimators. This is not the case in Phase I. However, when the number of samples m is large (m > 50), both limits match and the Tt2 statistic can be approximated by a 2 distribution with K degrees of freedom. Therefore, UCLðT 2 Þ ¼ 2K ; , where 2K ; is the 100(1)% percentile of the corresponding 2 distribution.
1.04.9.1.1(ii) Individual observations As commented in Section 1.04.3, there are situations where the subgroup size is naturally n ¼ 1. This occurs frequently in chemical and process industries and in measurement processes. In this case, the Hotelling T 2 statistic at stage t is given by Tt2 ¼ ðxt – x ÞT S – 1 ðxt – x Þ
ð16Þ
where xt is the (K 1) vector of measured variables at stage t, x T ¼ fx1 ; xP xK g is the sample mean vector 2 ; . . .; (estimate of the in-control (K 1) mean vector m), and S ¼ ðm – 1Þ – 1 mi¼1 ðxi – x Þðxi – x ÞT is the (K K) sample variance–covariance matrix (estimate of the in-control (K K) covariance matrix S) from the m samples. Following Tracy et al.,31 in Phase I the Tt2 =ðm – 1Þ2 m – 1 statistic follows a beta distribution with K/2 and (mK1)/2 degrees of freedom. Therefore, UCLs at significance level (Type I risk) can be obtained for Phase I as UCLðT 2 ÞPhase I; ¼
ðm – 1Þ2 BðK =2;ðm – K – 1Þ=2Þ; m
ð17Þ
where BðK =2;ðm – K – 1Þ=2Þ; is the 100(1)% percentile of the corresponding beta distribution. This can be computed from the 100(1)% percentile of the corresponding F distribution by using the relationship BðK =2;ðm – K – 1Þ=2Þ; ¼
ðK =ðm – K – 1ÞÞFðK ;m – K – 1Þ; 1 þ ðK =ðm – K – 1ÞÞFðK ;m – K – 1Þ;
ð18Þ
114
Statistical Control of Measures and Processes
In Phase II, the Tt2 =K ðm – 1Þ2 ½mðm – K Þ – 1 statistic follows an F distribution with K and (mK) degrees of freedom. Thus, the corresponding UCL at significance level (Type I risk) is given by UCLðT 2 ÞPhase II; ¼
K ðm2 – 1Þ FðK ;ðm – K ÞÞ; mðm – K Þ
ð19Þ
If the multivariate normality assumption is not reasonable, control limits may be obtained from resampling methods such as bootstrap.6
1.04.9.1.2
Multivariate Shewhart-type control charts for monitoring process variability The Hotelling T 2 control charts assume that process variance–covariance structure, summarized in the (K K) covariance matrix S, remains constant. This assumption is not necessarily true and must be validated in practice. There are two indices for measuring the overall variability of a set of multivariate data: (i) the generalized variance jSj, that is, the determinant of the covariance matrix, and (ii) the trace of the covariance matrix trS, that is, the sum of the variances of the K variables. F. B. Alt32 presents two control charts for monitoring process variability based on the two previous indices. Assume that the vector x follows an NK ðm; SÞ and that there are m samples of size n > 1 available. The first procedure is a direct extension of the univariate s2 control chart. The procedure is equivalent to repeated test of significance of the null hypothesis that the process covariance matrix is S. The control statistic to be charted at stage t is Wt ¼ – Kn þ Kn lnðnÞ – n lnðjAt j=jSjÞ þ trðS – 1 At Þ
ð20Þ
where At ¼ (n1)St, St is the sample covariance matrix of the tth subgroup, and tr is the trace operator. The Wt statistic follows an asymptotic 2 distribution with K(K þ 1)/2 degrees of freedom. Hence, the UCL at significance level (Type I risk) is UCLðW Þ ¼ 2ðK ðK þ1Þ=2Þ; . The second approach is based on the sample generalized variance jSj (the square root of this quantity is proportional to the area or volume generated by a set of data). By using the mean and variance of jSj, that is, pffiffiffiffiffiffiffiffiffiffiffiffi ffi E(jSj) and V(jSj) respectively, the control limits are set at EðjSjÞ 3 VðjSjÞ. Although jSj is a widely used measure of multivariate dispersion, it is a relative simplistic scalar representation of a complex multivariate structure. Therefore, its use can be misleading in the sense that different correlation structures can yield identical generalized variance. To face this problem, the use of univariate dispersion charts in addition to the control chart for jSj is proposed.32 Other references to multivariate SPC methods for controlling the process variability can be found in Mason et al.33
1.04.9.1.3
MEWMA control charts The Hotelling T 2 control chart is a Shewhart-type control chart, that is, it uses information only from the current sample. As a result, it is relatively insensitive to small and moderate shift in the mean vector m. The MEWMA, introduced by Lowry et al.,34 is the multivariate extension of the univariate EWMA to provide more sensitivity to small shifts in the mean vector m. This is particularly useful in Phase II studies. Assuming individual observations (n ¼ 1), the MEWMA statistics is defined as wt ¼ xt þ ð1 – Þwt – 1
ð21Þ
where is a smoothing constant (0 < 1) and xt is the tth, K-dimensional observation sampled from a Kdimensional normal distribution, x NK ðm; SÞ. The control statistic to be charted at stage t is –1 MEWMA2t ¼ wT t wt wt
ð22Þ
where the covariance matrix is S wt ¼
1 – ð1 – Þ2t S 2–
ð23Þ
Statistical Control of Measures and Processes
115
which is analogous to the variance of the univariate EWMA. The control limits and tuning parameters necessary for implementing the MEWMA control chart are dependent on the shift in the mean vector to detect the in-control ARL and the number of characteristics K to be monitored.35 MEWMA control charts, like their univariate counterparts, are robust to the assumption of normality, if properly designed. Bersimis et al.36 provide an extensive review of several types of MEWMA control charts for different purposes, for example, to account for multivariate time-dependent observations and to deal with monitoring changes in process variability. 1.04.9.1.4
Fault diagnosis Once the multivariate control chart signals an out-of-control alarm, it is required to diagnose an assignable cause for it. This involves two steps: first (diagnostic) find which measured variable(s) contributes to the out-ofcontrol signal and second (corrective) determine what happens in the process that upsets the behavior of these variables. For pursuing the first step (isolation of variables responsible for the out-of-control signal), several approaches have been reported in the literature. An extensive review and references of diagnostic procedures in original variables-based MSPC schemes can be found in Mason et al.33 Bersimis et al.36 and Kourti and MacGregor.37 Regarding the second step (identifying root causes of the problem), management and operator actions based on technical process knowledge will be required.
1.04.9.2
Latent Variables-Based MSPC Schemes
Although original variables-based MSPC is well sounded from a statistical point of view, it suffers from lack of applicability in data-rich environments, typical of modern processes where, as commented in Section 1.04.3, the subgroup size is naturally n ¼ 1. This serious drawback comes from the fact that the multivariate control statistics to be charted need the inversion of a covariance matrix (see Equations (13), (16), (20), and (22)). To avoid problems with this inversion, the number of multivariate observations or samples (m) has to be larger than the number of variables (K), and covariance matrix has to be well conditioned (slightly correlated variables). In addition, complete data (no missing values) are required to work out the Hotelling T 2 or MEWMA2 statistics for any particular sample (see Equations (13), (16), and (22)). Nevertheless, these requirements are not met in highly automated processes where not only a few product quality variables but hundreds (or even thousands) of process variables are measured at a higher frequency rate. For treating large and ill-conditioned data sets that are not full statistical rank (m < K), we advocate the use of latent variables-based MSPC in the way they were proposed by Kourti and MacGregor.37 Latent variable methodology exploits the correlation structure of the original variables by revealing the few independent underlying sources of variation (latent variables) that are driving the process at any time. Multivariate statistical projection methods such as principal component analysis (PCA)38 are used to reduce the dimensionality of the monitoring space by projecting the information in the original variables down onto low-dimensional subspaces defined by a few latent variables. The process is then monitored in these latent subspaces by using a few multivariate control charts built from multivariate statistics that can be thought of as process performance indices or process wellness indices.39 These charts retain all the simplicity of presentation and interpretation of conventional univariate SPC charts. However, by using the information contained in all the measured (process and quality) variables simultaneously, they are much more powerful for detecting out-of-control conditions. As commented by T. Kourti,40 by monitoring only the quality variables (as frequently done in original variables-based MSPC) we are in fact performing statistical quality control (SQC). For true SPC, one must look at all the process data as well. Monitoring the process variables is expected to provide much more information on the state of the process and to supply it more frequently. Furthermore, any abnormal events that occur will also have their fingerprints in the process data. Thus, once an abnormal situation is detected, it is easier and faster to diagnose the source of the problem, as we are dealing directly with the process variables. On the contrary, control charts on the quality variables will only signal when the product properties are no longer consistent with expected performance, but they will not point to the process variables responsible for the problem, making the fault diagnostic process more difficult. Another advantage of monitoring process data is that quality data may not be available at certain stages of the process. Sometimes, product quality is determined only by the performance of the product later, during another
116
Statistical Control of Measures and Processes
process. For example, if a catalyst is conditioned in a batch process before being used for polymer production, the quality of the catalyst (success of conditioning) is assessed by its performance in the subsequent polymer production. It would be useful to know if the catalyst will perform well before using it; monitoring the batch process variables would detect abnormal situations and would provide an early indication of poor catalyst performance. In some cases, the scarce properties measured on a product are not enough to define entirely the product quality and performance for different applications. For example, if only the viscosity of a polymer is measured and kept within specifications, any variation in end-use application that arises owing to variation in chemical structure (branching, composition, and end-group concentration) will not be captured. In these cases, the process data may contain more information about events with special causes that may affect the product structure and thus its performance in different applications. Finally, by monitoring process variables, other abnormal operating conditions may be detected such as a pending equipment failure or a process failure.40 Finally, by using latent variables-based MSPC, missing and noisy data are easily handled, and predictive models based on projection to latent structures, such as partial least squares (PLS) or principal component regression (PCR), can also be used.41 Given the practical relevance of this approach, in the following an overview of implementation of MSPC schemes based on principal component analysis (PCA-based MSPC), describing both Phase I and II, is presented. 1.04.9.2.1 (Phase I)
PCA-based MSPC: exploratory data analysis and offline process monitoring
As commented in Section 1.04.2, the main goal in Phase I is to model the in-control process performance based on a set of historical in-control (reference) data. This data set is one in which the process has been operating consistently (stable over time) in an acceptable manner, and in which only good quality products have been obtained. Occasionally, this historical in-control data set is not directly available, but has to be extracted from historical databases in an iterative fashion as commented below. This explorative analysis of historical databases is a useful technique for improving process understanding and detecting past faults in the process (out-ofcontrol samples). By correctly diagnosing their root causes, some countermeasures can be implemented, optimizing the future performance of the process. Consider that the historical database consists of a set of m multivariate observations (objects or samples of size n ¼ 1) on K variables (online process measurements, dimensional variables, or product quality data) arranged in an (m K) data matrix Z. Variables in matrix Z are often preprocessed by mean centering and scaling to unit variance. With mean centering, the average value of each variable is calculated and then subtracted from the data. This usually improves the interpretability of the model because all preprocessed variables will have mean value zero. By scaling to unit variance, each original variable is divided by its standard deviation and will have unit variance. Given that projection methods are sensitive to scaling, this is particularly useful when the variables are measured in different units. After preprocessing, matrix Z is transformed into matrix X. PCA is used to reduce the dimensionality of the process by compressing the high-dimensional original data matrix X into a low-dimensional subspace of dimension A (A rank(X)), in which most of the data variability is explained by a fewer number of latent variables, which are orthogonal and linear combinations of the original ones. This is done by decomposing X into a set of A rank 1 matrices X¼
A X a¼1
t a pT a þ
rankðXÞ X
T ta pT a ¼ TP þ E ¼ X þ E
ð24Þ
a¼Aþ1
P (K A) is the loading matrix containing the loading vectors pa, which are the eigenvectors, corresponding to the A largest eigenvalues of the covariance matrix of the original pretreated data set X, and define the directions of highest variability of the new latent A-dimensional subspace. T (m A) is the score matrix containing the location of the orthogonal projection of the original observations onto the latent subspace. The columns ta of the score matrix T (ta ¼ Xpa) represent the new latent variables with variances given by their respective eigenvalues ( a). These new latent variables summarize the most important information of the original K variables, and thus can predict (reconstruct) X, by means X ¼ TPT, with minimum mean square error. Matrix E (m K) contains the residuals (statistical noise), that is, the information that is not explained by the PCA model.
Statistical Control of Measures and Processes
117
The dimension of the latent variable subspace is often quite small compared with the dimension of the original variable space (i.e., A << rank(X)). Several algorithms can be used to extract the principal components. For large ill-conditioned data sets, it is recommended to compute the principal components sequentially via the NIPALS (noniterative partial least squares) algorithm42 and to stop based on cross-validation criteria or some other procedure.38,43 Another advantage of NIPALS algorithm is that it easily handles missing data (i.e., observation vectors from which some variable measurements are missing). The quality of the fitted PCA model can be evaluated by computing several parameters, such as R2, which measures the goodness of fit, or Q2, which indicates the predictive capability of the model.44 Equation (24) shows that the PCA model transforms each K-dimensional original observation vector xi (the ith row of matrix X) into an A-dimensional score vector tT i ¼ fti1 ;ti2 ; . . .; tiA g(the ith row of matrix T) and a residual vector ei (the ith row of matrix E). From the scores and the residuals (prediction errors) associated with each observation, two complementary (orthogonal or independent) statistics are derived: the Hotelling’s TA2 and the SPE (sum of squared prediction errors). The TA2 statistic for the ith observation is defined as –1 TA2 ¼ tT i Q ti ¼
A X t2 a
a¼1
a
ð25Þ
where Q (A A) is the covariance matrix of T (diagonal matrix of the highest A eigenvalues f 1 ; . . .; A g). This statistic is the Hotelling T 2 statistic when a reduced subspace with A components is used instead of the original variables space, and it represents the estimated Mahalanobis distance from the center of the latent subspace to the projection of an observation onto this subspace. Under the assumption that the scores follow a multivariate normal distribution (they are linear combinations of random variables), it holds31 that in Phase I, TA2 (times a constant) has a beta distribution TA2
ðm – 1Þ2 BA=2;ðm – A – 1Þ=2 m
ð26Þ
whereas in Phase II, TA2 (times a constant) follows an F distribution TA2
Aðm2 – 1Þ FA;ðm – AÞ mðm – AÞ
ð27Þ
The difference in both distributions comes from the fact that in Phase I, the same observation vectors xi collected in the reference data set are used for two purposes: (i) to build the PCA model and work out the control limits of the charts and (ii) to check whether they fall within these control limits. Therefore, observations in the reference data set are not independent of PCA model parameters used to derive the statistics to be monitored. In contrast, in Phase II, new observations (not used for model building) are checked against the control limits calculated from the in-control data and, therefore, independence is guaranteed. However, if a large reference data set is available, Equation (27) can also be used for approximating the distribution of the TA2 statistic in Phase I. On the contrary, the SPE statistic for the ith observation xi is given by T SPE ¼ eT i ei ¼ ðxi – xi Þ ðxi – xi Þ
ð28Þ
where ei is the residual vector of the ith observation and x i is the prediction of the observation vector xi from the PCA model. The SPE statistic represents the squared euclidean (perpendicular) distance of an observation from this subspace, and gives a measure of how close the observation is from the A-dimensional subspace. Assuming that residuals follow a multivariate normal distribution, several approximate distributions for such quadratic forms have been proposed.44–46 From these two statistics, in PCA-based MSPC, two complementary multivariate control charts are constructed. Shewhart-type control charts for individual observations are often used in practice. Nevertheless, other types of multivariate charts such as MEWMA charts (see Section 1.04.9.1.3) may be used. The latter may be specially suited for autocorrelated processes.
118
Statistical Control of Measures and Processes
The control limits of the multivariate control charts are calculated following the traditional SPC philosophy. In Phase I, an appropriate historical or reference set of data (collected from one or different periods of plant operation or analytical process when performance was good) is chosen, which defines the normal or in-control operating conditions for a particular process corresponding to common-cause variation. The in-control PCA model is then built on these data. Any periods containing variations arising from special events that one would like to detect in the future are omitted at this stage. The choice of the reference (in-control) data set is critical to the successful application of the procedure.37 Control limits for good operation on the control charts are defined based on this reference data set. In Phase II, values of future measurements are compared against these limits. UCL for the Shewhart TA2 chart at significance level (Type I risk) can be obtained for Phase I from Equation (26): UCLðTA2 Þ ¼
ðm – 1Þ2 BðA=2;ðm – A – 1Þ=2Þ; m
ð29Þ
where BðA=2;ðm – A – 1Þ=2Þ; is the 100(1)% percentile of the corresponding beta distribution that can be computed from the 100(1)% percentile of the corresponding F distribution by using the following relationship:31 BðA=2;ðm – A – 1Þ=2Þ; ¼
ðA=ðm – A – 1ÞÞFðA;m – A – 1Þ; 1 þ ðA=ðm – A – 1ÞÞFðA;m – A – 1Þ;
ð30Þ
For Phase II, the corresponding UCL from Equation (27) is given by UCLðTA2 Þ ¼
Aðm2 – 1Þ FðA;ðm – AÞÞ; mðm – AÞ
ð31Þ
Regarding the UCL for the Shewhart SPE chart, several procedures can be used. Jackson and Mudholkar46 showed that an approximate SPE critical value at significance level is given by " pffiffiffiffiffiffiffiffiffiffiffi #1=h0 z 22 h02 2 h0 ðh0 – 1Þ UCLðSPEÞ ¼ 1 þ1þ 21 1
ð32Þ
PrankðXÞ where k ¼ j ¼Aþ1 ð j Þk , h0 ¼ 1 – ð21 3 =322 Þ, j are the eigenvalues of the PCA residual covariance matrix ETE/(N1), and z is the 100(1)% standardized normal percentile. Alternatively, one can use an approximation based on the weighted 2 distribution ðg 2h Þ proposed by Box.45 Nomikos and MacGregor47 suggested a simple and fast way to estimate the parameters g and h, which is based on matching moments between a g 2h distribution and the sample distribution of SPE. The mean ð ¼ ghÞ and variance ð2 ¼ 2g 2 hÞ of the g 2h distribution are equated with the sample mean (b) and variance (v) of the SPE sample. Hence, the upper SPE control limit at significance level is given by UCLðSPEÞ ¼
v 2 2 2b ð2b =vÞ;
ð33Þ
where 2ð2b2 =vÞ; is the 100(1)% percentile of the corresponding 2 distribution. Another method based on the statistical test of equality of variances from normal distributions is proposed by Eriksson et al.44 Based on the SPE, they define the absolute distance to the model (DModX) of an observation as its (corrected) residual standard deviation sffiffiffiffiffiffiffiffiffiffiffiffiffiffi SPE DModX ¼ c ðK – AÞ
ð34Þ
where c is a correction factor (function of the number of observations and the number of components) to be used in Phase I. This correction factor takes into account that the DModX is expected to be slightly smaller for an observation in the reference set because it has influenced the model. This correction matters only if the number of observations in the reference set is small. In Phase II, c ¼ 1.
Statistical Control of Measures and Processes
119
They also define the normalized distance to the model (DModXnorm) as DModX s0
ð35Þ
K –A 2 s FðK – A;ðm – A – 1ÞðK – AÞÞ; c2 0
ð36Þ
DModXnorm ¼
qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi Pm PK 2 where s0 ¼ i¼1 k¼1 eik =ðm – A – 1ÞðK – AÞ is the pooled residual standard deviation. This is an estimation of the residual variability taking into account all the observations used to build the model (reference data set). Assuming that the statistic (DModXnorm)2 has an approximate F distribution with KA and (mA1)(KA) degrees of freedom for the in-control observations, the UCL for the Shewhart SPE chart at significance level is expressed as UCLðSPEÞ ¼
where FðK – A;ðm – A – 1ÞðK – AÞÞ; is the 100(1)% percentile of the corresponding F distribution. The normality assumption on which these calculations are based is usually quite reasonable in practice. Nevertheless, control limits for multivariate charts can be obtained from distribution free methods by repeated sampling. The only requirement is to have a large in-control data set from which the external reference distribution48 for any statistic can be obtained. The two multivariate control charts (TA2 and SPE) differ in their conceptual meaning. They are two complementary indices that provide a picture of the wellness of the process at a glance.39 The TA2 chart checks if the projection of an observation on the hyperplane defined by the latent subspace is within the limits determined by the reference (in-control) data. Thus, a value of this statistic exceeding the control limits indicates that the corresponding observation presents abnormal extreme values in some (or all) of its original K variables, even though it maintains the correlation structure between the variables in the model. This observation can be tagged as an abnormal outlier inside the PCA model (an extremist or severe outlier).41 On the contrary, the SPE chart checks if the distance (noise variation) of an observation to the latent hyperplane is inside the control limits. The SPE chart values exceeding the control limits are related to observations that do not behave in the same way as the ones used to create the model (in-control data), in the sense that there is a breakage of the correlation structure of the model. This chart will detect the occurrence of any new event that causes the process to move away from the hyperplane defined by the reference model. This kind of observations can be tagged as outliers outside the model (an alien or moderate outlier).41 Severe outliers are influential observations with high leverage on the model, that is, with strong power to pull the principal directions toward themselves, creating fictitious components and misleading the PCA model.44 Therefore, model validation is extremely needed in the Phase I stage to remove from the data matrix these dangerous outlier (out-of-control) observations and, afterward, recalculate the PCA model. However, before removing any observation from the data matrix, some diagnostics using contribution plots (discussed below) and process insight should be used to sort out false alarms from the real ones. This process of model building and validation is done iteratively until no multivariate control chart signals any real outlier. As a side effect of this debugging procedure, the root causes of the out-of-control observations can be discovered, improving process knowledge and future process performance.
1.04.9.2.2
PCA-based MSPC: online process monitoring (Phase II) Once the reference PCA model and the control limits for the multivariate control charts are obtained, new process observations can be monitored online. When a new observation vector zi is available, after preprocessing it is projected onto the PCA model yielding the scores and the residuals, from which the value of the Hotelling TA2 and the value of the SPE are calculated. This way, the information contained in the original K variables is summarized in these two indices, which are plotted in the corresponding multivariate TA2 and SPE control charts. No matter what the number of the original variables K is, only two points have to be plotted on the charts and checked against the control limits. The SPE chart should be checked first. If the points remain below the control limits in both charts, the process is considered to be in control. If a point is detected to be beyond the limits of one of the charts, then a diagnostic approach to isolate the original variables responsible for
120
Statistical Control of Measures and Processes
the out-of-control signal is needed. In PCA-based MSPC, contribution plots37 are commonly used for this purpose. Contribution plots can be derived for abnormal points in both charts. If the SPE chart signals a new out-ofcontrol observation, the contribution of each original kth variable to the SPE at this new abnormal observation is given by its corresponding squared residual: 2 ContðSPE; xnew;k Þ ¼ enew;k ¼ ðxnew;k – xnew;k Þ2
ð37Þ xnew;k
where enew,k is the residual corresponding to the kth variable in the new observation and is the prediction of the kth variable xnew;k from the PCA model. In case of using the DModX statistic, the contribution of each original kth variable to the DModX is given by44 ContðDModX; xnew;k Þ ¼ wk enew;k
ð38Þ
where wk is the square root of the explained sum of squares for the kth variable. Variables with high contributions in this plot should be investigated. If the abnormal observation is detected by the TA2 chart, the diagnosis procedure is carried out in two steps: (i) a bar plot of the normalized scores for that observation (tnew,a/ a)2 is plotted and the ath score with the highest normalized value is selected; (ii) the contribution of each original kth variable to this ath score at this new abnormal observation is given by Contðtnew;a ; xnew;k Þ ¼ pak xnew;k
ð39Þ
where pak is the loading of the kth variable at the ath component. A plot of these contributions is created. Variables on this plot with high contributions but with the same sign as the score should be investigated (contributions of the opposite sign will only make the score smaller). When there are some scores with high normalized values, an overall average contribution per variable can be calculated over all the selected scores.39 Contribution plots are a powerful tool for fault diagnosis. They provide a list of process variables that contribute numerically to the out-of-control condition (i.e., they are no longer consistent with NOCs), but they do not reveal the actual cause of the fault. Those variables and any variables highly correlated with them should be investigated. Incorporation of technical process knowledge is crucial to diagnose the problem and discover the root causes of the fault. Apart from the TA2 and SPE control charts, other charts such as the univariate time-series plots of the scores or scatter score plots can be useful (both in Phase I and II) for detecting and diagnosing out-of-control situations and also for improving process understanding.
1.04.9.3
Potential of Latent Variables-Based MSPC in Industry
As commented by Kourti,39 multivariate monitoring schemes based on latent variables methods have been receiving increasing attention by industrial practitioners in the last 20 years. Several companies have enthusiastically adopted the methods and have reported many success stories. Latent variables-based MSPC has been used for process monitoring, fault detection, and diagnosis in continuous and batch industrial processes. Kourti39 also provides a long list of publications reporting successful industrial applications in different fields (e.g., petrochemical, polymer, and chemical). In pharmaceutical industry, latent variables-based MSPC scheme is playing a critical role in the successful implementation of process analytical technology (PAT) supported by the United States Food and Drug Administration (FDA). PAT has been defined by the FDA as systems for the analysis and control of manufacturing processes based on timely measurements during processes of critical quality parameters and performance attributes of raw and in-process materials and processes to assure acceptable end product quality at the completion of the process.49 With this initiative, the FDA tries to motivate the pharmaceutical industry to improve process control strategies for highquality, cost-effective pharmaceutical products. Kourti50 discusses the critical role of latent variables-based MSPC methods for process understanding, abnormal situation detection, and fault diagnosis, as linked to PAT.
Statistical Control of Measures and Processes
121
Latent variables-based MSPC techniques can also be used to extract subtle information from digital images related to product quality and use such information for prediction, monitoring, and control. The methodology, based on multivariate image analysis (MIA),51 can be used to monitor solids and other heterogeneous materials (lumber, steel sheets, pulp and paper products, polymer films, multiphase streams, etc.). Image analysis provides informative, inexpensive, and robust online sensors for the solids industry. Therefore, it opens new ways for the successful monitoring and control of processes, which was traditionally difficult owing to lack of sensors. Several applications have been reported in the literature: online monitoring of time-varying images52 and lumber defects;53 monitoring and control of the amount of coating applied to the base food product and the distribution of the coating among the individual product pieces in snack food industry;54,55 defect detection and classification in ceramic tiles, artificial stone countertops, and orange fruits.56 Maintaining and controlling data quality is a key problem in large-scale microarray studies. In particular, systematic changes in experimental conditions across multiple chips can seriously affect quality and even lead to false biological conclusions. Model et al.57 introduce a novel approach for microarray process control based on MSPC. 1.04.9.3.1
Example 3 An example of successful implementation of latent variables-based MSPC in wastewater treatment plants is presented in Aguado et al.58 Data were collected from a sequencing batch reactor (SBR) operated for enhanced biological phosphorus removal from wastewater. The reactor was equipped with a mechanical mixer, an air diffuser, and several electronic sensors: temperature (Temp), pH, electric conductivity (Cond), oxidation– reduction potential (ORP), and dissolved oxygen (DO). Data from the five electronic sensors were recorded approximately every 1 min along the duration of the batch yielding 340 values per sensor and a total of K ¼ 1700 values per batch. The historical database available corresponding to 4 months of operation with the SBR was used to build the in-control model in Phase I. A PCA-based MSPC scheme was designed after several cycles of model validation and outlier detection (out-of-control batches). Finally, m ¼ 75 batches were selected to build the in-control model. Technical knowledge of the process was also incorporated to check if the real faulty batches in the database were detected and the responsible variables correctly identified (diagnosis ability). Note that in this case the structure of the matrix X (75 1700) makes unfeasible the use of original variables-based MSPC introduced in Section 1.04.9.1. Figure 9 illustrates the proposed PCA-based MSPC scheme. The upper part shows the DModX control chart (from Equation (34)) (Figure 9, top left) and TA2 control chart (from Equation (25)) (Figure 9, top right) at significance level ¼ 0.01. The first 30 batches represent in-control batches from Phase I. Batches 31 and 32 are new batches monitored in Phase II. Batch 31 looks in control, whereas batch 32 exceeds the UCL in both charts (values out of range), suggesting that this is an out-of-control batch. To isolate the variables responsible for this out-of-control situation, a contribution plot in DModX of out-of-control batch 32 is displayed (Figure 9, bottom left). The plot signals DO as the process variable related to the fault. The negative values of the contributions indicate that DO takes lower values than required, may be due to an aeration system fault. This is confirmed in Figure 9 (bottom right) by comparing the trajectory of DO for batch 32 (out-of-control) to that of an in-control batch. Around the time point 124, the DO dramatically drops in batch 32, confirming a fault in the aeration system control that should be fixed before a new batch starts. The potential of latent variables-based MSPC is really impressive nowadays and will be a key tool for success in the future high-tech environments.
1.04.9.4 Monitoring Measurement Systems: Applications of SPC in Analytical Laboratory Methods Every analytical or measurement process that works continuously (e.g., in routine analysis, or a method used for a long period of time) can be considered as a process with inputs (the samples) and output (the results of the analysis: measurements). Hence, the basics of SPC can be applied to monitor them leading to the so-called analytical process monitoring. In this case, it is the quality of the measurement that matters. Control charts can be used to attain and maintain statistical control of measuring systems. When the control chart signals an
122
Statistical Control of Measures and Processes
40
T 2 [7]
30 D crit (99%)
1.3 0.9
20 10 0
0.5 0
5
10
15 20 Batch
25
0
30
4
6
2
5
0 DO
–2 –4 –6 ORP
pH
–8
Cond
Temp
–10
4
DO
Contribution in DmodX
T 2 crit (99%)
(mg l–1)
DmodX [7]
1.7
2
5
10
Anaerobic
15 20 Batch
25
Aerobic Faulty batch Normal batch
30
Settle
3
1 0
0
340
680 1020 1360 Variable number
1700
0
50
100
150 200 Time instant
250
300
Figure 9 PCA-based MSPC scheme for SBR data. DModX control chart (top left) and TA2 control chart (top right): the solid red line represents control limits. Contribution plot in DModX of out-of-control batch 32 (bottom left) signaling DO as the process variable related to the fault. Trajectory of DO for batch 32 (out of control due to an aeration system fault) and for an incontrol batch (bottom right).
out-of-control measurement, the analyst must search for the assignable cause and fix the problem before reanalyzing some of the previous samples and continuing with other analyses.59 Instrumental methods of analysis are usually comparative in nature and the analytical instruments require periodic recalibration to compensate for the effects of uncontrollable environmental factors and gradual changes in instrument performance. To be confident of the test results, the analyst should have systems for monitoring and controlling the accuracy and precision of the test method. Standard or check samples (samples with known properties) are used to test analytical methods periodically to ensure that they are in control. For example, latex suspensions with known particle size are used to test the calibration in particle sizing methods, and samples with known concentration of an analyte are used to test gas chromatography. The results for these samples can be plotted on control charts to test that the value measured exhibits only common-cause variation. When the response obtained from the standard is not a single number (i.e., instead of composition of one analyte, we get compositions of several analytes), one should consider multivariate charts.40 In chromatography, process variables such as flow rate and pressure can be of significant importance for the performance of the chromatographic system, whereas, for example, column aging or a contaminated detector can take the system out of control. Nijhuis et al.60 present a PCA-based MSPC scheme of a chromatographic system for the quantification of the total content of trans fatty acids in vegetable oils by capillary gas chromatography using a check sample containing five analytes to test column performance. A TA2 chart and an SPE chart are used to monitor analyte peak area percentage of five analytes of the check sample along time. The multivariate monitoring scheme clearly outperforms the univariate approach in the sense of reducing the number of false alarms and allowing the detection of real out-of-control observations due to a change in correlation. Zhu et al.61 use online high-performance liquid chromatography (HPLC) to monitor a steady-state reaction over 35.2 h, with 197 chromatograms recorded as the reaction progresses. For each chromatogram, peaks are detected, baseline corrected, aligned, and integrated to provide a peak table consisting of intensities of 19 peaks,
Statistical Control of Measures and Processes
123
2 corresponding to the reactants, 1 to the product, and 1 to the solvent, the remaining being impurities, byproducts, or intermediates. A PCA-based MSPC scheme successfully detects out-of-control samples and provides diagnostic insight (using contribution plots) into why these samples are problematic. The TA2 charts appear to be better at monitoring the reactants and products, and could be employed, for example, to determine if there were problems with mixing or difficulties with instrument operation, whereas the SPE charts are best at detecting impurities during the reaction. When developing multivariate calibration models (as in SIA), it is crucial to ensure that the analytical system works under statistical control during the modeling process, and that it remains under control while the unknown samples are analyzed. To do so, a representative control sample is analyzed repeatedly before and during the multivariate model development to check the system’s stability. Rius et al.14 illustrate the use of a PCA-based MSPC approach in the development of an SIA method for determining sulfates in natural water using spectral data. As shown in the previous examples, many analytical methods are based on liquid chromatography and commonly the only measure of system stability is standards, injected repeatedly throughout the sequence. Fransson et al.62 present a liquid chromatographic process control (LCPC) system where the analytical run (chromatographic analysis) is treated as a process with the chromatographic data as the product. To ensure high quality of the analytical data, focus is on not only already existing postrun checks of retention time, plate numbers, resolution, and peak areas, but also the process monitoring itself. Several key parameters, such as the pressure at the column, the injection valve, the temperature, and the conductivity of the mobile phase, are monitored. LCPC provides better control of the analysis, enhancing the quality of the data obtained. This allows a reduction in the number of standards and replicates. Furthermore, troubleshooting is facilitated: Using contribution plots, system disturbance can easily be tracked back to the original parameters that caused the incident. The authors state that the approach can also be adapted to other flow systems, such as gas chromatography, capillary electrophoresis, capillary electrochromatography, and flow injection systems. Apart from monitoring spectral signals in analytical processes, latent-based MSPC techniques can also be used to diagnose the possible bias in analytical methods as a means of internal validation in analytical laboratories. Ortiz-Estarelles et al.63 illustrate the use of MSPC tools for automatic detection of possible errors in the method used for routine multiparametric analysis to design an internal multivariate analytical quality control (iMAQC) program. Such tools have the ability to signal possible failures in the analytical methods without resorting to any external reference, as they use their own analytical results as a source for the diagnosis of the method’s quality. The potential of the proposed approach is illustrated in the routine analysis of drinking water.
1.04.10 Software There exist many commercial software packages containing univariate SPC tools. A brief list includes Statgraphics Centurion from StatPoint Inc., Minitab, STATISTICA from StatSoft, SAS, and SPSS. Microsoft Excel also provides a basic univariate SPC module. Some of these packages (Statgraphics Centurion, Minitab, and STATISTICA) also include multivariate SPC tools in the original variables. Nevertheless, latent-based MSPC tools are not very common in commercial software; out of the already mentioned, only STATISTICA offers these tools. Another multivariate specialized software package is SIMCA from Umetrics. Some websites offer MATLAB freeware with specialized MSPC tools (e.g., the Multivariate Statistical Engineering Research Group website: http://mseg.webs.upv.es).
Acknowledgments The author gratefully acknowledges D. Aguado and the CALAGUA research group for providing the experimental data of the wastewater treatment process, and Professor M. P. Callao for providing the SIA sulfate determinations data set.
124
Statistical Control of Measures and Processes
References 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15. 16. 17. 18. 19. 20. 21. 22. 23. 24. 25. 26. 27. 28. 29. 30. 31. 32. 33. 34. 35. 36. 37. 38. 39.
Montgomery, D. C. Introduction to Statistical Quality Control, 5th ed.; Wiley: New York, 2005. Woodall, W. H. Controversies and Contradictions in Statistical Process Control. J. Qual. Technol. 2000, 32 (4), 341–350. Hawkins, D. M.; Olwell, D. H. Cumulative Sum Charts and Charting for Quality Improvement; Springer: New York, 1988. Box, G. E. P.; Lucen˜o, A. Statistical Control by Monitoring and Feedback Adjustment; Wiley: New York, 1997. Shewhart, W. A. Economic Control of Quality of Manufactured Product; Van Nostrand: New York, 1931. Reissued by the American Society for Quality: Milwaukee, WI, 1980. Liu, R. Y.; Tang, J. Control Charts for Dependent and Independent Measurements Based on Bootstrap Methods. J. Am. Stat. Assoc. 1996, 91, 1694–1700. McNeese, W. H.; Klein, R. A. Statistical Methods for the Process Industries (Quality & Reliability 28); Marcel Dekker: New York, 1991. Champ, C. W.; Woodall, W. H. Exact Results for Shewhart Control Charts with Supplementary Run Rules. Technometrics 1987, 29 (4), 393–399. Page, E. S. Continuous Inspection Schemes. Biometrika 1954, 41, 110–115. Page, E. S. Cumulative Sum Control Charts. Technometrics 1961, 3 (1), 1–9. Roberts, S. Control Chart Tests Based on Geometric Moving Averages. Technometrics 1959, 42 (1), 97–102. Hunter, J. S. Exponentially Weighted Moving Average. J. Qual. Technol. 1986, 18, 203–210. Rius, A.; Ruisa´nchez, I.; Callao, M. P.; Rius, F. X. Reliability of Analytical Systems: Use of Control Charts, Time Series Models and Recurrent Neural Networks (RNN). Chemom. Intell. Lab. Syst. 1998, 40, 1–18. Rius, A.; Callao, M. P.; Rius, F. X. Multivariate Statistical Process Control Applied to Sulfate Determination by Sequential Injection Analysis. Analyst 1997, 122, 737–741. Box, G. E. P.; Jenkins, G. M.; Reinsel, G. C. Time Series Analysis: Forecasting and Control, 3rd ed.; Prentice Hall: New Jersey, 1994. Callao, M. P.; Rius, A. Time Series: A Complementary Technique to Control Charts for Monitoring Analytical Systems. Chemom. Intell. Lab. Syst. 2003, 66, 79–87. Vander Wiel, S. A.; Tucker, W. T.; Faltin, F. W.; Doganaksoy, N. Algorithmic Statistical Process Control: Concepts and an Application. Technometrics 1992, 34, 286–297. Faltin, F. W.; Hahn, G. J.; Tucker, W. T.; Vander Wiel, S. A. Algorithmic Statistical Process Control: Some Practical Observations. Int. Stat. Rev. 1993, 61, 67–80. Box, G. E. P.; Kramer, T. Statistical Process Monitoring and Feedback Adjustment – A Discussion. Technometrics 1992, 34, 251–285. Capilla, C.; Ferrer, A.; Romero, R.; Hualda, A. Integration of Statistical and Engineering Process Control in a Continuous Polymerization Process. Technometrics 1999, 41, 14–28. Ferrer, A.; Barcelo´, S.; Hermenegildo, F. Engineering Statistical Process Control (ESPC) in a Tile Industry. Proceedings of the Industrial Statistics in Action 2000 International Conference. University of Newcastle upon Tyne, Newcastle, 2000; pp 145–158. Janakiram, M.; Keats, J. B. Combining SPC and EPC in a Hybrid Industry. J. Qual. Technol. 1998, 30, 189–200. MacGregor, J. F. On-Line Statistical Process Control. Chem. Eng. Prog. 1988, 84, 21–31. Montgomery, D. C.; Keats, B. J.; Runger, G. C.; Messina, W. S. Integrating Statistical Process Control and Engineering Process Control. J. Qual. Technol. 1994, 26, 79–87. MacGregor, J. F. Using On-Line Process Data to Improve Quality. Is There a Role for Statisticians? Are They Up for the Challenge? ASQC Stat. Div. Newsl. 1996, 16 (2), 6–13. Ferrer, A. Multivariate Statistical Process Control Based on Principal Component Analysis (MSPC-PCA): Some Reflections and a Case Study in an Autobody Assembly Process. Qual. Eng. 2007, 19, 311–325. Hotelling, H. Multivariate Quality Control. In Techniques of Statistical Analysis; Eisenhart, C., Hastay, M., Wallis, W. A., Eds.; MacGraw-Hill: New York, 1947; pp 111–184. Jackson, J. E. Multivariate Quality Control. Commun. Stat. Theor. Meth. 1985, 14 (11), 2657–2688. Wierda, S. J. Multivariate Statistical Process Control – Recent Results and Directions for Future Research. Stat. Neerl. 1994, 48 (2), 147–168. Fuchs, C.; Kenett, R. S. Multivariate Quality Control: Theory and Applications (Quality & Reliability 54); Marcel Dekker: New York, 1998. Tracy, N. D.; Young, J. C.; Mason, R. L. Multivariate Control Charts for Individual Observations. J. Qual. Technol. 1992, 24 (2), 88–95. Alt, F. B. Multivariate Quality Control. In The Encyclopedia of Statistical Sciences; Kotz, S., Johnson, N. L., Read, C. R., Eds.; Wiley: New York, 1985; pp 110–122. Mason, R. L.; Champ, C. W.; Tracy, N. D.; Wierda, S. J.; Young, J. C. Assessment of Multivariate Process Control Techniques. J. Qual. Technol. 1997, 29 (2), 140–143. Lowry, C. A.; Woodall, W. H.; Champ, C. W.; Rigdon, S. E. A Multivariate Exponentially Weighted Moving Average Control Chart. Technometrics 1992, 34, 46–53. Kulahci, M.; Borror, C. Advanced Statistical Process Control. In Statistical Practice in Business and Industry; Coleman, S., Greenfield, T., Stewardson, D., Montgomery, D. C., Eds.; Wiley: West Sussex, UK, 2008; Chapter 13, pp 211–237. Bersimis, S.; Psarakis, S.; Panaretos, J. Multivariate Statistical Process Control Charts: An Overview. Qual. Reliab. Eng. Int. 2007, 23, 517–543. Kourti, T.; MacGregor, J. F. Multivariate SPC Methods for Process and Product Monitoring. J. Qual. Technol. 1996, 28 (4), 409–428. Jackson, J. E. A User’s Guide to Principal Components; Wiley: New York, 2003. Kourti, T. Application of Latent Variable Methods to Process Control and Multivariate Statistical Process Control in Industry. Int. J. Adapt. Control Signal Process. 2005, 19, 213–246.
Statistical Control of Measures and Processes
125
40. Kourti, T. Process Analysis and Abnormal Situation Detection: From Theory to Practice. IEEE Control Syst. 2002, 22, 10–25. 41. Martens, H.; Naes, T. Multivariate Calibration; Wiley: New York, 1989. 42. Wold, H. Nonlinear Estimation by Iterative Least Squares Procedures. In Research Papers in Statistics; David, F., Ed.; Wiley: New York, 1966; pp 411–444. 43. Wold, S. Cross-Validatory Estimation of the Number of Components in Factor and Principal Component Models. Technometrics 1978, 20 (4), 397–405. 44. Eriksson, L.; Johansson, E.; Kettaneh-Wold, N.; Wold, S. Multi- and Megavariate Data Analysis: Principles and Applications; Umetrics AB, 2001. 45. Box, G. E. P. Some Theorems on Quadratic Forms Applied in the Study of Analysis of Variance Problems: Effect of Inequality of Variance in One-Way Classification. Ann. Math. Stat. 1954, 25, 290–302. 46. Jackson, J. E.; Mudholkar, G. S. Control Procedures for Residuals Associated with Principal Component Analysis. Technometrics 1979, 21 (3), 341–349. 47. Nomikos, P.; MacGregor, J. F. Multivariate SPC Charts for Monitoring Batch Processes. Technometrics 1995, 37 (1), 41–59. 48. Box, G. E. P.; Hunter, J. S.; Hunter, W. G. Statistics for Experimenters: Design, Innovation and Discovery, 2nd ed.; Wiley: Hoboken, NJ, 2005. 49. Guidance for Industry, PAT – A Framework for Innovative Pharmaceutical Development, Manufacturing, and Quality Assurance; U.S. Department of Health and Human Services, Food and Drug Administration, Center for Drug Evaluation and Research (CDER), Center for Veterinary Medicine (CVM), Office of Regulatory Affairs (ORA), Pharmaceutical CGMPs, September 2004. http://www.fda.gov/cder/guidance/6419fnl.pdf 50. Kourti, T. Process Analytical Technology beyond Real-Time Analyzers: The Role of Multivariate Analysis. Crit. Rev. Anal. Chem. 2006, 36, 257–278. 51. Geladi, P.; Grahn, H. Multivariate Image Analysis; Wiley: Chichester, UK, 1996. 52. Bharati, M. H.; MacGregor, J. F. Multivariate Image Analysis for Real Time Process Monitoring and Control. Ind. Eng. Chem. Res. 1998, 37, 4715–4724. 53. Bharati, M.; MacGregor, J. F. Softwood Lumber Grading through On Line Multivariate Image Analysis. Ind. Eng. Chem. Res. 2003, 42, 5345–5353. 54. Yu, H.; MacGregor, J. F. Multivariate Image Analysis and Regression for Prediction of Coating and Distribution in the Production of Snack Foods. Chemom. Intell. Lab. Syst. 2003, 67, 125–144. 55. Yu, H.; MacGregor, J. F.; Haarsma, G.; Bourg, W. Digital Imaging for On-Line Monitoring and Control of Industrial Snack Food Processes. Ind. Chem. Eng. Res. 2003, 42, 3036–3044. 56. Prats-Montalba´n, J. M.; Ferrer, A. Integration of Colour and Textural Information in Multivariate Image Analysis: Defect Detection and Classification Issues. J. Chemom. 2007, 21, 10–23. 57. Model, F.; Konig, T.; Piepenbrock, C.; Adorjan, P. Statistical Process Control for Large Scale Microarray Experiments. Bioinformatics 2002, 18 (Suppl 1), S155–S163. 58. Aguado, D.; Ferrer, A.; Ferrer, J.; Seco, A. Multivariate SPC of a Sequencing Batch Reactor for Wastewater Treatment. Chemom. Intell. Lab. Syst. 2007, 85, 82–93. 59. Paul, W. L.; Barnett, N. S. Control Charting Instrumental Analyses. Lab. Autom. Inform. Manag. 1995, 31, 141–148. 60. Nijhuis, A.; de Jong, S.; Vandeginste, B. G. M. Multivariate Statistical Process Control in Chromatography. Chemom. Intell. Lab. Syst. 1997, 38, 51–62. 61. Zhu, L.; Brereton, R. G.; Thompson, D. R.; Hopkins, P. L.; Escott, R. E. A. On-Line HPLC Combined with Multivariate Statistical Process Control for the Monitoring of Reactions. Anal. Chim. Acta 2007, 584, 370–378. 62. Fransson, M.; Spare´n, A.; Lagerholm, B.; Karlsson, L. On-Line Process Control of Liquid Chromatography. Anal. Chem. 2001, 73 (7), 1502–1508. 63. Ortiz-Estarelles, O.; Martı´n-Biosca, Y.; Medina-Herna´ndez, M. J.; Sagrado, S.; Bonet-Domingo, E. On the Internal Multivariate Quality Control of Analytical Laboratories. A Case Study: The Quality of Drinking Water. Chemom. Intell. Lab. Syst. 2001, 56, 93–103.
126
Statistical Control of Measures and Processes
Biographical Sketch
Alberto Ferrer is Professor of Statistics in the Department of Applied Statistics, Operation Research and Quality, and Head of the Multivariate Statistical Engineering Research Group of the Technical University of Valencia, Spain. He holds an M.Sc. in Agricultural Engineering and a Ph.D. in Statistics. His research focuses on statistical techniques for quality and productivity improvement, especially those related to multivariate statistical projection methods. Professor Ferrer is currently an associate editor of Technometrics, a member of the editorial board of Quality Engineering, a member of the Council of the International Society for Business and Industrial Statistics (ISBIS), and a member of the European Network for Business and Industrial Statistics (ENBIS) and the Spanish Chemometrics Network. He is also active as industrial consultant on Six Sigma and Process Analytical Technology (PAT).
1.05 Quality of Analytical Measurements: Univariate Regression M. C. Ortiz, M. S. Sa´nchez, and L. A. Sarabia, University of Burgos, Burgos, Spain ª 2009 Elsevier B.V. All rights reserved.
1.05.1 1.05.2 1.05.2.1 1.05.2.2 1.05.2.3 1.05.3 1.05.3.1 1.05.3.1.1 1.05.3.1.2 1.05.3.2 1.05.3.2.1 1.05.3.2.2 1.05.4 1.05.4.1 1.05.4.2 1.05.4.3 1.05.4.4 1.05.4.5 1.05.4.6 1.05.5 1.05.6 1.05.7 1.05.8 1.05.9 1.05.10 1.05.11 References
Introduction Linear Regression in Calibration: Elements and Procedure Estimation by Least Squares Estimation when the Distribution of " Follows a N(0, 2) The Correlation between Concentration, x, and Signal, y Statistical Validation of a Calibration Model Validity of the Functional Model F-test for significance of regression Lack of fit test Verification of the Hypotheses on the Residuals Graphical methods Inferential methods Confidence Intervals and Hypothesis Testing Confidence Intervals for the Intercept, the Slope, and the Residual Variance Joint Confidence Region for 0 and 1 Comparison of Two Regression Lines Confidence Interval for the Prediction Confidence Interval for the Determination of the Concentration of a Problem Sample The Intersection of Two Linear Regression Lines The Design of a Calibration The Capability of Detection, the Decision Limit, and the Capability of Discrimination Computed from a Regression Model Standard Addition Method Weighted Least Squares and Generalized Least Squares Robust Regression in Calibration Errors in Both Variables Final Remark
Symbols bj CC Cov(X,Y) E(X) ei F,v1,v2
Fv1,v2 Ha
estimated coefficients capability of detection (European Decision 2002/657) covariance between random variables X and Y expectation of random variable X residuals upper -perecentage point of an F distribution with v1 and v2 degrees of freedom so that ¼ pr{x > F,v1,v2} F distribution with v1 and v2 degrees of freedom alternative hypothesis
H0 N N(,2) NID(,2)
rxy s2p s2x s2yx t,v
128 129 129 131 133 134 134 134 135 136 137 138 140 140 141 142 144 145 147 148 151 153 154 159 163 164 165
null hypothesis number of observations (objects) normal distribution with mean and variance 2 independent random variables equally distributed as normal with mean and variance 2 sample correlation between x and y pooled estimated common variance variance (of x’s data) residual variance of the regression line upper -percentage point of a student’s t distribution with v degrees of freedom so that = pr{x > t,v}
127
128
Quality of Analytical Measurements: Univariate Regression
tv Var(X) x xc xd x xˆi y yˆi
student’s t distribution with v degrees of freedom variance of random variable X independent (predictor) variable decision limit (ISO 11843-1) capability of detection (ISO 11843-1) mean (of x’s data) predicted value of xi dependent (response) variable predicted value of yi significance level, probability of type I error, probability of false positive
j " xy 2 2v 2 ,v
probability of type II error, probability of false negative true coefficients random variable (experimental error) true mean correlation coefficient between x and y true variance (chi-squared) distribution with v degrees of freedom upper -percentage point of a chisquared distribution with v degrees of 2 } freedom so that ¼ pr{x > ,v
1.05.1 Introduction In Dux,1 a calibration model is defined as the verification of the response of an instrument to a material of known properties and, maybe, the correction by a factor to take the instrument to the corresponding mark, whereas standardization means to characterize the response of an instrument according to the known properties of the material. The standardization of an instrument is usually done by the ‘calibration curve’. These two terms are often mistakenly used interchangeably. Many of the present-day instruments consist of different systems and their complexity makes it difficult to calibrate them. In fact, what is done is standardizing the response by means of a series of samples of known concentration. These standards of concentration should be of enough purity, or a reference material with certified quality characteristics should be used. Many different meanings have been attributed to the term calibration: for example, the International Union of Pure and Applied Chemistry (IUPAC)2 recognizes two different versions: 1. Calibration function for species identification and qualitative analysis. That is, calibration of those analytical parameters that characterize types of chemical species by establishing a model for the purpose of identification and qualitative analysis (for instance, a model that identifies a compound on the basis of its chromatographic retention time). 2. Calibration function for quantitative analysis, whose main purpose is to obtain a function that allows one to calculate the concentration (or amount) of an analyte as a function of an instrumental signal. In most cases, the calibration function has to take into account the response relations of all relevant constituents and interferents. In the document from the International Organization for Standardization (ISO)3 related to the capability of detection, the practical relevance of the calibration is highlighted: ‘‘The quantitative calibration of a system of measurement is central in many chemical measurements. It is relevant for the trazability of the measurement, contributes to its uncertainty and many figures of merit of the analytical methods are directly related to the calibration.’’ The calibration curve is fit, usually, by the statistical technique of linear regression, which is a particular case of the more general framework of fitting models to experimental data. To fit empirical models involves specific peculiarities caused by the type of data used for the fitting; for this reason, a net chemometric approach has been adopted that emphasizes its utility for the calibration, the determination of the sensitivity of a method, the linear range, the detection of possible matrix and interferent effects, the comparison of two methods, the calculation of a detection limit, etc.
Quality of Analytical Measurements: Univariate Regression
129
1.05.2 Linear Regression in Calibration: Elements and Procedure We will establish the notation and the basic aspects of regression in the simplest case of univariate linear regression. Let x denote the independent variable and y the response. When calibrating, x will be the concentration of the analyte in each standard sample and y denotes the recorded signal. The random error supposed for the response is denoted as ". The model for the observed experimental responses is supposed to be the following: y ¼ f ðx Þ þ "
ð1Þ
where f(x) is the calibration curve to be estimated. To do this, N observations are recorded, yi, that correspond to k calibration standards. If there are no replicate observations, then N ¼ k, otherwise N > k. As usual, for each yi, the corresponding concentration of the standard is written as xi, without distinguishing whether they are equal (replicates) or not. Therefore, yi ¼ f ðxi Þ þ "i ;
i ¼ 1; . . .; N
ð2Þ
In many calibration problems, the following hypotheses are considered whose compatibility with the experimental results will have to be carefully evaluated: 1. The true calibration curve f is linear in the studied range of concentrations. Hence yi ¼ 0 þ 1 xi þ "i ;
i ¼ 1; . . .; N
ð3Þ
2. The values of the predictor variable (xi, i ¼ 1,2 , . . ., N) are controlled values and their experimental uncertainty can be neglected compared with the uncertainty in the determination of the response. 3. The random errors in the observed response are uncorrelated, Covð"i ; "j Þ ¼ 0; if i 6¼ j , have zero mean, E ð"i Þ ¼ 0; i ¼ 1; . . .; N , and the same variance, Var ð"i Þ ¼ 2 ; i ¼ 1; . . .; N , independent of 0, 1, and xi. As a result of the hypotheses on the errors, each response yi is a random variable whose mean is a linear function of the value of the predictor E ðyi Þ ¼ 0 þ 1 xi , its variance is Varðyi Þ ¼ 2 , and the observed values (y1, . . ., yN) are uncorrelated, that is, Covðyi ; yj Þ ¼ 0; if i 6¼ j . Therefore, the problem of estimating the model of Equation (1), or calibration line, is technically equivalent to estimating the parameters 0, 1, and from the experimental results of the calibration set C ¼ fðxi ; yi Þ; i ¼ 1; . . .; N g. To propose a calibration model for experimental data, as the one in Equation (3), is the consequence of the theoretical knowledge of the problem, but its validity has to be verified on the basis of a critical examination of the experimental results. The model, f(x), is a working hypothesis and must be modified if the experimental data are against it. However, the model is not directly observable, and therefore the decision on its validity depends on the success when distributing the experimental response in the two addends of Equation (1): random part and functional part. In other words, the hypotheses on the random error, which are those described in previous hypothesis (3), should also be made explicit and validated. When we say that a model is linear or nonlinear, we are referring to linearity or nonlinearity in the parameters. The value of the highest power of a predictor variable in the model is called the order of the model. For example, y ¼ 0 þ 1 x þ 2 x 2 þ "i
ð4Þ
is a second-order (in x) linear (in the ’s) regression model. The calibration model of Equation (4) is quadratic; it is not a linear calibration but a linear regression.
1.05.2.1
Estimation by Least Squares
The least squares (LS) method does not require any hypotheses under the distribution of the errors, ", in Equation (1). To obtain the estimates b0 and b1 of 0 and 1, respectively, the straight line that best describes the experimental data is determined. The argument is as follows: for each couple of values b0 and b1, a different line
130
Quality of Analytical Measurements: Univariate Regression
yˆ = b0 + b1x
eN
ei
e2 yˆ 1 e1
The linear calibration line fitted by least squares is the one that makes minimum the sum of squares of all these vertical distances
y1
x Figure 1 Illustration of least squares method to fit a calibration line.
is obtained, yˆ ¼ b0 þ b1 x , with the corresponding residuals ei ¼ yi – yˆi ; i ¼ 1; . . .; N . The LS estimators of 0 and 1 are the values that minimize the sum of squares (SSs) of residuals, Rðb0 ; b1 Þ ¼
N X
ei 2 ¼
i¼1
N X
ðyi – yˆi Þ2 ¼
i¼1
N X
ðyi – b0 – b1 xi Þ2
ð5Þ
i¼1
The line obtained when minimizing R as a function of b0 and b1 is the one that minimizes the sum of the distances ‘in vertical’ between the experimental data and the estimated ones with the fitted line (see Figure 1). It can be analytically proven that the function Rðb0 ; b1 Þ has a unique minimum (except for x1 ¼ x2 ¼ . . . ¼ xN , a clearly nonsense situation when calibrating) that can be explicitly determined. This solution provides the LS estimators as PN Þðxi – xÞ Covðx; y Þ i¼1 ðyi – y ¼ PN sx2 ð x Þ2 i¼1 i – x
b1 ¼
b0 ¼ y – b 1 x
PN
PN
ð6Þ ð7Þ
2 with y ¼ ¼ i¼1 yi =N the mean of all yi and x i¼1 xi =N the mean of all xi , and sx denotes the variance of the x’s. The procedure for obtaining the minimum shows that the LS line verifies N X
ei yˆi ¼ 0
ð8Þ
i¼1
and, if there is a 0 term in the model, as in our case, it is also true that N X
ðyi – yˆ i Þ ¼
i¼1
N X
ei ¼ 0
ð9Þ
i¼1
The LS method gives no estimator of 2 , but an estimator of 2 based on the LS estimators of 0 and 1 is the variance of the regression, defined as PN 2 syx ¼
– b0 – b1 xi Þ2 ¼ N –2
i¼1 ðyi
PN
PN 2 – yˆ i Þ2 e ¼ i¼1 i N –2 N –2
i¼1 ðyi
ð10Þ
Equations (6) and (7) imply that b0 and b1 are linear combinations of the random variables yi. Consequently, both estimates themselves are also random variables. As E ðyi Þ ¼ 0 þ 1 xi , from Equation (6), the following statement holds: E ðb1 Þ ¼
PN PN ÞE ðyi – yÞ Þð1 ðxi – xÞÞ i¼1 ðxi – x i¼1 ðxi – x ¼ ¼ 1 PN PN 2 Þ Þ2 i¼1 ðxi – x i¼1 ðxi – x
ð11Þ
Quality of Analytical Measurements: Univariate Regression
131
and, analogously from Equation (7), E ðb0 Þ ¼ Eð y Þ – 1 x ¼ 0
ð12Þ
Equations (11) and (12) mean that both estimates are unbiased estimates. The variance of the estimates is given by PN
Þ2 Varðyi Þ 1 2 i¼1 ðxi – x ¼ PN PN 2 2 2 ð x – x Þ i ð Þ x – x i¼1 i¼1 i ! PN 2 x 1 x2 2 þ PN Varðb0 Þ ¼ PN i¼1 i 2 2 ¼ 2 N N i¼1 ðxi – xÞ ð x – x Þ i¼1 i Varðb1 Þ ¼
x 2 2 ð x – x Þ i i¼1
Covðb0 ; b1 Þ ¼ – PN
ð13Þ
ð14Þ ð15Þ
Equations (13) and (14) show that the precision attainable in the estimates line P P of the coefficients of a calibration is the product of two factors; one of them depends on the magnitude, Ni¼1 ðxi Þ2 , and dispersion, Ni¼1 ðxi – xÞ2 , of the concentrations and the other one, 2, depends on the uncertainty of the observed response. The second factor is estimated with Equation (10) after making the experimental determinations of the calibration, but the other factor is previously known and implies the introduction of the idea of designing the calibration, analogous to any other experiment. This subject, together with other aspects to be taken into account as a consequence of the validation of the model, will be dealt with in Section 1.05.5. When analyzing Equation (15) one should remember that, in a calibration, the values xi cannot be negatives; thus, x > 0 and the covariance (correlation) between b0 and b1 is always negative. We consider the class of the unbiased estimators of 0 and 1 that are linear functions of the random variables yi. In this class, the LS estimators b0 and b1 of the linear model of Equation (3) with the three hypotheses detailed in Section 1.05.2 have smaller variances than any other estimators. This important theorem is often referred to as the Gauss–Markoff theorem and guarantees the coherence between the linear calibrations by LS and the accuracy (trueness and precision) demanded by the methods of chemical analysis. 1.05.2.2
Estimation when the Distribution of " Follows a N(0, 2)
The evaluation and comparative analysis of the estimated parameters in a calibration (b0, b1, and syx) is a problem of statistical inference because all of them are random variables that depend on the distribution of the error ". The only supposition on the distribution of the errors in the LS method is that they have common variance 2 independent of the concentration xi. However, it is necessary to dispose of additional information on their distribution to statistically infer anything about the parameters. When the distribution of the random error, " in Equation (1), is known, the estimation of 0, 1, and can be done by the maximum likelihood method, which, in essence, looks for the values of b0, b1, and s that maximize the probability of obtaining the experimental results that were observed. The method is based on the computation of the likelihood function L of the N random variables ðy1 ; y2 ; . . .; yN Þ, that is, the joint density, g, of these random variables considered as a function of the parameters 0, 1, and . Therefore, Lð0 ; 1 ; 2 Þ ¼ g ðy1 ; y2 ; . . .; yN ; 0 ; 1 ; 2 Þ. The maximum likelihood estimator is the value of 0, 1, and that maximizes L. If, further to the hypotheses on the errors established in hypothesis in point 3 in Section 1.05.2, the data are assumed to be normally distributed, the consequence is that for each x, the experimental response is a normal random variable N ð0 þ 1 x; 2 Þ. Furthermore, as Covðyi ; yj Þ ¼ 0; if i 6¼ j , the random vector ðy1 ; y2 ; . . .; yN Þt follows a multivariate normal distribution N ðh; 2 IÞ with vector of means h ¼ ð0 þ 1 x1 ; 0 þ 1 x2 ; . . .; 0 þ 1 xN Þt and the variance–covariance matrix, 2I, which is a diagonal matrix with all the terms equal to 2 . Figure 2 shows the meaning of the hypotheses assumed; for each concentration xi and due to the random character of ", any value of the signal can be obtained but those values close to 0þ 1xi are more probable, because this number is the expected mean value when using a probability density function of a normal distribution, independent from one concentration to another and with common variance 2 .
132
Quality of Analytical Measurements: Univariate Regression
y y = β0 + β1x
• •
•
•
•
•
•
•
XN
Xi
x
X1 Figure 2 Calibration model of Equation (3), assuming that the recorded signal comes from a normal distribution.
It can be proven that the likelihood function L is maximum when the sum of squares, R in Equation (5), is minimum. Consequently, the estimates b0 (Equation (7)) and b1 (Equation (6)) have the property of being maximum likelihood estimates. For 2 , the maximum likelihood estimate is PN s2 ¼
2 i¼1 ei
N
ð16Þ
which is biased, contrary to s2yx (Equation (10)). In any case, the relation ½N=ðN – 2Þs 2 ¼ syx2 holds. As a consequence of supposing normal distribution for the random errors, the following properties are obtained: 1. The estimates b0, b1, and syx2 (Equations (7), (6), and (10)) are of minimum variance in the class of all unbiased estimators, which is a class wider than the class of unbiased linear estimators. 2 =2 also does. 2. Ns 2 =2 follows a 2 distribution with N–2 degrees of freedom (d.f.), so ðN – 2Þsyx 3. b0 and b1 follow a bivariate normal distribution with covariance matrix whose elements are given by Equations (13)–(15). 2 . 4. b0 and b1 are independent of s 2 and, hence, of syx A last comment, if another distribution is supposed for the random term of the calibration line in Equation (3), the maximum likelihood estimate of 0, 1, and is obtained with other criteria. For example, instead of the normal distribution, let us suppose that the errors were independent and followed the Laplace distribution with null mean and variance 22, that is, its probability density function is f ð"i Þ ¼
1 – j"i j= e 2
ð – 1 "i 1Þ
ð17Þ
Then, application of the maximum likelihood principle for estimating 0 and 1 would involve the minimization of the sum of absolute errors N N X X j ei j ¼ jyi – yˆ i j i¼1
ð18Þ
i¼1
Of course, the LS solution is still available but it is not the maximum likelihood solution; thus, the statistical validity of the conclusions will be wrong. Consequently, the success in the selection of the distribution of the error is essential for a correct determination of the calibration line, including the criterion that should be followed to obtain the estimators. In the rest of the chapter, unless otherwise indicated, we will assume normal and independent residuals with zero mean and equal variance, that is, " is a NID(0,2) distribution.
Quality of Analytical Measurements: Univariate Regression
1.05.2.3
133
The Correlation between Concentration, x, and Signal, y
Usually, the correlation coefficient is used as an index of the linearity of a regression model. The validity of its application to a calibration is a subject of controversy because it seems to be valid only when it is applied with two random variables and the concentration of the standard samples is not random. In the following, this subject is analyzed. We consider two random variables, v and w, which follow some continuous joint bivariate probability distribution. The correlation coefficient between v and w is defined by Covðv; wÞ vw ¼ pffiffiffiffiffiffiffiffiffiffiffiffiffiffipffiffiffiffiffiffiffiffiffiffiffiffiffiffi VarðvÞ VarðwÞ
ð19Þ
For example, from Equations (13)–(15), the correlation coefficient between the two random variables b0 and b1 is sffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi N b0 b1 ¼ – x PN 2 i¼1 xi
ð20Þ
which depends only on the values of the concentration of the calibration samples and which is completely independent of the recorded signals, yi. In general, – 1 vw 1. The quantity vw is a measure of the linear association between the random variables v and w. If vw ¼ 1, v and w are perfectly positively correlated and their values all lie on a straight line with a positive slope in the (v, w) plane. If vw ¼ 0, the variables are said to be uncorrelated, that is, linearly unassociated with each other; note that this does not mean that v and w are statistically independent. If vw ¼ – 1, v and w are perfectly negatively correlated and their values again lie on a straight line, but this time with a negative slope. If a sample of size Nfðvi ; wi Þ; i ¼ 1; :::; N g is available from the joint distribution, the quantity PN Þðwi – w Þ i¼1 ðvi – v ffiqffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi rvw ¼ qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi PN PN 2 Þ Þ2 i¼1 ðvi – v i¼1 ðwi – w
ð21Þ
is called the sample correlation coefficient between v and w. It is an estimate of vw and provides an empirical measure of the linear association between v and w. Like vw , – 1 rvw 1. When ðvi ; wi Þ; i ¼ 1; :::; N , are all nonrandom values, that is, they are not sample values from some distribution, rvw can still be used as a measure of linear association, because the values ðvi ; wi Þ; i ¼ 1; :::; N , can be considered as a finite discrete bivariate distribution with uniform marginals. In this case, rvw is a population rather than a sample estimation, that is, vw ¼ rvw . In our case, xi, i ¼ 1, . . ., N, are nonrandom values that represent the values of a finite discrete distribution for concentrations and the corresponding recorded signals yi, i ¼ 1, . . ., N, are observed random variables. As a consequence, it is possible to define xy by Equation (19). Equation (21), with x and y replacing v and w, can still be used to estimate yx by ryx. There is another permanent discussion about the usefulness or uselessness of the correlation coefficient to evaluate the linearity of a calibration.4–8 The paper by Asuero et al.9 is an updated review on the correlation coefficient and its use either as a way to infer correlation between variables or to test linearity with chemical applications. Here, we will only show that between the coefficient of the regression (the slope of the calibration, b1) and the correlation coefficient ryx, the following relation holds: qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi ffi P pffiffiffiffiffiffiffiffiffiffiffiffiffi ðyi – yÞ2 =ðN – 1Þ Varðy Þ p ffiffiffiffiffiffiffiffiffiffiffiffiffiffi ryx b1 ¼ qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi ¼ r yx P Varðx Þ ðxi – xÞ2 =ðN – 1Þ
ð22Þ
Thus, b1 and ryx are closely related, although they provide different information about the calibration line. The unit-free and scale-free correlation coefficient ryx measures the degree of linear association between
134
Quality of Analytical Measurements: Univariate Regression
concentration and signal, whereas b1 measures the size of the change in signal when a unit change is made in concentration. Scale changes in the data will affect b1 but not ryx. It is easy to see that ryx ¼ ryyˆ
ð23Þ
That is, the correlation between the recorded signals yi and the estimated ones yˆi is the same (in absolute value) as the correlation between the concentrations xi and the signals yi.
1.05.3 Statistical Validation of a Calibration Model 1.05.3.1
Validity of the Functional Model
1.05.3.1.1
F-test for significance of regression Obviously, the dispersion of the recorded signal has two sources: the one due to the regression and the residual one. If the dispersion attributable to the regression is higher, in statistical terms, than the residual one, the conclusion is that the model is significant, that is to say, it explains the recorded signals. The variation in the y’s about their mean can be written as N N N N X X X X ðyi – yÞ2 ¼ ððyi – yˆ i Þ þ ðyˆ i – yÞÞ2 ¼ ðyi – yˆ i Þ2 þ ðyˆ i – yÞ2 i¼1
i¼1
i¼1
ð24Þ
i¼1
because the crossproduct term is null. Therefore, the total variation of the y’s, SST, is decomposed into the P variation that can be ascribed to the regression line, SSR, and the residual variation, SSE ¼ Ni¼1 ðyi – yˆi Þ2 , due to the fact that the experimental observations do not all lie on the regression line. In a compact notation, Equation (24) is SST ¼ SS E þ SSR
ð25Þ
If SSR SSE (equivalently, if R ¼ SSR =SST is near one), it is clear that the regression explains the recorded signal as a function of the concentration. The degrees of freedom (d.f.) associated with any sum of squares are the number of independent pieces of information that we need for its calculation. To compute SST, there are only N1 independent data because it is necessary to calculate y. The SSE has N2 d.f. because we need to previously compute b0 and b1. By subtraction, SSR has 1 d.f. For each term in Equation (25), the mean of squares (MSs) is defined as the quotient between the sum of squares and its d.f., MS ¼ SS=d:f : The result of these computations is presented as an analysis of variance (ANOVA) table like Table 1. MSR and s2yx ¼ MSE, which are shown in Table 1, are random variables because they are obtained from the variables yi. It can be shown that their mean values are as follows: 2
E ðMSR Þ ¼ 2 þ 12
N X ðxi – xÞ2
ð26Þ
i¼1
2 ¼ 2 E ðMSE Þ ¼ E syx
ð27Þ
In addition, if "i are NID(0,2) and 1 ¼ 0, the random variable Fcalc ¼
MSR MSE
ð28Þ
Table 1 Analysis of variance for significance of a linear regression Source of variation
Sum of squares
Degrees of freedom
Mean of squares
Fcalc
Regression Error or residual Total
SSR SSE SST
1 N2 N1
MSR MSE
MSR/MSE
Quality of Analytical Measurements: Univariate Regression
135
follows an F-distribution with 1 and N2 degree(s) of freedom in the numerator and denominator, respectively. If there is a linear relation between signal and concentration, the regression coefficient 1 must be different from zero. The appropriate hypotheses are H0 :1 ¼ 0
ð29Þ
Ha : 1 6¼ 0
and the null hypothesis would be rejected at significance level if Fcalc ¼ MSR =MSE exceeds the critical value at level , F,1,N2. In other words, the linear calibration is statistically significant at level . Instead of the critical value, it is usual to compute the probability pr{F1,N2 > Fcalc} called p-value. In this case, the null hypothesis is rejected (the model is significant) at level if the p-value is smaller than . In a calibration experiment, the null hypothesis of the test for significance of the regression is almost always rejected; hence the model is significant because the calibration is, by definition, based on the fact that the response of the instrument changes with the concentration of the standard solutions. Two additional relationships should be noted. In the explanation of Equation (25), we have defined R2 ¼ SSR =SST as an indicator of the variability of the signal that is explained by the variation of the concentration. R2 is called the determination coefficient and verifies for a fitted straight line that ryx ¼ ðsign of b1 ÞR
ð30Þ
where R is the positive square root of R2. Then, from Equation (23) we also have that, ryyˆ ¼ R
ð31Þ
1.05.3.1.2
Lack of fit test More interesting in practice is the lack of fit (LOF) test that allows us to decide if the supposed linear model in Equation (3) is compatible with the experimental data. The only information available on the discrepancy between the model and the experimental results is provided by the residuals of the fitting yi – yˆ i , which can be decomposed as yi – yˆ i ¼ ðyi – yˆ i Þ – ðE ðyi Þ – E ðyˆ i ÞÞ þ ðE ðyi Þ – E ðyˆ i ÞÞ ¼ ððyi – yˆ i Þ – E ðyi – yˆ i ÞÞ þ ðE ðyi Þ – E ðyˆ i ÞÞ ¼ qi þ Bi
ð32Þ
In Equation (32), Bi is the bias in the estimation when the concentration is xi. If the model is correct, then the bias is zero. Otherwise, Bi 6¼ 0 and its value depends on xi and on the postulated PN 2 model. The2 other term, the random variables qi, i ¼ 1, . . ., N, have null mean, are correlated, and E i¼1 qi ¼ ðN – 2Þ . Consequently, from Equation (32), it is deduced that MSE defined as MSE ¼
SSE ¼ N –2
PN
– yˆ i Þ2 N –2
i¼1 ðyi
ð33Þ
P is a random variable with mean 2 if the model is correct and 2 þ Ni¼1 Bi2 =ðN – 2Þ if the model is biased. Note that 2 ¼ MSE is an overestimation of 2 ¼ Varð"i Þ ¼ Varðyi Þ if the assumed model for the calibration is not correct, syx because it includes the bias of the model. The question is to have another independent estimate of 2 , for example, by previous knowledge of the system of measurement, and comparing it with MSE. If the latter is significantly greater, we say that there is LOF and we would reconsider the model for the calibration line. If no prior estimate of 2 is available, a usual procedure in calibration is to repeat measurements of the standard samples in such a way that we can use these replicates to obtain an estimate of 2 . It is said that this estimation represents the pure error because if xi is the same for several observations, then the variability among the observed responses is due to random fluctuations of measurements alone. It is important to understand that the replicated runs must be genuine replicates and not just repetitions of the same instrumental reading over the same analytical sample. The number and distribution of replicates in the concentration range is an important practical issue because it influences the variance of the estimates made from the calibration line and will be studied in Section 1.05.5.
136
Quality of Analytical Measurements: Univariate Regression
Pk Suppose that in each calibration standard xi (i ¼ 1,P . . . ,k) mi signals
2 yij are recorded so that i¼1 mi ¼ N . The mi internal variability of these observations is given by j ¼1 yij – yi with mi1 d.f. Therefore, pooling over the k levels xi, we have the so-called sum of squares of pure error: SSPE ¼
mi k X X
2 yij – yi
ð34Þ
i¼1 j ¼1
P with ki¼1 mi – 1 ¼ N – k d.f. Dividing SSPE by its d.f. we have MSPE that verifies E(MSPE) ¼ 2. On the other side, we have SSE ¼
N X
mi k X
2 X
2 yj – yˆ j ¼ yij – yˆ i i¼1 j ¼1
j ¼1
mi k X k X
2 X ¼ yij – yi þ mi ðyi – yˆ i Þ2 ¼ SSPE þSSLOF i¼1 j ¼1
ð35Þ
i¼1
In Equation (35), we can see that the sum of squares for LOF, SSLOF, is a weighted sum of squared deviations between the mean response yi at each xi level and the corresponding fitted value yˆi . Table 2 shows the decomposition of the sum of squares of Equation (35) with the corresponding d.f. If Fcalc ¼ MSLOF =MSPE is greater than F,k-2,Nk, the null hypothesis is rejected at level , that is, we conclude that the model proposed for the calibration line has LOF (is biased). Alternatively, if Fcalc does not exceed F,k-2,Nk, then there is no experimental evidence for LOF, and MSPE and MSLOF are often combined to estimate 2.
1.05.3.2
Verification of the Hypotheses on the Residuals
The residuals are the differences ei ¼ yi – yˆi ; i ¼ 1; . . .; N , where yi are the experimental values and yˆi the corresponding fitted values. The random variables "i are not observable. The intuitive idea is that, if the calibration model is correct, the residuals will show tendencies that make us think that "i are effectively NID(0,2). The difficulty arises when expressing ei as a function of "i :10 ei ¼ ð1 – hii Þ"i –
XN
h " j ¼1; j 6¼i ij j
ð36Þ
with
1 ðxi – xÞ xj – x hij ¼ þ PN N Þ2 i¼1 ðxi – x
ð37Þ
and the NID assumption regarding "i implies that the ei must be normally distributed with Varðei Þ ¼ ð1 – hii Þ2
ð38Þ
Thus, unless the concentration values are only two, the variance of the residuals Varðei Þ will be changing and will differ from 2 . To avoid this effect, which can be important if the calibration range is large, it is better to use the studentized residuals defined as Table 2 Analysis of variance for significance and lack of fit of a linear regression Source of variation
Sum of squares
Degrees of freedom
Mean of squares
Fcalc
Regression Error or residual Lack of fit Pure error Total
SSR SSE SSLOF SSPE SST
1 N2 k2 Nk N1
MSR MSE MSLOF MSPE
MSR/MSE MSLOF/MSPE
Quality of Analytical Measurements: Univariate Regression
ei 9 ¼
ei pffiffiffiffiffiffiffiffiffiffiffi syx 1 – hii
137
ð39Þ
As the studentized residuals come from a normal distribution with variance one, they have to vary between 3 and 3. 1.05.3.2.1
Graphical methods Usual graphics are the plots of residuals (or studentized residuals) versus the following: (1) the order of obtaining the results (possible bias due to the sequence of the experiments); (2) the responses yˆi computed with the model; and (3) the concentrations. In these plots, typical patterns like those of Figure 3 can be observed, which not only alert on the possible failure of the hypotheses but also aid in deciding on the actions to be taken to restore them. Figure 3(a) shows a suitable distribution of the residuals, whereas the rest display certain anomalies. For example, a tendency like the one observed in Figure 3(b), the variance of the residuals increases when increasing the variable represented in the abscissa, indicates a lack of homoscedasticity. Nevertheless, the tendency of Figure 3(c) shows a bias attributable to an inadequate model; a model of order 2 should be chosen as the calibration model. Figure 3(d) shows both anomalies, a failure of equality of variances and also a bias of the model. To evaluate normality, it is useful to draw the graph of normality of studentized residuals, which consists of plotting the cumulative relative frequency of each residual according to a normal distribution versus the residual itself. If the distribution is normal, the points have to appear aligned; additional details can be seen in Draper and Smith.11 When, instead of using the studentized residuals, their absolute values are used, the resulting graph is called a half-normal plot. The graphical possibilities of computers facilitate the drawing of these plots because the vertical scaling should be determined from the inverse function of a normal distribution. When " is not normally distributed, an additional problem can occur. Each hii is different for each estimated residual and, when it is large related to the residual itself, the corresponding ei will be approximately normally distributed because of the central limit theorem. When hii is small, the distribution of ei would be similar to that of "i, although closer to a normal distribution. Thus, we have the disturbing result that the residuals can have a distribution that is closer to normal than the distribution of the errors (even if they are not normally distributed). This property is known as supernormality by many authors. To rectify this problem, the use of (a) 2
(b) 2
1
1
0
0
–1
–1
–2
–2
(c) 2
(d) 2 1
1 0 0 –1 –1
–2
Figure 3 Schematic view of some possible tendencies on the residuals: (a) no tendency; (b) lack of homoscedasticity; (c) wrong model; and (d) both heteroscedasticity and wrong selection of model.
138
Quality of Analytical Measurements: Univariate Regression
a ‘simulation envelope’ is proposed for a half-normal plot together with the use of the ‘externally studentized pffiffiffiffiffiffiffiffiffiffiffi residuals’ defined as ei ¼ ei =syxðiÞ 1 – hii , where syxðiÞ is the residual standard deviation when in the regression model the pair (xi, yi) is not considered (is suppressed). The envelope boundaries are made up by the estimators of the 5th and 95th percentiles of the distribution for each ei obtained by bootstrap. These simulated residuals are normally distributed and the boundaries constructed essentially provide a limit on how non-normal the residuals can appear when the errors have normal distribution. This technique has been deeply studied and there are variations. A review of the subject with examples of application can be seen in Ryan.12 1.05.3.2.2
Inferential methods
Serial correlation The Durbin–Watson test is usually used to verify if each residual is independent of the previous one and therefore if some relation between the results exists related to the concentration of the standards or the sequence in which they were measured. This is frequent in the laboratory because of indirect causes, for example, degradation of reagents and instrumental drift. From the Durbin–Watson test, some procedures have been developed to detect the lack of linearity in real calibration experiments. A comparative analysis, including a recent proposal, is made in the paper by Bru¨ggemann et al.13
1.05.3.2.2(i)
The usual tests are those of Kolmogoroff and 2. The Kolmogoroff test detects modifications in the form of the empirical histogram made with n data with respect to the theoretical one, in our case with respect to that of a normal distribution. The 2 test, similar to the Kolmogoroff test, detects whether the histogram of some experimental data adheres suitably to the one of a theoretical distribution (in our case a normal one). Unlike the previous one, the data are grouped into k intervals so that it demands greater sample sizes. 1.05.3.2.2(ii)
Normality
Homoscedasticity (equal variances) To be able to decide whether the variance of the responses stays or is nonequal for the different values of the independent variable, it is necessary to have several measured responses yi for the same value of the predicting variable xi, at least in two levels of concentration. The Bartlett test and the Cochran test are both adequate and, to a great extent, complementary. The Bartlett test detects the existence of groups of values of the independent variable with variances in the response similar to each other but different from one group to another. It is very sensitive to the failure in the hypothesis of normality. The test of Cochran detects the existence of one discrepant variance.
1.05.3.2.2(iii)
Example 1 To determine cadmium by anodic stripping voltammetry, the intensity at 630 mV has been recorded for five different samples. The results obtained for five replicates of each sample are shown in Table 3. The calibration line fitted by LS is y ¼ – 0:497 þ 0:265x
with the standard deviation of slope and the independent term equal to 0.004 and 0.186 nA, respectively. The residual standard deviation is syx ¼ 0.308, the correlation coefficient is 0.997, and the coefficient of variation is R2 ¼ 0.994. Table 3 Data for the calibration of cadmium Concentration (nmol l–1)
Current (nA)
20.18 30.27 40.36 50.24 60.08
5.00 7.92 9.89 12.49 15.27
4.85 7.05 9.77 12.67 15.49
5.13 7.48 10.00 12.62 15.90
5.02 7.81 10.72 12.83 15.26
4.50 7.49 10.34 12.51 15.98
Mean
Variance
4.90 7.55 10.14 12.62 15.58
0.060 0.116 0.149 0.019 0.117
Quality of Analytical Measurements: Univariate Regression
139
Table 4 ANOVA for the determination of cadmium including the lack of fit test Source of variation
Sum of squares
Degrees of freedom
Mean square
Fcalc
p-value
Regression Error or residual Lack of fit Pure error Total
349.30 2.18 0.34 1.84 351.481
1 23 3 20 24
349.30 0.09 0.11 0.09
3686.37
<104
1.22
0.33
The ANOVA table, as defined in Table 2, is presented in Table 4. The p-value for the test on significance of the regression, Equation (29), is less than 104; thus, the linear calibration model is significant at a significance level of 0.05. Furthermore, Table 4 shows that there is no significant LOF; the p-value of the corresponding test is 0.33 and hence is greater than the usual significance level of 0.05. Therefore, there is no experimental evidence to reject the null hypothesis ‘the model is biased’. We can consider that the residuals follow a normal distribution (again at the significance level 0.05) according to the tests of 2, Kolmogoroff, Shapiro–Wilks, skewness, and kurtosis with p-values of 0.89, 0.85, 0.96, 0.57, and 0.28, respectively. Autocorrelation at the significance level of 0.05 does not exist either, because the p-value of the Durbin– Watson test is 0.07.
Externally studentized residuals
Regarding the equality of variances of the residuals by concentration levels, the Cochran and Bartlett tests have p-values of 0.79 and 0.42, respectively. Hence, none of the five variances is significantly greater or smaller than the others, nor there are groups of them that are significantly different. Figure 4 shows the externally studentized residuals of the fitted calibration line. No anomalous tendency is observed. The normality graph of the same externally studentized residuals is shown in Figure 5 (the straight line is just a guide), which shows that the residuals are reasonably aligned corresponding to a normal distribution. 2.9 1.9 0.9 –0.1 –1.1 –2.1 20
30
40 50 Cd (nmol l –1)
60
70
Percentage
Figure 4 Graph of the externally studentized residuals vs. the true concentration of cadmium.
99.9 99 95 80 50 20 5 1 0.1 –1.7
–0.7 0.3 1.3 Externally studentized residuals
Figure 5 Normal probability plot of the externally studentized residuals.
2.3
140
Quality of Analytical Measurements: Univariate Regression
1.05.4 Confidence Intervals and Hypothesis Testing Once the calibration model yi ¼ 0 þ 1 xi þ "i ; i ¼ 1; . . .; N , with NID(0,2) random errors "i is validated, it is useful to know how precise the estimated b0 and b1 are, as well as any other parameters obtained from them, as important as the concentration of a problem sample.
1.05.4.1
Confidence Intervals for the Intercept, the Slope, and the Residual Variance
Section 1.05.2.2 already explains that the estimates b0 and b1 follow a bivariate normal distribution with the 2 is an unbiased estimate of 2 independent of b0 variance–covariance matrix defined in Equations (13)–(15). syx and b1. In these conditions, the confidence intervals at significance level for the slope and the intercept are qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi
dðb1 Þ ¼ syx b1 – t=2;N – 2 sb1 ; b1 þ t=2;N – 2 sb1 with sb1 ¼ Var
b0 – t=2;N – 2 sb0 ; b0 þ t=2;N – 2 sb0 with sb0 ¼
sffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi 1 PN ð Þ2 i¼1 xi – x
sffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi 2 dðb0 Þ ¼ syx 1 þ P x Var N N Þ2 i¼1 ðxi – x
ð40Þ
ð41Þ
Moreover, we can test the null hypothesis i ¼ i0 , where i is equal to 0 or 1 and i0 is a nonrandom value previously fixed. For example, for the intercept when each measurement was blank-corrected, 0 ¼ 0. The hypothesis test for each one is H0 : i ¼ i0
ð42Þ
Ha : i 6¼ i0
The statistic for this hypothesis test is tcalc ¼
bi – i0 sbi
ð43Þ
which follows a Student’s t-distribution with N2 d.f. if H0 is true. Then the null hypothesis is rejected at significance level if jtcalc j > t=2;N – 2 . Example 2 With the calibration data in Example 1, we want to decide at significance level ¼ 0.05 whether the independent term is equal to zero. Thus, the hypothesis test is now H0 : 0 ¼ 0 versus Ha : 0 6¼ 0 and the statistic, Equation (43), is tcalc ¼ ð – 0:497 – 0Þ=0:186 ¼ – 2:672. As jtcalc j > t0:025;23 ¼ 2:069, the null hypothesis is rejected and we conclude that the independent term is significantly non-null. Two tests have been shown for testing H0 :1 ¼ 0 versus Ha :1 6¼ 0, an F-test (Equations (28) and (29)) and the t-test (Equations (42) and (43)). The two tests are equivalent because a random variable that follows an F-distribution with one and d.f. is equal to the square of a t-distributed variable with d.f., and it is simple to 2 in Equation (43). prove that the statistic Fcalc in Equation (28) is equal to tcalc 2 The confidence interval, at significance level , for the residual variance syx : 2 2 syx ðN – 2Þ syx ð N – 2Þ ; 2=2;N – 2 21 – =2;N – 2
! ð44Þ
2 2 The null hypothesis that the residual variance is equal to a known value H0 :syx ¼ 20 against Ha :syx > 20 is rejected at level if
2calc ¼
2 syx ð N – 2Þ
20
is greater than the critical value of a 2 distribution with N2 d.f.
ð45Þ
Quality of Analytical Measurements: Univariate Regression
141
In Draper and Smith11 and Asuero et al.,9 the procedure to compute a confidence interval for the correlation coefficient ryx can be found. Also, to test the null hypothesis H0 : ryx ¼ 0, where 0 is a specified value (e.g., zero), versus any of the alternative hypotheses Ha : ryx 6¼ 0, or ryx > 0, or ryx < 0. The relation between the correlation coefficient and the slope of Equation (22) makes the following relation holds: R2 ¼
ðb1 =sb1 Þ2 N – 2 þ ðb1 =sb1 Þ2
ð46Þ
Therefore, the significance of the slope of a calibration line, the significance of the regression (ANOVA), and the correlation and determination coefficients are related to each other in such a way that any of them can be obtained from the rest.
As a measure of the degree of linearity of a calibration, the use of the index ILC ¼ 1 – sb1 =b1 has been proposed.14,15 It is a hyperbolic transformation of the statistic of the t-test (for the significance of the slope) and the F-test (for the significance of regression) instead of evaluating the corresponding p-value. In fact, Equation (46) states that ILC is a transformation of the correlation coefficient that takes into account the size of the calibration set, N. Table 5 shows the values of the statistic b1 =sb1 between 1 and 100 together with the corresponding ILC, R2, ryx, and p-values supposing that the calibration size is N ¼ 10 or N ¼ 20. For example, stating that a calibration is good when ILC ¼ 0.95 is exactly the same as stating that the calibration has ryx ¼ 0.99015 (N ¼ 10) or ryx ¼ 0.97823 (N ¼ 20), which is equivalent, in terms of p-values, to 6.0 1010 and 2.7 1015, respectively. It is evident that to fix a threshold value for any of them is equivalent to fixing it in any other; the reader can verify that the practical meaning of all of them is the same.
1.05.4.2
Joint Confidence Region for 0 and 1
Equations (15) and (20) show that the estimates of b0 and b1 are correlated. Therefore, the joint confidence region at level 100(1–)% is not the Cartesian product of the corresponding individual confidence intervals, but is defined as the set of pairs (0, 1) that verify the following inequality: N ð0 – b0 Þ2 þ2
PN
i¼1 xi ð0
– b0 Þð1 – b1 Þ þ 2 2syx
PN
2 i¼1 xi ð1
– b1 Þ2
F;2;N – 2
ð47Þ
This region is useful to establish the accuracy of a method of analysis in a range of concentrations and to compare different analytical methods and/or calibrations. After the calibration model is obtained, it is used to estimate the concentrations xˆi corresponding to the calibration standards xi. Independent of the form of calibration (univariate, linear or nonlinear, multivariate, multiway, etc.), if the method is accurate, then the regression of values xˆi versus xi has to be a straight line with slope 1 and independent term zero. The regression 2 xˆ ¼ b 0 þ b 1 x and its corresponding syx are enough to calculate, Equation (47), the confidence region at significance level of ðb0 ; b1 Þ. If point (0, 1) belongs to that region, it can be concluded that the method is accurate. Table 5 Some values of x¼ b1 =sb1 and the corresponding index of linearity, ILC ¼ 1 – ð1=x Þ , coefficient of determination, R2, coefficient of correlation, ryx, and p-level for the t-distribution with the corresponding degrees of freedom (d.f.) R2
ryx
p-value
x
1(1/x)
N ¼ 10
N ¼ 20
N ¼ 10
N ¼ 20
t (8 d.f.)
t (18 d.f.)
1 5 10 20 50 100
0 0.8 0.9 0.95 0.98 0.99
0.11111 0.75758 0.92593 0.98039 0.99681 0.99920
0.05263 0.58140 0.84746 0.95694 0.99285 0.99820
0.33333 0.87039 0.96225 0.99015 0.99840 0.99960
0.22942 0.76249 0.92057 0.97823 0.99642 0.99910
0.08522 0.00013 3.9 107 6.0 1010 6.0 1014 5.5 1017
0.08231 0.00002 8.0 1010 2.7 1015 <1017 <1017
142
Quality of Analytical Measurements: Univariate Regression
The joint representation of these regions for several calibration methods fitted with the same analytical signals allows a comparison between them; the smaller the surface of the confidence region, the more precise the calibration method. Example 3 In the simultaneous determination16 of steroid hormones estrone (E1) and 17--ethinylestradiol (EE2) by gas chromatography with mass spectrometry detection (GC/MS), three calibration methods have been used with the same data that correspond to the signals recorded on three nonconsecutive days: 1. A three-way Parallel Factor Analysis (PARAFAC)-based calibration. 2. A univariate linear regression with standardized sum of six monitorized ions. 3. A univariate linear regression with standardized peak base area. For the EE2, Figure 6 shows the three joint confidence regions for the slope and the intercept of the three regression models xˆi versus xi . One can see that the three calibrates are unbiased and that the PARAFAC-based calibration model is the most precise. 1.05.4.3
Comparison of Two Regression Lines
Suppose that we have two calibration sets of size N1 and N2, with independent residuals and equal variances, 21 ¼ 22 . The equality of the estimated slopes of the two calibration lines, b11 and b12, is tested by the hypothesis H0 : 11 ¼ 12 versus Ha : 11 6¼ 12 The statistic for this hypothesis test is b11 – b12 tcalc ¼ sffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi 1 ðx – x1 Þ2 i¼1 i1
sp2 PN1
ð48Þ
þ PN2
1 ðx – x2 Þ2 i¼1 i2
where sp2 ¼
2 2 syx1 ðN1 – 2Þ þ syx2 ð N 2 – 2Þ
ð49Þ
N1 þ N2 – 4
is the pooled estimated common variance.
(c) (a)
Slope
(b)
–6
–4
–2
0 Intercept
2
4
6
Figure 6 Joint confidence region at 95% for the slope and intercept of the regressions ‘estimated concentration vs. true concentration’ for 3,17-di-TMS-EE2 with several calibration models: (a) PARAFAC; (b) AREAS; and (c) univariate.
Quality of Analytical Measurements: Univariate Regression
143
The null hypothesis is rejected at significance level if jtcalc j > t=2;N1 þN2 – 4 . For the case of significantly unequal residual variances at level , that is, if 2 syx1 2 syx2
> F;N1 – 2;N2 – 2
ð50Þ
the comparison is carried out by the statistic b11 – b12 ffi tcalc ¼ rffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi 2 2 syx1 syx2 PN1 P þ N2 2 2 i¼1
ðxi1 – x1 Þ
i¼1
ð51Þ
ðxi2 – x2 Þ
The distribution of tcalc in Equation (51) can be approximated by a t-distribution with d.f., where is the nearest integer to 1 c 2 =ðN1 – 2Þ þ ð1 – c Þ2 =ðN2 – 2Þ
ð52Þ
with c¼
PN1 ðxi1 – x1 Þ2 i¼1 ; P PN2 N1 2 2 2 = 2 = syx1 ð x – x Þ ð x – x Þ þ s i1 1 i2 2 yx2 i¼1 i¼1 2 syx1 =
if N1 N2
This value always lies between N12 and N1 þ N2 – 4. The adequate formulas for testing the equality of more than two regression lines can be found in Sachs.17 Another approximation to the same problem consists of considering a multivariate regression involving a dummy variable,11 z, which is zero for the values (xi ,yi) of the first calibration set C1 and one for the data of the second calibration set C2. The joint model is then y ¼ 0 þ 1 x þ 0 z þ 1 zx þ "
ð53Þ
The separate functional models for C1 and C2 lines are given by setting z ¼ 0 and z ¼ 1, respectively C1 calibration model: 0 þ 1 x
ð54Þ
C2 calibration model: ð0 þ 0 Þ þ ð1 þ 1 Þx ¼ 0 þ 1 x
ð55Þ
1. To test whether the two calibration lines are parallel (they have the same slope), we would fit model in Equation (53) and then test H0 : 1 ¼ 0 versus Ha : 1 6¼ 0. 2. To test whether the intercept is equal for both calibration lines, the null hypothesis used would be H0 : 0 ¼ 0 versus Ha : 0 6¼ 0. 3. To decide whether the two lines are equal, the null hypothesis is H0 : 0 ¼ 1 ¼ 0. In cases 1 and 2, the test is a t-test similar to those of Equations (42) and (43), that is, tcalc ¼ i =si but now si is the square root of the appropriate diagonal term of the (XtX)1s2 matrix, where s2 is the residual variance of the fitting of the model in Equation (53) and 0
1
B B 1 B B B B B B 1 X¼B B1 B B B. B .. B B @1
x1
0
x2
0
0
1
C 0 C C C C C C xN1 0 0 C C xN1 þ1 1 xN1 þ1 C C C C .. .. C . . C C xN1 þN2 1 xN1 þN2 A
ð56Þ
144
Quality of Analytical Measurements: Univariate Regression
The jtcalc j is compared with the percentage point t=2;N1 þN2 – 4 for a two-sided test on the null hypothesis H0 : i ¼ 0 versus Ha : i 6¼ 0. The joint test of case 3 is similar to that in Equation (47). The details of this test can be consulted in Chapter 3.02 and also in Chapter 14 of Draper and Smith.11 In the model of Equation (53), it is assumed that the variance of the residuals is equal in all experimental data, and this should be carefully verified before applying the hypothesis test to decide about the calibrations C1 and C2. In chemical analysis, the decision on the equality of two calibration lines is important to decide if there is a matrix effect and, if so, what type. If this is the case, calibration C1 is made with samples spiked on a matrix (e.g., sulfonamide in pig kidney) and calibration C2 would be formed by samples of similar concentrations in the dissolvent. If there is no matrix effect, the null hypothesis of case 3 will not be rejected. It is also possible that the effect is a constant or proportional bias, in these cases the null hypothesis of case 2 or the null hypothesis of case 1 will be rejected respectively. When 0 and 1 in Equation (55) are significantly non-null, then both biases exist. Another important application is the verification that the calibration stays day to day; in this case, C1 and C2 would be the calibration made on the first and second day, respectively. 1.05.4.4
Confidence Interval for the Prediction
For a concentration x0 where no experiments were carried out, the response is estimated by yˆ0 ¼ b 0 þ b 1 x 0
ð57Þ
However, as b0 and b1 are random variables that follow a bivariate normal distribution, yˆ0 is also a random variable that follows a normal distribution whose mean and variance as computed with Equations (11)–(15) are E ðyˆ 0 Þ ¼ 0 þ 1 x0
ð58Þ !
Varðyˆ 0 Þ ¼
1 ðx0 – xÞ2 2 þ PN 2 N ð x – x Þ i i¼1
ð59Þ
Varðyˆ 0 Þ has a minimum when x0 ¼ x and increases as x0 separates from x in either direction. In other words, the most precise prediction is expected to be in the mean of the calibration range and precision is lost as we separate from x. The confidence interval at level (1)100% for the true mean value of the response for a given x0 is then computed as
yˆ 0 – t =2;N – 2 s y0 ; yˆ0 þ t =2;N – 2 s y0 sffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi 1 ðx0 – xÞ2 d sy0 ¼ Varðyˆ 0 Þ ¼ syx þ PN N Þ2 i¼1 ðxi – x
ð60Þ
If we join up all the lower endpoints and all the upper endpoints of intervals defined in Equation (60) as x0 changes, we would obtain the two dotted hyperbolas as shown in Figure 7 for the data in Example 1, Table 3. The individual values of the random variable yˆ0 are distributed around its mean with variance 2 independent of Varðyˆ 0 Þ. Therefore, the variance of the prediction of an individual observation would be
2
1 ðx0 – xÞ2 1þ þ PN N Þ2 i¼1 ðxi – x
! ð61Þ
2 and the corresponding estimated value is obtained by substituting 2 by syx in Equation (61). The confidence interval for a new observation is then
sffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi 1 ðx0 – xÞ2 yˆ0 t =2;N –2 s yx 1 þ þ PN N Þ2 i¼1 ðxi – x
ð62Þ
Quality of Analytical Measurements: Univariate Regression
145
16
Current (nA)
14 12 10 8 6 4 20
30
40 50 Cadmium (nmol l –1)
60
70
Figure 7 Confidence intervals at 95% for the calibration data of Example 1, Table 3. Dotted hyperbolas are for the true mean. Continuous hyperbolas are for a new prediction with q ¼ 1 in Equation (63). The squares are the experimental points.
A confidence interval for the average of q new observations is obtained similarly as follows: sffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi 1 1 ðx0 – xÞ2 yˆ0 t =2;N – 2 s yx þ þ PN q N Þ2 i¼1 ðxi – x
ð63Þ
Again, by joining the corresponding endpoints of the confidence intervals for one new observation, we obtain two hyperbolas (those drawn with a continuous line in Figure 7). The rest of the hyperbolas that would be obtained for q > 1 would be located between the two ‘limiting’ hyperbolas (the dotted and the continuous lines in Figure 7). 1.05.4.5 Sample
Confidence Interval for the Determination of the Concentration of a Problem
The most important application of a calibration line is the determination of the concentration of an analyte in a problem sample. We have fitted the calibration line (y ¼ b0 þ b1 x with residual standard deviation syx) and now for a specified value of y, say y0, we wish to obtain an estimated value xˆ0, the value of sample concentration, as well as some sort of confidence interval for it. Initially, we suppose that y0 is the true mean value of the distribution of the signal for a sample of unknown concentration, then the solution will be adapted to the case of having one or several determinations on the same sample whose concentration we want to obtain. By using the calibration line, the estimated concentration is xˆ0 ¼
y0 – b0 b1
ð64Þ
A confidence interval for xˆ0 at level (1)100% will be made up by the values of the concentration x such that the confidence interval at level (1)100% for the corresponding value y ¼ b0 þ b1 x contains y0 . According to Equation (60), this is equivalent to considering all the x values that verify sffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi 1 ðx – xÞ2 jy0 – ðb0 þ b1 x Þj t=2;N – 2 syx þ PN N Þ2 i¼1 ðxi – x
ð65Þ
Besides being consequence of the distribution of the response, this interval is also a likelihood ratio confidence region as shown in Brown18 and also corresponds to the inversion of a similar test of the hypothesis x ¼ x0 (see Cox and Hinkley19). It is closely comparable to a Bayesian posterior distribution for x corresponding to vague prior knowledge.
146
Quality of Analytical Measurements: Univariate Regression
After squaring and rationalizing Equation (65), we obtain a second degree equation. Its solutions are:11,18. 1 1 syx x þ ðxˆ 0 – xÞ t =2;N – 2 1–g 1 – g b1
sffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi 1 ðy0 – yÞ2 ð1 – g Þ þ 2 PN N b1 i¼1 ðxi – xÞ2
ð66Þ
with g¼
t2 –2 =2;N ¼ P 2 = b12 = syx ðxi – xÞ2
t=2;N – 2 2 b1 =sb1
ð67Þ
Note that the denominator of Equation (67) is the square of the statistic of the t-test to test H0 : 1 ¼ 0 versus Ha : 1 6¼ 0 (Equation (43)), which is equal to the F-test for significance of regression (Equation (28)). Thus, the more significant b1 is (e.g., smaller p-value in Table 1), the larger the denominator of g and the smaller g and ð1 – g Þ > 1 will be. This is a quite usual situation; so, instead of using Equation (66), the following approximation for the confidence interval20,21 is used: sffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi xˆ0 t =2;N – 2
syx b1
1 ðy0 – yÞ2 þ 2 PN N b1 i¼1 ðxi – xÞ2
ð68Þ
The above calculations are true mean value calculations. Nonetheless, in calibration, y0 is regarded not as a true mean value but as the mean of q observations. Thus, Equation (63) will be used for Varðy0 Þ and the confidence interval at level (1)100% is syx xˆ0 t =2;N – 2 b1
sffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi 1 1 ðy0 – yÞ2 þ þ 2 PN q N b1 i¼1 ðxi – xÞ2
ð69Þ
Then q ¼ 1 provides the confidence interval for the concentration when we have an individual observation, y0 , of the signal. Equations (13), (14), and (61) prove that the variance of the slope, intercept, and yˆ0 are the multiplication of 2 2 (estimated by syx ) by a factor that depends on the following: (1) the proximity of x0 to x, (2) the number N of calibration samples, and (3) the value xi of the calibration standards. As regards the estimated variance of xˆ0 , from Equation (69), is obtains the estimated variance of xˆ0 as ðy0 – yÞ2 1 1 2 1 dðxˆ 0 Þ ¼ syx Var þ þ P b12 q N b12 Ni¼1 ðxi – xÞ2 ! 1 1 ðxˆ 0 – xÞ2 2 1 þ þ ¼ syx PN b12 q N Þ2 i¼1 ðxi – x
! ð70Þ
that depends on the calibration design (N and xi values), on the proximity between y0 and y , and also on the estimated slope. To obtain the most precise estimation of the concentration, the problem sample should provide a signal (or mean value of q signals), y0, close to the central value of those of the calibration concentrations in such a way that P the numerator of Equation (70) is small. Moreover, one obtains better precision if the sum of squares Ni¼1 ðxi – xÞ2 in the denominator is large, which is achieved if the concentration of the standards is far from the mean value. The remaining addends in Equation (70) also indicate that precision is better if the number of calibration standards N is larger and/or if the number of replicate determinations in the problem sample, q, is larger too. But we also obtain better precision in the estimated concentration if the slope b1 is larger, that is, as the sensitivity of the calibration increases, because b1 intervenes (squared) in the denominator of Equation (70). As in all confidence intervals, the value of N also influences through the value of t/2,N2, which can be important. For example, for ¼ 0.05, if N ¼ 5, t0.025,3 is 3.182, whereas for N ¼ 10, t0.025,8 is 2.306.
Quality of Analytical Measurements: Univariate Regression
147
Example 4 With the data of the calibration of cadmium, Example 1, Table 6 shows the variation of the confidence interval at 95% for the concentration of a problem sample as a function of its position in the calibration range and the number of replicates used. To clearly see the effect, the values of the signals y0 supposedly obtained were 4.75, 10.20, and 15.50 nA, which are values near to the minimum, the mean and the maximum respectively of the recorded calibration signals. Furthermore, in each case, the signal corresponds to one, three, or five replicate(s), that is, q ¼ 1, q ¼ 3, or q ¼ 5 in Equation (69). In our case, t=2;23 ¼ 2:0687. From the results in Table 6, one can observe that for the signal of 10.20 nA to consider 3 or 5 replicates causes a reduction in the radius of the confidence interval of 40 and 52%, respectively. This reduction is much higher than that caused by the position of the sample in the calibration range, which represents a reduction of 4% when only one replicate is used and 16% when five replicates are used. Note the interaction between the number of replicates and the position of the calibration range. In general, it is advisable to determine at least three replicates to make the precision increase significantly. In any case, it is quite simple using Equation (70) to explore the combined effect of the position of the signal of the sample and the number of replicates, similar to Table 6, for any calibration. The expressions of the intervals and confidence region, Equations (40), (41), (44), (60), and (68), show that all of them become ‘wider’ if syx is larger, that is, the estimates are less precise. Furthermore, in the statistics of the t-test, Equations (43), (48), and (51), the standard deviation of the estimate, for example, appears in the denominator and increases when increasing syx. Thus, tcalc decreases and it becomes more difficult to reject the null hypothesis. In summary, an overestimation of the variance of the residuals, 2, has a negative effect on all the inferences, including those related to the calculated concentration. At this point, the reader must remember the analysis of the LOF in Section 1.05.3.1.2. Concretely, Equations (32) and (33) show that the bias of the model necessarily causes an increase in syx. Therefore, much attention has to be paid to the problem of assuring the adequacy of the model because in case the model is biased, all the inferential aspects are altered.
1.05.4.6
The Intersection of Two Linear Regression Lines
In some analytical techniques, it is necessary to determine the abscissa of the point where two straight line segments of an analytical signal intersect (e.g., conductometric and photometric titrations). In this case, we have two series of measurements, {(xi1,yi1), i ¼ 1, . . ., N1} and {(xi2, yi2), i ¼ 1, . . ., N2}, before and after the break 2 2 and syx2 as estimated residual point, and fit linear regression lines y ¼ b01 þ b11x and y ¼ b02 þ b12x with syx1 variance, respectively. The estimated abscissa in the intersection point is xˆ0 ¼
b01 – b02 b0 ¼ b12 – b11 b1
ð71Þ
Table 6 Confidence interval for the determination of the concentration of a problem sample with the calibration in Example 1 q¼1 y0 (nA) xˆ0 4.75 10.20 15.50
dðxˆ 0 Þ Var
19.80 1.518 40.37 1.405 60.37 1.515
q¼3
q¼5
h (nmol l1) Var dðxˆ 0 Þ
h (nmol l1)
dðxˆ 0 Þ Var
h (nmol l1)
2.549 2.452 2.546
1.626 1.469 1.622
0.437 0.324 0.434
1.368 1.178 1.363
0.618 0.504 0.614
y0, signal of the problem sample; q, number of replicates used in the determination; h, radius of confidence interval ( Xˆ0h, xˆ0þh ) at 95% (Equation (69))
148
Quality of Analytical Measurements: Univariate Regression
The statistical problem of the intersection of two experimental lines has been tackled in depth.21,22–25 A (1)100% confidence interval on the intersection can be obtained as the roots of the following quadratic equation in xˆ0 :
2 2 xˆ0 2 b12 – t 2 sb – 2xˆ0 b0 b1 – t 2 Covðb0 ;b1 Þ þ b02 – t 2 sb ¼0 1 0
ð72Þ
where t denotes the critical value t=2;N1 þN2 – 4 . 2 2 Supposing that the variance of the error, syx1 and syx2 , is the same in both segments, the pooled estimated common variance of Equation (49) is used to obtain the variances and covariances of b0 and b1. They are computed by the propagation of errors method, which is exact in these cases because b0 and b1 are linear functions of random variables. 2 sb 1
2 sb 0
¼
sp2
¼
sp2
1 1 PN1 2 þ PN2 ð x – x Þ ð x 2 Þ2 i1 1 i¼1 i¼1 i2 – x
!
1 1 x1 2 x2 2 þ þ PN1 2 þ PN2 N1 N2 1 Þ 2 Þ2 i¼1 ðxi1 – x i¼1 ðxi2 – x
Covðb0 ; b1 Þ ¼
sp2
x1 2 x2 2 PN1 2 þ PN2 1 Þ 2 Þ2 i¼1 ðxi1 – x i¼1 ðxi2 – x
ð73Þ ! ð74Þ ! ð75Þ
The groups of symbols enclosed in brackets in Equations (73) and (74) have the form of a two-tailed t-test for significant differences of slopes and significant differences of intercepts, respectively. When the hypothesis of equal slopes is not rejected using the statistic in Equation (48), the coefficient of xˆ0 2 in Equation (72) becomes negative and no real roots are obtained; hence, the confidence interval embraces the entire x axis at the chosen level of confidence.
1.05.5 The Design of a Calibration In this section, the effect of the distribution of the calibration standards on the quality of the calibration line will be analyzed. In principle, it is necessary to answer a series of questions based on the prior knowledge of the analyst on the behavior of the system of measurement in terms of the model supposed for the calibration. In relation to the calibration set, which is the range of concentrations of interest? Is the total number, N, of measurements to be made limited? Is it possible to replicate in all the levels of concentration? Should the values of the concentration of the calibration standards be selected among some fixed values or can they take any value from the specified range? Other questions are relative to the calibration model such as is it probable that the relation signal concentration is not linear in the specified range? To decide this, one’s previous knowledge or knowledge of other experts should be used. Does one believe that the curvature, if it exists, can be modeled with a polynomial model of low order (for example, of order 2 or 3)? Because, if this is not so, a study with a model of higher order should be initiated. The answer to these questions will be reflected in the number of d.f. needed to estimate the LOF and even in the possibility of fitting a calibration model of higher order to the experimental data. Finally, questions about the variance of the error should be posed: Is there any a priori information about it? Should it be estimated together with the slope and the intercept? If the answer is yes, then replicates are mandatory, which are much more important if, in addition, the possibility of failure of the equality of variances in the chosen range of concentrations exists. Once the above questions are answered, the quality of a calibration model is related to the variances of b0 and b1 and the correlation between them; thus, Equations (13), (14), and (20) should be used. Equations (13) and (14) show that the variance of both parameters is equal to a factor, called variance inflation factor (VIF), that depends only on the values of xi and not on the recorded signals. The second factor, 2, is the variance of the
Quality of Analytical Measurements: Univariate Regression
149
signal, which is either known or will be obtained from the experimental results. In both cases, we must accept that it is a characteristic of the measurement system and cannot be modified during the calibration stage. It is evident that by designing the distribution of the calibration standards, the value of VIF can be decreased and, therefore, the variance of the estimators decreases. The previous argument is the same as that under the methodology for response surfaces (see Chapter 1.12 of the book dedicated to design of experiments) but there is a difference; basically, a calibration model is not used to predict the instrumental signal, y0, but the concentration of a problem sample. The structure of Equation (59) of the variance of the estimated response is also the product of 2 by a variance function. This function is quadratic and its minimum is 1/N,P which is reached in the mean value of the concentrations. The form of this function depends on the factor 1= Ni¼1 ðxi – xÞ2 , which is also the VIF of b1 (Equation (13)). Consequently, the quality of the design of a linear calibration will be evaluated with the VIF and with the correlation between the slope and the intercept. Let us now continue our discussion in terms of a specific example. Let us suppose that the analyst has decided to do N ¼ 17 determinations in concentrations around 80 ppb (a hypothetical permitted limit (PL) for a residue); the calibration range considered is between 0.5 and 1.5 PL, that is, between 40 and 120 ppb (as usual in official norms). Furthermore, she or he wants the sample with 80 ppb to be always among the calibration samples. Also, the analyst thinks that the relationship is probably first order but is not absolutely sure. Finally, it can be considered that the variance 2 does not change in the calibration range, although it is unknown and should be estimated with the calibration data. Figure 8 shows some of the possibilities. Each dot represents one experimental determination at the concentration specified in ppb. For the same experiments and to avoid excessively small numbers, instead of the VIF, Table 7 shows the square root of the VIF, VIF1/2, for the independent term and the slope at each of the possibilities in Figure 8. VIF1/2 is in fact the factor that when multiplied with the residual standard deviation gives the standard deviation of the parameter. The number of different standards, k, and the d.f. of pure error and those for the LOF have also been shown in Table 7. As can be seen, all the designs include concentration 80 ppb and cover the proposed range. Design (1) is not acceptable because it does not allow considering the pure error. Designs (8) and (9) are not acceptable either because the estimation of the LOF is made with only 1 d.f. (in this case, if the LOF was significant, the calibration line of order 2 will pass obligatorily through the mean value of the signals obtained in the three concentrations 40, 80, and 120 ppb). Nevertheless, the factors of design (9) can be taken as the reference of the minimal VIF1/2, whereas those of design (1) as the maximal. When comparing these two designs, it is clear that factor VIF1/2 for the slope can be improved by about 36%, and thus also its standard deviation by the same percentage. The remaining designs would fulfill the conditions imposed, allowing, in case of curvature, the fitting of a polynomial of degree two or three. Designs (4) and (5) are especially adequate because they have 5 d.f. to estimate the LOF, and with design (5) a higher reduction (23%) of the standard deviation of b1 is obtained. It is interesting to note that when decreasing the number k of different standards, the value of VIF1/2 is not necessarily reduced – compare designs (3) and (4), 1/2 (5) and (6), or (7) and (8). The values of VIF P 2 for the independent term follow a similar tendency, although they are much higher because of the term xi that appears in the numerator. The strong linear dependency between the slope and the independent term must be observed, which stays between 0.96 and 0.90. To study the effect of the design on the variance of the computed concentration, Equation (70) will be used, 2 =b12 and VIFðxˆ 0 Þ where which is the product of syx VIFðxˆ 0 Þ ¼
1 1 ðxˆ 0 – xÞ2 þ þ PN q N Þ2 i¼1 ðxi – x
! ð76Þ
2 The value syx =b12 will be obtained after fitting the calibration line; both the residual variance and the slope are specific to the system and do not depend on the design. However, VIFðxˆ0 Þ of Equation (76) depends on the design and on the estimated value, xˆ0 , of the concentration of the problem sample. The minimum is reached when xˆ0 ¼ x (and is ð1=q Þ þ ð1=N Þ), and the maximum is in one of the extremes of the calibration range. Table 7 also contains
150
Quality of Analytical Measurements: Univariate Regression
(1) 40 50 60 70 80 90 100 110 120 ppb (2) 40 50 60 70 80 90 100 110 120 ppb (3)
40 50 60 70 80 90 100 110 120 ppb (4) 40 50 60 70 80 90 100 110 120 ppb (5) 40 50 60 70 80 90 100 110 120 ppb (6) 40 50 60 70 80 90 100 110 120 ppb (7)
40 50 60 70 80 90 100 110 120 ppb (8)
40 50 60 70 80 90 100 110 120 ppb (9)
40 50 60 70 80 90 100 110 120 ppb Figure 8 Nine different possible designs for the calibration. Each dot represents an experiment at the concentration (ppb) in the horizontal axis.
the maximum values of VIFðxˆ0 Þ1=2 , which are the maximum inflaction factors of the standard deviation of xˆ0 for the nine designs analyzed. The minimum value when q ¼ 1, that is, when only one signal is recorded for the problem sample, is 1.0290 and the maximum varies from 1.1026 to 1.0589; nevertheless, when q ¼ 2 (two signals are recorded for the problem sample and the mean is taken), the minimum is 0.7475 and the maximum varies, depending on the design, from 0.8460 to 0.7882. In the case of q ¼ 5, the minimum is 0.5087 and the maximum varies from 0.6447 to 0.5669. It is clear that the effect due to the number of replicates in the problem sample is much greater than that due to the design. From Equation (76) it is clear that for q ¼ 1 is VIFðxˆ 0 Þ1=2 > 1, therefore it is advisable to make at least two replicates of the problem sample. In the example, this action diminishes the factor at least by 15%. Design (5) with q ¼ 2 is a suitable choice to obtain acceptable inflaction factors.
Quality of Analytical Measurements: Univariate Regression
151
Table 7 Characteristics of the different designs in Figure 8 VIF(x)1/2 Design
VIF(b0)1/2
VIF(b1)1/2
Corr(b0,b1)
q¼1
q¼2
q¼5
k
d.f. for PE
d.f. for LOF
1 2 3 4 5 6 7 8 9
0.8284 0.7695 0.6849 0.7375 0.6514 0.6927 0.6054 0.6262 0.5557
0.0099 0.0091 0.0080 0.0087 0.0076 0.0081 0.0069 0.0072 0.0063
0.9562 0.9490 0.9352 0.9444 0.9281 0.9367 0.9162 0.9220 0.8997
1.1026 1.0919 1.0777 1.0863 1.0725 1.0789 1.0657 1.0687 1.0589
0.8460 0.8320 0.8133 0.8247 0.8063 0.8149 0.7973 0.8013 0.7882
0.6447 0.6262 0.6012 0.6165 0.5918 0.6034 0.5747 0.5849 0.5669
17 9 9 7 7 5 5 3 3
0 8 8 10 10 12 12 14 14
15 7 7 5 5 3 3 1 1
VIF(.) represents the maximum of the variance inflation factor of xˆ0, VIF(xˆ0); PE, pure error; LOF, lack of fit; q, the number of replicates; k, the number of calibration standards; d.f., the degrees of freedom.
Introducing the individual cost of each determination and by varying N and q, the cost can be optimized in addition to the quality of the design. In Section 2.2.8 of Meier and Zu¨nd,21 it is shown from Equation (70) that using the wrong combination of N and q enormously increases costs.
1.05.6 The Capability of Detection, the Decision Limit, and the Capability of Discrimination Computed from a Regression Model One of the main reasons for the development of new instrumental techniques is to increase their capability to determine much smaller quantities. It is therefore necessary to establish a rule to ensure whether the analyte is present or not in the sample. In general, a detection rule should be as follows: affirm that the analyte is present in a sample if the response obtained is above a threshold value. When applying this decision rule, two mistakes might happen: (1) affirming that the analyte is present in the sample when it is in fact not present, in other words, giving a ‘false positive’, and (2) affirming that the analyte is not present in the sample when it is in fact present, or giving a ‘false negative’. The classic definition for the detection limit26 only evaluates the probability of a false positive but does not explicitly quantify the probability of a false negative. A good revision of the evolution of the stance of the IUPAC can be found in the paper by Currie.27 Chapters 5 and 6 of Gibbons’s book28 contain an interesting discussion of the methods proposed for the calculation of the detection limit and include a good selection of bibliography on the subject. The capability of detection for a given probability of false positive, , as defined by the ISO,3 is ‘‘the true net concentration of the analyte in the material to be analysed which will lead, with probability 1, to the correct conclusion that the concentration in the analysed material is larger than that in the blank material’’. The need to evaluate the probability of false positive, , and of false negative, , has also been recognized by the IUPAC, and in the EU it is mandatory for the identification and quantification of toxic residues or residues that come from veterinary treatments in products for human consumption (European Decision 2002/657/EC). The concept of capability of detection is applicable to any level of concentration x0 and not only for the case of concentration zero. The following one-tail test30 is performed to estimate the capability of detection: H0 : x ¼ x0 (the concentration of the analyte in the sample is x0) versus Ha : x > x0 (the concentration of the analyte in the sample is greater than x0). Thus, the capability of detection, named CC in the above-mentioned European decision and xd in the ISO 11843-1 standard, can be estimated in the univariate case31–33 through a linear calibration model signal versus concentration using the following expression: xd ¼
ð; Þwx0 syx b1
ð77Þ
152
Quality of Analytical Measurements: Univariate Regression
where (, ) is the value of noncentrality parameter of a noncentral t-distribution related to the probabilities and , and wx20 is the VIF(x0) already defined in Equation (76). When the analyte is banned, the minimum detectable net concentration is estimated with x0 ¼ 0 in Equations (76) and (77) and and are the probabilities of false positive and false negative, respectively. However, if the analyte is not banned and a PL has been fixed, the capability of detection must also be evaluated at those levels of concentration. In this case, the capability of detection is estimated31,32,34 as in Equations (76) and (77) with x0 ¼ PL and and are the probabilities of false noncompliance and false compliance, respectively. Regarding the decision limit (Equation (78)), the ISO 11843-2 standard has defined the critical value (at zero) of the net concentration as ‘‘the value of the net concentration the exceeding of which leads, for a given error probability , to the decision that the concentration of the analyte in the analysed material is larger than that in the blank material’’. xc ¼
t;N – 2 wx0 syx b1
ð78Þ
where wx0 is the same as in Equation (77). For substances with a PL, the decision limit also has to be estimated35,29,36 as in Equation (78) but substituting xˆ0 by PL in Equation (76) to compute wx0. As the decision of accepting or rejecting the presence of analyte in the sample is established based on the signal, for an unknown sample the null hypothesis has to be rejected if the recorded signal is above a threshold yd (signal of detection); hence, the critical region of the test is the set of signals beyond the signal of detection, and a calibration function that relates signal with concentration is needed. The program in Sarabia and Ortiz33 implements the previously described procedure when x0 ¼ 0 and calculates the characteristic curves associated with the hypothesis test. In this way, the statistical method establishes the capability of detection of a specific analytical procedure, given by the characteristic curves of the decision rule expressed as a hypothesis test in terms of the concentration taking into account the calibration line. This procedure (despite being published earlier than the year 2000) is the one proposed by ISO 11843 and Commission Decision 2002/657/EC, estimating the minimum detectable net concentration xd, taking into account the probabilities and . Example 5 In the determination of ascorbic acid in soft drinks by High Performance Liquid Chromatography – Diode Array Detection (HPLC-DAD) (see Garcı´a et al.37 for more details), we want to compute the capability of detection for the experimental data in Table 8. The LS regression line computes an intercept, b0, equal to 221.5 with sb0 ¼ 41.6 and a slope, b1, 308124.0 with sb1 ¼ 2688.2. The residual standard deviation of the regression is 89.6 a.u. and the correlation coefficient is 0.9994. Furthermore, it is checked that the regression model is significant (p-value < 104) and that there is no significant LOF (p-value 0.24). There is neither failure in the equality of variances (the p-values for the Cochran and Bartlett tests are 0.63 and 0.14, respectively) nor failure of the normality of residuals. Using DETARCHI,33 Equations (76) and (77) with xˆ0 ¼ 0, with one replicate (q ¼ 1) it is possible to detect 0.0011 mg l1 with a probability of false positive, , equal to 0.05 and a probability of false negative equal to 0.05. As the computed capability of detection is less than the lowest concentration of the calibration standards, this value cannot be considered and the only fact we can assert is that the proposed procedure has a capability of Table 8 Calibration of ascorbic acid to determine the capability of detection Concentration (mg l1)
Peak area (a.u.)
0.004
0.006
0.008
0.012
0.016
0.020
0.026
1454 1420 1437
2063 2198 2078
2597 2586
3863 4042
5259 5241
6463 6279 6255
8280 8117 8305
Quality of Analytical Measurements: Univariate Regression
153
detection equal to 0.004 mg l1 (the lowest concentration whose signal was recorded) with a probability of false positive equal to 0.05 and a probability of false negative less than 0.05. Garcı´a et al.37 contains a discussion about the capability of discrimination at several non-null nominal concentrations, xˆ0 6¼ 0. The decision limit and the capability of detection defined in Equations (77) and (78) have the property of being invariant for linear transformations of the response36 (the signal). As a consequence, one obtains the same capability of detection and decision limit using the regression estimated concentration versus true concentration. In practice, this is useful because even when using nonlinear univariate calibration methods, or calibrations based on multiway and multivariate regression methods or based on neural networks, etc., it is always true that the relation between the estimated concentration, xˆi , and that of the calibration standards, xi , must be linear. This regression enables the evaluation of the accuracy of different calibration methods (see Section 1.05.4.2) and also the capability of detection and the decision limit of any of them. The capability of discrimination is another piece of information of interest provided by the calibration line. A method is said to be sensitive (IUPAC,29 Chapter 10) if a small change in concentration, x, causes a large change in the measurement, y; that is, when the derivative dy=dx is large. In the case of a linear calibration line, dy=dx ¼ b1 . As the slope is a random variable, it is not enough to consider only its mean but also its probability distribution or at least its variance. The capability of discrimination, that is, the smallest difference jx – x0 j such that the signal of x can be distinguished from that of x0, will depend on the distribution of both and also on the assumed and (risk of, respectively, concluding that there is a difference when there is none and concluding that there is no difference when in fact there is difference). Formally, it is about testing H0 :jx – x0 j ¼ 0 versus Ha :jx – x0 j > 0. The following equation has been proposed for the sensitivity (capability of discrimination):36 jx – x0 j ¼
ð=2; Þwx0 syx b1
ð79Þ
As for the capability of detection, the capability of discrimination can be computed using Equation (79) for the regression model xˆ ¼ b 0 þ b 1 x and, therefore, it is of use with any kind of calibration method (multivariate or multiway). In the same paper,36 several applications with partial least squares (PLS) calibration models are shown.
1.05.7 Standard Addition Method A calibration line cannot be used to determine an analyte in a sample when it is known that the matrix of the sample interferes with the determination and no blank is available for spiking at levels around the expected (‘nominal’) level. A possible solution to this problem is to apply the method of standard addition in which the sample is used as a kind of ‘blank’ to which further analyte is spiked. To do this, several quantities of the analyte to be determined are added to aliquots of the unknown sample. These spiked samples as well as the unknown sample are measured. Figure 9(a) is the usual representation of the data signal versus concentration added, that (b) Signal y
Difference y–y0
(a)
y0
y0
0 Estimated concentration
Amount added
0
Amount added
Estimated concentration
Figure 9 Procedure for the standard addition method: (a) extrapolation and (b) interpolation.
154
Quality of Analytical Measurements: Univariate Regression
is, the pairs (xi, yi) i ¼ 1, . . ., N. If conditions about linearity and NID(0,2) errors are fulfilled, the model fitted by LS is y ¼ b0 þ b1 x and the quantity of analyte in the sample, xs, is obtained by extrapolating the line to the abscissa (y ¼ 0, xs ¼ – ðb0 =b1 Þ) if there is no systematic error. The estimated value for the concentration of the problem sample is thus xˆs ¼ b0 =b1 and its variance can be obtained from Equation (70) for value xˆs : 1 1 ð – ðb0 =b1 Þ – xÞ2 2 1 dðxˆ s Þ ¼ syx þ þ PN Var 2 b1 q N Þ2 i¼1 ðxi – x
! ð80Þ
This variance can be reduced if the regression line is made with the data (xi, yiy0), i ¼ 1, . . ., N, as indicated in Figure 9(b). It is clear that the slope of the new line of standard addition is the same as before. The concentration of the problem sample is the one corresponding to ordinate y0 in the graph, which is also the value b0 of the previous regression model, so its variance is 1 1 ððb0 =b1 Þ – xÞ2 2 1 dðxˆ s Þ ¼ syx þ þ PN Var b12 q N Þ2 i¼1 ðxi – x
! ð81Þ
less than the previous one, Equation (80), due to the positive sign of b0/b1. In a concrete problem, to guarantee the absence of bias in the determination and therefore that the standard addition method is valid, the following should be demonstrated: (1) that proportional error does not exist, verifying that the slope of the standard addition line is the same as the calibration line obtained for the pure analyte in the absence of the matrix, and (2) the absence of absolute systematic error that will be verified by means of reference spiked samples. The second point is the weakness of the method of standard addition because on many occasions it is not possible to have reference samples without an analyte to evaluate the recovery. The H-point standard addition method38 is a modification of the standard addition method that permits correcting for both absolute and proportional systematic errors. In the simplest case, an interferent whose spectrum is known, the method is based on obtaining a standard addition line at two wavelengths, 1 and 2, where the interferent shows the same absorbance. The two standard addition lines intersect at the so-called Hpoint (xH, yH) and the estimated concentration is xˆs ¼ – x H . This method has been generalized39,40 to obtain the total concentration or concentrations of different chemical forms of an analyte when the matrix of the sample is completely unknown. Another solution comes from the calibration methods based on the decomposition of three-way data that identify the multivariate signal of the analyte and separate it from the signal(s) of the remaining constituents of the sample. The procedure was initially developed41 for the ‘direct trilinear decomposition’ method known as the ‘second-order standard addition method’ (SOSAM). It has been applied with other decomposition methods such as multivariate curve resolution,42 or PARAFAC,43 or direct trilinear decomposition (TLD);44 even an adaptation of this method to handle the case in which the standard addition method causes a reduction of the rank has been proposed.45
1.05.8 Weighted Least Squares and Generalized Least Squares Sometimes the signals used in a calibration have different variance. This can be deduced from a priori knowledge, from graphics y versus x, or from the analysis of the residuals of an LS fit. With a transformation of the signal, y, and possibly of the concentration, x, it might happen that a linear relation between the new variables is valid and a constant-variance additive error is obtained. When this is not possible, the adequate estimation is obtained with a weighted least squares fitting. The presence of correlation in the errors leads to a generalized least squares fitting. To make the analysis easier, the matrix notation will be introduced for the model in Equation (3). ! 0 þ "i i ¼ 1, . . ., N, may be written in the matrix form as The set of N equations yi ¼ ð 1 xi Þ 1 y ¼ Xb þ e
ð82Þ
Quality of Analytical Measurements: Univariate Regression
155
where y is the column vector comprising the values of the response, " is the column vector of residuals, b is the column vector with the slope and the independent term, and finally X is the matrix 0
1
x1
1
C B B. . C X ¼ B .. .. C A @ 1 xN
ð83Þ
and EðeÞ ¼ 0; CovðeÞ ¼ 2 W; and e follows a N ð0; 2 W Þ
ð84Þ
where W, the variance–covariance matrix of the errors, is an N N positive definite square symmetric matrix. Because of the properties of W, it can be shown that it is possible to find a nonsingular symmetric matrix U such that U – 1 U ¼ Ut U ¼ UU ¼ W
ð85Þ
1
By writing e ¼ U e, it is easy to prove that Eðe Þ ¼ 0 and Covðe Þ ¼ 2 I
ð86Þ
In addition, as e is a linear combination of normal distributions, it also holds that e follows a N(0, 2I). Now, if in Equation (82) both members are multiplied by U1, the following new model is obtained: U – 1 y ¼ U – 1 Xb þ U – 1 e
ð87Þ
ð88Þ
y ¼ X b þ e
2
whose residuals are NID(0, ). The LS solution of Equations (82) and (88) is written in the matrix form as b¼
b0 b1
!
–1 ¼ ðX t X Þ – 1 X t y ¼ Xt W – 1 X Xt W – 1 y
ð89Þ
and its variance–covariance matrix is given by
–1 CovðbÞ ¼ Xt W – 1 X 2
ð90Þ
ˆ where The residuals that must be checked are those corresponding to e that are obtained for e ¼ U1(Y Y), Yˆ ¼ Xb and b is obtained from Equation (89). These weighted residuals are used by the Analytical Methods Committee4 to evaluate the LOF. An estimate of is given by s¼
rffiffiffiffiffiffiffiffiffiffiffi et e N –2
ð91Þ
Often the error covariance matrix W is not completely known and the information provided by the residuals can suggest a parametric specification with just a few unknown parameters. The Gauss–Markov optimality of ˆ except for large sample sizes. Equation (89) is lost when W is substituted by an estimate W According to the hypotheses on the residuals, several procedures have been studied in detail. When matrix W is not a diagonal matrix, that is, when the residuals of Equation (82) are correlated, it is known as generalized least squares (GLS) and, when it is diagonal, it is known as weighted least squares (WLS). In the last case, the recorded signals for the calibration samples, and hence the residuals, are independent but with different variance from one level of concentration to another. This is the most frequent situation in chemical analysis. ˆ i , bi is estimated, Also, an iterative procedure to estimate the diagonal W has been proposed. By taking a trial W ˆ then re-estimating Wiþ1 and biþ1 until convergence, it is named iterative weighted least squares (IWLS). See, for example, Ryan’s book.12 The generalized linear model (GLIM) is about the case in which the distribution of the error is an exponential one (the normal distribution is one of them) and/or additionally, h(y), a nonlinear, monotone,
156
Quality of Analytical Measurements: Univariate Regression
differentiable function of the vector of responses y, that verifies h(y) ¼ Xb þ e. The maximum likelihood solution for estimating the parameters, b of GLIM, involves the use of generalized (weighted) LS in an iterative way. Since the work by Nelder and Wedderburn,46 the GLIM procedure has become a technique of increasing use although it needs special computation programs that are more and more accessible. The complexity of GLIM prevents its in-depth discussion in the present chapter, and the interested reader can consult Chapter 16 of the book by Draper and Smith11 for an introduction and the references cited therein for further details. In many practical situations, it is difficult to initially have specific information on W, Equation (84). For this reason, usually it is necessary to make the hypothesis (mistaken) that W ¼ I and to try to discover something about matrix W examining the residuals of the usual LS regression. The LS method provides estimates of the coefficients as bLS ¼ ðXt XÞ – 1 Xt y and E ðbLS Þ ¼ ðXt XÞ – 1 Xt Xb ¼ b but CovðbLS Þ ¼ ðXt XÞ – 1 Xt WXðXt XÞ – 1 2 . If the correct analysis is done, according to Equation (90), –1 CovðbGLS Þ ¼ ðXt W – 1 XÞ 2 holds. The consequence of this wrong estimation is that the variances of the independent term and slope in bLS are greater than the correct ones in bGLS. However, as the estimate bLS is unbiased, the calculated values of the estimates are rarely affected by the fact that the correct regression has not been made. Therefore, the analysis of the residuals of the fitting by LS allows checking the lack of homogeneity of variances and even estimating the weights to be used, that is, matrix W. In an analytical calibration, it can be assumed that W is a diagonal matrix and, thus, the WLS approach proves to be adequate. Sometimes, the observation of the residuals allows fitting the variance as a function, f, of the predictor variable.11 Davidian and Haaland47 describe a procedure to estimate the GLS and the variance function, f, by means of an iterative procedure. It is also possible to use a GLIM to estimate the values of W. A common alternative consists of estimating the values wi of the main diagonal of matrix W by using replicates in each level of concentration, xi. In a calibration experiment, the estimate of the concentration of the problem sample, xu, is similar when using the LS or WLS regression, but its standard deviation will be different, and, further, depending on the position in the range of concentration, the incorrect estimation VarLS(xu) may be lesser or greater than VarWLS(xu).20,21,48 Tellinghuisen49 has analyzed recently this subject by using Monte Carlo simulation and showed that the use of replicates gives great improvement at small concentrations but can underperform LS regression in the mid-tolarge concentration region. Nevertheless, an estimation of the variance function approximates minimum variance even though the true variance functions are not well reproduced. The WLS regression for calibration has been recommended by several organizations;50,51 consequently, it is being used with increasing frequency in analytical chemistry. A survey of the analytical applications of the use of weighted linear regression is given in Table 10 of Asuero and Gonza´lez.52 Therefore, it is interesting to describe the basic inferential aspects of WLS regression. As much if the variance for each xi is estimated with replicates, si2, as if this is done by a function s2 ¼ f(x), it is usual to estimate the elements, wi, of the diagonal of W by 1=si2 (or 1=f ðxi Þ). The confidence intervals and the hypothesis tests for the slope and the independent term are computed as in the LS regression but taking the values of variances of Equation (90) and the residual standard deviation in Equation (91). Given a value of concentration, x0, the response is estimated by yˆ0 ¼ ð 1 x0 Þ
b0
! ð92Þ
b1
and the estimate of its variance is dðyˆ 0 Þ ¼ ð 1 Var
ˆ 1 X 1 x0 Þ Xt W
1 x0
! s2
ð93Þ
Thus, the confidence interval for the expected mean value at level (1)100% is yˆ 0 – t =2;N – 2
qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi dðyˆ 0 Þ; yˆ0 þ t =2;N – 2 Var dðyˆ 0 Þ Var
ð94Þ
Quality of Analytical Measurements: Univariate Regression
157
The estimated variance of the prediction for the average of q new observations is " dðyˆ 0 Þ ¼ Var
1 1 1 W1 X þ ð 1 x0 Þ Xt c w0 q x0
!# ð95Þ
s2
where w0 is the weight corresponding to x0. As a consequence, the confidence interval is vffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi !ffi u 1 1 u 1 c1 X þ ð 1 x0 Þ Xt W yˆ0 t =2;N – 2 s t w0 q x0
ð96Þ
Finally, once the calibration is over, for a problem sample with value y0, the mean of q recorded signals in q replicates, the estimated value of the concentration is xˆ0 ¼
y0 – b0 b1
ð97Þ
An argument similar to that used to obtain Equation (69) provides the corresponding approximate confidence interval s xˆ0 t =2;N – 2 b1
sffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi 1 1 ðy0 – yw Þ2 þ PN þ 2 PN w0 q b1 i¼1 wi ðxi – xw Þ2 i¼1 wi
where again w0 is the weight corresponding to x0 and yw ¼ are the weighted means.
PN i¼1
wi yi =
PN i¼1
ð98Þ
wi and xw ¼
PN i¼1
wi xi =
PN i¼1
wi
Example 6 For the determination of benzaldehyde by differential pulse polarography,53 a calibration was made with nine levels of concentration and four replicates in each level. Table 9 shows the intensity recorded together with the means and variances in each level of concentration. The LS regression for these N ¼ 36 data (xi, yi) is y ¼ 0.0021 þ 1.9401x The estimated standard deviations are 0.004697 and 0.04147 for the independent term and slope, respectively. Figure 10(a) shows the standardized residuals and evidences that the dispersion of the residuals increases with concentration. The Cochran test to decide if one of the nine variances is significantly different from the rest has p-value equal to 0.18; thus, there is no evidence to reject the null hypothesis at level 0.05. This is a quite general situation in analytical calibrations because even when the variances vary much, usually they vary progressively in such a way that none of variances is significantly greater or smaller than the others. However, Table 9 Experimental data (with means and variances) of the determinations of benzaldehyde Concentration (mmol l–1)
Current (mA)
0.0199 0.0450 0.0593 0.0884 0.0980 0.1270 0.1370 0.1640 0.1740
0.0330 0.0823 0.1233 0.1769 0.2068 0.2553 0.2825 0.3427 0.3660
0.0371 0.0914 0.1149 0.1573 0.1874 0.2243 0.2417 0.3175 0.3311
0.0340 0.0905 0.1134 0.1606 0.1913 0.2340 0.2612 0.2961 0.3097
0.0330 0.0845 0.1155 0.1680 0.1777 0.2456 0.2689 0.3078 0.3447
Mean, xi
Variance, si2
0.0343 0.0872 0.1168 0.1657 0.1908 0.2398 0.2636 0.3160 0.3379
3.73 106 2.00 105 1.96 105 7.58 105 1.47 104 1.83 104 2.90 104 3.93 104 5.59 104
158
Quality of Analytical Measurements: Univariate Regression
(b) 2.5
2.0
2.0
1.5
Standardized residuals
Standardized residuals
(a) 2.5
1.0 0.5 0.0 –0.5 –1.0 –1.5 –2.0 –2.5
1.5 1.0 0.5 0.0 –0.5 –1.0 –1.5
0
0.02 0.04 0.06 0.08 0.1 0.12 0.14 0.16 0.18 Benzaldehyde (mmol l –1)
–2.0
0
0.02 0.04 0.06 0.08 0.1 0.12 0.14 0.16 0.18 Benzaldehyde (mmol l –1)
Figure 10 Standardized residuals: (a) for the LS regression and (b) for the WLS regression.
the Bartlett test, which is adequate to detect groups of variances, has a p-value equal to 0.01, which means that, at a significance level of 0.05, the null hypothesis of homogeneity of variances should be rejected. It is interesting to note that the residuals in Figure 10(a) are compatible with the hypothesis of normal distribution. The tests of 2, Kolmogoroff, Saphiro–Wilks, skewness, and kurtosis have p-values of 0.43, 0.78, 0.97, 0.65, and 0.33, respectively; thus, clearly, at a significance level of 0.05, the hypothesis of normal residuals cannot be rejected. The plot of si2 versus xi suggests a quadratic relationship, which we estimate by LS as s 2 ¼ 0:00004 – 0:00195 x þ 0:02670 x2
which explains 96.27% of the variance of s2. Each individual xi value of concentration can be substituted in this equation, then estimate si2 (i ¼ 1, . . ., 36) and invert these values to give the estimated weights wˆi of the diagonal of matrix W. Then, from Equation (89), the WLS regression is y ¼ – 0:0021 þ 1:9565x
with estimated standard deviations 0.00196 and 0.03392 for the independent term and the slope, respectively. As the theory says, the estimated values of b0 and b1 are similar as both regressions provide unbiased estimates. However, the difference is in the standard deviation estimated for each one, which is reduced by 58% for the independent term and by 18% for the slope. Figure 10(b) shows the standardized residuals of WLS regression; unlike the LS case now they are practically equal for all the values of the concentration. Strictly speaking, they should have been represented pffiffiffiffiffi versus wi xi , but to make the comparison with the LS residuals easier, the unweighted values xi are maintained; this does not modify the clear pattern. Figure 11 shows the length of the confidence intervals (x–h, x þ h) at level 95%. For each abscissa value, x, the ordinate shows the two values þh and –h. The red ones are those obtained when using LS; the continuous line represents values that correspond to a mean signal (Equation (60)) and the dashed line those obtained in prediction of a future signal, q ¼ 1 in Equation (63). The length of the analogous intervals obtained with WLS is in blue; the continuous line represents the mean signal (Equation (94)) and the dashed line for the prediction (Equation (96)). The remarkable asymmetry in the length of the intervals introduced by the weighting is evident. Up to 0.1 mmol l1 of benzaldehyde, the intervals are shorter with WLS than with the incorrect LS, especially between 0.03 and 0.04 mmol l1. Much more interesting from the analytical point of view is the confidence interval (x0h, x0 þ h) for the concentration x0 determined in a problem sample, Equation (69) for LS and Equation (98) for WLS. In both cases, we use q ¼ 1 and 95% confidence level. Figure 12 shows along abscissas the values of x0 and along
Quality of Analytical Measurements: Univariate Regression
159
Length of the confidence interval
0.06 0.04 0.02 0.00 –0.02 –0.04 –0.06 0.00 0.02 0.04 0.06 0.08 0.10 0.12 0.14 0.16 0.18 0.20 Benzaldehyde (mmol l –1) Figure 11 Length of confidence intervals for LS (in red) and for WLS (in blue). Continuous lines mark values corresponding to the mean signal and the dashed lines those obtained for prediction of a future sample.
Length of the confidence interval
0.03
0.02
0.01
0.00
–0.01
–0.02
–0.03 0.00 0.02 0.04 0.06 0.08 0.10 0.12 0.14 0.16 0.18 0.20 Benzaldehyde (mmol l –1) Figure 12 Length of confidence intervals for LS (in red) and for WLS (in blue).
ordinates the values of h: red for LS and blue for WLS. Up to the concentration of 0.11 mmol l1, the interval determined by WLS is shorter than that determined when incorrectly using LS. The analysis of Figure 12 shows that the most important impact of WLS is in the decision limit and the capability of detection. When x0 ¼ 0, with the (incorrect) LS calibration model, the capability of detection is 0.0232 mmol l1 for ¼ ¼ 0.05. However, with the WLS calibration model, the capability of detection is 0.0118 mmol l1 for the same probabilities of false positive and false negative.
1.05.9 Robust Regression in Calibration In a linear univariate calibration, the visual inspection of the data can indicate the presence of outlier data, that is, data that are different from the majority and do not follow the general linear tendency. The first observation is that the outlier data are not necessarily incorrect or mistaken data. Sometimes, their presence indicates some
160
Quality of Analytical Measurements: Univariate Regression
phenomenon unexpected when the experimentation began but that has chemical or experimental explanation. The problem raised by outlier data is regarding their detection, after which the analyst will be able to evaluate their significance and act accordingly. The outlier data give rise to highly asymmetric distributions or distributions with great accumulated probability far from the central values. In these conditions, the hypothesis of normality is not acceptable. In addition, it is necessary to consider that, in a calibration, the sample sizes are not so large so that the normality tests behave conservatively and do not detect ‘lack of normality’. The effect of outlier data on LS regression may be notable because it first affects the estimates of the intercept, b0, the slope, b1, and the concentration, xˆ0 , of a problem sample. It also affects the precision of these 2 estimates through syx . On examination of the equations of the confidence intervals of the coefficients of the model (Equations (40), (41), (44), and (47)), those of the predictions (Equations (60), (62), and (63)), and those for the concentration of the problem sample (Equations (68) and (69)), and also considering all the hypothesis tests used on these 2 of the random error, ", of parameters, it is clear that they depend on a good estimate of the residual variance syx the model. It suffices to observe how this estimate is made in Equation (10) to deduce that the presence of outlier data inflates this value and, as a consequence, all the intervals become larger, that is, the estimates will be 2 less precise. On the contrary, in the hypothesis tests about the slope and intercept, the syx appears in the denominators, therefore causing the statistic tcalc to take smaller values, leading to the more frequent acceptance of null hypotheses when they should not be accepted. Several robust regression methods insensitive (to a greater or lesser extent) to the presence of outlier data have been developed.11,54 This is why they are able to detect the true linear calibration curve and allow the analyst to decide, with objectivity, which data disagree and to re-establish the conditions under which the LS regression is optimal. Different robust regression methods have already been introduced to analytical chemistry (updated revisions of these applications can be found in Ortiz et al.55 and Daszykowski et al.56). Also, the ISO has recommended its use (ISO 5725-5). Given the relevance of the subject, the present book devotes a complete chapter to the introduction of robust methods (Chapter 3.07). The robust regression methods are usually evaluated according to two indices: (1) The ‘sampling breakdown’: the smallest fraction of contamination that can cause the estimator to take on values arbitrarily far from the correct ones. (2) The ‘influence function’ that describes the effect on the estimator of an infinitesimal contamination in the sample. In a robust procedure, this function must be bounded. For LS regression, one outlier suffices to carry LS over all bounds. Therefore, for a sample of size n, its breakdown is 1/n, which tends to 0 for increasing sample size. With regard to the influence function, the LS estimator is not bounded on both axes; hence, it is very sensitive to any deviation of data as much in the abscissas (‘leverage point’) as in the ordinates (‘outlier point’), and, of course, both deviations can come together (‘outlier-leverage point’). Rousseeuw57 developed the so-called least median squares (LMS) regression and Massart et al.58 showed the advantages of its use in chemical analysis. The theory and versatility of its application can be seen in Rousseeuw and Leroy54 and Rousseeuw.59 The least median squares estimators of 0 and 1 are the values that minimize, instead of the sum of squares, the median of squares of residuals, that is, those that minimize the following function R:
Rðb0 ; b1 Þ ¼ median ei2 ; i ¼ 1; 2; :::; N ¼ median ðyi – yˆi Þ2 ; i ¼ 1; 2; :::; N
¼ median ðyi – b0 – b1 xi Þ2 ; i ¼ 1; 2; :::; N
ð99Þ
The LMS method has a bounded influence function on both axes and 50% breakdown. The effect of outlier data in an LS regression depends on their relative position in the cloud of points, which is measured by indices as the ‘leverage effect’.10 In a calibration, there will not be any leverage points, that is, samples whose concentration greatly differs from the rest. But if the experimental data include concentration values beyond the linear range of the analytical method, it is clear that they will exert a similar influence on regression to that of a leverage point. Furthermore, the LMS estimator has the property of the ‘exact fit’, which guarantees that if at least 50% of the data (x, y) follow a linear model, the LMS regression finds it. This property is of great interest when one
Quality of Analytical Measurements: Univariate Regression
161
needs to determine the range of linearity at the same time as the sensitivity of the analytical method, which is usually the case in calibration problems. The relation of LMS with the least trimmed squares (LTS) and a comparison with other 10 procedures (including fuzzy methods) can be seen in Ortiz et al.55 For more statistical background about LTS and other references, see Rousseeuw and Leroy54 and Stromberg;60 for computational advances, see Watson.61 The recent developments in PROGRESS62 and fast LTS63,64 have been included in a MATLAB library of robust statistical methods.65 The procedure for the calibration with LMS regression consists of the following: (1) obtaining with the calibration data the LMS regression line, (2) evaluating the standardized robust residuals to establish the data that can be considered outlier (if the residual in absolute value is greater than 2.5, the data are rejected), and (3) re-making the LS regression without the outlier data detected with the LMS regression. Traditionally, the detection of nonaligned data, decisive for the establishment of the range of linearity, is carried out by a detailed inspection of the residuals of the LS regression and many indices such as Cook distance, DFFITS, and DFBETAS.10,11 This procedure requires great experience in the analysis of regressions, which is perhaps not available in the routine working of a chemistry laboratory. The availability of a computerized method such as the LMS regression, with sufficient mathematical properties to successfully perform this task, is the key to the calibration proposed. Example 7 The linear range in the determination of nickel by square-wave adsortive-stripping voltammetry (SWAdSV) using dimethylglyoxime (DMG) was studied.66 The saturation of the electrode surface, caused by the voluminous complex of Ni–DMG, leads to an increasing curvature. Table 10 shows the experimental data corresponding to a calibration of this system. For more details, see Sanllorente et al.66 The LS regression estimates the slope as 193.12 and the independent term as 9.64 with standard deviations 14.05 and 3.37, respectively. The correlation coefficient is 0.96 and syx is 7.64 mA. The externally studentized residuals are not large; all of them are less than 2.5 in absolute value. Further, the p-values of the normality tests are 0.26, 0.77, 0.09, and 0.58 for the tests of 2, Kolmogoroff, Shapiro–Wilks, and skewness, respectively. However, the residuals show a systematic tendency, nonrandom, as can be seen in Table 10 Experimental data and residuals of the regression least squares (LS) and least median squares regression (LMS) for the calibration of nickel Nickel (mmol l–1)
Current (mA)
Externally studentized LS residuals
LMS standardized residuals
0 0.024 0.047 0.070 0.093 0.116 0.139 0.161 0.183 0.205 0.227 0.249 0.270 0.291 0.312 0.333 0.374 0.395 0.415
2.50 7.22 12.71 19.13 25.15 30.79 39.02 46.14 52.83 57.13 60.76 66.50 70.47 73.12 73.46 76.27 72.13 76.87 75.01
1.04 1.00 0.84 0.56 0.34 0.17 0.33 0.72 1.06 1.07 0.98 1.20 1.19 0.99 0.48 0.31 1.42 1.33 2.47
0.54 0.17 0.48 0.30 0.29 0.42 0.66 1.33 1.78 1.09 0.07 0.07 0.66 2.02 4.51 5.79 12.96 13.32 16.73
162
Quality of Analytical Measurements: Univariate Regression
Current (μA)
80 60 40 20 0 0
0.1
0.2 0.3 Nickel (μmol l –1)
0.4
0.5
Figure 13 Regression line by least squares for data of Table 10.
Figure 13. At the beginning, they are negative, then positive, and finally negative again. The most important conclusion is that the calibration line does not reproduce the experimental data in the region of the concentrations in which the electrode has not still saturated and the response is theoretically linear. The LMS regression, with independent term 1.389 and slope 261.91, points as outlier data those corresponding to the greatest five concentrations (fourth column of Table 10) because their standardized robust residuals are greater than 2.5 in absolute value. In addition, they show a systematic tendency for those five data; the residuals ei ¼ yi – yˆi are negative and decreasing. That is, the experimental data of intensity do not lie on a straight line, and deviate more and more from it with increasing concentration, which is concordant with the effect of saturation of the electrode. Once the outlier data are removed and the LS regression is repeated with the remaining 14 data, the intercept is 2.05 and the slope is 257.66 with standard deviations of 1.05 and 6.04, respectively. The standard deviation of the regression syx is now 2.04 and the correlation coefficient is 0.997. When comparing these results with those of the first LS regression, the negative effect of the outlier data on the estimates and, thus, on the confidence intervals and the capability of detection is evident. Figure 14 shows the calibration line obtained with LS without the outlier data detected by LMS regression. Still a systematic tendency is observed in the residuals, a tendency that is assumable given the correlation coefficient and the residual standard deviation, and taking into account the fact that the LOF cannot be tested because there are no replicates. As a consequence, the linear range of this calibration can be established as the interval of concentrations up to 0.291 mM. The property of the exact fitting of LMS regression allows removing the influence of the curvature in the determination of the end point in titration analysis, as graphically shown in Figure 15. This approach has been used67 to determine the equation of the linear segments in amperometric, conductimetric, spectrophotometric, and potentiometric titrations.
Current (μA)
80 60 40 20 0 0
0.05
0.1 0.15 0.2 Nickel (μmol l –1)
Figure 14 Regression line by LS without the outlier data.
0.25
0.3
163
Corrected signal
Quality of Analytical Measurements: Univariate Regression
V V′ Added volume Figure 15 V9 is the end volume calculated when the first linear section has been fitted by LS (dashed line). V is the end volume calculated when the first linear section has been fitted by LS without outlier data detected by means of LMS (solid line).
1.05.10 Errors in Both Variables In the previous sections, the predictor variable (e.g., the concentration of the standard samples) is supposed to be determined without random error, unlike the recorded signal that is affected by a random variation. In practice, there is no quantity experimentally determined without variability (random error) but, in many occasions, the error in the predictor variable is much less than that of the response; in this case, a model as the one in Equation (3) is adequate. However, consider for instance the case of comparing two analytical methods by linear regression; in this case, the abscissa contains the concentrations determined by one method and the ordinates those by the second method; therefore, the model supposed in Equation (3) cannot be adequate because both concentrations will have a similar experimental variability. The problem is posed by accepting that both the predictor variable, x, and the response, y, are yi ¼ i þ "i ;
i ¼ 1; :::; N
x i ¼ i þ i ;
i ¼ 1; :::; N
ð100Þ
and there is a linear relation between the true unobserved variables y ¼ 0 þ 1
ð101Þ
The solution to the problem of estimating the parameters 0 and 1 and the variance of the errors " and depends on the hypothesis under the distribution of the errors. Adcock68 suggested 130 years ago, minimizing the sum of squared perpendicular distances to the fitted line. Let us assume that "i are NID(0,2") and that i are NID(0,2) with " and uncorrelated. It can be proved11 that the solution by LS will provide bias in the following cases: (1) If 2 is small compared with the or no PN little 2 2 2 spread in the i values ( ¼ i¼1 i – =N ). (2) If the x’s are fixed by the experimenter. (3) If we want to fit yi ¼ i þ "i , where i ¼ 0 þ 1 xi (the observed xi) instead of Equation (101). The maximum likelihood estimates of 0 and 1 have an identifiability problem; the estimation cannot be carried out without some additional information being added, for example knowledge of the ratio ¼ 2" =2 . This situation can be assumed when analysis methods are compared because the repeatability of both methods can previously be known. If were known, maximum likelihood leads to estimates PN b1 ¼
P Þ2 – Ni¼1 ðxi – xÞ2 i¼1 ðyi – y þ P 2 Ni¼1 ðxi – xÞðyi – yÞ
b0 ¼ y – b 1 x
qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi P N
2 PN
2ffi P Þ2 – Ni¼1 ðxi – xÞ2 þ 4 Þðyi – yÞ i¼1 ðyi – y i¼1 ðxi – x P 2 Ni¼1 ðxi – xÞðyi – yÞ
ð102Þ
164
Quality of Analytical Measurements: Univariate Regression
qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi P P PN P Note that, if is estimated as Ni¼1 ðyi – yÞ2 = Ni¼1 ðxi – xÞ2 , then b1 ¼ Þ2 = Ni¼1 ðxi – xÞ2 , which is i¼1 ðyi – y the geometric mean functional relationship. In this case, b1 is the geometric mean of the slope in LS fit of y versus x and the inverse of slope of x versus y, so b1 is a compromising value lying in between the two equation slopes. In addition, the geometric mean functional relationship minimizes the sum of areas obtained by drawing horizontal and vertical lines from each data point to the fitted line. A more detailed discussion can be seen in Draper and Smith.11 An important particular case of Equation (102) corresponds to ¼ 1, that is, when both variances 2" and 2 are equal. The solution, called orthogonal regression, is the line that minimizes the sum of squares of perpendicular deviations from the line. This line is the same as the first principal component of the data (xi, yi), i ¼ 1, . . ., N, and is the solution proposed by Adcock. In Section 15.2 of Jackson’s book,69 the inferential procedures for the orthogonal regression line can be found. A very detailed and rigorous statistical analysis of diverse general models for the errors in Equations (100) and (101) can be found in the book by Fuller.70 Also, solutions based on LS have been proposed, that is, without supposing any distribution on the errors " and . In the approach by Ripley and Thompson,71 it is considered that " and are uncorrelated and heteroscedastic. The solution is then obtained by an IWLS procedure. By adequately modifying the weights, the previous iterative procedure allows considering correlated errors in both axes, the so-called bivariate LS procedure.72,73 Finally, it is proved that the total least squares (TLS)74 procedure estimates the true parameters of Equations (100) and (101) consistently, that is, the solution that provides TLS converges to the true values of 0 and 1 when N tends to infinity. This property of TLS estimates does not depend on any assumed distribution for the errors " and . TLS is a very useful method for the problem of errors in variable estimation.
1.05.11 Final Remark There are many regression procedures that can be of interest for calibration in analytical chemistry; however, there is no room for them in a chapter of introduction to the subject. Even so, the user of a regression for analytical calibration must draw her/his calibration data. The numerical indices that serve to evaluate the quality of a regression and therefore guarantee the analytical result and its precision are a limited vision of the calibration. The advantage of the univariate calibration is the possibility of the visual inspection of the data and it should not renounce to it, as well as to a careful reflection on the type of errors that are expected to obtain. The unthinking uses of a linear regression lead to seriously misleading conclusions. In this respect, the example of Anscombe75 is didactic; Table 11 shows the results of four linear regressions, N ¼ 11, with a single replicate. The equality of the estimates and their standard deviations is evident; thus, any other parameter deduced from them (e.g., the linearity coefficient, value and precision of the estimated response) will also be equal. Even the statistic for the significance of the regression indicates that they are ‘equally’ significant. It suffices to show the data graphically, Figure 16, to see the error made in the last three cases when using a linear regression because the data, except for case 1, were not compatible with the supposed model. Table 11 Analysis by means of a linear least squares regression of the Anscombe’s data
Intercept, b0 Slope, b1 Standard deviation of b0 Standard deviation of b1 Standard deviation of regression, syx Correlation coefficient, r Fcalc (Equation (28))a p-value of Durbin–Watson test a
p ¼ 0.002.
Case 1
Case 2
Case 3
Case 4
3.0001 0.5001 1.1248 0.1179 1.2366 0.82 17.99 0.010
3.0009 0.5000 1.1253 0.1180 1.2372 0.82 17.99 0.372
3.0024 0.4997 1.1245 0.1179 1.2363 0.82 17.97 0.402
3.0017 0.4999 1.1239 0.1178 1.2357 0.82 18.00 0.290
Quality of Analytical Measurements: Univariate Regression
(b)
12.2
Response variable, y
Response variable, y
(a)
10.2 8.2 6.2
4
6
8 10 12 Predictor variable, x
9.1 7.1 5.1
14
4
(d)
13.3
Response variable, y
Response variable, y
11.1
3.1
4.2
(c)
165
11.3 9.3 7.3
6
8 10 12 Predictor variable, x
14
12 14 16 Predictor variable, x
20
13.2 11.2 9.2 7.2 5.2
5.3 4
6
8 10 12 Predictor variable, x
14
8
10
18
Figure 16 Graphical representation of the data from Anscombe: (a) case 1; (b) case 2; (c) case 3; and (d) case 4 of Table 11. The lines are those fitted by least squares.
References 1. Dux, J. P. Handbook of Quality Assurance for the Analytical Chemistry Laboratory, 2nd ed.; Van Nostrand Reinhold: New York, 1990. 2. IUPAC Recommendation, Guidelines for Calibration in Analytical Chemistry. Part 1 Fundamentals and Single Component Calibration. Pure Appl. Chem. 1998, 70, 993–1014. 3. International Standard ISO 11843. Capability of Detection. Part 1. Terms and Definitions, 1997 and Part 2. Methodology in the Linear Calibration Case, 2000; International Organisation for Standardisation: Geneva, Switzerland, 1997 and 2000. 4. Analytical Methods Committee. Is my Calibration Linear? Analyst 1994, 119, 2363–2366. 5. Mulholland, M.; Hibbert, D. B. Linearity and the Limitations of Least Squares Calibration. J. Chromatogr. A 1997, 762, 73–82. 6. Huber, W. On the Use of the Correlation Coefficient r for Testing the Linearity of Calibration Functions. Accred. Qual. Assur. 2004, 9, 726. 7. Hibbert, D. B. Further Comments on the (Miss-)Use of r for Testing the Linearity of Calibration Functions. Accred. Qual. Assur. 2005, 10, 300–301. 8. Ellison, S. L. R. In Defence of the Correlation Coefficient. Accred. Qual. Assur. 2006, 11, 146–152. 9. Asuero, A. G.; Sayago, A.; Gonza´lez, A. G. The Correlation Coefficient: An Overview. Crit. Rev. Anal. Chem. 2006, 36, 41–59. 10. Cook, R. D.; Weisberg, S. Residuals and Influence in Regression; Chapman and Hall: New York, 1982. 11. Draper, N. R.; Smith, H. Applied Regression Analysis, 3rd ed.; John Wiley and Sons: New York, 1998; p 43. 12. Ryan, T. P. Modern Regression Methods; John Wiley and Sons: New York 1997; p 64. 13. Bru¨ggemann, L.; Quapp, W.; Wennrich, R. Test for Non-Linearity Concerning Linear Calibrated Chemical Measurements. Accred. Qual. Assur. 2006, 11, 625–631. 14. Cuadros, L.; Garcı´a, A. M.; Jime´nez, C.; Roma´n, M. Estimation of Performance Characteristics of an Analytical Method Using the Data Set of the Calibration Experiment. Anal. Lett. 1993, 26, 1243–1258. 15. de Beer, J. O.; de Beer, T. R.; Goeyens, L. Assessment of Quality Performance Parameters for Straight Line Calibration Curves Related to the Spread of the Abscissa Values Around their Mean. Anal. Chim. Acta 2007, 584, 57–65. 16. Arroyo, D.; Ortiz, M. C.; Sarabia, L. Multiresponse Optimization and Parallel Factor Analysis, Useful Tools in the Determination of Estrogens by Gas Chromatography–Mass Spectrometry. J. Chromatogr. A 2007, 1157, 358–368. 17. Sachs, L. Applied Statistics. A Handbook of Techniques; Springer-Verlag: New York, 1982; p 442. 18. Brown, P. J. Measurement, Regression and Calibration; Clarendon Press: Oxford, 1993; pp 26–29. 19. Cox, D. R.; Hinkley, D. V. Theoretical Statistics; Chapman and Hall: London, 1974. 20. Massart, D. L.; Vandeginste, B. G. M.; Buydens, L. M. C.; de Jong, S.; Lewi, P. J.; Smeyers-Verbeke, J., (eds.), Handbook of Chemometrics and Qualimetrics, Part A, Data Handling in Science and Technology 20A; Elsevier: Amsterdam, 1998. 21. Meier, P. C.; Zu¨nd, R. E. Statistical Methods in Analytical Chemistry, 2nd ed.; John Wiley and Sons: New York, 2000; pp 118–120. 22. Fisher, R. A. Statistical Methods for Research Workers, 13th ed.; Oliver and Boyd: Edinburg, 1963. 23. Lark, P. D.; Craven, B. R.; Bosworth, R. C. L. The Handling of Chemical Data; Pergamon: London, 1968. 24. Liteanu, C.; Rica, I. Statistical Theory and Methodology of Trace Analysis; Ellis Horwood: Chichester, 1980. 25. Carter, K. N. Jr.; Scott, D. M.; Salmon, J. K.; Zarcone, G. S. Confidence Limits for the Abscissa of Intersection of Two LeastSquares Lines such as Linear Segmented Titration Curves. Anal. Chem. 1991, 63, 1270–1278.
166
Quality of Analytical Measurements: Univariate Regression
26. International Union of Pure and Applied Chemistry. Analytical Chemistry Division. Commission on Spectrochemical and Other Optical Procedures for Analysis. Nomenclature, Symbols, Units and Their Usage in Spectrochemical Analysis-II. Data Interpretation. Spectrochem. Acta B 1978, 33, 241–245. 27. Currie, L. A. Detection and Quantification Limits: Origins and Historical Overview. Anal. Chim. Acta, 1999, 391, 127–134. 28. Gibbons, R. D. Statistical Methods for Groundwater Monitoring. Wiley: New York, 1994. 29. Incze´dy, J.; Lengyel, T.; Ure, A. M.; Gelencse´r, A.; Hulanicki, A. Compendium of Analytical Nomenclature, 3rd ed.; International Union of Pure and Applied Chemistry, Port City Press, Inc.: Baltimore 2000; p 74. 30. Clayton, C. A.; Hines, J. W.; Elkins, P. D. Detection Limits with Specified Assurance Probabilities. Anal. Chem. 1987, 59, 2506–2514. 31. Garcı´a, I.; Sarabia, L. A.; Ortiz, M. C. Detection Capability of Tetracyclines Analysed by a Fluorescence Technique: Comparison between Bilinear and Trilinear Partial Least Squares Models. Anal. Chim. Acta 2004, 501, 193–203. 32. Ortiz, M. C.; Sarabia, L. A.; Garcı´a, I.; Gime´nez, D.; Mele´ndez, E. Capability of Detection and Three-Way Data. Anal. Chim. Acta 2006, 559, 124–136. 33. Sarabia, L. A.; Ortiz, M. C. DETARCHI: A Program for Detection Limits with Specified Assurance Probabilities and Characteristic Curves of Detection. Trends Anal. Chem. 1994, 13, 1–6. 34. Verdon, E.; Hurtaud-Pessel, D.; Sanders, P. Evaluation of the Limit of Performance of an Analytical Method Based on a Statistical Calculation of its Critical Concentrations According to ISO Standard 11843: Application to Routine Control of Banned Veterinary Drug Residues in Food According to European Decision. Accred. Qual. Assur. 2006, 11, 58–62. 35. Ortiz, M. C.; Sarabia, L. A.; Herrero, A.; Sa´nchez, M. S.; Sanz, B.; Rueda, M. E.; Gime´nez, D.; Mele´ndez, M. E. Capability of Detection of an Analytical Method Evaluating False Positive and False Negative (ISO 11843) with Partial Least Squares. Chemom. Intell. Lab. Syst. 2003, 69, 21–33. 36. Sanz, M. B.; Sarabia, L. A.; Herrero, A.; Ortiz, M. C. Capability of Discrimination: Application to Soft Calibration Methods. Anal. Chim. Acta 2001, 446, 297–311. 37. Garcı´a, I.; Ortiz, M. C.; Sarabia, L.; Vilches, C.; Gredilla, E. Advances in Methodology for the Validation of Methods According to the International Organization for Standardization. Application to the Determination of Benzoic and Sorbic Acids in Soft Drinks by High-Performance Liquid Chromatography. J. Chromatogr. A 2003, 992, 11–27. 38. Bosch-Reig, F.; Campins Falco´, P. Campins Falco´ P., H-Point Standard Additions Method. Part 1 Fundamentals and Application to Analytical Spectroscopy. Analyst 1988, 113, 1011–1016. 39. Campı´ns-Falco´, P.; Verdu´-Andre´s, J.; Bosch-Reig, F.; Molı´ns-Legua, C. Generalized H-Point Standard Additions Method for Analyte Determinations in Unknown Samples. Anal. Chim. Acta 1995, 302, 323–333. 40. Tortajada-Genaro, L. A.; Campı´ns-Falco´, P.; Blasco-Go´mez, F.; Bosch-Reig, F. The Generalized H-Point Standard-Additions Method to Determine Analytes Present in Two Different Chemical Forms in Unknown Matrix Samples. Part I. General Considerations. Analyst 2000, 125, 771–776. 41. Booksh, K.; Henshaw, J. M.; Burgess, L. W.; Kowalski, B. R. A Second-Order Standard Addition Method with Application to Calibration of a Kinetics-Spectroscopic Sensor for Quantitation of Trichloroethylene. J. Chemom. 1995, 9, 263–282. 42. Saurina, J.; Tauler, R. Strategies for Solving Matrix Effects in the Analysis of Triphenyltin in Sea-Water Samples by Three-Way Multivariate Curve Resolution. Analyst 2000, 125, 2038–2043. 43. Sena, M. M.; Trevisan, M. G.; Poppi, R. J. Combining Standard Addition Method and Second-Order Advantage for Direct Determination of Salicylate in Undiluted Human Plasma by Spectrofluorimetry. Talanta 2006, 68, 1707–1712. 44. Herrero, A.; Zamponi, S.; Marassi, R.; Conti, P.; Ortiz, M. C.; Sarabia, L. A. Determination of the Capability of Detection of a Hyphenated Method: Application to Spectroelectrochemistry. Chemom. Intell. Lab. Syst. 2002, 61, 63–74. 45. Bahram, M.; Bro, R. A Novel Strategy for Solving Matrix Effect in Three-Way Data Using Parallel Profiles with Linear Dependencies. Anal. Chim. Acta 2007, 584, 397–402. 46. Nelder, J. A.; Wedderburn, R. W. M. Generalized Linear Models. J. R. Stat. Soc. Ser. A 1972, 135, 370. 47. Davidian, M.; Haaland, P. D. Regression and Calibration with Nonconstant Error Variance. Chemom. Intell Lab. Syst. 1990, 9, 231–248. 48. Miller, J. N. Basic Statistical Methods for Analytical Chemistry. Part 2. Calibration and Regression Methods. Analyst 1991, 116, 3–14. 49. Tellinghuisen, J. Weighted Least-squares in Calibration: What Difference Does it Make? Analyst 2007, 6, 536–543. 50. Analytical Methods Committee. Uses (Proper and Improper) of Correlation Coefficients. Analyst 1988, 113, 1469–1471. 51. EPA, Document # 815 R 05 006. Statistical Protocol for the Determination of the Single-Laboratory Lowest Concentration Minimum Reporting Level and Validation of Laboratory Performance at 0 Below the Minimum Reporting Level, 2004. 52. Asuero, A. G.; Gonza´lez, G. Fitting Straight Lines with Replicated Observations by Linear Regression. III. Weighting Data. Crit. Rev. Anal. Chem. 2007, 37, 143–172. 53. Sanz, B. Metodologı´a para determinar ı´ndices de calidad de te´cnicas analı´ticas que incorporan calibrados multivariantes. Aplicacio´n al Electroana´lisis. Ph. D. thesis, University of Burgos, 2002. 54. Rousseeuw, P. J.; Leroy, A. M. Robust Regression and Outliers Detection. John Wiley and Sons Inc.: Hoboken, NJ, 2001. 55. Ortiz, M. C.; Sarabia, L.; Herrero, A. Robust Regression Techniques. A Useful Alternative for the Detection of Outlier Data in Chemical Analysis. Talanta 2006, 70, 499–512. 56. Daszykowski, M.; Kaczmarek, K.; Vander Heyden, Y.; Walczak, B. Robust Statistics in Data Analysis – A Review. Basic Concepts. Chemom. Intell. Lab. Syst. 2007, 85, 203–219. 57. Rousseeuw, P. Least Median of Squares Regression. J. Am. Stat. Assoc. 1984, 79, 871–880. 58. Massart, D. L.; Kaufman, L.; Rousseeuw, P. J.; Leroy, A. Least Median of Squares: A Robust Method for Outlier and Model Error Detection in Regression and Calibration. Anal. Chim. Acta 1986, 187, 171–179. 59. Rousseeuw, P. J. Tutorial to Robust Statistics. J. Chemom. 1991, 5, 1–20. 60. Stromberg, A. J. Computing the Exact Least Median of Squares Estimate and Stability Diagnostics in Multiple Linear Regression. SIAM J. Sci. Comput. 1993, 14, 1289–1299. 61. Watson, G. A. Computing the Least Quantile of Squares Estimate. SIAM J. Sci. Comput. 1998, 19, 1125–1138.
Quality of Analytical Measurements: Univariate Regression
167
62. Rousseeuw, P. J.; Hubert, M. In L1-Statistical Procedures and Related Topics; Dodge, Y., Ed.; Institute of Mathematical Statistics: Hayward, CA, 1997; p 201. 63. Pison, G.; Van Aelst, S.; Willems, G. Small Sample Corrections for LTS and MCD. Metrika 2002, 55, 111–123. 64. Rousseeuw, P. J.; Van Driessen, K. Computing LTS Regression for Large Data Sets. Estadı´stica 2002, 54, 163–190. 65. Verboven, S.; Hubert, M. LIBRA: A MATLAB Library for Robust Analysis. Chemom. Intell. Lab. Syst. 2005, 75, 127–136. 66. Sanllorente, S.; Ortiz, M. C.; Arcos, M. J.; Lo´pez-Palacios, J. Application of an Optimization Procedure in Adsorptive Stripping Voltammetry to the Determination of Nickel with DMG. Electroanalysis 1996, 8, 285–292. 67. Ortiz, M. C.; Herrero, A. Regression by Least Median Squares, A Methodological Contribution to Titration Analysis. Chemom. Intell. Lab. Syst. 1995, 27, 231–243. 68. Adcock, R. J. A Problem in Least Squares. Analyst (des Moines, Iowa) 1878, 5, 53–54. 69. Jackson, J. E. A User’s Guide to Principal Components. John Wiley and Sons: New York, 1991. 70. Fuller, W. A. Measurement Error Models. John Wiley and Sons: New York, 1987. 71. Ripley, B. D.; Thompson, M. Regression Techniques for the Detection of Analytical Bias. Analyst 1987, 112, 377–383. 72. Lisy´, J. M.; Cholvadova´, A.; Kutej, J. Multiple Straight-Line Least-Squares Analysis with Uncertainties in All Variables. J. Comput. Chem. 1990, 14, 189–192. 73. Riu, J.; Rius, F. X. Univariate Regression Models with Errors in Both Axes. J. Chemom. 1995, 9, 343–362. 74. Van Huffel, S.; Lemmerling, P.; Eds. Total Least Squares and Errors-In Variables Modeling. Kluwer Academic Publishers: Dordrecht, 2002. 75. Anscombe, F. J. Graphs in Statistical Analysis. Am. Stat. 1973, 27, 17–21.
168
Quality of Analytical Measurements: Univariate Regression
Biographical Sketches
Marı´a Cruz Ortiz received her Ph.D. in chemistry from the University of Valladolid (Spain) in 1988. She is a member of the staff of analytical chemistry in the University of Burgos (Faculty of Sciences) since 1989. She does research and teaching on analytical chemistry and chemometrics. Her research activity has focused on experimental design, optimization, pattern recognition, quality assurance, and validation of analytical methods according to official regulations, multivariate and/or multiway regression models. All of these applied to problems in food chemistry, typification, and so on, which produce about 100 papers. She is at present the head of an active research group, the chemometrics and qualimetrics group of the University of Burgos.
M Sagrario Sa´nchez obtained her Ph.D. in mathematics from the University of Valladolid in 1997. She is working at the University of Burgos since 1991 and as a member of the permanent staff since 2002. The teaching activities are mostly directed to graduate and postgraduate courses in chemistry and science and food technology. She is also a permanent member of the chemometrics and qualimetrics group since its foundation. Her main research activities are developed into the areas of interest of the group that include modeling and analysis of n-way data, modeling of categories, design of experiments, optimization, and so on, by using classical methods as well as computationally intensive methods (such as neural networks or evolutionary algorithms).
Quality of Analytical Measurements: Univariate Regression
Luis A. Sarabia received his Ph.D. in Statistics from the University of Valladolid (Spain) in 1979. Since 1974, he has been teaching Statistics and Mathematics mostly to graduate and postgraduate students of Chemistry. At present, his research is centred on Chemometrics as a member of the Chemometrics and Qualimetrics group of the University of Burgos. His research activities include development of software and implementation of nonparametric and robust statistical methods, genetic algorithms, neural networks, etc. He is also involved in multivariate/multiway regression methods, methodology of the experimental design, quality assurance, and validation. He is the author of about a hundred papers on these matters.
169
This page intentionally left blank
1.06 Resampling and Testing in Regression Models with Environmetrical Applications J. Roca-Pardin˜as, University of Vigo, Vigo, Spain C. Cadarso-Sua´rez and W. Gonza´lez-Manteiga, University of Santiago de Compostela, Santiago de Compostela, Spain ª 2009 Elsevier B.V. All rights reserved.
1.06.1 1.06.2 1.06.3 1.06.3.1 1.06.4 1.06.4.1 1.06.5 1.06.5.1 1.06.6 1.06.6.1 References
Introduction to Bootstrap Bootstrap Resampling Methods for Regression Generalized Additive Models Fitting Generalized Additive Models Constructing CIs in GAMs Application to Air Pollution Data Generalized Additive Model with Interactions Estimation Procedure Bootstrap-Based Methods for Testing Interactions Application to Environmetrical Data
171 173 174 175 176 176 179 180 180 181 184
1.06.1 Introduction to Bootstrap The software revolution that has taken place in recent years has led to the development of different statistical methodologies of data analysis that rely on computer-based calculation. Preeminent among these techniques are the so-called resampling methods, which consist of generating a large number of samples to study the behavior pattern of given statistics. There are various procedures for generating artificial samples on the basis of an initial sample. Possibly the best known of these is the bootstrap method introduced by Efron in 1979,1 a type of procedure that uses simulation to assess statistical accuracy. At present, bootstrap methods can be regarded as a general tool for statistical work, tending to be used in combination with other statistical techniques rather than in isolation. These types of methods are applied to different statistical areas, including the construction of confidence intervals (CIs), testing of hypotheses, and regression or principal components analysis, among others (see, for instance, the monograph by Efron and Tibshirani2). The interest that bootstrap methodology has aroused among the statistics community is reflected in the considerable number of textbooks devoted to justifying its theoretical bases or to discussing its applications in specific areas such as biology, environment, or medicine.2–6 In addition, various packages have been developed by implementing a variety of bootstrap methods applied to different statistical methods, and most of these packages are in languages such as Fortran, R, or S-plus. In many practical situations, the goal of research is to make inferences about a given characteristic (or parameter), , of a variable of interest, X. Estimation of this parameter is obtained on the basis of a statistic, a statistical inference ˆ ¼ ˆðX1 ; . . . ; Xn Þ, calculated as a function of a sample of size n ðX1 ; . . . ; Xn Þ, for example, P about the mean population ¼ is based on the sample mean statistic ˆ ¼ X ¼ ni¼1 Xi =n. A good part of conventional statistics is based on laws that enable the sample distribution of a statistic ˆ to be approximated for sufficiently large sample sizes, e.g., where n is large enough, then, in accordance withpthe ffiffiffi central limit theorem, the distribution of sample mean X approximately follows the distribution N ð; S= nÞ, qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi ffi Pn 2 where S ¼ i¼1 ðXi – X Þ =n is the standard sample deviation. For instance, if one were seeking to construct pffiffiffi
a 95% CI for the population mean, this would be given by X 1:96S= n . 171
172
Resampling and Testing in Regression Models with Environmetrical Applications
Procedures of this type based on an asymptotic approach may display certain limitations in practice, as their performance will depend on the information furnished about the population by the sample. Hence, when not enough data are available, a good approximation of the distribution of ˆ will not be obtained, and so the asymptotic method will not produce good results. At other times, moreover, one has to work with statistics that have no asymptotic laws governing their sample distribution. This is the case, for example, of difference in sample medians, or of sample asymmetry the sampling distribution of which is unknown. In the face of these types of limitations, various alternative procedures that enable the sample distribution of the target statistic to be obtained have emerged, by simulating a large number of random samples directly constructed on the basis of initially observed data. Among these resampling techniques is the bootstrap method, which is described in detail below. As an alternative to the asymptotic method, the simplest version of the bootstrap method approximates the distribution of the statistic ˆ ¼ T ðX 1 ; . . . ; X n Þ, in accordance with the following procedure: 1. Based on sample X ¼X1 ; . . . ; Xn ; a random sample X b ¼ X1 b ; . . . ; Xn b is artificially simulated by resampling with replacement. In other words, after the extraction of an element, this is replaced in the original sample such that it can be chosen again. 2. For each sample obtained, the value of the statistic T b ¼ T ðX1 b ; . . . ; Xn b Þ is calculated. 3. Steps 1 and 2 are repeated a large number (B) of times so as to obtain bootstrap values T 1 ; . . . ; T B . At this stage in its development, computer software allows for the computational cost entailed in the generation of a large number of samples to be estimated. 4. Finally, the distribution of T and its corresponding quantiles is approximated by means of a histogram obtained on the basis of values T 1 ; . . . ; T B , that is, an empirical approach to the sample distribution of statistic T is obtained, without any assumptions having been made as to the theoretical distribution to which the latter conforms. The bootstrap resampling method outlined above is known as naive bootstrap. In step 1, the bootstrap samples are simulated by P means of resampling with replacement, that is, based on the empirical distribution Fˆn ðx Þ ¼ n – 1 ni¼1 I fXi x g of the sample. Other resampling approaches have also been considered: When the form of the population distribution is known, the use of parametric bootstrap allows for a better approximation of the sampling distribution of the test statistic. That is to say, if F is known to belong to a parametric family fF : PYg and ˆ is an estimator of (e.g., the estimator of maximum likelihood), then X can be taken as a random sample from Fˆ. For instance, assuming one knows that the observations came from a normal distribution X PN ð;Þ, then one would draw repeated bootstrap samples from a normal distribution N ðX; S Þ. Where the variable X is assumed to be continuous with the density function, f, the use of smoothed bootstrap might be more appropriate. Instead of resampling directly from the empirical distribution Fˆn ðx Þ, one first smoothes it out. RThis smoothed version is then x used to generate new samples from the smoothed distribution F˜n ðx Þ ¼ – 1 fˆn ðx Þ dx, where fˆn ðx Þ is an estimate of density function f (x). As previously pointed out, rather than being perceived as an isolated method, bootstrap is instead used in combination with other statistical techniques. As a review of all bootstrap applications published in recent years would prove extremely difficult, the main thrust of this chapter will focus on how bootstrap works in a regression context. In particular, the possibilities of bootstrap will be analyzed in the context of generalized additive models (GAMs). These types of models were proposed by Hastie and Tibshirani7 as a unifying family of flexible models covering a wide range of regression models with different types of responses (e.g., Gaussian, binomial, and Poisson). Rather than seeking to provide a comprehensive overview of all bootstrap applications in this regression context, this chapter will confine itself to discussing bootstrap resampling methods for achieving two different goals in a GAM context, namely (1) construction of CIs for covariate effects on response and (2) implementation of tests capable of detecting the statistical significance of interaction among the effects of the different covariates. This chapter is laid out as follows: Section 1.06.2 presents different bootstrap resampling methods for regression; Section 1.06.3 introduces GAMs and briefly discusses the process of estimating these models; and
Resampling and Testing in Regression Models with Environmetrical Applications
173
Section 1.06.4 addresses the construction of bootstrap-based CIs for covariate effects in GAMs. The advantage of bootstrap methodology versus a classical method of constructing asymptotic intervals is illustrated in Section 1.06.4.1, by reference to air pollution time-series studies.8 The effect of a given covariate on response may often vary with the values taken by another covariate, something that in turn leads to the concept of interaction. Section 1.06.5 introduces GAMs that include interactions among covariate effects. Section 1.06.6 poses the problem of test hypotheses for the detection of significant interactions in GAMs. Finally, Section 1.06.6.1 contains an application to real SO2 binary pollution time-series data.9
1.06.2 Bootstrap Resampling Methods for Regression In many fields of research, it is important to establish the relationship between a response variable of interest and one or more explanatory covariates. Regression studies enable mathematical models to
be obtained that ; . . . ; X . This section sets link the conditional mean ðXÞ ¼ E ðY jXÞ of response Y to p covariates X ¼ X 1 p
out different bootstrap resampling procedures for regression data sets X1; Y1 ; . . . ; ðXn ; Yn Þ of (X,Y). These procedures will be used in Section 1.06.4 to construct CIs for covariate effects, and in Section 1.06.6 to test the significance of interaction terms. Let us assume that the relationship between covariate X and response variable Y can be expressed
as Y ¼ ðXÞ þ ", where " is an error variable having zero mean E ð"jXÞ ¼ 0. Bootstrap samples X 1 ; Y1 ; . . . ;
Xn ; Yn are generated by being drawn from some estimated distribution Rˆn of the true distribution of (X,Y). If ˆ Rn is chosen as the empirical distribution of ðX1 ; Y1 Þ; . . . ; ðXn ; Yn Þ, then the bootstrap samples are generated by being drawn with replacement from the original sample. This bootstrap procedure is called pairwise resampling.10,11 Another alternative is the use of smooth bootstrap described above, where the bootstrap samples are drawn from a smoothed distribution estimate of the joint density of X and Y. For a more detailed discussion of this bootstrap, the reader should refer to Cao-Abad and Gonza´lez-Manteiga.12 In bootstrap procedures proposed until now, both X and Y are assumed to be random variables. This sample scheme is known as random design. However, there are applications where the researcher is able to control the values of the covariates X, and Y is the only random variable. The scheme known as fixed design will now be discussed. The first point to be borne in mind here is that the error variables "i are independently
and identically distributed (i.i.d.) with zero mean. In this case, the bootstrap samples X1 ; Y1 ; . . . ; Xn ; Yn can be obtained with Yi ¼ ˜ ðXi Þ þ "ˆi , where ˜ ðXi Þ is a pilot estimate of ðXi Þ and "ˆ1 ; . . . ; "ˆn is a random sample drawn from the empirical distribution of the centered residuals "ˆi ¼ Y i – ˜ ðXi Þ. This bootstrap procedure is called residual resampling. Errors cannot be assumed to be i.i.d. in all cases, for example, this condition is not fulfilled when the response variance depends on covariate X. In a situation of this kind, possibly the most popular resampling method is the so-called wild bootstrap13–15 introduced by Wu.16 It allows for heterogeneous variance in the residuals. In wild bootstrap, each residual "ˆi is drawn from a distribution that seeks to imitate the distribution of "ˆi up to the first three moments, namely, E ½"ˆ i ¼ 0, E ½"ˆ i 2 ¼ "ˆi 2 , and E ½"ˆ i 3 ¼ "ˆi3 . These pffiffiffi
conditions are pffiffiifffi "i is drawn from the two-point distribution, pwith ffiffiffi probability mass at a ¼ 0:5 1 – 5 "ˆi met and b ¼ 0:5 1 þ 5 "ˆi occurring with probabilities q ¼ 0:1 5 þ 5 and 1q, respectively. A fuller review of resampling methods in regression can be found in Schimek.17 The resampling methods proposed thus far have basically been designed for continuous response regression models. Yet this type of resampling does not adapt to other types of distribution, for example, for binary response models, values other than zero or one, Yi , will be obtained, with the result that the model will perform in quantitative but not qualitative terms. Similarly, should there be a Poisson response, there is no guarantee that the values Yi will not be negative. A general way of approaching these types of situations is to consider the exponential family,18 which includes the majority of distributions in statistics, that is, Gaussian, binomial, or Poisson. In the exponential family, the conditional density function f ðY jXÞ is given by y – b ðÞ f ðY jXÞ ¼ exp þ c ðy; Þ a ð Þ
ð1Þ
174
Resampling and Testing in Regression Models with Environmetrical Applications
where denotes the natural parameter, is the scale or dispersion parameter, and a, b, and c are known specific functions in the exponential family. The natural parameter depends on the conditional mean of the response ðX Þ ¼ E ðY jXÞ, and so one can write ¼ ððX ÞÞ. Moreover, the conditional variance ˆ ððXÞÞ, where V is of the response, VarðY jXÞ, depends on ðX Þ, via the relationship VarðY jXÞ ¼ V the so-called variance function, which is known and is determined by the exponential distribution chosen. Bootstrap resampling techniques for obtaining bootstrap samples are based on the conditional distribution of Y|X, determined by ðXÞ and , with the result that the density function f ðY jXÞ given in Equation (1) can therefore be written as f ð ðXÞ; Þ. The bootstrap technique consists of generating a large number (B) of
b b bootstrap replicates X1; Y1 ; . . . ; Xn ; Yn (b ¼ 1, . . . , B), with Yi b f ð˜ ðXi Þ Þ being based on the original sample ðX1 ; Y1 Þ; . . . ; ðXn ; Yn Þ, and ˜ ðXi Þ being the pilot estimates obtained on this basis.19,20 In special situations, the dispersion parameter is known and the conditional distribution of Y is specified by ðXÞ. For instance, for binary response Yi BernoulliððXi ÞÞ, the bootstrap replicates will be generated in line with Yi Bernoullið˜ ðXi ÞÞ. Likewise, for Poisson response Yi PoissonððXi ÞÞ, the bootstrap replicates will be generated in line with Yi Poissonð˜ ðXi ÞÞ. Otherwise, if is unknown, an estimate ˆ for ˆ ððXÞÞVarðY jXÞ ¼ V ˆ ððXÞÞ or equivalently parameter can be obtained, in line with VarðY jXÞ ¼ V
¼ VarðY jXÞ=V ððXÞÞ, from
ˆ ¼
n X ðYi – ˆ ðXi ÞÞ2 i¼1
nV ðˆ ðXi ÞÞ
where ˆ ðXi Þ are the estimates of ðXi Þ obtained on the basis of the original sample fðXi ; Yi Þgni¼1 . For instance, in the case of Gaussian response Yi – N ððXi Þ;Þ the parameter is unknown and coincides with ˆ where ˆ 2 ¼ n 2.P In such a case, the bootstrap responses will be generated in line with Yi – N ððX ˜ i Þ;Þ, n 2 –1 2 ˜ ðX i ÞÞ is the estimation of . It should be noted that in this approach, the variance 2 is a i¼1 ðYi – constant and the regression model is thus assumed to be homoscedastic. Nevertheless, our interest also lies in heteroscedastic regression models, where the variance 2 ¼ 2 ðXÞ depends on the vector of the covariates X. To estimate 2(X), a nonparametric regression model can be fitted to Ri ¼ ðYi – ˜ ðXi ÞÞ2 on X. Another alternative is to use the wild bootstrap method outlined above.
1.06.3 Generalized Additive Models The models that will be studied here can be viewed as a generalization of the well-known generalized 18 mean ðXÞ ¼ E ½Y jX of response Y depends on linear model (GLM). In these
GLMs, the conditional the covariates X ¼ X1 ; . . . ; Xp via X þ þ p Xp , where H is a known monotone ðXÞ ¼ H þ 1 1
link function and ; 1 ; . . . ; p is a vector of coefficients. In some instances, GLMs can be very restrictive, because they assume linearity in the covariates. This constraint can be avoided by replacing the linear index ¼ þ 1 X1 þ þ p Xp with a nonparametric structure. Accordingly, here we will concentrate on the GAM7, which is a generalization of the GLM, by introducing one-dimensional, nonparametric functions in lieu of linear components. Specifically, GAMs express the conditional mean (X) as
ðXÞ ¼ H þ f1 ðX1 Þ þ þ fp ðXp Þ
ð2Þ
where H is a known link function, is a constant, and f1 ; . . . ; fp are unspecified, unknown, zero-mean functions. In the case of an identity function, H, one speaks of an additive model. On assuming that effects are additive, GAMs maintain the interpretability of GLMs.18,21 Yet, at the same time, they incorporate the flexibility of nonparametric smoothing methods because, rather than following a fixed parametric form, the effect of each of the covariates, Xj, depends on a totally unknown function, fj, which is only required to possess a certain degree
Resampling and Testing in Regression Models with Environmetrical Applications
175
of smoothness so that it can be estimated. The adaptability of data belonging to an exponential distribution, together with the GAM compromise between flexibility, dimensionality, and interpretability, ranks these types of models among the statistical tools with the greatest capacity for data analysis in different fields of research. To learn more about these models, the reader should consult Gu,22 Ruppert et al.,23 and Wood.24 Although this chapter was mainly focused on GAMs, alternative statistical methodologies, such as the alternating conditional expectation (ACE) algorithm developed by Breiman and Friedman,25 can also be used in this context. This algorithm estimates the transformations of a response and a set of covariates in multivariate regression that produce the maximum linear effect between the transformed independent variables and the transformed response variable. These types of transformed models can be fitted to data by means of R routine ACE.
1.06.3.1
Fitting Generalized Additive Models
This section describes an algorithm for fitting the effects f1 ; . . . ; fp in the GAM in Equation (2). The algorithm shown below is a combination of local scoring and backfitting techniques.26 In brief, local scoring is analogous to the use of iteratively reweighted least squares18 for solving likelihood and nonlinear regression equations. At each iteration of the estimation algorithm, the estimates fˆj were obtained by applying some smoother to the liberalized transformed responses. Although our focus is on local scoring, there are other types of procedures that allow for nonparametric estimation of GAMs. Sperlich et al.27 presented methods based on marginal integration. Wahba28 and Guo29 proposed the use of smoothing spline analysis of variance (ANOVA) methods. Coull et al.30 and Ruppert et al.23 investigated alternative methods based on penalized splines (P-splines), and Wood31 used thin plate regression splines. Other studies, such as the paper by Brezger and Lang,32 also used P-splines, and developed Bayesian versions of GAMs and extensions to generalized structured additive regression. the local Given a sample ðX1 ; Y1 Þ; . . . ; ðXn ; Yn Þ of (X,Y), the steps ofP
scoring algorithm are
as follows: Initialization. Compute the initial estimates ˆ ¼ H – 1 n – 1 ni¼1 Yi and ˆi 0 ¼ 0, fˆj 0 Xij ¼ 0, for i ¼ 1; . . . ; n. Step 1. Construct the linearized response Y˜i ¼ ˆ 0i þ ðYi – H ðˆ i0 ÞÞ=H 9ðˆ i0 Þ and the weights Wi ¼ 2 H 9ðˆ i0 Þ =V ðˆ i0 Þ, where H 9ðÞ ¼ H = is the first derivative of H with respect to . Step 2. Fit a nonparametric additive regression model to Y˜i on Xi and compute the updates ˆi , using the following backfitting procedure:
Step 2.1. For j ¼ 1; . . . ; p, obtain the updated nonparametric estimates fˆj Xij by using a linear smoother P P given by the following weighted (local) average of the partial residuals Rji ¼ Y˜i k<j fˆk ðXik Þ – k>j fˆ k0ðXik Þ (obtained by removing the estimated effects of the other covariates): fˆj ðx Þ ¼
n X
W i W j Xij Rij
ð3Þ
i¼1
where W j is a weight function depending on a smoothing parameter j .
Step 2.2. Repeat Step 2.1 with fˆ1 0 ðXi1 Þ; . . . ; fˆp 0 Xip replaced by fˆ1 ðXi1 Þ; . . . ; fˆp Xip for i ¼ 1; . . . ; n, until convergence.
Step 3. Repeat Steps 1 and 2 with ˆi0 replaced by ˆi ¼ ˆ þ fˆ1 ðXi1 Þ þ þ fˆp Xip until there is no change in the estimates. In expression (3), different types of estimates will be obtained,
on the
type of smoother used to P depending obtain weights W j . For instance, if weights W j ðx Þ ¼ n – 1 ni¼1 K Xij – x = j are used, with K being any given univariate density function, the Nadaraya–Watson estimator introduced by Nadaraya33 and Watson34 in 1964 is obtained. Other commonly used smoothers in nonparametric regression are local linear kernel smoothers,35 smoothing splines,7 or P-splines.36
176
Resampling and Testing in Regression Models with Environmetrical Applications
At all events, it is well known that estimates depend heavily on the smoothing parameters j used in the estimation. The smoothing parameter j was selected automatically by minimizing the following weighted cross-validation error: CVj ¼
– iÞ
n X
2
Wi fˆ j – iÞ Xij – Ri j
i¼1
where fˆ j Xij indicates the fit at Xij, excluding the ith data point. Nowadays, the existence of standard software makes it easy to fit GAMs in practice. Thus, we may find a variety of freely available packages written in R language (http://cran.r-project.org/) using different estimation methods and smoothers. So far, the most widely used packages are gam,7 Semipar,23 VGAM,37 mgcv,24 and gamlss.38
1.06.4 Constructing CIs in GAMs To construct the CIs for partial effects, fj , in model (2), one has to know the distribution of estimations fˆj – f j , where fˆj is the nonparametric estimation of the true fj . As seen in Section 1.06.3.1, however, iterative algorithms based on nonparametric smoothers are used for estimation of the GAM in Equation (2), which makes obtaining asymptotic results for the distribution of fˆj – f j somewhat complicated. In Gu39,40 and Hastie and Tibshirani,7 smoothing splines are proposed as smoothers to construct the following (1) 100% limits for the CI for fj ðx Þ:
fˆj ðx Þz1 – =2 ˆ j ðx Þ
where ˆ j 2 ðx Þ is some estimate of the variance 2j ðx Þ of fˆj ðx Þ, and zp is the p-percentile of the standard Gaussian distribution. Yet, this procedure for constructing CIs does not always perform well. For instance, these types of approximations have been shown to suffer from an important limitation, in that the standard error j ðx Þ of the estimated effect of any covariate is underestimated and so the ensuing CIs are too narrow in cases where concurvity is present in the data.41 Briefly, concurvity is the nonparametric analog of multicollinearity, in which a partial function of one of the independent variables can be approximated by a linear combination of functions of the remaining variables. One way of eliminating the problem of underestimation of j ðx Þ in the presence of concurvity is to resort to bootstrap resampling techniques. The steps for construction of the bootstrap-based CIs for fj ðx Þ are as follows: Pp Step 1. Estimate the model in Equation (2) so as to obtain pilot estimates ˜ ðXi Þ ¼ H ˜ þ j ¼1 fˆj Xij for i ¼ 1, . . . , n, and then f˜j ðx Þ.
n Step 2. For b ¼ 1; . . . ; B generate the sample Xi ; Yi b i¼1 , where each Yi b is drawn from the estimated density function fˆð˜ ðXi ÞÞ (see Section 1.06.1 for a detailed description of this type of bootstrap resampling) and obtain bootstrap estimates fˆj b ðx Þ. Once this process has been completed, the (1) 100% limits of the CI for the true fj ðx Þ are given by fˆ j ðx Þ – fˆj ð1 – =2Þ ðx Þ; fˆj ðx Þ – fˆj ð=2Þ ðx Þ
where fˆj ðpÞ ðx Þ represents the p-percentile of the B differences fˆj 1 ðx Þ – f˜j ðx Þ; . . . ; fˆj B ðx Þ – f˜j ðx Þ. 1.06.4.1
Application to Air Pollution Data
Time-series studies with Poisson regression have been the reference method for analyzing the short-term effects of air pollution on health. To control for the possible confounding effect of unmeasurable variables that generate trend and seasonality in the series, different parametric functions (linear, quadratic, and sinusoidal of varying width) of the variable ‘time’ were initially used. However, these models could prove very ‘rigid’ and unadaptable in the case of confounding variables of seasonal behavior of varying width and frequency. Application of GAM-based nonparametric smoothing methods (smoothing splines or LOESS) has meant a great advance in that it has eradicated the need for researchers to specify the parametric forms of seasonality a
Resampling and Testing in Regression Models with Environmetrical Applications
177
priori and enabled flexible adaptation to seasonal behavior. Consequently, most studies published in recent years on the short-term effects of pollution on health have applied this methodology.42–44 As has already been stated, however, in cases where concurvity is present in the data, asymptotic CIs prove too narrow.41,45–47 Alternatively, CIs can be obtained using bootstrap resampling techniques. Indeed, a comparative study was undertaken into the coverages obtained with asymptotic and bootstrap CIs,8 using the time series Yt, denoting the number of deaths per day in the city of Vigo (Spain) in 1996, 1997, and 1998 (t ¼ 1; . . . ; 1096). For each day t, black smoke (BS) (shown in Figure 1) BSt and temperature tempt were recorded. This study considered the Poisson model
Yt Poissonðt Þ with logðt Þ ¼ þ BSt þ ftrend ðt Þ þ ftemp tempt
ð4Þ
in which ftrend is a partial function that depends on time (seasonal and trend component) and ftemp represents the functional form of the effect of temperature on mortality. The main objective was to obtain valid CIs for the coefficient , which denoted the log relative rate of Y associated with an increase in BS, based on the sample
1996 t ; tempt ; BSt ; Yt t ¼1 . Data were analyzed using routine gam of S-Plus-based smoothing splines, with the estimated coefficients ˆ ¼ 3:09 and ˆ ¼ 0:001 being obtained. The estimated effects fˆtrend and fˆtemp are plotted in Figure 2. The concurvity measure proposed by Ramsay et al.41 was based on the correlation between the fitted values obtained from the GAM, with pollution, BSt , as the response and with time and temperature as the predictors. Specifically, to assess the degree of concurvity in our data, one should (1) fit the GAM BSt ¼ g1 ðt Þ þ g2 ðtempt Þ þ "t , where the partial functions g1 and g2 are estimated using smoothing splines, ct , and "t is a zero mean error variable, and (2) then compute the squared correlation (R2) between BSt and BS using the fitted values from model in Equation (4), namely
2 ct ¼ gˆ1 ðt Þ þ gˆ2 temp concurvity ¼ correlation BSt ; BS t
In the data, concurvity was 0.29. To vary the concurvity and thereby assess its influence on CI coverage and ˆ in model (4), the original BS was replaced by a weighted average of the bias in the parameter estimate, , ct , and the difference BSt – BS ct . Specifically, the new variable BSst was calculated original BSt , the estimated BS
Daily mean of BS
200
150
100
50
0 0
200
400
600 Day (t )
800
1000
Figure 1 Daily mean black smoke (BS) levels in Vigo for 1096 days, in the period 1996–98. Reproduced from Figueiras, A,; Roca-Pardin˜as, J.; Cadarso-Sua´rez, C. A. Bootstrap Method to Avoid the Effect of Concurvity in Generalized Additive Models in Time-Series Studies of Air Pollution. J. Epidemiol. Community Health 2005, 59, 881–884, with permission from BMJ Publishing Group Ltd.
178
Resampling and Testing in Regression Models with Environmetrical Applications
0.4 2.9
fˆtrend
fˆtemp
0.3
2.4 0.2 1.9 0.1 1.4 0.0
0.9
–0.1
0.4 0
200
400
600 Day (t)
800
1000
0
5
10
15 Temp
20
25
Figure 2 Estimated seasonal and trend component, and estimated smoothed temperature effect. Reproduced from Figueiras, A,; Roca-Pardin˜as, J.; Cadarso-Sua´rez, C. A. Bootstrap Method to Avoid the Effect of Concurvity in Generalized Additive Models in Time-Series Studies of Air Pollution. J. Epidemiol. Community Health 2005, 59, 881–884, with permission from BMJ Publishing Group Ltd.
c t þ ð1 – w1 – w2 Þ"ˆt . Thus, by varying both weights w1 P½0; 1 and w2 P½0; 1, in line with BSst ¼ w1 BSt þ w2 BS concurvity may easily be modified from 0 (w1 ¼ w2 ¼ 0) to 1 (w1 ¼ 0, w2 ¼ 1). In the study in question, a number of scenarios were defined by using different values for w1 and w2, with the result that the degree of concurvity varied from 0.05 to 0.65. In each of the scenarios, 1000 samples
1096
t ; tempt ; BSst ; Yt t ¼1 were considered with Yt generated in accordance with Yt Poisson t , with
log t ¼ þ BSt þ ftred ðt Þ þ ftemp tempt
where ¼ 3:09, ¼ 0:001, mtrend ¼ fˆtrend and mtemp ¼ fˆtemp , and fˆtrend and fˆtemp were the estimated partial functions obtained with the original data. Lastly, the corresponding bootstrap and asymptotic 95% CIs were calculated for ¼ 0:001. CI coverages were calculated as the proportion of samples in which these included the true . In Figure 3, CI coverages are compared for different degrees of concurvity, using asymptotic GAM and GAM bootstrap. In this figure, the vertical line is that of the true concurvity, 0.29, obtained with w1 ¼ 1 and 1.00
Coverage
0.95
0.90 Asymptotic 0.85
Bootstrap
0.80 0.05
0.15
0.25
0.35 Concurvity
0.45
0.55
0.65
Figure 3 Effect of concurvity on 95% CI coverage. Reproduced from Figueiras, A,; Roca-Pardin˜as, J.; Cadarso-Sua´rez, C. A. Bootstrap Method to Avoid the Effect of Concurvity in Generalized Additive Models in Time-Series Studies of Air Pollution. J. Epidemiol. Community Health 2005, 59, 881–884, with permission from BMJ Publishing Group Ltd.
Resampling and Testing in Regression Models with Environmetrical Applications
179
w2 ¼ 0. It will be seen that, with the asymptotic approach, increases in concurvity were accompanied by a progressive decline in coverage, which dropped as low as 80% when concurvity was 0.6. With the bootstrap method, however, coverage remained above 94% throughout, irrespective of the degree of concurvity. Accordingly, the results of our study show that application of bootstrap techniques makes it possible for the CIs of contaminant-related effects to be calculated in time-series studies with optimum coverage, regardless of the degree of concurvity.
1.06.5 Generalized Additive Model with Interactions The results obtained with the pure GAM in Equation (2) are easily interpretable, as the effect of each of the covariates Xj is given by a single partial function fj via the known, monotone link function H. It is known, however, that in certain situations the effect of a given covariate on response may vary with the values taken by one or more of the remaining covariates, something that in turn leads to the concept of interaction. GAMs with interactions take the form p X X
ðXÞ ¼ H þ fj Xj þ fjk Xj ;Xk þ j ¼1
1j
X
!
fjkl Xj ;Xk ;Xl þ
1j
where functions fj correspond to the principal effects, fjk to second-order interactions, fjkl to third-order interactions, and so on successively. It is important to note here that the more one introduces higher-order interactions, the more complicated the interpretation of the model becomes, and moreover that problems in estimation arise due to the curse of dimensionality. Accordingly, we shall confine ourselves to studying GAMs that permit only second-order interactions, that is, GAMs such as ðXÞ ¼ H þ
p X
p X
f j Xj þ fjk Xj ;Xk
j ¼1
k¼j þ1
!! ð5Þ
Clearly, the representation given in Equation (5) is not unique and constraints
must be placed on the main effects f and interaction terms f . A convenient choice is E f X X ;X ¼ 0, E f jX j j jk j k j ¼ 0, and jk
j E fjk Xj ;Xk jXk ¼ 0 for 1 j < k p. Note that these conditions do not represent constraints on our model, because it can be changed to conform to these same identifiability conditions. The only purpose of such constraints is to define clearly which part of a function fjk is the main effect and which part, the interaction. It should be noted that the model in Equation (5) is hierarchical, that is, when the term fjk is introduced into the model, subterms fj and fk are also introduced. The advantage of hierarchical models is that they allow for differentiation of the principal effects associated with the covariates of the pure interaction terms. Not only does this render results more easily interpretable, but any possible interactions between the effect of Xj and that of Xk can also be identified by simply designing a test where the null hypothesis is H0jk : fjk ¼ 0. In recent years, a number of papers that address the problem of estimating and testing nonparametric regression models with interaction terms have appeared. For instance, Hastie and Tibshirani,7 Wood,24 Wahba,28 and Gu22 discussed various approaches in the GAM context using a variety of smoothers. Also, multivariate adaptive regression splines (MARSplines)48 can also be used. This technique makes use of the structure presented in Equation (5), based on the ANOVA decomposition. Briefly, MARSplines take the form of an expansion in product spline bases functions. The MARSplines are suitable for problems with higher number of covariates, where the curse of dimensionality would likely create problems for other techniques. This technique had certain spreading in chemometrics because it does not assume any particular class of relationship between the covariates and the outcome variable of interest. For more details about this technique, see Hastie et al.49 The MARS model can be easily fitted in R using the function contained in mda library.7 However, since nowadays there is no close theory to address interesting inferential issues, such as testing for interactions, the use of bootstrap techniques is recommended to approximate the distribution of the interaction tests.
180
Resampling and Testing in Regression Models with Environmetrical Applications
1.06.5.1
Estimation Procedure
To estimate the GAM with interactions in Equation (5), an adaptation of the local scoring algorithm outlined in Section 1.06.3.1 can be used. In Step 2 of this algorithm, a weighted additive (with interactions) regression model is now fitted to the transformed response Y˜ on X weighted by W, and the updates are computed using backfitting: ˆX i ¼ ˆ þ
p X X
fˆj Xij þ fˆjk Xij ; Xik j ¼1
1j
The backfitting algorithm cycles through the covariates Xj (j ¼ 1, . . . , p) and estimates each fj, by fitting a univariate weighted regression model to the partial residual Ri j on Xij (e.g., by applying local weighted kernel smoothers35) with Ri j ¼ Y˜i – ˆ –
p X
fˆl ðXil Þ –
X
fˆlm ðXil ; Xim Þ
1l<mp
l¼1 l6¼j
on Xij. Similarly, the algorithm cycles through each of the combinations Xj ; Xk , 1 j < k p, and the estimates fˆjk are obtained by fitting a bivariate weighted regression model to Rjki on ðXij ;Xik Þ R ji ¼ Y˜i – ˆ –
p X
fˆl ðXil Þ –
p X l¼1
fˆlm ðXil ; Xim Þ
1l<mp
l¼1 l6¼j
R jki ¼ Y˜i – ˆ –
X
fˆl ðXil Þ –
X
fˆlm ðXil ; Xim Þ
1l<mp ðl;mÞ6¼ðj ;kÞ
using, for instance, bivariate local weighted kernel smoothers.50 In each cycle of the algorithm, the bandwidths hj and ðhjk1 ; hjk2 Þ are selected automatically, by minimizing the 2 2 n
P P weighted cross-validation errors CVj ¼ ni¼1 Wi fˆ j– iÞ Xij – Rji and CVjk ¼ Wi fˆ jk– iÞ Xij – Rjki , respectively, where fˆj – iÞ and fˆjk– iÞ indicate the fits, excluding the ith data point.
i¼1
1.06.6 Bootstrap-Based Methods for Testing Interactions When a ‘pure’ principal effects GAM in Equation (2) (i.e., without interactions) is rejected in favor of a GAM with interactions in Equation (5), the ability to decide statistically which interaction terms are relevant and which should not be included in the model becomes crucial, something that leads one to the field of test hypotheses. Although this section focuses on regression models for binary response, all the procedures outlined below would be easily generalizable to any response belonging to the exponential family. Let Y be the binary (0/ 1) response. If we denote pðXÞ ¼ pðY ¼ 1jXÞ, then the logistic GAM with interactions takes the following form: log
p X X
pðXÞ fj Xj þ fjk Xj ; Xk ¼þ 1pðXÞ 1j
ð6Þ
The null hypothesis H0 : fjk ¼ 0, namely that there is no interaction between Xj and Xk for any given fixed pair (j, k), is considered for the GAM in Equation (6). Applying this test for any of the different pairs
Xj ; Xk 1 j < k p, the correct model can be established, thereby avoiding the introduction of
Resampling and Testing in Regression Models with Environmetrical Applications
181
nonsignificant interactions, which would entail a higher computational cost and increased variance in the estimations. The test for the null hypothesis X
p ðX Þ fj Xj þ ¼þ 1 – p ðX Þ j ¼1 p
H0 : log
X
flm ðXl ; Xm Þ
ð7Þ
1l<mp ðl;mÞ6¼ðj ;kÞ
is discussed below. Let fXi ; Yi gni¼1 be i.i.d. random variables from the model described above. In this testing problem, H0 and H1 are nested hypotheses, and so the likelihood ratio test can be written as T¼
n X
d pˆ 0 ðXi Þ; Y i – d ðXi Þ; Y Þ
ð8Þ
i¼1
where pˆ 0 ðXi Þ are the estimates of the true pðXi Þ under the null hypothesis and deviance d(p,y) is defined as d ðp; y Þ ¼ – 2½y logðpÞ þ ð1 – y Þlogð1 – pÞ. In other words, the test statistic given in Equation (8) represents the increase in deviance of the model without the (j,k) interaction with respect to the deviance of the general model (for more details see Roca-Pardin˜as et al.9). Based on the statistic T, the testing rule for checking fjk ¼ 0 with a significance level is that the null hypothesis is rejected if T is larger than its upper -percentile. It is well known that in this type of test there is no closed asymptotic theory that would enable this percentile to be ascertained, and so resampling methods, such as bootstrap, are used instead. The bootstrap-based testing rule consists of rejecting the null hypothesis if the original T is larger than the 1 percent the statistics T 1 ; . . . ; T B . Each of the bootstrap
of b nB bootstrap 1 b statistics T is obtained with the sample Xi ; Yi i¼1 with Yi drawn from a Bernoulliðp˜ 0 ðXi ÞÞ distribution, where p˜ 0 ðXi Þ is the pilot estimate of p(Xi) obtained under the null hypothesis. Software in R implementing the nonparametric model estimation (explained in Section 1.06.5) and the bootstrap-based tests (explained in Section 1.06.6) can be obtained by contacting the author J. Roca-Pardin˜as at [email protected].
1.06.6.1
Application to Environmetrical Data
Situated in the town of As Pontes, in the northeastern part of A Corun˜a (Corun˜a) Province in Spain, is a miningenergy complex belonging to the electricity utility company, Empresa Nacional de Electricidad, S.A. (ENDESA). The lignite from this mine is characterized by low calorific power and a high sulfur content. Hence, when the coal is fired to generate electricity, oxides of various kinds, including sulfur, are obtained as waste products and released through the power plant’s chimneys. Sometimes, such emissions fall to earth, producing high concentrations of these oxides. Under Spain’s environmental protection laws, average SO2 emission in any given 2-h period, as measured every 5 min at ground level, may not exceed a certain threshold. Any period in which this threshold is exceeded is therefore termed a ‘pollution period’, and the underlying idea is to alert the authorities, 1 h in advance, of the danger of pollution arising if suitable measures are not taken. Taking the present as our starting point, we specifically sought to estimate the probability of the threshold, say r, being exceeded within 1 h. For study purposes, the threshold was set at r ¼ 150 mg m3N1. Figure 4 depicts the passage of the pollution episode that occurred on 19 July 1998 from 08.21 to 19.21, with the temporal instants shown on the abscissa axis. In this figure, the unbroken line represents average 2-hourly concentrations Xt of SO2 and circles represent the response variable of interest Yt ¼ IfXt þ12 >150g (indicator of the occurrence of an episode). Let Xt be the value obtained by the series of 2-hourly averages for SO2 emissions at the present time t. The aim is to predict the probability pðXt þ12 > r Þ ¼ pðYt ¼ 1jXt ; Xt –1 ; . . .Þ. Previous studies51–53 have indicated that the two significant covariates are current pollution, Xt , and the increment on the previous quarter of an hour Xt – Xt – 3 . The nature of this SO2 time series has certain particularities, the most important of which is, perhaps, the fact that most of the time, Xt takes values close to zero. Indeed, long periods of several days, sometimes weeks, are randomly interrupted by a few hours during which values rise to high levels and then fall back to zero. Predictions must be based on data containing a reasonably large number of incidents.
182
Resampling and Testing in Regression Models with Environmetrical Applications
Indicator of episode Two-hour mean concentration Maximum permitted concentration
600
Concentration of SO2
500 400 300 200 100 0 08:21
09:21
10:21
11:21
12:21
13:21
14:21
15:21
16:21
17:21 18:21 19:21 Time (in format hh:mm)
Figure 4 Example of a SO2 pollution incident that occurred on 19 July 1998. The temporal instants are shown on the abscissa axis, with SO2 levels in mg m3 N1 represented by an unbroken line, and the binary series of interest indicating the occurrence of an episode, by circles. Reproduced from Roca-Pardin˜as, J,; Cadarso-Sua´rez, C,; Gonza´lez-Manteiga, W. Testing for Interactions in Generalized Additive Models: Application to SO2 Pollution Data. Stat. Comput. 2005, 15, 289–299, with permission from Springer Science and Business Media.
Roca-Pardin˜as et al.9 therefore constructed a sample fðXl ; Xl – Xl–3 ; Yl Þgt with n ¼ 2000 readings taken before, during, and a short while after pollution periods. To predict the probability pðXt ;Xt – Xt – 3 Þ ¼ pðYt ¼ 1jXt ;Xt – Xt –3 Þ, first we used the logistic GAM pðXt ;Xt –3 Þ ¼
expð þ f1 ðXt Þ þ f2 ðXt – Xt – 3 ÞÞ 1 þ expð þ f1 ðXt Þ þ f2 ðXt – Xt – 3 ÞÞ
ð9Þ
Furthermore, we extended the previous model, by including the interaction term f12 between Xt and Xt – Xt –3 , and so obtained the model pINT ðXt ; Xt –3 Þ ¼
expð þ f1 ðXt Þ þ f2 ðXt – Xt –3 Þ þ f12 ðXt ; Xt – Xt –3 ÞÞ 1 þ expð þ f1 ðXt Þ þ f2 ðXt – Xt –3 Þ þ f12 ðXt ; Xt – Xt –3 ÞÞ
ð10Þ
Figure 5 depicts the contour plot of the episode probability surfaces obtained with models in Equation (9) (left plot) and Equation (10) (right plot), respectively, on the basis of historical matrix entries. The figure shows a possible interaction between the effects of Xi and Xt – Xt –3 . To check this possible interaction statistically, we used test T in Equation (8), explained above, to test the null hypothesis of noninteraction H0 : f12 ¼ 0. The p-value obtained with this test was close to zero, so that the interaction term f12 proved to be statistically significant. Examination of Figure 5 revealed some differences between p and pINT : In the area corresponding to the start of episodes (a stage when immission levels are still low but nevertheless register an increase on the preceding quarter of an hour), the probability of an episode was higher for the model with interaction than for the model without interaction. As will be seen below, this means that the GAM with interactions in Equation (10) detects the commencement of episodes much earlier than does the pure GAM in Equation (9). Another area where differences between the two models are evident is that which correspond to very high, rapidly increasing immission levels. In this case, the probability obtained using the GAM with interactions in Equation (10) is lower than that yielded by the pure GAM in Equation (9). The explanation for this reduction in probability is that in this type of situation, evidence of a strong episode triggers the alarms at the power plant, and control room staff take the necessary measures (power output is lowered and emissions thereby reduced) to prevent SO2 immission levels
Resampling and Testing in Regression Models with Environmetrical Applications
No interaction
With interaction
70
70 1.0 0.8 0.6
–10 0.4
–50
1.0
30 In(t )–In(t –3)
30 In(t )–In(t –3)
183
0.2
0.8 0.6
–10 0.4
–50
0.2
0.0
0
100
200
300 In(t )
400
500
0.0
0
100
200
300 In(t )
400
500
Figure 5 Contour plots corresponding to the estimated probabilities of an episode. Reproduced from Roca-Pardin˜as, J,; Cadarso-Sua´rez, C,; Gonza´lez-Manteiga, W. Testing for Interactions in Generalized Additive Models: Application to SO2 Pollution Data. Stat. Comput. 2005, 15, 289–299, with permission from Springer Science þ Business Media.
from rising still further. It is precisely these prevention measures that lower the probability of an episode by the end of 1 h. Finally, when episodes are on the wane, no clear difference is seen between probabilities p and pINT . Lastly, we tested the performance of the proposed models in terms of their ability to predict the real immission episode that occurred on 19 July 1998. Shown in Figure 6 are the predictions and corresponding 1 Probability of episode
No interaction
(a)
0.8 0.6 0.4 0.2 0 08:24
11:45
13:26 Time
15:07
10:04
11:45
13:26 15:07 Time
16:48
18:14
19:55
18:14
19:55
1 Probability of episode
(b)
With interaction
10:04
0.8 0.6 0.4 0.2 0 08:24
16:48
Figure 6 GAM predictions and 95% confidence bands for the pollution episode (with the prediction shown in the plots for instant t being calculated 1 h prior to pollution, i.e., at instant t 12). Reproduced from Roca-Pardin˜as, J,; Cadarso-Sua´rez, C,; Gonza´lez-Manteiga, W. Testing for Interactions in Generalized Additive Models: Application to SO2 Pollution Data. Stat. Comput. 2005, 15, 289–299, with permission from Springer Science þ Business Media.
184
Resampling and Testing in Regression Models with Environmetrical Applications
95% CIs obtained for the episode by the models in Equations (9) and (10). It is important to note that the prediction indicated in the graph for any given instant t was calculated an hour in advance, that is, at instant t 12. From this figure, it can be seen that inclusion of the interaction enabled the start of the episode to be detected earlier. To evaluate the behavior pattern of the predictions numerically, we used deviance. This was 128 for the pure GAM versus 113 for the GAM with interactions; in other words, inclusion of interactions caused deviance to be reduced by 13% in the prediction of the episode in question.
References 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15. 16. 17. 18. 19. 20. 21. 22. 23. 24. 25. 26. 27. 28. 29. 30. 31. 32. 33. 34. 35. 36. 37. 38. 39. 40. 41. 42.
43.
Efron, B. Bootstrap Methods: Another Look at the Jackknife. Ann. Stat. 1979, 7, 1–26. Efron, E.; Tibshirani, R. J. An Introduction to the Bootstrap; Chapman and Hall: London, 1993. Good, P. Permutation, Parametric, and Bootstrap Tests of Hypotheses; Springer: USA, 2005. Hall, P. The Bootstrap and Edgeworth Expansion; Springer-Verlag: New York, 1992. Davidson, A. C.; Hinkley, D. V. Bootstrap Methods and Their Application; Cambridge: Cambridge University Press, 1997. Manly, B. F. J. Randomization, Bootstrap and Monte Carlo Methods in Biology; Chapman and Hall/CRC: Boca Rato´n, 2006. Hastie, T. J.; Tibshirani, R. J. Generalized Additive Models; Chapman and Hall: London, 1990. Figueiras, A.; Roca-Pardin˜as, J.; Cadarso-Sua´rez, C. A Bootstrap Method to Avoid the Effect of Concurvity in Generalized Additive Models in Time-Series Studies of Air Pollution. J. Epidemiol. Community Health 2005, 59, 881–884. Roca-Pardin˜as, J.; Cadarso-Sua´rez, C.; Gonza´lez-Manteiga, W. Testing for Interactions in Generalized Additive Models: Application to SO2 Pollution Data. Stat. Comput. 2005, 15, 289–299. McDonald, J. A. Projection Pursuit Regression with the ORION I Workstation: A 20-Minute, 16 mm Color Sound Film; Stanford University: Stanford, 1982. Dikta, G. Approximation of Nearest Neighbour Regression Function Estimators. J. Multivar. Anal. 1990, 32, 213–229. Cao-Abad, R.; Gonza´lez-Manteiga, W. Bootstrap Methods in Regression Smoothing. J. Nonparam. Statist. 1993, 2, 379–388. Ha¨rdle, W.; Mammen, E. Testing Parametric versus Nonparametric Regression. Ann. Stat. 1993, 21, 1926–1947. Ha¨rdle, W.; Marron, S. Bootstrap Simultaneous Error Bars for Nonparametric Regression. Ann. Stat. 1991, 19, 778–796. Mammen, E. When Does Bootstrap Work? Lecture Notes in Statistics; Springer-Verlag: New York,1992; Vol. 77. Wu, C. Jackknife, Bootstrap, and Other Resampling Methods in Regression Analysis (with Discussion). Ann. Stat. 1986, 14, 1261–1350. Schimek, M. G. Smoothing and Regression. Approaches, Computation, and Application; John Wiley & Sons: Canada, 2000. McCullagh, P.; Nelder, J. A. Generalized Linear Models; Chapman and Hall: London, 1989. Ha¨rdle, W.; Huet, S.; Mammen, E.; Sperlich, S. Bootstrap Inference in Semiparametric Generalized Additive Models. Econometric Theory 2004, 20, 265–300. Kauermann, G.; Opsomer, J. D. Local Likelihood Estimation in Generalized Additive Models. Scand. J. Statist. 2003, 30, 317–337. Nelder, J. A.; Wedderburn, R. W. M. Generalized Linear Models. J. R. Stat. Soc. 1972, 135, 370–384. Gu, C. Smoothing Spline ANOVA Models; Springer: New York, 2002. Ruppert, D.; Wand, M. P.; Carroll, R. J. Semiparametric Regression; Cambridge University Press: Cambridge, 2003. Wood, S. N. Generalized Additive Models. An Introduction with R; Chapman and Hall: Boca Rato´n, 2006. Breiman, L.; Friedman, J. H. Estimating Optimal Transformations for Multiple Regression and Correlation. J. Am. Stat. Assoc. 1985, 80, 580–678. Opsomer, J. D. Asymptotic Properties of Backfitting Estimators. J. Multivar. Anal. 2000, 73, 166–179. Sperlich, S.; Tjostheim, D.; Yang, L. Nonparametric Estimation and Testing of Interaction in Additive Models. Econometric Theory 2002, 18, 197–251. Wahba, G. Spline Models for Observational Data. Regional Conf. Ser. Appl. Math. 1990, 59. Guo, W. Inference in Smoothing Spline Analysis of Variance. J. R. Statist. Soc. B 2002, 64, 473–491. Coull, B.; Ruppert, D.; Wand, M. Simple Incorporation of Interactions into Additive Models. Biometrics 2001, 57, 539–545. Wood, S. N. Thin Plate Regression Splines. J. R. Statist. Soc. B 2003, 65, 95–114. Brezger, A.; Lang, S. Generalized Structured Additive Regression Based on Bayesian P-Splines. Comput. Statist. Data Anal. 2006, 50, 967–991. Nadaraya, E. A. On Estimating Regression. Theory. Prob. Appl. 1964, 10, 186–190. Watson, G. S. Smooth Regression Analysis. Sankhya Ser. A 1964, 26, 359–372. Wand, M. P.; Jones, M. C. Kernel Smoothing; Chapman and Hall: London, 1995. Eilers, P. H. C.; Marx, B. D. Flexible Smoothing with B-Splines and Penalties. Statist. Sci. 1996, 11, 89–121. Yee, T. W.; Wild, C. J. Vector Generalized Additive Models. JRSS-B 1996, 58, 481–493. Rigby, R. A.; Stasinopoulos, D. M. Generalized Additive Models for Location, Scale, and Shape. JRSS-C 2005, 54, 507–554. Gu, C. Penalized Likelihood: A Bayesian Approach. Stat. Sinica 1992, 2, 255–264. Gu, C. What Happens When Bootstrapping the Smoothing Spline? Commun. Statist. Theory Methods 1987, 16, 3275–3284. Ramsay, T. O.; Burnett, R. T.; Krewski, D. The Effect of Concurvity in Generalized Additive Models Linking Mortality to Ambient Particulate Matter. Epidemiology 2003, 14, 18–23. Saez, M.; Ballester, F.; Barcelo, M. A.; Pe´rez-Hoyos, S.; Bellido, J.; Teˆnias, J. M.; Ocan˜a, R.; Figueiras, A.; Arribas, F.; Aragoneˆs, N.; Tobias, A.; Cirera, L. A Combined Analysis of the Short-Term Effects of Photochemical Air Pollutants on Mortality within the EMECAM Project. Environ. Health Perspect. 2002, 110, 221–228. Le Tertre, A.; Medina, S.; Samoli, E.; Forsberg, B.; Michelozzi, P.; Boumghar, A.; Vonk, J. M.; Bellini, A.; Atkinson, R.; Ayres, J. G.; Sunyer, J.; Schwartz, J.; Katsouyanni, K. Short-Term Effects of Particulate Air Pollution on Cardiovascular Diseases in Eight European Cities. J. Epidemiol. Community. Health 2002, 56, 773–779.
Resampling and Testing in Regression Models with Environmetrical Applications
185
44. Samet, J. M.; Dominici, F.; Curriero, F. C.; Coursac, I.; Zeger, S. L. Fine Particulate Air Pollution and Mortality in 20 U.S. Cities, 1987–1994. N. Engl. J. Med. 2000, 343, 1742–1749. 45. Dominici, F.; McDermott, A.; Zeger, S. L.; Samet, J. M. On Generalized Additive Models in Time-Series Studies of Air Pollution and Health. Am. J. Epidemiol. 2002, 156, 193–203. 46. Samet, J. M.; Dominici, F.; McDermott, A.; Zeger, S. L. New Problems for an Old Design: Time-Series Analyses of Air Pollution and Health. Epidemiology 2003, 14, 11–12. 47. Lumley, T.; Sheppard, L. Time-Series Analyses of Air Pollution and Health: Straining at Gnats and Swallowing Camels? Epidemiology 2003, 14, 13–14. 48. Friedman, J. Multivariate Adaptive Regression Splines. Ann. Stat. 1991, 19, 1–141. 49. Hastie, T. J.; Tibshirani, R. J.; Friedman, J. The Elements of Statistical Learning; Springer-Verlag: New York, 2001. 50. Ruppert, D.; Wand, M. P. Multivariate Locally Weighted Least Squares Regression. Ann. Stat. 1994, 22, 1346–1370. 51. Garcı´a-Jurado, I.; Gonza´lez-Manteiga, W.; Prada-Sa´nchez, J. M.; Febrero-Bande, M.; Cao, R. Predicting Using Box-Jenkins, Nonparametric, and Bootstrap Techniques. Technometrics 1995, 37, 303–310. 52. Prada-Sa´nchez, J. M.; Febrero-Bande, M. Parametric, Nonparametric, and Mixed Approaches to Prediction of Sparsely Distributed Pollution Incidents: A Case Study. J. Chemom. 1997, 11, 13–32. 53. Prada-Sa´nchez, J. M.; Febrero-Bande, M.; Cotos-Ya´n˜ez, T.; Gonza´lez-Manteiga, W. Prediction of SO2 Pollution Incidents near a Power Station Using Partially Linear Models and an Historical Matrix of Predictor-Response Vectors. Environmetrics 2000, 11, 209–225.
186
Resampling and Testing in Regression Models with Environmetrical Applications
Biographical Sketches
Javier Roca Pardin˜as received B.S. and Ph.D. degrees in mathematics from the University of Santiago de Compostela, Santiago de Compostela, in 1996 and 2003, respectively. He is presently a Professor of Statistics at the University of Vigo, Vigo, Spain. His research interests include the nonparametric estimation of nonparametric curves, bootstrap inference, computational aspects, and applications to different research fields such as environmental sciences, biomedicine, epidemiology, and radiology.
Carmen Cadarso Sua´rez received B.S. and Ph.D. degrees in mathematics from the University of Santiago de Compostela, Santiago de Compostela, Spain, in 1982 and 1990, respectively. She is presently Full Professor of Biostatistics at the University of Santiago de Compostela. Her research interests include generalized additive modeling, bootstrap methods, and survival analysis, with applications in clinics, epidemiology, radiology, and physiology.
Resampling and Testing in Regression Models with Environmetrical Applications
Wenceslao Gonza´lez Manteiga is Full Professor and the Chair of the Department of Statistics and Operations Research at the University of Santiago de Compostela, Spain. He received his Ph.D. degree in Mathematics at this University. His main interests are in curve estimation, time series, bootstrap, spatial and spatio temporal data analysis, and survival analysis. He was Editor-in-Chief of TEST for 6 years, international journal of the Spanish Statistical Society, and he is associate editor in several international journals. Over the years he has collaborated with researchers in many fields visiting several universities, with work appearing, for example, in Technometrics, Chemometrics, Journal of the American Statistical Association, Annals of Statistics, Computational Statistics and Data Analysis. He has also collaborated in several projects of applied statistics in environmental data, in finances, etc.
187
This page intentionally left blank
1.07
Robust and Nonparametric Statistical Methods
I. Gijbels and M. Hubert, Katholieke Universiteit Leuven, Leuven, Belgium ª 2009 Elsevier B.V. All rights reserved.
1.07.1 1.07.2 1.07.2.1 1.07.2.2 1.07.2.3 1.07.3 1.07.3.1 1.07.3.2 1.07.4 1.07.4.1 1.07.4.2 1.07.4.3 1.07.5 1.07.5.1 1.07.5.2 1.07.6 References
Introduction Location and Scale Estimation The Mean and Standard Deviation Robust Estimators of Location and Scale Skewness and the Adjusted Boxplot Correlation and Covariance Parametric and Nonparametric Correlation Robust Correlation and Covariance Regression Robust Linear Regression Nonparametric Regression Robust Locally Weighted Regression Investigation Dependence Structures Association Measures and Copulas Nonparametric Estimation of a Copula Function Bibliographic Notes
Symbols C d F Fn f E G H hLTS hMCD h ¼ hn IA K KB KD Kd Kh KT m(x) ˆ LL m n r
copula function portion of data used in LOWESS method cumulative distribution function (cdf) of X empirical cdf density function expectation cumulative distribution function of Y joint cdf of (X, Y) number of samples used in LTS number of samples used in MCD bandwidth parameter in kernel estimation indicator function on the set A kernel function biweight kernel function a smooth cdf function (used as kernel) a density function (used as kernel) rescaled kernel function tricube kernel function nonparametric regression function local linear estimator of m number of observations Pearson correlation coefficient
ri rMCD rrob S s SXY
tK tS xi x(i) x¯ yi B 0 1 " q m
190 190 190 192 194 195 195 196 197 197 200 202 203 204 205 208 209
residual of object i in simple regression model robust correlation measure based on the MCD robust correlation measure sample covariance matrix of bivariate data sample standard deviation sample covariance between X and Y Kendall’s tau association measure Spearman’s rank correlation coefficient measurement of variable X for object i ith smallest observation among the xi sample mean measurement of variable Y for object i Blomqvist’s median correlation coefficient intercept in simple regression slope in simple regression error in simple regression regression parameters in simple regression model population center of bivariate data
189
190
Robust and Nonparametric Statistical Methods
S S
XY
K
population center loss function for M estimator population version of tS population covariance matrix of bivariate data
population scale population covariance between X and Y population version of tK standard normal cdf score function for M estimator
1.07.1 Introduction In statistical analysis, it often occurs that some observations possibly deviate compared to the global behavior of all other observations. Classical statistics, such as the empirical mean and the variance, are highly influenced by such aberrant observations. Robust statistical analysis deals with such outlying observations by reducing the influence that a few observations can have on the whole analysis. Parametric statistical analysis relies on a parametric assumption on, for example, the underlying distribution function, the mean influence that a covariate has on the response variable, and so on. Nonparametric statistical analysis does not make such restrictions and hence is less sensitive to departures from it. Often there is no justification for a specific parametric model to be considered, and then nonparametric modeling is a way to hopefully find such an appropriate parametric model. In addition, nonparametric models can suffer from a high sensitivity to outliers, and therefore robustifications of the developed methods are sometimes needed. This chapter is organized as follows. In Section 1.07.2 we review location and scale estimation starting from the usual mean and standard deviation and highlighting the need for robust location and scale measures. The section continues with a brief discussion on how to construct boxplots for skewed distributions. In Section 1.07.3 the emphasis is on appropriate (robust) measures of correlation and covariance. Sections 1.07.4 and 1.07.5 are devoted to the study of dependencies among variables. Regression models, including parametric as well as nonparametric models, are discussed in Section 1.07.4. In Section 1.07.5, dependencies are approached via association measures and the study of copulas; this section also reviews some nonparametric association measures and discusses recent developments in nonparametric estimation of copulas. Some further bibliographic notes and information on software availability are provided in Section 1.07.6.
1.07.2 Location and Scale Estimation 1.07.2.1
The Mean and Standard Deviation
The location-scale model states that the n univariate observations xi are independent and identically distributed (i.i.d.) with distribution function F. In the parametric setting, we assume that F ðxÞ ¼ F; ðxÞ ¼ F0 ððx – Þ=Þ for some known distribution F0. Typically, F0 is the standard Gaussian distribution function . We then want to find estimates for the center and the scale parameter (or for 2). The classical estimates of location and scale are the sample mean x ¼
n 1X xi n i¼1
and the standard deviation s¼
rffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi 1 ðxi – xÞ2 n–1
However, the mean and the standard deviation are very sensitive to aberrant values. Replacing only 1 out of n observations by a very large value can change the estimates completely. We say that the breakdown value of the sample mean and the standard deviation is 1/n, so it is 0% for large n. More precisely, the breakdown value is the smallest proportion of observations in the data set that need to be replaced to carry the estimate arbitrarily
Robust and Nonparametric Statistical Methods
191
far away. The robustness of an estimator can also be quantified by its influence function,1 which measures the effect of a small number of outliers. The influence function of the mean and the standard deviation is unbounded, which reflects its nonrobust behavior. To illustrate the behavior of the mean and the standard deviation, we consider the toxicity data set, analyzed in Maguna et al.2 and available in the robustbase package of R (http://www.r-project.com). The aim of the experiment was to predict the aquatic toxicity of carboxylic acids on the basis of several molecular descriptors. The data set contains 38 observations and 7 molecular descriptors. We consider the variable ELUMO (energy of the lowest unoccupied molecular orbital) listed in Table 1. Already from the raw data we see that most observations have an energy value between 4 and 5. Observation 20 has a somewhat smaller value (3.76), whereas cases 23, 32, 34, 35, and 36 have
Table 1 Variable ELUMO of the toxicity data set, and absolute standardized residuals obtained with different estimators of location and scale. Outlying samples with absolute standardized residual larger than 2.5 are italicized Absolute standardized residual based on Observation
ELUMO
Mean S.D.
Median MAD
Huber MAD
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38
4.81 4.68 4.86 4.83 4.80 4.90 4.82 4.62 4.74 4.88 4.97 4.83 4.86 4.86 4.86 4.90 4.87 4.35 4.35 3.76 4.44 4.44 2.77 4.40 4.44 4.42 4.72 4.74 4.72 4.54 4.62 1.86 4.72 2.67 1.86 2.39 4.39 4.39
0.549 0.396 0.608 0.573 0.537 0.655 0.561 0.325 0.467 0.632 0.738 0.573 0.608 0.608 0.608 0.665 0.620 0.007 0.007 0.689 0.113 0.113 1.856 0.066 0.113 0.089 0.443 0.467 0.443 0.231 0.325 2.928 0.443 1.974 2.928 2.304 0.054 0.054
0.424 0.077 0.617 0.501 0.385 0.771 0.463 0.308 0.154 0.694 1.041 0.501 0.617 0.617 0.617 0.771 0.655 1.349 1.349 3.623 1.002 1.002 7.439 1.156 1.002 1.079 0.077 0.154 0.077 0.617 0.308 10.946 0.077 7.824 10.946 8.903 1.194 1.194
0.771 0.270 0.964 0.848 0.732 1.118 0.809 0.039 0.501 1.041 1.388 0.848 0.964 0.964 0.964 1.118 1.002 1.002 1.002 3.276 0.655 0.655 7.092 0.809 0.655 0.732 0.424 0.501 0.424 0.270 0.039 10.599 0.424 7.477 10.599 8.556 0.848 0.848
192
Robust and Nonparametric Statistical Methods
ELUMO smaller than 3. The mean of all acids x ¼ 4.34 and the standard deviation s ¼ 0.85. However, if we only consider the acids with ELUMO larger than 4, the mean is 4.68 and the standard deviation is 0.20. We thus notice that the outlying acids have decreased the mean and increased the standard deviation considerably. Because of the sensitivity of the mean and the standard deviation to outliers, it is not recommended to use these estimators to detect outliers. A common rule pinpoints outliers as those observations whose absolute standardized residual jxi – xj=s exceeds, for example, 2.5 or 3 (the typical cutoff in quality control charts). If we apply this outlier detection rule on our data, we see from Table 1 that none of the observations exceeds the three-sigma cutoff, whereas only cases 32 and 35 (with maximal ELUMO) have a standardized residual larger than 2.5. This is an example of the masking effect: by using nonrobust estimators some or all of the outliers are not detected. Sometimes, swamping also occurs, which means that regular observations are incorrectly flagged as outliers. Outlier tests based on the sample mean and standard deviation are very common in statistical quality control and interlaboratory testing (see, e.g. ISO 57253). Some popular tests such as Dixon’s Q-test, Grubbs’ test, and Cochran’s test use the most extreme values, or the sample mean and standard deviation. This makes them very sensitive to outliers, and hence inappropriate for adequate outlier detection; see, for example, Davies4 and Pestiaux and Pic.5 1.07.2.2
Robust Estimators of Location and Scale
To obtain a correct classification of outliers and nonoutliers, robust estimators of and should be used. The philosophy of robust statistics is to accommodate the outliers by finding the location and center of the reliable observations; see, for example, the Analytical Methods Committee6 for more discussion and examples. The most popular robust location estimator is the median. Denote the ith-ordered observation as x(i). Then the median is defined as xððnþ1Þ=2Þ if n is odd and ðxðn=2Þ þ xððnþ1Þ=2Þ Þ=2 if n is even. Replacing only one observation by an arbitrary value will not change the median much. Its breakdown point is about 50%, meaning that the median can resist up to 50% of outliers, and its influence function is bounded. A robust measure of scale is the median absolute deviation (MAD) given by the median of all absolute distances from the sample median: MAD ¼ 1:483 median jxj – medianðxi Þj j ¼1;...; n
i¼1;...; n
ð1Þ
The constant 1.483 is a correction factor that makes the MAD unbiased at the normal distribution. Another alternative is the Qn estimator,7 defined as Qn ¼ 2:2219 cn fjxi – xj j; i < j gðkÞ h
!
n
!
=4 and h ¼
ð2Þ
n
þ 1. Here [z] denotes the largest integer smaller than or equal to z. Therefore, ! n ¼ nðn2– 1Þ possible distances xi – xj . This scale estimator is we compute the kth-order statistic out of the with k ¼
2
2
2
2
thus essentially the 1st quartile of all pairwise differences between two data points. The constant cn is a smallsample correction factor, which makes Qn an unbiased estimator. The breakdown value of both the MAD and the Qn estimator is 50%. Also popular is the interquartile range (IQR) defined as the difference between the 3rd and 1st quartiles, so roughly IQR ¼ x(3n/4) x(n/4). Its breakdown value is only 25% but it has an easy interpretation and it is commonly used to construct the boxplot (Section 1.07.2.3). For the ELUMO data, the median is 4.7, the MAD ¼ 0.26, Qn ¼ 0.24, and IQR ¼ 0.44. If we now compute the absolute standardized residuals as jxi – medianj ¼1;...; n ðxj Þj MAD
ð3Þ
Robust and Nonparametric Statistical Methods
193
we obtain a much more reliable outlier detection tool, as can be seen in Table 1. The most remote observations now have a very large standardized residual, whereas acid 20 also clearly exceeds the cutoff value 3. The robustness of the median comes at a cost: at the normal model it is less efficient than the mean. To find a better balance between robustness and efficiency, many other robust procedures have been proposed, such as trimmed means and M-estimators.8 The (k/n)-trimmed mean is the average of the data by removing the k smallest and the k largest observations. Its breakdown value is k/n and thus can be chosen as any value between 0 and 50% by an appropriate choice of k. A disadvantage of the trimmed mean is that it rejects a fixed percentage of observations on each side of the distribution. This might be too large if they are not all outlying, but even worse, this might be too small if many outliers are present. More adaptive procedures can be obtained using M-estimators. They are defined implicitly as the solution of the equation n X xi – ˆ ¼0 ˆ i¼1
ð4Þ
with being an odd, continuous, and monotone function. The denominator ˆ is an initial robust scale estimate such as the MAD. A solution to Equation (4) can be found by the Newton–Raphson algorithm, starting from the initial location estimate ˆ ð0Þ ¼ mediani ðx i Þ. From each ˆ ðk – 1Þ , the next ˆ ðkÞ is then computed by ˆ
ðkÞ
ðk – 1Þ
¼ ˆ
P ˆ ððxi – ˆ ðk – 1Þ Þ=Þ P þ ˆ i ðk – 1Þ Þ=Þ ˆ ˆ 9ððx – i i
with 9 being the derivative of . Often, a single iteration step is sufficient, which yields the one-step Mestimator. Both the fully iterated and the one-step M-estimator have a 50% breakdown value if the function is bounded. The influence function of M-estimators is (up to a constant factor) exactly equal to their function, and hence it is recommended to use bounded functions. Huber8 proposed the function (z) ¼ min{b, max{b, z}} where typically b ¼ 1.5. Residual values of xi – ˆ outside the interval – 1:5ˆ to 1:5ˆ thus have a bounded influence of 1.5 or 1.5 on the location estimate. Note that if we let b go to infinity, the Huber function converges to the identity (z) ¼ z which is unbounded. From Equation (4) it follows that the corresponding estimator is the sample average. For the toxicity data, the one-step Huber estimator with b ¼ 1.5 yields ˆ ¼ 4:62, whereas the fully iterated one is ˆ ¼ 4:61. The latter estimator yields the standardized residuals in the last column of Table 1. We see again that this procedure correctly flags the outliers. In addition, M-estimators of scale are defined as the solution of an explicit equation. They satisfy n x 1X i ¼ n i¼1 ˆ
ð5Þ
R with being an even, bounded function that satisfies (0) ¼ 0, (x) monotone for x 0 and ¼ (u) d(u). Typically choices for are Huber’s function (z) ¼ min{z2, b2} and the bisquare function (z) ¼ 1 (1 z2)3. To solve the equation iteratively, usually the MAD is taken as the initial scale estimate. Also simultaneous M-estimators of location and scale can be considered, but they have a smaller breakdown value, even in small samples.9 Note that all the mentioned estimators are location and scale equivariant. That is, if we replace our data set X ¼ {x1,. . ., xn} by aX þ b ¼ {ax1 þ b,. . ., axn þ b}, then the location estimator ˆ and the scale estimator ˆ satisfy ˆ ˆ Þþb ðaX þ bÞ ¼ aðX ˆ ˆ Þ ðaX þ bÞ ¼ jajðX
194
Robust and Nonparametric Statistical Methods
Robust measures of location and scale are for example, used in analytical chemistry to robustify one- and two-way analysis of variances (ANOVA),10–14 interlaboratory testing,4,15–18 and quality control.19,20 1.07.2.3
Skewness and the Adjusted Boxplot
Tukey’s boxplot is a graphical tool to visualize the distribution of a univariate data set and to pinpoint possible outliers. In this plot, a box is drawn from the 1st quartile Q1 x(n/4) to the 3rd quartile Q3 x(3n/4) of the data. Points outside the interval [Q1 1.5 IQR, Q3 þ 1.5 IQR], called the fence, are traditionally marked as outliers. If the data are sampled from the normal distribution, only about 0.7% of them will lie outside the fence. The boxplot clearly assumes symmetry, as we add the same amount to Q3 as what we subtract from Q1. At asymmetric distributions, this approach may flag much more than 0.7% of the data. For example, consider the boxplot in Figure 1 of the calcium (Ca) content in 428 soil samples, collected in different communities of the Condroz region in Belgium.21 From the analysis by these authors, it appeared that the distribution of the data was skewed, with six clear upper outliers (with very large Ca value) and three lower outliers (with very small Ca value). These outliers appeared to be measurements from communities at the boundary of the Condroz region, and hence, can be considered to be sampled from another distribution. We see from the standard boxplot in Figure 1 that a substantial number of observations are exceeding the upper whisker, leading to a black box in which the ‘outlying’ observations can no longer be recognized. Moreover, none of the cases is indicated as a lower outlier. The adjusted boxplot,22 also shown in Figure 1, has a longer upper whisker and a shorter lower whisker. The skewness of the underlying distribution is more pronounced and the shorter left tail is better reflected. Now, less observations exceed the upper cutoff, whereas the three smallest cases are marked lower outliers. Note that only two marks are visible, as two of the three smallest observations almost coincide. At right-skewed distributions, the skewness-adjusted boxplot labels observations as outliers if they lie outside the fence ½Q1 – 1:5e – 4MC IQR; Q3 þ 1:5e 3MC IQR:
ð6Þ 23
Here, MC stands for the medcouple, which is a robust measure of skewness. It is defined as MC ¼ median i;j
ðxj – medianÞ – ðmedian – xi Þ xj – xi
ð7Þ
where i and j have to satisfy xi median xj and xi < xj . From its definition it follows that the MC is a number between 1 and 1. If the data are symmetrically distributed, the MC is zero and the adjusted boxplot thus reduces to the standard boxplot. A positive (negative) MC corresponds to a right-skewed (left-skewed) distribution.
4000
Values
3000
2000
1000
0 Standard boxplot
Figure 1 Standard and adjusted boxplot of the calcium data.
Adjusted boxplot
Robust and Nonparametric Statistical Methods
195
1.07.3 Correlation and Covariance 1.07.3.1
Parametric and Nonparametric Correlation
When two variables X and Y are measured, a natural question of interest is whether these two variables are related to each other. The Pearson correlation coefficient, r¼
sXY sX sY
ð8Þ
P 2 with sXY ¼ n –1 1 ni¼1 ðxi – xÞðyi – yÞ being the sample covariance between X and Y and sX, sY the sample standard deviation of X and Y, measures the linear dependency between the two variables. It satisfies 1 r 1, with jr j ¼ 1 for variables that are perfectly linearly related. As this correlation coefficient is based on sample means and variances, it is again highly sensitive to outlying values. Consider the toxicity data set with X being the ELUMO variable and Y being the aquatic toxicity (measured as log(IGC501)). A scatter plot of the data is shown in Figure 2. The Pearson correlation coefficient for this data set equals r ¼ 0.002, which suggests that there is no linear dependency between ELUMO and aquatic toxicity. However, the estimated correlation could be corrupted by the outlying values for ELUMO. To reduce the effect of the outliers, one can rely on nonparametric measures of association or correlation. We briefly discuss two of such association measures. For n observations (x1, y1), (x2, y2),. . ., (xn, yn) from a vector (X, Y) of continuous random variables, there are ! n 2
distinct pairs (xi, yi) and (xj, yj) in the sample of observations. Each such pair (xi, yi) and (xj, yj) is either
concordant or discordant. A pair is called concordant if xi < xj and yi < yj or xi > xj and yi > yj . A pair is called discordant if xi < xj and yi > yj or xi > xj and yi < yj . In other words, a pair is concordant if large (small) values on X tend to be associated with large (small) values on Y. For discordance this is the opposite. Obviously, a pair (xi, yi) and! (xj, yj) is concordant if (xi xj)(yi yj) > 0 and is discordant when (xi xj)(yi yj) < 0. Among the
n 2
distinct pairs, denote by c the number of concordant pairs, and by d the number of
discordant pairs. Kendall’s tau association measure is then defined as tK ¼ ðc – d Þ=
n
! ð9Þ
2
1 0.8 0.6 0.4 Toxicity
0.2 0 −0.2 −0.4 −0.6 −0.8 −1 1.5
2
2.5
Figure 2 Scatter plot of the aquatic toxicity vs. ELUMO.
3 3.5 ELUMO
4
4.5
5
196
Robust and Nonparametric Statistical Methods
which is just the difference of the proportion of pairs in concordance and the proportion of pairs in discordance. Spearman’s rho is another measure of association. Its sample version is simply the Pearson correlation coefficient of the ranks of the ordered observations. Order separately the observations x1, x2, . . ., xn to get xð1Þ < xð2Þ < < xðnÞ , and denote by ui the rank (position among {1, 2, . . ., n}) that the observation xi takes in the ordered sequence. Similarly, denote by vj, the rank that the observation yj gets into the ordered sequence yð1Þ < yð2Þ < < yðnÞ . These ranks ui and vj are in fact observations from uniform U and V random variables. The sample version of Spearman’s rho is then simply the correlation coefficient between the uis and the vj s, that is, tS ¼
sUV sU sV
ð10Þ
P 2 with sUV ¼ n –1 1 ni¼1 ðui – uÞ vj – v being the sample covariance between the ranks ui and the ranks vj , and with sU and sV being the respective sample standard deviations. For obvious reasons, Spearman’s rho is also called Spearman’s ranks correlation coefficient. For the toxicity data set, with X the ELUMO variable and Y the aquatic toxicity, Spearman’s correlation rank coefficient is tS ¼ 0.313 and Kendall’s tau tK ¼ 0.259. 1.07.3.2
Robust Correlation and Covariance
Despite the advantages of the nonparametric correlation measures, they do not necessarily measure linear dependency. Hence, if one is mainly interested in linear relations, robust measures of correlation are more appropriate. According to Equation (8) they can be constructed from robust estimators of variance and covariance. For bivariate data, let Sˆ ¼
ˆ X 2
ˆ XY
ˆ XY
ˆ Y 2
! ð11Þ
be a robust covariance matrix estimate, then a robust correlation measure is obtained as rrob ¼
ˆ XY ˆ X ˆ Y
One well-known robust estimator of covariance is the minimum covariance determinant (MCD) estimator,24 which is described in detail in Chapter 3.07 of this book. The MCD looks for those hMCD observations in the data set (where the number hMCD > n/2 is given by the user) whose classical covariance matrix has the lowest possible determinant. The MCD estimate of location ˆ 0 is then the average of those hMCD points, whereas the MCD estimate of scatter Sˆ 0 is their covariance matrix multiplied with a consistency factor. The breakdown value of the MCD estimator is (n hMCD þ 1)/n, hence the smaller the hMCD, the more resistant MCD is toward outliers. Based on the raw MCD estimates, a reweighing step can also be added to increase the finitesample efficiency. For every bivariate point z ¼ (x, y), define its robust distance RDðz; ˆ 0 ; Sˆ 0 Þ ¼
qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi –1 ðz – mˆ 0 ÞT Sˆ 0 ðz – mˆ 0 Þ
ð12Þ
We can then, for example, give each data point a weight wi ¼ 1 if its squared robust distance is smaller than 22,0.975 (the 0.975 quantile of the 22 distribution) and wi ¼ 0 otherwise. The final MCD estimates ˆ MCD and Sˆ MCD are now given by the sample mean and covariance matrix of the observations with weight 1. In our example, with X ¼ ELUMO and Y ¼ aquatic toxicity, we have the robust correlation based on the MCD estimator rMCD ¼ 0.54, which is highly different from the Pearson coefficient. This has been illustrated in Figure 3. On the scatter plot of the data, we have added the classical tolerance ellipse based on the sample mean and covariance matrix, that is the set of two-dimensional points z whose Mahalanobis distance (MD) MDðzÞ ¼
qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi ðz – zÞT S z– 1 ðz – zÞ
ð13Þ
Robust and Nonparametric Statistical Methods
197
Tolerance ellipse (97.5%) 1 13
36 37
0.5 32 34
Toxicity
0 35 −0.5
23 −1
20 Robust Classical 2
2.5
3
3.5
4 4.5 ELUMO
5
5.5
6
6.5
Figure 3 Robust and classical tolerance ellipse of ELUMO and aquatic toxicity.
equals
qffiffiffiffiffiffiffiffiffiffiffiffiffi 22;0:975 . Here z ¼ ( x , y) is the sample mean and SZ ¼
sX2
sXY
sXY
sY2
! ð14Þ
is the sample covariance matrix. We see that this tolerance ellipse tries to encompass all observations. In contrast, the robust tolerance ellipse, which is based on the robust distances RDðz; ˆ MCD ; Sˆ MCD Þ, is much smaller and encloses the regular data points. Here, we have used hMCD ¼ [0.75n] but the results were similar for smaller hMCD. The computation of the raw MCD is complex, as it requires the evaluation of the covariance determinant of all possible subsets of size hMCD out of n. To avoid a full study of all subsets, an approximate fast algorithm (FAST-MCD) has been proposed.25 For an overview of other robust covariance estimators, refer, for example, to Chapter 3.07 and Rousseeuw et al.26
1.07.4 Regression In regression we not only try to quantify the dependency between X and Y, but also try to model their relation and to predict future Y values from X. Linear regression states that Y ¼ 0 þ 1 X þ
ð15Þ
where the errors are assumed to be i.i.d. with zero mean and constant variance . In nonparametric regression, we assume that Y ¼ m(X) þ for some unknown function m. 2
1.07.4.1
Robust Linear Regression
Following Equation (15) the linear simple regression model assumes that the n data points (xi, yi) satisfy yi ¼ 0 þ 1 xi þ i
ð16Þ
198
Robust and Nonparametric Statistical Methods
The parameter 1 is called the slope and 0 the intercept. We denote q ¼ ð0 ; 1 ÞT . Applying a regression estimator to the data yields regression coefficients qˆ ¼ ðˆ0 ; ˆ1 ÞT . The residual ri of case i is defined as the difference between the observed response yi and its estimated value yˆi: ˆ ¼ y i – yˆi ¼ y i – ðˆ0 þ ˆ1 x i Þ ri ðqÞ
The classical least-squares (LS) method for linear regression estimates by minimizing the sum of the squared residuals: minimize
n X
ð17Þ
ri2
i¼1
P The error variance 2 is estimated by s 2 ¼ ni¼1 ri2 =ðn – 2Þ. The LS method is very tractable as it yields an explicit computation of the coefficients, and as it is the most efficient approach when the errors are normally distributed. But when this assumption is not satisfied by all observations, the LS estimates can be very different from the true (population) coefficients. We illustrate the behavior of the LS estimator again on the toxicity data, where X ¼ ELUMO and Y ¼ aquatic toxicity. On the scatter plot of the data in Figure 4, we have added the LS line. We see that the line tries to make all residuals as small as possible, and consequently passes through the cloud of outlying points. A plot of the standardized residuals ri/s in Figure 5(a) shows no abnormal observations, as they are almost all inside the tolerance band 21,0.975 ¼ 2.24 and 2.24. Only acid 13 has a slightly larger residual, but it cannot be flagged as a strong outlying observation. A robust estimator tries to fit the bulk of the data points. When we apply the LTS estimator24 on these data, we find a regression line that is not pulled down by the outliers, as can be seen from Figure 4. The standardized LTS residuals in Figure 5(b) now clearly show the outlying acids. The least trimmed squares (LTS) estimator proposed by Rousseeuw24 is given by minimize
hLTS X ðr 2 Þi:n
ð18Þ
i¼1
where ðr 2 Þ1:n ðr 2 Þ2:n ðr 2 Þn:n are the ordered squared residuals. (They are first squared, and then ranked.) The value hLTS plays the same role as hMCD in the definition of the MCD estimator. For hLTS n/2 we
LTS and LS 1
13 36 37
0.5
Toxicity
32 0
34 LS
35 −0.5
23 LTS
−1
−1.5
1
1.5
2
2.5
3 3.5 ELUMO
4
4.5
5
5.5
Figure 4 Scatter plot of ‘aquatic toxicity’ vs. ‘ELUMO’, with least squares (LS) and least trimmed squares (LTS) line superimposed.
Robust and Nonparametric Statistical Methods
(a)
LS
(b)
LTS
3 10
2
35 Standardized LTS residual
Standardized LS residual
36
32
13
199
1
0
−1
8 34 6 4
23
13
37
2 0
−2
−2 −3
0
5
10
15
20 Index
25
30
35
40
0
5
10
15
20 Index
25
30
35
40
Figure 5 Standardized residual plot obtained with (a) LS; (b) LTS.
find a breakdown value of 50%, whereas for larger hLTS we obtain (n hLTS þ 1)/n. A fast algorithm for the LTS estimator (FAST-LTS) has been developed.27 To estimate the variance 2, we can use 2 ¼ c2hLTS ;n ˆ LTS
hLTS 2
1 X r i:n
hLTS
i¼1
where ri are the residuals from the LTS fit, and chLTS,n makes ˆ consistent and unbiased at Gaussian error distributions. The LTS estimator does not only provide a robust fit to the data, but also allows to detect the outliers. This has already been demonstrated by means of the standardized residual plot. Moreover, we can classify the outliers into three different types. Leverage points are observations (xi, yi) whose xi are outlying, that is xi deviates from the majority in x-space. We call such an observation a good leverage point if (xi, yi) follows the linear pattern of the majority. In contrast, if (xi, yi) does not follow this linear pattern we call it a bad leverage point. An observation whose xi belongs to the majority in x-space but where (xi, yi) deviates from the linear pattern is called a vertical outlier. A regression data set can thus have upto four types of points: regular observations, vertical outliers, good leverage points, and bad leverage points. To measure the outlyingness of xi in x-space, we compute its robust distance (Equation (12)). For univariate x-data, as here in the case of simple regression, this robust distance corresponds with the absolute standardized residual such as in Table 1. Here we use for ˆ and ˆ the univariate MCD estimator applied to the xi. (Note that for multiple linear regression, with more than one x-variable, the robust distance based on the multivariate MCD estimator can be used; see Chapter 3.07 of this book for examples.) The outlier map now plots the standardized LTS residuals versus the robust distances. When we construct the outlier map for the toxicity data, we obtain Figure 6(b). We see that observations 23, 32, 34, 35, and 36 are bad leverage points, case 20 is a good leverage point, and acids 13 and 37 are vertical outliers. When we look at the outlier map based on the LS estimator in Figure 6(a), we clearly see that the bad leverage points have attracted the LS solution and are converted into good leverage points. Apart from the LTS estimator, many other robust regression estimators have been proposed. The L1 estimator minimizes n X
jri j
i¼1
whereas more general M-estimators minimize n r X i i¼1
200
Robust and Nonparametric Statistical Methods
(a)
(b)
LS
36
4
10
2
36
37
1
32 34
0
35
−1
23 20
−2
32 35
13
−3
Standardized LTS residual
3 Standardized LS residual
LTS
8 34 6 13 37
4
23
2 20
0 −2
−4
−4 0
0.5
1
1.5 2 2.5 Mahalanobis distance
0
3
2
4 6 8 10 12 Robust distance computed by MCD
14
Figure 6 Outlier map obtained with (a) LS, (b) LTS.
These estimators are however not resistant to leverage points. MM-estimators28 combine high robustness with high efficiency. Tutorials and applications of robust regression methods in chemistry are reviewed in Rousseeuw,29 Ortiz et al.,30 and Rousseeuw et al.26 For robust multiple regression, refer to Chapters 3.07 and 3.24 of this book and references therein. 1.07.4.2
Nonparametric Regression
When there is no justification for assuming a linear influence of a variable X on the variable of interest Y, modeling Y ¼ 0 þ 1 X þ
is not appropriate, and if no other appropriate parametric form describing the influence of X on Y can be put forward, a way out is to consider a nonparametric regression model in which the n data points (xi, yi) satisfy yi ¼ mðxi Þ þ i
ð19Þ
where the i s are i.i.d. random errors with mean zero and variance 2. Without assuming a parametric form for the regression function m, a data point at a considerable distance away from x carries little information about the value of m(x). Hence, when estimating the influence m(x), only data in a neighborhood of x will be taken into account, and data points xi closer to x will have more weight in the estimation procedure. This leads to the calculation of a locally weighted average Pn Kh ðxi – xÞyi mˆ h ðxÞ ¼ Pi¼1 n i¼1 Kh ðxi – xÞ
ð20Þ
where Kh(?) is a rescaled version of a given symmetric probability density function K, namely Kh(y) ¼ h1K(y/h), with h ¼ hn > 0. Popular choices for K are a standard Gaussian density function or a biweight kernel KB ðtÞ ¼ ð1 – jt j2 Þ2 I½ – 1;1 ðtÞ
ð21Þ
where IA for a set A is the indicator function on that set, that is, IA(x) ¼ 1 if x falls in A and is 0 otherwise. The function K is called the kernel. The parameter h ¼ hn > 0 is the bandwidth parameter, and defines the size of the local neighborhood around x. Within this neighborhood the estimator mˆh(x) simply calculates a weighted average of the yi observations in this neighborhood. The estimator in Equation (20) is called the Nadaraya–
Robust and Nonparametric Statistical Methods
201
Watson estimator of m(x) and was introduced by Nadaraya31 and Watson.32 The estimator of the whole curve m(?) is obtained by calculating the estimator mˆh(x) over a grid of x-values covering the domain of the function m(?). Consider x, a fixed point, and denote by ¼ m(x) the target value that we would like to estimate. Another way of looking at mˆh(x) is then as mˆ h ðxÞ ¼ ˆ ¼ argmin
( n X
)
ðyi – Þ2 Kh ðxi – xÞ
i¼1
In other words, ˆ ¼ mˆ h ðxÞ is the result of finding the minimizer of n X
ðyi – Þ2 Kh ðxi – xÞ
i¼1
which is just a weighted version of the usual LS quantity, with larger weights for data points xi close to x. Hence, the Nadaraya–Watson estimator for m(?) is obtained by approximating the unknown function m(?) locally by a constant function and by fitting this constant via LS. An interesting extension of the above idea is to locally approximate the unknown function m by a linear function instead of by a constant function. This means, for z in a neighborhood of x, use a first-order Taylor expansion to write mðzÞ mðxÞ þ m9ðxÞðz – xÞ
The corresponding local LS problem would then be to find the minimizer of n X
ðyi – 0 – 1 ðxi – xÞÞ2 Kh ðxi – xÞ
ð22Þ
i¼1
where 0 ¼ m(x) and 1 ¼ m9(x). Finding (0, 1) that minimizes the weighted sum of squared residuals in Equation (22) would then yield an estimator not only for m(x) but also for m9(x), the derivative of the unknown regression function evaluated in x. Solving the minimization problem minimize 0 ;1
n X
ðyi – 0 – 1 ðxi – xÞÞ2 Kh ðxi – xÞ
i¼1
for a grid of x-values covering the domain of the variable X results in a nonparametric estimator for both the regression function and its first derivative. The resulting estimator of m is denoted by mˆLL and is called the local linear estimator of m. This estimator has a simple explicit expression given by mˆ LL ¼ ˆ0 ðxÞ ¼
n X xi – x w2;K – w1;K ðxi – xÞ K yi 2 hn w2;K w0;K – w1;K i¼1
where wk;K ¼
n X xi – x K ðxi – xÞk ; h n i¼1
k ¼ 0; 1; 2
is a sample version of the kth moment of the kernel function K. Using a Taylor expansion of m(z) in a neighborhood of a given x upto terms of order say p (p 0) leads to mðzÞ mðxÞ þ m9ðxÞðz – xÞ þ
1 1 m0ðxÞðz – xÞ2 þ ??? þ mðpÞ ðxÞðz – xÞp 2! p!
a polynomial of degree p. Using LS to locally fit this polynomial of degree p is called local polynomial regression and leads to estimates for the regression function m(?) and its derivatives upto order p. See Fan and Gijbels33 for an overview of this estimation method, its properties, and the application of it to a variety of estimation problems and application areas. See also, Ruppert and Wand34 for a detailed study of the extension to the multivariate case.
202
Robust and Nonparametric Statistical Methods
1.0
1.0 •
• •
•
•
•
•
0.5
•
0.5
• • •• •
0.0
• •
• •• • • •• • •• ••
–0.5
Toxicity
Toxicity
• • • • •
• • • •
•
•
40
• • • •
• • • •
• •
•
• •• • • •• • •• ••
–0.5 •
–1.0 30
0.0
• •
• • •• •
• •
•
•
•
•
–1.0
50 60 Surface tension
70
•
30
40
50 60 Surface tension
70
Figure 7 Scatter plot of ‘toxicity’ vs. ‘surface tension’. Left panel: Nadaraya–Watson estimator; right panel: local-linear estimator.
The above idea can of course be used for fitting locally any given parametric function. This estimation technique also clearly highlights the main difference between parametric fitting of a regression function and nonparametric fitting. The latter at most assumes a parametric model locally, whereas the former assumes a parametric model over the whole domain of the X variable. In Figure 7, the variable X is the surface tension plotted against the toxicity response variable Y. In the left panel of Figure 7, a Nadaraya–Watson estimator for the regression function is plotted. The right panel of Figure 7 depicts the local linear estimator. Note that both estimators have a little dip around 60. 1.07.4.3
Robust Locally Weighted Regression
Since the above weighted LS regression is based on the squared loss function, it will be sensitive to outliers similarly as the classical LS linear regression (see Section 1.07.4.1). Robustifications of the above locally weighted LS method have been proposed. A simple method of robustification of the local linear fitting method is to do an iterative procedure, in which at each iteration step the largest residuals from the fit of the previous step will be downweighted. This results in the well-known LOcally WEighted Scatter plot Smoothing (LOWESS) procedure implemented in the S-PLus and R software. The LOWESS method35 uses the tricube kernel KT ðtÞ ¼
70 ð1 – jt j3 Þ3 I½ – 1;1 ðt Þ 81
which descends smoothly to zero. In LOWESS, the local neighborhoods are determined by a nearest-neighbor type of bandwidth. For 0 < d 1 given, consider the portion of the data nd, and let r ¼ [nd] be its nearest integer value. For given observations xi, the neighborhood of an observation xk is determined by the bandwidth hk defined as the distance from xk to its rth nearest neighbor, that is,
hk ¼ xk – xj ; j ¼ 1; . . . ; n ðr Þ
the rth smallest number among all xk – xj . The LOWESS procedure starts by the locally weighted regression problem minimize 0 ;1
n X i¼1
ðyi – 0 – 1 ðxi – xk ÞÞ2 hk– 1 K
xi – xk hk
ð23Þ
Robust and Nonparametric Statistical Methods
1.0
1.0 •
• •
•
•
•
0.5
•
0.5
•
• • • •• •
0.0
• •
• •• • • •• • •• ••
–0.5
Toxicity
Toxicity
203
• • • •
•
•
40
• • • •
• •
• • • •
• •
•
• •• • • •• • •• ••
–0.5 •
–1.0 30
0.0
• •
• • • •
• • •• •
•
•
•
•
–1.0
50 60 Surface tension
70
•
30
40
50 60 Surface tension
70
Figure 8 Scatter plot of ‘aquatic toxicity’ vs. ‘surface tension’. Left panel: LOWESS estimator; right panel: Nadaraya– Watson estimator (solid curve), local-linear estimator (dashed curve), and LOWESS estimator (dotted curve).
From this initial fit, the fitted value for yk is given by yˆk ¼ ˆ0 ¼ ˆ0 ðx k Þ. Repeating Equation (23) for each observation xk gives the fitted values yˆ1, . . ., yˆn from which the residuals rk ¼ yk – yˆk
are then calculated. Using the biweight kernel KB(t) in Equation (21), the robustness weights k ¼ KB
rk 6 median ðjr1 j; . . . ; jrn jÞ
x i – xk is replaced by the are obtained. In the next iteration step of the procedure the initial kernel weight K hk xi – xk weight k K , which means a downweighting of observations with large residual values in the initial fit. hk The final fitted values after a number of iterations give the LOWESS estimator for the regression function m. In Figure 8, the LOWESS estimator of the scatter plot of the toxicity and surface tension data is depicted in the left panel. The right panel depicts the three estimators, the Nadaraya–Watson estimator, the local-linear estimator, and the LOWESS estimator. Note that the LOWESS estimator seems to be less influenced by the observation around 60, and also tends to be less high in the left-corner region. Other robustifications of the locally weighted LS method have been proposed. These include replacing the squared loss function (x) ¼ x2 by a loss function that is less influenced by aberrant observations such as the L1loss function ðxÞ ¼ jx j or Huber’s function 9(x) ¼ (x) ¼ min{b, max{b, x}}, among others.
1.07.5 Investigation Dependence Structures Modeling the influence of a variable X on a variable Y via a regression model, such as a linear regression model in Equation (15) or a nonparametric regression model as in Equation (19) is one way to investigate the dependence structure between X and Y. In the case of least-squares fitting, one in fact models the conditional expectation of Y given X, as for example EðY jX Þ ¼ 0 þ 1 X or via EðY jX Þ ¼ mðX Þ, with m(?) an unknown function. More robust alternatives are to look at the median regression curve instead of the above mean regression curve. Another way to investigate the dependence between X and Y is by measuring the association between variables X and Y via association measures, such as Kendall’s tau or Spearman’s rho. Yet, another alternative is
204
Robust and Nonparametric Statistical Methods
to try to model directly via the joint distribution function, the type of dependence that exists between X and Y. The latter approach is the idea behind copulas which is briefly explained below.
1.07.5.1
Association Measures and Copulas
In Equation (9) of Section 1.07.3.1 the sample version of Kendall’s tau was defined as the proportion of pairs in concordance minus the proportion of pairs in discordance. The population version of Kendall’s tau is simply the probability of concordance minus the probability of discordance. More precisely, let (X1,Y1) and (X2,Y2) be i.i.d. random vectors with the same joint distribution as the vector (X,Y). Then Kendall’s tau is defined as
K ¼ K;X ;Y ¼ P½ðX1 – X2 ÞðY1 – Y2 Þ > 0 – P½ðX1 – X2 ÞðY1 – Y2 Þ < 0
To define the population version of Spearman’s rho, consider (X1, Y1), (X2, Y2), and (X3, Y3) as three i.i.d. random vectors with the same joint distribution as the vector (X, Y). Spearman’s rho is defined to be proportional to the probability of concordance minus the probability of discordance for the two vectors (X1,Y1) and (X2,Y3), that is S ¼ S;X ;Y ¼ 3P½ðX1 – X2 ÞðY1 – Y3 Þ > 0 – P½ðX1 – X2 ÞðY1 – Y3 Þ < 0
With F denoting the distribution function of X and G denoting the distribution function of Y, and by putting U ¼ F(X) and V ¼ G(Y), Spearman’s rho can also be rewritten as CovðU ; V Þ S ¼ pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi VarðU ÞVarðV Þ
which clearly is the population version of the sample version in Equation (10). There exist several universal inequalities between the two association measures Kendall’s tau and Spearman’s rho. Some examples of these are – 1 3 K – 2S 1;
1 þ S 2
1 þ K 2
2 and
1 – S 2
1 – K 2
2
Such inequalities provide information on how different the two association measures, Kendall’s tau and Spearman’s rho can be. For example, for the toxicity data in Section 1.07.3.1 we obtain 3tK 2tS ¼ 0.151. For a detailed discussion of these and other association measures see, for example, Nelsen.36 Many other association measures exist, such as Gini’s measure of association and Blomqvist’s median correlation coefficient (see Blomqvist37). The latter coefficient involves the medians of X and Y and is defined as B ¼ P½ðX – medianðX ÞÞðY – medianðY ÞÞ > 0 – P½ðX – medianðX ÞÞðY – medianðY ÞÞ < 0
Among the many association measures, Kendall’s tau and Spearman’s rho are the most widely known scaleinvariant measures of association. Many, but not all, of these association measures are linked to the so-called copula function, which is a function that ‘couples’ a joint distribution function to its univariate marginals. Let H(x, y) denote the joint distribution function of the continuous random vector (X, Y), and let F be the marginal distribution function of X and G be the marginal distribution function of Y. Then, Sklar’s theorem38 tells us that there exists a unique bivariate function C that links the joint distribution function H with its marginal distribution functions: H ðx; yÞ ¼ CðF ðxÞ;GðyÞÞ
for all x and y
ð24Þ
The bivariate function C(?, ?) is called the copula function. A detailed treatment of copulas, their properties, appropriate parametric families of copulas, and their extension to multivariate distributions can be found in Nelsen.36 Note from Equation (24) that a copula function is always defined on the unit square I 2 ¼ [0, 1] [0, 1]. Many parametric families of copulas are available, and by specifying a parametric copula function, in some sense, one specifies a dependence structure between X and Y, leaving the marginals F and G unspecified.
Robust and Nonparametric Statistical Methods
205
An alternative to having to specify a parametric family of copulas is the nonparametric estimation of a copula function. This will be discussed in the next section. Of interest is to mention that several association measures such as Kendall’s tau and Spearman’s rho can be expressed in terms of the copula function ZZ
K ¼ 4
Cðu; vÞ dCðu; vÞ – 1 ¼ 4EðCðU ; V ÞÞ – 1 ZZ S ¼ 12 Cðu; vÞ du dv – 3
ð25Þ
I2
ð26Þ
I2
where U and V are uniform random variables. Obviously, when a parametric family of copulas is specified, depending on say a parameter , estimation of this parameter and substituting the parametric form of C(u, v) into expressions (25) and (26) lead to the corresponding ‘parametric’ estimators for Kendall’s tau and Spearman’s rho. 1.07.5.2
Nonparametric Estimation of a Copula Function
Nonparametric estimation of a copula function C relies on basic nonparametric estimation tools for a univariate distribution function. Assume we have a univariate variable X with a distribution function F. Based on a sample of i.i.d. observations x1, x2, . . ., xn from F, an obvious nonparametric estimator for F(x) ¼ P(X x), at a given point x, is simply the proportion of observations xi not exceeding x : Fn ðxÞ ¼
n n number of xik x 1 X 1X I½ – 1;x ðxi Þ ¼ I fxi x g ¼ n n i¼1 n i¼1
ð27Þ
where we adopted the notation I{x P A} instead of IA(x) for a given set A. The estimator (27) is a step function, showing jumps of size 1/n in each observation xi and staying constant between two consecutive observations. Improved estimators for F have been proposed in the literature. Note that the empirical distribution function is a step function, and if the unknown distribution function is smooth (i.e., continuous) then this is not a very accurate estimator for it. An improved version of Fn can be obtained by simply ‘smoothening’ the empirical distribution function Fn. A first obvious way to do this is by linear interpolation, in which the middle points of two subsequent constant parts in Fn are joined by (linear) lines. More generally, a smooth version of Fn is obtained by replacing the indicator function I fxi x g ¼ I fxi – x 0g in Equation (27) by a smooth function KD, a given distribution function (e.g., the standard Gaussian cumulative distribution ) leading to 1 Fˆn ðxÞ ¼ n
n X
KD
i¼1
x – xi hn
ð28Þ
where hn > 0 is a bandwidth parameter. If one takes KD, for example, the distribution function of a Uniform(1,1) distribution, then the function KD is 8 0 if > > < KD ðxÞ ¼ 0:5x þ 0:5 if > > : 1 if
x < –1 –1 x < 1
ð29Þ
x1
In other words, for a given point x, an observation xi that is in the neighborhood of the point x, defined by [x hn, x þ hn], will contribute with a weight n1 (0.5((x xi)/hn) þ 0.5) to the estimator (28) instead of with the weight n1 as in estimator (27). Observations xi that are closer to the given x contribute with a higher weight. Obviously, since KD is a distribution function, each observation xi contributes in estimator (28) with a weight between 0 and n1. This is in contrast to estimator (27) where the weight for each xi not exceeding x is n1. The ‘smooth’ assignment of weight in estimator (28) results in a ‘smooth’ version of the empirical distribution function.
206
Robust and Nonparametric Statistical Methods
Suppose that the density f of X exists. Then, a nonparametric estimator for the density f can be derived easily from Equation (28) by differentiating the expression with respect to x. This can be done if KD is differentiable and leads to the following nonparametric estimator for a density function f : n 1 X x – xi fˆn ðxÞ ¼ Kd n hn i¼1 hn
ð30Þ
with Kd now being a given probability density function (often taken to be symmetric) such as the standard gaussian density function. Figure 9 shows such a kernel-density estimator for the calcium data (left panel) together with a zoom-in version of it (right panel). The little bumps in the kernel density estimator to the right of the mode come from the six upper outliers. The little bump close to zero comes from the three lower outliers. The zoom-in version of the kernel density estimator (left panel) shows a possible secondary mode around 500. The empirical distribution function of the ELUMO variable in the toxicity data set is presented in Figure 10 (left panel). The right panel depicts a smooth version of this empirical distribution function using the kernel KD as in Equation (29). Figure 11 shows a kernel density estimator for the ELUMO data. Note the two smaller bumps around 1.8 and 2.7 that just reflect the few outliers around these values in this data set. The above estimation technique is also closely related to the kernel estimation of a regression function, since there too the local neighborhood is determined by a kernel function and a bandwidth parameter. See, for example, the Nadaraya–Watson estimator of the regression function in Equation (20), in which the denominator is in fact a nonparametric estimator of the marginal density of the X variable (compare with Equation (30)). Nonparametric estimation of a copula function C starts from its definition in Equation (24) by noting that Cðu; vÞ ¼ H ðF – 1 ðuÞ; G – 1 ðvÞÞ 1
ð31Þ
1
0.004
0.004
0.003
0.003
Density
Density
where F and G are the inverse functions of F and G respectively. Note that when F and G are continuous and strictly increasing, these inverse functions are easily defined.
0.002
0.002
0.001
0.001
0.0
0.0 0
1000
2000 Calcium
3000
4000
0
200
400 600 Calcium
Figure 9 Kernel density estimator for calcium data (left panel), and a zoom-in version of it (right panel).
800
1000
1.0
1.0
0.8
0.8
0.6
0.6
Distribution
Distribution
Robust and Nonparametric Statistical Methods
0.4
0.4
0.2
0.2
0.0
0.0 2.0
2.5
3.0 3.5 ELUMO
4.0
4.5
207
2.0
5.0
2.5
3.0 3.5 ELUMO
4.0
4.5
5.0
Figure 10 Empirical distribution function for the ELUMO variable in the toxicity data set (left panel), and a smooth version of it (right panel).
1.0
Density
0.8
0.6
0.4
0.2
0.0 1
2
3
4
5
6
ELUMO Figure 11 Kernel density estimate for the ELUMO variable in the toxicity data set.
In general, the inverse function of F is defined as F – 1 ðpÞ ¼ minfF ðxÞ pg for 0 p 1 x
ð32Þ
and similarly for G. With p ¼ 0.5 this leads to the population median of F. When F is continuous and strictly increasing this coincides with F(median(X)) ¼ 0.5, with median(X) being the population median.
208
Robust and Nonparametric Statistical Methods
An empirical estimator for F1(p) is just the inverse function of the empirical distribution function Fn(?) in Equation (27), that is, Fn– 1 ðxÞ ¼ minfFn ðxÞ pg x
for 0 p 1
ð33Þ
Recall the definition of a copula in Equation (31). The empirical copula function based on (x1, y1), (x2, y2), . . ., (xn, yn), i.i.d. observations from the joint distribution function H, is then simply obtained by generalizing the above univariate empirical distribution function estimator combined with the empirical version of the inverse of the univariate marginals. More precisely, the empirical version of the joint distribution function H(x, y), similarly defined as its univariate version in Equation (27), is Hn ðx; yÞ ¼
n 1X I fxi x; yi yg n i¼1
ð34Þ
Let Fn(x) and Gn(x) be its associated marginal distributions, that is Fn ðxÞ ¼ Hn ðx; þ1Þ
and
Gn ðyÞ ¼ Hn ðþ1; yÞ
ð35Þ
then the empirical copula function is defined as Cn ðu; vÞ ¼ Hn ðFn– 1 ðuÞ;Gn– 1 ðvÞÞ
ð36Þ
This estimator was studied by Deheuvels.39 The estimator in Equation (34) is a bivariate step function. Improved smooth versions of such an estimator can be obtained by using techniques similar to those used for obtaining a smooth version of a univariate distribution function (see Equation (28)). Analogous to Equation (28), one can start from a smoothed empirical distribution function 1 Hˆn ðx; yÞ ¼ n
n X i¼1
KD
x – xi y – yi ; hn hn
ð37Þ
where now KD is a given bivariate cumulative distribution function. Considering the associated estimators for the marginal distribution functions Fˆn (x) ¼ Hˆn (x, þ1) and Gˆn (y) ¼ Hˆn (þ1, y), as in Equation (35), this leads to the smoothed empirical copula function estimator
Cˆn ðu;vÞ ¼ Hˆn Fˆ n– 1 ðuÞ;Gˆ n – 1 ðvÞ
ð38Þ 40
The weak convergence of this estimator has been studied by Fermanian et al. Other, further improved, smooth versions of the empirical copula function can be obtained by using, for example, the ideas behind local linear fitting (see Section 1.07.4.2). For recent contributions, see Chen and Huang41 and Omelka et al.42 among others. Since Kendall’s tau and Spearman’s rho can be expressed in terms of the copula function C, it is clear from Equations (25) and (26) that nonparametric estimators for these association measures can also be obtained by replacing the unknown bivariate function C in these expressions by a nonparametric estimator. Substituting C in Equation (25) by the empirical copula function in Equation (36) leads to an estimator that is, for large samples, equivalent to the sample version of Kendall’s tau in Equation (9).
1.07.6 Bibliographic Notes In this chapter, we have included many references and review articles on robust and/or nonparametric methods. Here, we provide some additional references for further reading and give information on software availability. The robust methodology is introduced in the books by Huber8 and Hampel,1 and further developed by Rousseeuw and Leroy,43 Staudte and Sheather,44 and Barnett and Lewis.45 Recent books include Maronna et al.46 and Reimann et al.47 Nonparametric methods or smoothing methods are reviewed and studied by Fan
Robust and Nonparametric Statistical Methods
209
and Gijbels,33 Wand and Jones,48 Simonoff,49 and Schimek,50 among others. The books by Ha¨rdle51 and Bowman and Azzalini52 contain many examples and pay special attention to software applications. A basic reference for association measures and modeling via copulas is Nelsen.36 Software for robust procedures is available in the commercial software SAS and SPLUS, and in the freely available R software (http://www.r-project.org) as part of the packages ‘robust’, ‘robustbase’ and ‘rrcov’. Matlab functions are available as part of ‘‘LIBRA, Library for Robust Analysis’’,53 which can be downloaded from http://wis.kuleuven.be/stat/robust. Software for nonparametric methods, including some robust nonparametric methods, is also available in SPLUS and R, but also in Matlab toolboxes. Information on computational aspects for modeling and inference via copulas can be found in Yan,54 for example.
Acknowledgments The authors gratefully acknowledge financial support by the GOA/07/04-project of the Research Fund KULeuven. Support from the IAP research network no. P6/03 of the Federal Science Policy, Belgium, is also acknowledged.
References 1. Hampel, F. R.; Ronchetti, E. M.; Rousseeuw, P. J.; Stahel, W. A. Robust Statistics: The Approach Based on Influence Functions; Wiley: New York, 1986. 2. Maguna, F. P.; Nu´n˜ez, M. B.; Okulik, N. B.; Castro, E. A. Improved QSAR Analysis of the Toxicity of Aliphatic Carboxylic Acids. Russ. J. Gen. Chem. 2003, 73, 1792–1798. 3. ISO 5725, Accuracy, Trueness and Precision of Measurement Methods and Results, Part 5. Alternative Methods for the Determination of the Precision of a Standard Measurement Method; Gene`ve, 1994. 4. Davies, L. Statistical Evaluation of Interlaboratory Tests. Fresenius’ J. Anal. Chem. 1988, 331, 513–519. 5. Pestiaux, P.; Pic, C. The Limitations of the Cochran and Grubbs Outlier Tests in Round Robin Testing, SAE paper (2004), no. 200401-1894. 6. Analytical Methods Committee. Robust Statistics: How Not to Reject Outliers. Part 1. Basis Concepts. Analyst 1989, 114, 1693–1967. 7. Rousseeuw, P. J.; Croux, C. Alternatives to the Median Absolute Deviation. J. Am. Stat. Assoc. 1993, 88, 1273–1283. 8. Huber, P. J. Robust Statistics; Wiley: New York, 1981. 9. Rousseeuw, P. J.; Verboven, S. Robust Estimation in Very Small Samples. Comput. Stat. Data Anal. 2002, 40, 741–758. 10. Tukey, J. W. Exploratory Data Analysis; Addison-Wesley: Reading, MA, 1977. 11. Hoaglin, D. C.; Mosteller, F.; Tukey, J. W. Understanding Robust and Exploratory Data Analysis; Wiley-Interscience: New York, 1983. 12. Terbeck, W.; Davies, P. L. Interactions and Outliers in the Two-Way Analysis of Variance. Ann. Stat. 1998, 26, 1279–1305. 13. Kulinskaya, E.; Dollinger, M. B. Robust Weighted One-Way ANOVA: Improved Approximation and Efficiency. J. Stat. Plan. Inference 2007, 137 (2), 462–472. 14. Shane, K. V.; Simonoff, J. S. A Robust Approach to Categorical Data Analysis. J. Comput. Graph. Stat. 2001, 10, 135–157. 15. Analytical Methods Committee. Robust Statistics: How Not to Reject Outliers. Part 2. Inter-Laboratory Trials. Analyst 1989, 114, 1699–1702. 16. Rocke, D. M. Robust Statistical Analysis of Interlaboratory Studies. Biometrika 1983, 70 (2), 421–431. 17. Mu¨ller, C. H.; Uhlig, S. Estimation of Variance Components with High Breakdown Point and High Efficiency. Biometrika 2001, 88 (2), 353–366. 18. Pestiaux, P.; Pic, C. Robust Approaches for Mean and Variance Outliers Detection in Round Robin Testing, SAE paper (2007), no. 2007-01-2063. Q and RQ Charts: Robust Control Charts. J. R. Stat. Soc. Ser. D 1992, 41, 97–104. 19. Rocke, D. M. X 20. Giloni, A.; Seshardri, S.; Simonooff, J. S. Robust Analysis of Variance: Process Design and Quality Improvement. Int. J. Prod. Qual. Manag. 2006, 1, 306–319. 21. Vandewalle, B.; Beirlant, J.; Christmann, A.; Hubert, M. A Robust Estimator for Tail Index of Pareto-Type Distributions. Comput. Stat. Data Anal. 2007, 51, 6252–6268. 22. Hubert, M.; Vandervieren, E. An Adjusted Boxplot for Skewed Distributions. Comput. Stat. Data Anal. 2008, 52, 5186–5201. 23. Brys, G.; Hubert, M.; Struyf, A. A Robust Measure of Skewness. J. Comput. Graph. Stat. 2004, 13, 996–1017. 24. Rousseeuw, P. J. Least Median of Squares Regression. J. Am. Stat. Assoc. 1984, 79, 871–880. 25. Rousseeuw, P. J.; Van Driessen, K. A Fast Algorithm for the Minimum Covariance Determinant Estimator. Technometrics 1999, 41, 212–223. 26. Rousseeuw, P. J.; Debruyne, M.; Engelen, S.; Hubert, M. Robustness and Outlier Detection in Chemometrics. Crit. Rev. Anal. Chem. 2006, 36, 221–242. 27. Rousseeuw, P. J.; Van Driessen, K. Computing LTS Regression for Large Data Sets. Data Min. Knowl. Discov. 2006, 12, 29–45. 28. Yohai, V. J. High Breakdown Point and High Efficiency Robust Estimates for Regression. Ann. Stat. 1987, 15, 642–656.
210
Robust and Nonparametric Statistical Methods
29. Rousseeuw, P. J. Tutorial to Robust Statistics. J. Chemom. 1991, 5, 1–20. 30. Ortiz, M. C.; Sarabia, L. A.; Herrero, A. Robust Regression Techniques: A Useful Alternative for the Detection of Outlier Data in Chemical Analysis. Talanta 2006, 70, 499–512. 31. Nadaraya, E. A. On Estimating Regression. Theor. Probab. Appl. 1964, 9, 141–142. 32. Watson, G. S. Smooth Regression Analysis. Sankhya Ser. A 1964, 26, 359–372. 33. Fan, J.; Gijbels, I. Local Polynomial Modelling and Its Applications; Chapman and Hall: New York, 1996. 34. Ruppert, D.; Wand, M. P. Multivariate Weighted Least Squares Regression. Ann. Stat. 1994, 90, 1257–1270. 35. Cleveland, W. S. Robust Locally Weighted Regression and Smoothing Scatterplots. J. Am. Stat. Assoc. 1979, 74, 829–836. 36. Nelsen, R. B. An Introduction to Copulas, 2nd ed.; Springer: New York, 2006. 37. Blomqvist, N. On a Measure of Dependence between Two Random Variables. Ann. Math. Stat. 1950, 21, 593–600. 38. Sklar, A. Fonctions de re´partition a` n dimensions et leurs marges. Publications de l‘Institut de Statistique de L’Universite´ de Paris 1959, 8, 229–231. 39. Deheuvels, P. La fonction de de´pendance empirique et ses proprie´te´s, un test non parametrique d’inde´pendance. Bulletin de l’Acade´mie Royale de Belgique, Classe des Sciences 1979, 65, 274–292. 40. Fermanian, J.-D.; Radulovic, D.; Wegkamp, M. Weak Convergence of Empirical Copula Processes. Bernoulli 2004, 10, 847–860. 41. Chen, S. X.; Huang, T.-M. Nonparametric Estimation of Copula Functions for Dependence Modeling. Can. J. Stat. 2007, 35, 265–282. 42. Omelka, M.; Gijbels, I.; Veraverbeke, N. Improved Kernel Estimation of Copulas: Weak Convergence and Goodness-of-Fit Testing, Manuscript, 2008. 43. Rousseeuw, P. J.; Leroy, A. M. Robust Regression and Outlier Detection; Wiley-Interscience: New York, 1987. 44. Staudte, R. G.; Sheather, S. J. Robust Estimation and Testing; Wiley-Interscience: New York, 1990. 45. Barnett, V.; Lewis, T. Outliers in Statistical Data, 3rd ed.; Wiley: New York, 1994. 46. Maronna, R. A.; Martin, D. R.; Yohai, V. J. Robust Statistics: Theory and Methods; Wiley: New York, 2006. 47. Reimann, C.; Filzmoser, P.; Garrett, R.; Dutter, R. Statistical Data Analysis Explained: Applied Environmental Statistics with R; Wiley: New York, 2008. 48. Wand, M. P.; Jones, M. C. Kernel Smoothing; Chapman & Hall: New York, 1995. 49. Simonoff, J. S. Smoothing Methods in Statistics; Springer: New York, 1996. 50. Schimek, M. G. (ed.), Smoothing and Regression: Approaches, Computation, and Application; Wiley: New York, 2000. 51. Ha¨rdle, W. Smoothing Techniques: With Implementation in S; Springer: New York, 1991. 52. Bowman, A. W.; Azzalini, A. Applied Smoothing Techniques for Data Analysis. The Kernel Approach and S-PLUS Illustrations; Oxford Science Publications: Oxford, 1997. 53. Verboven, S.; Hubert, M. LIBRA: A Matlab Library for Robust Analysis. Chemom. Intell. Lab. Syst. 2005, 75, 127–136. 54. Yan, J. Enjoy the Joy of Copulas: With a Package Copula. J. Stat. Softw. 2007, 21 (4), 1–21.
Robust and Nonparametric Statistical Methods
Biographical Sketches
Ire`ne Gijbels received a university degree in mathematics from the Katholieke Universiteit Leuven, Belgium, and a Ph.D. degree in science, specialization in mathematical statistics from the Central Jury, Belgium. After her Ph.D. study she was a senior research assistant at the National Science Foundation Belgium. She was affiliated with the Institute of Statistics, Universite´ catholique de Louvain, Louvain-la-Neuve, Belgium for many years. Ire`ne Gijbels is now Professor of Statistics at the Department of Mathematics, Katholieke Universiteit Leuven, Belgium, and a member of the Leuven Statistics Research Center (LStat). Her current main research areas are semi- and nonparametric statistical methods, in particular curve and surface estimation, smoothing techniques and penalization, and regularization and deconvolution problems, among others. She is an associate editor of several international journals, author of a book on local modeling and applications, and author of many papers in international journals in statistics and probability.
Mia Hubert is an associate professor in the Department of Mathematics at the Katholieke Universiteit Leuven and a member of the Leuven Statistics Research Centre (LStat). She is teaching several courses for bachelor and master students of the faculty of science. She is specialized in the development of robust multivariate methods, with many applications in chemometrics and bioinformatics. Recently, robust methods have been developed for skewness, PCA, PCR, PLS, classification, ICA, and multiway analysis. This has resulted so far in more than 30 journal papers and 14 chapters in a book (among which several are review papers). Moreover she is the co-editor of a book on ‘Recent developments’ in Robust Statistics. Mia Hubert has been associate editor of Journal of Computational and Graphical Statistics. Currently she is associate editor of Computational Statistics and Data Analysis and member of the editorial board of Journal of Chemometrics.
211
This page intentionally left blank
1.08
Bayesian Methodology in Statistics
J. M. Bernardo, Universitat de Vale`ncia, Valencia, Spain ª 2009 Elsevier B.V. All rights reserved.
1.08.1 1.08.2 1.08.2.1 1.08.2.1.1 1.08.2.1.2 1.08.2.1.3 1.08.2.1.4 1.08.2.1.5 1.08.2.2 1.08.3 1.08.3.1 1.08.3.1.1 1.08.3.1.2 1.08.3.2 1.08.3.3 1.08.3.3.1 1.08.3.4 1.08.4 1.08.4.1 1.08.4.1.1 1.08.4.1.2 1.08.4.1.3 1.08.4.1.4 1.08.4.2 1.08.4.2.1 1.08.4.2.2 1.08.5 1.08.5.1 1.08.5.1.1 1.08.5.1.2 1.08.5.2 1.08.6 1.08.6.1 1.08.6.2 1.08.6.3 1.08.6.4 References
Introduction and Notation Axiomatic Foundations Probability as a Rational Measure of Conditional Uncertainty Probabilistic diagnosis Estimation of a proportion Measurement of a physical constant Prediction Regression Statistical Inference and Decision Theory Bayesian Methodology The Learning Process Nuisance parameters Domain restrictions Predictive Distributions Regression The simple linear model Asymptotic Behavior Reference Analysis Reference Distributions One parameter One nuisance parameter Many parameters Limited information Frequentist Properties Point estimation Interval estimation Inference Summaries Estimation Point Estimation Interval estimation Hypothesis Testing Discussion Coherence Objectivity Operational Meaning Generality
214 216 216 216 216 217 217 217 218 219 219 221 222 223 225 225 226 228 229 229 231 233 233 234 234 235 236 236 236 238 239 242 242 242 243 243 243
This chapter includes updated sections of the paper Bayesian Statistics, prepared by the author for the Encyclopedia of Life Support Systems, a 2003 online UNESCO publication.
213
214
Bayesian Methodology in Statistics
Symbols {A, Y, C}
Be(x|, ) ˜ d ðjDÞ
Fij(!)
I fT ; pðÞg
fpˆðx Þjpðx Þg L(a, ) N(x|, ) p(|D, C)
p(x|D) p(x|, C)
a decision problem with class of actions A, set of relevant events Y, and set of consequences C. the PDF of a beta distribution for x, with parameters and . expected intrinsic discrepancy ˜ and the true model, between pðDjÞ given data D. generic element of the Fisher information matrix of a general model of the form M X fpðxj!Þ;xPX ;!Pg. expected information about the value of to be provided by an observation tPT from p(t|), when the prior is p(). the logarithmic (Kullback–Leibler) divergence of pˆx from px. the loss of action a when occurs. the PDF of a normal distribution for x, with mean and variance 2. probability density of the unknown parameter given data D and conditions C. the predictive distribution of a future observation x, given data D. probability density of the observable random vector x given parameter and conditions C.
Pr(E|D, A, K) (|C) (|D, C)
(|M, P) S(!) St(x|, , )
˜ ) (, t ¼ t(D) U(a, ) {x, s2} !P
the probability of the event E, given data D, assumptions A and any other available knowledge K. a possibly improper prior function for . the posterior distribution of after data D have been observed, obtained by formal use of Bayes’ theorem with the prior function (|C). reference prior for given model M and class P of candidate priors. inverse of Fisher information matrix F(!). the PDF of a student distribution for x, with location parameter scale parameter and degrees of freedom. intrinsic discrepancy between p(D|) ˜ and p(D|). a general statistic, function of the data D. The utility of action a when occurs. The mean and the variance of a set of observations {x1,. . ., xn}. a general parameter vector in a model M X fpðxj!Þ;xPX ;!Pg.
1.08.1 Introduction and Notation Experimental results usually consist of sets of data of the general form D ¼ {x1, . . ., xn}, where the xis are somewhat ‘homogeneous’ (possibly multidimensional) observations. Statistical methods are then typically used to derive conclusions on both the nature of the process which has produced those observations, and on the expected behavior in future instances of the same process. A basic element of any statistical analysis is the specification of a probability model, which is assumed to describe the mechanism that has generated the observed data D as a function of a (possibly multidimensional) parameter (vector) w P W, sometimes referred to as the state of nature, about which only limited information (if any) is available. All derived statistical conclusions are conditional on the assumed probability model. Unlike most other branches of mathematics, frequentist methods of statistical inference suffer from the lack of an axiomatic basis; as a consequence, their proposed desiderata are often mutually incompatible, and the analysis of the same data may well lead to incompatible results when different, apparently intuitive procedures are tried; see Lindley1 and Jaynes2 for many instructive examples. In marked contrast, the Bayesian approach to statistical inference is firmly based on axiomatic foundations which provide a unifying logical structure, and guarantee the mutual consistency of the methods proposed. Bayesian methods constitute a complete paradigm to statistical inference, a scientific revolution in Kuhn’s sense. Bayesian statistics only require the mathematics of probability theory and the interpretation of probability, which most closely corresponds to the standard use of this word in everyday language: it is no accident that some
Bayesian Methodology in Statistics
215
of the more important seminal books on Bayesian statistics, such as the works of de Laplace,3 Jeffreys4, or de Finetti5, are actually entitled ‘Probability Theory’. The practical consequences of adopting the Bayesian paradigm are far reaching. Indeed, Bayesian methods (1) reduce statistical inference to problems in probability theory, thereby minimizing the need for completely new concepts, and (2) serve to discriminate among conventional, typically frequentist statistical techniques, by either providing a logical justification to some (and making explicit the conditions under which they are valid) or proving the logical inconsistency of others. The main result from these axiomatic foundations is the mathematical need to describe by means of probability distributions all uncertainties present in the problem. In particular, unknown parameters in probability models must have a joint probability distribution which describes the available information about their values; this is often regarded as the characteristic element of a Bayesian approach. Notice that (in sharp contrast to frequentist statistics) parameters are treated as random variables within the Bayesian paradigm. This is not a description of their variability (parameters are typically fixed unknown quantities) but a description of the uncertainty about their true values. A most important particular case arises when either no relevant prior information is readily available, or that information is subjective and an ‘objective’ analysis is desired, one that is exclusively based on accepted model assumptions and well-documented public prior information. This is addressed by reference analysis which uses information-theoretic concepts to derive formal reference prior functions which, when used in Bayes’ theorem, lead to posterior distributions encapsulating inferential conclusions on the quantities of interest solely based on the assumed model and the observed data. In this chapter it is assumed that probability distributions may be described through their probability density functions, and no distinction is made between a random quantity and the particular values that it may take. Bold italic roman fonts are used for observable random vectors (generally data) and bold italic greek fonts are used for unobservable random vectors (typically parameters); lowercase is used for variables and calligraphic uppercase for their dominion sets. Moreover, the standard mathematical convention of referring to functions, say f and g of x P , respectively by f (x) and g(x), will be used throughout. Thus, p(qjD, C) and p(xjq, C) respectively represent general probability densities of the unknown parameter q P Y given data D and Rconditions C, and of the observable random vector x P conditional on q and C. Hence, p(qjD, C) 0, Y p(qjD, C) dq ¼ 1 and R p(xjq, C) 0, p(xjq, C) dx ¼ 1. Bayesian statistics often make use of improper prior functions for the unknown parameters, that is positive functions whose integral over their dominion is not finite; possibly improper prior functions will be denoted by (qjC) and their corresponding posterior densities given data D and conditions C (obtained by formal use of Bayes’ theorem) will be denoted by (qjD, C). This admittedly imprecise notation will greatly simplify the exposition. If the random vectors are discrete, these functions naturally become probability mass functions, and integrals over their values become sums. Density functions of specific distributions are denoted by appropriate names. Thus, if x is a random quantity with a normal distribution of mean and standard deviation , its probability density function will be denoted by N(xj, ). Bayesian methods make frequent use of the concept of logarithmic divergence, a very general measure of the goodness of the approximation of a probability density p(x) by another density pˆ(x). The Kullback–Leibler or logarithmic divergence of a probability density pˆ(x) of the random vector x P from its true probability density p(x), is defined as Z fpˆ ðxÞjpðxÞg ¼
pðxÞlogfpðxÞ=pˆ ðxÞg dx
ð1Þ
ˆ It may be shown that (1) the logarithmic divergence is nonnegative (and it is zero if, and only if, pðxÞ ¼ pðxÞ almost everywhere), and (2) that fpˆ ðxÞjpðxÞg is invariant under one-to-one transformations of x. This chapter contains a brief summary of the foundations of Bayesian statistical methods (Section 1.08.2), an overview of the paradigm (Section 1.08.3), a detailed discussion of objective Bayesian methods (Section 1.08.4), a description of useful objective inference summaries, including estimation and hypothesis testing (Section 1.08.5) and some concluding remarks (Section 1.08.6). Good pioneering introductions to objective Bayesian statistics include Lindley,6 Zellner,7 and Box and Tiao.8 For more advanced monographs, see Berger9 and Bernardo and Smith.10 Bayesian works with specific reference to chemometrics include Rubin11, Dryden et al.12, and the review by Chen et al.13
216
Bayesian Methodology in Statistics
1.08.2 Axiomatic Foundations A central element of the Bayesian paradigm is the use of probability distributions to describe all relevant unknown quantities, interpreting the probability of an event as a conditional measure of uncertainty, on a [0, 1] scale, about the occurrence of the event under some specific conditions. The limiting extreme values 0 and 1, which are typically inaccessible in applications, respectively describe impossibility and certainty of the occurrence of the event. This interpretation of probability includes and extends all other probability interpretations. There are axiomatic arguments which prove the mathematical inevitability of the use of probability distributions to describe uncertainties; these are summarized later in this section. 1.08.2.1
Probability as a Rational Measure of Conditional Uncertainty
Bayesian statistics uses the word probability in precisely the same sense in which this word is used in everyday language, as a conditional measure of uncertainty associated with the occurrence of a particular event, given the available information and the accepted assumptions. Thus, Pr(EjC) is a measure of (presumably rational) belief in the occurrence of the event E under conditions C. It is important to stress that probability is always a function of two arguments, the event E whose uncertainty is being measured, and the conditions C under which the measurement takes place; ‘absolute’ probabilities do not exist. In typical applications, one is interested in the probability of some event E given the available data D, the set of assumptions A which one is prepared to make about the mechanism that has generated the data, and the relevant contextual knowledge K which might be available. Thus, Pr(EjD, A, K) is to be interpreted as a measure of (presumably rational) belief in the occurrence of the event E, given data D, assumptions A, and any other available knowledge K as a measure of how ‘likely’ is the occurrence of E under these conditions. Sometimes, but certainly not always, the probability of an event under given conditions may be associated with the relative frequency of ‘similar’ events under ‘similar’ conditions. The following examples are intended to illustrate the use of probability as a conditional measure of uncertainty. 1.08.2.1.1
Probabilistic diagnosis An industrial production is known to contain 0.2% of products, which will eventually fail within their guaranteed period (faulty items). A particular item, randomly selected from that population, is subject to a test which is known to yield positive results, indicating the likely existence of defects, in 98% of faulty items and in 1% of nonfaulty items, so that, if F denotes the event that an item will fail within its guaranteed period and þ denotes a positive test result, PrðþjF Þ ¼ 0:98 and Pr þjF ¼ 0:01: Suppose that the result of the test turns out to be positive. Clearly, one is then interested in PrðF jþ; A; K Þ, the probability that the item will fail, given the positive test result, the assumptions A about the probability mechanism generating the test results, and the available knowledge K of the proportion of faulty items in the population under study (described here by PrðF jK Þ ¼ 0:002). An elementary exercise in probability algebra, which involves Bayes’ theorem in its simplest form (see Section 1.08.3), yields PrðF jþ; A; K Þ ¼ 0:164. Notice that all the four probabilities involved in the problem have the same interpretation: they are all conditional measures of uncertainty. Besides, PrðF jþ; A; K Þ is both a measure of the uncertainty associated with the event that the particular item which tested positive is actually faulty, and an estimate of the proportion of items in that population (about 16.4%) that would eventually prove to be faulty among those which yielded a positive test. 1.08.2.1.2
Estimation of a proportion A survey is conducted to estimate the proportion of items in a population which share a given property. A random sample of n elements is analyzed, r of which are found to possess that property. One is then typically interested in using the results from the sample to establish regions of [0, 1], where the unknown value of may plausibly be expected to lie; this information is provided by probabilities of the form Prða < < bjr ; n; A; K Þ, a conditional measure of the uncertainty about the event that belongs to (a, b) given the information provided by the data (r, n), the assumptions A made on the behavior of the mechanism which has generated the data (a random sample of n Bernoulli trials), and any relevant knowledge K on the values of which might be available. For example, after a screening test for a particular defect where 100 items have been tested, none of which has turned out to have that defect, one may conclude that Prð < 0:01j0; 100; A; K Þ ¼ 0:844, that is a probability of 0.844 that the proportion of items with that defect is smaller than 1%.
Bayesian Methodology in Statistics
217
1.08.2.1.3
Measurement of a physical constant A team of scientists, intending to establish the unknown value of a physical constant , obtain data D ¼ {x1, . . ., xn}, which are considered to be measurements of subject to error. The probabilities of interest are then typically of the form Prða < < bjx1 ; . . . ; xn ; A; K Þ, the probability that the unknown value of (fixed in nature, but unknown to the scientists) lies within an interval (a, b) given the information provided by the data D, the assumptions A made on the behavior of the measurement mechanism, and whatever knowledge K might be available on the value of the constant . Again, those probabilities are conditional measures of uncertainty, which describe the (necessarily probabilistic) conclusions of the scientists on the true value of , given available information and accepted assumptions. For example, after a classroom experiment to measure the gravitational field with a pendulum, a student may report (in m s2) something like Prð9:788 < g < 9:829jD; A; K Þ ¼ 0:95, meaning that, under accepted knowledge K and assumptions A, the observed data D indicate that the true value of g lies within 9.788 and 9.829 with probability 0.95, a conditional uncertainty measure on a [0, 1] scale. This is naturally compatible with the fact that the value of the gravitational field in the laboratory may be well known with high precision from available literature or from precise previous experiments, but the student may have been instructed not to use that information as part of the accepted knowledge K. Under some conditions, it is also true that if the same procedure was actually used by many other students with similarly obtained data sets, their reported intervals would actually cover the true value of g in approximately 95% of the cases, thus providing a frequentist calibration of the student’s probability statement. 1.08.2.1.4
Prediction An experiment is made to count the number r of times that an event E takes place in each of n replications of a welldefined situation; it is observed that E does take place ri times in replication i, and it is desired to forecast the number of times r that E will take place in a similar future situation. This is a prediction problem on the value of an observable (discrete) quantity r, given the information provided by data D, accepted assumptions A on the probability mechanism which generates the ris, and any relevant available knowledge K. Computation of the probabilities fPrðr jr1 ; . . .; rn ; A; K Þg for r ¼ 0, 1, . . ., is thus required. For example, the quality assurance engineer of a firm that produces automobile restraint systems may report something like Prðr ¼ 0jr1 ¼ ¼ r10 ¼ 0; A; K Þ ¼ 0:953, after observing that the entire production of airbags in each of n ¼ 10 consecutive months has yielded no complaints from their clients. This should be regarded as a measure, on a [0, 1] scale, of the conditional uncertainty, given observed data, accepted assumptions and contextual knowledge, associated with the event that no airbag complaint will come from next month’s production and, if conditions remain constant, this is also an estimate of the proportion of months expected to share this desirable property. A similar problem may naturally be posed with continuous observables. For instance, after measuring some continuous magnitude in each of n randomly chosen elements within a population, it may be desirable to forecast the proportion of items in the whole population whose magnitude satisfies some precise specifications. As an example, after measuring the breaking strengths {x1, . . ., x10} of 10 randomly chosen safety belt webbings to verify whether or not they satisfy the requirement of remaining above 26 kN, the quality assurance engineer may report something like Prðx > 26jx1 ; . . .; x10 ; A; K Þ ¼ 0:9987. This should be regarded as a measure, on a [0, 1] scale, of the conditional uncertainty (given observed data, accepted assumptions, and contextual knowledge) associated with the event that a randomly chosen safety belt webbing will support no less than 26 kN. If production conditions remain constant, it will also be an estimate of the proportion of safety belts which will conform to this particular specification. 1.08.2.1.5
Regression Often, additional information of future observations is provided by related covariates. For instance, after observing the outputs fy 1 ; . . .; yn g which correspond to a sequence fx1 ; . . . ; xn g of different production conditions, it may be desired to forecast the output y which would correspond to a particular set of production conditions x. For example, the viscosity of commercially condensed milk is required to be within specified values a and b; after measuring the viscosities fy1 ; . . .; yn g which correspond to samples of condensed milk produced under different physical conditions fx1 ; . . .; xn g, the production engineers will require probabilities of the form Prða < y < b jx; ðy 1 ; x1 Þ; . . .; ðy n ; xn Þ; A; K Þ. This is a conditional measure of the uncertainty (always given observed data, accepted assumptions, and contextual knowledge) associated with the event that condensed milk produced under conditions x will actually satisfy the required viscosity specifications.
218
Bayesian Methodology in Statistics
1.08.2.2
Statistical Inference and Decision Theory
Decision theory not only provides a precise methodology to deal with decision problems under uncertainty, but its solid axiomatic basis also provides a powerful reinforcement to the logical force of the Bayesian approach. We now summarize the basic argument. A decision problem exists whenever there are two or more possible courses of action; let A be the class of possible actions. Moreover, for each a P A , let Ya be the set of relevant events which may affect the result of choosing a, and let c(a, q) P C a, q P Ya , be the consequence of having chosen action a when event q takes place. The class of pairs fðYa ; C a Þ; aPA g describes the structure of the decision problem. Without loss of generality, it may be assumed that the possible actions are mutually exclusive, for otherwise one would work with the appropriate Cartesian product. Different sets of principles have been proposed to capture a minimum collection of logical rules that could sensibly be required for ‘rational’ decision-making. All these consist of axioms with a strong intuitive appeal; examples include the transitivity of preferences (if a1 > a2 given C, and a2 > a3 given C, then a1 > a3 given C), and the sure-thing principle (if a1 > a2 given C and E, and a1 > a2 given C and not E, then a1 > a2 given C). Notice that these rules are not intended as a description of actual human decision-making, but as a normative set of principles to be followed by someone who aspires to achieve coherent decision-making. There are naturally different options for the set of acceptable principles (see, e.g., Ramsey,14 Savage,15 DeGroot,16 Bernardo and Smith,10 and references therein), but all of them lead basically to the same conclusions, namely (1) Preferences among consequences should be measured with a real-valued utility function U(c) ¼ U(a, q), which specifies, on some numerical scale, their desirability. (2) The uncertainty of relevant events should be measured with a set of probability distributions fðpðqjC; a Þ; qPYa Þ; aPA g describing their plausibility given the conditions C under which the decision must be taken. (3) The desirability of each action is measured by its corresponding expected utility, U ðajC Þ ¼
Z
Ya
U ða; qÞpðqjC; aÞ dq;
aPA
ð2Þ
It is often convenient to work in terms of the nonnegative loss function defined by Lða; qÞ ¼ sup fU ða; qÞg – U ða; qÞ
ð3Þ
aPA
which directly measures, as a function of q, the ‘penalty’ for choosing a wrong action. The relative undesirability of available actions a P A is then measured by their expected loss LðajC Þ ¼
Z Ya
Lða; qÞpðqjC; a Þ dq;
aPA
ð4Þ
Notice that, in particular, the argument described above establishes the need to quantify the uncertainty about all relevant unknown quantities (the actual values of the qs), and specifies that this quantification must have the mathematical structure of probability distributions. These probabilities are conditional on the circumstances C under which the decision is to be taken, which typically, but not necessarily, include the results D of some relevant experimental or observational data. It has been argued that the development described above (which is not questioned when decisions have to be made) does not apply to problems of statistical inference, where no specific decision making is envisaged. However, there are two powerful counterarguments to this. Indeed, (1) a problem of statistical inference is typically considered worth analyzing because it may eventually help make sensible decisions; a lump of arsenic is poisonous because it may kill someone, not because it has actually killed someone (Ramsey14), and (2) it has been shown (Bernardo17) that statistical inference on q actually has the mathematical structure of a decision problem, where the class of alternatives is the functional space of the conditional probability distributions p(qjD) of q given the data, and the utility function is a measure of the amount of information about q which the data may be expected to provide. Another, independent argument for the necessary existence of prior distributions which does not make use of decision theoretical ideas is based on the concept of exchangeability and the related representation theorems. For details, see Bernardo and Smith10, and references therein.
Bayesian Methodology in Statistics
219
1.08.3 Bayesian Methodology The statistical analysis of some observed data D typically begins with some informal descriptive evaluation, which is used to suggest a tentative, formal probability model fpðDjw Þ; w P Wg assumed to represent, for some (unknown) value of w, the probabilistic mechanism which has generated the observed data D. The argument outlined in Section 1.08.2 establishes the logical need to assess a prior probability distribution pðwjK Þ over the parameter space W, describing the available knowledge K on the value of w prior to the data being observed. It then follows from standard probability theory that if the probability model is correct, all available information about the value of w after the data D have been observed is contained in the corresponding posterior distribution whose probability density, pðwjD; A; K Þ, is immediately obtained from Bayes’ theorem, pðwjD; A; K Þ ¼ Z O
pðDjA; w ÞpðwjK Þ
ð5Þ
pðDjA; w ÞpðwjK Þ dw
where A stands for the assumptions made on the probability model. It is this systematic use of Bayes’ theorem to incorporate the information provided by the data that justifies the adjective Bayesian by which the paradigm is usually known. It is obvious from Bayes’ theorem that any value of w with zero prior density will have zero posterior density. Thus, it is typically assumed (by appropriate restriction, if necessary, of the parameter space O) that prior distributions are strictly positive (as Savage15 put it, keep the mind open, or at least ajar). To simplify the presentation, the accepted assumptions A and the available knowledge K are often omitted from the notation, but the fact that all statements about w given D are also conditional to A and K should always be kept in mind. Example 1. (Bayesian inference with a finite parameter space) Let pðDjÞ; Pf1 ; . . .; m g, be the probability mechanism which is assumsed to have generated the observed data D, so that may only take a finite number of values. Using the finite form of Bayes’ theorem, and omitting the prevailing conditions from the notation, the posterior probability of i after data D have been observed is pðDji ÞPrði Þ
; Prði jDÞ ¼ Pm j ¼1 p Djj Pr j
i ¼ 1; . . .; m
ð6Þ
For any prior distribution pðÞ ¼ fPrð1 Þ; . . .; Prðm Þg describing available knowledge on the value of ; Prði jDÞ, measures how likely i should be judged, given both the initial knowledge described by the prior distribution and the information provided by the data D. An important, frequent application of this simple technique is provided by probabilistic diagnosis. For example, consider again the simple situation where a particular test designed to detect a potentially faulty item is known from laboratory research to give a positive result in 98% of faulty items and in 1% of nonfaulty items. Then, the posterior probability that an item which tested positive is faulty is given by PrðF jþÞ ¼ ð0:98pÞ=f0:98p þ 0:01ð1 – pÞg as a function of p ¼ Pr(F), the prior probability of an item being faulty. As one would expect, the posterior probability is 0 only if the prior probability is 0 (so that it is known that the production is free of defects) and it is 1 only if the prior probability is 1 (so that it is known that all the population is faulty). Notice that if the proportion of faulty items is small, then the posterior probability of a randomly chosen item being faulty will be relatively low even if the test is positive. Indeed, say for Pr(F) ¼ 0.002, one finds PrðF jþÞ ¼ 0:164, so that in a population where just 0.2% of items are faulty only 16.4% of those testing positive within a random sample will actually prove to be faulty: most positives would actually be false positives.
1.08.3.1
The Learning Process
In this section, the learning process described by Bayes’ theorem is described in some detail, discussing its implementation in the presence of nuisance parameters, showing how it can be used to forecast the value of future observations and analyzing its large sample behavior. In the Bayesian paradigm, the process of learning from the data is systematically implemented by making use of Bayes’ theorem to combine the available prior information with the information provided by the data to
220
Bayesian Methodology in Statistics
produce the required posterior distribution. Computation of posterior densities is made easier by noting that Bayes’ theorem may be expressed as pðwjDÞ _ pðDjw Þpðw Þ
ð7Þ
(where _ stands for ‘proportional to’ and where, for simplicity, the accepted assumptions A and the available knowledge K have been omitted from the notation), since the missing proportionality constant –1 R may always be deduced from the fact that p(wjD), a probability density, must integrate O p ðDjw Þp ðw Þdw to 1. Hence, to identify the form of a posterior distribution it suffices to identify a kernel of the corresponding probability density, that is a function k(w) such that pðwjDÞ ¼ c ðDÞkðw Þ for some c(D) which does not involve w. In the examples which follow, this technique will often be used. R An improper prior function is defined as a positive function (w) such that O ðwÞdw is not finite. Equation (7), the formal expression ofR Bayes’ theorem, remains technically valid if p(w) is actually an improper prior function (w), provided that O pðDjwÞðwÞdw < 1, thus leading to a well-defined proper posterior density ðwjDÞ _ pðDjwÞðwÞ. In particular, as will be justified later (Section 1.08.4), it also remains philosophically valid if (w) is an appropriately chosen reference (typically improper) prior function. We will use the generic notation (w) for a possibly improper prior function of w and ðwjDÞ for the corresponding posterior density (whose propriety should always be verified). Considered as a function of w; lðwjDÞ ¼ pðDjwÞ is often referred to as the likelihood function. Thus, Bayes’ theorem is simply expressed in words by the statement that the posterior is proportional to the likelihood times the prior. It follows from Equation (7) that, provided the same (proper or improper) prior function (w) is used, two different data sets D1 and D2, with possibly different probability models p1 ðD1 jwÞ and p2 ðD2 jwÞ but yielding proportional likelihood functions, will produce identical posterior distributions for w. This immediate consequence of Bayes’ theorem has been proposed as a principle on its own, the likelihood principle, and it is seen by many as an obvious requirement for reasonable statistical inference. In particular, for any given prior function (w), the posterior distribution does not depend on the set of possible data values, or the sample space. Notice, however, that the likelihood principle only applies to inferences about the parameter vector w once the data have been obtained. Consideration of the sample space is essential, for instance, in model criticism, in the design of experiments, in the derivation of predictive distributions, and in the construction of objective Bayesian procedures. Naturally, the terms prior and posterior are only relative to a particular set of data. As one would expect from the coherence induced by probability theory, if data D ¼ fx1 ; . . .; xn g are sequentially presented, the final result will be the same whether data are globally or sequentially processed. Indeed, ðwjx 1 ; . . .; xiþ1 Þ _ pðxiþ1 jwÞðwjx 1 ; . . .; xi Þ for i ¼ 1; . . .; n – 1, so that the ‘posterior’ at a given stage becomes the ‘prior’ at the next. In most situations, the posterior distribution is ‘sharper’ than the prior so that, in most cases, the density ðwjx 1 ; . . .; xiþ1 Þ will be more concentrated around the true value of w than ðwjx1 ; . . .; x i Þ. However, this is not always the case: occasionally, a ‘surprising’ observation will increase, rather than decrease, the uncertainty about the value of w. For instance, in probabilistic diagnosis, a sharp posterior probability distribution (over the possible causes f!1 ; . . .; !k g of a syndrome) describing a clear’ diagnosis of disease !i (i.e., a posterior with a large probability for !i) would typically update to a less concentrated posterior probability distribution over f!1 ; . . .; !k g if a new clinical analysis yielded data which were unlikely under !i. For a given probability model, one may find that a particular function of the data t ¼ t(D) is a sufficient statistic in the sense that, given the model, t(D) contains all information about w which is available in D. Formally, t ¼ t(D) is sufficient if (and only if) there exist nonnegative functions f and g such that the likelihood function may be factorized in the general form pðDjwÞ ¼ f ðw; tÞgðDÞ. A sufficient statistic always exists, for t(D) ¼ D is obviously sufficient; however, a much simpler sufficient statistic, with a fixed dimensionality which is independent of the sample size, often exists. In fact, this is known to be the case whenever the probability model belongs to the generalized exponential family, which includes many of the more frequently used probability models. It is easily established that if t is sufficient, the posterior distribution of w only depends on the data D through t(D), and may be directly computed in terms of pðtjwÞ, so that ðwjDÞ ¼ ðwjtÞ _ pðtjwÞðwÞ. Naturally, for fixed data and model assumptions, different priors lead to different posteriors. Indeed, Bayes’ theorem may be described as a data-driven probability transformation machine, which maps prior distributions
Bayesian Methodology in Statistics
221
(describing prior knowledge) into posterior distributions (representing combined prior and data knowledge). It is important to analyze whether or not sensible changes in the prior would induce noticeable changes in the posterior. Posterior distributions based on reference ‘noninformative’ priors play a central role in this sensitivity analysis context. Investigation of the sensitivity of the posterior to changes in the prior is an important ingredient of the comprehensive analysis of the sensitivity of the final results to all accepted assumptions which any responsible statistical study should contain. Example 2. (Inference on a binomial parameter) If the data D consist of n Bernoulli observations with parameter which contain r positive trials, then pðDj; nÞ ¼ r ð1 – Þn – r , so that tðDÞ ¼ fr ; ng is sufficient. Suppose that prior knowledge about is described by a Beta distribution Be(j, ), so that p(j, ) _ 1 (1 )1. Using Bayes’ theorem, the posterior density of is p(jr, n, , ) _ r (1 )nr 1 (1 )1 _ r þ 1 (1 )nr þ 1 the Beta distribution Be(jr þ , n r þ ). Suppose, for example, that in the light of precedent surveys, available information on the proportion of citizens who would vote for a particular political measure in a referendum is described by a Beta distribution Be(j50, 50), so that it is judged to be equally likely that the referendum would be won or lost, and it is judged that the probability that either side wins less than 60% of the vote is 0.95. A random survey of size 1500 is then conducted, where only 720 citizens declare to be in favor of the proposed measure. Using the results above, the corresponding posterior distribution is then Be(j770, 830). These prior and posterior densities are plotted in Figure 1; it may be appreciated that, as one would expect, the effect of the data is to drastically reduce the initial uncertainty on the value of and, hence, on the referendum outcome. More precisely, Pr( < 0.5j720, 1500, H, K) ¼ 0.933 (shaded region in Figure 1) so that, after the information from the survey has been included, the probability that the referendum will be lost should be judged to be about 0.933.
1.08.3.1.1
Nuisance parameters The general situation where the vector of interest is not the whole parameter vector w, but some function q ¼ q(w) of possibly lower dimension than w, will now be considered. Let D be some observed data, {p(Djw), w P W} a probability model assumed to describe the probability mechanism which has generated D, (w) a (possibly improper) prior function describing any available information on the value of w, and q ¼ q(w) P Y a function of the original parameters over whose value inferences based on the data D are required. Any valid conclusion on the value of the vector of interest q will then be contained in its posterior probability distribution (qjD), which is conditional on the observed data D and will naturally also depend, although not explicitly shown in the notation, on the assumed model {p(Djw), w P O} and on the available prior information (if any) encapsulated by (w). The required posterior distribution (qjD) is found by standard use of probability calculus. Indeed, Bayes’ theorem yields (wjD) _ p(Djw) (w). Moreover, let l ¼ l(w) P L be some other function of the original parameters such that y ¼ {q, l} is a one-to-one transformation of w, and let J(w) ¼ (@y /@w) be the corresponding Jacobian matrix. Naturally, the introduction of l is not necessary if
30
p(θ | r, n, α, β) = Be(θ | 730, 790)
25 20 15 10
p(θ | α, β) = Be(θ | 50, 50)
5 0.35
0.4
0.45
0.5
0.55
0.6
0.65
θ
Figure 1 Prior and posterior densities of the proportion of citizens who would vote in favor of a referendum.
222
Bayesian Methodology in Statistics
q(w) is a one-to-one transformation of w. Using standard change-of-variable probability techniques, the posterior density of y is ðy jDÞ ¼ ðq; ljDÞ ¼
ðwjDÞ jJ ðwÞj w¼wðy Þ
ð8Þ
and the required posterior of q is the appropriate marginal density, obtained by integration over the nuisance parameter l, Z ðqjDÞ ¼
L
ðq; ljDÞ dl
ð9Þ
Notice that elimination of unwanted nuisance parameters, a simple integration within the Bayesian paradigm is, however, a difficult (often polemic) problem for frequentist statistics. For further details on the elimination of nuisance parameters see Liseo.18 1.08.3.1.2
Domain restrictions Sometimes the range of possible values of w is effectively restricted by contextual considerations. If w is known to belong to Oc O, the prior distribution is only positive in Oc and, using Bayes’ theorem, it is immediately found that the restricted posterior is ðwjD; w P Wc Þ ¼ Z
ðwjDÞ Oc
ðwjDÞ
;
w P Oc
ð10Þ
and obviously vanishes if w ‚ Oc. Thus, to incorporate a restriction on the possible values of the parameters, it suffices to renormalize the unrestricted posterior distribution to the set Oc O of parameter values that satisfy the required condition. Incorporation of known constraints on the parameter values, a simple renormalization within the Bayesian paradigm, is another very difficult problem for conventional statistics. Example 3. (Inference on normal parameters) Let D ¼ {x1, . . ., xn} be a random sample from a normal distribution N ðxj; Þ. The corresponding likelihood function is immediately found to be proportional to –n exp½ –nfs 2 þ ð x – Þ2 g=ð22 Þ, with n x ¼ i x i and ns 2 ¼ i ðxi – xÞ2 . It may be shown (see Section 1.08.4) that absence of initial information on the value of both and may formally be described by a joint prior function, which is uniform in both and log(), that is by the (improper) prior function (, ) ¼ 1. Using Bayes’ theorem, the corresponding joint posterior is ð; jDÞ _ – ðnþ1Þ exp½–nfs 2 þ ð x – Þ2 g=ð22 Þ
ð11Þ
Thus, using the gamma integral in terms of ¼ 2 to integrate out , Z
1
ðjDÞ _ 0
h n i – ðnþ1Þ exp – 2 ½s 2 þ ð x – Þ2 d _ ½s 2 þ ð x – Þ2 – n=2 2
ð12Þ
pffiffiffiffiffiffiffiffiffi which is recognized as a kernel of the Student density Stðj x ; s= n – 1; n – 1Þ. Similarly, integrating out , Z ðjDÞ _
1
–1
h n i ns 2 – ðnþ1Þ exp – 2 ½s 2 þ ð x – Þ2 d _ – n exp – 2 2 2
ð13Þ
Changing variables to the precision ¼ 2 results in ( jD) _ (n 3)/2ens /2, a kernel of the Gamma density Ga( j(n 1)/2, ns2/2). In terms of the standard deviation , this becomes (jD) ¼ p( jD)|@ /@| ¼ 23 Ga(2j(n 1)/2, ns2/2), a square-root-inverted gamma density. 2
Bayesian Methodology in Statistics
(a)
(b)
40
π(g | x, s, n)
30
40
223
π(g | x, s, n, g ∈Gc)
30
20
20
10
10 9.75
9.8
9.85
g
9.9
9.7
9.75
9.8
9.85
9.9
g
Figure 2 Posterior density (gjm, s, n) of the value g of the gravitational field, given n ¼ 20 normal measurements with mean m ¼ 9.8087 and standard deviation s ¼ 0.0428. (a) With no additional information, and (b) with g restricted to Gc ¼ {g; 9.7803 < g < 9.8322}. Shaded areas represent 95%-credible regions of g.
A frequent example of this scenario is provided by laboratory measurements made under conditions where central limit conditions apply, so that (assuming no experimental bias) those measurements may be treated as a random sample from a normal distribution centered at the quantity which is being measured, and with some (unknown) standard deviation . Suppose, for example, that in an elementary physics classroom experiment to measure the gravitational field g with a pendulum, a student has obtained n ¼ 20 measurements of g yielding (in m s2) a mean x ¼ 9.8087 and a standard deviation s ¼ 0.0428. Using no other information, the corresponding posterior distribution is p(gjD) ¼ St(gj9.8087, 0.0098, 19) represented in Figure 2(a). In particular, Pr(9.788 < g < 9.829jD) ¼ 0.95, so that with the information provided by this experiment, the gravitational field at the location of the laboratory may be expected to lie between 9.788 and 9.829 with probability 0.95. Formally, the posterior distribution of g should be restricted to g > 0; however, as immediately obvious from Figure 2(a), this would not have any appreciable effect due to the fact that the likelihood function is actually concentrated on positive g values. Suppose now that the student is further instructed to incorporate into the analysis the fact that the value of the gravitational field g at the laboratory is known to lie between 9.7803 m s2 (average value at the equator) and 9.8322 m s2 (average value at the poles). The updated posterior distribution will then be pffiffiffiffiffiffiffiffiffi Stðgjm; s= n – 1; nÞ pffiffiffiffiffiffiffiffiffi ; Stðgjm; s= n – 1; nÞ
ðgjD; gPGc Þ ¼ Z
gPGc
ð14Þ
gPGc
and zero if g ‚ Gc, where Gc ¼ {g; 9.7803 < g < 9.8322}. This is represented in Figure 2(b). One-dimensional numerical integration may be used to obtain a new 0.95-credible interval; indeed Pr(g > 9.792jD, g P Gc) ¼ 0.95, the shaded region in Figure 2(b). Moreover, if inferences about the standard deviation of the measurement procedure are also requested, the corresponding posterior distribution is found to be ðjDÞ ¼ 2 – 3 Gað – 2 j9:5; 0:0183Þ
ð15Þ
This has a mean E[jD] ¼ 0.0458 and yields Pr(0.0334 < < 0.0642jD) ¼ 0.95. 1.08.3.2
Predictive Distributions
Let data D ¼ {x1, . . ., xn}, xi P , be a random sample from some distribution in the family {p(xjw), w P O}, (w) a (possibly improper) prior function describing available information (if any) on the value of w, and consider now a situation where it is desired to predict the value of a future observation x P generated by the same random mechanism that has generated the data D. It follows from the foundations arguments discussed in Section 1.08.2 that the solution to this prediction problem is simply encapsulated by the predictive distribution p(xjD) describing the uncertainty on the value that x will take, given the information provided by D and any other available knowledge. Since p(xjw, D) ¼ p(xjw), it then follows from standard probability theory that Z pðxjDÞ ¼
O
pðxjwÞ ðwjDÞdw
ð16Þ
which is an average of the probability distributions of x conditional on the (unknown) value of w, weighted with the posterior distribution of w given D, (wjD) _ p(Djw) (w).
224
Bayesian Methodology in Statistics
If the assumptions on the probability model are correct, the posterior predictive distribution p(xjD) will converge, as the sample size increases, to the distribution p(xjw) that has generated the data. Indeed, about the best technique to assess the quality of the inferences about w encapsulated in (wjD) is to check against the observed data the predictive distribution p(xjD) generated by (wjD). For a good introduction to Bayesian predictive inference, see Geisser.19 Example 4. (Prediction in a Poisson process) sample from a Poisson distribution Pn(rj ) with parameter , so that Let D ¼ {r1, . . ., rn} be a random P p(Dj ) _ te n, where t ¼ ri . It may be shown (see Section 1.08.4) that absence of initial information on the value of may be formally described by the (improper) prior function ( ) ¼ 1/2. Using Bayes’ theorem, the corresponding posterior is ð jDÞ _ t e – n – 1=2 _ t – 1=2 e n
ð17Þ
the kernel of a gamma density Ga( jt þ 1/2, n), with mean (t þ 1/2)/n. The corresponding predictive distribution is the Poisson-Gamma mixture Z
1
pðr jDÞ ¼ 0
1 nt þ1=2 1 ðr þ t þ 1=2Þ Pnðr j ÞGað jt þ ; nÞ d ¼ ðt þ 1=2Þ r ! ð1 þ nÞrþt þ1=2 2
ð18Þ
Suppose, for example, that in a firm producing automobile restraint systems, the entire production in each of 10 consecutive months has yielded no complaint from their clients. With no additional information on the average number of complaints per month, the quality assurance department of the firm may report that the probabilities that r complaints will be received in the next month of production are given by Equation (18), with t ¼ 0 and n ¼ 10. In particular, p(r ¼ 0jD) ¼ 0.953, p(r ¼ 1jD) ¼ 0.043, and p(r ¼ 2jD) ¼ 0.003. Many other situations may be described with the same model. For instance, if meteorological conditions remain similar in a given area, p(r ¼ 0jD) ¼ 0.953 would describe the chances of there being no flash flood next year, given that there has been no flash floods in the area for 10 years. Example 5. (Prediction in a normal process) Consider now an example of prediction of the future value of a continuous observable quantity. Let D ¼ {x1, . . ., xn} be a random sample from a normal distribution N(xj, ). As mentioned in Example 3, absence of initial information on the values of both and is formally described by the improper prior function (, ) ¼ 1, and this leads to the joint posterior density (13). The corresponding (posterior) predictive distribution is Z pðxjDÞ ¼ 0
1
Z
rffiffiffiffiffiffiffiffiffiffiffi nþ1 Nðxj; Þð; jDÞ d d ¼ Stðxj x; s ; n – 1Þ n–1 –1 1
ð19Þ
If is known to be positive, the appropriate prior function will be the restricted function ( ð; Þ ¼
–1 0
if > 0 otherwise
ð20Þ
However, the result in Equation (19) will still basically hold, provided the likelihood function p(Dj, ) is concentrated on positive values. Suppose, for example, that in the firm producing automobile restraint systems, the observed breaking strengths of n ¼ 10 randomly chosen safety belt webbings have mean x ¼ 28.011 kN and standard deviation s ¼ 0.443 kN, and that the relevant engineering specification requires breaking strengths to be larger than 26 kN. If data may truly be assumed to be a random sample from a normal distribution, the likelihood function is only appreciable for positive values, and only the information provided by this small sample is to be used, then the quality engineer may claim that the probability that a safety belt randomly chosen from the same batch as the sample tested would satisfy the required specification Pr(x > 26jD) ¼ 0.9987. Besides, if production conditions remain constant, 99.87% of the safety belt webbings may be expected to have acceptable breaking strengths.
Bayesian Methodology in Statistics
1.08.3.3
225
Regression
Let data D ¼ {(x1, y1), . . ., (xn, yn)}, xi P , yi P Y , be a set of n pairs of probabilistically related observations, so that the observation yi is assumed to be generated from a distribution p(yijxi, w), which depends on the known observed vector xi and on an unknown parameter vector w P O, and the likelihood function is Yn
pðDjwÞ ¼
j ¼1
pð y i jxi ; wÞ
ð21Þ
Let (w) be a (possibly improper) prior function describing available information (if any) on the value of w. Consider now a situation where, for some x P , it is desired to predict the value of a future observation y P Y generated from p(yjx, w). It follows again from the foundations arguments discussed in Section 1.08.2 that the solution to this prediction problem is simply encapsulated by the predictive distribution p(yjx, D) describing the uncertainty on the value that y will take, given x and the information provided by D and any other available knowledge. Since p(yjx, w, D) ¼ p(yjx, w), it follows from standard probability theory that Z pðyjx; DÞ ¼
O
pðyjx; wÞðwjDÞdw
ð22Þ
which is an average of the probability distributions of y conditional on x and the (unknown) value of w, weighted with the posterior distribution of w given D, (wjD) _ p(Djw) (w). If the assumptions on the probability model are correct, the posterior predictive distribution p(yjx, D) will converge, as the sample size n increases, to the distribution p(yjx, w) that would generate y given x. 1.08.3.3.1
The simple linear model Let D ¼ {(x1, y1), . . ., (xn, yn)} be a set of n pairs of related real-valued observables, and suppose that the yi s may be assumed to be linearly related to the xis with normal homoscedastic errors, so that pðDj; ; Þ ¼
Yn j ¼1
Nðyi j þ xi ; Þ
ð23Þ
As discussed in Section 1.08.4, absence of relevant initial information on the values of , , and is formally described by the improper prior function (, , ) ¼ 1. Using Equation (22), this leads to the (posterior) Student t predictive distribution ˆ s pðyjx; DÞ ¼ Stðyjˆ þ x;
ˆx ; ˆ ¼ y –
ˆ ¼
2 sxy 2 sxx
;
rffiffiffiffiffiffiffiffiffiffiffi nhðxÞ ; n – 2Þ n–2
hðxÞ ¼ 1 þ
2 1 ðx – xÞ2 þ s xx 2 sxx n
ð24Þ
ð25Þ
which depends on the set of sufficient statistics x ¼
1 Xn x; j ¼1 j n
2 ¼ sxy
y ¼
1 Xn y; j ¼1 j n
1 Xn ðx – xÞðy j – yÞ; j ¼1 j n
s2 ¼
1 Xn ðx – xÞ2 j ¼1 j n
ð26Þ
1 Xn ˆ j Þ2 ðy – ˆ – x j ¼1 j n
ð27Þ
2 sxx ¼
Notice that the predictive densities are all Student t with n 2 degrees of freedom, centered at the regression ˆ and with a scale parameter which depends on the observed covariate x through the function h(x). line ˆ þ x, As Equation (25) indicates, h(x) attains its minimum value, (n þ 1)/n, when x ¼ x, where prediction is most precise, and increases as x moves away from x so that, as one would expect, prediction is less precise as the covariate x moves away from the data center.
226
Bayesian Methodology in Statistics
Example 6. (Calibration) In an environmental study, indirect, laser-based automatic Grimm measurements of the contents in the air of micro PM10 particles were to be calibrated with more precise, gravimetric Andersen measurements. A set of n ¼ 12 air samples were analyzed yielding the results presented in the following table, and plotted in the left pane of Figure 3.
Automatic Gravimetric
48.2 41.2
41.4 39.4
44.1 38.1
50.2 33.4
71.2 48.3
49.0 35.2
14.3 17.5
66.2 44.9
30.0 24.3
36.8 27.8
58.1 50.7
83.1 68.0
Using Equation (25), the corresponding regression line is y ¼ 6.089 þ 0.668x. Thus, for small pollution values (below 18 mg m3), automatic measurements underestimate the correct gravimetric value but, for the important large values, automatic values are noticeably larger than the gravimetric values. For instance, if the observed automatic value is 70 mg m3 (vertical dashed line in the left pane of the figure), the predictive density of the corresponding gravimetric value (right pane of the figure) is the Student St(yj52.83, 5.42, 10), whose more likely value is 52.83 mg m3, and has values between 40.74 and 64.92 with probability 0.95 (confidence bands in the left pane at x ¼ 70, and shaded region in the right panel of Figure 3).
1.08.3.4
Asymptotic Behavior
The behavior of posterior distributions when the sample size is large is now considered. This is important for, at least, two different reasons: (1) asymptotic results provide useful first-order approximations when actual samples are relatively large, and (2) objective Bayesian methods typically depend on the asymptotic properties of the assumed model. Let D ¼ {x1, . . ., xn}, x P , be a random sample of size n from {p(xjw), w P O}. It may be shown that, as n ! 1, the posterior distribution of a discrete parameter w typically converges to a degenerate distribution, which gives a probability 1 to the true value of w, and that the posterior distribution of a continuous parameter w typically converges to a normal distribution centered at its maximum likelihood ˆ with a variance matrix which decreases with n as 1/n. estimate (MLE) w,
100 Gravimetric
(μg m3)
p(Grav Auto = 70,D)
y=x
80 Regression line: y 6.089 + 0.668 x 60
40
20 μg m3
Automatic (μg m–3) 20
40
60
80
100
20
40
60
80
Figure 3 Calibration. Left panel: Data set, regression line and 0.95 credible lines. Right panel: Predictive density of the gravimetric value given an automatic value of 70 mg m3, with its (shaded) 0.95-credible region.
Bayesian Methodology in Statistics
227
Consider first the situation where O ¼ {w 1, w 2,. . .} consists of a countable (possibly infinite) set of values, such that the probability model that corresponds to the true parameter value w t is distinguishable from the others, in the sense that the logarithmic divergence {p(xjw i)jp(xjw t)} of each of the p(xjw i) from p(xjw t) is strictly positive. Taking logarithms in Bayes’ theorem, defining zj ¼ log[p(xjjw i)/p(xjjw t)], j ¼ 1,. . ., n, and using the strong law of large numbers on the n conditionally independent and identically distributed random quantities z1, . . ., zn, it may be shown that lim Prðw t jx 1 ; . . . ; x n Þ ¼ 1;
lim Prðw i jx 1 ; . . . ; xn Þ ¼ 0;
n!1
n!1
i 6¼ t
ð28Þ
Thus, under appropriate regularity conditions, the posterior probability of the true parameter value converges to 1 as the sample size grows. Consider now the situation where Pw is a k-dimensional continuous P parameter. Expressing Bayes’ theorem as ðwjx 1 ; . . .; xn Þ _ explog½ðwÞ þ nj¼1 log½pðxj jwÞ, expanding j log½pðx j jwÞ about its maximum (the ˆ and assuming regularity conditions (to ensure that terms of order higher than quadratic may be MLE w), ignored and that the sum of the terms from the likelihood will dominate the term from the prior), it is found that the posterior density of w is the approximate k-variate normal, ˆ SðD; wÞg; ˆ ðwjx1 ; . . . ; x n Þ Nk fw; S
–1
ðD; wÞ ¼
–
n X @ 2 log½pðxl jwÞ
@w i @w j
l¼1
! ð29Þ
A simpler, but somewhat poorer, approximation may be obtained by using the strong law of large numbers on ˆ n FðwÞ, ˆ where F (w) is Fisher’s information matrix, with the sums in Equation (29) to establish that S – 1 (D,wÞ general element Z
@ 2 log½pðxjwÞ dx @w i @w j
ð30Þ
ˆ n – 1 F – 1 ðwÞÞ ˆ ðwjx 1 ; . . . ; x n Þ N k ðwjw;
ð31Þ
F ij ðwÞ ¼ –
pðxjwÞ X
so that
Thus, under appropriate regularity conditions, the posterior probability density of the parameter vector w ˆ with a variance approaches, as the sample size grows, a multivariate normal density centered at the MLE w, matrix that decreases with n as n1. Example 2 (continued). (Asymptotic approximation with binomial data) Let D ¼ (x1, . . ., xn) consist of n independent Bernoulli trials with parameter , so that p(Dj, n) ¼ r(1 )n r. This likelihood function is maximized at ˆ ¼ r =n, and Fisher’s information function is F() ¼ 1(1 )1. Thus, using the above results, the posterior distribution of will be the approximate normal, ˆ sðÞ= ˆ pffiffinffiÞ; ðjr ; nÞ Nðj;
sðÞ ¼ fð1 – Þg1=2
ð32Þ
ˆ with mean ˆ ¼ r =n and variance ˆ ¼ ð1 – Þ=n. This will provide a reasonable approximation to the exact posterior if (1) the prior p() is relatively ‘flat’ in the region where the likelihood function matters, and (2) both r and n are moderately large. If, say, n ¼ 1500 and r ¼ 720, this leads to (jD) N(j0.480, 0.013), and to Pr( > 0.5jD) 0.940, which may be compared with the exact value Pr( > 0.5jD) ¼ 0.933 obtained from the posterior distribution that corresponds to the prior Be(j50, 50). It follows from the joint posterior asymptotic behavior of w and from the properties of the multivariate normal distribution that if the parameter vector is decomposed into w ¼ (q, ), and Fisher’s information matrix is correspondingly partitioned, so that F ðwÞ ¼ F ðq; lÞ ¼
F ðq; lÞ
F ðq; lÞ
F ðq; lÞ
F ðq; lÞ
! ð33Þ
228
Bayesian Methodology in Statistics
and Sðq; lÞ ¼ F – 1 ðq; lÞ ¼
S ðq; lÞ
S ðq; lÞ
S ðq; lÞ
S ðq; lÞ
! ð34Þ
then the marginal posterior distribution of q will be n o ˆ n – 1 S q; ˆ ˆ ðqjDÞ N qjq;
ð35Þ
while the conditional posterior distribution of given q will be n o –1 –1 ðljq; DÞ N ljlˆ – F q; lˆ F q; lˆ qˆ – q ; n – 1 F q; lˆ
ð36Þ
Notice that F1 ¼ S if (and only if) F is block diagonal, that is if (and only if) q and l are asymptotically independent. Example 3 (continued). (Asymptotic approximation with normal data) Let D ¼ (x1, . . ., xn) be a random sample from a normal distribution N(xj, ). The corresponding likelihood ˆ Þ ˆ ¼ ð function p(Dj, ) is maximized at ð; x ; sÞ, and Fisher’s information matrix is diagonal, with F ¼ 2. pffiffiffi Hence, the posterior distribution x ; s= nÞ; this may be compared with the exact pffiffiffiffiffiffiffiffiffi of is approximately Nðj result ðjDÞ ¼ Stðj x ; s n – 1; n – 1Þ, previously obtained under the assumption of no prior knowledge.
1.08.4 Reference Analysis Under the Bayesian paradigm, the outcome of any inference problem (the posterior distribution of the quantity of interest) combines the information provided by the data with relevant available prior information. In many situations, however, either the available prior information on the quantity of interest is too vague to warrant the effort required to have it formalized in the form of a probability distribution, or it is too subjective to be useful in scientific communication or public decision making. It is therefore important to be able to identify the mathematical form of a ‘noninformative’ prior, a prior that would have a minimal effect, relative to the data, on the posterior inference. More formally, suppose that the probability mechanism which has generated the available data D is assumed to be p(Djw) for some w P O, and that the quantity of interest is some real-valued function ¼ (w) of the model parameter w. Without loss of generality, it may be assumed that the probability model is of the form M ¼ fpðDj; lÞ;DPD ; PY; lPLg
ð37Þ
where l is some appropriately chosen nuisance parameter vector. As described in Section 1.08.3, to obtain the required posterior density of the quantity of interest p(jD), it is necessary to specify a (possibly improper) joint prior function (, l). It is now required to identify the form of that joint prior function (, ljM ; P ), the -reference prior, which would have a minimal effect on the corresponding posterior distribution of , Z ðjDÞ _
L
pðDj; lÞ ð; ljM ; P Þdl
ð38Þ
within the class P of all the prior distributions compatible with whatever information one is prepared to assume about (, l), which may just be the class P 0 of all strictly positive priors. To simplify the notation, when there is no danger of confusion, the reference prior (, ljM ; P ) is often simply denoted by (, l), but its dependence on the quantity of interest , the assumed model M , and the class P of priors compatible with the assumed knowledge should always be kept in mind.
Bayesian Methodology in Statistics
229
To use a conventional expression, the reference prior ‘would let the data speak for themselves’ about the likely value of . Properly defined, reference posterior distributions have an important role to play in scientific communication, for they provide the answer to a central question in the sciences: conditional on the assumed model p(Dj, l), and on any further assumptions of the value of on which there might be universal agreement, the reference posterior (jD) should specify what could be said about if the only available information about were some well-documented data D and whatever information (if any) one is prepared to assume by restricting the prior to belong to an appropriate class P . Much work has been done to formulate ‘reference’ priors which would make the idea described above mathematically precise. For historical details, see Bernardo and Smith10 (Section 2.17.2), Kass and Wasserman20 Bernardo and Ramo´n21 Bernardo,22 and references therein. This section focuses on an approach that is based on information theory to derive reference distributions, which may be argued to provide the most advanced general procedure available; this was initiated by Bernardo23,24 and further developed by Berger and Bernardo,25–28 Bernardo,22 Berger et al.,29 and references therein. For a general discussion on ‘noninformative’ priors, see Bernardo.30 In the formulation described below, the reference posterior exploits certain well-defined features of a possible prior, namely those describing a situation where relevant knowledge about the quantity of interest (beyond that universally accepted, as specified by the choice of P ) may be held to be negligible compared to the information about that quantity which repeated experimentation (from a specific datagenerating mechanism M ) might possibly provide. Reference analysis is appropriate in contexts where the set of inferences that could be drawn in this possible situation is considered to be pertinent. Any statistical analysis contains a fair number of subjective elements; these include (among others) the data selected, the model assumptions, and the choice of the quantities of interest. Reference analysis may be argued to provide ‘objective’ Bayesian solutions to statistical inference problems in just the same sense that conventional statistical methods claim to be ‘objective’, that is in the sense that the solutions provided only depend on the assumed model and the observed data.
1.08.4.1
Reference Distributions
1.08.4.1.1
One parameter Consider an experiment that consists of the observation of data D, generated by a random mechanism p(Dj), which only depends on a real-valued parameter P Y, and let t ¼ t(D)PT be any sufficient statistic (which may well be the complete data set D). In Shannon’s general information theory, the amount of information I {T, p()} which may be expected to be provided by D, or (equivalently) by t(D), about the value of is defined by I fT ; pðÞg ¼ fpðtÞpðÞjpðtjÞpðÞg ¼ Et
Z Y
pðjtÞlog
pðjtÞ d pðÞ
ð39Þ
the expected logarithmic divergence of the prior from the posterior. This is naturally a functional of the prior density p(): the larger the prior information, the smaller the information that the data may be expected to provide. The functional I {T, p()} is concave, nonnegative, and invariant under one-to-one transformations of . Consider now the amount of information I {Tk, p()} about that may be expected from the experiment, which consists of k conditionally independent replications {t1,. . ., tk} of the original experiment. As k ! 1, such an experiment would provide any missing information about , which could possibly be obtained within this framework; thus, as k ! 1, the functional I {Tk, p()} will approach the missing information about associated with the prior p(). Intuitively, a -‘noninformative’ prior is one that maximizes the missing information about . Formally, if pk() denotes the prior density that maximizes I {Tk, p()} in the class P of prior distributions which are compatible with accepted assumptions on the value of (which may well be the class P 0 of all strictly positive proper priors), then the -reference prior (jM ; P ) is the limit as k ! 1 (in a sense to be made precise) of the sequence of priors {pk(), k ¼ 1, 2, . . .}. Notice that this limiting procedure is not some kind of asymptotic approximation, but an essential element of the definition of a reference prior. In particular, this definition implies that reference distributions only depend on the asymptotic behavior of the assumed probability model, a feature which actually simplifies their actual derivation.
230
Bayesian Methodology in Statistics
Example 7. (Maximum entropy) If may only take a finite number of values, so that the parameter space is Y ¼ {1, . . ., m} and p() ¼ {p1, . . ., pm}, with pi ¼ Pr( ¼ i), and there is no topology associated to the parameter space Y, so that the is are just labels with no quantitative meaning, then the missing information associated to {p1, . . ., pm} reduces to lim I fT k ; pðÞg ¼ H ðp1 ; . . .; pm Þ ¼ –
Xm
k!1
p i¼1 i
logðpi Þ
ð40Þ
that is the entropy of the prior distribution {p1, . . ., pm}. Thus, in the non structured finite case, the reference prior (jM ; P ) (which in this case is obviously always proper) is that with maximum entropy in the class P of priors compatible with accepted assumptions. Consequently, the reference prior algorithm contains ‘maximum entropy’ priors as the particular case which results when the parameter space is a finite set of nonquantitative labels, the only case where the original concept of entropy as a measure of uncertainty is unambiguous and well behaved. In particular, if P is the class P 0 of all priors over {1, . . ., m}, then the reference prior is the uniform prior over the set of possible values, (jM ; P 0) ¼ {1/m, . . ., 1/m}. Formally, the reference prior function ð j M ; P Þ of a univariate parameter is defined to be the limit of the sequence of the proper priors pk(), which maximize I {Tk, p()} in the precise sense that, for any value of the sufficient statistic t ¼ t(D), the reference posterior, the intrinsic limit ðjtÞ of the corresponding sequence of posteriors fpk ðjtÞg, may be obtained from (jM ; P ) by formal use of Baye’s theorem, so that ðjtÞ _ pðtjÞðjM ; P Þ.A sequence fpk ð j tÞg of posterior distributions converges intrinsically to a limit ð j tÞ if the sequence of expected intrinsic R discrepancies Et ½fpk ð j tÞ; ð j tÞg converges to 0, where fp; qg ¼ minfkðp j qÞ; kðq j pÞg and kðp j qÞ ¼ Y qðÞ log½qðÞ=pðÞd. For details, see Berger et al.29 Reference prior functions are often simply called reference priors, even though they are usually not probability distributions. They should not be considered as expressions of belief, but technical devices to obtain (proper) posterior distributions, which are a limiting form of the posteriors that could have been obtained from possible prior beliefs which were relatively uninformative with respect to the quantity of interest when compared with the information which the data could provide. If (1) the sufficient statistic t ¼ t(D) is a consistent estimator of a continuous parameter , and (2) the class P contains all strictly positive priors, then the reference prior may be shown to have a simple form in terms of any asymptotic approximation to the posterior distribution of . Notice that, by construction, an asymptotic approximation to the posterior does not depend on the prior. Specifically, if the posterior density pð j DÞ has ˜ nÞ, the (unrestricted) reference prior is simply an asymptotic approximation of the form pðj; ˜ nÞj ˜ ðjM ; P 0 Þ _ pðj; ¼
ð41Þ
One-parameter reference priors are invariant under reparametrization; thus, if ¼ ðÞ is a piecewise one-toone function of , then the -reference prior is simply the appropriate probability transformation of the -reference prior. Example 8. (Jeffreys’ prior) If is univariate and continuous, and the posterior distribution of given {x1. . ., xn} is asymptotically normal ˜ pffiffinffi, then, using Equation (41), the reference prior function is ðÞ _ sðÞ – 1 . with standard deviation sðÞ= Under regularity conditions (often satisfied in practice, see Section 1.08.3.3), the posterior distribution of is asymptotically normal with variance n – 1 F – 1 ðˆ Þ, where F() is Fisher’s information function and ˆ the MLE of . Hence, the reference prior function under these conditions is ðjM ; P 0 Þ _ F ðÞ1=2 , which is known as Jeffreys’ prior. It follows that the reference prior algorithm contains Jeffreys’ priors as the particular case which obtains when the probability model only depends on a single continuous univariate parameter, and its posterior distribution is asymptotically normal. Example 2 (continued). (Reference prior for a binomial parameter) Let data D ¼ {x1, . . ., xn} consist of a sequence of n independent Bernoulli trials, so that pðx j Þ ¼ x ð1 – Þ1–x , with x P f0; 1g; this is a regular, one-parameter continuous model, whose Fisher’s information function is F ðÞ ¼ – 1 ð1– Þ – 1 . Thus, the reference prior is ðÞ _ – 1=2 ð1 – Þ – 1=2 , so that it is actually the (proper) Beta
Bayesian Methodology in Statistics
231
600 500
p(θ | r, n, α, β) = Be(θ|0.5, 100.5)
400 300 200 100 0.01
0.02
0.03
0.04
θ 0.05
Figure 4 Posterior distribution of the proportion of defective items in a production batch, given the analysis of n ¼ 100 items, none of which were defective.
distribution Be(j1/2, 1/2). Since the reference algorithm is invariant under reparametrization, the reference prior of pffiffiffi
ðÞ ¼ 2arcsin is ð Þ ¼ p ðffiffiÞ= ffi j@ =@=j ¼ 1; thus, the reference prior is uniform on the variance-stabilizing transformation ðÞ ¼ 2arcsin , a general feature under regularity conditions. In terms of , the reference P posterior is ðjDÞ ¼ ðj r ; nÞ ¼ Beð jr þ 1=2; n – r þ 1=2Þ, where r ¼ xj is the number of positive trials. Suppose, for example, that n ¼ 100 randomly selected items from a production batch have been tested for a particular defect and that all tested negative so that r ¼ 0. The reference posterior distribution of the proportion of items with the defect is then the Beta distribution Be(j0.5, 100.5) represented in Figure 4. Thus, on the basis of the assumed model and the observed experimental results, one may claim that the proportion of defective items is surely smaller than 5% (for the reference posterior probability of the event > 0.05 is 0.001), is smaller than 0.01 with probability 0.844 (area of the shaded region in Figure 4), it is equally likely to be over or below 0.23% (for the median, represented by a vertical line, is 0.0023), and the probability that a person randomly chosen from the population is infected is 0.005 (the posterior mean, represented in the figure by a black circle), since Prðx ¼ 1jr ; nÞ ¼ E½jr ; n ¼ 0:005. If a particular point estimate of is required (say a number to be quoted in the summary headline) the intrinsic estimator suggests itself (see Section 1.08.5); this is found to be ¼ 0.0032 (represented in the figure with a white circle). Notice that the traditional solution to this problem, based on the asymptotic behavior of the MLE, here ˆ ¼ r =n ¼ 0 for any n, makes absolutely no sense in this scenario. 1.08.4.1.2
One nuisance parameter The extension of the reference prior algorithm to the case of two parameters follows the usual mathematical procedure of reducing the problem to a sequential application of the established procedure for the single parameter case. Thus, if the probability model is pðtj; Þ; P Y; P L, and a -reference prior ð; jM ; P Þ is required, the reference algorithm proceeds in two steps:
(1) Conditional on , pðtj; Þ only depends on the nuisance parameter and, hence, the one-parameter algorithm may be used to obtain the conditional reference prior ð j; M ; P Þ. (2) If ð j ; M ; P Þ is proper, this may beR used to integrate the nuisance parameter, thus obtaining the oneparameter integrated model pðtjÞ ¼ L pðtj; Þð j; M ; P Þd , to which the one-parameter algorithm may be applied again to obtain ðjM ; P Þ. The -reference prior is then ð; jM ; P Þ ¼ ð j; M ; P ÞðjM ; P Þ, and the required reference posterior is ðjtÞ _ pðtjÞðjM ; P Þ. If the conditional reference prior is not proper, then the procedure is performed within an increasing sequence fLi g of subsets converging to L over which ( j) is integrable. This makes it possible to obtain a corresponding sequence of -reference posteriors fi ðjtg for the quantity of interest , and the required reference posterior is the corresponding intrinsic limit ðjtÞ ¼ limi i ðjtÞ. A -reference prior is then defined as a positive function ð; Þ, which may be formally used in Bayes’ theorem as a prior to obtain R the reference posterior, that is for any sufficient t P T (which may well be the whole data set D) ðjtÞ _ L pðtj ; Þ ð; Þd . The approximating sequences should be consistently chosen within a given model. Thus, given a probability model fpðxj wÞ; wPOg, an appropriate approximating sequence fOi g should be chosen for the whole parameter space O. If the analysis is done in terms of, say,
232
Bayesian Methodology in Statistics
y ¼ f 1 ; 2 gP(O), the approximating sequence should be chosen such that i ¼ ðOi Þ. A natural approximating sequence in location-scale problems is f; log gP½ – i; i2 . The -reference prior does not depend on the choice of the nuisance parameter ; thus, for any ¼ ð; Þ such that (, ) is a one-to-one function of (, ), the -reference prior in terms of (, ) is simply ð; Þ ¼ ð; Þ=j@ð; Þ=@ð; Þj, the appropriate probability transformation of the -reference prior in terms of (, ). Notice, however, that the reference prior may depend on the parameter of interest; thus, the -reference prior may differ from the -reference prior unless either is a piecewise one-to-one transformation of , or is asymptotically independent of . This is an expected consequence of the fact that the conditions under which the missing information about is maximized are not generally the same as the conditions which maximize the missing information about an arbitrary function ¼ ð; Þ. The nonexistence of a unique ‘noninformative prior’ that would be appropriate for any inference problem within a given model was established by Dawid et al.31 when they showed that this is incompatible with consistent marginalization. Indeed, if given the model pðDj; Þ, the reference posterior of the quantity of interest , ð jDÞ ¼ ðjtÞ, only depends on the data through a statistic t whose sampling distribution, pðtj; Þ ¼ pðtjÞ, only depends on , then one would expect the reference posterior to be of the form ðjtÞ _ ðÞpðtjÞ for some prior (). However, examples were found where this cannot be the case if a unique joint ‘noninformative’ prior was to be used for all possible quantities of interest. Example 9. (Regular two-dimensional continuous reference prior functions) If the joint posterior distribution of (, ) is asymptotically normal, then the -reference prior may be derived in terms of the corresponding Fisher’s information matrix, Fð; Þ. Indeed, if F ð; Þ ¼
F ð; Þ
Fq ð; Þ
F ð; Þ
F ð; Þ
! and
Sð; Þ ¼ F – 1 ð; Þ
ð42Þ
then the unrestricted -reference prior is ð; jM ; P 0 Þ ¼ ð jÞðÞ, where 1=2
ð jÞ _ F ð; Þ; PL
ð43Þ
If ð j Þ is proper, Z ðÞ _ exp
L
– 1=2 ð jÞ log½S ð; Þ d ;
PY
ð44Þ
If ð j Þ is not proper, integrations are performed on an approximating sequence fLi g to obtain a sequence fi ð jÞi ðÞg (where i ð jÞ is the proper renormalization of ð jÞ to Li Þ, and the -reference prior ð; Þ 1=2 – 1=2 is defined as its appropriate limit. Moreover, if (1) both F ð; Þ and S ð; Þ factorize, so that – 1=2
S
ð; Þ _ f ðÞg ð Þ;
1=2
F ð; Þ _ f ðÞ g ð Þ
ð45Þ
and (2) the parameters and are variation independent, so that L does not depend on , then the -reference prior is simply ð; Þ ¼ f ðÞg ð Þ, even if the conditional reference prior ð jÞ ¼ ð Þ _ g ð Þ (which will not depend on ) is actually improper. Example 3 (continued). (Reference priors for the normal model) The information matrix that corresponds to a normal model Nðxj; Þ is F ð; Þ ¼
–2
0
0
2 – 2
0
! ;
Sð; Þ ¼ F – 1 ð; Þ ¼ @
2 0
0
1
1 2A 2
ð46Þ
pffiffiffi 1=2 hence F ð; Þ ¼ 2 – 1 ¼ f ðÞg ðÞ with g ðÞ ¼ – 1 , and thus ðjÞ ¼ – 1 . Similarly, – 1=2 S ð; Þ ¼ – 1 ¼ f ðÞ g ðÞ, with f ðÞ ¼ 1, and thus ðÞ ¼ 1. Therefore, the -reference prior is
Bayesian Methodology in Statistics
233
ð; jM ; P 0 Þ ¼ ðjÞðÞ ¼ – 1 , as already anticipated. Moreover, as one would expect from the fact that Fð; Þ is diagonal and also anticipated, it is similarly found that the -reference prior is ð; jM ; P 0 Þ ¼ – 1 , the same as before. Suppose, however, that the quantity of interest is not the mean or the standard deviation , but the standardized mean ¼ =. Fisher’s information matrix in terms of the parameters and is Fð ; Þ ¼ J t Fð; ÞJ , where J ¼ ð@ð; Þ=@ð ; ÞÞ is the Jacobian of the inverse transformation; this yields F ð ; Þ ¼
1
–1
–2
–1
ð2 þ 2 Þ
! ;
0
1 2 B1þ 2 B Sð ; Þ ¼ @ 1 – 2
1 1 – C 2 C 1 2 A 2
ð47Þ
– 1=2
1=2 Thus, S
ð ; Þ _ ð1 þ 12 2 Þ – 1=2 and F ð ; Þ _ – 1 ð2 þ 2 Þ1=2 . Hence, using again the results in Example 9, ð ; jM ; P 0 Þ ¼ ð1 þ 12 2 Þ – 1=2 – 1 . In the original parametrization, this is ð; jM ; P 0 Þ ¼
–1 ð1 þ 12 ð=Þ2 Þ – 1=2 – 2 , which is very different from ð; j M ; P 0 Þ ¼ ð; j M ; P 0 Þ ¼ P . 1 2 – 1=2 The corresponding reference posterior of is ð jx1 ; . . . ; xn Þ _ ð1 þ 2 Þ pðt j Þ where t ¼ xj = P 1=2 xj2 , a one-dimensional (marginally sufficient) statistic whose sampling distribution, pðtj; Þ ¼ pðtj Þ,
only depends on . Thus, the reference prior algorithm is seen to be consistent under marginalization. 1.08.4.1.3
Many parameters The reference algorithm is easily generalized to an arbitrary number of parameters. If the model is pðtj!1 ; . . .; !m Þ, a joint reference prior ðm jm – 1 ; . . .; 1 Þ ð2 j1 Þ ð1 Þ
ð48Þ
may sequentially be obtained for each ordered parametrization f1 ðwÞ; . . .; m ðwÞg of interest, and these are invariant under reparametrization of any of the i ðwÞs. The choice of the ordered parametrization f1 ; . . .; m g precisely describes the particular prior required, namely the prior which sequentially maximizes the missing information about each of the i s, conditional on f1 ; . . .; i–1 g for i ¼ m; m – 1; . . .; 1. Example 10. (Stein’s paradox) Let D be a random sample from an m-variate normal distribution with mean m ¼ {1, . . ., m} and unitary variance matrix. The reference prior that corresponds to any permutation of the i s is uniform, and this prior leads indeed to appropriate reference posterior distributions for any P of the i s, namely pffiffiffi x i ; 1= nÞ. Suppose, however, that the quantity of interest is ¼ i 2i , the squared distance ði jDÞ ¼ N ði j of m to the origin. As shown by Stein,32 the posterior distribution of based on that uniform prior (or in any ‘flat’ proper approximation) has very undesirable properties; this is due to the fact that a uniform (or nearly uniform) prior, although ‘noninformative’ with respect to each of the individual i s, is actually highly informative on the sum of their squares, introducing a severe positive bias (Stein’s paradox). However, the reference prior that corresponds to a parametrization of the form f; 1 ; . . .; m–1 g produces, for any choice of the P nuisance parameters i ¼ i ðmÞ, the reference posterior ðjDÞ ¼ ðjt Þ _ – 1=2 2 ðntjm; nÞ, where t ¼ i xi 2 , and this posterior is shown to have the appropriate consistency properties. Far from being specific to Stein’s example, the inappropriate behavior in problems with many parameters of specific marginal posterior distributions derived from multivariate ‘flat’ priors (proper or improper) is indeed very frequent. Hence, sloppy, uncontrolled use of ‘flat’ priors (rather than the relevant reference priors) is very strongly discouraged. 1.08.4.1.4
Limited information Although often used in contexts where no universally agreed prior knowledge about the quantity of interest is available, the reference algorithm may be used to specify a prior, which incorporates any acceptable prior knowledge; it suffices to maximize the missing information within the class P of priors which is compatible with such accepted knowledge. Indeed, by progressive incorporation of further restrictions into P , the reference
234
Bayesian Methodology in Statistics
prior algorithm becomes a method of (prior) probability assessment. As described below, the problem has a fairly simple analytical solution when those restrictions take the form of known expected values. The incorporation of other type of restrictions usually involves numerical computations. Example 11. (Univariate restricted reference priors) If the probability mechanism which is assumed to have generated the available data only depends on a univariate continuous parameter P Y R, and the class P of acceptable priors is a class of proper priors which satisfies some expected value restrictions, so that P ¼
Z pðÞ; pðÞ > 0; pðÞ d ¼ 1; Y
Z Y
gi ðÞpðÞ d ¼ i ; i ¼ 1; . . .; m
ð49Þ
then the (restricted) reference prior is ðjM ; P Þ _ ðjM ; P 0 Þ exp
hXm
g ðÞ j ¼1 i i
i
ð50Þ
where ðjM ; P 0 Þ is the unrestricted reference prior and the is are constants (the corresponding Lagrange multipliers) to be determined by the restrictions which define P . Suppose, for instance, that data are considered to be a random sample from a location model centered at , and it is further assumed that E½ ¼ 0 and Var½ ¼ 20 . The unrestricted reference prior for any regular location problem may be shown to be uniform, so that here ðjM ; P 0 Þ ¼ 1. Thus, the restricted reference prior must be of the form R R 2 2 and ðjM ; P Þ _ expf1 þ 2 ð – 0 Þ2 g, with Y ðjM ; P Þ d ¼ 0 Y ð – 0 Þ ðjM ; P Þ d ¼ 0 . Hence, this is the normal distribution with the specified mean and variance, ðjM ; P Þ ¼ Nðj0 ; 0 Þ.
1.08.4.2
Frequentist Properties
Bayesian methods provide a direct solution to the problems typically posed in statistical inference; indeed, posterior distributions precisely state what can be said about unknown quantities of interest given available data and prior knowledge. In particular, unrestricted reference posterior distributions state what could be said if no prior knowledge about the quantities of interest were available. A frequentist analysis of the behavior of Bayesian procedures under repeated sampling may, however, be illuminating, for this provides some interesting connections between frequentist and Bayesian inference. It is found that the frequentist properties of Bayesian reference procedures are typically excellent, and may be used to provide a form of calibration for reference posterior probabilities. 1.08.4.2.1
Point estimation It is generally accepted that, as the sample size increases, a ‘good’ estimator q˜ of q ought to get the correct value of q eventually, that is to be consistent. Under appropriate regularity conditions, any Bayes estimator f of any function f(q) converges in probability to f(q), so that sequences of Bayes estimators are typically consistent. Indeed, it is known that if there is a consistent sequence of estimators, then Bayes estimators are consistent. The rate of convergence is often best for reference Bayes estimators. It is also generally accepted that a ‘good’ estimator should be admissible, that is not dominated by any other estimator in the sense that its expected loss under sampling (conditional to q) cannot be larger for all q values than that corresponding to another estimator. Any proper Bayes estimator is admissible; moreover, as established by Wald,33 a procedure must be Bayesian (proper or improper) to be admissible. Most published admissibility results refer to quadratic loss functions, but they often extend to more general loss functions. Reference Bayes estimators are typically admissible with respect to reasonable loss functions. Notice, however, that many other apparently intuitive frequentist ideas on estimation have been proved to be potentially misleading. For example, given a sequence of n Bernoulli observations with parameter resulting in r positive trials, the best unbiased estimate of 2 is found to be r(r 1)/{n(n 1)}, which yields ˜ 2 ¼ 0 when r ¼ 1; but to estimate the probability of two positive trials as zero, when one positive trial has been observed, is less than sensible. In marked contrast, any Bayes reference estimator provides a reasonable answer. For example, the intrinsic
Bayesian Methodology in Statistics
235
estimator of 2 is simply ( )2, where is the intrinsic estimator of described in Section 1.08.5.1. In particular, if r ¼ 1 and n ¼ 2 the intrinsic estimator of 2 is (as one would naturally expect) ( )2 ¼ 1/4.
1.08.4.2.2
Interval estimation As the sample size increases, the frequentist coverage probability of a posterior q-credible region typically converges to q so that, for large samples, Bayesian credible intervals may (under regularity conditions) be interpreted as approximate frequentist confidence regions: under repeated sampling, a Bayesian q-credible region of q based on a large sample will cover the true value of q approximately 100q% of times. Detailed results are readily available for univariate problems. For instance, consider the probability model fpðDjwÞ; wPOg; let ¼ ðwÞ be any univariate quantity of interest, and let t ¼ tðDÞPT be any sufficient statistic. If q ðtÞ denotes the 100q% quantile of the posterior distribution of , which corresponds to some unspecified prior, so that Z Pr½ q ðtÞjt ¼
q ðtÞ
ðjtÞ d ¼ q
ð51Þ
then the coverage probability of the q-credible interval {; q(t)}, Z Pr½q ðtÞ jw ¼
q ðtÞ
pðtjwÞ dt
ð52Þ
is such that Pr½q ðtÞ jw ¼ Pr½ q ðtÞjt þ Oðn – 1=2 Þ
ð53Þ
This asymptotic approximation is true for all (sufficiently regular) positive priors. However, the approximation is better, actually O(n1), for a particular class of priors known as (first-order) probability matching priors. For details on probability matching priors see Datta and Sweeting34 and references therein. Reference priors are typically found to be probability matching priors, hence they provide this improved asymptotic agreement. As a matter of fact, the agreement (in regular problems) is typically quite good even for relatively small samples. Example 12. (Product of normal means) Consider the case where independent random samples {x1, . . ., xn} and {y1, . . ., ym} have respectively been taken from the normal densities N ðxj!1 ; 1Þ and N ðyj!2 ; 1Þ, and suppose that the quantity of interest is the product of their means, ¼ !1 !2 (e.g., one may be interested in inferences about the area of a rectangular piece of land, given measurements {xi} and {yj} of its sides). Notice that this is a simplified version of a problem that is often encountered in the sciences, where one is interested in the product of several magnitudes, all of which have been measured with error. Using the procedure described in Example 9, with the natural approximating sequence induced by ð!1 ; !2 ÞP½ – i; i2 , the -reference prior is found to be ð!1 ; !2 jM ; P 0 Þ _ ðn !21 þ m !22 Þ – 1=2
ð54Þ
very different from the uniform prior !1 ð!1 ; !2jM ; P 0 Þ ¼ !2 ð!1 ; !2jM ; P 0 Þ ¼ 1, which should be used to make objective inferences about either !1 or !2. The prior (!1, !2) may be shown to provide approximate agreement between Bayesian credible regions and frequentist confidence intervals for ; indeed, this prior (with m ¼ n) was originally suggested by Stein in an unpublished report to obtain such an approximate agreement. The same example was later used by Efron35 to stress the fact that, even within a fixed probability model fpðDjwÞ; w P Og, the prior required to make objective inferences about some function of the parameters
¼ (w) must generally depend on the function . For further details on the reference analysis of this problem, see Berger and Bernardo.25 The numerical agreement between reference Bayesian credible regions and frequentist confidence intervals is actually perfect under special circumstances. Indeed, as Lindley36 pointed out, this is the case in those problems of inference which may be transformed to location-scale problems.
236
Bayesian Methodology in Statistics
Example 3 (continued). (Inference on normal parameters) Let D ¼ {x1, . . ., xn} be a random sample from a normal distribution N ðxj; Þ. As mentioned pffiffiffiffiffiffiffiffiffi before, the reference posterior of the quantity of interest is the Student distribution Stðj x ; s= n – 1; n – 1Þ. Thus, pffiffiffiffiffiffiffiffiffi normalizing , the posterior distribution of t ðÞ ¼ n – 1ð x – Þ=s, as a function of given D, is the standard Student Stðtj0; 1; n – 1Þ with n 1 degrees of freedom. In contrast, this function t is recognized to be precisely the conventional t statistic, whose sampling distribution is well known to also be a standard Student with n 1 degrees of freedom. It follows that, for all sample sizes, posterior reference credible intervals for given the data will be numerically identical to frequentist confidence intervals based on the sampling distribution of t. A similar result is obtained in inferences about the varaince. Thus, the reference posterior distribution of ¼ – 2 is the gamma distribution Gað jðn – 1Þ=2; ns 2 =2Þ and, hence, the posterior distribution of r ¼ ns 2 =2 , as a function of 2 given D, is a (central) 2 with n 1 degrees of freedom. But the function r is recognized to be a conventional statistic for this problem, whose sampling distribution is well known to also be 2 with n 1 degrees of freedom. It follows that, for all sample sizes, posterior reference credible intervals for 2 (or any oneto-one function of 2) given the data will be numerically identical to frequentist confidence intervals based on the sampling distribution of r.
1.08.5 Inference Summaries From a Bayesian viewpoint, the final outcome of a problem of inference about any unknown quantity is nothing but the corresponding posterior distribution. Thus, given some data D and conditions C, all that can be said about any function w of the parameters which govern the model is contained in the posterior distribution pðwjD; CÞ, and all that can be said about some function y of future observations from the same model is contained in its posterior predictive distribution pðyjD; CÞ. Indeed, Bayesian inference may technically be described as a decision problem where the space of available actions is the class of those posterior probability distributions of the quantity of interest which are compatible with accepted assumptions. However, to make it easier for the user to assimilate the appropriate conclusions, it is often convenient to summarize the information contained in the posterior distribution by (1) providing values of the quantity of interest which, in the light of the data, are likely to be ‘close’ to its true value and by (2) measuring the compatibility of the results with hypothetical values of the quantity of interest which might have been suggested in the context of the investigation. In this section, those Bayesian counterparts of traditional estimation and hypothesis testing problems are briefly considered. 1.08.5.1
Estimation
In one or two dimensions, a graph of the posterior probability density of the quantity of interest (or the probability mass function in the discrete case) immediately conveys an intuitive, ‘impressionist’ summary of the main conclusions which may possibly be drawn on its value. Indeed, this is greatly appreciated by users, and may be quoted as an important asset of Bayesian methods. From a plot of its posterior density, the region where (given the data) a univariate quantity of interest is likely to lie is easily distinguished. For instance, all important conclusions about the value of the gravitational field in Example 3 are qualitatively available from Figure 2. However, this does not easily extend to more than two dimensions and, besides, quantitative conclusions (in a simpler form than that provided by the mathematical expression of the posterior distribution) are often required. 1.08.5.1.1
Point Estimation Let D be the available data, which are assumed to have been generated by a probability model fpðDjwÞ; w P Og, and let q ¼ qðwÞ P Y be the quantity of interest. A point estimator of q is some function ˜ of the data q˜ ¼ qðDÞ which could be regarded as an appropriate proxy for the actual, unknown value of q. Formally, to choose a point estimate for q is a decision problem, where the action space is the class Y of possible
Bayesian Methodology in Statistics
237
q values. From a decision-theoretic perspective, to choose a point estimate q˜ of some quantity q is a decision to act as though q˜ were q, not to assert something about the value of q (although desire to assert something simple may well be the reason to obtain an estimate). As prescribed by the foundations of decision theory ˜ qÞ measuring the (Section 1.08.2), to solve this decision problem it is necessary to specify a loss function Lðq; ˜ consequences of acting as if the true value of the quantity of interest were q, when it is actually q. The expected posterior loss if q˜ were used is q˜ jD ¼ L½
Z Y
˜ qÞpðqjDÞ dq Lðq;
ð55Þ
and the corresponding Bayes estimator q is that function of the data, q ¼ q ðDÞ, which minimizes this expectation. Example 13. (Conventional Bayes estimators) For any given model and data, the Bayes estimator obviously depends on the chosen loss function. The loss function is context-specific, and should be chosen in terms of the anticipated uses of the estimate; however, a number of conventional loss functions have been suggested for those situations where no particular uses are envisaged. These loss functions produce estimates which may be regarded as simple descriptions of the location ˜ qÞ ¼ ðq˜ – qÞt ðq˜ – qÞ, then of the posterior distribution. For example, if the loss function is quadratic, so that Lðq; the Bayes estimator is the posterior mean q ¼ E½qjD, assuming that the mean exists. Similarly, if the loss ˜ qÞ ¼ 0 if q˜ belongs to a ball or radius centered in q and Lðq; ˜ qÞ ¼ 1 function is a zero-one function, so that Lðq; otherwise, then the Bayes estimator q tends to the posterior mode as the ball radius tends to zero, assuming ˜ Þ ¼ c 1 𘠖 Þ if ˜ , and that a unique mode exists. If is univariate and the loss function is linear, so that Lð; ˜ otherwise, then the Bayes estimator is the posterior quantile of order c2 =ðc1 þ c2 Þ, so that ˜ Þ ¼ c 2 ð – Þ Lð; Pr½ < ¼ c 2 =ðc 1 þ c2 Þ. In particular, if c1 ¼ c2, the Bayes estimator is the posterior median. The results derived for linear loss functions clearly illustrate the fact that any possible parameter value may turn out be the Bayes estimator: it all depends on the loss function describing the consequences of the anticipated uses of the estimate. Example 14. (Intrinsic estimation) Conventional loss functions are typically noninvariant under reparametrization. It follows that the Bayes estimator f of a one-to-one transformation f ¼ fðqÞ of the original parameter q is not necessarily fðq Þ (the univariate posterior median, which is invariant, is an interesting exception). Moreover, conventional loss functions focus on the ‘distance’ between the estimate q˜ and the true value q, rather than on the ‘distance’ between the probability models they label. Inference-oriented loss functions directly focus on how different
the ˜ li Þ; li PL and probability model pðDjq; lÞ is from its closest approximation within the family pðDjq; ˜ qÞ, defined as typically produce invariant solutions. An attractive example is the intrinsic discrepancy, ðq; the minimum logarithmic divergence between a probability model labeled by q and a probability model labeled ˜ When there are no nuisance parameters, this is given by by q.
˜ qÞ ¼ min ðqjqÞ; ˜ ˜ ; ðq; ðqjqÞ
Z ðqi jqÞ ¼
pðtjqÞ log T
pðtjqÞ dt pðtjqi Þ
ð56Þ
where t ¼ tðDÞPT is any sufficient statistic (which may well be the whole data set D). The definition is easily extended to problems with nuisance parameters; in this case, ˜ q; lÞ ¼ min ðq; ˜ li ; q; lÞ ðq; li PL
ð57Þ
˜ and the loss measures the logarithmic divergence from pðtjq; lÞ of its closest approximation with q ¼ q, function now depends on the complete parameter vector (q, l). Although not explicitly shown in the notation, the intrinsic discrepancy function typically depends on the sample size n; indeed, when the data consist R of a random sample D ¼ fx1 ; . . .; x n g from some model pðxjqÞ, then ðqi jqÞ ¼ n X pðxjqÞlog½pðxjqÞ=pðxjqi Þdx
238
Bayesian Methodology in Statistics
so that the discrepancy associated with the full model is simply n times the discrepancy which corresponds to a single observation. The intrinsic discrepancy is a symmetric, nonnegative loss function with a direct interpretation in information-theoretic terms as the minimum amount of information which is expected to be necessary to distinguish between the model pðDjq; lÞ and its closest approximation within the class
˜ li Þ; li PL . Moreover, it is invariant under one-to-one reparametrization of the parameter of interest pðDjq; q, and does not depend on the choice of the nuisance parameter l. The intrinsic estimator is naturally obtained by minimizing the reference posterior expected intrinsic discrepancy Z Z ˜ dðqjDÞ ¼
L
Y
˜ q; lÞðq; ljDÞ dq dl ðq;
ð58Þ
Since the intrinsic discrepancy is invariant under reparametrization, minimizing its posterior expectation produces invariant estimators. For further details on intrinsic point estimation see Bernardo and Jua´rez37 and Bernardo.38 Example 2 (continued). (Intrinsic estimation of a binomial parameter) In the estimation of a binomial proportion , given data D ¼ (n, r), the Bayes reference estimator associated with the quadratic loss (the corresponding posterior mean) is E½jD ¼ ðr þ 12 Þ=ðn þ 1Þ, while the quadratic lossbased estimator of, say, the log-odds ðÞ ¼ log½=ð1 – Þ, is found to be E½ jD ¼ ðr þ 12Þ – ðn – r þ 12Þ (where ðxÞ ¼ d log½ðxÞ=dx is the digamma function), which is not equal to ðE½ j DÞ. The intrinsic loss function in this problem is ˜ Þ ¼ n minfðjÞ; ˜ ˜ ði jÞ ¼ log þ ð1 – Þ log 1 – ð; ðjÞg; i 1 – i
ð59Þ
and the corresponding intrinsic estimator is obtained by minimizing the expected posterior loss R ˜ ¼ ð; ˜ ÞðjDÞ d. The exact value of may be obtained by numerical minimization, but a very dðjDÞ good approximation is given by ðr þ 13 Þ=ðn þ 23Þ. Since intrinsic estimation is an invariant procedure, the intrinsic estimator of the log-odds will simply be the log-odds of the intrinsic estimator of . As one would expect, when r and n r are both large, all Bayes estimators of any well-behaved function ðÞ will cluster around ðE½jDÞ. 1.08.5.1.2
Interval estimation To describe the inferential content of the posterior distribution of the quantity of interest pðqjDÞ, it is often convenient to quote regions R Y of given probability under pðqjDÞ. For example, the identification of regions containing 50, 90, 95, or 99% of the probability under the posterior may be sufficient to convey the general quantitative messages implicit in pðqjDÞ; indeed, this is the intuitive basis of graphical representations of univariate distributions like thoseRprovided by boxplots. Any region R Y such that R pðqjDÞ dq ¼ q (so that, given data D, the true value of q belongs to R with probability q) is said to be a posterior q-credible region of q. Notice that this immediately provides a direct intuitive statement about the unknown quantity of interest q in probability terms, in marked contrast to the circumlocutory statements provided by frequentist confidence intervals. Clearly, for any given q, there are generally infinitely many credible regions. A credible region is invariant under reparametrization; thus, for any q-credible region R of q, ðRÞ is a q-credible region of ¼ ðqÞ. Sometimes, credible regions are selected to have minimum size (length, area, volume), resulting in highest probability density (HPD) regions, where all points in the region have larger probability density than all points outside. However, HPD regions are not invariant under reparametrization: the image ðRÞ of an HPD region R will be a credible region for , but will not generally be HPD; indeed, there is no compelling reason to restrict attention to HPD-credible regions. In one-dimensional problems, posterior quantiles are often used to derive credible regions. Thus, if q ¼ q ðDÞ is the 100q% posterior quantile of , then R ¼ f; q g is a one-sided, typically unique q-credible region, and it is invariant under reparametrization. Indeed, probability centered q-credible regions of the form R ¼ f; ð1–qÞ=2 ð1þqÞ=2 g are easier to compute, and are often quoted in preference to HPD regions.
Bayesian Methodology in Statistics
239
Example 3 (continued). (Inference on normal parameters) In the numerical example about the value of the gravitational field described in Figure 2(a), the interval [9.788, 9.829] in the unrestricted posterior density of g is a HPD, 95%-credible region for g. Similarly, the interval [9.7803, 9.8322] in Figure 2(b) is also a 95%-credible region for g, but it is not a HPD. Decision theory may also be used to select credible regions. Thus, lowest posterior loss (LPL) regions are defined as those where all points in the region have smaller posterior expected loss than all points outside. Using the intrinsic discrepancy as a loss function yields intrinsic credible regions which, as one would expect from an invariant loss function, are coherent under one-to-one transformations. For details see Bernardo.39,40 The concept of a credible region for a function q ¼ qðwÞ of the parameter vector is trivially extended to prediction problems. Thus, a posterior q-credible region for xP is a subset R of the sample space with R posterior predictive probability q, so that R pðxjDÞd x ¼ q (see e.g., Example 6).
1.08.5.2
Hypothesis Testing
The reference posterior distribution pðqjDÞ of the quantity of interest q conveys immediate intuitive information on those values of q which, given the assumed model, may be taken to be compatible with the observed data D, namely those with a relatively high probability density. Sometimes, a restriction q P Y0 Y of the possible values of the quantity of interest (where Y0 may possibly consist of a single value q0) is suggested in the course of the investigation as deserving special consideration, either because restricting q to Y0 would greatly simplify the model, or because there are additional, context-specific arguments suggesting that q P Y0 . Intuitively, the hypothesis H0 X fq P Y0 g should be judged to be compatible with the observed data D, if there are elements in Y0 with a relatively high posterior density. However, a more precise conclusion is often required and, once again, this is made possible by adopting a decision-oriented approach. Formally, testing the hypothesis H0 X fqPY0 g is a decision problem where the action space has only two elements, namely to accept (a0) or to reject (a1) the proposed restriction. To solve this decision problem, it is necessary to specify an appropriate loss function, L(ai, q), measuring the consequences of accepting or rejecting H0 as a function of the actual value q of the vector of interest. Notice that this requires the statement of an alternative a1 to accepting H0; this is only to be expected, for an action is taken not because it is good, but because it is better than anything else that has been imagined. posterior loss of accepting, R Given data D, the optimal action will be to reject H0 if (and only if) the expected R Lða ; qÞpðqjDÞ is larger than the expected posterior loss of rejecting, Lða ; qÞpðqjDÞ dq, that is if (and dq, 0 1 Y Y only if) Z Z Y
½Lða0 ; qÞ – Lða1 ; qÞpðqjD dq¼
Y
LðqÞpðqjDÞ dq > 0
ð60Þ
Therefore, only the loss difference LðqÞ ¼ Lða 0 ; qÞ – Lða 1 ; qÞ, which measures the advantage of rejecting H0 as a function of q, has to be specified. Thus, as common sense dictates, the hypothesis H0 should be rejected whenever the expected advantage of rejecting H0 is positive. A crucial element in the specification of the loss function is a description of what is actually meant by rejecting H0. By assumption, a0 means to act as if H0 were true, that is as if qPY0 , but there are at least two options for the alternative action a1. This may either mean (1) the negation of H0, that is to act as if q‚Y0 or, alternatively, it may rather mean (2) to reject the simplification implied by H0 and to keep the unrestricted model, qPY, which is true by assumption. Both options have been analyzed in the literature, although it may be argued that the problems of scientific data analysis where hypothesis testing procedures are typically used are better described by the second alternative. Indeed, an established model, identified by H0 X fq P Y0 g, is often embedded into a more general model, fq P Y; Y0 Yg, constructed to include possibly promising departures from H0, and it is required to verify whether presently available data D are still compatible with q P Y0 , or whether the extension to qPY is really required. Example 15. (Conventional hypothesis testing) Let pðqjDÞ; q P Y, be the posterior distribution of the quantity of interest, a0 be the decision to work under the restriction q P Y0 , and a1 be the decision to work under the complementary restriction q‚Y0 . Suppose, moreover, that the loss structure has the simple, zero-one form given by fLða0 ; qÞ ¼ 0; Lða1 ; qÞ ¼ 1g if qPY0
240
Bayesian Methodology in Statistics
and, similarly, fLða0 ; qÞ ¼ 1; Lða 1 ; qÞ ¼ 0g if q‚Y0 , so that the advantage L(q) of rejecting H0 is 1 if q‚Y0 and it is 1 otherwise. With this loss function, it is immediately found that the optimal action is to reject H0 if (and only if) Prðq‚Y0jDÞ > PrðqPY0jDÞ. Notice that this formulation requires that PrðqPY0 Þ > 0, that is the hypothesis H0 has a strictly positive prior probability. If q is a continuous parameter and Y0 has zero measure (e.g., if H0 consists of a single point q0), this requires the use of a nonregular ‘sharp’ prior concentrating a positive probability mass on Y0. For details see Kass and Raftery41 and references therein. Example 16. (Intrinsic hypothesis testing) Again, let pðqjDÞ; qPY, be the posterior distribution of the quantity of interest, a0 be the decision to work under the restriction qPY0 , and a1 be the decision to keep the general, unrestricted model qPY. In this case, the advantage LðqÞ of rejecting H0 as a function of q may safely be assumed to have the form LðqÞ ¼ ðY0 ; qÞ – , for some > 0, where (1) ðY0 ; qÞ is some measure of the discrepancy between the assumed model pðDjqÞ and its closest approximation within the class fpðDjq0 Þ; q0 PY0 g, such that ðY0 ; qÞ ¼ 0 whenever qPY0 , and (2) is a context-dependent utility constant which measures the (necessarily positive) advantage of being able to work with the simpler model when it is true. Choices for both ðY0 ; qÞ and which may be appropriate for general use will now be described. For reasons similar to those supporting its use in point estimation, an attractive choice for the function ðY0 ; qÞ is an appropriate extension of the intrinsic discrepancy; when there are no nuisance parameters, this is given by ðY0 ; qÞ ¼ inf minfðq0 jqÞ; ðqjq0 Þg q0 PY0
R
ð61Þ
where ðq0jqÞ ¼ T pðtjqÞ logfpðtjqÞ=pðtjq0 Þg dt , and t ¼ tðDÞPT is any sufficient statistic, which may well be the whole dataset D. As before, if the data D ¼ fx 1 ; . . .; x n g consist of a random sample from pðxjqÞ, then Z ðq0 jqÞ ¼ n
pðxjqÞ log X
pðxjqÞ dx pðxjq0 Þ
ð62Þ
Naturally, the loss function ðY0 ; qÞ reduces to the intrinsic discrepancy ðq0 ; qÞ of Example 14 when Y0 contains a single element q0. Besides, as in the case of estimation, the definition is easily extended to problems with nuisance parameters, with ðY0 ; q; lÞ ¼
inf
q0 PY0 ; l0 PL
ðq0 ; lo ; q; lÞ
ð63Þ
The hypothesis H0 should be rejected if the posterior expected advantage of rejecting is Z Z dðY0 jDÞ ¼
L
Y
ðY0 ; q; lÞðq; ljDÞ dq dl >
ð64Þ
for some > 0. As an expectation of a nonnegative quantity, d(Y0, D) is obviously nonnegative. Moreover, if
¼ ðqÞ is a one-to-one transformation of q, then dð ðY0 Þ; DÞ ¼ dðY0 ; DÞ so that, as one should clearly require, the expected intrinsic loss of rejecting H0 is invariant under reparametrization. It may be shown that, as the sample size increases, the expected value of d(Y0, D) under sampling tends to one when H0 is true, and tends to infinity otherwise; thus d(Y0, D) may be regarded as a continuous, positive measure of how inappropriate (in loss of information units) it would be to simplify the model by accepting H0. In traditional language, d(Y0, D) is a test statistic for H0 and the hypothesis should be rejected if the value of d(Y0, D) exceeds some critical value . In sharp contrast to conventional hypothesis testing, this critical value is found to be a context-specific, positive utility constant , which may precisely be described as the number of information units which the decision maker is prepared to lose to be able to work with the simpler model H0, and does not depend on the sampling properties of the probability model. The procedure may be used with standard, continuous regular priors even in sharp hypothesis testing, when Y0 is a zero-measure set (as would be the case if q is continuous and Y0 contains a single point q0). Naturally, to implement the test, the utility constant which defines the rejection region must be chosen. Values of d(Y0, D) of about 1 should be regarded as an indication of no evidence against H0, since this is
Bayesian Methodology in Statistics
241
precisely the expected value of the test statistic d(Y0, D) under repeated sampling from the null. It follows from its definition that d(Y0, D) is the reference posterior expectation of the log-likelihood ratio against the null. Hence, values of d(Y0, D) of about log[12] 2.5, and log[150] 5 should be respectively regarded as an indication of mild evidence against H0, and significant evidence against H0. In the canonical problem of testing a value ¼ 0 for the mean of a normal distribution with known variance (see below), these values correspond to the observed sample mean x, respectively lying 2 or 3 posterior standard deviations from the null value 0. Notice that, in sharp contrast to frequentist hypothesis testing, where it is hazily recommended to adjust the significance level for dimensionality and sample size, this provides an absolute scale (in information units) which remains valid for any sample size and any dimensionality. For further details on intrinsic hypothesis testing see Bernardo and Rueda42 and Bernardo and Pe´rez.43 Example 17. (Testing the value of a normal mean) Let the data D ¼ {x1,. . ., xn} be a random sample from a normal distribution Nðx j ; Þ, where is assumed to be known, and consider the problem of testing whether these data are compatible or not with some specific sharp hypothesis H0 X f ¼ 0 g on the value of the mean. The conventional approach to this problem requires a nonregular prior which places a probability mass, say p0, on the value 0 to be tested, with the remaining 1 p0 probability continuously distributed over R. If this prior is chosen to be ðj 6¼ 0 Þ ¼ N ðj0 ; 0 Þ, Bayes theorem may be used to obtain the corresponding posterior probability, Pr½0 jD; ¼
B01 ðD; Þp0 ð1 – p0 Þ þ p0 B01 ðD; Þ
n 1=2 1 n exp – B01 ðD; Þ ¼ 1 þ z2 2n þ
ð65Þ
ð66Þ
pffiffiffi where z ¼ ð x – 0 Þ=ð= nÞ measures, in standard deviations, the distance between x and 0 and ¼ 2/20 is the ratio of model to prior variance. The function B01(D, ), a ratio of (integrated) likelihood functions, is called the Bayes factor in favor of H0. With a conventional zero-one loss function, H0 should be rejected if Pr[0jD, ] < 1/2. The choices p0 ¼ 1/2 and ¼ 1 or ¼ 1/2, describing particular forms of sharp prior knowledge, have been suggested in the literature for routine use. The conventional approach to sharp hypothesis testing deals with situations of concentrated prior probability; it assumes important prior knowledge about the value of and, hence, should not be used unless this is an appropriate assumption. Moreover, see Barlett,44 the resulting posterior probability is extremely sensitive to the specific prior specification. In most applications, H0 is really a hazily defined small region rather than a point. For moderate sample sizes, the posterior probability Pr[0jD, ] is an approximation to the posterior probability Pr[0 < < 0 jD, ] for some small interval around 0 which would have been obtained from a regular, continuous prior heavily concentrated around 0; however, this approximation always breaks down for sufficiently large sample sizes. One consequence (which is immediately apparent from the last two equations) is that for any fixed value of the pertinent statistic z, the posterior probability of the null, Pr[0jD, ], tends to one as n ! 1. Far from being specific to this example, this unappealing behavior of posterior probabilities based on sharp, nonregu