Modern Applied U-Statistics
Jeanne Kowalski Division of Oncology Biostatistics Johns Hopkins University Baltimore, MD
Xin M. Tu Department of Biostatistics and Computational Biology University of Rochester Rochester. NY
m - 4
BICENTENNIAL
1807
~WILEY
:2 0 0 7
I
WILEY-INTERSCIENCE A John Wiley & Sons, Inc., Publication
This Page Intentionally Left Blank
Modern Applied U-Statistics
T H E w ILEY B ICE NTE NNIAL-KNOW LEDG E
FOR G E NE RATI ONS
~
G a c h generation has its unique needs and aspirations. When Charles Wiley first opened his small printing shop in lower Manhattan in 1807, it was a generation of boundless potential searching for an identity. And we were there, helping to define a new American literary tradition. Over half a century later, in the midst of the Second Industrial Revolution, it was a generation focused on building the future. Once again, we were there, supplying the critical scientific, technical, and engineering knowledge that helped frame the world. Throughout the 20th Century, and into the new millennium, nations began to reach out beyond their own borders and a new international community was born. Wiley was there, expanding its operations around the world to enable a global exchange of ideas, opinions, and know-how. For 200 years, Wiley has been an integral part of each generation’s journey, enabling the flow of information and understanding necessary to meet their needs and fulfill their aspirations. Today, bold new technologies are changing the way we live and learn. Wiley will be there, providing you the must-have knowledge you need to imagine new worlds, new possibilities, and new opportunities. Generations come and go, but you can always count on Wiley to provide you the knowledge you need, when and where you need it! 4
WILLIAM J. PESCE PRESIDENT AND CHIEF
EXECUTIVE OFFICER
PETER BOOTH WILEV CHAIRMAN OF THE BOARD
Modern Applied U-Statistics
Jeanne Kowalski Division of Oncology Biostatistics Johns Hopkins University Baltimore, MD
Xin M. Tu Department of Biostatistics and Computational Biology University of Rochester Rochester. NY
m - 4
BICENTENNIAL
1807
~WILEY
:2 0 0 7
I
WILEY-INTERSCIENCE A John Wiley & Sons, Inc., Publication
Copyright 02008 by John Wiley & Sons, Inc. All rights reserved Published by John Wiley & Sons, Inc., Hoboken, New Jersey. Published simultaneously in Canada. No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, electronic, mechanical, photocopying, recording, scanning, or otherwise, except as permitted under Section 107 or 108 of the 1976 United States Copyright Act, without either the prior written permission of the Publisher, or authorization through payment of the appropriate per-copy fee to the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, (978) 750-8400, fax (978) 750-4470, or on the web at www,copyright.com.Requests to the Publisher for permission should be addressed to the Permissions Department, John Wiley & Sons, Inc., 111 River Street, Hoboken, NJ 07030, (201) 748-601 1, fax (201) 748-6008, or online at http://www.wiley.com/go/permission. Limit of Liability/Disclaimer of Warranty: While the publisher and author have used their best efforts in preparing this book, they make no representations or warranties with respect to the accuracy or completeness of the contents of this book and specifically disclaim any implied warranties of merchantability or fitness for a particular purpose. No warranty may be created or extended by sales representatives or written sales materials. The advice and strategies contained herein may not be suitable for your situation. You should consult with a professional where appropriate. Neither the publisher nor author shall be liable for any loss of profit or any other commercial damages, including but not limited to special, incidental, consequential, or other damages. For general information on our other products and services or for technical support, please contact our Customer Care Department within the United States at (800) 762-2974, outside the United States at (317) 572-3993 or fax (317) 572-4002. Wiley also publishes its books in a variety of electronic formats. Some content that appears in print may not be available in electronic format. For information about Wiley products, visit our web site at www.wiley.com. Wiley Bicentennial Logo: Richard J. Pacific0
Libra y of Congress Cataloging-in-Publication Data is available.
ISBN 978-0-471-68227-1 Printed in the United States of America 1 0 9 8 7 6 5 4 3 2 1
Contents Preface
ix
1 Preliminaries 1.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.1.1 The Linear Regression Model . . . . . . . . . . . . . 1.1.2 The Product-Moment Correlation . . . . . . . . . . 1.1.3 The Rank-Based Mann-Whitney-7 ‘i :oxon Test . . . 1.2 Measurability and Measure Space . . . . . . . . . . . . . . . . 1.2.1 Measurable Space . . . . . . . . . . . . . . . . . . . . 1.2.2 Measure Space . . . . . . . . . . . . . . . . . . . . . . 1.3 Measurable Function and Integration . . . . . . . . . . . . . . 1.3.1 Measurable Functions . . . . . . . . . . . . . . . . . . 1.3.2 Convergence of Sequence of Measurable Functions . . 1.3.3 Integration of Measurable Functions . . . . . . . . . . 1.3.4 Integration of Sequences of Measurable Functions . . . 1.4 Probability Space and Random Variables . . . . . . . . . . . 1.4.1 Probability Space . . . . . . . . . . . . . . . . . . . . . 1.4.2 Random Variables . . . . . . . . . . . . . . . . . . . . 1.4.3 Random Vectors . . . . . . . . . . . . . . . . . . . . . 1.5 Distribution Function and Expectation . . . . . . . . . . . . . 1.5.1 Distribution Function . . . . . . . . . . . . . . . . . . 1.5.2 Joint Distribution of Random Vectors . . . . . . . . . 1.5.3 Expectation . . . . . . . . . . . . . . . . . . . . . . . . 1.5.4 Conditional Expectation . . . . . . . . . . . . . . . . . 1.6 Convergence of Random Variables and Vectors . . . . . . . . 1.6.1 Modes of Convergence . . . . . . . . . . . . . . . . . . 1.6.2 Convergence of Sequence of I.I.D. Random Variables . 1.6.3 Rate of Convergence of Random Sequence . . . . . . . 1.6.4 Stochastic op (.) and 0, (.) . . . . . . . . . . . . . . . . V
1 2 2 5 7
9 9 12 14 15 16 18 21 24 24 26 27 30 30 33 36 38 41 41 43 44 48
vi
CONTENTS
1.7 Convergence of Functions of Random Vectors . . . . . . . . . 50 1.7.1 Convergence of Functions of Random Variables . . . . 50 1.7.2 Convergence of Functions of Random Vectors . . . . . 55 1.8 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58 2 Models for Cross-Sectional Data 63 2.1 Parametric Regression Models . . . . . . . . . . . . . . . . . . 64 2.1.1 Linear Regression Model . . . . . . . . . . . . . . . . . 65 2.1.2 Inference for Linear Models . . . . . . . . . . . . . . . 68 2.1.3 General Linear Hypothesis . . . . . . . . . . . . . . . 81 2.1.4 Generalized Linear Models . . . . . . . . . . . . . . . 87 2.1.5 Inference for Generalized Linear Models . . . . . . . . 101 2.2 Distribution-Free (Semiparametric) Models . . . . . . . . . . 104 2.2.1 Distribution-Free Generalized Linear Models . . . . . 105 2.2.2 Inference for Generalized Linear Models . . . . . . . . 107 2.3 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115
3 Univariate U-Statistics 119 3.1 U-Statistics and Associated Models . . . . . . . . . . . . . . . 120 3.1.1 One Sample U-Statistics . . . . . . . . . . . . . . . . . 121 3.1.2 Two-Sample and General K Sample U-Statistics . . . 131 3.1.3 Representation of U-Statistic by Order Statistic . . . . 137 3.1.4 Martingale Structure of U-Statistic . . . . . . . . . . . 144 3.2 Inference for U-Statistics . . . . . . . . . . . . . . . . . . . . . 150 3.2.1 Projection of U-statistic . . . . . . . . . . . . . . . . . 152 3.2.2 Asymptotic Distribution of One-Group U-Statistic . . 157 3.2.3 Asymptotic Distribution of K-Group U-Statistic . . . 164 3.3 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 170
4 Models for Clustered Data 175 4.1 Longitudinal versus Cross-Sectional Designs . . . . . . . . . 176 4.2 Parametric Models . . . . . . . . . . . . . . . . . . . . . . . . 179 4.2.1 Multivariate Normal Distribution Based Models . . . 179 4.2.2 Linear Mixed-Effects Model . . . . . . . . . . . . . . . 184 4.2.3 Generalized Linear Mixed-Effects Models . . . . . . . 191 4.2.4 Maximum Likelihood Inference . . . . . . . . . . . . . 193 4.3 Distribution-Free Models . . . . . . . . . . . . . . . . . . . . . 198 4.3.1 Distribution-Free Models for Longitudinal Data . . . . 198 4.3.2 Inference for Distribution-Free Models . . . . . . . . . 201 4.4 Missing Data . . . . . . . . . . . . . . . . . . . . . . . . . . . 209
vii
CONTENTS
4.4.1 Inference for Parametric Models . . . . . . . . . 4.4.2 Inference for Distribution-Free Models . . . . . . 4.5 GEE I1 for Modeling Mean and Variance . . . . . . . . . 4.6 Structural Equations Models . . . . . . . . . . . . . . . 4.6.1 Path Diagrams and Models . . . . . . . . . . . . 4.6.2 Maximum Likelihood Inference . . . . . . . . . . 4.6.3 GEE-Based Inference . . . . . . . . . . . . . . . 4.7 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
. . . . . . .
. . . . . . .
.
. 211 . 214 . 223 . 230 . 230 . 237 . 241 243
Multivariate U-Statistics 249 5.1 Models for Cross-Sectional Study Designs . . . . . . . . . . . 250 5.1.1 One Sample Multivariate U-Statistics . . . . . . . . . 250 5.1.2 General K Sample Multivariate U-Statistics . . . . . . 273 5.2 Models for Longitudinal Study Designs . . . . . . . . . . . . . 279 5.2.1 Inference in the Absence of Missing Data . . . . . . . 280 5.2.2 Inference Under MCAR . . . . . . . . . . . . . . . . . 284 5.2.3 Inference Under MAR . . . . . . . . . . . . . . . . . . 294 304 5.3 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
6 Functional Response Models 6.1 Limitations of Linear Response Models . . . . . . . . . . . . 6.2 Models with Functional Responses . . . . . . . . . . . . . . 6.2.1 Models for Group Comparisons . . . . . . . . . . . . 6.2.2 Models for Regression Analysis . . . . . . . . . . . . 6.3 Model Estimation . . . . . . . . . . . . . . . . . . . . . . . . . 6.3.1 Inference for Models for Group Comparison . . . . . 6.3.2 Inference for Models for Regression Analysis . . . . . 6.4 Inference for Longitudinal Data . . . . . . . . . . . . . . . . 6.4.1 Inference Under MCAR . . . . . . . . . . . . . . . . 6.4.2 Inference Under MAR . . . . . . . . . . . . . . . . . 6.5 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
309
. 311 . 316 . 317
. 326 333 . 335 . 347 . 352 . 353 . 360 367
References
373
Subject Index
375
This Page Intentionally Left Blank
Preface This book is an introduction to the theory of U-statistics and its modern applications by in-depth examples that cover a wide spectrum of models in biomedical and psychosocial research. A prominent feature of the book is its presentation of U-statistics as an integrated body of regression-like models, with particular focus on longitudinal data analysis. As longitudinal study designs are increasingly popular in today’s research and textbooks on U-statistics theory that address such study designs are as yet non-existent, this book fills a critical void in this vital research sector. By integrating U-statistics models with regression analyses, the book unifies the two classic dueling paradigms - the U-statistics based non-parametric analysis and model-based regression analysis - t o present the theory and application of U-statistics in an unprecedented broad and comprehensive scope. The book is self-contained, although the content of the text does require knowledge of the classic statistical inference theory at a level comparable to the book by Casella and Berger (1990) and familiarity with statistics asymptotics or large sample theory. As U-statistics are not presented as an isolated entity, but rather as natural extensions of single-response based regression models to multisubject-defined functional response models, the book is structured to reflect the constant alternation between U-statistics and regression analysis. In Chapter 1, we review the theory of statistics asymptotics, which forms the foundation for the development of later chapters. In Chapter 2, we discuss regression analysis for cross-sectional data, contrast the classic likelihood-based approach with the modern estimating equation based distribution-free inference framework and explore the pros and cons of these two dueling paradigms. In Chapter 3, we introduce univariate U-statistics and discuss their properties. In Chapter 4, we revert back to regression and discuss inference for regression models introduced in Chapter 2 within the context of longitudinal data. We focus on distribution-free inference and discuss the popular generalized estimating equation (GEE) and weighted GEE (WGEE). In Chapter 5 , we return to the discussion of Ustatistics by extending the theory in Chapter 3 to multivariate U-statistics for applications to longitudinal data analysis. In Chapter 6, we introduce a new class of function response models as a unifying paradigm for regression by bringing together U-statistics based models discussed in Chapters 3 and 5 and regression models covered in Chapters 2 and 4 under a single regression-like modeling framework and discuss inference by introducing a new class of U-statistics based GEE (UGEE) and WGEE (UWGEE) under complete and missing data scenarios. We opted to use a dedicated website, ix
X
Preface
http://www.cancerbiostats.onc.j hmi.edu/Kowalski/ustatsbook, as a venue to post real data applications and software, and to continuously update both to reflect current interest and timely research. We consider this preferable to the traditional approach of including real data examples in the book, since application interests change rapidly in today’s research environment and real data examples will likely become obsolete much sooner than examples that focus on modeling principles. Likewise, we have only included some key references in the book. With the worldwide web and powerful search engines. such as Google, enabling the retrieval of references and related information at unprecedented speed and with only a few key strokes, static media, such as books, are no longer the best choice for documenting and finding references for research on a given topic. This book may be used as a text for a one-semester topic course on U-statistics by focusing on Chapters 3 , 5. and 6. This approach assumes that students have a course on generalized linear models (GLM), longitudinal data modeling, and distribution-free inference with GEE and WGEE. Alternatively, one can precede such a topic course by using Chapters 1. 2, and 4 either as a primary or secondary textbook in a one-semester course on GLM and longitudinal data analysis. The book is intended for second and third year graduate students in a Biostatistics or Statistics department, but may also serve as a self-learning text or reference for researchers interested in the theory and application of U-statistics based models. Many people have contributed to this book project. We are very grateful to Mr. Yan Ma, M s . Cheryl Bliss-Clark, Dr. Changyong Feng, Ms. Haiyan Su, Dr. Wan Tang and h4s. Qin Yu in the Department of Biostatistics and Computational Biology at the University of Rochester, and to Ms. Hua Ling Tsai and Mrs. Alla Guseynova in the Division of Biostatistics at the Sidney Kimmel Comprehensive Cancer Center at Johns Hopkins for their help with reading parts of the manuscript and helping make numerous corrections during the various stages of the manuscript development, as well as website design, construction, and development. We are especially indebted to Ms. Bliss-Clark, who painstakingly proofread the entire book numerous times and helped rewrite many parts of the book to improve the presentation, to Mr. Ma, who diligently read drafts of all chapters of the book and made countless corrections and changes to help eradicate typos and technical errors, and to Dr. ThJan Tang, who selflessly shared with us his knowledge and expertise in the theory of semiparametric models for longitudinal data and helped edit many sections and exercises of the book. We would not have finished the book without the tenacious help and dedication from these three marvelous individuals.
Preface
xi
Finally, we would like to thank Mr. Steve Quigley from Wiley for his unrelenting support of this book project, and Ms. Jacqueline Palmieri and Ms. Melissa Yanuzzi for their careful review of the manuscript and assistance with the publication of the book. Jeanne Kowalski Baltimore, Maryland Johns Hopkins University Xin Tu Rochester, New York University of Rochester October 2007
This Page Intentionally Left Blank
Chapter 1
Preliminaries This chapter serves to provide a systematic review of the concepts and tools that are fundamental to the development of the asymptotic and U-statistics theories for inference of statistical models introduced in the book. Since the majority of concepts and techniques discussed build upon the theory of probability, it is necessary to introduce and discuss them within the progression from the theory of probability to statistics asymptotics. This chapter comprehensively covers a spectrum of topics in probability theory in order to form a foundation for the development of statistics asymptotics utilized throughout the book. We distinguish statistics asymptotics or the large sample behavior of statistics from other asymptotics in that statistics asymptotics studies the convergence of random variables and vectors, as opposed to a sequence of fixed quantities, as in the study of asymptotics. In the interest of space, most technical results will be presented without proof (some left as exercises). Readers familiar with statistics asymptotics may opt to skip this chapter except for Section 1.1 without loss of continuity, and instead use the materials contained within as a reference when reading subsequent chapters. The concepts and techniques presented focus on investigating the large sample behavior of statistics, particularly within the context of parameter estimates that arise from statistical models. Because of the frequent difficulties encountered in applying and extending such techniques, we begin this chapter by motivating discussion through examples that require asymptotic theory to address a wide range of problems arising in biomedical and psychosocial research. We then highlight the fundamental concepts and results that underlie the foundation of statistics asymptotics and its applications to inference of statistical models. Among the topics reviewed, those pertaining 1
2
CHAPTER 1 PRELIMINARIES
to boundedness and convergence of sequences of independently and identically distributed, or i.i.d., random variables play a particularly important role in the theoretical development of subsequent chapters.
1.1 Introduction Asymptotic theory has an indispensable role in the inference of modern statistical models. What is statistics asymptotic theory? In brief, it is the study of the behavior of statistics, particularly model estimates, as the sample size approaches infinity. What is the advantage to applying such large sample theories? First, statistics asymptotic theory provides a powerful and unified theoretical framework to systematically simplify otherwise algebraically messy and often intractable computations in order to enable inference based on parameters of interest. Second, statistics asymptotics provides a theoretical basis for studying the properties of estimates of statistical models, such as bias and efficiency (or precision) to aid in the development of robust and optimal estimates, as well as to select such estimates among competing alternatives. Third, statistics asymptotics enables the study of distributions of statistics arising from statistical models and, as such, provides a framework in which to develop inference for such models t o facilitate both data and power analysis. In this section, we illustrate the importance of statistical asymptotic theory with some prototype models that are highlighted throughout the book. These models will be frequently visited in later chapters, along with their extensions to address more complex data types and study designs.
1.1.1 The Linear Regression Model Regression models are widely used to model an outcome of interest, or dependent variable, as a function of a set of other variables, or independent variables, in almost all areas of research, including biomedicine, epideniiology, psychology, sociology, economics and public health. The dependent variable, or response, may be continuous or discrete. The independent variables are also referred to as explanatory variables, predictors, or covariates, depending on their roles in the statistical model. A clarification on the taxonomy used to delineate different types of independent variables is presented in Chapter 2. When the dependent variable is continuous, linear regression is the most popular choice for modeling a relationship with other variables.
3
1.1 INTRODUCTION
Consider a study with n subjects. Let yi denote a continuous response T and xi = (xil,xi'. . . . , xip) a p x 1 vector of independent variables from the ith subject (1 5 i 5 n). The linear regression for modeling yi as a function of xi is defined by the following model statement: T
y i = xi /3
+~
i ,
N
i.i.d.
N (0, 0 ' ) , 1 5 i 5 n
(1.1)
r where p = (pl, p2,. . . , p p ) is the vector of parameters, ~i denotes the error term in modeling yi through xTP, N ( p , 02) denotes a normal distribution with mean p and variance u2. In (l.l),~i 4 . i . d . N (0, g2) means that ~i are independently and identically distributed with the same normal distribution N ( 0 , ~ ' ) . The linear function q i = xT,B is often called the linear predictor. In most applications, zil = 1 so that the linear predictor of the model includes an intercept, that is,
, maximum Given the distribution assumption for the error term ~ i the likelihood method is widely used to estimate and make inference about p. Let T T = (Xl,. .. ,xn) Y = (Yl,.. . , y n > ,
x
p
Then, the maximum likelihood estimate (MLE) of ,El is = (XTX)-' XTy (see Chapter 2 for details). Further, under the assumption of a constant matrix X, the sampling distribution of is a multivariate normal (see Chapter 2 for details):
p
Note that we do not distinguish estimate and estimator and will use estimate consistently throughout the book. The assumption of a constant X is reasonable for some studies, especially controlled experiments. For example, in a study to compare two treatments with respect to some response of interest y, we first determine the sample size for each treatment and then assign patients to one of two treatment conditions. We can use two binary indicators to code the treatment received by each patient as:
xik
1 if treatment = k =
0 if otherwise
, k=l,2,
ISiSn
(1.3)
4
CHAPTER 1 PRELIMINARIES
With the sample size nk (k = l , 2 ) for each treatment fixed, the design matrix X with xi = (xi1,xi2)~ is a constant and p1 and p2 in the linear predictor q i = P1xil xi2p2 repres_ent the mean response of y within each treatment condition. The MLE P in this particular case reduces to
+
p = (&,&)
T
1
= @I., T72.)T, where V k . =
( k = 1 , 2 ) . The sampling distribution of exercise) :
ci=l zircyi and nk ci=l n
n
=
Xik
in (1.2) also simplifies to (see
When u2 is known, the above distribution describes the variability of the estimates V k . of the mean responses of the two treatment groups. In most applications, however, u2 is unknown. By estimating u2 using the following pooled sample variance, n
1 n-2
-2 = -
2=1
2
n-2
i=l
we can use the bivariate t distribution (see Chapter 2 for details),
for inference about P. For example, we can use (1.4) or (1.5) to test whether there is any between-group difference, that is, PI - p2 = 0, depending on whether u2 is known or estimated. However, in many applications, especially in observational studies arising in epidemiological and related research, the response yi and independent variables xi are typically observed concurrently. For such study designs, it is not sensible to hold x, fixed when considering the sampling variability of yi. Even with controlled randomized trials, such as the example for comparing two treatment conditions discussed above, it is not always possible to have a constant design matrix. For example, if the between-treatment difference is significant in that example, we may want to know whether such a difference varies across subgroups defined by different demographic and comorbid conditions such as age, gender, and race. Such moderation analyses have important implications in biomedical and psychosocial research for determining treatment specificity and deriving optimal treatment regimes
5
1.1 INTRODUCTION
(Baron and Kenny, 1986). Within the context of the study for comparing two treatments, if we want to test whether age moderates the treatment effect, we can fit the model in (1.1)with the linear predictor, 0 qi = P1.il
-k x t 2 P 2
+ xi3P3 + xt2xi3P4
where xi3 is the covariate of subject’s age and z i k ( I c = 1 , 2 ) are defined in (1.3). If p4 # 0, then age is a moderator. Since moderation analyses are often post-hoc (after treatment difference is established) and moderators can be discrete as well as continuous variables, it is practically impossible to fix their values prior to subjects’ recruitment. When X varies from sample to sample, the matrix (XTX) in the sampling distribution (1.2) changes as a function of the sampling process. The sampling-dependent variance matrix of the estimate not only violates the definition of a sampling distribution, but more importantly, it leaves open the question of whether (1.2) is valid for inference about p. This issue is addressed in greater detail in Chapter 2 by utilizing the theory of statistics asymptotics. The normal distribution in the linear regression (1.1) places a great restriction on applications of the model to real study data for inference. In recent years, many distribution-free modeling alternatives have been developed to address this critical limitation. These alternative approaches do not assume any parametric distribution for the response and thus apply to a much wider class of data types and distributions arising in real study applications. In the absence of a parametric assumption imposed on the distribution of the response, likelihood-based approaches, such as the maximum likelihood, cannot be applied for inference of model parameters; in this case, statistics asymptotics plays a critical role in the development of alternative approaches to inference. We will discuss such distribution-free regression models and inference procedures for cross-sectional study designs in Chapter 2 and longitudinal study designs in Chapter 4.
1.1.2
The Product-Moment Correlation
Correlation analysis is widely used in a range of applications in biomedical and psychosocial research for assessing rater reliability, gene expression relations, precision of diagnosis, and accuracy of proxy outcomes (Barnhart et al. 2002; King and Chinchilli, 2001; Schroder et al. 2003; Kowalski et al. 2004; Kowalski et al. 2007; Tu et al. 2007). Although various types of correlation have been used, the product-moment correlation is the most popular and is a major focus within the context of correlation analysis throughout this
6
CHAPTER 1 PRELIMINARIES
book. Thus, the word correlation is synonymous to the product-moment correlation within this book unless otherwise noted. Like regression, correlation analysis is widely used to assess the strength of a linear relationship between a pair of variables. In contrast t o linear regression, correlation analysis does not distinguish between a dependent and an independent variable and is particularly useful for modeling dynamic relationships among concurrent events and correlates of medical and psychiatric conditions, such as heart disease and depression, and for assessing rater agreement in diagnoses, test-retest reliability in instrument development and fidelity of psychotherapy intervention. Let (xi,yi) denote an i.i.d. sample of bivariate continuous outcomes (1 5 i 5 n ) . The product-moment correlation is defined as:
where Cov ( 2 ,y) denotes the covariance between z and y and V a r (u)the variance of u (u= z or y). If we assume a bivariate normal distribution for the i.i.d. pairs, (xi,yi), the maximum likelihood estimate of p is readily
A
where U.= Er=lui (u= z or y). The above estimate, known as Pearson’s correlation, is widely used as a measure of association between two variables. However, unlike linear regression, the exact distribution of i; no longer has a known parametric form. Statistics asymptotic theory is required to study the sampling variability of i; for large samples. More generally, as in the discussion of the linear regression model, the normal assumption may not apply to outcomes arising from real study data, making it necessary to develop inference procedures for p without the bivariate normal assumption imposed on the data distribution. A key difference between linear regression discussed in Section 1.1.1and the product-moment correlation defined in (1.6) is that linear regression models the conditional (on covariates) mean response, or first order moment of y, given 2 , while the product-moment correlation centers on cross-variable variability between n: and y, as measured by the second order moments of 2 and y. This difference is not confined to linear regression and in fact, most regression models including the more general generalized linear models
1.1 INTRODUCTION
7
are based upon modeling the mean or first order moment of a response of interest (see Chapter 2 for details). Thus, the correlation model represents a significant departure from this mainstream modeling paradigm, in terms of modeling high order moments. Of note, the difference in modeling of moments carries over to drastically different inference paradigms. Indeed, as we demonstrate in further detail in Chapters 3, the typical approach to asymptotics-based inference theory developed for mean-based models, such as linear regression, does not in general apply to higher order moment based models. Although standard asymptotic approaches as typified by direct applications of the law of large numbers and central limit theorem may still be applied to study the sampling variability of the Pearson estimate (see Section 1.7), such applications require much more work both computationally and analytically. A more elegant solution is to utilize the theory of U-statistics. As we discuss in Chapters 5 and 6, such an alternative approach also allows us to generalize this popular measure of association to longitudinal study designs, as well as to effectively address missing data.
1.1.3 The Rank-Based Mann-Whitney-Wilcoxon Test When comparing two treatment groups, one can use the linear regression model as discussed in Section 1.1.1 provided that the data are normally distributed. In the presence of severe departure from normality, especially when the data distribution is highly skewed, the Mann-Whitney-Wilcoxon rank sum test is a popular alternative to regression. This test compares the distributions of the response of interest between two groups based on the rankings of the values of the response so that no analytical distribution is imposed upon the response, at least in regard to its shape or form. Thus, the test applies to a much wider range of data distributions. Consider a study with two treatment groups with nk subjects constituting the kth sample ( k = 1 , 2 ) . Let yki be two i.i.d. samples from some continuous response of interest (1 5 i 5 n k , k = 1 , 2 ) . We are interested in testing whether there is any difference in mean response between the two samples. However, unlike the regression setting, we do not assume analytic form for the distributions of the two samples, except that they differ by a location shift, that is, if Fl is the cumulative distribution function (CDF) of yli, the CDF of yzj is given by: F2 (y) = F1 (y - 6'), where 6' is some unknown parameter (see Section 1.5 for definition of CDF). The MannWhitney-Wilcoxon test for the equality of distributions, that is, 6' = 0, is based on rank scores that are created as follows. First, the responses from the two groups y k i are combined and ordered
8
CHAPTER 1 PRELIMINARIES
from the smallest to the largest. If there are ties among the observations, they are arbitrarily broken. The ordered observations are then assigned rank scores based on their rankings. For tied observations, average rankings are assigned to them. For example, if n1 = 5 and 7x2 = 6, and the observations of y l i and y2j are given by yli : 4, 7, 3, 15, 1 ;
y2j :
10, 14, 9, 3, 17, 24
then the ordered observations of combined groups when arranged from the smallest (left) to the largest (right) are given as follows: 1, 3, 3, 4, 7, 9, 10, 14, 15, 17, 24
The rank scores of the ordered observations are the rankings of the ordered sequence starting from 1 for the smallest observation with tied observations assigned the average rankings, that is, 1
72 +2 3 , 2 + 3 , 4, 5, 6, 7, 8, 9, 10, 11 2
The rank scores R k i for the two groups are given by
Note that for two continuous variables y l i and yaj, the probability of having a tie either within the variable or between the variables is 0 (see Section 1.5). Although an improbable event in theory, it is possible to have tied observations in practice because of limited precision in measurement. Wilcoxon (1945) and Mann-Whitney (1947) independently developed two statistics for testing the equality of the distributions of yki based on the rank scores R k i for the observations from the two groups. More specifically, Wilcoxon proposed the rank sum statistic: Wilcoxon rank sum test :
Wn =
2
Rli
i=l
Note that the sum of the rank scores Eyzl Rzj from the second group may also be used as a statistic. However, it is readily shown (see exercise) that W, CyL, R2j = where n = n1 n2. Thus, only one of the
+
v,
+
9
1.2 MEASURABILITY AND MEASURE SPACE
sums of the rank scores can be used as a statistic. Mann-Whitney is 121
Mann-Whitney test :
The test initiated by
n2
U, =
I{y23-ylolo) (1.7) i=l j=1
where I{,lo} is a set indicator with I{,lo)= 1 if u 5 0 and 0 if otherwise. in the absence of ties (see exercise), the two Since W, = U, tests are equivalent (see Chapter 3 ) . Throughout this book, we will use the Mann-Whitney form of the test and refer to it as the Mann-WhitneyWilcoxon rank sum test. To use the test in (1.7), we must find the sampling distribution of U,. Although it is possible to find the exact distribution (see Chapter 3), we are also interested in the behavior of the statistic in large samples, since such behavior will enable the study of efficiency and extend the test to a more general setting (see Chapter 5 and 6). However, unlike the linear regression model and product-moment correlation, it is more difficult to study the asymptotic behavior for such a statistic, as it cannot be expressed in the usual form of an independent sum of random variables. In Chapter 3, we will carefully characterize the asymptotic behavior of this statistic and generalize it in Chapters 5 and 6 to address design issues arising in modern study trials by using the theory of U-statistics.
+
1.2
Measurability and Measure Space
Measurable sets and measures of such sets are the most basic elements of the theory of probability. In this section, we review these fundamental concepts.
1.2.1
Measurable Space
Let R be a sample space and F some collection of subsets of R. A a-field or a-algebra F is a collection of subsets of R, which has the following defining properties: 1. 4 E F (empty set is contained in F ) . 2. If A E F , then A" E f ( F is closed under complementation). 3. If Ai E f ,then Uzl Ai E F (f is closed under countable unions). Note that under (2) fl E F iff (if and only if) E f . Thus, (1) may be equivalently stated as: 1'. R E f (empty set is contained in f )
10
CHAPTER 1 PRELIMINARIES
The pair S = (R. F) is called a measurable space. So, a measurable space is a sample space equipped with some a-field. For any sample space R;there always exist two special a-fields. One is F = {R,q5} and the other is f = {all subsets of R including 4 and R}, representing the smallest and largest a-field, respectively. The latter is often called the power set of R. Example 1. Consider rolling a die. There are six possible outcomes, 1. 2, , , . ,6, the die will register when thrown. Let
R = {1,2,3,4,5,6}, F = {all subsets of R including R and #} Then, R is the sample space containing all possible outcomes from a toss of the die and the power set f is a a-field. Thus, S = (R, F) is a measurable space. Example 2. In Example 1, consider a collection of subsets of R defined by: F2 = { 4 , R , {1,3,5) , {2,4,6}) It is readily checked that F2 is also a a-field. Thus, S = (R, F2) is again a measurable space. When compared to f in Example 1, this a-field has fewer subsets. In other words, F2 is contained in F, that is, f 2 C f . Example 3. Let R = R,the real line, and
Then, F2 and F3 are both a-fields. Clearly, F3 C F2. The a-fields in all three examples above are quite simple, since each has only a finite number of subsets. In general, it is impossible to enumerate the members in a a-field if it contains infinitely many subjects. One way of constructing a a-field in this case is to use the following result. Proposition. The intersection of a-fields is a a-field. Let A denote a collection of subsets of R. To construct a a-field including all subsets in A, we can take the intersection of all the a-fields containing A. Since the power set of R (the collection of all subjects of R) is always a a-field (see exercise), the intersection always exists. This smallest a-field is also called the a-field generated by A, denoted by a ( A ) . Example 4. Let R = R and the Bore1 a-field B be the a-field generated by any one of the following class of intervals:
1.2 MEASURABILITY A N D MEASURE SPACE
11
More generally, let R = Rk = ( ( 2 1 , ..., zk) ;xiE R, 1 5 i 5 k } be the kdimensional Euclidean space. The Borel a-field Bk is the a-field generated by any one of the following class of subsets:
{&,
( - m , a i ] ; a iE R } ,
{@f=l(az,bi];ai,bz E R}
(1.9)
where @$=,A,= A1 8 A2 . . . @ A k denotes the Cartesian product of sets such that {(XI,..., zk) : xi E Ai, 1 5 i 5 k } . The Borel a-field contains all open and closed sets. For example, let 0 C Rk be some open set. For any point a E 0, the defining properties of an open set ensure that we can find an open interval 0, C 0 that contains a. Since the set of rational numbers is dense, we can find a rational number in 0, t o index such an open interval. The collection of all such open intervals indexed by a subset of the rational numbers Q covers all points in 0. Since Q is countable, 0 is a countable union of open intervals. It thus follows from (3) of the definition of a-field that 0 E B k . To show that any closed set is in B', simply note that the complement of a closed set is an open set. The a-field is an abstract notion of information and is a building block of probability theory and its vast applications in all areas of research. In many experiments and studies, we may not be interested in every single outcome in the sample space and we can use a-field to represent the information pertaining to our interest. For example, if we only care whether the outcome registered by a toss of a die is an odd or even number, the simpler a-field F2 in Example 2 suffices to communicate this interest. The defining properties of a a-field ensure that we can perform meaningful operations regarding the pieces of information in a a-field as well. Example 5 . In Example 2, let A = {{1,3,5}}. This collection has a single subset communicating the interest of observing an odd outcome when tossing the die. However, A does not contain sufficient information to describe all possible outcomes, since the event of even outcomes, {2,4,6}, is not in A. The a-field generated by A, F2 = {d, R, {1,3,5 } , {2,4,6}}, has all the information needed to describe all potential outcomes in terms of the event of interest. For relatively simple sample spaces like the one in the above example, we can readily construct a a-field containing the events of interest. For more complex sample spaces, especially those with an infinite number of elements
12
CHAPTER 1 PRELIMINARIES
such as R, it is difficult to assemble such a a-field through enumerating all its subsets. However, for any events of interest represented by a collection of subsets A, we can always find a a-field containing A, such as the one generated by A, a (d), as guaranteed by the Proposition.
1.2.2
Measure Space
Given a measurable space S = (a, f ) , let p define a mapping or function from f to R U {m}, with the following properties: 1. p ( A ) 2 0 for any A E f . 2. (Countable additivity) If Ai E f is a disjoint sequence for i = 1 , 2 , . . ., then 00
If R has a finite number of elements, the countable additivity property becomes p (U21Ai) = C:Il p (Aj). Such a mapping p is called a measure. A measure space is defined by adding p to S and denoted by the triplet M = (0,L 4 . Thus, a measure is defined for each element of the a-field f , that is, subsets rather than each individual element of the sample space 0. This distinction is especially important when R has an uncountable number of elements such as the real line R. If a 0-field represents the collection of information to describe some outcome of interest, a measure then allows us to quantify the pieces of information in the a-field in a certain fashion according to our interest. For example, in the theory of probability, such a probability measure is used t o quantify the occurrence of event as represented in a a-field in the study of random phenomena (see Section 1.4 for the definition of probability measure). Example 6 . Within the context of Example 1, define a set function p as follows: p ( A ) = IA/, A E f where lAl denotes the number of elements (singletons) in A. Then, p is a measure. This measure p assigns 1 to each singleton. In general, a measure may be constructed by assigning a value to each singleton, and the measure of a subset is then the sum of the measures of all its members. The countable additivity of such a measure follows directly from the additivity of arithmetic addition. For example, t o check
13
1.2 MEASURABILITY A N D MEASURE SPACE
the countable additivity property (or finite additivity for this example, since R is finite), consider A = {1,2) = (1) U (2). Then, we have I-1 ( A ) = P ((1))+ P ((2)) = 2
Since p ( A ) counts the number of elements in A, it is called the count measure. Note that by restricting p to f p in Example 2, p becomes a measure for the measurable space S = (R, f 2 ) . Example 7. Consider the number of surgeries performed in a hospital over a period of time. It is difficult to put an upper bound on such an outcome. As a result, this outcome of interest has a theoretical range from 0 to 00. Thus, an appropriate sample space for the number of surgeries is the countable space containing the non-negative integers, R = (0,1,2,. . .}. Let f be the a-field containing all subsets of R including R itself and 4, that is, the power set of R. The measure defined in Example 6 may be extended to the countable space. It is readily checked that p ( A U B ) = p ( A )+ p ( B )for any A, B E f with AnB = 4. In addition, the countable additivity also holds true. Thus, p is a well-defined measure. However, unlike the count measure in Example 6, p ( R ) = co. Since R is a countable union of measurable sets with finite measure (e.g. R = U:=o { k ) ) , p is called a a-finite measure. For a sample space with a finite or countable number of elements, we can define a measure by first limiting it for each element, as in Examples 5-7. For a sample space with an uncountable number of elements, such as R, this approach generally does not work. For example, for the Bore1 measurable space S = ( R , B ) ,B contains all different types of intervals as well as points. It is impossible to develop a measure by first defining its values for all individual points of the sample space R. This is because a proper interval such as (0,4) contains an uncountable number of points and as such it is not possible to extend such a point-based function to a measure that is defined for proper intervals using the defining countable additivity property. It is also generally difficult to define a measure by assigning its values to all subsets in a a-field directly because such assignments must satisfy the relationship given by the following theorem. Theorem 1. Given a measure space M = (a, f ,p ) , we have 1. (Continuity from below) If A, E f , A E f and A, T A, then
T (4. (Continuity from above) If A, E f , A E f and L p ( A ) , provided that at least one p ( A , ) is finite.
P (An) P
2.
p(A,) 3. (Continuity subadditivity) If Ai
E f ,then p
A,
A, then
(U,"=,Ai) 5 Cgl p (Ai).
14
CHAPTER 1 PRELIMINARIES
4. (Rule of total measure) If {Ci E f ;i = 1 , 2 , . . .} is a partition of a, then p ( A )= Czl p ( An Ci). A2 C . . . G A, G . . . Note that A, 1' A ( A , J A ) means that A1 and A = Ai, while p (A,) I' p ( A ) ( p (A,) J p ( A ) )implies that p (A,) increases (decreases) to p ( A ) . Because of these difficulties, a measure for a a-field is generally constructed by first defining it for a class of subsets that generates the a-field. For example, if a class of subsets A is a ring, that is, A U B E A (closed under pairwise union) and A i'7 BCE d (closed under relative complement) for any A, B E A, then any measure defined on A may be uniquely extended to the a-field generated by it. This is known as the Caratheodory's extension theorem (e.g., Ash, 1999). Example 8. Consider again the measurable space S = (R',B k ). Since Bk is generated by subsets of the form A = { (ai, bi] ; ai,bi E R} and d is a ring, it follows that we can create a measure p on 0' if we first define it for such sets. For example, if we measure each (ai, bi] by its volume
UZl
n (bi
@tzl @t=l
k
-
ai),we obtain the Lebesgue measure for B'.
i=l
Example 9. In Example 8, let Ic = 1 and consider S = (R, B ) . Let F be some non-decreasing, right continuous function F on R. For intervals of the form ( a , b ] , let p ((a, b ] ) = F ( b ) - F ( a ) . Then, by Example 8, p can be uniquely extended to the Bore1 a-field B. In particular, if F ( z ) = exp (-ax) if IC 2 0 and 0 if otherwise, where a is a known positive constant, this Finduced measure gives rise to the exponential distribution (see Section 1.4 for details about distribution functions).
1.3
Measurable Function and Integration
Functions and their integration form the foundation of mathematical science. The Riemann integral is the first rigorous definition of integration of a function over an interval in R or subset in R ' and has been widely used in applications, including statistics. However, Riemann integration has several limitations. First, as the definition of Riemann integral relies on R k , it is difficult to extend the notion of integration to general measurable spaces. Second, within the Euclidean space, a wide class of functions of theoretical interest is not Riemann integrable, that is, the integral does not exist. In this section, we review measurable functions and their integrals, which address the fundamental limitations of Riemann integration.
15
1.3 MEASURABLE FUNCTION AND INTEGRATION
1.3.1 Measurable Functions Consider two measurable spaces, S1 = (R1, F i ) and S2 = (R2, f 2 ) . mapping from R1 to R2, T : 01 + R2 is measurable F i / F 2 if
T-~A E
fl,
for any A E
f 2
A
(1.10)
where T - l A = { w E F l , T ( w ) E A } is the inverse image of A E F2 in F l . If S2 = ( R ,B ) , T is called a real measurable function and often denoted by
f.
Note that T-l should not be interpreted as the inverse mapping of T , since T is not necessarily a one-to-one function. Note also that unlike measures that are defined as set functions based on a-fields, T is a pointbased function defined for each element of the sample space R1. Let a ( T )denote the a-field generated by T , that is, a ( T )= a {T-'A,
for any A E F2)
Then, T is measurable a ( T )/F2. Thus, for any mapping T , we can always make it measurable with respect to some a-field. Example 1. Let S1 = S2 = ( R ,B ) . Let f be a continuous real function from S1 to S2. Consider any open set 0 E F2 = B. By the definition of a continuous function, f-'0 is also an open set in F l = B. Thus, f is measurable. Example 2. Consider the measurable spaces S1 = ( R , B ) and S2 = ( R ,F2) in Example 3 of Section 1.2. Define the identity mapping T from R to itself, that is, T (z) = z for any z E R. Then, T is measurable B/F2. This follows since for any A E F2, A is either 4, (-m,O], ( 0 ,m) or R, all of which are elements of B. Thus, T-'A = A E B for any A E F2. Note that it is readily checked that the reverse is not true, that is, T is not measurable Fz/B. It is inconvenient if not impossible to check measurability of a mapping by the defining condition in (1.10). For example, even for a very simple a-field F2 and a mapping in Example 2, we must inspect every element A of f 2 to ensure that T-'A E B. When Fa contains an infinite number of subsets, such as B, it is not possible to perform such a task. Fortunately, we can use the following results to circumvent the difficulties. Theorem 1. Let S k = (Rk,Fk) denote three measurable spaces (1 5 k < - 3 ) . Let A c F2, acollection of subsets contained in F2, and F2 = a ( A ) . Let G be a a-field for S1. Let TI and T2 be measurable mappings from S1 to S2 and from S2 to S,, respectively. Then,
16
CHAPTER 1 PRELIMINARIES
1. If TCIA E f l for each A E A, then TI is measurable f l / f 2 . 2. If TI is measurable Fl/f2 and f 1 c G, then TI is measurable G/F2. 3. If TI is measurable f l / F 2 , and T2 is measurable F 2 / f 3 , then the composite mapping from 01 to Q3, T2 o TI is measurable f l / f 3 . Thus, by Theorem 1/1, we can check measurability of a mapping TI from 01 to R2 by limiting the defining condition in (1.10) t o a collection of subsets generating the o-field F2. In addition, we can use Theorem 1/2 and 1/3 to construct new mappings for a given purpose. Example 3. In Example 2, let A = ( ( - 3 0 , O ] } . Then, A C f 2 . Consider again the identity mapping T ).( = x for any x € R. It is readily checked that T-'A = (-m,O] E B. Since f 2 = o ( A ) ,it follows from Theorem 1/1 that T is measurable B/f2. In comparison t o Example 2 , we only checked the measurability of T with respect to a single subset A = (-m,O] in F2. Example 4. Let S1 = S 2 = S3 = ( R , B ) . Since f(x) = sin(x) and g (x) = 1x1 are both continuous functions, it follows from Example 2 that f is measurable f l / F 2 = B / B and g is measurable f 2 / F 3 = B / B . It = g (f (x))= g o f (x)is measurable follows from Theorem 1/3 that Isin )I(. fl/F3 = B / B . In other words, lsin(z)l is a real measurable function on
R.
Theorem 2. Let S = (R, f ) denote a measurable space. Let f and g be real measurable functions. Then, P
are all measurable.
1.3.2
Convergence of Sequence of Measurable Functions
Limit and convergence are key to studying the behavior of measurable functions and their integrations. In this section, we discuss the notion of convergence of sequence of measurable functions and in particular review two popular modes of convergence. Consider a measurable space S = (R, f ) . For each n (21): let f, be a real measurable function defined on S , that is, f n is a mapping from S to ( R ,B ) and FIB measurable. For such a sequence of measurable functions, we can always construct two sequences of functions, g, = maxllks, fk and h, = minl
17
1.3 MEASURABLE FUNCTION AND INTEGRATION
element w E R, the limits gn and limn h, exist if 00 is considered to be a “real” value. We denote these limits by sup,>1 - fn = limn+oogn and inf,?l f, = h,. Based on these two limit functions we may further define two sequences f, and infn>k - f , ( k 2 1). It is straightforward t o check of functions, that these two sequences are monotone (see exercise). Thus, their limits exist, denoted as lim sup f n and lim inf f,, respectively. It is easy to show that fn converges iff limsup f, = liminf f, and the limit is denoted by limnjco f n . The following asserts that all these functions constructed are also measurable (see exercise). Theorem 3. Let S = ( R , f ) be a measurable space. Let fn be a sequence of real measurable functions. Then, 1. The following functions, jot,
are all measurable. 2. The set { w : lim sup f, = lim inf f,} is measurable. In particular, if limn+oo fn exists, then the limiting function is measurable. As will be seen in Section 1.3.3, we can alter the values of a real measurable function in a set of measure 0 without changing the integral of the function. Thus, two functions f and g that differ on a set of measure 0, that is, p ({x : f (x)# g (x)}) = 0, are not distinguishable by integration and are said to be equivalent almost everywhere (a.e.). By applying this notion of equivalence to a sequence of measurable functions, we say that f, converges to f almost everywhere, fn + f a.e., if the set on which f, does not have a limit has measure 0, that is, p ( { w : lim sup f,
# lim inf f,})
=0
Thus, under a.e. convergence, f, can have points on which limn+m f n does not exist, but the set of such points must have a measure of 0. The a.e. convergence is defined for each element R. Another measure of convergence for f, is t o look at the difference between f, and its limit f as quantified by p directly. More specifically, f, is said to converge to f in measure, denoted by f, -+p f , if for any 6 > 0, the deterministic sequence, d,,s = p - f i > S], converges to 0 as n -+ 30. Stated formally, for any 6, E > 0, we can find some integer NE.bsuch that
[Ifn
18
CHAPTER 1 PRELIMINARIES
As will be seen in Section 1.6, this measure of convergence is widely used in statistics asymptotics to study large sample behaviors of statistics and/or model estimates. The two measures of convergence for sequences of functions are not equivalent as the following example demonstrates. Example 5 . Let fnk be functions from S = ( R , B ) to itself as defined by
Let p be the Lebesgue measure on S . Then, the sequence, fll, fil, fiz, f31, f 3 2 , f33,. . . , converges to function 0 in measure since p (lfnk - 01 > 6) 5 1 n -+ 0 for any S > 0. However, the sequence does not converge a.e. and in fact, it converges nowhere since it may take values 0 and 1 for sufficiently large n. On the other hand, consider f n (x)= I{z>n)( n 2 1). Then, fn (x)+ 0 for all IC E R. However, since p (Ifn - 01 > 6) = p (x > n ) = 30 for any S > 0 ( S < l),fn does not converge in measure. In the above example, p ( R ) = ca. This is not incidental, as in the construction of the second part of the example, as the following theorem indicates. Theorem 4. Let S = ( Q F ) be a measurable space. If p ( n ) < 00, then convergence a.e. implies convergence in measure. Proof. Let fn (x)+ f(x) a.e. Then, since
it follows that p ({w : limsup,,, Ifn ( w ) - f ( w ) l > S}) = 0 for any 6 > 0. From the continuity properties of measure (with finite total measure), p ({w : supKn (w)- f(w)l> S}) -+ 0. Thus
Ifn
P ( { w : Ifn (w)- f(u)l > S>>
+
0
and fn (x)-+ f(x) in measure. Thus, a.e. convergence is generally stronger than convergence in measure.
1.3.3 Integration of Measurable Functions In Riemann calculus, the definition of integral relies critically on the metric of the Euclidean space. For example, consider integrating a function f
1.3 MEASURABLE FUNCTION A N D INTEGRATION
19
over an interval [a, b]. By partitioning the interval into n subintervals, a = a0 < a1 < ... < a, = b, we obtain a Riemann sum c y = l f (xi)(ui- ai-1) with xi E Ai = [ui-l,ai] ( 1 5 i 5 n). If such a sum converges as 6, = maxlli
+
20
CHAPTER 1 PRELIMINARIES
of f as follows:
Then, fn is a monotone increasing sequence of non-negative simple measurable functions and f = limn+m fn (see exercise). Define Ja f d p = limn+m Ja fndp. As in 2 , it can be shown that the integral does not depend on the choice of fn and thus the integral is uniquely defined (see exercise). 4. For general f, write f = f + - f-, where f+ = max(f,O) and f - = max (-f,0) are the positive and negative parts of f and both are nonnegative. The integral of f is defined as f d p = Ja f + d p - Ja f - d p , provided that at least one of Ja f + d p and Ja f - d p is finite. The integration of a measurable function f on a measurable set C E F is defined as f d p = f I A d p . If M = ( R ,B, p ) and p is the Lebesgue measure, J A f d p is called the Lebesgue integral and is often denoted by
la
lA
JA
la
f (4d x .
From the definition, it is clear that f d p = 0 if f = 0 a.e.. Thus, functions that differ on set of measure 0 have the same integrals. Example 6. For the measure space in example 7 of Section 1.2, any function on it is measurable. The integral of a real function f on a subset A is actually the sum of its values of points in A, that is,
For a finite subset A, the above is well defined. If A contains infinitely many points, C(f(i)ZO)nA and Cf(i)
Jb.,
fdp =
i0.,
IAdP = I-1 ( A )= 1
and f is Lebesgue integrable. In general, it is difficult to compute the integral of a measurable function defined on a general measure space. As in Riemann calculus, the change
1.3 MEASURABLE FUNCTION AND INTEGRATION
21
of variable formula is very useful in converting such an integration problem into the more concrete Euclidean space. Let M1 = (01,f 1 , p l ) and M2 = (a2,F2, p 2 ) be two measure spaces and h be a measurable mapping from M I t o M2. If f is a real measurable function on M2, then the composite g = f o h is a real measurable function from MIto S = (R, B). If p2 is induced by p 1 by h, then the integral of f with respect to A42 can be computed from the integral of g with respect to M2 and vice versa, as the following theorem asserts. Theorem 5 (Change of variable formula). Under the above assumptions, we have
The equality is understood as if one side of the equation exists, then both exist and agree with each other. The theorem is most useful when M2 = ( R , B , p ) . In this case, integration over an abstract measure space is changed to integration on the familiar space R. We will illustrate applications of Theorem 5 with respect to calculations of moments of random variables in Section 1.5. Most functions used in statistics are Riemann integrable. A real-valued function f on [u,b]is Riemann integrable iff it is bounded and continuous almost everywhere (that is the measure of the set of non-continuity points is 0). For such functions, the Lebesgue and Riemann integrals are equal to each other. Thus, integration of such functions with respect to the Lebesgue measure can be carried out by the usual Riemann integration calculus using powerful tools such as Newton-Leibnitz and integrate by parts formulas. The Fubini theorem may be used when integrating functions over R'. The theorem states that if the absolute value of the function is integrable, then the Lebesgue integral over Rk may be computed through iterated integrals, and the order of iteration does not matter. For example, if If ( 2 ,y)l is integrable over R2,then,
1.3.4 Integration of Sequences of Measurable Functions As seen in Section 1.3.3, functions that are equal to each other almost everywhere have the same integral. Thus, if a sequence of measurable functions, f n -+ f a.e., that is, p ({z : f n (z)+ f (x)}') = 0, we would expect that the sequence of the integral of the sequence, J f n d p + J f d p , as n + m. This
22
CHAPTER 1 PRELIMINARIES
is indeed true under some sensible assumptions. Listed below are some important properties with respect to integration of functions and sequences of functions. 1. ( L i n e a r i t y ) If f and g are integrable, then J (af bg) dp = J afdp+ J bgdp for any a , b E R. 2. If fi are integrable, then J (CElfi) dp = J fidp. 3. If f is an integrable function and Ai are disjoint measurable subsets, then A, f d p = JA, f dp. 4. J f dp 2 0 if f is integrable and f 2 0 a.e. The equality is achieved if and only if f = 0 a.e. 5. (Fatou's L e m m a ) If f, 2 0 and are integrable, liminf,,, J fndp 2 J(h inf,,, f,)dp. 6. ( M o n o t o n e Convergence The ore m) If f, are integrable and 0 5 f, t f a.e., then J f n d p J fdp. 7. ( D o m i n a t e d Convergence The ore m) If f, are integrable, f, -+ f a.e., 5 g a.e. for all n and J g d p < x,then J f,dp + J fdp. The special case of (7) in which g is constant and the integration is over a subset with finite measure, is called the bounded convergence theorem. Part (5)-(7) are particularly useful in measure and probability theories as they permit the change of the order of integration and limit. In statistic asymptotic theory, we often face the problem of integrating a sequence of r a n d o m variables (a special class of measurable functions), which is difficult to carry out because of the known measure p. However, such sequences often converge t o a simple limit, the normal random variable, and integration of such a limit is easy to compute. We discuss random variables and convergence of sequences of such variables in Section 1.6. Let M = (n,F , p ) be a measure space and f be some real measurable function defined on M . Then, by applying the integral properties above, it is readily shown that v ( E ) = & f d p is a measure for any E E f (see exercise). In addition, v is absolutely continuous with respect to p in the sense that p ( A ) = 0 implies v ( A )= 0 for any A E F . The theorem below claims that the reverse is also true. Theorem 6 ( R adon- Ni k ody m) . If v and p are a-finite measures F and v is absolutely continuous with respect to p, then there is a g 2 0 such that v ( E ) = JE g d p . The function g is called the Radon-Nikodym derivative of v relative to p. From the Radon-Nikodym theorem, all absolute continuous measures relative to the usual Lebesgue measure on the Bore1 g-field B have the form v ( E ) = J E g d p for E E B. Let F (z)be an antiderivative of g , that is,
+
zzl
2=1
lfnl
cz1
1 . 3 MEASURABLE FUNCTION A N D INTEGRATION
23
F'(x)= %d F
= g(x). Then, the measure 'u on B is determined by F ( x ) and vice versa since 'u ((a, b ] )= F ( b ) - F ( u ) . In Section 1.2, we discussed how t o construct a measure on M = ( R ,8 ,p ) from a nondecreasing, right continuous function F (x). This F-induced measure is known as the Lebesgue-Stieltjes measure. There are three basic types of F (z). 1. F (x) is a step function. The jumps can only occur at a countable number of points (see exercise). The F-induced measure at such points is positive and 0 elsewhere. Thus, such a measure concentrates on these jump points and is called the discrete measure. A discrete measure is not absolute continuous relative to the Lebesgue measure. 2. F ( x ) is absolutely continuous. For such a function, its derivative f (x)exists a.e., and further, F (x)= F ( u ) f ( t ) d t . The corresponding measure 'u is given by 'u ( E ) = f (x)dx. For such a function, F,' (z) exists and F,' (z) = 0 3. F (x) is singular. a.e., but F (x)is in general not an antiderivative of F,' (x),unless F (z) is a constant. The corresponding measure is called a singular measure on the Bore1 a-field B. In general, it has a positive measure for a subset with Lebesgue measure 0, but no point has a positive measure. The singular measure is not absolutely continuous relative to the Lebesgue measure. The following theorem states that a general F (x)can be decomposed as a sum of the three basic types of functions. Theorem 7. A right, continuous function F (x)on M = ( R ,B, p ) has the following decomposition:
'1
+ laz
Proof. The measure of a point u is F ( u ) - F (a+). Thus, a point in R has a positive measure if and only if F is discontinuous at the point. Let Fd (x) = F ( x ) - F (x+) > 0. Then, the number of such points x is countable and F (x)- Fd (x)is continuous (see exercise). Its derivative, f ( x ) = ( F ( x )- F d ( X ) ) ' , exists a.e.. Let Fa (x) be an anti-derivative of f (x),that is a function such that Fa (x)= Fa ( u ) f ( t )d t . Then, Fa (x) is absolutely continuous, and F ( x ) - Fd(x)- Fa (x)is a singular function. Note that all three terms are not necessary to appear in a decomposition. For statistical applications, F (x)is either discrete or absolutely continuous, or sometimes a combination of the two. Singular measures are usually of theoretical interest.
+ laz
24
CHAPTER 1 PRELIMINARIES
1.4 Probability Space and Random Variables 1.4.1
Probability Space
A probability space, P = (R, f ,P ) , is a special case of measure space when the measure P is a probability measure satisfying P ( R ) = 1. Thus, as in the case of general measure space, a probability measure is defined for each element of F , that is, subsets rather than each individual element of R. Likewise, P satisfies the defining as well as the additional properties of a measure given by Theorem 1 of Section 1.2. Example 1. In Example 6 of Section 1.2, we can normalize the count measure to obtain a probability measure P as follows:
1 P ( k ) = -, 1 5 k 5 6 6 P(AUB)=P(A)+P(B), forAnB=4
A,BEF
For example, to check the countable additivity property, or rather f i n i t e additivity since R is finite, consider A = {1,2} = (1) U {a}. Then, we have
+
2
P (A) = P (1) P (2) = 6 Note that by restricting P t o f a in Example 2, P becomes a probability measure for the measurable space S = (R, f 2). In general, if the total measure of the sample space is finite, we can normalize it into a probability space, that is, rescale the original measure p by a factor where p (0)is the total measure of the sample space. Example 2. In example 7 of Section 1.2, we discussed a measure space with countable elements, R = { 0 , 1 , 2 , .. .}, together with a count measure. Since the total measure of the sample space is oc,the count measure cannot be normalized to form a probability measure. To define a probability measure P , we may assign to each outcome k of 9 a value P k = P ( k ) satisfying:
h,
(1.12) k=1
Since CElppk is a convergent series, P k must converge t o O as k + 00 (see exercise). Although various choices of p k are available, the most popular is to define P as follows:
P ( k ) = -Pekx p ( - p ) , k!
p
> 0. k
=O,l,
...
(1.13)
25
1.4 PROBABILITY SPACE A N D RANDOM VARIABLES
It is readily checked that P satisfies the defining properties of a measure (see exercise). Thus, P is a well-defined probability measure and is known as the Poisson probability law. If we want to construct an absolute continuous probability measure P with respect to the Lebesgue measure, then by the Radon-Nikodym theorem, P (-00,00) = J-", f (z)d z for some non-negative function f (z), and the requirement of total probability measure of the sample space =1 means that J-", f (z)dz = 1.
(-$)
Example 3. Consider the positive function, f (z) = 1 exp . Jz;; It is readily checked that J-", f (z)dz = 1. Define a function P for each interval of the type (--00, u] by
(-$) dz Since A, = (-m,n]
A, it follows from Theorem 1/1 of Section 1.2 that
P ((-00,00)) = lim P ((-m, n])= lim n-cc
exp
(-$) dx
" 1
As 1 exp
6
xa
(--
is continuous and bounded, the integral can be completed
s-", & (-$)
exp dz = 1 by Riemann calculus. It is readily shown that (see exercise). Thus, P is a probability measure for S = (R, a). Example 4. Consider the measurable space S = (R, F2) defined in Example 3 of Section 1.1. Let
0 ifA=$
$
if A = (-00,0], or A = (0, x)
1 ifA=R
Then, P2 is a probability measure for the measurable space S = ( R ,Fa). Here: the sample space R is uncountable, but F2 contains a finite number of subsets. The probability measure Pz is defined for each of the four subsets of Fa.
26
1.4.2
CHAPTER 1 PRELIMINARIES
Random Variables
A random variable, X, is a real measurable function from a probability F ) , to the measurable space, R = ( R ,B ) . Since intervals of space, S = (a, the form ( - w , a ] (aE R ) generate B, it follows from Theorem 1 of Section 1.3 that X is a random variable iff
x-'
((-m,a])= { X
5 u}
E f , for any a E
R
(1.14)
The above often serves as the definition of a random variable. Note that we can replace (-m,a] in (1.14) by any one class of the intervals discussed in Example 4 of Section 1.2. For example, X may also be defined by the intervals of the form [a,00):
x-'
([a, 00)) = { X
2 a> E
F,
for any a E R
For a random variable X , we denote by 0 ( X ) the a-field generated by X . Example 5. Consider Example 1 of Section 1.2 for the possible outcomes resulting from tossing a die and let X be the number of dots registered on the die when tossed. Then, X is a random variable defined on the measurable space S = (R:F ) introduced in that example. To see this, consider intervals of the form (-m, a ] . It is readily checked that ifa<1
{X 5 a } =
{ 1 , 2 , . . . , [ a ] } if 1 5 a 5 6 { 1 , 2 , . . . ,6}
if a
(1.15)
>6
where la1 denotes the largest integer function, that is, la1 is an integer with a - 1 < 1. 5 a. Since subsets of the form { 1 , 2 , . . . , [ a ] }are all in B , (1.15) shows that X is a random variable. In addition, it is readily checked that .(X) = F . Example 6 . Within the context of Example 2 of 1.2, let X = 1 if the number of dots registered in a toss of the die is an even number and 0 if otherwise. Since cp
ifa
{1,3,5} if 0 5 a
R
ifa21
<1
1.4 PROBABILITY SPACE A N D RANDOM VARIABLES
27
and { 1,3,5} E F 2 , it follows that X is a random variable. Also, it is readily checked that a (X) = { { 1 , 3 , 5 } , { 2 , 4 , 6) , a, R} = F2, the a-field f 2 defined in that example. Note that X is also a random variable with respect to the measurable space S = (0,f ) in Example 1 of Section 1.1. This follows from Theorem 1/2 of Section 1.3 since a ( X 2 ) C f . Example 7. Consider a measurable space, S = ( 0 ,f ) .Let X ( w ) = c , a real function that maps every element w E R to a constant c E R. Then,
Thus, ( X ) = {a,R}, the smallest a-field. Thus, X is a random variable. This example shows that any constant is a random variable. Example 8. Let f be a continuous function from S 1 = ( R , B ) to Sz = (R, 23). Then, for any open set 0 E F2 = 23, f - l O E F1 = 23. Thus, f is a random variable defined on the measurable space S1 = ( R ,23).
1.4.3 Random Vectors Random vectors are defined in the same way. A k-dimensional random vector, X = ( X I , . . , Xk)T, is a measurable mapping from a measurable space, S = (0,F) to 72'" = ( R k ,B k ) , where Rk denotes the k-dimensional Euclidean space and 23'" the associated Bore1 a-field. Since rectangles of the form (-30,all x . . . x (-30,ah] generate a', a random vector is also defined as : k
{XI < G I , . . . , Xk
a l , . . . ,a k E R
(1.16)
As in the case of a random variable, we may replace the rectangles in (1.16) by other classes of rectangles such as [ a l ,m) x . . . x [ a k ,30) and even mixtures of different classes of rectangles such as (--oo: a11 x [a2, )..
x ... x
[ a k - l , -oo>
x [ a k , -oo)
Example 9. If X = ( X I , .. . , X k ) T is a random vector, then any of its components X i is a random variable. For example, let a,j = n for all j except j = i. Then, as
A,
=
{ X I 5 n,.. . , X i 5 a i , . . . , Xk 5 n} E f
28
CHAPTER 1 PRELIMINARIES
it follows that {Xi 5 ui} = U =:l A , E f . Theorem 1. Let S = (R, F ) denote a measurable space. Then, g is 1. Let fi : R + R (1 5 i 5 k ) and g = (fl,.. . , f k ) T . measurable f if and only if fi is measurable F / B . 2. If f is a continuous mapping from Rk to Rm, then f is measurable
/ak
B"B". 3. If X is a k-variate random vector and f is a continuous (or measurable) mapping from R' to Rm, then f (X) is measurable FIRm. Thus, by Theorem 1 the reverse in Example 10 is also true: if each of the components of X is a random variable, then X is a random vector. In addition, Theorem 1 generalizes the results in Example 9 to a general case of random vector. Example 10. Consider randomly drawing a point from the unit disk in R2. The sample space is R = { ( x , g ) : x2 y2 5 l}. Since R E B2, ( X , Y ) is a random vector with respect to f = { A ;A = R n B , B E B 2 } , the 0field B2 restricted to R. It follows from Theorem 1 that the X-coordinate, X, of ( X , Y ) is a random variable. It is easy to check that if a < -1, {X 5 a} = a, and if a > 1, then {X 5 a} = R. Example 11. Let S = (R, f ) be a measurable space. Let f (X,Y) = X Y be a mapping from R2 to R. If X and Y are random variables, then it follows from Theorem 1 that 2 = f ( X , Y ) = X Y is also a random variable. Similarly, 2 = X Y , 2 = X / Y (Y # 0) and 2 = X - Y are also random variables. Example 12. Let Xi be a finite sequence of n random variables. By Theorem 3 of Section 1.3, maxl
+
+
+
{x1< x2}= U {xl< T > n { T < x2}
(1.17)
T€Q
For each T E Q, {XI < T } and { T < Xz} are both measurable. Since B is closed under intersection and countable unions, the conclusion follows.
1.4 PROBABILITY SPACE AND RANDOM VARIABLES
29
Example 14. Let X = (XI,Xz, . . . , Xk)T be a random vector. Then, the ordered vector:
is also a random vector, where ((1) ? ( 2 ) , . . . , ( k ) } is some permutation of the integer set { 1 , 2 , . . . , k } . Let i = {il, 22, . . . ,ik} index a permutation of {1,2, . . . , k } and Pk denote the collection of all such permutations. Then, Xi = (Xi,, Xi,, . , Xik)T is a random vector and thus the subset Bi = {Xi, 5 Xi, I 5 Xik} is measurable. Then, for any al,. . . ? ak E R (see exercise), + .
The measurability of A follows. Note that since a, are not ordered, it is necessary t o intersect Bi with { X t 3 Iaj} in (1.19) to ensure the identity since otherwise, we only
n,"=,
(&
have A C & p k {Xi,I a j } ) . An alternative is to define bJ = minjgsk al. Then, it is readily checked that (see exercise)
from which it follows that A =
u [ (&
{ X i j I b j } ) ] E Bk
i E P
The ordered vector in (1.18) is a rearrangement of the original X in the order from the smallest to the largest. If Xi are i.i.d. random variables (1 5 i 5 n ) ,then (Xp)?X(2),, . , ,X(k)) is known as the order statistic.
30
C H A P T E R 1 PRELIMINARIES
1.5 Distribution Function and Expectation 1.5.1 Distribution Function Let P = (R, f ,P ) be a probability space and X a random variable defined on the measurable space (a: F ) . The cumulative distribution function of X or CDF, denoted by F (.), is defined by
F ( u ) = P ( X 5 u ) . for a E R
(1.21)
For any value a E R, F (a)yields the probability of the event that X 5 a. The CDF has the following properties 1. P ( u < 5 b) = F ( b ) - F ( a ) . 2. (Continuity from above) F (-cc) = lim,+-m F ( u ) = 0. 3. (Continuity from below) F (m) = lim,-,m F ( u ) = 1. Proof. 1. This follows from:
x
F ( b ) = P (--00 < x 5 b) = P ( { --oo < x 5 u } u { u < x 5 b } ) = P(-cc < x 5 u ) + P ( u < x 5 b ) =F(u)+P(u<X I b ) 2. Without the loss of generality, assume a i -co. Let n = [Iai], where r.1 denotes the largest integer function. It follows that (1.22) Let
A,
= {-cc
< X 5 - n } , B, = {-cc < X 5 -n+ I}
Then, as n x,A, i 4 and B, J, 4 and thus by the continuity properties of measure (Theorem 1 of Section 1.2), --f
lim P(A,) = lim P(B,) = P ( 4 )
n+-w
(1.23)
,+-Cc
It follows from (1.22) and (1.23) that lim,+-mF(u) = 0. Kote that the continuity properties of measure only apply to a countable sequence. As A, = { -m < X 5 u } is not such a sequence, it is necessary to construct the countable sequences A, and B, before applying the continuity properties of measure. 3. This follows from an argument similar to 2. Since P = (R, F , P ) is a measure space and X a real measurable function from P = (R, F , P ) to the measurable space S = (8,a),it follows from the
31
1.5 DISTRIBUTION FUNCTION AND EXPECTATION
discussion in Section 1.3.3 that the CDF defined in (1.21) is the value of the X-induced measure for the interval (-m, b ] ,
F ( b ) = px ((-m: b ] ) = P (x-l(-oo, b]) Following the discussion for general measures on S = ( R ,B ) , F also generates the same measure as px. Further, F can be decomposed into three parts consisting of a step function, an absolutely continuous function (with respect to the Lebesgue measure) and a singular function. In statistical applications, F is either a step or an absolutely continuous function, depending on whether X represents a discrete or continuous outcome. In the discrete case, F can have at most a countable number of discontinuity points x k , that is, fk = F (xk)- F (xi) > 0. k = 1,2 , . . . Thus. F in this case is completely determined by {fk: k = 1 , 2 , . . .}. which is called the probabzlzty dzstrzbutzon or mass function. If F is absolutely continuous, F (a)= f(x)dx. In this case. the probabzlzty denszty f u n c t z o n (PDF), f(x), is often used to identify the distribution. Following the discussion in Section 1.3 for general real measurable functions, the probability of a single point for a continuous X is zero. px ({x}) = 0 and the PDF is unique up to a set of Lebesgue measure 0. Note that both the probability distribution and probability density functions are real measurable functions with respect to the Bore1 a-field. Through X and F , we can calculate the probability of any event of interest in the probability space P = (R, F , P ) . This high level of abstraction makes it possible t o study any random phenomenon through the single znduced probability space P' = (R.B. px) without referring to the nature and context of the original probability space P . For this reason, we often denote the probability P ( X - ' A ) of any event A E B simply by P r ( X E A ) , that is, Pr ( X E A ) = px ( A )= P ( X - l A )
s-",
Example 1. Consider a constant random variable X known constant. Since
= c,
where c is a
F ( a ) is a step function with a jump of size 1 at a = c, which is known as the degenerate distribution. The induced probability measure in this case
32
CHAPTER 1 PRELIMINARIES
is given by
Px (A) =
1 ifcEA
0 ifc@A
Thus, a constant random variable induces a probability measure px on S = ( R , B )with a point mass at c. Example 2. Consider the probability space in Example 2 of Section 1.4, P = (R, f ,P ) , where R = { 0 , 1 , 2 , .. .}, f is the power set of R and P satisfies (1.12). Let X = k be a real function from R to R. Then, X is a random variable (see exercise). The CDF F is a step function with jumps, F ( k ) - F ( k - ) = p k , at k . If R is viewed as a subset of R,the induced probability measure px is identical to P. Nonetheless, they are conceptually different since px is defined for a completely different measurable space S = ( R ,B ) . If P satisfies (1.13), X is said to have the Poisson distribution. Example 3. Non-negative integrable functions can be used to construct distribution functions for continuous random variables. For example, consider a non-negative function f (x)= I{o
s-",
s, the normalized g
(- (S;b:)').
s-",
1 v 5 2 f (x)is a PDF of a normal distribution N ( p , c?) with mean p and variance 0 2 . Finally, consider f (x)= a e x p (-ax)I{z2o>with a > 0. Then, it follows that f (x)dx = 1, yielding the PDF of an exponential distribution. The corresponding CDF is F (x)= (1 - exp (-ax)) I{z2~). Example 4. Let F (a)be the CDF of some random variable X defined on a probability space P = (R, f P ) . Define the inverse function of F as follows: (1.24) F-' (4) = inf {z; F (x)2 q } , for any o < q < 1 =
s-",
If F (x)is strictly increasing, the above reduces to
F-' (4) = {x;F (x)= q ) , for any For any 0 < q
o
< 1, { F (4 2 4 )
=
[F-' ( 4 ) 7 .c>E
Since intervals of the form [a, m) generate B , F (x)is B measurable when viewed as a real function from S = ( R , B ) to itself. Thus, Y = F ( X )
1.5 DISTRIBUTION FUNCTION AND EXPECTATION
33
is a measurable function f / B and thus a random variable in its own right. Similarly, we can show that F-l ( Y )is also a random variable (see exercise). Example 5 . (Probability integral transformation). Let X be a continuous random variable. Then, it is readily checked that
F (F-' (4)) = q ,
for any
o
(1.25)
Note that the above is generally not true for a discrete random variable. In fact, in general, we have
F (F-'(4)) 2 q ,
for any
o
It follows from (1.25) that ~y
(y) = P r (Y 5 y) = Pr ( F ( x )5 y) = P r ( X 5 = F (F-1
F-' (y))
(y)) = y
Thus, Y = F ( X ) is a uniform random variable between [0,1]. Example 6. In Example 3, let U be a uniform random variable on [0,1]. Then, it follows from Example 4 that X = F-l ( U ) is a random variable with the CDF given by F . This result is useful for generating random variables with particular distributions, such as in simulation studies. For example, t o generate a random variable with a n exponential distribution, F (z)= 1 - exp (--:), we first generate values u from the uniform U [0,1]and then generate II: through the inverse relationship: z = F-l (u)= -310g(l - u ) .
1.5.2
Joint Distribution of Random Vectors
Let X = ( X I , .. . , X k ) T be a random vector defined on the measurable space (0,F ) . The joint CDF of X is defined by
F (a) = P r (X 5 a) = P r ( X I 5 al,...,XI, 5
Uk)
for a = (u1,u2, ..., uk)T E R k For any value a, the function F ( a ) yields the probability of the event {X 5 a}. As in the random variable case, for continuous X, we can also del , , , a a k , which is again fine the probability density function (PDF), f (a)= d aakF(a) unique up to a set of Lebesgue measure 0. For discrete X, we can similarly define the probability distribution function, fZl,...ik
= Pr
(Xl= zil,
= F (Xil
=zZk)
, '", X i k ) - F z;, ..., z;) > 0
34
CHAPTER 1 PRELIMINARIES
where {xil, ...,x i k }indexes all points with a positive mass {ij
2 1,1 5 j 5 k } .
E x a m p l e 7. Let X i denote the number of surgeries performed in a hospital in year i and X = ( X 1 , X z )T ( i = 1 , 2 ) . Then, F ( k , m ) = Pr ( X I 5 k , X2 5 m) is the probability that the hospital performed at most k and m surgeries in year 1 and 2, while fk,, = Pr ( X I = k , X z = m ) is the probability that it performed exactly k and m surgeries ( k ,rn = 1 , 2 , . . .). Similar to the random variable case, the CDF of a random vector has the following properties: 1. P (a < X 5 b) = F (b) - F (a). 2. l i m F ( a ) -+ 0 if ai -+ -co for some i (1 5 i 5 k ) . 3. F (co)= lima+m F (a)= 1 with a -+ 00 denoting a, -+ 00 for all i (1 5 i 5 k ) . As Example 9 of Section 1.4 shows, if X = ( X I , . . , Xk)T is a random vector, then any of its components X i is a random variable. In addition, let Fi denote the CDF of X i , which is known as a marginal distribution of X. Then, we have 4. lim F (a) --+ Fi (ai)if the ith component of a is fixed at ai while all the rest of a approach co. It follows from (4) that marginal distributions are completely determined by the joint CDF of X. The reverse is generally not true, except for a special class of independent random variables. To discuss such random variables, we first review the notion of independence within a collection of measurable sets as well across such collections. Two measurable sets A and B are said to be independent if Pr ( A n B ) = Pr ( A )Pr ( B ) . A finite or countable collection of such sets, A = {Ail i 2 1}, is defined to be independent if any finite subcollection of A is independent, that is, Pr Ai) = Pr (Ai),where denotes product and J some finite subset of the integers 1 , 2 , . . .. Likewise, two collections of measurable sets A and B are independent if for any A E A and B E B , A and B are independent. A finite or countable sequence of such collections {An,n 1) is independent if any finite subsequence is independent, that is, {Ak,k E J } is independent for any J indexing a finite subsequence. Let { f n , n 1 1) with f = a (An)be a sequence of a-fields generated by {An,n 2 1). The following shows that independence on {An,n 2 1) carries over to the sequence of a-fields it generates. Proposition. If the finite or countable sequence { d n , n2 l} is independent, then { f n , n 2 l} is also an independent sequence. Example 8. Consider R = { 1 , 2 : 3 , 4 , 5 , 6 } , where each point, is assigned aprobability Let A = {1,2)3}, B = {1,2,3} and C = {3,4,5,6}. Then,
(n,,
n,,,
n
>
k.
35
1 . 5 DISTRIBUTION FUNCTION A N D EXPECTATION
it is readily checked that 1 P r ( A ) = -, 2
1 P r ( B ) = -, 2
4 Pr(C) = -
6
1 P r ( A n B n C ) = P r ( A ) P r ( B ) P r ( C )= 6 But, A, B, and C are not independent since P r (A n B) # P r (A) P r ( B ) . Thus, it is important to check all subcollections for independence. Example 9. A subset A is independent with itself iff P r ( A )= 0 or 1. This follows from Pr (A) = Pr (A n A ) = P r (A) Pr (A). A finite or infinite sequence of random variables {Xi;i 2 l} is independent if the sequence of the a-fields { D (Xi) ; i 2 l} is independent. By the Proposition, we can equivalently define such an independence through any sequence of collections of sets that generates {o (Xi) ; i 2 l}. For example, since {Xi5 ui} generates a (Xi), the sequence {Xi; i 2 l} is independent iff {Xi5 ai;i 2 l} is a collection of independent sets for any ai E R (i 2 1). If {Xi5 ai:i 2 l} is an independent sequence and Xi all have the same CDF F , it is called an independently a n d identically distributed, or i.i.d., sequence of random variables. Independent random variables play an important role in the theory and application of statistics. In most applications, it is difficult if not impossible to verify independence directly by the definition. Instead, by interpreting c-fields as collections of pieces of information about events of interest, we check independence by examining whether such pieces of information are independent across the different o-fields or random variables. Example 10. In Example 7, X1 and X2 are generally dependent as the numbers of surgeries performed by the same hospital should not be too different in two consecutive years unless there is some disease outbreak or a major population change in the area served by the hospital. If X1 and X2 represent the surgeries performed by two hospitals that serve different populations from say distinct localities in the same year, then they are independent. If the components of a random vector X = (XI,. . . , Xk)Tare independent, then {Xi5 ui;1 5 i 5 k } is a collection of independent sets for any ai E R (1 5 i 5 k ) . It follows that / k
\
k
k
So, the distribution function of such a random vector is determined by the CDFs of its individual components. It is readily checked that the P D F or
36
C H A P T E R 1 PRELIMINARIES
probability distribution function (for discrete X) is the product of the density or probability distribution functions of the component random variables (see exercise). As a special case, if Xi have the same distribution, then the joint PDF or probability mass function of X is given by k
k
i=l
j=1
ewhere f and fi, denote the PDF and probability mass function o f . pending on whether X is continuous or discrete. Such a joint PDF or probability mass function is known as the likelihood function, which is the premise for maximum likelihood and related inferences in statistics.
1.5.3 Expectation Let P = (R,F,P)be a probability space. Let X be a random variable defined on ( Q , f ,P ) . Then, we define the expectation of X as:
E ( X )= L X d P
(1.26)
provided that at least one of E ( X + ) = Jn X f d P and E ( X - ) = JQX - d P is finite, where X + = max ( X ,0) and X - = max ( - X , 0). Since P (a) = 1, E ( X ) is an average of X . Example 11. Let X = c, a constant random variable. Then, E ( X ) = Jn cdP = C . Example 12. Let B E B and X = I B . Then,
E ( X )=
I
IBdP = P r ( X = 1)
the average of the values of X weighted by the probability of X = 1. Example 13. Consider a probability space P = (a, f 2 , P ) , with R and F2 defined in Example 2 of Section 1.2 and P being some probability measure. A random variable X with respect to F2 must be constant over each of the points in {1,3,5} and {2,4,6}. Let a and b be the two constants of X on {1,3,5} and {2,4,6}, respectively. Then,
E ( X )= a P r ( { X = k ;k
=
1,3,5})
+ bPr ( { X = rn;rn = 2 , 4 , 6))
the average of a and b weighted by the probabilities when X equals the respective constants. In general, if X is discrete, then its expectation is the
37
1.5 D I S T R I B U T I O N F U N C T I O N A N D E X P E C T A T I O N
weighted average of the values of X according to its probability distribution function. Suppose that E < 30 and E / Y /< 00. Then, the expectation has the following properties: 1. E (ax b) = u E X b. 2. If X 2 Y , then E X 2 E Y . 3. (Cauchy-Schwartz inequality) E ( U V ) 5 v ’ m , where U = X - E ( X ) and V = Y - E ( Y ) . 4. (Jensen’s inequality) Suppose p is convex (concave), then E (p ( X ) )2 p ( E ( X ) )( E (p( X ) )5 p ( E ( X ) ) ) provided , that both expectations exist. 5. (Chebyshev’s inequality) Suppose cp : R + R, with p 2 0 and When (p(x)= nondecreasing. Then, for any a > 0, P ( X 2 a ) 5
1x1
+
+
x2,we have for X > 0, P ( X
2a ) 5
012. E(X2)
w.
Proof. (1) and (2) follow from similar properties for integration with respect to general measures discussed in Section 1.3. The Cauchy-Schwartz, Jensen’s and Chebyshev’s inequalities are also readily proved (see exercise). Let X be a random variable defined on P = F, P ) and p x the induced probability measure. Then, by the change of variable formula, we have
(a,
Thus, integration with respect to P can be carried out by integration with respect to p x . If X is continuous with PDF f,then we can compute E ( X ) by integration with respect to the Lebesgue measure, (1.27) More generally, if g is a real measurable function from R = ( R , B ) to itself, then Y = g ( X ) is a random variable. Let py denoted the induced measure. Then, it follows from the change of variable formula that
As in the case of X , the above integration can be carried out using the Lebesgue measure if Y is continuous. Example 14. Let X be a random variable following a normal distrib-
38
CHAPTER 1 PRELIMINARIES
ution, N ( p , a 2 ) . Then, it follows from (1.27) that
E(X)=
L
zf(z)dx=
Let Y = g ( X ) = ( X
-P
L
x-
G
1
o
exp
(-
(X
- P ) ~ dz ) 2a2
) ~ .Then, by (1.28),
The mean p and variance o2 completely characterize the normal distribution. More generally, for each positive integer k , E ( X k ) ( E ( X - E ( X ) ) ' ) is called the kth (centered) moment of X . Thus, the mean is the first moment and variance the second centered moment of X . The first two moments play an important role in distribution-free modeling, which we will discuss in detail in later chapters.
1.5.4
Conditional Expectation
Let P = (R, f ,P ) be a probability space and let X and Y be two random variables with respect to P. Let a ( Y ) denote the a-field generated by Y . Then, the conditional expectation of X given Y , E ( X I Y ) ,is a o ( Y ) measurable random variable satisfying:
The conditional expectation has the interpretation of being a smoothed version of X with respect to Y . Note that E ( X 1 Y ) is unique up to a set of probability 0, that is: if there is another random variable, g ( Y ) ,satisfying (1.29), then, E ( X I Y ) = g ( Y ) a.e., or, P r ( E ( X 1 Y ) # g ( Y ) )= 0.
39
1.5 DISTRIBUTION FUNCTION AND EXPECTATION
Conditional expectation of a random variable X may also be defined with respect to a a-field, G. In this case, E (X j G) is defined by substituting 6 for a ( Y ) in (1.29). Example 15. Let Y = c be a constant. Then, a (Y) = {a,R}. Since E ( X 1 Y )is a (Y)-measurable, it must be a constant on Q. Thus, it follows that
E ( X )= L X d P
=L
E ( X 1 Y )dP
=E(X
i
1Y)
dP = E ( X 1 Y )
The conditional expectation in this case has smoothed X to a single value. This example shows that E ( X ) can be viewed as a conditional expectation corresponding to the trivial a-field {a,R}. Example 16. Let Y be a binary random variable with the value 0 or 1. Then, a ( Y ) = ( @ , O , Y - l (0) ,Y-' (l)},where Y-l ( u ) denotes the preimage of Y = u in f of P. Since E ( X 1 Y ) must be a constant on Y-l (0) and Y-I (l),it follows that
E ( X 1 Y = I ) P r ( Y = 1) =
L
l
E ( X I Y = 1)dP XdP =E
=
(XI{Y=I))
s,=l
Thus, the conditional expectation E (X 1 Y) is given by
(1.30) In this example, Y has been smoothed into a random variable X with two values, E (Pr(Y=O) X ' { Y = o } ) and E (Pr(Y=1) X 1 { y = l } ) , which are the averages of X over the sets {Y = 0} and {Y = l} defined by the respective levels of the binary Y. Example 17. Let A , B E f , X = I A and Y = 1,. It follows from (1.30) that
40
CHAPTER 1 PRELIMINARIES
The last quantity is used to define the conditional probability of A given B , Pr ( A 1 B ) . The formula P r ( A 1 B ) = Pr(AnB) is called the Bayes’ theorem. W B ) Note that E ( X I Y ) is a random variable while Pr (A 1 B ) is a value equal to E ( X 1 Y = 1). Example 18. Let X and Y be two continuous random variables with a joint PDF f x y (x,y). Denote by f x (x) and f y (y) the PDFs of the marginal X and Y . Consider G = {Y 5 y} E a ( Y ) . Then, by the change of variable formula, we have: ~
On the other hand, since JG E ( X 1 Y )dP = JG X d P by definition, it follows
Since G generates a ( Y ) ,it follows from the discussion in Section 1.3 that the above holds for any subset Q E a ( Y ) . Thus,
E (X I Y )dP
=
L
x . fxjy(x)dxdP, a.e.
The function f x l y ( x ) is called the conditional density of X given Y . This example shows that the conditional expectation can be evaluated by the conditional density for continuous X and Y . Theorem 1. Let X and Y be some random variables, and GI, 62 C: F some a-fields. 1. (Iterated conditional expectation) If GI C G2, E [E( X 1 G2) I 411 = E ( X I Gl). 2. (Linearity) E ( a x bY 1 G)= aE ( X I 6 ) bE (Y 1 G). 3. If X and Y are independent, then E ( X 1 Y ) = E ( X ) . 4. If X is measurable with respect to G,then E ( X I G)= X. 5. (Jensen’s inequality) If g is convex, then E (g ( X ) 1 6 ) 2 g ( E ( X I G)).
+
+
41
1.6 CONVERGENCE O F RANDOM VARIABLES AND VECTORS
Proof. Here we will prove (1). The others can be shown in a similar fashion (see exercise). First, note that the left side is GI-measurable. Now, by definition of E [ E( X I 6 2 ) 1 611, we have
However, if G E 61, then G E vields:
92. Applying the definition of E ( X 1 G2)
Thus,
(1.32) It follows from (1.31) and (1.32) that
Example 19. Let G1 = {a,O } and G2 = u ( Y ) . Then, clearly 41 C G2 and it follows from Theorem 1/1 that E [ E( X 1 Y ) ]= E ( X ) . This identity has wide applications in the theory and application of statistics.
1.6
Convergence of Random Variables and Vectors
Convergence of sequence of random variables (vectors) plays a key role in the study of behavior of statistics in large samples and is a concept fundamental to the theory of statistics asymptotics. In this section, we review the notion of convergence of random sequence and important results concerning the convergence of a particular class of random sequences consisting of independently and identically distributed random variables (vectors).
1.6.1 Modes of Convergence Let X , ( n 2 1) be a sequence of random variables defined on some probability space, P = (a,f ,P ) . As random variables are real measurable functions, all results discussed in Section 1.3 for the convergence of such a seX quence apply to the sequence of random variables. For example, if X , for all w E 0 except for a subset with probability measure 0, then X , -+ X ---f
42
CHAPTER 1 PRELIMINARIES
a.e. (almost everywhere). Since this definition of convergence relies on the original probability space, it is not widely used in statistics. The definitions most relevant to statistical theory and application are convergence in probability and convergence in distribution. We discuss convergence in probability below as well as in Section 1.6.2. Convergence in distribution will be reviewed in Section 1.6.3. Convergence in probability is convergence in measure for real measurable functions when rephrased with respect to the probability measure P. Thus, X , converges to a random variable X in probability, X , - f P X , if for any 6 > 0, the deterministic sequence, d , > ~ = P r [IX, - XI > 61 or c,,~ = P r [IX, - X I 5 61 converges to 0 or 1 as n 00, that is, for any E > 0, we can find some integer N,,6 such that for all n 2 NE.6, --f
dn,6 = Pr [IX, - XI
> S] < E ,
c,,~
= Pr
[IX, - XI 5 S] 2 1 - E
(1.33)
Thus, under (1.33) the variable or random nature of X , is captured by a single deterministic sequence dn,6 or c,,6 so that its convergence can be evaluated by the criteria for deterministic sequence. In most statistical applications, X is often an unknown constant p that describes some features of a population of interest such as age, disease prevalence or number of hospitalizations, while X , is an estimate of p. If X, p, X , is a consistent estimate of p. In this case: if v, denotes the X,-induced measure on S = (R, a),X , -+p p can be stated as - f P
~ , ( ( p - - S , p + 6 ) ~ ) - + O or
wn([p-S,p++])-+1
n-oo
In this sense, the notion of convergence in probability or consistency of parameter estimate is independent of the underlying probability space. Example 1. Let f (x) be a continuous function on R. If X , --+P p , then f ( X n >+p f (4. Let K > lpl > 0. Then, for any E > 0,
Since f is uniformly continuous on [-K, K ] ,there exists d , , ~> 0 such that
z.
Given such a K,, we then We first choose K , so that Pr [IX,l > K,] < select S,,K to ensure P r [IX, - pi > S € , K ] < 5 . Since X, -+P p , we can find
1.6 CONVERGENCE O F RANDOM VARIABLES A N D VECTORS
E
43
E
<-+--=€
2
2
The conclusion follows from (1-34)-( 1.36). Example 2. Let X , N , that is, X , follows a normal disN
tribution with mean p and variance sequence. For any S > 0,
$
(n>_ 1).
Then, X , is a random
where @ (.) denotes the CDF of standard normal N ( 0 , l ) . It follows that X , - f p p, a consistent estimate of p In this example, if X , does not follow the normal distribution, it may be difficult t o evaluate Pr [IX, - pi > S ] . Fortunately, in many statistical applications, X , is typically the average of i.i.d. random variables and in this case, it is possible t o determine the behavior of P r [IX, - pi > 61 for large n as we discuss next.
1.6.2
Convergence of Sequence of I.I.D. Random Variables
Statistics and model estimates of interest arising in many applications are often in the form of an average of i.i.d. random variables. For such random sequences, we can determine whether they converge (in probability) as well as the converged limits by the law of large numbers (LLN). Theorem 1 (Law of large numbers). Let X i be i.i.d. random variables, with finite mean p = E ( X i ) and variance o2 = V a r ( X i ) . Let 5?, = Cy==l X i be the average of the first n random variables. Then, the random sequence 5?, converges t o the mean p = E ( X i ) in probability, 5?, -+p p, as n -+ 30. Proof. For any S > 0, it follows from the assumptions that
From the above, we immediately obtain the Chebyshev’s inequality: (1.37)
44 For each
CHAPTER 1 PRELIMINARIES
E,
by setting Nt,6 = Pr
",I[
-p
[iu21+ 1, it follows from (1.37) that 2 611 _< E ,
for all n
2 NE,h
The above LLN is known as the weak law of large numbers. A similar version for the almost everywhere convergence of an i.i.d. random sequence is called the strong law of large numbers. In addition, the assumptions of Theorem 1 can also be relaxed so that it can be applied to a broader class of i.i.d. random sequences. But, the version of LLN stated in Theorem 1 suffices for the development of later chapters in this book. The law of large numbers is widely applied to facilitate the process of determining the convergence of 5?, when it has an unknown distribution. Random variables arising from almost all applications have a limited range, (the variables have the value 0 outside this range) and thus not only the first two moments , but all higher order moments are finite. For this reason, we apply Theorem 1 without citing the assumption throughout the book unless otherwise noted. Example 3 (Monte Carlo integration). Let f (x)be a continuous b]. Let I ( f ) = f (x)dx be the integral o f f over [a, b]. Let function on [a, X i be i.i.d. continuous random variables with PDF g (z) > for all a 5 x 5 b. Then, by LLN,
s,"
s ( x , ) . Thus, if where Y , = fo
s," f (z)dz has no closed form solution, we can h
approximate I ( f ) by a random sum I, ( f ) =
T,
A C:=l #.If both a and b
are finite, the simplest way to compute ( f ) is to sample X , from a uniform U [a, b ] . This Monte Carlo integration plays an important role in Bayesian analysis. Example 4. Let X , i.i.d. ( p ,u 2 ) and = C:=l X,, where (p,u 2 ) denotes a distribution with mean p and variance u2. For T > 0, let f (z) = zT. Then, by applying LLN and Example 2, 5?: = f (F,) - f p f ( p ) = p'. As a special case, by letting r = k (= 1 , 2 , . . .), it follows that sample moments yi converge to the corresponding population moments ,ukof all orders. N
x,
1.6.3 Rate of Convergence of Random Sequence In the theory and application of statistics, we need to know not only whether an estimate is consistent, but also its sampling variability. To study the
1.6 CONVERGENCE OF RANDOM VARIABLES A N D VECTORS
45
latter, we must normalize the estimate using the rate of Convergence of the estimate since convergence in probability does not distinguish random sequences with different rates of convergence as shown by the next example. Example 5. Let X i i.i.d. N ( p ,02).Then, by LLN, X , -+p p or equivalently, X , - p +p 0. NOW, let Y, = n+ - p ) By applying Chebshev’s inequality, we have:
-
(x,
1
1
-
So, Y, +p 0. However, since na 00, Y, = nz ( X , - p ) has a slower rate of convergence than 5?, - p. The above example shows that we need a new concept of convergence to delineate the different rates of convergence for sequences that converge to the same limit in probability. Let X , be a random sequence with CDF F, ( n 2 1) and X a random variable with CDF F . The sequence X , converges in distribution to X , X , -+d X , if for every continuity point c of F , lim F, (c) = F (c)
,--to3
Since F is right continuous, continuity points are those c at which F ( c ) = lim,TcF (x) = F (c-), the limit of F as x approaches c from left, that is, x < c. Convergence in probability is stronger than convergence in distribution. However, when X is a constant, the two become equivalent as the following example shows. Example 6. X , +p p iff +d p. We only establish the first part, that is, X , tp p implies that X , -+d p. The reverse is similarly proved (see exercise). By viewing the constant p as a special random variable, the CDF of p Such an F is known as a degenerate CDF. Thus, any is Fp (c) = c # p is a continuity point of Fp (.). To establish the conclusion, we need to show that for any c # p,
x,
F, (c) = Pr ( X , 5 c) -+ Fp ( c )
46
CHAPTER 1 PRELIMINARIES
First, consider c
< p. Since X ,
-+p
p and p
-
c
# 0 , it
follows that
F, ( c ) = Pr ( X , 5 c ) = P r [ X , - p 5 c - p] 5 P r [ / X , - pl 2 p
- c] + 0
Likewise, we can readily show that for c > p , F, ( c ) + 1. Thus, Fn (c) + Fp ( c ) for all continuity points c ( p ) of Fp. However, the most important use of this new notion of convergence in statistics is to define the rate of convergence of a consistent estimate so that we can study its sampling variability. A random sequence X , (n2 1) has a rate of convergence n p if
x,
-+,jX, n p
asnioo
where X has a nondegenerate distribution. Example 7. In Example 5, since fi converges to 0 at the rate of 1
-
(A)
(A) ;.
(x, p ) -
N
N (0,u 2 ) ,X n
-p
Similarly, the rate of convergence for
( X , - p ) is $, slower than 5?, - p. If is non-normal, it is generally difficult to find its rate of convergence. The central limit theorem (CLT) is applied to facilitate the evaluation of I?, in general situations. Theorem 2 (Uniuariate central limit theorem). Let Xi be i.i.d. random variables, with finite mean p = E ( X i ) and variance u2 = V a r ( X , ) . Then, fi - p) +,j N (0, u 2 ) . Since N (0, u 2 ) has continuity points on R1 (the real line), this implies that for all x E R1,
n4
x,
(x,
where CP (.) denotes the CDF of standard normal N ( 0 , l ) . As in the case of LLN, the moment assumption of Theorem 2 is satisfied in most applications. For the mean of i.i.d. random variables, the CLT refines the conclusion of LLN by providing both the rate of convergence for the centered mean I?, - p and the limiting distribution for the normalized sample mean fi - p ) . The latter, known as asymptotic normality, has wide applications in the theory and application of statistics asymptotics. It asserts that for large n, follows approximately a normal distribution, N ( p , i u 2 ) , which provides the basis for inference for p. Throughout the book, we state such an approximation by AN ( p , i u 2 ) and refer t o AN ( p , ; g 2 ) as the asymptotic distribution and u2 as the asymptotic variance of T,. Example 8. Let Xi be i.i.d. random variables following a Bernoulli distribution with mean p , that is,
x,
(x,
x,
x, -
Pr [ X i = 11 = p ,
Pr [ X i = 01 = 1 - p = q
1 . 6 CONVERGENCE OF R A N D O M VARIABLES A N D VECTORS
F,
xy=l
F,
47
Let =$ Xi. Then, by CLT, AN ( p , i p g ) . Thus, we can use this asymptotic distribution to compute probabilities for binomial random variables. For example, let r = CG1Xi,the number of times when X i = 1. Then, N
The above approximation works well if p is too small. We can also use the asymptotic distribution to approximate confidence intervals for fin:
where
24
denotes the
4 percentile of the standard normal N (0,I), that is,
@ z- = 4 . .;> We conclude this section by generalizing the concepts of convergence to random vectors. Let X, E Rk be a sequence of k-dimensional random vectors. Then, X, converges to X in probability, X, -+p X, if IIX, - X(I -fP 0, where denotes the Euclidean distance in Rk. Let F, (n2 1) be the CDF of X, and F the CDF of X. The sequence X, converges in distribution to X, X, - f d X, if for every continuity point c of F , limn+co F, ( c ) = F ( c ) . Again, convergence in probability implies convergence in distribution (see exercise). Also, Theorem 2 is similarly generalized to sequences of random vectors. Theorem 3 (Multivariate central limit theorem). Let Xi E Rk be i.i.d. random vectors, with mean p = E (Xi) and variance C = Var (Xi). If p and C are finite and C is full rank, then f i - p ) -+ N (0,C), that is, for all y E R k ,
(
~~~~~
(x,
where N ( p , C ) denotes a k-dimensional normal distribution with mean p and variance C, and Q, (.) the CDF of such a distribution with p = 0 and
c = Ik.
In many applications, we can often verify whether a sequence of random vectors X, converges in probability by examining its moments. More specifically, let r (> 0). The sequence X, convergence in r t h mean, X, -+T X, if E j/X, - XII' -+ 0, n cc It is readily shown that X, +T X implies that X, -tpX (see exercise). ---f
48
CHAPTER 1 PRELIMINARIES
1.6.4 Stochastic
0,
(.) and 0, (.)
As in the study of convergence of deterministic sequence, we often just want to know the rate of convergence of a random sequence rather than its exact limit. To this end, we use the stochastic op (.) and 0, (.). First, we review the stochastic op (.), which, like its counterpart o (.) for deterministic sequence, is used to indicate a faster rate of convergence for a random sequence to converge to 0 in probability. For a random sequence X,, X, = op (1) if X, +p 0. More generally, if T, (> 0) -+ 0, X, = 0, (T,) is defined as TL'X, = op (1). Example 9. Let Xi i.i.d. ( p , 0 2 ) . Then, it follows from LLN that X , - p -+, 0. Thus, X, - p = op (1). Example 10. Let X i and 5?, be defined as in Example 9. Consider another random sequence, Y, = (57, - p ) . Then, nY, = (5?, - p ) - f p 0. Thus, Y, = 0, and Y, converges to 0 in probability at a rate faster than N
(A)
1 n'
Unlike its counterpart 0 (.) for deterministic sequence, the stochastic 0,(.) is used to indicate stochastic boundedness rather than equivalent rate of convergence. We first review the notion of stochastic boundedness. Example 11. Let X, = n or 0 with Pr [ X , = 01 = 1 and P r [ X , = n] = 1Then, X, has a large mass centered at n and the probability of X , at this mass point increases to 1 as n 00. For any M > 0,
-
A.
P r [-Ad 5 X ,
1
5 M ] = 1 - Pr [X, > M ]= - -+0 n
Thus, X, is not bounded in the sense that the probability that X , is confined t o any finite interval [ - M , M ] decreases to 0 as n + 00. In other words, we cannot find a sufficiently large interval [-M, M ] t o trap a desired amount of mass of X,, as a nonzero mass escapes to infinity. A random sequence X, is stochastically bounded or bounded in probability, denoted by 0, (l), if for any E > 0, we can find M , and N, such that Pr [IX,l 5 M,] Pr[lXnl
2 1- E ,
2 M,] i E ,
all n 3 N , or equivalently all n 2 N,
For a bounded random sequence, we can always find a sufficiently large interval so that the probability that the sequence lies outside the interval can be made arbitrarily small. Note that in many applications, Pr [-M 5 X, 5 M ] has a limit for each fixed M as n + 00. To check stochastic boundedness in this case, we only
49
1.6 CONVERGENCE OF RANDOM VARIABLES A N D VECTORS
need to show that for any P r [IX,l 5 Me]+ a
E
2 1- E
> 0, we can find Me such that - or
equivalently
-
P r [IXnl 2 Me]-+ b 5
E
Example 12. Let X be some random variable and X , = X . Then, it is readily shown that X , is bounded (see exercise). Let X, = ( ~ ~ 1. ., , x. n k l T E R'. We write X, = 0, (1) if X, +p 0. We define X, = 0, (1) if for any E > 0 we can find M, and N , such that P r [IIX,ll P r [llX,il
I Me]2 1 - E , for all n 2 N,, 2 Me]5 E , for all n 2 N ,
or equivalently
(1.38)
Example 13. If x, - f d x, then X, = 0,(1). Let ml and m2 denote some continuity points of the CDF F (x) of X. For any E > 0, we can find continuity points mle and m2Esuch that
It follows from X,
-+d
X and the above that
and the conclusion follows. Example 14. Let Xi i.i.d.(p,C), with ( p ,C) denoting a multivariate distribution with mean p and variance C. Then, by CLT, f i - p ) -+d N ( p ,C ) . Thus, fi (X,- p ) = 0,(1). Let T, (> 0) -+ 0. If TL'X, = 0, (I), we write X, = 0,(T,). Example 15. Consider again Y, = na (5?, - p ) in Example 7. Since
(x,
N
(
1
n 2Y,= f i (57, - p ) = 0,( 1) , Y,= 0, ( 7) . Example 16. Let T, (> 0) -+ 0 as n oc. Then, 0, -+ 00 as n --+ 00, it follows that For any E > 0, since --f
(T,)
= 0, (1).
Thus, 0, (T,) 0 and the conclusion follows. Let T,. q, (> 0) -+ 0 and X,, Y , E R', Listed below are some additional properties regarding op(.) and 0, (.) (see exercise). - f P
50
CHAPTER 1 PRELIMINARIES
+ + +
1. 0, (1) 0,(1) = 0,(1). 2. 0, (1) o p (1) = 0,(1). 3. o p (1) op (1) = 0, (1). 4. 0,(1)0, (1) = 0, (1). 5. 0,(r,) = 0, (1). 6. X, = op(1) iff X,j = op (1) for all 1 5 j 5 k . 7. X, = 0, (1) iff X,j = 0, (1) for all 1 5 j 5 k . 8. If X, = 0, (T,) and Y, = 0, (q,), then X, + Y , = 0, {max (r,>4,)) and XZY, = X,Y, = op (r,q,).
Cf=,
Example 17. Let Xi Then, s: -+ 0 2, . First, we have
By LLN,
N
ix;=,[(Xi - p)2 -
i.i.d.(p,0 2 ) and s: =
0’1 =
iE:=,( X i - 5?,) 2 .
op (1). In addition, by Property 5, we
Thus, s,2 - 02 = op (1)
+ o p (1) = 0, (1)
1.7 Convergence of Functions of Random Vectors 1.7.1 Convergence of Functions of Random Variables In many inference problems in statistics, we often need to determine the distribution of a function of a sequence of random variables. For relatively simple functions, such as a linear combination of random variables, we can apply Slutsky’s theorem to determine the limiting distribution of the sequence. For more complex functions, we typically first linearize the function using a stochastic Taylor series expansion and then apply CLT and/or Slutsky’s theorem to find the asymptotic distribution of the sequence. The latter approach is also known as the Delta method. Theorem 1 (Slutsky’s theorem). Let X , - f d X and Y, +, c, where c is a constant. Then, 1. X , + Y , - + d X + C 2. x,Y,*dcx.
51
1.7 CONVERGENCE OF FUNCTIONS OF RANDOM VECTORS
3. If c # 0, X,/Y, -+d x / c . Proof. We only prove ( l ) ,with (2) and (3) following a similar argument. First, note that the continuity points of the CDFs of X and X c are the same. For a continuity point a of the CDF of X, the following is true for any b > 0:
+
Let n
-+ 00,
we get
Hence, limPr(X,Iu-b)
IlimPr(X,+Y,Ia)
Since a is a continuity point of X , we obtain the identity by letting b -+ 0 and the conclusion follows. Example 1. Let X i i.i.d.(p, 02) and r?, = Ey=lXi. Let us find the limit and the asymptotic distribution of the sequence of the second order moments By LLN:r?, +p p. We have shown in Section 1.6 that all moments of a convergent random sequence converge to the corresponding moments of the distribution. Thus, p2 and is a consistent estimate of p 2 . By CLT and LLN, we have N
x2.
xz
zz
h ( X , - P ) -+d
N (0,02),
-
x, + p -+p
2P
It follows from Slutsky’s theorem that
6(r?: fi
Thus, 4pV.
-
2)= h (x, p ) (r?, + p )
r2 ”> X,
-p
-
+d
N (0, 4p2a2)
also has a limiting normal with asymptotic variance
Example 2. In Example 1, consider the limit and the asymptotic 2 distribution of the sample variance, s i = Cy=l( X i - r?,) . It follows from LLN and Example 1 that
c(xz
2 1 , s, = n i=l
-
r?,)
2
cx:
1 ,
=n
i= 1
-
r?;
-+
E
(xi’)- p 2
a
I
2
52
CHAPTER 1 PRELIMINARIES
Thus, s i is a consistent estimate of 0’. To find the asymptotic distribution of s i , first, note that
By CLT,
In addition,
It then follows from Slutsky’s theorem that
In Theorem 1, if X , -+p a,a constant, then the results also hold when convergence in distribution (‘-id7’ is replaced by convergence in probability “+,” (see exercise). For example, if X , -+p a and Y, c (f 0), then X7l Yn +P
+,
5
For complex f such as J , we cannot apply Theorem 1 to find the limiting distribution. As in the study of deterministic function, Taylor series expansion is often used to facilitate the computation. The following is an analogue of such an expansion for random functions. Theorem 2 (Stochastic Taylor series expansion). Let X , be a sequence of random variables such that X , = a + 0, (T,), where T, +, 0 as n -+ 00. If f has continuous derivatives up to order s 1 in a neighborhood of a, then
+
where f ( k ) (a)denotes the kth order derivative. Note that f (X,) above is a random sequence, since both ( X n - a)k and op (T,) are random sequences. Proof. First, note that since T, + 0, X, = a 0, (T,) = a op (1). For x # a,consider the deterministic Taylor series expansion:
+
+
53
1.7 CONVERGENCE O F FUNCTIONS O F RANDOM VECTORS
where [ E ( u , x ) . Let
Then, h (u)= 0. In addition, h is continuous at a. By Example 1 of Section 1.6, we have h(X,)=h(u)+op(l)=op(l) (1.40) It follows from (1.39)-(1.40) that
=op
(4
The last equality above follows from the property that o p ( l ) O ; ( ~ , ) = op (rn). Example 3. In Example 2, consider the limit of s, = From Example 2, s i -+p o2 and thus s i = 0’ op (1). Let f (x)= &. By Theorem 2, we have
+
s, = 0
1 +( s2, -
0’)
2 f i
a.
+ op (1) = 0 + op (1) + op (1) = + op (1) 0
The last equality follows from the properties of op (1). Thus, s, is a consistent estimate of CJ. Note that in this simple example, we may find the limit of s, without using the Taylor series expansion. Since f (z)= & (x > 0) is continuous and s i -+p g 2 , it follows from Example 1 of Section 1.6 that f ( s i ) -+
f
(02)
= 0.
Example 4. In Example 1, consider the sequence Y, CLT, fi (5?, - p ) - f d N (0, 0’) and from Example 3, s, follows from Slutsky’s Theorem that
In Example 4, if Xi
fiv
N
fiv. -
=
-+p
0.
By Thus, it
i.i.d.N ( p , D’), then it is well known that Y, =
has a t distribution with n - 1 degrees of freedom. For non-normal
54
CHAPTER 1 PRELIMINARIES
data, Y, no longer follows the t distribution. However, the above example shows that Y, has an approximate normal distribution. Since t converges to the standard normal as n + 00, the difference between t and normal diminishes as sample size increases. Example 5. In Example 3, consider the asymptotic distribution of s,. Since fi ( s i - 0 ” ) * d N it follows that s:
-
.
u2 = 0,
By Taylor series expansion,
Thus,
=
-6 1
(S,2
2&
-+d
-
2)+ 0,
(1)
N (0, k v a r ((Xi- 1 0 2 ) )
Note that although s: is an unbiased estimate of a 2 , s, is a biased 1 estimate of 0. Since f (x)= x2 is a concave function, it follows from the Jensen’s inequality (see Section 1.5) that
Thus, s, always underestimates o in finite samples. If X N ( p ,0 2 ) then , it follows from the properties of the normal distribution that any linear combination of X also has a normal distribution, that is, uX b N (up b, u 2 a 2 ) ,where a and b are constants. This property also carries over to random sequences with asymptotic normal distributions as the next example shows. Example 6. If X , A N p, , then for any a , b E R, N
+
+
N
N
(
u2)
fi[ ( u X +~ b ) - ( U P + b ) ] = ~ (f X ni
-
Thus, u X ,
+b
N
AN up + b, 7 . u2g2)
(
p)
-td
N (0, a 2 0 2 )
1 . 7 CONVERGENCE OF FUNCTIONS O F RANDOM VECTORS
55
1.7.2 Convergence of Functions of Random Vectors In Section 1.6, we considered convergence of a sequence of random vectors X,. By applying a vector-valued function f (x) to such a sequence, we get a sequence of vector-valued functions of random vectors f (X,). As in the scalar case, the following theorem is quite useful to determine the convergence of such a sequence. Theorem 3. Let X, E R ' be a sequence of random vectors. If X, - X = op(1) and f (x) is a continuous mapping from R' t o Rm,then f (X,) - f (X) = op( l ) ,that is, f (X,) - f p f (X). Likewise, for such a sequence of vector-valued functions of random vectors, we are also interested in its asymptotic distribution. We first discuss a useful result that enables the study of convergence of random vectors through random variables. Theorem 4 ( Cram&- Wold device). Let X, E R ' be a sequence of random vectors. Then, X, +d X iff XTX, --id XTX for any constant vector X E Rk. The Cram&-Wold device is a useful tool for extending univariate asymptotic results t o the multivariate setting. One important application is in defining the asymptotic normal distribution for sequence of random vectors. The random sequence X, has an asymptotic multivariate normal distribution with mean p and variance C if for all X E R ', &AT
(X,
- p ) -'d
N (0,XTCX)
,
or
In Theorem 3 of Section 1.6, we stated a version of asymptotic multivariate normal distribution when the asymptotic variance C is full rank so that the PDF exists. This alternative definition require no such assumption and thus applies to degenerate multivariate n o r m a l distribution as well. In most applications, however, C is full rank and two versions become identical. By applying Theorem 4, we can immediately show that all linear vectorvalued functions of X, are asymptotically normal. Example 7. Let X, AN ( p , AX). Then, the linear function AX,+b is also asymptotically normal, AX, b AN ( A p b, ;AXAT), where A is some m x Ic constant matrix and b a m x 1 constant vector. For any X E Rm, N
AT [(AX,
+
N
+ b) - ( A p + b)] = XTA (X,
Since ?; = XTA E R and
+
- p) =
x i (X, - p )
6 = XTb E R,it follows from the assumptions and
56
CHAPTER 1 PRELIMINARIES
Theorem 4 that
XT ( X , - p )
+
N
AN
+
Thus, AT ( A X , b ) AN (AT ( A p b ) , ;AT ( A C A )A) and the conclusion follows. For general non-linear functions of X,, the asymptotic distributions are determined by the Delta method. Theorem 5 (Delta m e t h o d ) . Let X, A N ( p , i C ) and g ( x ) = (91 (x) ! . . . , gm ( x ) )be ~ a continuous vector-valued function from Rk to Rm. If g (x) is differentiable at p , then g ( X , ) A N ( g ( p ), A D T C D ) , where is an rn x k derivative matrix with &g defined as: D =g ,d N
N
N
Proof. Let X, = ( & I , ,
,
. , Xnk)T. It follows from Theorem 4 that
6(Xnj - P j ) +d +
& 0
N
(o,& ,
(1 5 j 5 Ic). Thus, Xnj = p j 0, each gl ( X , ) (1 5 1 5 m ) , we have:
1 Ij
Ik
By a Taylor series expansion for
When expressed in the vector form, the above becomes:
It follows from Example 7 that where op(.) = ( o , ~(.) , . . . , opm ( s ) ) ~ . DT (X,- p ) AN (0,i D T C D ) . Thus, for any X E Rm, N
57
1.7 CONVERGENCE OF FUNCTIONS OF RANDOM VECTORS
The conclusion follows from Theorem 4. Example 8. Let X i i.i.d. ( p , a 2 ) . Then, the statistic, t, = *, is invariant under linear transformation. In other words, if Y-, = aXi b
-
+
-
t,, for any a (> 0), b E R, then Y , i.i.d.(pg,oi) and t , Syn where sin denotes the sample variance of Y,. However, the measure of variability does not have such a nice property. For example, if Y , = a X i , then a; = a2u2 and the scale transformation has increased the variance of X i by a multiplicative factor a2. A scale-invariant measure of variability is the coeficient of variation, IT = $. We can estimate 7 by 7n = *. It follows from Slutsky's theorem that XTl 7 , is consistent, that is, = Yn-h
=
h
To find the asymptotic distribution of Fn, let f (z,y) = $. Then, ?, = f sn) and by the Delta method, 7, AN ( T , k D T C D ) , where C is the asymptotic variance of (F,,s), and D is the derivative of f (z,y) with
-
(x,,
In the above approach, it may not be straightforward to find the asymptotic variance C (see exercise). An alternative is t o express the estimate ?, differently. To this end, let
Then, it is readily checked that ?, = g (U,, V,). In addition, by CLT,
By the Delta method, ?, and D bv
-
AN
(7, i D T C D ) ,where
C is given by the above (1.42)
Example 9. Let Zi = (Xi,Y,)Tbe i.i.d. random vectors (1 5 i 5 n). Let
58
CHAPTER 1 PRELIMINARIES
Then, f i (y, - y) has an asymptotic normal distribution. Let W, = ( X , , ~ . X , ~ and ) T f ( w ) = w3 - w1w2. Then, Tn = f (W,) = f (57,,F,,=,), where En= CZ1XiY,. It follows from CLT that 6(Wn- E ( ~ 2 ) ) - ~ c iN (0, C) where C = V a r (Wi) is readily computed and estimated. method, 9, AN (y, i D T C D ) , where
By the Delta
N
Similarly, we can show that the Pearson correlation coefficient h
Pn =
x;'lm (Xi 57,) (yz Y n ) (1.43) J JW -
-
is a consistent estimate of p = Corr ( X I ,Y1) =
cOw(xl~yl) .JVov(X1)Var(Y1)
and asymp-
totically normal (see exercise).
1.8
Exercises
Section 1.1 1. For the linear regression model discussed in Section 1.1.1 for the comparison of two treatment conditions, , , T
(a) show that the MLE of ,B is -
yk. =
1
=
(/31,/32) = (jJl,,jJ2.)T, where
n ci=1 XikyZ;
(b) verify (1.4) by showing
2. Let R k i denote the rank scores for yk, ( k = 1 , 2 ) as defined in Section 1.1.3. Let Wn = CyLl Rli. Show (a) W, + Ev2 R? = where n = n1 122. 3=1 3
w,
nl(nl+l)
(b) W , = U n + 7. 3 . In Problem 2 above, show nl(nl+nz+l) (a> E(W7L)= 2
+
59
1.8 EXERCISES
(b) Var (W )
n -
nln2(nl+n2+1) 12
Section 1.2 1. Let R be some sample space. A power set P of R is a collection of all subsets of R including 4 and R. Show that P is a a-field. 2. Prove the proposition. 3. For any a E R, the real line, show that (-00,a] can be expressed as a countable union of the intervals of the form: (a) ( b , c] with b, c E R. (b) ( a , b ) with a , b E R. Thus, intervals of the form (--00, a ] , ( a ,b] and ( a , b) all generate the same Bore1 a-field B. Since a E R, the complement of (-m. a] is ( a ,00),B is also generated by intervals of the form ( a ,m). 4. In problem 2 above, show that [a.m) can be expressed as a countable union of ( a ,m) for any a E R. Thus, in light of problem 2 above, the class of intervals [a,m) generates B. Since (-m, a ) is the complement of [a,m ) , B is also generated by the intervals of the form (-m. a ) . 5. Prove Theorem 1. Section 1.3 1. Let S = (R, f ) and f, be a real measurable function defined on S (n= 1 . 2 , . . .). Let gn = rnaxln f k . Show that both are monotone sequences and thus their limits exist. 3. Prove Theorems 2 and 3. 4. I f f = CzIA, = x:=1dzlBZ,show x:=1 czp (A,)= dtp (Bz). Thus, the integral Jn f d p = Er=lc,p ( A , ) is well defined. 5 . For f 2 0, show (a) f, (z) defined in (1.11) is a monotone increasing sequence of real measurable functions. (b) limn+m f, ( w ) = f ( w ) for each w E R. (c) if gn ( w ) is also a monotone increasing sequence of real measurable functions and f = gn, then limn+m JQf,dp = limn+m Jn g,dp. 6. Let M = (R, f , p ) be a measure space and f some real measurable function define on M . Let v ( E ) = JE f d p for any E E f . Show that v (.) is also a measure. 7. Let F (z)be a nondecreasing, right continuous, step function. Let Fd (z) = F ( x ) - F (z’). Show (a) the number of points z such that Fd (z) > 0 is countable.
x:=i
c:=i
60
CHAPTER 1 PRELIMINARIES
(b) F (x)- Fd (x)is nondecreasing and right continuous. 8. Let S = ( R ,B , p ) with p denoting the Lebsgue measure. Show that for any absolutely continuous measure w with respect t o p, any single point a E R has a measure 0, that is, v ( { a } ) = 0. Section 1.4 1. If a sequence of non-negative p k satisfies (1.12), show lirnk-" p k = 0. 2. For the Poisson probability law defined by (1.13) in Example 2, show (a) For any A, B E F with A n B = 4, P ( AU B ) = P ( A ) P ( B ) . (b) For a sequence of disjoint sets A, ( n = 1 , 2 , . . .), P (U= :, A,) = Cr=1P (An). 3. To show S"exp -$) dx = 1, let
+
&
(
1,& (-%) "
I -exp
Then, " =
1
/k2
exp
kexp
dx = a
sU
(-5)dx -"& (-$) dy
(-v)
&
1
-exp
dxdy
(-w)
Now, prove that J JR2 exp dxdy = 1 by using the polar coordinate transformation, that is, x = T cos 0 and y = T sin 0. 4. Verify (1.17)) (1.19), and (1.20). Section 1.5 1. Let P = (R, f ,P) be a probability space, where R = { 0 , 1 , 2 , .. .}, f is the power set of R and P satisfies (1.12). Let X = k be a real function from Q to R. Show that X is a random variable from ( 0 ,f ) to ( R , B ) . 2. Let F (x)be the CDF of a random variable X and F-l (x)be defined by (1.24) in Example 4. Then, Y = F ( X ) is a random variable. Show that 2 = F-l ( Y )is also a random variable. 3. If the X i of X = ( X I , .. . , X k ) T are independent, show that the PDF or probability mass function is the product of the density or probability mass functions of X i (1 5 i 5 k ) . 4. Prove the Cauchy-Schwartz, Jensen's and Chebyshev's inequalities. 5. Show part (2)-(6) of Theorem 1. Section 1.6 1. Show If X n -+d p, then X n -tPp. 2. Let X n E Rk be a sequence of random vectors. Show
61
1.8 EXERCISES
X+ E R’” , implies that X, -+d X. (a) X, -(b) if X = p , a constant, X, * d x implies that x, +p x. 3. In Example 2, show that X, X implies that X, +p X. 4. Let X be some random variable and X , = X ( n 2 1). Show that the random sequence X , is stochastically bounded. 5. Show that an equivalent definition of stochastic boundedness in (1.38) is that for any E > 0, we can find Me such that --fr
P r [llXll, 5 Me]2 1 - E, P r [IIXll, 2 KI 5 E,
all n, all n
or equivalently
0, as n --+ 30. Show the properties regarding Let T,, gn (> 0) op (.) and 0, (.) in Section 1.6.4. 7. Let X,,X E R‘. Show that if X, -ir X, then X, +p X. Section 1.7 1. Prove part (b) and (c) of Slutsky’s theorem. 2. In Theorem 1, if X, +p a, a constant, show that (1)-(3) still hold true when “-+d” is replaced by “ _ t P ” , Y, = iC:=,X,”and Z, = 3. Let Xi Ni.i.d.(p,a2). Let X , = 6.
--f
x,,
( X n ,KIT. (a) Show that Z, has a bivariate asymptotic normal distribution and find the asymptotic variance. (b) Find an appropriate functional f (x,y) so that f ( X n : Y n )= s i = 2 n 2 = 1 (Xi-r?,) . (c) Use the Delta method to find the asymptotic distribution of s i . 4. In Example 8, T (a) find the asymptotic distribution of Z, = (T,, s,) . (b) use the result in (a) in conjunction with the Delta method to find the asymptotic distribution o f ? = X, (c) show that the asymptotic variance from (b) is DTCD with C given in (1.41) and D in (1.42). 5 . Prove Theorem 3. 6. Let X,,X E R‘“,and F, (x) and F (x) be the CDF of X, and X. Let
*.
4, (t) = J
R k
exp (itTx) dF, (x) ,
4( t )=
/
exp (itTx) dF (x)
Rk
denote the characteristic functions of X, and X, respectively. Show (a) X, --+d X is equivalent to convergence of 4, (t) to 4 (t) for every t (E Rm).
62
CHAPTER 1 PRELIMINARIES
(b) use (a) to prove Theorem 4 (Cramdr-Wold Device).
7. In Problem 9, show (a) Fn is a consistent and asymptotically normal estimate of p. 2 (b) if Zi i.i.d.N ( p ,E), the asymptotic variance of Fn is (1 - p 2 ) N
Chapter 2
Models for Cross-Sectional Data In this chapter, we discuss regression models for cross-sectional study data analysis. Cross-sectional study designs employ a single assessment time to obtain a snapshot of a study population with respect to the outcomes of interest. Cross-sectional data can also arise from longitudinal and other related study designs. For example, in longitudinal study designs, a cohort of study subjects is followed up for a period of time and assessed repeatedly during the study period. If we look only at the data collected from a single assessment, such as the end of the study, the data is cross-sectional. Longitudinal study data captures both between-individual differences and within-subject dynamics, offering the opportunity to study more complicated biological, psychological and behavioral hypotheses, especially those involving changes in outcomes of interest over time such as causal effects and disease progression. In contrast] cross-sectional data can only be used to examine relationships among outcomes at a single time point. Thus, any relationship or association among outcomes can be seriously confounded by individual differences among the study subjects and must be interpreted with caution. In particular, cross-sectional study data in general does not permit causal inference about the outcomes of interest. Despite such limitations] however, analysis of cross-sectional data is still quite popular in many areas of research. First, cross-sectional designs are much less costly than their longitudinal counterparts and logistically are much easier to carry out. Although inappropriate for causal inference, analyses of cross-sectional data still provide useful information for assessing and generating hypotheses concerning association among outcomes of
63
64
C H A P T E R 2 MODELS FOR CROSS-SECTIONAL DATA
interest that serve as a precursor for formal investigation using longitudinal studies. Second, cross-sectional data can also arise from longitudinal study designs. For example, many longitudinal studies employ a pre- and post-treatment design in which each subject is assessed first at baseline before some treatment of interest is given and then again at some point post-treatment. For such a special longitudinal design, treatment effect can be analyzed using methods for cross-sectional data such as the analysis of covariance model (see Section 2.1 for more details). Such analyses even provide valid inference when there are missing data or subject dropout at the post-treatment assessment (see Chapter 4 for more discussion on the topic of missing data). Third; when studying causal or risk factors for some disease of interest, the standard of randomized controlled clinical trial may not be an option; in these situations, the case-control retrospective study can be a useful alternative. For example, it is impossible to carry out a randomized controlled trial to study the causal effect of smoking on lung cancer because of the inability to control exposure status by randomly assigning subjects to the smoking and no smoking condition. Finally, for a rare disease, controlled longitudinal or observational cohort study designs may not yield a sufficiently large event rate to ensure adequate power. Although increasing sample size is an option, logistic considerations and prohibitively high costs often argue against such a prospective design in favor of a case-control retrospective alternative. Linear regression is probably the most popular approach for modeling a response or dependent variable as a function of a set of independent variables. However, a major limitation is that it only applies to continuous response. In addition, the classic normal distribution assumption further limits the applicability of such models to real study data, as data distributions in many studies are often at odds with such a mathematical model. Nonetheless, it remains popular among practitioners and serves a good starting point to discuss the concepts of regression and parametric inference. In Section 2.1, we first discuss the linear regression model and then its extension to the generalized linear model to address the first major limitation. In Section 2.2, we address the second distribution concern by studying a class of distribution-free models.
2.1
Parametric Regression Models
Parametric models are widely used for data analysis in studies in the biomedical, behavioral and social sciences. For regression analysis, one of the
2.1 PARAMETRIC REGRESSION MODELS
65
study outcomes is designated as the dependent or response variable and is modeled as a function of a set of other outcomes or independent variables. For parametric regression models, some parametric distribution is assumed for the conditional distribution of the response given the independent variables. Such an assumption provides the premise for statistical inference of model parameters of interest. Although a variety of parametric models have been developed and used in statistical applications, linear regression is probably the most popular choice for modeling continuous response. In this section, we discuss this class of models along with inference for model parameters.
2.1.1
Linear Regression Model
Consider a study with n subjects and let yi be a continuous dependent variable of interest and xi = (zil,. . . , zip)T a p x 1 vector of independent variables from the ith subject (1 5 i 5 n). Note that unlike Chapter 1, we will use lowercase letters to denote random variables and vectors throughout the rest of the book unless otherwise noted. Interest lies in modeling yi as a function of xi. The linear regression for modeling such a relationship is defined by the following model statement: y, = xTP
+
E,.
T
E,
-
i.i.d. N ( 0 , ~ ”,) 1 5 i 5 n
(2.1)
where P = (pl,p2,.. . ,in,) is a p x 1 vector of parameters, E , denotes the error term in the model, N ( p , a 2 )denotes a normal distribution with mean p and variance c2,and i.i.d. stands for independently and identically distributed. In (2.1), E , 4 . i . d . N (0, a”) means that E , are independently and identically distributed with the same normal distribution N (0, a”). The linear function 7, = x,’P is often called the h e a r predzctor. In the linear regression model (2.1), we are interested in estimating P and making inference about this parameter vector. By viewing (2.1) as a relationship in which y, depends on x,, a known P will enable us to predict the mean response yz for a given x,. This is the reason why y, and x, are often called the dependent (or response) and the zndependent (or predzctor) variables, respectively. In many applications, especially in epidemiologic studies, a subset of x is of primary interest and the variables within such a subset are often called explanatory variables. The remaining variables in x are used to control for heterogeneity of study sample such as differences in demographic and clinical outcomes, and for this reason, they are often called covarzates or
66
CHAPTER 2 MODELS FOR CROSS-SECTIONAL DATA
confounding variables. The presence of such variables in the model (2.1) is to control for the additional variability in yi that reflects the heterogeneity of the subjects in the study sample not explained by the explanatory variables or predictors. In this book, we will not carefully distinguish the different types of independent variables, although such a distinction plays a critical role in causal inference. Because of scale differences between yi and the components of xi, it is typically necessary to include an intercept term in the model (2.1) to calibrate such differences. For this reason, one of the components of xi, say z i l , is usually set to 1 so that the corresponding parameter p 1 serves as this calibration coefficient and the linear predictor of the model has the form:
Example 1 ( A N O V A ) . Consider a study comparing g different treatment conditions, each of which has n k subjects (1 5 j 5 nk, 1 5 k 5 9 ) . Let yi denote the continuous response of interest. We assume that subject i k is in the kth treated group if Cfit n1 1 5 i 5 nl (no= 0 , l 5 k 5 9). Thus, y 1 , . . .,ynl represent data from the first group, ynl+l,. . .,ynl+n2 denote the data from the second group, etc. Define g binary indicators for each of the treatment conditions as follows:
+
1 if subject i receives treatment k
xik =
0 if otherwise Then, the linear model in (2.1) with the linear predictor q i = P 1 x i 1 x i g P g is equivalently expressed as: yi =
+
6 z,’
k-1
k
l=O
1=0
ei
-
+. .+
i.i.d. N (0,o’)
(2.4)
In the above, PI, is the mean of the response yi across the subjects within the kth treatment group. By rewriting yi as y k j when Cfz:n1 15 i 5 CkO nl and changing p k to p k , (2.4) becomes the familiar analysis of variance (ANOVA) model:
+
ykj = pk
f Ekj,
ckj
i.i.d. N ( 0 , 0 2 ) 1 5 j 5
nk,
15 k 5 9
(2.5)
The most popular approach for studying treatment difference is t o compare the mean response p k across the different treatment groups. Thus, under
67
2.1 PARAMETRIC REGRESSION MODELS
this classic treatment comparison paradigm, we are interested in testing the following hypothesis:
H~ : Pk
=p
for all k
versus Ha : ,uk
# pl for some k # 1
(2.6)
For example, if g = 2, we have: Ho : p1 = p2 versus Ho : p1 # p 2 . For g > 2, we are not only interested in whether the group means p k are the same, but also how the difference arises when the means do differ across the groups. Thus, if HO is rejected, we then follow with post-hoc comparisons among p k t o identify the groups that differ from each other. In Example 1, if we use the linear predictor with an intercept, qi = Po + Plzil . . . sigPg,the ANOVA becomes:
+ +
yi = Po
+ PI,+ ~ i , ~i
k-1
k
l=O
1=0
N
i.i.d. N (0, 0 2 )
(2.7)
In this case, P k are not identifiable since the model above involves more than g number of parameters. To address identifiability, some constraint must be imposed. One popular approach is to set one of PI, to 0. For example, by setting P, = 0, we can identify the remaining g parameters, Po, pl,. . . ,pg-l. This cell reference coding method is equivalent to including only the first g - 1 binary indicators z i l , 2 i 2 , . . . , Under this parameterization scheme, Po is interpreted as the mean response for the gth treatment and PI, represents the difference in mean response between the kth and gth treatment conditions. Another common method for identifiability is to impose the constraint, = 0, and then solve for p,. This effect coding method is also widely used by popular software packages, such as SAS. The hypothesis for testing treatment difference in mean response takes different forms depending on the coding schemes used. For example, under the reference cell coding method, (2.6) is equivalent to
C",=,
Ho : Pk = 0 for all 1 5 k 5 g - 1 versus Ha : PI,# 0 for some 1 5 k 5 g - 1
(2.8)
For example, if g = 2, the above yields Ho : P1 = 0 versus Ha : P1 # 0. In many applications, differences in patients with respect to their disease history and comorbid conditions often alter the effect of treatment. Understanding such a relationship not only enables us to obtain unbiased
68
CHAPTER 2 MODELS FOR CROSS-SECTIONAL DATA
treatment effect, but also affords us an opportunity to develop individualized treatment therapy. The next example generalizes ANOVA to model the impact of patients differences on treatment effect. Example 2 (ANCOVA). Within the context of Example 1, let Z k i denote the vector of covariates from the ith subject within the kth treatment condition. Consider the following linear model:
The above is similar to the ANOVA model in ( 2 . 5 ) except for the extra term p z z k , to account for the effect of covariates z k i on treatment differences. If p k = p, that is, z k i does not interact with the treatment conditions, (2.9) is known as the analysis of covariance model (ANCOVA). In this case, ANCOVA allows us to derive treatment effect by controlling for z k i . Otherwise; there exists covariate by treatment interaction. In treatment studies, it is important to identify such interactions to study the differential effect of a treatment when applied to subjects from different study populations, such as different ethnicity, gender, medical and psychiatric history and comorbid conditions so that effective treatment interventions can be develop to account for such patients’ differences.
2.1.2
Inference for Linear Models
For parametric models, the method of maximum likelihood is the most popular approach for inference about model parameters. We first discuss computations and properties of parameter estimates for general parametric models obtained under this approach and then specialize the results to the linear regression model. Consider a sample of n subjects and let yz denote some outcome of interest (1 i s n ) . We assume that the distribution of yi is prescribed by a family of probability density or distribution function (PDF) f (yi I 8) indexed by a q x 1 vector of parameters 8 depending on whether yz is continuous or discrete. Note that since the theory of maximum likelihood applies to both continuous and discrete outcomes, we will refer to either the probability distribution or density function as PDF throughout the rest of the book for the sake of convenience. When the distinction between the two is important, we will be specific about which type of distribution is being considered. The likelihood function is defined as the product of the PDFs from the n responses: L, (8) = f (yi I 8 ) . For a fixed 8, as noted in
<
ny=l
69
2.1 P A R A M E T R I C REGRESSION MODELS
Chapter 1, f (. I 8 ) is a real measurable function with respect to the Bore1 0-field and as such the composite function f (yi I 8 ) is a random variable. Thus, as the product of these random variables, the likelihood L, (8) is also a random variable. When applied to data in a real study, the likelihood becomes a function of 8 conditional on the observed yi. The maximum likelihood estimate (MLE) is defined as the value of 8 that maximizes the likelihood function L, (8)when viewed as a function of 8. As will become clear shortly, the log of the likelihood defined below plays a much more critical role for inference about 8 than the likelihood function itself n
In ( 0 ) = log (Ln( 6 ) )=
C log f (yi I 0 ) i= 1
Since log (.) is a monotone increasing function, the MLE of 8 can be equivalently defined as the value that maximizes the log-likelihood, 1, ( 8 ) . To discuss computations, as well as properties of MLE, we will be taking first and second order derivatives of 1, (8). It is helpful to review the definition of vector-valued function and some basic properties. Let f ( 8 ) be a n x 1 and h ( 8 ) a 1 x m vector-valued function of 8. The derivatives of &f and &g are defined as follows:
(2.10)
Thus, &g = (&gT)T. As a special case, if f ( 8 ) is a scalar function, it T
follows from (2.10) that = (&f,. . . , a f ) is a 4 x 1 column vector. Let A be a m x n matrix of constants, g ( 8 ) a m x 1 vector-valued function of 8 and h ( 8 ) a function of 8. Then, we have (see exercise): 1. (Af) = ( g f ) AT. 2. gj ( h f ) = ( & h ) f T h g j f . 3. (gTAf) = Af ATg. If the maximum of l n ( 8 ) is achieved at an internal point then this value should be a critical point of I , ( O ) , that is, must satisfy the following condition:
8
+ (gg) + (gf)
5,
(2.11)
70
CHAPTER 2 MODELS FOR CROSS-SECTIONAL DATA
As noted earlier, the derivative u, (0) of 1, (0) is a q x 1 column vector and is known as the score statistic vector (when viewed as a function of the random variables yi). The equation in (2.11) is known as the score equation. Like 1, (O), the score statistic vector u, (0) plays an important role in the study of MLE. Example 3 (One-group ANOVA with known variance). If the linear model contains only the intercept, that is, xi = zil 3 1, then (2.1) reduces to the following simpler model:
+
yi = p1 ~ i , ~i
-
i.i.d. N ( 0 , ~ ' ),
15 i 5 n
In this special case, the intercept p1 = E (yi) is the mean of yi and the above linear regression models the mean of yi from a single study population. Assume that a2 is known so that p1 is the only parameter of the model. The log-likelihood function is given by 1,
n
( P I ) = -- log
2
c n
n 1 (an) - - log ( a 2 )- 2 2 2a z.= 1
(Yi -
P1I2
(2.12)
The score statistic un(pl) is given by
By setting u, (pl) = 0 and solving for pl, we obtain the MLE: = B, = 1 ;Cy=lyi. It follows from the law of large numbers (LLN) that the MLE is a consistent estimate of p1 (see Chapter 1 for LLN and definition of consistency). Further, as p1 is a linear combination of i.i.d. normal variates yi, it follows from the properties of normal distribution that N .Ol, . This sampling distribution allows us t o make inference about L72) PI. For example, consider testing the hypothesis: HO : p1 = 0 versus Ha: # 0. Under Ho, we have: p1 N 0, . Thus, the test statistic,
p1
h
p1
(
h
fi'
t=Q, (.)
N
L7
-
(
N
L72)
N (0.1) and the p-value is given by: p = 2 (1 - Q, (It\)),where
denotes the cumulative distribution function (CDF) of the standard normal N ( 0 , l ) . Example 4 ( ANOVA with unknown variance). Now, assume that a2 is unknown in Example 3. Then, the normal sampling distribution of derived in the above example cannot be used for inference about pl. We must consider a different statistic.
p1
71
2.1 PARAMETRIC REGRESSION MODELS T
Let 8 = (p1,02) . The log-likelihood function is the same as the one given in (2.1), but the score statistic vector u (0)now contains the additional component for the derivative of the log-likelihood with respect t o 0 2 :
By setting u ( 8 ) = 0 and solving for 8, we obtain the MLE
=
: (-P1, z2IT
x:=l
2 (yi - y,) is the usual sample variance. It is a wellwhere s i = (n-l)s2 known fact that and s i are independent and xi-l,a x2 distribution with n - 1 degrees of freedom (see exercise). Thus, the MLE of is still the sample mean as in Example 3. Unlike Example 3, however, we cannot perform inference about PI based on the , as u2 is unknown. Fortunately, normal sampling distribution N (pl,
pl
N
B1
<)
by the independence between
pl and s i and the fact that
(n-113: o2
2 Xn-1,
sn t n - l , a univariate t distribution with n - 1 degrees we have: Jsl('l-al) of freedom. This t sampling distribution is widely used for inference about N
P1.
z2
is not the same as the sample variance s i . Note that the MLE of However, as we have shown in Chapter I, S2 and s i are both consistent estimates of u2 with the same asymptotic distribution. Thus, they are asymptotically equivalent. Example 5 (Linear regression). Consider the general linear regression model (2.1) with a general xi and unknown u2. Let 8 = ( pT , oZ ) T denote the vector of model parameters. The log-likelihood function is given by
Using the properties of differentiation of vector-valued function, the score
72
CHAPTER 2 MODELS FOR CROSS-SECTIONAL DATA
vector equation is readily calculated (2.14)
By solving for 6 , we obtain the MLE
=
(BT,32)T (2.15)
If
xi
is regard as fixed, then it is readily shown (see exercise) that (2.16)
( ~ - ~ ) M s-En;i2 where M S E = h
-
7
a2 I& n-P
2 Xn-P
is known as the error mean square.
Again, as in
Example 4, p and S2 are independent (see exercise). As o2 is unknown, we cannot use the multivariate normal distribution in (2.16) for inference about p. However, using the distribution result for in (2.16)) it can be shown that follows a multivariate t distribution (see exercise) :
z2
where t (,u,C,u) denotes a multivariate t with mean p , variance C and degrees of freedom (or shape parameter) u.with the following PDF:
The multivariate t distribution has nice properties similar to those of the multivariate normal. In particular, it is readily shown that any sub-vector
73
2.1 PARAMETRIC REGRESSION MODELS
also has a t distribution (see exercise). Thus, we can make inference for each individual pk using the marginal univariate t distribution of When using the distribution result in (2.17), C:=l is presumed t o be a constant matrix. As noted in Section 1.1 of Chapter 1, the response yi and the set of independent variables xi are often concurrently observed and as such C:=l X ~ X ? also varies from sample to sample. Thus, unless for the special case of a constant xi, the sampling distribution of the MLE in general does not follow the t distribution as given in (2.17). In such applications, we must rely on the asymptotic distribution of MLE for valid inference of model parameters. To discuss the asymptotic distribution of the MLE for linear regression, we first present results concerning maximum likelihood estimates for general parametric models as summarized by the following theorem. To avoid a lengthy list of assumptions for the legitimacy of performing certain mathematical operations, such as exchanging the order of differentiation and integration, we follow the convention of subsuming all such operations under the phrase, “under mild regularity conditions,” throughout the book. Theorem 1. Let yi be i.i.d. random variables with the P D F f (yi 1 0) (1 5 i 5 n ) . Let G denote the maximum likelihood estimate of 8. Then, under mild regularity conditions, we have 1. The maximum likelihood estimate G is consistent and asymptotically normal. 2. The asymptotic variance of G is given by
X~XT
pk.
3
(2.19)
4
T
&
where aeae log f (yi I 8)= (& log f (yi I 0)) . Proof. First, we show that the normalized score statistic vector, u, _ _&2, (O),is asymptotically normal. Note that the normalized u, slightly different from the score statistic vector u, (0) used in the examples with the normalizing factor used to control the rate at u, (8)converges t o 0 in probability. For notational brevity, let wi (0) = log f (yi I 8). Since
A
A
(8) = (8)is above which
&
(2.20)
74
CHAPTER 2 MODELS FOR CROSS-SECTIONAL DATA
it follows from LLN that
By CLT,
Further, since
it follows that
5.
Now, consider the distribution of MLE For convenience and without the loss of generality, assume a scalar 8 in which case, u, (8) is also a scalar. First, we show consistency of 8. Consider a neighborhood N ( 8 ) of 8. h
Then,
75
2.1 PARAMETRIC REGRESSION MODELS
It then follows from a Taylor series expansion (Chapter 1) that
where
5 E N (6’) and 15‘1 < 1.
a
-un(0)=E
-Wi(O)
ax
= -E =
-vg
It follows from LLN that
1
[a x2
1
+ o p ( l ) = E 7 l o g f ( y z 10) + o p ( l )
[axa 1% f (YiI 0) + o p (1)
1’
+ op (1)
Thus,
where z, = op (1). For each integer k , let
Since z, = op (l),we can find N i such that k
1
For such n, we have: 1
U,
(x2) = --21 (e)-k + zn < 0,
un
(xl)= (e)-k1 + x, > o
Since u, (A) is continuous in A, the interval [XI, A21 contains a solution of the likelihood, that is, u, (7) = 0 for some r j E [XI, A,]. Choose some n > Ni k and let
76
CHAPTER 2 MODELS FOR CROSS-SECTIONAL DATA h
Then, Q,+ is a random sequence (see exercise). Further,
Thus, S,+is a consistent estimate of 19. Next, we show the asymptotic normality of$. It follows from a stochastic Taylor series expansion that U,
(Q)
1
(5- 8 ) + ZChn (5- Q )
2
(2.23)
151 <
1. Since [(h,l 5 lhnl,Ch, = 0, (1). Solving the above for - Q) yields
where
(g
a (5) = un( Q ) + -un ae
Since
a
aQ
-Un.
( 0 ) + op (1) +p -vQ,
h U n
( 0 ) +d N (0,'Ug)
it follows from Slutsky's theorem that
By generalizing (2.24) to the case of vector 8, we have:
with the asymptotic variance CQgiven in (2.19). The results in this theorem provide the basis for inference about 8 for parametric models f (y 18) when the sampling distribution of MLE 6 is not in the form of an available analytic distribution such as the normal or t distribution. In particular, we can use Theorem 1 to address inference about p for the linear regression model in Example 5 when is not a constant matrix. Note that the asymptotic variance CQof 8 in Theorem 1 is not the variance of 6, but rather the variance of the limiting normal of & 8 - 8 . In fact, 8 follows approximately a normal distribution
x7ElX~XT
(-
1
h
N (8,;CQ) with mean 8 and variance
for large samples.
77
2.1 PARAMETRIC REGRESSION MODELS
Example 6 . For the linear regression model in Example 5, it follows from (2.14) that the second derivative of the log density function log f ( y i I 0 ) with respect to 8 = (PT ,cr2 ) T is
By taking the expectation of the above (see Chapter 1 for review of expectation and conditional expectation), we have
Since
it follows that
Thus, Co is the asymptotic variance of the MLE 6 = (BT,C2) T. Further, it follows from (2.27) and properties of multivariate normal distribution that the marginal vector P is also asymptotically normal with the asymptotic variance Cp = cr2E-l ( X i X T ) . By Slutsky’s theorem, a consistent estimate of Cp is given by Cp = l?2 xixT)-’. Thus, for large n, follows h
h
(ixy=l
B
78
C H A P T E R 2 MODELS FOR CROSS-SECTIONAL DATA
approximately the following normal distribution:
where A N (8,C) denotes the asymptotic or approximate normal distribution with mean 8 and variance C . Note that the variance estimate ig3 in the asymptotic distribution of in (2.28) is the estimated asymptotic variance gp divided by the sample size n. By comparing the above with (2.16), it is clear that even when xi varies from sample to sample, the normal sampling distribution of in (2.16) still provides valid inference of ,kl for large sample
B
n.
-T
It is seen from (2.27) that ,kl and z2 are uncorrelated in the asymptotic distribution. Since uncorrelatedness implies independence for variables fol-T lowing the multivariate normal distribution, p and 3' are asymptotically independent. This independence makes it possible to use Cp as the asymptotic variance for inference about ,kl since otherwise the inverse of CQ would also involve the covariance between and Note that the asymptotic variance of the marginal C 2 in (2.27) suggests that we can also make inference about the nuisance scale parameter u2 using the asymptotic normal distribution: Z2 AN rather than the x2 distribution in (2.16) for large samples. For fixed xi, this asymptotic distribution can also be obtained from the xi-p distribution of ;i2 given in (2.16) (see exercise). One of the primary reasons for the popularity of maximum likelihood inference is the optimal property that maximum likelihood estimates are asymptotically most efficient. This is best discussed by introducing the notion of information matrix. As before, let 1, ( 8 ) denote the log-likelihood and let
8
N
e2.
(02,g)
The I: ( 8 ) above is known as the observed information matrix. By taking expectation, we obtain the expected or Fisher information matrix as follows:
Thus, it follows from Theorem 1 that the asymptotic variance of 6 can be expressed as the inverse of the expected information matrix, that is,
79
2.1 PARAMETRIC REGRESSION MODELS
Ce = [If(0)]-'. The identity (2.29) shows that the expected information for a sample of size n is the sum of the individual expected information from each individual If (0). It is readily checked that I: (0) = V a r (u, (f?)), where u, (0) = &ln (0) is the score statistic vector (see exercise). Example 7. In Example 6, we computed the second derivatives of the log-likelihood function for the linear regression model (2.1). It follows from (2.25) that
(2.30)
As noted earlier in Example 6, the off-diagonal block of I: (0) is zero. Thus, we can simply use the inverse of the marginal information for P , [ I f as the asymptotic variance for inference about the parameter vector of interest p. Note that the off-diagonal block of the observed information matrix I: (0) also vanishes when substituting the MLE 6 in place of 8 . For a scalar variable 8, I , (0) is a function of 8 conditional on the observed yi (1 5 i 5 n). It follows from calculus that for a concave (down) function g, the second derivative of g provides the degree of curvature for measuring sharpness of the function around the point at which g achieves its maximum. Thus, 1; (0) measures the curvature of the log-likelihood function I , (0) around the MLE while If (8) measures the same curvature averaged over the joint distribution of yi. A highly peaked 1, (8) around yields a large second order derivative (in absolute value) which in turn leads to a smaller asymptotic variance 0 ; or more accurate estimate 0 of 8, while a more blunt curve from I , ( 8 ) corresponds to a lower second derivative and a larger asymptotic variance or less accurate estimate. For vector 0, the interpretation is similar except that I; (0) and If (0) measure the curvature of a surface defined by I , (0) around the MLE 6. The Fisher information matrix If(0) not only provides the asymptotic variance of the MLE, and but also furnishes an asymptotic bound for all estimates satisfying certain regularity conditions. To illustrate, consider a parametric model for continuous response defined by the density function f (y; 0 ) . For convenience, assume a scalar 8. Given an i.i.d. sample yi from this model (1 5 i 5 n ) , let be the MLE of 8 and D; the asymptotic
,'-I)@
5,
5
h
5
80
CHAPTER 2 MODELS FOR CROSS-SECTIONAL DATA h
variance of 8. If T, = T,(yl,.
. . ,y,)
is any unbiased estimate of 8, then
E(GJ = / T , f , (yl,. . . , Y n ; 8) dyl -dy,
=
n
f (yi; 8) is the joint density or likelihood func-
where f, (yl, . . . , y,; 8) = i=l
tion. It follows that
cov (T,, u,)
= E(Tnun)=
/
T n y f n(YI,. . . , y,; 8) dyl . . dy,
(2.31)
=1
As in Theorem 1, we assume in (2.31) the regularity conditions t o ensure the validity of the change of order of integration and differentiation. By Cauchy-Schwartz inequality, Cov (T,, u,) 5 J V a r (T,) V a r (u,) (see Chapter 1). Thus, it follows from (2.31) and Theorem 1 that (2.32) The above inequality, known as the Cram&-Rao bound, shows that the variance of any unbiased estimate T, is as large as the asymptotic variance of the MLE The Cramkr-Rao bound in (2.32) also holds for vector 0. In this case, T, is a vector, I: (0) and Is (0) are matrices, Ce is the asymptotic variance of the MLE 6 (or MLE-based estimate $ 0 ) and 1 in (2.32) means that
2.
(3
the symmetric matrix Var (T,)- [I; (0)I-l is non-negative. More generally, for any vector-valued, differentiable function $ (O), it follows from the Delta method (see Chapter l ) ,
lh ($ (6) - $ (0)) -'d
g.
N (0, Z$ = DT(0) COD (0))
+
(2.33)
where D = If T, is any unbiased estimate of (0), then it is readily shown that (2.32) is also true with 0 ; replaced by the asymptotic variance
Eg of the estimate $
given in (2.33) (see exercise).
81
2.1 P A R A M E T R I C R E G R E S S I O N MODELS
For consistent estimates, the Cram&-Rao bound does not apply directly. However, it is natural to anticipate that there may be a similar asymptotic bound for comparing the asymptotic variance of consistent estimate with that of the MLE. This turns out to be true and there is an asymptotic version for the Cram&-Rao bound for consistent estimates that are regular and asymptotically linear. For this reason, the Fisher information can be used to compare and define asymptotically efficient estimates in large sample theory. An estimate is asymptotically efficient at 8 for estimating a differentiable ( 8 ) if it is regular at 8 and asymptotically normal with the asymptotic variance given by the Cram&-Rao bound. Thus, the MLE is asymptotically efficient for parametric models.
+
2.1.3
General Linear Hypothesis
When making inference about a single model parameter, such as regression coefficient /3 for a continuous or binary independent variable, hypotheses can be expressed as:
Ho : p
= a, versus
Ha : p # a
(2.34)
where a is a known constant. For example, if there are two treatment conditions in Example 4 of Section 2.1.1, g = 2 and under the cell reference coding method, p1 represents the difference in mean response between the two treatment conditions. The null HO in (2.34) with a = 0 tests the hypothesis of no between-treatment difference. Testing for hypotheses of the form (2.34) can be carried out using the distribution of the parameter estimate. Example 8 ( Two-group A NOVA). Consider comparing two treatment conditions using the linear model in Example 4 of Section 2.1.1. Under the cell reference coding method, yz = Po
+ plxzl + ez,
ez
-
i.i.d. N (0, a 2 )
where yz is from the first treatment if 1 5 i I n1 and from the second if nl+l I iI 722 ( n = n1 n2). Let ,B = (Po,Bl)T. It follows from Example
+
5 of Section 2.1.2 that the MLE
fi(B-’) MSE
has a bivariate t distribution,
t 0, XB = (i CG1x,x;)-l , n - 2).
Since the marginal
distribution, we can test the null Ho :
P1 = 0 using the statistic
(
Bl
also has a t
-
t 0~:~. n - 2), where a$l is the diagonal element of Xp corresponding to
(
P1.
82
C H A P T E R 2 MODELS FOR CROSS-SECTIONAL DATA
Many hypotheses of interest, however, cannot be expressed in this simple form. For example, if testing the hypothesis in Example 8 under the cell mean or ANOVA model (2.5) in Example 3 of Section 2.1.1, the hypothesis is given by: Ho : p l - p2 = 0 versus p, - p2 # 0 (2.35) which is not in the form of (2.34) as it involves more than a single parameter. The general linear hypothesis for a model parameterized by a p x 1 vector 8 has the following form:
Ho : KO = b versus. Ha : KO # b
(2.36)
where K is some full rank matrix of dimension 1 x p and b some 1 x 1 column vector, all consisting of known constants ( I 5 p ) . When b = 0, the null in (2.36) is called a linear contrast. Example 9 (Three-group ANOVA). In Example 8 above, if comparing three treatment conditions, the linear model under the cell reference coding is given by yi = Po
+ plzil + P2zi2 + ~ i , ~i
N
i.i.d. N (0, c 2 )
cj";:
Cj"=,
where yi is from the kth treatment if nj+l 5 i 5 n j , with no E 0 and n k denoting the sample size of the kth treatment group (1 5 k 5 3). The null of no difference across treatments can be expressed as a linear contrast:
For the linear regression in (2.1), if xi are fixed such as by design in Example 7, follows a multivariate normal given in (2.16). From the definition of x2 distribution, we have
3
3
Since and MSE are independent, so are Q i and MSE. Thereby, it follows from (2.16) and (2.37) that
(KB-b)T [K(C:=,xix:)-'KT]-'
Qill MSE/cr2
-
MSE
(Kb-b)
T
-
q n - p
83
2.1 PARAMETRIC REGRESSION MODELS
where F,,, denotes an F distribution with the degrees of freedom u for the numerator and w for the denominator. This F distribution is widely used for inference about the general linear hypotheses. When xi are concurrently sampled with yi, we use the asymptotic distribution of the MLE for inference about the general linear hypothesis. The two most popular approaches for testing the null of linear hypothesis are the Wald and likelihood ratio tests. If 6 A N (0, ;&I), then it follows from the properties of multivariate normal distribution that K6 N (KO,i K C 6 K T ) . Under the null, K6 AN (b,; K & K T ) . By a slight abuse of notation, let Q i denote the Wuld statistic obtained from (2.37) by replacing Cp with g p . It follows from Slutsky’s theorem that QL has an asymptotic xf distribution for large n. Alternatively, we can use the likelihood ratio test (LRT) to examine the general linear hypothesis. Let L, (0) denote the log-likelihood function and let 6 the MLE of 8. Let 60 be the MLE of 8 under the null Ho in (2.36). Then, the likelihood N
N
N
ratio based statistic, -2 In R (60) = -2 In
(-)
, has an asymptotic xf
distribution under HO as asserted by the following theorem. Theorem 2. Under mild regularity assumptions,
Proof. Let u, (6) = :$&. We prove (2.38) in three steps. Case 1. First, assume K = Ip. In this case, the null in (2.36) reduces to Ho : 8 = b. For notational brevity, we continue to use 8, but with the understanding that 8 is a known vector under Ho. By (2.23), we have
(-1
o = u, e
a (e)+ -u, (e) (e e) + op (d) aeT h
= un
-
It follows that
(G - e)
T
u,
(el = - (G - e) T -u,a
aeT
(el (6 - e) + (G - e)
T
op
(n-i)
84
C H A P T E R 2 MODELS FOR CROSS-SECTIONAL DATA
(e), we have
By substituting $$$!j for u,
(2.39)
Since
it follows from (2.39) and (2.40) that
(6 - 6 ) Now, expand 1, 1%
e -~,(e) (3
81, ( 0 ) ae
=
(G - e)
T
I;
( e l . (G - e) + op (1)
(2.41)
(-1
8 around 1, ( 6 ) to obtain
(G-e)
81, ( 0 )
+ 1 (e- - e)
a21n
=
ae
(2.42)
+
aeaeT (G - e> + op (1)
By combining (2.41) and (2.42), we have
=
(G - e ) '
I;
(e) (G - e) + op (1)
Since
6(6 - 0 ) *d
N
(01(1; (el)-') ,
1
111; ( 0 ) * p If (e)
(2.44)
the conclusion follows from (??), (2.44), and an application of Slutsky's theorem. Case 2. Assume K = (Il,O) with 11denoting the 1 x 1 identity matrix. Then, the null becomes HO : 81 = b. In this case, the first 1 x 1 subvector
85
2.1 PARAMETRIC REGRESSION MODELS
T T T of - 0 = (6, ,8, ) is known under HO and the MLE of 8 under HOis simply 62 (b),which may depend on b. Note that we use different notation for the MLE under HOsince 6 2 is generally different from the corresponding ( p - 1) -T -T 6 = (0, , 0 2 ) . For convenience, we use 81 for b and denote 5, (b) simply by (32.
subvector
92
Let I: =
of the unconstrained MLE
(
In11 Ig12
)
notational
. By a similar argument as in the proof of Case
1, we have
By the consistency of that
6 and 6 2
under
Ho,it
can be shown (see exercise)
It follows from (2.45) and (2.46) that
Case 3. Consider HO in the general form given in (2.36) with 1 < p . By viewing the rows of K as 1 linearly independent vectors in Rp, it follows from the theory of linear algebra that there exists p - 1 linearly independent vectors such that together with the rows of E( they form a basis for R p . Let G be a matrix whose rows are formed by these p - 1 vectors. Then,
86
C H A P T E R 2 MODELS FOR CROSS-SECTIONAL DATA
((T, (z)T
[ = (KT,GT)T 8 = is a linear transformation of 8. Under Ho, El =-KO = b and the MLE of E2 is (b). It follows from Case 2 that +d X r . The conclusion follows from expressing $ and (El) -2
i2
wg
i2
L(0
e2 )
-T
in terms of 8 as 6 = (KT,GT)-T and 60 = ( K T ,GT)-T (bT, and the invariance of the likelihood function under such linear transformations. Example 10. In Example 9, let xi = ( l , x i l , x ~and ) ~ consider the linear hypothesis:
Let Eo = Po,
E l = P1 - B 2 ,
Then, when expressed in terms of ~i = Eo
+ (51 +
E 2 ) xi1
E
= (E0,
El, E2)T,
+ E2xi2 + ~ i ,
Under Ho, El = 0 and (2.48) reduces to
+ E2 (xi1 + xi2) + ~
yi = to h
h
i ,ci
(2.47)
E2 = P 2
the linear model is
-
i.i.d. N (0,0 2 )
(2.48)
-
i.i.d. N (0: 0 2 )
(2.49)
T
h
L e t t = ([0,t1,E2,272) a n d t o = ( ~ o ~ ~ 2 , 3 2 ) T b e t h e I \ I L=E(ET,02)T of~ for the full and reduced models in (2.48) and (2.49), respectively. Then, T the MLEs of 8 = @',a2) for the respective full and reduced models are readily obtained by transforming and to the scale of 8 by (2.47). For example, the MLE 6 for the full model is given by
2
h
to
h
Po=Eo,
B1=&+Z2,
It follows from Theorem 2 that -2 logR
xl
B2=Z2
(50)
=
-21n
(a) has an
Ln(@>
approximate distribution for large n, where L, likelihood functions corresponding to the full and reduced models in (2.48) and (2.49). Some software packages may only implement tests for linear contrasts. When using such software, we have to rewrite the general linear hypothesis in terms of linear contrast:
HO: KP - b = 0 versus Ha : K P - b # 0
87
2.1 PARAMETRIC REGRESSION MODELS
By performing the transform y = ,B - KT (KTK)-’ b and reparameterizing the linear model in terms of y, it follows from (2.1) that yi
- xTKT (KTK)-’ b = xiT y
+ ~ i , ~i
N
i.i.d.
Thus, by redefining the response, zi = yi - xTKT (“‘“)-I b, we can test the original null through a linear contrast Ho : K y = 0. Alternatively, we can use an oflset term without performing the linear transformation. We discuss this alternative approach in Section 2.1.5.
2.1.4
Generalized Linear Models
The linear regression model discussed in the preceding sections assumes a continuous response y. In applications, other types of dependent variables also arise. For example, binary response is quite popular in many areas of research such as depression diagnosis in mental health studies and infection of the human immunodeficiency virus (HIV), in investigation of infectious disease. Since such a variable only takes on two values and the linear predictor 17 of the linear regression model is unconstrained and can in theory range from --x to m, linear regression is generally inappropriate for modeling such a type of response. The generalwed h e a r model (GLM) is a generalization of linear regression to accommodate different types of response, including binary response. This general class of models frames a wide range of seemingly disparate problems of statistical modeling and inference within an elegant unifying framework. This new integrated modeling approach provides great power and flexibility to model a wide variety of response variables. Examples of generalized linear models include linear regression, logistic regression and Poisson log-linear models. To appreciate the limitation of linear regression and set the stage for GLM, let p, = E (9, I xz),the conditional mean of yz given xz.and express the linear regression model in (2.1) equivalently as: Yz I x,
p, =
-
N (P,, 0”)
(yz 1 x,) = r l z = x
(2.50)
iI n p , 1I
where i.d. denotes zndependently dzstrzbuted. It is clear from (2.50) that p, is a linear function of x,. In addition, the right side of the model, qz = XTP, has range in the real line R, which concurs with the range of p, on the left side. If y, is binary. however, pi = E (yi I xi) = P r [yi = 1 1 xi]
(2.51)
88
CHAPTER 2 MODELS FOR CROSS-SECTIONAL DATA
The conditional mean pi in (2.51) is a value between 0 and 1. In this case, it does not make sense to equate pi with the linear predictor qi as for the linear regression model in (2.50). In addition, the normal assumption does not apply to binary response. The generalization of linear regression to accommodate other types of response is made possible by modifying the following two components of the linear regression in (2.50): 1. Random component. This part specifies the conditional distribution of the response y given the dependent variables x. In the linear regression, a normal distribution is assumed. 2. Systematic component. This part links the conditional mean of y given x to the linear predictor x by a one-to-one function g: g ( p ) = q = p1.1
+ + p,. * * *
=x
'p
(2.52)
In the linear regression model, g ( p ) = p, an identity function. In the nomenclature of generalized linear models, g (.) is called a link function. By varying the distribution function for the random component and the link function in the systematic part, we can use the class of GLM to model many different types of response. E x a m p l e 11 (Models for binary response). Consider a sample of n subjects. Let yi be a binary response of interest and xi a set of independent variables. Logistic regression is the most popular choice to model the relationship of yi as a function of xi. As a member of GLM, the random component specifies the conditional distribution of yi given xi as a Bernoulli or binomial with sample size 1, with the probability of success given by E (yi I xi) = T (xi)= ~ i . For the systematic part, the conditional mean pi is linked to the linear predictor by the logit function: g(n) = log ( T / (1 - T)). Thus, this logistic model in terms of a GLM is expressed as: yi 1 x i ~ i . d B . I ( T ( x % ) ; l ) ,g(T2)=log(n,/(l-.rri)),
l l i l n
where BI(T; 1) denotes a binomial distribution with sample size 1 and probability of success T. The log transform of the odds T/(1 - T ) is also known as the logit transform of T . In addition to the logit link, other popular functions used in practice for modeling binary response include 1. The probit link: g ( T ) = @-'(T),where a(.)is the cumulative distribution function of the standard normal. 2. The complementary log-log function: g(x) = log(- log(1 - T)).
89
2.1 PARAMETRIC REGRESSION MODELS
Example 12. In Example 11, consider a special case with a single binary predictor xi such as gender with xi = 1 for male and xi = 0 for female subjects in the study. Then, ~ ( x=) P r ( Y = 1 I x) is the conditional probability of response y = 1 given x. The systematic component of the logistic model is logit [ ~ ( x = ) ]logit [Pr(Y = 1 1
X
= x)] = log
( :t&) Po + =
PlX
It follows from the above that the log odds ratio OR(odds of response of male to that of female) is
=
(Po + P d
-
Po = P1
Thus, the parameter P1 is the log odds ratio or, equivalently, exp(P1) is the odds ratio for comparing the response between the male and female subjects. As in linear regression, we can develop ANOVA like GLMs for comparing multiple treatment groups using the different coding methods discussed in Section 2.1.1. For example, within the context of Example 11, suppose there are three treatment groups and interest lies in comparing the mean response of yi across three treatment conditions. Let X i k be a binary indicator for the kth treatment condition defined in (2.3) with 1 5 k 5 3. Then, under cell reference coding, the linear predictor is q i = = Po ,B1xil P2xi2 and the systematic component of the logistic model is
x T ~ +
logit xi)] = log
(:
$12))
= Po
+
+ PlZil + P22i2
The log odds for each treatment condition is given by (2.53) For group 3 : log
(""> 1
- 7r3
= Po
where 7rk is the mean response of y for group Ic. From (2.53), we obtain the log odds ratios for comparing groups 1 and 2 to group 3:
90
CHAPTER 2 MODELS FOR CROSS-SECTIONAL DATA
Thus, under this coding method, the intercept Po is identified as the log odds for group 3 . while PI and Pz represent the log odds ratios for comparing each of the remaining groups to this group. Note that there is no particular reason to use group 3 as the reference level and any treatment condition can be designated as the reference group. In a case-control study on examining the relationship between some exposure variable and disease of interest, we first select a random sample from a population of diseased subjects or cases. Such a population is usually retrospectively identified by reviews of patients’ medical history and records. We then randomly select a matched sample of disease-free individuals or controls from a nondiseased population based on similar demographic and clinical variables. Thus, unlike most prospective studies, the number of diseased and nondiseased subjects are fixed a priori, while the number of exposed and nonexposed subjects are random under such a retrospective study design. Example 13 (Logistic model f o r case-control study). In example 11, suppose that yz and x, represent the status of disease and exposure of interest. Let z be a dummy variable denoting whether a subject is sampled or not from a population of interest. Since the sampling process is independent of the exposure status, we have p l (x)= Pr [ z = 1 I y = 1.1~1 = Pr [ z = 1 I y = 11 = p l
~o(x)=Pr[z=1/y=0,z]=Pr[x=lIy=O]=po It follows from the above and the Bayes’ theorem that the disease probability among sampled individuals with exposure status IC is
+
where p;F, = Po log(pl/po). We can see that the logistic model for the retrospective case-control study sample has the same coefficient B1 for the exposure variable xi as the model for the prospective study sample, except for a different intercept. For this reason, the logistic model is also widely used in case control studies to assess the relationship between exposure and disease variables, provided that the intercept is interpreted with caution. All three link functions are similar (McCullagh and Nelder, 1989). In particular, the logit and probit functions are almost identical over the interval 0.1 7r 0.9. Thus, it is usually difficult t o discriminate between
< <
91
2.1 PARAMETRIC REGRESSION MODELS
these two link functions on the grounds of model parameter estimates. Although any of the three link functions above [and other functions that map ( 0 , l ) onto to R] can be applied to link the mean of response to the linear predictor, the logit link is the most popular. One reason for its popularity is its simple interpretation as the logarithm of the odds of response and the fact that odds ratios are functions of model parameters. Second, the logit link is the only link function that can be applied to both prospective and respective case-control study designs without altering the interpretation of model parameters. The logit link is also known as the canonical link for modeling binary response. This term is derived from the exponential family of distributions. If y is from the exponential family of distributions, the PDF of y can be expressed in the form:
(2.54) where 6' is the canonical parameter (if 4 is known) and 4 is the dispersion or scale parameter. The exponential family includes many distributions, such as normal, binomial, and exponential. For example, in the normal distribution case N (p,a'), 6' = p, $ = a2 and a (a') = a'. However, 6' is not the mean (or probability of success) for the Bernoulli or binomial BI(.lr,1) with sample size 1. The canonical link is defined as a one-to-one function g (.) that maps the mean p = E (y) of y to the parameter 8, that is, g ( p ) = 6'. It is readily shown that for the exponential family defined above, p = $ b (6') = h (0) , and hence the canonical link is g ( p ) = h ( p ) - l (see exercise). When modeling binary response, y follows a binomial BI(n-, 1) with the following distribution function:
f (y 1 n-) = n-y (1 - 7 r ) l - Y where
.lr
= exp
CY 1% (n-/ (1 - 4) + 1% (1 - 741
= E (y). It follows from (2.54) that for this particular model,
By definition, the canonical link is g (T)= log A. Alternatively, since d ee the canonical link is also given by the inverse of this derivBbb6') = - 7 ative, g (n-) = log (A). Thus, the logit function is the canonical link for modeling binary response. Similarly, it is easily shown that the identity function is the canonical link for modelling a continuous random variable following the normal distribution.
92
CHAPTER 2 MODELS FOR CROSS-SECTIONAL DATA
Except for these key differences, however, all three link functions have similar interpretations. For example, as they are all monotone increasing functions, the corresponding model parameters ,B indicate the same direction of association (signs) even though their values are generally different across the three different models. In many applications, we often encounter multi-level response variables which cannot be collapse into two-level binary response. For example, classification of blood types is a four-level response with qualitative categories 0, A, B, AB. In cancer research, cancer patients are classified into four stages from I to IV to indicate disease severity and prognosis. In mental health research, subjects may be diagnosed as one of the three categories, no depression, subsyndromal depression (SSD) and major depression, with patients diagnosed with major depression being more paralyzed mentally than those with SSD, which typically show some symptoms of depression. If we study such polytomous responses further, we can identify them as one of the following types: 1. N o m i n a l scale. The categories of such a response variable are regarded as exchangeable and totally devoid of structure. For example, the color of cars and the type of disciplines for graduate studies belong to such a response type. 2 . Ordznal scale. The categories of this type of response are ordered in terms of preference or severity. For example, cancer staging and depression diagnoses produce responses that are ordered based on the severity of disease of interest. For nominal response, the generalazed logit or multinomial response model is the most popular approach. Suppose that a nominal response y has J categories. This model designates one category as a reference level and then pairs each other response category to this reference category. Usually the last category is chosen as the reference level. Of course, for nominal response, the “last” category is not well defined, as the response categories are exchangeable. Thus, the selection of the reference level is arbitrary and is usually a matter of convenience. To define the generalized logit model under GLM, let yz = (yzl,. . . , ~J,J)’ be a vector of binary responses yzJ from the ith subject with yzg= 1 if the response is in the j t h category and yzg= 0 if otherwise (1 5 j 5 J ) . Let ~ i =j ; ~ r j(xi)= P r
( y23. . = 1 I xi), j = 1 , .. . , J
the conditional probability of response in the j t h category given xi. The
93
2.1 PARAMETRIC REGRESSION MODELS
generalized logit model is defined as follows: yi I xi
N
i.d. MN (ri; 1),
ri = (ril,. . .
.
ri~)
(2.55)
where M N ( r ; 1) denotes the multinomial distribution with sample size 1 for the random component and a = ( ~ 1 , ... , a~ - 1 is) a~ vector of parameters of the baseline generalized logits in the absence of x. When J = 2, (2.55) reduces to the logistic regression for binary response. These J - 1 logits determine the parameters for any other pairs of the response categories:
From the defining equation of q j , we obtain the probability of response of the j t h category:
By setting c t ~= 0 and p J = 0 and including j = J in the modeling of generalized logit, we can express the probability of response for all J categories in a uniform fashion as:
Example 14 (Generalized logit model for nominal response). Consider a three level depression diagnosis outcome y with 0 for no depression, 1 for SSD and 2 for major depression. Although these categories are ordinal, for illustration purposes, we treat them as nominal and model the response using the generalized logit model. For convenience, consider a single binary predictor, gender denoted by z,with z = 1 for male and z = 0 for female subjects. If y = 0 is selected as the reference level, then the generalized logit model is given by
94
CHAPTER 2 MODELS FOR CROSS-SECTIONAL DATA
It then follows that
as the conditional odds of SSD for the subjects, we may interpret PI as the log odds ratio of SSD for comparing the male and female subjects within this subgroup. Similarly, p2 may be interpreted as the log odds ratio of major depression for comparing the male and female subjects within the subgroup of major depression and no depression subjects. For ordinal response, it is natural to model the cumulative response probabilities. Consider a J-level ordinal response y. Again, let yi = T (yil, , . . , y i ~ ) be a vector of binary responses with yij = 1 if the response is in category j and yij = 0 if otherwise (1 5 j 5 J). The popular GLM for ordinal response is the proportional odds model: yi I xi
N
i.d. MN (ri; 1) ,
ri = ( n i l , .. . , r i ~ )
(2.56)
k is the cumulative probability of response up to where yJ (xi) = C”,=,~ i (xi) and including category j conditional on the vector of independent variables xi. Unlike the generalized logit model, logit transform is applied to the cumulative probabilities yj (x). Since y3 (x) increases as a function of j , the logit transform of y3 (x) is also a monotone increasing function of j and as a result the aj’s must satisfy the order constraint in (2.56). Of course, these two sets of probabilities yj (x) and riTg (x) are equivalent, that is, one completely determines the other. However, models based on cumulative probabilities are easier to interpret for ordinal response than similar models based on the categorical probabilities of individual responses. Example 15 (Proportional odds model f o r ordinal response). Consider Example 14, but now model depression diagnosis using the proportional odds model. The systematic part of the model is given by:
It then follows that
95
2.1 P A R A M E T R I C REGRESSION MODELS
For j = 0, the right side above is the log odds ratio of no depression (versus SSD or depression) for comparing the male and female subjects, while for j = 1, it becomes the log odds ratio of no depression or SSD (versus depression) for comparing the male and female subjects. Under the proportional odds model, the two log odds ratios (or odds ratios) are assumed to be the same. This is in stark contrast with the generalized logit model where the generalized log odds ratios are allowed to change as a function of the response categories. In applications, we can check the proportionality by first assuming different ,Bj and then testing the null hypothesis of a common ,B. With the general specification of GLM, not only can we readily model other types of response, such as continuous and binary, but also the same type of response with different link functions, such as logistic and probit. The next example shows that we can also change the random component when modeling the same response variable. Example 16 (Linear regression with t distribution). Consider the linear regression in (2.50). The random component assumes a normal distribution for the continuous response yi. A major limitation of the normal distribution is that its tails decay to 0 too rapidly at an exponential rate, making it ill-suited for modeling data distributions with relatively thicker tails. A popular alternative is to use a distribution that does not decay at such a fast rate, such as the t distribution. Let
yi I xi
i.d. t ( p i , a 2 , w )
I
pi = E (yi xi) =
XTP,
(2.57)
15 i 5 n
where t ( p , 0 2 , w ) denotes a t distribution with mean p, scale parameter cr2 and degrees of freedom w. The above GLM has the same systematic part as the normal-based model in (2.50), except for a different distribution assumption in the random component. In comparison to the normal-based counterpart, the model in (2.57) accounts for thicker tails in the conditional distribution of yi given xi through an appropriate choice of the degree of freedom w. Since t ( p , cr2, w) converges t o N ( p ,cr2) as w + 00, the t-based linear regression (2.57) is expected to provide estimates similar to those of its normal-based counterpart (2.50) for large w. Another popular type of outcome in many applications is count response. Count response such as the number of heart attacks, suicide attempts, abortions and birth defects arises quite often in studies in the biomedical, behavioral and social sciences. Since count response only takes on non-negative integers, linear regression is inappropriate for modeling this type of response.
96
CHAPTER 2 MODELS FOR CROSS-SECTIONAL DATA
On the other hand, unlike the other discrete responses considered above, count response cannot be expressed in the form of proportions as the range of such a response is theoretically unbounded. Example 17 (Poisson log-linear model for count response). Consider a study of n subjects and let yi be a count response. The most popular approach for such a response is the Poisson log-linear model with the random and systematic components specified as follows: yi I xi
-
(2.58)
i.d. Poisson(pi) T
log ( p i ) = log ( E (yi 1 xi)) = q i = xi ,B, 1 5 i
I: n
where Poisson(p) denotes a Poisson distribution with the following probability distribution function:
(2.59) Since p E (0, cm),log (.) is an appropriate link as it maps (0,co)to (--00, m ) , the range of the linear predictor 7 . One major problem with the Poisson log-linear model is overdispersion. Under the model assumption, it is readily checked that the conditional mean E (yi I xi) and variance V a r (yi I xi) of yi given xi satisfy (exercise):
E (yi I xi) = Var (yi
1 xi) = p i
(2.60)
In many applications, the conditional variance V a r (yi I xi) often exceeds the conditional mean pzr causing overdispersion and making the Poisson model inappropriate for modeling such data. When overdispersion occurs, the standard errors of the parameter estimates of the Poisson model are artificially deflated, giving rise to exaggerated effect size estimates and false significant findings. One common cause of overdispersion is data clustering. Within the context of cross-sectional studies, clustered data often arise when subjects sampled from within a common habitat (cluster), such as families, schools and communities are more similar than those sampled across different habitats, leading to correlated responses within a cluster. The existence of data clusters invalidates the usual independent sampling assumption, rendering statistical methods developed based on independence of observations inapplicable to such data. Overdispersion can often be empirically detected by goodness of fit statistics for generalized linear models. When deemed present, overdispersion
2 . 1 PARAMETRIC REGRESSION MODELS
97
may be corrected post hoc by using robust estimates of the asymptotic variance rather than the one based on the MLE in Theorem 1. We discuss this option in Section 2.2 after introducing the distribution-free generalized linear models. A better alternative is to use models that directly address data clustering, the cause of overdispersion. One popular approach is the mixed-effects model, which employs latent variables or random eflects t o model the effect of data clustering. We discuss this general approach for modeling clustered data in Chapter 4. Below, we consider one particular example of this class of models for count response. Example 18 (Negative binomial log-linear model for count response). In Example 17, if overdispersion occurs, we can no longer model the count response using the Poisson log-linear model. A popular approach for modeling such overdispersed data is the negative binomial model: yi
I xi
N
i.d. NB ( p i ,a )
(2.61)
where NB(p, a ) denotes the negative binomial distribution with the following probability distribution function: (2.62) p>0,
a>0,
y = 0 , 1 , . ..
It is readily shown that the mean and variance of the NB distribution are given by (see exercise):
Thus, unless Q = 0, the variance is always larger than the mean p , addressing the issue of overdispersion. Note that Q indicates the degree of overdispersion and as a I 0 (decreases to 0) this dispersion tends t o 0 and thereby f N B (y) -+ f p (y) (see exercise). Data clustering is not the only source of overdispersion for count response. Too often, the distribution of count response arising in biomedical and psychosocial research is dominated by a preponderance of zeros that exceeds the expected frequency of the Poisson or negative binomial model. In this book, we use the term structural zeros to refer to excess zeros that fall either above or below the number of zeros expected by a statistical model such as the Poisson. For example, when modeling the frequency of condom
98
C H A P T E R 2 MODELS FOR CROSS-SECTIONAL DATA
protected vaginal sex over a period of time as in a sexual health study among heterosexual, adolescent females, the presence of structural zeros may reflect a proportion of subjects who either practice abstinence or engage in unprotected vaginal sex, thereby inflating the proportion of sampling zeros under the Poisson law. As in the case of data clustering, the presence of structural zeros also causes overdispersion. However, an even more serious problem with applications of the Poisson model to such data is that structural zeros not only impact the conditional variance, but also the conditional mean, leading to biased estimates of model parameters. Thus, unlike the situation with data clustering, we can no longer fix the problem by only considering the conditional variance. Rather, we must tackle this issue on both fronts. The zero-inflated Poisson (ZIP) model is a popular approach to address the twin effects of structural zeros on both the mean and variance. Since ZIP is developed based on finite mixture, we first illustrate the notion of mixture through a simple example. Example 19 (Mixture of two normals). All models considered so far are unimodal, that is, there is one unique mode in the conditional distribution of the response y given x. When mixing two or more such unimodal distributions, we obtain a mixture of distributions which typically has more than one mode. Let y be sampled from N ( p l ,09) or N ( p 2 ,0 : ) according to some probability p (> 0) and (1 - p ) , respectively. Let zi = 1 if yi is sampled from N ( p l ,a:) and 0 if otherwise. It is readily shown that y is distributed with the following density function (see exercise):
where dlC(y I p k , c r i ) denotes the PDF of N ( p k , o t ) ( k = 1 , 2 ) . Shown in Figure 2.1 is an equal mixture ( p = 0.5) based on two normals: N (3, 0.3) and N (1.0.2). As seen from the plot, there are two modes in this distribution. It follows from (2.64) that the mean of y is E (y) = p p l (1 - p ) p 2 , the weighted average of the means of the two component normals (see exercise). Distributions with multiple modes arise in practice all the time. For example, consider a study to compare efficacy of two treatment conditions. At the end of the study, if treatment difference does exist, then the distribution of the entire study sample (subjects from both treatments) will be bimodal with the models corresponding to the respective treatment conditions. In general, with the codes for treatment assignment, we can identify the two groups of subjects and therefore model each component of the bimodal distribution using, say a normal distribution, N ( p k ,a t ) ( k = 1.2).
+
99
2.1 PARAMETRIC REGRESSION MODELS
0
1
2
3
4
Figure 2.1. Probability density function of an equal mixture of two normals (solid line), along with the probability density function of each normal component, N (1,0.2) (dotted line) and N (3,0.3) (dashed line). T
The parameter vector in this case is 8 = ( p l , p 2 , 0 5 , a i ) . We can then assess treatment effect by comparing the means of the two normals p k using the ANOVA. Now, suppose that the treatment assignment codes are unavailable. In this case, we have no information to identify the subjects assigned to the two treatment conditions. As a result, we cannot tease out the two individual components of the bimodal distribution by modeling the distribution of y for each group of subjects. For example, if y follows N ( p k ,0 ; ) for the kth group, we do not know whether yi is from N ( p l , 05) or N ( p 2 ,0;). Thus, we have to model the combined groups as a whole using a mixture of two normals. The parameter vector for this two-component normal mixture, T 8 = ( p , p l , p2,a?,0 : ) , now also includes the mixing proportion p . Thus, when differences exist among a number of subgroups within a study sample, we can explicitly model such between-group differences using ANOVA or ANOVA like GLMs (for discrete response) if the subgroups are identifiable based on information other than the response such as treatment conditions as in an intervention study. When such subgroups cannot be identified, we can implicitly account for their differences using finite mixtures. By viewing structural zeros as the result of a subgroup of subjects who, unlike the rest of the subjects, are not susceptible t o experiencing the
100
CHAPTER 2 MODELS FOR CROSS-SECTIONAL DATA
event being modeled by the count response, we can employ finite mixtures t o account for their presence. Example 20 ( T h e zero-inflated Poisson model). In Example 17, suppose that there are structural zeros in the distribution of yi when modeled by the Poisson. Consider a two-component mixture model consisting of a Poisson and a degenerate distribution centered at 0 defined by
where fp (y) is the Poisson distribution given in (2.59) and fo (9)= I{y=~} denotes the distribution of the constant 0 By reexpressing this ZIP model, we obtain
+
Thus, the Poisson probability at 0, fp (0), is modified by: f~ (0) = p (1 - p) fp (0):with the degenerate distribution component fo (y) to account for structural zeros. Since 0 5 f~ (0) 5 1, it follows that < p 5 1. Thus, p can be negative. When 0 < p < 1, p represents the amount of positive structural zeros above and beyond the sampling zeros expected by the Poisson distribution fp (y). A negative p implies that the amount of Structural zeros falls below the expected sampling zeros of the Poisson. Although not as popular as the former, the latter zero-inflated or negative structural zero case may also arise in practice. For example, in the sexual health example discussed earlier, the distribution of condom protected vaginal sex may initially exhibit positive structural zeros at the beginning study because of a large number of people who never consider using condoms. If study contains some intervention for promoting safe sex, then behavioral changes may take place and many of these initial non-condom users may start practicing safe sex during or after the intervention. Such changes will reduce or remove the positive structural zeros, and may even result in negative structural zeros in the distribution if a substantial number of these non-condom users change their sexual behaviors at the end of the study. The mean and variance of ZIP defined in (2.65) are given by (see exercise)
If 0 < p < 1, E ( y ) < V a r ( y ) and thus the presence of positive structural zeros leads to overdispersion. By comparing (2.67) to (2.60) and (2.63), it is seen that the positive structural zeros also affect the mean response E (y).
2.1 PARAMETRIC REGRESSION MODELS
101
For regression analysis, we need to model both the Poisson mean p and the amount of structural zeros p. Let ui and vi be two subsets of xi. Note that ui and vi may overlap and thus generally do not form a partition of xi. The ZIP regression model is given by
Thus, the Poisson mean pi and structural zeros pi are linked to the two linear predictors uTPUand vTPv by the log and logit functions, respectively. Note that mixture-based models such as the normal mixture and ZIP do not fall under the rubric of generalized linear models. For example, for the ZIP regression defined in (2.68), the mean response, E (yz 1 xT) = (1 - p i ) pi, is not directly linked to a linear predictor as in the standard GLM such as the NB model in (2.61). This distinction will become clearer when we discuss the distribution-free generalized linear model in Section 2.2.
2.1.5 Inference for Generalized Linear Models Like linear regression, inference for generalized linear models as well as the mixture-based models is carried out by the method of maximum likelihood. Thus, the results in Theorem 1 apply to this general class of models as well. However, unlike linear regression, closed-form solutions to the score equation are usually not available for maximum likelihood estimates and numerical methods must be employed to find the MLE. The Newton-Raphson method is the most popular approach for computing the MLE numerically. For a given GLM, let 0 be the vector of model parameters and 1, (0) the log-likelihood function of the model. As discussed in Section 2.1.3, the maximum likelihood estimate is a solution to the score equation:
When the above is not solvable in closed form, the Newton-Raphson method is used to obtain numerical solutions. This method is based on a Taylor series expansion of the score statistic vector u, (0) around some point do) near the MLE:
102
C H A P T E R 2 MODELS FOR CROSS-SECTIONAL DATA
where 11. 1 1 denotes the Euclidean norm and o ( 6 ) a p x 1 vector with its length approaching 0 faster than 6 L O . By ignoring the higher order term o (110 - do)l1) in (2.69) and solving for 8, we obtain: (2.70)
where I: (0) = -=Tl, 6 2 (0) is the observed information matrix. Substiin (2.70) and solving the resulting equation yields a new tuting dl)for do) d2).By repeating this process, we obtain a sequence d k ) which , converges t o the MLE 0 as n + 03, that is, 8 = limk,, d k ) .In practice, the process ends after a finite number of iterations once some convergence criterion is achieved. The Newton-Raphson algorithm converges very rapidly. For most models, convergence is usually reached after 10-30 iterations. However, starting values are critical for the iterations t o converge. Figure 2.2 depicts what could happen with two different starting values for the case of a scalar parameter 8. In this case, the algorithm would converge, when initiated with some value 8(') as depicted on the left part of the diagram. A different starting value as shown on the right part of the diagram would drive dk)to 00. When convergence is not reached in about 30 iterations, it is recommended that the algorithm be stopped and restarted with a new initial value. General linear hypothesis can be similarly tested using the Wald or likelihood ratio tests. However, unlike linear regression, it is not possible to test nonlinear contrasts by simply reparameterizing the model. For example, consider the general linear hypothesis (2.36) with b # 0. As in the linear model case, we can perform the transformation y = fi - K T ( K T K ) - ' b so that the null hypothesis can be expressed as a linear contrast in terms of y, Ho : K y = 0. For GLMs, the linear predictor has the form: h
h
g (pi)= xiT fi = xTKT ( K T K ) - ' b
+ xTy = ci + x iT y
(2.71)
For mixture-based models such as ZIP, the linear predictors can be expressed in the above form. Unlike linear regression, however, we can no longer transform the response yi to absorb the extra ofSset term ci in (2.71). Most software packages, such as SAS allow for the specification of such ogset terms when fitting GLMs. Note that ci cannot be treated as a covariate since it has a known coefficient 1.
103
2.1 PARAMETRIC REGRESSION MODELS
Figure 2.2. Diagram showing the effect of the starting point on the convergence of Newton-Raphson algorithm. Example 21. Consider Example 17 again. The log-likelihood function
is
Thus, the score function and observed information are given by (2.72)
By applying (2.70) to u, ( p ) and I: ( p ) above, we can numerically obtain the MLE It follows from Theorem 1 that the asymptotic distribution of is A N ,B,E p = [If( p ) ] - ' ) . Since IF ( p )cannot be expressed in closed
P.
(
form, we estimate it by: 11' (p). It follows from Slutsky's theorem that a n. consistent estimate of EBis given by
[
1 1 %=n 4; n
(P)]
-1
=
CI;: (p)]
-l
(2.73)
Example 22. Consider the ZIP regression model in Example 20. The
104
CHAPTER 2 MODELS FOR CROSS-SECTIONAL DATA
conditional PDF of yi given
xi
is
The log-likelihood function is given by:
where I{.} denotes a set indicator. The score un ( 6 ) and observed information matrix I: ( 6 ) are also readily calculated, albeit with more complex expressions than those in Example 16.
2.2
Distribution-Free (Semiparametric) Models
In Section 2.1, we discussed the classic linear regression model for continuous response and its extension, the generalized linear model (GLM), to accommodate other types of outcome variables such as binary and count response. The key step in the generalization of linear regression to the class of G L N is the extension of the link function in the systematic part of the linear model so that the mean response can be meaningfully related to the linear predictor for other types of response. In this section, we consider generalizing the other, random, component of the GLM. More specifically, we will completely remove this component from the specification of GLM so that no parametric distribution is imposed on the response variable. The resulting models are distribution-free and are applicable to a wider class of data distributions. This distributionfree property is especially desirable for power analysis. Without real data, it would be impossible to verify a parametric distribution model, and the
105
2.2 DISTRIBUTION-FREE (SEMIPARAMETRIC) MODELS
robust property of the distribution-free models will ensure reliable power and sample size estimates regardless of the data distribution. The removal of the random component of GLM, however, entails serious ramifications for inference of model parameters: without a distribution model specified in this part of GLM, it is not possible to use the method of maximum likelihood for parameter estimation and inference. Thus, we must introduce and develop a new and alternative inference paradigm.
2.2.1
Distribution-Free Generalized Linear Models
Recall that the parametric GLM has the following components: 1. Random component. This part specifies the conditional distribution of the response y given the independent variables x. 2. Systematic component. This part links the conditional mean of y given x to the linear predictor by a link function: g ( p ) = 7 = Po
+ PlXl + . . . + P,zp
=x
'p
(2.74)
By removing the random component, we obtain the distribution-free GLM with only the systematic part specified in (2.74). Example 1 (Linear regression). Consider a sample of n subjects. Let yi be a continuous response and xi a vector of independent variables of interest. The normal-based linear regression has the following form: yi
1 xi
-
i.d. N (pi, a 2 ),
T
g (pi)= pi = qi = xi p,
-
15 i
5n
(2.75)
N ( p i ,a 2 ) from (2.75), we By excluding the distribution specification yi obtain the distribution-free version of the normal based linear regression for modeling the conditional mean of yi given xi as a linear function of xi. In the special case when yi represent individual responses from a study comparing g treatment conditions and xi are defined by the g binary indicators as in Example 1 of Section 2.1.1, we obtain the distribution-free ANOVA: k-1
k
1=0
1=0
or equivalently,
E
(ykj) = pk,
15 j 5 nk,
15 k
5g
where y k j denotes the response from the j t h subject within the kth treatment group.
106
C H A P T E R 2 MODELS F O R CROSS-SECTIONAL DATA
Example 2 (Linear regression with t distribution). In Example 1, suppose that we want to model yi using a t distribution to accommodate thicker tails in the data distribution under the parametric formulation, that is, yi I xi
N
i.d. t (pi, 02? u) ,
T
g (pi)= pi = q i = xi
0, 1 5 i 5 n
(2.76)
where t ( p , g 2 ,u) denotes a t distribution with mean p , scale parameter g2 and shape parameter u. This alternative model differs from the normal based linear regression in (2.75) only in the random component. Thus: by eliminating this component, we obtain from (2.76) the same distribution-free linear model, p, = xTP, as in Example 1. The normal-based linear model in Example 1 may yield biased inference when the response yi follows a t distribution. Likewise, the t-based model in Example 2 may produce biased estimates of ,8 if the assumption of t distribution is violated (e.g., skewness in the data distribution). Being independent of such parametric assumptions, the distribution-free linear model yields valid inference regardless of the shape or form of the distribution of the response so long as the model in the systematic part is specified correctly. Example 3 (Log-linear model). In Example 1, now suppose that yi is a count response. As discussed in Section 2.1.4, the most popular parametric GLM for such a response variable is the Poisson log-linear model: yi
I xi
N
i.d. Poisson ( p i ) , log (pi)= q i = xiT P , 1 5 i 5 n
(2.77)
By removing the distribution specification yi -Poisson(pi), (2.75) yields the distribution-free log-linear model, log (pi)= q i = xTP. Example 4 (Log-linear model with negative binomial distribut i o n ) . In Example 3, suppose that data are clustered. As discussed in Section 2.1.4, the Poisson-based log-linear model generally yields biased inference when applied to such data. A popular remedial alternative is the following negative binomial model: yi 1 xi
-
i.d. NB ( p i :a ) ,
log ( p i ) = x
n ~ P ,1 5 i I
(2.78)
By comparing the two models in (2.77) and (2.78), we see that the systematic components of Poisson- and NB-based log-linear models are identical. Thus, with the random component excluded, (2.77) and (2.78) define the same distribution-free log-linear model.
107
2.2 DISTRIBUTION-FREE (SEMIPARAMETRIC) MODELS
The distribution-free version of the model in all the examples above can be expressed in a general form:
g ( p i ) = g ( E ( y i I x i ) ) = r7i = xTpI 1 5 i
In
(2.79)
where g (.) is a link function appropriate for the type of response variable in the regression model. The link function is specified in the same way as in the systematic component of parametric GLM. For example, for binary response, only functions that map ( 0 , l ) to (-00,oo) such as logit and probit are appropriate. Although no parametric model is specified for the distribution of the response, the systematic component in (2.79) still involves parametric assumptions for the link function and the linear predictor. For this reason, distribution-free GLMs are often called semi-parametric models.
2.2.2
Inference for Generalized Linear Models
Inference for parametric GLM is carried out by the method of maximum likelihood, which is discussed in Section 2.1.5. As the log-likelihood function is required for deriving model estimates and their inference] the method of maximum likelihood does not apply to the distribution-free GLM. The most popular approach for inference for this new class of models is the method of estimating equation. We first illustrate the ideas behind this approach with examples and then discuss the properties of estimates obtained under this alternative inference paradigm. Example 5 . Consider the normal-based linear model in Example 1 of Section 2.2.1: yi 1 xi
N
i.d. N ( p i ]a')
,
pi = xTP,
15 i 5 n
(2.80)
The log-likelihood function and score statistic vector with respect to the parameters of interest p are given by (2.81)
Let
D. - -
(2.82)
108
CHAPTER 2 MODELS FOR CROSS-SECTIONAL DATA
Then, the score equation defined by u, in (2.81) can be expressed as: n
(2.83) Unlike the score vector in (2.81), the above equation only involves p, which suffices to define the MLE of this parameter vector of interest. Example 6. Consider the Poisson log-linear model in Example 3. The likelihood and score statistic are given by
Let V, = p i , but define Di and Si the same way as in (2.82). Then, the score equation can again be expressed in the form of (2.83). Example 7 (Logistic regression). Consider a sample of n subjects. Let yi be a binary response and xi a vector of independent variables of interest. The parametric GLM for modeling yi as a function of xi is given by yi I xi i.d. BI ( p i , 1) , g ( p i ) = xTP, 1 5 i 5 n where g (pi)is some appropriate link function, such as the logit and probit. The log-likelihood and score vector are given by N
n
1n
(P>
C
1
+ (1 - ~ i log ) (1 - pi11
[ ~ log i (pi)
i=l
By letting V, = pi (1 - p i ) , but leaving Di and Si in (2.82) unchanged, we can again express the score equation in the form of (2.83). In the three examples above, the score equation follow a common, general form given in (2.83). This is not a coincidence and in fact this expression holds true more generally if the conditional distribution of yi given xi in the random component of parametric GLM is from the exponential family of distributions defined in (2.54). For such a distribution, the log-likelihood function and score statistic vector with respect to p are given by
(2.84)
2.2 DISTRIBUTION-FREE (SEMIPARAMETRIC) MODELS
109
Further, it is readily checked that (see exercise),
It follows from (2.84) and (2.85) that
Thus, by setting u, to 0 and eliminating a ( 4 ) ,we obtain a score equation with respect to p that has the same form as the estimating equation in (2.83). We know from the discussion in Section 2.1.4 that inference for parametric GLM by the method of maximum likelihood is really driven by the score equation. The likelihood function provides the basis for constructing the score statistic vector, but computations as well as asymptotic distributions of maximum likelihood estimates can all be derived from the score statistic vector. The method of estimating equation is premised on a score-like statistic vector. Consider the class of distribution-free GLM defined by (2.79). Although no parametric distribution is assumed for the response yi, Di = $$ and Si = yi - p i are still well defined given the model specification in (2.79). Thus, given some choice of V , = u (pi),we can still define a score-like vector w, ( p ) and associated estimating equation: n
w,
d ( p ) = CDiy-92 = 0 , D . - - p . i=l
“-ap
V, = u ( p i )
(2.86)
In the above, Si is called the theoretical residual (to differentiate it from the observed residual with estimated p i ) and is the only quantity that contains the response yi. The quantity V, is assumed to a function of p i . As discussed earlier, with the right selection of V,, the estimating equation in (2.86) yields the MLE of p for the parametric GLM when yi is modeled by the exponential family of distributions. Estimating equation estimates are defined in the same way as the MLE - by solving the equation in (2.86). As in the case of MLE, a closed-form
110
CHAPTER 2 MODELS FOR CROSS-SECTIONAL DATA
solution typically does not exist and numerical estimates can be obtained by applying the Newton-Raphson algorithm. For example, by expanding w, (p) around the estimating equation estimate using a stochastic Taylor series expansion and ignoring the higher order terms, we have
By setting w,
p('"+') 0
= 0 and solving for
dk+'), we obtain
where A-T = (A-l)T for a matrix A. Starting with some initial ,do), we use (2.87) to iterate until some convergence criterion is reached and the limit is the estimating equation estimate of p. The calculation of $w, ( p )may be complicated in some cases. using the following algorithm:
p An alternative is to compute the estimate p
While this algorithm may converge at a slower rate, it avoids the calculation a of -w,. ap Example 8 . Consider the distribution-free log-linear model specified by only the systematic component in Example 3. Let V , = pi. The estimating equation is given by
It is readily checked (see exercise) that (2.89)
Thus, for this particular example, the two updating schemes (2.87) and (2.88) are identical.
111
2.2 DISTRIBUTION-FREE (SEMIPARAMETRIC) MODELS
When using the estimating equation in (2.86) to find estimates of P for the distribution-free GLM, we must select some function for V,. We know from the above discussion that if the conditional distribution of y, g'iven x, is a member of the exponential family, V, can be determined to enable the estimating equation to yield the MLE of 0. In most applications, however, this distribution is unknown and the selection of V, becomes arbitrary. Fortunately, the estimating equation still yields consistent estimates of P regardless of the choice of V,. Theorem 1 below summarizes the asymptotic properties of the estimating equation estimates. Theorem 1 (Properties of estimating equation estimates). Let denote the estimate obtained by the method of estimating equation. Let B = E (D,V,- 1D,T) and V , = v (p,), a known function of p,. Then,
a
1. 2.
B-,P
6(3 P ) -p -
N ( 0 ,Xp), where E p is given by
Proof. The proof follows from an argument similar to the one used to prove Theorem 1 of Section 2.1.2 for the asymptotic properties of MLE. DiV,-lSi, is asFirst, note that the score-like vector, w, ( p ) = $ ymptotically normal,
cy=l
This follows immediately from an application of CLT to wn (P) and the following identities:
I Xi)] = 0 (s;I Xi) DT] = E
(2.92)
E (DZV,-lSz) = E [D&-1E (SZ
V a r (D$-1sz)
=E
[DzV,-%
[Dili;-2Vur (yi
I X i ) D?]
h
Now, consider the estimating equation estimate P. By an argument similar to the proof of consistency for MLE, we can show that P is consistent regardless of the choice of V , (see exercise). To prove asymptotic normality, consider a Taylor series expansion of w, around w, (P): h
(a)
w,
(a)
(6%(a)) (P T
= w,
(PI +
-
P)
+ op (I) (p - p)
112
C H A P T E R 2 MODELS FOR CROSS-SECTIONAL DATA
from which it follows that
By LLN, &wn ( p ) that (see exercise)
- f P
E
[& ( D i K - ' S i ) ] .
Further, it is readily checked
(2.94)
=
-E
(DiVi-
Di
=
-B
T>
Thus, by applying the Delta method and Slutsky's theorem, it follows from (2.91) and (2.93) and (2.94) that f i +d N ( 0 , Xp) with the asymptotic variance Xp given in (2.90). There are two ways to estimate the asymptotic variance Cp in (2.90) depending on the choice of V,. If Var ( y i I xi) = cr2V,, (2.90) simplifies to
(p p>
C f B = B-lE [Dil/;-2Var( y i 1 x i ) D T ] B-' = cr2B-l
(2.95)
If cr2 is known, it follows from an application of LLN that a consistent estimate of Cp is given by: (2.96)
&,
denote the estimates of Di, V , and B obtained by where and substituting in place of p. If yi is modeled by the exponential family of distributions, cr2 = q5 and C r B in (2.95) is the asymptotic variance of the MLE of p and above is a consistent estimate of C r B (see exercise). For example, for the Poisson log-linear model in Example 6, o2 = 1 and 5 f Bis a consistent estimate of the asymptotic variance of the MLE of P. If cr2 is unknown, we can substitute the following estimate based on the Pearson residuals in place of cr2 in (2.96):
3
srB
cr = -2
- Q p2 ; n
n Q2
-
i=l
w
-'( p i )(yi 2
-
= g-'
(xTp)
(2.97)
2 . 2 DISTRIBUTION-FREE (SEMIPARAMETRIC) MODELS
113
Note that when u2 is unknown, Cp in (2.90) is generally different from the asymptotic variance of the MLE of ,B under the parametric GLM even when y, is modeled by the exponential family of distributions. This is because the MLEs of p and other (nuisance) parameters of the distribution may be asymptotically correlated and as such the off-diagonal block of the Fisher information matrix that represents such asymptotic correlations may not be zero. However, if y, given x, follows a normal distribution, Cp in (2.90) yields the asymptotic variance of the MLE of p (see Example 9 below). If V a r (y, 1 x,) # a2V,or if y, conditional on x, does not follow the exno longer estimate ponential family of distributions, the model-based Cp in (2.90). Since
%FB
Cp = B-lE [D,K-2Var (y,
I x,) D,
B-l = B-'E (D,K-2S:DT)
B-l
it follows from Slutsky's theorem that we can estimate E p using the following sandwach estamate: (2.98)
gi and 6 denote the estimates of Di, V,, Si and B obtained where Ei, by substituting in place of ,L?. Example 9. Consider the distribution-free linear regression model in Example 1 without the random component. Let V, = 1. Then, we have
E,
(2.99)
Thus, a consistent estimate of the asymptotic variance of the estimating equation estimate is given by
If V a r ( y , 1 xi) = u2, we can use the model-based asymptotic variance, C f B = u 2 E 1 (xix:), for more efficient inference about p. Further, if yz given x, follows a normal distribution, C F B actually becomes the asymptotic variance of the MLE of 0. This follows from comparing C f B with
114
C H A P T E R 2 MODELS FOR C R O S S - S E C T I O N A L DATA
the asymptotic variance of the MLE of p derived in Example 5 of Section 2.1.2. Example 10. Consider the distribution-free log-linear model specified by the systematic component in Example 3. Let V, = pi. Then,
B = E (pixipix:)
(2.100)
= E (pixixT)
The estimating equation in (2.86) yields a consistent estimate of p with the asymptotic variance Cp regardless of the distribution of y,. As in Example 9, if Var (y, 1 x,) = p, or if y, given x, follows a Poisson, the model-based cM P B = E-1 ( p , x , x ~ )should be used to provide more efficient inference about p. As discussed in Section 2.1.4, overdispersion may occur when there is data clustering. The MLE of p from the Poisson log-linear model is still consistent. However, its asymptotic variance E r B = E-l ( p 2 x z x ~un) derestimates the sampling variability. We can detect overdispersion by examining some goodness of fit statistics. For example, under the Poisson model,
1
u-2
(g,)(y, - G,)
1
= j2T5 (y, -
g,) = 1 so that the Pearson goodness
%
of fit statistic Q$ in (2.97) is close to n for large n. Thus, if is significantly larger than 1, overdispersion is indicated and 5gw should be used for
(h
inference. Alternatively. we may simply use z l p = 82 C:=l p,x,xT)-' to correct for overdispersion. This simpler estimate is based on the particular form of V, and may be biased if V a r (y, I x,) # 02p,. For example, if y, given x, follows a negative binomial, then V a r (y, 1 x,) = pz(l a p Z ) for some known constant a. In this case, only the sandwich estimate is consistent and valid for inference. When y, is modeled by a distribution from the exponential family, the estimating equation can yield the MLE with the right choice of V,. As shown by the next example, this is generally not true when the distribution does not belong to the exponential family. Example 11. Consider the linear regression model with y, conditional on x, given following the t distribution in Example 2. The log-likelihood function is given by (see exercise): h
+
1, = nlog
21
r (;)
(0%)
5
i=l
V
(2.101)
115
2.3 EXERCISES
Suppose that w and u2 are both known. Then, the MLE of the score equation (see exercise):
P is defined by
+
Although similar, the above is not an estimating equation since V, = 1 U2 2 (yi - p i ) depends on the response yi. Inference for general linear hypotheses proceeds in the same fashion as for parametric models except that the likelihood ratio test no longer applies as it relies on parametric distribution assumptions. For example, for testing the null of the form, Ho : KP = b, we can use the Wald statistic,
B
where is the estimating equation estimate of p and %p some consistent estimate of the asymptotic variance EDof P. For parametric models, the MLE is optimal in that it is asymptotically most efficient. Similar optimal properties exist for estimating equation estimates from distribution-free GLMs. However, as the distribution of response is totally unspecified, that is, no analytic form is posited as in the case of parametric models, the vector of parameters for distributionfree models is of infinite dimension. As a result, the Fisher information cannot be computed directly to provide the optimal lower bound for the variance of such parameter estimates. Asymptotic efficiency of estimates for a distribution-free model is defined as the infimum of the information over all parametric submodels embedded in such a model ( e g . , Bickel et a1.1993; Tsiatis,2006). For estimates defined by the estimating equation, their efficiency depends on the choice of V. If specified correctly, that is if V = V a r (y 1 x) , the estimating equation estimate is asymptotically efficient. Thereby, in theory, there exists an asymptotically efficient estimate in the classes of estimating equation estimates. In most applications, however, this correct V is unknown. Various approaches have been considered for the selection of V which we will not delve into in this book. We emphasis that regardless of the choice of V, the estimating equation estimates are always consistent. This robustness property is what we value most.
2.3 Exercises Section 2.1
h
116
C H A P T E R 2 MODELS FOR CROSS-SECTIONAL DATA
1. Verify the two basic properties regarding differentiation of vectorvalued function. 2. Show that and a2defined in (2.15) are stochastically independent. As a special case, it follows that and Z 2 defined in (2.13) are independent. 3. Let s? be defined in (2.13). Show that x?-~. 4. Let y be a p x 1 random vector and X a random variable. Suppose that y and X are independent and satisfy the following:
p1
p1
N
yIX--N(p,XC),
V p :, v>o,
C>O
where C > 0 means that the symmetric p x p matrix C is positive definite. (a) Show that the unconditional distribution of y has a multivariate t distribution, t ( p ,C, w), with the density function given by (2.17). (b) Show that any subvector of y also has a multivariate t distribution. (c) Show that for any vector a of known elements any linear combination aTy follows a univariate t distribution, t (aTp,aTCa, v). 5. Consider the MLE and a2 in the linear regression model given in (2.15). (a) Show that and are independent. with MSE defined in (2.16). (b) Show that (c) Show that follows a multivariate t distribution given in (2.17). 6. For fixed xi, a2 is a x i p p variable as shown in (2.16), that is,
p
a2
& l (a2 -
- xi-p
2 )= &iu2
(+
-
I)
Show that ,,hi (a2- 0 2 )" d N (0,2 0 4 ) . 7. Show that ( k 2 1) defined in (2.22) in the proof of Theorem 1 is a sequence of random variables. 8. Use (2.20) and (2.21) to show that 1; (8) = V a r (u, ( O ) ) , where u, (8) = &l, ( 8 ) is the score statistic vector. 9. Consider a parametric model f ( 9 ;8). Let (8) be some smooth function of 8 and T, an unbiased estimate of ( 6 ) . Let 6 be the MLE of 8, CQ the asymptotic variance of 6 and Eg the asymptotic variance of (6) given in (2.33). Show V a r ( T , ) 2 ~ C Q . 10. Verify (2.46) in the proof of Theorem 2. 11. For the exponential family of distributions defined in (2.54), show (4 (Y) = $ (6). (b) V a r (Y) = a ( 4 ) (6).
2,
+
+
$&
+
117
2.3 EXERCISES
12. Show that for the Poisson log-linear model defined in (2.58), the conditional mean E (yi I xi) and variance V a r (yi I xi) are identical and given by (2.60). 13. Let y be a continuous response following the negative binomial distribution defined in (2.62). Verify that the mean and variance of y are given by (2.63). 14. Show that the probability distribution function f N B for the negative binomial model defined in (2.62) converges to the Poisson distribution function fp in (2.59) when a decreases to 0. 15. Let y be sampled from a mixture of fk (y I O h ) , where fk (y I 0,) denotes two PDFs ( k = 1 , 2 ) . Let z = 1 if y is sampled from f1 and z = 0 if otherwise. Let p = E ( z ) > 0. (a) Show that the PDF of y is given by
f (Y I P ?01,02) = Pfl (Y I 01) + (1 - P)f 2 (Y I 02) (b) If f k (y I 0,) = N (&? a;), find the mean and variance of y. (c) In (b), find the maximum likelihood estimates of p , pk and ug ( k = 1,2). 16. Verify (2.67). Section 2.2 1. Let yi and xi be some continuous response and vector of independent variables of interest (1 5 i 5 n ) . Consider a GLM for relating yi to xi with the distribution of yi from the exponential family defined in (2.54). Verify (2.84) and 2.85). 2. Show that at convergence the estimates of p obtained from (2.87) and (2.88) are identical. 3. Verify (2.89) and (2.92). 4. Show that the estimating equation estimate is consistent regardless of the choice of V,. 5. Verify (2.94) by applying the properties regarding differentiation of vector-valued function in Section 2.1.2. 6. Within the context of Example 1, suppose that q5 is known. (a) Find the Fisher information matrix 1; = E --;;a 1.) , where 1, is the log-likelihood function in (2.84). of the MLE of p is the (b) Show that the asymptotic variance same as the model-based asymptotic variance of the estimating equation estimate given in (2.95). 7. Show that if a is known, the negative binomial distribution of yi given xi in Example 2 is a member of the exponential family of distributions.
(
118
C H A P T E R 2 MODELS FOR CROSS-SECTIONAL DATA
8. Use the density function for the t distribution in (2.18) to verify (2.101) and (2.102).
Chapter 3
Univariate U-St at ist ics In Chapter 2, we discussed various regression models for modeling the relationship between some response of interest and a function of other independent variables. We started with the normal based linear regression and then generalized this class of classic models to address different response types and distribution assumptions, culminating in the development of the distribution-free, or semiparametric, generalized linear models (GLM). This general class of regression models not only applies to all popular types of response but also imposes no parametric distribution assumption on the response, yielding a powerful and flexible platform for regression analyses of cross-sectional data arising in practical studies. In the distribution-free GLM, a parametric structure is assumed t o link the mean of a designated dependent or response variable t o the linear predictor (a combination of other independent variables) in order t o characterize the change in the dependent variable in response to a change in the linear predictor. This systematic component is at the core of GLM as it provides the essential bridge to connect the dependent and the set of independent variables. Thus, the distribution-free GLM places minimal assumptions on modeling the relationship between the mean of the response and the independent variables. Although such minimal specifications yield more robust inference when applied t o the wide array of data distributions arising in real study data, important features of the data distribution of the response may get lost. For example, as discussed in Chapter 2, the Poisson and negative binomial become indistinguishable when implemented under distributionfree GLM, resulting in an inability t o detect data clustering and a lack of efficiency. Since the difference between the Poisson and negative binomial is reflected in the variance, it is necessary to include models the variance to 119
120
CHAPTER 3 UNIVARIATE U-STATISTICS
obtain the required information in order to distinguish the two models to address data clustering and efficiency. In addition to the negative binomial model, many other statistical models are not amenable to treatment by GLM. Two such examples, the Pearson correlation and the hfann-Whitney-Wilcoxon test, are discussed in Chapter 1. Because the Pearson correlation models the concurrent changes between two variables rather than the mean of one variable as a function of the other, GLM is not applicable. The difference between these two modeling approaches is also amply reflected in their analytic forms; the Pearson correlation involves second order moments, while the distribution-free GLM models only the mean response, or first order moment. The form of the rank-based Mann-Whitney-Wilcoxon test deviates even more from that of GLM as it models the difference between two data distributions without using any of the moments of the outcome variables. In this chapter, we introduce univariate U-statistics and discuss their applications to a wide array of statistical problems some of which address the limitations of the distribution-free GLhf including modeling higher order moments, while others require models that cannot be subsumed under the rubric of GLM or treated by the conventional asymptotic methods discussed in Chapter 1. We limit our attention t o cross-sectional data without data clustering. In Chapters 5 and 6, we take up multivariate U-statistics and discuss their applications to repeated, correlated responses within a longitudinal or clustered data setting along with addressing missing data within such a setting.
3.1
U-Statistics and Associated Models
Since Hoeffding’s (1948) foundational work, U-statistics have been widely used in both theoretical and applied statistical research. What is a UStatistic and how is it different from the statistics that we have seen in Chapters 1 and 2? From a technical point of view, U-statistics are different both in their appearance and in the methods used to study their asymptotic behaviors. U-statistics are not in the usual form of a sum of i.i.d. random variables either expressed exactly or asymptotically through a stochastic Taylor series expansion as are all the statistics and model estimates discussed in Chapters 1 and 2. Take for example an i.i.d. sample yz with mean p and variance 0 2 . The sample mean Tj, and variance s i given below are unbiased and consistent
121
3.1 U-STATISTICS A N D ASSOCIATED MODELS
estimates of p and variance
02:
The sample mean is a sum of i.i.d. random variables yi. Although the sample variance is not in such a form, it can be expressed as an i.i.d. sum by applying a Taylor series expansion around p (see Chapter 1):
i=l
For statistics and/or estimates that are in the form of an i.i.d. sum such as yn and s:, we can find their asymptotic distributions by applying the various methods and techniques discussed in Chapter 1. As will be seen shortly, the distinctive appearance of U-statistics makes it impossible to express them as an i.i.d. sum in either closed form or asymptotic approximation using a Taylor series expansion. From an applications standpoint, statistics and estimates that can be expressed as an i.i.d. sum either exactly or through a Taylor series expansion typically arise from parameters of interest defined by a single-subject response. For example, the mean p = E (yi) and variance o2 = E (9') E 2 (yi), all defined by a single-subject yi. More generally, the class of distribution-free GLM is also defined in such a fashion since the random component in the defining model, g ( E ( y i 1 xi)) = involves only a single-subject response yi as discussed in Chapter 2. Many parameters and statistics of interest, such as the Mann-Whitney-Wilcoxon test, cannot be defined by such single-response based statistical models. Further, even for those that can be modeled by a single-subject response, the study of the asymptotic behavior of their associated statistics and estimates can be greatly facilitated when formulated within the U-statistics setting. In this Section, we first introduce the one-sample U-statistic and then generalize it to two and more samples. We then discuss some classic properties of U-statistics. We develop inference theories for univariate U-statistics in Section 3.2. -
xT~,
3.1.1
One Sample U-Statistics
Consider an i.i.d. sample of n subjects. Let yi be some response of interest with mean p and variance o2 (1 5 i 5 n ) . In most applications, we are
122
CHAPTER 3 UNIVARIATE U-STATISTICS
interested in estimating the population mean p . As discussed in Chapter 2, we can model this parameter of interest using either a parametric or non-parametric GLNI. For example, we may assume that yi follows a normal distribution, in which case we obtain the maximum likelihood estimate (MILE) of p and can use this estimate along with its sampling distribution for inference about p . If we are unwilling to posit any parametric model for the distribution of yi, we can still make inference about p by modeling yi using the distribution-free GLM. In many other applications, we may be interested in inference about the variance 02. For example, when comparing two effective interventions for treating some medical and/or psychological condition of interest, we may be interested in examining difference in variances if both treatments yield similar mean responses. If one treatment has significantly smaller variability in subjects’ responses, it is likely to be the preferred treatment choice for the disease, as it is more likely to give patients the expected treatment effect. Such considerations may be especially important in effectiveness research, where the potency of an effective treatment is often diluted when given to patients with comorbid health conditions. A treatment with smaller variability across patients with different comorbid conditions compromises less than its competing alternative in terms of treatment efficacy when prescribed to patients with diverse comorbid conditions.
As in the case of inference for p , we can use the MLE of o2 and its sampling distribution to make inference about o2 under a parametric GLM. However, if we are unwilling to use a parametric model for the data distribution of yi, we will not be able to model this variance parameter using a distribution-free GLM. Fortunately, in this simple problem, we know that we can estimate o2 using the sample variance s i . Further, by using the asymptotic distribution of this estimate that we developed in Chapter 1, we can also make inference about u2. This example shows the limitation of GLM and points to the need to develop a new approach for systematically modeling higher order moments such as the variance of ys in this example, as it may be difficult to find estimates and their asymptotic distributions when dealing with other more complex problems. The power of U-statistics lies exactly in addressing this type of problem. Let us consider again the estimate s i . As shown below, we can rewrite
3.1 U-STATISTICS AND ASSOCIATED MODELS
123
it in a U-statistic, or symmetric, form involving pairs of responses:
(3.3)
where C; = { ( i , j ); 1 5 i < j 5 n} denotes the set of all distinct combinations of pairs (i,j ) from the integer set { 1 , 2 , . . . , n}. Note that two pairs (i,j ) and ( k , 1 ) are not distinct if one can be obtained from the other through a permutation. For example, ( l , 2 ) and ( 2 , l ) are not distinct, but ( 1 , 2 ) and (1,3) are two distinct pairs. By comparing the above expression to the one in (3.1), it is seen that the U-statistic expression employs pairs of responses to express s i as a function of between-subject variability rather than deviations of individual response from the mean 5, of the distribution yi. As will be seen shortly, this shift of modelling paradigm is especially critical for studying statistics defined by responses from multiple subjects. Let h (pi, yj) = (yi - yj)2. This kernel function is symmetric with respect to the pair of responses yi and yj, that is, h (yi, yj) = h (yj, yi). The alternative U-statistic expression in (3.3) shows that s i is the sample mean of this kernel function over all pairs of subjects in the sample:
(3.4) Further,
Thus, it follows immediately from (3.4) and (3.5) that the sample variance s i is an unbiased estimate of 02. In comparison, it is more involved to show unbiasedness of s i using the traditional expression in (3.1). Not only can we express the variance, or centered second order moment, estimate s:, but also other order moments, such as the sample mean Z, in a U-statistic form similar to (3.4) involving a symmetric kernel h. Example 1 (U-statistic for sample m e a n ) . Let yi be an i.i.d. sample with mean 8 and variance o2 (1 5 i 5 n). Now consider estimating the kth order moment 8 = E (y') (assuming it exists). Let h (y) = y'. In
124
C H A P T E R 3 UNIVARIATE U-STATISTICS
this case, the one-argument kernel function h ( y ) is, of course, symmetric. Further, -1
n
n i=l
5
As E ( h (y)) = 8, it immediately follows that is an unbiased estimate of 8. The above is the usual estimate for the kth-order moment. Example 2 (U-statistic for proportion). In Example 1, let zi = I { y L > ~(1} 5 i 5 n). Let h ( z ) = zi. Then, 6' = E ( h ( z ) ) = P r (yi > 0) and
5
where I{.} denotes a set indicator. Thus, the U-statistic is the usual proportion of yi > 0. The next example shows that it is also easy to construct U-statistics to estimate powers of the mean p k ( k = 1 , 2 , . . .). Example 3 (U-statistic for powers of mean). In Example 1, con) y1y2. Then, sider estimating the square of the mean p2. Let h ( y l , y ~ =
E [h( Y l , Y2)l = E ( w 2 ) = E ( Y l ) E (Y2) = P2 the parameter of interest. The U-statistic based on this kernel is:
Again, Un is an unbiased estimate of p 2 . In Chapter 1, we indicated that we could use the theory of U-statistics to facilitate the study of the asymptotic behavior of the Pearson correlation estimate. In the example below, let us first express a related quantity in the form of a U-statistic. Example 4 ( U-statistic for covariance). Let zt = (xi,yi)T be i.i.d. pairs of continuous random variables (1 5 i 5 n). Consider estimating the covariance between xi and yi, 8 = Cov (xi,yi). Let
1
h (Zi,Z j ) = - (Xi - Z j ) (Yi 2 Then,
- Yj)
3.1 U-STATISTICS AND ASSOCIATED MODELS
125
Thus, the U-statistic below:
is an unbiased estimate of 0 = Cow ( 2 , ~ ) . Note that as the study of the Pearson correlation estimate requires multivariate U-statistics, we will defer the discussion of this important statistic to Chapter 5 after we introduce multivariate U-statistics. In all the examples above, the U-statistics based estimates for these popular statistical parameters of interest can also be expressed as traditional sample averages over single subjects or functions of such sample averages. For example, U, in (3.6) call be reexpressed in the familiar form as (see exercise): (3.7)
c
1 =-
n-1
( 2 2 - :n)
(92 - Y,)
2=1
However, as will be seen shortly, many statistics and estimates of interest can only be expressed in the U-statistic form by using multiple subjects' responses. However, the power of U-statistics not only lies in studying such intrinsic or endogenous multisubject based statistics and estimates, but also in facilitating asymptotic inference for parameters amenable to the traditional treatment discussed in Chapters 1 and 2. Before proceeding with more examples, let us give a formal definition of U-statistics. Consider an i.i.d. sample of p x 1 random vectors yi (1 5 i 5 n). Let h (y1,.. . , y m ) be a symmetric function with m arguments or input vectors, that is, h (yl,. . . , Y m ) = h (yil,. . . , yi,) for any permutation ( i l , . . . , im) of (1,2, . . . , m ) . We define a univariate one-sample, m-argument U-statistic as follows:
where (7; = { ( i l , . . . , im) ; 1 5 i l < . . . < im 5 n} denotes the set of all distinct combinations of rn indices (21, . . . , im) from the integer set { 1 , 2 , . . . , n } . Under this formal definition, all U-statistic based estimates in the above examples have two arguments, except for the ones in Example 1 and 2
126
CHAPTER 3 UNIVARIATE U-STATISTICS
where the estimates are expressed as one-argument U-statistics. [h( Y l , . . . , Y m ) l , then,
Let Q =
=Q Thus, U, is an unbiased estimate of 8. Thus, the i.i.d. Note that unzwarzate refers to the dimension of U,. sample y z in the definition of a univariate U-statistic can be multivariate such as in Example 4. In each of the examples above. the kernel function h is symmetric. For cases in which a kernel is not symmetric, it is easy t o construct a symmetric version. Example 5 . Consider again the problem of estimating u2 in Example 1. Let 2 h ( Y z , Y , ) = Y , -YzY,, 1 5 % J < n ,2 f . i Then,
[h(Yz,YJl
=E
(2)E (Yz) E (Y,) = u2 -
However, h (yz,3,) is not symmetric, that is, h (yz,y,) readily construct a symmetric version of h (zz,2 , ) by
1
= - (Xi - Z j )
# h (y,,
yz). We can
2
2
which gives rise to the same kernel as in (3.5). This example shows that kernels for a U-statistic are not unique. More generally, if h (yl,. . . , y m ) is not symmetric, we can readily construct a symmetric version by
where Pm denotes the set of all permutations of the integer set { 1,, . . . , m}. One major limitation of the product-moment correlation for two continuous outcomes is its dependence on a linear relationship to serve as a
127
3.1 U-STATISTICS AND ASSOCIATED MODELS
valid measure of association between the two outcomes. For example, if two outcomes follow a curved relationship, the product-moment correlation may be low even though the two outcomes are closely related, giving rise to incorrect indication of association between the two variables. One popular alternative to address this limitation is Kendall’s 7 . Example 6 (Kendall’s 7 ) . Let z, = ( ~ % , ybe~ i.i.d. ) ~ continuous bivariate random variables (1 5 i 5 n ) as defined in Example 4. Unlike the Pearson estimate of the product-moment correlation, which relies on moments to capture the relationship between x and y, Kendall’s 7 defines association between x, and y, by considering pairs of zz based on the notion of concordance and discordance. More specifically, two pairs z, = (xz, Y , ) ~and z3 = (x3.y3)T are concordant if xt < x3 and y, < y3, or x, > x3 and y, > y3 and discordant if xi
< xj and
yi
> yj,
or
xi > xj and yi < y j
If a pair of subjects share the same value in either x or y or both, that is, the two subjects have a tie in either x (xi = xj) or y (yi = yj) or both, then it is neither a concordant nor a discordant pair. For continuous outcomes, the probability of having such tied observations is 0, though in practice it is likely to have a few tied observations because of limited precision in measurement. If there are more concordant pairs in the sample, x and y are positively associated, that is, larger x leads to larger y and vice versa. Likewise, more discordant pairs imply negative association. A similar number of concordant and discordant pairs in the sample indicates weak association between the two variables. We can express concordant and discordant pairs in a compact form as: Concordant pairs: (xi- xj) (yi - yj) > 0 Discordant pairs: (xi- xj) (yi - yj) < 0 Kendall’s 7 is defined based on the probability of concordant pairs: Kendall’s 7 : p, = Pr [(xi- xj)(yi
- yj)
> 01
(3.11)
A value of p , closer to 0 (1) indicates a higher degree of negative (positive) association. In practice, we often normalize p, so that it ranges from -1 to 1 to have a similar interpretation as the product-moment correlation: Normalized Kendall’s
IT :
T
= 2p, -
1
(3.12)
128
CHAPTER 3 UNIVARIATE U-STATISTICS
Now, define a kernel for the normalized Kendall’s 7 in (3.12) as follows:
h (zz, 2 3 )
= 2 ~ { ( ~ ~ - 5 3 ) ( y z - y 3 ) > o}
1
Clearly, h ( z z , z I Iis) symmetric and E [ h ( z z , z J )=] 7 . Thus, the following U-statistic based on this kernel is an unbiased estimate of 7 :
Unlike all the previous examples, the U-statistic in (3.13) is intrinsically or endogenously multisubject based, as it is impossible to express it as a sum of i.i.d. random variables. Another classic statistic that is not subjected to the treatment of traditional asymptotic theory is the one-sample Wilcoxon signed rank statistic. This statistic also employs ranks, but unlike the twosample Mann-Whitney-Wilcoxon rank sum test discussed in Section 1.1 of Chapter 1, it tests whether an i.i.d. sample from a population of interest is distributed symmetrically about some point 8. More specifically, let yz be an i.i.d. sample from a continuous distribution (1 5 i 5 n). Let Ri be the rank score of yi based on its absolute value (yil (see Section 1.1 in Chapter 1 for details on creation of rank scores). We are interested in testing whether the point of symmetry p is equal to some hypothesized known value po, that is, HO : p = po. Since in most applications, po = 0 and the consideration for the general situation with po # 0 is similar (by a transformation yz - po), we focus on the special case po = 0 without the loss of generality. Example 7 (One-sample Wilcoxon signed rank t e s t ) . The Wilcoxon signed rank statistic, W:, for testing HO is defined by n
Wilcoxon signed rank test : W$ = x I { y t > o } R i
(3.14)
i=l
This statistic ranges from 0 to n(n+1) (see exercise). Under Ho, about onehalf of the yi’s are negative and their rankings are comparable to those of positive yi’s because of the symmetry. It follows that W$ has mean ,4n ( n + l ) half of the range n ( n + l ) (see exercise). Further, we show in Section 3.1.3 that W$ can be expressed as: n
(3.15) i=l
l
129
3.1 U-STATISTICS A N D ASSOCIATED MODELS
It then follows from (3.15) that (see exercise)
-
2 n-1
un~+ un2
In the above, Unl is a U-statistic in Example 1 and Un2 is also a U-statistic defined by the symmetric kernel h2 (yl, y2) = I{y,+y,>o). Later in Section 3.2, we will show that G)-'W$ and Un2 have the same asymptotic distribution and thus inference for Ho can be carried out equivalently by using only the asymptotic distribution of Un2 for large samples. The U-statistic Un2 is an unbiased estimate of 6 = E (h2 (yi, yj)) = Pr (yi y j > 0). By taking expectation on both sides of the equality in (3.16) under Ho : po = 0 we have (see exercise)
+
(3.17) n
It follows that
r)
1 e=-+--
2 Thus, we can express the null
1 n-1
1
'2,
asn-+oo HO: po = 0 in terms of 6 as:
1 2
Ho : 6 = Pr(yi + y j 5 0) = -
All the examples above involve continuous outcomes. Many important statistics for discrete or categorical data can also be expressed in U-statistics form. Consider two categorical ordinal outcomes u and 'u. Suppose u has K and u has M levels indexed by k and m, respectively. The notion of concordant and discordant pairs has also been applied to categorical outcomes to create indices of association between two categorical ordinal outcomes such
130
CHAPTER 3 UNIVARIATE U-STATISTICS
as Goodman and Kruskal’s y and Kendall’s T b , as we discuss in the next example. Example 8 (Kendall’s T b and Goodman and Kruskal’s 2 ) . Consider an i.i.d. sample of bivariate ordinal outcomes z , = ( u z ,v,)T (1 5 i 5 n ) defined above. For a pair of subjects z , and zJ, concordance and discordance are defined similarly as for continuous outcomes:
(z2,z3E )
I
concordant if u,< (>)u3,vz < (>)v3 discordant if u,> ( < ) u 3 ,w2 neither
< (>)w3
if otherwise
Note that unlike continuous response, there is usually a sizable number of tied observations. Let p , and pdp denote the probability of concordant and discordant pairs. Also, let p , and p , be the probabilities of having ties in u and v , respectively. Kendall’s T b and Goodman and Kruskal’s y are defined as: Tb =
Pep
-
Pdp
d(1- P u ) ( l
-
Pu)’
Y=
Pep Pcp
- Pdp
+ Pdp
(3.18)
where p , = Pr (uz= u3)and p , = Pr (w, = v3). Both coefficients range from -1 to 1 (see exercise). Both indices are widely used to measure association between two ordinal outcomes. If u and w are independent, pep - pdp = 0 and thus T b = y = 0. Then, the mean Let h (zz,2 3 ) = ~ { ( u * - u J ) ( v z - - v ~ ) >-o~}{ ( , z - U 3 ) ( V Z - t J ) < 0 } . of h ( z , , z 3 )is
- Pep - Pdp
In addition, since
h ( z i ,z j ) =
I
1
if yi and y j are concordant
-1 if
0
yi
and yj are discordant
if otherwise
it follows that the U-statistic based on h ( z i ,z j ) has the following form:
3.1 U-STATISTICS A N D ASSOCIATED MODELS
131
where C and D denote the number of concordant and discordant pairs in the sample, respectively. Under the null of no association between u and u, S = p , - pdp = 0 and the U-statistic can be used to test this null. For nominal outcomes, Chi-squares and Fisher's exact tests are widely used to test independence between two such outcomes. We can also develop U-statistics to test independence between two nominal outcomes. In the next example, we consider a relatively simple case with two binary outcomes, as the general setting involving two multilevel nominal outcomes requires multivariate U-statistics and thus will be deferred to Chapter 5. Example 9 ( T e s t for independence between two binary outc o m e s ) . Within the context of Example 8, assume that u and u are both binary outcomes. For convenience, we recode u and u so that they all have values 0 and 1. Independence between u and 'u is equivalent to the following condition p,, = Pr(uv = 1) = Pr(u = 1)Pr(v = 1) = pup,
(3.19)
Let h zi,z j ) = 31 (ui- uj)(ui- v j ) . Define the following U-statistic:
This U-statistic is an unbiased estimate of S = E ( h (zi,zj)) = p,, - pup,. The condition for independence in (3.19) is equivalent to null HO : 6 = 0. Thus, we can use the distribution of Un to test the independence between u and u.
3.1.2
Two-Sample and General K Sample U-Statistics
Now consider two i.i.d. samples of random vectors, yli (1 5 i 5 n1) and y2j (1 5 j 5 n2). We define a (univariate) two-sample U-statistic with m l and m2 arguments as follows:
where hiil:...,ii,, ;2il ,...,2i,, = h (yii,, . . . , YE,, ; mi,, . . . , y2im,) and h is some symmetric kernel function. Note that symmetry is defined with respect to arguments within each sample, that is, h is unchanged when permuting ( i l , .. . , i m l )within each 1 (I = 1,2). Note also that although the random
132
C H A P T E R 3 U N I V A R I A T E U-STATISTICS
vectors yki may have varying dimensions, they usually have the same dimension in most applications. As in the one-sample U-statistics case, U, is an unbiased estimate of the mean of the kernel statistic:
8 = E [~(Ylil;...,Y1i,1,;Y2il,.'.rY2i ,2,)] As noted in the beginning of Section 3.1, in many applications, we are also interested in comparing variances among different treatment groups. The example below defines a U-statistic for comparing variances between two samples. Example 10 (Model for comparing two group variances). Let yki be two i.i.d. samples from two continuous distributions with mean P k and variance a: (1 5 i 5 n k , 1 5 k 5 2). Define a symmetric kernel as follows: h ( Y l i , Y l j ; Y2m, Y2l) = 5 ( Y l i 1
- Ylj)
2
-
21 (Y2m
-
Y2l)
(3.21)
Then, it follows that
8 = E [h( Y l i ! Y l j ; Y2m, Y 2 d 1 If B = 0, then yli and U-statistic given by
y2j
= a: - 0;
have the same variance. Thus, the two-sample
can be used to test the null of equal between-sample variance, Ho : a: = 0;. Note that kernel functions are also not unique for two-sample U-st'atistics. For example, we can define a different kernel for this problem as follows:
9 ( Y l t , Y l j ; Y2m, Y 2 d =
2 2 (Yli - YliYlj) - (YZk
-
Y2mY21)
(3.22)
This kernel is not symmetric. As in the one-sample U-statistics case, we can readily create a symmetric version using an argument similar to (3.10) (see exercise). Further, h ( y l i , y y ; yzm, y2l) in (3.21) is a symmetric version of 9 ( Y l i , y y ; Y2m, Y 2 l ) ' Like the one-sample case, the power of U-statistics lies in studying stmatistics that are intrinsically defined by responses from multiple subjects. For example; in Section 1.1, we discussed the Mann-Whitney-Wilcoxon rank sum statistic t o test the equality of two distributions that differ by a location shift. As noted there, this statistic has two representations that were
133
3.1 U-STATISTICS A N D ASSOCIATED MODELS
introduced independently by Wilcoxon (1945) and Mann-Whitney (1947) but are equivalent. In the example below, we develop the U-statistic for the Mann-Whitney form of the test. We will show their equivalence in the next section after we introduce order statistics. Example 11. ( M a n n - Whitney- Wilcoxon rank sum test). Let yli and y2j be two i.i.d. samples from two continuous distributions with CDFs F1 and F 2 , respectively (1 5 i 5 n l , 1 5 j 5 n2). We assume that Fk differ by a location shift, that is, F1 (y) = F 2 (y - p ) , where p is some unknown scalar. Interest lies in testing the null of no location shift, HO: p = 0 or equivalently,
Ho : Pr (yli - y2j
1
5 0) = -2
versus
Ha : P r (yli - y2j 5 0) #
1
5
(3.23)
Let h (312;Y2j) = ~ { y , z - y 2 3 ~ o } . Then,
8 = E [h(yii; y2j)l = Pr (yii - y2j
5 0)
Following (3.20) and (3.23), the associated U-statistic is given by
The above U, is an unbiased estimate of 8. The null in (3.23) can be expressed in terms of 8 as: Ho : 8 = Diagnostic tests are widely used in biomedical and psychosocial research for screening diseases and disorders of interest. Many diagnostic tests yield continuous readings. However, to facilitate decision making in practice, such test outcomes are often dichotomized to give binary outcomes. Sensitivity and specificity are used to provide a measure of accuracy of a diagnostic test, determined by some cut-off point in dichotomizing the continuous test out come. Consider a diagnostic test for detecting some medical or mental health condition, such as HIV or depression. Consider two i.i.d. samples with one consisting of n1 diseased and the other of 722 disease-free subjects. Let yli and y2j denote the continuous test outcomes from the diseased and nondisease samples (1 5 i 5 n1,l 5 j 1. nz). We assume that larger values of the test outcome indicate the presence of the disease. Let Fk ( t ) = Pr (yki 5 t ) be the CDF of yki ( k = 1,2). Given a cut-point c, the test sensitivity and specificity are defined by
i.
134
CHAPTER 3 UNIVARIATE U-STATISTICS
We can estimate the sensitivity and specificity by the following statistics:
Following Example 1 of 3.1.1, both estimates can be viewed as one-sample U-statistics. Since q (c) and 5 (c) depend on the cut-off c, they vary as c changes. By varying c, we can bring either sensitivity or specificity arbitrarily close to 1. Thus, to compare performance between two diagnostic tests, we must examine sensitivity and specificity simultaneously. The Receiver Operating Characteristics (ROC) curve is widely used as an index of performance of test kit by studying q (c) and [ (c) jointly along the continuum c. An ROC curve is the plot of the bivariate (1 - q (c) ,[ (c)) as a function of c with a theoretical range (--oo,o;;). A good test kit should maintain high values of both q (c) and E (c) across all values of c. The area under the ROC curve (AUC) 8 can be expressed as (see exercise):
8=
i T q ( E ) d[ =
0
[l - F1 ( t ) ]dF2 ( t ) = 1 -
--co
il
F1 ( t )dF2 ( t )
(3.24)
-m
In the next example, we show how to construct a U-statistic for estimating this parameter 8. Example 12 (U-statistic f o r single ROC curve). Let yli and y2j denote the continuous test outcomes from the diseased and non-disease samples as defined above (1 5 i 5 n 1 , l 5 j 5 n2). Define a symmetric kernel as: h (yii; y2j) = I{Y2j
(3.25) We can readily extend the two-sample U-statistics to general K-sample U-statistics. Consider K i.i.d. samples of random vectors, yki (1 5 i 5 n k , l < k < K ) . Let hlil ....) l i m l...; ; Ki1 ....,K i m ,
= h ( y l i l , . . . , Yliml ; *
9
. ; Y K i l . ,YKZ,, j
* *
3.1 U-STATISTICS A N D ASSOCIATED M O D E L S
135
be some symmetric function with respect t.o input arguments within each sample, that is, h is constant when y k i l , . . . , Yki,, are permuted within each kth sample (1 5 k 5 K ) . A K-sample U-statistic with r n k arguments for the kth sample has the general form:
= E (h). ROC curves are widely used to compare accuracy of multiple diagnostic tests. For example, in biomedical research such analyses are used t o compare performance of tests by different manufacturers for detecting infectious diseases such as HIV. In mental health and psychosocial research, ROC curve analyses are often used to compare different instruments for diagnoses of certain mental health conditions and behavioral patterns. Within the context of Example 12, now suppose that we have two test kits and want t o compare their AUCs. Either model below can be applied depending on the design of the study. Example 13 ( U-statistic for comparing A UCs with independent samples). If we have two additional i.i.d. samples from the same diseased and disease-free study populations as in Example 12, then we can apply the second test kit to these two samples to obtain outcomes for computing the AUCs between the two test kits. Let the diseased and disease-free sample have 123 and 724 subjects, respectively. Denote the outcomes from the two samples by y3i and y4j (1 5 i 5 723, 1 5 j 5 724). Then, as in Example 12, we can use the U-statistic obtained by substituting y3i and y4j in place of yli and y2j and n3 and 724 in place of n1 and 722 in (3.25) to estimate AUC for this second test. If we are interested in the null of no difference in AUC between the two test kits, we can define the following U-statistic to test such a null:
As in the one- and two-sample case, Un is an unbiased estimate of I9
The above is a four-sample U-statistic defined by the kernel function h = '{YZl
-
'{Y41
The mean
Of
is
136
C H A P T E R 3 UNIVARIATE U-STATISTICS
where 81, denotes the AUC of test kit k (= 1 , 2 ) . Thus, the null HO : 81 = 82 is equivalent to the null HO : 6 = 0. It is interesting to note that the foursample U-statistic can be expressed as the difference between two two-sample U-statistics:
The study design in the above example does not sufficiently control for subjects' sampling variability. If sample sizes are small, differences in AUCs may also reflect heterogeneity between the two samples within the same disease status. In applications. multiple tests are often applied to the same subject to control for heterogeneity across subjects if the order in which the tests are administered has no effect on test outcomes. Unlike the study design in Example 13. test outcomes become correlated since they are obtained from the same subjects. Thus, new statistics must be constructed to test the equality of AUCs between two test kits, as we discuss in the next example. Example 14 ( U-statistic f o r comparing A UCs with correlated outcomes). Consider applying two tests to each of the two i.i.d. samples of diseased and disease-free subjects as assumed in Example 12. Let ylz and ~ 3 denote % the test outcomes from the first and second tests when applied to the diseased, and ~ 2 and % ~4~ the test results from the first and second kits when administered to the disease-free sample. The AUC for each of the test kits is estimated by the U-statistic in (3.25) in Example 12. However, to test the null of no between-kit difference, the U-statistic in (3.27) does not apply, as there only two i.i.d. samples in the design of this study. To define an appropriate U-statistic, let ~ 1 = % ( y l z ,y32)T and zaJ = T (y2%, Y4z) . Then, h ( ~ 1z z~J ;) = I{yzJsylz) - I{y4Jsy3z) is a symmetric kernel and this kernel-based U-statistic,
is an unbiased estimate of the difference in AUCs between the two kits,
y = E [ h ( z l i ; z p j ) ]= 81 - 82 with 8 k again denoting the AUC of test Ic (= 1 , 2 ) . The null of no between-test difference corresponds to the null Ho:y=O.
3.1 U-STATISTICS AND ASSOCIATED MODELS
3.1.3
137
Representation of U-Statistic by Order Statistic
U-statistics have a close relationship with order statistics, which play an important role in statistical inference. In this section, we first introduce and discuss some properties of order statistics. We then provide a derivation of U-statistics from the perspective of order statistics. Let yi (1 5 i 5 n ) be an i.i.d. sample from a continuous distribution. Let y(q denote the order statistic of the sample, that is, Y(1)
< Y(2) <
* * *
< Y(n-1) < Y(n)
T
Let v = ( y ~ )y,p ) , . - , q,)) be a vector of the above order statistic. As shown in Chapter 1, v is a random vector. Although yi are i.i.d. random variables, the ordered qi)are not. Clearly, qi)are not independent. The theorem below carefully characterizes the marginal as well as the joint distribution of the ordered multivariate statistic vector v, which in particular shows that y(!l are not identically distributed. Theorem 1. Let f (x)and F (x)denote the PDF and CDF of yi. Then, 1. The marginal density function of the j t h component of v is given by:
2. The joint density of the order statistic vector v is given by:
Proof. 1. Let Gj (u)be the CDF of uj = Y ( ~ ) .Then, for any u E R, we have
Gj (x)= Pr (uj5 u)= Pr ( q j )5 u)= P r (at least j y's 5 u)
The last integral is the incomplete Beta function. Now, let H(.) be the CDF of the Beta distribution, Beta ( a ,p), with a = j and p = n - j 1.
+
138
C H A P T E R 3 UNIVARIATE U-STATISTICS
Then, we have: Gj (u)= H ( F (u)).Differentiating Gj (u)with respect to u yields (see exercise): gj (4= h ( F (4) f (4
(3.29)
+
where h (.) denotes the PDF of Beta ( a = j , ,O = n - j 1). 2. Without the loss of generality, consider the case with n = 2 and T v = (y(l),y(2)) . In addition, we need the following lemma (see exercise). Lemma. Let x E Rm be a random variable with a joint PDF f (x). Let y = g ( x ) E R" be a transformation of x. Suppose that there exists a partition, Dk c R", of the support of x and associated transformations, g k (x), defined on each DI, such that each g k (x) is a one-to-one mapping (1 5 k 5 K ) . Then, y has the following PDF:
1
I
where & g i l (y) is the usual Jacobian matrix. T Now let y = (y1, y2) and consider the transformation, v = h (y), which is not one-to-one. However, within each DI, ( k = 1 , 2 ) defined below,
h ( y ) becomes one-to-one. Denote by hk the component of h ( y ) when restricted to DI, ( k = 1 , 2 ) . Then, it follows from the lemma that
The example below illustrates the lemma with a relative simple case of a random variable. Example 15. Let z N ( 0 , l ) and y = h (z)= x2. Then, y = h (z) is not a one-to-one function. By partitioning R into D1 U DZ = (-m,O] U (0, oc),we can define two components h ~that , are one-to-one when restricted to DI,, h k = h (z) if IC E Dk. The inverse of hk exists on each partition Dk:
-
z= h,
1
(y) =
-&
if z E ~
1
, = h;' z
(y) = & if
zE
~2
139
3.1 U-STATISTICS A N D ASSOCIATED MODELS
It follows that the density of y is given by
Thus, x2 follows x:, a x2 distribution with one degree of freedom. Example 16. Let yi be an i.i.d. sample from a continuous distribution with CDF F ( 9 ) . Let ui = F (yi) be the probability integral transformation of yi. We have shown in Chapter 1 that ui has a uniform distribution between 0 and 1, that is, ui i.i.d. U [0,1]. Now consider a similar transformation of the order statistic qi), vi = Further, it follows from Unlike ui, vi are not independent. F (~(~1). Theorem 1 that the marginal wi has a beta distribution, Beta ( a ,p), with cy = j and /3 = n - j 1, while the joint lii has the following density function: N
+
f"
(Ul,
. ,V n ) =
n! if v1 < w2 <, . . . , < v,
' *
0 if elsewhere
Example 17 ( Wilcoxon signed rank t e s t ) . In Example 7 of Section 3.1.1, we discussed the one-sample Wilcoxon signed rank statistic and claimed the equality in (3.15). We are now in a position to show this identity using the properties of order statistics. First, it is readily checked that (see exercise): (3.30) Then, it follows from the above that n
where
U1,
n
i
n
i
and U2n are defined in (3.16) for Example 7 of Section 3.1.1.
140
C H A P T E R 3 UNIVARIATE U-STATISTICS
Example 18 ( M a n n - Whitney- Wilcoxon rank sum t e s t ) . In Section 1.1 of Chapter 1, we presented two forms of the two-sample MannWhitney-Wilcoxon rank sum test for testing whether there is a location shift between two distributions. More specifically, consider a study with two treatment groups with n1 subjects in the first and n2 in the second group. Let yli (1 f i 5 n l ) and y2j (1 5 j 5 n2) denote the continuous response of interest from the first and second groups, respectively. Let Ri denote the rank scores assigned to the observations from the first group that are created based on the combined observations. The two statistics are given by: Wilcoxon rank sum statistic: Wn =
R,!
(3.32)
i=l nl nz
Mann-Whitney statistic: Un =
7;I{yz3-ylz
i=l j Z 1
We have shown in Example 11 of Section 3.1.2 that the normalized Mann-Whitney form, &Un, is a U-statistic. As in Example 17 above, we can also express the rank variables R, as a function of the order statistic y ( k i ) as follows (see exercise): (3.33) By using this order statistic based expression, it is readily shown that under the null of identical CDF between the two samples (see exercise),
(3.34) Further, under the null the mean of Wn is E(W,) = n1(n1+n2S1) 2 (see exercise). By taking expectation of the two sides in (3.34), we obtain (see exercise). This result was also derived in Ex(I{Y(zJ)-Y(l%)50} = ample 11 based on the symmetry of the distribution of the combined sample of y2j and yli under the null. Thus, the null of no location shift corresponds to Ho : ( q Y ( 2 3 ) - 9 ( 1 t ) < 0 } = for the Mann-Whitney statistic. Examples 17 and 18 indicate a close relationship between order- and Ustatistics. In fact, U-statistics can be derived from order statistics. Before proceeding with such an undertaking, let us first derive some useful results for evaluation of conditional expectation.
)
)
3.1 U-STATISTICS AND ASSOCIATED MODELS
141
Let y i (I 5 i 5 n) be an i.i.d. random sample and v = (Y(~),. . . ,Y(~))T the corresponding order statistic vector. Let 7r be a permutation of the integer set (I, ...,n}. Recall that a permutation of a set is a one-to-one mapping from the set to itself. We use 7r ( k ) to denote the image of k under the permutation. For example, if 7r maps (1,2,3) to ( 2 , 3 , l), 7r (1) = 2. As shown in Chapter 3, v is a random vector. Since v is determined by the sample observations y = (yl, . . . ,Y , ) ~ , the joint density of fv,y (w1, * * * , wn,y1,. . . , yn) equals the marginal density of fy (31,. . . , yn). Hence, for any permutation 7r of (1, ...,n}, it follows from Theorem 1 and the Bayes' theorem that
From the above joint distribution of y, we can readily derive the marginal distribution of any subset of y conditional on the order statistic vector v. First, consider the event (y1 = q}. Since this event consists of (n- l)! events of the form (y1 = w7r(l), . . . , yn = where the permutations 7r satisfy 7r (1) = i, it follows that 1 Pr (YI = vi I V ) = n!
1 Pr ( y 1 = w7 r ( l ) , . . . , Yn = %(n) I v) = n
(3.36)
7rE PG
where Pc denotes the set of all permutations of (1, ,.., n} satisfying 7r (1) = i. Now consider the event (y1 = vi,y2 = wj} (i # j). This corresponds to all (n- 2)! events (y1 = u ~ ( ~. ). ,,pn. = with 7r (1) = i and T (2) = j . It follows that 1 Pr (y1 = wi,y2 = wj I v) = (3.37) n(n - 1) and 3 1 (3.38) Note that the equality of the sets (yl, 742) = {vi,uj} in (3.38) implies two . . distinct events: (y1 = wi,y2 = wj} and (y1 = wj,y2 = wi}, which have the same probability in the above setting. More generally, let (il, ...,im} be a subset of (1, ..., n}. The event (y1 = u i l , y2 = wi2,...,ym = vim} is the union of ( n - m)!events of the form
142
CHAPTER 3 UNIVARIATE U-STATISTICS
{ ~ =i ~ , ( 1 ) ., . . , yn = W ~ ( ~ ]with } Thus. it follows that
T
satisfying
T
Pr (91= uil,y2 = viZ,...,ym = uim I v) =
(k)
= ik
( n - m)! n!
for k = 1, ..., m.
(3.39)
Now we are ready to provide an order statistic perspective of the Ustatistics. Let yi, y and v be defined as before. Let g (-) be a Bore1 measurable function from Rm to R. Then, g (yl, . . . , ym) is a random variable. For m = n, it follows from (3.35) that
where Pn denotes the set of permutations of { 1, . . . , n } . If g (y1, . . . , yn) is symmetric, then the above reduces to:
Below, we discuss some examples of how the notion of conditional expectation is used t o show unbiasedness of symmetric functions as defined in a U-statistic. For the concept of conditional expectation and associated properties, the reader is referred to Chapter 1. Example 19. Let h (y) be a symmetric and measurable function. Then, we have .
n
(3.41) To justify this, we can use several approaches. First, let g ( y l , . . . , y n ) = h (yi). Then, g (yl, . . . , yn) is clearly a random variable. It follows from (3.40) that
from which (3.41) follows. Alternatively, we can apply (3.35), which immediat,ely yields (3.41). Example 20. Consider a symmetric and measurable function with m arguments h ( y l , . . . , ym). Following the first approach in the above
3.1 U-STATISTICS A N D ASSOCIATED MODELS
143
example, let g (yl, . . . , y,) = h (yl, . . . , ym). Then, (3.40) yields:
Alternatively, by applying (3.35), we also obtain the above. The two examples above show that a one-group, m-argument U-statistic defined by a symmetric kernel h can be viewed as the conditional expectation of the kernel given the order statistic ~ ( ~. .1. , ~ ( ~ 1From . this perspective, we can readily show some optimal properties of the U-statistic. Let Tn (yl, . . . , yn) be some unbiased estimate of some parameter 8 of interest. Define a U-statistic by (3.43) Then, it follows from the law of iterated conditional expectation that
U n = E [Tn( ~ 1 ,. .. ,~
I
n )v3
Since g (x)= x2 is a convex function, it follows from Jensen’s inequality for conditional expectation (see Chapter 1) that
(~2
E~ (Tn I V) = ( E (Tn 1 v ) ) 5 E ( 9 (Tn)1 V) E I V) By taking expectation again on both sides in (3.44), we have
E
(u:)= E [
E (Tn ~ 1 v)] 5 E [ E (T: I v)]
=E
(3.44)
(~2)
Since E (U,) = E (T,) = 8 , it follows that V a r (Un) 5 Var (T,). Thus, given any unbiased estimate, we can construct a more efficient U-statistic. It follows consequently that the estimate with minimal variance among all unbiased estimates (if it exists) must be a U-statistic. In the above example, if T, itself is a U-statistic, then Un = T, and thus the procedure cannot be continued indefinitely. Example 21. Let yi be an i.i.d. sample with mean p and variance 0 2 . Let 81 = T, ( y l , . . . , y n ) = y1. This single-observation based estimate is unbiased since E ($1) = p. The variance of 81 is V a r ($1) = 0 2 . Now consider the estimate U, in (3.43) defined by T,. The U-statistic in this special case yields h
h
The above Un is the well-known sample mean with a much smaller variance V a r (u,) = 2 n .
144
3.1.4
CHAPTER 3 UNIVARIATE U-STATISTICS
Martingale Structure of U-Statistic
An increasing family of a-fields, {F,; n 1 l}, is called a filtration, that is, n 5 m. A sequence of random variables 2 , is called a martingale if for any n (= 1,2,. . .), 1. z, is measurable .En, 2. E (2,+1 I Fn) = G L . It immediately follows from these defining properties and the law of iterated conditional expectation that for any integers 1 1 k ,
F, C Fm for any
E (xl 1 Fk) = E [ E(zl I 3 - 1 ) 1 Fk] = E
(21-1
1 Fk) = . . . = E ( z k + l I Fk)
= %k
E (zd = E [ E( Z l = E (xk)
I Fdl
Thus, a martingale xn has a constant mean. As discussed in Chapter 1, the conditional expectation E(z,+1 1 F,) represents the average of x,+1 with respect to the a-field 3,. If we interpret n as time, x,+1 is a future event whose outcome is unknown to F,, afield only containing information up to and including the present time n. Thus, E(x,+I I F,) is our best guess or prediction of a future outcome using all available information at the present time in F,. We will exploit such an interpretation in the evaluation of conditional expectation for several examples later. The reader may want to review the definition, properties and examples of conditional expectation in Chapter 1 before proceeding further. Example 22. Let yz be an i.i.d. random sample with mean E (y2) = 0. Let n
2, =
CY2,Fn =
0(21..
*
. .2,)
= ff
(Yl, . . ,yn) *
2=1
where o ( 2 1 , . . . , z,) denotes the a-field generated by the n random variables x2 (1 5 i 5 n). It is clear that F1 C Fz.. . C 3,. . . and thus {F,:n2 l} is a filtration. Since 2 , is measurable F,,it follows that
c+ n
E
(&+1
I
F 7 L )
=
Yz
2=1
= 5,
E
(Yn+l
I Fn) = 2 , + E (Yn+l)
+ 0 = 2,
Thus, z, is a martingale. Note that in this example, 3, does not provide information about yn+l as yi are independent. As a result, E (yn+l I F,)=
145
3.1 U-STATISTICS A N D ASSOCIATED MODELS
E(y,+l) is simply the average of gn+l itself without borrowing any information from the y2 variables observed prior to time n 1. Example 23. Let y2 be a sequence of random variables with mean E ( y 2 ) = 0 (i 1 1). Instead of assuming independence as Example 22, let E (yz 1 y1, .... y2-1) = 0 for i = 2 . 3.... Let
+
n
2,
=
CY,
F n = a ( w . .. ,En)= a ( ! & .
* *
,Yn)
2=1
Since 2 , is measurable F,, it follows that
c+ n
E
(%+1
I Fn) =
Yz
2=
E
(Yn+l
I 6%) = 272 + E (Yn+l)
1
= 2,
+ 0 = 2,
Thus, 2 , is again a martingale. In the above example, if we interpret yn as the earnings on the nth play by a person in a gambling game, z, represents the gambler’s fortune at time n. The quantity E (yn+l I Fn)represents the expected earnings on the (n 1)th play by the gambler. Since this expected earning is 0, the gambler has a fair chance to win or lose on the (n 1)th play. The a-field Fnrepresents the information available t o the gambler right before the ( n 1)th play and E (2,+1 1 .En)the expected fortune after the (n 1) play. The martingale structure in this example represents a fair gambling game. Example 24. In Example 22, let y2 be an independent sample with mean E ( y 2 ) = p # 0. Let zn and Fnbe defined as in that example. It is readily checked that {Fn: n 2 l} is again a filtration. However, since
+
+
+
+
c+ n
E
(&+1
I Fn) =
YZ
E
(Yn+l
2=1
= xn
I Fn) = Yn + E (Yn+l)
+ P # xn
For p > 0, E (zn+lI Fn)> 2 , and z, is called a submartzngale, while for p < 0, E (x,+1 1 Fn) < zn and 2 , is called a supermartangale. Within the context of a gambling game, the submartingale structure of 2 , implies that the game is not fair and in favor of the gambler, while the supermartingale structure indicates an unfair game against the gambler. Since (%+l) = E [ E(&+1 1 Fn)]= E +P xn is not a martingale.
(4
146
CHAPTER 3 UNIVARIATE U-STATISTICS
the gambler is expected to win (lose) p amount of earnings on each additional plan with a total fortune (debt) of n p after n plays if p is positive (negative). The martingale structure also has applications in other areas of research. For example, for a short period of time, the day-to-day fluctuation of a stock price or investment may follow a martingale structure because of the relatively stable economy. However, the monthly or annual yields of a stock price or investment are likely to follow a submartingale as they are augmented by the infusion of company and investment profits. Let h (yl, . . . , ym) be a symmetric kernel of a one-group U-statistic with m arguments and 6' = E [h( y l , . . . , ym)]. Let FO= {a,R} be the trivial cfield and Fk = c (yl, . . . , yk) for 1 5 k 5 m. For an integer k (1 5 k 5 m ) , let
Then, hk ( y l , . . . , y k ) is a random variable since it is the conditional expectation of h given 91,. . . , yk. Since E (hk)= 0, the centered version, &, has mean E (Lk) = 0. Since Fk C Fl for k 5 1, it follows from the law of iterated conditional expectation that for 1 5 k 5 m
Note that for notational brevity, we expressed E ( h k I y1,. . . , yk-1) using E ( h k I Fk-1)in (3.45). For 1 5 k 5 m, define
To appreciate the above, consider three special cases with k = 1, 2 and 3
3.1 U-STATISTICS AND ASSOCIATED MODELS
147
for which the above reduces to
By applying (3.45) to the above, we obtain
By exchangeability and symmetry, we have
E
[92 (Yl,Y3)
I Yl] = E [g2 (y2, y3) 1 y2] = E [g2 (yl,7J2) I yl] = 0
(3.47)
It follows from (3.45) and (3.47) that
More generally, it is similarly shown that (see exercise)
Note that since yi are an i.i.d. sample, yi are exchangeable. Thus, (3.48) holds true for any k-member subset yil, . . . , yik with Fk-1defined by any of the k - 1 variables of this subset. The following theorem shows the martingale structure of a one-group U-st at ist ic.
148
C H A P T E R 3 UNIVARIATE U-STATISTICS
Theorem 2. Let
Then, we have: 1. For each k , when considered as a random sequence of n, Skn is a martingale with respect to the filtration, {Fn = 0 (91,. . . , yn) ; n L k } . 2. The one-sample U-statistic, U,, defined in (3.8) has the following representation: m
(3.50) k=l
Proof. We consider the case with m = 2 and the more general case is similarly considered. By (3.49), we have n
n
i=l
i=l
It follows that
The last expression is the same as the right side in (3.50) when written out in terms of San and S1,.
149
3.1 U-STATISTICS A N D ASSOCIATED MODELS
To show that both S1, and S2nare martingales with respect to Fn, first consider S1,. For 1 5 1 5 n, if i > 1, then yi and Fl are independent. It follows that
E [Sl(Yi)
El
=
[g1(Yi)]= E [jEl(?/i)]
Otherwise, for 1 5 i 5 1, gi (pi) is measurable g1 (yi). It follows that n
=0
Fl and E [gl(yi) 1 5 1=
1
Now, consider S2n. If i > I, then yi and from (3.48) that
Fl are
independent.
It follows
E 192 (Yi,Yj) I fil = E [ E(92(Yi,Yj) I Yj,Y1, * > YZ) I El =E I E(92(Yi,Yj) I Yj) I El = E (0 I 6)
(3.51)
=o
Similarly, it is readily checked that the above also holds true if j exercise). Thus, it follows that
> 1 (see
Example 25. Let yi be an i.i.d. random sample and consider the sample mean: Un = y,, which is a one-sample U-statistic with one argument. Thus, m = c = 1 and the kernel h (y) = y. It follows from Theorem 2 that n
and S1, is a martingale with respect to the filtration
We can also verify this directly using the definition of martingale:
E (Sqn+l) I Fn) = s 1 n +
(Yn+l - 0
I Fn) = s1,
150
CHAPTER 3 UNIVARIATE U-STATISTICS
Note that it follows from (3.50) that
n
n
m
i= 1
c=2
Thus, the centered U-statistic-like quantity Un - 8 (Un - 8 is a U-statistic if 8 is known) can be expressed as an i.i.d. random sum CZ1hl (yi) plus an extra term (:)-'Scn. This decomposition plays an important role in the study of asymptotic properties of U-statistics. If the second term in (3.52) is a s y m p t o t i c a l l y negligible, that is, Un and the first term have the same asymptotic distribution, then inference for 8 can be based on the asymptotic distribution of the i.i.d. sum. This is indeed the case as we show in Section 3.2.
EL2(T)
3.2
Inference for U-Statistics
In this section, we discuss inference for univariate U-statistics. More specifically, we want to investigate the asymptotic properties of such statistics when used as estimates of parameters of interest. In the case of U-statistics, however, it is not possible to directly apply conventional methods [e.g., law of large numbers (LLN), central limit theorem (CLT)] to examine the large same behavior of such statistics, as in the case of the study of regression models in Chapter 2. To illustrate this point, consider again the problem of estimating the variance, g2, of an i.i.d. random sample y introduced in the beginning of Section 3.1. Based on the preceding section, we can represent the usual sample variance s: in the form of a one-group U-statistic as follows:
By taking expectation, it immediately follows that s i is an unbiased estimate of cr2. However, if we want to study the asymptotic behaviors of this statistic, such as consistency, the U-statistic expression above is not in a form that is conducive to the usual treatment by direct applications of LLN and CLT. The problem lies in the unique definition of U-statistic.
151
3.2 INFERENCE FOR U-STATISTICS
Unlike the statistics we studied in Chapter 2, U-statistics are defined by responses from multiple rather than a single subject. For example, in the case of the sample variance s i , the U-statistic in (3.53) expresses the variability of yi by averaging squared differences over all pairs of responses, yielding a sum of correlated random variables. Although we have seen sums of correlated random variables before, they are fundamentally different from the U-statistic in (3.53). For example, the conventional expression of s i in (3.1) is also a sum of dependent variables. However, by rewriting s t in the following form: .
n
(3.54) we can apply LLN, CLT, and Slutsky's theorem to obtain the asymptotic properties of s: (see Chapter 1 for details). The key difference between the two approaches is that the quantity that causes the dependence in the conventional expression, y,, converges to p in probability so that the dependence can be removed asymptotically by substituting p in place of Yn as expressed in (3.54). However, the dependence in the U-statistic expression in (3.53) cannot be eliminated in such a substitution fashion. The basic idea behind tackling the dependence of a U-statistic is to create a random quantity, called projection, in the usual form of a sum of i.i.d. random variables that approximates the original U-statistic so well that they become indistinguishable asymptotically, that is, they have the same asymptotic distribution. Since we know how to find asymptotic distributions of sums of i.i.d. random variables, the creation of such a projection of random variables will then allow us to find the asymptotic distribution of the underlying U-statistic. The existence of such a sum of i.i.d. random variables was suggested by Theorem 2 of Section 3.1.4. As noted earlier in the discussion of this theorem, we can express a one-sample U-statistic with m arguments as follows:
where 8 = E ( h ) and hl (yl) = E [(hl(yl, . . . , ym) - Q) I yl]. The first term above is a sum of i.i.d. random quantities. Thus, if the second term in (3.55) is asymptotically negligible, that is, Un and the first term have the same asymptotic distribution, then hl (yi) will be the projection we
2cr=l
152
C H A P T E R 3 UNIVARIATE U-STATISTICS
seek. We now formalize this approach and systematically study such a decomposition of a U-statistic.
3.2.1
Projection of U-statistic
Consider again the class of one-sample U-statistics with m arguments: (3.56) where C&{ (21, . . . , im) ; 1 5 i l < . . . < im 5 n} denotes the set of all distinct combinations of rn indices ( i l , . . . , im)from the integer set { 1 , 2 , . . . , n } and h (yi,, . . . , y,i) a symmetric kernel function defining an i.i.d. sample of p x 1 random vectors yz (1 5 i 5 n). The projection of U,,denoted by is defined as follows:
G,,
where 0 = E [h(yi,, . . . ,yi,)] is the parameter of interest. The appearance of E ( U , I yi) is a bit deceiving since the fact that U, is a statistic easily carries the impression that Cy=lE (U,1 yi) is also a statistic. However, as is seen shortly in Example 1, E (U, I yi) is generally a function of 8 and thus is not a statistic. Nonetheless, what is important about is the fact that it is an i.i.d. sum to allow us to readily study its properties using conventional asymptotic techniques. Note that the first term in (3.57) is a sum of i.i.d. random variables, E ( U , 1 yi), which is the conditional expectation of U, given each individual response yi (1 5 i 5 n),while the second part is a normalizing factor so that E U, = 8. Thus, U, can be viewed as projecting U, onto each yi of the i.i.d. sample (1 5 i 5 n ) . Example 1 (U-statistic for sample m e a n ) . Let yi be an i.i.d. sample with mean p and variance c2 (1 5 i 5 n). Consider estimating p with the sample mean, U, = y,; which is a U-statistic with kernel h (y) = y. Since E ( y j I yi) = yi if j = i and E (yj 1 yi) = p if otherwise, it follows that
fin
(- 1
1 --yi
n
+ ( n n- 1)c1 ~
h
3.2 INFERENCE FOR U-STATISTICS
153
(3.58)
Gn
In this case, the projection is the same as the original U-statistic. However, it is seen that the first term, C:=l E (Un 1 yi), involves the unknown parameter p and thus is not a statistic. Nonetheless, the projection in this particular application is a statistic and is the same as the original statistic g,. Example 2 ( U-statistic for sample variance). In Example 1, consider the sample variance s: as an estimate of v2. As shown in the preceding section, the U-statistic expression of s: has a symmetric kernel h (yj, yk) = $ (yj - y ~ c ) ~It. follows that
fin
To compute E ( h (yj, y k ) I yi), notice that when ( j , k ) rai zes over CF, either j or k equals i or neither is equal to i. In the first case, that is, j = i or k = i, we have by symmetry of h (yj, yh):
1
=2
(Y? - 2PLyi + E (Y?))
In the second case, that is, j
# i and k # i, it follows that
Thus, E ( h ( y j , y k ) 1 yi) can be characterized by whether i E { j , k } . Let hi ( ~ i )= E ( h (yj, yk) I yi) when i E { j , k } . Then, we can compactly express (3.59) and (3.60) as:
(3.61) Note that because of the exchangeability of yi, (3.61) can be rewritten as: h~ (yi) = E ( h (yl, y2) 1 y l ) . We introduced hl as well as its centered version
154
C H A P T E R 3 UNIVARIATE U-STATISTICS
-
hl = hl- 8 in Section 3.1.4 for a general one-sample U-statistic with m arguments in the form of hl(Y1) =E[h(Yl,...,Ym) I Y11
where 8 is the mean of the kernel h. This conditional expectation plays a critical role in the study of asymptotic behavior of U-statistics. It follows from (3.61) that
where Djk = ( ( 1 , m) : (1, m) E CF, 1 # i, m # i}. tion of U, = s i is given by
xy=l
It follows that the projec-
Unlike Example 1, neither the i.i.d. sum E (Un 1 yi) nor the projection itself is a statistic. In fact, the projection bears no resemblance to the original U-statistic in this example at all. Nonetheless, Gn has the desired form of a sum of an i.i.d. random quantities. For the general Gn in (3.57), by applying an argument similar to the projection in Example 2, we have
=
m n-m -h1 (Yi) -e n n
+
155
3.2 INFERENCE FOR U-STATISTICS
where Gi = { i l ,. . . , im}. as :
h
It follows that the projection U, can be expressed n
(3.62) As U, is an unbiased estimate of the parameter of interest 19,it is convenient to center U, as follows when studying its asymptotic behaviors:
-
where h = h - 8. Analogously, we can define a centered projection:
i=l
i= 1
It follows from (3.62) that
i=l
Since E
( X I (yi))
=E
[ E( h 1 yi)
-
e] = 0, it immediately follows from (3.65)
(Gn)
h
= 0. Thus, both U, - 0 and U, - 8 have mean zero. that E Example 3. Consider Example 1 again. We have U, = y, and h (y) = y. Since
it follows that
It immediately follows from LLN and CLT that
En +p e,
fi (ti, - e)
= fi (y, -
e) +d
N
(o,02)
For this U-statistic, we can also apply CLT directly to U, to obtain the same asymptotic distribution as above.
156
C H A P T E R 3 UNIVARIATE U-STATISTICS
Example 4. In Example 2, we have 1
Gn - 2 = -
c n
2 [hl (yi) - 2 1
i=l
By applying LLN and CLT, we have
Gn -+p
o2 and
d i n 2 [hi(yi) - 01 +p 6(a,- g2) = -
N (0,4Var (hl (yl)))
(3.66)
i=l
To find V a r (hl ( y l ) ) , first note that
1 2 1 = - “91 2
= - [y? - Y1P -
+ E I)&(
d 2-
I”.
- cT2 =
1
5 (YT - Y1P + P2 +)’.
Thus, (3.67) 1
=
4 (P4 - g4)
where p4 = E (y1 - p)4denotes the fourth centered moment of yi. It follows from (3.66) and (3.67) that
In this simple example, we can also find the asymptotic distribution of s; directly using a Taylor stochastic series expansion as follows:
By applying CLT and Slutsky’s theorem to the above decomposition, we obtain & (3; i - 2)f p N (0, p 4 - 04) Both Examples 3 and 4 show that the projection and the U-statistic have the same asymptotic distribution. This is not a coincidence and is in fact
157
3.2 INFERENCE FOR U-STATISTICS
true for the general U-statistics, as we show in the next section. As noted earlier, the projection in general is not a statistic and thus the projection itself can not be used for statistical inference. The role of the projection is to serve as a vehicle from which to derive the asymptotic distribution of the underlying U-statistic. Note that in (3.68), we must assume p4 # u4 since otherwise the asymptotic distribution is degenerate and s2 becomes asymptotically a constant. In practice, the situation with p4 = u4 is unlikely to occur. A notable exception is the Bernoulli random variable. If yi is a binary variable with the value 0 and 1, then p=E(y1),
i,
a2=p(1-p),
p4=
(i)4, (i)4
(1-d4P$-P4(1-P)
When p = p4 = u4 = and thus p4- u4 = 0. In this special from the theoretical mean case, the squared individual deviation (yi is a constant By taking variability in the sample mean in the definition of s i into consideration, we have
i)2
i.
Thus, s2 is asymptotically a constant
i
i.
3.2.2 Asymptotic Distribution of One-Group U-Stat ist ic To show that the centered projection, Gn - 0, has the same asymptotic distribution as the centered U-statistic, U, - 0, consider the following relationship: &(u~-o)=&(G~-o (3.69) h
where en is the normalized difference between Un and U,,
en = f i ( U n
- 0)
-
f i (Gn
-
0)
=
fi (Un - Gn)
(3.70)
If en = op (l),then it follows from Slutsky’s theorem that U, - 0 and Gn - 0 have the same asymptotic distribution. This is indeed the case, but requires some effort t o prove. We summarize the asymptotic results regarding the one-sample U-statistic in a theorem below.
158
CHAPTER 3 UNIVARIATE U-STATISTICS
Theorem 1 (Univariate one-sample). Let
hl ( Y l ) = E [h( Y l , . . . , Ym) I Y l ] , e, =
6(U, - fin)
,
0;
-
hl ( Y l ) = hl ( Y l )
-
6
= Var F 1 (yl)]
Then, en +p 0. Thus, 1. Un is consistent, that is, U, +p 8. 2. U, is asymptotically normal, f i (u, - -'d N (0, cr$ = m2ai). Proof. In light of the relationship between U, and as expressed in (3.69) and (3.70), the conclusions follow if we can show e, +p 0. Since E ( e i ) -'p 0 implies e, - f P 0 (see exercise in Chapter l ) ,we prove E ( e i ) +p 0 below. To this end, we first prove the following lemma: Lemma. For 1 5 1 5 m, let
e)
hl ( y l , . . . , y l > = ~ 0;
6,
-
( I yhl , .. . , X I , h1= h2 - e
= V a r (hl ( Y l , . . . , Yl)) = E [Ll ( Y l , . . . , Yl)]
2
Then, we have
(3.71) =
m" --ar n
+
(hl (y1)) 0 (n-2)
where 0; = 0. Proof of Lemma. By expanding Var (U,), we obtain
Let ( k l , . . . , k l ) be the indices common to then follows that
( i l , . . . , im)
and
( ~ ' 1 ,... , j m ) .It
159
3.2 INFERENCE FOR U-STATISTICS
Since the number of distinct choices for two such sets having exactly I elements in common is: (see exercise), it follows that
(E) (7)(z-7)
=
2
(;)-I(y)
(n-m).:. rn-1
1=0
This proves the first equality in (3.71). Note that since 0; = 0 in (3.73), the summation may start from 0. To show the second equality in (3.71), let c (n,m, 1 ) = (,)n -1 (m) Then, we have:
(z-7).
[(n- rn) . . . ( n - 2 m + 1 + 111 [(n- 2m + 1)!12 2 (n- m ) . . . ( n - 2m + I + 1) . = I! n . . . (n- m 1)
(7)
+
where o (.) denotes the deterministic small o (see Chapter 1 for definition and properties). The last equality above follows from the fact that as n --+ 00,
+ 1 + 1) = nm + o (nm) n . . . (n- m + 1) = nm-' + o ( n m )
( n- m) . . ( n - 2m +
(3.74)
Thus, it follows from (3.72) and (3.73) that
= -0: m2
+o
(nP2) n Now, consider proving E (e:) 0 for the theorem. E ( e i ) , we have --f
E (ei) = n v a r
(6.)
- 2 n ~ o w(un,6n)
By writing out
+ n v a r (un>
160
C H A P T E R 3 U N I V A R I A T E U-STATISTICS
The proof involves straightforward calculations of each of three three terms above. First, let us find nVar Un . Since Un is a sum of i.i.d. random terms as expressed in ( 3 . 5 7 ) ,it follows immediately that h
(- )
If i q!
( i l , . . . , im),
If i E
(21,.
then
. . , im), we have
4 1 ,...,i,,i
-
=
[ R ( Y i l ?. . . , Yi,)
= Var
hl (Yi) 1 Yij
(h(Yi))
Since for each i, the event {i E (il, . . . , im)} occurs cise), it follows that
(z::) times (see exer(3.76)
= m2Var ( h l ( Y l ) )
Finally, consider nVar have
(Un).By applying the lemma to V a r (iYn),we
nvar (un) = m2Var (hl (yl)) = m2Var (
+ o (n-') ~ (yl)) 1 + o (1)
(3.77)
161
3.2 INFERENCE FOR U-STATISTICS
Thus, it follows from (3.75), (3.76) and (3.77) that
E (en) 2 = m2 ~ a (hl r ( y 1 ) ) - 2 m 2 ~ a r(hl (yl))+m2Var( h l ( y l ) ) + o (1)
o
j P
With the asymptotic results in place, we can now make inference for statistical parameters of interest modeled by U-statistics. We demonstrated how to find the asymptotic distributions for the projections of the U-statistics for the sample mean and variance in Examples 3 and 4. For these two relative sample statistics, we are also able to derive their asymptotic distributions by direct applications of standard asymptotic methods. Below are some additional examples to show how to evaluate the asymptotic variance of the U-statistic. For the statistics in these examples, however, it is not possible to find their asymptotic distributions by directly applying the conventional asymptotic techniques. Example 5 (One-sample Wilcoxon signed rank t e s t ) . In Section 3.1, we showed that one-sample Wilcoxon signed rank test is asymptotically equivalent to the following U-statistic:
By Theorem 1, U, is asymptotically normal. To find the asymptotic variance, first note that
Thus, the asymptotic varianc~eof U, is given by (3.78)
Now, consider testing the null that yi are symmetrically distribution around 0, HO : 8 = 1 2. Let F (y) denote the CDF of yi. Since yi are symmetric about 0 under Ho, yi and -yi have the same CDF F ( y ) . To evaluate the first term in (3.78), note that given y1 = y ,
Let ui = F (yi) be the probability integral transformation of yi. As shown in Chapter 1, under Ho, ui follows U [0,1], a uniform distribution on (0,l).
162
CHAPTER 3 UNIVARIATE U-STATISTICS
Thus, under Ho, we have
Under Ho, Un has the following asymptotic distribution:
Example 6 (U-statistic f o r covariance). Let zi = ( ~ i , y i be )~ i.i.d. pairs of continuous random variables (1 5 i 5 n ) . Consider inference for 0 = Cow ( z i , y i ) , the covariance between xi and yi. In Section 3.1, we introduced a U-statistic Un in (3.6) to estimate 8 defined by the kernel function: h (zi, zj) = (xi- zj)(yi - y j ) . It follows from the Theorem 1 that U, is unbiased and asymptotically normal, with the asymptotic variance given by u2 = 4Var hl (z1) .
1
(-
To find u 2 ,pi (zi, p ) = (xi- p,) (yi that (see exercise)
1 =2 [kl
-
P,)
(Y1
-
Py)
- py).
Then, it is readily checked
+ 01
It follows that
Thus, the asymptotic variance of Un is u2 = V a r (p1 (z1, p ) ) . Under the assumption that zi is normally distributed, o2 can be expressed in closed form in terms of the elements of the variance matrix of zi (see exercise). Otherwise, a consistent estimate of u2 is given by
3.2 INFERENCE FOR U-STATISTICS
163
where pz (Fy)denotes some consistent estimate of px ( p y ) such as the sample mean. Example 7 (Kendall’s 7 ) . Let zi = ( ~ % , y be % i.i.d. ) ~ pairs of continuous bivariate random variables (1 5 i 5 n ) . In Section 3.1, we discussed Kendall’s T as a measure of association between xi and yi and estimating this correlation measure using U-statistics. Unlike the product-moment correlation, Kendall’s 7 defines association based on the notion of concordance and discordance and as such provides a more robust association measure. In this example, we consider inference for this important correlation index. Kendall‘s 7 is defined by p , = E [I{(S1-z1)(Y’-Y3)>0}]. It is often normalized so that it ranges between 0 and 1 to have the interpretation as a correlation coefficient. For convenience, we illustrate the consideration with Pc-
The U-statistic based estimate of 0 = p , is given by
h
It follows from Theorem 1 that 0 is consistently and asymptotically normal. The asymptotic variance has the form:
Unlike the previous example, it is not possible to evaluate a; in closed form. However, we can readily find a consistent estimate of 0;. Since
we can express
CT;
as:
Let g (z1, z2, z3) be a symmetric version of h (z1, zz) h (z1, z3). As noted in Section 3.1, such a symmetric function is readily constructed. Then, we
164
CHAPTER 3 UNIVARIATE U-STATISTICS
can estimate
C = E (hf ( z 1 ) ) using the following U-statistic:
By Slutsky's theorem, we can estimate the asymptotic variance
4
(T
-
by Z i =
g2).
kxample shows that we can estimate the asymptotic variance of 0 using U-statistics. This is true for general one-sample U-statistics. If U, is defined by a symmetric kernel h ( y l , . . . , y m ) , then E (hl ( y l ) ) can be expressed as (see exercise) :
h
' This
E (h: (
~ 1 )= )
=
E [h( Y I ,~
.,
2 , . . ~ mh )( y i ,ym+i,. . .
am-^)]
(3.82)
19 ( Y l , Y 2 , .. * ,Yzm-1)1
Thus, by defining a symmetric kernel based on g ( y l ,y2, . . . , yzm-l), we can use U-statistics to estimate E (hl ( y l ) ) .
3.2.3 Asymptotic Distribution of K-Group U-Statistic The projection and the asymptotic results in Theorem 1 are readily generalized to K-sample U-statistics. First, consider a two-sample U-statistic with ml and m2 arguments for the first and second samples:
where h is a symmetric kernel with respect to arguments within each sample and yki is an i.i.d. sample of p x 1 column vectors ( 1 5 i 5 n k , 1 5 k 5 2). The projection of U, is defined as follows:
(3.83) k=l
i=l
where 0 = E ( h ) . By applying arguments similar to those used for onesample the projection in Section 3.2.1, we can readily express the projection and its centered version in the following form (see exercise):
165
3.2 INFERENCE FOR U-STATISTICS
-
where hlk
(yki) =
hlk (yki) - 0 and
hlk ( ~ k l= ) E
[h( ~ 1 1 : .* *
?
hlk
(yki) is defined by
,~ 2 m z )I ykl]
~ l m ~l 2 1 * ,*
?
k
=
1,2
Using an argument similar to that for the one-sample U-statistic, we can also show that Un and Gn have the same asymptotic distribution (see exercise). To find the asymptotic distribution of Gn, let n = n1 n2 and assume limn+m $ = p; < m ( k = 1,2). Then,
+
= Snl
Let
(-
+ Sn2
= V a r hkl ( y k i )
flik
Snk +d
Since Snl and h
un + p
0:
Sn2
).
By applying CLT to Snk, we have
N (0;P;m;var
(hkl (Ykl)))
k
?
=
1?2
are independent, it follows from Slutsky theorem that
fi (C?" - 0)
= snl
+ sn2 A
d
N (0, ~2l m2 l 2o h+~~22 m2 220 h ~ )
We summarize the asymptotic results for the two-sample Un in a theorem below. Theorem 2 (Univariate two-sample). Let n = n1 122 and assume limn.+m $ = p i < 00 ( k = 1,2). Then, under mild regularity conditions, we have:
+
un + p
0,
6(un- 0) +d
N (0,fl;
2
2 2
= Plmlohl
2 2 2 + P2m20h2)
(3.85)
Note that the assumption limn+m $ = p i < 00 in the theorem is to ensure that that nk increase at similar rates so that + c E (0,00) as n + m. We may define n differently without affecting the expression for the asymptotic distribution of U,. For example, if n = rnin(nl,nz), then we require limn+m A = q i < 00 to ensure + c E (0, co) ( k = 1 , 2 ) . 121, The asymptotic distribution of Un remains the same form as given in (3.85) except for replacing p i by c; in the asymptotic variance g;. Example 8 (Model for comparing variance between two groups). In Example 10 of Section 3.1, we discussed a U-statistic for comparing the variances of two i.i.d. samples yki (1 5 i 5 nk, 1 5 k 5 2) defined by the following kernel:
2
2
h (Yli,
1
Y l j ; Y21, Y2m) = 2
2
(y12 - y y ) -
1
2
(y21 - y2m)
166
CHAPTER 3 UNIVARIATE U-STATISTICS
The mean of the kernel is 0 = E ( h ) = a: - oi,the difference between the two samples’ variances. To find the asymptotic variance of this U-statistic, first note that
Thus, h11 (yii) = hi1 (yii) - 8 =
[(yii
2 - p1)
-
c7:]
and its variance is
where p k 4 denotes the centered fourth moment of yki ( k = 1 , 2 ) . Similarly, we find
Thus, the asymptotic variance of the U-statistic is given by
Under the null of no difference in variance between the two groups, Ho : 0 = a: - 0: = 0, (3.86) reduces to
where o2denotes the pooled variance over the two samples. We can estimate 0; in (3.86) by substituting moment estimates for &4 and and as estimates for p i (IC = 1,2). Example 9 ( Two-sample Mann- Whitney- Wilcoxon rank sum test). In Example 11 of Section 3.1, we discussed the U-statistic version of the two-sample Mann-Whitney-Wilcoxon rank sum test:
02
It follows from Theorem 2 that
167
3.2 INFERENCE F O R U-STATISTICS
To find
oil, first note that
Thus,
denote the CDF of Y k i . Under the null Ho : 8 = 1 or F k ( 9 ) = F (y), the probability integral transformation, F (yki) U [0,1]( k = 1 , 2 ) . It follows that Let
-
F k (y)
2
= E(1-
q2- -41 = 1
-
1
1
dl= E [ E (I{yl%5yzJ) I Yli)] - 4=E
- F (Yli)I2 -
4
1 1 1 2E(U) + E ( U 2 ) - - = - - 4 3 4
1 12
-
Similarly, cri2 =
A. Thus, we have
Given the sample sizes
nk:
the asymptotic variance of U, is estimated by
Now consider the general K-sample U-statistic U, defined in (3.26) of Section 3.1.2, we define the projection of U, as follows: K
nk
6,= x x E ( u n I y k i ) k = l i=l
c n k (k:l
The centered projection can be expressed as:
where 8 = E (h)and
)
1 8
(3.87)
168
CHAPTER 3 UNIVARIATE U-STATISTICS
Gn
As before, it can be shown that Unand have the same asymptotic distribution (see exercise). By employing an argument similar to the one used for the two-sample case, we can also readily determine the asymptotic distribution of the projection and therefore the asymptotic distribution of U,. We state these results in a theorem below. n -Theorem 3 (Univariate K-sample). Let n = c kK= l nk and nk p i < 00 (1 5 k 5 K ) . Let
Gn
Then, under mild regularity conditions, we have
where
mk
is the number of arguments corresponding to the kth group.
As noted in the two-sample U-statistics case, n may be defined differently
2
so long as the requirement + Ckl E ( 0 , ~ is) ensured for any nl and n k ( I 5 k , l 5 K ) . The asymptotic distribution of Un retains the same form regardless of how n is defined. Example 10 ( U-statistic f o r comparing A UCs with independent samples). Consider the problem of comparing the AUCs of ROC curves for two diagnostic tests with four independent samples discussed in Example 13 of Section 3.1.1. The four-sample U-statistic is defined by the kernel:
It follows that
where 81 is the AUC for the 1th test (1 = 1,2). Under the null of no difference between the two test kits, 6 = 81 - 6 2 = 0 and 81 = 6 2 = 8. In this case,
It follows that for each 1 5 k 5 4,
169
3.2 INFERENCE FOR U-STATISTICS
and the asymptotic variance of the U-statistic is To find o i k ,first consider k = 1. Then,
-
hll
(Ykl) =
=E
0;
('{yZl
- I{y41
(I{YZl
I Y11)
-
2 2 = C k4 = l pkahk.
I Y11)
(3.91)
02
It follows from (3.90) and (3.91) that 4
1
=
[ ( E( I I Y Z l 5 Y l l }
=E
[E(I{Y215Yll}
=E
[ E( G Y Z l 5 Y l l )
I 911) - 0212]
IYd2]
-
I Y11) E I Y l l ) ] + 0; I I Yll)]
(3.92)
I Yl4l -
(~{,215,11}
- 2QE[ E ( l i Y Z l 5 Y l l )
=E -
E
[ E( I { Y Z l < Y l l } (GY215Y11)
{Y21
I{ Y 2 1 1 Y l Z d
- 20102
I Y l l ) ] + 0;
[E( 4 Y Z l 5 Y l l )
-
20201 + 0;
+ 0;
Similarly, we can find oik for k = 2 , 3 , 4 (see exercise): 4
2
gi3
= E (I{YZl5Yll)
I{ Y 2 2 < Y 1 1 ) )
- 20102
= =
I{Y41
- 2e1e2
(I{y41
+ 0; + '? +
(3.93)
'?
I{Y?2
(3.94)
k=l1=1 Since the mean of the kernel of u k n in (3.94) is e k , we can estimate each 8 k using u k n ( k = 1,2). Alternatively, we can pool the two samples for each disease status to estimate the common 0 under the null Ho : 01 = 02 = 0 using the following U-statistic:
170
CHAPTER 3 UNIVARIATE U-STATISTICS
where n1 + n 3 and 122 + n4 are the sample sizes for the combined diseased and disease-free samples, respectively. Now, consider estimating the first term in the expression for o i k as shown in (3.92) or (3.93). For convenience, consider C1 = E (I{y21
is a consistent estimate of C1. Similarly, we can find the U-statistics for estimating the first term for the other three o i k (see exercise). Like Kendall's T , it is not possible to evaluate in the asymptotic variance in closed form (1 5 k 5 4). As in the case of one-sample U-statistics, we can U-statistics to construct a consistent estimate of u i k and consequently a consistent estimate of the asymptotic variance o;.
oik
3.3 Exercises Section 3.1 T 1. Let Zi = ( ~ i , y i be ) ~ an i.i.d. sample with mean p = (pz.py) , 2 variance oz, ot and covariance ozy (1 5 i 5 n ) . Show that the sample covariance Zzy has two representations:
where ?En = 1 Ey=lxi and Cg denotes the set of all distinct pairs ( i , j ) from the integer set { 1 , 2 , . . . , n}. 2. Let yi be an i.i.d. sample from a continuous distribution (1 5 i 5 n). Let Ri denote the rank of yi. For the Wilcoxon signed rank test defined in (3.14), show n(n+l) (a) W$ ranges between 0 and 7. + and Var(WZ)= (b) E ( W n ) (c) Under the null Ho : 0 = 0, the sampling distribution of W$ is given by ifoIkLT n(n+1) =k) = Pr 0 if otherwise
n(n+lJy+l)
(~n+
$
171
3.3 EXERCISES
Qi
where is the number of subjects from the integer set { 1 , 2 , . . . ,n} for which the sum is Ic. 3. Within the context of Problem 2, verify the steps in (3.16) and (3.17). 4. For the ordinal categorical outcomes in Example 8 of Section 3.1.1, show that both indices, T b and y,range from -1 to 1. 5. Symmetrize the kernel g (yli, ylj, y2m, y 2 ~ )in (3.22) and show that h (yii, y i j , ~ 2~ 2 ~1 in ) ,(3.21) is a symmetric version of g (yli, ylj, ~2~~ ~ 2 1 ) . 5. Verify (3.24). 6. For the four-sample U-statistic in (3.27) for ROC analysis, show that it can be expressed as the difference between two two-sample U-statistics as shown in (3.28). 7. Consider the proof of Theorem 1. (a) Verify (3.29). (b) We can also prove Theorem 1/1 using a simpler, heuristic argument as follows. Consider a small interval around t , (t - A,t A), and let
+
A = ( ~ ( 1 ) < t - A;1 5 1 5 j
C = { Y ( ~ )> t
-
I},
+ A;j+ 1 5 rn 5 n }
B = { y ( j , E (t - A,t
+ A)}
Then, since
it follows that gj
( t )A
-
+
+
(A n B n C ) o (A) = Pr ( A )P r ( B )P r (C) o (A) ( n - I)! nFj-' (t - A) [l - F ( t A)]"-j f ( t )A ( j - l)!( n - j ) !
= Pr
+
+ o (A)
where o (.) denotes the deterministic 0. We obtain the density gj (u)in (3.29) by dividing both sides of the above equation by A and letting A + 0. (c) Prove the Lemma. 8. Verify (3.30) and (3.31). 9. Consider the two forms of the two-sample Mann-Whitney-Wilcoxon rank sum test in (3.32). (a) Use the expression in (3.33) to show (3.34). (b) Show ( q Y ( 2 j ) - Y ( l i ) 5 0 } = [Hint:Consider taking expectation of both sides in (3.34).]
) i.
172
CHAPTER 3 UNIVARIATE U-STATISTICS
10. For the Wilcoxon rank form of the test in (3.32), show and V a r(W,) = n l n 2 ( n l + n z + 1 ) (4 E ( W n )= 121 (n1+nz+l) 2 12 (d) Under the null of identical CDFs between the two samples, the sampling distribution of W, is given by: n -1Qk
Pr(W, = k ) =
(,1)
.n2
ni(ni+l)
in if 7 Ik
I
7L1 (n1+2n2+1)
2
if otherwise
+
where n = n1 722 and Q:,,,, is the number of unordered subsets of n2 integers taken without replacement from { 1 , 2 , . . . ,n } for which the sum is k. 11. Consider the proof of Theorem 2. (a) Verify (3.48) (b) Verify (3.51) for the case when i > 1 (1 5 i,l 5 n ) . (c) Extend the proof of Theorem 2 for the general case with k 2 3, that is, for each k , when considered as a random sequence of n,Skn is a martingale with respect to the filtration, {F,= u (yl, . . . , y,) ; n 2 k } . Section 3.2 1. Consider the integer set { 1 , 2 , . . . ,n}. For 1 I m I n,let (21,. . . , im) be a permutation of ( 1 , 2 , .. . ,n ) . Show (a) The number of choices for two such permutations (21,. . . , im) and ( j l , . . . j m ) to have exactly 1 elements in common is given by: (",--";).
(k)(7)
(b) For a given integer i, the event {i E ( i l , . . . , im)}, that is, i is one of il, occurs ( ;I;) times (1 5 I 5 m). 2. Verify the two equalities in (3.74). ) ~ i.i.d. pairs of continuous random variables 3. Let zi = ( ~ i , y i be (1 5 i 5 n). Consider the U-statistic U, defined by the kernel h (zi,z j ) = 1 5 (Xi - Z j ) (Yi- Yj). (a) Verify (3.70). (b) If zi -i.i.d.N ( p ,C), show that u2 = V a r (p1 (21, p ) ) has a closed form expression defined by the elements of C , where pl (21, p ) is given by (3.75). 4. Consider a one-sample U-statistic U, with m arguments defined by a symmetric kernel h ( y l , . . . , ym). Show that then E (h?(yl)) can be expressed as the mean of a U-statistic in (3.82). 5. Let yki be two independent samples (1 5 i 5 n k ; 1 5 k 5 2). For the projection of the two-sample U-statistic defined in (3.83) of Section 3.2.3, show that the centered projection is given in (3.84).
3.3 EXERCISES
173
6. Prove Theorem 2 for two-sample U-statistics by extending the argument used in the proof of Theorem 1. 7. For a general K-sample U-statistic defined in (3.87) of Section 3.2.3, show (a) The centered projection is given in (3.88). (b) Under the assumptions of Theorem 3, the projection Gn has the asymptotic distribution:
(c) Show that 6,and U, have the same asymptotic distribution. 8. Consider the four-sample U-statistic defined by (3.89). (a) Verify (3.93). (b) Find the U-statistic for estimating the first term of each of the u& in (3.93).
This Page Intentionally Left Blank
Chapter 4
Models for Clustered Data This chapter introduces models for clustered data. Longitudinal studies are the most popular examples of such data, as outcomes from repeated assessments of an individual over time are not stochastically independent. Clustered data also arise from cross-sectional study designs in survey and epidemiological research, where subjects sampled from within a common habitat (cluster), such as families, schools and communities, are more similar than those sampled across different habitats, leading to correlated responses within a cluster. We touched upon the notion of data clustering in Chapter 2 when discussing the issue of overdispersion that arises from modeling count responses using standard Poisson-based log-linear models. Recall that a major problem with the Poisson-based log-linear model is that under the model assumption, the variance of the response equals the mean response. In practical studies, the variance is often larger than the mean, making this model inappropriate for such overdispersed data. A common cause of overdispersion is data clustering. In this chapter, we provide a systematic treatment of models for clustered data, especially those arising from longitudinal study designs, and discuss both parametric and distribution-free (semiparametric) approaches for modelling such data. In Section 4.1, we contrast longitudinal study data with clustered data from cross-sectional studies and highlight the often subtle, yet fundamental differences between the two types of data clustering. In Section 4.2, we describe parametric models for longitudinal data. Within the parametric modeling paradigm, the mixed-effects (or latent variable) modeling approach is the most popular for modeling such data. However, our discussion of parametric models, particularly mixed-effects models, will not be in-depth given the large number of excellent textbooks on this topic and the
175
176
CHAPTER 4 MODELS FOR CLUSTERED DATA
focus of this book on distribution-free and U-statistics based models. In Section 4.3, we discuss the mean response based distribution-free model or GEE I for longitudinal data analysis. One major drawback of the parametric models for longitudinal study data is the multiple layers of distribution assumptions that are often at odds with real study data, particularly those from research in the behavioral and social sciences, making inference prone to error. This popular distribution-free alternative GEE I addresses this fundamental problem with parametric modeling. In Section 4.4, we address missing data. Again, we will concentrate on distribution-free models given the focus of the book and the fact that inferences for mixed-effects models are robust to missing data mechanisms in most applications. In Section 4.5, we will discuss an extension of the mean based distribution-free model, known as GEE I1 for modeling variance parameters. As noted in Chapters 2 and 3, many important indices and models of interest in the behavioral and social sciences are defined by second order moments and multiple responses, which are beyond the scope of the mean based distribution-free models. GEE I1 can address some of the problems in a limited and ad hoc fashion. We will provide a systematic treatment of second and higher order moments modeling in Chapters 5 and 6 by extending the univariate U-statistics theory developed in Chapter 3 to a multivariate and longitudinal data setting. In Section 4.6, we discuss structural equations models (SEM), which are widely used in the behavioral and social sciences for modeling causal mechanisms. Although similar in appearance, linear SEM are fundamentally different from multiple regression models. Since there are good textbooks out there on the general theory and applications of linear SEM, we will not provide an in-depth treatment of the models. Rather, we will highlight the major distinctions and unique features of SEM and discuss the use of GEE I1 as an alternative, distribution-free inference procedure for SEM. Since linear SEM is essentially a class of models involving second order moments, it can be treated as a special case of theU-statistics based functional response models for high order moments in Chapter 6. Thus, we will take up S E N again in Chapter 6.
4.1
Longitudinal versus Cross-Sectional Designs
In statistics, cluster is a generic term used to describe groups of subjects and/or data that are more similar within each group than across groups. Longitudinal study data is the most common example of data clustering, although many cross-sectional study designs in epidemiologic and survey
4.1 LONGITUDINAL VERSUS CROSS-SECTIONAL DESIGNS
177
research and psychosocial interventions also produce such data. In a longitudinal study, each subject receives multiple assessments over time and data from such repeated assessments are correlated and clustered around the subject. Studies in the fields of epidemiology, behavioral and social sciences often sample subjects from a common habitat, yielding clustered responses. For example, in a cross-section study on academic performance with students sampled from multiple schools, those within the same school are likely to have their grades on some standardized test clustered together because of similar curriculums received. In psychosocial research, many psychotherapy and behavioral intervention studies also result in clustered data because of the unique mode of treatment delivery. In psychotherapy studies, multiple subjects receive treatment from a common therapist and the heterogeneity among therapists due to different training programs and level of experience makes subjects’ responses cluster around each therapist. In group based treatment studies, interventions are delivered in group formats with each group led by a trained instructor(s). The between-instructor heterogeneity again creates clustered responses. Although both types of study designs give rise to clustered responses, there are important differences between them as we highlight below. First, clustered responses arising in cross-sectional studies that sample multiple subjects from a common habitat such as school usually have a relatively simple correlation structure. For example, in most cases, any pair of responses within a cluster will have the same correlation. However, clustered responses from most longitudinal studies do not have such a symmetric correlation structure; correlations between two responses from a subject at different times are likely to depend on the length of time between the two assessments. Second, although varying cluster sizes are common to both study designs, they arise for quite different reasons and therefore must be differentially addressed. Varying cluster size in cross-sectional studies as in survey research and epidemiologic research is often the result of design consideration and generally has no impact on model estimation and inference. For example, in an epidemiologic study on estimating disease prevalence, varying cluster size is employed to ensure sufficiently large sample sizes to reliably estimate the disease prevalence in regions where the prevalence is relatively low. For longitudinal study data, however, different cluster sizes are the product of missing data, which must be carefully addressed in order to provide valid inference. Shown in Figure 4.1 are two possible scenarios with observed responses that may arise for a hypothetical longitudinal study with six subjects and
178
CHAPTER 4 MODELS FOR CLUSTERED DATA
1
T i m e of A s s e s s m e n t 2 3
Figure 4.1. Observed responses for a longitudinal study with six subjects and three assessment points with yit denoting ith subject’s response at time t and dots missing data.
three assessments. In scenario (a), we observed all three responses (yit) for each of the subjects, while in (b), each subject (cluster) had a varying number of observed responses due to missing data (dots). Missing data are commonplace for almost all longitudinal studies, no matter how well the study is planned, designed and executed. Thus, scenario (a) is atypical while pattern (b) is commonly encountered in data analysis. One approach is to perform analysis based only on the complete responses from the first three subjects. Alternatively, we may want to use all available data from the six subjects. It is obvious that an approach using only those subjects with complete responses is likely to be less powerful; what is not so obvious is the serious flaw in such an approach. We will take up these issues in Section 4.4. For now, it suffices to say that the presence of missing data in longitudinal studies makes the analysis of such study data more complex than the analysis of clustered data arising from cross-sectional study designs. Thus, from the model development perspective, we will focus on methods for longitudinal data analysis since they may be adapted to and/or modified for cross-sectional clustered data analysis.
4.2 PARAMETRIC MODELS
4.2
179
Parametric Models
As in the analysis of cross-sectional study data, parametric models have been and still are a primary analysis tool for modeling longitudinal data. For continuous outcomes, multivariate normal distribution-based models remain the most popular in applications, especially in the behavioral and social sciences. In particular, the multivariate analysis of variance (MANOVA) and multivariate linear models have played a significant role in shaping the landscape of quantitative analysis and methodological research in the behavioral and social sciences that we see today. Although the proliferation in recent years of new and cutting-edge methods for longitudinal and other multi-level clustered data designs, such as the mixed-effects and structural equations models, has addressed some fundamental limitations of such classic models and enabled researchers t o study complex individual, social and cultural processes that unfold over time, most of these methods have roots in the multivariate normal distribution based linear models. Thus, whether from a historic perspective or for teaching purposes, it is important t o understand the theory, applications and limitations of the classic multivariate linear models.
4.2.1 Multivariate Normal Distribution Based Models The multivariate normal distribution forms the basis for many parametric models that have been developed for longitudinal and other clustered data analysis. Although the classic MANOVA and multivariate linear models have been phased out and replaced by state-of-the-art approaches, such as mixed-effects and structural equations models, the theory of multivariate normal distribution still plays a significant role in developing new models in many research studies especially in the behavioral and social sciences. In this section, we introduce some of the normal-based popular models. These models will be revisited in later sections of this chapter, as well as in Chapters 5 and 6 as we relax the normal distribution assumption and address other limitations such as missing data. Example 1 ( M A N O VA/Repeated measures A N O VA). Consider a longitudinal study for comparing treatment differences with m assessment times. Suppose that there are g (> 1) different treatment conditions and we are interested in modeling differences in the mean response over time across the different treatment groups. At each assessment time t , the analysis of variance (ANOVA) model discussed in Chapter 2 can be used to compare the mean response across treatment conditions. However, in most longitudinal
180
C H A P T E R 4 MODELS FOR CLUSTERED DATA
studies, it is also of primary interest to see how differences in group means change over time. Inferences about such time trend by group interactions require joint modeling the group means across the different assessment times, which introduces data clustering among the repeated responses collected over time. Let ykit denote the response from the ith subject within the kth group at the tth assessment time for 1 5 i 5 n k , 1 5 k 5 g and 1 5 t 5 m. For each subject i within a treatment group k , ykit are correlated over different assessment times t since they are obtained from the same subject. Note that throughout the book, t = 1 will be referred to as baseline. For randomized clinical study trials, baseline typically designates the time when subjects are assessed before treatment initiation. Thus, interest usually lies in treatment differences at follow-up times such as t = 2, 3, and so on. Let
The classic multivariate analysis of variance (MANOVA) model is to assume that yki follow a multivariate normal distribution with mean Pk and variance E k , that is, Y k i N ( p k ,C k ) (1 5 k 5 9 ) . As in the case of ANOVA, treatment differences under MANOVA can be assessed by testing appropriate linear hypotheses of the form: N
Ho : Cp = a versus Ha : C p # a
)
T
(44
where p = p: . . . pz and C is some full rank p x (gm)matrix with known elements. In most applications, a = 0 and Ho in (4.2) is known as a linear contrast. For example, suppose that the study has three treatment conditions and three assessment times. Then, g = m = 3. If we are interested in testing the hypothesis of no mean difference across the groups at the two follow-up times t = 2, 3, we can express the null hypothesis Ho as a linear contrast as follows:
(
181
4.2 PARAMETRIC MODELS
with C given by
C=
0100-1
0 0 0
0
0 1 0 0 0
0 0-1
0
0 0 1 0 0 -10 0 0 1 0 0
0
0
0 0 0 -1
A fundamental factor that sets research in behavioral and social sciences apart from their biomedical counterpart is outcome assessment. Many mental health disorders, such as depression and anxiety, are latent constructs and thereby not directly observable. Outcomes for measuring such disorders in psychosocial research are generally derived based on instruments (questionnaires) that delve into multifaceted personal feelings and private life rather than unidimensional, quantitative changes, such as blood pressure, heart rate and CD4 counts in drug treatment studies. For example, the popular Hamilton rating scale for depression severity consists of a total of 24 items to assess a wide range of emotional and physical problems, such as guilty feelings, suicide ideation and insomnia. Most items are rated either by a binary response to indicate the presence or absence of some psychological and physical phenomena and symptoms or by a Likert scale with integer values such as 0,1,2, etc. to quantify the extent of suffering, such as from “Not at all” to “A little”, “A moderate amount”, “Very much”, and “An extreme amount”. Item scores are then totaled to provide an assessment of the underlying latent construct. Note that items in some instruments may be grouped t o form measures of different domains of the construct, such as suicide ideation and insomnia, as in the case of Hamilton rating scale for depression severity. Item scores may be totaled within each domain to provide domain-specific assessment. Also, when an instrument (domain) contains both positive and negative items, one of the two types of the items must be reverse scored before the total scores within the instrument (domain) are computed. For example, in the 26-item version of the instrument for quality of life assessment, WHOQOL-BRIEF, developed by the World Health Organization (WHO), two of the questions in the domain of physical health are as follows: 1. How m u c h do y o u need a n y medical treatment to f u n c t i o n an y o u r daily life ? 2. Do y o u have enough energy for everyday life ? Both are scored by a five-item Likert scale: 1 for “Not at all”, 2 for “A little”, 3 for “A moderate amount”, 4 for “Very much” and 5 for “An
182
CHAPTER 4 MODELS FOR CLUSTERED DATA
extreme amount”. The first is a negative, while the second is a positive item. If the negative item is reverse scored, the response Q1 for the first question will be rescored as 6-Q1 when totaling the domain score for each respondent. For the total score of an instrument (domain) to be a meaningful dimensional measure, the item scores within the instrument (domain) must be positively correlated. A common index for assessing such internal consistency is the Cronbach coefficient alpha. Example 2 ( Cronbach coeficient alpha). Consider an instrument or a subscale of an instrument with m number of individual items, which is administered to a group of n subjects. Let yik denote the response from the ith subject to the kth item of the instrument. Let
where [ o k ~ denotes ] the rn x m variance matrix of yi with crkl denoting the klth element. Let pkl denote the correlation between the kth and Zth item scores. Cronbach’s coefficient Q is defined as:
-z(
C k f l *k*lPkl
-
m-1
cr=1*;+ C k j l
Ok*lPkl
where pkl denote the between-item score correlations. Thus, Cronbach’s alpha compares the sum of the covariances to that of the total elements of the variance matrix C. In the theory of measurement for item-based instrumentation, the true score for measuring some latent construct of interest is assumed to be the limit of the average of item scores with items randomly selected from an infinite pool of possible items (Nunnally and Bernstein, 1994). Thus, measurement error tends to zero as the number of items grows unbounded. The between-item correlations pkl indicate the extend to which a common core exists among the different items. If all such between-item score correlations are zero, then Q: = 0. In this extreme case, the items are totally unrelated and share nothing in common, and thus should not be used together t o measure the latent construct. At the other end of the spectrum, if all the item scores are perfectly correlated with a common variance, then Q = 1. In this special case, all the items are identical and the construct is not really
183
4.2 PARAMETRIC MODELS
latent as it can be measured by a single item without error. Of course, these extreme cases are hypothetical scenarios. In practice, 0 < a! < 1 and a! 1 as m + m provided that pkl > 0 for all (1 5 k , l 5 m). Thus, Cronbach alpha is an index of quality of measurement for an instrument with Ic addable items. In fact, it can be shown that it is an estimate of the squared product-moment correlation between the true score and the averaged item scores of the instrument (Nunnally and Bernstein, 1994). Although the index a above is well defined regardless of the data distribution, yi is typically assumed to follow a multivariate normal with mean p and variance C in most applications because of the difficulty in making inference about Q without such a parametric distribution assumption. However, item scores in most instruments used in the behavioral and social sciences are defined by a binary or Likert scale and as such the normal distribution assumption is fundamentally at odds with the data distribution of item scores for most instruments. We will discuss nonparametric inference about Q by removing the normal assumption in Chapter 5 . In many longitudinal studies, regression analysis is often required to evaluate treatment effects even though such studies only involve comparisons of treatment conditions. For example, it is often necessary to control for baseline covariates such as age and gender. When including covariates that contain continuous variables, we must use regression models for comparing treatment conditions. Regression models are also necessary for modeling relationship between a continuous response and a set of independent variables. Example 3 (Multivariate linear regression). Consider a longitudinal study with n subjects assessed at m different time points. Let yit denote the response and xit the vector of independent variables from the i t h subject at assessment time t . Let --f
Yi = ( Y i l , .
9
.
1
yz,)
T
, xi = (x;, . . . ,Xi,)T
'
,
xi = (X i l , . .
Then, we model the linear relation between yit and
yit = x;p ~i =
+
€o , t,
( ~ 1 ,. . , ~
or
i , )N ~
yi = X @
xit
,X i , )
T
as follows:
+ ~i
(4.5)
i.i.d.N (0, C)
where €it (q)denotes the model error. Note that if different Pt are used for different times, the model is known as the seemingly unrelated regression. In many controlled, randomized longitudinal trials, we are only interested in treatment difference at posttreatment. In this case, we can apply
184
CHAPTER 4 MODELS FOR CLUSTERED DATA
ANOVA discussed in Chapter 2 for comparing different treatment conditions or ANCOVA if we want to control for covariates at baseline. These methods yield valid inference in the absence of missing data. However, when there is missing data, both generally gives rise to biased estimates when implemented within the distribution-free inference framework discussed in Chapter 2. We address this missing data issue in Section 4.4. Although multivariate normality based approaches are widely used in biomedical and psychosocial research, many studies have indicated problems with using estimates derived from such models because of the strong distribution assumption imposed on the response. As noted earlier, most instruments used in psychosocial research are based on item scores, and so are intrinsically discrete. Thus, the treatment of such variables as though they were continuous is purely for analytic simplicity. For variables with a relatively large range, such as the total Hamilton rating scale for depression severity, such a treatment is convenient and sensible. However, because these variables are inherently discrete, normal-based parametric models are fundamentally flawed when used for such outcomes. Thus, whenever possible, distribution-free alternatives should be considered.
4.2.2
Linear Mixed-Effects Model
The two major limitations of the classic multivariate normal distribution based approach are (1) its limited ability to deal with missing data and (2) its requirement of a set of common assessment times for all subjects. The mixed-effects (or latent variable) modeling approach and distribution-free models provide an effective solution to both issues. We start with the linear mixed-effects model for continuous response, which is a direct extension of the classic multivariate linear models. The linear mixed-effects model (LMM) is a general class of models widely used in biomedical and psychosocial research to model the linear relationship between a response and a set of independent variables within a longitudinal and clustered data setting. As data clustering arises in research studies employing longitudinal study designs and multilevel sampling strategies (e.g., sampling subjects in classes nested within schools) across a wide range of disciplines, various applications of this general class of models are found under different guises, such as random coefficient models, random regression, hierarchical linear models (HLM), latent variable models, mixed models and multilevel linear models (Goldstein, 1987; Raudenbush and Bryk, 2002; Laird and Ware, 1982). The linear mixed-effects model employs random effects or latent variables to account for within-cluster correlations, rather
185
4.2 PARAMETRIC MODELS
than directly correlating the clustered responses as in the classic multivariate linear model. As a result, this approach enables one to address the difficulty in modeling correlated responses arising from varying assessment times as in some longitudinal cohort studies. Example 4 (Linear growth curve model with constant assessment times). Consider a longitudinal study with n subjects. Assume first a set of fixed assessment times for all subjects, denoted by t = 1, 2, . . . , m. If yit is a linear function of time t , then the classic linear model has the form: git = PO
+ P l t + €it,
l l i l n ,
€it = (cil,. .
9
,€im)-
N
i.i.d.N ( 0 ,C) ,
(4.6)
l
In the above, C is the variance of yi = ( y i l , . . . ,yim)-' , which contains both the between- and within-subject variation. The idea behind the linear mixed-effects model is to break up the two sources of variation by modeling each separately. For each subject i, let bio and bil denote the intercept and slope of the response yit and let bi = (bio, bil)-. Then, for each subject i, we model the within-subject variation as follows: Yit
I Y i = Po + Plt + bio + bilt + €it
fit
N
(4.7)
i.i.d. N ( 0 , ~, ~ 1) 5 t 5 m
In other words, for the ith individual, the response yit is modeled as a linear function of time with intercept, Po bi0, and slope, P1 bil. Thus, by modifying the population mean ,kl = (Po,Pl)T using the individual specific bi to account for between-subject differences in the linear predictor, the error terms ~ i lEQ,. , . . , eim, in (4.7) can be assumed to be i.i.d. This is in stark contrast to the assumption of the error term for the classic multivariate linear regression in (4.6) in which the model error €it are correlated over t to account for individual differences. By letting bi vary across the subjects, we obtain a linear mixed-effects model that accounts for both between- and within-subject variations. In many applications, bi are assumed to follow a multivariate normal N (0,D). By combining the two-level specifications, we can express this linear mixed-effects model in a hierarchical form:
+
it = P o
+ tP1 + bio + hilt +
yi = XiP + Zibi + ~i €it
N
i.i.d.N (0,02) ,
+
= x;,kl
+ z i b i + €it
bi = N (bio, bil)-
N
i.i.d. N ( 0 , D )
(4.8)
186
CHAPTER 4 MODELS FOR CLUSTERED DATA
where xit = zit = (1,t ) T and Xi = Zi = (xil,xi2,.. . , ~ i ~ The ) ~ linear . predictor of the LMM in (4.8) has two parts; the first part xLP describes the variation of the population mean over time, while the second zLbi models the deviation of each individual subject from the population mean. Since bi is random, bi (or zzbi) is called the r a n d o m eflect. The population vector of parameters ,B (or x:P) is called the fixed eflect, and hence the name of the linear mixed-effects model. An alternative perspective is to view bi as a latent quantity since it is unobservable, which gives rise to the term of latent variable model. The covariance between the sth and tth assessment points for the classic (4.6) and linear mixed-effects (4.7) models are given by
If assessment times vary across individuals, cist will depend on i and become inestimable except in special cases with some particular covariance structures. For example, if the covariance between pis and yit follows the uniform compound symmetry assumption, ust = u and are estimable. However, this issue does not arise for the linear mixed-effects model since D and u2 are well-defined regardless of the structure of assessment. Unlike the classic multivariate linear model, the variance parameters for the random effect in LMM has an interesting and useful interpretation. The diagonals of D measure the variability in individual intercepts and slopes among different subjects in the study population. Thus, in addition to P, inference about D is also often of interest to assess the variability of individual intercepts or slopes or both. Example 5 (Linear growth curve model with varying assessment t i m e s ) . In Example 4, suppose that each subject has a varying number as well as times of assessments. For each subject i, let mi denote the number of assessments and t i j the assessment times (1 5 j 5 mi).Then, the LMM in this case can be expressed as:
€it N
i.i.d. N (0, .")
, bi = (bio, b
i ~ ) i.i.d.N ~ (0,D ) N
In comparison to (4.8), yi (Xi) above is a mi x 1 (mi x 2) vector (matrix) of varying dimension, rather than a fixed m x 1 ( m x 2) vector (matrix) as in the case with fixed assessment times.
187
4.2 PARAMETRIC MODELS
A general LMM has the following form: T
~ z t , ,= X,t,,P
b,
N
T + ztt,,bz + Ezt,,,
i.i.d.N (0, D ) ,
E,
N
~z
= XzP
+ Zzbz +
i.i.d.N (0, .’I,,)
,
15
where x i p is the fixed and zLb, the random effect,
bzJ-ez
€2,
_L
J
(4.10)
5 m,
denotes stochastic
T
independence and W, = w,tZl,. . . , w,t,_,) (W = X or 2 ) . For growthcurve analysis (modeling change of yzt over time as in longitudinal study), z,t is often equal to x,t. It follows from the assumptions of the LMM that:
(
(Yz I X , , 2 2 ) = XzP,
V a r (Yz
1 x,, 2%) = Z,DZ,T + 021m,
(4.11)
Clustered data also often arise from cross-sectional studies. For example, psychotherapy or behavioral intervention programs in the behavioral and social sciences are often delivered in a group format. Because therapists may differ in their skill set or ability to form a therapeutic bond and alliance with their patients, which are important considerations for intervention effect, there are often real differences between therapists. Such variability leads to different treatment outcomes in patients who are treated by different therapists and should be accounted for in assessing treatment effect in such studies. Example 6 ( L M M for controlling f o r therapists’ eflect). Let n denote the number of therapists and mi the number of patients seen by the ith therapist in a cross-sectional study comparing two treatment conditions (1 5 i 5 n ) . By treating n as the number of clusters and mi as the size of each cluster, patient’s response, yij, can be modeled using the following LMM: yij = Po ~ i jN
+ zipl + bi + ~ i j ,
i.i.d.N (0,o’) ,
bi
N
i.i.d.N (0,o;)
1 5 i 5 n,
(4.12)
1 5 j 5 mi
where the binary z i indicates the treatment conditions. In the above, bi accounts for the individual effect of the ith therapist with o; measuring the variability across therapists. Thus, concerns about therapists’ variability can be examined by testing the null hypothesis: Ho : oi = 0. Note t.hat in T this example, xit = (1,xi) # zit = 1. Example 7. For the LMM in (4.12) of Example 6, we have: (4.13)
188
CHAPTER 4 MODELS FOR CLUSTERED DATA
where C, ( p ) denotes the uniform compound symmetry correlation matrix of dimension m with a constant correlation p = Corr (yij, yih). The intru-
A,
class correlation (ICC), p = measures the proportion of therapists’ variability in the total variability of patients’ outcomes. If all therapists are the same, then CT; = 0 and p = 0. In this case, yij become uncorrelated across j within each i and there is no cluster in the data. At the other extreme, if g; = m, that is, all therapists are really different, then p = 1. In this case, yit only reflects the large variation in the therapists. In this example, p has a more interesting interpretation than 0;. Example 8 ( L M M for site diflerence i n multisite t r i a l ) . Consider a randomized, multicenter study trial on comparing two treatment conditions. For simplicity, we only consider modeling treatment difference at one time point only, such as the posttreatment assessment. Let yij denote some response of interest from the j t h subject within the ith site. Then, an appropriate LMM is given by Yij =
Po + ~ i j P 1+ bi + ~ i j , bi
N
i.i.d.N (0,o;),
N
i.i.d.N ( 0 , g ’ )
where ~ i indicates j the treatment received by subject j at site i and bi denotes the (random) effect of site. We can assess whether there is significant site effect by testing the null: HO : 0; = 0. If this null is not rejected, we can simplify the model by dropping bi. When the number of sites is small such as two or three, we may want to model potential site difference using a fixed effect. In this case, site difference is interpreted with respect to the particular sites in the study. If there is a large number of randomly selected sites, it is sensible to model site difference using random effects. A significant random effect implies differences not only among the participating sites, but also across similar sites not included in the study. This difference in interpretation is usually not important in applications. In Section 4.2.1, we discussed a popular index, Cronbach coefficient alpha, for measuring interitem consistency. Another major concern with respect to quality of outcome in psychosocial research is the intem-uter or between-observer agreement when an instrument is rated by different evaluators to assess mental disorders, disease conditions and behavior patterns in a person. Such agreement is also known as test-retest reliability when such an instrument is administered repeatedly over time. Different approaches are employed for assessing interrater consistency, depending on whether the outcome is continuous or discrete (ordinal or categorical) or treated as such. The product-moment and intraclass cor-
4 . 2 PARAMETRIC MODELS
189
relations (ICC) are the most widely used measures of interrater agreement and test-retest reliability for continuous outcomes. For two raters, the product-moment correlation is used to indicate agreement, while for more than two judges, the ICC measures overall agreement by averaging all such pair-wise correlations among the observers. Consider a sample of n subjects, each of which is assessed by M independent observers. Let yi, denote the rating from the mth judge on the ith subject and yi = (yil, . . . , y i ~T ). Let p = E (yi) and C = V a r (yi) and N ( p ,C = [ a k a l p k l ] ) . Then, p k l = Corr (yik, yil), is the assume that yi product-moment correlation between judges k and 1 (1 5 k < 1 5 M ) . The product-moment based ICC as an index for overall agreement is defined as: N
As we discuss shortly, this product-moment correlation based ICC has a serious drawback and may provide misleading information in some applications. Example 9 ( L M M for ICC with no judges’ bias). A popular alternative t o the product-moment correlation based ICC above is the modelbased ICC. This approach is more appealing since it frames the problem under the more familiar regression framework. Consider the following LMM:
where the fixed effect ,LL denotes the mean rating over the raters, the random effect X i explains individual differences across the subjects, and the model error ei, accounts for variability among the judges. T Under (4.15), yi = (yil, . . . , y i ~ ) has a multivariate normal with mean and variance:
where 1, is an m x 1 column vector of 1’s and J, an m x m matrix of 1’s. It then follows that the product-moment correlation between any two raters is, P L M M I C C = Corr (yi,, yil) = -&, a constant across the raters. This model-based between-rater correlation ~ L n l M I c cis often used as the definition of intraclass correlation as it elegantly explicates the effect of the interrater variability on the variability of the subject’s ratings. If g2 is
190
CHAPTER 4 MODELS FOR CLUSTERED DATA
small (larger) relative to a:, P L ~ ~ ~ ~will I Cbe C close to 1 (0), indicating good (poor) interrater agreement. It follows from (4.16) that the LMM in (4.15) implies a common mean rating E(yi,) = p across the M observers. When there is consistent difference or bias between judges’ ratings, this LMM no longer applies. By allowing judges’ mean ratings to vary, we can readily revise the LMM to address such judges’ bias. Example 10 ( L M M for ICC with judges’ bias). In Example 9, consider replacing the constant mean p by the individual judge’s mean rating p m in (4.15). The resulting LMM is defined by yim = pm eirn
N
+ X i + ~i,,
Xi
N
i.i.d.N (0,a:) ,
i.i.d.N ( 0 , ~, ~ 1 ) 5 i 5 n,
(4.17)
Xil~i,
15 m 5 M
Under this revised M-fixed factor LILIM, the mean rating, p, = E (yzm), is allowed to differ across the judges. However, the correlation between any two raters is still a constant, P L M M I C C = Cow (yi,, yzl) = Thus, although the above model extends the single-factor LMM in (4.15), it still assumes constant variance and covariance between the judges’ ratings, a structure not imposed by the product-moment correlation based model in (4.14). When there is judges’ bias, both versions of ICC in (4.14) and (4.17) may yield incorrect information about interrater agreement. Consider a hypothetical study with six subjects and suppose that ratings from two judges are 3, 4, 5, 6, 7 , 8 and 5, 6, 7, 8, 9, 10, respectively. Both the productmoment and multifactor LMM based ICC is 1, indicating perfect agreement between the judges. However, there clearly exists judges’ difference in rating the subjects and neither version of ICC accounts for such a difference. We will discuss another agreement index in Chapter 5 to address the limitation of ICC. Note that an application of the single-factor LMM in Example 6 to the data yields an ICC close to 0.56. Although this low ICC provides a more reasonable conclusion about the discrepancy between the two judges’ ratings in this example, it does not provide the right interpretation of the poor agreement between the two raters. Since this single-factor model assumes a common mean rating p, it too ignores judges’ bias. The low ICC does not indicate large random variability between the raters as it is designed to measure, but rather the fixed bias in the judges’ ratings.
$&.
191
4.2 PARAMETRIC MODELS
4.2.3
Generalized Linear Mixed-Effects Models
A major limitation of the linear mixed-effects model is that it only applies to continuous response. By extending the generalized linear models (GLM) for cross-sectional study data considered in Chapter 2 to a longitudinal and clustered-data setting using random effects, we can develop similar mixedeffects models for other types of response such as binary and categorical outcomes. Consider a longitudinal data study with n subjects and m assessments. For notational brevity, we assume a set of fixed assessment times, 1 5 t 5 m. Let yzt denote the response and xit a vector of covariates of interest from the ith subject at time t (1 5 i 5 n,1 5 t 5 m). The principle of extending GLM to a longitudinal data setting is the same as in generalizing the classic univariate linear regression to LMM. For each subject, we first model the within-subject variability using a generalized linear model: yit
I X i t , Z i t , , bi
-
id.
f ( p i t ),
g ( p i t ) = xLP
+ z&bi
(4.18)
where g (.) is a link function, zit is a subvector of xit, bi denotes the random effects, and f ( p ) some probability distribution with mean p. Note that the model specification in (4.18) is quite similar to the cross-sectional GLM except for the additional individual effect bi. Then, by adding a distribution for bi to explain the between-subject variation, we obtain from (4.18) the class of generalized linear mised-eSfects model (GLMM). As in the case of LMM, bi is often assumed to follow a multivariate normal bi N (0,D ) (a common assumption in most studies), although other types of more complex distributions such mixtures of normals may also be specified. Example 11 ( G L M M f o r binary response). In (4.18), if yit is a binary response, then f ( p ) is a binomial distribution with mean p and sample size 1, BI(p; 1) (1 5 i 5 n,1 5 t 5 m). The most popular link for modeling such a response is the logit function. If we model the trajectory of yit over time as a linear function of t with a bivariate normal random effect for the mean and slope, the GLMM becomes: N
yit
-
i.d. BI ( p i t ;1) ,
As in the LMM case, Po + P t describes the change over time for the population average, while the random effect bio deviations.
+ hilt accounts for individual
192
CHAPTER 4 MODELS FOR CLUSTERED DATA
Example 12 (Poisson log-linear model). In Example 11, let yzt be a count response. By assuming a Poisson distribution f(p), we obtain a random-effects based log-linear model for the trajectory of a count response yit over time: yit
N
i d . Poisson ( p i t ),
log ( p i t ) = Po
+ P t + bio + hilt
(4.19)
bi=(bio,bii)TNi.i.d.N(O,D), l < i < n , l < t < m
The fixed and random effects have the same interpretation as in Example 11. In Chapter 2, we discussed some major limitations of the Poisson based log-linear model when data exhibit overdispersion and discussed alternative models for addressing the underlying problems causing such overdispersion, such as the negative binomial and zero-inflated Poisson regression models. Overdispersion can also occur in longitudinal data analysis and thus it is of interest to generalize such models to a longitudinal data setting. Example 13 (Negative binomial based log-linear model). Under the Poisson log-linear model in Example 9, the mean and variance of yzt conditional on the random effect b, are identical. In many applications, the variance is often larger than the mean, a phenomenon known as overdispersion. By substituting a negative binomial distribution, N B ( p ,a ) in place of the Poisson model in (4.19), we obtain a GLMM that addresses over dispersion. Note that the negative binomial distribution has an extra parameter a that describes the degree of overdispersion in the conditional variance of yzt given b,: V a r (yzt 1 b,) = ptt (1 a p z t ) ,which is larger than the mean pzt by an amount of ap:. This extra term accounts for overdispersion in modeling yzt. Overdispersion is not the only problem with count response. As discussed in Chapter 2, count response arising in biomedical and psychosocial research often contains structural zeros. Such structural zeros not only cause overdispersion in the variance, but also affect the mean of the response. For example, under the Poisson or negative binomial model, the presence of positive structural zeros (excess amount of zeros) will pull the mean response down towards zero, biasing model estimates. Thus, even estimates for descriptive analyses, such as the mean response, are incorrect. The zero-inflated Poisson (ZIP) model considered in Chapter 2 can be extended t o address structural zeros within a longitudinal data setting. Example 14 (Zero-inflated Poisson model). Within the context
+
193
4.2 PARAMETRIC MODELS
of Example 13, consider the following mixed-effects ZIP:
where ZIP(p,p ) denotes the mixture distribution for ZIP with parameters p and p, X k i t (.kit) denotes the covariates for the fixed (random) effect for modeling the occurrence of structural zero and the Poisson mean, respectively. In addition to binary and count response, we can similarly generalize the cross-sectional generalized logit and proportional odds models for multilevel nominal and ordinal responses discussed in Chapter 2 to a longitudinal data setting (see exercise).
4.2.4
Maximum Likelihood Inference
Maximum likelihood is the most popular inference procedure for the classic multivariate linear model as well as the modern GLMM. Given our focus on distribution-free models and the large number of references on multivariate linear models and GLMM, we only provide a brief review and highlight the main steps in performing such inference. We begin with the classic multivariate linear model. The multivariate linear models considered in Section 4.2.1 have the following general form: Yi = Pi ~i =
+ ei,
(GI,.. . ,
(4.21)
Y i = (Yil,* . . , Y i m )T ~
i
-
~ i.i.d.N(O, ) ~ C),
1
where pi = p , P k or X i p depending on whether we have a homogeneous onesample, MANOVA or multivariate linear regression model. Note that in the MANOVA case, yi are expressed in the form of y k i with p k = E ( y k i ) , where Y k i denotes the response vector from the ith subject within the kth group (1 5 i 5 n k , 1 5 k 5 9 ) . We discuss maximum likelihood inference for the general linear model defined by (4.21) with p i = X @ . The general theory is readily adapted and applied to other specific models such as MANOVA. The likelihood and log-likelihood functions for the multivariate linear
194
C H A P T E R 4 MODELS F O R C L U S T E R E D DATA
regression model with p i = Xip are given by n,
where the constant c = --?log (27r) is dropped from the log-likelihood since it is irrelevant for estimation of p and C. By taking the first and second order derivatives of the log-likelihood l, we obtain the score vector equation for the parameters of interest p and C. The classic multivariate linear model is among the few classes of multivariate models for which the maximum likelihood estimate (MLE) can be expressed in closed form. By taking derivatives of the log-likelihood 1 with respect to and C, we obtain the score vector (see exercise):
a
n
n
(C-lei) = i=l
i=l
(--apd ( X @ ) )
(C-lei)
(4.23)
n
=
Cx$-'
(yi - xip)
i= 1
is an rn x m matrix. By setting the score vector where a = [&1,] above to 0 and solving for p and C, we obtain (4.24)
2 = -l
c n
(yz
-
xiq(yi - x,qT
i=l
The above solution can be viewed as maximizing the log-likelihood by conditioning on either p or C, that is, given C, is the MLE of p, while given p, 2 is the MLE of C. Such a conditional maximization also converges, albeit at a slower rate than the Newton-Raphson algorithm. However, the benefit of using this alternative algorithm is that it allows us to derive closed-form expressions of the MLE of p and C. In most regression analyses primary
195
4.2 PARAMETRIC MODELS
interest lies in inference about the parameters p in the mean response. In this case, we are only interested in the MLE of p. However, many quantities of interest in the behavioral and social sciences are also defined by second order moments in which case it is also of interest to make inference about C. To find the MLE of ,tJ and C in (4.7), we can also apply the NewtonRaphson algorithm directly t o find the MLE and its asymptotic variance. Let 8 = (pT,cecT (C))T with wec(C) denoting the m(m-1) x 1 column vector containing the unique elements of C, that is,
vec (C) = ( g i l l ,ai12,. . . , ailm,ai22, Starting with some initial value gence to obtain the MLE of 8:
. . , a i 2 m , . . . , aimm)T
do), we iterate the following until
conver-
The asymptotic variance is calculated and estimated the same way as before. Now, consider inference for the LMM defined in (4.10). Let 8 = (p', vecT ( D ), c2)T . The log-likelihood is given by
where fy,z,z,b (y, 1 X , , Z,, b,) denotes the density function of a multivariate normal N ( p , ,C,) with mean X , p Z,b, and variance C, = Z,DZT a21m, and f b ( b , ) the density of the normal random effect b,. The integral in (4.25) is the result of integrating out the latent b,. One major technical problem with inference for mixed-effect models is how t o deal with such an integral. Fortunately for LMM, under the normal distribution assumptions, this integral can be completed in closed form. In fact, the marginal of y, is again normal: N (X,p,V, = (2,DZ; + a&,)). By this property, the log-likelihood function is given by
+
+
n
n
1
where N = C7=lmi. Note that although the dimension of D is fixed, V , and ei = yi - Xip both have a varying dimension from subject t o subject. The MLE of 8 is obtained by maximizing the log-likelihood I , (8).
196
CHAPTER 4 MODELS FOR CLUSTERED DATA
Although straightforward in principle, it is actually quite difficult to maximize 1, ( 8 ) in (4.26), since obtaining the derivatives of Z(8) in closed form is quite a daunting task. Because of the analytic complexity in calculating the derivatives, several algorithms have been proposed and used. Among them, a popular approach is to use the expectation/maximization (EM) algorithm and its various enhanced versions, such as the expectation/conditional maximization either (ECME) algorithm. These approaches aim to obtain the maximum likelihood estimate (MLE) by maximizing the alternative, expected rather than the observed-data log-likelihood. Because expected log-likelihood functions can often be evaluated in closed form, applications of such algorithms bypass the difficulty of evaluating high dimensional integrals when maximizing the observed-data log-likelihood. However. it is well known that the convergence of EM is extremely slow in many applications, while the enhanced versions such as ECME are often difficult to implement, with unknown convergence rates. Further, these algorithms only produce the estimate of 8 and require additional methods to compute its asymptotic variance, the latter of which is often quite complex and a problem in its own right. Thus, while using EM-type algorithms for a particular application at hand may be convenient and effective, such an approach is not practical for developing models for routine use and wide applications. Most software packages use the Newton-Raphson algorithm (Lindstrom and Bates, 1988; Wolfinger, 1993; Demidenko, 2004). By deriving the first and second order derivatives of the log-likelihood, the MLE 6 of 8 is obtained by applying the Newton-Raphson method. Inference for GLMM faces similar difficulties. In particular, it is generally not possible to obtain analytic derivatives. Even with the most popular normal random effects, integration cannot be completed in closed form for most models. The approach implemented in SAS by the NLMIXED procedure utilizes advanced Gaussian quadrature methods for high dimensional integration to maximize the observed data log-likelihood and can produce estimates as close to MLE as required by improving precision in the numerical integration (SAS Institute Inc. 2006). Thus, estimates so obtained can be treated as MLE for any practical purposes with well known asymptotic properties. The SAS NLMIXED procedure not only provides default routines for fitting some popular GLMM models, such as linear mixed-effects models, but also offers a flexible and powerful programming environment for developing inference procedures for very general random-effects models, such as those with nonlinear random effects.
Example 15. Consider the GLMM for binary response in Example 8.
197
4.2 PARAMETRIC MODELS
Let a = uecT ( D ) and 8 = is given by
(PT,a T )T .
Since bi is latent, the log-likelihood
m
where fylz,z.b
(yi I bi, P ) =
pyit (1 -
is the conditional joint den-
i=l
sity of m independent binary components of yi = (yil, . . . , y i m )T given the random effect bi with mean pit = exp (Po P t bio hilt) and fb (bi I a ) is the joint density of the binary normal bi = ( b i o , b i ~ ) ~The . MLE of 8 is defined by the solution to the score vector equation &ln (8)= 0. We can use the NLMIXED procedure in SAS to numerically obtain the MLE of 8 and the observed information matrix. As in the univariate case, hypotheses concerning 8 for most applications can be expressed in the following form:
+ + +
HO : C8 = b versus Ha : C8 # b
(4.27)
where C is some known full rank k x p matrix with p (2Ic) denoting the dimension of 8. If b = 0, HO becomes a linear contrast. As discussed in Chapter 2, both the Wald and likelihood ratio tests can be used to examine the general linear hypothesis in (4.25). If 8 only consists of parameters for the fixed-effect, that is, 8 = P, we can reexpress the linear hypothesis in terms of a linear contrast by performing the transformation: y = ,8 CT (CTC)-'b. When expressed in the new parameter vector y , the linear predictor will contain an offset term. For example, the linear predictor for the GLMM in (4.18) under this transformation becomes
where cit = x; is the offset. Except for LMM, where cit can be absorbed into the response by redefining the dependent variable as zit = yit -tit, the offset must be specified when fitting GLMM using software packages.
198
C H A P T E R 4 MODELS F O R CLUSTERED DATA
4.3 Distribution-Free Models The GLMM is developed by introducing random effects to model the withinsubject variation. A fundamental problem with using random effects to account for correlated responses is the difficulty in empirically validating the distribution assumption for the random effects since they are latent variables. To further exacerbate this problem, GLMM also relies on parametric assumptions for the distribution of the response for inference. If either of these assumptions is violated, estimates will be biased and inconsistent. Unfortunately, such assumptions are often taken for granted and unacknowledged, laying the basis for inconsistent and spurious findings. A remarkable breakthrough underlying cutting-edge distribution-free modeling is the elimination of both sets of assumptions, leading to estimates that are robust to data distributions and within-cluster correlation structures.
4.3.1 Distribution-Free Models for Longitudinal Data Since the seminal work of Liang and Zeger (1986), distribution-free regression models have been widely used as a robust alternative to mixed-effects models. In comparison to LMM and GLMM, such models offer more robust estimates and inference. By modeling the marginal mean of the response at each assessment time, this approach eliminates both layers of assumptions and thereby provides consistent estimates regardless of the complexity of the correlation structure and the distribution of the response. Unfortunately, many applications of GLMM in studies in the behavioral and social sciences fail to acknowledge the impact of distribution assumptions and the issue of robustness. Outcome data from most instruments in these studies are especially prone to violations of distribution assumption, because they are not even continuous let alone do they follow the normal distribution. In Chapter 2, we discussed distribution-free models for nonclustered data and introduced estimating equations for inference. Such cross-sectional models are readily extended to a longitudinal data setting. Consider a longitudinal study with n subjects and m assessment times. Again, for notational brevity, we assume a set of fixed assessment times 1 5 t I: m. Let yit denote a response and xit a set of independent variables of interest from the ith subject and at time t . By applying the distribution-free GLM discussed in Chapter 2 to each time t , we obtain a class of distributionfree regression models for the current context of longitudinal data:
4.3 DISTRIBUTION-FREE MODELS
199
where g (.) is some link function and p is a p x 1 vector of parameters of interest. Note that at each time t , the distribution-free GLM above is exactly the same as the distribution-free GLM for cross-sectional data. Example 1 ( M A N OVA/repeated measures A N O V A ) . Consider a longitudinal study for comparing g treatment groups, with each group involving n k number of subjects (1 5 k 5 9 ) . Let Y k i t denote the continuous response of interest from the ith subject within the kth group at the tth assessment (1 5 i 5 n k , 1 5 k 5 g, 1 5 t 5 m). The distribution-free MANOVA/repeated measures ANOVA models the mean of each group as follows: (4.29) In comparison to the normal based MANOVA/repeated measures ANOVA in Section (4.2.1), the distribution-free version only specifies the mean response of each group at each time t with no assumption regarding the joint T distribution of the within-subject response vector yki = (ykil, . . . ,ykim) . As a result, inference is valid regardless of how Yki are distributed. Example 2 (Group comparison f o r binary response over t i m e ) . In Example 1, let Ykit be a binary response. In this case, the mean response Pkt = E ( Y k i t ) represents the probability of the occurrence of the event ykit = 1 for each group over time t. Example 3. In Example 1, if ykit is a count response, then the model in (4.28) can be used t o compare the mean frequency of the response across the different groups. In parametric analysis, gkit is assumed to follow an analytic model, such as the Poisson. In comparison, the distribution-free model only specifies the mean of the response ykit. Thus, the distribution-free model provides valid inference even if ykit does not follow a Poisson distribution. For example, in Section 4.2.3 we discussed the negative binomial model to account for overdispersion. The negative binomial distribution has the same mean as the Poisson, but differs from Poisson in that it has a larger variance. Since the distribution-free model only involves the mean response, it yields valid inference regardless of whether ykit follows the Poisson, negative binomial or any other distribution. Example 4 (Linear model f o r growth-curve analysis). Consider a growth curve analysis for a longitudinal study with n subjects and m assessments. Assume a linear growth with one baseline covariate IC. Let xit = (1,zi,t , z,t)T. Then, (4.28) yields
200
CHAPTER 4 MODELS FOR CLUSTERED DATA
If z is a binary indicator for two treatment conditions, the above models the mean response over time with pz indicating the growth rate for one treatment group and &+p3 the growth rate for the other group. Treatment difference is indicated by a significant group by time interaction, p3 # 0. If z represents some demographic variable, such as gender and age, the above models the growth curve of a single treatment condition with the group by time interaction denoting the moderating effect of this covariate; z moderates the growth rate if p3 # 0 and has no effect if otherwise. Note that yit in (4.28) can be a continuous, binary or count response. However, as in the case of cross-sectional data analysis, this model does not apply t o multi-level nominal or ordinal response. We can readily generalize the distribution-free models for such types of response in Chapter 2 to the longitudinal data setting. Example 5 (Distribution-free models f o r multilevel categorical response). Within the longitudinal study context in Example 4, consider a categorical response with J distinct outcomes T I , . . . , T J . Let xit be a vector of independent variables from the ith subject at time t. For each T time t , let yit = ( y i l t , . . . , yijt) be a vector of binary responses yijt from the ith subject at time t with yijt = 1 if the response is in the j t h category and yijt = 0 if otherwise (1 5 j 5 J ) . For nominal response, we discussed in Chapter 2 the generalized logit or multinomial response model for such a response type. Within the current longitudinal data setting, let the conditional probability of response in the j t h category given generalized logit model is defined as:
xit.
The
where a = ( a l ,. . . , a ~ - 1T) is the vector of parameters of the baseline generalized logits in the absence of x and ,B = (,BT,. .., the parameter vector of interest. Similarly, we generalize the proportion odds model for ordinal response in Chapter 2 to the longitudinal data context. Let Y~~~ (xit) = Ci=,pikt (xit) denote the cumulative probability of response up t o and including category j conditional on the vector of independent variables xit at time t . The proportional odds model is defined by (4.32)
4.3 DISTRIBUTION-FREE MODELS
20 1
where CY = (cq,. . . , aj-1) T and p have the same interpretation as the generalized logit model. For ordinal response, it is preferable to model the cumulative response because of the invariance property when response categories are combined. For example, suppose that there are five ordered responses, rl < 1-2 < . . . < 7-5. If the first two categories are combined, we will have four categories, F1 < 7-2 < - . < 7-4. This change in the response categories has a negligible effect on the ordinal model as it only reduces the number of parameters without affecting the interpretation of the remaining parameters, particularly ,B.
4.3.2 Inference for Distribution-Free Models Consider the class of distribution-free models in (4.28). At each time t, (4.28) reduces to the distribution-free model for cross-sectional data discussed in Chapter 2. For such a model, the most popular inference approach for inference is the estimating equation:
(4.33) where Git (xit) is some known matrix function of xit. As noted in Chapter 2, estimates obtained as solutions to the equation in (4.33) are consistent regardless of the choice of Git (xit) so long as E (yit - pit) = 0. In most applications, Git (xit) is selected based on the following to achieve efficiency:
(4.34) If yit given xit follows the exponential family of distributions, w (.) is readily determined and the estimate equation in (4.33) yields the MLE of p. The generalized estimating equation (GEE) is developed by extending the estimating equation (4.33) at a single time t to multiple times across all assessments. This is achieved by capitalizing on the fact that the use of a wrong correlation matrix has no impact on the consistency of the GEE estimate of p, just as in the univariate case the misspecification of w (.) does not affect the consistency of the estimating equation estimate. Let Si = yi - pi with
202
CHAPTER 4 MODELS FOR CLUSTERED DATA
In analogy to (4.33), the generalized estimating equation (GEE) is defined bY n
n
3
where Gi (xi) is some matrix function of xi. The GEE estimate is obtained by solving the above vector equation. As in the univariate case, ,B is consistent independent of the choice of Gi (xi) provided that E (yi - p i ) = 0 (see Theorem 1 below). It is readily checked that under the model in (4.28), this condition holds true (see exercise). In addition, E(w,) = 0, that is, the GEE is unbiased. In most applications, Gi (xi)has the form: h
(4.36)
where R ( a )denotes a working correlation matrix parameterized by a and diag(v ( b i t ) )a diagonal matrix with u ( p i t ) on the tth diagonal. The phrase "working correlation" is used to emphasize the fact that R ( a ) is not necessarily the true correlation matrix. For example, the simplest choice is R = I,. In this case, the correlated components of yi are treated as if they are independent. In addition, there is no parameter associated with this particular working independence model. Another popular choice is the uniform compound symmetry correlation matrix, R ( a ) = C, ( p ) , which assumes a common correlation p for any pair of the component responses of y i . The working correlation matrix involves a single parameter, a = p. Under the specification in (4.36), (4.35) can be expressed as n
n
The above is identical to the estimating equation in (4.33) for cross-sectional data analysis except that Di in (4.36) is a p x m matrix rather than a p x 1 vector. Although the GEE in (4.37) is a function of both ,B and a , we express it explicitly as a function of 0 to emphasize the fact that (4.37) is used to obtain the estimate of the parameter vector of interest p. If a is known ( e g . , for the working independence model), we can obtain the
203
4.3 DISTRIBUTION-FREE MODELS
solution of
p
to (4.37) by either of the recursive algorithms: (4.38)
where ,do) denotes some initial value (see exercise). As in the univariate case, the second equation above is easier to compute. The GEE estimate is obtained by iterating the selected algorithm until convergence. When a is unknown, we must estimate it so that (4.37) can be used t o find estimates of 0.We illustrate these considerat,ions below. Example 6 ( M AN O VA/repeated measures A N O VA). Consider the distribution-free model for group comparison in Example 1. In this case, YkZ = ( Y k Z l , . *
T
* 1
To find w ( p i t ) , assume g,+it Thus, Vji = w, ( p ) =
YkZm)
-
(
p,+1
' * *
Ilkrn
)
T
(4.39)
N ( p k l ,0 ' ) . Then, w ( p i t ) = 1 and A,+i= Irn.
( a h ) Ati =
2
pk =
!
R,+(a,+).
R i l (a,+) Ski
R i l (a,+) (y,+i- p k )= 0
=
(4.40)
k = l i=3
,+=l i=l
Regardless of the choice of R k (a,+), the GEE estimate of p,+ is given by p k = Cy21y,+i (1 5 k 5 9 ) . Further, if y,+i N ( o , o ' R ~( a , + )ji,+ ) , is actually the MLE of pk (see exercise). Example 7 (Distribution-free linear regression model). Let yit and x,t be the response and vector of independent variables of interest from the ith subject assessed at time t in a longitudinal study with n subjects and m assessments. Consider the distribution-free linear regression model:
-
d
E
(yit
I xit) = x i p ,
As in Example 6, if we assume Thus, Ai
= Im,
yit
v, = R ( a ) ,
1 5 i 5 n,1 5 t 5 m
1
xit
pi =
-
(4.41)
N (xzp,o'), then w ( p i t ) = 1.
a xzp, D "aap . - -p.
=
x:
204
CHAPTER 4 MODELS FOR CLUSTERED DATA
where Xi = (xil,. . . ,xirn)T . It follows that W,
(p)=
n
n
i=l
i=l
C XTR-l ( a )Si = C XTR-'
( a )(yi - Xip) = 0
(4.42)
The above is readily solved to obtain
i
p = Cxp2-1( a )xi -'(2xTR-1 ( a )yi) ( i=l n
(4.43)
i=l
3
If a is known, is the GEE estimate of p. Otherwise, a must be estimated in order to obtain estimates of p. As an example, consider the completely unstructured working correlation matrix, R ( a )= [pst]. We can estimate pst by the Pearson correlation: Pst =
E;=1 ( T i s - 7.4ti.(
&;=,
si.(
- 7.t)
x;!l
- Q2
(.it
,
Tit = yzt -
Txitp
(4.44)
- F.t)2
where F.t = 1 n T i t . If yi 1 Xi N (Xip, a2R( a ) )and a2 is known or estimated by (4.44), is the MLE of ,8 (see exercise). Example 8 (Distribution-free logistic regression model). In Example 7, assume a binary response yit and consider the logistic model: N
E
(Yit
1 X i t ) = Pit,
log
(A) 1 - Pit
=xip,
1 5 i 5 n, 1 5 t 5 m
(4.45)
As discussed for the univariate distribution-free GLM in Chapter 2, for each t , the estimating equation in (4.33) with Git and Dit given in (4.34) and ( p i t ) = pit (1 - p i t ) yields the MLE of ,8 if yit I xit wBI(pit,1). Thus, we may choose Ai =diag(pit (1 - p i t ) ) for the GEE to estimate p for the multivariate model in (4.45). As in Example 7 , if a is unknown, it must be estimated. However, since the correlation pi,st = Cow (yis, yit I xis,xit) for binary response generally varies across subjects, a constant R ( a ) is hardly ever the true correlation matrix except for some special cases. In addition, pi,st must also satisfy an additional set of Frechet bounds (see exercise):
205
4.3 DISTRIBUTION-FREE MODELS
Because of these constraints, a is typically selected by some ad hoc rules rather than estimated. For example, under the uniform compound symmetry assumption, R ( p ) = C, ( p ) , we may select p that satisfies the Frechet bounds in (4.46). Although quite similar in form, GEE differs from the univariate estimating equation considered for the distribution-free GLM discussed in Chapter 2 in that Gi or V, may involve an unknown vector of parameters a. While primary interest lies in p, a must be estimated t o proceed with the computation of the GEE estimate of p. Although consistency of the GEE estimate does not depend on how a is estimated, judicious choices of the type of estimates of a used in (4.35) not only ensure the asymptotic normality but also simplify the asymptotic variance of Theorem 1 (GEE estimates). Let denote the estimate obtained by solving the generalized estimating equation in (4.35). Let B = E ( G J J T ) . Then, under mild regularity conditions, we have I. is consistent. 2. If f i ( & - a ) = 0, ( l ) , is asymptotically normal with its asymptotic variance I=b given by
B
B. B
B
B
cg= B - 1 ~(G,S,S;G:) where BPT=
B-T
(4.47)
A consistent estimate of E p is given by n
(4.48) i=l
where A^ denotes the estimated A obtained by replacing p and a with their respective estimates. If Gi = DiV,-', then B = E (DiV,1Di T ). In this case, (4.47) and (4.48) reduce to
->
cp= B - ~ E(D~V,-'S~S;V,- D~ B-l
(4.49)
Proof. The proof follows from an argument similar to the one used for the estimating equation estimates for the univariate distribution-free GLM studied in Chapter 2.
206
CHAPTER 4 MODELS FOR CLUSTERED DATA
E (GiSi) = E (GiE (SiI xi)) = E [GiE(Si1 xi)]= 0 V a r (GiSi) = E [GiE (SiST I xi) GT]
=E
(4.50)
(GiSiSTGl)
Thus, it follows from CLT that
By an argument similar to the proof of consistency for the univariate estimating equation estimate, we can show that the GEE estimate is consistent regardless of the choice of Gi (see exercise). By a Taylor series expansion, we have
Thus.
It is readily checked that (see exercise):
207
4.3 DISTRIBUTION-FREE MODELS
It follows that &wn
(0, a ) = 0, (1) and thereby
Thus, (4.52) reduces to:
It is readily checked that (see exercise):
The asymptotic normality of follows from (4.51), (4.55), (4.56) and Slutsky's theorem with the asymptotic variance Xcp given in (4.47). Note that if fi(6 - a ) = 0, ( l ) ,that is, (& - a ) is stochastically bounded, & is called +-consistent. As seen from the proof of the theorem, T this assumption allows us to ignore the term (&wn (p,a ) ) (& - a ) in the asymptotic expansion of &wn (p,a ) in (4.52) to obtain the asymptotic normality and compute the asymptotic variance of p. As discussed in Chapter 1, &(& - a ) = 0, (1) holds true if fi(& - a ) converges in distribution. In most applications, 2 is asymptotically normal and thus is +-consistent. Example 9 ( M AN O VA/repeated measures A NOVA) . For the distribution-free model in Example 6, it follows from Theorem 1 that the GEE estimate fi is consistent and asymptotically normal. In addition, it follows from (4.49) that the asymptotic variance for the GEE estimate f i k is given by
+
+
h
In this particular case, is the variance of yki and independent of the choice of working correlation matrix Rk ( a ) . Further, if Y k i N (O,Ck), C I , is the asymptotic variance of the MLE of f i k (1 5 k 5 9 ) . N
208
CHAPTER 4 MODELS FOR CLUSTERED DATA
Example 10 (Linear regression). In Example 7, the GEE estimate If a is known, is also asymptotically normal. The asymptotic variance is given by
a
3 given in (4.43) is consistent.
x:)
C = B-lE (XTR-1( a )szS,TR-1 ( a ) where B = E
(XTR-l( a )X-) If yi I X i C = B-IE
N
B-l
N (Xip, a2R( a ) )then ,
(XTR-1( a )SiSlR-1( a )xi'>B-l
= a2E (XTR-'
(4.57)
(a) X->
3
In this case, is the MLE of and C is the asymptotic variance of (see exercise). If a is known and estimated by (4.44), a is fi-consistent and therefore If yi 1 Xi by Theorem 1, C in (4.57) is the asymptotic variance of N ( X @ ,a2R( a ) ) , becomes of the MLE and C is the asymptotic variance of the MLE Example 11. In Example 10, let C = V a r ( y i I X i ) = [asatpst]with V a r (yit I xit) = a: and Corr (yit I xit) = pst. If a: f. a2, that is, a nonconstant across time, then given in (4.43) will not be the MLE of p when yi I Xi N(X@,C). For the GEE in (4.37) to yield the MLE of ,kl in this case, we can set R ( a )= C = [asa.tpst]and A, = I,. Of course, R ( a ) no longer has the interpretation of being the correlation matrix. We can estimate the entries of R ( a )by the residuals T i t = yit - xLp. The GEE in (4.35) only applies to the distribution-free model defined in (4.28). We must develop a different approach for inference about the model for multilevel categorical response discussed in Example 5. We consider the generalized logit model for nominal response below. Similar considerations apply to the proportion odds for ordinal response. Example 12 (Generalized logit model). For the nominal response model in Example 5, we can equivalently express (4.31) as follows:
3.
3.
3
N
h
j
1
1, ..., J - 1
N
209
4.4 MISSING DATA
Let Yit = (Yilt,. . . , ! / i ( J - l ) t )
I
T >
Pit = (/Liltr. . . > P i ( J - l ) t )
(4.59)
For each t , yit has a multinomial distribution with mean pit and variance Aitt given by
1
1
Thus, let V, = A: R ( a )A: with R ( a )and Ai defined by
where R j k are some ( J - 1) x ( J - 1) matrices parameterized by a. Define the GEE in the same form as in (4.37) except for using this newly defined V , above, and Di and Si in (4.59). As in the case of binary response, a constant working correlation matrix R ( a ) is generally not the true correlation matrix and similar Frechet bounds exist for the elements of R ( a )(see exercise). Thus, unless we use the working independence model, R ( a ) = I m ( J - l ) x m ( J - l ) r we must be mindful about the bounds when selecting R ( a ) .
4.4
Missing Data
Two key issues arising in the analysis of longitudinal data are the withinsubject correlation among repeated assessments and missing data. Up to this point, we have discussed how to address the former issue under both
210
CHAPTER 4 MODELS FOR CLUSTERED DATA
parametric and distribution-free modeling frameworks. In this section, we address the latter missing data issue. In most longitudinal studies, missing data is inevitable, even in well designed and executed clinical trials. In longitudinal studies, subjects may simply quit the study or they may not show up at follow-up visits because of problems with transportation, weather conditions, health status, relocation, and so on. In clinical trials, missing data may also be the results of patients’ deteriorated or improved health conditions due t o treatment, treatmentrelated complications, treatment response, and so on. Some of the reasons for missing data are clearly treatment related while others are not. In statistics, we characterize the impact of missing data on model estimates through assumptions or missing data mechanisms. Such assumptions allow statisticians to ignore the multitude of reasons for missing data and focus on addressing their impact on estimation of model parameters. The missing completely at random (MCAR) assumption is used to define a class of missing data that does not affect model estimates when completely ignored. For example, in a treatment study, missing data resulting from patient’s relocation and conflict of schedules fall into this category. MCAR corresponds t o a lay person’s notion of random missing, that is, missing data are completely random with absolutely nothing to do with treatment effect. The next category is the missing at random (MAR) assumption, which generalizes MCAR to deal with a popular class of treatment-related missing data. In many clinical trials, missing data is often associated with the treatment interventions under study. For example, a patient may quit the study if he/she feels that the study treatment has deteriorated his/her health conditions and any further treatment will only worsen the medical or psychological problems. Or, a patient may feel that he/she has completely responded to the treatment and therefore does not see any additional benefit in continuing the treatment. In such cases, missing data does not follow the MCAR model since they are predicted by treatment related response. By positing that the occurrence of a missing response at an assessment time depends on the response history or observed pattern prior to the assessment point, MAR constitutes a plausible and applicable statistical approach to model this class of treatment related missing data. Missing data that satisfies either the MCAR or MAR model is also known as ignorable missing data. Nonignorable missing data is defined as a type of missing data the occurrence of which can depend on unobserved or future response (in a longitudinal setting). Since the mechanism of non-ignorable missing data involves unobserved response, it is generally quite difficult to
211
4.4 MISSING DATA
model such missing data in real studies without additional data or information regarding the relationship between the missing and observed data. Note that the term "ignorable missing data" may be a misnomer. For parametric models, we can indeed ignore such missing data since maximum likelihood estimates are consistent when obtained based on the observeddata. For distribution-free models, however, GEE estimates will generally not be consistent when missing data follows the MAR model. Thus: alternative estimating equations must be constructed to provide consistent model estimates. In this book, we focus on MCAR and MAR, which apply to most studies in biomedical and psychosocial research. In addition, we only consider missing data that occurs in the response variable. Although missing independent variables are also common in real study applications, it is more complex to model them, especially in the presence of missing response.
4.4.1 Inference for Parametric Models Consider a longitudinal study design with n subjects and m assessments. Let yit be the response and xit a vector of independent variables of interest. For each subject, we define a missing (or rather, observed) data indicator as : 1 if yit is observed rit = , ri = (ril, (4.61) 0 if yit is missing
i
We assume no missing data at baseline t = 1 such that ril = 0 for all l l i l n . T T T Let yi = (yil, ...,yim)T and xi = (xil, ...,xim) . Let yg and yy denote the observed and unobserved responses, respectively. Thus, yg and yy form a partition of yi. Under likelihood based parametric inference, we jointly model the response yi and missing data indicator ri. The joint density or probability distribution, f (yi, ri I xi), can be factored into the product of marginal and conditional distributions using two different approaches, giving rise to two distinct classes of models known as the selection and mixture models. We outline the two approaches below. One way to factor the joint distribution is as follows: (4.62)
Selection models are developed based on the above factorization with the term selection reflecting the probability of observing a response or a selection
212
CHAPTER 4 MODELS FOR CLUSTERED DATA
process. Under MAR, the distribution of ri only depends on the observed responses, yp, and we thus have
It follows from (4.62) and (4.63) that:
If 8, and 8,1r are disjoint, then following (4.64) the log-likelihood based on the joint observations (yp, ri) is given by
i ( 8 ) = 11 (6,)
+ 12 (Oylr)
(4.65)
Thus, inference about the regression model can simply be based on I1 (8,). In other words, missing data can be "ignored" if interest is centered on modeling the relationship between yi and xi. It should be emphasized, however, that if 8, and 8,IT are not disjoint, inference based on f (y; I xi;8,) may be incorrect. In practice, it is difficult to validate this disjoint assumption between 8, and t9,1T, which makes it a potential weakness for applications of selection models. Under MCAR, it follows from (4.63) and (4.64) that
f (.i I Y f , xi,Oylr)
f
= i.(
I xi,e,,,)
Unlike the MAR case, f (ri 1 xi,8y,r)is independent of yp, lending support t o the disjoint assumption between 8, and 81 ,., Thus, only under MCAR can missing data be completely ignored when making inference concerning the relationship between between yi and xi. Alternatively, we can factor f (yi, ri 1 xi) as
Since the marginal distribution of yi obtained by integrating out ri in the factorization above is in the form of a mixture with mixing probability
213
4.4 MISSING DATA
f ( r i I xi), models developed based on (4.66) are known as mixture models. This approach is frequently used when the missing pattern itself is also of interest. For example, in a saturated pattern-mixture model, the relationship between yi and xi are modeled separately for each pattern. The overall relationship is then a mixture of the different missing data patterns. Let p denote the number of distinct missing data patterns. Then, it follows that (see exercise) :
where 4; denotes the parameter vector for modeling the relationship between yi and xi for the kth pattern (1 k p ) . If 4; = q5y, that is, a common relationship across all patterns, then missing data follows MCAR and may be totally ignored if interest lies in 4y. Under MAR, f (y&I ri, xi, 4;) depends on missing data patterns. If missing response can occur at any time t , there are potentially Zm-l different patterns, making it difficult to model this relationship. Fortunately, as noted in the beginning of Section 4.4, missing data in most longitudinal studies typically occurs as the result of subject dropout, reflecting the subject's deteriorated/improved health conditions and other related conditions. In this case, missing data follows the monotone missing data pattern (MMDP) and has p = m distinct patterns. Under MMDP, yit is observed only if yis are all observed prior to time t (1 s 5 t m ) . Such structured missing data patterns greatly simplify modeling f (yf I ri, xi,4;) and other relationships involving T i t . Suppose that missing data satisfies the MMDP assumption. Then, there are m distinct patterns defined by the last observed response at t = 1 , 2 , . . . , rn. Thus, we can index the m patterns by p = t (1 5 p 5 rn). Let T Yit = (yil, . . . , ~ i ( ~ - 1 ) ). It is readily shown that MAR is equivalent to the condition: (4.68) f (Yit I Y i t , xi,p = j ) = f (Yit I Yit, xi,P 2 t )
< <
<
<
for all t 2 2 and j < t . The above condition is known as the available case missing value restriction. Example 1 (Multivariate linear regression). Consider a longitudinal study with two assessments. Let git be a response and xit a vector of
214
CHAPTER 4 MODELS FOR CLUSTERED DATA
independent variables of interest and assume a multivariate linear regression: yi = Xip
+ ci,
~i = ( c i l , ~
2
N
i.i.d.N ) ~ (0, C)
(4.69)
where yi = ( y i 1 , y i ~ ) ' and Xi = ( x i l , ~ ~ )There ~ . are t'wo missing data patterns. Under the saturated pattern-mixture model assumption, we have yi I p = k , Xi
N
i.d.N (Xip,, XI,) ,
15 k 5 2
It follows from the properties of multivariate normal distribution that
where 0 j l k denotes the j l t h element of (4.68) that p k and CI, must satisfy: x31+
Under MCAR,
0121
- (Y210111
x;pl)
Ck.
=xAP2
Under MAR, it follows from 0122 +(Yil 0112
-
xLL32)
p1 = p2 and C 1 = C 2 , and the above holds true.
4.4.2 Inference for Distribution-Free Models For parametric models, missing data can be ignored even when it follows the MAR model. Unfortunately, this is not true for distribution-free models. More specifically, the generalized estimating equation discussed in Section 4.3.2 only guarantees consistent model estimates when missing data follows MCAR. In this Section, we discuss inference using a revised estimating equation known as the weighted generalized estimating equation (WGEE). We motivate the development of WGEE by a relatively simple longitudinal study design with only two assessment points. Example 2 (One-sample repeated measures ANOVA under M C A R ) . Consider a longitudinal study with two assessments and a homogeneous study group with n subjects. Let yit denote a continuous response from the ith subject at time t (t = 1,2). We are interested in estimating the mean of yz = (ylt, By treating this one-sample problem as a special case of the repeated measures ANOVA, it follows from the discussion in Section 4.3 that the GEE estimate of the mean /.L = E ( y , ) of y, is the sample mean.
215
4.4 MISSING DATA
Under our assumption, yil is always observed, but yi2 may be missing. Let ri2 be a missing data indicator for yi2, with ri2 = 1 if yi2 is observed and 0 if otherwise. Consider the following estimate of p : n
n2 =
C
7-22
(4.70)
i=l
It is readily seen that the above is simply a fancy way t o express the sample mean of y22 based on observed data. If yz2 is not subject to missingness, j2 above yields the GEE estimate of p in the complete data case. Otherwise, if missing yz2 follows MCAR, rz2 is independent of yzl as well as yz2. Thus, we have (4.71) 7rz2 = Pr (rz = 1 I Yzl,YZ2) = Pr (rz = 1) Intuitively, G2 should be consistent. This is indeed the case; it follows from LLN, Slutsky’s theorem and (4.71) that
Example 3 (One-sample repeated measures ANOVA under M A R ) . In Example 2, if the missingness depends on yZ1,(4.71) no longer holds and
F2
in general will not be consistent.
For example, if yzl and
yz2 are positively correlated with higher values of yzl more likely leading
to missing y22, F2 will be downwardly biased. In treatment studies, this type of response-dependent missingness may happen if a patient feels that he/she has responded to the treatment and decides not t o undergo additional treatment. To construct a consistent estimate, we may somehow “recover” the missing response yz2 for these study dropouts and include them in the estimation of pa. Of course, such a recovery process is not possible in most studies. However, with a probability model for the missingness data indicator r z , we can “statistically recover” such missing yz2. Since the missingness of yzl only depends on yzl as guaranteed by the MAR model, we have:
For each subject i, we observe yi2 with probability 7ri2. Thus, this subject subjects with response yil at time 1 who are represents a subgroup of unobserved at time 2. By augmenting each observed subject with the weight
&
216
C H A P T E R 4 MODELS F O R CLUSTERED DATA
in the following estimate, we are in effect statistically recovering function and including such “ghosts” in the estimation of p 2 :
h
1-12 =
1
Cy=1 -1 ri2
n
-rizyi2
(4.74)
i=l
We can think of the denominator as representing the total number of subjects (observed plus ghosts) and the numerator the sum of response at time 2 over all such subjects. The weighted estimate g2 is indeed consistent. To see this, observe first that
Then, by applying an argument similar to (4.72), it follows from (4.75) that g2 is consistent (see exercise). E x a m p l e 4 (One-sample repeated measures ANOVA under nonignorable missingness). In Example 3, suppose that the missingness also depends on yi2, that is,
In this case, p2 defined in (4.74) is still consistent (see exercise). However, unlike the model in (4.73), it is generally not possible to estimate 7ri2 defined in (4.76) based on observed data when 7 r is ~ unknown as in most applications. For example, we can model and estimate 7ri2 as a function of yil in (4.73) using a logistic model. But, this approach does not work for 7ri2 in (4.76) as yi2 is not observed ri2 = 1. Additional information and/or data source (other than the data at hand) are needed to help identify the parameters in such a logistic model. Examples 2-4 cover the three major missing data mechanisms employed in statistical modeling of missing data and illustrate how to construct consistent model estimates under each situation. The weighted generalized estimating equation (WGEE) is developed based on the same principles by using weight functions to account for the contribution of missing response from the ghost. E x a m p l e 5 ( W G E E for one-sample repeated measures ANOVA
217
4.4 MISSING DATA
under M A R ) . Within the context of Example 3, let
d D. - -p a
-
dp
1
= 12 ,
Si = yi - p ,
1
V = ASR ( p ) At? = R ( p )
Define a weighted generalized estimating equation (WGEE) as follows: n
(4.77)
i=l
The WGEE above is quite similar to the GEE in (4.37) except for the extra term weight function Ai. The presence of Ai is essential t o ensure the unbiasedness of the above equation, that is, E (wn ( p ) )= 0, under MAR, which in turn provides consistent estimates of p. This readily follows from (see exercise):
The equation in (4.77) is readily solved t o yield
which is identical to the weighted estimate of p in (4.74) constructed in Example 3. Thus, the estimate of p obtained from the WGEE in (4.77) is consistent. Note that R-' ( p ) in this special case does not play any role in inference for p . The WGEE for this relatively simple one-sample ANOVA with a pre- and posttreatment study design is readily generalized to the general setting with more complex models and multiple assessments. Consider a longitudinal study with n subjects and m assessments. Let yit be a response and xit a vector of independent variables of interest. Consider the distribution-free regression model in (4.28), which for convenience is copied below:
E
(pit
I xit) = pit,
g ( p i t ) = x$P,
1It
L m, 1 I i I n
(4.80)
218
C H A P T E R 4 MODELS FOR C L U S T E R E D DATA
where g (.) is some known link function and p a p x 1 vector. Let Tit denote a binary indicator with the value 1 if yit is observed and 0 if otherwise. Let 7rit =
P r ti.(
I
= 1 xi, yi) ,
Tit
Ait = -,
Tit
Ai
= diag,
(&)
(4.81)
The WGEE for the class of models in (4.80) has the form: n
n
where G, (x,), S,, y, and h, are defined the same way as for the unweighted GEE in (4.35) in the absence of missing data. The WGEE in (4.82) is unbiased, that is, E (wn ( p ) )= 0, if E (AZS,)= 0. By using an argument similar to proving (4.78) in Example 5, it is readily checked that E (A,S,) = 0 is guaranteed if the model in (4.81) (and of course, the model for regression in (4.80) as well) is specified correctly (see exercise). To be able to observe representatives to statistically recover all ghosts, From an 7r,t > 0 must hold true for all 1 5 t 5 m and 1 5 z 5 n. inference point of view, this requirement ensures that representatives of all subgroups are observed. Otherwise, some subgroups will be left out, yielding biased estimates of ,B. If 7rtt is very small, the WGEE estimate obtained from (4.82) may be unstable. Thus, we assume that 7r,t 2 c > 0. Also, G, may depend on some parameter vector a , and for most applications, G, (x,) = (&A,h,) V,-l = D,A,y-l, where D, and V , are defined the same way as for the unweighted GEE in (4.36). As in the case of unweighted GEE, the consistency of the estimate from (4.82) is independent of the type of estimates of a used, but the asymptotic normality of the estimate is ensured when a is substituted by some &-consistent estimate. We summarize these asymptotic properties of the WGEE estimate in a theorem below. The proof of the theorem follows by applying an argument similar to the proof of Theorem 1 for unweighted GEE in Section 4.3.2 (see exercise). Theorem 1 (WGEE estimates). Let denote the estimate obtained by solving the WGEE in (4.82). Assume that 7r,t are known and 7r,t 2 c > 0 (1 5 z 5 n, 1 5 t 5 m). Let B = E (G,A,Dd). Then, under mild regularity conditions, we have I. is consistent. 2. If Ei is &-consistent, p is asymptotically normal with the asymptotic variance EDgiven by
b
fi
h
(4.83)
219
4.4 MISSING DATA
A consistent estimate of X p is given by
(4.84) where A^ denotes the estimate of A obtained by substituting the estimates of p and a in place of the respective parameter vectors. If Gi (xi) = D i A i Y - l , then B = E (Di&Y-'&DT), and (4.83) and (4.84) reduce to
cp= B
- 1 ~
(D ~ A ~ ~ - ~ s ~ s : T / , - ~ AB-T ~DT)
(4.85)
In most applications, Tit is unknown and must be estimated. Under MCAR, ri is independent of xi and yi and thus T i t = P r ( T i t = I) = T t . In this case, 7rt is a constant independent of xi and y i and is readily estimated by the sample moment: ?t = $ CyzlT i t (2 5 t 5 m). Under MAR, Tit becomes dependent on the observed xi and y i . As discussed in Section 4.4.1, it is generally difficult to model and estimate 7rit without imposing the MMDP assumption because of the large number of missing data patterns. The structured patterns under MMDP greatly reduce not only the number of missing data patterns, but also the complexity in modeling T i t . - ~ ?it ) )=~( y i l , ..., yi(t-l))T ( 2 5 t Im ) , deLet Xit = (x;, ...,x T~ ( ~ and noting the vectors containing the independent and response variables prior to time t , respectively. Let Hit = {Zit, ?it; 2 I t 5 m } denote the observed data prior to time t (2 I t 5 m, 1 5 i < - n ) . Then, under MAR we have
Under MMDP yit is observed only if all yis prior to time t are observed. We can first model the one-step transition probability pit of the occurrence of missing data and then compute 7rit as a function of pit by invoking the MMDP assumption. = 1,Hit) denote the one-step transition Let pit = E ( T i t = 1 I probability from observing the response at t - 1 to t. We can readily model pit using a logistic regression model:
220
C H A P T E R 4 MODELS F O R CLUSTERED DATA T
where yt = ([,, 7 ): denotes the model parameters and gt ( q t ,Zit, ?it) some functions of (qt,5&,?it). For example, if Yit and xit are all continuous, T T we can set gt (qt,Z i t , F i t ) = q;&t r&?it with qt = (qZt, q:t) . More complex forms of gt ( q t ,Zit, ?it) as well as cases with other types of yit and xit, such as binary, are similarly considered. Under R/IMDP, we have (see exercise) :
+
t Tit
( 7 )= pit P r
(Ti(t-1) =
1I
Hi@-1))=
n p i s ( 7 s ),
(4.88)
s=2
25t
l
T
where y = (yl , . . . ,yA) . Thus, we can estimate Tit from pit in (4.87) using the above relationship. Note that when modeled by (4.87) and (4.88),Tit in theory ranges between 0 and 1. However, xit can be assumed t o be bounded and the requirement 2 c > 0 is satisfied for all practical purposes. Note also that when Tit is estimated by (4.88), the asymptotic variance in (4.83) or (4.85) is likely to underestimate the variability of Unlike the parameter vector a in Gi, the variability of 7cannot be ignored even for fi-consistent estimate. If 7 is obtained from GEE or maximum likelihood method, we can readily find the asymptotic variance of obtained from (4.82) with y substituted by 7. Let 7be the solution to the equation: u, (y) = Uni (y) = 0, where u,i is the score (for maximum likelihood estimation) or score-like vector (for GEE estimation) for the ith subject (1 5 i 5 n). As in the proof of Theorem 1 of Section 4.3.2, let u, and w, denote the normalized versions of u, and w, in (4.82), that is,
b.
9
z:=l
i=l
i=l
Then, by a Taylor series expansion, we have
221
4.4 MISSING DATA
By applying an argument similar to the proof of Theorem 1 of Section 4.3.2, it follows from the above that (see exercise):
J;E(a-fJ)= (-$wn)-TJ;E
[ w n + (&wn)'(?-7)]
+
(4.89)
where
3
It immediately follows from (4.89) that is asymptotically normal with the asymptotic variance given by (see exercise):
A consistent estimate of Cp is readily obtained by substituting consistent estimates of the respective quantities in (4.90). In comparison to (4.83), the asymptotic variance has an additional correction factor. So far, we have assumed that we can model 7rit correctly when it is unknown. If the model for 7rit is wrong, that is, Tit # E(rit = 1 1 I&), the condition E ( A i S i ) = 0 may not be true and the WGEE may yield biased estimates of fJ. In some cases, however, we can still consistently estimate ,B if we have a correct model for the missing yit. We illustrate this double robustness idea again with a relatively simple pre-post study design involving two assessment times. As noted in Section 4.2.1, ANOVA and ANCOVA may be applied to compare treatment difference at posttreatment for controlled randomized longitudinal trials. Under MAR, these models generally yield biased estimates when applied directly to the observed data at posttreatment. By generalizing Example 5 to multisample repeated measures ANOVA, we can immediately address MAR using WGEE. Example 6 (Multisample repeated measures A NOVA under M A R ) . Within the pre-post study design in Example 5 , consider comparing g treatment conditions with nk subjects in the kth treatment condition
222
CHAPTER 4 MODELS FOR CLUSTERED DATA
(1 5 k I: 9 ) . Let ykit denote a continuous response from the ith subject within the kth treatment at time t (t = 1 , 2 ) . Let T
Yki = ( Y k i l , Y k i 2 ) T k i 2 = E (rki2
pk = E
>
1 ykil)
7
(4.91)
(Yki) = (pk1, pk2lT
nkil =
1,
nki2 =
rki2
-> Tki2
&i
= diagt (&it)
By substituting 12, Aki and yki - p k in place of Gi,Ai and solving for p k , we immediately obtain
Si in (4.82) and
The WGEE estimate of & has the same form as in (4.79) except for the obvious generalization to the g groups. The estimate Pk2 in (4.92) may be biased if T k 2 2 is not modeled correctly. As an alternative, we may use an ANCOVA-like model to address MAR without positing any model for T k 2 2 . Example 7 (ANCOVA f o r p r e - p o s t s t u d y d e s i g n u n d e r M A R ) . Within the context of Example 6, suppose that we have a linear model for Yka2 as a function of ykzl as follows: Y k z 2 = pkz2
Ekz
N
+ Ekz,
pkz2 = T k
i.i.d. ( 0 , a 2 ),
+ p k y k z l = X;$k
15 i 5 nk.
(4.93)
15 k 5g
T apply (4.93) t o the where x k z = ( l , y k z l ) and 81, = ( ~ k , p ~ ) If~ we . observed response y k 2 2 , we can estimate 81, by the following unweighted GEE:
nk
nk
Gt ( Y k z l ) z=1
GZ( Y k t l ) rk22 ( Y k t 2 - p k 2 2 ) = 0
rkz2sz =
(4.94)
z=l
w,
It is readily checked that E (rk22Sz)= 0 so that the above GEE is unbiased (see exercise). In addition, by setting G, (ykzl) = Dkz = we obtain from (4.94) the GEE estimate of 8 k :
The consistency of G k above follows immediately from the unbiasedness of the GEE in (4.94), but can also be checked directly by using its expression in
223
4.5 G E E I1 FOR MODELING M E A N A N D VARIANCE
(4.95) (see exercise). Note that G k in (4.95) is also the maximum likelihood estimate of 8 k if ~ k -i.i.d.N i (0, 02). Thus, we can estimate p k 2 by Ek2 = h
2 nk
x&ek.
Example 7 shows that we can estimate p k 2 in Example 6 without relying on modeling rki2 if we have a correct model for relating Y k i 2 to ykli. In practice, we may not know which model is correct. By combining the approaches in the two examples, we can develop a modified WGEE that will deliver consistent estimates of p if one of these models is correctly specified. Example 8 (Double-robust estimate f o r pre-post s t u d y design u n d e r M A R ) . Consider estimating p k 2 in Example 6. Let
where &,2 = xL8k with Xkz and 81, defined in (4.93) of Example 7. Assume that 6 k is known or substituted by a consistent estimate. It is readily checked that E (Sk,)= 0 if E (rk,2 I y k , ~ )= 7rk22 or the model in (4.93) is correct. Thus, by substituting Sk, in (4.96) together with 12 and Ak, in (4.92) in place of S,, G, and A, in (4.82), we obtain an unbiased WGEE for estimating pk if one of the models for Q.22 and y k z 2 is correct. Note that if p k = p and E ( y k , l ) = p l in Example 7, (4.93) becomes ANCOVA. In this case, we can use this model directly to examine posttreatment difference by comparing ~k since 7 k - q = pk2-pz2(1 Ic, I 5 9 ) . As in the univariate distribution-free GLM case, the Wald statistic can be used to test the general linear hypothesis of the form in (4.27). Similarly, general linear hypotheses concerning the parameter vector p in the linear predictor can be tested through linear contrasts by including an offset term in the linear predictor or by performing a linear transformation on the response yzt for linear models.
<
4.5
GEE I1 for Modeling Mean and Variance
A major drawback of the distribution-free model discussed in Section 4.3 and Section 4.4 is that it only models the conditional mean of the response y given the vector of independent variables x. However, as noted in Section 4.2.1, it is also of great interest to model second and even higher order moments. For example, Cronbach coefficient alpha and intraclass correlations are all defined by the variance parameters of the response variable. As a result,
224
CHAPTER 4 MODELS FOR CLUSTERED DATA
the mean based distribution-free model is unable to provide inference for such indices of interest. To overcome this limitation, we can generalize the mean response based distribution-free model in (4.28) to model simultaneously the mean and variance of a response variable. By augmenting the model for the mean response with a variance component, we can jointly model the conditional mean and variance of y given x. Example 1 (Linear mixed-effects model). Consider the linear mixed-effects model:
xLP + ZLbi + € i t ,
yit
bi
N
i.i.d.N (0, D ) ,
~i
yi = Xi0 N
+ Zibi + ~i
i.i.d.N (0,021m),
b&i,
(4.97) 15 i 5 n
For notational brevity, we have assumed fixed assessment times. If we only model the mean response, we then obtain from (4.97) the distribution-free counterpart as follows:
E (yit I xit) =
XLP,
1 5 t 5 m,
15 i 5n
(4.98)
The above model only involves the parameter vector ,B for the mean response or fixed effects. However, in many applications, we are also interested in the parameters D for the random effects and u2. This is not a problem for the parametric LMM in (4.97) since P, D and o2 are simultaneously estimated from the likelihood function. But the distribution-free version in (4.98) only involves P and as such does not provide inference about D and 02.
To address this limitation, we can add to (4.98) a component for modeling the variance of yit to obtain
E
1
(Yit Xit) = P at. = x. aTt P T
E (sist I xis,xit) = zisDzit
+ u2 Jst,
(4.99) 15 i
5 n, 1 5 s < t 5 m
where sist = (yis - p i s )(yit - p i t ) and Jst = 1 if s = t and 0 if otherwise. Thus, by complementing the mean response based distribution-free model with a component for the conditional variance, we are able to conduct distrubution-free inference for all the parameters in the parametric LMM. Example 2 (Models f o r count response). In Example 1, assume a count response yit. Then, the distribution-free log-linear model has the following form: (4.100)
225
4.5 GEE I1 FOR MODELING MEAN A N D VARIANCE
The above model provides robust inference about ,B regardless of whether there is overdispersion in the data. For example, this model applies if conditional on xit yit follows either a Poisson or negative binomial (NB) distribution. Suppose now that we want to test over- or underdispersion in the data distribution, which occurs when Var (yit I xit) # pit for some t (1 5 t 5 rn). We can add to (4.100) a component for modeling this conditional variance to obtain the following model:
(4.101)
where u ( p i t ) is some function of pit. For example, we may assume u ( p i t ) = $:pit and test for over- or underdispersion by considering the null Ho : 4: = 1 for all 1 5 t 5 rn. If we believe that yit given xit follows an NB like distribution, we may posit a different model, u ( p i t ) = pit (1 Atpit), and assess for over- or underdispersion by testing Ho : At = 0 for all 1 5 t 5 rn. Note that unlike the parametric formulation, we have the flexibility to allow At to be either positive or negative so that we can use the variance form of the NB model to test either over- or underdispersion. Of course, only a positive A t corresponds to a NB model for the conditional distribution of git given xit.
+
In general, consider the distribution-free model for both the mean and variance given by
As illustrated in Examples 1 and 2, for some applications we may only specify the conditional variance V a r (yit 1 xit) in which case we can set s = t in (4.102). The generalized estimating equation discussed for mean response based distribution-free models in Sections 4.3 is readily extended to provide inference for the model defined in (4.102). Let ,B be the parameter vector for the regression component, C the vector containing the remaining parameters in the conditional variance in (4.102),
226
CHAPTER 4 MODELS FOR CLUSTERED DATA
and 8 = ( pT ,
T T
)
the q x 1 column vector of all model parameters. Let (4.103)
(m-1)m
where si and oi are both x 1 column vectors with sist and gist defined (m+l)m in (4.102) and Si is a 7 x 1 column vector. In analogy to the GEE for the mean response based model, we define the generalized estimating equation for (4.102) as follows
matrix function of xi. Again, the Newtonwhere G, is some q x Raphson algorithm can be applied to compute the estimate of 8. The generalized estimating equation in (4.104) is known as GEE I1 to differentiate it from GEE or GEE I used for the mean response based model. As in GEE I, the estimating equation estimate 6 obtained from (4.104) is consistent independent of the choice of Gi. Further, 6 has an asymptotic normal distribution (see exercise): h
,
zo= -n1B - ~ E ( G ~ S ~ S T GB-T T)
(4.105)
where B = E (G,Dd). The above still holds true if G, is defined by some parameter vector a and 6 is obtained from (4.104) with a substituted by some &-consistent estimate. A consistent estimate of Co is similarly obtained as in the case of GEE I by substituting consistent estimates of 8, B and E ( G , S , S ~ G in ~ )place of the respective parameters. Although similar, there is still some important difference between GEE I in (4.35) and GEE I1 in (4.104). When G, = DLV,-l in GEE I with D, = & p a , B = E (D,V,- 1D,T ) and the asymptotic variance of the GEE estimate is
1
zB= -nB - ~ E
(4.106)
227
4.5 GEE I1 FOR MODELING MEAN A N D VARIANCE
&
T
For GEE 11, if we set Gi = DiV,-' with Di = (p:, 0:) , the asymptotic variance Eo of still has the above form. However, B = E (FiV,-'D:), where Fi = -&Si (see exercise). Since &Si # E (DiV,- 1Di T). More importantly, the use of Di =
-& (p:, 0:) T , B
-&Si
#
in GEE I1 may
yield inconsistent estimates of 8. Unlike GEE I, Si is not linear in yit and thus &Si generally depends on yit. As a result, Gi = DiV,-' may not be a function of xi only as required to ensure the unbiasedness of the GEE I1 in (4.104). E x a m p l e 3. Consider the LMM in Example 1. It is readily checked that (see exercise)
where
< = (uecT ( D )
,02)'.
Let
where Zi = ( z i l , .. . ,zim)T and &2 ( a )is some (m-1)m (m-1)m working variance matrix parameterized by a. By setting Gi = DiV,-l, it follows from (4.104) that
If a is known, such as by setting I 4 2 2 = I ( m - l ) m 2
(m-l)m,
we can readily solve
2
the above equation to obtain the estimate of 8. Otherwise, a fi-consistent estimate of a is required to use (4.106) for inference about 8. As noted earlier, the asymptotic variance of this GEE I1 estimate of 8 is still given a by (4.106) except for a revised B = E (FiV,- 1DiT ) with Fi = -mSi.
&
and GEE 11s defined by setting In this example, -&Si # (p:, Di = -&Si is no longer guaranteed to be unbiased to yield consistent estimates of 8. Note also that the selection of a block diagonal V, in (4.108) has an advantage to ensure the consistency of estimate of p from (4.109) if only the mean response in (4.99) is modeled correctly, that is, if
228
CHAPTER 4 MODELS FOR CLUSTERED DATA
E (yit I xit) = xLP holds true. If a nonblock diagonal V, is used, the GEE I1 has the following form:
The above GEE I1 may yield biased estimates of P if the variance component in (4.99) is not specified correctly. Example 4. Consider the model in (4.100) from Example 2 with o (pit) = &pit. In this case, sit=(yit-Pit) bi = ( O i l , O i 2 :
2 7
'
,
T ~ i = ( ~ i 1 t ~ i 2 , * * * i 3 ~ i Oit=&Pit m ) T T Oim) = ..
,
c
( 4 , . , 4:)
It follows that (see exercise)
Let Ai = diagt ( 4 f p i t ) and R ( a 1 ) ( I 4 2 2 ( a 2 ) ) be some m x m working correlation (variance) matrix parameterized by al ( a 2 ) . Set
By substituting (4.110) and (4.111) into (4.109), we obtain a GEE I1 for simultaneous inference about /3 and <. In particular, we can use it to formally test for overdispersion, that is, whether 4; = 1 for all 1 5 t 5 m. As in Example 3, the asymptotic normality of the estimate follows from T (4.105) provided that a = (a7,a;) is known or substituted by a &consistent estimate. Note that for this example,
Thus, as in Example 3, Di is a not function of xi only and GEE 11s defined by (4.112) may yield inconsistent estimates of 8.
4.5 GEE I1 FOR MODELING MEAN A N D VARIANCE
229
In both Examples 3 and 4 above, the GEE 11s yields When there is missing data, the approach in Section 4.4.2 for GEE I is readily generalized to the current setting. For each subject i, define a missing data indicator T i t as in (4.61) and let
As a special case, if s = t , we obtain from the above Tit = P r ( T i t = 1 I xi, yi) and A,,, which are defined in (4.81) of Section 4.4.2 in the discussion of WGEE I in the presence of missing data. For the mean response yit - pit, it follows from an argument similar to (4.78) that E [A, (git - pit)] = 0. For the variance component sist, we have for s 5 t ,
Let
where diag(6) denotes the diagonal matrix with the vector S forming the diagonal. As in the case of GEE I, we revise the estimating equation in (4.109) in the presence of missing data as follows: n
wn(8)= C G i A i S i
=0
(4.116)
i=l
The weighted GEE (WGEE) I1 above is unbiased, that is, E (wn (8))= 0, if E ( A i S i ) = 0. Thus, as in WGEE I, the estimating equation in (4.116) yields consistent estimates of 8 if the regression model in (4.102) and missing data indicator model in (4.109) are both specified correctly. Further, the
230
CHAPTER 4 MODELS FOR CLUSTERED DATA
asymptotic results in Theorem 1 of Section 4.4.2 for WGEE I are similarly generalized to WGEE I1 estimates obtained from (4.116) (see exercise). As in WGEE I, we must estimate 7rit in most applications. Under MCAR, ri is independent of xi and yi and thereby
The constant weight function 7rst is readily estimated by the sample moment: 1 ; rist (2 5 s 5 t 5 m). Under MAR, 7rist becomes dependent = A. on xi and yi. As in WGEE I, under MMDP, Tist = 7rit and Aist = ILL Ttt. at for 1 5 s 5 t 5 m. Thus, we can obtain estimates of 7rist by modeling 7rit as discussed in detail in Section 4.4.2. h
7rst =
4.6
xy=l
Structural Equations Models
In treatment studies, it is often of great interest to identify and study mechanisms by which an intervention achieves its effect. By understanding the mediational process through which treatment affects study outcomes, not only can we further our understanding of the pathology of the disease and treatment, but also provide alternative treatments for the disease with efficient use of resources. However, regression analysis is ill-suited for modeling such a causal relationship. The structural equations model (SEM) provides a formal modeling and inference framework for this and other types of causal problems. In this book, we discuss only the classic linear SEM for continuous response. For convenience, we refer to linear SEM as SEM throughout the rest of the book. Note that our intention for this chapter is to provide the necessary background for SEM so that we can discuss distribution-free inference for such models in Section 4.6.3 as well as in Chapter 6. Thus, detailed considerations about many of the important issues, such as latent variables and model identifiability, are not provided. Readers may want to consult other texts such as Bollen (1989) for an in-depth treatment of the theory and application of linear SEM.
4.6.1
Path Diagrams and Models
SEM is a system of linear models linked together to capture the complex and dynamic relationships among a web of variables. Although similar in appearance, SEM is fundamentally different from linear regression. In a regression model, there exists a clear distinction between dependent and independent variables. In SEM, such concepts only apply in relative terms
4.6 STRUCTURAL EQUATIONS MODELS
231
since a dependent variable in one model equation can become an independent variable in other components of the SEM system. It is precisely this type of reciprocal role a variable plays that enables SEM to infer causal relationships. SEMs are best represented by path diagrams. A path diagram consists of nodes representing the variables and arrows showing relations among the variables. SEM, or path analysis, frequently employs latent variables t o represent abstract constructs that are not directly observable, but are measured by multiple variables, or indicators, in the SEM literature. For example, depression is a latent construct in mental health research and popular measures include the structured clinical interview for depression, the Hamilton rating scale and the Beck depression inventory. In a path diagram, latent variables are distinguished from their observed counterparts in convention by using a circle or ellipse rather than the rectangular or square box used for the observed variables. Error terms are not observed, but since they are not part of the causal chain, they are not treated as latent variables. When we want to indicate that a variable is measured with error, we add an error term with an arrow from it to the variable. Arrows are generally used to represent relationships among the variables (observed or latent). A single straight arrow indicates a causal relation from the base of the arrow to the head of the arrow. Two straight single-headed arrows in opposing directions connecting two variables indicate a reciprocal causal relationship. A curved two-headed arrow indicates there may be some association between the two variables. For example, in most path diagrams error terms are not connected, indicating stochastic independence across the error terms. But if we suspect association between them, they should be connected by curved two-headed arrows. The path diagrams in Figure 4.2 depict a mediational process involving a predictor z,a mediator z and a response depending on whether the response is directly observable or not. The path diagram in (a) shows a mediational process with an observable response y, while the one in (b) represents a more complex situation involving a latent response 7 measured by the observable indicator y. The two arrows emanating from z indicate the causal effect of z on both the mediator and response and the single arrow stemming from z defines the effect of the moderator on the response variable. The strengths of the relationships are specified by the Greek letters, yll, y21, p, next to the arrow with the error in relating the variables denoted by eg (g = x,y, 7 ) .
Example 1 ( S E M for mediational process without latent variable). Consider the diagram in (a) of Figure 4.2. The SEM for this
232
C H A P T E R 4 MODELS FOR C L U S T E R E D DATA
Figure 4.2. Path diagrams for a mediational process with a single predictor, mediator and observed response in (a) and latent response in (b). mediation model is given by
Y = 710 + Y l l Z + P z + Ey, E E
(Ey) = 0, (Ez)
= 0,
c o v ( 2 ,Ey) c o v (z, E,)
= 0,
= 720
+7212 + E,
(4.117)
cow ( z ,Ey) = 0
=0
In the above, 2 is called an exogenous variable, while z and y are called endogenous variables, as the latter two are determined by other variables within this SEM. As a special case, if p1 = 0, (4.117) yields a full mediational process with no residual direct effect of 2 on y. Note that in the nomenclature of regression, y is a response variable for the first and z a response variable for the second model equations in (4.117). However, such terms do not apply to SEM since a variable can serve as both a dependent and independent variable, such as z in (4.117). Since a primary goal of SEM is to make causal inference, stochastic independence is not taken for granted. The usual independence assumption is replaced by zero correlation (or covariance), which unlike independence can be empirically checked and validated. For example, to assess the causal effect of II: and z in (4.117), it is critical that neither be correlated with ey in the first equation and that z not be correlated with E, in the second equation. Such a zero-correlation assumption is known as pseudo-isolation in SEhf. Example 2 ( S E M f o r mediational process with latent response). Similarly, by using the path diagram in Figure 4.2 (b) as a guide, we can express the SEM for the mediational process involving a latent endogenous
233
4.6 STRUCTURAL EQUATIONS MODELS
response q as
+ Y l l Z + P z + €7, 2 = 720 + Y2lZ + y = q + 6,
(4.118)
rl = 7 1 0
EZ,
E
(€7) = 0,
( 4= 0,
E (6,) = 0, cow (z, E7) = 0 c o v ( z , € 7 ) = 0, c o v (z, E , ) = 0,
c o v ( q ,6,)
=0
In (4.118), 6, denotes the error in measuring 77 with y, which is assumed t o be uncorrelated with q with mean 0. As illustrated by the next example, the pseudo-isolation assumption plays a critical role to ensure consistent estimation of model parameters. Example 3. In Example 1, assume /3 # 0. Suppose that P z is mistakenly dropped from the first equation in (4.117), that is, y i = a o + a 1zi
(€it)= 0,
E
(EzJ
+EiO, = 0,
zi = 7 2 0
+ 7213% + Ezz
cow (z2,E t z )
(4.119)
=0
By comparing (4.117) and (4.119), it is seen that
If yZl # 0, we have (see exercise):
i and E.T/~ in (4.119) are correlated. Let 61 denote the least In other words, z squares estimate of Q I . It then follows that (see exercise)
= 711
+ PY21
Thus, a1 in the misspecified model in (4.119) does not estimate the direct effect, but rather the (biased) total effects of z on y. Path diagrams not only indicate causal paths, but between-variable correlations as well. For example, the connectors between z1 and z2 in both path diagrams in Figure 4.3 indicate that these exogenous predictors are correlated. Similarly, unlike the error terms in part (a), eyl and in Diagram (b) are correlated.
234
C H A P T E R 4 MODELS FOR CLUSTERED DATA
Figure 4.3. Path diagrams with correlated and uncorrelated error terms.
Example 4. The SEMs for the two path diagrams in Figure 4.3 share the following core components:
The difference between the two lies in the assumptions about the error terms:
In SEN1 analysis, path diagrams are also useful for calculating direct, indirect and total effects of a causal variable. For example, for both diagrams in Figure 4.3, 21 has a direct effect y21 on 92, while 2 2 has a direct effect y12 on 91,and a direct effect yZ2and an indirect effect P21y12on 39. Thus, the total effect of 2 2 on y2 is y22 P21y12. We can also derive these effects through the model form. For example, by substituting the first equation in place of y1 in the second in (4.122), we obtain
+
which gives the direct, indirect and total effects of
22
on 92.
235
4.6 S T R U C T U R A L EQUATIONS MODELS
Example 5 ( S E M f o r linear regression with measurement err o r ) . Consider a simple linear regression model with measurement errors in both the response and predictor: r72
E
= YE2
(€11,) = 0, (Sy,)
= 0,
+ Erl,'
zz = 5,
+
JZ,,
Cov ( E z , E 1 l , ) = 0, Cov (Ez, S Z Z ) = 0,
Yz = 72 + S,,
(4.124)
E(d,,) = 0
c0-0(72,sy,>
=0
where the latent response qz (predictor &) is observed by yz (zz). Note that for convenience, we have set the intercept to 0 in (4.124). In addition, we assume that E ~ , Syz , and S,, are uncorrelated with each other as well as with Ez and qz. Primary interest lies in estimating y. If we ignore measurement error and fit a linear regression based on the observed variables. we have
Yi = ?*Xi + Ey,,
E
= 0,
(Ey,)
c o v (zz, Ey,) = 0
(4.125)
It is readily shown that (see exercise) (4.126) where
0; =
V a r (&) and a:, = Var (S,,).
The ICC,
7
=
A,is also u<+'62
known as the reliability ratio in measurement error models. Unless a:, = 0, that is, E is observed without error, the regression model in (4.125) based on the observed z and y will yield downwardly biased estimates of y. It is interesting to note that measurement error in the response 7 does not create any bias in the estimation of y. Consider the mediation model with no latent variable in (4.117) of Example 1. Let Y = (Yl,Y 2 I T = (Y,Z I T
,
x = ( 5 1 , .dT = (1,.IT
(4.127)
We can then express this SEM in a matrix form as follows: y = BY
+ rx + ey,
E (ey) = 0, cov (x,E , ) = o
(4.128)
In the above, y is a vector consisting of the endogenous variables, while the vector x contains the intercept and the exogenous variable z. The matrices
236
CHAPTER 4 MODELS FOR CLUSTERED DATA
B and r contain the parameters of interest. The matrix form also applies to a general SEM with more endogenous and exogenous variables. More generally, when either or both endogenous and exogenous variables contain latent variables, SEM can be expressed as
where 7 (5)denotes the column vectors containing all endogenous (exogenous) variables and y (x) the observed indicators for the latent endogenous (exogenous) variables. The parameters of primary interest are the matrices B and r. It is important to emphasize that SEM is not a multivariate linear model. For example, consider the SEM in (4.129) with all variables observed, that is, 7 = y and E = x. If y is an rn x 1 vector, (4.129) can be equivalently expressed as: y = (I,
E
(E;)
- B)-
1
(rx+ E , )
= (I, - B)-1 E
(Ey)
=
(I,
rx+ €,;
-~ 1 - l
(4.130)
= 0, cow (x,€j) = cow (x,EY ) = 0.
Since (I, - B)-’r is generally a nonlinear function of parameters, the above is not a multivariate linear model, although the equation is linear in x. Example 6. We can express the SEM in (4.117) from Example 1 in the matrix form of (4.130) as below:
It is clear that the above SEM is not linear in the parameters.
4.6 S T R U C T U R A L EQUATIONS MODELS
4.6.2
237
Maximum Likelihood Inference
First, consider the SEM in (4.128) with all observed variables. The classic approach utilizes the second order moments (or variance-covariance matrix) for estimation. T Let ui = (yT7 , @ = Var (xi)and XD = V a r (eY,). Then, it follows that (see exercise)
xT)
(4.131)
Note that although primary interest lies in B and r, we must also simultaneously estimate the nuisance parameters @ and m in (4.131). Let 8 denote the vector containing all these parameters. If we assume ui N (0,C (O)), then ui has the following density function: N
(4.132) where p denotes the length of the column vector ui. Given a sample of n i.i.d. observations, the log-likelihood function is given by (see exercise): (4.133) where S, denotes the sample variance of ui and tr (.) the trace of a matrix. Aside from technical details, inference about 8 is straightforward based on the asymptotic distribution of the MLE of 8. In addition to maximum likelihood, other alternative inference procedures have also been proposed. For example, the unweighted (ULS) and weighted (WLS) least-squares methods minimize the following objective functions:
where W is some symmetric matrix with known elements. Thus, ULS minimizes the sum of squares of elements in the residual matrix s, - C (O), while WLS minimizes a similar sum with such squared terms weighted differently
238
CHAPTER 4 MODELS FOR CLUSTERED DATA
according to W . For example, by setting W = S,, elements of S,-C ( 6 )are weighted by the corresponding entries of S, in the weighted sum of FLVLS. Example 7 ( S E M for mediational process without latent variable). For the mediation model in Example 1, it follows from (4.127) that (see exercise) : (4.135)
Q=
O
($11
0
),
rWT+Q=
(
$22
$11
+ 7?1422
711721422 $22
711721422
+-dl422
)
Thus, we have (4.136)
Go
+
2PG1+
P2G2 G1+ PG2 0 G2
c (el =
4 2 2 (711
+ PY21)
0
721422
0
0
0
422
I
(4.137)
.
(4.138)
By removing the row and column containing only zeros, we have: Go
c ( 6 )=
+ 2PG1+ P2G2 G1+ PG2 4 2 2 (711 + P721) G2
721422
422
In (4.138), C ( 6 )has six unique elements and 8 = (P, yll, 721,5hZ2,$11, $22)T has six parameters. Thus, the MLE, ULS, and WLS estimates of 6 are all identical and given by the unique solution to the equation C ( 6 ) = S,.
239
4.6 STRUCTURAL EQUATIONS MODELS
Note that the row and column with only zeros in (4.137) are the result of including the intercept terms, ylo and y20,in the model in (4.127), which are not estimated. To avoid such rows and columns in C ( 0 ) at the outset, we can center the SEM in (4.128) on E (xi) to obtain
For example, when applied to this mediation model, (4.127) can be expressed as :
When calculated based on this centered model, C ( 0 ) has the form in (4.138) (see exercise). Now consider inference for the SEM in (4.129) with latent variables. Let @ =
V a r (0>
A, = Var (6,)
j
A,
= V a r (6,) ,
Q = V a r (El))
Let 8 denote the vector consisting of all the parameters. If we assume T N (0,C (e)), then C ( 6 ) has the same partition as in u = (yT,xT) (4.131), but with the individual components given by N
c,, ( e ) = A,@A,T + A,! c,, (e) = A,
c,, ( e ) = A,
(I,
-~ 1 -
(I, - B ) - [email protected],T ~ (4.140) (rwT l + Q) (I, - B ) - T A,T + A,
where rn is the length of y. The density and log-likelihood functions have the same form as those in (4.132) and (4.133). Also, inference proceeds in the same way as before using either the maximum likelihood or least-squares approach. Example 8 ( S E M for linear regression with measurement error). Consider the SEM in (4.124) for linear regression with measurement error in Example 5. Assume that only the predictor E is measured with error by r replicates zij, that is,
+
) ~S ,, = (S,,,, . . . , d,,.)'. Let x = ( x i j .. . , ~ r and d,,, Then, xi = E i l r where 1, denotes an r x 1 vector of 1's. It is readily checked that (see
240
CHAPTER 4 MODELS FOR CLUSTERED DATA
exercise)
giz = V a r (Szz,) and where J , is an m x rn matrix of 1’s) of = V a r (ti), g2= VW ( E ~ , ) . Thus,
+
+
The matrix C (0) has a total of ( r 1) ( r 2) unique elements, which is > 4 only if r 2 2. Thus, we must have at least two replicates t o identify the parameter vector 8 = y,o;,q2)oiz)
(
T
for this model.
Example 9 ( S E M for mediational process with latent response). Consider the SEM in (4.118) from Example 2. Let y1 = y, y2 = z , 2 1 = 1 and x2 = 2 . By expressing the model in matrix form, we have
Thus.
241
4.6 S T R U C T U R A L EQUATIONS MODELS
where
g2
6Yl
= V a r (dyl). It follows that
(4.142)
where Go, GI and G2 have the same interpretation as those in (4.136). By removing the row and column containing only zeros, we can express C (0) as :
c (el =
r
Go
+ 2PG1+ P2G2+ “62
Y1
G1+ PG2
4 2 2 (711
G2
+ PY21)
7 2 14 2 2 422
The above matrix has 6 unique elements, but the vector of parameters, T
0iY,> has
seven parameters. Thus, the model is unidentifiable. As in Example 8, one way to identify the model is to have replicate observations y.
0 =
(P>?11,?21, 4 2 2 , $11, $22,
4.6.3 GEE-Based Inference The problem with the maximum likelihood estimation is the lack of robustness. Although least-squares based alternatives such as ULS attempt to address this issue, they do not consider missing data. By framing the inference problems under GEE, we can address both issues. Consider first the SEM in (4.128) with all observed variables. We can express the model in terms of the mean and variance as follows:
E (y,l x,) = pz = (I, V a r (y,i Xi) = c, = (I,
-
~ 1 - rx,. l
-B
) y a (I,
(4.143) T
-
B)-’
Let 0 denote the vector containing all the parameters in B , I?, and a. Note that since we condition on xi, we no longer need to consider estimating the
242
C H A P T E R 4 MODELS FOR CLUSTERED DATA
variance matrix Q, of xi as in the classic approaches discussed in Section 4.6.2. Since E (yil xi) and V a r (yii xi) both contain the parameters of interest, we can use GEE I1 for inference about 8. As in Section 4.5. let
where oi is an Ci and Si an given by
(m-l)m
x 1 column vector containing the unique elements of x 1 column vector.
The GEE I1 for estimating 8 is
m+l m
matrix function of xi with q denoting the where Gi is some q x length of 8. Inference for 8 follows from the discussion in Section 4.5. In particular, if Gi is parameterized by a , the estimate of 8 that solves the equation in (4.145) is asymptotically normal with the asymptotic variance given in (4.105) if a is known or substituted by a &consistent estimate. Example 10 ( S E M f o r mediational process without latent variable). For the mediation model in Example 1, we have
cz =
(
P2$22
+ $11
&2) $22
=
(%ll
':.12) vi22
Let
where I422 ( a )is parameterized by a and &pi is readily calculated (see exercise). By substituting Gi given in (4.147) into (4.145), we can estimate 8.
243
4.7 EXERCISES h
For example, if setting
( a )= I ( m - l ) mx+ m - - l ) m ,
I422
the estimate 8 obtained
also follows an asymptotic normal distribution given in (4.105). Now, consider the SEM involving latent variables defined in (4.129). Let
ui = (YLX:) Sist
=si.(
si =
-
T
c = V a r (uz) = [Q]
,
, u = vec ( C )
- .t)
a s ) (.it
( s i l l ,si12,. . . , silp, si22, s i 2 3 , .
. . ,s i 2 p , . . . ,s i p p )
T
where p denotes the length of the column vector ui and C has the form in (4.131) with the component matrices C,,, C,, and C,, defined by (4.140). The GEE I1 for estimating 8 is given by: n
n
2=1
i=l
(4.148)
As before, Gi may contain a vector of unknown parameters a. The estimate of 8 obtained by solving the above equation has an asymptotic normal distribution given in (4.105) if a is known or substituted by a fi-consistent estimate. Example 11 (Least squares estimate). Consider the weighted least squares estimate (WLS) of 8 for the SEM model in (4.129) obtained by minimizing the objective function FWLS in (4.134) with some weight matrix W . Let w = v e c ( W - ' ) denote x 1 column vector containing the x diagonal unique elements of W and V-l = diag(w), the matrix with w forming the diagonal. Then, by setting Gi = ($0)Y-l, we obtain from (4.148) the same WLS estimate of 8 as defined by FWLS in (4.134) (see exercise). Inference under missing data follows from the discussions in Sections 4.4 and 4.5 for parametric and distribution-free models. The procedures are again straightforward for parametric models under both MCAR and MAR. For distribution-free inference, we can apply WGEE 11. Under MAR, if the weight function is unknown, the feasibility of applying such an approach depends on whether the weight function can be estimated.
v
4.7
9
Exercises
Section 4.2 1. Consider Cronbach's alpha defined in (4.4). Show that for a standardized instrument, that is, all items have the same standard deviation, or
244
CHAPTER 4 MODELS FOR CLUSTERED DATA
C = Im, cv = 1 if all items are perfectly correlated and Q = 0 if all items are stochastically independent. 2. Verify (4.11) for the LMM defined in (4.10). 3. Verify (4.16) for the LMM defined in (4.15). 4. Let y follow the Poisson distribution:
I
fP (Y P ) =
Puy
7exp (-4 , p > 0, Y = 031, Y*
Let b be a random variable such that exp ( b ) follows a gamma distribution with mean 1 and variance a. Consider a mixed-effects model: y 1b
N
Poisson ( p ) , log ( p ) =
+ b,
y = 0 ’ 1 , .. .
Show that y has a marginal negative binomial distribution N B ( p ,a ) with the following density function:
p>o,
a>0,
y=O,l,..
5. Let M be some p x p full rank constant matrix and a some p x 1 column vector. Show (4 &loglMl = ( MT ) -1 * (b) (aTM-’a) = -hl-laaTn/;r-l. 6. Consider the multivariate normal linear regression model in (4.21) with p, = &,El. (a) Verify the likelihood and log-likelihood expressions in (4.22). (b) Use the identities in Problem 6 and those regarding differentiation of vector-valued function discussed in Chapter 2 to verify the score equation in (4.23). (c) Show that the estimates of and 6 in (4.24) are the maximum likelihood estimates of ,El and a. 7. Consider the LMM defined in (4.10). Show that
&
3
Yz
I xt,zt
N
(X,P,K= (Z,DZ? + .;Im%))
8. By using the normal random effects, extend the generalized logit and proportional odds models for nominal and ordinal response to a longitudinal study setting. 9. Consider the Cronbach’s alpha defined in (4.4).
245
4 . 7 EXERCISES
(a) Find the MLE G of a. (b) Show that the asymptotic variance u i of Ei is given by
ua =
+
2m2 [ (1;@lm) (tr (a2) tr2 (@)) - 2tr (a)
(~AQ~I.~)]
(m - 1) (1,@1rn)3
where tr (.) denotes the trace of a matrix. [Hint: See van Zyl et al., 2000.1 Section 4.3 1. Assume that the distribution-free model in (4.28) is specified correctly. Show (a) (Yi - P i ) = 0. (b) E (w,) = 0. 2. Show that at convergence, both recursive equations in (4.38) yield the GEE estimate of p. 3. Consider the repeated measures ANOVA in Example 6. Show (a) j& satisfies the GEE defined in (4.40) under the working independence model. (b) c k is the MLE Of P k if y k i N (0,x k ) for 1 5 k 5 9. 4. Consider the GEE estimate in (4.43) for the distribution-free linear model defined in (4.41) and assume y z 1 X i N ( X $ , u2R( a ) ) .Show (a) is the MLE of ,kJ if a is known. (b) is the MLE of p if a is unknown and estimated by (4.44). 5. Verify the Frechet bounds in (4.46) for the distribution-free model (4.45) with a binary response yit. [Hint: see Prentice, 1988.1 6. Verify (4.50), (4.53) and (4.56) in the proof of Theorem 1. 7. Use the asymptotic normality of the score-like vector w, (p)in (4.51) in the proof of Theorem 1 to show that the GEE estimate is consistent regardless of the choice of Gi. 8. Derive the Frechet bounds for the correlation matrices R j k in (4.60) for the generalized logit model defined in (4.58). Section 4.4 1. Consider the general LMM defined in (4.10). Under MAR, find (a) the observed-data log-likelihood Z1 (6,) in (4.65). (b) the MLE of p. 2. Show that under MMDP MAR is equivalent t o the available case missing value restriction condition in (4.68). [Hint: See Verbeke and Molenberghs, 2000.1 3. Extend the available case missing value restriction condition for the linear regression model in (4.69) to a longitudinal study with three assessment points.
B
N
B
N
3
B
246
C H A P T E R 4 MODELS FOR CLUSTERED DATA
4. Consider the estimate G2 of p 2 defined in (4.74). Show (a) is consistent under MAR, that is, if (4.73) holds true. (b) i7i2 is consistent if missing data is non-ignorable, that is, if 7 r in ~ (4.76) is a function of both yil and y i 2 . (c) Verify (4.78). 5. Consider the WGEE I defined in (4.82). Show (a) E (wn ( p ) )= 0,if E (AiSi) = 0. (b) E ( A i S i ) = 0 if the models in (4.80) and (4.81) for regression and ri are both specified correctly. 6. Prove Theorem 1 by employing an argument similar to the proof of Theorem 1 in Section 4.3.2. 7 . Verify (4.88), (4.89) and (4.90). [Hint: For verifying (4.89), note
c2
-1
+
that fi(7- 7) (-&$un) fiun o p (I).] 8. Consider the model in (4.93). (a) Show that E [ ~ k i (2y k i 2 - pki2)]= 0 (1 5 k 5 9 ) . Thus, the estimating equation in (4.94) is unbiased. (b) Let 7rki2 = E ( r k i 2 I ykil) and G k be defined in (4.95). Show
and
E E
(7rki2YkiZ) = P k E ( 7 r k i 2 Y k i l )
( 7 r k i 2 Y k i l Y k i 2 ) = P k E (.ki2Y;il)
+ -rkE ( 7 k i 2 ) + T k E (7rki2Ykil)
(c) Use the results in (b) to show G k -tPO k . (d) Assume Pk = /3 (1 5 k 5 9 ) . Show the GEE below defines a consistent estimate of e = (71,.. . , T g , /3klT:
(e) Find the estimate G by solving the equation in (d). Section 4.5 1. Let 6 denote the estimate of 8 obtained from the GEE I1 defined in (4.104). By employing an argument similar to the proof of Theorem 1 in Section 4.3.2, show (a) G is consistent regardless of the choice of Gi.
247
4.7 EXERCISES
(b) 6 has the asymptotic distribution in (4.105) if a is either known or substituted by a &-consistent estimate. (c) B = E (F,K--'DT) if G, = D,V,-' with D, = (p:, u, T)T.
6
2. Consider the distribution-free model defined in (4.101) with ptt (1
+ AtpZt). Discuss inference for 8 = (pT,AT)
T
with X = (XI,
0
(ptt) =
. . . ,)A,
T
using GEE 11. 3. Verify (4.107), (4.109), (4.110) and (4.112). 4. Consider the LMM for therapists' variability in (4.12) of Section 4.2.2. It follows from (4.12) and (4.13) that (Ytj
VdEre
I z%)= pt
sajk
(Ytj
= BO
+ 2zP1,
E
(%jk
- pt) ( Y z k - &)> 0$ =
1 2%)= O&jjk
0:
+
+ &p ff2
g 2 1
(1 - d j k )
and
p = ff$Dz
= 1 if
6,k T
and 6,k = 0 if otherwise. Discuss inference about 8 = (o$, p ) using GEE 11. 5. Consider the WGEE I1 defined in (4.116). Show (a) E (wn ( p ) )= 0 , if E (&St) = 0. (b) E (AJ,) = 0 if the models in (4.102) and (4.113) for regression and r, are both specified correctly. 6. Within the context of Problem 5 , generalize the asymptotic results in Theorem 1 of Section 4.4.2 for WGEE I to WGEE I1 estimates obtained from (4.116). Section 4.6 1. Verify (4.120) and (4.121). 2. The SEM for a multiple linear regression with two predictors is by
J = tk
Y = Yo + 71x1 + Y 2 2 2 + Ey E
(Ey) =
0,
cow (21, E y ) = 0.
c o v (XI, Ey)
=0
For convenience, we have set the intercept t o 0 in the above model. Suppose that 2 2 is mistakenly dropped, that is, y = a0
+ a121 +
E;,
E
(E;)
= 0.
cov
(21, E ; )
(a) Show
(b) Let &I be the least-squares estimate of 01. Show
#0
248
CHAPTER 4 MODELS FOR CLUSTERED DATA
Thus, unless 21 and x2 are uncorrelated, that is, Cov ( 2 1 , z2) = 0, GI is not a consistent estimate of yl. 3. Consider the simple linear regression model with measurement errors in (4.124) and the model based on the observed variables in (4.125). (a) Show 2
c o v (E, 7 ) = 7 4 2 , cow ( 2 ,Y) = 7 4
2
= 7*%
(b) Use (a) to confirm (4.126). 4. Verify (4.131), (4.132) and (4.133). 5. Verify (4.135) and (4.136). 6. Find the variance matrix C (8) = Var (y, z , x) for the centered model in (4.139) and show that it is given by (4.138). 7. Verify (4.141) and (4.142). 8. Find Di = &pi in (4.147) by using the expressions for pit and vist in (4.146) (1 5 s < t 5 2). 9. Verify the claim in Example 11 that with Gi = (&c)V,-' the GEE I1 in (4.148) yields the same weighted least squares estimate 8 as defined by FWLS in (4.134).
(
r>
Chapter 5
Multivariate U-Statistics In Chapter 3, we introduced univariate U-statistics and studied the properties of such a class of statistics, especially with respect to their large sample behaviors, such as consistency and asymptotic distributions. We also highlighted their applications to an array of important problems including the classic Kendall’s tau and the Mann-Whitney-Wilcoxon rank based statistics as well as modern applications in comparing variances between two independent samples and modeling areas under ROC curves between two test kits. The focus of this chapter is twofold. First, we generalize univariate U-statistics and the associated inference theory to a multivariate setting so that the classical U-statistic-based statistics can be applied t o a wider class of problems requiring such multivariate U-statistics. Second and more importantly, we want to extend the U-statistics theory to longitudinal data analysis. As highlighted in Chapter 4,longitudinal study designs are employed in virtually all scientific investigations in the fields of biomedical, behavioral and social sciences. Such study designs not only afford an opportunity to study disease progression and model changes over time, but also provide a systematic approach for inference of causal mechanisms. In Chapter 4, we discussed the primary issues in the analysis of data arising from such study designs and how to address them for regression analysis, particularly within the distribution-free inference paradigm. As in Chapter 2, these models are primarily developed for modeling the mean, or first order moment, of a response variable. Many quantities of interest in applications also involve higher order moments and even functions of responses from multiple subjects, such as the product-moment correlation and Mann-WhitneyWilcoxon rank based tests. Although GEE I1 may be applied to address 249
250
C H A P T E R 5 MULTIVARIATE U-STATISTICS
second order moments, models and statistics defined by multiple subjects, such as Kendall’s tau and rank based tests, are amenable to such treatment, as we have seen in Chapter 3. By extending the univariate U-statistics theory to a longitudinal data setting, we will be able to apply such classic statistics to modern study designs. In addition to generalizing the univariate U-statistics models in Chapter 3 to longitudinal data analysis, we also introduce some additional applications, particularly those pertaining to reliability of measurement. As discussed in Chapter 4, measurement in psychosocial research hinges critically on the reliability of instruments designed to capture the often latent constructs that define a disorder or a condition. Internal instrument consistency and external rater validity are of critical importance for deriving reliable outcomes and valid study findings. Most available methods for such reliability assessments are developed based on the parametric modeling paradigm. As departures from parametric distributions are quite common, applications of such methods are liable for spurious results and misleading conclusions. In Section 5.1, we consider multivariate U-statistics arising from crosssectional study designs and develop the asymptotic theory for such statistics by generalizing the univariate theory of Chapter 3. In Section 5.2, we apply multivariate U-statistics and the inference theory to longitudinal study data. Within longitudinal data applications, we first consider the complete data case in Section 5.2.1, and then address missing data in Sections 5.2.2 and 5.2.3.
5.1 5.1.1
Models for Cross-Sectional Study Designs One Sample Multivariate U-Statistics
As discussed in Chapter 3, many problems of interest in biomedical and psychosocial research involve second and even higher order moments, which are often defined by multiple correlated outcomes of different constructs and/or dimensions. Thus, even with a single assessment point, such models involve multiple correlated outcomes that are generally different from the focus of regression analysis, since the latter is primarily concerned with the mean of a single-subject response. Although regression may sometimes be used to model the quantity of interest, it is at best cumbersome. We illustrate this key difference and thereby motivate the development of the theory of multivariate U-statistics and its applications by considering the product-moment correlation.
251
5.1 MODELS FOR CROSS-SECTIONAL STUDY DESIGNS
Example 1 (Pearson correlation). Consider a study with n subjects and let zi = (xi,yi)T be a pair of variables from the ith subject (I 5 i 5 n ) . Let 2 2 osy = c o v (Xi,yz) , os = V a r (Zi) , oy = V a r (yz)
z,
For the product-moment correlation between xi and yi, p = Corr (xi,yi) = the Pearson estimate of p is defined by
Although the above estimate can be obtained from a linear regression model relating yi to xi or vice versa, inference still requires separate treatment since the theory of linear regression does not provide inference for p (see exercise). To develop distribution-free inference for 2using multivariate U-statistics, let 1 1 2 (xi- Z j ) (5.2) hl (Zi,Zj)= 2 (xi- Xj) (yz - y j ) , h2 (Z2,Zj)= -
As we discussed in Chapter 3, the symmetric kernel hl (zi,zj) gives rise to a univariate U-statistic that provides an unbiased estimate of the covariance Cov (xi,yi) between xi and yi. Although useful for testing whether xi and yi are correlated, it does not provide a measure for the strength of the association. Now consider a vector statistic defined by
The above has the same form as a one-sample, univariate U-statistic except that h (zi,zj) is a vector rather than a variable. In addition, each component 81,of 8 is a univariate U-statistic, estimating the corresponding component of the parameter vector 8; 81 estimates osy while 192 and 0 3 are unbiased estimates of oz and 0:. h
h
h
h
h
It is readily checked that
2=
'l
Thus, if we know the asymptotic
distribution of 6, we can use the Delta method to find the asymptotic distribution of 2. However, as 6 is a vector statistic, the theory for univariate
252
CHAPTER 5 MULTIVARIATE U-STATISTICS
U-statistics discussed in Chapter 3 does not apply and must be generalized to derive the joint distribution of G. Note that in this particular application, we can also apply GEE I1 for inference about 2 (see exercise). In this case, we first model both the mean and variance:
where s: = (xi- p z )2 and oz = E (szy) if II: = y. The model parameter, 2 2 T C = ( p Z ,p y ,oz, oy,ozy) , can be estimated by GEE 11. We can then apply the Delta method to find the asymptotic distribution of = cz based
&
on the WGEE I1 estimate (see exercise). Further, we can define a GEE I1 so that it yields the Pearson estimate in (5.1). Before proceeding with more examples, a formal definition of a multivairate U-statistic vector is in order. Consider an i.i.d. sample of p x 1 column vector of response yi (1 5 i 5 n). Let h (yl, . . . , y,) be a symmetric vector-valued function with rn arguments. We define a one-sample, rn-argument multivariate U-statistic or U-statistic vector as follows:
where C z = { ( i l , . . . ,im) ; 1 5 il < . . . < ,i 5 n} denotes the set of all distinct combinations of rn indices ( 2 1 , . . . , im) from the integer set { 1 , 2 , . . . , n}. As in the univariate case, U, is an unbiased estimate of 8 = E(U,) = E (h,) * In Chapter 3, we discussed two measures of association, the Goodman and Kruskal’s y and Kendall’s 7 b , for two ordinal outcomes. Both measures are defined based on the concept of comparing probabilities between concordant and discordant pairs, but are normalized to range from -1 to 1 so that they can be interpreted as correlations like the product-moment correlation for association between two continuous outcomes. We developed a U-statistic as an unbiased estimate of the difference between the concordant and discordant pair probabilities and used it to test the null of no association between two ordinal categorical outcomes. In the next examples, we apply multivariate U-statistics to develop estimates of Goodman and Kruskal’s y and Kendall’s T b .
5.1 MODELS FOR CROSS-SECTIONAL STUDY DESIGNS
253
Example 2 (Goodman and Kruskal’s 7). Consider an i.i.d. sample with bivariate ordinal outconies z, = (u,,t~Z)T (1 5 z 5 n ) . Suppose u,has K and v, has M levels indexed by k and rn, respectively. For a pair of subjects z, = ( / ~ , r nand ) ~ zII= ( / ~ ’ , r n ’ )concordance ~, and discordance are defined as follows:
(Zz, z3)
I
concordant if u,
< (>)uII,v, < (>)vII
discordant if u,> (<) u J , 21% < (>) vJ neither
if otherwise
Let p , and p d p denote the probability of concordant and discordant pairs. The Goodman and Kruskal’s y is defined by y = ~ : ~ ~ ~ d k p Let p.
hl (zz, 4
= ~{(uz-uJ)(v2-vJ)>o}> h2 (zz, z3) = I{(u2-uJ)(v,--21J)
(5.5)
h (zz, Z J ) = (hl,h2IT where I{ 1 denotes a set indicator. Then, since (l{(uz-uJ)(vt-vJ)>O))
pcp =
Pdp =
7
(1{(u2-uJ)(vt-vJ)
it follows that the bivariate U-statistic,
is an unbiased estimate of 8 = = ( ~ ~ ~ , p d p Thus, ) ~ . as in the case of the product-moment correlation, once we determine the asymptotic distribution of 6, we can use the Delta method to find the asymptotic distribution 9 = 81-82 for inference about y. &+02 Example 3 (Kendall’s T b for ordinal outcome). Within the setting- of Example 2, consider Kendall’s T b . Unlike Goodman and Kruskal’s y, Kendall’s Tb normalizes pcp - p d p by using a different scaling factor, Pcp -Pdp J(1 --pu)(l -pv), that is, T b = , where -
A
J
P,
b
U
= P r (ui= uj),
.
)(l-Pv)
p , = P r (ui= vj)
(5.7)
254
C H A P T E R 5 MULTIVARIATE U-STATISTICS
T is an unbiased estimate _. of - 8 - = ([,pu,pv) . As in the earlier examples, we estimate T b by ?b = and apply the Delta method to determine
-4
its asymptotic distribution from that of 5. Multivariate U-statistics are usually defined by vector-valued outcomes yz rather than scalar variables. As order statistics are not well defined for random vectors, most of the results and properties of univariate U-statistics derived based on order statistics in Chapter 3 are not generalizable to a multivariate setting. However, the concept of projection can be extended to our current setting along with the asymptotic relationship between a U-statistic and its projection. Thus, as in the univariate case, we can characterize asymptotic distributions of multivariate U-statistics by their projection counterparts. Consider the general one-sample U-statistic vector defined in (5.4). Define the projection of Un by n
cn=):E(Un
(5.8)
lyz>-8(n-1)
z= 1
Compared to projections defined for univariate U-statistics, the multivariate projection Gn above has exactly the same form except for the obvious difference in dimensionality. Analogously, the centered projection U, - 8 has a similar form: h
.
?here hi (yi) = E (hl (yl, . . . ym) 1 y1) and hl (yl) = hl (yl) - 8. As Un is a sum of i.i.d. random vectors, an application of LLN and CLT
5.1 MODELS FOR CROSS-SECTIONAL STUDY DESIGNS
255
immediately yields
As in the univariate case, the next step is to show that 6,and U,-have the same asymptotic distribution so that the limiting distribution of U, above can be used for inference about 6 = E(U,). We prove this and state the fact formally in a theorem below.
Theorem 1 (Multivariate one-sample).
Let C h = Var [hl (yl)] = J
[Ki
(YI)
h r (yi)].
Then, under mild regularity conditions,
Proof. The proof follows the same steps as in the univariate case. First, write
&(u, - e) = 6 (6,- 6) + & (u, -6,)= Then, as in the univariate case, we show e,
-+p
(en- 6 ) +en 6) Again, as in
0 so that &(Un
U, - 6 have the same asymptotic distribution. () the univariate case, we prove e , 0 by showing that E (eLe,) and
& L
+p
-
-+ 0
(see
exercise). Lemma. For 1 5 1 5 m, let
Then, we have
m
1=0
=
Proof of Lemma.
Let
m2 -VW n
( k l ,.
m-1
(5.10)
+
(hi ( ~ 1 ) ) 0 ( n - 2 )
. . , ki) be
the indices that are common to
256
CHAPTER 5 MULTIVARIATE U-STATISTICS
As in the univariate case, the number of distinct choices for two such sets having exactly 1 elements in common is Thus, it follows that
(k)(7)(x-y).
Also, as shown in the proof of the univariate case, (5.13) It follows from (5.11) and (5.12) that
where 0 (.) is the vector version of 0 (.) (see Chapter 1 for definition and properties). Now, consider proving E (e:e,) -+ 0 for the theorem. First, we have
E
(eae,)
= nvar
u,, 6, + n v a r (u,). (-U , 1 - 2 n ~ o v0
As in the univariate case, we calculate each term above. Since U, is a sum of i.i.d. random terms, it follows immediately that
(z)2 n
nVar
(6,)
i=l
Var (hi (yi))= m2Var (hl (yi))
(5.15)
5.1 MODELS FOR CROSS-SECTIONAL S T U D Y DESIGNS
If i E
(ii,
. . . , im),
this term equals:
Since for each i the event {i E ( i l , . . . , im)} that
ncov
257
(un,Cn)= e m ( ; ) - 1 (
occurs
(krt)times, it follows
m- 1l ) V u r ( h l ( y l ) )
(5.16)
i=l
= nm(;)-'(
m- 1' ) V u r (hl ( y l ) )
= m2Var(hl ( y l ) )= m 2 C h
Thus, it follows from (5.10), (5.15), and (5.16) that
E (.:en)
+
+ o (n-2) -fP o
=m 2 ~ h 2 m 2 ~ h m 2 ~ h
Example 4. Consider again the U-statistic vector for the prodoctmoment correlation in Example 1. In this case, m = 2 and it follows from Theorem 1 that
To find Cg, let
258
C H A P T E R 5 MULTIVARIATE U-STATISTICS T
where p = ( p x , p y ) . Then, it is readily checked that (see exercise)
Thus,
hl(Z1) = hl
1
( a )- 8 = -2 [Pl (z1, p ) - 81
(5.18)
It follows that
Ce
=~
(-
V U hTi
)
(ZI) = V
U(pi ~(~1))
Under the assumption that zi is normally distributed, Var (p1 (z1)) can be expressed in closed form in terms of the elements of the variance matrix of zi (see exercise). Otherwise, a consistent estimate of Ce is given by:
(Fx,
Fx
T
where @ = Py) with (Py) denoting some consistent estimate of px ( p y ) ,such as the sample mean. By applying the Delta method, we obtain the asymptotic distribution of h
P:
&(?
- p) +p
N
For multivariate normal
(0,0:
=D
zi, 0; =
(el CODT ( 8 ) ) , n
--+
00
(5.19)
2
(1 - p 2 ) (see exercise). Otherwise, we
can estimate 0; by: 2; = D In the above example, we estimated the asymptotic variance of the Ustatistic vector by first evaluating the asymptotic variance in closed form and then estimating it by substituting consistent estimates in place of the respective parameters. Alternatively, we can estimate the asymptotic variance
259
5.1 MODELS F O R CROSS-SECTIONAL S T U D Y DESIGNS
without evaluating the analytic form by constructing another U-statistic as discussed in Chapter 3 for the univariate case. We illustrate this alternative approach for the Goodman and Kruskal’s y index discussed in Example 2. Example 5 . Consider again the U-statistic vector defined in (5.6) for estimating the numerator and denominator of the Goodman and Kruskal’s y in Example 3. The kernel vector is
It follows that
By applying the law of iterated conditional expectation, we have:
+ 8BT
=E
(hl (z1) h:
=E
[E (h(zl, z2) I zl) E (hT(zl: z2) I zl)]
(81)) -
2BE (h:
[E (h z2) hT z3) I = E [h (zl, z2)hT(zl, z3)] - BeT =E
(z1,
(ZI,
(z1))
zl)]
-
-
2eeT
+ BeT
€JOT
To estimate Ch, we must estimate +h = E [h (z1, z2) hT (z1, z3)] and 8. For 8, we can use the U-statistic in (5.6). To estimate ah, we can construct another multivariate U-statistic. Let g ( Z I , z2: z3) = h (21,22)hT ( z l , z3) and g (z1, z2, z3) be a symmetric version of g (z1, z2, z3). For example, given the special form of h (z1, z2) hT (z1, z3), the following is such a symmetric function: 1
ii(21,552,z3) = -3 (g (z1: z2, z3) + g (z2, z1, z3) + g (z3, z2, z1)) Then, the U-statistic vector,
is a consistent estimate of
ah.
260
CHAPTER 5 MULTIVARIATE U-STATISTICS
21
Table 5.1. Classification of two categorical variables with cell counts and cell proportions p k ,
nk,
In Chapter 3, we discussed a U-statistic based approach to test independence between two binary outcomes. In the next example, we generalize this approach to consider testing independence between two categorical outcomes. Example 6 ( T e s t f o r independence between two categorical outcomes). Consider two categorical variables, u and v. Assume u has K and 'u has M levels indexed by r k and qm, respectively. Denote the joint cell counts (probabilities) by n k m (pk,) and the marginal cell counts (probabilities) by n k + and n+3 ( p k and p J ) . The different combinations of the two categorical outcomes are typically displayed in a contingency table as in Table 5.1. A primary interest is in testing the stochastic independence between the two variables. Chi-squares or Fisher's exact tests are often used to test such a null hypothesis. In this example, we describe a similar asymptotic test based on multivariate U-statistics. Let xzk
Yz
~ { z L , = T ~ } ,
Yzm
T
(Yzl,. . . > Y z M )
= I{v,=q,},
Pk = E (xzk)
, xz =
. . ,x z K )
(221,
*
,
P m = E (Yzm) T
T 7
zz = (XTj Y:>
Then, we have: d k m = E ( Z z k Y z m ) - E ( x z k ) E (Yzm) = P k m - P k
Pm
Thus, u and 'u are independent iff the null hypothesis, HO : d k m = 0 (1 5 k 5 K , 1 5 m 5 M ) , holds true. Note that since
5.1 MODELS FOR CROSS-SECTIONAL STUDY DESIGNS
261
there are only ( K - 1) ( M - 1) independent 6 k m value. Let 1 hkm (Xik~Zjk? Yim, Y j m ) = - ( Z i k - Z j k ) (Yim - Yjm) 2 h ( Z i , Zj) = (hll,. * . > hl(M-l)? h21, * ' * > h(K-q(M-1))
6 = (611,. . . , J l ( M - l ) ,
621,. . . >6(K-l)(M-l))
T I
where 6 contains the ( K - 1) (Ad- 1) independent 6km and h (zi, z j ) the corresponding h k m . The null hypothesis of independence between u and w corresponds to the condition E (h (zi, z j ) ) = 6 = 0. Thus, we can use the following U-statistic defined by the kernel h (zi, zj) to test the null hypothesis:
"=(;)-'
h(zi,zj) (i,j)EC,n
By Theorem 1,
2 has the following asymptotic distribution:
To find Cg, note that h (zi, z j ) , we have
h l (z1) =
[hkm (z1, z2)
E [h (z1, z2) 1
1 5511 = ZE [(Zlk
-
I el].
For a component
Z 2 k ) (Ylm - Y2m)
- 1 - - [ ( Z l k - P k . ) (Ylm - P.m)
2
It follows that the kmth component
Ihl,km
hl.km (z1) =
[hkm (Zl, z2)
1 = - [(Zlk 2
(z1) of
hkm
of
I z13
+ Skm]
hl (z1) is given by
I Zl] - S k m
- P k . ) (Ylm - P.m) -
&4
Let % , k m = ( Z i k - P k . ) (Yim - P.m) ?
42 =
( % , l l ,*
*
.
7
%.l(M-l), %21?
T
. . . ,%,(K-l)(M-l))
Then, we can express the asymptotic variance as Cg = V u r (41).A consistent estimate of Cg is given by:
262
CHAPTER 5 MULTIVARIATE U-STATISTICS
An advantage of the U-statistics derived test is that when the null is rejected, we can perform post hoc analyses t o see which cells contribute to the rejection of Ho. Since
6 k m is interpreted as the covariance between Z i k and yim. Thus, relatively larger estimates of b k m are indications of likely dependence for the corresponding cells. Instrument reliability is a broad issue affecting scientific and research integrity in many areas of psychosocial and mental health research, such as in assessing accuracy of diagnoses of mental disorders, treatment adherence and therapist competence in psychotherapy research, and individuals’ competence for providing informed consent for research study participation in ethics research. In Chapter 4, we discussed intraclass correlations for assessing interrater agreement or test-retest reliability for continuous outcomes. For discrete outcomes, different indices are used, with Kappa being the most popular for comparing agreement between two raters. Example 7 (Interrater agreement between two categorical outcomes). Consider a sample of n subjects to be classified based on a categorical rating scale with G categories: q ,7-2, . . . , rG. Two judges independently rate each of the subjects based on the rating scale and their ratings form a total of G2 combinations, resulting in a G x G contingency table. For example, if the ratings from raters 1 and 2 are identified as the variables u and in Example 6, the joint distribution of their ratings can be displayed in a table similar to Table 5.1 with n k m (&m) denoting the number of subjects (proportions) classified in the kth and mth categories by the first and second raters. The marginal cell counts n k + (n+,) and proportions (?zm) describe the number (proportions) of subjects classified by the first (second) rater according the rating categories. The widely used Cohen’s weighted kappa statistic 6 for two raters is defined by
(5.20) where the weight W k m is a known function with 0 5 W k m 5 1. Note that p l k p 2 m is the proportion under the assumption of independence between the two judges. As a special case, if w k m = 1 for k = q and 0 otherwise, the h
h
5.1 MODELS FOR CROSS-SECTIONAL STUDY DESIGNS
above yields the unweighted
263
K:
(5.21) Thus, the unweighted K requires absolute between-rater agreement, while the weighted version allows for between-rater difference weighted by W k m . To define the population K , let Y ~ I= ~ C I{uz=Tk}, Y22 = (Y221, * * .
E
(Yz2m) = P 2 m ,
~ 2 2 m= I{ut=rm}, 1
T Y22G)
Pkm =
E
)
yZl= ( ~ t 1 1 , .. . , ~
YZ = (YA, Y A )
2
1
(5.22) ~ ) ~
T
E
(Yzlk) =Plk
(YzlkY22m)
Then, we can represent the observed proportions in the contingency table using the random variables y i l k and yiam as follows: .
n
.
.
n
Thus, in light of (5.21), the population
K
n
is defined by (5.24)
To express i? as a function of U-statistics, let G
c=
G <2 =
ci =
521T >
1-
c
G
CC
Wkm ( ~ k m ~ 1 k ~ 2 ,m )
(5.25)
k = l m=l
G
WkmPlkP2m
k=l m = l
Then, the population kappa is a function of
C:
=
=f
(C). Let (5.26)
I
k=l m=l
264
C H A P T E R 5 MULTIVARIATE U-STATISTICS A
Then, 5 is a bivariate one-sample U-statistic with kernel h (yi,yj) = uij. Further, it is readily checked that E = (see exercise). Thus, 2 = A
f
is a consistent estimate of It follows from Theorem 1 that
(2)
=
c (3
c
K.
By the Delta method, we obtain the asymptotic distribution of 2: (5.28) h
Given a consistent estimate of Cc, we immediately obtain a consistent estimate of a: by Z: =
& f (2) 2, (4-f(e)>r.
To find a consistent estimate of Cc, note that (see exercise) (5.29)
Thus, by Slutsky’s theorem, we obtain a consistent estimate of Ec:
h
where E (uij I yi) denotes the estimated E (uij I y i ) in (5.29) obtained by substituting j?km, j?lk and j?zm in place of the respective parameters. In Chapter 4, we introduced the product-moment based intraclass correlation as a measure of overall agreement over multiple raters, but did not discuss its inference because of the lack of theory to determine the joint asymptotic distribution of multiple correlated Pearson correlations. We are now in a position to tackle the complexity in deriving this joint distribution.
265
5.1 MODELS FOR CROSS-SECTIONAL S T U D Y DESIGNS
Example 8 (Product-moment based I C C ) . We consider a setting with n subjects and M judges. Let yik denote the continuous rating from the kth rater for the ith subject. Let pkm = Corr (yik, y i m ) denote the product-moment correlation between the kth and mth judges' ratings (1 I k < m 5 M ) . The product-moment based ICC is defined by averaging all such pairwise correlations: (5.31) For inference about pPMICC,we can apply the Delta method once we have the joint distribution of estimates of Pkm. For each pkm, we derived the asymptotic distribution of the Pearson estimate in Example 4. Here, we generalize this approach to find the joint asymptotic distribution of such estimates. T Let zi,km = (yik, yim) be a bivariate variable containing the ratings for the ith subject from the kth and mth observers (1 5 k < m 5 M ) . Let zi =
P= Okm =
e=
f (Okm) = q e ) = (f ( 0 1 2 ) f ( 8 1 3 ) > . > f ( e l M ) > 7
* *
..> f
T
(@(M-l)M))
Then, p = f ( 8 ) . As Example 4, we now construct U-statistics based estimates e of 8 so that f 8 yields the Pearson correlation for each component
(-1
h
pkm of p, and determine the asymptotic distribution of 8 using the results in Theorem 1. Let
266
C H A P T E R 5 MULTIVARIATE U-STATISTICS
Then, E [h (zi,zj)] = 8 and thus the following U-statistic is an unbiased estimate of 8:
By Theorem 1, the asymptotic distribution is given by
6(5 - 8) +d
N (0,EQ), XQ
= 4VUr 1'(
To find Cg, let us first look at the kmth component Let
(81))
hl,km
(zi) of
(5.32)
hi (zi).
It follows from (5.18) in Example 4 that
Let
Then, it follows from ( 5 . 3 3 ) that the asymptotic variance of 5 in ( 5 . 3 2 ) is given by, %3 = 4Var = V a r (pi (zi, p , ) ) . A consistent estimate is given by
Note that G, above denotes any consistent estimate of p,, such as the sample moment. From ( 5 . 3 2 ) , we immediately obtain the asymptotic distribution of = by applying the Delta method. By expressing the estimate of pphr1cc
26 7
5.1 MODELS FOR CROSS-SECTIONAL STUDY DESIGNS
we also obtain by the Delta the asympin (5.31) as &nlIcC = liM(M-llp, T totic distribution of jjPnlICCfor inference about this product-moment-based ICC. As seen from the development in Example 8, inference based on the product-moment based ICC is quite complex. Primarily for this reason, the alternative definition based on linear mixed-effects model is more widely used in mental health and psychosocial research. We discussed inference for this model based index within the context of linear mixed-effects model in Chapter 4. As linear mixed-effects models place strong distribution assumptions, such as normality on both the latent effect and model error, parametric inference often yields unreliable results. We now consider a distribution-free approach to provide more robust inference about this model based ICC. Example 9 (Linear mixed-eflects model based I C C ) . As in Example 8, let y,k denote the rating of the kth observer on the ith subject (1 < - k < M , 1 5 i 5 n). Consider the following normal based linear mixed-effects model for y,k: y,k = p Ezk
N
+ A, + ~ , k ,
A,
i.i.d.N (0,o’) ,
N
i.i.d.N (0,o;)
(5.34)
1 5 i 5 n, 1 5 k 5 hf
In the above model, the latent variable A, measures the between-subject variability and E , , the judges’ variability in rating the same subject. Under the model assumptions, we have:
Var (yzk)
= o;
+ 2,
2
Var (A,) = c o v (yzk,yzl) = OA
Thus, O? is interpreted as the between-observer covariance, and the betweenrater correlation is a constant and is the ratio of the between-subject variability to the total variability (between-subject and between-judges). This conU2
stant between-rater correlation is the model based ICC, ~ L w I n r I c c= -&. Parametric inference for P L ~ ~ ~ I I depends CC on the normal assumption for both X i and ~i, in (5.34) and as such lacks robustness. For distribution-free inference, first define a bivariate U-statistics kernel as follows:
268
CHAPTER 5 MULTIVARIATE U-STATISTICS
Then, it is readily checked that
Thus, rather than using the variance and covariance, we can express CT; o2 in terms of the mean of h (yi,yj), that is,
+
0:
and
It follows that we can express the model based ICC as a function of the mean 8 of h (yi,yj),that is, ~ L h l h i I c c= g ( 8 ) = Thus, the following bivariate U-statistic based on h (yi, yj) in (5.35) is a consistent estimate of 8:
2.
(">.
We estimate PLMMICC by &,MMICC = g By Theorem 1 and the Delta is given by: method, the asymptotic distribution of ZLhIhlIc~
(-
) , let
To find Ce = 4Vur hl (yl)
5.1 MODELS FOR CROSS-SECTIONAL STUDY DESIGNS
269
Then, it is readily checked that (see exercise)
-
hl ( Y l ) = E [h( Y l , Y2) I Y l ]
(5.36)
-6
Thus, a consistent estimate of Ce is given by
In the above, of $c
can be any consistent estimate-of p. A consistent estimate
is given by: C; = &g
(G)
1
2 0 (&g
(6))
As noted in Chapter 4, neither the product-moment nor the model based ICC takes into account raters’ bias. For example, if ratings from two judges on six subjects in a hypothetical study are 3, 4, 5, 6, 7, 8 and 5, 6, 7, 8, 9, 10, the LMM in (5.34) does not apply because of the difference in the mean ratings between the two observers. Although the model based ICC derived from a revised LMM by replacing p with pk in (5.34) can be applied to the data, it would yield an ICC equal to 1, providing a false or deceptive indication of perfect agreement. The product-moment based ICC has the same problem. A popular index that addresses the limitation of ICC is the concordance correlation coefficient. Example 10 (Concordance correlation c o e f i c i e n t ) . Within the context of Example 8, consider only two raters so that A4 = 2. Let
The concordance correlation coefficient (CCC) is defined as follows: (5.37)
2 70
C H A P T E R 5 MULTIVARIATE U-STATISTICS
Like the product-moment and LMM based ICC, pccc ranges between -1 and 1 and thus can be interpreted as a correlation coefficient. Unlike these popular indices, however, pccc is sensitive to differences in mean rating pk between the observers. For example, pccc # 1 for the hypothetical data discussed above. Thus, pccc provides a sensible solution to address rater's bias. Note that CCC is closed related to the product-moment correlation; Pccc =
7
PC
c = -2
-1
P1 - P 2
+( 4 7 2 ) +
(01/02)-1
where p is the product-moment correlation and C is a function of (scale shift) and (location shift relative to scale).
'H
01/02
with 81, defined by
Let 8 =
el = 2gl2, e2 = (pl - P Then, pccc = f (8)= h(Y2,Yj) =
(
&.
0;-
(5.38)
2a12
Let
hl (Yi, Y j h2
+ +
~ )0::~
;) ( =
(Yi, Y j
(Yil - Y j l ) ( Y i l - Yi2)2
(Yi2 - Y j 2 )
+ (Yjl - Yj2)2
)
(5.39)
Then, we have E [h ( y i , yj)] = 8 (see exercise). Thus, the U-statistic vector h
8
n -1
= (2)
x(i,j)EcF h (yi, yj) is an unbiased and asymptotically normal (-
)
estimate of 8 with the asymptotic variance given by: CQ= 4Var hl (yi) . By the Delta method,
Fee,
=
8 is also asymptotically normal with the
(-1
d
T
asymptotic variance i;,,, = & f (8)Ce ( m f (8)) . To find the asymptotic variance & of 6, note that (see exercise)
-
hl (Yi) =
=(
(h (Yi, Y j ) I Yi)
+012 +P l P 2 + 04 + 0; - 2 0 1 2 + ( P I - P 2 I 2 ]
YilYi2 - Y i 2 P l - Y i l P 2
51 [(Yil
- Yi2I2
= q (Yi, C )
where C = ( p l ,p2,g?,Q;, is given by n
012)~.
i=l
(5.40)
1
It follows that a consistent estimate of EQ
5.1 MODELS FOR CROSS-SECTIONAL STUDY DESIGNS
271
t.
We can where q yi, C denotes q (yi, C ) in (5.40) with C replaced by *> and 0 1 2 . F'rom the above, we estimate C by moment estimates for &, readily obtain a consistent estimate of the asymptotic variance of Fccc. In Chapter 4, we considered Cronbach alpha for assessing internal consistency of multiple items within a domain of an instrument or an instrument itself and discussed inference of this index under a multivariate normal assumption for the joint distribution of the items. However, as items in most instruments used in the behavioral and social sciences expect discrete, Likert scale responses, the normal distribution assumption is fundamentally flawed for such item scores. We describe next a distribution-free approach for inference of alpha. Example 11 (Cronbach's coeflcient alpha). Suppose that a domain of an instrument has m number of individual items. Let yik denote the response from the ith subject to the kth item (1 5 i 5 n, 1 5 k 5 m). Let
(
02
pk = E ( Y i k ) >
0;
m
k= 1
k=l
Yi =
(Yil,
= Var ( % k ) m
0 k l = CoV ( ? h k , YZl)
?
kfl
.
Cronbach's coefficient
We estimate 8 and then use the Delta method to make inference about a = f (el. It is readily checked that (see exercise) m
(5.41) k=l
1
T
82 =
T
1 E [(Yi- P ) (Yi- P ) T ] 1= 1 E
Let
h (Yi,Y j )
=-
(Yi - Y j )
T
"5
1
(yz - yj) (yz - Y j ) T 1
(Yi - Y j ) T
1 ( [I(Yi T - Y j ) ] [I (Yi T -Yj)]
)
272
CHAPTER 5 MULTIVARIATE U- S TAT IS TIC S
Then,
where s; and Skl denote the sample variance and covariance of the elements of yi, respectively. By Theorem 1 and the Delta method, we have
TOfind the asymptotic variance Ce = 4Var
, h12 (
=.[a(
hl (3'1) = (hll (YI)
~ 1 )= ) ~
[h (YI, ~
I
(5.43)
2 )yi]
(Y1 - Y2)- (y1 - y2)
PT(Y1 - Y2)]
[I(y1 T - yz)]
T)
lyl]
Further, it is also readily checked that (see exercise) (5.44)
1
=2 [(Yl - dT(Y1 - P )
+ 811
Also, it follows from (5.18) in Example 4 that hl2
(Yl) = 1 E
[: -
1
(y1 - y2) (yl - y 2 ) T I y1 1
= 2 [IT [(Yl - P ) (Y1 - PITI 1
+ l T E[(Yi - p ) (yi
(5.45) -
P ) T ] I]
5.1 MODELS FOR CROSS-SECTIONAL STUDY DESIGNS
By substituting (5.44) and (5.45) into (5.43), we have:
hl (Yl)= -
"
(Y1 - PIT (Y1 - P )
IT (Y1 - P ) (y1 -
)
1
273
I.+
A consistent estimate of the asymptotic variance Ce is given by:
By substituting
$0
above and
&f (6) in place of Ce and
&f(0) in (5.42),
we obtain a consistent estimate of the asymptotic variance of
0:.
5.1.2 General K Sample Multivariate U-Statistics The generalization of univariate two and K sample U-statistics to a multivariate setting is carried out similar to that of the one-sample case. Consider K i.i.d. samples of random vectors, yhi (1 5 i 5 n k , 1 5 k 5 K ) . Let
be some symmetric vector-valued function with respect to input arguments within each sample, that is, h is constant when yk1,. . . ,Y k m k are permuted within each kth sample (1 5 k 5 K ) . A K-sample U-statistic with r n h arguments for the kth sample has the general form:
As in the one-sample case, U, is an unbiased estimate of 0 = E (h). In Chapter 3, we discussed the importance of examining differences in variances when comparing different treatment groups. For example, if two treatments have the same mean response, but different variances, then the
274
CHAPTER 5 MULTIVARIATE U-STATISTICS
one with the smaller variance may be preferable in terms of treatment generalizability and cost effectiveness, both of which are of great importance in effectiveness research. Also, in applications of the analysis of variance (ANOVA) model t o real study data, it is critically important to check variance homogeneity t o ensure valid inference, since the F-test in ANOVA for inference about difference in group means are sensitive to this assumption of variance homogeneity. However, testing for differences in variance is beyond the mean based, distribution-free GLM discussed in Chapter 2. In Chapter 3, we discussed a U-statistic based distribution-free model for testing equality of variance between two groups. With the introduction of multivariate U-statistics, we can readily extend this approach to compare both the mean and variance simultaneously between two groups, as we consider in the next example. Example 12 (Model f o r mean and variance between two groups). Consider two i.i.d. samples yki with mean P k and variance a; (1 5 i 5 YLk, 15 k 5 2). Let
h(Yli,Yy;Yzl,Y2m) =
(
Yli 2)
~
-
-
( Y l i - Y2j)
(
Y21 2 )
~
(5.47)
- (Y21 - Y2m)
It follows that
6=
[h ( Y l i , Y l j ; Y21, Y2m)l =
(:; I.;)
Thus, the two components of 8 represent the difference in the mean and variance between the two groups. If the null HO : 6 = 0 holds true, then g k i have the same mean and same variance and vice versa. Since h i s not symmetric (e.g. h (Y11, Y l 2 ; Y 2 1 , Y 2 2 ) we symmetrize it to obtain
g(Yli,Ylj;Yzl,Yam) =
(; (Yli
+ Y l j )2 )
( Y l i - Yy)
-
# h ( 3 1 2 , Y11;Y21, Y 2 2 ) ) ,
(; (Y21
+ Y2m)2 )
(Y21 - Y2m)
For notational brevity, we denote g by h. The U-statistic vector based on
275
5.1 MODELS FOR CROSS-SECTIONAL STUDY DESIGNS
the above symmetric kernel is
& c:zl
1
-2
2
where &j = ,y, = yki and g k = S;, = 3E:ll ( Y k i - y), , Thus, the the usual sample mean and variance of each group k (= 1,2). U-statistic vector in (5.47) generalizes the distribution-free ANOVA for comparing two groups as it can test the equality of both group means and variances simultaneously. In Chapter 3, we discussed the two-sample Mann-Whitney-Wilcoxon and F2 that differ by rank sum test for detecting whether two CDFs a location shift, F1 (y) = I72 (y - p ) , are identical, that is, Ho : p = 0 or F1 (y) = F2 (y). By using multivariate K-sample U-statistics, we can generalize this classic test to comparing the equality of multiple CDFs if they differ by a location shift. We consider such an extension for the special case of three samples ( K = 3) in the example below. Example 13 (K-sample Mann- Whitney- Wilcoxon t e s t ) . Let ygi denote three i.i.d. samples from three continuous CDFs that differ by a location shift, that is, Fg (y) = F (y - p g ) , for some pg E R (1 6 i I ng, 1 5g I 3). Let
Consider the bivariate U-statistic, 121
7L2
7L3
(5.49)
T
The mean Of u7L is = (h)= ( E ('{vlt-?jzJ
276
CHAPTER 5 MULTIVARIATE U-STATISTICS
shift, Ho : pg = 0 (1 5 g 5 3), corresponds t o the identity:
Thus, in terms of the parameter vector 8 of the mean of the U-statistic U,, 1 1 T the null of no location shift is: Ho : 8 = (z, z) . Thus, we can ascertain whether there is any location shift across three such CDFs by testing this null Ho regarding the parameters of the U-statistic defined in (5.49). The asymptotic distribution of the K-sample U-statistic vector in (5.46) is established in a similar fashion as in the one-sample case by utilizing the projection of the U-statistic and its asymptotic equivalence to the distribution of the U-statistic counterpart. More specifically, define the projection of U, as follows:
k=l i = l
where 8 = E (U,). Then, it is readily checked that the centered-projection is given by (see exercise):
where for 1 5 k 5 K ;
As in the one-sample case, by applying CLT to (5.51), we immediately obtain the asymptotic distribution for the projection 6,.The asymptotic distribution of the U-statistic vector U, defined in (5.46) is then obtained by establishing the fact that 6,and U, have the same asymptotic distribution. Such an asymptotic equivalence between U, and U, is proved by showing (as in the proof of Theorem 1) that the difference sequence, en = fi (Un converges to 0 in square mean, that is, E (e:e,) 0 (see exercise). We summarize the asymptotic properties of the multivariate K-sample U-statistic in the following theorem. h
en),
--f
277
5.1 MODELS FOR CROSS-SECTIONAL STUDY DESIGNS
Theorem 2 (Multivariate K-sample). Let n = that = p i < 00 (1 5 k 5 K ) . Let h k l (Ykl) =E
(h 1
Ykl)
,
K
c k = l n k
and assume
h k l (Ykl) = h k l (Ykl) - 0
Under mild regularity conditions, we have:
u,
j
e, &(u,-
p
0) +d N
(
0,
K
=xpkmkChk k=l 2 2
)
(5.53)
Note that as in the proof of Theorem 1 for the one-sample case, the asymptotic distribution in (5.53) is actually derived based on the projection of U, by directly applying CLT to the sum of i.i.d. random vectors of the centered projection of U, in (5.51) (see exercise). Also, as noted in the univariate K-sample U-statistics case, n may be defined differently and the assumption 2 = p; < oc is to ensure that nI; increase at similar "k rates so that + Ckl E (0, m) for any n l and n k as n 00 (1 I k, 5 K). Further, the asymptotic distribution of U, retains the same form as given in (5.53) regardless of how n is defined. Example 14. Consider again the U-statistic vector for simultaneously comparing the mean and variance between two independent samples discussed in Example 12. In this application, K = 2, ml = m 2 = 2 . In addition, we have
2
hll
--f
(Yll) = E (h(Yll!Y12!Y2l!Y22)
I Y11) =
i 5
iY11 + &l - P2 2 (Y11 - 11.1)
+ in: -
)
0;
It follows that
Let fly,
&4
= E (Yk1 - &)
4
, the fourth-moment of
yki.
Since E
(~11
it follows that fly C O V (Y11, (911 - PJ2) Pl4 -
4
pl)2 =
2 78
CHAPTER 5 MULTIVARIATE U- S TAT IS TIC S
Similarly, we have
Thus, the asymptotic distribution of the estimate
6 in (5.48) is given by:
Given a real data set, we can readily use the above expressions to estimate the asymptotic variance C h k ( k = 1 , 2 ) . For example, we may estimate the parameters in C h k by
We estimate h
Pk
by
pk
= nl+n2 nk
( k = 1,2). Denote the estimates of
Then for large sample sizes following normal distribution: chk.
chk
by
h
n1
and
n2,
8 follows approximately the
(5.55)
z,
As noted above, we may define n differently such as 6 = min ( n l ,712). Then are estimated by the same expression, = albeit a different value E. By substituting the different definition of n and 2 : into (5.55), we immediately obtain
Fg
Pi
The above confirms the prior assertion that the asymptotic distribution of 8 is independent of how n is defined. In the above example, we estimated the asymptotic variance of 0 by first obtaining a closed form expression for each C h k . Alternatively, as in the onesample case, we can construct a U-statistic t o estimate each C h k without
h
h
279
5.2 MODELS FOR LONGITUDINAL STUDY DESIGNS
attempting to evaluate these variance matrices. estimating C h l , which can be expressed as: Ch1 =
=
[ E (h(911,Y12, Y21, Y22)
To illustrate, consider
I Y11)I2 wT -
ahl eeT -
Then, the U-statistic matrix,
is a consistent estimate of a h l . This approach is especially effective when the asymptotic variance cannot be evaluated analytically.
5.2
Models for Longitudinal Study Designs
The longitudinal models discussed in Chapter 4 are mostly generalizations of the models for cross-sectional study designs developed in Chapter 2 and are primarily used for modeling the conditional mean of a single response given a set of independent variables. As seen in Chapter 3, many important statistics, such as the classic two-sample Mann-Whitney-Wilcoxon rank sum test, cannot be framed as regression models and are best studied by the theory of U-statistics. Thus, in parallel to the development in Chapter 4, we consider generalizing these U-statistics based models to a longitudinal data setting in this section. In Section 5. 1,we discussed models that require multivariate U-statistics because of the number of parameters involved in defining and estimating the quantities of interest. In recent years, longitudinal studies have not only become more popular in biomedical and psychosocial research but also more complex in their designs. In such applications, we have multiple outcomes
280
CHAPTER 5 MULTIVARIATE U-STATISTICS
from the same subject due to repeated assessments. Interest for such longitudinal studies usually centers around patterns of response over time. Thus, even for parameters defined by a single response, studying the change patterns of such parameters over time requires considering multiple responses collected together at various assessment times, giving rise to multivariate outcomes. However, because of the opportunity for missing responses, the methods discussed in Section 5.1 no longer apply to data from such study designs. One approach is to simply use subjects with no missing response at any of the assessment points, or complete data. However, such an approach at best does not utilize all available data and thereby lacks efficiency in inference of model parameters. Further, as we discussed for regression analysis in Chapter 4, such a complete-data approach generally gives rise to biased estimates if missing data follows the missing at random (MAR) model. In this Section, we generalize the results in the preceding section to address missing data. As in Chapter 4, we focus on the missing completely at random (TUICAR) and MAR mechanisms.
5.2.1
Inference in the Absence of Missing Data
The multivariate U-statistics discussed in Section 5.1 arise from the need to construct multidimensional statistic vectors t o estimate the parameters of interest. Thus, even for cross-sectional study designs, such problems require multivariate U-statistics. Another common type of problem that requires multivariate statistics is analysis of longitudinal study data. As discussed in Chapter 4, longitudinal study designs offer many benefits and address the fundamental limitation of cross-sectional study - the inability to provide causal inference. On the other hand, longitudinal study designs yield multiple correlated responses, which turn otherwise simple univariate statistical analyses into complex models involving multivariate distributions. To differentiate the two types of multivariate statistics, we refer to the multivariate U-statistics that are required to express the parameters of interest as endogenous while those resulting from longitudinal study designs as exogenous type. Thus, although the theory of multivariate U-statistics can be applied to exogenous type of multivariate problems, we must address the inherent missing data issue. In this section, we assume no missing data and illustrate how the theory of multivariate U-statistics developed for endogenous problems can be applied to address exogenous multivariate outcomes. Example 1 (Model for variance ower time). Consider a longitudinal study with n subjects and rn assessment times. Let yit denote the response from the ith subject at time t (1 5 i 5 n, 1 5 t 5 m ) . We are
281
5.2 MODELS FOR LONGITUDINAL S T U D Y DESIGNS
interested in modeling the variance of the response git over time.
Let
A primary area of interest in longitudinal studies is trend over time. Within the current context, we are interested in testing variance homogeneity over time, that is HO
: a: = 0 2 , for all 1 5
H , : a; # a:,
t 5 m versus for some 1 5 s < t 5 m
In this example, the quantity of interest at each assessment time is a: and the associated univariate U-statistic based estimate has been discussed in Chapter 3. However, testing the above hypothesis requires jointly modeling all these univariate statistics simultaneously. To this end, consider the following multivariate U-statistic vector:
Since E
(6)
h
=
E [h ( y i ,yj)] = 8, 8 is an unbiased estimate of 8.
The
asymptotic distribution of 6 is readily obtained using the development in Section 5.1. In particular, the asymptotic variance is given by (see exercise): T
C B = Var (pi (~1))
pi ( y i ) = ( ( ~ i-i ~
1
, .). . ,~(gim - k d 2 )
(5.58)
With such a joint distribution, we can readily test the hypothesis of variance homogeneity using linear contrasts. Example 2 (Mann- Whitney- Wilcoxon f o r longitudinal data). Suppose that we have two groups of subjects who are followed over time in a longitudinal study with m assessment times. Let ykit denote the response from the ith subject within the kth group at time t (1 5 i 5 n k , k = 1 , 2 , 1 5 t 5 m). At each time t , we can compare the two groups using the Mann-Whitney-Wilcoxon rank sum test as discussed in Chapter 3. In this example, we generalize this classic test for no location shift between the two CDFs of y k i t at a single time to a longitudinal data setting involving multiple assessment times.
282
CHAPTER 5 MULTIVARIATE U-STATISTICS
For each time t , let Bt = P r [ y l i t 5 g2it] (1 5 t 5 m). Then, it follows from the discussion of the two-sample Mann-Whitney-Wilcoxon test in Chapter 3 that the hypothesis of no location shift can be expressed as: 1
Ho : 8t
=
-,
for all 1 5 t 5 m versus
Ha : 8t
#
2,
for some 1 5 t 5 m
2 1
(5.59)
As in Example 1, the quantity of interest is univariate, but the longitudinal study design requires that we model Qt simultaneously over all t. Let
Then, the U-statistic vector, (5.60)
5
is an unbiased estimate of 8. The asymptotic distribution of is readily computed using the theory of multivariate U-statistics in Section 5.1 to provide inference for testing the hypothesis in (5.59) (see exercise). One major limitation of the product-moment correlation between two continuous variables x and y is the assumption of a linear relationship between x and y. In Chapter 3, we discussed some popular alternatives that require no such assumption. One such alternative is Kendall’s T , which assumes no functional relationship between the variables. In Chapter 3, we discussed inference for 7 using the theory of univariate U-statistics. In Example 3, we consider extending this approach to data from a longitudinal study design. Example 3 (Kendall ’s 7 for continuous outcome). Consider a longitudinal study with n subjects and m assessment times. Let z,t = (It,t, y z t )T be an i.i.d. sample from a continuous bivariate distribution (1 5 t 5 m, 1 5 i 5 n). As discussed in Chapter 3, for each time t , we can develop a univariate U-statistic based on the kernel, h ( z z t ,z 3 t )= I{(,zt-”3t)(Yzt-Ylt)>o}. t o test whether there is association between x, and yz. To generalize this univariate test to the current longitudinal data setting.
283
5.2 MODELS FOR LONGITUDINAL STUDY DESIGNS
let T
(5.61) T
h(zz,z3= ) ( h l , ... , h m )
, 15 i < J 5 n
Then, T = E [h (z,, z3)] is the vector containing Kendall’s rs from all the n -1 m assessments. The U-statistic vector, ? = ( 2 ) Cy=lh (z,, z3), is an unbiased estimate of T. The asymptotic distribution again is readily derived using the theory in Section 5.1 and can be used to test the null of no association between x,t and yzt over the study period, HO : T = 0 (see exercise). If this overall test is rejected, we may apply the procedure to a subvector of T to carry out follow-up tests to determine the assessment times over which x,t and yzt are associated. In addition, we may want to perform followup tests to compare magnitudes of association across different assessment times. For example, we can test whether there is any change in association between times 1 and 2 by testing the linear contrast, Ho : T I = 7 2 . In Example 3 of Section 5.1.1, we also discussed inference for Kendall’s T b , a version of T for association of between two ordinal outcomes. The example below extends this index to a longitudinal data setting. Example 4 (Kendall’s T b for ordinal outcome). In Example 3, let T T zzt = ( u , t , ~ , tand ) ~ z, = (zzl,.. . , z L ) be an i.i.d. sample of bivariate ordinal outcomes. Assume that u has K and u has M levels indexed by Ic and m, respectively. To generalize the kernel function for T b developed in Section 5.1.1 to the current longitudinal context, let
h 1t (Zzt , 23t 1 = I{ (u, t -u3 t ) (w, -71J t ) >O>
- I{ (uzt-uj t ) ( U z t -213 t ) <0 )
h2t (Zzt, Z g t ) = I{Uzt=ujt}, h3t (Zzt, Z J t ) = I { 7 J Z t = W J t }
ht
(Zzt, Z J t ) =
(hlt, h2t, h3dT ,
h (zz,zJ
=
(hL
T *
’ >
hL)
Then, the U-statistic vector:
is an unbiased estimate of 8 = E (h (zi, zj)). We can readily determine the asymptotic distribution of 6 by applying Theorem 2 of Section 5.1.
284
CHAPTER 5 MULTIVARIATE U- S TAT ISTIC S
Now let
(->
Then, ?b = f 8 is a consistent and asymptotic normal estimate of T b . The asymptotic variance of ?b is readily obtained from the Delta method (see exercise). As in Example 4, we can use this asymptotic distribution to test hypotheses concerning T b or a subset of T b .
5.2.2 Inference Under MCAR In all the above examples, the multivariate U-statistics are required because of the repeated assessments in longitudinal study designs. In such studies, missing data is a common and persistent problem, and as a result, the approaches discussed can only be applied to subjects with complete data. However, under random missingness, such an approach still provides valid inference. As discussed in Chapter 4, a formal definition of random missingness is the missing completely at random (MCAR) assumption. Under MCAR, the occurrence of missing response is unrelated to or independent of the response and thus estimates based on the observed data are still consistent. Although such a complete-data approach still provides valid inference, it may be quite inefficient. For example, if the occurrence of missing data is random over time and across subjects, the number of subjects with complete data may be substantially smaller than the original sample. As a result, such a strategy of removing subjects with missing data may seriously affect power. An alternative and indeed a more efficient way is to use all observed data. As discussed in Chapter 4, missing data arising in many longitudinal trials is the result of treatment response patterns and thus is often predicted by observed responses. Such a response-dependent dropout process or missing at random (MAR) mechanism invalidates the MCAR assumption. Consequently, estimates based on the complete-data subsample become biased or inconsistent. As pointed out in Chapter 4, such a dependence of occurrence of missing data on the observed response must be specifically modeled and explicitly accounted for when making inferences for mean based distributionfree generalized linear and related regression models. This MAR mechanism or response-dependent dropout process exerts a similar effect on U-statistics based distribution-free models discussed in the above sections and therefore
5 . 2 MODELS FOR LONGITUDINAL S T U D Y DESIGNS
285
must be carefully addressed. In this section, we first generalize the missing data techniques employed for the distribution-free regression models in Chapter 4 to develop approaches for inference for U-statistics based models under the relatively stronger, but more easily addressed MCAR assumption. Example 5 (Model for variance over time).Consider the model for variance in Example 1 of Section 5.2.1. Under the complete-data approach, only subjects with observed yit over all assessment times t contribute to inference (1 5 t I:m). Let n,bs denote the number of such subjects. In this case, we estimate 8 in Example 1 by applying the procedure to only these subjects. Although unbiased, such estimates are based on an effective sample size, n,b,. If missing data occurs sporadically over time and across subjects, n,b, may severely undercut the original sample size n. Now consider the more efficient approach that uses all observed data. For each time t , we can estimate a: based on all available responses. Although it is not difficult to compute such estimates, it is notationally complex to express the asymptotic distribution of these joint estimates. For example, consider only two time points, s and t (1 5 s < t 5 m). Let n, and nt denote the sample size at each of the times and let {il, . . . ,in,} and {jl,. . . , j n , } denote the subsets of the indices for the subjects with observed responses at times s and t , respectively. Then, we can express the two estimates of a: and a: as follows: (5.64)
02
and a:, since $: and 3; are correlated The difficulty lies in comparing and the joint asymptotic distribution is needed to make inference concerning these parameters. It is at best cumbersome to derive this joint distribution directly using the expressions in (5.64). The major problem is tracking the indices for the subjects with observed data at each of the two times points when deriving the asymptotic distribution. This problem becomes more challenging when considering more time points. We now describe an alternative approach based on a similar development for addressing missing data for the distribution-free generalized linear models in Chapter 4. This approach allows us to effectively deal with the notational complexity discussed above and model the effect of missing data
286
CHAPTER 5 MULTIVARIATE U-STATISTICS
in a systematic fashion. This approach employs missing data indicators to track the occurrence of missing response and defines estimates as a function of such missing data indicators, thus alleviating the algebraic complexity in expressing observed responses using subsets of subject and time indices. In the following example, we describe how to adapt this approach for Ustatistics-based models. Example 6. Consider again the model for variance in Example 1. Rather than employing subject and time indices to express the observed data, we can do so alternatively using binary indicators. For each subject i, define a set of missing (or rather, observed) data indicators as follows: rit =
1 if yit is observed
,
ri = ( ~ i l ..., , Tim)T
(5.65)
0 if yit is missing
** . .
Ri = diag (ri) =
,
l
h
By using ri, we can readily revise the U-statistic estimate 8 defined in (5.57) for this problem to reflect its dependence on observed data. For example, consider the tth component ht (yit, yjt) of the kernel h (yi,yj) for this U-statistic estimate in (5.57). Although ht (yit, yjt) is not defined when one or both of yit and yjt is missing, the following revised kernel gt (yit, it, yjt, r j t ) is well defined ht gt
1
2
bit, Yjt) = -2 (Yit - Y j t )
k i t , T i t , yjtr ~ j t = ) ritrjtht
(5.66)
1
2
( y i t , y j t ) = - ritrjt (Yit - Yjt)
2
Unlike ht ( g i t , y j t ) , gt ( y i t , ~ i t , ~ j t , r j is t ) also a function of it and rjt. It is readily checked that the estimate 5: in (5.64) can be expressed in terms of this kernel function as follows (see exercise): (5.67) However, unlike (5.64), the above does not involve the use of a subset of indices of subject and time for tracking observed data, making it particularly easy to derive the joint asymptotic distribution of 5: as we develop below.
287
5.2 MODELS FOR L O N G I T U D I N A L S T U D Y DESIGNS
Let zi = ( y a ri) , T and 8 = (al, 2 g22 , .. . ,a,) 2 the form of a vector, we have
T.
By expressing (5.66) in
It is readily checked that g (zi,zj) is a symmetric function with respect to zj. Thus, the U-statistic
zi and
is an unbiased estimate of E ( g (zi,zj)). Under MCAR, independent. It follows that
Ri
and yi are
where E2 (Ri) = [E (&)I2. If E (Ri) = I,, that is, there is no missing data, E ( g (zi,zj)) = 8 and U, is an unbiased estimate of 8. Otherwise, missing data occurs with a positive probability and U, is not an estimate of 8. However, we can construct a consistent estimate of 8 by generalizing the univariate estimate 5: in (5.67). Let
Now define an estimate of 8 by
288
CHAPTER 5 MULTIVARIATE U-STATISTICS
It is readily checked that (see exercise) (5.69)
[
E iritrjt (Yit -+P
- Yjt)']
E2 ( T i t )
-
E2 (.it)
0;
E2 (.it)
2
= ut
where the second to the last equality follows from the independence between Ri and yi under the MCAR assumption. Thus, 6 in (5.67) is a consistent estimate of 8. Since 8 can be expressed as 8 = G-l C(i,j)EC,n RiRj8, it follows that (see exercise) G-I
c c
g ( z i , z j ) - G-l
(i,j)EC,.
=6G-l
c (idEC,.
RiRj [h (yi,yj) - 81
RiRj8
1
(5.70)
(i,j)EC,.
=
(;)G-'&V,
where v (zi, zj) = RiRj [h (yi,yj) - 81 and V, =
(n 2 ) -1
C(i.j)Ecg v (zi, zj).
As v (zi, zj) is symmetric, V, is a U-statistic-like random vector, that is, V, is a U-statistic if 8 is known. By Theorem 1 of Section 5.1, we have:
The asymptotic normality of (5.71):
fi (6 - e )
-+p
6 follows by
applying Slutsky's theorem to
N ( 0 ,= ~E ~- ( ~ R~ c )~ E (R~)) - ~
(5.72)
5.2 MODELS
FOR LONGITUDINAL STUDY DESIGNS
289
To estimate the asymptotic variance Co in (5.72), we need to find Cv. Consider the tth component Vt (z1) of Vl (z1) (1 5 t 5 m ) . It is readily checked that
T
where Pt = E ( Y i t ) . Let Pi (Yi, P ) = ((Yil - P d 2 ,. * . , (Yim - P m ) 2 ) . BY expressing Vt (z1) in (5.73) in terms of the vector Vl ( z l ) , we have
Vl
(z1) =
1
f (R1)R1 [Pl (Yl?PI
- 61
(5.74)
Thus, we have (see exercise):
c v = 4Var (V1 (Zl)) = 4E [V1 ( X I ) =E
iq (zl)]
(5.75)
(R1) Var (RlP1 (Yll P I ) E (R1)
Thus, it follows from (5.75) that
CQ= K 2(R1) = E-l ( R l ) Va7-(
(5.76)
(R1)
R m (Yl, P ) )E-l (R1)
When expressed in terms of the elements of Co = [o,t], we have: o s t = E - l ( T l s ) E-l ( T l t ) E (TlsTlt) c o v (Pls,Plt)
(5.77)
A consistent estimate of Co is obtained by substituting consistent estimates for E ( r ~ tE) (rlsrlt) ~ and Cov (pl,,plt). For example, we can estimate these quantities by the following moment estimates: n
n
i=l
i= 1
In Examples 5 and 6, the U-statisics for the parameters of interest are univariate and the multivariate U-statistics are created by the repeated assessments as dictated by longitudinal study designs. The above considerations are similarly applied to endogenous multivariate U-statistics within a longitudinal data setting as illustrated by the next example.
290
CHAPTER 5 MULTIVARIATE U-STATISTICS
Example 7 (Product-moment correlation for longitudinal d a t a ) . Consider the product-moment correlation problem in Example 1 of Section 5.1.1 within a longitudinal study setting. Even within a cross-sectional study design, this problem requires consideration of a three-component multivariate U-statistic. Here, we extend the development to a longitudinal data setting with m assessments. T Let zit = ( z i t , y i t ) be an i.i.d. sample of bivariate variables rn assessments. Let gxyt =
c o u (.it,
Yit) ,
et = ( g x y t ,g:t, .it)’ gxyt
Pt=
d a
= ft
g x2 t = Var (.it)
>
g y2 t =
(5.78)
Var ( Y i t )
T , e = (el,.. . ,ern)
(&I,
T
P = f ( 6 ) = (fl(01), 3 . . . > f m ( O m ) )
We are interested in inference for p as well as hypotheses concerning linear combinations p. First, consider the problem without missing data. For each time t , let h k t ( z i t ,z j t ) be defined as in (5.2) of Example 1 in Section 5.1 for zit and let h t (Zit,Zjt) =
(hlt (Zit,Z j t ) , h 2 t
^p = f
(G)
= (fl ( G l )
,
(Zit,Zjt) h3t (Zit,Zjt)IT
! , . . . , fm ( G m ) )
T
By applying the results in Example 4 of Section 5.1.1 to each Ft, it follows that ^p is a consistent estimat.e of p. Also, by the Delta method, we immediately obtain the asymptotic distribution of ^p:
29 1
5.2 MODELS FOR LONGITUDINAL STUDY DESIGNS
TOestimate EQ,let p, = E ( x i ) and py = E ( y i ) . Then, for each t , it follows from Example 4 that (5.79)
T
Pit (Zi,P x t , P y t ) = ( ( Z i t - P d (Yit - P y t ) ,ti.(
-
P d 2 , (Yit
-
PYtJ2)
Thus, it follows from (5.79) that
where pi (zi,p z ,p,) = given by
(p;,
. . . , FL)
T
. A
consistent estimate of CQ is
Now consider the missing data case. Let rit be defined as in (5.65), T except that git is replaced by zit = ( z i t , y i t ) . Thus, we assume that xit and yit are either observed or missing together as a pair, that is
rit =
1 if the pair (zit,yit) is observed
(5.80)
0 if otherwise Let ri and Ri be defined as in (5.65). For real study data, this assumption is plausible if missing data is the result of missed study visits. We relax this assumption t o allow xit and yit to have their own missing data patterns and discuss this more general case in Section 5.2.3. In the presence of missing data, we define the Pearson estimate at each assessment time t as follows:
292
CHAPTER 5 MULTIVARIATE U-STATISTICS
where
Thus, Ft not only depends on zit but on h
p=
- T (Pl,P2,...,Prn) A
Tit
and rjt as well (1 5 i 5 n). Let
h
*
Before embarking on the task to find the asymptotic distribution of 6, let us first establish the consistency of 6 as an estimate of p. To this end, let
where I3 denotes the 3 x 3 identity matrix. For each t , define an estimate of Ot as follows:
(5.83)
where
n -1 U n t = (2)
C(i,j)EcF gt (zit,z j t , ritrjt)
is a three-dimensional U-
statistic vector. Thus, although the estimate & above is not a U-statistic as in the complete data case, it is a function of such statistics since (;)-'Gt is also a U-statistic. Further, under MCAR, it is readily shown that & -tP0t
(1 5 t 5 rn) (see exercise). Thus, 6 = f
(G)
-+P
f (0) = p and
6 defined in
(5.81) is a consistent estimate of p. Since defined in ( 5 . 8 3 ) is consistent, as in the complete data case, we can use the Delta method to find the asymptotic variance of 6 once we have
5.2 MODELS
FOR LONGITUDINAL STUDY DESIGNS
the asymptotic distribution of
G.
293
To find the latter, let
where diagt (Rit)denotes the 3 m x 3 m block diagonal matrix with Rit forming the tth block diagonal. Then, we have (see exercise):
,/E
(6 - e) = hc-1
c
[g (zi,zj, R ~R, ~-)R ~ R ~ ~(5.84) I
(i,AECzn
where
is a U-statistic-like vector with mean 0 as in Example 6. asymptotically normal:
&iUn
+d
Thus, Un is
N ( 0 , Eu = 4Var (GI( ~ 1.I))) ,
(5.85)
It follows from (5.84), (5.85) and Slutsky's theorem that
6 (G - e)
-+d
N ( 0 , E8 = 413-2 ( R ~E ) ~ E (-R ~~ ) )
To find Eu in the asymptotic variance that (see exercise)
&? above,
it is readily checked
61 (z1, .1) = E [R1R2 (h(z1,z2) - 0) I z1, r11 1 = -R1E (R1)[Pl (Zl!P,, Py) - e] 2 Thus,
(5.86)
294
C H A P T E R 5 MULTIVARIATE U- STAT IS TIC S
from which it follows that
Ee = E-l
( R i )Var (Rip1 ( 2 1 , y,,
yy))E-l (Ri)
(5.88)
Note that (5.87) and (5.88) are quite similar to (5.75) and (5.76), except for the difference in p1 (21, y,, yy)and R1, as these are all 3m x 3m matrices in the current example. A consistent estimate of C Qis obtained by substituting consistent estimates for the respective parameters, such as the ones given below: n i=l
5.2.3
n i=l
Inference Under MAR
In Chapter 4, we discussed inference for distribution-free generalized linear and other related models under response-dependent missingness or MAR within the context of generalized estimating equation (GEE). The idea is to use weight function to statistically “recover” missing data for study dropouts and include them in estimation of model parameters. Although we do not consider GEE for inference in this section, the general strategy is equally applicable to address selection bias when missing data follows the MAR model. Consider the model for variance within a longitudinal data setting introduced in Example 1 of Section 5.2.1. We developed an approach that employs missing data indicators to facilitate the construction of a consistent T estimate 8 of 8 = (a:, . . . , a&) as well as the derivation of the asymptotic distribution of under MCAR in Example 6 of Section 5.2.2. Under MAR, however, 6 is generally not consistent and a new estimate must be introduced. To see why in general 6 does not consistently estimate 8, let us revisit the steps to show the consistency of 5: in (5.69). The crucial step is the following identity: h
(5.89)
5 . 2 MODELS FOR LONGITUDINAL S T U D Y DESIGNS
295
Under MCAR, rit and yit are independent and thus
Without the independence between rit and yit, such as under MAR, the identity in (5.89) generally does not hold true, and as a result constructed in Example 6 of Section 5.2.2 may not be consistent estimate of 8. In the following example, we discuss how to construct a consistent estimate of 8 for this particular model as a way of introducing a general approach to address MAR in our context. Example 8 (Model f o r variance ower t i m e ) . Within the context of Example 6 of Section 5.2.2, assume that missingness depends on the response vector yi. Let T i t = Pr (rit = 1 1 y i ) denote the probability of observing the response yit given y i . For the moment, assume that T i t is known (1 5 i 5 n,
l
Consider the following estimate of a;: (5.90)
It follows from the law of iterated conditional expectation that (see exercise)
E
(8)=
(5.91)
[gt ( P i t , Y j t , T i t , r j t ) ]
= at"
Thus, a; is an unbiased estimate of a;. By the consistency of U-statistics, T (5.90) defines a consistent estimate of a? and 8 = (a:,?;, . . . , Z i ) is a consistent estimate of 8 = (a:, a;,. . . ,arn) 2 7 -. h
296
C H A P T E R 5 MULTIVARIATE U-STATISTICS
Note that under MCAR, since T i t is independent of yi, (5.90) reduces to (5.92)
On the other hand, it follows from the consistency of U-statistics that (5.93)
Thus, it follows from Slutsky's theorem that G: defined in (5.90) has the same asymptotic distribution as the estimate of g; definition in (5.67) in Example 6 under MCAR. The difference between the two estimates is that the version in (5.90) requires a known or an estimate of T i t , while the estimate in (5.67) does not. To find the asymptotic distribution of let
g,
Then, it follows from the theory of multivariate U-statistics in Section 5.1 that the asymptotic distribution of is given by: (5.94) The asymptotic variance Ce in (5.94) is evaluated as in Example 6 (see exercise). Likewise, we can readily construct a consistent estimate of Ce by following the same steps in that example (see exercise). In Example 8, we showed how to incorporate weight functions into estimates to address selection bias due to missing data under MAR. Such an approach has been used for regression analysis within the context of GEE in Chapter 4. As discussed there, it is critical that > 0 for all 1 5 i 5 n and 1I :t I :m so that the observed responses can serve as representatives for their missing counterparts. Otherwise, we will not be able to statistically recover all missing data to include them in estimation to ensure consistent
297
5.2 MODELS FOR LONGITUDINAL STUDY DESIGNS
parameter estimates. Further, we often assume 7rit 2 c > 0, that is, X i t is bounded away from 0, to obtain stable estimates of parameters of interest ( I I i l n ,1 < t < m ) . While Tit may be known in some study designs (e.g., deliberately discontinuing some subjects in a multiphase clinical trial), the relationship between ri and yi is unknown in most applications. Under MAR, 7rit only depend on the observed responses. As discussed in Chapter 4, it is generally difficult to model Tit without imposing the monotone missing data pattern (MMDP) assumption because of the large number and complexity of missing data patterns. The structured patterns under MMDP eliminate a potentially large number of missing data patterns and make it practical to model and estimate 7rit under MAR. Example 9. In Example 8, assume MAR, that is, 7rit is a function of observed yit. Further, we assume no missing data at baseline t = 1. Under MMDP, riit only depends on the observed responses and thus can be expressed as: 7rit = Pr ( T i t =
1I
yi) = Pr ti.(
Yit = ( Y i l , ...,?li(t-l))T,
=
1 I ?it)
(5.95)
2I tI m
T
where Yit = (yil, . . . , the sub-vector of yi containing y variables up to and including time t - 1. As discussed in Chapter 4, we can readily model 7rit in (5.95) under MAR and MMDP using logistic regression. More specifically, we first model the one-step transition probability from observing the response at t - 1 to t using a logistic regression as follows: logit ( p i t ) = logit [Pr ( T i t = 1 1 25t<m
ri(t-.l)=
1,F i t ) ] = at
+ plyit
(5.96)
T
Let yt = (at,@-) denote the parameters of the logistic model above. Then, by invoking MMDP, we obtain nit
(y) = P r [riS= 1,T i t = 1 1 ?is, ?it] = P r [Bit = 1 I Fit] = P r [ T i t = 1 I T i ( t - 1 ) = 1,YitI p r [ T q t - I ) = 1 I Y q - l ) ]
(5.97)
t
=nPi,(y,),
2
15iFn
s=2 T
where y = (y:, . . . ,yA) . We can estimate riit (y) by Tit (;3) with obtained from the logistic regression for one-step transition probability in (5.96).
298
CHAPTER 5 MULTIVARIATE U-STATISTICS
Note that when Tit is estimated by (5.97), the asymptotic variance in (5.94) is likely to underestimate the variability of g. In Chapter 4, we discussed how to account for the variability of the estimate 7 in the model for 7rit ( 7 ) . If T is obtained from GEE or the maximum likelihood method, we can readily find the asymptotic variance of g adjusted for the variability of 7. Consider a one-sample U-statistic vector with 4 arguments of the form U, (y),where y is the parameter vector describing the missing data model as in the above example. It follows from the theory of multivariate statistics that
As in Chapter 4, let T be the solution to the estimating equation: w, (y) = 1 n Cr=lw,i (y) = 0 , where w,i is the score (for maximum likelihood estimation) or score-like vector (for GEE estimation) for the ith subject (1 5 i 5 n ) . By a Taylor series expansion, we have -T a h ( 7 - 7 ) = (-yW.)
h w ,
+ o p (1)
(5.99)
T
where H = E (&wni) . It follows from (5.98) and (5.99) that (see exercise) :
h (Un (7)- 0) = h [Un(7)- u n ( 7 )+ u n ( 7 )- 01
(5.100)
T
where C = E
(&h1 (yi,ri,y)) .
By applying CLT to (5.100), we see
that is asymptotically normal with the asymptotic variance given by (see exercise) :
5.2 M O D E L S FOR L O N G I T U D I N A L S T U D Y D E S I G N S
299
Thus, the extra term ip in (5.101) accounts for the additional variability due to 7. A consistent estimate of is readily obtained by substituting consistent estimates for the respective quantities in (5.101). Now consider extending the considerations for the product-moment correlation discussed in Example 7 of Section 5.2.2 to account for responsedependent missingness under MAR. Although we can apply the same approach in Example 9 to the product-moment correlation, we consider a more general problem of inference about the cross-variable, cross-time productmoment correlation pxyst between xis and yit, as well as by allowing zit and yit to have their own missing data patterns and in the process introduce a version of MMDP for bivariate responses. Example 10 (Product-moment correlation). Let zit = (zit, be an i.i.d. sample of bivariate outcomes over time t (I 5 t 5 m, 1 5 i 5 n). Here, we generalize the consideration in Example 7 to discuss inference about the cross-variable, cross-time product-moment correlation pxyst between xis and yit (1 5 s , t 5 m ) . Further, we relax the missing data assumption considered earlier to allow zit and yit to have their own missing data patterns, that is, zit and yit may be missing at different time points. We define two missing data indicators, one for each of the outcomes as follows: 1 if uit is observed ? u=x,y (5.102) riut = 0 if uit is missing
{
Let 7riuvst = Pr ( T i u s =
1,rivt = 1 I zi) , u = 2 , y
(5.103)
The above define three sets of probabilities, 7rixyst, 7rixst and 7riyst. For notational brevity, we adopt the convention that Tiuvst = 7riust if u = and s 5 t so that we can use i7iuvSt to represent all three. Note that unlike 7riuvst, 7riust = Tiuts for u = z, y and thus we only consider 7riust for s 5 t. While our interest lies in estimating pstr for convenience, we discuss inference for the covariance oxystbetween xis and yit (1 5 s, t 5 m). It is straightforward to generalize the consideration to develop a similar procedure for inference about pst. As before, we first assume that n-iuvSt is known and derive a consistent estimate of oxystand its asymptotic distribution. We then discuss how to model and estimate 7riuvst under MAR. Note that for our consideration of oxyst,we only need rrisyst. However, when considering inference about pst, we also need 7rixst and 7riyst for constructing estimates of oxst and ovst (see exercise). Thus, we consider modeling the general 7riUvst below
300
C H A P T E R 5 MULTIVARIATE U-STATISTICS
Let 1 2
(5.104)
h s t ( Z i , Z j ) = - ( X i s - X j s ) (Yit - Y j t ) gst (Zi,
zj, 'i,
-1
= 7 r ~ l y s t ~ j x y s t r i x s r i y t r j x s r j y t h (sZt i , Z j )
'j)
The above generalizes (5.82) to account for dependence of the occurrence of missing data on zi. It follows from the law of iterated conditional expectation that (see exercise) (5.105)
I ZZ,Zj)] = E [hst ( Z i , Z j ) ] = g x y s t
Thus, by the theory of U-statistics, the statistic,
is an unbiased and consistent estimate of uxyst. Further, it follows from Theorem 2 of Section 5.2 that the asymptotic distribution of Zxyst is given by
f i (Zxyst - g x y s t )
-+d
N (0,
& =4 ~ a r
(zi, '1)))
(5.106)
Since Var (&ti
( ~ 1'1)) ,
=
[%ti
(zi, z 2 , '1,
'2)
g s t l ( ~ 1 , 5 3 3 '1, , '311
a consistent estimate of & is readily constructed using U-statistics. In the above, we only considered inference for a single oxyst. However, we can readily extend the consideration to the vector statistic g = T (911,.. . , glm, ~ 2 1 , .. . , gmm) so that we can make joint inference of oXy = (Uxyll,
' '
, CJxylm, g x y 2 1 , . , g z y m m ) T .
As in Example 9.
* *
are rarely known in practice. Under MCAR, As a result, 7riuwst= E ( r i u s r i v t ) , which are readily estimated by sample moments of the form: Cy=lr i u s r i w t . Further, if r i x t = r i y t = Tit, that is, zi and yi are observed or missing together, Tiuwst = E (r,,rit) = 7rst and are again readily estimated. When MCAR fails, 7riuvst becomes dependent on zi. Note that 7riuwst is different from 7rit discussed in Example 9 as it is a function of two outcomes, 7rixyst
riut are stochastically independent of zi.
5.2 MODELS
FOR LONGITUDINAL S T U D Y DESIGNS
30 1
xis and yit, and the approach for modeling Tit does not apply to i7iuwst. As in the case of T i t , it is difficult to model Tiuwst as a function of zi in general without imposing strong assumptions on the relationship between the occurrence of missing data and outcomes. Under MAR, 7riuwst only depends on the observed zi. Unfortunately, it is still quite complex to model the occurrence of missing data and a common approach is to further impose the monotone missing data pattern (MMDP) assumption as we discussed earlier in Example 9. For the current bivariate response setting, the singleresponse version of MMDP can still create a potentially large number of distinct patterns. For example, if m = 5, zit may be observed at all times, while yit may be last observed at time t = 3 , 4 or 5. Although possible, it is difficult to formulate a model for Tiuwstgiven the large number of missing data patterns. Thus, it is necessary to limit our consideration to some special cases. To this end, we first introduce an index to categorize the different MMDPs within a bivariate response setting. For each subject i, let li denote the absolute difference between the times of the last observed zit and yit. This lag time can vary between 0 and m - 1. For example, if zit is last observed at t = 1 (baseline), while pit is observed at all times, then li = m - 1. Let k = max(1i; 1 5 i 5 n } denote the largest difference in lag time across all subjects in the study. This index k characterizes the lag time of bivariate MMDP (BMMDP) for the study sample. We focus on the special cases when k = 0 and k = 1 and discuss modeling Tiuwst under the assumption that riut only depend on zit = { z i s ;1 5 s 5 t - l}, the history of responses observed prior to time t .
-
For BMMDP with lag 0, T i z t = Tiyt = T i t . In this case, z and y are either observed or missed together. By BMMDP, we have 7rist = Pr
-
(riS = 1,T i t = 1 1 Zi,, Zit) = Pr ( T i t = 1 I zit) = T i t , 1 5 s
5t5m
We can estimate 7rist through a process similar to the steps defined by (5.96) and (5.97) for modeling Tit in Example 9, except that the predictor of the logistic regression in (5.96) is defined by the bivariate outcomes -zit = (zit,yit) T , that is logit
( p i t ) = logit
(Pr ( T i t = I
I
= 1,Zit)) = at
+ ,@zit
25t5m If k = 1, the lag times between the last observed responses of zit and yit can differ at most by 1 across all subjects. Given that (xi,,yi,) is observed at time t - 1, four possible missing data patterns emerge at time t , as illustrated
302
... ...
CHAPTER 5 MULTIVARIATE U- STATISTIC S
Assessment times t-I
t
t+ 1
~1
...
... ...
... ...
...
...
Assessment times
r i l FI t+ 1
... ...
r7l FI ... rq r-q ... r7l rl ... (b)
... ...
(d 1
Figure 5.1. Four possible missing data patterns at time t given observed pair ( q - l ) , yi(t-l)) at time t - 1. in Figure 5.1, with Patterns (a) and (d) correspond to the two missing data patterns for the BMMDP case with k = 0. Denote the four patterns by
Let Iitl = 1 if witl is observed and 0 if otherwise (1 5 1 5 4). As discussed in Chapter 2, a popular model for such nominal response is the generalized logit or multinomial response model:
T
Ptl = ( P t l l , *
*
’
,& ( t - 1 ) )
T >
Ytl =
(7111,
*
. . ,7 t 1 ( t - l ) )
Under the assumptions, zit are observed for all subjects at baseline t = 1 and so pill = 1 if 1 = 1 and 0 if otherwise. Thus, starting at t = 2, we apply (5.107) successively to model the occurrence of missing data patterns at each time t. We then estimate riuust using a relationship between piti and 7riUuStsimilar to (5.97) for the univariate T i t .
5.2 MODELS FOR LONGITUDINAL S T U D Y DESIGNS
303
By applying BRIIMDP, we have for s = t :
t
for s
< t:
and for s > t , s-1
Similarly, for u = u,we have
In many applications, pztl may only depend on the most recently observed responses, zit = zz(t-l), and the predictor of the model (5.107) under this ,BLzi(t-l). Markov condition is simply: By assuming BMMDP in the above example, the total number of missing data patterns reduces from m2 to m for k = 0 and t o 3m - 2 for k = 1 (see exercise). For BMMDPs with lag time k larger than 1, the number of potential missing data patterns at each time t will increase at the rate of ( k 1)2 (see exercise). Although possible, it is difficult to model and estimate parameters given insufficient replications of missing data patterns in most applications. On the other hand, BMMDPs with large lag times are unlikely to occur under MAR in real study applications. Since MAR is often posited for modeling subject dropout due to deteriorated/improved health
+
+
304
CHAPTER 5 MULTIVARIATE U-STATISTICS
and related conditions, it is unlikely that one outcome will be continuously observed while the other is not. Thus, the BMMDPs with lag 0 and 1 provide good approximations for modeling the occurrence of missing data under MAR in most applications. As in Example 9, we may want to adjust the asymptotic variance of Zzyst in (5.106) to account for the variability of in the estimated 7rizyst. The discussion parallels that for Example 9 and in particular, the formula in (5.101) can be applied t o obtained the adjusted asymptotic variance.
5.3
Exercises
Section 5.1 1. Consider a study with n subjects and let zi = (xi,yi)' be a pair of variables from the ith subjects (1 5 i 5 n). Consider a simple linear regression: yi = Po
+ Plxi + ~ i , ~i
N
i.i.d. (0, 0 2 ),
15 i 5 n
pl
(a) Find the least-square estimate of pl. (b) Show that p1 = G S n yp , where snz = [Cr=, (zi- 2n)2/ ( n - l)] and
2 is the Pearson correlation given in (5.1). 2. Consider the model in (5.3). (a) Define an GEE I1 of the form w,(O) = C:'"=,iSi
c = (Pz,
Py,
4,g;,
T
QZY)
to estimate
*
(b) Select an appropriate G, and Si so that the GEE I1 in (a) yields the sample mean, variance and covariance for the respective components of 3. Let e , be defined in the proof of Theorem 1 of Section 5.1.1. Show E (eLe,) 0 implies that en 0. 4. For the one-group U-statistic vector U, defined in (5.4), show
c.
- f p
---f
5. Consider the product-moment correlation in Example 4 of Section 5.1.1. Verify (5.19). T N ( p , E ) ,a bivariate 6. In Problem 5, assume that zi = ( x i , y i ) ' normal with mean p and variance E = [ o k l ] . (a) Find Ce = V a r hl (z1) and express it in terms of a k l . N
(-
1
(b) Show that the asymptotic variance of (1 - p2)%.
2 in
(5.19) reduces t o
0;
=
305
5.3 EXERCISES
7. Consider the Concordance correlation coefficient pccc defined in (5.37). (a) Show E (h (yi, yj)) = 8, where h (yi,yj) is defined in (5.39) and 8 in (5.38). (b) Find a bivariate U-statistic vector U, = (U,1? Un2)T such that
E
(unl) = 2a12’
E
2 ( u n 2 ) = 01 - 2012
+ 0; + (PI
-~
2
)
~
(c) Find the asymptotic distribution of Un. (d) Find the asymptotic distribution of the estimated CCC, ~,,, = 8. Verify (5.26)) (5.29)’ (5.36), (5.40)’ (5.41)’ and (5.44). 9. For the kernel function h (yli, y l j ; y21, yzm) defined in (5.47), find a symmetric version of this kernel. 10. For the K-group U-statistic vector defined in (5.46). (a) Show that the projection of the U-statistic can be expressed as:
2.
where h k l (ykl) is defined in (5.52). (b) Use the result in (a) to verify (5.51). (c) By applying the central limit theorem to the centered projection - 8 in (5.51)’ show that 6, has the asymptotic distribution given by (5.53). 11. Let U, and 6, be defined in (5.46) and (5.51). Let en = fi (U, - 6,).Show that E (e;e,) + 0 as n -+ 00. 12. Find a symmetric version of the kernel function in (5.56). Section 5.2 1. Consider the estimate defined by a multivariate U-statistic in (5.57). Show that the asymptotic variance of can be expressed in (5.58). 2. Consider the U-statistic vector 6 in (5.60) for testing the hypothesis of no location shift between two CDFs in a longitudinal data setting in (5.59). (a) Find the asymptotic variance Ce of 5. (b) Construct a consistent estimate of Ce in (a). 3. Consider the U-statistic vector ? = (;)-’ h ( z t ,zJ)defined by the kernel h ( z t ?z J )in (5.61) for Kendall’s T . (a) Find the asymptotic distribution of ?. (b) Use the asymptotic distribution o f ? in (a) to develop an asymptotic test for the null of no association between xzt and yzt over time, Ho : r = 0.
en
5
5
x:=l
306
C H A P T E R 5 MULTIVARIATE U-STATISTICS
4. Consider the U-statistic vector G defined in (5.62) for inference about Kendall's r b within a longitudinal data setting. (a) Find the asymptotic distribution of G. (b) Find the asymptotic distribution of ?b = f 8 , where f (.) is defined in (5.63). 5. Show that (5.64) and (5.67) define the same estimate of 3; based on the observed data. 6. Verify (5.69), (5.70), (5.73), (5.75), (5.76) and (5.77). [Hznt: E ( r l t ) = E (r2t) in (5.73).] 7. Consider the estimate 6 defined in (5.83). (a) Show that G is consistent, that is, G 8, where 6 is defined in (5.78). (b) Verify (5.84) and (5.86). [Hznt: E (Rz)= E (R1) in (5.86).] -2 T 8. Consider the estimate 8 = (3;,G;, . . , ,a,) with 3: defined in
(-1
- f P
h
(5.90). (a) By verifying (5.91), show that 8 is an unbiased estimate of 6 = (a:, a;, . . . , a&)T with 0: denoting the variance of yzt. (b) Show that 3; has the same asymptotic distribution as the estimate of l?: definition in (5.67) by applying the results in (5.92), (5.93), and Slutsky theorem. (c) Evaluate the asymptotic variance Ce of G given in (5.94). (d) Use the result in (c) to construct a consistent estimate of Ce. -2 -2 9. Consider the estimate 6 = (ol, a 2 , .. . ,sf)' defined in (5.90). Let T z t (y) be modeled by (5.96) and (5.97). (a) Assume y is known and evaluate its asymptotic variance CQ = 4Var (El ( y l , rl)) given in (5.94). (b) Evaluate Ce = 4 [Var (Gl ( y 1 , q . y ) ) +a] with Q, given in (5.101) if y is estimated by 7satisfying (5.99). 10. Verify (5.100), (5.101), and (5.105). 11. In Example 10, let A
h
1 -1 g2st (zz, Z J , rz,q)= ~ ; s ~ J x , ~ z z t ~ J x t h (22 2s . t Z J ) -1
-1
g3st (zz, Z J ,rz,r J )= T z y s T J y s r z y t r J y t h 3 s t(2%) ZJ)
1
h2st (zz, Z J ) = - ( G s h 3 s t (zz, Z J ) =
Assume
Tizyst
are known.
2 1
-
q s )
- (Yzs - YJS) 2
(Gt -2Jt) (Yzt - Y J t )
307
5.3 EXERCISES
(a) Show that
E (g2st ( Z i , z j , 'i,
' j ) ) = COV ( X i s , Zit)
>
E (g3st (Zi,z j , '2,
' j ) ) = flyst
(b) Let C(i,j)Ec;gst ( z i , z j , r i , r j )
h
Pxyst
=
\/C(i,j)ECzn g2ss ( Z i , z j : I'i, ' j ) C(i,j)Ec; g2tt ( Z i , z j , r i , ' j )
where gst ( z i ,z j , ri, rj) is defined in (5.104). Show that and asymptotically normal estimate of pxyst. (c) Find the asymptotic variance of Fxyst. h
(4 Let Cxy = (Fxyll,. . . , Pxylm, Pxy21, . . . h
h
:
FxYStis a consistent
T
Pxymm) . Find the asymp-
totic distribution of CXy. 12. In Problem 11, assume that Tixyst are unknown and modeled using the approach described in Example 10. Let y denote the parameter vector in the model for 7rixyst. (a) Find the asymptotic variance of Fxyst. (b) Find the asymptotic variance of Fxy. 13. Consider a study with bivariate outcomes zit = ( Z i t , it)^ collected over rn assessment times (1 5 t 5 m, 1 5 i 5 n). Let k denote the lag time defined in Example 10 to characterize BMMDPs. Show (a) The total number of missing data patterns is 2m. (b) The total number of missing data patterns under MMDP on each xit and yit is m2. (c) The total number of missing data patterns under BMMDP is m if lag time k = 0 and 3m - 2 if lag time k = 1. (d) The total number of missing data patterns at each time t is ( k 1)2 (1 5 t 5 m).
+
This Page Intentionally Left Blank
Chapter 6
Functional Response Models In Chapter 5, we introduced multivariate U-statistics, developed an inference theory for such vector-valued statistics for longitudinal data analysis and discussed applications of this new theory t o address an array of important statistical problems involving modeling second order moments and statistics defined by responses from multiple subjects. From a methodologic standpoint, the theory extended the many classic univariate U-statistics based models and tests t o a longitudinal data setting and addressed missing data under the general MAR assumption so that these models and tests can provide more powerful applications in modern-day research studies populated by longitudinal study designs. From a practical application perspective, Chapter 5 demonstrated the widespread utility of multivariate U-statistics by providing effective solutions to problems that involve second order moments, such as in modeling the variance and correlation: and functions of multiple subjects’ responses in the fields of biomedical and psychosocial research. A distinctive feature of the new methods is the paradigm shift from classic, univariate- and continuous-variable based U-statistics to applications of discrete outcomes within the context of longitudinal study data. This methodological breakthrough opens the door to many new applications that we continue to expand upon in this chapter. At the same time, the new theoretical development presented in Chapter 5 to address new applications of U-statistics also highlighted some critical limitations - in particular, the lack of a systematic approach to modeling complex high order moments and functions of responses from multiple subjects. Typically, in such cases, models are developed to address specific problems in an ad hoc fashion with no formal framework for either model estimation or inference. Addition-
309
310
C H A P T E R 6 FUNCTIONAL RESPONSE MODELS
ally, the inability to accommodate covariates within such a modeling setting seriously limits the potential for the broad application of such methods. In this chapter, we introduce a new class of distribution-free or semiparametric regression models to simultaneously address all these limitations. By generalizing traditional regression models through defining the response variable as a general function of several responses from multiple subjects. this class of functional response models (FRM) subsumes linear, generalized linear and even nonlinear regression models as special cases. In addition. FRM provides a general platform to integrate many classic nonparametric statistics and popular distribution-free models, as discussed in Chapters 25, within a single. unified modeling framework to systematically address the various limitations of these statistics and models. For example, by viewing the classic, nonparametric, two-sample Mann-Whitney-Wilcoxon rank sum statistic as a regression under FRM, we are readily able to generalize it to account for multiple groups and clustered data as in a longitudinal study setting. Such extensions are especially important for effectiveness studies which by their design introduce variability, as well as for microarray studies in which additional heterogeneity in expression may be introduced simply from the type of sample used, such as biopsies from skin tissue. With FRI4, we are also able to formally address the limitations of the classic mean based, distribution-free regression model for inference about second and higher order moments, such as the random effects parameters in the linear mixedeffects model, and causal-effect parameters in structural equation models. both of which are popular in psychosocial research and more recently. in cancer vaccine trials with molecular endpoints. To further achieve the over arching scope of this book for analyses of modern day applications, we also systematically address MCAR and MAR assumptions. In Section 6.1, we highlight the limitations of the classic distributionfree regression model discussed in Chapter 2 and 4 for modeling second and higher order moments and how the classic models can be extended to overcome this major difficulty by using functional response. In Section 6.2, we formally define FRM and discuss their applications in integrating classic non-parametric statistics and distribution-free models discussed in Chapters 2-5. In Section 6.3, we discuss inference for FRM by developing a new class of U-statistics based estimating equations. In Section 6.4, we address missing data. We discuss the impact of missing data on model inference under different missing data mechanisms and develop procedures for consistent estimation of model parameters under the two most common WICAR and MAR assumptions.
6 . 1 LIMITATIONS OF LINEAR RESPONSE MODELS
6.1
311
Limitations of Linear Response Models
In this section, we illustrate some of the key limitations associated with the classic, mean-based distribution-free regression models discussed in Chapter 2 for modeling second and higher order moments as well as statistics defined by multisubject responses by way of examples. We start with a relative simple analysis of variance model (ANOVA) model. Example 1 ( A N O V A ) . Consider the ANOVA model discussed in Chapter 2 for comparing K treatment conditions. Let y k z denote some continuous response of interest from the ith subject within the kth group for 1 5 i 5 n k , where n k denotes the sample size of group k (1 5 k 5 K ) . Under the distribution-free ANOVA setup discussed in Chapter 2, we model the mean of Y k J as follows:
E
(ykz) = p k ,
15 i 5
nk,
15 k 5 K
(6.1)
where E ( . ) denotes mathematical expectation. The above model is then used to test hypotheses concerning the mean responses. p k , across the different treatment groups. As noted in Chapter 3, for many real data applications, especially in the area of effectiveness research, there may also exist difference in variance in addition to the mean, in which case, a comparison of such second order variability among the different groups is also of interest. For example, if two treatments have the same mean response, but different variances. the treatment associated with the smaller variance in response may be preferable in terms of its generalizability and cost effectiveness two important issues in effectiveness research. However, a test of difference in variance across the treatment conditions is beyond the capability of the mean response based model in (6.1), since it is restricted to modeling only the mean of Y k z . In Chapter 3, we discussed a U-statistics based approach for comparing the variance between two treatment groups. By setting K = 2 within the context of Example 1 and denoting the variance of Ykz by ci ( k = 1 , 2 ) , we can use the following U-statistic to compare the variance between the two groups:
Since 8 = E ( U ) = 07 - cg,we can use this U-statistic to test the null hypothesis of equal variance between the two groups, Ho : el = cg. Although one may generalize this statistic to compare the variance for more
312
C H A P T E R 6 FUNCTIONAL RESPONSE MODELS
than two groups using multivariate U-statistics as illustrated in Chapter 5 for extending the two-sample Mann-Whitney-Wilcoxon rank sum test to a multi-group setting, such an approach is ad hoc. Alternatively, a regression model provides a general framework to systematically address this and other similar problems involving higher order moments. Example 2 ( Generalized A NOVA for comparing variance). Consider a generalization of the ANOVA in Example 1 to compare variances across the K groups. To this end, let = V a r ( y k i ) denote the variance of yki (1 5 k I: K ) . Let
02
Then, the mean of the two-subject response based functional f given by
(ykz,y k j )
is
The above model has a similar form as the mean-based distribution-free ANOVA in (6.1). If we want to test the hypothesis of variance homogeneity, we can simply test the following linear contrast: = o:,
for all 1 5 I , rn 5 K
H, : O! # 0:.
for some 1 5 1 # m
HO
:
O!
versus
IK
The mean based ANOVA model in (6.1) differs from the one in (6.2) in several respects. First, the classic mean based ANOVA only involves a single response ykz. while the latter model involves a pair of responses, ykZ and yk3, from two different subjects (within the same group). Second, the dependent variable in the mean based ANOVA is Y k z , whereas the dependent variable for the model in (6.2) is defined by a more complex, quadratic function of two individual responses ykz and yk3. It is these extensions in the dependent variable that allow the distribution-free model in (6.2) to overcome the fundamental limitation of the mean based ANOVA and provide inference for hypotheses concerning variance or second order moment of the response. Example 3 (Random-factor ANOVA). A classic one-way ANOVA with random factor levels is defined by
ekz
N
+ + ekz,
i.i.d.N (0.0;) i.i.d.N ( o , o ~ ) ,XI,I € k l < 1 5 i 5 n, 1 5 k 5 K
ykz = Y,
XI, N
(6.3)
313
6.1 LIMITATIONS OF LINEAR RESPONSE MODELS
where p, g; and c2 are parameters, N ( p ,g2) denotes a normal distribution with mean p and variance g2 and I stochastic independence. Unlike the fixed-factor ANOVA in Example 1, the random factor level effect, Ak, is now a latent variable and as a result, yki are not independent across the subjects within each cluster k , though yki are independent across the different clusters k (1 k IT). As discussed in Chapters 4, the random-factor ANOVA in (6.3) is a special case of the linear mixed-effects model and widely used to model rater agreement and instrument reliability in biomedical and psychosocial research. As shown in Chapter 4, the variance of Y k i and covariance between any pair of within-cluster responses, yki and ykj, are given by
< <
Var ( y k i ) = V a r (A,) Cow (yki, y k j )
= Var (A,)
+ V a r ( E k i ) = + o2
(6.4)
0:
2
= oA, 1 5
i
< n, 1 < k < K
Thus, the correlation between two within-cluster responses given by
yki
and
ykj
is
Under the model assumption, this within-cluster or intraclass correlation (ICC) is a constant and a ratio of the between-cluster variability 0; to the total variability u: + c 2 . A major drawback of the model in (6.3) is the normal-normal distribution assumption for the response and random factor. In Chapter 5: we discussed a U-statistics based approach to facilitate distribution-free inference about p. Here, we develop a regression model to provide distribution-free inference for the between-cluster variance 0 ; . In many applications, we may want to test the null hypothesis, Ho : 0: = 0. If this null is not rejected, we can then ignore the between-cluster variability and use a simpler model for y k i without any random factor. To model 0: directly, first observe that under the model assumptions in (6.3), if ( i , j )# (i’:j’), that is, both i and j are different from i’ and j’, we have
In other words, the parameter of interest u: is the mean of a functional of response of the form: f (yki, y k i t ; ylj, ylj’) = (yki - ylj) (yki’ - y l j t ) .
314
C H A P T E R 6 FUNCTIONAL RESPONSE MODELS
To express (6.6) in a vector form, let yk = ( y k l , . . . , ykn) T . Then, yk are independent across k , though the components within y k are correlated. Let
( Y k ,Y l ) = (fl2.34,. . . , f l 2 . ( n - l ) n * f13.24: . . . > f(n-S)(n-2).(n-l)n)
h
T
(d)= a,l3c,. 2
where 1 3 ~ ; denotes the 3C,“ x 1 column vector of 1’s. The number 3Cg is the dimension of the vector f (yk,yl),which is also the total number of combinations of { i , j } and {i’,j’}: CtC; (see exercise). The factor accounts for the fact that the order of { i , j } and {i’,j’} does not matter. Then, we can express (6.6) succinctly as:
i
i
Note that unlike the original parametric model in (6.3), the nuisance parameter, a2, no longer appears in the distribution-free model in (6.7). Note also that the above model differs from the one in (6.1) for Example 1 in that it involves a vector-valued functional response. This difference reflects the need to group correlated responses into independent outcomes and is analogous to modeling clustered data such as arises from longitudinal study designs. In Chapter 4, we introduced the notion of internal consistency for grouping item scores in an instrument to form dimensional measures of latent constructs in psychosocial research and discussed a normal distribution based model for Cronbach’s alpha coefficient for assessing such consistency. As noted there, the major difficulty in applying such a model to real data, particularly in psychosocial research, is the distribution assumption which is fundamentally at odds with the data distribution of item scores for most instruments. We addressed this problem using theory of multivariate Ustatistics in Chapter 5 . In the next example, we show how to model this index using a regression-like formulation. Example 4 ( M o d e l for Cronbach’s alpha). Consider an instrument (or questionnaire) with m individual items and let yik denote the response from the ith subject to the kth item for a sample of n subjects. The Cronbach’s coefficient alpha for assessing internal consistency of questionnaire with m items is defined as follows:
6 . 1 LIMITATIONS O F LINEAR RESPONSE MODELS
315
where 0; = V a r (y,k) and g k l = Cov (y,k, y,~) denote the variance and covariance of the item responses, respectively. In (6.8). cy is expressed as a function of two parameters 7 and and thus we need to define two response functionals to model this parameter of interest. To this end, let
fl ( Y i , Y j ) = 21 (Yi- Y j ) T (Yi- Y j ) f2
(Yi,Y j )
= IT (Yi- p ) (Yi- p)T 1
Then, we have:
or simply,
Thus, by defining two functionals of response in (6.9), we are able to model a complex function of second order moments using an ANOVA-like model in (6.10). As in the two previous examples. (6.10) are fundamentally different from the mean based ANOVA by the form of the dependent variable. In this example, the mean response, h ( a , is a more complex nonlinear function of the parameters. The three examples above demonstrate the limitations of existing regression models for modeling complex functions of second and high order moments. Although the examples are contrasted with the relatively simple, distribution-free ANOVA model, the comparisons nonetheless illustrate an inherent weakness with the classic distribution-free regression model. For example, the class of distribution-free generalized linear model (GLM) for continuous. binary and count response is defined by
a),
~ ( y I ,x,) = h ( X T P ) ,
15 i I n
(6.11)
where y, denotes a response and x, a p x 1 vector of independent variables from the ith individual of a sample with n subjects, E ( y , I x,) the conditional mean of y, given x,, P a vector of parameters and g (.) a link function. Although (6.11) generalizes the multiple linear model to accommodate more complex types of response variables such as binary, the left side retains the
316
C H A P T E R 6 FUNCTIONAL RESPONSE MODELS
same form as in the mean-based ANOVA, in that it is restricted to a single subject response yi. Although GEE I1 discussed in Chapter 4 is capable of modeling second order moments, its applications are limited to relatively simple problems. Further, GEE I1 does not address many popular statistics that are complex functions of response from multiple subjects, such as the Mann-W hit ney- W ilcoxon test . Note that the distribution-free GLM in (6.11) has a slightly different form than the definition in Chapter 2 in that the conditional mean of yi given xi is expressed as a function of the linear predictor q?,= using the inverse of the link function g ( . ) for the convenience of discussion in this chapter. But, other than this difference in appearance, (6.11) defines exactly the same distribution-free GLM as in Chapter 2. Example 5 ( Wilcoxon signed rank test). Consider the one-sample Mann-Whitney-Wilcoxon signed rank test. Let yi denote the continuous response of interest from an i.i.d. sample of n subjects (1 5 i 5 n). Let
x T ~
where I{.}denotes a set indicator. Then, the mean of the functional response f (yi, yj) is given by (6.12) As discussed in Chapter 3, the parameter of interest 1-1 in the above model can be used to test whether yi are symmetrically distributed around 0. The dependent variable, f (yi, yj), in the model defined in (6.12) is again a function of multiple subjects’ responses as in the previous examples. However, what is different from the other examples considered earlier is that the parameter of interest p in this model cannot be expressed as a function of moments. For such an intrinsic or endogenous multisubject based functional, moment-based methods such as GEE I1 do not apply.
6.2
Models with Functional Responses
The models discussed in the examples in the previous section differ from the distribution-free GLMs discussed in Chapters 2 and 4 in that the response variable can be a complex functional involving multiple subjects’ responses rather than a linear function of a single subject response. This generalization not only widens the class of existing regression models to accommodate new challenges in modeling real data applications, but also
6.2 MODELS WITH FUNCTIONAL RESPONSES
317
provides a general framework to generalize classic nonparametric methods such as the Mann-Whitney-Wilcoxon rank sum test and unify them under the rubric of regression analysis. Let y, be a vector of response and x, a vector of independent variables from an i.i.d. sample (1 5 i 5 n). By broadening the response variable as a general non-linear functional of several responses from multiple subjects, we define the general functional response model as follows:
[f (Yz,. . . . , Y z q ) 1
. . , X Z q ] = h (x,,,.. . ,xzq;P) ( i l , * .. ,i4) E c 4"
where f (.) is some function, h (.) some smooth function (with continuous second order derivatives), C t denotes the set of )(: combinations of q distinct elements ( i l , . . . , i4) from the integer set (1, . . . , n} and P a vector of parameters. By generalizing the response variable in such a way, this new class of FRM provides a single distribution-free framework for modeling first and higher order moments, such as mean and variance, and endogenous multisubject based statistics, such as the Mann-Whitney-Wilcoxon test. More importantly, we can readily extend such models t o clustered and longitudinal data settings and address the inherent missing data issue under a unified paradigm. As noted earlier, the distribution-free regression models discussed in Chapters 2 and 4 are all defined based on a single-subject response. The generalized linear model extends linear regression to accommodate more complex types of response such as binary and count variable. Yet, the fact remains that GLM is still defined by a single subject response. The extension only occurs to the right side of the model specification by generalizing the linear predictor vz = xTP in linear regression to a more general function of x,'p in GLM. But the left side of the model definition remains identical to the original linear regression model. For ease of exposition, we define the FRM and discuss its applications for comparisons of multiple samples (or treatment groups) and regression analyses separately. We start with models for comparing multiple samples involving inference based on K (22) groups.
6.2.1
Models for Group Comparisons
Consider the following class of models for comparing K independent groups: [f (YE,
, . . . ,Y i i q ; . . . ;Y
K ~ .~. ,,y.~ j , ) = ] h (6) (il,.
. ,i4) E c;,, 1 5 q *
(6.13)
318
C H A P T E R 6 FUNCTIONAL RESPONSE MODELS
where y k 3 denotes a vector of response from the j t h subject within the kth group, f some vector-valued function, h (0) some vector-valued smooth function (with continuous derivatives up to the second order), 8 a vector of parameters of interest. q some positive integer, n k the sample size of group k and C:k the set of combinations of q distinct elements ( i l , . . . , i4) from the integer set (1,. . . , n k } (1 k K ) . Since no parametric distribution is assumed, (6.13) defines a class of distribution-free GLM-like models for inference about the K groups. Note that Y k 3 are assumed to be stochastically independent across both k and j . The response f of the FRM above is a function of multiple individual responses, y k z l , . . . , ykz,, from within each of the K groups. Example 1 (Generalized A N O V A for modeling variance). Consider the ANOVA like variance model in (6.2). Based on the definition above, this model is an FRM with K independent groups and
(y)
< <
Example 2 (Random-factor A N O V A ) . We can also express the distribution-free ANOVA with random factor levels in (6.6) as an FRM:
Note that although similar in appearance, the random-effect ANOVA is different from its fixed-effect counterpart. For the fixed-effect ANOVA; the responses ykz within the kth group are independent across i, while for the random-effect version, these responses are dependent within each kth group. Thus, for the random-effect ANOVA, we must group the responses within each random factor t o form independent vectors yk when defining the FRM for this model. The one- and two-sample Mann-Whitney-Wilcoxon rank based tests are often used as a robust alternative to parametric tests for comparing mean differences. We now define the FRMs that give rise to these test statistics. Example 3 ( Wilcoxon signed rank t e s t ) . Let yi denote a continuous response of interest from an i.i.d. sample of size n. Let
319
6 . 2 MODELS W I T H F U N C T I O N A L R E S P O N S E S
+
Then, p = E [ f (yz,y,)] = P r (yz y, 5 0). Thus, the null hypothesis that yz are symmetrically distributed around zero can be expressed in terms of the model parameter as HO : p = Note that the above FRM has the same form as the model in (6.12) considered in the previous section for the discussion of limitations of the classic regression paradigm. Example 4 ( M a n n - Whitney- Wilcoxon rank sum t e s t ) . Let ykt denote some continuous response from two i.i.d. samples (1 5 i 5 nk, k = 1.2) . Define an FRM as follows:
i.
K=2,
(6.14)
q = l
f ( Y l z , Y 2 J ) = I{ylz
h ( p ) = p.
1L
I 721,
1L j 5
722
As discussed in detail in Chapters 3, the model parameter p = P r (ylz 5 y2,) can be used to express the null hypothesis of no location shift when comparing the CDFs of the two samples, that is. HO : p = Thus, by using a functional response, we can frame this classic test within a regression-like model in (6.14). Example 5 (ROC analysis). Consider the problem of assessing the quality of a diagnostic test for detecting some medical or mental health condition of interest, such as an infectious disease or depression. Let y12 and ~2~ denote the continuous test outcomes from the diseased and nondiseased samples (1 5 i 5 n1,l 2 j 5 722). As discussed in Chapter 3, the area under the receiver operating characteristics curve (AUC), 6' = P r (ylz 5 y2,), is widely used to evaluate the performance of a diagnostic test. By identifying ykz as the two samples in Example 4, we can see that this parameter of interest 6' is precisely the model parameter p in (6.14). Thus, we can define the same FRM as in the previous example to model the AUC of the diagnostic test. Although the two FRILIs are the same. interest is no longer centered around testing the null, HO: 6' = within the current context. In fact, 19 is well interpreted in its entire range (0.1). If the diagnostic test determines disease status purely in a random fashion. that is, with a probability 0.5 for issuing disease and nondisease outcomes, the ROC curve, the plot of (F1 ( c ), Fz ( c ) )with FI, ( c ) denoting the CDF of ykz, is a straight line with a 45" angle between 0 and 1 and thus the corresponding AUC is 0.5. On the other hand, if the test has some diagnostic value, that is, detecting disease status with a probability higher than 0.5, then p > 0.5. Thus, within the current context, it is of interest to test null hypotheses of the form: HO : p = a,where a is some known constant between 0.5 and 1. The real advantage of expressing these popular models in the form of an FRhI is to generalize them. For example, we generalized the two-sample
i.
i,
320
C H A P T E R 6 FUNCTIONAL RESPONSE MODELS
Mann-Whitney-Wilcoxon rank sum test to compare three samples in Chapter 5. Within the framework of FRM, we can readily generalize this classic test to a general setting involving any number of groups without running into the notational complications found with multivariate U-statistics in Chapter 4, as illustrated in the next example. Example 6 (K-sample Mann- Whitney- Wilcoxon rank s u m test). Let Y k i denote some continuous response of interest from K i.i.d. samples (1 5 i 5 n k , 1 5 k 5 K ) . Let L = );( and q = 1. Consider the following FRM:
It follows that
E
[f( y k t , Y h j ) ] = h ( 0 ) = o k h ,
15 i
15 j 5 nh, 1 5 k
If ykJ have the same distribution across the K samples, then o k h = $ for all 1 5 k < h 5 K and vice versa. Thus, we can test the null hypothesis, Ho : Okh = for all 1 5 k < h 5 K , to determine whether there is any location shift in CDF across the K independent samples. In Chapter 5, we developed a U-statistics based approach for simultaneously comparing the mean and variance between two groups. By combining the classic mean-based ANOVA with the generalized ANOVA for variance discussed earlier) we can readily generalize this approach by developing an FRM to jointly compare the mean and variance of a response across multiple samples, as illustrated below. Example 7 (Model f o r K-sample mean and variance). Consider K i.i.d. samples and let y k B denote the response of interest from the ith subject within the kth group with mean ,%k and variance cri ( 1 5 i 5 n k , 1 5 k i K ) . Let
(6.15)
Then, f ( Y k i , Y k j ) = h ( 8 ) = 81, and the parameter vector 81, contains both the mean and variance of the kth sample. If HO : 81, = 8 for all 1 2 k 5 K ,
6.2 MODELS WITH FUNCTIONAL RESPONSES
321
then yki not only have the same mean, but the same variance across the K groups as well. As discussed in Chapter 4, we can also perform joint inference about the mean and variance by using GEE 11, which extends the mean-based distribution-free GLM t o include additional models and estimating equations for the variance parameters. The difference is that the model for the variance component in GEE I1 is not really in the form of a regression since the dependent variable on the left side of the model specification also involves parameters. For example, consider again the problem for comparing the mean and variance of y k i across K groups in Example 7. If we use GEE I1 to model both the mean and variance simultaneously, we have (6.16) 2
where S k i = (yki - p k ) . Since the dependent variable S k i for the variance component in (6.16) is a function of the group mean p k , (6.16) is not a regression in the usual sense. In contrast, the dependent vector f (yki, ykj) of the FRM in (6.15) does not contain any unknown parameter and thus conforms to the usual notion of regression analysis. Count response arises quite often in biomedical and psychosocial research. The standard approach for modeling such a response is the Poisson distribution. Under the Poisson model, the mean and variance are constrained to be the same. However, in many applications, the variance is often larger than the mean, a phenomenon known as overdispersion. As discussed in Chapter 2, a common cause for overdispersion is data clustering and a popular approach for modeling this type of overdispersed data is the Negative Binomial (NB) model. In practical applications, overdispersion can be assessed by the deviance and Pearson’s chi-square statistics (see Chapter 2). The appropriateness of an NB as a parametric model to address overdispersion can also be formally assessed by the Vuong likelihood-ratio test (Vuong, 1989). Within the framework of FRM, we can readily develop a distribution-free model to relax the parametric assumption in KB and to assess the appropriateness of fitting such an NB-like model. Example 8 (Negative binomial model). Consider K i.i.d. samples and let yki be some count response of interest from the ith subject within the kth group (1 5 i 5 nk,1 5 k 5 K ) . Following Chapter 2, the standard parametric approach for modeling such a response type is the Poisson model. By removing the random component of GLM, we obtain a distribution-free, Poisson-like log-linear model as follows:
(6.17)
322 If
yki
CHAPTER 6 FUNCTIONAL RESPONSE MODELS
does follow the Poisson distribution for each k , then we have
When overdispersion occurs, the above will not hold true. If KB model. we have:
yki
follows the
(6.18)
where a k is a parameter indicating the degree of overdispersion within each sample (1 1. k K ) . By combining (6.17) and (6.18), we arrive at an FRM that addresses overdispersion without imposing any parametric model on the data distribution:
<
As in the parametric model case, we can assess overdisperson under this distribution-free, NB-like FRM (6.19) by testing the hypothesis:
Ho :
=0
versus
Ha: a k > 0,
15 k
5K
Note that in the parametric NB model N B ( p ,a ) , a = 0 does not correspond to a valid NB distribution since the PDF of N B ( p . a ) becomes undefined in this case. In addition, a > 0. Thus, the constraint a > 0 must be imposed when fitting the parametric NB model. In contrast, this boundary issue does not arise for the FRM defined in (6.19) since a,+can be negative as long as the right side in (6.18) stays positive. Note also that the Poisson model is not a member of the NB family, though it can
323
6.2 MODELS WITH FUNCTIONAL RESPONSES
be viewed as the limiting case of the NB distribution when a -+ 0. As a result, assessing the appropriateness of Poisson within the NB context is complicated since the Poisson model corresponds to the boundary value 0 in the NB family and regular maximum likelihood theory may not apply. However, for the distribution-free FRM defined in (6.19), this boundary issue is again irrelevant since a can be 0 or even negative, in which case, there is under-dispersion in the data. As discussed in Chapter 2, count outcomes arise in many studies in biomedical and behavioral research that do not follow either the Poisson or NB model because of structural zeros in the data distribution. In such cases. the frequency of zeros in the data exceeds their expected frequency by either the Poisson or YB model, also causing overdispersion. The zero-inflated Poisson (ZIP) is widely used to address this type of overdispersion. We discussed the parametric ZIP and the difficulty in developing a distributionfree version of the model under the distribution-free GLM in Chapter 2. The major problem in the latter is the general lack of information to identify model parameters from a mixture-distribution based model such as ZIP. By including a component for modeling the second order moment. we will be able to define an FRM to identify the parameters of ZIP in a distribution-free setting, as we discuss in the next example. Example 9 (Zero-inflated Poisson). Within the context of Example 8, now consider modeling ykz in the presence of structural zeros using the ZIP model. This ZIP approach models the structural zeros separately using a dedicated point mass at zero so that the remaining amount of sampling zeros, together with the positive responses. can be modeled by the Poisson. Let fo (y) denote the PDF of a degenerate distribution centered at zero and f p , k (y) the PDF of a Poisson with mean p k . Then, the PDF of the parametric ZIP for ykz is defined by the following mixture of two component distributions. fo (y) and f p . k (y), fZIP,k (Ykz) =
(1 - p k ) f P , k ( y k z ) + P k f O
ykz=O.l
(ykt)
(6.20)
...., 1 < k < K
In the above, fo (.) absorbs the structural zeros, while f p , k (.) models the remaining observations t o follow the Poisson distribution. Under the distribution-free GLM, only the conditional mean response given the kth sample is specified, that is, (Ykz) =
(1 - P k ) exp ( P k ) '
1I k I K
(6.21)
The above specification will not yield unique estimates of p and p since the mean-based GLhl in (6.21) is overparameterized. One approach to address
324
C H A P T E R 6 FUNCTIONAL RESPONSE MODELS
this issue of model identifiability without reverting back to the parametric ZIP defined in (6.20) is to add a component for modeling the variance of yki. Under this approach, we supplement the mean response with a component for the variance to obtain
By combining (6.21) and (6.22): we obtain a distribution-free, ZIP-like model that rectifies the identifiability issue with the distribution-free GLM in (6.21):
The functional response f ( y k z , ykJ) of this FRM is the same as the one defined in (6.19) for the Poisson-like log-linear FRM. However, the FRM in (6.23) differs from the one in (6.19) in the mean response t o reflect the assumptions between the NB and ZIP models. All the examples above concern cross-sectional data. As in the case of distribution-free GLM, FRM also applies to longitudinal study designs. In Chapter 4, we discussed models for longitudinal study data and contrasted the two popular approaches, the generalized linear mixed-effects model (GLMM) and the generalized distribution-free GLM (or GEE), in terms of their relative strengths and weaknesses. In the next example, we develop an FRM to integrate the strengths of the linear mixed-effects model (LMM) with the GEE. Example 10 (Linear mixed-eflects m o d e l ) . Consider a longitudinal study with K independent treatment groups and m assessment times. Let ykzt denote some response of interest from the ith subject at time t within the kth group (1 5 i 5 nk. 1 5 t 5 m, 1 5 k 5 K ) . We are interested in modeling the growth-curve over time and comparing such curves across the different treatment conditions. For convenience, we assume a linear
325
6.2 MODELS WITH FUNCTIONAL RESPONSES
time trend for each group and allow the slope to vary across the treatment groups. The normal-based LMM has the following form: Ykzt = Pic0 f t P k l b k z = (bkon,
-k bkOz
-
+ t b k l z + Ekzt
(6.24)
i.i.d.N (0,D k )
€kz = ( ~ k ~ l .. .., Ektm)T
-
i.i.d.N (O,G= diag (a:))
where diag (a:) denotes a diagonal matrix with a: on the tth diagonal. In this model, PkO+Pklt represents the mean response of Ykzt for the kth group and bkoz+tbkz the deviation of the ith individual from the group mean. Thus, the fixed-effects parameters, (Pk0,Pkl),represent the intercept and slope for the group response, while the random-effects parameters, Dk,measure the variability of individual deviations from the group mean within each sample. The model error Ekzt captures the variation in response over time within each individual. Inference for parametric LMM under the normal distribution assumption has been discussed in Chapter 4. We now consider distribution-free inference about the parameters of interest by developing an appropriate FRM. First, it is readily checked that ( Y k z t ) = PkO r,
+tPkl
(6.25)
1
aibl
where a&O, and a k b o l denote the diagonal and off-diagonal elements of DI, and S,t = 1 if s = t and 0 if otherwise. By (6.25), we define the vectors of functional and mean response as follows:
(
h = hTl, . . . , hLl, hll, . . . , hL2)
T
326
C H A P T E R 6 FUNCTIONAL RESPONSE MODELS
where 8 denotes the collection of all the model parameters, pk0, Pkl, DI, and G (1 5 k 5 K ) . It follows from (6.25) that the FRM for distribution-free inference of the normal-based LMM in (6.24) is given by
With this FRM, we are able to not only test hypotheses concerning the parameters for the mean response, pko and Pkl, but also those pertaining to the variability of individual subjects, Dk.
6.2.2 Models for Regression Analysis In this section, we discuss FRM for regression analysis. Consider a class of distribution-free regression models defined by [f
(yi,,. . . , yi,) I xil, . . . , xi,] = h (xil,. . . , xi,;8) (ii , . . . , i4) E Cp, 11 q , 1 5i 1. n
(6.26)
where f , h, 8, n and q all have the same interpretation as before. Note that as in the case of group comparison, the vectors of individual response, yi, . . . , y n , are assumed to be stochastically independent. The FRM defined in (6.26) subsumes all regression models discussed in Chapters 2 and 4 as special cases. For example, by setting:
4 = 1,
f (Yi)= Y i ,
hi = h (xi;P )
where h ( . ; . )denotes some nonlinear function, we obtain from (6.26) the distribution-free, nonlinear regression model:
E (yi
1 xi) = h (xi;0) ,
15 i 5 n
(6.27)
If h (xi;P ) = h (xTp),we obtain the distribution-free GLM. Of course, if h ( x i ;P ) = xTP, we obtain the distribution-free linear model. The general FRM in (6.26) also applies t o clustered and longitudinal study data. For example, consider a longitudinal study with m assessment times. Let yit and xit denote the response and vector of independent variables of interest from the ith subject at the tth assessment (1 5 i I n, 1 5 t < r n ) . Let
6.2 MODELS W I T H FUNCTIONAL RESPONSES
327
We then immediately obtain from (6.26) the generalization of the class of models in (6.27) for repeated measures data arising from longitudinal study designs:
In particular, if ht (xit;p) = ht (x$p), (6.28) reduces to the distributionfree GLM for continuous, binary and count response studied in Chapter 4. The classic mean-based distribution GLM only models the conditional mean of the response given the independent variables. As a result, parametric models that differ in second and higher order moments such as the Poisson and negative binomial log-linear models are not distinguishable when modeled under this paradigm. It is this insensitivity to parametric distribution assumptions that enables the distribution-free GLM to provide robust estimates and valid inference under a wider class of data distributions. A potential drawback of this classic approach is that unlike its parametric counterpart it may not provide asymptotically efficient estimates when the data at hand does support the parametric assumption. This limitation is exacerbated by our interest in modeling second and higher order moments in many studies. We can address this weakness with an ad-hoc approach similar to the one discussed in Chapters 4: by adding to (6.28) a component for modeling the conditional variance of the mean given the independent variables. As noted in Example 6, such a GEE I1 model does not constitute a formal extension of (6.28) since the dependent variable for the variance component involves unknown parameters and thus GEE I1 is not really a regression in the usual sense. Within the general FRM framework, we can readily accommodate modeling components for variance or higher order moments without such a conceptual difficulty, as illustrated by the next example. Example 11 (Model for overdispersion). Consider a cross-sectional study with n subjects. Let yz and xi denote some count response and vector of independent variables of interest from the ith subject. The classic distribution-free Poisson-like log-linear model is given by (6.29)
As discussed in Chapter 2, (6.29) provides valid inference under a wide class of distributions including Poisson and negative binomial. If we are interested in formally testing over- or underdispersion, we may use a GEE I1
328
C H A P T E R 6 FUNCTIONAL RESPONSE MODELS
by including a component for modeling the conditional variance of yi given
xi as follows:
This GEE I1 defined by (6.29) and (6.30) allows us to make simultaneous inferences about P and X2. Thus, we can test for over- or underdispersion by considering the following hypotheses: Overdispersion : Ho : X2 = 1 versus
Ha : X2 > 1
Underdispersion : Ho : X2 = 1 versus
Ha : X2 < 1
The dependent variable si depends on pi which is a function of the parameter vector P, and (6.30) is not a regression model. To define an FRM as a formal regression alternative to GEE 11, let = Yi
+ Yj,
hl ( x i : x j ; P : X 2 )= Pz
+pj,
fl ( Y 2 , Y j )
(fi;f 2 I T ,
f =
f2
2
(Y2,Y-j)= (Yi- Yj)
h2 ( x i , x j ; P , X 2 )= X2 (pz
+ Pj) + ( P i - p j ) 2
h = ( h l , h2)T
Then, it is readily checked that the FRM below defines the same model as GEE I1 above (see exercise):
E (f (yi,~
I
j )x i , xj) = h ( x i ,x j ; P
, A ~, ) 1 5 i < j I n
(6.31)
However, unlike the GEE 11, the FRM in (6.31) is a generalization of regression model. One of the most popular models for regression analysis of longitudinal data with continuous response is the linear mixed-effects models (LMM). Consider a longitudinal study with n subjects and m assessment points. Let yzt denote some response, and uzt and v,t vectors of independent variables from the ith subject at the tth assessment. Note that u,t and v,t may overlap and share common variables. Let x,t = (u;,v:)'. The LMM is specified as follows: Yzt = E,
N
uLP + v i b , + e z t ,
b,
-
i.i.d.N ( 0 . 0 )
i.i.d.N (0,G = diag ($)) .
b, 1 E,.
(6.32)
1 5 i 5 n, 1 5 t 5 m
where b, = ( b z l , .. . , b , k ) T , E, = (€,I,. . . , E , ~ ) ~u,t, (v,t) denote the vectors of independent variables for the fixed (random) effect, diag (0;)a m x rn
329
6.2 MODELS W I T H FUNCTIONAL RESPONSES
diagonal matrix with a: on the tth diagonal and D the variance matrix of bi. In (6.32), the parameter vector p describes the population or fixed effects, the elements of the matrix D measure individual deviations from the popular mean or random effects and the error term € i t accounts for the within-individual variation in the response over time. Under the model assumptions, inference for ,f3 and D based on the likelihood theory has been discussed in Chapter 4. A primary limitation of the normal-based LMM in (6.32) is the strong distribution assumption on both the response yi and the random effects bi. For robust inference, we can consider the following distribution-free GLM as an alternative to model the longitudinal data:
However, the above model can only be used to make inference about the fixed effect p. In many applications, we are also interested in inference about the random effect parameters D . The classic mean-based distribution LMM in (6.33) does not contain information for inference about this parameter matrix. By capitalizing on the strength of FRM for modeling second order moments, we can readily develop an FRM model to address this issue. Example 12 (Linear mized-eflects model). To derive a distributionfree FRM for the LMM in (6.32), first define the functional response as follows: fit fist
(Yi,Yj 1 xi,Xj) = Yit + y j t , ( Y i ,Yj I xi, Xj) = (92s
1I t I rn
- Yjs) (Yit - Y j t ) ,
fi = (fll,. . . , f l m )
(6.34) 1I s, t I m
T
f 2 = (f211, f212, ’ . . , fZ(m-l)m, f2mm)
f= Next. we define the mean of functional response by:
T
330
CHAPTER 6 FUNCTIONAL RESPONSE MODELS
where 8 includes all t'he parameters, /3: D and 0 = (o?, . . . , ~ 7 % ) . Then, it is readily checked that under (6.32) f and h satisfy the following (see exercise) :
Structural equations models (SERI) are widely used in psychosocial research. One important application of SEM in this area is modeling the mediational process. We discussed SEM and its applications in mediation analysis in Chapter 4. We also discussed inference for SEM using both the classic likelihood theory and modern GEE 11. As many applications of SEM involve a large number of variables, likelihood methods based on parametric assumptions (often multivariate normal) do not provide reliable estimates. Within the framework of FRM, we can readily develop a distribution-free approach to address this key limitation. In the next example, we illustrate how such a development is carried out for a relatively simple SEM for mediation analysis.
Example 13 (Structural equation model). The primary hypothesis of interest in a mediation analysis is to test whether the effect of a factor 20 (e.g., an intervention) on the response y can be mediated by a change in the mediating variable z. In a full mediation process, the effect of 20 on y is 100% mediated by z , that is, in the presence of z , the pathway connecting zo and y is completely broken so that 20 has no direct effect on y. In most applications, however, partial mediation is more common, in which case z only mediates part of the effect of 20 on y, that is, 20 has some residual direct effect after z is introduced into the model. Consider an i.i.d. sample with n subjects. In addition to the response yz, predictor zzo and mediator zz, let ~ ~ 1. . ,, xzP . denote a set of covariates
from the ith subject (1 5 i 5 n). When modeled using SEM, the mediation process has the following form:
331
6.2 MODELS W I T H FUNCTIONAL RESPONSES
Let T
xi = ( G o , G l , ' ' * >zip) ,
Yi = ( Y i l , Yi2)
T
=
T
(Yi,Zi)
(6.38)
Under the normal distribution assumption, we can express (6.37) in matrix form as follows: y ' - B-1 (rxi
+ ei) ,
ei = (eyi, ~
-
~
i.i.d.N i ) ( 0 ,C,) ~
(6.39)
where B and r are parameter matrices of interest and the matrix C, consists of nuisance parameters. To derive an FRM for distribution-free inference, define the functional response as follows: flt
(Yi)= Y i t , fl
1 2
f i s t ( Y i , Y j ) = - (Yis - Yjs) (Yit - Y j t )
= (fll,f12)',
fi = (f211, f212, f 2 2 2 I T ,
f =
(fT, fgT
Similarly, define the mean of the functional response as: T
hit ( 6 ) = b,Trxi, hist ( 6 ) = b,Tr (xi - xj) (xi - xj)
r T bt + bTC,bt
where b: denotes the tth row of B-l. If z is a mediator, then y20 # 0 and p # 0. The hypothesis of a full mediation process is supported if the condition yzO# 0 and 3!, = 0 holds true and vice versa. In Section 6.2, we discussed FRM based negative binomial (NB) and zero-inflated Poisson (ZIP) models for count response for comparing multiple groups when there is over- or underdispersion in the data. We next discuss similar FRM-based models for regression analysis. Example 14 (Negative binomial model). Consider a sample of n subjects and let yz be a count response and x, a set of independent variables. As in the group comparison case, if there is overdispersion due to data clustering, the Poisson log-linear model is inappropriate and the NB log-linear model is often used to address this issue. The mean-based distribution-free Poisson- or NB-like log-linear model has the form:
(Yz I x,) = A, 1% (&)
=XTP,
1 5 i 5 ,n
(6.40)
332
C H A P T E R 6 FUNCTIONAL RESPONSE MODELS
Since NB has the same mean as the Poisson, the above specification does not distinguish between the two models. As in the group comparison case, we can add a component for modeling the variance to tease out such a second order difference between the two models. Under the NB model, the conditional variance of yi given xi is
V a r (yi I xi) = pi (1
+api),
15 i 5 n
(6.41)
Thus, we define an FRM by adding the following component to the meanbased model in (6.40): (6.42)
where a is a parameter indicating the degree of overdispersion. Note that by adding the above variance component, we have implicitly imposed another constraint upon modeling the relationship between yi and xi,that is, in addition to satisfying the specification for the mean response (6.40), the conditional second order moment must also follow a particular relationship as defined in (6.42). If the latter relationship holds true, then the FRM increases the efficiency for inference about p. Otherwise, inference based on the FRM model may be incorrect. For example, if yi does follow a negative binomial distribution conditional on xi, then (6.42) is true and inference based on the FRM is more efficient than the mean-based model in (6.40). However, if the conditional variance of yi given xi does not satisfy (6.40), inference about p based on the FRM may be incorrect. Under the FRM, we can assess whether there is overdispersion by testing the following hypothesis,
Ho : ct = 0 versus Ha : Q > 0 As in the discussion of the NB model within the context of comparing multiple samples in Section 6.2.1, ct can be negative in the FRM model. Thus, the boundary issue for a in the parametric NB model does not exist and the FRM defined by (6.40) and (6.42) can be used to assess either over- or underdispersion in the data distribution. Example 15 (Zero-inflatedPoisson). Within the context of Example 14, if there are structural zeros in the distribution of yi, then the FRM developed is not appropriate and ZIP must be used to address this type of overdispersion.
333
6.3 MODEL ESTIMATION
As discussed in Chapter 2, the parametric ZIP is based on the following two-component mixture model: fZIP (Pi I Xi) = P (Xi)fo yi = 0 , 1 , . * *
hi)+ [I - p (Xi)]fp (yi 1 X i )
(6.43)
where fo (yi) denotes a degenerate distribution at 0 to account for the structural zeros, fp (yi I xi) the Poisson distribution to model the remaining observations and p ( x i ) a mixing proportion of the mixture model. The two components p (xi) and fp (yi I xi) are typically modeled to depend on two different subsets of independent variables, ui and vi, which may overlap and share some common variables. The Poisson mean of yi and the mixing proportion p ( u i ) are modeled as a function of ui and vi, respectively, as follows:
Inference under the parametric assumption has been discussed in Chapter 2. For distribution-free inference under FRM, we first specify the mean response of yi based on the mixture model as follows:
As in the group comparison case discussed in Section 6.2.1, the above mean specification is not sufficient to identify the parameters, P, and P,. Now let fi (yi,yj) = (yi - ~ j ) ~It .is readily checked that under the parametric ZIP assumption, we have
E
If2
(Yi,Yj) I xi,Xjl
= Pi (1 - Pi) (1
+ Pipi) +
(6.45)
+ P j (1 - P j ) (1 + P j P j ) + + [(I - P i h
-
(1 - P j ) P j I 2
Thus, by combining (6.44) and (6.45), we obtain an FRM for distributionT free inference about the parameters of interest, ,8 = (PL,P ); .
6.3 Model Estimation In Chapters 2 and 4, we discussed inference for regression models under the two dueling likelihood theory and estimating equation based paradigms.
334
CHAPTER 6 FUNCTIONAL RESPONSE MODELS
Regardless of which approach is used, an assumption fundamental to both is independence in the response variable across the sampling subjects. For example, consider a cross-sectional study consisting of n subjects and let yi and xi denote some continuous response and p x 1 vector of independent variables of interest. Suppose that we model the relation between yi and xi using a generalized linear model: (6.46) where ,B is the p x 1 vector of parameters of interest. Then, as discussed in Chapter 2, irrespective of the inference approach used, estimates of p are obtained as solutions to the following equation: n
n
i=l
i=l
(6.47) where Gi (xi) is some p x 1 function of xi (not yi). In (6.47), w, ( p )is a sum of i.i.d. random quantities Gi (xi)Si. As seen in Chapters 2, this particular form of w, ( p ) allows us to apply standard asymptotic techniques, such as the central limit theorem t o show large sample properties of estimates of p obtained from (6.47). Although exceptions occur when responses become clustered such as those arising from repeated assessments on the same subject in longitudinal studies, the basic form of the estimating equation does not change. For example, as discussed in Chapter 4, for longitudinal data analysis, estimates for distribution-free inference are defined by equations identical in form to (6.47). For maximum likelihood inference, the score vector equation is also a sum of i.i.d. random quantities, although the summand may not have an explicit partition involving the theoretical residual (yi - hi) as does the estimating equation for distribution-free inference. For an FRM, the dependent variable is defined by a function of responses from multiple subjects and as a result, estimating equations of the form for the single response based dependent variable given in (6.47) do not apply to such a class of models. To illustrate this important difference as well as to motivate the development of a new class of estimation equations for FRM, consider the generalized ANOVA for modeling variance heterogeneity discussed in Section 6.1. For convenience, consider a single sample so that K = 1. The FRM in this case is given by
E[f(yi,yj)]=E
=02,
1
(6.48)
335
6 . 3 MODEL ESTIMATION
Clearly, the estimating equation for the distribution-free GLM defined in (6.47) does not apply to the above model, since the dependent variables f (pi,yj) are defined by a function of two individual subjects’ responses, yi and y j , and thus are not stochastically independent across i and j . For this simple problem, we know that the sample variance of yi is an unbiased estimate of 0 2 . Further, from the consideration of this model in Chapter 3, we know that we can express the unbiased estimate of g2 as a U-statistic as follows:
Now by applying some simple algebra, we can readily show that Z 2 is the solution to the following U-statistics defined equation (6.50) (,.,)ECzn
( t 3)ECzn
where UnzJ = f ( g Z , g , ) - o2 denotes the theoretical residual for the FRM in (6.48). Thus, by identifying itself as an equation for g2, the estzmatzng equatzon in (6.50) suggests a generalization of (6.47) to accommodate dependent variables defined by functions of responses from multiple individuals. In addition, since Un,, is symmetric with respect to y, and y,, that is, U,, = U,,, Un defined in (6.50) is a U-statistic-like quantity (Un is not a statistic as it contains the unknown parameter 02). Thus, the theories of U-statistics in Chapters 3 and 5 can be utilized to provide the basis to develop estimating equations as well as to study the asymptotic properties of estimates obtained from such estimating equations for FRM. As in Section 6.2, we consider models for group comparison and regression analysis separately.
6.3.1 Inference for Models for Group Comparison Consider the class of FRM for comparing K independent samples defined in 6.13 of Section 6.2.1. Let
, ill , . . . > y ~ i l , ;. . ; YKiKl, . . . , Y K ~ K ~ )
Sil ,...)iK = f i l , ...,i K - h, fil,...?iK = f
,
i k = ( i k ~ .,. . i k q )
where ( i k l , . . . , i k q ) E C:k (1 5 k 5 K ) . For notational brevity, assume that fil ,,,,,iK is symmetric with respect to each i k , that is, fil ,,..,ik,,,,,iK =
336
C H A P T E R 6 FUNCTIONAL RESPONSE MODELS
,,,,;i; ,,.,,iK for any permutation ik of i k (1 5 k 5 K ) . For nonsymmetric f , a symmetric version is readily constructed (see Chapters 3 and 5 ) . Define the U-statistics based generalized estimating equation (UGEE) as follows:
fi,
(6.51)
where G is some known p x 1 matrix function of 8 with p and 1 denoting the dimension of 8 and S i l , , , , , i K . The above UGEE generalizes the estimating equation in (6.47) and GEEs (GEE I and 11) for single response based distribution-free models discussed in Chapter 4 to accommodate dependent variables defined by nonlinear functions of multiple individual responses. The Newton-Raphson algorithm is readily applied to solve the UGEE in (6.51) to obtain estimates of 8. If E(Sil,.,,liK) = 0, it follows that (see exercise) : E (un(6))= E (GSil,...,i K ) = 0 (6.52) il E C ~,.. . ,i K Ec,"
C
Thus, as in the case of GEEs, the estimating equation defined in (6.51) is unbiased provided that the FRM is specified correctly. Like GEEs, the choice of G is not unique. In most applications, G = DV-' = ( g h )V-l, where V denotes some known 1 x 1 matrix. As in the case of GEEs, V ( a )may be parameterized by some vector a . If a is unknown, it must be estimated before the UGEE in (6.51) can be solved for 8. For GEEs, we showed in Chapter 4 that consistency of GEE estimate is independent of how a is estimated, while asymptotic normality of such estimate is guaranteed only when fi-consistent estimates of a are used. As we show shortly, these nice asymptotic properties carry directly over to UGEE estimates. Example 1 ( Generalized A NOVA for variance). Consider the FRM for the one-group variance ANOVA in (6.48) again. In this case,
Set G = D =
g.
Since
=
= 1, the estimating equation for 8 = g2
6.3 MODEL ESTIMATION
337
is given by
The above is the same estimating equation as given in (6.48). By solving for 0 2 ,we immediately obtain (6.49). Example 2 (Generalized ANOVA f o r mean and variance). In Example 1, now consider adding a component for the mean of yi such that the FRM has the form:
Let
Set G = D = $. Since $ = 1 2 with the UGEE in (6.51) for 6 is given by
h
12
denoting the 2 x 2 identity matrix,
T
By solving for 8 , we obtain 8 = (y,,si) , where y, and s i denote the sample mean and variance of yi. The UGEE estimate defined by (6.51) has asymptotic properties that parallel those of GEE studied in Chapter 4. The following theorem summarizes these nice asymptotic results. Theorem 1 (UGEE estimates for group comparison). Let
C H A P T E R 6 FUNCTIONAL RESPONSE MODELS
338
Then, under mild regularity conditions, we have 1. 8 is consistent. 2. If fi(ti - a ) = 0,(11, is asymptotically normal: h
5
(6.56)
For each 1 5 k 5 K: let
be some symmetric kernel of Ui,,,..,i,, ,...,i, UC(O),...L ( 1 ),... ,iK (0) ’ with respect to each jk (1 5 T 5 K ) . Then, a consistent estimate of CQ is given by Let
@ k j l ,,,,j,
(6.58)
where A^ denotes the estimated A obtained by substituting estimates 6 and ti in place of their respective parameters 8 and a. Proof. For convenience, we assume a is known. The case with a substituted by a fi-consistent estimate is similarly considered (see exercise). 1. Since Unil,,,,,iK is symmetric with respect to i k , it follows that the -lU, is a K-sample U-statistic-like quantity. For normalized notational brevity, denote this normalized vector by U,. By applying the theory of multivariate U-statistics in Chapter 5, we have
nf==, (z,,)
K
0, X U = q2
pkck k=l
(6.59)
339
6 . 3 MODEL ESTIMATION
where CI, and p k are defined in (6.55). Given that U, is asymptotically normal with mean 0, consistency of 8 follows from an argument similar to the one employed in proving consistency of maximum likelihood estimate in Chapter 2 (see exercise). 2. It follows from a Taylor series expansion that h
q - 8 ) =
(-gun)-’& U n + o p ( l )
where op (.) denotes the stochastic o
(6.60)
(see Chapter 1) and
( 9 )
Since Unil,,,,,iK is symmetric with respect to i k , it follows that $Un is also a K-sample U-statistic-like quantity. Thus, by applying the theory of multivariate U-statistics to (6.61), we have (6.62) where B is defined in (6.55). The conclusion follows from (6.60), (6.62) and an application of the Delta method. 3. Since E [E(Unil ,...,iK I ~ i k l ) = ] E [GE(Sii,,,.,iK1 y i k l ) ]= 0, it follows that
XI,= V a r ( E (Unil,...,iK I y i k 1 ) ) =E =E
(uL1,...,iK I ~ i k i ) ] [ E ( ~ ~ i l : . ~ ~ ~ i ~ ~...,i~(l)~...~iK(o) . ~ ~ i K - ~ ~ I~Ylk (l ) o] ) , [ E ( u n i 1,...,iK
I
E
~ i k i )
I
-
- 1
= E (unil, ...,i k ...,iKuT nil-(0),...,i k ( 1);.... i (0) ~
where y k (1) and Tk (0) are defined in (6.56). Let j, be defined in (6.57). T Then?it is readily checkedthat V n j l , ~ . . i K= Unil,...,ik...siKUnTl(0),,,,,Tkc(l),,,,,-iK(0) depends on i k , & (0) and& (1) only through j,, . . . ,j (see exercise). Thus, if @jl,,,,,jK is a symmetric kernel of Vnjl,,,,jK, then it follows from the theory of multivariate U-statistics and Slutsky’s theorem that % k in (6.58) is a consistent estimate of C I , = V a r ( E (Unil;,.,:iK I y i , , ) ) . The conclusion follows by applying this result and the Delta method.
340
CHAPTER 6 FUNCTIONAL RESPONSE MODELS
Example 3 ( Generalized A NOVA for variance). Consider again the FRM in Example 1 for the one-group variance ANOVA. Here, q = 2 and the estimating equation for u2 is given by (6.53). Thus, G = D = 1 and B = E (GUT) = E (DD’) = 1 It follows from Theorem 1 that the asymptotic variance $2 = 4
of ;i2is
~ ~ [ Ea(Uir j yi,)] B = 4 ~ a [rE (fi - u2 I yi,)]
However,
E
@2
(fi- (72 I Y i l )
=
“
5 (Yil
- Yi2)2
1 yi,
Thus,
I
-u =
5 (yz, - p )2 - u2
where p4 = E (yf) denotes the 4th-order moment of yi. A consistent estimate of 4’ is obtained by substituting consistent estimates of p4 and u2 in the above such as the sample moments. Yote that we can also use Theorem 1 / 3 to construct a U-statistics based estimate of the asymptotic variance 4’ (see exercise). But, in this simple example, it is necessary to do so since we can evaluate V a r [ E(Ui 1 yi,)] in closed form. Example 4 (Generalized A N O V A for mean and variance). Consider the FRM in Example 2. The UGEE for 8 = ( p ,0”)’ is given by (6.54) a h = 1 2 , it follows that with q = 2. Since G = D = B
B = E (GDT) = E (DD’) h
By Theorem 1, the asymptotic variance of 8 =
= 12
(gn,s i ) T is given by
Co = 4 B V a r [E( U n i I ~ i , ) ] B = 4Var [E(fi - 8 I yi,)]
Since
341
6.3 MODEL ESTIMATION
it follows that
Example 5 (Generalized K-sample ANOVA for mean and variance). We have extended the FRM in Example 2 to a more general setting with three samples in Example 7 of Section 6.2.1. Here, we consider inference for this three-sample FRM for simultaneously modeling the mean and variance. Let y k i denote the response of interest from three i.i.d. samples (1 5 i 5 n k , 1 5 k 5 3 ) . Also, let ik = (ikl! i k 2 )
!
Yik = (ykikl
T I
Ykik2)
It follows from (6.51) that the UGEE for this three-sample FRM is given by
The solution to the above equation is in closed form T
A
(6.64)
where Ykn and sin denote the sample mean and variance of the kth sample (1 5 k 5 3 ) . is asymptotically normal with the It follows from Theorem 1 that asymptotic variance given by
342
CHAPTER 6 FUNCTIONAL RESPONSE MODELS
where
n with n nk
Pk = limniW
3
=ck=1 nk.
To find
&, note first
that
Similarly,
E
(Uil,iz,i3
E
(Uil,iz,i3
Thus, we have @I
1
c 1 = -
4
0 0 0 0 0
0 @ 0 2 00 )
,
1 &=4
..-(
0 0 0
0
a3
where a;,P k 3 and &4 denote the variance, the third and forth moment of Y k i , respectively. It follows that the asymptotic variance of 8 is given by h
(6.66)
343
6 . 3 MODEL ESTIMATION
Note that we may also obtain the above asymptotic results by applying Slutsky’s theorem. For each k , it follows from Example 4 that
It follows from Slutsky’s theorem that
yielding the same asymptotic variance as in (6.66). Example 6 ( Wilcoxon signed rank t e s t ) . Consider the FRM for the Wilcoxon signed-rank test introduced in Example 5 of Section 6.2.1:
The above is readily solved, yielding
5=
(2) n -1
x(il,i2)E~2 I{yil+yizIo~
This UGEE estimate is precisely the Wilcoxon one-sample signed rank statistic. By Theorem 1, 0 is asymptotically normal. Since B = 1, it follows that the asymptotic variance of 0 is given by h
h
Unlike the previous examples, we cannot evaluate the above in closed form. We can use Theorem 1/3 to construct a U-statistics based estimate of 0; (see exercise). As discussed in Chapter 3, the Wilcoxon signed-rank statistic is used to test whether yi is distributed symmetrically around 0. Under the null, 0 = f and the asymptotic variance u; can be evaluated in closed form 0; = The Mann-Whitney-Wilcoxon signed rank test provides an alternative to the paired t test for comparing treatment difference in a pre-post study design. As discussed in Chapter 4, the pre-post comparison is a special
344
CHAPTER 6 FUNCTIONAL RESPONSE MODELS
case of the longitudinal study design with two assessment times. The classic application of pre-post study design is in evaluating the effect of a treatment of interest by examining the change or difference of the outcome from preto posttreatment assessment. In this case, the study subjects are assessed at pre- and posttreatment times, yielding outcomes of interest before and after the administration of the intervention. We can then assess treatment effect by testing whether the mean change of the outcome is significantly different from zero using the paired t test. Alternatively, we can use the signed-rank test for this purpose. If we let yi be the change score between pre- and posttreatment assessment, then no treatment effect corresponds to the null that yi is distributed symmetrically about 0, that is, Ho : 8 = for the parameter 8 defined by the Wilcoxon signed-rank statistic.
Example 7 (Generalized K-sample Mann- Whitney- Wilcoxon rank s u m test). Consider the generalized K-sample Mann-WhitneyWilcoxon test introduced in Section 6.2.1. For notational brevity, consider the three group case, that is, K = 3. Let
Set G = D = ga h . Then, D = B a h = 13 and B = E (GDT) = 13. The UGEE for this model is given by
Let 6 denote the estimate obtained from solving the above equation. Then, 8 is asymptotically normal, with the asymptotic variance given by
h
345
6 . 3 MODEL ESTIMATION
where Fk denotes the CDF of y k i (1 5 k 5 3). To further evaluate consider two case scenarios. First, consider testing the null hypothesis:
Ho : Fl ( Y l i , )
&, we
(6.69)
= F 2 (Y222) = F3 (Y323).
Under Ho, Fk are all uniformly distributed between 0 and 1. It follows that (see exercise) :
'1
= Var [ E(u21,i2,23
Similarly, we have for
&=
C2
and
(A
1 ylil)] =
:1):
[' !)
C3:
0 0 0
11
c 3 =
3
CQ=4):pkck=
[ ("2:: 0
p2
(6.71)
12 12 1 12
Thus: it follows from (6.70) and (6.71) that
k=l
(6.70)
0
1 1 2
1
(6.72)
346
CHAPTER 6 FUNCTIONAL RESPONSE MODELS
The asymptotic variance in (6.72) is derived under the null hypothesis in (6.69) concerning the equality of distribution functions of the response. Now, consider an alternative approach that does not require such a strong assumption. First, note that
347
6 . 3 MODEL ESTIMATION
Then, the following U-statistic matrix is a consistent estimate of CQ:
where i k = ( z k , j k ) and f i k denotes the estimated Hk obtained by substituting ?kl in place of 8 k l (1 5 k < 1 5 3).
6.3.2
Inference for Models for Regression Analysis
Now consider the FRM for regression analysis defined in (6.26) of Section 6.2.2. Let
where i = ( 2 1 , . . . , ip) E CF. As in Section 6.3.1, assume that fi is symmetric with respect to i, that is, fi = fj for any permutation j of i. Define the U-statistic based estimating equation for 8 as follows: (6.77) where Gi (Xi) is some known p x 1 matrix function of xi with p and 1 denoting the dimension of 8 and Si. As in the group comparison case, this new class of U-statistics based estimating equations is a generalization of the GEES discussed in Chapter 4 for distribution-free inference of single-response based regression models to the current context of FRM where dependent variables are defined by general nonlinear functions of multisubject responses. Again, the Newton-Raphson algorithm is readily applied to solve the UGEE in (6.77) to obtain estimates of 8. Also, it is again readily shown that the UGEE is unbiased, that is, E (U, ( 8 ) )= 0, if the FRM is specified correctly (see exercise). As in the case of UGEE for group comparison discussed in Section 6.3.1, the choice of Gi (xi) is not unique and consistency of the UGEE estimate defined in (6.77) is independent of the selection of Gi (xi). In most applications, we set Gi (xi) = DiY-', where Di = &hi and % is some 1 x 1 matrix. The matrix ( a )may depend on some paramet,er vector a . As in Theorem 1 for the group comparison case, the UGEE estimate obtained from (6.77) is guaranteed to be asymptotically normal if a is substituted by
348
CHAPTER 6 F U N C T I O N A L RESPONSE MODELS
some &-consistent. These asymptotic properties are summarized by the following theorem. Theorem 2 (UGEE estimate for regression analysis). Let
C u = VUT( E (U,i
I yil
xi,))
B
== E
(GiDi'>
Then, under mild regularity conditions, we have I. G is consistent. 2. If fi (5 - a ) = 0,(11,G is asymptotically normal: (6.78) 3. For i = ( i l , i 2 , . . . , i q ) ,let
-
-
i = ( i l , i , + ~ ,. . , tkq-l), j = i u i = ( i l , . . . , i,, iq+l,.. .
(6.79)
Let @j denote some symmetric kernel of U,iULy with respect to j . Then, a consistent estimate of & is given by
where A^ denotes the estimated A obtained by substituting estimates 6 and 6 in place of the respective parameters 8 and 6. Proof. Again, for convenience, we only consider the case with known a. The general case with a substituted by a &-consistent estimate is similarly established (see exercise). 1. As in the proof of Theorem 1, we first show that the normalized U-statistic-like vector (E)-lUn (0) in (6.77) is asymptotically normal with mean 0. For notational brevity, we denote the normalized vector by U, (8). The asymptotic normality of U, (8) is established by following the same argument as used to prove this property for the similar U-statistic-like vector U, ( 8 ) in Theorem 1 (see exercise). The consistency of G then follows by employing an argument similar to the one used in proving consistency of maximum likelihood estimate in Chapter 2 (see exercise). 2. Similar to the proof of Theorem 1 / 2 in Section 6.3.1, we obtain from a Taylor series expansion the following:
h (G - 8 ) =
a
(-+)
-T
h u , + op(1)
349
6 . 3 MODEL ESTIMATION
Since U,i is symmetric with respect to i, it follows that &U, is a U-statisticlike quantity. Thus, by convergence results for multivariate U-statistics in Chapter 5, we have
where i = ( i i , . . . ,i 4 )E CF. The conclusion follows from an application of the Delta method. 3. Since E [ E(U,i 1 y i , , xi,)] = 0,it follows from an argument similar to the proof of Theorem 1/3 that
-
where i = ( i l , i 2 , . . . , i 4 )and i = ( i l , z q + l ,.. . , i 2 4 - 1 ) . As in the proof of Theorem 1/3, it is readily checked that any symmetric kernel of U,,iULT depends on i and through j, where j is defined in (6.79). Thus, if some symmetric kernel of U,iUi-, then
@j
is
(6.81)
is a U-statistic-like vector and thus a consistent estimate of the asymptotic variance = q2E U,;U, of U,. The conclusion follows from (6.78), T, (6.81) and the Delta method. Example 8 (Negative binomial F R M ) . Consider the NB-like FRM for regression analysis with clustered data discussed in Example 14 of Section 6.2.2:
(
fl ( Y i , Yj) = Yi + Yj,
hl (Xi, xi; 6 ) = pi + pj h2 (Xi,xi; 6) = pz f =
+ pj +
f2
CK
(Yi,Y j ) (p?
= (Yi- Y j ) 2
,
+ p i ) + (pi - p j )
log ( p i ) = x:p 2
(fl,fi)T , h = (hl,h2)T, e = (pT,a )
Also, let Si = fi
-hi,
G.
- D. -
a
- - h e
I-ae
i = (i,j)
T
350
CHAPTER 6 FUNCTIONAL RESPONSE MODELS
Then, the estimating equation for this FRM has the form:
(6.82)
=o Unlike the UGEEs in the examples for group comparisons discussed in Section 6.3.1, Gi is not an identity matrix. In addition, the estimating equation in (6.82) does not have a closed form solution, but estimates of 8 are readily computed numerically by using the Newton-Raphson method. Let 6 denote the solution t o the equation. It follows from Theorem 2 that 6 is asymptotically normal with the asymptotic variance given by
co = 4B-lCuB-1
= 4B-lVur
( E (Uni 1
yi, Xi)) B-1
(6.83)
where B = E (DiD’). Although Co cannot be evaluated analytically, a consistent estimate is readily obtained as we illustrate below. First, we estimate B by (6.84)
The above is a U-statistic-like quantity and thus consistency of as an estimate of B follows from the theory of multivariate U-statistics and Slutsky’s theorem. To find a consistent estimate of Cu = Vur ( E (U,i / yi, xi)), define a symmetric kernel of U(i>j) U(+) by
Then, by applying Theorem 2, we obtain a consistent g L r given by (6.85) h
where @ ( i , j , k ) denotes the estimated @ ( i , j , k ) with 8 replaced by 6. By combining (6.83); (6.84) and (6.85), we obtain a consistent estimate of &.
35 1
6.3 MODEL ESTIMATION
Note that the choice of G i = D i = & h i may not be optimal. As in the case of GEES for single-response based distribution-free regression models, we may use G i = DiK-’ to improve efficiency, where the matrix T“, may be parameterized by an unknown vector a. If a is substituted by some fi-consistent estimate, the UGEE estimate 6 obtained from (6.82) is still asymptotically normal with the same asymptotic variance CQgiven in (6.83). We can still use &,r in (6.85) t o estimate Cu. Of course, @ ( i , j , k ) in this case is not only a function of 6 , but 2 as well. Example 9 (Linear mixed eflects model). Consider the FRM for distribution-free inference about the linear mixed-effects model introduced in Example 12 of Section 6.2.2: h
E [fit(Yi, Y j ) I xi,xjl [f2st (Yi, Y j ) 1 xi,X j ]
(6.86)
= hlt ( 0 ) = h2st ( 8 )
where 8 denotes the collection of all the distinct parameters in and: flt f2st
(Yi,Y j ) = Yit
+ Yjt,
h l t (Xi!xj;0 ) = (Uit
+ UjtIT P
P
and D (6.87)
(Yi,Y j ) = ( Y i s - Yjs) (Yit - Y j t )
h2st (Xi,xj;0 ) =
T PT (Uis - Ujs) (Uit - U j t )T P + VisDVit+
v&D v j t
where uit and Let
vit
+ 2~$6,t
are subsets of xit (1 5 t 5 rn, 1 _< i 5 n ) . T
(6.88) , f l m ) , hl = ( h l l , * . , h l m I T T f2 = (f211, f212, * . , f21m, f222,. . . , f22m, , f2(m-l)m, f2mm) T h2 = (h211, h212,. . . , h21m, h222,. . , h22m, ’ . , h2(m-l)m, h2mm) fl = ( f l l ! . .
*
*
*
*
1
*
*
f = (f[,f>)T
h = (h:,hl)T,
d D. - - h i , i =(i,j) ‘-88 Then, we estimate 8 by solving the following equation: Si = fi - h i ,
Un ( 8 ) =
C icC,”
G.
-
GiSi =
C
Di
[f (yi,y j ) - h (xi,xj;8 ) ] = 0
(6.89)
(i> 1j € C?
Again, the UGEE estimate 6 is readily obtained numerically. The asymptotic variance of 6 and a consistent estimate are computed similarly to the
352
C H A P T E R 6 F U N C T I O N A L R E S P O N S E MODELS
previous example. Also, we may set Gi = DiY-' and select y-' based on the variance of f . For example, since the variance of f1 is readily obtained, we can set K to a block diagonal matrix and use the variance of f1 and the identity matrix to form the first and second block diagonal of K. The consideration is similar to the selection of G for GEE I1 discussed in Chapter 4. Example 10 (Zero-inflatedPoisson). Consider the FRM for the distribution-free ZIP model discussed in Example 15 of Section 6.2.2: (Yi I Xi) = (1 - P (Ui)) P (Vi)= (1 - P i ) Pi
where ui and
vi
are subsets of
xi.
(6.90)
Let
The UGEE for 8 has the same form as the one given in (6.89). The estimate of 8 and its asymptotic distribution are computed similarly to the previous example (see exercise).
6.4
Inference for Longitudinal Data
When analyzing data from longitudinal studies, missing data is a common issue. In Chapter 4, we discussed inference for single-response based, distribution-free regression models, including the popular class of generalized linear models under missing data. In Chapter 5, we applied and extended such missing data considerations for inference about multivariate U-statistics under the two most popular missing data mechanisms - the MCAR and the more general MAR assumption. In this section, we extend the discussions in Chapters 4 and 5 to address missing data when making inference for FRM. We first consider missing data following the MCAR mechanism.
353
6.4 INFERENCE FOR L O N G I T U D I N A L DATA
6.4.1 Inference Under MCAR Under MCAR, the probability of occurrence of missing response does not depend on the response of interest. As discussed in Chapters 4 and 5, the important issue is how to use all available data so that more power is achieved for inference of model parameters since the traditional approach based only on the subjects with complete data does not utilize all available data and thus may be quite inefficient. We discussed in these chapters a missing-data indicator approach to facilitate the development of asymptotic distribution for inference of model parameters based on all observed data within the context of each chapter. Such an approach is readily extended to the current FRM setting. Example 1 ( Generalized A NOVA f o r variance). Consider the one-group variance ANOVA in Example 1 of Section 6.3.1 within the setting of a longitudinal study with n subjects and two assessments at pre- and posttreatment times indexed by t (t = 1 , 2 ) . Let yi = ( y i l , yi2)T denote the bivariate response from the ith subject with yil and yi2 denoting the outcome at the pre- and posttreatment assessment points (1 5 i 5 n). Assume that missing data only occurs at posttreatment so that yi2 is partially observed. Let (6.92)
Then, the FRM defined by (6.93)
02
can be used for inference about the variance parameters or their difference between the pre- and posttreatment assessments. Setting G = D = &,h (0) = 1 2 , the UGEE given by (6.94)
can be used to provide inference about 8 in the absence of missing data.
354
CHAPTER 6 FUNCTIONAL RESPONSE MODELS
Solving the above for 8 yields
which is the familiar sample variance estimate at each time t ( t = 1,2). When ys2 is partially observed, the UGEE in (6.94) can only be applied to subjects with complete observations at the pre- and posttreatment assessment times, that is, those with both yil and yi2 observed. As discussed in Chapters 4 and 5 , although such a complete-data approach still yields consistent estimates of 8 under MCAR, it may severely undercut power, especially when many subjects have intermittent missing responses. Within the context of the example, the loss of power or efficiency depends on the number of subjects who have missing data at posttreatment. To utilize all available data to estimate 8, let rzt = 1 if yzt is observed and r,t = 0 if otherwise (1 5 i 5 n, 1 5 t 5 2). Since the response is always observed at pre-treatment, ril z 1. Now consider the following revised estimating equation:
(6.96)
Equation (6.96) is well defined for all subjects regardless of their missing response status. For example, if yi2 or yj2 or both are missing, then r i 2 r j 2 = 0 and in this case +ri2rj2 (yi2 - y j ~ = ) 0, ~ a well-defined quantity. The equation is readily solved to yield (6.97)
6.4 I N F E R E N C E F O R LONGITUDINAL DATA
355
h
-2 -2 T The estimate 8 = (ol,02) in (6.97) has been derived for the same problem using a similar missing data indicator approach within the context of multivariate U-statistics in Chapter 5. It was shown there that G is both consistent and asymptotically normal. It is also readily checked that 2; is the usual sample variance estimate calculated based on the observed yi2. We can also readily show the consistency and asymptotic normality of G by using Theorem 1 of Section 6.3. To see this, let
i = (i,j),
(6.98)
Then, the estimating equation in (6.96) can be expressed in the following form: (6.99) It is readily checked that E(U,) = 0 and thus the above equation is unbiased (see exercise). Further, U, is a U-statistic-like quantity and thus it follows from Theorem 1 of Section 6.3 that (6.99) defines consistent and asymptotically normal estimates of 8 . Example 2 (Generalized ANOVA f o r mean and variance). In Example 1, now consider adding a component for the mean response at each assessment to simultaneously model the mean and variance of git. Let
The FRM has the same general form as in the previous example, except for the added component for the mean response. Again, set G = & h ( 0 ) = 14. Then, in the absence of missing data, the UGEE is given by
(6.100)
We obtain from (6.100) the usual sample mean and variance yit as estimates of pUtand 0 ; .
356
CHAPTER 6 FUNCTIONAL RESPONSE MODELS
In the presence of missing data, we can readily generalize the approach in Example 1 to accommodate the extra component for the mean response. Let R i j 2 = ~ i 2 ~ j 2 1and 2 define the estimating equation as follows:
i l s in Example 1, the estimating equation above is well defined. Further, it is readily checked that (6.101) yields the same estimates of u: and ui as those in Example 1 (see exercise). The estimates of p1 and p 2 are given by (see exercise) n
As in Example 1, it is readily shown that the estimate obtained from (6.101) with Pt and t?: given in (6.102) and (6.97) is consistent and asymptotically normal (see exercise). Example 3 (Product-moment correlation). Consider again the product-moment correlation for longitudinal study data discussed in ChapT ter 5. Let zit = (zit,y,t) denote the continuous bivariate outcome from the ith subject at time t from a longitudinal study with n subjects and m assessment times (1 i n, 1 t 5 m). Let pt = Cow ( Z i t ; yit) denote the product-moment correlation between x,t and yit and p = ( p l , , . . . , p,) T . We discussed inference about p under both MCAR and MAR within the context of multivariate U-statistics in Chapter 5. Alternatively, we can also model p using an FRM. Let
< <
<
357
6.4 INFERENCE FOR LONGITUDINAL DATA
Also, let
0t
2
= ( P t , ost, .,t)
2
T
,
e=
T (01,...,0,)
Define the FRM as follows: (ft ( z i t , z i t ) ) = ht
tI m (0), 1 I
(6.104)
In the FRM model above, p is part of the model parameter vector 0. In Chapter 5, we first employed multivariate U-statistics to model the covariance ozyt= Cow (zit,g i t ) together with the variance components 02t and o$ and then obtained inference about p by applying the Delta method, as pt is a function of these parameters. In contrast, the FRM in (6.104) directly models p as part of its parameters. Now let T
f
=
(fT, . . . , fT) ,
h = (h:
, . . . , h;)
T
,
( L,
T
z . = z . . . . ,z h )
Then, by setting G = D = g h ( 0 ) = I S x m , the estimating equation for 0 in the absence of missing data is given by
U, (0) =
C
DSij
ilj€C,”
=
C
[f ( z i , z i )- h] = 0
(6.105)
i,j€C,”
The above UGEE can be solved in closed form to obtain the estimate 6 of 0. In addition, it is readily checked that Ft are the Pearson correlation estimates and and are the sample variances of zit and yit, respectively (see exercise). When there is missing data, let Tit = 1 if z,t is observed, that is, if both zit and yzt are observed. Also, let
&
$it
Define the estimating equation in the presence of missing data by
Like the estimating equations for Examples 1 and 2 discussed above, (6.106) is well defined. It is again readily checked that the estimate obtained as the
358
C H A P T E R 6 FUNCTIONAL RESPONSE MODELS T
solution to (6.106) is 8 = (&, &, Z i t ) , where & is the Pearson correlation between xit and yzt, while t?2t and Zit are the respective sample variance estimates of zit and y i t calculated based on the observed data (see exercises), that is, h
h
The consistency and asymptotic normality of the estimate 8 also follows from an argument similar to the one used in Example 2. Note that in the missing data considerations above, we assumed that both xzt and y z t are either observed or missing together. As in the case of the multivariate U-statistics approach discussed in Chapter 5 for the productmoment correlation, we can relax this assumption and allow x2t and yzt to have different missing data patterns. The above considerations are readily generalized to this more general situation (see exercise). Example 4 (Linear mixed-eflects model). Consider the FRM defined by (6.86) and (6.87) in Example 9 of Section 6.3.2 for distribution-free inference of the linear mixed-effects model. Let fi, f 2 , hl, h2, fi, hi, Si,and i be defined in (6.88). As discussed in that example, the UGEE in (6.89) defines consistent estimates of the parameter vector of interest 8 under complete data. In the presence of missing data, this estimating equation can be applied to the subset of the data formed by the subjects with complete response gzt over all assessment times to provide valid inference about 8. However, as noted earlier, such an approach generally leads to less efficient estimates. To define an appropriate estimating equation to take advantage of all available data, let r2t = 1 if y z t is observed and 0 if otherwise. We assume no missing data at t = 1 so that = 1. Also, let rt =
( ~ 2 1 .~
T
. . ,~ t r n ) ,
2 2 , .
r223 = ( T t l T J l , TZlTJ2,.
.
1'123 = ( ~ t i ~ ~ ~~ 2 i ~. 2. .2 ,T ., , T ~ ~ )
T
. . , TZlTJrn! T22TJ2,. . . ,T Z ( r n - l ) T J r n , TZrnTJrn)
(6.108) T
6 . 4 INFERENCE FOR LONGITUDINAL DATA
359
+ im + + im +
( m 1)) x 1 In the above, rlij is an m x 1 vector, while r2ij is an ( m vector. Thus, rij is a w x 1 vector, where 'u = 2m (rn 1). Let i = ( i , j ) and Ri be a 'u x 'u diagonal matrix with ri forming the diagonal of the matrix. Define the estimating equation for 8 as follows:
U, (8) =
C
Uni
C GiRiSi = 0
(8)
ieCF
(6.109)
iEC2
To see if the above estimating equation is well defined, consider the term RiSi. This term is a w x 1 column vector. The lth component Ail of RiSi has the following form:
A, =
i
ritrjt [flt (Yi, Yj)-
if 1 5 1 5 m
hltl
risritrjsrjt [fm (yi,yj) - h t ] if
m
+ 1 I1 I
'u
For 1 5 1 I m, Ail = ritrjt [fit (yi,yj)- hit], with 1 5 t 5 m. If yit or is missing, then one of rit and rjt will be 0. So, Ail = 0 regardless of what value is assigned to the missing response. For m 1 5 1 5 w, Ail = risritrjsrjt [ f 2 s t (yi,yj)- hzt],with 1 I s < t 5 m. If one of yis, yjs, yit and g j t is missing, then one of ris, rjs, T i t and rjt is 0 and E (Ail) = 0. Thus, the estimating equation for 8 in (6.109) is well defined. In addition, it is quite similar to the one defined in (6.89) for estimating 8 in the absence of missing data, with the only difference being replacing Si with the revised residual term RiSi. Under MCAR, the missing data indicator vector ri is stochastically independent of yi. Thus, it follows from the iterated conditional expectation that for 1 5 15 m, yjt
+
E
1
I
(nil x i , x j ) = E [ritrjt ( f l t ( y i , ~ j-) hlt) x i , x j ]
= E ( W j t ) E [(flt
(Yi,Yj)- hlt) 1 xi, Xj]
=o Similarly, for m
E (Ail 1
+ 1 5 1 5 w,we have
x i , x j )= E [Tisritrjsrjt ( f 2 s t
(yi,yj) - h2t) 1 xi, x j ]
1
= E (rkritrjsrjt)E [ ( f 2 s t (yi,yj) - h2t) X i , xj]
=o It follows that
360
CHAPTER 6 FUNCTIONAL RESPONSE MODELS
Thus, the estimating equation defined in (6.109) is unbiased. In addition, as U,i (6) are U-statistic like quantities, it follows from Theorem 2 that the estimate of 6 obtained from (6.109) is consistent and asymptotically normal.
6.4.2
Inference Under MAR
When missing data are MAR, the missingness will depend on observed responses. Under such response-dependent missingness, the estimating equations discussed in the preceding section no longer guarantee consistent estimates for the models considered. In Chapter 4, we discussed how to deal with this issue for the mean-based distribution-free regression models. In Chapter 5, we generalized the approaches in Chapter 4 to address MAR for multivariate U-statistics-based models. These considerations can also be applied to address MAR for the functional response model. Example 5 ( Generalized A NOVA f o r variance). Consider again the bivariate response model in Example 1 of Section 6.4.1 arising from a pre-post treatment study design. As before, assume that missing data only occurs at posttreatment. But, now suppose that the occurrence of missing response yi2 at posttreatment depends on pretreatment response ~ $ 1 . Under this response-dependent missingness or MAR, the consistency of the estimate of 8,or more specifically D ; , obtained from the estimating equation in (6.94) can no longer be assured and a new estimating equation must be developed. As seen in the discussion of Example 1,we can estimate 6 from a modified UGEE given by
where Ri is given in (6.98), i = ( 2 1 , i 2 ) E Cg and G can be parameterized by a vector a . We can see that the diagonal elements, rit = riltri2t,are binary variables indicating whether the tth component fit of the functional response fi = ( f i l , fi2)T is missing. Thus, rit is a generalization of the single-response missing data indicator rit to the functional response of FRM. n’ow we can generalize the weighted generalized estimating equation (WGEE) approach for single-response based regression models discussed in Chapter 4 to address MAR for inference about F R M within our current setting. TOthis end, let Tit = E (Tit 1 yi) and assume that for some c > 0, Tit 2 c for all i E CF and 1 5 t 5 2, that is, Tit are bounded away from 0. This assumption is a generalization of a similar condition discussed in Chapters
6.4 INFERENCE FOR LONGITUDINAL DATA
361
4 and 5 for weighting function employed for single-response based regression models and multivariate U-statistics to ensure consistent estimation and stable estimates of the model parameter vector 8 when Tit are used as weights to statistically "recover" missing functional responses. Following the discussion in these chapters, we first assume that Tit are known and consider T estimation of 8 = (a:, 0;) . We then discuss the more practical situation when n i t are unknown and modeled by observed study data.
Note that the second equality above follows since yil is always observed and
I Yil)
1 Yi,) = TiltiTTizt,
i = (ii,i 2 ) E C; (6.111) Consider the following U-statistics-based weighted generalized est,imating equation (UWGEE): Tit =
(Tilt
(7-izt
(6.112) iEC2
iEC2
The UWGEE is quite similar t o the unweighted UGEE in (6.110) for the same model defined by (6.92) and (6.93). The only difference is that Ai in (6.112) depends not only on the missing data indicators Tit, but also on their conditional probabilities Tit. Now consider the special case with G = D = & h ( 8 ) = 1 2 . The UWGEE in (6.112) becomes:
As in Example 1 of Section 6.4.1, it is readily checked that the above equation is well defined (see exercise). Further, it follows from the iterated conditional expectation that
E
(uni2) =
E [ E (Rz1n;217-z2Tj2
=E
1 -1
[G T32
= E [Gln;1 =E
(fz.72
-
h2)
( T z 2 5 2 (fz32 - h 2 ) (fz,2 - h 2 )
(7-,27-,2
I yz, y,)l
4
I Yz, I Yzl,Y 4
[GITZ1(fz32 - h 2 ) E (Tz2 I Yzl)
= E (fz32 - h 2 ) = 0
(6.114)
(7-32
I
4
362
CHAPTER 6 FUNCTIONAL RESPONSE MODELS
Thus, E(U,) = 0 and the UWGEE in (6.113) is unbiased. Since U, is a U-statistic-like random vector, it follows from Theorem 1 of Section 6.3.1 that the estimate of 8 obtained from the estimating equation in (6.113) is consistent and asymptotically normal. Note that for notational brevity, we expressed i as i = ( i , j ) instead of i = ( i l , i z ) in equation (6.114). We will continue t o employ this notation whenever convenient provided that no confusion arises. The UWGEE in this simple example can be solved in closed form: -2
01
=
-2
=
02
(6.115)
In comparison to the estimate in (6.97) obtained under MCAR, it is seen that -2 n2 has the additional weight 7riT,27rJ2. As a special case, if 7r,2 is independent of y,l, that is, missing data is MCAR, then 7ri2 = 7r2 and (6.115) reduces to (6.97). Thus, the UWGEE estimate in (6.115) is a generalization of the unweighted version in (6.97) to account for response-dependent missingness under MAR. We can also directly verify that is consistent. The consistency of follows immediately from the univariate U-statistics theory. To show that -2 o2 is consistent, note that
5
5
$1
Since both the numerator and denominator are univariate U-statistics, it follows from the consistency of U-statistics and Slutsky's theorem that
By applying Theorem 1 of Section 6.3, it is readily shown that the asymptotic variance of the UWGEE estimate 6 in (6.115) is given by (see exercise): Co = 4Cu = ~ V U( TE (A,jS,J 1 y i , r,)) (6.116)
363
6.4 INFERENCE FOR LONGITUDINAL DATA
A consistent estimate of the asymptotic variance Ce is readily constructed following Theorem 1/3 (see exercise). In practical applications, T i t , or rather 7ri2, are unknown and some estimates of this weight function must be substituted into (6.113) before estimates of 8 can be computed. Given the relationship in (6.111), we can model bivariate 7ri2 through single-response based ~ i 2 .Under MAR, we can readily model 7ri2 using a logistic regression as follows: logit
(7ri2)
1
= logit [Pr ( r i 2 = 1 yil)] = Q
+ pyil
(6.117)
7
We can estimate 7 ~ i 2for each pair of subjects i by the fitted 7ri2 (?) 7riT,,2(7)7ri22 (?) based on estimates of the parameter vector y = ( a ,p) of the above logistic regression and then substitute them into (6.113) for computing estimates of 8. We can also use (6.116) to estimate the asymptotic variance of the resulting UGEE estimate G. However, as discussed in Chapters 4 and 5 , Ce in (6.116) may underestimate the variability of 8 . By following the discussion in these chapters, we can readily derive the asymptotic variance that accounts for this extra variability. As noted in Chapters 4 and 5 , we can estimate the parameter vector y = ( 0 ! , / 3 )of ~ the logistic model using either parametric or distributionfree procedures. Regardless of the approach taken, ? can be expressed as the solution to an estimating equation of the form: h
(6.118)
where Wni is the score (for maximum likelihood estimation) or score-like vector (for GEE estimation) for the i t h subject (1 5 i 5 n). By a Taylor series expansion, we have (6.119)
(
where H = E &wni)'. By slightly modifying the asymptotic expansion of Un in (6.60) in the proof of Theorem 1 t o account for variability in the
364
C H A P T E R 6 FUNCTIONAL RESPONSE MODELS
estimated
y, we have (see exercise)
-
T
where C = E d h l ( y i , ri,y)) and hl ( y l ,q ,y) = E (Un12 1 y1, q ) . Note 87 that the coefficient 2 in the last equality of (6.120) is from the asymptotic expansion of U, around its projection. By applying CLT to (6.120): we obtain the asymptotic variance of 8 (see exercise):
(
h
CQ= 4B (Cu
+ a) B
(6.121)
where Cu is defined in (6.116). Thus, the extra term in (6.121) accounts for the additional variability due to y. A consistent estimate of Q, is readily obtained by substituting consistent estimates for the respective quantities in (6.121) (see exercise). Note that the above consideration only applied to the special case with G = D = A h ( 8 ) . If G is also parameterized by a vector a , the UWGEE estimate G from (6.112) is still consistent when a is substituted by some estimate 2. Further, if 2 is a &-consistent estimate, we can estimate the asymptotic variance of G by (6.116) if 7ri2 are known or by (6.121) if 7ri2 are unknown and estimated according to (6.118). In other words, the variability of 2 does not affect the asymptotic variance of G. This is readily established by a slight modification of the asymptotic expansion in (6.120) (see exercise). These asymptotic properties parallel those of the WGEE estimates for single-response-based regression models discussed in Chapter 4. Example 6 (Linear mixed-effects model). Consider the FRM for distribution-free inference of the linear mixed-effects model in Example 4 of Section 6.4.1. Under MAR, the estimating equation in (6.109) generally does not provide consistent estimates of 8. As in Example 5 above, a weighted estimating equation must be constructed to ensure valid inference. As discussed in Example 4, we can use the following modified UGEE to obtain consistent estimates of 8 :
U, (8) =
c
iGg
U,i ( 8 ) =
c
iEC,"
GiRiSi
GiRi (fi - hi) = 0
= iEC,"
(6.122)
365
6.4 I N F E R E N C E F O R LONGITUDINAL DATA
where fi, hi, Si and Ri are defined in Example 4 of Section 6.4.1. In (6.122)) Ri is a 'u x ZI diagonal matrix with the vector ri of binary indicators forming the diagonal of the matrix and u = (m2 3m). The Zth element of ri has the form: ril = r,1sr,ltrz2srz2t, where r,t is a binary missing data indicator for the ith subject defined in Example 4. As in Example 5, ril may be viewed as the missing data indicator for the functional response fi of the FRM. Let Ti1 = E (ril 1 xi, yi) and assume again that r i l are known and bounded away from 0 (1 5 i 5 n, 1 5 Z 5 u). Note that unlike Example 5, ~ i may l also depend on x, in addition to y,. Note also that in this particular example, if ril 1rz1srzltrzzsrz2t, then
4
Til =
E
(rzlsrzlt
+
1 xz1>yz2) E (T22srtZt I
= rzlstrzzst
~ 2 2 ~, 2 2 )
(6.123)
i = ( i l , i 2 ) E CF Let Ail = r i l r i l .
Ai = diagl (Ail)
(6.124)
where diagl (Ail) denotes the 'u x 'u diagonal matrix with Ail on the Zth diagonal. Define the UWGEE for 8 as follows:
where fi, hi and Si are defined in (6.122). The estimating equation above is similar to the one defined in (6.122) except for replacing Ri with Ai to include the weights in (6.124). Like (6.122), the estimating equation in (6.125) is well defined regardless of whether the components of the functional response fi are observed (i E C;). To show that the estimating equation is unbiased, consider the Zth component Ail, which is given by
Note that we switched to the notation i = (i, j ) for clarity in the above. Let zi = {xi,yz}denote the collection of both xi and yi. For 1 5 1 5 m, it follows from the iterated conditional expectation that
E
(Ail
I xi, xj) = E
( T i t r j t ~ -ritrjt ~
(fit (yi,yj) - hit) I zi,Z j ) ]
366
C H A P T E R 6 FUNCTIONAL RESPONSE MODELS
Similarly, for rn
E
(Ail
+ 15 1
I xi, X j ) = E
Thus, the UWGEE in (6.125) is unbiased. It follows from Theorem 2 of Section 6.3 that the estimate from the UWGEE in (6.125) is consistent and asymptotically normal. It is also readily checked that the asymptotic variance of 8 is given by
5
h
Co = ~ B - ~ C U B - ~
(6.126)
X U = V a r ( E (Uni I yzl, xzl, r z l ) ) , B = E GiniDi
(
A consistent estimate of Co is readily constructed based on (6.126) (see exercise). As in the previous example, the weights 7rtt = E(r,t I xt,yt) in most applications are unknown and must be estimated before the UWGEE defined in (6.125) can be solved to yield estimates of 8. In light of the identity in (6.123), we can estimate through r,,t = Pr (rzs= l , r t t = 1 I xL,y,). Since the number of assessments rn in this example can be > 2. 7iZstcannot be directly modeled by logistic regression as in the previous example. In fact, it is generally not possible to model rzstwithout placing some restrictions on the missing data patterns even under MAR. We discussed this issue in Chapters 4 and 5, focusing on the monotone missing data pattern (MMDP) assumption to make this modeling task feasible. Under MMDP, we first model the one-step transition probability p,t of the occurrence of missing data and then compute 7rtst for s 5 t as a function of p,t using the following relation between the two:
6 . 5 EXERCISES
367
where Hzt = {xzs,y z s ;1 5 s 5 t - l} denotes the observed data prior t o time t (2 5 t 5 rn, 1 5 i 5 n). The one-step transition probability p,t is readily modeled by logistic regression as discussed in Chapter 4. As in Example 5, we can use (6.126) to estimate the asymptotic variance of the UWGEE estimate derived by substituting 7ril (y)into (6.125), with 7 denoting the estimated parameter vector of the model for estimating 7ril (y) as discussed above. Alternatively, we can derive a version corrected for the variability in the estimated y by applying an argument similar to Example 5 (see exercise). The considerations above for Examples 5 and 6 are readily extended to the general FRM defined in Section 6.2. The only caveat is that modeling the occurrence of missing data for FRMs can become quite complicated when models are defined by multiple outcomes. In both Examples 5 and 6, we are able to estimate the weight function r i l for the functional response fi through modeling the weight function for individual response y z , but this is not always possible. For example. if we generalize the FRM for the product-moment correlation in Example 3 of Section 6.4.1 to include crossvariable, cross-time correlations, psyst = Corr (xZs,yzt), and allow xzt and yzt to have their own missing data patterns, then it is readily checked that the weight function ril arising from the context cannot be estimated by the weight function for individual responses of the form TZt,but rather by the more complex function 7rZxyst = P r (T,,, = 1,rzyt= 1 I x,,y z ) (see exercise). As discussed in Chapter 5, we must impose a stronger bivariate monotone missing data pattern (BMMDP) assumption to be able to feasibly model 7rZzyst. Models for weight functions can become even more complex when an FRM is defined by more than two outcomes within a longitudinal study setting. Thus, inference for FRM under MAR depends on whether we can model the weight function.
6.5
Exercises
Section 6.1 1. Consider the random factor analysis of variance (ANOVA) model in (6.31, (a) Show that the variance of Y k i and covariance between y,+i and yli are given by (6.4). (b) Verify (6.6). 2. Show that the dimension of the vector f (yk,y l ) , 3C2, is also the total number of combinations of ( z , j ) and ( i ’ , j ’ ) , C$C$, where ( i , j )E C;l“ and
368
C H A P T E R 6 FUNCTIONAL RESPONSE MODELS
( i ' , j ' ) E C;l are disjoint with 1 5 k , 1 5 K . Section 6.2 1. Show that the FRM in (6.31) defines the same model as the GEE defined by (6.29) and (6.30). 2. Symmetrize the functional f (yki,ykj) defined in (6.15) so that the associated FRM is defined by a symmetric function of y k i and ykj. 3. Verify (6.18) under the negative binomial distribution. 4. Verify (6.22) under the parametric zero-inflated model defined in (6.20). 5. Verify (6.25) under the normal based LMM defined in (6.24). 6. Show that f and h defined in (6.34) and (6.35) satisfy (6.36) under the model assumptions in (6.32). 7. Verify the identity in (6.45) under the parametric ZIP model in (6.43). Section 6.3 1. Show that (6.52) holds true if E ( S i ,,...,i x ) = 0. 2. Consider the proof of Theorem 1. (a) Complete the proof of Theorem 1/1 by showing that the estimate obtained from the UGEE estimating equation (6.51) is consistent. (b) Verify (6.62). ( c ) verify that Unil :...,i k ...,iKuTdepends on ik, & (0) nil (0),...,i k (1),...,i ~ ( 0 )
and y k (1) only through j1,. . . ,j,, where (0) and y k (1) are defined in (6.56) andjk in (6.57). (d) Prove Theorem 1/2 under the assumption that a is unknown and replaced by a &-consistent estimate. 3. Consider the UGEE for the generalized three-sample ANOVA for mean and variance defined in (6.63). (a) Show that the solution to the UGEE is given by (6.64). (b) Verify (6.65). 4. Find a consistent estimate of the asymptotic variance gz of the Wilcoxon signed-rank statistic given in (6.67). 5. Consider the FRM for the generalized K-sample Mann-WhitneyWilcoxon rank sum test with K = 3 in Example 7. (a) Show that under the null hypothesis in (6.69), (6.70) follows from (6.68). (b) Verify (6.68) and (6.73). (c) Verify that H1 defined in (6.76) is a symmetric kernel function of G1 (ylzl?Y222 , Y232 y3z.3, y333) defined in (6.73).
369
6.5 EXERCISES
(d) Find a symmetric Hk for GI, (ylil, y2iz,y2jz,y3i3, y3j3) defined in (6.74) and (6.75) for k = 1 , 2 . 6. Show that the estimating equation defined in (6.77) is unbiased, that is, E (U, ( 8 ) )= 0,if E (Si) = E (fi - hi) = 0 . 7. Consider the proof of Theorem 2. (a) Show that the U-statistic-like vector U, ( 6 ) in (6.77) is asymptotically normal with mean 0. (b) Use (a) t o complete the proof of Theorem 2 / l by showing that the estimate 6 obtained from the UGEE in (6.77) is consistent. (c) Verify (6.80). (d) Prove Theorem 2/2 under the assumption that a is unknown and replaced by a &-consistent estimate. 8. Consider the FRM for the linear mixed-effects model defined by (6.86) and (6.87). (a) Develop a UGEE with Gi = DiY-' and determined based on the mean of fi. (b) Find the asymptotic variance of the UGEE estimate 6. (c) Use Theorem 2 to construct a consistent estimate of the asymptotic variance of 8. 9. Consider the FRhl for the linear mixed-effects model defined by (6.90) and (6.91). (a) Develop a UGEE with Gi = Di. (b) Find the asymptotic variance of the estimate obtained from the UGEE in (a). (c) Apply Theorem 2 to construct a consistent estimate of the asymptotic variance of 8. Section 6.4 1. Show that i?; defined in (6.97) is the usual sample variance estimate calculated based on the observed yi2. 2. Show that the estimating equation defined in (6.99) is unbiased, that is, E (U, ( 8 ) )= 0. 3. Show by solving (6.100) that the estimates of pt and a; at each time are given by the sample mean and sample variance. 4. Consider the estimating equation (6.101). h
h
5
T
(a) Show that the solution to the equation is = (&, 3:) with j& and i?: defined by (6.97) and (6.102). (b) Show that 8 is consistent and asymptotically normal by applying Theorem 1 of Section 6.3. h
370
CHAPTER 6 FUNCTIONAL RESPONSE MODELS
5. Consider the FRM for the product-moment correlation defined in (6.103) and (6.104). (a) Verify that the Pearson correlation estimate Ft and the sample variance -2 ozt of xit and of yit satisfy the UGEE for the FRM defined in (6.105). (b) Verify that the generalized Pearson correlation estimate & and the sample variance of zit and of yit given in (6.107) satisfy the UGEE defined in (6.106) in the missing data case. (c) Generalize the UGEE in (6.106) so that it provides consistent estimates of 8 when zit and yit are allowed to have different missing patterns, that is, xit and yit may not be missing at the same time. 6. Consider the estimating equation (6.112) or more specifically (6.113) for the FRM for the generalized ANOVA used in modeling the variance of a response within the context of pre-post study design. Assume that 7 r i p are known. (a) Verify that this estimating equation is well defined regardless of whether yi2 is observed. (b) Show that the estimates 6 = (3:,Z i ) T given in (6.115) is the solution to the equation in (6.113). (c) Show that Co defined in (6.116) is the asymptotic variance of the UWGEE estimate G in (6.115). (d) Find a consistent estimate of Cg. 7. Consider the UWGEE in (6.112) for the generalized ANOVA defined by (6.92) and (6.93). Let G ( a ) be parameterized by a vector a and let 7ri2 (y) be modeled according to (6.118). (a) Show that the UWGEE estimate G from (6.112) has the following asymptotic expansion:
$it
$2t
$zt
where B and C are defined in (6.118) and D = E
T &K1 (yi,ri, y))
.
~
T
&hl (yi,ri, y)) = 0. ( Use the results in (a) and (b) to show that the asymptotic variance
(b) Show that D = E
(c) is independent of the variability of Ci if Ci is a fi-consistent estimate. 8. In Problem 6, assume that 7ri2 are unknown and modeled by (6.117). (a) Verify (6.120) and (6.121). (b) Find a consistent estimate of Co based on the expression in (6.121). 9. Consider the UWGEE defined in (6.125) for the FRM in Example 6. Assume that nil are known. of
6.5 EXERCISES
371
(a) Verify (6.126). (b) Use (6.126) to construct a consistent estimate of the asymptotic variance of the UWGEE estimate G from the equation in (6.124). 10. In Problem 9, assume that 7ril are unknown and modeled according to (6.127). Let y denote the parameter vector of the model for 7ril. Assume that the estimate of y has the asymptotic expansion in (6.119). (a) Show that the asymptotic variance Co of the UWGEE estimate G has an expression similar to the one in (6.121). (b) Construct a consistent estimate of Co based on the expression obtained in (a). 11. Let zit = (zit,yit)T denote the continuous bivariate outcome from the ith subject at time t from a longitudinal study with n subjects and rn assessment times (1 5 i 5 n,1 5 t 5 m ) . Let CZyst = Cov (zis,yit) denote the covariance between xis and yit. Assume that Zit and yit can be missing at different times. (a) Develop an FRM to model Czyst. (b) Discuss inference under MCAR. (c) Discuss inference under MAR and show that the weight function for the functional response is of the form 7rizyst = P r (rizs= 1,riyt = 1 1 xi,yi).
This Page Intentionally Left Blank
References Ash, R.B. (1999). Probability and Measure Theory. New York: Academic Press. Barnhart, H. X., Haber, M. and Song, J. (2002). Overall concordance correlation coefficient for evaluating agreement among multiple observers. Biometrics 5 8 , 1020-1027. Baron, R. M. and Kenny, D. A. (1986). The moderator-mediator variable distinction in social psychological research: Concept, strategic and statistical considerations. Journal of Personality and Social Psychology 51,1173-1182. Bickel, P.J., Klaassen, C.A.J., Ritov, Y. and Wellner, J.A. (1993). Eficient and Adaptive Estimation for Semiparametric Models. Baltimore, MD: Johns Hopkins University Press. Bollen, K.A. (1989). Structural Equations with Latent Variables.New York: Wiley. Casella, G. and Berger, R. (2002). Duxbury Press.
Statistical Inference.
Pacific Grove, CA:
Demidenko, E. (2004). Mixed Models: Theory and Applications. New York: Wiley. Goldstein, H. (1987). Multilevel Models in Educational and Social Research. Oxford: Oxford University Press. Hoeffding, W. (1948). A class of statistics with asymptotically normal distribution. Annals of Mathematical Statistics 19, 293-325. King, T. S. and Chinchilli, V. M. (2001). A generalized concordance correlation coefficient for continuous and categorical data. Statistics in Medicine 20, 21312147. Kowalski, J . and Powell, J. (2004). Nonparametric inference for stochastic linear hypotheses: Application to high dimensional data. Biometrika 91, 393-408. Kowalski, J., Blackford, A . , Feng, C., Mamelak, A.J. and Sauder, D.N. (2007). Nested, nonparametric, correlative analysis of microarrays for heterogenous phenotype characterization. Statistics in Medicine 26, 1090-1101. Laird, N. and Ware, J. (1982). Random-effects models for longitudinal data. Biometrics 38, 963-974.
373
374
References
Liang, K.Y. and Zeger, S.L. (1986). Longitudinal data analysis using generalized linear models. Biometrika 73, 13-22. Lindstrom, L4.J. and Bates, D.M. (1988). Newton-Raphson and EM algorithms for linear mixed-effects models for repeated-measures data. Journal of the American Statistical Association 83, 1014-1022. Mann, H.B. and Whitney, D.R. (1947). On a test of whether one of two random variables is stochastically larger than the other. Annals of Mathematical Statistics 18, 50-60. McCullagh, P. and Nelder, J. A. (1989). Generalized Linear Models. New York: Chapman and Hall. Nunnally, J.C. and Bernstein, I.H. (1994). Psychometric Theory. New York: McGraw Hill. Prentice, R.L. (1988). Correlated binary regression with covariates specific to each binary observation. Biometrics 44, 1033-1048. Raudenbush, S.W. and Bryk, A.S. (2002). Hierarchical Linear Models: Applications and Data Analysis Methods. Xewbury Park, CA: Sage Publications. SAS Institute Inc. S A S 9.1.3 Release. Cary, NC: SAS Institute Inc. Schroder, K. E. E., Carey, M. P. and Vanable, P. A. (2003). Methodological challenges in research on sexual risk behavior: 11. Accuracy of self-reports. Annals of Behavioral Medicine 26, 104-123. Tsiatis, A.A. (2006). Springer.
Semiparametric Theory and Missing Data.
New York:
Tu, X.M>Feng, C., Kowalski, J., Tang, W., Wang, H., Wan, C. and Ma, Y. (2007). Correlation analysis for longitudinal data: Applications to HIV and psychosocia1 research. Statistics in Medicine 26, 4116-4138. van Zyl, J. M., Neudecker, H. and Nel, D. G. (2000). On the distribution of the maximum likelihood estimator of Cronbach's alpha. Psychometrika 6 5 , 271280. Verbeke, G. and Molenberghs, G. (2000). Linear Mixed Models for Longitudinal Data. New York: Springer-Verlag. Vuong, Q.H. (1989). Likelihood ratio tests for model selection and non-nested hypotheses. Econometrica 57, 307-333. Wilcoxon, F. (1945). Individual comparisons by ranking methods. Biometrics 1, 80-83. Wolfinger, R. (1993). Laplace's approximation for nonlinear mixed models. Biometrika 80, 791-795.
Subject Index Absolutely continuous distribution, 22, 31 Q coefficient, see Cronbach coefficient Q Analysis of covariance, 68 Analysis of variance cell reference coding. 67 effect coding, 67 multivariate, 180, 193, 199, 203, 207 repeated measures, 179, 199. 203, 207, 214, 216, 217. 221 univariate, 66, 81, 82, 105 ANCOVA, see Analysis of covariance ANOVA, see Analysis of variance Area under receiver operating characteristic curve, 134-136, 168, 319 Asymptotic efficiency, 79. 81, 115 Asymptotic normal distribution, 46 Asymptotic variance. 46 AUC, see Area under receiver operating characteristic curve Available case missing value, 213
Concordance correlation coefficient, 269 Consistent estimate, 42 Convergence almost everywhere, 17, 42 in distribution, 42 in measure, 17 in probability, 42 in r t h mean, 47 Cramitr-Rao bound, 80 Cramitr-Wold device, 55 Cronbach coefficient a, 182, 271, 314 Cumulative distribution function multivariate, 33 univariate. 30
Degenerate distribution, 31, 55, 100 Delta method, 56 Direct effect, 234 Distribution multivariate normal. 55 t , 72 univariate Bayes theorem, 40, 90, 141 Bernoulli, 46 BMMDP, see Monotone missing data binomial, 47 pattern, bivariate exponential, 32 Bore1 a-field, 10 multinomial, 93 Cauchy-Schwartz inequality, 37, 80 negative binomial, 97 CCC, see Concordance correlation coefnormal, 32 ficient Poisson. 25. 96 CDF, see Cumulative distribution funct , 71 tion uniform. 32 Characteristic function, 61 Distribution-free regression model Chebyshev’s inequality, 37, 43 cross-sectional data, 105 Cohen’s K , 262 longitudinal data, 198 Complementary log-log link, 88 Double robustness, 223
375
376 ECME algorithm. see Expectation/conditional maximization either algorithm EM algorithm. see Expectation/maximization algorithm Endogenous variable. 232 Estimating equation. 109 Exogenous variable, 232 Expectation, 36 conditional, 38 iterated conditional, 40 Expectation / conditional maximization either algorithm, 196 Expectation / maximization algorithm, 196
SUBJECT INDEX
Generalized logit model, 92, 200, 208, 219, 302 GLM, see Generalized linear model GLMM, see Generalized linear mixedeffects model Goodman and Kruskal’s 7, 130, 253, 259 Hierarchical linear model, 184 HLM. see Hierarchical linear model
i.i.d.. see Independently identically distributed ICC, see Intraclass correlation coefficient Independence, 34, 35 Independently identically distributed, 35 Indirect effect, 234 Information matrix Filtration, 144 Fisher, 78 Frechet bounds, 204 observed, 78 FRM, see Functional response model Integration Functional response model, 317, 326 Lebesgue. 20 measurable function, 19 GEE. see Generalized estimating equaRiemann, 19 tion, mean. unweighted sequence of measurable functions, 21 GEE I, see Generalized estimating equaInternal consistency of an instrument, 182 tion. mean, unweighted Interrater agreement, 188 GEE 11, see Generalized estimating equaIntraclass correlation coefficient. 188-190, tion, mean and variance 265, 313 General linear hypothesis. 82 Generalized ANOVA K , see Cohen‘s K mean and variance. 320. 337, 340. Kendall’s 7 355 continuous outcome, 127, 163, 282 variance, 312, 318. 336, 340, 353, ordinal outcome, 130. 253. 283 360 Generalized estimating equation Latent variable model. 184 mean Law of large numbers unweighted. 202 strong, 44 weighted, 218 weak, 44 mean and variance Least square method unweighted. 226 unweighted, 233, 237 weighted. 229 weighted, 237, 243 Generalized linear mixed-effects model. Likelihood function, 36 191. 197 Likelihood ratio test, 83 Generalized linear model Likert scale. 181 cross-sectional data, 87 Linear growth curve model longitudinal data, 198 constant assessment times, 185. 199
377
SUBJECT INDEX
varying assessment times, 186 Linear mixed-effects model, 184-190, 224, 227, 324, 329, 351, 358, 364 Linear regression model multivariate, 183, 193, 194, 203, 208, 214 univariate, 3, 65, 71, 77, 79, 95, 105, 113 Link function, 88 LMM, see Linear mixed-effects model Log-linear model distribution-free, 106, 114, 224 FRM, 321, 331 negative binomial, 97, 192, 349 Poisson, 96, 192 Logistic regression, 88, 90, 108, 204, 297, 363 Logit link, 88 Mann-Whitney-Wilcoxon rank sum test K-sample, 275, 320, 344 two-sample, 7, 133, 140, 166, 281. 319 MANOVA, see Analysis of variance, multivariate MAR, see Missing data, missing at random Marginal distribution, 34 Martingale, 144 Maximum likelihood estimation, 69, 73, 194, 197 MCAR, see Missing data, missing completely at random Measurability function, 15 set, 9 Measure Lebesgue, 14 measurable space, 12 probability, 24 Mediational process, 231, 232, 238, 240, 242, 330 Missing data ignorable missing, 210 missing at random, 210
missing completely at random, 210 nonignorable missing, 210 Mixed model, 184 Mixture model missing data, 211 structural zero, 100 MMDP, see Monotone missing data pattern, univariate Moments of a random variable, 38 Monotone missing data pattern bivariate, 301 univariate, 213, 219 Multilevel linear model, 184 Multinomial response model, 92, 302 Newton-Raphson algorithm, 101 Order statistic, 29, 137 Overdispersion, 96, 97, 100, 114, 192, 225, 228, 322, 323, 327, 331, 332, 349 Parametric regression model cross-sectional data, 65 longitudinal data, 179 Path diagram, 231 PDF, see Probability density function Pearson correlation coefficient, 6, 204, 251, 357 Pearson residual, 112 Probability density function multivariate, 33 univariate, 31 Probability distribution function multivariate, 33 univariate, 31 Probit link, 88 Product-moment correlation, 6, 183, 188, 251, 270, 290, 299, 356, 367 Projection of a U-statistic multivariate K-sample, 276 one-sample, 254 univariate K-sample, 167
378
SUBJECT INDEX
U-statistic based generalized estimating equation unweighted, 335. 347 weighted, 361, 365 U-statistics Radon-Nikodym theorem, 22 multivariate Random coefficient model, 184 K-sample. 273 Random regression, 184 one-sample, 252 Random variable, 26 univariate Random vector, 27 K-sample. 135 Receiver operating characteristic curve, one-sample, 125 134 two-sample, 131 ROC curve, see Receiver operating charUGEE. see U-statistic based generalized acteristic curve estimating equation, unweigh&-consistent estimate, 207 ted ULS, see Least square method. unweighSandwich estimate. 113 ted Score equation, 70 UWGEE. see U-statistic based generScore statistic. 70 alized estimating equation, Score vector, 70 weighted Seemingly unrelated regression, 183 Selection model, 211 Wald statistic, 83 SEM, see Structural equations model WGEE, see Generalized estimating Semiparametric regression model equation. mean. weighted cross-sectional data, 107 Wilcoxon signed rank test, 128, 139, 161, longitudinal data. 198 316, 318, 343 g-field. 9 WLS. see Least square method, weightSlutsky’s theorem, 50 ed Space Working correlation. 202, 209, 228 Euclidean, 11 Working independence, 202 measurable, 10 Zero-inflated Poisson, 100, 103, 193, 323, probability, 24 332, 352 sample, 9 ZIP, see Zero-inflated Poisson Stochastic op (.) and 0, (.). 48 Stochastic Taylor series expansion, 52 Structural equations model, 230, 330 Structural zero, 97 Study design clustered, 177 cross-sectional, 63 longitudinal. 177 prospective, 64, 90 retrospective. 90 one-sample, 152 two-sample, 164 Proportional odds model, 94
Total effect, 234
WILEY SERIES IN PROBABILITY AND STATISTICS ESTABLISHED BY WALTER A. SHEWHART AND SAMUEL S. WILKS Editors: David J. Balding, Noel A . C. Cressie, Nicholas I. Fisher, Iain M.Johnstone, J. B. Kadane, Geert Molenberghs, David W. Scott, Adrian F. M.Smith, Sanford Weisberg Editors Emeriti: Vic Barnett, J. Stuart Hunter, David G. Kendall, Jozef L. Teugels The WiZey Series in Probability and Statistics is well established and authoritative. It covers many topics of current research interest in both pure and applied statistics and probability theory. Written by leading statisticians and institutions, the titles span both state-of-the-art developments in the field and classical methods. Reflecting the wide range of current research in statistics, the series encompasses applied, methodological and theoretical statistics, ranging from applications and new techniques made possible by advances in computerized practice to rigorous treatment of theoretical approaches. This series provides essential and invaluable reading for all statisticians, whether in academia, industry, government, or research.
*
* *
ABRAHAM and LEDOLTER . Statistical Methods for Forecasting AGRESTI . Analysis of Ordinal Categorical Data AGRESTI . An Introduction to Categorical Data Analysis, Second Edition AGRESTI . Categorical Data Analysis, Second Edition ALTMAN, GILL, and McDONALD . Numerical Issues in Statistical Computing for the Social Scientist AMARATUNGA and CABRERA . Exploration and Analysis of DNA Microarray and Protein Array Data ANDEL . Mathematics of Chance ANDERSON . An Introduction to Multivariate Statistical Analysis, Third Edition ANDERSON . The Statistical Analysis of Time Series ANDERSON, AUQUIER, HAUCK, OAKES, VANDAELE, and WEISBERG . Statistical Methods for Comparative Studies ANDERSON and LOYNES . The Teaching of Practical Statistics ARMITAGE and DAVID (editors) . Advances in Biometry ARNOLD, BALAKRISHNAN, and NAGARAJA . Records ARTHANARI and DODGE . Mathematical Programming in Statistics BAILEY . The Elements of Stochastic Processes with Applications to the Natural Sciences BALAKRISHNAN and KOUTRAS . Runs and Scans with Applications BALAKRISHNAN and NG . Precedence-Type Tests and Applications BARNETT . Comparative Statistical Inference, Third Edition BARNETT . Environmental Statistics BARNETT and LEWIS . Outliers in Statistical Data, Third Edition BARTOSZYNSKI and NIEWIADOMSKA-BUGAJ . Probability and Statistical Inference BASILEVSKY . Statistical Factor Analysis and Related Methods: Theory and Applications BASU and RIGDON . Statistical Methods for the Reliability of Repairable Systems BATES and WATTS . Nonlinear Regression Analysis and Its Applications
*Now available in a lower priced paperback edition in the Wiley Classics Library. ?Now available in a lower priced paperback edition in the Wiley-Interscience Paperback Series.
*
t
*
BECHHOFER, SANTNER, and GOLDSMAN . Design and Analysis of Experiments for Statistical Selection, Screening, and Multiple Comparisons BELSLEY . Conditioning Diagnostics: Collinearity and Weak Data in Regression BELSLEY, KUH, and WELSCH . Regression Diagnostics: Identifying Influential Data and Sources of Collinearity BENDAT and PIERSOL * Random Data: Analysis and Measurement Procedures, Third Edition BERRY, CHALONER, and GEWEKE . Bayesian Analysis in Statistics and Econometrics: Essays in Honor of Arnold Zellner BERNARD0 and SMITH . Bayesian Theory BHAT and MILLER . Elements of Applied Stochastic Processes, Third Edition BHATTACHARYA and WAYMIRE . Stochastic Processes with Applications BILLINGSLEY . Convergence of Probability Measures, Second Edition BILLINGSLEY . Probability and Measure, Third Edition BIRKES and DODGE . Alternative Methods of Regression BISWAS, DATTA, FINE, and SEGAL . Statistical Advances in the Biomedical Sciences: Clinical Trials, Epidemiology, Survival Analysis, and Bioinfonnatics BLISCHKE AND MURTHY (editors) . Case Studies in Reliability and Maintenance BLISCHKE AND MURTHY . Reliability: Modeling, Prediction, and Optimization BLOOMFIELD . Fourier Analysis of Time Series: An Introduction, Second Edition BOLLEN * Structural Equations with Latent Variables BOLLEN and CURRAN . Latent Curve Models: A Structural Equation Perspective BOROVKOV . Ergodicity and Stability of Stochastic Processes BOULEAU . Numerical Methods for Stochastic Processes BOX . Bayesian Inference in Statistical Analysis BOX . R. A. Fisher, the Life o f a Scientist BOX and DRAPER . Response Surfaces, Mixtures, and Ridge Analyses, Second Edition BOX and DRAPER . Evolutionary Operation: A Statistical Method for Process Improvement BOX and FRIENDS . Improving Almost Anything, Revised Edition BOX, HUNTER, and HUNTER . Statistics for Experimenters: Design, Innovation, and Discovery, Second Editon BOX and LUCERO . Statistical Control by Monitoring and Feedback Adjustment BRANDIMARTE . Numerical Methods in Finance: A MATLAB-Based Introduction BROWN and HOLLANDER . Statistics: A Biomedical Introduction BRUNNER, DOMHOF, and LANGER . Nonparametric Analysis of Longitudinal Data in Factorial Experiments BUCKLEW . Large Deviation Techniques in Decision, Simulation, and Estimation CAIROLI and DALANG . Sequential Stochastic Optimization CASTILLO, HADI, BALAKRISHNAN, and SARABIA . Extreme Value and Related Models with Applications in Engineering and Science CHAN . Time Series: Applications to Finance CHARALAMBIDES . Combinatorial Methods in Discrete Distributions CHATTERJEE and HADI . Regression Analysis by Example, Fourth Edition CHATTERJEE and HADI * Sensitivity Analysis in Linear Regression CHERNICK . Bootstrap Methods: A Guide for Practitioners and Researchers, Second Edition CHERNICK and FRIIS . Introductory Biostatistics for the Health Sciences CHILES and DELFINER . Geostatistics: Modeling Spatial Uncertainty CHOW and LIU . Design and Analysis of Clinical Trials: Concepts and Methodologies, Second Edition CLARKE and DISNEY . Probability and Random Processes: A First Course with Applications, Second Edition COCHRAN and COX . Experimental Designs, Second Edition
*Now available in a lower priced paperback edition in the Wiley Classics Library. ?Now available in a lower priced paperback edition in the Wiley-Interscience Paperback Series.
*
*
*
*
*
*
*
CONGDON . Applied Bayesian Modelling CONGDON . Bayesian Models for Categorical Data CONGDON . Bayesian Statistical Modelling CONOVER . Practical Nonparametric Statistics, Third Edition COOK * Regression Graphics COOK and WEISBERG . Applied Regression Including Computing and Graphics COOK and WEISBERG . An Introduction to Regression Graphics CORNELL . Experiments with Mixtures, Designs, Models, and the Analysis of Mixture Data, Third Edition COVER and THOMAS . Elements of Information Theory COX . A Handbook of Introductory Statistical Methods COX . Planning of Experiments CRESSIE . Statistics for Spatial Data, Revised Edition CSORGO and HORVATH . Limit Theorems in Change Point Analysis DANIEL * Applications of Statistics to Industrial Experimentation DANIEL . Biostatistics: A Foundation for Analysis in the Health Sciences, Eighth Edition DANIEL . Fitting Equations to Data: Computer Analysis of Multifactor Data, Second Edition DASU and JOHNSON . Exploratory Data Mining and Data Cleaning DAVID and NAGARAJA . Order Statistics, Third Edition DEGROOT, FIENBERG, and KADANE . Statistics and the Law DEL CASTILLO . Statistical Process Adjustment for Quality Control DEMARIS . Regression with Social Data: Modeling Continuous and Limited Response Variables DEMIDENKO . Mixed Models: Theory and Applications DENISON, HOLMES, MALLICK and SMITH . Bayesian Methods for Nonlinear Classification and Regression DETTE and STUDDEN . The Theory of Canonical Moments with Applications in Statistics, Probability, and Analysis DEY and MUKERJEE . Fractional Factorial Plans DILLON and GOLDSTEIN . Multivariate Analysis: Methods and Applications DODGE . Alternative Methods of Regression DODGE and ROMIG . Sampling Inspection Tables, Second Edition DOOB . Stochastic Processes DOWDY, WEARDEN, and CHILKO . Statistics for Research, Third Edition DRAPER and SMITH . Applied Regression Analysis, Third Edition DRYDEN and MARDIA . Statistical Shape Analysis DUDEWICZ and MISHRA * Modern Mathematical Statistics DUNN and CLARK . Basic Statistics: A Primer for the Biomedical Sciences, Third Edition DUPUIS and ELLIS . A Weak Convergence Approach to the Theory of Large Deviations EDLER and KITSOS . Recent Advances in Quantitative Methods in Cancer and Human Health Risk Assessment ELANDT-JOHNSON and JOHNSON * Survival Models and Data Analysis ENDERS . Applied Econometric Time Series ETHIER and KURTZ . Markov Processes: Characterization and Convergence EVANS, HASTINGS, and PEACOCK . Statistical Distributions, Third Edition FELLER . An Introduction to Probability Theory and Its Applications, Volume I, Third Edition, Revised; Volume 11, Second Edition FISHER and VAN BELLE . Biostatistics: A Methodology for the Health Sciences FITZMAURICE, LAIRD, and WARE . Applied Longitudinal Analysis FLEISS . The Design and Analysis of Clinical Experiments FLEISS * Statistical Methods for Rates and Proportions, Third Edition
*Now available in a lower priced paperback edition in the Wiley Classics Library. ?Now available in a lower priced paperback edition in the Wiley-Interscience Paperback Series.
FLEMING and HARRINGTON . Counting Processes and Survival Analysis FULLER . Introduction to Statistical Time Series, Second Edition t FULLER . Measurement Error Models GALLANT . Nonlinear Statistical Models GEISSER . Modes of Parametric Statistical Inference GELMAN and MENG . Applied Bayesian Modeling and Causal Inference from Incomplete-Data Perspectives GEWEKE * Contemporary Bayesian Econometrics and Statistics GHOSH, MUKHOPADHYAY, and SEN . Sequential Estimation GIESBRECHT and GUMPERTZ . Planning, Construction, and Statistical Analysis of Comparative Experiments GIFI . Nonlinear Multivariate Analysis GIVENS and HOETING . Computational Statistics GLASSERMAN and YAO . Monotone Structure in Discrete-Event Systems GNANADESIKAN . Methods for Statistical Data Analysis of Multivariate Observations, Second Edition GOLDSTEIN and LEWIS . Assessment: Problems, Development, and Statistical Issues GREENWOOD and NIKULIN . A Guide to Chi-Squared Testing GROSS and HARRIS . Fundamentals of Queueing Theory, Third Edition * HAHN and SHAPIRO . Statistical Models in Engineering HAHN and MEEKER . Statistical Intervals: A Guide for Practitioners HALD . A History of Probability and Statistics and their Applications Before 1750 HALD . A History of Mathematical Statistics from 1750 to 1930 HAMPEL ' Robust Statistics: The Approach Based on Influence Functions HANNAN and DEISTLER . The Statistical Theory of Linear Systems HEIBERGER . Computation for the Analysis of Designed Experiments HEDAYAT and SINHA . Design and Inference in Finite Population Sampling HEDEKER and GIBBONS . Longitudinal Data Analysis HELLER . MACSYMA for Statisticians HINKELMANN and KEMPTHORNE * Design and Analysis of Experiments, Volume 1: Introduction to Experimental Design, Second Edition HINKELMANN and KEMPTHORNE . Design and Analysis of Experiments, Volume 2: Advanced Experimental Design HOAGLIN, MOSTELLER, and TUKEY . Exploratory Approach to Analysis of Variance * HOAGLIN, MOSTELLER, and TUKEY . Exploring Data Tables, Trends and Shapes * HOAGLIN, MOSTELLER, and TUKEY . Understanding Robust and Exploratory Data Analysis HOCHBERG and TAMHANE . Multiple Comparison Procedures HOCKING . Methods and Applications of Linear Models: Regression and the Analysis of Variance, Second Edition HOEL . Introduction to Mathematical Statistics, Fifth Edition HOGG and KLUGMAN . Loss Distributions HOLLANDER and WOLFE . Nonparametric Statistical Methods, Second Edition HOSMER and LEMESHOW . Applied Logistic Regression, Second Edition HOSMER, LEMESHOW, and MAY . Applied Survival Analysis: Regression Modeling of Time-to-Event Data, Second Edition IHUBER . Robust Statistics HUBERTY . Applied Discriminant Analysis HUBERTY and OLEJNIK . Applied MANOVA and Discriminant Analysis, Second Edition HUNT and KENNEDY . Financial Derivatives in Theory and Practice, Revised Edition *Now available in a lower priced paperback edition in the Wiley Classics Library. ?Now available in a lower priced paperback edition in the Wiley-Interscience Paperback Series.
HURD and MIAMEE . Periodically Correlated Random Sequences: Spectral Theory and Practice HUSKOVA, BERAN, and DUPAC . Collected Works of Jaroslav Hajekwith Commentary HUZURBAZAR . Flowgraph Models for Multistate Time-to-Event Data IMAN and CONOVER . A Modern Approach to Statistics JACKSON . A User’s Guide to Principle Components JOHN . Statistical Methods in Engineering and Quality Assurance JOHNSON * Multivariate Statistical Simulation JOHNSON and BALAKRISHNAN . Advances in the Theory and Practice of Statistics: A Volume in Honor of Samuel Kotz JOHNSON and BHATTACHARYYA . Statistics: Principles and Methods, Fifth Edition JOHNSON and KOTZ . Distributions in Statistics JOHNSON and KOTZ (editors) . Leading Personalities in Statistical Sciences: From the Seventeenth Century to the Present JOHNSON, KOTZ, and BALAKRISHNAN . Continuous Univariate Distributions, Volume 1, Second Edition JOHNSON, KOTZ, and BALAKRISHNAN . Continuous Univariate Distributions, Volume 2 , Second Edition JOHNSON, KOTZ, and BALAKRISHNAN . Discrete Multivariate Distributions JOHNSON, KEMP, and KOTZ . Univariate Discrete Distributions, Third Edition JUDGE, GRIFFITHS, HILL, LUTKEPOHL, and LEE . The Theory and Practice of Ecenomejrics, Second Edition JURECKOVA and SEN . Robust Statistical Procedures: Aymptotics and Interrelations JUREK and MASON Operator-Limit Distributions in Probability Theory KADANE . Bayesian Methods and Ethics in a Clinical Trial Design KADANE AND SCHUM * A Probabilistic Analysis of the Sacco and Vanzetti Evidence KALBFLEISCH and PRENTICE . The Statistical Analysis of Failure Time Data, Second Edition KARIYA and KURATA . Generalized Least Squares KASS and VOS . Geometrical Foundations of Asymptotic Inference KAUFMAN and ROUSSEEUW . Finding Groups in Data: An Introduction to Cluster Analysis KEDEM and FOKIANOS Regression Models for Time Series Analysis KENDALL, BARDEN, CARNE, and LE . Shape and Shape Theory KHURI . Advanced Calculus with Applications in Statistics, Second Edition KHURI, MATHEW, and SINHA . Statistical Tests for Mixed Linear Models KLEIBER and KOTZ . Statistical Size Distributions in Economics and Actuarial Sciences KLUGMAN, PANJER, and WILLMOT . Loss Models: From Data to Decisions, Second Edition KLUGMAN, PANJER, and WILLMOT . Solutions Manual to Accompany Loss Models: From Data to Decisions, Second Edition KOTZ, BALAKRISHNAN, and JOHNSON . Continuous Multivariate Distributions, Volume 1, Second Edition KOVALENKO, KUZNETZOV, and PEGG . Mathematical Theory of Reliability of Time-Dependent Systems with Practical Applications KOWALSKI and TU . Modern Applied U-Statistics KVAM and VIDAKOVIC . Nonparametric Statistics with Applications to Science and Engineering LACHIN . Biostatistical Methods: The Assessment of Relative Risks LAD . Operational Subjective Statistical Methods: A Mathematical, Philosophical, and Historical Introduction LAMPERTI . Probability: A Survey of the Mathematical Theory, Second Edition 3
*Now available in a lower priced paperback edition in the Wiley Classics Library. ?Now available in a lower priced paperback edition in the Wiley-lnterscience Paperback Series.
*
*
LANGE, RYAN, BILLARD, BRILLINGER, CONQUEST, and GREENHOUSE . Case Studies in Biometry LARSON . Introduction to Probability Theory and Statistical Inference, Third Edition LAWLESS . Statistical Models and Methods for Lifetime Data, Second Edition LAWSON . Statistical Methods in Spatial Epidemiology LE . Applied Categorical Data Analysis LE . Applied Survival Analysis LEE and WANG . Statistical Methods for Survival Data Analysis, Third Edition LEPAGE and BILLARD . Exploring the Limits of Bootstrap LEYLAND and GOLDSTEIN (editors) * Multilevel Modelling of Health Statistics LIAO . Statistical Group Comparison LINDVALL . Lectures on the Coupling Method LIN . Introductory Stochastic Analysis for Finance and Insurance LINHART and ZUCCHINI . Model Selection LITTLE and RUBIN . Statistical Analysis with Missing Data, Second Edition LLOYD . The Statistical Analysis of Categorical Data LOWEN and TEICH . Fractal-Based Point Processes MAGNUS and NEUDECKER . Matrix Differential Calculus with Applications in Statistics and Econometrics, Revised Edition MALLER and ZHOU * Survival Analysis with Long Term Survivors MALLOWS . Design, Data, and Analysis by Some Friends of Cuthbert Daniel MA", SCHAFER, and SINGPURWALLA . Methods for Statistical Analysis of Reliability and Life Data MANTON, WOODBURY, and TOLLEY . Statistical Applications Using Fuzzy Sets MARCHETTE . Random Graphs for Statistical Pattern Recognition MARDIA and JUPP . Directional Statistics MASON, GUNST, and HESS . Statistical Design and Analysis of Experiments with Applications to Engineering and Science, Second Edition McCULLOCH and SEARLE . Generalized, Linear, and Mixed Models McFADDEN . Management of Data in Clinical Trials, Second Edition McLACHLAN . Discriminant Analysis and Statistical Pattern Recognition McLACHLAN, DO, and AMBROISE * Analyzing Microarray Gene Expression Data McLACHLAN and KRISHNAN . The EM Algorithm and Extensions, Second Edition McLACHLAN and PEEL . Finite Mixture Models McNEIL . Epidemiological Research Methods MEEKER and ESCOBAR . Statistical Methods for Reliability Data MEERSCHAERT and SCHEFFLER . Limit Distributions for Sums of Independent Random Vectors: Heavy Tails in Theory and Practice MICKEY, DUNN, and CLARK . Applied Statistics: Analysis of Variance and Regression, Third Edition MILLER ' Survival Analysis, Second Edition MONTGOMERY, PECK, and VINING . Introduction to Linear Regression Analysis, Fourth Edition MORGENTHALER and TUKEY . Configural Polysampling: A Route to Practical Robustness MUIRHEAD . Aspects of Multivariate Statistical Theory MULLER and STOYAN . Comparison Methods for Stochastic Models and Risks MURRAY . X-STAT 2.0 Statistical Experimentation, Design Data Analysis, and Nonlinear Optimization MURTHY, XIE, and JIANG . Weibull Models MYERS and MONTGOMERY . Response Surface Methodology: Process and Product Optimization Using Designed Experiments, Second Edition MYERS, MONTGOMERY, and VINING . Generalized Linear Models. With Applications in Engineering and the Sciences
:Now available in a lower priced paperback edition in the Wiley Classics Library. T N o available ~ in a lower priced paperback edition in the Wiley-Interscience Paperback Series
'
t
*
7
*
* *
'*
*
NELSON . Accelerated Testing, Statistical Models, Test Plans, and Data Analyses NELSON . Applied Life Data Analysis NEWMAN . Biostatistical Methods in Epidemiology OCHI . Applied Probability and Stochastic Processes in Engineering and Physical Sciences OKABE, BOOTS, SUGIHARA, and CHIU . Spatial Tesselations: Concepts and Applications of Voronoi Diagrams, Second Edition OLIVER and SMITH . Influence Diagrams, Belief Nets and Decision Analysis PALTA . Quantitative Methods in Population Health: Extensions of Ordinary Regressions PANJER . Operational Risk: Modeling and Analytics PANKRATZ . Forecasting with Dynamic Regression Models PANKRATZ . Forecasting with Univariate Box-Jenkins Models: Concepts and Cases PARZEN . Modern Probability Theory and Its Applications PEfiA, TIAO, and TSAY . A Course in Time Series Analysis PIANTADOSI . Clinical Trials: A Methodologic Perspective PORT . Theoretical Probability for Applications POURAHMADI . Foundations of Time Series Analysis and Prediction Theory POWELL . Approximate Dynamic Programming: Solving the Curses of Dimensionality PRESS . Bayesian Statistics: Principles, Models, and Applications PRESS . Subjective and Objective Bayesian Statistics, Second Edition PRESS and TANUR . The Subjectivity of Scientists and the Bayesian Approach PUKELSHEIM . Optimal Experimental Design PURI, VILAPLANA, and WERTZ . New Perspectives in Theoretical and Applied Statistics PUTERMAN . Markov Decision Processes: Discrete Stochastic Dynamic Programming QIU . Image Processing and Jump Regression Analysis RAO . Linear Statistical Inference and Its Applications, Second Edition RAUSAND and HBYLAND . System Reliability Theory: Models, Statistical Methods, and Applications, Second Edition RENCHER . Linear Models in Statistics RENCHER . Methods of Multivariate Analysis, Second Edition RENCHER . Multivariate Statistical Inference with Applications RIPLEY . Spatial Statistics RIPLEY . Stochastic Simulation ROBINSON . Practical Strategies for Experimenting ROHATGI and SALEH . An Introduction to Probability and Statistics, Second Edition ROLSKI, SCHMIDLI, SCHMIDT, and TEUGELS . Stochastic Processes for Insurance and Finance ROSENBERGER and LACHIN . Randomization in Clinical Trials: Theory and Practice ROSS . Introduction to Probability and Statistics for Engineers and Scientists ROSSI, ALLENBY, and McCULLOCH . Bayesian Statistics and Marketing ROUSSEEUW and LEROY . Robust Regression and Outlier Detection RUBIN . Multiple Imputation for Nonresponse in Surveys RUBINSTEIN and KROESE . Simulation and the Monte Carlo Method, Second Edition RUBINSTEIN and MELAMED . Modem Simulation and Modeling RYAN . Modern Engineering Statistics RYAN . Modem Experimental Design RYAN . Modem Regression Methods RYAN . Statistical Methods for Quality Improvement, Second Edition SALEH . Theory of Preliminary Test and Stein-Type Estimation with Applications SCHEFFE . The Analysis of Variance SCHIMEK . Smoothing and Regression: Approaches, Computation, and Application SCHOTT . Matrix Analysis for Statistics, Second Edition SCHOUTENS . Levy Processes in Finance: Pricing Financial Derivatives
*Now available in a lower priced paperback edition in the Wiley Classics Library. ?Now available in a lower priced paperback edition in the Wiley-Interscience Paperback Series.
t t
7
t *
SCHUSS . Theory and Applications of Stochastic Differential Equations SCOTT . Multivariate Density Estimation: Theory, Practice, and Visualization SEARLE . Linear Models for Unbalanced Data SEARLE . Matrix Algebra Useful for Statistics SEARLE, CASELLA, and McCULLOCH . Variance Components SEARLE and WILLETT . Matrix Algebra for Applied Economics SEBER . A Matrix Handbook For Statisticians SEBER . Multivariate Observations SEBER and LEE . Linear Regression Analysis, Second Edition SEBER and WILD . Nonlinear Regression SENNOTT . Stochastic Dynamic Programming and the Control of Queueing Systems SERFLING . Approximation Theorems of Mathematical Statistics SHAFER and VOVK . Probability and Finance: It’s Only a Game! SILVAPULLE and SEN . Constrained Statistical Inference: Inequality, Order, and Shape Restrictions SMALL and McLEISH . Hilbert Space Methods in Probability and Statistical Inference SRIVASTAVA . Methods of Multivariate Statistics STAPLETON . Linear Statistical Models STAPLETON . Models for Probability and Statistical Inference: Theory and Applications STAUDTE and SHEATHER . Robust Estimation and Testing STOYAN, KENDALL, and MECKE . Stochastic Geometry and Its Applications, Second Edition STOYAN and STOYAN . Fractals, Random Shapes and Point Fields: Methods of Geometrical Statistics STREET and BURGESS . The Construction of Optimal Stated Choice Experiments: Theory and Methods STYAN . The Collected Papers of T. W. Anderson: 1943-1985 SUTTON, ABRAMS, JONES. SHELDON, and SONG . Methods for Meta-Analysis in Medical Research TAKEZAWA . Introduction to Nonparametric Regression TANAKA . Time Series Analysis: Nonstationary and Noninvertible Distribution Theory THOMPSON . Empirical Model Building THOMPSON . Sampling, Second Edition THOMPSON . Simulation: A Modeler’s Approach THOMPSON and SEBER . Adaptive Sampling THOMPSON, WILLIAMS, and FINDLAY . Models for Investors in Real World Markets TIAO, BISGAARD, HILL, PEfiA, and STIGLER (editors) . Box on Quality and Discovery: with Design, Control, and Robustness TIERNEY . LISP-STAT: An Object-Oriented Environment for Statistical Computing and Dynamic Graphics TSAY . Analysis of Financial Time Series, Second Edition UPTON and FINGLETON . Spatial Data Analysis by Example, Volume 11: Categorical and Directional Data VAN BELLE . Statistical Rules of Thumb VAN BELLE, FISHER, HEAGERTY, and LUMLEY . Biostatistics: A Methodology for the Health Sciences, Second Edition VESTRUP . The Theory of Measures and Integration VIDAKOVIC . Statistical Modeling by Wavelets VINOD and REAGLE . Preparing for the Worst: Incorporating Downside Risk in Stock Market Investments WALLER and GOTWAY . Applied Spatial Statistics for Public Health Data WEERAHANDI . Generalized Inference in Repeated Measures: Exact Methods in MANOVA and Mixed Models WEISBERG . Applied Linear Regression, Third Edition
*Now available in a lower priced paperback edition in the Wiley Classics Library. ?Now available in a lower priced paperback edition in the Wiley-Interscience Paperback Series.
*
WELSH . Aspects of Statistical Inference WESTFALL and YOUNG . Resampling-Based Multiple Testing: Examples and Methods for p-Value Adjustment WHITTAKER . Graphical Models in Applied Multivariate Statistics WINKER * Optimization Heuristics in Economics: Applications of Threshold Accepting WONNACOTT and WONNACOTT . Econometrics, Second Edition WOODING . Planning Pharmaceutical Clinical Trials: Basic Statistical Principles WOODWORTH . Biostatistics: A Bayesian Introduction WOOLSON and CLARKE . Statistical Methods for the Analysis of Biomedical Data, Second Edition WU and HAMADA . Experiments: Planning, Analysis, and Parameter Design Optimization WU and ZHANG . Nonparametric Regression Methods for Longitudinal Data Analysis YANG . The Construction Theory of Denumerable Markov Processes YOUNG, VALERO-MORA, and FRIENDLY . Visual Statistics: Seeing Data with Dynamic Interactive Graphics ZELTERMAN . Discrete Distributions-Applications in the Health Sciences ZELLNER . An Introduction to Bayesian Inference in Econometrics ZHOU, OBUCHOWSKI, and McCLISH . Statistical Methods in Diagnostic Medicine
*Now available in a lower priced paperback edition in the Wiley Classics Library. ?Now available in a lower priced paperback edition in the Wiley-Interscience Paperback Series.
This Page Intentionally Left Blank
