Selected Proceedings of the Symposium on Inference for Stochastic Processes

Institute of Mathematical Statistics LECTURE NOTES-MONOGRAPH SERIES Selected Proceedings of the Symposium on Inference ...

Author: S.V. Basawa | C.C. Heyde | R.L. Taylor (editors)

18 downloads 1464 Views 28MB Size Report

This content was uploaded by our users and we assume good faith they have the permission to share this book. If you own the copyright to this book and it is wrongfully on our website, we offer a simple DMCA procedure to remove your content from our site. Start by pressing the button below!

Report copyright / DMCA form

DOWNLOAD PDF

Institute of Mathematical Statistics LECTURE NOTES-MONOGRAPH SERIES

Selected Proceedings of the Symposium on Inference for Stochastic Processes

S.V. Basawa, C.C. Heyde and R.L. Taylor, Editors

Volume 37

Institute of Mathematical Statistics LECTURE NOTES-MONOGRAPH SERIES Volume 37

Selected Proceedings of the Symposium on Inference for Stochastic Processes I.V. Basawa, C.C. Heyde and R.L. Taylor, Editors

Institute of Mathematical Statistics Beachwood, Ohio

Institute of Mathematical Statistics Lecture Notes-Monograph Series Series Editor: Joel Greenhouse

The production of the IMS Lecture Notes-Monograph Series is managed by the IMS Business Office: Julia A. Norton, IMS Treasurer, and Elyse Gustafson, IMS Executive Director.

Library of Congress Control Number: 2001 135427 International Standard Book Number 0-940600-51-X Copyright © 2001 Institute of Mathematical Statistics All rights reserved Printed in the United States of America

Table of Contents SECTION 1: INTRODUCTION

1

An Overview of the Symposium /. V. Basawa, C. C. Heyde, and R. L. Taylor

3

Shifting Paradigms in Inference C. C Heyde

9

Section 2: Stochastic Models: General

23

Modelling by Levy Processes Ole E. Barndorjf-Nielsen

25

Extreme Values for a Class of Shot-Noise Processes W. P. McCormick and Lynne Seymour

33

Statistical Inference for Stochastic Partial Differential Equations B. L. S. Prakasa Rao

47

Fixed Design Regression Under Association George G. Roussas

71

Dependent Bootstrap Confidence Intervals Wendy D. Smith and Robert L. Taylor

91

Section 3: Time Series

109

Kolmogorov-Smirnov Tests for AR Models Based on Autoregression Rank Scores Faouzi El Bantli and Marc Hallin

111

Estimation of the Long-Memory Parameter: A Review of Recent Developments and an Extension Rajendra J. Bhansali and Piota S. Kokoszka

125

Stability of Nonlinear Time Series: What Does Noise Have to Do With It? Daren B. H. Cline and Huay-min H. Pu

151

Section 4: Population Genetics

171

Testing Neutrality of mtDNA Using Multigeneration Cytonuclear Data Susmita Datta

173

Inference on Random Coefficient Models for Haplotype Effects in Dynamic Mutation Using MCMC Richard M. Huggins, Guoqi Qian and Danuta Z. Loesch

185

Section 5: Semiparametric Inference

203

Semiparametric Inference for Synchronization of Population Cycles P. E. Greenwood and D. T. Haydon

205

Plug-In Estimators in Semiparametric Stochastic Process Models Ursula U. Muller, Anton Schίck and Wolfgang Wefelmeyer

213

Section 6. Estimating Functions

235

Nuisance Parameter Elimination and Optimal Estimating Functions T. M. Durairajan and Martin L. William

237

Optimal Estimating Equations for Mixed Effects Models with Dependent Observations Jeong-gun Park and I. V. Basawa

247

Section 7. Spatial Models

269

Reconstruction of a Stationary Spatial Process from a Systematic Sampling Karim Benhenni

271

Estimating the Variance of the Maximum Pseudo-Likelihood Estimator Lynne Seymour

281

A Review of Inhomogeneous Markov Point Processes Eva B. Vedel Jensen and Linda Stougaard Nielsen

297

Section 8. Perfect Simulation

319

Perfect Sampling for Posterior Landmark Distributions with an Application to the Detection of Disease Clusters Marc A. Loizeaux and Ian W. McKeague

321

A Review of Perfect Simulation in Stochastic Geometry Jesper Mφller

333

LIST OF CONTRIBUTORS Name Bantli, Faouzi El Barndorff-Nielsen, Ole E. Basawa, Ishwar Benhenni, Karim Bhansali, Rajendra Cline, Daren B. H. Datta, Susmita Durairajan, T. M. Greenwood, Priscilla Hallin, Marc Haydon, D. T. Heyde, C. C. Huggins, Richard Jensen, Eva B. Vedel Kokoszka, Piota S. Loesch, Danuta Z. Loizeaux, Marc A. McCormick, William McKeague, Ian W. M0ller, Jesper Muller, Ursula U. Nielsen, Linda Stougaard Park, Jeong-gun Pu, Huay-min H. Qian, Guoqi Rao, B. L. S. Prakasa Roussas, George Schick, Anton Seymour, Lynne Smith, Wendy D. Taylor, Robert L. Wefelmeyer, Wolfgang William, Martin L.

Affiliations Universite Libre deBruxelles, Belgium University of Aarhus, Denmark University of Georgia Universite Pierre Mendes-France University of Liverpool, U.K. Texas A&M University Georgia State University Loyola College, India University of British Columbia & Arizona State University Universite Libre deBruxelles, Belgium Centre for Tropical Veterinary Medicine, U.K. Columbia University & Australian National University LaTrobe University, Australia University of Aarhus, Denmark University of Liverpool, U.K. LaTrobe University, Australia Florida State University University of Georgia Florida State University Aalborg University, Denmark Universitat Bremen, Germany University of Aarhus, Denmark Harvard University Texas A&M University LaTrobe University, Australia Indian Statistical Institute University of California at Davis Binghamton University University of Georgia U.S. Census Bureau University of Georgia Universitat of Siegen, Germany Loyola College, India

Institute of Mathematical Statistics LECTURE NOTES — MONOGRAPH SERIES

AN OVERVIEW OF THE SYMPOSIUM ON INFERENCE FOR STOCHASTIC PROCESSES I. V. Basawa University of Georgia C. C. Heyde Columbia University and Australian National University R. L. Taylor University of Georgia Abstract Some historical snap-shots of developments in the general area of stochastic processes and statistical inference are given. An overview of the papers appearing in this volume is then presented.

1

Introduction

The Symposium on Inference for Stochastic Processes was held at the University of Georgia from May 10, 2000 to May 12, 2000. The Symposium was cosponsored by the Institute of Mathematical Statistics and was a satellite meeting of the Fifth World Congress of the Bernoulli Society. The major focus of the symposium was to provide a forum for the interchange of information on inference for stochastic processes and related applications. Partial funding by the University of Georgia's "State-of-the-Art" Conference Program, a National Security Agency Grant and a National Science Foundation Grant contributed to the success of the Symposium and is gratefully acknowledged. The Symposium attracted 79 registered participants from several countries including Australia, Belgium, Canada, Denmark, France, Germany, India, Iran, Portugal, Sweden, Taiwan, the United Kingdom and the United States of America. The program consisted of 17 sessions with 49 speakers. This volume represents selected proceedings of the Symposium.

4

2

BASAWA, HEYDE AND TAYLOR

Some Historical Perspectives

Before describing the current research and applications of inference for stochastic processes, it is important to consider some historical developments in this subject area. Our most influential ancestors in the subject were very strongly motivated by the scientific needs of their times. They did applied work of the highest quality for which they developed theoretical tools as necessary. To illustrate, we might start with what is arguably some prehistory, namely with Daniel Bernoulli. He was one of the famous Swiss Bernoulli family and in 1760 he produced the first epidemic model - a deterministic model for tracking the spread of smallpox. This disease was still a major concern at that time and variolation, the first attempt at vaccination, was new and very topical. Bernoulli was a strong advocate of its advantages, for which he provided important quantitative support. Of course, the first stochastic epidemic model came much later, what is now called the chain binomial model, first published by the Russian physician Pyotr En'ko in 1889. He had measles data, and he did careful estimation and fitting. Next let us move to Prance in the early 1840's. At the time there was considerable interest in the demographics of aristocratic families. Benoiston de Chateauneuf did considerable statistical work in which he reached conclusions such as, on average, an aristocratic family name dies out in around 300 years. There was interest and concern in these matters, inheritance being a key factor in the operation of society. This motivated I.-J. Bienayme, then a civil servant, to model the extinction of family names. He developed the branching process model, and was able to state the correct form of the criticality theorem. Galton and Watson took up the same subject in England in 1873-4, but they did not know of Bienayme's earlier work of 1845, and they did not find the correct form of the criticality theorem. Unfortunately, it is only their names that are generally remembered for the work. Prom the start of the 20th Century the practical use of stochastic processes and associated inference began to flourish. In 1900 Louis Bachelier published the theory of Brownian motion, in his thesis, five years before the work of Einstein on the subject. Bachelier's motivation was the modelling of the stock market. But unfortunately his work was distinctly unappreciated till late in his life. Indeed, he did not manage to obtain a tenured university appointment until he was age 47. It was actually Kolmogorov in 1931 who first recognized the importance of his work, but his fame was established only in the 1960's and 1970's. Now we even have a Bachelier Society. The next of our ancestors from the turn of the century who can be usefully mentioned is the Swede Filip Lundberg. To him we owe the basic theory and practice of collective risk as it is applied by insurance companies, his contributions beginning with his thesis of 1903. His starting point was the

SYMPOSIUM OVERVIEW

5

description of the total claim by what we would now describe as a compound Poisson process. He went on to spend his working life in the most senior and influential positions in the insurance industry in Sweden. One exception in terms of practical motivation was the work of Markov and his contributions on Markov chains which date from 1906. Markov's motivation was actually to strike a blow on behalf of the St. Petersburg School (founded by Chebyshev) against Nekrasov, then leader of the rival Moscow School. Nekrasov had unwisely asserted that independence was a necessary condition for the weak law of large numbers, and Markov pounced on this, developing the idea of chain dependence to show that Nekrasov was wrong. Philosophical and religious differences underpinned a bitter enmity between them. But practical use of the Markov chain idea had significantly predated Markov. In 1846 Quetelet used a two-state Markov chain to model the weather type from one day to the next. He noted from the available data that independence did not hold, rain being more likely to be followed by rain etc. Much of our modern queueing theory comes from the Dane Agner Erlang. He worked for the Copenhagen Telephone Company from 1908 until his death in 1929, and he was involved in all facets of queueing performance. Indeed, he is reputed to have been regularly seen walking the streets of Copenhagen accompanied by a workman with a ladder. He was hunting for network loss sources. Another very practical man was the British engineer Harold Hurst. It is to him that we owe the ideas of long-range dependence. These ideas were developed in the 1940's and 1950's when he played a key role in the design of the Aswan High Dam on the river Nile. Hurst had abundant data, and his numerical work convinced him that standard ARMA time series models could not match the data. Much new theory, largely developed by Benoit Mandelbrot, came out of Hurst's pioneering work. These are some of our innovative predecessors who did much to influence the development of stochastic processes and its associated inference. They had good models and genuine data. They did first-rate science and they are good role models for us. More details can be found in articles in Heyde and Seneta (2001). The mathematical foundations for the modern theory of statistical inference were laid by R. A. Fisher in the 1920's. Neyman and Pearson developed the theory for hypotheses testing in the 1930's. Subsequently, the work by Wald, LeCam, C. R. Rao and the others unified various developments in the theory of inference. Most of the pioneering work on inference was, however, devoted to the classical framework of independent and identically distributed observations. Grenander (1950) addressed the problem of extending the classical inference theory to stochastic processes. See also Grenander (1981).

6

BASAWA, HEYDE AND TAYLOR

Billingsley (1961) discussed inference problems for Markov processes. Hall and Heyde (1980) gave a general treatment of likelihood based inference using martingales. The monograph by Basawa and Prakaso Rao (1980) gave a comprehensive survey of the general area of inference for stochastic processes. See also, Basawa (2001) for a recent review on this topic. The recent monograph by M. M. Rao (2000) gives a rigorous mathematical treatment of the theory of inference for stochastic processes.

3

An Overview

This volume containing the Selected Proceedings of the Symposium on Inference for Stochastic Processes includes twenty referred articles in addition to this overview. These papers are grouped into eight sections. The introductory Section 1 contains the overview article and Chris Heyde's foundational paper on shifting paradigms in inference. Section 2 has five papers on applications to various stochastic models: Barndorff-Nielsen discusses applications of Levy processes in finance; McCormick and Seymour study extreme value results for a shot-noise model; inference problems for stochastic partial differential equations are discussed by Prakasa Rao; Roussas considers design problems under association and finally, Smith and Taylor present their results on bootstrap confidence intervals for dependent data. Section 3 contains three papers on time series: El Bantli and Hallin discuss Kolmogorov-Smirnov tests for autoregressive models based on ranks; Bhansali and Kokoszka review long-memory parameter estimation and discuss an extension; the problem of stability of nonlinear time series is studied by Cline and Pu. Two papers on population genetics are included in Section 4: Susmita Datta studies the problem of testing neutrality using multigeneration data; Huggins, Qian and Loesch discuss inference problems for random coefficient models for haplotype effects. Two papers on semiparametric models are included in Section 5: Greenwood and Haydon discuss semiparametric inference for synchronization of population cycles; Muller, Schick and Wefelmeyer study estimation for semiparametric stochastic processes. Section 6 contains two papers on optimal estimating functions: Durairajan and William discuss nuisance parameter elimination in optimal estimating functions; Park and Basawa study optimal estimating functions for mixed effects nonlinear models with dependent observations. Section 7 contains three papers on spatial models: Benhenni discusses systematic sampling from a stationary spatial process; Seymour studies variance estimation for the pseudo-likelihood estimator; Vedel Jensen and Nielsen review inhomogeneous spatial Markov point processes. Finally, Section 8 contains two papers on perfect simulation: Loizeaux and McKeague

SYMPOSIUM OVERVIEW discuss perfect sampling from posterior distributions in a spatial model; M0Uer reviews perfect simulation in stochastic geometry. Markov Chain Monte Carlo (MCMC) methods and more recently perfect simulation techniques provide a powerful link between stochastic processes and inference. The papers in Sections 7 and 8 illustrate the use of these methods. Acknowledgements The editors of these selected proceedings are very grateful to the many referees who carefully reviewed all papers which were submitted for publication in the proceedings. Also, the cooperation of individual authors in producing this volume in the IMS Lecture Notes Series is greatly appreciated. Special thanks go to Connie Durden for the preparation of this Volume. Ms. Durden worked tirelessly and patiently with the various authors in securing software files of their papers, providing uniformity of margins, spacing and similar editorial changes which greatly enhanced the general appearance of this Volume.

References Basawa, I. V. (2001). Inference in Stochastic Processes. In Handbook of Statistics, Vol 19, 55- 77, Eds.: D. N. Shanbhag, and C. R. Rao. Elsevier, Amsterdam Basawa, I. V. and B. L. S. Prakasa Rao (1980). Statistical Inference for Stochastic Processes, Academic Press, London. Billingsley, P. (1961). Statistical Inference for Markov Processes, Univ. Chicago Press, Chicago. Grenander, U. (1950). Stochastic Processes and Statistical Inference. Arkiv fur Mat 1, 195-277. Grenander, U. (1981). Abstract Inference, Wiley, New York. Hall, P. and Heyde, C. C. (1980). Martingale Limit Theory and its Applications. Academic Press, New York. Heyde, C. C. and E. Seneta (Eds.) (2001). Statisticians of the Centuries, Springer, New York. Rao, M. M. (2000). Stochastic Processes: Inference Theory, Kluwer, Boston.

Institute of Mathematical Statistics LECTURE NOTES — MONOGRAPH SERIES

SHIFTING PARADIGMS IN INFERENCE C.C. Heyde Australian National University and Columbia University Abstract Some personal perspectives on changing paradigms in inference are presented. The topics discussed include the changes from independence to dependence, estimators to estimating functions and from adhoc methods to Fisher information based methods. Recent trends in time series, general theory of inference, estimating functions and information based techniques are discussed.

1

Introduction

The advent of the new millenium gives us a particularly good excuse to take stock of changing paradigms in inference, or more particularly inference for stochastic processes. Our subject can reasonably be thought of as roughly a century old, and it was strongly practical and model based from the outset. Our distinguished ancestors such as Pyotr En'ko in 1889 with the chain binomial model for epidemics, Louis Bachelier in 1900 with Brownian motion and the modelling of the sharemarket and Filip Lundberg in 1903 with collective risk for insurance application, were very much motivated by the scientific needs of their times. Any list of paradigms is, of course, rather subjective. The ones that I will treat in this paper are undoubtedly important, and are ones which have influenced me personally. But there are arguably others, and certainly one other that I would have liked to discuss. That is the advent of the computer as a tool for model exploration and simulation. I have learned a lot from the newly available technologies. A lot about bad models, poor behaviour of limit theorems, slow rates of convergence etc. But the constraints of the occasion have precluded discussion of these issues. So let me pass to the topics which I will discuss. These are the changes: 1. Independence => Dependence. 2. Estimators => Estimating Functions. 3. Ad hoc methods => Fisher Information based methods.

10

HEYDE

In connection with the first of these, I should remark that I am not seeking to minimize the role of independence in stochastic models. Regeneration in stochastic models is a key phenomenon and our capacity to simulate is strongly tied to independence. My focus is on what we can do in inference.

2

Independence to Dependence

The history of Inference for Stochastic Processes has two major strands of development: (1) General theory of inference. (2) Inference for time series. We shall examine the impediments to the development of each of these. 2.1

Time Series

Time series as an autonomous area of study basically dates from the 1920s. There were many contributors but I will particularly mention the name of George Udny Yule who wrote papers in 1926, 1927 which laid the foundation of autoregressive process theory. The idea of a linear model in terms of a finite past of the process together with a stochastic disturbance was natural and immediately successful in a wide range of problems. The subject exolved into the autoregressive moving average (ARMA) and autoregressive integrated moving average (ARIMA) forms and entered the modern era in large part through the computational implementation of Box and Jenkins (1970). Prom the outset there was a special focus on the case of Gaussian time series. Indeed, the theory was developed for second order stationary Gaussian processes, which are fully characterized by their means and covariances {^(k) = cov(Xn, Xn+k)}. Also, maximum likelihood (ML) and least squares (LS) estimation procedures were used from the outset, and continue to be used. Amongst the most influential early results was the following: Wold Decomposition (1938). If {Xj} is a purely non-deterministic (physically realizable) stationary process with zero mean and finite variance it is representable in the form oo j=0

where {βj} is a stationary, uncorrelated, zero mean process and Σj^Lo aj2 < oo. If the X's are normally distributed the e's can be taken as independent and identically distributed (iid) and normally distributed.

11

SHIFTING PARADIGMS

This theorem implicitly suggested that all second order stationary processes could be reasonably approximated by ARMA processes of sufficiently high order, a suggestion which was further reinforced by results such as the following: Theorem If (7(.)} is any covariance function such that ^(k) —> 0 as k —> oo, then there is a causal AR(K) process whose autocovariance function at lags 0,1, ...,if coincides with j{j),j = 0,1, ...K. The results lulled users into a false sense of security concerning the breadth of applicability of ARIMA models, and it was not for some decades that the need to deal with long-range dependent processes, and various nonlinear phenomena, could no longer be denied. In the meantime, the subject proceeded via development of inference for the Gaussian case and then the Gaussian assumption was dropped and replaced by that of iid innovations e. There was complete reliance on the Strong Law of Large Numbers (SLLN) and Lindeberg-Feller Central Limit Theorem (CLT) for sums of independent random variables to develop the consistency and asymptotic normality results which underpinned a useful inferential theory. This is where the subject stood at the end of the 1960s. The 1970s saw the development of limit theory, in particular the SLLN and CLT, for martingales, subsuming the earlier results for independent random variables. With the martingale theory it became possible to treat issues such as when is a linear model appropriate. There is, indeed, a simple answer to this question. Consider a stationary finite variance process {Xn} and write e

X

j — j ~~

where the βj are the prediction errors, {Fn} are the past history σ-fields and E{Xj\Tj-ι) is the best one-step predictor of Xj. Then, it turns out that the best linear predictor is the best predictor if and only if the e 's are martingale

differences (Hannan and Heyde (1972)). In recent times there has been an increasing realization of the role of non-linear models, but much of the development has been coming from other professions, such as physicists (see for example Kantz and Schreiber (1997)). Dynamical systems, often with striking associated properties such as chaos, have attracted much attention and proponents of deterministic theory have thrown out a challenge to the stochastic community to which there has been all too little in the way of a reasoned response. 2.2

General Theory of Inference

This was developed in a setting of a random sample of iid rv (and is still typically taught in that setting!). Much of the theory rests on asymptotic

12

HEYDE

normality (or mixed normality) of estimators, and when it was developed there were nice CLT results only for independent rv. The first attempts at a discussion of inference for stochastic processes in a general setting came only in the 1960s and 1970s. This can be seen in the books of Billingsley (1961) and Roussas (1972) in a setting of stationary ergodic Markov chains. Soon thereafter, general central limit results for martingales became available subsuming independence results such as the Lindeberg-Feller CLT. Only then did it become possible to give a very general discussion of inference for stochastic processes in the traditional likelihood based setting. The basic framework is as follows. We have a sample {XijX^ •••^n} whose distribution depends on a parameter θ (which we take as scalar for convenience). The likelihood Ln(θ) is assumed to be differentiate with respect to θ. Then, ordinarily, the score function

is a martingale. The whole classical theory of the maximum likelihood estimator (MLE) carries over in its entirety to the general setting. One uses martingale limit theory on Un(θ) and local linearity with a Taylor expansion in the neighbourhood of the MLE. Details are given in Hall and Heyde (1980, Chapter 6). The essence of the results is as follows. If In(θ) = Σ * = 1 E(ui2\!Fi-ι) is generalized Fisher information, and if In(θ) —> oo, and In(θ)/EIn(θ)^η2(θ) for some η(θ) > 0 a.s., 'up' denoting uniform convergence in probability, then with little else one has optimality of the MLE in terms of producing minimum size asymptotic confidence intervals for θ and the classical theory is nicely subsumed. Martingale theory provides the natural setting but - 20 years later - these things have regrettably not yet become part of the statistical consciousness. The consequence is that many contemporary developments in inference, for example on the "general" linear model, missing data and the EM algorithm, multiple roots of the score function,... - are carried out in an independence setting while a much more general treatment is possible. Martingales are not yet part of the statistical mainstream. They are still regarded as belonging to the domain of the probabilists. It is notable that, by contrast, the Econometricians have not been reluctant to embrace the theory. See, for example, Davidson (1994, p. xiii).

13

SHIFTING PARADIGMS

3

Estimators to Estimating Functions

An estimating function (EF) is a function of data and parameter, typically with mean zero, which when equated to zero gives a parameter estimator as its root. The use of estimating functions is close to universal in statistical practice. It is just that there has been little focus to date on EF's themselves. Their usage dates back at least to Karl Pearson's method of moments (1894). For example, if Xi are iid with EX\ = μ and varXi = σ 2 , then

are estimating functions for θ = (μ,σ 2 )'. Maximum likelihood (ML) and least squares and its variants (LS, WLS) are basically EF methods, the parallel between them being shown below. ML Likelihood L(θ) Form the score dlogL(θ)/dθ Equate to zero and solve

LS/WLS Sum(weighted sum) of squares S(θ) Form dS(θ)/dθ Equate to zero and solve

The score function is a benchmark (eg. Godambe (I960)). It is the score function, rather than the MLE which comes from it, which is fundamental. Indeed, the optimality properties which we ascribe to the MLE are really optimality properties of the score function. For example: • Fisher information is an EF property (Fisher information is varU). • The Cramer-Rao inequality is an EF property. It gives varU as a bound on the variances of standardized estimating functions. EF's have significant advantages over the estimators derived therefrom. • EF's with information about an unknown parameter can be readily combined. • EF's usually have straightforward asymptotics. That for the estimator is derived therefrom using local linearity plus regularity. For a discussion of optimal inference it is best to choose an EF setting and to focus on optimality of the EF and not optimality of an estimator derived therefrom. The origins of this theory go back to the 1960s but it began a serious surge of development in the mid 1980s, much of the impetus being provided by Godambe's 1985 paper. A detailed treatment of the subject has been provided in book form in Heyde (1997). The theory, labelled as quasilikelihood since it closely mimics the features of classical likelihood theory, is outlined below.

14 3.1

HEYDE General QL Principles

The setting is of a sample {Zut e T} from some stochastic system whose distribution involves θ. The θ to be efficiently estimated is a vector of dimension p. The approach is via a chosen family of EFs the GT being vectors of dimension p with EGτ(θ) = 0 and the p x p matrices EGT =

1

being assumed nonsingular. Comparisons are made using an information criterion (generalized Fisher information) E(Gτ) = (EG)r(EGτGτYι{EGτ) for GT € Q. We choose G*χ £ Q to maximize £(ζ?τ) in the partial order of non-negative definite matrices. (This amounts to a reformulation of the Gauss-Markov theorem.) Such a G*τ is called a quasi-score estimating function (QSEF) within Q. It should be emphasized that the choice of the family Q is open and should be tailored to the particular application. The estimator θ*τ obtained from G*τ(θ*τ) = 0, termed a quasi-likelihood estimator, has under broad conditions, minimum size asymptotic confidence zone properties for 0, at least within Q. The basic properties are those of the MLE, but restricted to Q. The theory does not require a parametric setting, let alone the existence of a score function Uτ{θ). Important features of the theory include: • It applies to general stochastic systems. • It allows for the control of the problem of misspecification. This control is in the hands of the experimenter. No more than means and variances are required in many contexts. • It carries with it all the classical theory of ML and LS. There is no extra baggage required for the discussion of inference for stochastic processes. But for this setting the detailed asymptotics does require modern limit theory (especially that for martingales). QSEFs can usually be found with the aid of the following result (Heyde (1997), Theorem 2.1, p.14): Proposition G*τ € Q is a QSEF within Q if {EGτ)'ιEGτG*τf = Cτ (3.1) for all GT € Q, where CT is a fixed matrix. Conversely, if Q is convex and G*τ is a QSEF then (3.1) holds.

15

SHIFTING PARADIGMS

3.2

Finding Useful Families of EFs

Statistical models can generally, perhaps after suitable transformation, be described as data = signal + noise where the signal is a predictable trend term and the noise is a zero-mean stochastic disturbance. This can then be conveniently reformulated in terms of a special semimartingale representation as Xt = Xo + At{θ) + Mt{θ) where At is a predictable finite variation process and Mt is a local martingale. This provides a natural route to estimating parameters in the signal. Discrete time processes and most continuous time processes with finite means have this kind of representation, via a suitable rewrite if necessary. Thus, for example, if {Xt = Σ*_i χί} ι s a discrete time process, with past history σ-fields {^i}, it can be rewritten in special semimartingale form as

A general strategy is to try the Hutton-Nelson family of EFs

G = {Gτ : Gτ = [T as(θ)dMs(θ) = Γ as(θ)d(Xs - As(θ))} Jo

Jo

for which the QSEF is

1

where Mt = Y?8=i ms in the discrete time case and

[T(E(dMa\Γa-))'(d(M)a)-dMa

Jo

in the continuous time case. Here (M)t is the quadratic characteristic and the minus superscript denotes the generalized inverse. Example. The membrane potential V across a neuron is well described by a stochastic differential equation dV(t) = (-pV(t) + λ)dt + dM{t) (eg Kallianpur (1983)), where M(t) is a martingale with a (centered) generalized Poisson distribution. Here (M)t = σ 2 t, σ > 0.

16

HEYDE

The QSEF for the Hutton-Nelson family on the basis of a single realization {V(t),0 < t < T} is / ί-V(t) l)'(dV(t) - (-pV(t) + X)dt). Jo The estimators p and λ are then obtained from the estimating equations

ίTV(t)dV(t) = ίT(-βV(t) + \)V(t)dt JO

Jθ

V{T) - V{0) =

0

For more details about this subject, including such things as how to deal with parameters in the noise component of the semimartingale model, see Heyde (1997, Chapter 2). An important role for semimartimgales in inference for stochastic processes has been evident since the papers of Hutton and Nelson (1986) and Thavenaswaran and Thompson (1986). There are few contexts where these methods cannot play a vital role. Now there is a book on inference associated with semimartingale models (Prakasa Rao (1999)), although not with a focus of the kind that has been outlined above. Amongst the (rare) processes which are not semimartingales is the fractional Brownian motion BH(t),0 < H < l,fί φ \. The case H = \ corresponds to ordinary Brownian motion, which is a semimartingale. Fractional Brownian motion has a Gaussian distribution and the self-similarity property BH(ct)±cHBH(t),c>0. It has been widely used to model possible long-range dependence (see for example Beran (1994)). An example where this process has been used is in modelling departures from the standard geometric Brownian motion model of Black and Scholes for the price of a risky asset which may now exhibit long-range dependence. Here the price Pt of the asset at time t is modelled by the stochastic differential equation (sde) -σdBH{t)] where Bπ(t),0 < H < 1, is a fractional Brownian motion process. The standard model corresponds to the case H = \ and the nonstandard model, H φ ^, is not amenable to the analysis described above. Substitute methods, however, have recently been developed. See, for example, Mikosch and

SHIFTING PARADIGMS

17

Norvaisa (2000) for a discussion of the above sde, and Norros, Valkeila and Virtamo (1999) for a discussion of the parameter estimation. All the methods of inference, semimartingale based or not, make use, in some sense, of information or empirical information. Consistency results can generally be obtained via the martingale SLLN and (asymptotic) confidence intervals via the martingale CLT. For the latter, the most general results deal with the self-normalized case [M]~2 Af, [M] being the quadratic variation (Heyde (1997)). Ideas on information are the subject of the next, and last, section.

4

Fisher Information as a Statistical "Law of Nature"

The essential points that I wish to make are: (1) The role of Fisher information in the general theory of inference has already been described (in the previous section). (2) Fisher information has a key role as a scientific tool (for example in Physics). (3) Many, perhaps most, statistical procedures rely on Fisher information in ways which have not hitherto been acknowledged. Recently there has been some quite striking work in Physics based on the idea of Fisher information. The book Frieden (1998) caught the interest of the science journalist community. For example, it led to an article in New Scientist (Matthews (1999)). On the basis of this I bought the book and I found it both fascinating and frustrating. I wove consideration of it into a seminar course which I gave at Columbia University in the Fall of 1999. The ideas certainly warrant very serious consideration by the statistical community. Frieden's thesis is that: • All physical laws may be unified under the umbrella of measurement theory. • With each phenomenon there is an associated Lagrangian, natural to the field. All Lagrangians consist entirely of two forms of Fisher information - data information and phenomenological information. An informal explanation is that each context requires solution to some extremum problem. At the basis of this is a scalar function called the Lagrangian (like the likelihood). The solution of the problem can be phrased in terms of a pde involving the Lagrangian. The parallels with statistics look good at face value. But the reality is much more complex. Much of the book treats physical systems where information decreases over time. Of course, statistical problems are typically ones where information increases over time - corresponding to the collection

18

HEYDE

of more data. The sort of context where information decreases over time is, say, when the position of a particle is observed subject to noise. Over time the particle moves and its position is known with decreasing precision. 4.1

Procedures Involving Information

Most statistical procedures seem to be associated, directly or indirectly, with measures of information, and it is arguably of value to make the connection explicit as an aid to the development of useful methods. Also, it is important to note that there is a close connection between comparisons of information content and statistical distance. For example, the formulation of a quasiscore estimating function in terms of maximizing generalized Fisher information can be equivalently recast into a formulation in terms of minimizing dispersion distance (from the (generally unknown) score function) (Heyde (1997), p. 12). A new book focusing on the use of statistical distance is Lindsay and Markatou (2001). We now proceed to examine two applications in which information based ideas are not immediately apparent, in order to see the role that they can play. Choosing the order of an autoregression The most widely used procedure, AIC, involves choosing the order k to minimize: AΙC(k) = -2logL(θk) + 2k

(4.1)

where θk is the MLE of θ restricted to Rk and 2k is a penalty function. It is assumed that the order k < K for some fixed K. Akaike's original proof uses the Kullback-Liebler entropy KL given by KL

--/'

r(x)

measuring the distance between two pdf's p, r and it should be noted that Fisher information can be thought of as a form of local entropy (Friedan (1998), pp. 31-32)). The proof shows that, asymptotically, the order minimizing EK(Θ, θk) is the same as the one minimizing (4.1). It proceeds via the likelihood ratio statistic for testing the null hypothesis Ho : θ E Rk versus the alternative H\ : θ G Rκ — Rk and suggests a QL generalization of AIC based on generalized Fisher information. Note the route to treating problems where one does not have estimating functions differentiable with respect to the parameter of interest. Here the variable in question is discrete.

19

SHIFTING PARADIGMS

Stochastic Resonance The core idea here is of a weak signal operating in a noisy environment which is normally undetectable. However, by suitably increasing the noise a "resonance" can be set up making the signal apparent. Resonance may be a very important phenomenon scientifically. It has, for example, been proposed as a possible explanation for the ice ages. There is a burgeoning literature on the phenomenon which can be conveniently accessed via the the stochastic resonance web site http://www.umbrars.com/sr based in Perugia, Italy, which in turn has links to similar web sites in San Diego, USA and Saratov, Russia. A very simple example concerns the tunable model dXt = {Asinωt)dt + σdW{t)

(4.2)

where W is standard Brownian motion and the amplitude A is subthreshold (ie A < AQ). We want to estimate ω and the issue is the optimum choice of σ. When estimating a frequency from discretely observed data Xt = Xt — Xt-ι, the conventional wisdom is to work with the periodogram

where ωp = 2πp/N,p = 0,1, ...[N/2], The theory, which originated back with Fisher (1929), tells us to use the estimator ω corresponding to the ωp for which max p Ip obtains. Consistency and rate of convergence results are available. An information formulation can proceed as follows. Prom the semimartingale representation

say, derived from (4.2), we obtain the estimating function

2

and the empirical information associated with this is | | G P | | . Asymptotically, | | G P | | 2 and Ip are maximized for the same p. Of course in the stochastic resonance problem we do not observe the {xt} process, but rather the censored process {Tt = xtl(\xt\ > Ao)}. But it seems that the periodogram based approach is still appropriate. As a general conclusion, it seems profitable to think about statistical problems in a setting of maximizing an information. I see this as an important unifying principle.

20

HEYDE References Beran, J. (1994). Statistics for Long-Memory Processes, Chapman & Hall, New York. Billingsley, P. (1961). Statistical Inference for Markov Processes, Univ. Chicago Press, Chicago. Box, G.E.P. and Jenkins, G.M. (1970). Time Series Analysis, Forecasting and Control Holden-Day, San Francisco. Davidson, J. (1994). Stochastic Limit Theory. Oxford University Press Advanced Texts in Econometrics, Oxford. Fisher, R.A. (1929). Tests of significance in harmonic analysis. Proc. Roy. Soc. Ser. A 125, 54-59. Frieden, B.R. (1998). Physics from Fisher Information. Cambridge U. Press, Cambridge.

A Unification,

Godambe, V.P. (1960). An optimum property of regular maximum- likelihood estimation. Ann. Math. Statist. 3 1 , 1208-1211. Godambe, V.P. (1985). The foundations of finite sample estimation in stochastic processes. Biometrika 72, 419-428. Hall, P.G. and Heyde, C.C. (1980). Martingale Limit Theory and its Application, Academic Press, New York. Hannan, E.J. and Heyde, C.C. (1972). On limit theorems for quadratic functions of discrete time series. Ann. Math. Statist. 43, 2058-2066. Heyde, C.C. (1997). Quasi-Likelihood and its Application. A General Approach to Optimal Parameter Estimation, Springer, New York. Hutton, J.E. and Nelson, P.I. (1986). Quasi-likelihood estimation for semimartingales. Stochastic Process. Appl. 22, 245-257. Kallianpur, G.P. (1983). On the diffusion approximation to a discontinuous model for a single neuron. In P.K. Sen, Ed., Contributions to Statistics: Essays in Honor of Norman L. Johnson. North-Holland, Amsterdam, 247-258. Kantz, H. and Schreiber, T. (1997). Nonlinear Time Series Analysis. Cambridge University Press, Cambridge. Lindsay, B.G. and Markatou, M. (2001). Statistical Distances: A Global Framework for Inference. Springer, New York, to appear. Matthews, R. (1999). / is the law. New Scientist, 30 January 1999, 24-28.

SHIFTING PARADIGMS

21

Mikosch, T. and Norvaisa, R. (2000). Stochastic integral equations without probability. Bernoulli 6, 401-434. Norros, L, Valkeila, E. and Virtamo, J. (1999). An elementary approach to a Girsanov formula and other analytical results on fractional Brownian motion. Bernoulli 5, 571-587. Prakasa Rao, B.L.S. (1999). Semimartingales and their Statistical Inference. Chapman & Hall/CRC, Boca Ratan. Roussas, G.G. (1972). Contiguity of Probability Measures, Cambridge Univ. Press, London and New York. Thavaneswaran, A. and Thompson, M.E. (1986). Optimal estimation for semimartingales. J. Appl. Prob. 23, 409-417.

25

Institute of Mathematical Statistics LECTURE NOTES — MONOGRAPH SERIES

MODELLING B Y LEVY PROCESSES Ole E. Barndorff-Nielsen MaPhySto1 University of Aarhus

1

Introduction

A considerable body of recent work uses Levy processes to model and analyse financial time series. Section 2 provides a brief review of this work. The review is to a large extent based on two papers Barndorff-Nielsen and Shephard (2001a,b) where more detailed information may be found. See also Barndorff-Nielsen and Shephard (2001c,d,e). The models in question aim to incorporate one or more of the main stylised features of financial series, be they stock prices, foreign exchange rates or interest rates. A summary of these stylised features, and a comparison with related empirical findings in the study of turbulence, is given in Section 3. (In fact, the intriguing similarities between finance and turbulence have given rise to a new field of study coined 'econophysics'.)

2

Levy Processes in Finance

A Levy process is a stochastic process (in continuous time) with independent and homogeneous increments. The study of such processes, as part of probability theory generally, is currently attracting a great deal of attention, see Bertoin (1996,1999), Sato (1999), Barndorff-Nielsen, Mikosch and Resnick (2001), and references given there. It is by now well recognised that Brownian motion generally provides a poor description of log price processes of stocks and other financial assets. Improved descriptions are obtained by substituting Brownian motion by suitably chosen alternative Levy processes, for instance hyperbolic Levy motion, normal inverse Gaussian Levy motion and, more generally, one of the Centre for Mathematical Physics and Stochastics, funded by the Danish National Research Foundation

26

BARNDORFF-NIELSEN

generalised hyperbolic Levy motions. See Eberlein (2001), Prause (1999), Mantegna and Stanley (1999), Barndorff-Nielsen and Prause (2001). Merely changing from Brownian motion to another, more suitable, Levy process does not, however, provide a modelling of the important quasi long range dependencies (cf. Section 3) that pervades the financial markets. But such dependencies may be captured by further use of Levy processes, as innovation processes driving volatility processes in the framework of SV (Stochastic Volatility) models. Discrete time models of this kind were considered in Barndorff-Nielsen (1998b). That approach has since been developed, in joint work with Neil Shephard, into the continuous time setting, and the rest of the present note consists mainly in a summary of that work (Barndorff-Nielsen and Shephard (2001a,b; cf. also 2001c,d,e)). The stochastic volatility models considered are of the form dx*(t) = {μ + βσ2(t)} dt + σ(t)dw(t)

(2.1)

where, for concretenes, we may think of x*(t) as the log price process of a given stock. In (2.1), w(t) is Brownian motion and σ 2 (t), which represents the fluctuating and time dependent volatility, is a stationary stochastic process, for simplicity assumed independent of w(t). Of particular interest are cases where σ2(t) is of OU type (Ornstein-Uhlenbeck type) or is a superposition of such processes. In the former instance, σ2(t) satisfies a stochastic differential equation of the form dσ 2 (t) - -λσ 2 (t)dt + dz(λt)

(2.2)

where z(t) is a Levy process with positive increments; thus z(t) is a subordinator. Because of its role in (2.2), z(t) is referred to as the Background Driving Levy Process (BDLP, for short). The correlation function 2 r(u) of the (stationary) solution σ (t) of (2.2) has the exponential form r(u) = exp(-λu). 2 In particular, choosing the volatility process so that σ (t) follows the inverse Gaussian law /G(ί, 7) with probability density

one obtains that the increments of the log returns over a lag Δ, i.e. x*(t + Δ) — x*(t), are approximately distributed according to a normal inverse Gaussian law, and these laws are known to describe the distributions of log returns well. More generally, one may consider the generalized inverse Gaussian distribution GIG(λ,δ,j) (Ί/S)x

2Kλ(δΊ)

_Λ_

27

LEVY MODELLING 2

where K\ is a Bessel function, as the law for the volatility σ (t). The corresponding approximate laws of the increments z*(£ + Δ) — x*(t) are then of the generalized hyperbolic type which, besides the hyperbolic and normal inverse Gaussian distributions, inter alia includes the variance gamma laws, the Student distributions, and the Laplace distributions. It is important here to note that it is essential for the construction of these OU processes that the G/G(λ, ί, 7) distributions are selfdecomposable (cf. Barndorff-Nielsen (1998b)). 2 Furthermore, whatever the choice of the process & (t), if the parameter β is (approximately) 0 then the autocorrelations of the sequence of log returns will be (approximately) 0, reflecting another important stylized fact. The dependency structure in the log price process (as it manifests itself for instance in the autocorrelations of the absolute or squared returns) may be modelled by endowing the σ2(t) process with a suitable correlation structure. This can be done by taking σ 2 (t) to be a superposition of independent OU processes, while keeping the chosen marginal law of o2{t). Already the superposition of just two OU processes (with different regression parameters λi and λ2) may go a long way in describing the observed dependency structure of x*(t) (see, for instance, Barndorff-Nielsen (1998b; Figure 1)). However, even processes with real long range dependence can be constructed in this way (Barndorff-Nielsen (2001)). Finally, the so-called leverage effect (see Section 3) can be modelled by adding an extra term in equation (2.1), defined using again the BDLP z(t).

3

Stylized Features of Finance and Turbulence

A number of characteristic features of observational series from finance and from turbulence are summarised in table 1. The features are widely recognized as being esssential for understanding and modelling within these two, quite different, subject areas. In finance the observational series concerned consist of values of assets such as stocks or (logarithmic) stock returns or exchange rates, while in wind turbulence the series typically give the velocities or velocity derivatives (or differences), in the mean wind direction of a large Reynolds number wind field. For some typical examples of empirical probability densities of logarithmic asset returns, on the one hand, and velocity differences in large Reynolds number wind fields, on the other, see, for instance, Eberlein and Keller (1995) and Shephard (1996), respectively Barndorff-Nielsen (1998a). A very characteristic trait of time series from turbulence as well as finance is that there seems to be a kind of switching regime between periods of relatively small random fluctuations and periods of high 'activity'. In turbulence this phenomenon is known as intermittency whereas in finance one

BARNDORFF-NIELSEN

28

speaks of stochastic volatility or conditional heteroscedasticity. For cumulative processes x* (t) in finance a basic expression of the volatility is given by the quadratic variation process [#*](£), defined as

i=\

where 0 = ίo < h < ••• < ί n _i < tn — t and the limiting procedure is for the grid size max(ij — ίj_i) tending to 0. Similarly, in turbulence intermittency is expressed as the energy dissipation rate per unit mass at position ξ: i

/'

ξ

Jξξ-r/2

Here u = u(x) is the velocity at position x in the mean direction of the wind field. For detailed and informative discussions of the concepts of intermittency and energy dissipation, see Frisch (1995).

varying activity semiheavy tails asymmetry aggregational Gaussianity 0 autocorrelation quasi long range dependence scaling/selfsimilarity TABLE

Finance volatility

Turbulence intermittency

+ + + + + [+]

+ + + [+] +

1. Stylised features.

The term 'semiheavy tails', in table 1, is intended to indicate that the data suggest modelling by probability distributions whose densities behave, for x —> ±oo, as const. \x\p± exp(—σ± \x\)

for some p-f-,p- € R and σ+,σ_ > 0. The generalised hyperbolic laws exhibit this type of behaviour. Velocity differences in turbulence show an inherent asymmetry consistent with Kolmogorov's modified theory of homogeneous high Reynolds number turbulence (cf. Barndorff-Nielsen, 1986). Distributions of financial asset returns are generally rather close to being symmetric around 0, but for stocks there is a tendency towards asymmetry stemming from the fact that the equity market is prone to react differently to positive as opposed to negative

LEVY MODELLING

29

returns, cf. for instance Shephard (1996; Subsection 1.3.4). This reaction pattern, or at least part of it, is referred to as a 'leverage effect' whereby increased volatility tends to be associated with negative returns. By aggregational Gaussianity is meant the fact that long term aggregation of financial asset returns, in the sense of summing the returns over longer periods, will lead to approximately normally distributed variates, and similarly in the turbulence context2. For illustrations of this, see for instance Eberlein and Keller (1995) and Barndorff-Nielsen (1998a). The estimated autocorrelation functions based on log price differences on stocks or currencies are generally (closely) consistent with an assumption of zero autocorrelation. Nevertheless, this type of financial data exhibit 'quasi long range dependence' which manifests itself inter alia in the empirical autocorrelation functions of the absolute values or the squares of the returns, which stay positive for many lags. For discussions of scaling phenomena in turbulence we refer to Frisch (1995). As regards finance, see Barndorff-Nielsen and Prause (2001) and references given there. In addition, it is relevant to mention the one-dimensional Burgers equation du du _ d2u dt dx dx2 This nonlinear partial differential equation may be viewed as a 'toy model' version of the Navier-Stokes equations of fluid dynamics and, as such, have been the subject of extensive analytical and numerical studies, see for instance Frisch (1995; p. 142-143) and Bertoin (2001), and references given there. In finance, Burgers' equation has turned up in work by Hodges and Carverhill (1993) and Hodges and Selby (1997). However, the interpretation of the equation in finance does not appear to have any relation to the role of the equation in turbulence.

References [1] Barndorff-Nielsen, O.E. (1986): Sand, wind and statistics. Ada Mechanica 64, 1-18. [2] Barndorff-Nielsen, O.E. (1997): Normal inverse Gaussian distributions and stochastic volatility modelling. Scand. J. Statist. 24, 1-14. [3] Barndorff-Nielsen, O.E. (1998a): Probability and statistics: selfdecomposability, finance and turbulence. In Acccardi, L. and Heyde, 2

However, in turbulence a small skewness generally persists, in agreement with Kolmogorov's theory of isotropic turbulence.

30

BARNDORFF-NIELSEN C.C. (Eds.): Probability Towards 2000. Proceedings of a Symposium held 2-5 October 1995 at Columbia University. New York: SpringerVerlag. Pp. 47-57.

[4] Barndorff-Nielsen, O.E. (1998b): Processes of normal inverse Gaussian type. Finance and Stochastics 2, 41-68. [5] Barndorff-Nielsen, O.E. (2001): Superposition of Ornstein-Uhlenbeck type processes. Theory Prob. Its Appl. (To appear.) [6] Barndorff-Nielsen, O.E., Mikosch, T. and Resnick, S. (Eds.) (2001): Levy Processes - Theory and Applications. Boston: Birkhauser. [7] Barndorff-Nielsen, O.E. and Perez-Abreu, V. (1999): Stationary and selfsimilar processes driven by Levy processes. Stoch. Proc. Appl. 84, 357-369. [8] Barndorff-Nielsen, O.E. and Prause, K. (2001): Apparent scaling. Finance and Stochastics, 5, 103-113. [9] Barndorff-Nielsen, O.E. and Shephard, N. (2001a): Modelling by Levy processes for financial econometrics. In Barndorff-Nielsen, O.E., Mikosch, T. and Resnick, S. (Eds.): Levy Processes - Theory and Applications. Boston: Birkhauser, 283-318. [10] Barndorff-Nielsen, O.E. and Shephard, N. (2001b): Non-Gaussian OU based models and some of their uses in financial economics (with Discussion). J. Roy. Statist. Soc, B, 63, 167-241. [11] Barndorff-Nielsen, O.E. and Shephard, N. (2001c): Econometric analysis of realised volatility and its use in estimating stochastic volatility models. J. Roy. Statist. Soc, J5, 64, (to appear). [12] Barndorff-Nielsen, O.E. and Shephard, N. (2001d): Integrated OU processes. (Submitted). [13] Barndorff-Nielsen, O.E. and Shephard, N. (2001e): Normal modified stable processes. (Submitted). [14] Bertoin, J. (1996): Levy Processes. Cambridge University Press. [15] Bertoin, J. (1999): Subordinators: Examples and Applications. In Bernard, P. (Ed.): Lectures on Probability Theory and Statistics. Ecole d Έ t e de St-Flour, 1997. Berlin: Springer. Pp. 1-91.

LEVY MODELLING

31

[16] Bertoin, J. (2001): Some properties of Burgers turbulence with white or stable noise initial data. In Barndorff-Nielsen, O.E., Mikosch, T. and Resnick, S. (Eds.): Levy Processes - Theory and Applications. Boston: Birkhauser. Pp. 267-279. [17] Eberlein, E. (2001): Application of generalized hyperbolic Levy motion to finance. In Barndorff-Nielsen, O.E., Mikosch, T. and Resnick, S. (Eds.): Levy Processes - Theory and Applications. Boston: Birkhauser. Pp. 319-336. [18] Eberlein, E. and Keller, U. (1995): Hyperbolic distributions in finance. Bernoulli 1, 281-299. [19] Eberlein, E. and Raible, S. (1999): Term structure models driven by general Levy processes. Math. Finance 9, 31-54. [20] Prisch, U. (1995): Turbulence. Cambridge University Press. [21] Hodges, S. and Carverhill, A. (1993): Quasi mean reversion in an efficient stock market: The characterization of economic equilibria which support the Black-Scholes option pricing. Economic J. 103, 395-405. [22] Hodges, S. and Selby, M.J.P. (1997): The risk premium in trading equilibria which support the Black-Scholes option pricing. In Dempster, M. and Pliska, S. (Eds.): Mathematics of Derivative Securities. Cambridge University Press. Pp. 41-53. [23] Mantegna, R.N. and Stanley, H.E. (1999): Introduction to Econophysics: Correlation and complexity in finance. Cambridge University Press. [24] Prause, K. (1999): The Generalized Hyperbolic Model: Estimation, Financial Derivatives and Risk Measures. Dissertation. Albert-LudwigsUniversitat, Freiburg i. Br. [25] Sato, K. (1999): Levy Processes and Infinite Divisibility. Cambridge University Press. [26] Shephard, N. (1996): Statistical aspects of ARCH and stochastic volatility. In D.R. Cox, D.V. Hinkley and O.E. Barndorff-Nielsen (Eds.): Time Series Models - in econometrics, finance and other fields. London: Chapman and Hall. Pp. 1-67.

33 Institute of Mathematical Statistics LECTURE NOTES — MONOGRAPH SERIES

EXTREME VALUES FOR A CLASS OF SHOT-NOISE PROCESSES W. P. McCormick Department of Statistics University of Georgia Athens, GA 30602 USA

Lynne Seymour Department of Statistics University of Georgia Athens, GA 30602 USA Abstract

The distribution of the maximum of a shot-noise process based on amplitudes which are heavy tailed and follow a chain-dependent structure is analysed. Asymptotic results are obtained. The process is seen to have a strong local dependence and it's extremal index is computed. A simulation study shows the finite sample size performance of an asymptotic approximation to the distribution of the maximum.

1

Introduction

In the paper we are concerned with the asymptotic behavior of the extreme values for a class of shot noise processes. Shot noise processes provide a wide class of stochastic models that are particularly well suited to modeling time series with sudden jumps. Such processes have been applied to modeling river flow data where a rise in the riverflow level could, for example, be attributed to rainfall, Lawrance and Kottegoda (1977) and Weiss (1973). Moreover, rainfall data, itself, has been modeled via shot noise processes, Waymire and Gupta (1981). The basic model under study here takes the form

Y τ

k

) ,

t>0

where {A^} is a sequence of random amplitudes, {r^} forms a point process of event times and h is the impulse response function, typically, taken to be nonincreasing with support in [0,oo). In the current investigation, we take the {A^} to be a stochastic process of heavy-tailed random variables. In applications the sequence of shocks or amplitudes exhibits dependence. It may be that large shocks tend to occur in succession followed by periods

34

MCCORMICK AND SEYMOUR

of mild or small shocks. To model this dependence, we assume that {Ak} form a sequence of chain dependent variables. A chain dependent process operates in the following way. The observed process {A^} is linked with a secondary Markov chain {ξk} Conditional on the values of {£*;}, the Afc's are independent with the distribution of A^ depending on the value of ξk-ι Chain dependent processes form a useful class of stochastic models which, for example, have been applied with success in modeling extremes of precipitation data, Guttorp (1995) p. 74. Theorectical work on extremes for chain dependent models has been done in Resnick and Neuts (1970), Denzel and O'Brien (1975) and more recently in McCormick and Seymour (2001). In section 2 we present an extreme value analysis of the shot noise model. Extremes for shot noise processes have been considered by several authors under various assumptions. However, all previous work have taken the {Ajς} sequence to be iid. On the other hand, in practice, the data often appear to contradict such an assumption. When the {A^} are constant and the {r^} form a homogeneous Poisson process, i.e. in the case of a filtered Poisson process, Hsing and Teugels (1989) have obtained results on the limiting distribution of the maximum. Doney and O'Brien (1991) provide an extension to the results of Hsing and Teugels (1989) while working under the assumption of constant amplitude. The case of light-tailed amplitudes, viz. Gamma or Weibull distribution, was considered in Homble and McCormick (1995) and the heavy-tailed amplitude case, e.g. Pareto distribution, was developed in McCormick (1997). The process under consideration has a strong local dependence quantified by a value referred to as its extremal index. The method developed in Chernick et al. (1991) for calculating extremal index is conveniently applied here and represents an essential step in obtaining the asymptotics for the maxima. In section 3 the results of a simulation study are shown.

2

Asymptotics

Let {ξn} be a stationary finite state space Markov chain with probability transition matrix P — {pij), 1 < i,j < r. Further, let {TΓJ,1 < j < r} denote the stationary probability measure for the chain. Next, we define a chain-dependent sequence associated with the Markov chain {ξn} as follows. Let iί^(x), 1 0} be a renewal process with a fixed renewal at TQ = 0. We assume that the sequence {τn} is independent of the sequence {Aniξn} and

35

EXTREME VALUES define a shot noise process X by X(t)=

Σ

i > 0

Akh{t-τk),

(2.1)

0<τk
where h is the impluse response function for which we assume its support to be a compact subset of [0, oo). To define the stationary process associated with X, we first introduce the stationary renewal process {δj,j G Z} where we label the points so that -oo <

< ί_i < <S0 < 0 < <$i <

• < oo,

that is, SQ is the last nonpositive point. Further, let the interpoint distance d.f. for {rn} be F(x) = P{τn - τ n _i < x}. We assume F(0+) = 0 and μ = Jo°° χdF(x) < oo. Then, take {δj} such that (i)

δj - δj-uj φ 1 i.i.d. with δj - δj-ι - FJ Φ 1

and

(ii) δu - ί 0 ~ G(x) = - Γ ( l - F(t))dt

(2.2)

μ Jo

and

1 ί°° = (1 μ Jx+y

(iii) P{δ0 >x,δι>y}

F{t))dt.

With the choices made in (2.2), setting NS(A) = Σj€δj{A) where ex(A) = /^(α;), x G M, A C K, we have for any fe > 1 and sets A\,..., Ak in B(R) that *)), ί 6 R

(2.3)

Now define

ί>0.

Σ

(2.4)

Note by letting jVί = Σ j l - o o ^ Λ " ) ' a P°^ n t P Γ 0 C e s s with points (δj.Aj) G β = R x [0, oo), j E Z, we have the representation

Y(t)= f gt(s,x)dM

(2.5)

JE

with gt(s,x) = x/i(t — 5). For t G K and doubly infinite sequences 5 = (... ,d_i,io?ii? —00 < ... < J_i < Jo < 0 < δι < ... < 00, define an integer mt by mt = max{i : δi+t

< 0}.

•) with

36

MCCORMICK AND SEYMOUR

Let δ' be the doubly infinite sequence defined by

Then stationarity of the point process Ns( ) = X ^ ^ ( - ) is equivalent to δ' = δ for any t G R Under the assumption that {δn} is independent of the sequence {An, ξn}, it is easily checked that for the doubly infinite sequences A = ( , A-\, A$, A\, and A' = ( , A'.^Ό, Arl7 •) with A[ = Ai+mt, we have A = Af. Hence, letting M! = ^ ^ . ^ e^/ ^/.), we have M = M1 so that

Y(h)

-

ί gtl(8,χ)dM

JE

=

ί gtl{s,x)dM'

JE

The same argument shows (Y(t1),...,Y(tk))±(Y(tι-t),...,Y(tk-t)) which establishes stationarity of (Y(t),t E R). Let us further observe that the marginal distribution for X(t) is such that

since we assumed that cl{x : h(x) φ 0} C [0, oo). In the following we make the assumptions that (i)

support (h) = cl{x : h(x) φ 0} C [0,1]

(ii)

h is strictly decreasing

(iii)

0 < Λ(l) < Λ(0) = 1.

(2.7)

Furthermore, we assume that the d.f.'s Hi, 1 < i < r, are such that Hi{x)=H{x/κi),

l
(2.8)

37

EXTREME VALUES

where S(x) = x~aL(x) for some slowly varying function L and a a positive constant. Our first lemma establishes the tail behavior of the stationary distribution for the shot noise process defined in (2.1). To that end define a variable

(2.9)

Y = jΓίAjh(σj) .7 = 1

where {Aj,ξj} is a chain-dependent process independent of {σj}. Assume that the d.f.'s Hi associated with the chain dependent process satisfies (2.8) and that {σj} forms a delayed renewal process. Lemma 2.1 Let Y be given as in (2.9). Then P{Y > x} ~ cx~aL(x)

as x -> oo

where c = (Σ)i=i ^f71"*)^ Σ^Li ha(σj). In the special case that σ^ = Jj, j > 1, i.e. σi ~ G and σ^ — σ^-i ~ F, j > 2 in (2.2), r /I

C =

Proof: Consider for 0 < s\ < ... < sn < 1 x|σ^ = SJ, j = 1,..., n, n

Let κ(ί) = Ki and let {Xj} be iid with d.f. H and be independent of {ξj}. Then P{Y > x\σj = Sj,

1<J<

n, J=I

where Px denotes taking probability w.r.t. {-Xj} only. Since n

<ξ3-ύHsj)Xj

> x} ~ ^ ^ ( ξ j - O Λ 0 ^ ) ^ )

as x -> ex,

38

MCCORMICK

AND SEYMOUR

we have n

P{Y > x\σj — Sj, 1 < j < n,σ n +χ > 1} ~ EN^κa(ξj-ι)ha(sj)H(x)

as x —> oo. (2.10)

Finally, it follows from (2.10) as in McCormick (1997) that oo

r 1

oo j=l

~

""

) asx^oo.

(2.11)

In the stationary point process case, i.e. σ\ ~ G, we have j

(2.12)

jj

Thus the lemma holds from (2.11) and (2.12).

D

Consider the stationary sequence -τj),

k = 0,1,...

(2.13)

where we define TJ with j < 0 such that {—r_j,j = 1,2,...} is an independent copy of {τj,j = 1,2,...}. By Lemma 2.1 P{W0 >x)~ cx~aL{x)

as x -> oo

(2.14)

where c = ( ^ ^ «f π2) ΣZ=ι J<ί h{s)Fn*{ds) with F n * denoting the n-th fold convolution of F. As a first step in determining the asymptotic distribution for the maximum of the shot noise process, we obtain the asymptotic distribution for the maxima of the Wk To that end define intermediary sequences k

WJP=

J2

AjhiTk-Tj),

k =0,1,...

(2.15)

j—k-m

We will establish that the sequences {W™,k > 0}, m — 1,2,3,..., satisfy mixing conditions D(un) and D^m+ι\un) for a suitable sequence un. By Corollary 1.3 in Chernick et al (1991), this implies that the sequence {W™, k > 0} has an extremal index θm for each m. We recall the definition of D(un) and D^k\un). For a stationary sequence ^ } and sequence of constants {un}, set for 1 < I < n — 1

α n ? / = snp\P{Wj < un,j e AuB}-P{Wj

< un,j e A}P{Wj < un,j E

EXTREME

39

VALUES

where the supremum is taken over all A, B such that i c { l , . . . , f c } and B C {k + /,...,n} for some k with 1 < k < n — I. The condition D(un) is said to hold for the stationary sequence {Wk} if for some sequence ln = o(n), anjn -> 0 as n -> oo. If D(un) holds for {VFfc}, we say that condition Dk(un) holds provided there exist sequences of integers {sn} and {/n} with sn -> oo, s n α n ? / n -* 0, snln/n -^ 0 and lim nP{Wχ > un >

7l-»OO

i=2

where rn = [n/sn] and \/i signifies the maximum as the index varies over i to j . For the sequence {un} consider for any β > 0, un = un(/?) satisfying for all n sufficiently large un = i T - ( l - ^ ) (2.16) n where we take H^~{x) — inf{y : iϊ(y) > x} and ϋ" is as given in (2.8). Note regular variation of 1 - H(x) implies n ( l — H(un)) -> β as n —)• oo. First, we note from Denzel and O'Brien (1975) that the chain-dependent process {An} is strong mixing, and since the m-tuples (h(τk—Tfc_m),..., hfa — Tfc_i)), k = 0 , 1 , . . . are m-dependent, we see that {W™,k > 0} is strongmixing. Thus condition D(un) holds. Next, we consider D^m^(un). Observe m-i-l

rn

v wr

>un>\ιwr,

2=771+2

i-2

Now observe that for i > m + 2 un, W™ > un\ξk,

k<ι]

i

j-i)Xjh{Ti-Tj) > un} j=l-m

j=i-m

where Pτ,x denotes taking the probability with respect to the r and X sequences only. Thus if K* — Vi=i «i? 771 + 1

> un,Wt

m

> un} 0, be as defined in (2.13). Then {Wk} has an extremal index θ given by

Proof. We begin by computing the extremal index for {W™}. Since conditions D(un) and D^m+ι\un) hold for {W™}, we have by Corollary 1.3 in Chernick et al. (1991) that the extremal index for the {W™} exists and is given by # m where m+l

θm = lim P{ V W™ < un\W^ > un}. v

n-+oo

i=2

Now observe that m+l

nP{W?

>un>\/

W™} m+l

=

nEP{W^

>un > \J W™\τk,ξk,

k<m + l}

i=2

=

1

m+l i

ΠEPX{ ^

«(ίj-l)Λ(n -Tj)Xj >Un > \J ( Σ

j=l-m

Kiξj-lM^-Tj).

ϊ~2 j=i-m

Following the development in Chernick et al (1991), it is checked that as n -» oc, 1

m+l i τ

y ^ κ>{ζj-ι)h{τ~i — j)Xj > ^n ^ \f ( / ^ κ(ζj-i)h(ri

nEPχ{

j=l-m 1

j+m

j=l-m

i=2

1

β Σ

j+rn

E{ha\rx-τ3)-\l

j=l-m 1

β Σ j=l-m

~ Tj)

i=2 j=i-m

Λ ° ( T i - T ^ I + ^ & -I)

i—2 r E ha

a

l ^-^)-h (τ2-r3))(Σ<^ 1

r

fn).

(2.17

EXTREME

41

VALUES

Now by Lemma 2.1 and (2.16) m

~κ?πi)(l +EJ2

«i

/

a

n

h (s)F *(ds)).

72=1

l=-\

Thus we obtain _1-Eh«(τm+1) Finally, one argues as in McCormick (1997) that {Wk} has extremal index θ given by oo a

1

θ= Km θm = {l + EYh {τn))-

(2.19)

1

completing the proof. The following result is immediate from Lemma 2.2. T h e o r e m 2.3. Let {W^} be the stationary sequence defined in (2.13). Then with un given in (2.16), lim P{ max Wk < un} — exp{—

n—>oo

\
Following Hsing and Teugels (1989) we obtain the limit result for maxima of the shot noise process. Corollary 2.4. Let {X{t)} be the shot noise process defined in (2.1). Then r

X(t) < uT) = exp{-(β/μ) ^

nfm}

where μ — Eτ\. Remark. We have that

-V[l-iί z (^)]π z ->^5]<π,- as T-^ oo. μ

ι=l

μ

t=l

Thus, we obtain the following asymptotic approximation P{ ι max X(t) <x}π o
exp{

T 4

A

r

42

3

MCCORMICK AND SEYMOUR

Simulation Study

The performance of the approximation given in the Remark above is now investigated via simulation. The steps needed to carry out this simulation are specified as follows. First, the distribution of the maximum of T observations of a shot noise process {X{t)} (2.1) must be simulated. This simulation will compare T = 20 and T = 100. Generate an r-state Markov chain {ξn} of length L with transition matrix P and stationary distribution π. The value of L depends stochastically on T, as indicated in the generation of the shot noise process below. Take r = 3, and let the probability transition matrix

P = drive the Markov chain {ξn}

0.15 0.15 0.01

0.80 0.70 0.89

0.05 0.15 0.10

The stationary distribution is then given by

π = (0.1318 0.7379 0.1303). Then, generate the chain-dependent process {An} of length L, with chain-state distributions defined via 1 — Hi(x) = KiX~a, i = 1,2,3. Take K{ = ΐ, and let a take the values 2.5 and 1.5 in order to compare the performance of the approximation under finite- and infinite-variance conditions, respectively. Next, generate a renewal process {τn} of length L, with mean μ. Take a simple Poisson process with mean μ — 1. Use {An} and {τn} to generate the shot noise process {X(t)} of length T, where T < 7χ. Three impulse response functions h( ) are compared: h(x) = 1 for 0 < x < 1 and 0 otherwise; h(x) = 1 - x for 0 < x < 1 and 0 otherwise; and finally, h(x) = exp(—x) where x > 0, just to test the assumptions made on h( ) . Finally, find the maximum of the shot noise process. This procedure is followed 500 times in order to approximate the true underlying distribution. To complete the simulation study, the approximation is computed and then compared to the empirical cumulative distribution function (CDF) of the simulated process. The results of this exercise are shown in Figure 1 (for a = 2.5, or finite variance) and Figure 2 (for a — 1.5, or infinite variance). Each figure is a table of graphs, with the cases indicated by the value of T across the top, and by the impulse response function h(x) down the left-hand side. Each individual graph shows the approximation of the distribution of max X(t) 0
for that particular case (the solid line), along with 10 different realizations

EXTREME

43

VALUES

of the empirical CDF (the dotted lines) computed from simulations of the true under lying distribution of max X(t) outlined above. v y ° o
Chernick, M. R., Hsing, T. and McCormick, W. P. (1991). Calculating the extremal index for a class of stationary sequences. Adv. Appl. Prob. 25, 835-850. Denzel, G. E. and O'Brien, G. L. (1975). Limit theorems for extreme values of chain-dependent processes. Ann. Probab. 3, 773-779. Doney, R. A. and O'Brien, G. L. (1991). Loud shot noises. Ann. Appl. Prob. 1, 88-103. Guttorp, P. (1995). Stochastic Modeling of Scientific Data. Chapman and Hall, London. Homble, P. and McCormick, W. P. (1995). Weak limit results for the extremes of a class of shot noise processes. J. Appl. Prob, 32, 707-726. Hsing, T. L. and Tengels, J. L. (1989). Extremal properties of shot noise processes, Adv. Appl. Prob. 21, 513-525. Lawrance, A. J. and Kottegoda, H. T. (1977). Stochastic modeling of riverflow time series. J. R. Statistic. Soc A 140, 1-14. McCormick, W. P. (1997). Extremes for shot noise processes with heavy tailed amplitudes. J. Appl. Prob. 34, 643-656. McCormick, W. P. and Seymour, L. (2001). Rates of convergence and approximations to the distribution of the maximum of chain-dependent sequences. Extremes, 4, 23-52. Resnick, S. I. and Neuts, M. F. (1970). Limit laws for maxima of a sequence of random variables defined on a Markov Chain. Adv. Appl. Prob. 2, 323-343. Waymire, E. and Gupta, V. K. (1981). The mathematical structure of rainfall representations 3. Some applications of the point process theory to rainfall processes. Water Resources Res. 17, 1287-1294. Weiss, G. (1973). Filtered Poisson processes as models for daily streamflow data. PhD thesis. Imperial College, London.

44

MCCORMICK AND SEYMOUR

Figure 1: Approximation Compared to 10 Simulated Distributions, α = 2.5

T = 100

T = 20

10

20

30

40

50

20

40

60

80

100

20

40

60

80

100

I

0.8 0.6 U.4 0.2 n

f

I

I

I /

Appx Emp. CDF 20

40

60

80

100

EXTREME VALUES

45

Figure 2: Approximation Compared to 10 Simulated Distributions, α = 1.5

T=100

0.8 0.6 0.4 0.2 100

200

300

0

200

400

600

800

1000

800

1000

1 0.8 0.6 0.4 0.2

100

300

0

0.8

0.8

3-0.6

0.6

X α>

200

400

0.4

II

3; 0.4 x:

600

Appx 0.2

0.2 100

200

300

Emp. CDF 200

400

600

800

1000

47

Institute of Mathematical Statistics LECTURE NOTES — MONOGRAPH SERIES

STATISTICAL INFERENCE FOR STOCHASTIC PARTIAL DIFFERENTIAL EQUATIONS B.L.S. Prakasa Rao Indian Statistical Institute, New Delhi Abstract Stochastic partial differential equations (SPDE) are used for stochastic modelling , for instance, in the study of neuronal behaviour in neurophysiology, in modelling sea surface temparature and sea surface height in physical oceanography , and in building stochastic models for turbulence. Probabilistic theory underlying the subject of SPDE is discussed in Ito (1984) and more recently in Kallianpur and Xiong (1995) among others. The study of statistical inference for the parameters involved in SPDE is more recent. Asymptotic theory of maximum likelihood estimators for a class of SPDE is discussed in Huebner, Khasminskii and Rozovskii (1993) and Huebner and Rozovskii( 1995) following methods in Ibragimov and Khasminskii (1981). Bayes estimation problems for such a class of SPDE are investigated in Prakasa Rao (1998, 2000) following the techniques developed in Borwanker et al.(1971). An analogue of the Bernstein-von Mises theorem for parabolic stochastic partial differential equations is proved in Prakasa Rao (1998). As a consequence, the asymptotic properties of the Bayes estimators of the parameters are investigated using the asymptotic properties of maximum likelihood estimators proved in Huebner and Rozovskii (1995). Nonparametric estimation of a linear multiplier for some classes of SPDE are studied in Prakasa Rao(2000a,b) by the kernel method of density estimation following the techniques in Kutoyants(1994).

48

PRAKASA RAO

Key words: Bernstein-von Mises theorem, Stochastic partial differential equation, Maximum likelihood estimation, Bayes estimation, Nonparametric inference, Linear multiplier.

AMS Subject classification (2000): Primary 62M40; Secondary 60H15, 35R60.

1

Introduction

Stochastic partial differential equations(SPDE) are used for stochastic modelling, for instance, in the study of neuronal behviour in neurophysiology, in modelling sea surface temperature and sea surface height in physical oceanography and in building stochastic models for the behaviour of turbulence (cf. Kallianpur and Xiong (1995)). The probabilistic theory of SPDE is investigated in Ito (1984), Rozovskii (1990), Kallianpur and Xiong (1995) and De Prato and Zabczyk (1992) among others. Huebner et al.(1993) started the investigation of maximum likelihood estimation of parameters for a class of SPDE and extended their results to parabolic SPDE in Huebner and Rozovskii (1995) following the approach of Ibragimov and Khadsminskii(1981). Bernstein -von Mises type theorems were developed for such SPDE in Prakasa Rao (1998,2000) following the techniques in Borwanker et al.(1971) and Prakasa Rao (1981). Asymptotic properties of the Bayes estimators of parameters for SPDE were discussed in Prakasa Rao (1998,2000). Statistical inference for diffusion type processes and semimartingales in general is studied in Prakasa Rao (1999a,b). Our aim in this paper is to review some recent work of us on the Bernstein-von Mises type theorems for parabolic SPDE and to present some new results on the problem of estimation of a linear multiplier for a class of SPDE using the methods of nonparametric inference following the approach of Kutoyants (1994).

49

STOCHASTIC PDE

2

Main Results

2.1

Bernstein-von Mises theorem

Let (Ω,^*, P) be a probability space and consider a stochastic partial differential equation (SPDE) of the form duΘ(t,x)

= Aθuθ(t,x)dt

+ dW(t,x),0

(2.1)

where Aθ — ΘA\ + AQ, A\ and AQ being partial differential operators, θ E Θ C R and W(t,x) is a cylindrical Brownian motion in L,2(G),G being a bounded domain in Rd with the boundary dG as a C°°-manifold of dimension (d — 1) and locally G is totally on one side of dG. For the definition of cylindrical Brownian motion, see, Kallianpur and Xiong (1995), p.93. The order Ord(A) of a partial differential operator A is defined to be the order of the highest partial derivative in A. Let mo and m\ be the orders of the operators Ao and A\ respectively. We assume that the operators ^o and A\ commute, m\ is even and

(CO)

mi> \{Ord{Aθ)-d)

in the following discussion. Suppose the solution uθ(t,x) of (2.1) has to satisfy the boundary conditions uθ(0,x)=uQ(x)

(2.2)

and (2 3 )

2?Mt,z)|βσ = 0

(24)

for all multiindices 7 such that — | y| = m — 1 where 2m = max(mi,mo). Here

> '

with | 7 | = 71 +

+ 7d Suppose that

A(x)u = -

Σ

{-lt\Da{af{x)D^u)

\a\,\β\
(2.5)

50

PRAKASA RAO

where

af{x)eC°°(G).

(2.6)

aaβ(θ,x) = θaf(x)+af(x).

(2.7)

Let

Suppose ΘQ is the true parameter. We follow the notation introduced in Huebner and Rozovskii (1995). Assume that the following conditions hold. (HI) The operators AQ and A\ satisfy the condition / Aiuvdx = / uAivdx,u,v E CZ°(G),i = 0,1.

JG

JG

(H2) There is a compact neighbourghood Θ of θo so that {AQ, θ E Θ} is a family of uniformly strongly elliptic operators of order 2m = mαa;(rai,mo). For s > 0, denote the closure of C£°(G) in the Sobolev space WS>2{G) by W05'2 The operator Aθ with boundary conditions defined by (2.3) can be extended to a closed self-adjoint operator C$ on Li2{G) (Shimakura(1992)). In view of the condition (H2) , the operator CQ is lower semibounded, that is there exists a constant k(θ) such that —CQ + k(θ)I > 0 and the resolvent {k(θ)I - Cθ)~ι is compact. Let Λ# = (k(θ)I - Cθ)^. Let hi(θ) be an orthonormal system of eigen functions of Λ#. We assume that the following condition holds. (H3) There exists a complete orthonormal system {hi, i > 1} independent of θ such that

The elements of the basis {h{,i > 1} are also eigen functions for the operator CQ, that is i = μi hi

where

51

STOCHASTIC PDE

For s > 0, define H$ to be the set of all u G L<2{G) such that CO

λ

Nk* = (Σ f ( For s < 0, Hβ is defined to be the closure of L2(G) in the norm |M| S j 0 given above. Then Hρ is a Hubert space with respect to the inner product (., .)Sj0 associated with the norm \\.\\sβ a n d the functions hsiθ = \~s(θ)hι,i

>

1 form an orthonormal basis in HQ. Condition (H2) imples that for every 5 ,the spaces H$ are equivalent for all θ. We identify the spaces HQ (denoted by Hs) and the norms !|.|| s ,0 for different θ € θ. In addition to the conditions (H1)-(H3), we assume that (H4)u 0 G H~a where a > \. Note that u0 e L 2 (G), (H5) the operator Ai is uniformly strongly elliptic of even order mi and has the same system of eigen functions {h^i > 1} as Cg. The conditons (H1)-(H5) described above are the same as those in Huebner and Rozovskii (1995). Note that u0 E H~a. For θ <E Θ, define θ

(2.8)

u 0ι = (uo,h-«)-a. Then the random field 00

θ

Σ

θ

(

x

)

(2-9)

1=1

is the solution of (2.1) subject to the boundary conditions (2.2) and (2.3) where u\ (ί) is the unique solution of the stochastic differential equation du\{i) = μθiUi(t)dt + \-a{θ)dWi{t),0 < t < T,

u{f)(0)=uθ0ι.

(2.10)

(2.11)

52

PRAKASA RAO Let πN be the orthogonal projection operator of H~a onto the subspace

spanned by {h~θa, 1 < i < N}. Let uN'θ{t,x)

= πNuθ(t,x)

(2.12)

θ

where u τ(t) is the solution of (2.10) subject to (2.11). Note that duNfi(t,

x) = AeuNfi{t,

x)dt + dWN(t,

(2.13)

x),0
with (2 1 4 ) and

WN(t,x) = ΣKaWi(t)h-θa(x). Here {Wi(t),t

(2.15)

> 0},2 > 1 are independent standard Wiener processes.

Let Pf be the probability measure generated by uNfi Let h~a denote h~^,uN

on C([0,T]; RN).

denote uN>θ° and u denote uθ° when #o is the true

parameter. It is known that, for any θ E Θ, the measures P^ and P^ are absolutely continuous with respect to each other and T r

fJpN lOg

dP*(uN)

(ύ2 _ ύ2\

= (θ-θo)J(AlUN(s),duN(s))o-[

θo

0) 2

0

T r

J \\A,uN{s)\\lds 0

T

-(θ - ΘQ) J(AιUN{s),

AouN{s))ods.

(2.16)

o It is easy check that (cf. Huebner and Rozovskii (1995)) N

N

J(A1u (s),dW (s))0 ΘN-ΘO

(2.17)

=^

f\\AιU"(sψods 0

where ΘN is the maximum likelihood estimator of ΘQ. Huebner and Rozovskii (1995) studied the asymptotic properties of this estimator under the

53

STOCHASTIC PDE conditions (H1)-(H5). Further more the Fisher information is given by T

IN = Ej\\AιuN'θ°(s)\\ld8.

(2.18)

o Note that IN -* oo as N -> oo from the Lemma 2.1 of Huebner and Rozovskii (1995). Suppose that Λ is a prior probability measure on (Θ,i?) where B is the σ-algebra of Borel subsets of set Θ C R. We assume that the true parameter #o £ Θ°, the interior of Θ. Further suppose that Λ has the density λ( ) with respect to the Lebesgue measure and the density λ( ) is continuous and positive in an open neighbourhood of θo, the true parameter. Let τ = IιJ2(θ-θN)

(2.19)

and

p*(τ\uN) = Γ^PΨN

+ τΓNll2\uN)

(2.20)

where ^ ( 0 1 ^ ) is the posterior density of θ given uN. Note that P(θ\uN) = —$

(2.21)

S^(u^)\{θ)dθ and let p*(τ\uN)

denote the posterior density of ij

(θ — §w). Let

dP? θ

In view of (2.16), it follows that 2

(2.23)

\ogvN{τ) = -\r l^ I WA^WWlds since HA1uN(s),duN(s)-A0(s)uN(s)d8)0 ΘN = 2 0

.

(2.24)

54

PRAKASA RAO

Let oo

CN = I uN(r)λ(θN + T / - 1 7 V

(2-25)

—oo

It can be checked that + r/-1/2).

p*(τ\uN) = C^uN(r)X(θN

(2.26)

T 1

Note that (Cl)

N

βN = I^ J \\Aχu {s)\\lds -> 1 a.s. [Pθo] as N -> oo. o from the Lemma 2.2 of Huebner and Rozovskii (1995). Then the following relations hold: (i)

iv

limo^(r)=exp(-ir2)a.s.[P(,0],

(ii) for any 0 < 7 < 1, 1

o.

x

for every r for sufficiently large JV, and (iii) for every 5 > 0, there exists 7' > 0 such that sup

vN(τ) <

exp{j'Iΰ1

as N ->• 00.

Further more (C2) the maximum likelihood estimator (9# is strongly consistent, that is ΘN -> #o a.s. [P(90] as TV -> 00. from the Lemmas 2.1 and 2.2 in Huebner and Rozovskii (1995). Suppose that (C3) K( ) is a nonnegative function such that, for some 0 < 7 < 1, 00

ί

<

00.

(C4) For every 77 > 0 and δ > 0, K(τI-l/2)X(θN

+ τ)dτ -> 0 a.s

55

STOCHASTIC PDE as N —> oo.

We now have the following main theorem which is an analogue of the Bernstein-von Mises theorem (cf.Prakasa Rao (1981, 1984)) for diffusion processes and diffusion fields. A special case of this result for some classes of SPDE's was recently proved in Prakasa Rao (2000).

Theorem 2.1: Suppose the conditions (C3) and (C4) hold in addition to the conditions (H1)-(H5) stated earlier where λ( ) is a prior density which is continuous and positive in an open neighbourhood of #o? the true parameter. Then / i

\

1/2

K(τ)\p*(τ\uN) - ( - )

e-Γ \dτ = 0 a.s. [Pθo].

(2.27)

-oo

As a consequence of Theorem 2.1, it is easy to get the following result.

Theorem 2.2: Suppose the conditions (H1)-(H5) hold. In addition suppose that: (Dl) λ( ) is a prior density which is continuous and positive in an open neighbourhood of θo, the true parameter; and (D2)

m

/ \θ\ λ(θ)dθ < oo for some integer m > 0.

Then OO

lim

1.

mm

NN

/ \τ\ \p*(τ\u \p*(τ\u ) ) -( (± ^ λ)* e-^\dr

= 0 a.s. [Pθo].

(2.28)

Remarks: It is obvious that the condition (D2) holds for m = 0. Suppose the condition (Dl) holds. Then it follows that 0 0

1/2

lim ί \p*(r\uN) - (±-)

e-^ldr - 0 a.s. [P,o].

(2.29)

56

PRAKASA RAO

This is the analogue of the Bernstein-von Mises theorem in the classical statistical inference. As a particular case of Theorem 2.2, we obtain that ~ θo)}m -> E[Z)m as N -> oo

(2.30)

where Z is JV(O,1).

For proofs of Theorems 2.1 and 2.2, see Prakasa Rao(1998).

3

Bayes Estimation

We define an estimator ΘN for θ to be a Bayes estimator based on the path uN corresponding to the loss function L(θ,φ) and the prior density λ(0) if it is an estimator which minimizes the function

BN(φ) = jL(θ,φ)p(θ\uN)dθ,φ(Ξ Θ where L(θ,φ) is defined on Θ x Θ. Suppose there exist a Bayes estimator 0/v. Further suppose that the loss function L{θ,φ) satisfies the following conditions: (El) L(θ,ψ)=L(\θ-φ\)>0; (E2) L(t) is nondecreasing for t > 0; (E3) there exists nonnegative functions (a) RNL{τI~1/2)

and G(τ) such that

RN,K(T)

< G{r) for all N>1;

I/O

(b)

RNL(TIN

' ) -> K(τ) as TV -> oo uniformly on bounded intervals

of τ; (c) the function oo _ !

/

2

K(r + ra)e~2r dr

—oo

achieves its minimum at m — 0, and (d) G(τ) satisfies the conditions similar to (C3) and (C4).

STOCHASTIC

57

PDE

The following result can be proved by arguments similar to those given in Borwanker et al. (1971). We omit the proof.

Theorem 3.1: Suppose the conditions (D1)-(D2) of Theorem 2.2 hold in addition to (H1)-(H5) stated earlier. In addition , suppose that the loss function L{θ,φ) satisfies the conditions (El) - (E3) stated above. Then Oa.s. [Pθo] asJV->oo

(3.1)

and lim RNBN(ΘN) N-+00

=

lim RNBN(ΘN)

(3.2)

N—ΪOO 1/2

Huebner and Rozovskii (1995) proved that ΘN -> 0O a.s. [Pθo] as N -> oo

(3.3)

and I]!2ΦN

- θ0) 4 iV(0,1) as N -> oo

(3.4)

under the conditions (H1)-(H5). As a consequence of Theorem 3.1, it follows that 07V -> 0o a.s [Pθo] as TV -> oo

(3.5)

and

l]ί2φN

- 0O) 4 7V(0,1) as N -> oo.

(3.6)

In other words the Bayes estimator ΘM of the paramaeter 0 in the parabolic SPDE given by (2.1) is strongly consistent, asymptotically normal and asymptotically efficient as N —> oo under the conditions (H1)-(H5) of Huebner and Rozovskii (1995) and the conditions stated in Theorem 3.1. Remarks:

A general approach for the study of asymptotic properties

of maximum likelihood estimators and Bayes estimators is by proving the local asymptotic normality of the loglikelihood ratio process as was done in

58

PRAKASA RAO

Prakasa Rao (1968), Ibragimov and Khasminskii (1981) in the classical i.i.d. cases and by Huebner and Rozovskii (1993) for some classes of SPDE. Our approach for Bayes estimation, via the comparison of the rates of convergence of the difference between the maximum likelihood estimator and the Bayes estimator, is a consequence of the the Bernstein - Von Mises type theorem .

We now consider a nonparametric version of the problem discussed earlier for a class of SPDE.

4

Stochastic PDE with Linear Multiplier

Let (Ω,^1*, P) be a probability space and consider the process u e (£,x),0 < x < 1,0 < t
+ εdWQ(t,x)

(4.1)

where Δ = J ^ . Suppose that ε —> 0 and θ £ Θ where Θ is a class of real valued functions 0(i),O < t < T uniformly bounded , k times continuously differentiate and suppose that the λ -th derivative θ^k\.) satisfies the Lipschitz condition of order α G (0,1], that is, |0(*)(t) _ 0(*)( 5 )| < \t - s\a,β = k + a.

(4.2)

Further suppose the initial and the boundary conditions are given by ue(O,a0 = /(α0,/€L2[O,l]

( 43 )

u e (ί,0) = u e ( t , l ) = 0 , 0 < ί < T and Q is the nuclear covariance operator for the Wiener process WQ(<, X) taking values in L2[0,1] so that WQ{t,x)=Qι/2W(t,x) and W(t, x) is a cylindrical Brownian motion on ί-2[0,1] Then, it is known that (cf. Rozovskii (1990), Kallianpur and Xiong (1995)) OO

) = Σql/2ei(x)Wi(t) i=\

a.s.

(4.4)

59

STOCHASTIC PDE

where {Wi(t),0 < t < T},z > 1 are independent one - dimensional standard Wiener processes and {e^} is a complete orthonormal system in Z^IP, 1] consisting of eigen vectors of Q and {^} eigen values of Q. We assume that the operator Q is a special covariance operator Q with βfc = sin(kπx),k

> 1 and λ^ = (πk)2,k > 1. Then {e^} is a complete

orthonormal system with eigen values qι = (1 + AJ" 1 ^' > 1 for the operator Q and Q = (/ - Δ)"" 1 . Note that dWQ = Qιl2dW.

(4.5)

We define a solution u£(t,x) of (4.1) as a formal sum oo

Uε{t,x) = (cf. Rozovskii (1990)). It can be checked that the Fourier coefficient Uiε(t) satisfies the stochastic differential equation duiε{t) = (0(t) - \τ)uιε{t)dt +

,S V Λi + 1

dWj{t):

0 < t < T

(4.7)

with the initial condition uιε(0) = vu vi=

ί f(x)eι(x)dx. Jo

(4.8)

We assume that the initial function / in (4.3) is such that Vi = / f{x)et(x)dx>0, Jo

i > 1.

Estimation of linear multiplier We now consider the problem of estimation of the function θ(t), 0 < t < T based on the observation of the Fourier coefficients Uiε(t), 1 < i < N over [0, T] or equivalently the projection uε ' (ί, x) of the process uε(t, x) onto the subspace spanned by {ei,..., ejy} in L2[0,1]. We will at first construct an estimator of #(.) based on the path {ixie(t), 0 < t < T}. Our technique follows the methods in Kutoyants (1994), p.155.

60

PRAKASA RAO Let us suppose that sup sup |0(t)| < L 0 .

(4.9)

θeθθ
Consider the differential equation J i l l ! = (θ(t) - \i))ui(t),Ui{0) =Vi,0
(4.10)

and hence Ui(t) >vle~m1Q
(4.11)

where Mz = L 0 + λi.

(4.12)

^From the Lemma 1.13 of Kutoyants (1994), it follows that sup \uie(s)-Ui(s)\< o<s<τ almost surely. Let

,/ eM* sup \Wi(s)\ yλi +1 o<5<τ

« = {ω :Qmf ul£{s) > \vie-M^}

(4.13)

(4.14)

I)

and Ai = A^. Note that A[ contains the set A{ for 0 < t < T. Define the process {Y;ε(£),0 < t < T} by the stochastic differential equation

+u-1(t)χ(A?)duiε(t),0

(4.15)

where χ(E) denotes the indicator function of a set E. Let φε —> 0 as ε -> 0 and define

STOCHASTIC

61

PDE

Γ

ΓG()dYiε(s) (4.16) Φε where G(.) is a bounded kernel with finite support , that is , there exists JO O

constants a and b such that G(u)du — 1, G{u) = 0 for u < a and u > b.

(4.17)

We suppose that a < 0 and b > 0. Further suppose that the kernel G(.) satisfies the additional condition roo / G{u)uJdu J—oo

= 0,j

=

l,...,

(4.18)

Note that

ΓG{t—^)dYiε{s) O

(4.19)

Φε

-u-ε2{s))χ{Aψ)ds

Hence

0

-E[x(Aτ)φJι

Φε T

ΓG(t

JO

+E\χ{Ai)φ7ι Γ G(t Jo Note that 1

f

Jθ

- λi)ds]

62

PRAKASA RAO

Therefore, for sufficiently small ε > 0,

E[θiε(t)-Θ(t)}

= E\χ(Ai)φJι fo

G{t-^-){θ{s)-θ{t))ds\

-E[x(A<)(θ(t)

- Xi)

i)φ? ίT'G(tQ JQ

φε

(4.20)

+O(JUε) since, for 0 < t < T, ΦΊl iTG{~)ds JO

(4.21)

=l

Φε

for sufficiently small ε > 0 by the conditions imposed on the kernel G. Therefore, for 0 < t < T, for sufficiently small ε > 0,

\E[θιε(t) - θ(t)}\2

< 4{φJι ίT G ( ^ ) ( 0 ( s ) - θ(t))ds}2 JO

Φε

(4.22)

Applying the Taylor series expansion and properties of the kernel G(.) and the function #(.), it is easy to see that the first term is bounded by roo

{

\G{u)uP\duγ

(4.23)

J -oo

where C\ is a constant depending only on the the constants Lo in (2.9) and the smoothness parameter k of θ(.). Note that

P(Aξ)

= P 0
*

since P( sup \Wi(t)\ > α) < min(2,

~\lγ)

from Kutoyants (1994), p.28. The third term is bounded by

where C2 is an absolute constant. Relations (4.22) -(4.24) show that

\E[θiε(t)-θ(t)}\2

<

Cz[φf{Γ

\G{u)uβ\du}2

J -ΌO

+O(J2iε)

(4.25)

where C3 is an absolute constant. Hence

\E[θιε(t) - θ(t)}\2 < C,[φ2/ + e-^ε~\θ(t)

- \τ)2 + ε% + Jl%ε]

(4.26)

where C4 is a constant depending on the kernel G(.) and the Lipschitz constant Lo,

di = '^e

4M

'^Xi

+ 1 }

(4.27)

and

Following computations as given above(cf. Kutoyants (1994), p. 157), we can show that E[θiε(t)-Θ(t)}2

< Cb[φ2εβ + e-d^2{θ{t)-\i)2

+ ε%+ε2φ-ιki

+ J2iε] (4.29)

PRAKASA RAO

64

where C5 is a constant depending on the kernel G(.) and the Lipschitz constant LQ. Choosing φε such that

2

we obtain that φε = ε 2

E[θiε(t)-Θ(t)}

(2/3+1

) and we have d

ε

2

(4.30)

4

and 2 ε (E[θiε(t) - θ(t)}) < CA[e*tfc + e-* ~\θ(t)

2

2

- Xτ) + ε% + J Uε}.

(4.31)

Note that 0;ε(£),l < i < N are independent estimators of θ(t) since the processes W^ 1 < i < AT are independent Wiener processes. The above inequalities imply that sup % ( ί ) - 9 ( t ) ] 2
< C 6 [ ε ^ + e - ( i n f l ^ " ^ ε ~ 2 sup {θ(t) - Xtf +ε

sup fci

sup

\<ι
+ sup J2iε] Note that

(4.32)

2{L0+N2π2)T

e

sup fei < — l

Ώ

Ϊ

τ

V

- /3 N (say)

(4.33)

and inf di > ( inf v2) Ki
Therefore

K

~ Kτ
l J

= -yN (say).

(4.34)

\T

sup

+ sup

(4.35)

Ki
In particular sup Var(θιε{t))

<

sup Ki
(4.36)

65

STOCHASTIC PDE

We assume that the following conditions hold for 1 < i < N. Let ηε — . Suppose that

Under the above condition , it follows that the estimators θiε(t), 1 < i < N are independent estimators of θ(t) such that 2/3

sup lEtfieit) - θ(t)]\ < Csεw+i

(4.37)

l
and sup E[θie(t) - θ(t)]2 < C9ε^

(4.38)

where C% and Cg are constants depending on the kernel G(.), the Lipschitz constant Lo and N. Note that the estimators 0χε(ί), 1 < i < N are the best estimators of θ(t) as far as the rate of mean square error are concerned by Theorem 4.6 in Kutoyants (1994). We now combine these estimators in an optimum fashion to get an estimator using all the information available. It is easy to check that

Γ

JO

(4.39) =

χ(Ai)Jιiε

+ J2iε (say)

(4.40)

Note that E{Jfiε) = 0(7f J u J where JO

Φε

^

λi -t L

ψ

(4.41)

as defined earlier. In addition to (CΊ), assume that

for 1 1 as ε ->• 0, it follows by the Central limit theorem for stochastic integrals (cf. Kutoyants (1994), Prakasa Rao (1999a)) that 1 ^

W(0, ^

^

f°° ^

/

θ\u)Λu)

(4.42,

66

PRAKASA RAO

as ε -> 0 for 1 < i < N. Define &Ne{t) — —

l = l

6

"

N*

2—"

'

(4.43)

Note t h a t t h e r a n d o m variable ί?Λ/ε(ί) is n o t a n estimator of θ(t) as t h e functions Ui(t) depend on t h e function θ(t). However t h e r a n d o m variable θχε(t)

is a linear function of independent r a n d o m variables θiε(t), 1 < i < N.

^From t h e earlier calculations , it can b e checked t h a t E(θNε(t)-θ(t)f

2

= Var(θNε(t)) +

(E(θNε(t)-θ(t)) 43

4/3

ϊ

(4.44)

As a consequence, we have the following result. Theorem 4.1: Under the conditions stated earlier, for 0 < t < T, (i) (ii)

E(θNε(t))

(iii) lim ε ^o E(θNε{t)

- θ(t))2 -+ as ε -+ 0;

(iv) limsup ε ^ 0 E(θ^ ε (t) — θ(t))2ε20+1 < oo. (v) ε^h{θNε{t)

- θ{t)) 4 ΛΓ(0,σ2(ί)) as ε -)• 0

where N(0, σ2(t)) denotes the normal distribution with mean zero and variance σ 2 (ί)) given by 1

=

Let

where

roo

G2(u)du.

(4.45)

67

STOCHASTIC PDE

ύiε(t) = υ , e / o ( ^ ω - λ ^ .

(4.47)

Note that for any 1 0

(4.48)

Jo

for 1 0

(4.49)

for 1 < i < N. In view of (4.30), it follows that the estimator θ*Nε(t) is a consistent estimator of θ{t). Theorem 4.2: Under the conditions stated above, for 0 < t < T, fl vεW "^ θ(t) as ε -^ 0. Note that

ΣϊLi Since (i) 7ε(4(ί) - θ(t)) 4 ΛΓ(0,σ2(ί)) as ε -> 0 for 1 0 for 1 < i < N,

(4.50)

68

PRAKASA

RAO

and since the estimators θiε(t),l < i < N are independent random variables, it follows that the estimator θ*Nε(t) is asymptotically normal and we have the following theorem. Theorem 4.3: Under the conditions stated earlier, for 0 < t < T, Ίε{θ*Nε{t) - θ(t)) 4 iV(0, σ2{t)) as ε -> 0

(4.51)

= ε-2lτϊ

(4.52)

where

Ίζ

and

Remarks l:If k = 0 and /? = 1, that is, the function θ(.) G Θ where Θ is the class of uniformly bounded functions which are Lipschitzian of order one, then it follows that ε'Hθ*Nε{t) - θ(t)) 4 N(0,σ2{t))

as ε -> 0.

(4.54)

Remarks 2: It is known that the probability measures generated by stochastic processes satisfying the SPDE given by (4.1) are absolutely continuous with respect to each other when θ(.) is a constant (cf. Huebner et al.(1993)). There are classes of SPDE which generate probability measures which are singular with respect to each other when θ(.) is a constant. One can study the problem of nonparametric inference for a linear multiplier for such a class of SPDE by the above methods(cf. Prakasa Rao (2000b)).

References Borwanker, J.D., Kallianpur, G. and Prakasa Rao, B.L.S. 1971 The Bernstein-von Mises theorem for Markov processes, Ann. Math Statist. 42, 1241-1253.

STOCHASTIC PDE

69

Da Prato, G. and Zabczyk, J. 1992.Stochastic Equations in Infinite Dimensions, Cambridge University Press, Cambridge. Huebner, M., Khasminskii, R. and Rozovskii. B.L. 1993. Two examples of parameter estimation for stochastic partial differential equations, In Stochastic Processes : A Festschrift in Honour of Gopinath Kallianpur, Ed. S.Cambanis, J.K.Ghosh, R.L.Karandikar, P.K.Sen, Springer, New York, pp. 149-160. Huebner, M., and Rozovskii, B.L. 1995.On asymptotic properties of maximum likelihood estimators for parabolic stochastic SPDE's. Prob. Theory and Relat Fields, 103, 143-163. Ibragimov, LA., and Khasminskii, R. 1981. Statistical Estimation: Asymptotic Theory, Springer-Verlag, Berlin. Ito, K. 1984.^0^7-1^207^5 of Stochastic Differential Equations in Infinite Dimensional Spaces, Vol. 47 of CBMS Notes, SIAM, Baton Rouge. Kallianpur, G., and Xiong, J. 1995. Stochastic Differential Equations in Infinite Dimensions , Vol. 26, IMS Lecture Notes, Hayward, California. Kutoyants, Yu. 1994. Identification of Dynamical Systems with Small Noise , Kluwer Academic Publishers, Dordrecht. Prakasa Rao, B.L.S. 1968. Estimation of the location of the cusp of a continuous density, Ann. Math. Statist., 39, 76-87. Prakasa Rao, B.L.S. 1981. The Bernstein - von Mises theorem for a class of diffusion processes, Teor. Sluch. Proc.,9, 95-101 (In Russian). Prakasa Rao, B.L.S. 1984. On Bayes estimation for diffusion fields. In Statistics : Applications and New Directions, Ed. J.K. Ghosh and J. Roy, Statistical Publishing Society, Calcutta. Prakasa Rao, B.L.S. 1998. Bayes estimation for parabolic stochastic partial differential equations, (Preprint, Indian Statistical Institute, New Delhi).

70

PRAKASA

RAO

Prakasa Rao, B.L.S. 1999a. Statistical Inference for Diffusion type Processes, Arnold, London and Oxford university Press, New York. Prakasa Rao, B.L.S. 1999b. Semimartingales and their Statistical Inference, CRC Press, Boca Raton , Florida and Chapman and Hall, London. Prakasa Rao, B.L.S. 2000. Bayes estimation for some stochastic partial differential equations , J. Statist. Plan. Infer. , 9 1 , 511-524. Prakasa Rao, B.L.S. 2000a. Nonparametric inference for a class of stochastic partial differential equations, Tech. Report No. 293, Dept. of Statistics and Actuarial Science, University of Iowa. Prakasa Rao, B.L.S. 2000b. Nonparametric inference for a class of stochastic partial differential equations II, Statist. Infer, for Stock. Proc. (To appear). Rozovskii, B.L. 1990. Stochastic Evolution Systems, Kluwer, Dordrecht. Shimakura, N. 1992. Partial Differential Operators of Elliptic Type, AMS Transl. Vol. 99, Amer. Math. Soc, Providence.

71

Institute of Mathematical Statistics

LECTURE NOTES — MONOGRAPH SERIES

FIXED DESIGN REGRESSION UNDER ASSOCIATION George G. Roussas University of California, Davis Abstract For n = 1,..., 7i, let xni,i = 1,..., n, be points in a compact subset in Sftd,d > 1, at which observations Yn{ are taken. It is assumed that these observations have the structure Yni = g(xni) + εni> where g is a real-valued unknown function, and the errors (e n i, ^nn) coincide with the segment (£χ,... ,f n ) of a strictly stationary sequence of random variables ξi, & » — F° r each x G 5Rd, the function g(x) is estimated by gn(x]Xn) = i2?=iwni(x\xn)Yni, where xn = (xnli... ,x n n ) and Wnϊ( ; •) are weight functions. Under suitable conditions on the underlying stochastic process £1,62, and the weights wni( ; •), it is shown that the estimate gn(x\Xn) is asymptotically unbiased, and consistent in quadratic mean. By adding the assumption of (positive or negative) association of the sequence £i,&» • •> it is shown that ρ n (^;^n), properly normalized, is also asymptotically normal.

Key words and phrases: Fixed design regression, stationarity, weights, fixed design regression estimate, asymptotic unbiasedness, consistency in quadratic mean, association, asymptotic normality.

1

Introduction

For each natural number n, consider the design points xni, % = 1,... ,n in Sβd, d > 1, which, through a real-valued (Borel) function g defined on $ϊd, produce observations Ym, subject to errors ε n i, 1 1, is a (strictly) stationary and (positively or negatively) associated (see Definition 1.1) sequence of random variables (r.v.s). The problem we are faced with here is that of estimating

72

ROUSSAS

the function g in terms of the YniS and xniS, and establishing optimal properties for the proposed estimate. Following established tradition in this line of work, for each x G 5Rd, the contemplated estimate is gn(x\ xn) given by

gn{x;xn) = ^2wni(x;xn)Ynii

(1.2)

2=1

where xn = (xnι,... ,£ n n), and wni( ), 1 < i < n, are suitable weight functions. It will be shown that, under appropriate regularity conditions, the proposed estimate is asymptotically unbiased, consistent in quadratic mean, and asymptotically normal. Properties of this nature and for specific choices of the weight functions were established by Priestly and Chao (1972), and Gasser and Mύller (1979). This problem was also investigated by Georgiev and Greblicki (1986) and Georgiev (1988). In all of these cases, the errors εUu i = l , . . . 5 n , were assumed to be independent identically distributed (i.i.d.). When independence is replaced by strong mixing, the above cited results were established in Roussas (1989) and Roussas et al. (1992). In the present contribution, independence is suppressed again and is replaced by association. For a brief review on the significance of the concept of association, some of its applications, and a summary of some (statistical) results under association, the interested reader is referred to the review paper Roussas (1999). Relevant are also the papers of Cai and Roussas (1999 a,b). Important results on some limit theorems for dependent r.v.s, and, in particular, negatively dependent r.v.s may be found in Bozorgnia et al. (1996), Patterson and Taylor (1997), Taylor and Patterson (1997), and Taylor et al. (1999a,b). The paper is organized as follows. Asymptotic unbiasedness and consistency in quadratic mean are established in Section 2 after suitable assumptions are spelled out. Asymptotic mormality is proved in Section 3 along with a number of auxiliary results. Assumptions under which these results hold are also stated in this same section, and they are followed by some comments. This section is concluded with the definition of association. Definition 1.1. For a finite index set 7, the r.v.s {Xi\ i G /} are said to be positively associated (PA), if for any real-valued coordinatewise increasing functions G and H defined on SR7, Cσυ[G{Xuiεr),H(XJ9j

εη]>0,

provided EG2(Xi,i G I) < oo,£H2(XjJ G J) < oo. These r.v.s are said to be negatively associated (NA), if for any nonempty and disjoint subsets A and B of /, and any coordinatewise increasing functions G and H with G :

REGRESSION

73

UNDER ASSOCIATION

5ft and # : 5ftβ -> 5ft with SG2(Xi,i

e A) < oo,SH2(XjJ

i,z e A),H(XjJ

Eΰ)
e B)] < 0.

If / is not finite, the r.v.s {Xi\i G /} are said to be PA or NA, if any finite subcollection is a set of PA or NA r.v.s, respectively. Finally, it is mentioned at the outset that all limits are taken as n —> oo unless otherwise stated, and C stands for a generic (positive) constant.

2

Asymptotic Unbiasedness and Consistency in Quadratic Mean

Assumptions (A) (Al) For a compact subset S of 5ftrf, the function g : S —> 5? is continuous. (A2) For 1 1, the errors εniS have expectation 0. For each x G S and with xn = (x n χ,... , x n n ) ϊ ^ m € 3?d, i = 1 , . . . ,n, the weights wni(x]xn) are 0 for i > n, and satisfy the following requirements for 1 1, for a positive constant B.

(A4) Σ ? = i k f e « n ) h l . (A5) For any c> 0, £ " = 1 |^ n ί (a;;x n )|/ ( || X n ._ ; r || > c ) (a;) -> 0, where || || is any one of the familiar norms in 3?d. All results in this paper hold for all x G 3id and with xn as defined above. Theorem

2.1 (asymptotic unbiasedness). £gn{x',xn)

Proof. Writing gn(x) and wni(x)

Under assumptions (Al) - (A5),

->g(χ)

instead oί gn(x\xn)

and wni(x;xn),

re-

74

ROUSSAS

spectively, we have n

Sgn(x) - g(x) I =:| ^2wni{x)g{xni)

- g{x) \

2=1

n

7/j

ίr^ II niΎ \ — n(τ\

I Wnι\?) II 9 ^ m J

9VXJ I i (||xni-^||'

i=l

For every ε > 0 and sufficiently small c = c(ε), consider those xnjS for which || ίr^i — re ||< c. Then | g(xni) — g{%) |< ε, and therefore | g{xni) — g(x) I /(||χ ni -χ||< c )(^) < ε Thus, for all sufficiently large n, (2.1) yields I £gn(x) — g(x) ϊ< 2Cε + εC + εC = 4εC, where C is a suitable bounding constant. This completes the proof. • Before the formulation of the second main result, assumptions (A) are augmented as follows. Assumptions (B) (Bl) For each n > 1, (ε n i,..., εnn) is equal in distribution to (ξi,..., ξ n ), where {ξn}5 n > 1, is a (strictly) stationary sequence of r.v.s, £ξf = σ 2 < oo, (B2) For each x G 3id and x n as above, {\wni(x] xn)\ 1 0. Theorem 2.2 (consistency in quadratic mean). Under assumptions (Al) (A5) and (Bl) - (B2), ε[gn(x;xn)-g{x)]2->0. Proof. For further notational simplification, write just wni instead of v>ni{x) — Wmθz;#n), and recall that wn = max{\wni\; 1 < i < n}. Then, by assumptions (A3) and (B2), (2.2)

REGRESSION UNDER ASSOCIATION

75

Next, 2

2

£ [9n(x) - 9(x)] = ε [gn(x) - εgn{x)}

2

+ [£gn(x) - g(x)] ,

and the second term on the right-hand side above tends to 0, under assumptions (Al) - (A5), by Theorem 2.1. So, it suffices to show that Var(gn(x)) ->> 0. To this end,

Var(gn(x)) = Var Σ w

)

[ ^2 wniwnjS

£ε

li li

{εniεnj)

i=\

(2.3)

jS {εniεnj).

Since the first term on the right-hand side of (2.3) tends to 0 by (2.2), it suffices to show that 0. By assumption (Bl), jZ

(εniεnj)

iξj)

n—\

t=l

n-l

•\Cσo(ξltξn-i+ι)\] 2=1

(by stationarity) n—1

t=l

n—i

s.i=l

76

ROUSSAS

by assumptions (A3), (Bl) and (B2). • Remark 2.1. At this point, it is to be observed that Theorems 2.1 - 2.2 were established without reference to association. The property of association is used only in Theorem 3.1, stated and proved in Section 3.

3

Asymptotic Normality

Introduce the following notation by suppressing the argument z. Set Zni

= σ~ιwniεni,

σl

=Var

(gn>)

=

equal in distribution to σ~ιwniii, v

a r

t^rι

βM

,CΛ

Ί

,

ϊ

(όΛ)

.

Also, for m = 1,..., fc, let

Jm

={(m

and define ynm, y'nm and y^ by:

ynm = Σ Zni, y'nm = Σ Znj, y l = Σ ielm

jeJm

Z

^

(3 3)

l=k(p+q)+l

and let y'nm, ra=l

K = V'n

(3-4)

- Sgn).

(3.5)

m=l

We wish to show that Sn Λ ΛΓ(0,1),

where Sn = σ-\gn

Clearly, Sn = Tn+T'n+Tl

(3.6)

and (3.5) will be established by showing that Tn Λ JV(0,1),

(3.7)

and

ε{τ n γ + ε(τχf -> o.

(3.8)

77

REGRESSION UNDER ASSOCIATION

These assertions hold true under the set of assumptions stated below. Although some of the assumptions spelled out below coincide with assumptions previously made, we choose to gather all of them here for easy reference. Assumptions (C) (Cl) The sequence {£n}5 n > 1, is (either positively or negatively) associated and (strictly) stationary. 2+δ

(C2) Sξx = 0, S\ξι\

< oo for some δ > 0, and Σ ^ |Coυ(fi>£j+i)l < °°

(C3) For 1 1, (ε n i,... ,ε n r ι ) is equal in distribution to (fl,---,ίn). With wn = max{\wni(x\ # n ) | , 1 oo, and let k = kn be the largest integer for which k(p + q) < n. Then select p and q as just described, and also to satisfy the requirements: (C5)

(i) p = o{np), p = 2(ϊ^y (the same δ as in (C2)).

(ϋ) *?->!. Comments on some assumptions (a) The choice of p, q, and fcasθ
that * M -> 1 and J -> 0 (since J = ^

^ ) .

(b) $ - M implies f ->• 0 and J -»• 0 (since ^ψ± = & + f and both *ίH±ώ,$-> 1, and 2 = f - + 0 ) . n 1

(c) If p = o(n) (which is implied by (C5)(i)), then fc -> oo (since £ =

5) -»° b y ( b ))

£±£

78

ROUSSAS

(d) Choices of p and q as described above and satisfying condition (C5) are readily available. Indeed, for 0 < 62 < δ\ < p, take p ~ nδl and q ~ nδ2 (where xn ~ yn means | ^ ->• 1). This choice of p is consistent with (C5)(i) (since £ = ;£-• ^ 7 Γ "^ °) Furthermore, A; ~ n 1 "* 1 (since ^ = ^ • & and k = ^ . ^ . . ^ which tends to 1). Therefore $ = ^

^

^ 1.

(e) That δ in assumptions (C2) and (C5)(i) must be the same stems from the proof of Lemma 3.2(iii). (f) Assumption (C4)(ii) is borrowed from Roussas et al.(2000) (see Remark 2.1(ii), page 265). Theorem 3.1. Under assumptions (Cl) - (C5), the convergence asserted in (3.5) holds; that is,

where Sn is defined in (3.5), gn = gn{xm, xn) is given in (1.2), and σ\ = σ^(x)

= Var(gn). The proof of the theorem follows by combining the two propositions below. The propositions, as well as the three lemmas employed in this section, hold under all or parts only of assumptions (Cl) - (C5). However, these lesser assumptions will not be explicitly stated. Proposition

3.1. The convergence asserted in (3.8) holds; that is,

ε{τ'nγ + ε{τ£γ -+ o, where T'n and T% are given in (3.4). Proposition

3.2. The convergence asserted in (3.7) holds; that is,

τnΛ;\r(o,i), where Tn is given in (3.4). Assuming for a moment that Propositions 3.1 and 3.2 have been established, we have Proof of Theorem 3.1. It follows from Propositions 3.1 - 3.2 and relation

(3.6). • The following three lemmas will be required in various parts of the proofs of Propositions 3.1 - 3.2.

79

REGRESSION UNDER ASSOCIATION f

Lemma 3.1. Let ynm and y nm be defined by (3.3). Then: (i) KKr
and (ϋ) " 3=q

l
where, it is recalled from assumption (B2) that, wn = max{\wni\] 1 < i < n}, and σ\ is given in (3.1). Proof, (i) From the definition of the y n m s, and with the 7 m s as defined in (3.2), it is clear that

Coυ(ynl,ynr)

COV

Kl
Kl
^ Σ YJYJ

Ccro{ZnilZnj)

i<ι
<ί Σ where

(3.9)

A

-= Σ

(3.10)

l
However, by stationarity of the

jeh

-«ΣΣ|

Σ <

(fe-l)p

+ (p -

+

ROUSSAS

80 ... +

\Cov(ξ1,ξ{p+q)+p)

+

(k - 2) [p|Coι;(ξi,ξ2(P+,)+i)| + (P - 1)

(p-

) Uk-l)[(p-l)\Cσv(ξ2,ξ(p+q)+1)\

+. . . +

(k - 2) [(p - l)|c< W (ξ 2 ,ξ 2 ( P + ρ ) + i)| + 2[(p-]

•••+

(3-11)

+

where An\ and An2 stand for the first and second bracket in (3.11), respectively. However,

Anι

< pk jeh

i€/2

and

\Cσv(ξi,ξj+i)

(3.12)

REGRESSION UNDER ASSOCIATION

81

2(p+9)

P+Q

£

Coυ(ξ1,ξj+1)\+

An2
\Cσυ(ξuξj+1)

j=2{p+q)-(p-l) (fc-2)(p+g)

Σ

I

+

p+q J=Q

J=(P+Q)+Q

Σ

I

(k-2){p+q)

Σ

Cσv{ξuξj+1)

Cσυ{ζuξj+1)\ j=(k-2)(p+q)+q

j=(k-3)(p+q)+q

(3.13) Relations (3.11) - (3.13) imply that An < 2pk Y^Lq\Coυ(ξuξj+ι)l (3.9) and (3.10) yield

so that

Coυ(ynl,ynr) \
which is what part(i) asserts. (ii) It follows as in part (i) upon replacing the ynms by the y'nπβ. Corollary 3.1. It holds: Λ
Cov(ynhynr)

-+ 0, and (ii)

0.

Proof, (i) The right-hand side of the expression in Lemma 3.1(i) is written as: ( 5 ) ( $ ) K ) ΣT=q \Cov(ξuξj+ι)l and this is bounded by: CY%Lq \Coυ(ξι,ξj+ι)\, on account of assumptions (C4)(ii), (C5)(ii) and (C4)(i). However, this last expression tends to 0 by assumption (C2). (ii) Likewise, the right-hand side of the expression in Lemma 3.1(ii) is written ^: ( ? ) ( ί ) K ) ΣT=P \Cσv(ξi,ξj+i)\ ^ 0 as in part (i). •

ROUSSAS

82 We may now proceed with the proof of Proposition 3.1. Proof of Proposition 3.1. First,

y^) , (3.14) m=l

m=l

KKr
and

Var{y'nm) = Var I ] Γ Zni I =

σ2qwl

σ2qw2n

+

so that, by assumptions (C4)(ii), k

Var{y'nm) < C^qk

= C (&)

(nwn) -+ 0;

(3.15)

m=l

this is so because nwn = 0(1) and ^ ->• 0, which is implied by ^ ->• 1 (see Comment (b) after Assumptions (C)). Relations (3.14) - (3.15) and Corollary 3.1(ii) show that S{T'n)2 -> 0. Next, 2

= Var

Cov{ZnUZnj)\, i=k(p+q)+l

(3.16)

REGRESSION UNDER ASSOCIATION

83

and

Var{Zni) < i=k(p+q) + l 2 2/

\

2

σ

= -^rV ( 1 + - ) < C^-p σ σ n \ P/ n

(since - -^ 0, by (Comment(b)) P

< Cwnp = C- -> 0 (by (C4)(i) and (C5)(i)).

(3.17)

n

Thus, relation (3.17) implies that Var(Zni)

-+ 0.

(3.18)

i=k(p+q)+l

Finally, Cov(ZnUZnj)

2

=5

n-k(p+q)-l

Σ {l"-

< C*ή-(p + q) = C^fp

< c?- -+ 0 n

(1 + ^

(by (3.17)).

< C ^ P (by Comment (b))

(3.19)

Relations (3.16) - (3.19) show that £(T%)2 -> 0. The proof of the proposition is completed. •

84

ROUSSAS

For the formulation of the second lemma, introduce the following notation. Let Ynm, m = l,...,fc be independent r.v.s with Ynm having the distribution of ynm, set si = Σm=iVar(Ynm), and let Xnm = ^ with distribution function Fnm, m = 1,..., k. Then the r.v.s Xnm, rn — 1,..., k are independent with 8Xnm = 0 and Σm=i Var(Xnm) = 1. Finally, for ε > 0, set *2dFnm(x).

(3-20)

Then we have Lemma 3.2. Let T n and gn(ε) be given by (3.4) and (3.20), respectively, and recall that si = Σ,km=ι Var(Ynm). Then: (i) εTl -> 1, (ii) si -> 1, and (iii) gn{ε) -+ 0. Proof, (i) From (3.5), £5;; = 1, whereas from (3.6),

= εs2n + ε{τ'n + τχ)2 - 2ε [sn{τ'n + rj)] = l + ε{τ'n + τχ)2 - 2ε [sn(τ^ + τ;')]. But by Proposition 3.1,

εHr^ + TZ)2 < εHTtf + εHTZ)2 -> o,

(3.21)

and

2

2

< (εϊsήεh (τ'n + τή = ε\ (τn + τή -+ o (by (3.21)). Thus,

ετl -41. (ii) Prom (3.4) again,

J] m=l

m=l

), Kl
Cov(ynl,ynr) (3.22)

85

REGRESSION UNDER ASSOCIATION and Σι<ι 1.

-* 0 by Corollary 3.1(i). Then relation (3.22)

(iii) We have

x2dFnm(x)

J

2

= ε [x2nml{\Xnm\ 2s

> ε) ] = s~2ε [y2nml (\ynm\ > εsn) ]

1

< s~ £s \ynm\\2s P * (\ynm\ > εsn) 2s

2

< s~ ε~s \yn

2s

(ε~ s- 2 s ε \yn

(where s, t > 1 with - + - = 1 ) s t \Vn

2s

\

125

(3.23) At this point, take s = ^ Then (3.23) becomes

and t = ^ , so that 2s = 2 + δ = v, and ^ = δ.

x2dFnm(x)

[

J(\x\>ε) However, by assumption (Bl)

ε \ynm\ =ε

y

zni

<ε

ieim

ε

<\-\t s

n

> \zni\

(3.24)

\Vn

< -±t

i

(3.25)

/

\ieim

and

so that i=i \ii\Y
/

l(\x\>ε) J(\x\>ε) Hence, with C = e~ιε \ ξι \u,

and therefore, by (3.24) - (3.25),

x2dFnm(x)<(^ε\ξA^. \ε

J

σ s

nn

(3.26) However, (327)

ROUSSAS

86

by assumption (C4)(ii), part(ii) of the present lemma and assumption (C5)(ii), and

7*=εlΫwniξA

2

2

=σ γw ni + 2 Y

, ξj)

i=l

= Cnw*, i=\

so that ^ <

(3.28)

Then, by (3.26) - (3.28), and assumption (C4(i)),

n2

(3.29)

However, the right-hand side in (3.29) tends to 0 on account of the choice of p in assumption (C5)(i); namely, p — o(n2<-1+s'>) or p1+δ = o{nϊ). The proof of the lemma is completed. •

Lemma 3.3. With the ynms denned by (3.3) and for any t € R, it holds

(3.30) m=l

Proof. Clearly,

REGRESSION UNDER ASSOCIATION

87

k-1

H Σ J/nm m=l _

£e

It Σ !/n m=l

e

m=l

m=l ife-i

Σ

«*

Σ 1/nm

m=l

_ ^

k-1

k-1

it Σ Σ ynn m

m=l

e

=l

_

m=l

k-1

£e

it Σ ynrn

m=l

k-l

m=l

and, by a repetition of the argument, inequality (3.30) becomes k-1

it Σ ynrn

β m=l

it Σ 2

γ Ύ

g

_ II ,

τn=l

m=l fc-2 it Σ 2M

(3.31) At this point, use relations (3.1) and (3.3) to define the functions fm{xu i € Im) = elt(Tn ΣieinVniXi χ

?m

= i 5 . . . 5 fc? σ

1

and observe that, for each i G / m , farfm( ii i ^ Λn) ^ |^ n ^ i It follows that, for each i E / m , m = 1,..., i, and I = 1,..., fc,

^ 7 11/m(^<,ie/m) 1

m=l

(3.32)

ROUSSAS Therefore, applying Lemma 1 in Bulinski(1996) to each one of the terms on the right-hand side of (3.31), we obtain by way of (3.32)

it Σ Vnm m=l -4-

71

jeh leh

>

7 \UθV

jG(/l+/2)^/3

+•••+

Σ

By utilizing stationarity of the &s and repeating the arguments used in relations (4.3) - (4.5) in Roussas(2000), inequality (3.33) becomes

m=l

{ζuξj)\ 0,

j=q

(3.34)

by assumptions (C4), (C5)(ii) and (C2). The proof of the lemma is completed. • Proof of Proposition 3.2. Lemma 3.2(ii) implies that Σm=i Ynmd^N(0,1) by the Feller-Lindeberg Criterion (see, for example, Loeve (1963), page 280). This fact along with (3.34) yield the result Σm=i ynmd^N(0,1) or Tnd^N(0,1), as the proposition asserts. •

Acknowledgements This work was supported in part by a research grant from the University of California, Davis. The author is indebted to the referees and one of the editors whose thoughtful comments led to the removal of an inconsistency in the assumptions and helped improve a previous version of this paper.

REGRESSION UNDER ASSOCIATION

89

References Bozorgnia, A., Patterson, R.F., and Taylor, R.L. (1996). Limit theorems for negatively dependent random variables. Proceedings of the First World Congress of Nonlinear Analysis, Tampa, Florida, August 19 26, 1992. Bulinski, A.V. (1996). On the convergence rates in the CLT for positively and negatively dependent random fields. In Probability Theory and Mathematical Statistics (LA. Ibragimov and A.Yu. Zaitsev, Eds), pp. 3-14. Gordon and Breach Publishers, Amsterdam. Cai, Z., and Roussas, G.G.(1999a). Weak convergence for smooth estimator of a distribution function under negative association. Stochastic Analysis and Applications 17, 245 - 268. Cai, Z., and Roussas, G.G.(1999b). Berry-Essen bounds for a smooth estimator of a distribution function under association. Journal of Nonparametric Statistics 11, 79 - 106. Gasser, T., and Mύller, H. G. (1979). Kernel estimation of regression function. In Smoothing Techniques for Curve Estimation (T. Gasser and M. Rosenblatt, Eds.) Lecture Notes in Mathematics, Vol. 757, pp. 23-68. Springer-Verlag, Berlin - New York. Georgiev, A. A. (1988). Consistent nonparametric multiple regression: The fixed design case. Journal of Multiυariate Analysis 25, 100 - 110. Georgiev, A. A., and Greblicki, W. (1986). Nonparametric function recovering from noisy observations. Journal of Statistical Planning and Inference 13, 1 - 14. Loeve, M. (1963). Probability Theory, 3rd ed. Van Nostrand, Princeton, NJ. Newman, C M . , (1980). Normal fluctuations and the FKG inequalities. Communications in Mathematical Physics 75, 119 - 128. Patterson, R.F., and Taylor, R.L. (1997). Strong laws of large numbers for negatively dependent random elements. Nonlinear Analysis, Theory, Methods and Applications 30, 4229 - 4235. Priestley, M. B., and Chao, M. T. (1972). Nonparametric function fitting. Journal of the Royal Statistical Society Ser. B 34, 385 - 392.

90

ROUSSAS

Roussas, G.G. (1989). Consistent regression estimation with fixed design points under dependent conditions. Statistics and Probability Letters 8, 41 - 50. Roussas, G.G., Tran, L. T., and Ioannides, D. A. (1992). Fixed design regression for time series: Asymptotic normality. Journal of Multivariate Analysis 40, 262 - 291. Roussas, G.G. (1994). Asymptotic normality of random fields of positively or negatively associated processes. Journal of Multivariate Analysis 50, 152 - 173. Roussas, G.G. (1999). Positive and negative dependence with some statistical applications. In Asymptotics, Nonparametrics and Time Series (S. Ghosh, Ed.), pp. 757 - 788. Marcell Dekker, Inc., New York. Roussas, G.G. (2000). Asymptotic normality of the kernel estimate of a probability density function. Statistics and Probability Letters 50, 112. Taylor, R.L., and Patterson, R.F. (1997). Negative dependence in Banach spaces and laws of large numbers. Nonlinear Analysis, Theory, Methods and Applications 30, 4249 - 4256. Taylor, R.L., Patterson, R.F., and Bozorgnia, A. (1999a). A strong law of large numbers for arrays of rowwise negatively dependent random variables. Technical Report Number 9 9 - 1 , Department of Statistics, University of Georgia, Athens, Georgia. Also, to appear in Journal of Stochastic Analysis and Applications. Taylor, R.L., Patterson, R.F., and Bozorgnia, A. (1999b). Weak laws of large numbers for arrays of rowwise negatively dependent random variables. Technical Report Number 99 - 10, Department of Statistics, University of Georgia, Athens, Georgia. Also, to appear in the Journal of Applied Mathematics and Stochastic Analysis.

91

Institute of Mathematical Statistics LECTURE NOTES — MONOGRAPH SERIES

DEPENDENT BOOTSTRAP CONFIDENCE INTERVALS Wendy D. Smith and Robert L. Taylor Department of Statistics The University of Georgia Athens, GA 30602-1952 Abstract A dependent bootstrap is shown to produce estimators which have smaller variances but which are still consistent and asymptotically valid. Simulated confidence intervals are used to examine possible gains in coverage probabilities and interval lengths.

1

Introduction

Inference for stochastic processes seeks to provide more appropriate dependent models to situations where the assumption of independence is not plausible. The use of independent models is prevalent in bootstrapping. Efron (1979) introduced the bootstrap as a tool to estimate the standard error of a statistic, and an enormous amount of applied and theoretical research on the bootstrap technique has followed in the past two decades. While much of this ensuing research has been methodological adaptions and theoretical validity verifications for different statistics, considerable research has been directed toward shortcomings and possible improvements to the basic bootstrap technique. The traditional resampling of the sample observations (with replacement) produces independent and identically distributed (bootstrap) random variables (conditional on the original sample), and many of the theoretical justifications of the bootstrap procedures are crucially related to techniques involving independent random variables. Resampling without replacement produces dependent random variables (actually negatively dependent) which are still identically distributed (and in fact has the desirable property of exchangeability). The purpose of this paper is to consider some estimation using a form of dependent bootstrapping. In particular, confidence interval comparisons will be given for the traditional bootstrap procedure and the dependent bootstrap procedure.

92

SMITH AND TAYLOR

Resampling without replacement is not new. The majority of research on resampling without replacement has been for application in finite population sampling. Gross (1980) introduced the concept and many others (Bickel and Freedman, 1984; Chao and Lo, 1985; Sitter, 1992; Booth,Butler and Hall, 1994; and others) have extended this research. In 1994 Politis and Romano examined resampling without replacement from a data set to approximate the sampling distribution of a statistic Tn. Under weak assumptions, they showed that the empirical distribution of the suitably normalized values of the statistic computed for all subsamples of size b from the original data is first order asymptotically valid for the true sampling distribution of Tn. This is a generalization from Wu (1990) who studied the same method in the i.i.d case for statistics which are asymptotically normal. Bertail (1997) showed second order correctness of this method for an adequately chosen resample size. Their investigations differ from this proposed research because they sample without replacement from the original data rather than an enriched collection with a fixed number of copies of each observation. Their procedure has been termed "ra out of n" where m(« n) is the bootstrap sample size and n is the sample size of the original sample. Bickel, Gόtze, and Van Zwet (1997) investigated the gains and losses for "ra out of n" resampling where ra = o(ή). Praestgaard and Wellner (1993) showed that "ra out of fcn" could allow larger bootstrap sample sizes and some asymptotic results using exchangeability arguments. Babu and Singh (1985) and Babu and Bai (1996) showed that Edgeworth expansions could be used to obtain approximation results for estimators based on samples drawn without replacement from a finite population. Their approximation results provide for weak convergence of normalized absolute differences of original sample statistics and bootstrap statistics. This paper will compare the coverage probabilities and lengths of the more generally used bootstrap confidence intervals for the traditional bootstrap and the dependent bootstrap. The formal definition of the dependent bootstrap procedure and the theoretical properties of consistency and asymptotic validity are listed in Section 2. Section 3 provides the description and results of the simulations for the confidence intervals.

2

Properties of the Dependent Bootstrap

Consider the random sample observations X\, X2,..., Xn, that is, identically distributed random variables with distribution function F. Often the random variables X\yX2,... ,Xn are also assumed to be independent, but may be dependent as when sampling from a finite population. A dependent boot-

BOOTSTRAP

93

CΓS

strap is defined as the sample of size ra, denoted by X^i, iXnmi drawn without replacement from the collection of kn items made up of k copies each of the sample observations, Xι,..., JΓnj where m < kn. This dependent bootstrap is proposed as a procedure to reduce variation of estimators and to obtain better confidence intervals. Let £?*, Var*, and P* denote the conditional expectation, variance, and probability given Xi, X2,..., Xn- It can be shown that (2.1)

E*X*nm = Xn, Var*(X*nj)

j = l,...,m,αnd

= Sl

j

j

n

(2.2)

— - 2 ^ i = i ( A ι - Λn)

.

Notice that the multiplier in the variance of the dependent bootstrap sample mean, ^irf, is similar to the finite population correction factor. Random variables X and Y are said to be negatively dependent (ND) if P[X <x,Y

P[X < x]P[Y < y]

(2.4)

for all α;, y G R. Negative dependence includes independence, and the terminology of negative relates to (2.4) which (by dividing by P[Y < y]) implies P[X < x\Y
P[X < x]

Examples and properties of negative dependent random variables and the applications to the dependent bootstrap are given by Smith and Taylor (2001). In particular, they showed that the dependent bootstrap produces negatively dependent random variables. A collection of random variables {X\, X2,.. , Xn} is said to be exchangeable if the joint distribution of (Xi,X27 ,Xn) is invariant with respect to permutations of the indices 1,... ,n. It can be shown (cf: Smith and Taylor (2001)) that the dependent bootstrap random a r e variables, X* l 5 X* 2 ,..., Xnπn exchangeable in addition to being negative dependent. Theorem 2.1 gives the consistency of the bootstrap mean for this dependent bootstrap procedure established by Smith and Taylor (2001), while Theorem 2.2 establishes the consistency of the dependent bootstrap variance. The technique of the proof for Theorem 2.1 follows similar techniques for obtaining consistency in the i.i.d. bootstrap given by Hu and Taylor (1997) and Bozorgnia, Patterson and Taylor (1997). Moreover, it is important to observe that required moment conditions in Theorem 2.1 are identical to the traditional i.i.d. bootstrapping procedure (cf: Athreya, Ghosh, Low and Sen (1984)).

94

SMITH AND TAYLOR

Theorem 2.1 Let Xi,X2 5 ,Xn, - be i.i.d. random variables with mean ι+δ μ and E\X\\ < oo for some δ > 0. Along almost all sample sequences X\,X<ι,... given (X\,...,Xn), X^m ~^ f1 wϊth conditional probability 1.

(2.5)

Theorem 2.2 Let Xi, X2, . . , Xγn be i.i.d. random variables with mean μ, variance σ2 and E\Xι\2+δ < 00 for some δ > 0. Along almost all sample sequences Xχ5 X2? given (X\,..., Xn), Snm ~*σ<1 with conditional probability 1, nj ~

(2.6)

Λ

nm)

Theorems 2.3 and 2.4 are also from Smith and Taylor (2001) and provide the asymptotic validity for the dependent Kolmogorov-Smirnov bootstrap statistic. Theorem 2.3 Let Xi,X25 ?-^n? be i.i.d. random variables with distribution function F. Along almost all sample sequences Xi,X2 5 given (Xι,... ,Xn), Fm(x) ~^ F(x) where F*(x) =

ith conditional probability 1,

(2.7)

±Σ7=iI[x:j<*]-

Theorem 2.4 Let X\,X2,... tribution function F. Then, D*m =

w

sup

^Xn-,

be i.i.d. random variables with dis-

\F^{x) - F(x)\ -> 0 with conditional probability 1. (2.8)

—oo<x
Theorems 2.1 - 2.4 were obtained using negative dependent limit theorems and are valid for all k and for all m such that liminf n ^ > 0. Finite population versions of these results were also obtained (cf: Smith and Taylor (2001)). For the validity (asymptotic normality) for the dependent bootstrap a stronger (more restrictive) form of negative dependence is needed, namely negative association. Random variables {Xn} are said to be negatively associated (NA) if for each k > 2 and every pair of disjoint subsets Ai, A2 of {1,2,..., k} Cσυ(f(Xi,i

€ AxUiXjJ

e A2)) < 0

(2.9)

whenever / and g are monotone increasing, Borel functions. Using a combination of exchangeable and negative association results, Patterson, Smith, Taylor, and Bozorgnia (2001) obtained the following central limit thoerem for the dependent bootstrap.

BOOTSTRAP

95

CΓS

Theorem 2.5 If {Xn} are i.i.d. random variables such that E\Xι\2+δ < oo for some δ > 0, then conditionally on almost all sample paths

y/nsn

(2.10)

converges in distribution

to a iV(0,1) random variable where sn = It is important to observe that the normalizing factor sn in (2.10) includes A;, and that the result holds for all choices of k = k(n) > 2 as n -» oo. Moreover, Theorem 2.5 can be extended to include sample observations from finite populations and bootstrap sample sizes m = m(n) such that 0 < inf '¥- < sup - ^ < 1. n kn n kn

(2.11)

In addition, the asymptotic validity of the dependent bootstrap distribution function estimator F^(x) follows from these techniques. This result is stated for i.i.d. random variables but is also obtainable when sampling from finite populations. m m

n

Theorem 2.6 Let Fn(x) = £ g I[Xi<x] and F^(x) = i g I[X'ni<χ], let (2.11) hold, then along almost all samle sequences XijX^ given {Xι,... ,Xn) F* (x) — F 1(x) —^ TT^ converges in distribution to a N(0,l) random variable, sn{x) where s2n(x) = Var*(F^(x)) = i ^ F n ( a : ) ( l -

3

Fn(x)).

Simulations

In this section the simulation results are given to compare the coverage probabilities and average lengths of confidence intervals calculated using two methods of calculating CΓs for both traditional and dependent bootstrap methods. Also included in these simulation is the computation of the traditional normal theory confidence intervals for comparison, where the limits are computed using

Xn±zA. 2 vn

(3.1)

While there are many methods of computing bootstrap confidence intervals (cf: DiCiccio and Efron (1996)), these simulations employed the general

96

SMITH AND TAYLOR

methods of percentile and bootstrap-t. Percentile method confidence limits are [Λnm

i^nm

J,

Δ

\° )

where Xnm is the B pth ordered value in the list of the B bootstrap sample means, Xnm , b = 1,..., B. For the bootstrap-t method the limits are given by [Xn-z^

2)—,χn-z^2)-7=\ y/n \Jn

(3.3)

where z^' is the B ^th ordered value in the list of the B standardized bootstrap sample means, Z*^ — *n™^bfn, b = l,...,i? and se*^ is the estimated standard error of X*(b\

When resampling using the traditional

bootstrap method, &\ ' — - ^ , where Sζ is the sample standard deviation of the 6th bootstrap sample, while for the dependent bootstrap, se^ = ^ly mtkn-m) a s P r o v i d e d in Theorem 2.5. For these simulations 1500 samples of size n = 20,40,100,200 were generated from the following distributions, each having mean 4 and variance 8: normal (μ = 4, σ 2 = 8), χ2 with 4 degrees of freedom, double exponential with μ = 4 and σ = 2, and mixture of two normal distributions: 80 % 7^(4.8,6.45) and 20 % #(0.8,1.4). For each original sample, the traditional normal theory 90% confidence interval was calculated and the coverage probability and length of the 1500 intervals was computed. The 90% confidence level was chosen (rather than the usual 95%) so that there would be sufficient number of confidence intervals not including the mean of 4 for interesting comparisons. Next, for each original sample the dependent bootstrap 90% CI was formed by drawing 2000 dependent bootstrap samples of size m = n from each of the original samples for varying replication factors, fc, specifically, k = 2,4,6,8,10,20. Using the 2000 bootstrap samples, the percentile and bootstrap-t confidence intervals were obtained. The same procedure was followed using the traditional bootstrap. The estimated coverage probabilities and average lengths are reported in Tables 1-4. The results show that for all distributions there is little difference between the coverage probabilities and the lengths of the normal theory CI, the traditional bootstrap and the dependent bootstrap procedure, when the bootstrap-t method of CI computation is used. However, for the percentile method of confidence interval computation, the dependent bootstrap performs poorly when compared to the traditional bootstrap and the normal theory method (cf: Tables 5-8). For all distributions, the dependent bootstrap confidence intervals displayed far lower coverage probabilities than the

BOOTSTRAP

97

CΓS

other methods while yielding much shorter lengths. This result is to be expected since the variance of the dependent bootstrap mean estimator (cf: (2.3)) is smaller than that of the sample mean or the traditional bootstrap mean, and hence the distribution of the dependent bootstrap mean is narrower. Thus, an adjustment to the percentile method is needed for the dependent bootstrap confidence intervals to achieve desired coverage probabilities while trying to maintain shorter lengths. Specifically, since the percentiles are functionally related to the standard deviation, instead of using (3.2) in obtaining the confidence interval limits, use

where p* satisfies 1 — Φ(Z(Q ^)) = *£• with

for 0 < θ < 1 and where Zs. is a standard normal (1 — §) percentile. Using a = .1, this becomes

Note that θ — 0 is the usual percentile method described above whereas for 0 < θ < 1 the attempt is to improve the coverage probabilies of the confidence intervals while maintaining shorter lengths. Using the same distributions and sampling plan as before, these new simulations were carried out with results for θ — 0,0.5., 0.75,1 given in Tables 9-12. For all distributions, even for moderate k (k = 4,6,8) and θ = 0.75, the dependent bootstrap confidence intervals achieve comparable coverage probabilities while retaining shorter lengths. The simulation studies presented in this paper suggest some directions for future research. While θ = 0.75 appears to be the best choice from these simulations, more investigations might determine if θ is dependent on fc, n, or the distribution the original sample was drawn from, or some combination of these factors.

References Athreya, K. B., Ghosh, M., Low, L. Y. and Sen, P. K. (1984). Laws of large numbers for bootstrapped U-statistics. J. Statist. Planning and Inf., 9, 185-194.

98

SMITH AND TAYLOR

Babu, G.J. and Bai, Z.D. (1996). Mixtures of global and local Edgeworth expansions and their applications. J. Multivariate Anal, 59, 282-307. Babu, G.J. and Singh, K. (1985). Edgeworth expansions for sampling without replacement from finite populations. J. Multivariate Anal, 17, 261-278. Bertail, P. (1997). Second-order properties of an extrapolated bootstrap without replacement under weak assumptions. Bernoulli, 3, 149-179. Bickel, P.J., Gόtze, R. and van Zwet, W.R. (1997). Resampling fewer than n observations: gains, losses, and remedies for losses. Statistica Sinica, 7, 1-31. Bickel, P.J. and Freedman, D.A. (1984). Asymptotic normality and the bootstrap in stratified sampling. Ann. Stat, 12, 470-482. Booth, J.G., Butler, R.W. and Hall, P. (1994). Bootstrap methods for finite populations. J. Amer. Stat. Assoc, 89, 1282-1289. Bozorgnia, A., Patterson, R. F. and Taylor, R. L. (1997). Chung type strong laws for arrays of random elements and bootstrapping. Stock. Anal, and AppL, 15, 651-669. Chao, M.T. and Lo, S.H. (1985). A bootstrap method for finite populations. Sankaya, Series A., 47, 399-405. DiCiccio, T. J. and Efron, B. (1996). Bootstrap confidence intervals. Statist. Scl, 11, 189-228. Efron, B. (1979). Bootstrap methods: Another look at the jackknife. Ann. Stat, 7, 1-26. Gross, S. (1980). Median estimation in sample surveys. Proc. Sect. Survey Res. Methods, Amer. Stat. Assoc, 181-184. Hu, T. -C. and Taylor, R. L. (1997). On the strong law for arrays and the bootstrap mean and variance. Intern. J. Math, and Math. Sci., 20, 375-382. Patterson, R. F., Smith, W. D., Taylor, R. L. and Bozorgnia, A. (2001). Limit theorems for negatively dependent random variables. NonLinear Analysis, 47, 1283-1295. Politis, D.N. and Romano, J.P. (1994). Large sample confidence regions based on subsamples under minimal assumptions. Ann. Stat., 22, 2031-2050. Praestgaard, J. and Wellner, J. (1993). Exchangeably weighted bootstraps of the general empirical process. Ann. Prob., 21, 2053-2086. Singh, K. (1998). Breakdown theory for bootstrap quantiles. Ann.

Stat,

BOOTSTRAP CΓS

99

26, 1719-1732. Sitter, R.R. (1992). Comparing three bootstrap methods for survey data. Canad. J. Stat, 20, 135-154. Smith, W. and Taylor, R.L. (2001) Consistency of dependent bootstrap estimators. Amer. J. Math, and Management Sciences, 21. Wu, C.F.J. (1990). On the asymptotic properties of the jackknife historgram. Ann. Stat, 18, 1438-1452.

SMITH AND TAYLOR

100

Table 1: 90% Bootstrap-1 Confidence Intervals for Normal(μ = 4, σ 2 = 8)

n 20 40 100 200

Normal Approx. 0.877 0.907 0.892 0.912

Traditional Bootstrap 0.901 0.911 0.895 0.912

Coverage Probabilities Dependent Bootstrap k = 2 Jfe = 4 k = 6 fc = 8 fc = 10 fc = 12 0.897 0.892 0.898 0.896 0.899 0.899 0.914 0.912 0.913 0.918 0.914 0.915 0.891 0.894 0.890 0.890 0.897 0.891 0.914 0.913 0.912 0.911 0.917 0.916

n 20 40 100 200

Normal Approx. 2.049 1.463 0.930 0.658

Traditional Bootstrap 2.181 1.503 0.940 0.661

Interval Lengths Dependent Bootstrap k = 2 fc = 4 k = 6 k = 8 As = 10 2.137 2.157 2.164 2.168 2.172 1.491 1.497 1.499 1.501 1.502 0.938 0.939 0.939 0.939 0.939 0.661 0.661 0.661 0.661 0.661

k = 12 2.173 1.501 0.940 0.661

Table 2: B o o t s t r a p - t m e t h o d 90% Confidence Intervals for C h i Square(d/ = 4)

n 20 40 100 200

Normal Approx. 0.867 0.863 0.901 0.893

Traditional Bootstrap 0.852 0.883 0.911 0.895

Coverage Probabilities Dependent Bootstrap k = 2 k = A k = 6 k=8 As = 10 0.895 0.887 0.886 0.891 0.891 0.883 0.881 0.885 0.886 0.881 0.909 0.911 0.911 0.910 0.913 0.893 0.889 0.892 0.895 0.893

n 20 40 100 200

Normal Approx. 2.028 1.449 0.925 0.654

Traditional Bootstrap 2.295 1.538 0.949 0.663

Interval Lengths Dependent Bootstrap k = 2 k = A k=6 k=8 A; = 10 2.234 2.262 2.274 2.277 2.282 1.521 1.530 1.534 1.534 1.536 0.944 0.946 0.947 0.947 0.947 0.661 0.662 0.663 0.662 0.663

A; = 12 0.889 0.883 0.909 0.897

k = 12 2.283 1.536 0.948 0.663

BOOTSTRAP CΓS

101

Table 3: Bootstrap-t method 90% Confidence Intervals for Double Exponential(μ = 4, σ = 2)

n 20 40 100 200

Normal Approx. 0.877 0.894 0.905 0.902

Traditional Bootstrap 0.865 0.887 0.899 0.896

Coverage Probabilities Dependent Bootstrap A; = 2 A: = 4 A; = 6 A; = 8 A; = 10 0.865 0.863 0.863 0.867 0.867 0.891 0.894 0.889 0.889 0.891 0.897 0.902 0.902 0.901 0.899 0.901 0.897 0.898 0.896 0.897

n 20 40 100 200

Normal Approx. 2.008 1.458 0.927 0.656

Traditional Bootstrap 1.954 1.438 0.922 0.654

Interval Lengths Dependent Bootstrap k = 2 A; = 4 A; = 6 A; = 8 A; = 10 2.184 2.187 2.191 2.193 2.193 1.517 1.515 1.514 1.514 1.515 0.941 0.940 0.941 0.939 0.940 0.662 0.661 0.661 0.661 0.661

A; = 12 0.863 0.888 0.900 0.899

A; = 12 2.195 1.516 0.940 0.661

Table 4: Bootstrap-t method 90% Confidence Intervals for mixture of two normal distributions

n 20 40 100 200

Normal Approx. 0.885 0.888 0.901 0.903

Traditional Bootstrap 0.905 0.902 0.907 0.902

n 20 40 100 200

Normal Approx. 2.069 1.469 0.929 0.656

Traditional Bootstrap 2.197 1.509 0.940 0.660

Coverage Probabilities Dependent Bootstrap A; = 2 A: = 4 A; = 6 A; = 8 A; = 10 0.895 0.903 0.902 0.904 0.904 0.896 0.897 0.899 0.901 0.903 0.905 0.906 0.906 0.907 0.905 0.904 0.905 0.905 0.904 0.905 Interval Lengths Dependent Bootstrap A; = 2 A; = 4 A: = 6 A; = 8 A; = 10 2.185 2.145 2.168 2.179 2.183 1.506 1.491 1.501 1.504 1.506 0.939 0.936 0.938 0.938 0.940 0.660 0.659 0.659 0.660 0.659

A; = 12 0.905 0.902 0.907 0.902

A; = 12 2.188 1.507 0.939 0.660

SMITH AND TAYLOR

102

n 20 40 100 200

Table 5: Percentile 90% Confidence Intervals for Normal(μ = 4, σ 2 = 8) Coverage Probabilities Dependent Bootstrap Normal Traditional Approx. Bootstrap k = 2 A; = 4 A; = 6 k=8 k = 10 0.849 0.728 0.814 0.830 0.840 0.877 0.869 0.886 0.907 0.767 0.855 0.879 0.887 0.901 0.867 0.892 0.892 0.775 0.835 0.853 0.865 0.912 0.907 0.893 0.783 0.861 0.875 0.889

A; = 12 0.852 0.893 0.876 0.893

n 20 40 100 200

Interval Lengths Dependent Bootstrap A; = 2 A; = 4 A; = 6 A; = 8 A; = 10 1.433 1.743 1.834 1.877 1.903 1.374 1.029 1.256 1.323 1.355 0.657 0.803 0.846 0.867 0.878 0.465 0.569 0.600 0.614 0.623

k = 12 1.920 1.386 0.888 0.629

Normal Approx. 2.049 1.463 0.930 0.658

Traditional Bootstrap 1.997 1.443 0.925 0.656

Table 6: Percentile method 90% Confidence Intervals for Chi Square(cίf = 4) Coverage Probabilities Normal Traditional Dependent Bootstrap n Approx. Bootstrap A: = 2 A; = 4 A; = 6 A; = 8 A; = 10 A; = 12 20 0.867 0.852 0.726 0.806 0.824 0.833 0.840 0.840 40 0.863 0.852 0.859 0.735 0.813 0.830 0.841 0.845 100 0.901 0.903 0.763 0.850 0.867 0.878 0.883 0.885 200 0.893 0.887 0.732 0.833 0.852 0.864 0.868 0.875

n 20 40 100 200

Normal Approx. 2.028 1.449 0.925 0.654

Traditional Bootstrap 1.973 1.428 0.920 0.653

Interval Lengths Dependent Bootstrap A; = 2 A; = 4 A; = 6 A; = 8 A; = 10 1.423 1.725 1.813 1.855 1.881 1.021 1.244 1.311 1.341 1.362 0.653 0.799 0.842 0.862 0.874 0.463 0.566 0.597 0.613 0.620

A; = 12 1.895 1.373 0.882 0.626

BOOTSTRAP CΓS

103

Table 7: Percentile method 90% Confidence Intervals for Double Exponential(μ = 4, σ = 2)

n 20 40 100 200

Normal Approx. 0.877 0.894 0.905 0.902

Traditional Bootstrap 0.859 0.885 0.900 0.897

Coverage Probabilities Dependent Bootstrap k = 2 k = A k=6 k=8 k = 10 0.721 0.802 0.822 0.833 0.837 0.738 0.833 0.855 0.866 0.869 0.754 0.839 0.855 0.870 0.876 0.755 0.849 0.869 0.876 0.881

A; = 12 0.840 0.874 0.878 0.885

n 20 40 100 200

Normal Approx. 2.008 1.458 0.927 0.656

Traditional Bootstrap 1.954 1.438 0.922 0.654

Interval Lengths Dependent Bootstrap k = 2 k = 4 fc = 6 k = 8 k = 10 1.408 1.708 1.797 1.838 1.862 1.027 1.253 1.318 1.349 1.370 0.655 0.801 0.845 0.863 0.876 0.464 0.568 0.598 0.613 0.622

fc = 12 1.879 1.382 0.884 0.627

Table 8: Percentile method 90% Confidence Intervals for Mixture of Two Normal Distributions n 20 40 100 200

Normal Approx. 0.885 0.888 0.901 0.903

Traditional Bootstrap 0.875 0.885 0.901 0.900

Coverage Probabilities Dependent Bootstrap k = 2 k = 4 fc = 6 A; = 8 k = 10 0.854 0.733 0.812 0.838 0.847 0.866 0.749 0.837 0.854 0.862 0.877 0.759 0.849 0.868 0.875 0.884 0.759 0.844 0.865 0.879

k = 12 0.856 0.873 0.886 0.885

n 20 40 100 200

Normal Approx. 2.069 1.469 0.929 0.656

Traditional Bootstrap 2.018 1.451 0.925 0.654

Interval Lengths Dependent Bootstrap k = 2 fc = 4 λ; = 6 λ; = 8 Jfc = 1 0 1.448 1.759 1.852 1.895 1.921 1.262 1.361 1.379 1.032 1.330 0.879 0.656 0.803 0.846 0.868 0.622 0.464 0.568 0.599 0.613

A; = 12 1.938 1.393 0.886 0.628

SMITH AND TAYLOR

104

Table 9: Modified Percentile 90% Confidence Intervals: 2 Normal(μ = 4, σ = 8)

θ 0

.5

0.75

1

θ 0

0.5

0.75

1

Coverage Probabilities Dependent Bootstrap Traditional Bootstrap k = 2 fc = 4 k = 6 k = S 0.892 0.898 0.896 0.899 0.901 0.914 0.915 0.912 0.913 0.911 0.894 0.890 0.890 0.897 0.895 0.912 0.914 0.913 0.912 L0.911

k = 10 0.899 0.914 0.891 0.917

n 20 40 100 200

Normal Approx. 0.877 0.907 0.892 0.912

20 40 100 200

0.877 0.907 0.892 0.912

0.869 0.901 0.892 0.907

0.801 0.853 0.830 L0.858

0.842 0.883 0.866 0.887

0.855 0.893 0.878 0.896

0.858 0.894 0.877 0.904

0.861 0.895 0.880 0.903

20 40 100 200

0.877 0.907 0.892 0.912

0.869 0.901 0.892 0.907

0.855 0.892 0.876 0.904

0.865 0.898 0.881 0.905

0.863 0.899 0.883 0.907

0.865 0.901 0.883 0.909

20 40 100 200

0.877 0.907 0.892 0.912

0.869 0.901 0.892 0.907

0.839 0.881 0.863 0.887 0.864 0.901 0.890 0.913

0.869 0.901 0.889 0.912

0.873 0.903 0.891 0.910

0.868 0.901 0.891 0.915

0.869 0.902 0.887 0.911

n 20 40 100 200

Normal Approx. 2.049 1.463 0.930 0.658

Traditional Bootstrap 2.181 1.503 0.940 0.661

20 40 100 200

2.049 1.463 0.930 0.658

1.997 1.443 0.925 0.656

1.713 1.240 0.792 0.562

7.874 1.356 0.867 0.614

1.919 1.385 0.889 0.630

1.938 1.400 0.899 0.637

fc = 10 2.172 1.502 0.939 0.661 1.954 1.411 0.903 0.640

20 40 100 200

2.049 1.463 0.930 0.658

1.997 1.443 0.925 0.656

1.849 1.343 0.861 0.610

1.939 1.396 0.897 0.636

1.963 1.416 0.907 0.643

1.968 1.423 0.913 0.647

1.980 1.428 0.915 0.648

20 40 100 200

2.049 1.463 0.930 0.658

1.997 1.443 0.925 0.656

1.982 1.448 0.927 0.658

1.997 1.447 0.929 0.659

2.003 1.446 0.929 0.659

2.002 1.446 0.929 0.659

2.004 1.448 0.926 0.657

Interval Lengths Dependent Bootstrap k = 2 A; = 4 λ; = 6 k = 8 2.137 2.157 2.164 2.168 1.501 1.491 1.497 1.499 0.938 0.939 0.939 0.939 0.661 0.661 0.661 0.661

BOOTSTRAP

105

CΓS

Table 10: Modified Percentile 90% Confidence Intervals: Chi-Square (df = 4)

θ 0

0.5

0.75

1

θ 0

0.5

0.75

1

Coverage Probabilities Traditional Dependent Bootstrap Bootstrap k = 2 fc = 4 fc = 6 fc = 8 0.852 0.726 0.806 0.824 0.833 0.859 0.735 0.813 0.830 0.841 0.903 0.763 0.850 0.867 0.878 0.887 0.732 0.833 0.852 0.864

n

Normal Approx.

20 40 100 200

0.867 0.863 0.901 0.893

20 40 100 200

0.867 0.863 0.901 0.893

0.852 0.859 0.903 0.887

0.798 0.805 0.844 0.823

0.831 0.841 0.879 0.863

0.841 0.848 0.888 0.875

0.841 0.851 0.890 0.877

0.847 0.855 0.893 0.877

20 40 100 200

0.867 0.863 0.901 0.893

0.852 0.859 0.903 0.887

0.827 0.833 0.876 0.862

0.845 0.851 0.893 0.880

0.849 0.853 0.894 0.881

0.850 0.855 0.897 0.887

0.852 0.857 0.899 0.888

20 40 100 200

0.867 0.863 0.901 0.893

0.852 0.859 0.903 0.887

0.849 0.860 0.899 0.890

0.859 0.861 0.901 0.893

0.860 0.863 0.899 0.894

0.857 0.862 0.903 0.888

0.855 0.863 0.902 0.895

n

Normal Approx.

Traditional Bootstrap

20 40 100 200

2.028 1.449 0.925 0.654

1.973 1.428 0.920 0.653

1.423 1.021 0.653 0.463

1.725 1.244 0.799 0.566

1.813 1.311 0.842 0.597

1.855 1.341 0.862 0.613

20 40 100 200

2.028 1.449 0.925 0.654

1.973 1.428 0.920 0.653

1.688 1.225 0.787 0.558

1.850 1.342 0.862 0.611

1.897 1.372 0.884 0.628

1.917 1.386 0.894 0.634

1.881 1.362 0.874 0.620 1.962 1.398 0.898 0.638

20 40 100 200

2.028 1.449 0.925 0.654

1.973 1.428 0.920 0.653

1.819 1.326 0.854 0.607

1.914 1.384 0.893 0.633

1.939 1.403 0.902 0.639

1.947 1.408 0.909 0.644

1.958 1.415 0.909 0.645

20 40 100 200

2.028 1.449 0.925 0.654

1.973 1.428 0.920 0.653

1.947 1.427 0.920 0.654

1.972 1.432 0.924 0.655

1.978 1.432 0.924 0.655

1.980 1.433 0.923 0.655

1.980 1.434 0.921 0.654

Interval Lengths Dependent Bootstrap k = 2 k = A A; = 6 k = 8

fc = 10 0.840 0.845 0.883 0.868

k = 10

SMITH AND TAYLOR

106

Table 11: Modified Percentile 90% Confidence Intervals: = 4,σ=2)

θ 0

0.5

0.75

1

θ 0

0.5

0.75

1

n

Normal Approx.

20 40 100 200 20 40 100 200 20 40 100 200

0.877 0.894 0.905 0.902

Coverage Probabilities Dependent Bootstrap Traditional fc = 2 fc = 4 fc = 6 fc = 8 Bootstrap 0.721 0.802 0.822 0.833 0.859 0.738 0.833 0.855 0.866 0.885 0.754 0.839 0.855 0.870 0.900 0.897 0.755 0.849 0.869 0.876

0.877 0.894 0.905 0.902

0.859 0.885 0.900 0.897

0.877 0.894 0.905 0.902

20 40 100 200

n 20 40 100 200 20 40 100 200

fc = 10 0.837 0.869 0.876 0.881

0.856 0.883 0.893 0.893

0.845 0.879 0.889 0.889 0.853 0.884 0.895 0.896

0.849 0.885 0.889 0.893 0.855 0.885 0.895 0.897

0.861 0.887 0.899 0.899

0.859 0.885 0.901 0.898

0.857 0.885 0.903 0.898

0.830 0.870 0.872 0.877 0.850 0.876 0.889 0.893

0.839 0.877 0.884 0.885

0.859 0.885 0.900 0.897

0.793 0.829 0.833 0.845 0.827 0.867 0.869 0.874

0.877 0.894 0.905 0.902

0.859 0.885 0.900 0.897

0.860 0.890 0.901 0.901

0.861 0.888 0.903 0.899

Normal Approx. 2.008 1.458 0.927 Lθ.656 2.008 1.458 0.927 0.656

Traditional Bootstrap 1.954 1.438 0.922 0.654 1.954 1.438 0.922 0.654

Interval Lengths> Dependent Bootstrap k = 2 fc = 4 fc = 6 fc = 8 k = 10 1.862 1.408 1.708 1.797 1.838 1.027 1.253 1.318 1.349 1.370 0.876 0.655 0.801 0.845 0.863 0.622 0.464 0.568 0.598 0.613 1.834 1.914 1.672 1.880 1.900 1.408 1.233 1.351 1.380 1.396 0.789 0.863 0.886 0.897 0.900 0.639 0.560 0.612 0.629 0.636

20 40 100 200

2.008 1.458 0.927 0.656

1.954 1.438 0.922 0.654

1.801 1.335 0.857 0.608

1.898 1.394 0.895 0.635

1.922 1.412 0.903 0.641

1.928 1.418 0.911 0.647

1.937 1.425 0.912 0.646

20 40 100 200

2.008 1.458 0.927 0.656

1.954 1.438 0.922 0.654

1.925 1.434 0.923 0.656

1.953 1.442 0.926 0.657

1.961 1.443 0.927 0.656

1.961 1.443 0.926 0.657

1.961 1.442 0.923 0.655

BOOTSTRAP CΓS

107

Table 12: Modified Percentile 90% Confidence Intervals: Mixture of Two Normals

Coverage Probabilities θ 0

0.5

0.75

1

θ 0

0.5

0.75

1

n

Normal Approx.

Traditional Bootstrap

k =2

20 40 100 200

0.885 0.888 0.901 0.903

0.875 0.885 0.901 0.900

0.733 0.749 0.759 Lθ.759

20 40 100 200

0.885 0.888 0.901 0.903

0.875 0.885 0.901 0.900

0.803 0.831 0.840 0.843

0.847 0.861 0.875 0.877

0.862 0.873 0.883 0.886

0.863 0.875 0.889 0.894

20 40 100 200

0.885 0.888 0.901 0.903

0.875 0.885 0.901 0.900

0.843 0.856 0.868 0.871

0.857 0.875 0.885 0.893

0.871 0.879 0.891 0.897

0.871 0.877 0.898 0.895

0.896 0.880 0.889 0.891 0.872 0.881 0.893 0.899

20 40 100 200

0.885 0.888 0.901 0.903

0.875 0.885 0.901 0.900

0.869 0.889 0.901 0.897

0.874 0.889 0.899 0.900

0.875 0.885 0.896 0.903

0.878 0.885 0.897 0.902

0.876 0.887 0.905 0.904

n 20 40 100 200

Normal Approx. 2.069 1.469 0.929 0.656

Traditional Bootstrap 2.018 1.451 0.925 0.654

20 40 100 200

2.069 1.469 0.929 0.656

2.018 1.451 0.925 0.654

1.732 1.246 0.792 0.560

1.891 1.360 0.866 0.613

1.939 1.392 0.889 0.629

1.959 1.407 0.900 0.636

1.974 1.418 0.902 0.639

20 100 200

2.069 1.469 0.929 0.656

2.018 1.451 0.925 0.654

1.872 1.350 0.861 0.608

1.956 1.404 0.897 0.635

1.983 1.423 0.907 0.642

1.991 1.429 0.913 0.646

1.998 1.434 0.914 0.647

20 40 100 200

2.069 1.469 0.929 0.656

2.018 1.451 0.925 0.654

2.008 1.455 0.930 0.657

2.016 1.453 0.929 0.657

2.023 1.453 0.928 0.657

2.022 1.452 0.930 0.657

2.023 1.455 0.926 0.655

40

Dependent Bootstrap fc = 8 0.812 0.838 0.847 0.837 0.854 0.862 0.849 0.868 0.875 0.844 0.865 0.879

k = ± k =6

Interval Lengths Dependent Bootstrap fc = 2 k = A fc = 6 fc = 8 1.852 1.448 1.759 1.895 1.032 1.262 1.330 1.361 0.656 0.803 0.846 0.868 0.464 0.568 0.599 0.613

k = 10 0.854 0.866 0.877 0.884

fc = 10 1.921 1.379 0.879 0.622

Ill Institute of Mathematical Statistics LECTURE NOTES — MONOGRAPH SERIES

Kolmogorov-Smirnov Tests for AR Models Based on Autoregression Rank Scores Faouzi El Bantli1 I.S.R.O. Universite Libre de Bruxelles Brussels, Belgium

Marc Hallin2 I.S.R.O., E.C.A.R.E.S., and Departement de Mathematique Universite Libre de Bruxelles Brussels, Belgium Abstract

Tests of the Kolmogorov-Smirnov type are constructed for the parameter of an autoregressive model of order p. These tests are based on autoregression rank scores, and extend to the time-series context a method proposed by Jureckova (1991) for regression rank scores and regression models with independent observations. Their asymptotic distributions are derived, and they are shown to coincide with those of classical Kolmogorov-Smirnov statistics, under null hypotheses as well as under contiguous alternatives. Local asymptotic efficiencies are investigated. A Monte Carlo experiment is carried out to illustrate the performance of the proposed tests. Keywords: Autoregressive models, Autoregression quantiles, Autoregression rank scores, Kolmogorov-Smirnov test, Local asymptotic efficiency.

1

Introduction

Consider the autoregressive model of order p Yt = θ i Y t - ι + •- + θvYt-v

+ε

u

t e Z

(1.1)

{AR(p) model), where p > 1 is a fixed integer, θ = (0 X ,..., θp)' ap-dimensional vector of unknown autoregressive coefficients, and {ε*, t G Z} a process of 1

Research supported by the Fonds d'Encouragement a la Recherche de 1'Universite Libre de Bruxelles. 2 Research supported by an A.R.C. contract of the Communaute franςaise de Belgique and the Fonds d'Encouragement a la Recherche de ΓUniversite Libre de Bruxelles.

112

EL BANTLI AND HALLIN

independent and identically distributed random variables, with unspecified cumulative distribution function F and probability density / satisfying and

ί xdF(x) =0

0 < σ) := ί x2dF{x) < oo.

(1.2)

Letting P

zp -J^θiZ^

θ:=lθe

φ 0 Mz G C, \z\ < 1 L

(1.3)

we shall assume throughout the paper that 9 G Θ (the usual causality assumption) : {εt, t € Z} then is the innovation process, and / the innovation density, of the AR(p) process {Y^, t G Z}. Denote by Y n := (YΊ,..., Y^) a realization of length n of some solution of (1.1); we do not require Y n to be stationary, since all solutions are asymptotically stationary (see Hallin and Werker (1998) for a detailed discussion on this issue). Assume furthermore that (YLp+i,..., YQ) also are observed; in case they are not, they safely can be put equal to zero without affecting asymptotic results. Koul and Saleh (1995), extending to the time-series context Koenker and Bassett (1978) 's concept of regression quantiles, defined the a-autoregression quantile for model (1.1) as the solution

ϋin\a) := {θ^(a),θin\aή

, flW(α) G K, θin)(a) G W

of the minimization problem (n

£ ρa (Yt - θ0 - xj.^),

\

(1.4)

where the minimum is taken over all ϋ = (θo,θ) such that ΘQ € K. and θ € W, ρa(u) := |u| {al[u > 0] + (1 - a)I[u < 0]} , u € R,

aβ [0,1],

and x ( := (Yt, ..., Yt-p+ι)', t = 0,..., n - 1. This α-autoregression quantile can be obtained as the component ΰ(n)=

ίθ^\θinΛ

€ W+1 of the optimal solution ( d ( n ) , f + f - )

G E2n+P+1

of the linear program α l n r + + (1 — a)lnr~

:= min

Y n - l n 0 O - XnZ = Γ+ - ΓzeW,

i^eEl,

0<α
,

(1.5)

113

K-S TESTS FOR AR MODELS

where l n stands for the n-dimensional vector (1,..., 1)', and X.n is the n x p matrix with rows x ^ , 0 < t < n — 1. The corresponding dual program is Yna

:= max

ϊnsi = n(l - a)

(1.6)

Following the idea of Jureckova and Gutenbrunner (1992), later adapted to the autoregressive context by Koul and Saleh (1995), the solutions a π (a) = άι (α), ...,ά n n (α)J, 0 < a < 1 of (1.6) are called the a-autoregression rank scores.

The formal duality between (1.5) and (1.6) implies that, for alH = 1,..., n and α e [0,1], 1 if

Yt>θo(a)+^1θin\a)

0 if

Yt<§0(a)+xft_/n\a)

while, for t E < t Yt = ^o(^) + &'t-\θ {&) \> the components of a n (a) are determined by the equality constraints in (1.6). Clearly, the sample paths {a n (α), 0 < a < 1} are continuous, piecewise linear, and such that Qt (0) — 15 and άtn (1) = 0. An obvious modification of the algorithms of Koenker and dΌrey (1987 and 1994) allows for an efficient computation of the solutions ΰ

(a) and a n (α) over the whole interval [0,1].

A crucial property of autoregression rank scores is their autoregressioninvariance, i.e., denoting by άtn (α, Y n ) the solution of (1.6), the fact that ά; n ) (α,Y n + * l n + X n z) =aί n ) (α,Y n ),

(z,z) e KP+1,

(1.7)

which immediately follows from (1.6). Some further algebraic relations between autoregression quantiles and the corresponding autoregression rank scores are provided in Lemma 2.1 of Hallin and Jureckova (1999). Quite remarkably, no preliminary estimation of θ is needed in order to compute autoregression rank score statistics. This is in sharp contrast with the more familiar aligned rank methods (Hallin and Puri, 1994), where ranks are computed from estimated residuals; see Jureckova (1991), Gutenbrunner and

114

EL BANTLI AND HALLIN

Jureckova (1992), Gutenbrunner et al. (1993), Hallin et al. (1997a, 1977b), Harel and Puri (1998), or Hallin and Jureckova (1999) for details and numerical applications. In the present paper, new tests based on a autoregression rank score version of the traditional Kolmogorov-Smirnov statistic are introduced for model (1.1). The asymptotic behaviour of these tests is investigated in Section 3, where we show that the limiting distributions of the test statistics coincide with those of the classical Kolmogorov-Smirnov statistics, both under the null hypothesis as under contiguous alternatives. Our results extend those of Jureckova (1991) from regression models to autoregression models. The local asymptotic efficiency of these tests is also investigated. Finally, the performance of the proposed tests is illustrated on simulated AR series with Normal, Laplace and Cauchy innovation densities, respectively.

2

Limiting Distributions of Kolmogorov-Smirnov Statistics Based on Autoregression Rank Scores

Assume that the density / of the innovations in the autoregressive model (1.1) remains unspecified within the family T of exponentially tailed densities satisfying (1.2) and the following conditions (borrowed from Hallin and Jureckova, 1999) : (Fl)

f(x) is positive for all x E M, and absolutely continuous, with a.e. /

d e r i v a t i v e /'a n d

finite

Fisher information l(f)

:=

£f ί

\\ ^

/ 1 . 1

f(x)dx

J \}\x)j < oo; moreover, there exists K/ > 0 such that, for all \x\ > Kf, f has two bounded derivatives, / and / " , respectively; (F2)

/ is monotonically decreasing to 0 as a; —> ±oo and, for some b = bf > 0, r = 77 > 1, -log(l-F(s))

-logFjx) z lim »oo

b\x\r

i m

l z->oo

b\x\r

We will focus on the problem of testing null hypotheses of the form Ή-o

' θp = 0,

0(!) := (0i,..., θp-ι) unspecified,

against alternatives U\ : θp φ 0,

0 ( 1 ) := {θu ..., 0p_i) unspecified.

Such tests play a crucial role, for instance, in the order identification process (see Garel and Hallin 1999).

115

K-S TESTS FOR AR MODELS Write the AR{p) model (1.1) as ^(l) + Xn;2^p + β n ,

(2.1)

where X n := ( X n ; i:X n ; 2 I is the (n x p) matrix with rows x^_i, 1 < t < n, Xn;2 := (Y-p+u , ^n-p)' (hence, Xn;2,t = Yt-p, t = 1,..., n), and ε n := ; (εi,...,ε n ) . Denote by P n := X n ; i(X n ; 1 X n ; i) 1

the (random) matrix projecting W onto the linear space spanned by the columns of X n; χ. Define ^ ; 2 := Xn;2 ~ Xn;2 ~ [Xn,2 ~ Xn;2J l n

where n-p a

n

d

t=l

t=-p+l

and let )

:

^ = n" 1 (X n ; 2 - Xnflln)'

[In - P n ] (X n ; 2 - X n ; 2 l n ) .

Denoting by 7it(d), A; = 1,... the autocovariances of the stationary solution of (1.1), the consistency under AR(p) dependence of empirical autocovariances implies that D\ converges in probability, as n —> oo, to (

TO W

71W

•••

Ίv-2{θ)\

-1

71

Ίp-ιiβ)

which, in view of classical Yule-Walker equations, under HQ reduces to p-1

a simple scale factor that does not depend on 0, nor on the shape of the innovation density /. Let a n (a) = Γά/1 (α),...,ά n n (cm, 0 < a < 1, be the autoregression rank scores computed under HQ , i.e., corresponding to the submodel Yn =

+ εn

(2.2)

116

EL BANTU AND HALLIN

(though of course a " (α), as a statistic, does not depend on 0(i)) For each n, consider the process {Tn(a) : 0 < a < 1} defined by Xn;2,tάt

;

(α),

0 < α < 1.

(2.3)

t=l

This process has trajectories in the space C[o,i] °f continuous functions α ι-> c(a), a 6 [0,1] (as usual, C[o,i] is equipped with the Borel σ-field C associated with the uniform metric ||ci — C21| := maxo<α
h± (c( )) := max^ |c(α)|.

and

The one- and two-sided autoregression rank score-based Kolmogorov-Smirnov statistics we are considering in the problem of testing KQ are +

1

K+ := h (Tn( )) = n-^D-

max £ X ^ ά ^

(α)

(2.4)

and 71

71

n

n

0
(n) U

~ *

'

'

respectively. The following results provide the asymptotic distributions of K^ and K^ under the null hypothesis ΉQ . Theorem 2.1 Assume that (F1)-(F2) are satisfied. Then, under Ήξ, x

lim P(K+ <x) = \

n->oo

—

2

1 l — exp(—2x )

~

(2.6)

x > ϋ

and

{

U;

1

2

2

l-2f)(-l)*- «p(-2ifc x ) The proof is based on the following lemma.

<0

x>0

•

(2

'

7)

Lemma 2.1 Define the scores 0 a*t (a) := { Rn.t - not 0

no. < Rn]t - 1 Rn,t - 1 < not < i? n ; ί Rn t < na ,

(2.8)

117

K-S TESTS FOR AR MODELS

where Rn]t denotes the rank of εt among εi,..., εn. Assuming that (F1)-(F2) are satisfied, let at(a) := I[εt > F~ι{a)],

1 < t < n,

0 < a < 1.

Then, sup n

-1/2

Σ [(*n;2,t " *n;2) ^ (α) - X^fi,(<*)] A 0

t=l

O
sup n

-1/2

(2.9)

I

[(Xn;2)ί " Xn;2) ^ (a) - X^O^a)]

A 0,

(2.10)

0
as n -> oo. Moreover, the process {Tn(a) : 0 < a < 1} converges in distribution in (C[Oji],C) fo ίΛe Brownian bridge {Z(a) : 0 < a < 1}. Proof of Lemma 2.1. The convergence (2.9) is a direct application of Theorem 3.2 of Hallin and Jureckova (1999). The approximation results on the rank score process given in Hajek (1965) (see also Jureckova 1999) then yield (2.10). The asymptotic behaviour of {Tn(a) : 0 < a < 1} follows from the fact that

}

{ I

)

t=i

converges weakly to the Brownian bridge {Z(a}} : see Theorem 1 in Section V.3.5 of Hajek and Sidak (1967). D Proof Theorem 2.1. Prom Theorem V.3.3.a of Hajek and Sidak (1967), we have, for positive x and y and all 0 < a < 1, P [Z(a) < x(l - a) + ay] = 1 - exp(-2:ry) and

P [-x(l -a)-ay<

Z(a) < x(l - a) + ay] = 1-2 k=l

The asymptotic distributions (2.6) and (2.7) directly follow from letting x = y. Π Thus, the one-sided test based on K+ rejects WR at (asymptotic) prob1 /9

ability level a whenever K+ is larger than the critical value ί— ^logαj

,

118

EL BANTLI AND HALLIN

while the two-sided based on K^ rejects ΉQ at probability level a whenever Kn exceeds the α-quantile q(a) := Q~ι(a) of the distribution function oo

k 1

2

2

Q(x):=2Σ(-l) - exp(-2k x ). k=ι

This function Q(x) has been tabulated by Smirnov (1948). Note that both critical values are entirely distribution-free, as they do not depend neither on / nor on the nuisance θ^. We now turn to a study of the power of the tests based on K+ (the case of K^ follows along similar lines, and is left to the reader). The following result provides asymptotic powers against local alternatives of the form Un : θp = 0£ := rCxl2τ,

r E R,

0 ( 1 ) = (θu ..., 0p_i) G I T " 1

unspecified.

Theorem 2.2 Assume that (F1)-(F2) are satisfied. Let pn(θ{ιy,τ)

:=Pθn |uf+ > (^-I

where P$n is computed under θn := (0(i),0p) Then, for any 0 < α < 1,

,

(2-11)

where f\ and F\ stand for the standardized versions of f and F, respectively. Furthermore, for τ —> 0 (the notation £(τ) ~ ζ(τ) means that the ratio ξ(τ)/ζ(r) tends to one as τ -> 0; Φ, as usual, stands for the standard normal distribution function), 1/2

( ( 1 λ ί1 ι2 B(a,/,r) — a~ I 2σfT~ ' (f)a ί — -logα 1 /

\ φ(u,f)ψ(a,u)du)r, (2.12)

ψ(a,u) ~ 2Φ U - I l o g α ) 1 ' (2u - l)(tι(l - u))" 1 / 2 ! - 1.

Proof. This proof, as well as the proof of Theorem 3.1, heavily relies on Sections VL4.5 and VΠ.2.3 of Hajek and Sidak (1967); the various constants appearing there here are to be taken as ct = n-ι'2X^t

(hence, c = 0),

p 2 = 1,

K-S TESTS FOR AR MODELS

119

and

Theorem VΊ.3.2 in Hajek and Sidak (1967) entails that, under 1-Ln, the process converges in distribution to the Brownian bridge {Z(a)}. Since K+ = maxo<α
La* for any α; (2.11) follows from noting that σff(F~1(u)) = fι(F^ι(u)). As for the linear approximation (2.12), it is an immediate consequence of Theorem 1 in Section VI.4.5 of Hajek and Sidak (1967). D

3

Local Asymptotic Efficiency of the Test Based onK+

Pitman asymptotic relative efficiencies of the tests based on K* with respect to the corresponding Gaussian Lagrange multiplier procedures, or with respect to the autoregression rank score test proposed by Hallin and Jurckova (1999) cannot be computed as easily as in the usual case of asymptotically normal or chi-square test statistics, for which AREs are obtained as ratios of noncentrality parameters. Actually, the analytical form of asymptotic powers, for given θ and r does not allow for any simple result, and AREs typically will depend on α, /, θ and r. This problem already appears in the classical case of Kolmogorov-Smirnov tests for linear models with independent observations; see Hajek and Sidak (1967), Section VII.2.3. However, local ARE results can be obtained from the linear approximation (2.12) of the asymptotic power B(a,f,τ) as r —>• 0. More precisely, denote by e(α,/,β,r) the ARE of the test based on K+ with respect, for instance, to the locally optimal Gaussian test : the following result, inspired from Section VII.2.3 of Hajek and Sidak (1967), shows that the limit e(α,/):=lime(α,/,β,τ)

(3.1)

does not depend on θ and r anymore, and thus allows for local or approximate comparisons of asymptotic performance.

120

EL BANTLI AND HALLIN

Theorem 3.1 Under (F1)-(F2), the local asymptotic relative efficiency of K^ with respect to the locally optimal Gaussian test is

e(a,f) =4πα 2 (-logα)J- 1 (/)exp(z2) [j|%(u,/W(α, u) du] ,

(3.2)

where za denotes the (1 — a)-quantile of the normal distribution. Moreover,

Jme(α)=Γ 1 (/) [^ φ(u, f)sgn(2u - 1) du]

(3.3)

Proof. The proof readily follows from Theorem VΠ.2.3 of Hajek and Sidak (1967).

D For instance, under the Laplace (double-exponential) density f(x) = ^ — | x | ) , we have φ(u, f) = sgn(2w - 1),

0 < u < 1.

One can easily check that / Φ(ui /)sgn(2u - 1) du = 1 - 2Φ ( - ( - log α ) 1 / 2 ) /α, so that (3.2) becomes

e(α) =4π(-logα)exp(4_ α ) [α - 2Φ (-(-logα) 1 / 2 )]'. In general, however, the local asymptotic efficiency e(α, /) still involves the actual innovation density /. If we apply the Cauchy-Schwarz inequality to (3.2), we obtain the uniform (in /) upper bound e(α) := sup e(α, /) = 4 π α 2 ( - logα) exp(z2_a) / ψ(a, u)2 du. fer Jo Using the fact that lim 2πx2 exp(x2)(l

— Φ(x))2 — 1,

it follows that 2

Jhn o 4πα (-logα)exp(^_ α ) = 1, hence lim e(a) = 1.

a—>o

One can easily check by computations that e(0.01) = 0.864, e(0.05) = 0.852, and g(0.1) = 0.844.

K-S TESTS FOR AR MODELS

4

121

Simulation Results

The null hypothesis θ = 0 has been considered in the AR(3) model Yt = 0.5Yt_i - 0.2Yt_2 + 0Yt-3 + ε t .

(4.1)

More precisely, N = 2000 replications of the AR(3) series of length n — 100 generated by (4.1) with initial values Yl2 = Y-i = YQ = 0, θ = 0, θ = 0.1, θ = 0.2, θ — 0.3, and θ = 0.4, and three innovation densities (standard normal, standard Laplace, and standard logistic) have been subjected to the + two Kolmogorov-Smirnov tests described in this paper (based on K and 1 1 if * ), respectively), and to the Gaussian Lagrange multiplier test (based on a quadratic form Qc\ see Garel and Hallin 1999). For each of these tests, relative rejection frequencies (at probability levels a = 5% and a = 1%) are reported in Table 1. Inspection of the θ = 0 column of Table 1 reveals that the three tests considered all are rather conservative, thus biased. The bias seems less severe, though, for the test based on K+ than for the K^ and Qc ones. Since the same one-sided test based on K+ is also significantly more powerful, under all alternatives considered here, than the two-sided one based on K±^ the two-sided Kolmogorov-Smirnov procedure might well be non admissible. Under θ = 0.1, the test based on K+ is slightly more powerful than the Gaussian Lagrange multiplier test, even under Gaussian innovations, despite the local asymptotic optimality of the latter. Under larger values of 0, this advantage of K+ over its competitors is clear under the three densities considered, but particularly marked under Laplace densities—as expected. Though more extensive simulations should be undertaken, Table 1 nevertheless indicates that Kolmogorov-Smirnov techniques, especially the onesided, based on nonlinear test statistics, can be expected to beat locally asymptotically optimal tests based on linear statistics.

Acknowledgement. The authors thank the Editor, Professor Ishwar Basawa, and an anonymous referee for their careful reading of the manuscript and helpful comments.

References [1] Garel, B., and M. Hallin (1999). Rank-based AR order identification. Journal of the American Statistical Association 94, 1357-1371 . [2] Gutenbrunner, C., and J. Jureckova (1992). Regression rank scores and regression quantiles. Annals of Statistics 20, 305-330.

122

EL BANTLI AND

HALLIN

[3] Gutenbrunner, C , J. Jureckova, R. Koenker, and S. Portnoy (1993). Tests of linear hypotheses based on regression rank scores. Journal of Nonparametric Statistics 2, 307-331. [4] Hajek, J. (1965). Extension of the Kolmogorov-Smirnov test to the regression alternatives. In Bernoulli (1713), Bayes (1763), Laplace (1813) : Proceedings of an International Research Seminar (L. Le Cam and J. Neyman, eds), 45-60. Berlin, Springer-Verlag. [5] Hajek, J., and Z. Sidak (1967). Theory of Rank Tests. Academic Press, New York. [6] Hajek, J., Z. Sidak, and P.K. Sen (1999). Theory of Rank Tests (2nd edition). Academic Press, New York. [7] Hallin, M., J. Jureckova, J. Kalvova, J. Picek, and T. Zahaf (1997). Nonparametric tests in AR models, with applications to climatic data. Environmetrics 8, 651-660. [8] Hallin, M., J. Jureckova, J. Picek, and T. Zahaf (1998). Nonparametric tests of independence of two autoregressive time series based on autoregression rank scores. Journal of Statistical Planning and Inference 75, 319-330. [9] Hallin, M., and J. Jureckova (1999). Optimal tests for autoregression models based on autoregression rank scores. Annals of Statistics 27, 1385-1414. [10] Hallin, M., and M.L. Puri (1994). Aligned rank tests for linear models with autocorrelated errors. Journal of Multivariate Analysis 50, 175237. [11] Hallin, M., and B.J.M. Werker (1998). Optimal testing for semiparametric autoregressive models : from Gaussian Lagrange multipliers to regression rank scores and adaptive tests. In Asymptotics, Nonparametrics, and Time Series (S. Ghosh, Ed.), 295-358. M. Dekker, New York. [12] Harel, M., and M.L. Puri (1998). Autoregression quantiles and related rank score processes for generalized random coefficient autoregressive processes. Journal of Statistical Planning and Inference 68, 271-294. [13] Jureckova, J. (1991). Tests of Kolmogorov-Smirnov type based on regression rank scores. In Transactions of the 11th Prague Conference on Information Theory, Statistical Decision Functions, and Random Processes (J.A. Vίsek, Ed.), 41-49. Academia, Prague.

K-S TESTS FOR AR MODELS

123

[14] Jureckova, J. (1999). Regression rank scores against heavy-tailed alternatives. Bernoulli 5, 1-18. [15] Koenker, R., and G. Bassett (1978). Regression quantiles. Econometrica 46, 33-50. [16] Koenker, R., and V. dΌrey (1987). Algorithm AS 229 : Computing regression quantiles. Journal of the Royal Statistical Society Series C 36, 383-393. [17] Koenker, R., and V. dΌrey (1994). Remark AS R92. A remark on Algorithm AS 229 : Computing dual regression quantiles and regression rank scores. Journal of the Royal Statistical Society Series C 43, 410414. [18] Koul, H.K., and A.K.Md.E. Saleh (1995). Autoregression quantiles and related rank scores processes. Annals of Statistics 23, 670-689. [19] Smirnov, N.V. (1948). Table for estimating the goodness of fit of empirical distributions. Annals of Mathematical Statistics 19, 279-281. Marc HALLIN Faouzi EL BANTLI [email protected] [email protected] Institut de Statistique et de Recherche Operationnelle Campus de la Plaine Universite Libre de Bruxelles CP210 B 1050 Brussels Belgium

EL BANTLI AND HALLIN

124 θ =0

f

θ = 0.1

θ = 0.2

α = 5%

a = 1%

α = 5%

α = 1%

α = 5%

a. = 1%

θ = 0.3

θ = 0.4

ct = 59% α = 1% a = 5?ro

a = 1%

K+

Normal Laplace Logistic

0.035 0.038 0.043

0.006 0.009 0.008

0.374 0.525 0.377

0.155 0.226 0.176

0.578 0.854 0.675

0.238 0.452 0.287

0.708 0.873 0.769

0.428 0.663 0.514

0.908 0.982 0.934

0.753 0.915 0.797

AT*

Normal Laplace Logistic

0.041 0.036 0.050

0.003 0.007 0.005

0.248 0.462 0.263

0.131 0.174 0.138

0.403 0.661 0.477

0.153 0.325 0.185

0.604 0.785 0.654

0.324 0.554 0.382

0.845 0.955 0.883

0.634 0.843 0.714

Qn

Normal Laplace Logistic

0.035 0.031 0.046

0.003 0.003 0.006

0.335 0.324 0.221

0.144 0.112 0.102

0.483 0.442 0.402

0.224 0.211 0.147

0.723 0.733 0.683

0.366 0.521 0.383

0.854 0.731 0.868

0.747 0.742 0.783

0.004

0.002

0.011

0.009

0.011

0.011

0.011

0.011

.010

0.011

standard errors

Table 1. Rejection frequencies of the null hypothesis θ = 0 for n = 100, under standard Normal, Laplace, and Logistic innovation densities /, respectively, and for various values of 0, a = 5% and a = 1%. The number of replications is N = 2000; maximal standard errors are provided for each column.

125

Institute of Mathematical Statistics LECTURE NOTES — MONOGRAPH SERIES

Estimation of the Long-memory Parameter: a Review of Recent Developments and an Extension R. J. Bhansali P. S. Kokoszka The University of Liverpool Abstract Current methods of estimating the memory parameter, d, of a longrange dependent stationary time series are reviewed, and a new method of estimating d by fitting a fractionally-differenced autoregression of order p is introduced. Under the assumption that p approaches infinity simultaneously with the observed series length, n, the estimators are shown to be consistent both when the innovations have finite variance and when their distribution follows an infinite-variance stable law with exponent α £ (1,2). The relative finite sample behaviour of the estimators is investigated by a simulation study and by an application to a real data set involving ethernet traffic.

1

Introduction

Let {Xt} be a discrete-time covariance-stationary process with 0 mean, covariance function R(u) = E(XtXt+u) {t, u = 0, ±1,...) and spectral density function /(λ). Then, {Xt} is said to exhibit long-memory with memory parameter d, 0 < d < 0.5, if /(λ) may be written as /(λ) = |λΓ 2 d L(l/λ)

(1.1)

where L(λ) is slowly varying at infinity. A characterizing property of a long-memory time series is that its covariance function is not absolutely summable. By contrast, {Xt} is said to be a short-memory process if its covariance function is absolutely summable, implying that /(λ) is continuous and bounded on [O,τr]. Although the definition (1.1) above is classical, it does not readily extend to processes which are not covariance-stationary, see Heyde and Yang (1997) and Hall (1997). Nevertheless, in what follows, we use this definition for ease of exposition, though in Section 3, when considering the class of stable processes with infinite variance, we restrict the admissible values of d.

126

BHANSALI AND

KOKOSZKA

For general surveys discussing stochastic properties of long-memory processes, see Hosking (1981), Granger and Joyeux (1980), Andel (1986) and Beran (1992, 1994), among others. This class of processes has been found to be useful for modelling time series occurring in many fields of applications, including Hydrology, Geophysics, Economics, Ecology and telecommunication traffic, see Willinger et al. (1995) for a discussion of the last application. In recent years, there has also been much development on the general question of estimating the long-memory parameter, d, from a finite realization of length n from {Xt}> An objective of this paper is to provide an overview of these developments and also to study the empirical behaviour of a new method of estimating d by fitting fractional autoregressive, FAR, models; a consistency result for the estimator is also established. The FAR method may be viewed as an extension to the class of long-memory processes of the well-known autoregeressive model fitting approach of estimating the spectral density, the linear predictor and related parameters of a short-memory stationary process with a bounded spectral density; see Parzen (1969), Berk (1974), Shibata(1980, 1981), Bhansali(1978, 1980), among others. Now, however, the spectral density is unbounded and the theory developed by these authors does not apply. A related reference is Bardet et al. (1999) who review the current state of the art for estimating d from a slightly different perspective.

2

An Overview of the Current Methods for Estimating d

The existing methods for estimating d may be grouped under four broad headings: graphical, parametric, non-parametric and semi-parametric. Two of the early methods for estimating d are graphical and are based on the use of the R/S statistic and/or the variance-time plot; see, e.g. Leland et al. (1994). A main difficulty in using the graphical methods is that, especially for moderate values of n, they typically have large biases; their asymptotic behaviour has recently been investigated by Giraitis et al. (1999). A commonly-used parametric method estimates d by postulating that {Xt} follows a fractional autoregressive moving average model of order (p, d, q), ARFIMA(p, d, q), i.e., the dth fractional difference follows a standard ARM A model. On the assumption that the order (p, q) is known a priori, the model parameters, θ(p, q), say, including d, are estimated by a likelihood procedure. The resulting estimates, θ(p,q), say, are known to be asymptotically efficient, in the sense that, under appropriate regularity conditions, a s n - ) oo, ι 2 n l {θ{p, q) — θ(p, q)} converges in distribution to a Normal random variable with a 0 mean vector and covariance matrix W, where W~ι denotes the corresponding Fisher information matrix. This result was established by Fox

ESTIMATION OF LONG MEMORY

127

and Taqqu (1986) for the estimate obtained by adopting the Whittle approximation to the likelihood function; see also Giraitis and Surgailis (1990) for an extension to non-Gaussian processes and an alternative proof. Dahlhaus (1989) extended the result to exact maximum likelihood estimates of θ(p, q)\ also, Beran (1995) considers the use of an 'autoregressive' approximation to the likelihood function and allows for all d G (—0.5, oo), where a value of d > 0.5 would indicate that {Xt} is a non-stationary accumulated process considered by Box and Jenkins (1970) and {Xt} could be transformed to a stationary and possibly a long-memory process by differencing it an appropriate number of times. The algorithmic aspects of how to compute the likelihood estimates are discussed by Hosking (1984), Haslett and Raftery (1989), Sowell (1992), among others. Pai and Ravishanker (1996, 1998) and Ravishanker and Ray (1997) consider a Bayesian approach to the estimation of θ(p, q), while Chan and Palma (1998) investigate the use of a state space approach and discuss how this approach is particularly useful for time series with missing observations. A main difficulty in implementing the parametric approach, however, is that the order (p, q) is invariably unknown, and, for an observed time series, there may not even be a 'true' value of this order. The simulation results of Taqqu and Teverovsky (1998) demonstrate that the parametric method may not be improved upon when the order (p, q) is correctly specified but it performs rather poorly when the order is misspecified. Beran et al. (1998) consider the question of model selection for the class of fractional autoregressive, FAR(p,d), models with p finite and d G (—0.5,oo) and derive an appropriate version of the Akaike information criterion, AIC, for this class of processes and show that it is of the same form as in the standard short-memory situation, but with d treated as an additional parameter. Asymptotic sampling properties of the order selected by AIC are also studied and in particular it is shown that, as in the short- memory case, whereas AIC does not provide a consistent order selection procedure for this class of processes, the corresponding versions of the BIC and HIC criteria of Schwarz (1978) and Hannan and Quinn (1979) do so. A related reference is Crato and Ray (1996) who carry out an extensive simulation study for investigating the empirical behaviour of various order selection procedures for ARFIMA as well as non-AFRIMA models. Unlike the parametric methods, the non-parametric methods seek to estimate d under few prior assumptions concerning the spectral density of {Xt} and, in particular, without specifying a finite parameter model for the dth difference of {Xt}> A successful procedure, called the GPH method, is due to Geweke and Porter-Hudak (1983), and it is based on the observation that if in (1.1), L(λ) = C, a constant, for all λ, log/(λ) follows a linear function of log λ in a neighbourhood of the origin, and, hence, an estimate of d may

128

BHANSALI AND

KOKOSZKA

be found by regressing log-periodogram on log-λ; moreover, the logarithm of a smoothed and/or tapered periodogram, in place of the raw periodogram, may also be employed for estimating d. The GPH method has been widely studied in the literature. Thus, it has been recognised that the band of frequencies over which the aforementioned regression is performed needs to be chosen carefully in that the frequencies too close to or too far away from the origin should be excluded so as to avoid the estimate being 'contaminated' by their inclusion, see, for example, Hurvich and Beltrao (1993), who also show that the behaviour of the periodogram near the origin in this case is non-standard. Moreover, Agiakloglou et al. (1993) show that for small values of n, this estimator could be badly biased and investigate the sources of this bias for particular models. Robinson (1995a) has derived the asymptotic distribution of a GPH estimator, dcPHt say, such that the estimator is obtained by regressing the log-periodogram, or a smoothed version thereof, on log λ for λ G [A^, λ[/], where λ^ and Xjj are 'trimming' numbers and specify the actual band of frequencies, not too close or too far away from the origin, over which the regression is performed. Robinson shows that, under regularity conditions, including those on λ^ and λ^/, as n -> oo, n^{dGPH — d} is asymptotically distributed as normal with mean 0 and variance rra//(ra), where m specifies the degree of smoothing, called pooling, applied to the periodogram before taking the logarithms and performing the regression, ψ'(m) denotes the trigamma function and the precise value of β depends upon m and the trimming numbers used. The value of mψ'(m) decreases as m increases and it converges to one as m —> oo; however, even with an optimal choice of m and the trimming numbers, β < 2/5. An additional reference is Hurvich et al. (1998) who derive asymptotic expressions for the mean squared error of the original GPH estimator, in which none of the low frequency periodogram ordinates are omitted, as functions of M, where M determines the number of periodogram ordinates included in the regression and tends to infinity simultaneously but sufficiently slowly with n, and find the optimal M minimizing the asymptotic mean squared error. A main difficulty in using the GPH estimator with an observed time series, however, is the determination of an optimal band of frequencies over which the regression is performed; a systematic procedure for doing so is not yet available. A second difficulty is that the method exhibits high variability, since its rate of convergence, n~^, β < 0.4, is much slower than that for the parametric estimator. In the semi-parametric approach also, see Kunsch (1987), Robinson (1995b), a precise parametric form for the spectral density of {Xt} is not specified and it is assumed to be of the form given in (1.1), but with L(λ) = G(d), a continuous function of the long-memory parameter, d, in a neighbourhood of the origin. An estimate of d is constructed by a likelihood procedure based

ESTIMATION OF LONG MEMORY

129

on the Whittle likelihood but the maximization of the likelihood is only over a neighbourhood of the zero frequency which degenerates slowly to zero, as n increases. Thus, this approach combines the parametric and non-parametric methods discussed above and it is often called 'local Whittle'. However, it suffers from the same difficulties as described above for the nonparametric approach in that currently a procedure is not available for choosing systematically the bandwidth of freqencies to be used for computing the estimate and which must be chosen in an arbitrary fashion; also, the optimal rate of convergence of the estimator is still n~^, β < 0.4, and it is not as fast as that for the parametric approach. The estimates of d based on the non-parametric and semi-parametric approaches are, however, not comparable to those based on the parametric approach, since whereas in the latter the behaviour of the spectral density of {Xt} is specified for all frequencies, in the former it is only specified for frequencies in a neighbourhood of 0 and in this sense the parametric approach uses more information than the non-parametric or semi-parametric approaches; see also Beran (1997). Recently, Moulines and Soulier (1999) and Hurvich and Brodsky (2001), have considered an FEXP approach to the estimation of d in which the logarithm of the spectral density of the short-memory process is postulated to possess an infinite Fourier series expansion and thus the fractionally-differenced short memory process, {Yi}, say, is specified to follow the exponential model of Bloomfield (1973). An estimator of d and of the Fourier coefficients, c(j), say, is found by first truncating the infinite Fourier series at some finite value, A;, say, and then by applying an ordinary linear least-squares procedure in which the logarithm of the smoothed periodogram, smoothed over m non-overlapping blocks, is regressed on —21og|l — e~ zλ | and on the Fourier cosine functions. On the assumption that k —> oo as n —» oo, but sufficiently slowly, and under additional regularity conditions, the authors establish the asymptotic normality of their estimator, dER, say, and show that y/n/k{dER — d} is asymptotically normal with mean 0 and variance mψf(m), where ^'(m), as before, denotes the trigamma function. Moreover, if the c(j) converge to zero at an exponential rate then k may be taken to be proportional to log n and the FEXP approach yields an estimator of d with a convergence rate of y/ϊognjn which is faster than that for the non-parametric or semi-parametric approaches, but without specifying a finite parameter model for the short-memory process, Next, in Section 3, we discuss an alternative FAR approach to the estimation of d in which the short-memory process is postulated to follow an infinite autoregressive process and an FAR(p,d) model is fitted to an observed time series, where p is such that p -» oo simultaneously but sufficiently slowly with n. Thus, in common with the FEXP method, in our

130

BHANSALI AND KOKOSZKA

approach also a finite parameter model for the short-memory process, is not specified and an estimate of d is constructed by fitting a hierarchical family of parametric models. However, unlike the former, we adopt a likelihood-based method for estimating the model parameters and, in our approach, estimates of d and of the autoregressive parameters are obtained by minimizing the Whittle likelihood and, for an observed time series, the value of p, the autoregressive order to be fitted, will be selected by minimizing AIC or a related order selection criterion considered by Beran et al. (1998). A consistency result for the parameter estimates is established in Section 4 both for the situation in which the innovation process, {Z*}, has finite variance and when the Zt have a distribution in the domain of attraction of an infinite-variance stable law. The results of a simulation study aimed at comparing the finite sample behaviour of the FAR estimate of d with that of the parametric and non-parametric estimates are described in Section 5, while in Section 6 the empirical behaviour of the FAR approach is illustrated with teletraffic ethernet data.

3

Assumptions, Parameters and Estimates

We suppose that the observed time series X i , . . . , Xn is a (part) realization of a process {Xt} satisfying the following assumption: ASSUMPTION

1 That {Xt}

(t = . . . , -1,0,1,...) satisfies (1 - B)dXt = Yu

(3.1)

ΣύYt-j = Zu

(3.2)

where {Yt} satisfies j=0

the aj are absolutely summable real coefficients satisfying oo

oo

Σ h , |
Σα^yo,

3=0

3=0

\z\
(3.3)

for some ε > 0 and {Zt} is a sequence of independent identically distributed random variables each with mean zero and satisfying either Assumption 2A or Assumption 2B below: 2A That EZt2 = VarZ* = σ2 < oo, (ί = . . . , -1,0,1,...). ASSUMPTION 2B The common distribution of the Zt is in the domain of attraction of an α-stable law with 1 < a < 2, i.e. ASSUMPTION

P(\Zt\ >x) = χ-"L(x),

(3.4)

131

ESTIMATION OF LONG MEMORY where L is a slowly varying function, and

P{Zt > x)/P(\Zt\ >x)^a,

P(Zt < -x)/P{\Zt\ > x) -* 6,

(3.5)

as, x —> oo, where a and b are nonnegative numbers satisfying a + b = 1. Assumption 2A is equivalent to requiring that the sums n" 1 / 2 Σ™=1 Z* converge to a normal distribution, i.e. are in its domain of attraction. It is known, see e.g. Samorodnitsky and Taqqu (1994), that, in addition to a normal limit, the only other possible limit for a normalized sum of iid random variables is an α-stable distribution with 0 < a < 2, which has infinite variance. Under conditions (3.4) and (3.5), the sum ]CtLi %t divided by nιla and appropriately centred tends to an α-stable distribution. A process satisfying Assumptions 1 and 2A is well-known to be stationary if d < 1/2. In the sequel we assume that 0 < d < 1/2, i.e. we consider only the long-memory case, see e.g. Hosking (1981). Theorem 2.1 in Kokoszka (1996) (see Kokoszka and Taqqu (1995, 1996a) for purely stable case) implies that if d < 1 — 1/α, there is a unique strictly stationary process satisfying Assumptions 1 and 2B. In view of this result, in the following assumption we restrict the admissible values of d. ASSUMPTION

3 That 0 < d < 1 - 1/α.

(3.6)

(The case a = 2 corresponds to Assumption 2A.) In the following, we use the superscript 0 to denote the true values of the various parameters, e.g. d°,a^. In order to define the parameter space, consider the Banach space F of absolutely summable sequences β = (d, αi, cz2,. ) τ with the norm

3=1

We now define the parameter space by the following assumption. 4. That the parameter space, denoted E, is a compact subset of F containing an open neighbourhood of the true parameter β° = ASSUMPTION

(d°,α?,α°,...)τ. The idea of fitting an FAR(p, d) model can be most conveniently formalised by introducing the restricted parameter spaces Έp = {β G E : α& = 0 for fc>p},p=l,2, The Whittle (an approximate maximum likelihood) estimate βpn obtained by fitting an FAR(p, d) model is then defined as the value of β G E p which minimizes the integral

G(β) = / J —π

/n(λ

^λ,

(3.8)

132

BHANSALI AND KOKOSZKA

where i\t

2πn

(3.9)

t=i

is the periodogram and -2 ΛX\d

(3.10)

is the power transfer function, which is proportional to the spectral density of the process {Xt} in the finite variance case. Under Assumption 2A, the corresponding Whittle estimate of σ2 is given by

=Γ J-

'p,nJ

EXAMPLE. TO give an example of a parameter space satisfying Asssumption 4, consider a sequence of positive numbers x$, xi,... such that Σ ^ o Xj < oo and \βj\ < Xj. The set {β eF : \βj\ < Xj, j = 0,1,...} is then a parameter space satisfying Assumption 4. Moreover, now the corresponding set, {βp : \βj\ < xj >3 — 0,1,... ,p;;Sj = 0,j > p}, is an example of the parameter space E p .

4

Consistency of the Estimates

The consistency of the Whittle estimators /3Pj7l and σ2(p) obtained by fitting fractionally differenced autoregressive models is established below in Theorem 4.1. In common with related studies, see, for example, Berk (1974), we assume that p = p(n) is a sequence of integers such that p(n) -> oo, p(n)/n -> 0, as n —>> oo. For an observed time series, the value of p will be determined by appealing to the AIC, or BIC, criterion introduced in Section 5 by equations (5.1) and (5.2), respectively, or by a related criterion. Thus, in practice, p = p(n) will invariably be a random sequence of integers and Theorem 4.1 does not apply to this situation. In Section 5, we demonstrate the usefulness of the FAR approach in this situation by a simulation study, and, also, compare the relative behaviour of some of the alternative estimates of d discussed in Section 2 with that of the FAR estimate. THEOREM 4.1 Suppose p = p(n) is a sequence of integers such thatp(n) < n and p(n) —> oo, as n -> oo, and that Assumptions 1-4 hold. 2 2 (i) Under 2A, with probability one, βp -* β° and σ (p) -> σ . (ii) Under 2B, βp —>• β° in probability.

ESTIMATION OF LONG MEMORY

133

The proof of Theorem 4.1 is similar to the corresponding proofs for the correctly specified models, see Fox and Taqqu (1986) and Kokoszka and Taqqu (1996b). Modifications include Lemma 4.1 and the proof of Proposition 4.1. We present the proof only under Assumption 2A because the argument under Assumption 2B is essentially the same, but with the difference that one now works with the self-normalized periodogram and the a.s. convergence in Proposition 4.1 and Lemma 4.2 is replaced by convergence in probability, see analogous proofs in Kokoszka and Taqqu (1996b) and Mikosch et al. (1995). LEMMA

4.1 For any compact subset E ofF and any sequence p = p(n) such

that p -> oo, as n —» oo ;

lim sup y^ \aΛ = 0. PROOF.

(4.1)

Observe that for each fixed p, the function Tp:F3

(d,αi,α 2 , ..)

\cij\ G (0, oo) j=p+l

is continuous on F . Since for every β G F, l i m ^ - ^ Tp(β) = 0 and Tp+\ < Tp, the convergence is uniform on any compact subset of F. I Denote ΛX\d

gp{λ,β) = k=0

PROPOSITION 4.1 Suppose that the assumptions of Theorem 4-1 hold. Then, as n -» oo (and so p —> oo)j with probability one,

0(λ,/3°)

sup Denoting Ap(λ,β)

PROOF.

iXk

= Σ{=0 ake ,

dλ

A{λ,β)

(4.3)

0. αfe

= ΣT=o Ofce ,

= ζfj(λ,β°)(g(X,β))-1dλ, observe that 2

Gp(β)-σ (β) = jΓ

—e

d\

(4.4)

BHANSALI AND KOKOSZKA

134

iλ

{\Ap(X,β)\2 - \A(λ,β)\2} Jn(λ) 1-e

2d

dλ

\A(λ,β)\2 L(λ) - ^ By Lemma 4.2 below, the last term in (4.4) tends to zero uniformly on E with probability one. Thus it remains to verify that with probability one,

l^{\A(X,β)\2-\Ap(λ,β)\2}ln(X)

sup

1-e

iλ

2d

dX

0.

(4.5)

Note that (4.6) j=p+l

and ιλ sup l-e \

< oo.

(4.7)

/3GE

Observe also that since every linear process is ergodic, we have with probability one,

Γ In(X)dλ = - ΣX2

-> EX2.

(4.8)

Relation (4.5) now follows from (4.6) combined with (4.1), (4.7) and (4.8).

LEMMA 4.2 Suppose that the assumptions of Theorem 4-1 hold, andu(λ^β) is a function continuous on [—π,π] x E. Then, with probability one,

sup Γ u(\,β)In{\)d\ -£- Γ u(Kβ)g(\β°)dλ

/36E |«/-π

^7Γ

0.

J-π

PROOF. This is essentially a restatement of Lemma 1 of Fox and Taqqu (1986), the only difference being that in our setting Gaussianity is not assumed. However, since {Xt} is linear, it is also ergodic and the proof of Lemma 1 of Hannan (1973) applies. • PROOF OF THEOREM

4.1: The proof relies on the fact that for any β φ β° (4.9)

Relation (4.9) follows immediately from the inequality

/

^(λ,/3i) (\ / 3 \ ^ ^ ^ π '

whenever /3X ^ /3 2 ,

(4.10)

135

ESTIMATION OF LONG MEMORY

see Lemma 3.1 of Kokoszka and Taqqu (1999). For ARMA spectral densities, relation (4.10) is the content of Lemma 10.8.1 of Brockwell and Davis (1991) which was extended to fractional ARIMA models in Lemma 2.1 of Kokoszka and Taqqu (1996b). In our setting, (4.10) follows, for example, from Lemma 3.1 of Kokoszka and Taqqu (1999). In the sequel, all random quantities are evaluated at a fixed elementary event from the set on which (4.3) holds. Suppose ad absurdum that βpn

does not converge to β°. Since E is

compact, there is a subsequence \βp(m\m \ of {βp(n),n \ > denoted for brevity 2

by {/3r}, such that βr -> β' φ β°. By (4.3), lim Gr{βr) = σ (β'). On the 2 other hand, Gr(βr) < Gr(β°), so again by (4.3) limsup Gr(βr) < σ (β°). 2 2 Thus, σ (β') < σ (/3°), which contradicts (4.9). •

5

Simulations

We illustrate the efficacy of the FAR method of estimating the memory parameter, d, in situations where the short-memory process, Y^, is not necessarily purely random, by applying this method to simulated realizations of several different ARFIMA(p, d, q) processes, with both p and q taking varying values over the region 0 < p < 2, 0 < q <2. Our principal aim is to apply this method in plausible realistic conditions rather than to investigate the validity of the asymptotic consistency result proved in Section 4 with finite values of n. Hence, although the estimator (3.8) and the proof of its consistency are based on the Whittle approximation to the likelihood function, in our simulation study we use the S+ function arima.fracdiff to compute the estimate of d and of the associated AR parameters. The main reason why we opted to use the S+ software package is that it is widely available, and this in turn should help towards ensuring the reproducibility of the simulation results. The estimation procedure based on the Whittle likelihood and the algorithm used in the S+ package both seek to provide estimates which are equivalent to the exact maximum likelihood estimates, but without being as expensive computationally. Hence, these two and the related (Kunsch, 1987) maximum likelihood procedures may also be expected to be asymptotically equivalent to each other, even though with a finite n they could produce substantially different estimates. A rigorous proof this result is however so far not available, though for an ARFIMA(p, d, q) process with the order (p, q) treated as known, Beran (1995) has shown that the estimates obtained from an alternative autoregressive approximation to the likelihood function have the same asymptotic distribution as those obtained by minimising the Whittle likelihood function. Secondly, as discussed ealier in Section 4, we select the order, p, of the

136

BHANSALI AND

KOKOSZKA

FAR model to fit by an order selection criterion, and thus in our simulations we do not treat p as a determinstic function of n. The model selection procedures used are described below for completeness: FAR AIC: For each p = 0,1,2,3,4,5, the parameters of an FAR(p, d) model were estimated by calling the S+ function arima.fracdiff and the corresponding value of the Akaike information criterion, AIC, was computed using the formula AIC(p) = -2loglik + 2p,

(5.1)

where loglik is the logarithm of the Gaussian maximum likelihood for each model, returned by the S+ function arima.fracdiff λ. Then, the adopted estimate of d was that given by the function arima. f racdif f for an FARQ3, d) model, with p being the value of p minimizing (5.1). FAR BIC: The procedure used was basically the same as that described above, but instead of the AIC criterion, (5.1), the order of the FAR model was selected by the following BIC criterion: BIC(p) = -2loglik + (1 + lnn)p

(5.2)

where n is the length of the series. It should be noted that the definition of the BIC criterion given in (5.2) above is consistent with a Bayesian modification of the AIC introduced by Akaike (1978), but uses an approximation for this criterion suggested by Priestley (1981, p. 375). This definition is however slightly different from an alternative Bayesian criterion introduced by Schwarz (1978), see also Hannan (1980), in whose approach the second term to the right of 5.2 is replaced by As discussed in Section 2, for the class of FAR(p, d) models, with a finite but unknown p, Beran et al. (1998) have studied the asymptotic behaviour of the order selected by the AIC and BIC criteria introduced above and shown that whereas the AIC criterion does not provided a consistent estimator of the order, the BIC criterion does do so. Moreover, by their Theorem 2, both versions of BIC discussed above provide consistent order selection for an FAR(p, d) process with a finite p. The penalty for introducing an additional parameter is, on othe other hand, always greater by 1 in 5.2; thus, for n = 1000, this penalty is 7.91 while for the Schwarz criterion it is 6.91, and for n = 500 the values of these penalty terms are 7.21 and 6.21, respectively. lf

Γhe value loglik of arima.fracdiff is actually the maximized log-likelihood, not just a quantity proportional to it as indicated in the S+ help file. Thus the loglik of arima.fracdiff is different from the loglik of arima.mle which is —2x Gaussian maximum log-likelihood.

ESTIMATION OF LONG MEMORY

137

In our simulation study, we examine the extent to which either of these two criteria are useful for selecting the value of the autoregressive order, p, to use for implementing the FAR approach discussed in this paper. We also compare the relative performance of the FAR estimate of d with the estimates obtained by adopting some of the alternative methods discussed in Section 2. Our aim, here, is to compare the overall performance of the different estimators over a wide range of models, rather than point out which estimators are best for certain models. We first describe the methods that are compared. 1. Periodogram: This method, as discussed earlier in Section 2, is based on the observation that for a long memory series {Xt}, /(λ) ~ Const|λ|~ 2d , as λ —>- 0, where the periodogram I(λ) is defined by (3.9). Thus, for low frequencies the log-log plot of J(λ) versus λ should follow a straight line with slope — 2d. Following Taqqu and Teverovsky (1998) we used the lowest 10% of the frequencies to fit the regression line. Our own simulations showed that for models with moderate AR and MA coefficients using various frequency bands between 3% and 12% does not significantly affect the estimates. 2. Semiparametric: This method has been rigorously investigated by Robinson (1995b), who assumes that the spectral density /(λ) ~ G(d)|λ|~ 2 d , as λ -» 0. As the approximate Gaussian likelihood is maximized in a neighbourhood of the zero frequency, it is also known as, see Taqqu and Teverovsky (1998), Kύnsch (1987), "local Whittle" . As with the periodogram method, however, it is not clear how to choose the optimal frequency band over which the local likelihood function is maximized. Following the recommendation of Taqqu and Teverovsky (1998), we used the lowest 1/32 of all frequencies. Our own limited simulation study showed that the results remain essentially the same for any band from 1/50 to 1/20 of all frequencies. 3. ARFIMA(l,d,l): Here d was estimated by approximate Gaussian maximum likelihood and assuming that the simulated time series truly follows a fixed ARFIMA(l,d,l) process, whether or not the undelying simulated model was of this particular form. The maximization was carried out using the S+ function arima.fracdiff. 4. ARFIMA(2,d,2): Same as above, but ARFIMA(2,d,2) model was fitted. 5. ARFIMA AIC: In this method we considered ARFIMA(p, d, q) models for p, q = 0,1,2. The model for which d was estimated was selected

138

BHANSALI AND

KOKOSZKA

by minimizing the following AIC criterion: AIC{p, q) = -2loglik + 2{p + q).

(5.3)

For each value of the order p and q, the estimation was carried out using the S+ function arima.fracdiff. 6. ARFIMA BIC: Same as above, but with AIC(p,q) replaced by BIC{p, q) = -2loglik + (1 + In n)(p + q).

(5.4)

The periodogram and semiparametric methods were chosen for comparison because the extensive simulation study of Taqqu and Teverovsky (1998) demonstrated that these two methods are more robust and accurate than any other non-parametric method considered by them. The ARFIMA(1, d, 1) and ARFIMA(2, d, 2) methods were considered for two reasons: first, for models VI and VII described below their use corresponds to fitting the true generating processes respectively and thus for these two models, the simulation results should throw some light on possible effects of model selection on the estimation of d. Secondly, for the other six simulated models, their use corresponds to under- or over-fitting the generated process and the simulation results may again be expected to provide some information about how misspecification of the short memory model structure in this way influences the estimation of d. The ARFIMA AIC and ARFIMA BIC methods were considered in order to see if anything is lost or gained by fitting only fractional autoregressive models. We next describe the long memory models used in our study. Recall that d Yt = (1 — B) Xt denotes the fractionally differenced process and {Zt} is the noise sequence. The simulated process is {Xt}' I ARFIMA(0,d,0): Yt = Zt. II ARFIMA(l,d,O): Yt = .5Yt-ι + Zt. III ARFIMA(2,d,0): Yt = -.5Yt-ι

- .25Yt_2 + Zt.

IV ARFIMA(2,d,0): Yt = .5Yt-ι - .25Y*_2 + Zt. V ARFIMA(0,d, 1): Yt = Zt + .5Z t _i. VI ARFIMA(l,d, 1): Yt - .5Yt_i + Zt + .5Zt-i. VII ARFIMA(0,d,2): Yt = Zt-

.5Zt_i + .25Z*_2.

VIII ARFIMA(2,d,2):Yt = ,5Yt-ι - .25Yt-2 + Zt + .5Zt-i -

ESTIMATION OF LONG MEMORY

139

Note that all models have moderate AR and MA coefficients so as not to favour a priori any of the model fitting methods. The models were decided upon before any simulations were done. For comparing the behaviour of the estimates under Assumption 2A, the Zt were simulated as independent Gaussian deviates, each with mean 0 and variance 1, using the S+ function arima.fracdiff .sim. Only one value of n, namely n — 1000 is considered and the number of simulations for each (model, method) configuration was 250. The nominal value of d was set to equal 0.3. The simulation results for the Gaussian innovations are shown in Table 1, where for each cell corresponding to each (model, method) configuration, the bias, standard deviation, and the square root of the mean squared error are shown. For convenience, a summary of the simulation results is also given in Table 2, where the average squared roots of the mean squared errors averaged over all models and separately for only the fractional autoregressive, FAR, models are shown. As our simulation study is restricted to only the class of ARFIMA models the two periodogram-based non-parametric and semiparametric methods perform rather poorly: for all the simulated models the magnitudes of their biases and standard deviations are much larger than for other methods. Consider next the method of fitting either a fixed order ARFIMA(2,d,2) or ARFIMA(l,d,l) model. For Model VII the former coincides with the actual generated model and it performs particularly well, with a similar remark applying to the method of fitting a fixed order ARFIMA(l,d,l) for Model II. However, for all other models possible effects of misspecifying the short memory model on the estimation of d may be gleaned from our simulation results. Thus the fitting of an ARFIMA(2,d,2) model to any of the models I-VII tantamounts to fitting a model with too many parameters. The main effect of this overfitting as compared with the fitting of an FAR model with the order selected by the BIC criterion is seen to be an increase in the simulated variance; moreover the bias is also much larger for all models except Model VI. For Model VII, on the other hand, fitting a fixed ARFIMA(l,d,l) corresponds to fitting a model with too few parameters and the main effect of this underfitting is seen to be an increase in the bias though the variance is smaller than when the correct ARFIMA(2,d,2) is fitted. A possible explanation of these results is that when a model with too many parameters is fitted, the additional parameters attempt to model the long memory component, effectively altering the generated value of d. At the same time, the variance in estimating d increases because of the excess variability introduced by the estimation of the redundant parameters. It

BHANSALI AND KOKOSZKA

140

T A B L E 1. COMPARISON OF THE ESTIMATE OF d PROVIDED BY DIFFERENT METHODS IN 2 5 0 SIMULATIONS OF VARIOUS GAUSSIAN MODELS.

Method Periodogram

Semiparametric

ARFIMA(l,d,l)

ARFIMA(2,d,2)

FAR AIC

FAR BIC

ARFIMA AIC

ARFIMA BIC

a) b) c) a) b) c) a) b) c) a) b) c) a) b) c) a) b) c) a) b) c) a) b) c)

Model I II III IV V VI VII VIII .066 .070 .065 .084 .067 .089 .053 .059 .206 .192 .179 .208 .187 .192 .199 .194 .204 .188 .224 .206 .203 .216 .199 .212 -.031 .334 -.306 .267 .091 .494 -.252 .474 .022 .062 .028 .026 .023 .027 .028 .036 .495 .260 .475 .038 .340 .307 .268 .094 .078 -.170 -.078 -.055 .004 -.126 -.018 -.056 .094 .088 .097 .049 .047 .048 .037 .047 .049 .134 .051 .109 .086 .176 .118 .112 -.083 -.104 -.035 -.036 -.126 -.106 -.063 -.034 .074 .125 .116 .103 .065 .120 .122 .059 .177 .146 .160 .069 .082 .157 .121 .073 -.028 -.061 -.037 -.043 -.072 -.087 -.051 -083 .122 .106 .071 .107 .064 .081 .096 .075 .076 .107 .042 .150 .118 .112 .058 .120 -.005 -.054 -.019 -.021 -.039 -.064 .019 -.063 .027 .092 .037 .054 .101 .103 .100 .074 .027 .107 .042 .108 .121 .102 .058 .097 -.021 -.071 -.025 -.040 -.029 -.067 -.037 -.045 .114 .046 .070 .063 .032 .111 .090 .076 .073 .135 .052 .075 .043 .130 .097 .088 -.011 -.070 -.009 -.032 -.015 -.041 -.016 -.090 .034 .092 .045 .051 .033 .087 .051 .078 .036 .117 .046 .060 .036 .096 .053 .119

a) BIAS = SIMULATED MEAN - 0.3, b) STANDARD DEVIATION,

ESTIMATION OF LONG MEMORY

141

TABLE 2. AVERAGE SQUARE ROOTS OF MEAN SQUARE ERRORS OF VARIOUS ESTIMATES OF D IN ALL GAUSSIAN MODELS AND F A R MODELS IN INCREASING ORDER.

Method FAR BIC ARFIMA BIC FAR AIC ARFIMA AIC ARFIMA(l,d,l) ARFIMA(2,d,2) Periodogram Semiparametric

FAR Models .059 .065 .071 .083 .103 .114 .205 .238

Method ARFIMA BIC FAR BIC ARFIMA AIC FAR AIC

ARFIMA(l,d,l) ARFIMA(2,d,2) Periodogram Semiparametric

All

Models .070 .081 .087 .098 .104 .123 .207 .293

should be noted that the situation here is slightly different from when a pure short memory is being overfitted in which case the estimation of additional parameters increases the variance but does not unduly influence the bias in estimating the non-zero parameters. On the other hand, when a model with too few parameters is fitted, the short memory component is not adequately modelled and this introduces bias in the estimation of d because the spectral density of the generated process is being approximated by the spectral density of the underparametrized model. The variance in estimating d is, however, reduced because fewer parameters are estimated. Consider now the method of fitting a fractional autoregressive model proposed in this paper and its relative behaviour in comparison with the fitting of ARFIMA models, with order selected by the AIC or the BIC criterion. The method of fitting a fractional autoregressive model is seen to provide good results for all models, but especially for models I-IV, where the generated model for {Yf} is a finite autoregression. It should be noted, however, that the ARFIMA BIC method has a smaller mean squared error for models VI-VII, all of which have q > 0. This finding is probably not surprising because if the selected model coincides with the generated model the resulting estimate of d is known to be asymptotically efficient. It should be emphasised, nevertheless, that the full ARFIMA models were fitted only up to order 2 and the chance of selecting an incorrect model is quite small in our simulations. As regards the question of whether the AIC or BIC criterion should be used for implementing the FAR method suggested in this paper, the simulation results appear to favour the latter, probably because

142

BHANSALI AND

KOKOSZKA

the AIC criterion is known to frequently select an "overparametrized" model resulting in a large mean square error as explained above when discussing possible effects of overfitting the simulated models. Even though the main goal of the present simulation study is to examine the overall performance of the estimators for typical stationary models with moderate AR and MA coefficients, it is of interest to see how the methods perform for nearly non-stationary or non-invertible models, and thus to examine the limits of their applicability. We considered the following models for achieving this objective. 1) Almost unit root AR(1) models of the form Yt = φYt_x+Zu with φ = .9, .95, .99, 2) almost unit root MA(1) models of the form Yt = Zt + ΘZt-ι with θ = .9, .95, .99. Our findings can be summarized as follows. Focusing first on AR models, the periodogram and semiparametric methods fail. The FAR BIC method by contrast gives estimates almost as good as for AR processes with moderate coefficients considered in Table 1. By contrast, for the almost unit root MA(1) models, the performance of the FAR BIC method is worse than that for the models considered in Table 1, and, also, as compared with the ARFIMA BIC method. A main reason why the performance of the ARFIMA(1, d, 1) method is noticeably better for this class of models than that of all other methods is that the ARFIMA(l,d, 1) model just includes the simulated ARFIMA(0,d, 1) and ARFIMA(l,d,O) models as its special case and yet avoids the parameter identification difficulties (Hannan (1970), p. 388) associated with fitting ARFIMA(p,d,g) models with p > l,g > 1. A detailed analysis of the simulation results for MA(1) models revealed that the ARFIMA(l,c/, 1) method yields estimates of the autoregressive and moving average coefficients which are within 0.1 of the correct values 0 and 0, respectively, and thus its estimate of d is based on an almost MA(1) short memory component. To illustrate the relative behaviour of the estimators in the infinite variance setting, we simulated symmetric α-stable innovations Zt with a = 1.5 and unit scale parameter. The nominal value of d is .2, which lies approximately in the middle of the stationary invertible range 0 < d < 1 — I/a specified in Assumption 3. Figures 1-4 show histograms of the estimated values of d in 50 replications of models I, II, V and VI. Only six methods were considered, namely, Periodogram, ARFIMA(l,d, 1), FAR AIC, FAR BIC, ARFIMA AIC and ARFIMA BIC. For all four models, the periodogram method provides a biased as well as a highly dispersed estimate of d, a finding that accords with the results reported above in Tables 1 and 2 for the the Gaussian case. Somewhat surprisingly, however, a similar comment applies to the ARFIMA BIC method, which, unlike the Gaussian case, now provides a biased estimator for all four models, and for models V and VI the estimator is also highly dispersed. A

ESTIMATION OF LONG MEMORY

143

Figure 1: FIGURE 1. Histograms of the estimated values of d for model I (ARFIMA(0,d,0)) with d = 0.2 and stable innovations. Periodogram

FARIMA(1,d,1)

I

FAR AIC

I 0.0

0.1

0.2

0.3

0.2

0.3

0.4

0.5

0.0

0.1

FARIMA AIC

FAR BIC

0.0

0.1

0.4

0.5

0.2

0.3

0.4

0.5

O.4

0.5

FARIMA BIC

0.4

0.5

0.0

0.1

0.2

I 0.3

plausible explanation for this behaviour is not easily given, but the simulations appear to indicate that the question of model selection for an α-stable ARFIMA process requires further investigation and that a naive use of the BIC criterion (5.4) may not be recommended in a situation where "outliers" may be present. The simulation results for the FAR BIC method, by contrast, broadly support the asymptotic consistency property established in Theorem 4.1. For models I and VI, in particular, the histogram is centred around the actual generated value of d = 0.2 with a relatively small dispersion around this value. For model V, however, the estimate is biased, probably reflecting the difficulty of estimating d for processes with a short memory MA component. In conclusion, the simulations indicate that when the long-memory process generating the observed series can be well approximated by a fractional autoregressive process, the method proposed in this paper provides a good estimator of d even when the order of the generating process, finite or infinite, is unknown, and in this sense it is "non-parametric". The method, moreover, compares favourably with the periodogram based non-parametric methods, which in this situation tend to have a larger variance and often a greater bias.

BHANSALI AND KOKOSZKA

144

Figure 2: FIGURE 2. Histograms of the estimated values of d for model II (ARFIMA(l,d,O)) with d - 0.2 and stable innovations. Periodogram

0.0

0.1

0.2

0.3

0.4

0.5

FAR BIC

0.2

6

0.3

FAR AIC

FARIMA(1,d,1)

_ι I. 0.0

0.1

0.2

0.3

0.4

0.5

0.0

J I. 0.1

0.4

0.5

L.

0.2

0.3

0.4

0.5

0.4

0.5

FARIMA BIC

FARIMA AIC

0.0

0.1

0.2

0.3

Data Example

We consider the Ethernet traffic data studied by Leland et al (1994), Willinger et al (1995), Taqqu and Teverovsky (1997), among others. The data, described in detail by Leland et al (1994), were collected between August 1989 and February 1992 at the Bellcore Morristown Research and Engineering Center and represent the number of bytes per 10 milliseconds passing through a monitoring system during a "normal traffic hour" in August 1989. The periodogram of the data has a number of very sharp peaks at non-zero frequencies suggesting that the data may not necessarily follow an ARFIMA model. Taqqu and Teverovsky (1997) used a graphical analysis of the periodogram and the semi-parametric method to infer that the true value of d lies between 0.31 and 0.35. We consider here 18 consecutive 200 second long time periods making up the "normal traffic hour". Thus, we consider 18 extremely long series, each consisting of 20,000 observations. Because of the self-similarity property, the long memory parameter for each of the 18 time series must be the same, and the estimates should lie in the range between .31 and .35. Figure 5. shows the estimates obtained using the FAR BIC, ARFIMA BIC, and Semiparametric

ESTIMATION OF LONG MEMORY

145

Figure 3: FIGURE 3. Histograms of the estimated values of d for model V (ARFIMA(O,d,l)) with d = 0.2 and stable innovations. Periodogram

0.0

0.1

0.2

O.3

FARIMA(1,d.1)

O.4

0.5

.Hi..

0.0

0.1

FAR BIC

0.0

0.1

0.2

0.3

FAR AIC

0.2

0.3

0.0

0.1

FARIMA AIC

0.4

0.5

0.0

-

0.1

0.2

0.3

0.2

0.3

0.4

0.5

0.4

0.5

FARIMA BIC

0.4

0.5

0.0

0.1

0.2

0.3

methods. Following the analysis of Taqqu and Teverovsky (1997), we used for the semiparametric method the lowest 1/128 of all frequencies to get clear of the peaks at non-zero frequencies. The estimates obtained using the FAR BIC and ARFIMA BIC methods appear to be more stable over time than those obtained using the semiparametric method. It is possible that the intensity of long range dependence decreased in the 7th period and increased in the 8th period, but the semiparametric estimates probably overestimate the magnitude of the change. Acknowledgements. The software used in Section 5 for the periodogram and semiparametric methods and for simulating α-stable ARFIMA series was kindly made available to us by Murad Taqqu and Vadim Teverovsky, who also gave us plentiful advice on how to use it. Our colleague Simon Fear has invariably been willing to help us deal with intricacies of WΓ^X. and S+. We also thank Jan Beran for a discussion which stimulated the present research and Murad Taqqu for providing us with the Ethernet data studied in Section 6. Clifford Hurvich, Carenne Ludeήa, Adrian Raftery and Gennady Samorodnitsky also offered valuable comments.

BHANSALI AND KOKOSZKA

146

Figure 4: FIGURE 4. Histograms of the estimated values of d for model VI (ARFIMA(l,d,l)) with d = 0.2 and stable innovations. FARIMA(1,d,1)

Periodogram

0.0

0.1

0.2

0.3

0.4

0.0

0.5

0.1

0.0

0.1

0.2

0.2

0.3

0.4

0.2

0.5

0.3

0.4

0.3

0.4

0.5

0.4

O.S

FARIMA BIC

FARIMA AIC

FAR BIC

0.5

0.0

0.1

0.2

0.3

Figure 5: FIGURE 5. Estimated long memory parameter for 18 consecutive Ethernet data series by various methods. FAR BIC: continuous, ARFIMA BIC: dotted, Semiparametric: dashed.

Y

,- \ ••

:

Λ. *

\\ / /

^

5

'

\

//••»'

"ΪJi ψj 1O

V •••. \

\

OS

\ "\ V

V

\

15

Series Number

References Agiakloglou, C, Newbold, P. and Wohar, M. (1993). Bias in an estimator of the

ESTIMATION OF LONG MEMORY

147

fractional difference parameter. Journal of Time Series Analisis, 14, 235-246. Akaike, H. (1978). A Bayesian analysis of the minimum AIC procedure. Ann. Inst. Statist. Math., 30 A, 9-14. Andel, J. (1986). Long memory time series models. Kybernetika, 22, 105-123. Bardet, J-M., Moulines, E. and Soulier, P. (1999). Recent advances on the semiparametric estimation of the long-range dependence coefficient. In ESAIM Proceedings, pp. 23-43. Societe de Mathematiques Appliquees et Industrielles. Beran, J. (1992). Statistical methods for data with long-range dependence. Statistical Science, 7, number 4, 404-416; With discussions and rejoinder, pages 404-427. Beran, J. (1994). Statistics for long-memory processes. Chapman & Hall, New York. Beran, J. (1995). Maximum likelihood estimation of the differencing parameter for invertible short and long memory autoregressive integrated moving average models. J. Royal Statist. Soc. B, 57, 659-673. Beran, J. (1997). Discussion of 'Heavy tail modelling and teletraffic data'. The Annals of Statistics, 25, 1852-1856. Beran, J., Bhansali, R. J. and Ocker, D. (1998). On unified model selection for stationary and nonstationary short- and long-memory autoregressive processes. Biometrika, 85, 921-934. Berk, K. N. (1974). Consistent autoregressive spectral estimates. The Annals of Statistics, 2, 489-502. Bhansali, R. J. (1978). Linear prediction by autoregressive model fitting in the time domain. The Annals of Statistics, 6, 224-231. Bhansali, R. J. (1980). Autoregressive and window estimates of the inverse correlation function. Biometrika, 67, 551-566. Bloomfield, P. (1973). An exponential model for the spectrum of a scalar time series. Biometrika, 60, 217-226. Box, G. E. P. and Jenkins, G. M. (1970). Time series analysis; forecasting and control. Holden Day, New York. Brockwell, P. J. and Davis, R. A. (1991). Time Series: Theory and Methods. Springer-Verlag, New York. Chan, N. H. and Palma, W. (1998). State space modeling of long-memory processes. The Annals of Statistics, 26, 719-740. Crato, N. and Ray, B. K. (1996). ModeΓselection and forecasting of long-range dependent processes: results of a simulation study. Journal of Forecasting, 15, 107-125. Dahlhaus, R. (1989). Efficient parameter estimation for self similar processes. Ann. Stat, 17, number 4, 1749-1766.

148

BHANSALI AND KOKOSZKA

Fox, R. and Taqqu, M. S. (1986). Large-sample properties of parameter estimates for strongly dependent stationary Gaussian time series. The Annals of Statistics, 14, 517-532. Geweke, J. and Porter-Hudak, S. (1983). The estimation and application of long memory time series models. Journal of Time Series Analysis, 4, 221-238. Giraitis, L., Robinson, P. and Surgailis, D. (1999). Variance-type estimation of long memory. Stochastic Processes and their Applications, 80, 1-24. Giraitis, L. and Surgailis, D. (1990). CLT for quadratic forms in strongly dependent linear variables and application to asymptotical normality of Whittle's estimate. Prob. Th. Rel Fields, 86, 87-104. Granger, C. W. J. and Joyeux, R. (1980). An introduction to long-memory time series and fractional differencing. J. Time Series Anal, 1, 15-30. Hall, P. (1997). Defining and measuring long-range dependence. Fields Institute Communications, 11, 153-160. Hannan, E. J. (1970). Multiple Time Series. Wiley, New York. Hannan, E. J. (1973). The asymptotic theory of linear time series models. J. Appl. Prob., 10, 130-145. Hannan, E. J. (1980). The estimation of the order of an ARMA process. The Annals of Statistics, 8, 1071-1081. Hannan, E. J. and Quinn, B. G. (1979). The determination of the order of an autoregression. J. Royal Statist. Soc. B, 41, 190-195. Haslett, J. and Raftery, A. E. (1989). Space-time modelling with long-memory dependence: assesing Ireland's wind power resource. Appl. Statist, 38, number 1, 1-50. Heyde, C. C. and Yang, Y. (1997). On defining long-range dependence. J. Appl. Probab., 34, 939-944. Hosking, J. R. M. (1981). Fractional differencing. Biometrika, 68, number 1, 165-176. Hosking, J. R. M. (1984). Modeling persistence in hydrological time series using fractional differencing. Water Resources Research, 20, number 12, 1898-1908. Hurvich, C. M. and Beltrao, K. I. (1993). Asymptotics for the low-frequency ordinates of the periodogram of a long-memory time series. Journal of Time Series Analysis, 14, 455-472. Hurvich, C. M. and Brodsky, J. (2001). Broadband semiparametric estimation of the memory parameter of a long-memory time series using fractional exponential models. Journal of Time Series Analysis, 22, 221-249. Hurvich, C. M., Deo, R. and Brodsky, J. (1998). The mean squared error of Geweke and Porter-Hudak's estimator of the memory parameter of a long memory time series. Journal of Time Series Analysis, 19, 19-46.

ESTIMATION OF LONG MEMORY

149

Kokoszka, P. S. (1996). Prediction of infinite variance fractional ARIMA. Probability and Mathematical Statistics, 16/1, 65-83. Kokoszka, P. S. and Taqqu, M. S. (1995). Fractional ARIMA with stable innovations. Stochastic Processes and their Applications, 60, 19-47. Kokoszka, P. S. and Taqqu, M. S. (1996a). Infinite variance stable moving averages with long memory. Journal of Econometrics, 73, 79-99. Kokoszka, P. S. and Taqqu, M. S. (1996b). Parameter estimation for infinite variance fractional ARIMA. The Annals of Statistics, 24, 1880-1913. Kokoszka, P. S. and Taqqu, M. S. (1999). Discrete time parametric models with long memory and infinite variance. Mathematical and Computer Modelling, 29, 203-215. Kύnsch, H. (1987). Statistical aspects of self-similar processes. Bernoulli, 1, 67-74. Leland, W. E., Taqqu, M. S., Willinger, W. and Wilson, D.V. (1994). On the selfsimilar nature of Ethernet traffic (extended version). IEEE/ACM Transactions on Networking, 2, 1-15. Mikosch, T., Gadrich, T., Klύppelberg, C. and Adler, R. J. (1995). Parameter estimation for ARM A models with infinite variance innovations. The Annals of Statistics, 23, 305-326. Moulines, E. and Soulier, P. (1999). Broad band log-periodogram regression of time series with long range dependence. The Annals of Statistics, 27, 1415-1439. Pai, J. S. and Ravishanker, N. (1996). Bayesian modelling of arfima processes by markov chain monte-carlo methods. Journal of Forecasting, 15, 63-82. Pai, J. S. and Ravishanker, N. (1998). Bayesian analysis of autoregressive fractionally integrated moving average. Journal of Time Series Analysis, 19, 99-102. Parzen, E. (1969). Multiple time series modelling. In Multivariate Analysis II, New York (ed. P. R. Krishnaiah). Academic Press. Priestley, M. B. (1981). Spectral Analysis and Time Series: Volume 1. Academic Press. Ravishanker, N. and Ray, B. K. (1997). Bayesian analysis of vector arfima processes. Austral J. Statist, 39, 295-311. Robinson, P. M. (1995a). Log-periodogram regression of time series with long range dependence. The Annals of Statistics, 23, 1048-1072. Robinson, P. M. (1995b). Gaussian semiparametric estimation of long range dependence. The Annals of Statistics, 23, 1630-1661. Samorodnitsky, G. and Taqqu, M. S. (1994). Stable Non-Gaussian Random Processes: Stochastic Models with Infinite Variance. Chapman & Hall. Schwarz, G. (1978). Estimating the dimension of the model. The Annals of Statistics, 6, 461-464. Shibata, R. (1980). Asymptotically efficient selection of the order of the model for estimating parameters of a linear process. The Annals of Statistics, 8, 147-164.

150

BHANSALI AND KOKOSZKA

Shibata, R. (1981). An optimal autoregressive spectral estimate. The Annals of Statistics, 9, 300-306. Sowell, F. B. (1992). Maximum likelihood estimation of stationary univariate fractionally integrated time series models. Journal of Econometrics, 53, 165188. Taqqu, M. S. and Teverovsky, V. (1997). Robustness of Whittle-type estimates for time series with long-range dependence. Stochastic Models, 13, 723-757. Taqqu, M. S. and Teverovsky, V. (1998). On estimating the intensity of long-range dependence in finite and infinite variance series. In A practical guide to heavy tails: Statistical techniques for analyzing heavy tailed distributions, Boston (eds R. Adler, R. Feldman and M. S. Taqqu), pp. 177-217. Birkhauser. Willinger, W., Taqqu, M. S., Leland, W. E. and Wilson, D.V. (1995). Self-similarity in high-speed packet traffic: analysis and modeling of Ethernet traffic measurements. Statistical Science, 10, 67-85.

151 Institute of Mathematical Statistics

LECTURE NOTES — MONOGRAPH SERIES

STABILITY OF NONLINEAR TIME SERIES: WHAT DOES NOISE HAVE TO DO WITH IT? Daren B.H. Cline and Huay-min H. Pu Department of Statistics Texas A&M University Abstract We survey results on the stability of various nonlinear time series, both parametric and nonparametric. The emphasis will be on identifying the role that the "error term" has in determining stability. The error term can indeed affect stability, even when additive and for simple, common parametric models. The stability of the time series is not necessarily the same as that of its related (noiseless) dynamical system. In particular, this means that care must be taken to ensure that estimates are actually within the valid parameter space when analyzing a nonlinear time series.

Key Words: ergodicity, Markov chain, nonlinear time series

1

Introduction

Fitting time series with nonlinear models has become increasingly popular, especially since the emergence of nonparametric function estimation methods (Collomb and Hardle (1986), Hardle and Vieu (1992), Chen and Tsay (1993a,b), TJ0stheim and Auestad (1994a,b), Masry and Tj0stheim (1995)). No matter what model is fit, a critical part of the estimation procedure is determining whether the model is stable or whether the parameters are within the appropriate parameter space (TJ0stheim (1994)). Additionally, knowing the stability properties of a particular model makes it possible to develop simulation and resampling procedures to be used for inference. For these procedures, as well as for the obvious questions of limit theorems and robustness, the nature of the noise (error) distribution is clearly a significant concern. What is not so clear, however, is how this distribution can affect — if it does at all — the stability question itself. Habit with

152

CLINE AND PU

linear models has made it seem as though the error distribution is essentially irrelevant for stability or, as in the case of bilinear and ARCH models, the magnitude of the error variance can appear to be all that is relevant. On the contrary, even when additive, noise often plays a large and critical role in determining the stability of many nonlinear time series models. In particular, the assumption that the process is stochastic is fundamental. In a sense, the issue is between taking a stochastic view of the reasons underlying a process and taking a deterministic view. In the stochastic view, noise is not just a nuisance of observation and inference, a proxy for the uncertainty of scientific investigation. Rather it is integral to the behavior of the process. Of course this is no surprise to those who study stochastic processes: the effect of noise on the values of the time series persists due to dependence. This persistent effect, in turn, affects the stability of the process itself. Determinism as a tenet of science is ingrained into us at our earliest experiences with the scientific method. Linear models, it turns out, naturally lend themselves to this point of view. One's objective as a scientist is to identify the hidden principle, to strip away the flesh (as it were) of noise and distracting factors and with Occam's razor lay bare the skeleton of a true mechanism for movement in the process. This is determinism. If instead we choose to view nature as stochastic (and we will not argue whether this is a wise choice) then we also recognize the biases of traditional determinism, and this includes our view of the stability of nonlinear time series. In this paper, then, our objective is to identify the role that noise plays in determining stability conditions for nonlinear time series. This will involve general approaches to the problem of stability, illustrated with specific examples. We will also directly compare stability conditions for time series with those of dynamical systems which are deterministic and can sometimes be thought of as noiseless "skeletons" of the process (Tong (1990)). For some examples the stability conditions coincide and for others they do not. As have most authors studying stability, starting with Priestley (1980), we take the approach of embedding the time series (say, {&}) in a suitable Markov chain referred to as the state space model and defining stability in terms of the ergodicity, null recurrence or transience of that chain. State space models may vary with the application and for our purposes they need not be observable. For example, an autoregressive type of process can be embedded in (1) for some order p, whereas a process that also has a moving average component of order q could use the state space model {(Xu Ut)} = {(&,..., ξt-p+uet,..., e t - 9 +i)}

(2)

153

NONLINEAR TS STABILITY

The time series is most suitably stable — and the properties of estimators behave best — when the Markov chain is geometrically ergodic (Nummelin (1984), Athreya and Pantula (1986), Chan (1989,1993a,b), Meyn and Tweedie (1992)), meaning the chain converges to its stationary distribution at a uniform geometric rate. Throughout the paper we set aside the questions of irreducibility and aperiodicity even though the errors can have a role in determining these properties as well. So unless we say otherwise, the stability conditions discussed below are to be taken in the context of a subspace on which the process is irreducible and/or with time lags according to the periodicity. Likewise, we assume continuity of the transitions in the sense of a T-chain (cf. Tuominen and Tweedie (1979), Meyn and Tweedie (1993), ClineandPu (1998)). In addition, there is a distinction between geometric ergodicity of a Markov chain and geometrically stable drift of a Markov chain. A chain with geometrically stable drift tends to decrease geometrically in magnitude when it becomes too large and it will satisfy a drift condition such as those in Theorems 1 and 2 below, but such drift is not necessary for geometric ergodicity. For most of the paper we will focus on geometrically stable drift as it most directly compares with the notion of geometric stability of a dynamical system skeleton (see (5)). In section 7, however, we return to this distinction in a discussion of the role that the noise distribution tails can play. The error sequence will be denoted {e*} and is assumed iid. It may or may not contribute additively to the time series. Sections 2 and 3 review stability of linear, bilinear and ARCH models, as well as stability of models that can be tied directly to their skeletons. Sections 4 and 5 go on to describe the standard uses of Foster-Lyapounov test functions with examples that behave like their skeletons and an example that does not. Section 6 presents a new approach to using such test functions to analyze models which either do not have skeletons or are characteristically different from their skeletons. Finally, in section 7 we discuss improvements possible when errors have sufϊicently light distribution tails. 2. Linear, Bilinear and ARCH Models. Example 1. Viewing stability deterministically works especially well for a linear model. When in reduced form, the usual linear ARMA(p, q) model & = aιξt-ι H + apξt-p + et + bιet-ι H h bqet-q (3) has (2) as its state space model and + ••• + apx*t_p

(4)

154

CLINE AND PU

as its skeleton. The skeleton, in other words, is the time series stripped of its noise terms. Here, it is a linear dynamical system. A dynamical system {xt} is defined to be geometrically stable when a bounded solution exists for each initial condition and there exist K < oo and p < 1 such that I*;I < K(l + p*||a;o||) for all t > 1 and x0 = (sj,.. -, xLp).

(5)

The system (4) is geometrically stable precisely when the solution is attracted to 0 regardless of the initial condition. An equivalent algebraic condition is that the eigenvalues of the so-called companion matrix,

( 10... 0 0 1 0, have maximum modulus less than 1. Obviously, geometric stability for (3) (more precisely, for (2)) is thus identical to geometric stability for (4). Example 2. The usual condition for stability of bilinear models also is essentially algebraic though it does depend on the error variance σ1. The model is expressed as it = a'Xt-ι + XUBUt-ι + et + dUt-u where et has zero mean, Xt-ι = (&_i,..., &- p ), Ut-\ = (et-ι,..., e t _ 9 ), B is a matrix and a and c are vectors. If B is subdiagonal, there are appropriately defined matrices A\, B\ such that if A\ ® A\ -j-σ2B\ ® B\ has spectral radius less than 1 then the state space process (e.g., (2)) is geometrically ergodic (Pham (1985,1986)). See Bhaskara Rao et al (1983), Guegan (1987), Liu and Brockwell (1988), Liu (1992) and Pham (1993) for a treatment with more general B. Weaker conditions may actually suffice (Liu (1992)). These also depend on the distribution of et, again principally through the scale parameter. Example 3. Combined autoregressive and autoregressive conditionally heteroscedastic (AR-ARCH) models such as it = a0 + aiit-i + -" + apξt~p + (bo + &i&2-i + 2

likewise have an algebraic condition depending on σ :

1

^

155

NONLINEAR TS STABILITY

(Tong (1981), Quinn (1982), Lu (1998b), cf. also Diebolt and Guegan (1993), Lu (1996,1998a), Borkovec (2000)). The relevant Markov chain is (1). Liu, Li and Li (1997) provided a similar algebraic condition for stability of a nonlinear AR-ARCH model with piecewise constant coefficient functions. We return to a generalization of this example in section 6. 3. Nonlinear AR(p) Models. A common nonlinear model is the autoregressive model with functional coefficients (FCAR), ξt = ao{Xt-ι) + αi(X t -i)6-i +

+ ap{Xt-i)ξt-P

+ c(et; X t -i),

(6)

where c(e*; x) has mean 0 for each x E W and the state space model (1) defines the embedding Markov chain. The coefficient functions αo(rr),..., ap(x) usually are assumed bounded. The error terms may be additive (c(et',x) = et) but more generally {c(et; x)} has some regularity condition such as being uniformly integrable across the choices of x. Self-exciting threshold (SETAR) models are special cases where the coefficient functions are piecewise constant, whereas threshold-like models only require the functions to be asymptotically piecewise constant, as ||α;|| -» oo. Examples of all of these have found successful application (cf. Tong (1990)). Nonparametric fitting of the FCAR model is now common (cf. Tj0stheim (1994), Chen and Hardle (1995), Hardle et al (1997)). The skeleton of (6) is the dynamical system, (7)

x*t = a{x*t_ι,...,x*t_p),

where a(x) =

CLQ(X)

+aι(x)xχ

H

\-ap{x)xp,

x = ( x i , . . . ,xp).

(8)

Explicit conditions for geometric stability of (7) are difficult to state, and not always known. Chan and Tong (1985) and Chan (1990) have shown, however, that if (7) is geometrically stable and a(x) is Lipschitz continuous then (6) is likewise geometrically stable. That is, {Xt} is a geometrically ergodic Markov chain satisfying a geometric drift condition. Cline and Pu (1999a, Thm. 3.1; 1999b, Thm. 2.5) have extended this result to the case a(x) is asymptotically Lipschitz as min^i,...^ \x{\ —>• oo, in other words, for x far away from the axial hyperplanes. It has also been extended to nonlinear ARMA models (Cline and Pu (1999c)). The condition is essentially deterministic as it depends only on the dynamical system and not on the noise. For the models where α(x) is sufficiently smooth, the condition is sharp and thus stability of the time series

156

CLINE AND PU

coincides with stability of the skeleton. This includes any threshold model where a(x) is piecewise linear and continuous. Unfortunately, the smoothness condition disallows the many threshold models for which the coefficient function is piecewise linear but not continuous. Example 4- An example where this approach gives good results is the simple SETAR(l) model ξt = (αoi + απ&-i)lξt-io + c(ξt-ι)et which has a piecewise linear and Lipschitz continuous autoregression function. Assuming c(x) is bounded, this process has geometrically stable drift if and only if agreeing exactly with the geometric stability of its skeleton (Petruccelli and Woolford (1984), Chan et al (1985), Guo and Petruccelli (1991)). This example and others naturally bring the following questions to mind: when are stability of the time series and stability of the skeleton equivalent? That is, when is stability determined independently of the errors? Is it usual for stability of the time series and stability of the skeleton to be equivalent or does the error distribution normally play a critical role?

4. The Foster-Lyapounov Drift Condition. The connection between stability of a stochastic process and stability of a dynamical system is not superficial, even if it is not always as simple as one might like. To verify that a dynamical system is stable, one approach is to show that within some finite time the system (or some appropriate function of it) is sure to "drift" toward an attracting set. (See (5), for example.) A simple condition to check this is known as Lyapounov's drift condition (La Salle (1976)): for some nonnegative function V, K < oo and compact set C

V(x*t) < V(xi_x) + Klc(xϊ-i)

f o r a11

*>

L

Foster (1953) likewise showed that if the mean transition of a Markov chain on the nonnegative integers was uniformly negative for large states then the chain is certain to drift toward the origin whenever it gets too large, and therefore is ergodic. The method was generalized by Tweedie (1975,1976,1983a), Popov (1977), Nummelin and Tuominen (1982), Meyn and Tweedie (1992) and others (cf. Nummelin (1984), Meyn and Tweedie (1993)) to what is now called the Foster-Lyapounov drift condition for ergodicity of an irreducible, aperiodic Markov chain {Xt}' for some function V taking values in [l,oo), K < oo and "small" set C,

E (y(Xt)

- V(X t _i) I Xt-ι =x)<

Klc(x)

- 1.

(9)

157

NONLINEAR TS STABILITY

(A set C is small if there exists m > 1 and measure v such that P{Xm G B \ Xo = x) > v(B) for all x 6 C and all B (cf. Nummelin (1984), Meyn and Tweedie (1993)). Typically, compact sets on an appropriately defined topological space are small.) The function V is then called a Foster-Lyapounov test function. Furthermore, a Markov chain (or more precisely, the sequence of distributions generated by the transition kernel from an initial distribution) is in fact a dynamical system on the space of probability distributions and thus (9) can be interpreted as an ordinary Lyapounov drift condition for that system. This is the connection with dynamical systems, therefore, to be exploited. Indeed, Meyn and Tweedie (1992,1993) have explored the depth of this concept, and especially for the stronger drift condition for geometric ergodicity: for some function V taking values in [l,oo), K < oo, p < 1 and small set C,

E (y(Xt) I Xt-i =x)< pV{x) + Klc(x).

(10)

This drift condition ensures, among other things, F-uniform ergodicity and a geometric rate of convergence of the marginal distributions to the stationary distribution (Nummelin (1984), Chan (1989), Meyn and Tweedie (1992)), a strong law of large numbers for ^ Σ?=i h(Xt) if \h{x)\ < V(x) (Meyn and Tweedie (1992)), and a central limit theorem for \ Σ? = i h(Xt) if (h(x))2 < V{x) (Meyn and Tweedie (1992), Chan (1993a,b)). If the test function satisfies ||x|| r < V(x) < M + if ||a;|| r for some finite K and M then (10) is a condition for geometrically stable drift of the chain. (See Theorem 1 in the next section.) Example 5. An example of the application of the drift condition (10) is to the FCAR model (6). Chan and Tong (1985, 1986) have shown that if £sup|ai(z)|
(11)

x

then (10) may be verified for a test function of the form V(x) = 1 + ΣPi=\ci\χi\- Chen and Tsay (1993a) prove geometric ergodicity with the same condition using a slightly different approach. Condition (11) depends on the FCAR representation but An and Huang (1996) have shown that the somewhat weaker limsup

|α(x)

! , <1 (12) max|:z| also suffices, using V(x) = l+maxi< p |a;i| and the m-step approach discussed in the next section. Unfortunately, not even (12) includes all geometrically stable linear models. For example, the AR(2) model ξt = αi£ £ +

158

CLINE AND PU

is stable if and only if max(|α2|, |αχ| + 02) < 1 but (12) holds only if |αχ| + |α 2 | < 1. 5. Approaches to Using Drift Tests. The benefit realized in applying a drift condition clearly relies on the nature of the test function. The choice of a test function has been described as something of an art and the functions are often constructed specifically for the model at hand. Test functions which are based on a simple kind of norm (e.g., V(x) = 1 + ||x|| r or V(x) = 1 + maxi
0 and λ is bounded and bounded away from 0. Typically λ is primarily a function of the direction of x rather than its magnitude. Optimally choosing λ is equivalent to optimally choosing Tj0stheim's m (see below). Furthermore, the geometric drift condition can be expressed in terms of the drift of the logarithm of V(Xt) (the log-drift condition). For the following theorems let Λ be the class of all nonnegative measurable functions on W that are bounded and bounded away from 0. Theorem 1 Assume {Xt} is an aperiodic, φ-irreducible T-chain in W such that E (||ΛΊ||7(1 + ||z|| r ) I XQ = x) is bounded for some r > 0. The following are equivalent conditions, each sufficient for {Xt} to be geometrically ergodic.

(i) ί m s u p | M | _ , o o g ( Λ ( ^ ) ) | | ^ J | r I Xo = x) < 1 for some λ e A, r > 0.

(it) l i m s u p H s i i ^ E ( i j p ί | X°

= x

) < l for

some

r>0,n>m>0.

(in) l i m s u P | N K o o E (log (δ + ^ j j ^ l " ) | *o = x) < 0 for some δ > 0, λeΛ. (iv) limsupiiaju^oo E (log {δ + l^\\χ^) m > 0.

Xo = x) < 0 for some

δ>0,n>

NONLINEAR TS STABILITY

159

Proof The equivalences follow from Cline and Pu (1999a, Lena. 4.1, Lem. 4.2) and the sufficiency for geometric ergodicity from condition (10) with test function V(x) = 1 + λ(z)||z|| r . D If the noise terms in a FCAR(p) model do not dominate when the time series is very large then they may be at least partly ignored, as the following theorem suggests. Theorem 2 Assume {Xt} is an aperiodic, φ-irreducible T-chain defined by r (1) and (6) such that αo,...,α p are bounded, sup|j a .|| < M 2?(|c(ei;£)| ) < oo r r for some r > 0 and all M < oo, and lim||x||_>ooi£ (|c(ei;x)| /||α:|| ) = 0. Let a be given by (8) and θ{x) — (a(x),xι,... ,xp-ι)/(l + ||a?||) for x = (xι,..., xp). The following are equivalent conditions, each sufficient for {Xt} to be geometrically ergodic. (i) U m s u P | W K o o £ ( ^

I Xo = x) \\θ(x)\\r < 1 for some r > 0, λ G Λ.

(it) l i m s u p | N K o o E (j[nj=m ||0(-Xj )IΓ \XQ = X) 0, n > m > 0: (in) l i m s u p | N K o o E (log (δ + ^ 1 | | 0 ( * ) | | ) | Xo = x) < 0 for some δ > 0, λΛ λ(EΛ. (iv) U m s u p | W K o o S ( Σ j = m log ( ί + 1 1 0 ( ^ ) 1 1 ) \Xo = x) 0, n > m > 0. Proof This essentially is Theorem 3.2 in Cline and Pu (1999a).

D

Similar theorems express equivalent conditions for transience of a Markov chain (e.g., Cline and Pu (2001, Thm. 2.2)). Example 6. We refer to conditions (iii,iv) in Theorem 1 and (iii,iv) in Theorem 2 as "log-drift" conditions. A model which illustrates the benefits of using a log-drift condition is the Periodic FCAR(l) model,

where the coefficient function a\(x) is periodic with period r. This is an unusual model but there is a suggestion of periodicity in the coefficient functions fitted by Chen and Tsay (1993a) for the sunspot number data, and it was at the urging of Prof. Chen that we studied the stability of this particular model. At any rate, it is a useful example. With n = m = 1, the condition in Theorem 2(iv) can be reexpressed as limsupE (log(ί + K(ξi)|) ξo = x) < 0 for some δ > 0.

(13)

160

CLINE AND PU

(This can also be determined from the 2-step condition with V(x) — 1 + |x|.) The function E (log(£ + |αχ(£i)|) | ξo = %) is close to being periodic in |αi(x)x|. Thus the left-hand side of (13) is only a limsup, not a limit, which has the unfortunate consequence that condition (13) is not sharp. The solution is to choose n = m — 2 instead. Assume a$x) is continuously differentiate with a derivative that is 0 on a set of measure 0. Then the function E (log(ί + |αi(ξ2)|) I ξo = x) does in fact have a limit and therefore does lead to a sharp condition, namely if

Γ log(|αi(u)|)*ι<0

(14)

JO

then {ξt} is geometrically ergodic and if f£ log(\aι(u)$du > 0 then {ξt} is transient (Cline and Pu (1999a, Thm. 3.4; 2001, Thm. 3.2)). Note that the skeleton process, xt = ao(xt-ι) + aι{xt-i)zt-i is geometrically stable if and only if sup|αi(α:)| < 1. Therefore, this is an example where stability of the time series does not coincide with stability of its skeleton. For a specific example, suppose a\{x) = c + dcos(x) with \c\ + \d\ > 1, M < |d| < 2. Then the time series is geometrically ergodic but the skeleton is not stable. Although condition (14) does not explicitly refer to the noise distribution, the noise does play a major role in determining the condition by causing the drift to be averaged. Example 7. The directional method, on the other hand, seems to work well with many threshold models of order 1, including ones that employ a delay. Consider, for example, the TAR(l) time series with delay d given by ξt = ao(Xt-ι) + aι(Xt-i)ξt-i + eu

Xt-ι = (ξt-u

>6-d

ao(x) bounded and a\{x) depends only on (sgn(xχ),... ,s x = (xi,..., Xd). There are thus 2d regions i?χ,..., R2d each corresponding to a coefficient: a\j = 01(1), x G Rj. Notice that the thresholds — boundaries of the regions — are the axial hyperplanes. As long as the time series remains large (which is all that is of concern for stability), ξt avoids the thresholds and hence Xt cycles among some subset of the regions. There may be several such subsets possible, depending on the signs of the αij's, and it is the drift of the "worst case" cycle that is critical. Thus the geometric drift condition for stability is exactly (Cline and Pu (1999b, Cor. 2.4))

max

161

NONLINEAR TS STABILITY

where the maximum is taken over the possible cycles described above. This corresponds precisely to the geometric stability of the skeleton. The test function for establishing this condition takes the directional form, V(x) = 1 4- A(rr)||a;||r, and the optimal choice for X(x) is constant on each of the 2d regions. 6. The Piggyback Method. To this point we have presented examples studied with what one might call the traditional methods of drift analysis. Not all models yield to this analysis, however, including some surprisingly simple models. In this section we present a new approach, as yet somewhat informal, that employs a much more sophisticated ra-step test function. We call it the piggyback method because it relies on finding a stable Markov chain similar to a process embedded in the one of interest and building a new test function on top of one that works for the known stable chain. We will first present a sketch of the piggyback method and then provide three examples. The sketch, however, is quite rough because in fact the method is applied somewhat differently for each of the examples. (See the papers referenced below.) Indeed, the concept is elegant but its application is messy and as yet we do not know how generally useful it may prove to be. The time series {ξt} is embedded in a Markov chain {Xt} At the same time we consider another Markov chain {Yt} similar to a simpler process embedded in {Xt}> The chain {Yt} is assumed to be geometrically ergodic and, in particular, to satisfy the geometric drift condition with test function V\{y). Its stationary distribution we denote G. If there is a function H(y) which somehow exemplifies (or bounds) the relative change in magnitude of X\ when XQ is large and YQ — y then, intuitively, the stationary value of H(Yt) will measure the geometric drift of {Xt}- Thus, a log-drift condition for geometric stability would be exp ( ίlog(ί + H(y))G(dy)\ < 1 for some δ > 0.

(15)

To obtain a test function that will yield such a condition, we first define

and let y(x) identify the "embedding" of Yt into Xt An integer m is chosen suitably large, a "correction" function c(x) constructed and the ultimate test function is (something like) l/m

V(x)=c(x)[l[E(h(Yj)\Y0 =

162

CLINE AND PU

The key point for this paper is that the piggyback method and the resulting condition for ergodicity capture the implicit stochastic behavior of {F t }, not the behavior of a deterministic skeleton of {Xt} or {£*}. Even if such a skeleton can be identified, its stability properties will not coincide with those of {&}. Alternatively, one may think of {Xt} as having a sort of stochastic skeleton which must be analyzed for stability. Example 8. Our first example is a bivariate threshold model (Cline and Pu (1999a, Ex. 3.2; 2001, Ex. 3.2)). Indeed it is the simplest such model that is not just two independent univariate models joined together. Suppose

where ai(xχ) = α ϋ l X l < o + α^lxjXb i = 1?2. Note that the nonlinearity of the second component Xt^ is driven by the univariate TAR(l) process {Xt,ι} The latter is our "embedded" process and is stable when max(αii,αi2,αiiαi2) < 1.

(16)

Let G be its stationary distribution. The function |α2(#i)| plays the role of H(y) so that the resulting (sharp) stability condition is, in addition to (16),

which represents the stationary value of the relative change in magnitude of Xti2 when it is very large. This condition neither implies nor is implied by the stability condition for the corresponding skeleton process: (16) plus

Example 9. The second example (cf. Cline and Pu (1999c)) is the threshold ARMA(l,ςr) model (TARMA) with a delay d:

ξt = oo(Xt-i) + aι(Xt-i)ξt-i

+et + &i(-ϊt-i)et-i + • +

bq(Xt-ι)et-q,

where Xt-ι = {ξt-i,..., ξt-d), «o3 &i, , bq are bounded and aχ{x) is (asymptotically) piecewise constant. We further assume the thresholds are affine which implies the regions on which a\(x) is constant are cones in Md. This is the simplest interesting example of a TARMA process. See also Brockwell, Liu and Tweedie (1992) and Liu and Susko (1992). In the case q > 0, the time series must be embedded in the Markov chain {(Xt,Ut)} where Xt is as above and Ut = ( e t , . . . , e t _ g +i). Threshold ARMA models have not seen a lot of study, perhaps in part because the moving average terms can affect the irreducibility and periodicity

NONLINEAR TS STABILITY

163

properties of the chain in complicated ways as yet not well understood. (See, for example, Cline and Pu (1999c)). This by itself is a major role played by the noise but we will pass by it here. Let the regions i?χ,..., Rm be the partition of W such that a\(x) is constant on each region, with a n , . . . , a i m being the corresponding constants. There are basically two types of situations that arise in these models when ξt is very large: cyclical and noncyclical. For the cyclical situation, {Xt} essentially cycles close to certain rays having the form

(Uϊ:lalji,nPiZΪalji,...,l)xu

(17)

if all are in the interior of the conical regions. Noise plays no role in determining the stability in this situation since Xt avoids the thresholds; all that matters is the product of coefficients realized by moving through the cycle. For a model which is purely cyclical the stability condition is based on the "worst case" cycle, it is deterministic and corresponds to that of the skeleton process, and it is very much like that of the TAR(l) process with delay discussed in section 5. The model may also have, however, situations where one or more of the rays of type (17) actually lie on a threshold. In such a case, Xt can fall on either side of the threshold, and thus into one of two possible regions, at random but depending on both the present error et and the past errors e t _i,..., et-q. If Jt denotes the region that Xt is in then {(Jt, Ut)} behaves something like a Markov chain where the first component is one of a finite number of states and the second component is stationary. (If q = 0 then {Jt} itself is like a finite state Markov chain.) We relate {(Jt, Ut)} to such a Markov chain denoted, say, {(JtiUt)}. This chain is not necessarily irreducible or aperiodic but clearly every invariant measure is finite. Indeed it may be decomposed into a finite number of uniformly ergodic subprocesses. The coefficients |αχj| play the role of H(y) in this model. Now let G be any stationary distribution for {(Jt, Ut)} and define ΈJ = IRQ G(j,du). If (condition (15))

regardless of the choice of G then {(XuUt)} is geometrically ergodic aiίid, again, the condition is sharp. Because at least one ray lies on a threshold, the noncyclical models are special cases, but the stability condition for a noncyclical process can be quite different from that of nearby purely cyclical processes. See the parameter spaces for the TARMA(1,1) with delay 2 in Cline and Pu (1999c).

164

CLINE AND PU

Example 10. The third example combines the nonlinearity of piecewise continuous coefficient functions with a piecewise conditional heteroscedascity, a model called the threshold AR-ARCH time series: ξt = a(Xt-i) + b(Xt-i)et + c(et; X t -i), where a{x) and b(x) are piecewise linear, {c(ei a )} is uniformly integrable and Xt-i = (ξt-i, ,ζt-p) We further suppose a(x) and b(x) are homogeneous, b{x) is locally bounded away from zero except at x = 0, the thresholds are subspaces containing the origin and the regions of constant behavior are cones. Note that these assumptions need only hold asymptotically (in an appropiate sense) as x gets large. Once again, the Markov chain under study is {Xt} The basic idea on which we piggyback is that the process {Xt} collapsed to the unit sphere behaves very much like a Markov chain. The compactness of the unit sphere serves to make this chain stable and then the stability condition for the original chain can be computed. More specifically, define

S = α ( X t _ 0 + &(*_!)*,

X; = (ξl...,ξ*t-p+ι)

and θ*t = X*/\\X*t\\.

Then, due to the homogeneity of a(x) and 6(x), {0£ } is a Markov chain on the unit sphere and is uniformly ergodic with stationary distribution G, say. By the piggyback method, therefore, Xt has geometrically stable drift if

E (flog(\a(θ) + b(θ)e1\/\θι\)G(dθ)yj < 0. For a simple demonstration, suppose p = 1, a(x) = (αil x o)x, b{ Φ 0, i = 1,2. Then {#£} is a Markov chain on {—1,1} with transition probabilitiesp_i5i = P(a\+b\e\ < 0) = 1—p-^-i and pi,_i = P{a2 + b2eλ < 0) = 1 - p M . If +

+

P

p^y +P

log

(|α2

)

then {ξt} is geometrically ergodic. This example and generalizations of it will be considered fully in a forthcoming paper (Pu and Cline (2001)). Example 11. A very simple model which has not been analyzed fully is the ordinary TAR(2) model with additive noise, ξt = αi(X t -i)ξt-i + a2(Xt-ι)ξt-2

+ eu

where Xt-ι = (ξt-i^t-2) &nd αi(#) and a2(x) are piecewise constant. The precise stability condition is not known even when there is but one affine threshold. The results of this section, however, suggest that the key will be to identify an appropriate stochastic skeleton process to study.

165

NONLINEAR TS STABILITY

7. The Role of the Noise Distribution Tails. Spieksma and Tweedie (1994) pointed out how, with appropriate assumptions on the error distribution tails, an ordinary drift condition (such as (9) with V(x) = 1 + \x\) can be boosted to ensure geometric ergodicity of the process. We generalize the result as follows. Theorem 3 Assume {Xt} is an aperiodic, φ-irreducible T-chain in W and V : W —> [l,oo) is locally bounded. Suppose there exists a random variable W(x) for each x such that V{X\) < W(x) whenever XQ = x, {\W(x) — V(x)\ + er(w(x^~v(x^} is uniformly integrable for some r > 0 and limsupE (W(x) - V(x)) < 0. Then there exist s > 0 and V\(x) = esV^ ergodic (and hence geometrically ergodic).

(18)

such that {Xt} is V\-uniformly

Proof This follows directly from the drift condition for ^-uniform ergodicity (cf. Meyn and Tweedie (1993, Thm. 16.0.1)) and uniform convergence (cf. Cline and Pu (1999a, Lem. 4.2)). (See also the proof to Theorem 4.) D Essentially, this is the log-drift condition in another guise: if the test function in (10), for example, is replaced with V\{x) = esV^ with some sufficiently small s > 0 then (18) is a log-drift version of the condition. As a bonus, if V(x) is norm-like, satisfying ||x|| < V(x) < M +jfif||a;||, one gets strong laws and central limit theorems for all the sample moments (Meyn and Tweedie (1992), Chan (1993a,b)) and exponentially damping tails in the stationary distribution (Tweedie (1983a,b)). Example 12. For example, consider the FCAR(p) model discussed in section 3. If the noise term c(et;Xt-ι) is such that sup x E(er^eux^) < oo for some r > 0 then it frequently is possible to satisfy the requirements of Theorem 3 with a norm-like V(x). To illustrate how this can work, consider the FCAR(l) process, Xt = ξt = αi(6-i) + Φ t ; 6 - i ) with

-L < a\{x) < a\\x + αoi Ίix < -L,

, *

L > a\(x) > a\2X + αo2 if x > L, where a\\a\i — 1, a\\ < 0 and L < oo. We assume here that E(c(e\\ x)) = 0 for all x G R and sup x E(er^eux^) < oo for some r > 0. For the special case of equality on the right in (19) (the SETAR(l) model of Example 4), Chan et al (1985) showed {ξt} is ergodic if and only if 7 = 011^02 + «oi < 0. We thus assume 7 < 0. Let λi = λ^"1 = \J—a\\ and choose δ{ > 1, i = 1,2 so that -λiα O 2 + δχ- δ2 = X2OΌI + h - δ\ = λ 2 7/2. Define

V{x) = (\ι\x\ + δλ)lx<0

+ (X2\x\

166

CLINE AND PU

Then it is a simple computation to show that for some e > 0 and K < oo, V(Xι) - V(x) < (λ 2 lχ< 0 - λil x >o)c(ei; x) + X2j/2 + K\c(eγ,x)\l\c{ei]x)\>€\x\ χ ιs when |Xo| — \ \ sufficiently large, which satisfies the conditions of Theorem 3 with the limit in (18) being λ27/2. The time series is thus geometrically ergodic. On the other hand its skeleton, while stable, is not geometrically stable since a\\aγι = 1. In fact we would say both have only a linear drift. Tanikawa (1999) studied this example and Cline and Pu (1999b) looked at similar first order threshold-like models, but with a possible delay. Using a similar approach but with stronger stability conditions, Diebolt and Guegan (1993) studied multivariate examples and An and Chen (1997) investigated FCAR(p) models withp > 1. One of the drawbacks to a log-drift condition such as the one in Theorem l(iii) is that it guarantees geometric ergodicity only with test functions of the form V(x) = l-hλ(α;)||x|| r where r may be arbitrarily small and therefore it fails to imply needed limit theorems for sample moments. To be able to conclude Vί-uniform ergodicity with an exponential-like Vί, the condition must again be boosted and then the desired limit theorems will hold. Theorem 4 Assume {Xt} is an aperiodic, φ-irreducible T-chain in W and V : W —> [l,oo) is locally bounded and V(x) -> oo as \\x\\ -> oo. Suppose there exists a random variable W(x) for each x such that V(X\) < W(x) whenever Xo = x, {\ log{W{x)/V{x))\ + e ( ^ W ) r - ( ^ ) ) Γ } is uniformly integrable for some r > 0 and limsup£ (log{W(x)/V(x))) < 0.

(20)

IMI->oo

Then there exist s > 0 and V\{x) = e^v^s such that {Xt} is V\-uniformly ergodic (and hence geometrically ergodic). Proof For v > w > 1 and 0 < s < r, we have i {ew3~vS - 1) < ^ ( ^ - l) and log(w/v) < j fe - l j < 0. By the uniform integrability of {log(W(x)/V(x))}, truncation and uniform convergence (as s I 0),

Λi ((ii ((e("W-(v(*))« ("W(v(*)) __ Λ

< limsupE (\og(W(x)/V(x))lw{x) 0 and 5 > 0 small enough.

Λ

+ e < -e,

(21)

167

NONLINEAR TS STABILITY For w > v > 1 and 0 < s < r/2, we have 0 < \og(w/υ) < - (ewS-υ° - l) < - (ewT~vT

-

and if wΓ-υr < K, v > M > 1 then ± (β 10 '-"* - 1) < g £ £ . By the uniform integrability of {e^ί*))'"^**))'}, truncation and V{x) -> oo as ||z|| ->• oo,

0 <

\imsupE(log(W(x)/V(x))lw{x)>v(x})

\\x\\-κx>

)M^)) _ Λ i

) <6,

(22)

for e > 0 and s > 0 small enough. From (20)-(22), therefore, we conclude there exists s > 0 small enough that limsupE (- (eWi»MV(*)) _ Λ IX o = \ Also, sup|| x ||< M J5 M v ( X l ) ) s

χ0 —xλ < oo for all M < oo, and hence geo-

metric ergodicity is assured with test function V\.

D

Example 13. We again consider an FCAR(l) model, ξt = «i(6-i) + c{et\it-ι), satisfying (19) but now we assume a n < 0 < anau < 1 and |c(ei;z)| < ci|a;|^|ei| where cx > 0, 0 < β < 1 and E(eη^) < oo for some η > 0. Let λi = y/-an, λ 2 = V~ α i2 and F(x) = 1 + (λil x o)|^|. Then for \XQ\ = \x\ sufficiently large and some e > 0 and K < oo, V{X\) < (1 - e)V(x) + ίί|x|^|ei|, which satisfies Theorem 4 with r < 1 - β. See also Diebolt and Guegan (1993) and Guegan and Diebolt (1994) for related results. When the errors are bounded, an otherwise unstable model can sometimes be stable. See Chan and Tong (1994) for an example. References An, H.Z. and Chen, S.G. (1997). A note on the ergodicity of non-linear autoregressive model, Stat. Prob. Letters 34, 365-372. An, H.Z. and Huang, F.C. (1996). The geometrical ergodicity of nonlinear autoregressive models, Stat Sinica 6, 943-956. Athreya, K.B. and Pantula, S.G. (1986). Mixing properties of Harris chains and autoregressive processes, J. Appl. Probab. 23, 880-892. Bhaskara Rao, M., Subba Rao, T. and Walker, A.M. (1983). On the existence of some bilinear time series models, J. Time Series Anal 4, 95-110. Borkovec, M. (2000). Extremal behavior of the autoregressive process with ARCH(l) errors, Stock. Proc. Appl 85, 189-207.

168

CLINE AND PU

Brockwell, P.J., Liu, J. and Tweedie, R.L. (1992). On the existence of stationary threshold autoregressive moving-average processes. J. Time Series Anal 13, 95-107. Chan, K.-S. (1989). A note on the geometric ergodicity of a Markov chain, Adv. Appl. Probab. 21, 702-704. Chan, K.-S. (1990). Deterministic stability, stochastic stability, and ergodicity, Appendix 1 in Non-linear Time Series Analysis: A Dynamical System Approach, by H. Tong, Oxford University Press (London). Chan, K.-S. (1993a). A review of some limit theorems of Markov chains and their applications, Dimensions, Estimation and Models, ed. by H. Tong, World Scientific (Singapore), 108-135. Chan, K.-S. (1993b). On the central limit theorem for an ergodic Markov chain, Stock. Proc. Appl 47, 113-117. Chan, K.-S., Petruccelli, J.D., Tong, H. and Woolford, S.W. (1985). A multiple threshold AR(1) model, J. Appl Probab. 22, 267-279. Chan, K.-S. and Tong, H. (1985). On the use of the deterministic Lyapunov function for the ergodicity of stochastic difference equations, Adv. Appl Probab. 17, 666-678. Chan, K.-S. and Tong, H. (1986). On estimating thresholds in autoregressive models, J. Time Series Anal 7, 179-190. Chan, K.-S. and Tong, H. (1994). A note on noisy chaos, J. Royal Stat. Soc. 56, 301-311. Chen, R. and Hardle, W. (1995). Nonparametric time series analysis, a selective review with examples, Bulletin of the International Statistical Institute, 50th session of ISI, August, 1995, Beijing, China. Chen, R. and Tsay, R.S. (1991). On the ergodicity of TAR(l) process, Ann. Appl Probab. 1, 613-634. Chen, R. and Tsay, R.S. (1993a). Functional-coefficient autogressive models, J. Amer. Stat Assoc. 88, 298-308. Chen, R. and Tsay, R.S. (1993b). Nonlinear additive ARX models, J. Amer. Stat Assoc. 88, 955-967. Cline, D.B.H. and Pu, H.H. (1998). Verifying irreducibility and continuity of a nonlinear time series, Stat & Prob. Letters 40, 139-148. Cline, D.B.H. and Pu, H.H. (1999a). Geometric ergodicity of nonlinear time series, Stat Sinica 9, 1103-1118. Cline, D.B.H. and Pu, H.H. (1999b). Stability of nonlinear AR(1) time series with delay, Stock. Proc. Appl 82, 307-333. Cline, D.B.H. and Pu, H.H. (1999c). Stability of threshold-like ARMA time series, Statistics Dept., Texas A&M Univ. Cline, D.B.H. and Pu, H.H. (2001). Geometric transience of nonlinear time series, Stat Sinica 11, 273-287. Collomb, G. and Hardle, W. (1986). Strong uniform convergence rates in robust nonparametric time series analysis and prediction: kernel regression estimation from dependent observations, Stock. Proc. Appl 23, 77-89. Diebolt, J. and Guegan, D. (1993). Tail behaviour of the stationary density of general non-linear autoregressive processes of order 1, J. Appl. Probab. 30, 315-329.

NONLINEAR TS STABILITY

169

Foster, F.G. (1953). On the stochastic matrices associated with certain queueing processes. Ann. Math. Stat. 24, 355-360. Guegan, D. (1987). Different representations for bilinear models, J. Time Series Anal. 8, 389-408. Guegan, D. and Diebolt, J. (1994). Probabilistic properties of the β-ARCH model, Stat. Sinica 4, 71-87. Guo, M. and Petruccelli, J.D. (1991). On the null-recurrence and transience of a first order SETAR model, J. Appl. Probab. 28, 584-592. Hardle, W., Lϋtkepohl, H. and Chen, R. (1997). A review of nonparametric time series analysis, Internal. Stat Review 65, 49-72. Hardle, W. and Vieu, P. (1992). Kernel regression smoothing of time series, J. Time Series Anal 13, 209-232. La Salle, J.P. (1976). The Stability of Dynamical Systems, CMBS 25, Society for Industrial and Applied Mathematics (Philadelphia). Lim, K.S. (1992). On the stability of a threshold AR(1) without intercepts, J. Time Series Anal. 13, 119-132. Liu, J. (1992). On stationarity and asymptotic inference of bilinear time series models, Stat. Sinica 2, 479-494. Liu, J. and Brockwell, P.J. (1988). On the general bilinear time series models, J. Appl. Probab. 25, 553-564. Liu, J., Li, W.K. and Li, C.W. (1997). On a threshold autoregression with conditional heteroscedastic variances, J. Stat. Plan. Inference 62, 279300. Liu, J. and Susko, E. (1992). On strict stationarity and ergodicity of a nonlinear ARMA model, J. Appl. Probab. 29, 363-373. Lu, Z. (1996). A note on the geometric ergodicity of autoregressive conditional heteroscedasticity (ARCH) model, Stat. Prob. Letters 30, 305-311. Lu, Z. (1998a). Geometric ergodicity of a general ARCH type model with applications to some typical models, in Advances in Operations Research and Systems Engineering, J. Gu, G. Fan and S. Wang, eds., Global-Link Informatics Ltd., 76-86. Lu, Z. (1998b). On the geometric ergodicity of a non-linear autoregressive model with an autoregressive conditional heteroscedastic term, Stat. Sinica 8, 1205-1217. Masry, E. and Tj0stheim, D. (1995). Nonparametric estimation and identification of nonlinear ARCH time series: strong convergence and asymptotic normality, Econometric Theory 11, 258-289. Meyn, S.P. and Tweedie, R.L. (1992). Stability of Markovian processes I: Criteria for discrete-time chains, Adv. Appl. Probab. 24, 542-574. Meyn, S.P. and Tweedie, R.L. (1993). Markov Chains and Stochastic Stability, Springer-Verlag (London). Nummelin, E. (1984). General Irreducible Markov Chains and Non-negative Operators. Cambridge University Press, Cambridge. Nummelin, E. and Tuominen, P. (1982). Geometric ergodicity of Harris recurrent Markov chains with applications to renewal theory, Stoch. Proc. Appl. 12, 187-202. Petruccelli, J.D. and Woolford, S.W. (1984). A threshold AR(1) model, J. Appl. Probab. 21, 270-286.

170

CLINE AND PU

Pham D.T. (1985). Bilinear Markovian representation and bilinear models, Stock. Proc. Appl. 20, 295-306. Pham, D.T. (1986). The mixing property of bilinear and generalised random coefficient autoregressive models, Stock. Proc. Appl. 23, 291-300. Pham, D.T. (1993). Bilinear times series models, Dimensions, Estimation and Models, ed. by H. Tong, World Scientific Publishing (Singapore), 191-223. Popov, N. (1977). Conditions for geometric ergodicity of countable Markov chains, Soviet Math. Dokl. 18, 676-679. Priestley, M.B. (1980). State-dependent models: A general approach to non-linear time series analysis, J. Time Series Anal. 1, 47-71. Pu, H.H. and Cline, D.B.H. (2001). Stability of threshold AR-ARCH models, tech. rpt., Dept. Stat., Texas A&M University (forthcoming). Quinn, B.G. (1982). A note on the existence of strictly stationary solutions to bilinear equations, J. Time Series Anal. 3, 249-252. Spieksma, F.M. and Tweedie, R.L. (1994). Strengthening ergodicity to geometric ergodicity for Markov chains, Stock. Models 10, 45-74. Tanikawa, A. (1999). Geometric ergodicity of nonlinear first order autoregressive models, Stock. Models 15, 227-245. Tj0stheim, D. (1990). Non-linear time series and Markov chains, Adv. Appl. Probab. 22, 587-611. Tj0stheim, D. (1994). Non-linear time series: a selective review, Scand. J. Stat. 21, 97-130. Tj0stheim, D. and Auestad, B.H. (1994a). Nonparametric identification of nonlinear time series: projections, J. Amer. Stat. Assoc. 89, 1398-1409. Tj0stheim, D. and Auestad, B.H. (1994b). Nonparametric identification of nonlinear time series: selecting significant lags, J. Amer. Stat. Assoc. 89, 1410-1419. Tong, H. (1981). A note on a Markov bilinear stochastic process in discrete time, J. Time Series Anal. 2, 279-284. Tong, H. (1990). Non-linear Time Series Analysis: A Dynamical System Approach, Oxford University Press (London). Tuominen, P. and Tweedie, R.L., (1979). Markov chains with continuous components, Proc. London Math. Soc. (3) 38, 89-114. Tweedie, R.L. (1975). Sufficient conditions for ergodicity and recurrence of Markov chains on a general state space. Stock. Proc. Appl. 3, 385-403. Tweedie, R.L. (1976). Criteria for classifying general Markov chains. Adv. Appl. Probab. 24, 542-574. Tweedie, R.L. (1983a). Criteria for rates of convergence of Markov chains with application to queueing and storage theory, in Probability, Statistics and Analysis, London Math. Society Lecture Note Series, ed. by J.F.C. Kingman and G.E.H. Reuter, Cambridge Univ. Press (Cambridge). Tweedie, R.L. (1983b). The existence of moments for stationary Markov chains, Adv. Appl. Probab. 20, 191-196.

173

Institute of Mathematical Statistics LECTURE NOTES — MONOGRAPH SERIES

TESTING NEUTRALITY OF mtDNA USING MULTIGENERATION CYTONUCLEAR DATA Susmita Datta Department of Mathematics and Statistics Georgia State University Abstract The neutrality theory of evolutionary genetics assumes that DNA markers distinguishing individuals and species are neutral and have little effect on individual fitness (Kimura, 1983). Under this hypothesis, the action of genetic drift or genetic drift in combination with mutation or migration can be used to describe the evolution of most DNA markers. In recent years, scientists have set up experiments to collect cytonuclear data over several generations to test whether the empirical evidence is consistent with this theory. In this paper, we review the existing statistical tests for neutrality based on such data and propose a new test that we believe is vastly superior. The new test arises from the likelihood theory after embedding the neutral model in a larger class of selection models, where the selection effect takes place due to a difference in fertility of various gametes. A power study based on Monte Carlo simulation is presented to demonstrate the superior performance of the new test.

1

Introduction

A major debate amongst evolutionary geneticists in recent years is whether most DNA markers distinguishing individuals and species are neutral and have little effect on individual fitness (Kimura, 1983). As a profound application of this theory, DNA sequence differences between extant species have been used to reconstruct the history of life. The classical theoretical developments of random genetic drift is built around this assumption. Under this hypothesis, the action of genetic drift or genetic drift in combination with mutation or migration can be used to describe the evolution of most DNA markers. The recent attacks on the neutrality theory are twofold. Firstly, it has been pointed out that in some cases non-neutral models can also explain behavior consistent with empirical evidence. For example, Gillespie (1979) showed that his model of selection in a random environment has the same

174

DATTA

stationary distribution as the infinite allele neutral model. Therefore, the agreement between observations and that predicted by the infinite allelic model noted by Fuerst et al. (1977) can be used with equal strength to support Gillespie's model of natural selection. In another context, Rothman and Templeton (1980) showed that, under some departure from the model assumptions, a neutral model (Watterson, 1977; Ewens, 1972) can yield frequency spectra and homozygosity similar to those expected from heterosis. In addition to the above results, a number of recent experiments suggest apparent non-neutral behaviors of mtDNA markers (Clark and Lyckegaard, 1988; MacRae and Anderson 1988; Fos et al., 1990; Nigro and Prout, 1990; Pollak, 1991; Arnason, 1991; Kambhampati et al., 1992; Scribner and Avise, 1994a, b; Hutter and Rand, 1995; etc.). Singh and Hale (1990) suggested that the apparent "non-neutral" behavior may also be caused by mating preference and that any attempt to understand the role of selection on mtDNA variants should first begin with simpler conspecific variants rather than with interspecific variants; however see MacRae and Anderson (1990), Jenkins et al. (1996). Multi-locus empirical comparisons have been undertaken by Karl and Avise (1992; also see McDonald, 1996), Berry and Kreitman (1993), McDonald (1994). In view of these recent experimental developments it is important to test whether the apparent non-neutral behavior of the markers are indeed statistically significant. Consequently it is more important than ever to devise appropriate statistical tests for testing the neutrality of a mtDNA marker. As we will see in Section 3, the existing statistical tests are often too limited to take full advantage of the multi-generation cytonuclear data that are now available. As a result, a new test based on the recent works by Datta (1999, 2001) is proposed. This test is based on an approximate likelihood for the full available data constructed from a broad parametric selection model and is therefore expected to perform well in practice. The data collection scheme and the underlying model of random drift for genetic evolution is introduced in the next section. This neutral model serves as the null model for the statistical tests which are introduced in Sections 3 and 4. A numerical power study based on Monte Carlo simulation is reported in Section 5. The paper ends with some concluding remarks in Section 6.

2

Data Collection Scheme and the Random Drift Model

In recent kitty-pool experiments, there are two potential sources of variation in cytonuclear frequencies, namely, genetic sampling variation and statistical sampling variation (Weir, 1990). Genetic sampling variation arises

NEUTRALITY OF mtDNA

175

from genetic drift, the sampling of gametes from a finite breeding pool of individuals in nature to constitute the next generation. Statistical sampling variation arises from sampling individuals from a population and using the genotypic frequencies from the sample in subsequent calculation. In Datta et al. (1996), test statistics based on cytonuclear disequilibria were constructed which can account for both sources of variation. The sampling scheme is described below. Such sampling schemes were introduced by Fisher and Ford (1947) and subsequently considered by Schaffer et al. (1977). Kiperskey (1995) also collected data on the fruit fly Drosophila melanogaster following such a scheme. We feel that these types of sampling schemes will become increasingly important in prospective tests for selection (White et al., 1998) using molecular markers in which a cytoplasmic marker is included as a control. Consider a population propagating through discrete non-overlapping generations. Although this is a simplifying and restrictive assumption, it can be achieved for an experimental population with specially selected species, such as Gambusia and fruit flies. At each generation, a portion of the adult population is collected by simple random sampling and sent for genotyping after they form the next generation; eggs by random mating. The eggs are then collected and placed in a cage to form the next generation. Thus, in this case, only the sample genotypic relative frequencies are available and are therefore subject to the additional source of sampling variation. We let g denote the number of consecutive generations from which samples were drawn. Throughout the rest of the paper, we will simultaneously concentrate on a nuclear site with possible alleles A and a and a cytoplasmic site with possible alleles Cand c. The various relative frequencies at the genotypic and the gametic levels are indicated in Tables 1 and 2, respectively. Note that since the cytoplasmic marker is only maternally inherited, its representation remains the same at both levels. Also, if needed, we will denote the generation number (i.e., time) in parenthesis and the corresponding quantities at the sample level will be indicated by the hat notation. Table 1 Genotypic frequencies Nuclear Allele Cytoplasm AA Aa aa Total C V2 P3 q Pi c P4 P5 Pe 1-q V u w 1 Total

DATTA

176

Table 2 Gametic frequencies

Cytoplasm C

Nuclear Allele A a Total eι

c

Total

q

e4 P

l-q 1

Under the action of genetic drift alone, the evolution of the population through generations can be modeled by the following Markov chain. Under the RUZ (random union of zygotes) model (Watterson, 1970), the probability of observing an offspring which received gametic types / and m, respectively, from the two parents is e/e m . Thus, the probability distribution of the counts X(t + 1) = (Xn(t +1), , Xu(t + 1)) in generation t + 1,given Ww> the gametic combination counts up to time ί, is multinomial and is given by (1)

(2) where JVi+i = Σ/,m x /m(^+l) is the size of the ί+1 generation. Finally note that this in turn determines the distribution of the genotypic and the gametic proportions p(t + 1) and e(t + 1), since they are just linear combinations of the x(t + l);viz, pk(t) = {Σf,m<*fmkXfm{t)}/Nt and φ) = ΣkβikPk(t) The coefficients a and β are given in Tables 3 and 4, respectively. For example, since the mtDNA is only maternally transmitted, the genotype AA/C can be formed by either A/C from the father and A/G from the mother, or by A/c from the father and A/C from the mother leading to = {xn(t)+X2i{t)}/Nt.

NEUTRALITY OF mtDNA

177

Table 3 The coefficients aifmk

f

m

1

2

k 3

4

5

6

1 2 3 4

1 0 0 0

0 0 1 0

0 0 0 0

0 1 0 0

0 0 0 1

0 0 0 0

1 2 3 4

0 0 0 0

1 0 0 0

0 0 1 0

0 0 0 0

0 1 0 0

0 0 0 1

lor 2

3or4

Table 4 The coefficients i k 1 2 3 4

5 6

3

1 1 1/2 0 0 0 0

2 0 0 0 1 1/2 0

3 0 1/2 1 0 0 0

4 0 0 0 0 1/2 1

Existing Neutrality Tests Based on Multigeneration Data

Here we review two existing tests of neutrality based on multigeneration data. The first one, due to Schaffer et al. (1977), compares the relative frequencies at a single locus with that expected under random drift over the generations. The test due to Datta et al. (1996) takes advantage of simultaneous data collected at a nuclear site and a cytoplasmic site over generations and compares the pathways of association measures, called the cytonuclear disequilibria, with their expected values over time under random drift. Other existing tests for neutrality generally compare various empirical characteristics based on the very last generation data with the corresponding

178

DATTA

asymptotic value reached at the equilibrium distribution under random drift. The obvious criticisms of such methods are (i) they don't take full advantage of the multigeneration data and (ii) the theoretical basis is questionable unless the population has been in existence for a long time.

3.1 The Schaffer-Yardley-Anderson

tests: This test is a modifica-

tion of a classical test due to Fisher and Ford (1947). They considered the variance stabilizing angular transformation ( 2 sin~ ^relative frequency ) of the proportions at a single locus and compared them with their constant expected value under the action of genetic drift leading to the asymptotically chi-squared distributed test statistic:

where

^ _ l'W~ιY ~ 1W-1Γ Y is the vector of transformed relative gene frequencies, 1 is the vector of ones and W is the matrix whose elements are given by μ

z-1

,

k-l/N

k=l

i y k

j=ϊ \

,*

i <j . ±y

3 J

In an effort to improve the power properties of their test, Schaffer et al. (1977) also proposed an alternative test which effectively tests for linear trend in the transformed frequencies. Although such a selection model may be hard to justify biologically, this approach does lead to a usable test statistic.

3.2 The disequilibria test due to Datta et al.: The Schaffer et al. tests do not make full use of cytonuclear data because they were constructed for tracking the information at a single locus. Datta et al. (1996) proposed testing the dynamics of the sample cytonuclear disequilibria coefficients (see Arnold 1993) with those expected under a drift model. This resulted in a test statistic of the form

T2 = (D-μYΣ^φ-μ) which has an approximate chi-squared distribution under the null hypothesis of random drift. Here D is the vector of sample cytonuclear disequilibria, μ is its expectation under the neutral model of random drift and Σ is its estimated variance-covariance matrix. The above test is somewhat difficult to implement in the sense that the formulas for Σ are complicated. Moreover its power properties may be poor due to its omnibus nature.

NEUTRALITY

4

179

OF mtDNA

A New Test Based on a Selection Model

In order to take full advantage of the multigeneration cytonuclear data, very recently Datta (2000) considered a fairly broad selection model that includes the neutrality model of random drift as a special case. The selection effect takes place because of a difference in the fertility of various gametes. We propose the resulting likelihood based score test for testing the neutrality hypothesis. Not only does it arise very naturally, it incorporates the entire cytonuclear information present in the data; furthermore since it is derived from embedding the null hypothesis of random drift into a fairly rich parametric alternative selection model, it should enjoy reasonable power properties at least when the selection model holds. A simulation based power comparison study reported in the next section shows that this is indeed the case. Consider the following selection model as an alternative to the random drift model describing the evolution of the population. The probability of observing an offspring which received garnetic types / andra, respectively, w e from the two parents is ef(w)em(w)where eι(w,t) = Wieι(t)/ yΣ j j{t))\ here W{ denote the relative fertility of gametic type i, 1 < i < 4. Therefore, the distribution of the counts x(t + 1) given the t-th generation is given by (1) with the product ef(t)em(t) replaced by ej{w, t)em(w, t). Note that when W{ = 1/4, one has the random drift model. Therefore, a test of neutrality can be based on the score statistic s(wo), with wo = (1/4, , 1/4). Since s, the derivative of the log-likelihood based on the population gametic relative frequencies, is not computable from the observed data, Datta (2001) suggested using an approximate version of it obtained by replacing them with the corresponding sample versions. This results in additional terms for the variance-covariance matrix, all of which can be consistently estimated from the observed data. One can show that the approximate log likelihood has a simple closed form expression given by g-i

6

Άw) = Σ Σ Nt+ιPk(t + 1) log(ifc(ti;,i)) t=l

k=l

Lι(w,t) = e\(w,t) +

eι(w,t)e2{w,t),

L2{w,t) = 2e1(ϊi;,ί)e3(iί;,t) + e2(w,t)es{w,t) + eι(w,t)e4(w,t) Ls(w, t) = e\(w, t) + e3(w, t)e4(tu, t),

180

DATTA

t) = ej(w,t) +es(w,t)eA(w,t)

1

and βi{w,t) = ei{t)wi/ ί Σej{t)wj J , 1 < i < 4. See Datta (2001) for the details of the algebraic calculations. Of course, the approximate score is defined as s(w) ^ (d/dw) I (w) where we interpret it as a function of w = (wι,W2, ws) ( i.e., replace w^ by I — wι — w2 — w^). Datta (2001) was able to calculate the estimated asymptotic variancecovariance matrix of s given by Σ y = (d/dw)(s(w))\w=g

+ C f d + CξC2 + ••• +

CjCg,

where

ds{w;p{l), -,p(g)) dP(t)

-1I/2

\{diag{p(t))-p(t)p(t)T)/nt]

1 < < ^ 9 Therefore, a test statistic for the neutrality hypothesis is given by T = S {WQ)Σ^S{W ) with w0 = (1/4,1/4,1/4). The neutrality hypothesis would be rejected if T exceeds Xχ_Q(3). T

0

5

Power Studies

We now report the results of a simulation study where we compare the power of Datta's (2001) test with those of earlier tests by Schaffer et al. (1977). The experimental setup is as follows. The w are parametrized by a single parameter μ so that w\ = w2 = 1/2 - μ; ^3 = W4 = μ. We simulated 2000 multigeneration samples each of g = 5 generations. The (constant) population size Nt equaled 1000 and (constant) sample size nt equaled 100. The initial population frequencies were given by pi — Pz — PA = P6 = 0; P2 = 0.5, p$ = 0.5. The counts x at the population level of successive generations are generated recursively using the multinomial model described in the second paragraph of Section 4. Next the genotypic and the gametic proportions are obtained by the formulas pk = Σ/,m afmkXfmlN and βi = Σk βikPk Finally, at every generation given the population p^ the sample pk are generated by multinomial (n; pi, • ,pβ) sampling. We simulated the powers of three different tests; (i) the omnibus test by Schaffer et al. for the mitochondrial locus, (ii) the linear trend test by Schaffer et al. and (iii) the new approximate score test by Datta described in the previous section. A nominal level of α = 0.05 was used for each case.

NEUTRALITY OF mtDNA

P _

181

*

0.8

1

•

Scor

s

SYA

1-

Unw

0.4

5-

0.2

s

ie

e

.

»

0.0

Λ

0220

0225

0230

0235

0.240

0.245

0.250

mu

Figure 5.1: Power of 5% tests at selected values of μ Figure 5.1 describes the findings. To reduce computational time involved we computed the power at only four values at and near the null hypothesis value of μ = 0.25. The figure clearly illustrates the superior performance of Datta's approximate score test over the older tests. Whereas the power of this test reaches nearly one for μ = 0.22, the power for the other tests remain flat in the entire range of μ values under consideration.

6

Concluding Remarks

Testing the neutrality of DNA markers is an important problem in evolutionary biology. In recent years, experiments have been designed in a controlled setting where a population with discrete generation can be allowed to propagate at the same time random samples from each generation are collected. Often genotyping is done at two loci, say one nuclear and one cytoplasmic, simultaneously. Utilization of such multigeneration cytonuclear data in a set up where both genetic and statistical variabilities are present is a challenging problem. We present some recent work by Datta (2001) in this direction, where a neutrality test is constructed by correctly identifying an approximate likelihood for such a setup. A numerical comparison of power with earlier tests show great promise. It will be interesting to investigate the

182

DATTA

power properties more extensively covering a broad range of non-neutrality models, possibly going beyond the selection model considered here. Such a study is underway and will be reported elsewhere.

References

Arnold, J., 1993: Cytonuclear disequilibria in hybrid zones. Annu. Rev. Ecol. Syst. 24, 521-554. Arnason, E., 1991: Perturbation-reperturbation test of selection vs. hitchhiking of the two major alleles of Esterase-5 in Drosophila pseudoobscura. Genetics 129, 145-168. Berry, A. J. and Kreitman, M., 1993: Molecular analysis of an allozyme cline: alcohol dehydrogenase in Drosophila melanogaster on the east cost of North America. Genetics 134, 869-893. Clark, A. G. and Lyckegaard, E. M. S., 1988: Natural selection with nuclear and cytoplasmic transmission. III. Joint analysis of segregation and mtDNA in Drosophila melanogaster. Genetics 118, 471-481. Datta, S., 2001. Estimation of Selection Parameters using Multi-generation Cytonuclear Data, Biometrical Journal 43, 219-233. Datta, S., 1999: Hypotheses testing for different selection models using multigeneration cytonuclear data. In Proceedings of American Statistical Association, Biometrics Section, 157-161, Alexandria, USA. Datta, S., Kiparsky, M., Rand, D. M. and Arnold, J., 1996: A statistical test of a neutral model using the dynamics of cytonuclear disequilibria. Genetics 144, 1985-1992. Ewens, W. J., 1972: The sampling theory of selectively neutral alleles. Theor. Popul. Biol. 3, 87-112. Fisher, R. A. and Ford, E. B., 1947: The spread of a gene in natural conditions in a colony of the moth Panaxia dominula L. Heredity 1, 143-174. Fos, M., Dominguez, M. A., LaTorre, A. and Moya, A., 1990: Mitochondrial DNA evolution in experimental populations of Drosophila pseudoobscura. Proc. Natl. Acad. Sci. USA 87, 4198-4201. Fuerst, P. A., Chakraborty, R. and Nei, M., 1977: Statistical studies on protein polymorphism in natural populations. I. Distribution of single locus heterozygosity. Genetics 86, 455-483.

NEUTRALITY OF mtDNA

183

Gillespie, J. H., 1979: Molecular evolution and polymorphism in a random environment. Genetics 74, 175-195. Hutter, C. M. and Rand, D. M., 1995: Competition between mitochondrial haplotypes in distinct nuclear genetic environments : Drosophila pseudoobscura vs Drosophila persimilis. Genetics 140, 537-548. Jenkins, T. M., Babcook, C, Geiser, D. M. and Anderson, W. W., 1996: Cytoplasmic incompatibility and mating preference in Colombian Drosophila pseudoobscura. Genetics 142, 189-194. Karl, S. A. and Avise, J. C, 1992: Balancing selection at allozyme loci in oysters: implications from nuclear RFLPs. Science 256, 100-102. Kambhampati, S., Rai, K. S. and Verleye, D. M., 1992: Frequencies of mitochondrial DNA haplotypes in laboratory cage populations of the mosquito Aedes albopictus. Genetics 132, 205-209. Kimura, M., 1983: The Neutral Theory of Molecular Evolution. Cambridge University Press, New York. Kiparsky, M., 1995: Cytonuclear genetics of experimental Drosophila melanogaster population. Unpublished Honor's Thesis, Department of Ecology and Evolutionary Biology, Brown University. MaCrae, A. and Anderson, W. W., 1988: Evidence of non-neutrality of mitochondrial DNA haplotypes in Drosophila pseudoobscura. Genetics 120, 485-494. MaCrae, A. and Anderson, W. W., 1990: Can mating preference explain changes in mtDNA haplotype frequency? Genetics 124, 999-1001. McDonald, J. H., 1994: Detecting natural selection by comparing geographic variation in protein and DNA polymorphisms. In Non-Neutral Evolution, B. Golding, Ed., 88-100, Chapman and Hall, New York. Mcdonald, J. H., 1996: Lack of geographic variation in anonymous polymorphisms in the American oyster Crassostrea virginica. Mol. Biol. Evol. 13, 1114-1118. Nigro, L. and Prout, T., 1990: Is there selection on RFLP differences in mitochondrial DNA? Genetics 125, 551-555. Pollak, P. E., 1991: Cytoplasmic effects on components of fitness in tobacco hybrids. Evolution 45, 785-790. Rothman, E. D. and Templeton, A. R., 1980: A class of models of selectively neutral alleles. Theor. Popul. Biol. 18, 135-150. Schaffer, H. E., Yardley, D. and Anderson, W. W., 1977: Drift or selection: test of gene frequency variation over generations. Genetics 87, 371-379.

184

DATTA

Scribner, K. T. and Avise, J. C , 1994: Population cage experiments with a vertebrate: genetics of hybridization in Gambusia fishes. Evolution 48, 155-171. Singh, R. S. and Hale, L. R., 1990: Are mitochondrial DNA variants selectively non-neutral? Genetics 124, 995-997. Watterson, G. A., 1970: The effect of linkage in finite random-mating population. Theor. Popul. Biol. 1, 72-87. Watterson, G. A., 1977: Heterosis or neutrality? Genetics 85, 789-814. Weir, B. S., 1990: Genetic Data Analysis. Sinauer Assoc, Sunderland. White, T., Marr, K. A. and Bowden, R. A., 1998: Clinical, cellular, and molecular factors that contribute to antifung drug resistance. Clinical Microbiology Reviews 11, 382-402.

185

Institute of Mathematical Statistics LECTURE NOTES — MONOGRAPH SERIES

INFERENCE ON RANDOM COEFFICIENT MODELS FOR HAPLOTYPE EFFECTS IN DYNAMIC MUTATIONS USING MCMC. Richard M. Huggins1, Guoqi Qian1 Sz Danuta Z. Loesch2 department of Statistical Sciences, La Trobe University, Bundoora, 3083, Australia. 2 School of Psychological Science, La Trobe University, Bundoora, 3083, Australia. Abstract Modern genetic research involves complex stochastic processes and difficult inference problems and research in this area is necessarily a collaboration with geneticists and biologists. The major difficulty is in defining stochastic processes which are biologically meaningful yet amenable to analysis. To illustrate this we examine a class of random effects models for dynamic mutations. Dynamic mutations characterize several inherited disorders in humans. In these disorders a mutated segment of the gene typically increases in size as it is transmitted from generation to generation until the gene fails. Biological interest is in the effect of various genetic markers on the rate of expansion of the mutated segment and to what extent these markers describe alternate pathways. We concentrate on the widely studied fragile X disorder as there are data sets available for statistical analysis. We use heirarchical Bayes models fitted via MCMC methods to examine some data and hence determine a class of random coefficients, branching, time series models which have applications in genetical research.

1

Introduction

The realistic modelling of the stochastic processes occurring in biology quickly leads to analytically intractable models. However, these models may be of enormous practical importance. Here we examine a model for the recently discovered phenomenon of dynamic mutations, that lead to a number of genetic disorders. In these disorders rather than an on-off allele being transmitted from a parent to their offspring the mutation itself changes upon transmission. The mutation arises through expansion at a region of the gene, i.e. the insertion of extra genetic material in the form of trinucleotide

186

HUGGINS, QIAN AND LOESCH

repeats at that site. This induces instability at that site with the size of the expansion usually increasing from generation to generation. Ultimately the expansion becomes large enough to affect the functioning of the gene resulting in phenotypic expression of the disorder. A feature of these dynamic mutations is that they are clinically undetectable until the expansion crosses a threshold and phenotypically abnormal individuals arise. The fragile X Syndrome is the most widely studied dynamic mutation. It is a common X-linked genetic disorder associated with intellectual disability. The fragile X syndrome arises as the result of progressive CGG trinucleotide repeat expansion in the FMR1 gene, which culminates in the failure of gene expression. The mechanism for the expansion of the repeat size has not been determined. The fragile X syndrome is classified according to the number of CGG repeats. The premutation (55-200 repeats) and full mutation (200+ repeats) categories are unstable and some instability has also been reported within grey zone (35-55 repeats) alleles. Previously the rate of expansion of CGG repeats on transmission has usually been modelled as a function of the size of CGG repeat sequence on the maternal X chromosome. However, some molecular characteristics of the gene other than the size of the CGG repeat in this chromosome have recently been shown to affect the rate of expansion. These genetic covariates are certain combinations of microsatellite markers (haplotypes) flanking the repeat sequence, and the number and position of the AGG triplets which normally interrupt CGG repeats at intervals of 9-10 units. Eichler et al. (1996) have recently hypothesised that at least two different mutational pathways leading to the fragile X syndrome related to those genetic covariates are in action in the population. However, the data obtained by Dana et al (2000) from an Afro-American population showed that this hypothesis should be viewed from a broader evolutionary perspective. Several mathematical models for the transmission of fragile X have been proposed. These include comprehensive transmission models (Ashley and Sherman 1995, Morton & Macpherson 1992, Morris et al 1995), iterated branching models (Gawel and Kimmel 1996) and Bernoulli counting process models (Bat et al 1997). The model of Ashley and Sherman (1995) accounts for the dynamics of fragile X mutation by assuming the multi-step expansion of CGG repeats (meiotic expansion in either sex, and mitotic expansion restricted to somatic alleles of maternal origin). They also included population assumptions, such as selection against full mutation, as well as molecular mechanisms, such as the loss of AGG interspersions, which in their model constitutes the initial mutation in the FMR1 gene. These previous models were biologically meaningful but were difficult to analyse statistically and tended to become outdated as more data on molecular characteristics of the gene came to hand.

RANDOM COEFFICIENT MODELS

187

Huggins, Loesch & Sherman (1998) introduced a multi-step non-linear time series models for the transmission of dynamic mutations which could be analysed using standard statistical methods. They used a random intercepts model to examine the effects of haplotypes or genetic markers on the transmission of the mutation. Here we extend that model in several directions, with an emphasis on the effect of the genetic markers on the rate of transmission. Firstly, we use cubic splines to model the relationship between the length of the repeat sequence in offspring X chromosomes to that in the parent X chromosome. Secondly we allow the rate of transmission to be random, rather than just a random intercept. Available data consists of observations on several generations of affected families as, due to the lengths of human generations, it is not possible to collect DNA to construct longer chains. However, one purpose of our modelling is to be able to model distantly related affected individuals who are known to have a common ancestor. We illustrate the inference method using published data of Murray et al (1997) who report a variety of haplotypes which may be related to the rate of expansion in fragile X. The biological meaning of our results shall be discussed elsewhere, as will formal tests of hypotheses.

2

Notation

We label the observed haplotypes by h = 1,2,..., H. We suppose there are Nh families with haplotype /ι, labelled /ι/, / = 1,..., iV^. Let n^f denote the number of parents that produce observed offspring in family /&/, labelled /ι/fc, k = 1,..., rihf. Also, let Whfk be the number of offspring of individual hfk. The offspring of individual hfk are denoted by hfkl, I = 1,..., Whfk Note that individuals may be labeled both as parents and as offspring of their parents. This notation aids in constructing the conditional densities required below. In this paper, inference is conducted conditional on the initiating individual in each family, the observed family structure and the observed haplotypes. Further, the probands, or the individuals initially detected with the disorder in each family, have been omitted. This was done on order to reduce the ascertainment bias as the probands were usually detected by their having an extreme phenotype and hence genotype. Let Y be the vector of observations of the CGG triplet repeat lengths on all offspring. Namely, Y = {Yhfki : (Λ,/,M) e S], where S is defined by S = {(Λ,/,M): h = 1,2,-- ,£Γ; /|Λ = 1,2,--.,ΛΓΛ; k\hf = 1,2, -- , n Λ / ; l\hfk = 1,2, ,whfk.}. Here f\h is understood as the value of / given h.

188

3

HUGGINS, QIAN AND LOESCH

The Hierarchical Model

Our full model for the length of the CGG triplet repeat sequence in offspring I of individual k is of the general form { h)

(βh + β f )IF(hfk)}

+ δFIFζhfkl

+ σεh}ku

(3.1)

h = 1,... ,iϊ, f\h = 1,... ,ΛΓΛ, k\hf = 1,..., n Λ /, Z|/ι/fc = 1,..., wh}k, where sa n the C/ι/Jfc/' d tfifkis are independent normal random variables with 0 means and variance 1, Xhfk is a design matrix and IF(hfk) takes the value 1 if individual hfk is female and zero otherwise. The response g(Yhfkι) is taken m to be log log Yhfki this paper as this gave errors that appeared normally distributed, although other choices are possible. We suppose βh follows a common probability distribution for all individuals with haplotype h and βi ' follows a common distribution for all individuals in family / within haplotype h. The introduction of the sex effect allows us to mimic the Ashley-Sherman (1995) model and the subsequent model of Huggins et al (1998) which were two step models with an initial step common to transmissions from males and females, and a second step that only occurs in transmissions from females. This is difficult to directly model in a linear framework so instead we allow different rates in transmissions from males and females. Available data, see the plots of Huggins & Loesch (1998), suggest there is little difference between the haplotypes or families in transmissions from males and a single vector β is used to model this transition. The conditional mean Xhfk{β + (βh + βf^I^hfk)} of g(Yhfkι) as a function of the parameters and log Yhfk was modelled using cubic splines (Dierckx 1993). The corresponding B-splines are contained in Xhfk- As the present work is exploratory ten degrees of freedom were used in the cubic splines to allow flexible modelling. This gave similar results to models with different degrees of freedom (6, 8 and 14). Let m denote the number of columns of Xhfk- The main interest is in the posterior distributions of βh

and βf\

We will use the following prior distributions for the parameters /?, βh, βf \σ2 and tip:

2. βh - b2(βh\Σho) = DMVN(0, ΣΛO), 3. iβJ Λ) ~6i(i9j Λ) |Σ / o) = MyΛΓ(O1Σ/o), 4. σ 2 ~ 6 4 ( σ > , λ ) = = IGa(i//2,i/λ/2), 5. 4 ~ 6 5 ( 4 1 ^ , λ F ) = IGa(ι/F/2, I/

h = 1,..., £Γ, f\h = 1,... ,JVΛ, Λ = l , . . . ,

189

RANDOM COEFFICIENT MODELS

which are all assumed to be independent. Here MVN denotes the multivariate normal distribution, IGa the inverse Gamma distribution (e.g. Robert 1994, p. 153). The values of v, λ, up and Xp do not have any significant effect on the analysis so will be specified in the study (see Appendix A). The hyperprior distributions for the hyper-parameters in the above prior distributions are taken to be: 1. βo ~ Wo|&,Σ) = DMVN{b,Σ), 2. Σ o - W - H ξ o + m + l^oiϊo),

ξo>m

ι

3. Σfco ~ W- (ξh0 + m + l,ξhoRhθ), ι

4. Σ/o - W- (ξf0

+ m + l,ξ/OΛ/o),

ξhx> > m, ξ/o > m,

which are also assumed to be independent. Here W7^1(ξ. + m + l.ξ.R.) denotes an inverted Whishart distribution for an m x m random matrix Σ. with degrees of freedom £ + m + 1 and positive definite matrix R. It can be shown that ^ ( Σ " 1 ) = R~ι and E(Σm) = {(£ - m - ljξ.β.}" 1 if these exist (Muirhead 1981, p.113). The values of &,Σ,fθj-Ro?fΛθ?Λ/ιθjf/o a n d Rfo will be specified in Appendix A. We denote the last three densities by wo(Σo|ξo,i?o), u>hθ(Zho\ξhθ,Rhθ) and^ /o (Σ/o|ξ/o,Λ/o) respectively. In order to simplify the presentation, we use Θ and 7 respectively to denote the parameters and hyper-parameters in the hierarchical model defined here. Namely, Θ = ( / ^ / ^ / ^ σ 2 , ^ ) * and 7 = (/3b,Σo,ΣΛO,Σ/o), where Ph — vPi?

7 P/f J 5 P/ — vP/

5

? P/

a n α

P/

— vPi

/ι = 1, , H. Note that Θ is a vector of dimension (1 + H + Σ Λ ^ I Nh)m + 2 and 7 is an m x (1 + 3m) matrix. The equation (3.1) together with the prior distribution of Θ and hyperprior distribution of 7 comprise a hierarchical random effects model for the CGG triplet repeat sequence. In order to make inference about Θ and 7, and in particular βh and βj ' from the data, we need the posterior distributions of βh and βί '. Because of their mathematical complexity we will not be able to make inferences directly from these posterior distributions. Rather we will first generate a sample from these distributions and then make inferences from the sample. The Markov chain Monte Carlo (MCMC) technique has shown to be powerful in simulating a sample from those probability distributions which are analytically intractable. Recall that the basic idea of MCMC is to use a relatively simple transition probability kernel to generate a Markov chain in such a way that its invariant distribution is that from which we want to generate a sample. After a sufficient number of generations of the chain have been simulated, subsequent generations can be regarded as a sample from the target distribution. Two most basic algorithms used in MCMC methods are Gibbs sampling and Metropolis-Hastings

190

HUGGINS, QIAN AND LOESCH

algorithm. We use a mixture of these two algorithms which is called MCMC block-at-a-time algorithm (Chib and Greenberg, 1995). Several methods are available (Robert 1998) for monitoring the convergence to stationarity of the resulting Markov chain. We employ the GelmanRubin statistic (Gelman and Rubin 1992) which measures the correlation between the within- and between-chain variations of the simulated multiple Markov chains. When the Gelman-Rubin statistic becomes very close to 1 (usually less than 1.2 or 1.1 is enough in practice), the chains can be regarded as having achieved convergence. Once a sample has been generated from the joint posterior distribution, it can be examined to detect the random effects of haplotypes and families. The sample can also be used to simulate the posterior predictive distribution for the response variable CGG repeats which, after comparison with the true CGG observations, allows determination of the goodness of fit of the model. The various joint and conditional densities and the MCMC procedure used to fit the model are described in the appendices.

4

Results

The data of Murray et al (1997) contain observations on CGG repeat sequences for 124 individuals and their parents after the probands were omitted. The remaining 124 individuals are the offspring of 86 observed parents who come from 57 different families, nested in 18 different haplotypes. In addition, 10 of the 86 observed parents are fathers who between them have 18 offspring. When applying the hierarchical model of section 3, the parameter vector Θ contains 762 components and the hyper-parameter 7 is a 10 x 31 matrix. For each of these parameters we generated 3 Markov chains of length 10,000 from which the posterior samples were formed. We first check the convergence of the simulated Markov chains. It was found that for the current model at most 2400 transitions would be sufficient for the simulated Markov chain to be stationary. In the simulations the slowest convergence occurred at the Markov chain of Θ(88) = /?s (8) (component 8 of haplotype 724) among all parameters and at 7(6,17) = E^o^β] (the 6th diagonal element of Σ^o) among all hyper-parameters. Figure 1 displays the sub-chain of a simulated Markov chain (by filter [fc/50], i.e., taking every 50th value) for each of /?s(8) and Σ^oίθjθ) and their Gelman-Rubin statistic value sequences based on the 3 parallel chains. To form a posterior sample for each parameter or hyper-parameter, we discard the first half of each of the 3 generated chains and combined the rest into one sequence; then we filter this sequence by the operator [Λ /50] and take the resulting sequence of length 300. Shown in Figure 2 are histogram densities of the posterior samples for the 10 components of the haplotype

RANDOM COEFFICIENT MODELS

191

tt> 6

I G-RvaluβsforSigr ω

^

y-1.2 " V-1-1 40

100

160

Figure 1: Row 1 gives Markov chains generated for β$(8) and Έho(6,6). Row 2 gives the corresponding Gelman-Rubin statistic values based on the three parallel chains.

8 effect 0 8 (the 81st-90th elements of Θ, i.e. Θ[81 : 90] and the 10 components of the family effects β[2) = Θ[261 : 270] (effects of the first family in the second haplotype which are the 261-270th components of Θ). These histograms are also typical of other haplotype and family effects. The haplotype effects can be analysed by examining whether or not there is any heterogeneity among the marginal posterior distributions of {βh(ϊ), h = 1, , 18} (i = 1, , 10). Shown in the first column of Figure 3 are the Xbar-chart and S-chart with 95% confidence level for posterior samples of {0i(8), ,0is(8)}, and the histogram density and the QQ-plot for the corresponding posterior means. The other two columns of Figure 3 are for {0i(6), ,/?i8 (6)} and {0i (3), ,0is(3)}. Figure 3 gives evidence of haplotype effects in the expansion process. The family effects given the haplotype effects can be similarly analysed based on the marginal posterior distributions of βf(i) (i — 1, , 10). The three columns in Figure 4 give the results of these effects from the posterior samples of 0/(8) 's, 0/(4)'s and 0/(1) 's respectively. Again family effects are evident in Figure 4. The goodness of fit of the model can be assessed by comparing the response observations g(Y) to their posterior predictive distributions (Gelman et al 1995, sec. 6.3). For each response observation g(Yhfki) a sample of size 300, regarded as the replicates of g(Yhfki) which could have been observed,

HUGGINS, QIAN AND LOESCH

20

20 40 60

0

0

40 60

192

1 A3 A bθta_8(3)

s

Jh J L

°

•10 bita.Vf

s

1

°

"

s

5 -10 -5 0 5 10 bβta_8(7)

•10

0

1

jjj^H

-5bβ°te_8δ)10

1 S

s

L

3 d

H

^

—

-

0-5 0 5 10 bβta_8(10)

S

J

o

5 bθta_1Λ{(2)}(5)

bθta_1Λ{(2)}(3)

bβta_1A{(2)}(2)

i

b 5 βtL8?8) 1 °

s

w -5

bθta_8(5)

S

l riL -10

bθta_8(4)

s

A!

jL

A (2)

Figure 2: Histograms for all the 20 components of βs and β\

II

I„

»^umber beyond

V\/ \/\

- Λ

ii

-o Λ Λ beyond ϊmits=0

Λ. ^

Number beyond id bmi(s=0 bmi(s=0

s Chart for beta.h.8

\ /°

A

V-

π

7\

' \i i&>

_

/

\ / X<

Number beyond lirn"ite=2 °

s Chart for beta.h.3

s Chart rorbeta.h.6 "*

effects.

/\

.•"'„••

\/

Number beyond limils=2

"

I:

*

o

V

'

/

"

•

%

-

.

Nurnber bβvond lirr |s=2

11

Figure 3: Effects of haplotype factors evaluated at components 8, 6 and= 3.

RANDOM COEFFICIENT MODELS

1101

^.. ^..^.. . X Al

Number beyond Km ls=2

Number beyond limits=4

s Chart lor bθta.f.8

I: " ^ I3

193

1 4 7 10 13 I t in

s Chart for bela.f.4

ΐ

s Chart (orbβta.1.1

g oil "juj'jj ib^'jis,"

111

πv*ϊ

κ

N U j mber beyond Bmits=0

π Figure 4: Effects of family factors evaluated at components 8, 4 and 1.

was generated from its posterior predictive distribution. Then the Bayes pvalue was calculated for each sample, which is the proportion of the replicates that are greater than g{Yhfki) The histogram density for the replicates of log log Yg, which is typical for all 124 response observations, is given in Figure 5. A histogram of the 124 Bayes p-values is also given in Figure 5. Since most of the Bayes p-values are very close to 0.5 (all but one are between 0.4 and 0.563), it implies that the response observations are typical under the posterior predictive distributions. In addition, we provide in Figure 5 the plots of log log Ynew IY — the sample means of the posterior predictive distributions against= log log Y, and £?(loglog Y|Θ p o 5 ) — the fitted response values using the posterior means of the distributions of the /?'s against log log Y. These plots show the reasonable fit of the proposed model to the data which is also supported by the values of the goodness of fit statistic computed from (2) below. Actually it was found that || loglog Y-loglog Yne™|Y||2 = 14.10 and 11 log log Y - E(loglog Y | Θ p o 5 ) | | 2 = 7.26. However, the histogram density in Figure 5 shows that the posterior predictive distribution of Y has a quite wide range and a large variance, particularly considering the nature of a loglog( ) transformation for the response Y. This implies that the proposed model has a large variation in its predictability although it is not the main interest of this paper. One could possibly obtain better predictions if another transformation g(Y) or a discrete model such as a Poisson distribution for Y were used. Finally we propose to evaluate the goodness of fit of the model by the

HUGGINS, QIAN AND LOESCH

194

summary(bayesPIO) Min. 1st Qu. Median Mean 3rd Qu. Max. 0.317 0.477 0.497 0.497 0.517 0.563

-15-11-7-3 1 5 9 13 17 Histogram density of log(log(Y_9)) from a

0.25 0.31 0.37 0.43 0.49 0.55 0.61 Histogram for θayes p-vakies with individual

v obs. parent is father t-obs. parent is mother l

••

+ obs. parent is father +obs. parent is mother

k>g(log(Y))

Figure 5: Model assessment based on posterior predictive distributions.

following statistic N

N

i=l hfkl

Var(Yhfkl\θi)

(4.1)

where θ i , , θ # is a posterior sample of θ . Although no rigorous justification is available, it sounds sensible to approximate T by a χf 24 distribution without being too erroneous. Based on the posterior θ sample generated in our study we found T = 87.26, which corresponds to 0.5-percentile of a χ{24 and thus indicates that the proposed model gives an adequate fit.

5

Discussion

Many of the advances in inference for stochastic processes over the past few decades have arisen from branching processes and time series models. A population model for the transmission of dynamic mutations combines time series models with branching processes, for, as well as (3.1) holding the number of offspring Whfk of individual hfk has a distribution that depends on both the sex of individual hfk and the value of Yhfk- The outstanding problem is to find conditions under which the population model has a stationary distribution. We have concentrated on developing and fitting time series type models for the transmission of a dynamic mutation given the observed haplotypes

RANDOM COEFFICIENT MODELS

195

and family structures. The model can be viewed as an extension of the simpler bifurcating autoregressive models for cell lineages. The work on cell lineage studies starting with Cowan (1984), Cowan &: Staudte (1986) and others considers the bifurcating autoregressive process for cell lineage studies. Cell lineage trees arise from the binary splitting of cells and the processes of interest are time series, typically low order autoregresive models, down lines of descent with correlated errors for sister cells and in later models (Huggins & Basawa 1999) correlations between individuals in the same generation. Bui & Huggins (1999) considered random effects models for the bifurcating autoregressive(l) model. The processes of interest here have a random number of offspring rather than just the two of the bifurcating autoregressive process and the models down each line of descent are more sophisticated than the ARMA type processes. Moreover, individuals with a given identified haplotype may differ at other gene sites, which naturally gives rise to the random effects models. Hence we take the transition rates to be random, which are common for individuals in the same families and there is communality between unrelated individuals with the same haplotype. Markov models are typically used to model the transmission of a characteristic from parent to offspring. That is, the genetic make up of the offspring only depends on that of the parents. The indiscriminant use of such Markov models may be unrealistic in the branching situations that occur in family studies. For example, in related work on cell lineage studies, Staudte et al (1996) determined that the correlation between cousins in the data examined by them was larger than what was predicted by AR(1) type Markov models. In Markovian AR(1) models for cell lineages, correlations drop off as powers of the mother-daughter correlation θ. That is, mother-daughter correlation is 0, the= sister-sister correlation is θ2 (plus environment effects) the cousin-cousin correlation is 04 etc. Huggins & Basawa (1999) proposed several models which allow higher correlations between cousins and other more distant relatives including an AR(2) type model as well as random effects models of generation effects. This situation may also occur in dynamic mutations unless genetic covariates that affect the expansion rates are taken into account. This motivated our hierarchical random effects model. The models examined here are based on transmissions over one or two generations and may not reflect the full complexity of the dynamic mutations. Moreover, the processes are sparsely observed and as the process is unobserved in its initial stages, there is little information concerning the mutation rate currently available. The nested random effects, which depend on haplotypes that are themselves evolving according to some stochastic process, add another level of complexity. However, the haplotypes occur on a portion of the gene which does not appear to code for a protein hence they should not be exposed to selection so that their evolution should be depen-

196

HUGGINS, QIAN AND LOESCH

dent only on random mutations and rare recombinations and be simpler than that of the the FMR1 gene itself. Future work will involve the collection and modelling of data on distantly related affected individuals, where the evolution of the haplotypes will need to be considered. The strength of the relationship between individuals will be determined either through family trees or through genetic markers with known mutation rates. It is not necessary to use MCMC to estimate random effects although it was convenient in the present example. For example, Park & Basawa (2000) have developed an optimal estimating function approach. However, they have not yet fully developed the inferential procedures. The use of cubic splines has advantages in that the models are linear and may to some extent be regarded as non-parametric. Moreover, it was not necessary to introduce a threshold as in Huggins et al (1998). However, the non-linear models of Huggins et al (1998) did have some biological advantages in that they more naturally modelled the expansions in the meiotic and mitotic phases. In the non-linear case it was possible to let the parameters in the model depend on the size of the parents repeat sequence which is more difficult in linear models. The increasing interest in molecular genetics will result in new and complex stochastic processes. However, as in cell lineage studies and the example considered here, these new processes are combinations of familiar simpler stochastic processes with perhaps extended error structures. Nevertheless, it appears certain that many new problems in statistical inference and applied probability will arise in this area. Acknowledgements This research was supported by NICHID grant # HD36071 and an Australian Research Council small grant.

References [1] Ashley, E.A. & Sherman, S.L (1995) Population dynamics of a meiotic/mitotic expansion model for the fragile X syndrome. Am. J. Hum. Genet 57: 1414-1425. [2] Bui, Q.M. & Huggins, R.M. (1999) The random coefficient bifurcating autoregression model in cell lineage studies. J. Stat. Plan. & Inf. 81; 253-262. [3] Bat O, Kimmel M, & Axelrod DE (1997) Computer simulation of expansions of DNA triplet repeats in the fragile X syndrome and Huntington 92s disease. J.Theor.Biol. 188: 53-67

RANDOM COEFFICIENT MODELS

197

[4] Chib, Siddhartha and Greenberg, Edward (1995) Understanding the Metropolis-Hastings Algorithm American Statistician V. 49, No. 4 327335. [5] Cowan, R. (1984). Statistical concepts in the analysis of cell lineage data. Proceedings of the 1983 Workshop on Cell Growth and Division., 18-22, LaTrobe University. [6] Cowan, R. and Staudte, R.G. (1986). The bifurcating autoregression model in cell lineage studies. Biometrics. 42, 769-783. [7] Dana C. Crawford, Charles E. Schwartz, Kellen L. Meadows, James L. Newman, Lisa F. Taft, Chris Gunter, W. Ted Brown, Nancy J. Carpenter, Patricia N. Howard-Peebles, Kristin G. Monaghan, Sarah L. Nolin, Allan L. Reiss, Gerald L. Feldman, Elizabeth M. Rohlfs, Stephen T. Warren, and Stephanie L. Sherman (2000) Survey of the Fragile X Syndrome CGG Repeat and the Short-Tandem-Repeat and SingleNucleotide-Polymorphism Haplotypes in an African American Population. Am. J. Hum. Genet. 66 480-493. [8] Dierckx, P. (1993). Curve and surface fitting with splines, Clarendon: New York. [9] Eichler, E.E., Macpherson, J.N., Murray, A. Jacobs, P.A., Chakravarti, A. and Nelson, D.L. (1996) Haplotype and interspersion analysis of the FMR1 CGG repeat identifies two different mutational pathways for the origin of the fragile X syndrome. Human Molecular Genetics. 5 319-330. [10] Gawel B & Kimmel M(1996) The iterated Galton-Watson process. J.Appl.Prob. 33: 949-959. [11] Gelman, A. and Rubin, D.B. (1992). Inference from iterative simulation using multiple sequences (with discussion). Statistical Science 7,457511. [12] Gelman, A., Carlin, J.B., Stern, H.S. and Rubin, D.B. (1995). Bayesian Data Analysis, Chapman & Hall: London. [13] Huggins, R.M. & Loesch, D.Z. (1998) On the analysis of mixed longitudinal growth data. Biometrics [14] Huggins, R.M., Loesch, D.Z., & Sherman, S.L. (1998) A branching nonlinear autoregressive model for the transmission of the fragile X dynamic repeat mutation. Ann. Hum. Genet. 62 337-347. [15] Huggins, R.M. & Basawa, I.V. (1999) Extensions of the bifurcating autoregressive model for cell lineage studies. J.Appl.Prob. 36 1225-1233.

198

HUGGINS, QIAN AND LOESCH

[16] Morris, A., Morton, N.E., Collins, A., Macpherson, J., Nelson, D. and Sherman, S. (1995) An n-allele model for progressive amplification in the FMR1 locus. Proc. Natl. Acad. Sci. USA 92 4833-4837. [17] Morton, N.E. and Macpherson, J.N. (1992) Population genetics of the fragile X syndrome: multiallelic model for the FMR1 locus. Proc. Natl. Acad. Sci. USA 89 4215-4217. [18] Muirhead (1981).Aspects of multivariate statistical theory, John Wiley: New York. [19] Murray, M.A., Macpherson, J.N., Pound, M.C., Sharrock, A., Youings, S.A., Dennis, N.R., McKechnie, N., Lineham, P., Morton, N.E., Jacobs, P.A.(1997). The role of size, sequence and haplotype in the stability of FRAXA and FRAXE alleles during transmission. Hum. Mol. Genet. 6:173-84. [20] Park, Jeong-gun &; Basawa, I.V. (2000) Optimal estimating equations for mixed effects models with dependent observations. Technical report 2000-16, Department of Statistics, University of Georgia. [21] Robert, C.P. (1994) The Bayesian Choice. New York: Springer-Verlag [22] Robert, C.P. (1998). Discretization and MCMC Convergence Assessment, Springer-Verlag: New York. [23] Staudte, R.G., Zhang, J., Huggins, R.M. and Cowan, R. (1996). A reexamination of the cell lineage data of E. O. Powell. Biometrics., 52, 1214-1222.

199

RANDOM COEFFICIENT MODELS

Appendices In the appendices we derive the joint and conditional densities required to estimate the model parameters using MCMC, and then give the MCMC algorithm.

A

Joint Densities

Using the notations introduced in the paper, it follows that the joint conditional density of g(Y) given Θ, 7 and the observed ancestor of each family is H

{g(Y)\θ,Ί}

P

Nh

nh=fhfk

ί>{5(Y)|ΘJQΠ Π h=lf=l nhf

H Nh

ΐ[p{9(Yhfki)\β,βh,βf\σ2,δ2F}

k=l 1=1

h=l /=1 k=l

{g(Yhfki)-Xhfk[β 2(PpIF(hfk)

+ σ*) (A.1)

The joint prior density of the parameter Θ given 7 is H Nh

π(θ|7)

H

Σ

Π Π Wrl /o) Π W Λ o ) h=lf=l

h=l

H Nh /ι=l/=l H X h=l

{

1

ff

Λ=l/=1

200

HUGGINS, QIAN AND LOESCH

x|Σ Λ 0 Γ* / 2 exp {-^ x|ΣoΓ 1/2 exp {—tr&^iβ

- βo)(β - βoΫ}]

(A.2) The joint prior density of the hyper-parameter 7 is τr(7)

= ω/0(Σ;o|ξ/θιΛ/o) ωω(Σ!fto|ξfco,ΛAθ) ^o(Σo|ξo,i?o) • bo{βo\b,Σ)

x(2π)-"/ 2 |ΣΓ 1 / 2 exp{-i(ft,-6)'Σ- 1 (A-6)}

(A.3)

where Γ m ( ) is the multivariate gamma function (see Muirhead 1981, pp. 61-63). In the simulations, the value of b was taken to be the estimate of β in the spline regression model g(Yhfki) = Xhfkβ We also used the following values in the simulation in such a way that they should have very weak effects on the simulation results: v = λ = up = \ F = 10,= Σ = RQ = Rho — Rfo — 10^10 with Jio being the 10 x 10 identity matrix, and £o = 6ιθ = ζfo = 50.

B

Posterior Conditional Densities

In order to simulate the posterior distributions using MCMC methods we need a number of posterior conditional densities. The joint posterior density of Θ, 7 given the observed ancestor of each family is

The conditional posterior density of θ given 7 and the observed ancestor of each family is p{g(Y)|Θ,γ}7r(Θ|Ύ)

fR

.

201

RANDOM COEFFICIENT MODELS

The conditional posterior density of 7 given Θ and the observed ancestor of each family is

C

Simulating the Posterior Densities by MCMC

To simulate the posterior distributions of the β^s and βί s, we first generate a random sample for (©,7) from the posterior density π(θ,7|Y). Then we extract those values of βh and βf in the sample, which clearly comprises a 1 random sample from the posterior distributions of (/?£, βj) . So the question is how to generate a random sample from the posterior density π(θ,7|Y). To answer this question we apply an Markov chain Monte Carlo method (MCMC) which uses a Metropolis-Hastings acceptance-rejection algorithm alternately to the two blocks of the random quantity (©,7). In other word, this is the so-called MCMC block-at-a-time algorithm (refer to Chib and Greenberg, 1995). We list the algorithm in the following: Obtain the initial values (7(0), 1. Generate a matrix 7^°) from the prior π(7). 2. Generate a vector θ<°) from the prior 7r(e Update ( 7 ϋ), θ ^ ) to {^j+ι\ θ ^ + 1 ) ) . Repeat for j = 0,1,

, N - 1.

1. Use an M-H acceptance-rejection algorithm to generate a matrix 7^ + 1 ) from the conditional posterior density π ( 7 | Y , θ ^ ) : - Generate an initial matrix 70 from τr(7). - Repeat for i = 0,1, , / — 1. - Generate a matrix 7' from π(7) and u from the uniform distribution ZY(/, 00). - If u < Oij(ji ,V), set 7^\ = 7 ; . The acceptance probability is defined by

a 7(7ί ( τ ω τ7 V mini )

' ~

π ( θ | 7 )

U(Θ(%?Y

l

- Finally set 2. U s e a n M - H a c c e p t a n c e - r e j e c t i o n a l g o r i t h m t o g e n e r a t e a v e c t o r QU+ι) f r o m t h e c o n d i t i o n a l p o s t e r i o r d e n s i t y π ( ^ ^

202

HUGGINS, QIAN AND LOESCH

- Generate an initial vector ΘQ fr°m 7r(θ|7^ + 1 )). - Repeat for q = 0,1, , Q — 1. - Generate a matrix θ ' from π(θ|7^ + 1 ^) and u from the uniform distribution Z//(/, oo). - If u < OLQ(QΨ,&), bility is defined by

set θ ^ i = ©'. The acceptance proba-

- If u > aθ(θφ,&), set Θ% = θ[j). - Finally set θ^ + 1 ) = θ § } . Return the sequence {(7<°\ β ^ ) , ( 7 ( 1 ) , θ W ) , • • •, (7 ( Λ Γ ) , Θ(JV))}. As in any MCMC method, the sequence generated by the above algorithm is actually a Markov chain with τr(θ,7|Y) a s i*s invariance density. So by ignoring the first iVo values the sequence can be approximately regarded as a random sample from the posterior density τr(θ, 7|Y), provided that both NQ and N are sufficiently large. In our simulations we found that when taking / = Q = 50 the simulated Markov chain became convergent after No > 48 runs.

205

Institute of Mathematical Statistics LECTURE NOTES — MONOGRAPH SERIES

SEMIPARAMETRIC INFERENCE FOR SYNCHRONIZATION OF POPULATION CYCLES RE. Greenwood Dept. of Mathematics, Arizona State University, Tempe, Arizona and Dept. of Mathematics, University of British Columbia, Canada D.T. Haydon Centre for Tropical Veterinary Medicine Easter Bush, Roslin, Midlothian, U.K.

1

Introduction

We consider a dynamic random field. On each of a discrete array of sites is located a hyperbolic dynamical system perturbed by noise. We assume that the dynamics are identical, with a unimodal limit cycle and that the perturbations are independent, centered, and with the same distribution. In addition the individual processes are coupled with one another in a homogeneous pattern. The coupling may be global, in which case we are thinking of a mean-field type of system. Or the coupling may be local, the coupling strength between each site and its neighbors attenuating with distance. The application we have in mind is to cycling populations of animals, where the log of each local population increases roughly linearly to an apparent critical point from which it falls precipitously to a minimum. The data is discrete in space and time, being based on periodic reports from catchment or reporting regions. In a previous study [1] we focussed on data from Canadian lynx populations. Lynx population cycles are known to follow those of snowshoe hare. Previous analyses of this data have been concerned with inferring the length and regularity of the evident population cycles. We consider in [1] the very different challenge of estimating a parameter identified as strength of coupling among populations. That paper is primarily addressed to data analysis and interpretation. The emphasis in this report is on the steps involved in arriving at a suitable model and estimator. We explore some of the difficulties posed by this rather unusual problem. In forthcoming studies we, with colleagues, will apply the method described here to cycling population data from Canadian muskrat and mink, and from the greysided vole of Hokkaido.

206

2

GREENWOOD AND HAYDON

Development of a Coupled Evolving Phase Field Model

To begin, let us introduce a general definition of synchronization applicable to random fields. Let X = {Xa,i € I,t = 0,1,2,...} denote the values at site i and time t of a dynamic random field. Let S = {£(•)*, t > 0} denote a real-valued functional or collection of functionals of {Xis, i € / , s < *}. Note that we allow S evaluated at time t to depend on data up to and including time t. Define S'-synchronization to mean {SX)t = 0, almost surely, for t in some time-set T. This gives us a flexible definition of exact synchronization. But a random field will not be exactly synchronized. In order to formulate a statistically useful definition we need to allow for and to measure departures from synchronization. For example, we probably want the mean and variance of S applied to random data to be small. We are led to the following: Definition. A dynamic random field X departs from 5-synchronization by no more than e over the time interval T if E(SX)Ί < e for all t eT. One might use other norms, e.g. Lι or Kullbach-Leibler instead of L 2 . However this definition has a familiar form and is computationally convenient. At each site i in an array of sites / is located a "skeleton" process which, in the absence of any noise or coupling can be written -ΪM+i = HXith

i e /, t = 0,1,2,....

(2.1)

In this treatment we write the dynamics as of first order in discrete time. We assume there is a unimodal limit cycle t(t),t = 0, . . . , p , ^(0) = t{p), where, since / is the same for each i, the form of the cycle, ί^ and its period, p, are the same for each site i. For definiteness let ^(0) be the minimum point of the unimodal cycle. If we picture the deterministic field (1) running in equilibrium, we see the periodic cycle ί{ ) being executed at each site, the only difference between sites being the phase, which will differ if the initial points X^o were not identical. Now suppose we watch, instead, the stochastic field, X^t+ι=f(Xit)

+ eiu

t 6 / , t = 0,l,2,...

(2.2)

where en represents a small, identically distributed, centered noise. A theorem of deterministic dynamics called the "shadowing lemma" implies that the stochastic paths of (2) shadow the paths of (1), where "shadow" means that they remain in a distributional neighborhood of the deterministic path up to a time shift. The differences among the paths at the various sites, then, in addition to the small width of this neighborhood, are the phases, which change in a random way. The neighborhood width depends on the

207

INFERENCE FOR SYNCHRONIZATION

noise variance. This picture motivates a model for coupling based on the randomly evolving phases of the components. We now move our attention from the evolving random field of population levels, (2), to a corresponding evolving random field of phases. Going back to the deterministic model (1) we can unambiguously define the phase φa at site 2, time £, to be the time fraction of the current cycle, ^(0),... ,^(p), which has been accomplished at time t. Then each phase φu is in the interval [0,1). The arithmetic for phases is modi. We think of the set of points {Φit,i G /} as a set on the circle of circumference 1. As time advances the points progress around the circle. Since in fact we wish to consider the phase field associated with the stochastic field (2), where the "limit cycle" is a stochastic perturbation of the deterministic one, we may not see a unique minimum for each cycle, and the definition of φu may be ambiguous. We will assume that the noise en is small enough so that the ambiguity can be resolved by a device described in the next section. For the moment let us ignore this problem and assume that in the stochastic model the phase φn is defined as the fraction of the current orbit which has been traversed at time t by each path X{ of the stochastic field (2). We describe the structure of the phase field φa, i G /, t = 0,1,2,..., by writing Φiί+\ = Φit + 9it + tit

mod 1,

(2.3)

where gu is the fraction of the current orbit traversed at site i at time t. Now we introduce a hypothetical coupling force into the phase field which will shift the phase at each site i and time t in the direction of the "mean phase". For this purpose we need to devise an appropriate definition of mean for a set of random points on a circle. As mentioned above we identify the phase values with points on a circle of circumference 1. Between points x,y on this circle, let Δ(x,y) denote the signed smallest arc measured counter-clockwise between them. Then Δ(x,y) is positive or negative according to whether x leads or lags y, and (x,y)|<0.5. For each set of phases φu with t fixed, let φt be a solution of ψt)=0.

(2.4)

Then φt is almost surely uniquely defined and can be interpreted as the mean phase at time t on the circle, or equivalently on [0,1) mod 1. To model phase coupling among sites i in the array /, we insert a coupling term in expression (3) which moves the phases at time t + 1 in the direction of the mean phase at time t, and write Φi,t+i = Φit + 9it ~ cA{φiu φt) + eit mod 1,

i e /, ί = 0,1,....

(2.5)

208

GREENWOOD AND HAYDON

The coupling strength c can take values in [0,1). Expression (5) says that in each time step, from t to t + 1, three changes occur in the phase at each site: (a) it advances according to phase dynamic ga inherited from the limit cycle of the "skeleton" process (1), (b) it is shifted in the direction of the mean of the phases as defined by (4) by a proportion c of its distance from this mean, and (c) it is perturbed by a centered random effect. About the en we assume only that for each t they are conditionally independent given the past of the process {φu}, with the same conditional distribution for each i and t. In order to simplify our computations we would like to omit the modi in relation (5). This is possible if it does not happen, or happens only rarely, that the combined effect of (a), (b), and (c) at a particular timestep t is large enough that the phase increment at a single step exceeds 1. In the model we make this assumption. If it is occasionally violated in the data we make an appropriate adjustment. Assuming, then, that the noise is sufficiently small relative to the coupling, (5) can be written as +i,^+1)

= (1 - c)Δ(φiuφt)

+ eit + {git - φt+i + Φt),

<2 ^

The last grouped term in (6) has mean 0 as can be seen from considerations of symmetry. In addition, for each t the various values in i are conditionally independent given the past of the phase field. We have assumed that the noise increments en are small and centered and conditionally independent given the past of the phase field. Without altering this assumption we can regard the last grouped term in (6) as part of the noise and write (6) as Δ(&, t + i, φt+ι) = (1 - c)A{φiu φt) + ηiu

i E /, t = 0,1,....

(2.7)

Relation (7) describes a stochastic process of ARl-type. It differs from the usual AR1 model in that the process is a vector-valued function of a distinct vector process. The underlying process has vectors of phases as its values, whereas the process defined by (7) has values which are vectors of differences of these phases from the evolving central phase. Nevertheless the least squares type estimator is consistent for (1 — c) and that is what we used in [1]. One needs to know that there is a stationary law for the process defined by (7). This can be shown by an argument along the lines of arguments in Meyn and Tweedie's book [2], The fact that the state space is bounded simplifies the situation.

209

INFERENCE FOR SYNCHRONIZATION

If we wish to model local rather than global coupling, then the central phase as seen from site i should be computed from the phases at sites near i. We then define the local centre of phases at i, to be the almost surely unique phase φit which is the solution of
i e J, t = 0 , 1 , . . . ,

(2.8)

3

where the numbers α^ are weights reflecting the fraction of the total coupling at site i to be attributed to interaction between sites i and j , Σ aij — 1. We _

_

3

arrive at a version of (7) with φit replacing φt in each equation. If edge effects are disregarded, and if α^ depends only on i — j , the system remains spatially homogeneous in law. In any case, a stationary distribution exists for the process and the least-squares-type estimator is consistent for 1 — c. Let us return to the notion of departure from 5-synchronization by no more that e, as defined early in this section. Perfect synchronization corresponds to departure zero, and higher levels of departure correspond to less synchronization. The synchronization measure we choose here is the meansquare deviation of the phase field from its mean φt, or φit, as defined by (4) or (8),

The departure from synchronization, then, is defined as

E{SX)2t = ±-ΣEA(φiuφt)2.

3

(2.10)

Estimation

In an initial application of the model presented here [1], we study synchronization in the well-known Canadian lynx data sets. Here we summarize a few of the results and refer to [1] for the full treatment and references. The first data set was compiled by Elton and Nicholson, in 1942, [4] spans the years 1821 to 1891 and is organized over six trading regions of the Hudson's Bay Company. The second data set, compiled by Statistics Canada, [5] is organized over eight Canadian provinces and territories and 2 spans the years 1919-1990. We used estimators of (1 - c) and σ where η is the noise increment in (7), which are analogous to well-known linear estimators for parameters of AR1 time series. The data analysis involves two difficulties peculiar to the phase-coupled model (7). As mentioned earlier it was not possible to identify uniquely

210

GREENWOOD AND HAYDON

the time points where minima of cycles occur in the data. Our treatment in [1] involves identifying the sets of possible minima and repeating the estimation procedure many times using random choices from these sets. We are exploring other possible ways to handle this problem in other data sets. A second difficulty involves discontinuities in the function A(φu,φt) at ±0.5, and a very few instances where A(φt+1^φt) < 0. Our handling of these is detailed in [1]. We mention these problems here to indicate the novel difficulties posed by data analysis in this context. The estimators we use for θ — 1 — c and for σ% are

θ = ϊ=i-ί=i n

i=\

i\ — l

_

2^ i\{φit,Φty t=l

(3.1)

/n(iV - 1) - 2. We can distinguish by looking at the residuals from these estimates the times at which synchrony is maintained by intermittent synchronizing events from the time periods of constant phase-coupling dynamics as expressed in expression (7). At the times of synchronizing events we see outliers in the residuals, whereas corresponding to periods of consistent coupling, data is clustered around the regression line. Times, thus detected, of synchronizing events correspond to times of decrease of the estimator (9) of asynchrony. The coupling estimates we obtained were c ~ 0.054 for the earlier data set and c ~ 0.011 for the later one. With local weighting the estimate doubled to c ~ 0.096 for the earlier data set and remained about the same at c ~ 0.005 for the later data set. The numbers for the earlier data set are significant according to tests using the estimated variances, and are significantly greater than estimates produced from simulated cycles with randomized phases from a model fitted to this data. Thus, although the estimates c from the lynx data seem small, we believe that they measure the phenomenon of coupling as intended. An alternative statistical approach to the study of synchronization of population cycles uses the correlation structure of the logged abundance data as in e.g. Jolliffe [3]. With this method, amplitude variability may confound the detection of synchronization. Further, since phase coupling is not modelled, correlational methods measure phase coupling only very indirectly.

INFERENCE FOR SYNCHRONIZATION

211

Acknowledgements This work was supported by NSERC Canada and the Crisis Points Group of the Peter Wall Institute of Advanced Studies at U.B.C. The second author acknowledges support from the Wellcome Trust.

References [1] Haydon, D.T. and Greenwood, P.E. (2000). Spatial coupling in cyclic population dynamics: models and data. Theoretical Population Biology, 58, 239-254. [2] Meyn, S.P. and Tweedie, R.L. (1993). Markov Chains and Stochastic Stability. Springer, Heidelberg. [3] Jolliffe, I.T. (1986). Principal Components Analysis. Springer, New York. [4] Elton, C. and Nicholson, M. (1942). The ten-year cycle in numbers of lynx in Canada. J. Anim. EcoL, 11, 215-244. [5] Statistics Canada. 1921-1990. Catalogue 23-207, Fur Production, Ottawa, Canada.

213

Institute of Mathematical Statistics LECTURE NOTES — MONOGRAPH SERIES

PLUG-IN ESTIMATORS IN SEMIPARAMETRIC STOCHASTIC PROCESS MODELS Ursula U. Muller Universitat Bremen Anton Schick * Binghamton University Wolfgang Wefelmeyer Universitat Siegen Abstract Consider a locally asymptotically normal semiparametric model with a real parameter ϋ and a possibly infinite-dimensional parameter F. We are interested in estimating a real-valued functional a(F). If ά# estimates a(F) for known $, and ϋ estimates #, then the plug-in estimator ά^ estimates a(F) if ϋ is unknown. We show that ά^ is asymptotically linear and regular if ά# and $ are, and calculate the influence function and the asymptotic variance of ά^. If a(F) can be estimated adaptively with respect to #, then ά^ is efficient if ά# is efficient. If a(F) cannot be estimated adaptively, then for ά^ to be efficient, ΰ must also be efficient. We illustrate the results with stochastic process models, in particular with time series models, and discuss extensions of the results. Key Words: Empirical estimator, asymptotically linear estimator, influence function, regular estimator, Markov chain model, nonlinear regression, residual distribution, nonlinear autoregression, innovation distribution, stochastic equicontinuity, stochastic differentiability.

1

Introduction

Let Vn = {PnϋF : ^ G Θ,F G 7 } denote a sequence of semiparametric models, with θ one-dimensional and T a possibly infinite-dimensional set. We are interested in estimating a real-valued functional a(F). For each ϋ Research partially supported by NSF Grant DMS 0072174

214

MULLER, SCHICK AND WEFELMEYER

let ά# be an estimator for a(F) when ΰ is known, and let ΰ be an estimator for #. The plug-in estimator for a(F) is ά^, obtained by substituting ΰ for ΰ in ά#. In Section 2 we introduce a semiparametric version of local asymptotic normality, and a concept of asymptotically linear estimators for functionals on such models. This concept requires embedding the semiparametric model into a "nonparametric" supermodel which is to some extent arbitrary, although for specific types of stochastic models there usually is a natural choice. We recall two results which are essentially known, at least for i.i.d. models and Markov chain models. The first characterizes regular estimators among asymptotically linear ones; the second characterizes efficient estimators. In Section 3 we show that ά^ is asymptotically linear if ά# and ϋ are, and calculate the influence function of ά^. In Section 4 we assume that both ά# and ϋ are asymptotically linear and regular, and show that then ά^ is also regular. The characterization of regular estimators then allows comparison of the asymptotic variance of the plug-in estimator with the minimal asymptotic variance. In particular, if a(F) can in principle be estimated adaptively with respect to #, i.e., if knowledge of ΰ does not contain information about a(F)j then ά^ is efficient whenever ά# is efficient for known ϋ. lΐa(F) cannot be estimated adaptively with respect to #, then both ά# and ΰ must be efficient for ά^ to be efficient. For i.i.d. observations and estimators of the form ά# = a(F$), these results are due to Klaassen and Putter (1999). Sections 5 and 6 contain applications to models with i.i.d. observations and to Markov chain models, respectively. The discussion is heuristic in these two sections. Some extensions of the results are outlined in Section 7.

2

Characterizing Regular and Efficient Estimators

In this section we recall briefly characterizations of regular and of efficient estimators of real-valued functional in semiparametric models. For the i.i.d. case we refer to Bickel, Klaassen, Ritov and Wellner (1998), Chapter 3 and in particular Section 3.4. For Markov chains see Wefelmeyer (1999); for general models and counting process models see Andersen, Borgan, Gill and Keiding (1993, Chapter VIII); for general parametric models see Le Cam and Yang (1990). We will embed the semiparametric model in a "nonparametric" supermodel. The reason is that we want a sufficiently rich class of "asymptotically linear" estimators, and asymptotic linearity will be defined in terms of statistics approximating local likelihood ratios; see below. For n G N let Vn denote a collection of probability measures on some

215

PLUG-IN ESTIMATORS

measurable space (Ωn, Tn). Fix a sequence Pn in Vn and assume that the model is locally asymptotically normal at Pn in the following sense. There f is a linear space H with inner product (h,h ) and corresponding norm ||/ι||, there is a sequence of random linear functionals Sn on H, and for each h E H there are perturbations Pnh of Pn within Vn such that (2.1)

\\h\\ + oPn{l), under P n ,

(2.2)

with N denoting a standard normal random variable. The linear space H may be interpreted as (approximate) tangent space oίVn at Pn. Now consider a sequence of semiparametric submodels Vn — {PnϋF '• ^ G Θ J F G ^ } o f ? n , with θ one-dimensional and T an arbitrary, possibly infinite-dimensional set. We fix ΰ and F and consider perturbations ϋnu = ϋ + n~1/2u of ϋ and Fnυ of F, with v in some linear space V. In general, the appropriate rate may be different from n" 1 / 2 , but it will be n" 1 / 2 in our applications, which is why we have taken it to be n~ιl2 here. We assume that (2.1) and (2.2) hold at Pn = P n #F 5 and that the semiparametric model is locally asymptotically normal in the following sense. There are an element m in H and a linear operator D : V ->• H such that for Pnuυ = PnϋnuFnv * 7

log ^

= Sn(um + Dv) - I||um + Dt;|| 2 +

oPnϋF{l).

Prom (2.2), Sn(um + -Dv) => ||i4m + 2?v||JV

under Pn#F

The tangent space of the semiparametric model is H=[m] + DV, with [m] denoting the linear span of m. The one-dimensional space [m] is the tangent space for known F, obtained by perturbing ΰ. The space DV is the tangent space for known #, obtained by perturbing F. We assume that DV is closed, and that m does not belong to DV. We may think of m as the score function for ΰ. Let α(#, F) be a real-valued functional. It is called differentiable at (#, F) with gradient g Ίΐ g £ H and n 1 / 2 (α(τ? n ί z ,F n v ) - α(tf,F)) -> {g,um + Dv) for u G Λ, v € V.

(2.3)

The canonical gradient g is the projection of g onto if; it is therefore of the form ΰm + Dv.

216

MULLER, SCHICK AND WEFELMEYER

An estimator ά is called asymptotically linear for α(#, F) with influence function bifbeH and 1 2

n / (ά-a(ϋ,F))

= Sn(b) + oPnΰF(l).

(2.4)

An estimator ά is called regular for α(#, F) with ZzYm£ L if L is a random variable such that ι/2

n (ά

- a(βnu, Fnv)) => L under P n m ; for all u G # , Ϊ; G V.

(2.5)

The convolution theorem of Hajek (1970) says that L = \\g\\N + M

in distribution,

with M independent of N. This justifies calling a regular estimator a efficient for a(ΰ,F) if it is asymptotically normal with variance ||ff||2, nιl2{a - α(0, F)) =• \\g\\N under P n i 9 F . We have the following two characterizations: (1) An asymptotically linear estimator is regular for α(i?,F) if and only if its influence function is a gradient of α(??,F). (2) A regular estimator is efficient for α(#, F) if and only if it is asymptotically linear with influence function equal to the canonical gradient of a(ΰ,F). Now it becomes clear why we have introduced the "nonparametric" model. If we restrict attention to the semiparametric model, then there is only one gradient, the canonical one, and all regular and asymptotically linear estimators are asymptotically equivalent. In the examples of Sections 5 and 6 we will need to consider estimators whose influence functions are non-canonical gradients. The concept of asymptotically linear estimators is arbitrary in that it depends on the choice of "nonparametric" model; see Wefelmeyer (1991).

3

Asymptotic Linearity of Plug-in Estimators

Consider the problem of estimating a real-valued functional a(F) in the semiparametric model Vn = {PnΰF '• ΰ Eθ,F G T}. Fix ϋ and F. Suppose that for each τ near ΰ we have an asymptotically linear estimator aT of α(F), with influence function 6T. We assume that asymptotic linearity holds locally uniformly in shrinking neighborhoods of 7?, sup |n1/2(ai?n]ί _ a(F)) - Sn(bκJ\

= o P n ,,(l),

(3.1)

217

PLUG-IN ESTIMATORS and that Sn(bΰnu)

is stochastically differentiable at u = 0,

sup \Sn(bΰnJ

- Sn(b#) + (m,bϋ)u\ = oPnΰF(l).

(3.2)

Now let ϋ be asymptotically linear for ΰ, with influence function c,

By conditions (3.1) and (3.2), the plug-in estimator ά^ fulfills =

Sn(b^) +

=

Sn(bΰ -

oPnΰF(l) (m,b#)c)+oPnϋF(l).

This means that ά^ is asymptotically linear for a(F) with influence function (3.3)

b = b#-{m,bΰ)c

If (m, &#) = 0, then we can relax the assumption that ϋ is asymptotically linear to n1/2-consistency and still get that the plug-in estimator is asymptotically linear, now with influence function 6 = &#. Asymptotic linearity (3.3) of the plug-in estimator also follows if we replace conditions (3.1) and (3.2) by a non-uniform version of (3.1), - a(F)) = Sn(b#) + o P n , F ( l ) ,

(3.4)

and an expansion of ά^,

n1/2(^ ~ &*) = - K bϋ)nι'2φ

-ΰ) + oPnΰF(1).

(3.5)

An application is Example 3 in Section 6. R e m a r k 1. (Plug-in and sample splitting.) Our requirements (3.1) and (3.2) are stronger than the following conditions. For every bounded sequence

Sn(b*nun) - Sn(b»)

=

Sn(bΰnun)

+ o P τ ι , F (l),

(3.6)

=

-un(mM

+ oPnΰF(l).

(3.7)

Property (3.7) appears quite frequently in the literature. It has been been verified by Drost, Klaassen and Werker (1997), Jeganathan (1995), Koul and Schick (1997) and Kreiss (1987) for some time series models. A simple sufficient condition for (3.7) is given in Schick (2000) in the context of i.i.d. observations.

218

MULLER, SCHICK AND WEFELMEYER

It was shown by Klaassen and Putter (1999) in the context of i.i.d. observations that under these weaker conditions one can use sample splitting techniques to construct a modified version of the plug-in estimator with influence function b as in (3.3). Their construction can be generalized to stationary and ergodic Markov chains using the sample splitting techniques developed in Schick (2001). But this will not be pursued here. Remark 2. (Stochastic differentiability.) We will check stochastic differentiability (3.2) for specific types of processes in Sections 5 and 6, using stochastic equicontinuity of empirical processes. Here we indicate that (3.2) also follows from a locally uniform version of local asymptotic normality. Compare the proof of Theorem 6.2 in Bickel (1982). Since F is fixed in (3.2), we will omit it from the notation. Fix ϋ and set τ = ϋnu = ldΛ-n~ι'2u. Assume that the parametric family {Pn<β : ΰ G Θ} is locally asymptotically normal at #, log - ^

= uSnΰ(m)

- -u2\\m\\l +

oPnΰ(l).

Assume also that for τ near ϋ there is a tangent space Hτ such that for each bτ £ Hτ we have local asymptotic normality at r, l o g % ^ = Snτ(bτ) - hbT\\2T +

oPnr(l).

Then 1 2

2

1

2

2

2

If bτ is continuous at r = ΰ in an appropriate sense, PUτbτ will be approximately equal to Pnϋ,um+bϋ (For a more explicit argument it would be convenient if the sequence of "nonparametric" supermodels Vn were indexed by an infinite-dimensional parameter; see LeCam (1986, Chapter 11) and Greenwood and Wefelmeyer (1991, Section 4).) Hence log dPnτbg

dPnΰ

=

log- dpn*,™+*>*

=

Sn4um + bΰ)--\\um + bΰ\\l + oPnϋ(l).

°g

dPnϋ

(3.9)

If both Snτ and || | | τ are continuous at r = ϋ in an appropriate sense, we obtain from (3.8) and (3.9): = o P n i 9 (l). Stochastic differentiability (3.2) requires that the supremum over |ιt| < Δ is stochastically negligible. For this, we need a corresponding strong version of local asymptotic normality, as introduced in Fabian and Hannan (1985, Section 9.1).

219

PLUG-IN ESTIMATORS

4

Efficient and Adaptive Plug-in Estimators

We continue the discussion of plug-in estimators under additional assumptions. As in Section 3, let a(F) be a real-valued functional of F. Fix # and F. For r near ϋ let άτ be a locally uniformly asymptotically linear estimator of a(F) in the sense of (3.1), and let ΰ be an asymptotically linear estimator of ΰ. Now assume, in addition, that a(F) is differentiate (2.3) at (ΰ,F), that ά# is regular at (#,F) for known ΰ, and that ϋ is regular at (ΰ,F). Then we can decompose the canonical gradient of a(F) and the influence function of the plug-in estimator ά^ as follows. Let cp denote the canonical gradient of ϋ when F is known. It is of the form CF = tm with t determined by -ΰ) = u = {tm, um), i.e., t = ||m||~ 2 and cF = \\m\\~2m. The squared length ||m|| 2 of m may be called the Fisher information for ϋ when F is known. Let Dvm be the projection of m onto DV. When ϋ is unknown, the canonical gradient of ϋ is characterized by three properties. It is in H = [m] + DV, orthogonal to DV, and its projection onto [m] is cp Hence it is of the form c = t(m — Dvm) , with t determined by (c — ϋp,m) = 0, i.e., t = (m — Dvm, m)~ι = ||m — Dvm\\~2. Hence the canonical gradient of ΰ is c=\\m-Dvm\\-2(m-Dvm). The squared length \\m — Dvm\\2 of m — Dvm is the Fisher information for ΰ. Let g# denote the canonical gradient of a(F) when ϋ is known. The canonical gradient ~g of a(F) for unknown ΰ is characterized by three properties. It is in H — [m] + DV, orthogonal to m, and its projection onto DV is gϋ. Hence g — g# is of the form tc, with t determined by (m,p) = 0, i.e., t = -(m,g#), i.e., g = gΰ-{m,g#)c.

(4.1)

Since ϋ is regular for ϋ, we obtain from characterization (1) of Section 2 that the influence function of ϋ is a gradient of ΰ, say c. Hence (4.2) Since ά# is regular for a(F) when ϋ is known, we obtain from characterization (1) of Section 2 that the influence function of ά# is a gradient of a(F) for known ΰ, say g#. Hence (4.3)

220

MULLER, SCHICK AND WEFELMEYER

From (3.3) we obtain that the plug-in estimator ά^ is asymptotically linear for a(F), with influence function (4.4)

9 = 9ΰ -(m,g#)c. Prom (4.1) and (4.4) we obtain 9 -9" = 9ΰ -9# - (m,g# - g#)c - {m,gϋ)(c - c).

(4.5)

It follows from (ra, c) = 1 and relations (4.2) and (4.3) that g-g is orthogonal to Ή. Hence g is a gradient. Characterization (1) of Section 2 now implies that the plug-in estimator ά^ is regular. Since g — g and c — c are orthogonal to if, the asymptotic variance of the plug-in estimator ά^ is -\\m - Dυm\\-2{m,gΰ

- gΰ)2 - 2(c - c,gϋ - S#)(m,3#

If ά# is efficient for a(F) when i? is known, and ΰ is efficient for i?, then 9ΰ — 3i9 a n d c = c. Hence g — ~g by (4.5), and the plug-in estimator ά^ is efficient for a(F). By (4.1), its asymptotic variance is

llsll2 = ll^ll2 + \\m - Dvm\\-2(m,gϋ)2.

(4.7)

If ά^ is efficient for a(F) when ΰ is known, then by (4.5) the influence function of the plug-in estimator ά^ is g = g-(m,gΰ){c-c),

(4.8)

and by (4.6) the asymptotic variance of α^ is 2

2

2

(4.9)

If ϋ is efficient for #, then by (4.5) the influence function of the plug-in estimator ά^ is 9 = 9 + 9ϋ -9ϋ ~ {™>,9ϋ ~ g#)c,

(4.10)

and by (4.6) the asymptotic variance of ά^ is

llίll2 + \\g* - VoW2 ~ \\m - Dvm\\-2(m,gϋ - gϋf.

(4.11)

We say that a(F) can be estimated adaptively with respect to ΰ if the asymptotic variance bound for a(F) is not decreased by knowing ΰ. This is the case if and only if Ίj = ~gϋ. By (4.1), this is equivalent to ( m , ^ ) = 0. Then ~g does not depend on c. Hence the plug-in estimator ά$ is efficient whenever ά# is efficient for known #, and the asymptotic variance is ||g#|| 2 . We say that F can be estimated adaptively with respect to ϋ if for every differentiable functional a(F) the asymptotic variance bound for a(F) is not decreased by knowing ΰ. This is equivalent to orthogonality of m and DV. Then ϋ can also be estimated adaptively with respect to F , and c = cp = 2

221

PLUG-IN ESTIMATORS

5

The i.i.d. Case

If we have i.i.d. observations ΛΊ,... ,X n , then a natural candidate for the "nonparametric" model of Section 2 is the usual nonparametric model, with completely unknown distribution Q(dx) of X{. (Larger nonparametric models are obtained by allowing the observations to be dependent.) Fix Q. Set Qh = J h(x)Q{dx) and H = L2fl(Q) = {he L2(Q) : Qh = 0}. For h E H set The approximation is meant in the sense of Hellinger differentiability. The joint law of X L , . . . , Xn is Pn = Qn. We have local asymptotic normality:

(Qhψ2N, with N standard normal. This is (2.1) and (2.2) with

V

,

(h,ti) = Q(hti).

Remark 3. (Stochastic differentiability.) In the stochastic differentiability condition (3.2), the parameter F is fixed, and we may omit it. Let {Q$ : ϋ G Θ} be a parametric family of distributions of X{. For r near ϋ let bτ G ^2,o(Qr) Stochastic differentiability (3.2) follows if bτ is differentiate at r = ϋ in an appropriate sense. By Taylor expansion, sup

H<Δ'

=*P»«U)

From QT6T = 0 we obtain by taking the derivative under the integral, (5.2) with nriΰ = dτ=#dQτ/dQΰ the score function for ϋ. Relations (5.1) and (5.2) imply stochastic differentiability (3.2). A proof for fixed bounded sequences u = un is in Schick (2000). For (5.2) it is essential that Qτbτ = 0, in other words, that bτ is in the nonparametric tangent space, i.e., that bτ(Xi) is a statistic which approximates a local likelihood. n -i/2 γj}=ι

222

MULLER, SCHICK AND

WEFELMEYER

For stochastic differentiability (3.2) to hold, the function b# need not be differentiable. For r near ϋ consider the empirical process

If the collection of functions 6 r , r near #, fulfills an appropriate bracketing condition, then vnτ is stochastically equicontinuous at r = ϋ\ For each ε, η > 0 there is δ > 0 such that limsupP n #( sup

< ε.

\vnτ -vnϋ\>η)

Such a result was first proved by Daniels (1961) and Huber (1967) to obtain asymptotic normality of the maximum likelihood estimator under weak conditions on the score function; for a general version see Pollard (1985). For T = ΰnu we obtain sup =

OPn

,(l).

(5.3)

From QTbT = 0 we obtain

= Qΰbτ-Qτbτ

= ~(τ (5.4)

Relations (5.3) and (5.4) imply stochastic differentiability (3.2). Example 1. (Nonlinear regression.) Let (Xi, YΊ),..., (X n , Yn) be pairs of real observations with The εi are i.i.d. with density f(y) which has mean zero and finite variance but is unknown otherwise. The X{ are independent and independent of the Si and have unknown distribution function G(y). The pair (Xi,Yi) plays the role of Xι in the general setting. The model is semiparametric. The distribution of (Xi, Yι) is

Q»fG(dx,dy)

= dG{x)f(y - r(β,x))dy.

Fix 7?, / and G. We introduce perturbations

fnz(y) dGnw{x)

= =

ι 2

dG{x){\+n- l w(x)).

223

PLUG-IN ESTIMATORS

The density fnz must integrate to one and have mean zero, so z runs through

Z = {zβ L2(f) : Jz(y)f(y)dy = 0, J yz(y)f(y)dy = 0}. The function Gnw must be a distribution function, so w runs through L2fl{G) = {we L2(G) : j w{x)dG(x) = 0}. The perturbed distribution of (X^Yi) is dx d

Qnuzw(dx,dy) = Q#nufnzGnw( i y) 12

= QϋfG{dx, dy) (l + n- '

= dGnw{x)fnz{y

-

r(ϋnu,x))dy

(ur(0, x)i{y - r(0, *))

where r(ϋ,x) is the derivative of r(ΰ,x) with respect to #, and ^(y) = —f'{y)lf{y) is the score function for location of the error distribution. Hence the tangent space H of the nonlinear regression model consists of functions h{x, y) = ur(ϋ, x)i(y - r(tf, x)) + z(y - r(tf, x)) + tι (a ). It is therefore of the form ~H = [m] + DV of Section 2, with V = Zx L2,o(G), m(z,y) = r(i9, x)^(y - r(tf, a:)),

Dv(a:, y) = z(y - r(ΰ, x)) + w(x).

Note that by taking the derivative under the integral,

E(ε£(ε)) = Ix£(x)f{x)dx = - ί xf'(x)dx = 1.

(5.5)

Note also that w(x) is orthogonal to both m(x, y) and z(y — r(ΰ, #)), so that both ϋ and / are adaptive with respect to G. We want to estimate the expectation

= Jk(y)f(y)dy of an /-square-integrable function k under the error distribution. The usual estimator is the empirical estimator based on the estimated errors,

n

with ii = Yi—r(ΰ, Xi). A natural estimator of?? is the least squares estimator i?, which solves

224

MULLER, SCHICK AND WEFELMEYER

The least squares estimator is asymptotically linear with influence function (Er(ϋ,X)2)-ιr(ΰ,x)(y-r(ϋ,x)).

c(x,y) =

We have (m,c) = Er{ϋ,X)2 by (5.5), and c J_ DV since Jyz(y)f{y)dy = 0 for z G Z. Hence c is a gradient of ΰ. The empirical estimator of a(f) = Ek(ε) based on true innovations is

Its influence function is

For v = (z,w) G Z x I/2,o(G) we have

n^WA.) - α(/)) -> £(*(φ(e)) = (», Dυ). Hence g$ is a gradient of α(/) when Ί9 is known. It fulfills

Hence by Remark 3, an appropriate bracketing condition on the collection of functions bτ(x,y) = k(y — TX) — Ek(ε) implies stochastic differentiability (3.2) of the form

sup M<Δ

Σ ι uEr(4,X)E(l(ε)k(ε))\=oPa,/G(l). i=

It follows from (3.3) that the plug-in estimator ά^ is asymptotically linear for a(f) with influence function 9

=

9ϋ-{m,gϋ)c

-

k(ε) -Ek(ε)

-Er(ΰ,X)E(ί(ε)k(ε))(Er(ΰ,X)2)-1r(ΰ,x)ε.

Efficient estimators for ΰ are constructed in Schick (1993). The canonical gradient ~g and an efficient estimator for Ek{ε) are in Mύller and Wefelmeyer (2000a).

225

PLUG-IN ESTIMATORS

6

Markov Chain Models

Let XQ, ..., Xn be observations from a homogeneous and uniformly ergodic Markov chain with transition distribution Q(x)dy) and invariant law π(dx). Assume for simplicity that the chain is stationary. The natural "nonparametric" model of Section 2 is described by the collection of all such transition distributions. Fix Q. Set Qxh = f Q(x,dy)h(x,y) and H = {he L2(π ® Q) : Qxh = 0}. For h e H set Qnh(x,dy) = Q(x,dy)(l + rΓ The approximation is meant in the sense of Hellinger differentiability for Markov chains. The joint law oϊ X$,... ,Xn is Pn{dx0, . ,dx n ) = π(dxo)Q{xo,dx\)

Q(xn-Udxn).

We have local asymptotic normality:

i = ι

with N standard normal. This is (2.1) and (2.2) with 1

2

Sn(h) = n" / Σ h{Xi-UXi),

(Λ, ti) = π ® Q(hti).

2=1

Remark 4. (Stochastic differentiability.) The arguments of Remark 3 translate to stochastic process models. Stochastic equicontinuity for Markov chains was obtained by Ogata (1980) in connection with asymptotic normality of maximum likelihood estimators. Results for general discrete-time stochastic processes are in Andrews (1994) and Andrews and Pollard (1994). Let {Q$ : ϋ G Θ} be a parametric family of transition distributions of X{. For r near ϋ let bτ be π r ® Qr-square-integrable with f Qr(x,dy)bT(y) = 0 for all x. The score function for ϋ is

226

MΪJLLER, SCHICK AND WEFELMEYER

If the functions bτ fulfill an appropriate bracketing condition for r near #, we have stochastic differentiability (3.2) of the form sup H<Δ

Example 2. (Nonlinear autoregression.) The observations XQ, ... ,Xn are real with The ε; are i.i.d. with density f(x) which has mean zero and finite variance but is unknown otherwise. Conditions for uniform ergodicity are in Bhattacharya and Lee (1995) and An and Huang (1996). The model is semiparametric, with transition distribution (x, dy) = f(y - r(0, x))dy. Fix ϋ and /. We introduce perturbations ϋnu fnv(x)

= ϋ + n-ι'2u, = f(x){l +

n-ι'2v{x)).

As in the regression example, Example 1, the function v runs through V = {υ e L2(f) : ίv{x)f(x)dx = θjxv(x)f(x)dx

= 0}.

The perturbed transition distribution is = fnυ(y x, dy) (l + n-1'2(ur{ϋ,χ)l(y

-

r{ϋnu,x))dy

- r(0, x)) + υ(y - r(ΰ,

where r(ϋ,x) is the derivative of r(ϋ,x) with respect to ϋ, and i(x) = —f'(x)/f(x) is the score function for location of the innovation distribution. Hence the tangent space H of the nonlinear autoregressive model consists of functions h{x,y) = ur(ϋ, x)i(y - r(0, x)) + v{y - r(0, x)). It is therefore of the form H = [m] + DV of Section 2, with m(z, y) = r(ΰ, x)i(y - r(ΰ, x)),

Dυ(x, y) = v(y - r(#, x)).

227

PLUG-IN ESTIMATORS We want to estimate the expectation

= Jk(x)f(x)dx of an /-square-integrable function k under the innovation distribution. The usual estimator is the empirical estimator based on the estimated innovations,

ά= -n ] < with έi = Xi — r(ϋ,Xi-\). estimator #, which solves

A natural estimator of ΰ is the least squares

The least squares estimator is asymptotically linear with influence function

We have (ra,c) = Er(ΰ,X)2 by (5.5), and c J_ DV since Jxv(x)f(x)dx = 0 for v € V. Hence c is a gradient of ΰ. The empirical estimator of α(/) = Ek(ε) based on true innovations is 1^ Its influence function is

9ϋfa y) = k(y - ^ ) - Ek(ε). We have n 1/2 (α(/™) - α(/)) -> B(fc(e)t;(ε)) = (g*,Dv)

for t; G V.

Hence g$ is a gradient of a(f) when i9 is known. It fulfills " (m,gϋ) =

Er(ϋ,X)E(ί(e)k(ε)).

By Remark 4, an appropriate bracketing condition on the functions bτ(x, y) = k(y - rx) - Ek(ε) implies stochastic differentiability (6.1). It follows from (3.3) that the plug-in estimator ά^ is asymptotically linear for a(f) with influence function 9

= gϋ-{rn,g$)c = jfc(ε) - Ek{ε) -

Er{ϋ,X)E{i{ε)k{ε))(Er{ΰ,X)2)-ιr{ϋ,x)ε.

228

MULLER, SCHICK AND WEFELMEYER

Efficient estimators for ϋ are constructed in Drost, Klaassen and Werker (1997) and Koul and Schick (1997). The canonical gradient g and an efficient estimator for Ek(ε) is in Schick and Wefelmeyer (2000). Example 3. (Heteroscedastic linear autoregression.) The observations Xo5 , X n are real with

The Si are independent and, for simplicity, standard normal. Conditions for uniform ergodicity and efficient estimators for ϋ are in Maercker (1997) and Schick (1999). The model is semiparametric, with transition distribution 1

/υ — ϋx

where φ is the standard normal density. Fix ϋ and 5. Introduce perturbations tfmi = 0 + n~ 1 / 2 u, snυ{x) = s(x)(l + n-l'2v{x)). The function υ runs through V = ^ ( Z ) , where / is the stationary density. The perturbed transition distribution is Qnuv(x,dy) = QΰnuSnv{χ,dy) = Qϋs{x,dy)(l

= —-7-τy( y G

+ n-1/2{um{x,y)

T\

(

+ Dv(x,y))),

with

Since the normal distribution is symmetric, m and DV are orthogonal, and s can be estimated adaptively with respect to ΰ. Suppose we want to estimate the functional 2

a(s) = / s(x) dx. Jo For all u E R and v EV we have n1/2(a(snυ)

- a(s)) -> 2 / s(x)2v(x)dx = (Dva,Dυ + um) Jo 2 with va = l[ 0) i]S //. Hence a(s) is differentiate at (i?,s), with canonical gradient — ϋx\^

\

229

PLUG-IN ESTIMATORS Assume first that ΰ is known. Then we can estimate a(s) by l

/ h(x) Jo where

Here wn(x) = c~ιw(c~ιx), where w is a continuously differentiable symmetric density with compact support [—1,1], and cn is a bandwidth of order n " 1 / 3 . We show that a$ is asymptotically linear with influence function Dva. We do so under the assumption that 5 is twice continuously differentiable. Write (Xi - ΰXi^)2 = β(Xi-i) 2 (ε? - 1) + s(X - i ) 2 . Expand s(Xi_i) around s(x) to obtain , x Γ1A(x) aϋ - a(s) = / J o

+ 2s(x)8'{x)f1(x) j — fo{x)

2

dx

where A(X)

=

-

To . To

Λ

1=1

i=\

The assumptions imply that / is twice continuously differentiable. Hence we obtain uniformly for x G [0,1],

EA{x)2 = O(n-ιc-1) = O(n E(fo(x) - /(z)) 2 = 0{n-χc-1 + c4n) = O(n" 2 / 3 ), E(fι(x) - cnf'(x))2 = Oin-'c-1 + ci) = O(n- 2 / 3 ). We can also show that s u p o < x < i |/o(#) — / ( # ) | converges to zero in probability. From this and the facfthat / is bounded away from zero on [0,1], we can conclude that άΰ - a{s) = I -jγ^dx Now write

+ 0Pnΰs

(cn).

230

MULLER, SCHICK AND WEFELMEYER

with I

n(y) = Jo

1

j^wn(y-x)dx.

It is easy to check that In converges in L,2(f) to the indicator of [0,1]. Combining the above lets us conclude that α# has influence function Dυa. 1 2 Suppose now that ϋ is unknown. Let ϋ be a n / -consistent estimator of ΰ. We prove that the plug-in estimator ά^ is efficient. We have already shown above that ά$ fulfills (3.4) with b$ = Dva. By the argument of Section 3, it remains to show (3.5). Since (m,Dva) = 0 by adaptivity, (3.5) reduces l 2 to asymptotic equivalence of ά^ and ά#, i.e., n / (a$ — ά#) = opnϋs(l). To prove this, we note first that

~ ^ / o fo(x) where B{x) = -Ys{Xi-ι): n ϊΞ{ Since / 0 B(x)/fo(x)dx converges to zero in probability, we obtain the desired result.

7

Extensions

1. We have assumed ϋ and a(F) to be one-dimensional. Extension to finitedimensional a(F) is straightforward; infinite-dimensional a(F) require additional technicalities. In nonlinear regression, Example 1, we may, e.g., be interested in estimating the error distribution function F, defined by F(t) — P(ε < t). For linear regression we refer to Klaassen and Putter (1999). Extension to finite-dimensional ϋ is also straightforward. We note that it may happen that a(F) is adaptive with respect to certain components of ί? only. For efficiency of ά^, efficient estimators are required only for the non-adaptive components of ΰ. Extensions of nonlinear regression, Example 1, are treated in Mύller and Wefelmeyer (2000a). Extensions of nonlinear autoregression, Example 2, are treated in Schick and Wefelmeyer (2000). 2. We have restricted attention to functionals a(F) of F only. The results may be extended to functionals α(i9, F) which depend also on ΰ. An interesting application is estimation of invariant distributions of time series, for example in linear autoregression Xι = ΰXi-ι + E{. Since Y^LxWej is distributed as the invariant law, we can write the expectation of a function k under the invariant law as j

Ek{X) = Ek(Σΰ εj)

= a(ΰ,F),

231

PLUG-IN ESTIMATORS

where F is the invariant distribution function. Hence Ek(X) can be estimated by a von Mises statistic or a [/-statistic based on estimated innovations; see Schick and Wefelmeyer (2001). 3. The results extend from semiparametric models {PnϋF : t f 6 θ , F E 7 J } to parametric families {Vn>β : ϋ G θ } of nonparametric models. This is of interest when we start from a nonparametric model Vn and impose a restriction which depends on an unknown parameter, say r#(Pn) = 0, leading to Vn* = {Pn r#(Pn) = 0}. For example, let XQ, . . . , Xn be observations from a Markov chain with transition distribution fulfilling /Q(x,dy)y = r{ϋ,x) for some ϋ. This is thenonlinear autoregressive model X{ = r($,Xi_i) + ε^, where the ε% are martingale increments, not i.i.d. as in Example 2. For estimators of ϋ see Wefelmeyer (1994), (1996), (1997a), (1997b); for estimators of the stationary law see Schick and Wefelmeyer (1999). The model may be written as a semiparametric model by introducing transition distributions F(x,dy) with / F(x, dy)y = 0 and writing Q{x,dy)=F{x,dy-r(ΰJx)). This is, however, technically inconvenient because we perturb ΰ and would need differentiability of F. Another example are i.i.d. observations (-XΊ, YΊ),..., ( I n , Yn) with joint law fulfilling the constraint E(a(X, Y, ΰ)\X) = 0, where α(X, Y, ϋ) is a given function. For plug-in estimators in such models see Mύller and Wefelmeyer (2000b). A special case is a{X,Y,ϋ) = Y - r(tf,X), i.e., Y{ = r(ϋ,Xi) + εu which differs from Example 1 in that we do not assume Z{ and X{ to be independent.

References Andersen, P. K., Borgan, 0., Gill, R. D. and Keiding, N. (1993). Statistical Models Based on Counting Processes. Springer Series in Statistics, Springer, Berlin. Andrews, D. W. K. (1994). Asymptotics for semiparametric econometric models via stochastic equicontinuity. Econometrica 62, 295-314. Andrews, D. W. K. and Pollard, D. (1994). An introduction to functional central limit theorems for dependent stochastic processes. Internat. Statist Rev. 62, 119-132. An, H. Z. and Huang, F. C. (1996). The geometrical ergodicity of nonlinear autoregressive models. Statist. Sinica 6, 943-956.

232

MULLER, SCHICK AND WEFELMEYER

Bhattacharya, R. and Lee, C. (1995). On geometric ergodicity of nonlinear autoregressive models. Statist. Probab. Lett. 22, 311-315. Bickel, P. J. (1982). On adaptive estimation. Ann. Statist. 10, 647-671. Bickel, P. J., Klaassen, C. A. J., Ritov, Y. and Wellner, J. A. (1998). Efficient and Adaptive Estimation for Semiparametric Models. Springer, New York. Daniels, H. E. (1961). The asymptotic efficiency of a maximum likelihood estimator. Proc. Fourth Berkeley Symp. Math. Statist. Probab. 1, 151163. Drost, F. C, Klaassen, C. A. J. and Werker, B. J. M. (1997). Adaptive estimation in time-series models. Ann: Statist. 25, 786-817. Greenwood, P. E. and Wefelmeyer, W. (1991). Efficient estimating equations for nonparametric filtered models. In: Statistical Inference in Stochastic Processes (N. U. Prabhu, I. V. Basawa, eds.), 107-141, Marcel Dekker, New York. Fabian, V. and Hannan, J. (1985). Introduction to Probability and Mathematical Statistics. Wiley, New York. Hajek, J. (1970). A characterization of limiting distributions of regular estimates. Z. Wahrsch. Verw. Gebiete 14, 323-330. Huber, P. J. (1967). The behavior of maximum likelihood estimates under nonstandard conditions. Proc. Fifth Berkeley Symp. Math. Statist. Probab. 1, 221-233. Jeganathan, P. (1995). Some aspects of asymptotic theory with applications to time series models. Econometric Theory 11, 818-887. Klaassen, C. A. J. and Putter, H. (1999). Efficient estimation of Banach parameters in semiparametric models. Technical Report, Department of Mathematics, University of Amsterdam. Koul, H. L. and Schick, A. (1997). Efficient estimation in nonlinear autoregressive time series models. Bernoulli 3, 247-277. Kreiss, J.-P. (1987). On adaptive estimation in stationary ARMA processes. Ann. Statist. 15, 112-133. Le Cam, L. (1986). Asymptotic Methods in Statistical Decision Theory. Springer-Verlag, New York. Le Cam, L. and Yang, G. L. (1990). Asymptotics in Statistics. Springer Series in Statistics, Springer, Berlin.

233

PLUG-IN ESTIMATORS

Maercker, G. (1997). Statistical Inference in Conditional Heteroskedastic Autoregressive Models. Shaker, Aachen. Mύller, U. U. and Wefelmeyer, W. (2000a). Estimating parameters of the residual distribution in nonlinear regression. In preparation. Mύller, U. U. and Wefelmeyer, W. (2000b). Regression type models and optimal estimators. In preparation. Ogata, Y. (1980). Maximum likelihood estimates of incorrect Markov models for time series and the derivation of AIC. J. Appl. Probab. 17, 59-72. Pollard, D. (1985). New ways to prove central limit theorems. Econometric Theory 1, 295-314. Schick, A. (1993). On efficient estimation in regression models. Ann. Statist. 21, 1486-1521. Correction: 23 (1995), 1862-1863. Schick, A. (1999). Efficient estimation in a semiparametric heteroscedastic autoregressive model. Technical Report, Department of Mathematical Sciences, Binghamton University, h t t p : //math. binghamton. edu/ant on/preprint. html Schick, A. (2000). On asymptotic differentiability of averages. Probab. Lett. 51, 15-23.

Statist.

Schick, A. (2001). Sample splitting with Markov chains. Bernoulli, 7, 33-61. Schick, A. and Wefelmeyer, W. (1999). Efficient estimation of invariant distributions of some semiparametric Markov chain models. Math. Meth. Statist. 8, 426-440. Schick, A. and Wefelmeyer, W. (2000). Estimating the innovation distribution in nonlinear autoregressive models. To appear in: Ann. Inst. Statist. Math. Schick, A. and Wefelmeyer, W. (2001). Estimating invariant laws of linear processes by [/-statistics. Technical Report, Department of Mathematics, University of Siegen. http://www.math.uni-siegen.de/statistik/wefelmeyer.html Wefelmeyer, W. (1991). A generalization of asymptotically linear estimators. Statist. Probab. Lett. 11, 195-199. Wefelmeyer, W. (1994). Improving maximum quasi-likelihood estimators. In: Asymptotic Statistics (P. Mandl, M. Huskova, eds.), 467-474, Physika-Verlag, Heidelberg. Wefelmeyer, W. (1996).

Quasi-likelihood models and optimal inference.

234

MULLER, SCHICK AND WEFELMEYER Ann. Statist

24, 405-422.

Wefelmeyer, W. (1997a). Adaptive estimators for parameters of the autoregression function of a Markov chain. J. Statist. Plann. Inference 58, 389-398. Wefelmeyer, W. (1997b). Quasi-likelihood regression models for Markov chains. In: Selected Proceedings of the Symposium on Estimating Functions (I. V. Basawa, V. P. Godambe and R. L. Taylor, eds.), 149-173, IMS Lecture Notes-Monograph Series, Institute of Mathematical Statistics, Hayward, California. Wefelmeyer, W. (1999). Efficient estimation in Markov chain models: an introduction. In: Asymptotics, Nonparametrics, and Time Series (S. Ghosh, ed.), 427-459, Statistics: Textbooks and Monographs 158, Dekker, New York.

237

Institute of Mathematical Statistics LECTURE NOTES — MONOGRAPH SERIES

NUISANCE PARAMETER ELIMINATION AND OPTIMAL ESTIMATING FUNCTIONS T. M. Durairajan and Martin L. William Loyola College, Madras, India

Abstract In the context of obtaining optimal estimating functions for interesting parameters in the presence of nuisance parameters in parametric models, a method of elimination of nuisance parameters is proposed in this paper. The proposed method is direct and does not impose any 'factorization' conditions on the likelihood. In this direction, a sequence of lower bounds for the variance-covariance matrix of estimating functions is derived. A recipe which gives a transparent approach for obtaining optimal estimating functions is suggested. It is shown that minimum variance unbiased estimators could be obtained using the recipe. Keywords and Phrases: Lower bounds, nuisance parameter elimination, optimal estimating function.

1

Introduction

In the theory of estimating functions applied to parametric models involving nuisance parameters, the 'elimination' of nuisance parameters to obtain optimal estimating functions (EF) for interesting parameters is a very important task. In a pioneering work, Godambe (1976) suggested a method of eliminating nuisance parameters by multiplying and adding suitable functions to the score function and formally established the optimality of conditional score function. Lloyd (1987) and Bhapkar and Srinivasan (1993) claimed the optimality of marginal score function. However, that there are errors in the results of Lloyd (1987) and Bhapkar and Srinivasan (1993) has been pointed out by Bhapkar (1995, 1997) who imposed some more conditions and established the optimality of marginal score function. The conditional and marginal factorization properties were used by the above authors in the elimination of nuisance parameters. Heyde (1997) proposed a method of

238

DURAIRAJAN

AND WILLIAM

obtaining optimal EF by eliminating nuisance parameters from a suitably chosen function that possesses the 'likelihood score property'. Heyde gives the optimal EF of 'first order theory' but not of the higher orders. The present work is an attempt in this direction. In this paper, a straight forward recursive method of elimination of nuisance parameters without going into the factorization aspects of the likelihood is proposed. In this direction, a theorem which gives a sequence of lower bounds for the variance-covariance matrix of the EFs is established in Section 2. This is achieved by considering higher order derivatives with respect to the nuisance parameters drawing inspiration from Godambe (1984). Consequently, a recipe which gives a systematic approach for possible elimination of nuisance parameters leading to optimal EF is suggested. Section 3 presents several examples to illustrate the recipe. In Section 4, as another outcome of the main result of Section 2, a sequence of lower bounds for the variance - covariance matrix of unbiased estimators of the interesting parameters is given. This sequence is different from the sequence of Bhattacharya bounds both in context and in content. Further, it is shown that minimum variance bound unbiased estimators of the interesting parameters could be obtained by the suggested recipe.

2

The Main Result and the Recipe

Let X be a random vector with sample space X and probability density function p(x; ω) with respect to some σ-finite measure μ on (X, B(X)). The family of densities is indexed by ω = {θ,φ) E Ω with θ G Ωi C SRr, φ E Ω2 C 3?m, Ω = Ωi x Ω2. The interesting parameter is θ and the nuisance parameter is φ and estimation of θ in the presence of φ is considered. We assume the usual regularity conditions on the density function 'p' and the EFs g = (gu .. .,gr)': X x Ωx -» W (refer Godambe (1976, 1984), Bhapkar (1995, 1997)). Let Dg = ((E(dgi/dθj))). Let the class of EFs satisfying the regularity conditions be denoted as Go let Mg(ω) = D~ιE{g.g')(D'g)~ι, the variance-covariance matrix of standardized EFs. In the sequel, the following notations are used: le = (dlog p/dθu...,dlogp/dθry In = E(lθ l'θ)

(2.1) (2.2)

NUISANCE PARAMETER ELIMINATION

239

with l

φ \ — h\ )

&

l

•'22 — -^ [*0 " φ J

^ J

/ i i a n d / ^ a r e a s s u m e d non-singular a n d for simplicity, we write I,1' = lφ,

/&> = /i2, I$? = hi, J& = hi- Let (2.6)

, fc = 1 , 2 , . . .

Since / ^ are positive definite we have Bk+i > Bk. Also, E[L •0(*)

'j = BΓ1

V k.

(2.7)

Theorem 2.1: For every 3 € C/o, M5 > 5 fc , k = 1,2,..., with equality if and only if g = 1(0,0) L ^ where A(θ, φ) is a non-singluar matrix and the functions Lg ' are defined recursively in (2.4) and (2.5). Proof: For g e Go, we observe that E[glφk)>) = 0, E[gLf]'} Now, considering the n.n.d. matrix E

r(*)

9,' — =

-Dg

= -Dg\/k.

(2.8)

E(g.g')

and applying matrix theory arguments, we get Rank

- Rank

0 E{g.g')

Also, Bkι — M~ι > 0 by the non-negative definiteness of the full matrix in (2.8). This gives Mg > Bk. Further, B~ι - M~ι = 0 if and only if the rank of the matrix in (2.8) is r. ' for some non-singular matrix A(θ,φ), then Now, if g = A(θ,φ) clearly the matrix in (2.8) has rank r so that Mg = B^. Conversely, if Mg = Bk i.e. the rank of the matrix in (2.8) is r, then as B^1, Dg and E(g.g') are all non-singular, it is necessary that for some non-singular matrix A(θ, φ), g = A(θ, φ) - Lf\ Hence the theorem. Remark 1: If for some k > 1, Bk is attained by the EF A(θ,φ)Lf) for a suitable choice of A(θ,φ) (i.e. A(θ,φ)Ly is free of φ), then we have /Jί;+1) = 0 which gives L^+l) = L{θk) and Bk+λ = Bk. Similarly, V s > k + 1, /W = 0, L{θs) = Lf] and Bs = Bk.

240

DURAIRAJAN AND WILLIAM In view of this, we now propose the following.

Definition 2.1: If there exists a g* G Go such that Mg* attains a lower bound Bk for some A;, then g* is said to be minimum variance bound estimating function (MVBEF). Based on the theorem established and the above remark, we suggest the following recipe for eliminating nuisance parameters and obtaining MVBEF: "Starting with the score function /#, consider recursively the functions Lθ

= IQ — I ^ I ^ l φ ' i LQ = L Q — l[2 I22

lφ

a n

ds 0 forth. If t h e nuisance

parameters are essentially eliminated or appear as a multiplicative factor of an EF in some recursion, stop the process as the EF thus obtained is optimal and further recursions would result in the same EF". Remark 2: The first order bound in the sequence of bounds derived above namely Bι = (In — Iul^hi)"1 was derived as the lower bound by Chandrasekar and Kale (1984) who gave a different line of proof. In a latest book, Heyde (1997) has proposed a 'first order' theory for obtaining optimal EF. He suggests the possibility for higher order theory but does not give any explicit method for the same. The recipe suggested above gives a formal and transparent method to proceed to second and higher order theories and carries out Heyde's suggestion. However, the forms of the optimal EFs obtained in Theorem 2.1 do not follow as a consequence of the technique of Heyde (1997).

3

Applications

In this section, a number of examples are discussed to illustrate the recipe suggested in the previous section. Throughout this section, we reserve the symbol θ for the interesting (real or vector) parameter. Example 3.1: Let x = (xi,...,x n ) and y = (j/i,. ,J/n) be independent where X{ are i.i.d. with density φexp(-φx), x > 0 and yι are i.i.d. with density φθ~ιexp(-φθ~λy), y > 0, 0, > 0 . Here,

is the MVBEF attaining the bound B\. Example 3.2: Let z\,..., zn be i.i.d. with z% = (x^, yu > ? VH) where X{ are i.i.d. exponential with mean φ and for each fixed j — 1,..., r, y^ are i.i.d. n

exponential with mean φθj. Denote x = Σ x^ yj = Y%=i yji, j = 1,..., r.

241

NUISANCE PARAMETER ELIMINATION

Here, φL™ = (rf,... ,g*r) is MVBEF, with g* = | - ^

^

Σ |

Remark 3: Examples 3.1 and 3.2 have been discussed respectively by Lloyd (1987) and Bhapkar and Srinivasan (1993) in the context of marginal factorization of the likelihood. These authors claimed that a marginal score function is the optimal EF. However, from the above discussion, we find that the optimal EFs obtained above do not coincide with the EFs claimed by these authors as optimal. In Example 3.1, we have Mg{\)+ — 2θ2/n whereas for the EF of Lloyd (1987) namely g0 = n/θ - 2n/(θ + we have Mgo = 2(n + I)θ2/n2 > M (i)*. This shows that Lloyd's claim that go is optimal EF is incorrect. The errors in Lloyd (1987) and Bhapkar and Srinivasan (1993) has been pointed out also by Bhapkar (1995, 1997) who has found the correct optimal EF for the model in Example 3.1 but not for Example 3.2. In contrast, the recipe of Section 2 and the explicit form L,θ ' for the optimal EF have enabled us to achieve this for Example 3.2 as well in an elegant manner. Example 3.3: Let x\, yi,..., x n , yn be independent normal with E(xi) = 0, E(yi)

=θ + φi, V{xi) = V(yi) = 1. Here, L^ = Σ {x{ - θ) is the MVBEF. i l

Example 3.4: Let a i, j/i,... .Xn^Vn be independent normal with E(x{) = θ + φu E{yi) = φi, V{xi) = V{yi) = 1. Here, the MVBEF is L^ = n(x - y - θ)/2. Remark 4: Examples 3.3 and 3.4 were discussed by Godambe (1976) in the context of nuisance parameter elimination when conditional factorization property for the likelihood holds good. He has shown that the same EFs are optimal. In the above discussion, we have demonstrated the straight forward applicability of our recipe without investigating the factorization aspects which is required in Godambe's approach. Example 3.5: Let x = (xi,... ,xn) and y = (yi,... ,y m ) be independent where xι are i.i.d. Poisson (0), yj are i.i.d. Poisson (), θ,φ > 0. Here,

4

υ

= mn(x - θy)/(θ(nθ + m)) is the MVBEF.

This example with m — n — \ has been discussed by Reid (1995) in illustrating the roles of conditioning in inference in the presence of nuisance parameters, wherein the estimation is based on conditioning upon a statistic called a 'cut' (Barndorίf-Nielsen 1978) and involves a suitable reparametrization. In contrast, our approach is straight forward and does not require reparametrization. Example 3.6: Consider the linear model of a randomized block design Vij = μ + U + bj + eij, i — 1,..., fc, j = 1,..., r where e^ are i.i.d. iV(0, σ 2 ),

242

DURAIRAJAN AND WILLIAM

Σt{ = Σbj

= 0. Suppose estimation of t h e effect of the first treatment 't'λ

alone is of interest. 2

T h a t is θ = t\, φ = ( μ , ΐ i , . . . ,ίjfe_i,&i, {

Here, (σ (k

2

5

&r-i )

= ylm - y.. - h is MVBEF, where ylm =

- l)/(kr))L ^

Example 3.7: Consider a 2 x 2 contingency table ((n^)), i, j = 1,2, following a multinomial distribution with fixed sample size n = ΣΣn^ and with probabilities ((TΓJJ)), i,j — 1,2, ΣΣπ^ = 1. Let θ = πn be the parameter of interest with φ = (πi2,π2i). Here Lβ ' = {n\\ — nθ)/(θ(l — θ)) is the MVBEF. This example was discussed by Bhapkar (1989) in investigating conditioning and loss of information in the presence of nuisance parameters. Example 3.8: Let {X(t),t > 0} and {Y(t),t > 0} be two independent Poisson processes with parameters (θ + φ) and φ respectively, 0, φ > 0. Suppose data on the states of {X(t)} at times
( Let

1

lt is f o u n d

t h a t

-1

1

2^

2

get 5 ( )* = A(θ, φ) 4

^2

n-lV

2_

^ = ώ . Σ K - %) 2 Choosing Z— 1

1 ,n

2 )

= (Sf - σf,...

Φ) =

—

) '

j

V(: as the

|4'

Ί4) We

MVBEF attaining

the bound B2. This MVBEF was shown as optimal EF not attaining the bound j?i by Chandrasekar and Kale (1984) and in their framework served as an example of an optimal EF which is not MVBEF. From our discussion above, it is found that g^* is MVBEF attaining, however, the higher bound B2 Example 3.10: Let xi, yi,..., x n , yn, be independent normal with E(xι) = E(yi) = φi, i = 1,..., n and Vfa) = V(yi) =θ Vi = 1,..., n. Here, L^ is

NUISANCE

PARAMETER

ELIMINATION

243

not an E F for 0 and so we proceed to the second recursion to get (2) θ

n 1 A Q 9Λ ~*~ AΛ2 ^ Λ X * ~" y*) 2=1

a s

MVBEF attaining the bound B2.

The above example has been discussed by Godambe (1976) in the context of conditional factorization wherein he contrasts the consistency of the solution of the above optimum estimating equation with the inconsistency of the usual maximum likelihood estimate discussed by Neyman and Scott (1948). Again, investigation into factorization aspects is not required in our approach. Example 3.11: Let #1, yi,..., xni yn be independent normal with E(xi) = 0, E{yi) = φ, V(xi) = V{yι) = 0, φ e 5R, 0 > 0. Here, L J 1 } is not an EF. So, we proceed to the second recursion to get Lg ' = —^[nθ2 4- (2n — 1)0 — n s ( l + sl + x2)] a s the MVBEF attaining bound B2. Here, si = ^(xi - x)2 and 5^ = ^Σ(yj — y) 2 . The solution of the equation LQ = 0 is a consistent estimate namely Λ

^ 4

[(2n - I) 2 + 4n 2 [s2x + ^ + x 2 ) ] V 2 - (2n ^

Minimum Variance Unbiased Estimators

The main result of Section 2 leads us to minimum variance bounds for unbiased estimators of the interesting parameters. Prom Theorem 2.1, it is immediately evident that, for unbiased estimators T = (Ti,...,T r ) ; of Var - Coυ{T) > Bk, fc = 1,2,...

where Bk's are given in (2.6). This is verified by considering EFs of the form T-0. If any of the L ^ ' s defined recursively in (2.4) and (2.5) is such that A{θ,φ)L{θk) is of the form T* - θ for a suitable choice of A(θ,φ), then T* is minimum variance unbiased estimator (MVUE) of θ. Thus, the recipe of Section 2 could possibly be of help in finding T*. The following examples illuminate this point. Example 4.1: Consider the model in Example 3.3. Here LQ ' /n — x — θ so that x is MVUE of θ attaining bound B\. Example 4.2: Consider the model in Example 3.4. Here 2L,Θ '/n = x — y — θ so that x — y is MVUE of θ attaining bound B\.

244 Example 4.3: 2

(σ (fc - l)/kr)L^

DURAIRAJAN AND WILLIAM Consider the linear model in Example 3.6.

Here,

= ylm - y.. - tλ so t h a t ylm - ymm is M V U E of tλ.

Example 4.4: Consider Example 3.7. Hear, 0(1 - θ)Ly /n = n\\jn - θ so that nn/n is MVUE of θ. Example 4.5: Consider Example 3.8. Here, ((0 + φ)sntm)~1Lg [SnX(tm) - tmy(Sn)]/(Sntm) as MVUE of θ. Example 4.6: Consider Example 3.9. (σ\,..., σ2) attaining bound i?2

= 0 gives

Here, (S?,... ,Sj?) is MVUE of

Example 4.7: Consider the Neyman-Scott model in Example 3.10. Here 2θ2L%)/n = Σ{xi - yi)2/(2n) - θ so that Σ{Xi - y;)2/(2n) is MVUE of θ attaining bound B<ι. It is noted that the MVBEF need not produce MVUE when it cannot be reduced to the form T* — θ. This is evident in Examples 3.1, 3.2, 3.5 and 3.11. This fact is also reported by Thavaneswaran and Abraham (1988) who considered EFs for non-linear time-series models. Acknowledgement We thank the referee for bringing to our notice some of the important references in this area and for the valuable comments which improved the quality and content of the paper substantially.

References Barndorίf-Nielsen, O.E. (1978) Information and Exponential Families in Statistical Theory. Wiley, New York. Bhapkar, V.P. (1989). Conditioning on ancillary statistics and loss of information in the presence of nuisance parameters. J. Statist. Plann. Inference, 21, 139-160. Bhapkar, V.P. (1990). Conditioning, marginalization and Fisher information functions. Proc. R.C. Bose Symposium, Delhi. (Ed. R.R. Bahadur). Wiley Eastern Limited, New Delhi, 123-136. Bhapkar, V.P. (1991). Loss of information in the presence of nuisance parameters and partial sufficiency. J. Statist. Plann. Inference, 28, 185-203. Bhapkar, V.P. (1995). Completeness and optimality of marginal likelihood estimating equations. Comm. Statist. A - Theory Methods, 24, 945-952.

NUISANCE PARAMETER ELIMINATION

245

Bhapkar, V.P. (1997). Estimating functions, partial sufficiency and insufficiency in the presence of nuisance parameters. Proc. Symposium on Estimating Functions, Georgia. (Ed. I.V. Basawa, V.P. Godambe and R.L. Taylor). Institute of Mathematical Statistics: Lecture Notes Monograph Series. 32, 83-104. Bhapkar, V.P. and Srinivasan, C. (1993). Estimating functions : Fisher Information and optimality. Probability and Statistics. Proc. First International Triennial Calcutta Symposium on Probability and Statistics. (Ed. S.K. Basu and B.K. Sinha). Narosa Publishing House, New Delhi, 165-172. Bhapkar, V.P. and Srinivasan, C. (1994). On Fisher information inequalities in the presence of nuisance parameters. Ann. Inst. Statist. Math. 46, 593-604. Chandrasekar, B. and Kale, B.K. (1984). Unbiased statistical estimation functions for parameters in presence of nuisance parameters. J. Statist. Plann. Inference, 9, 45-54. Godambe, V.P. (1976). Conditional likelihood and unconditional optimum estimating equations. Biometrika, 63, 277-284. Godambe, V.P. (1984). On ancillarity and Fisher information in the presence of a nuisance parameter. Biometrika, 71, 626-629. Heyde, C.C. (1997). Quasi-Likelihood and its application - A general approach to optimal parameter estimation. Springer, New York. Lloyd, C.J. (1987). Optimality of marginal likelihood estimating equations. Comm. Statist. A - Theory Methods, 16, 1733-1741. Neyman, J. and Scott, E.L. (1948). Consistent estimates based on partially consistent observations. Econometrica, 16, 1-32. Reid, N. (1995). The roles of conditioning in inference. Statistical Science, 10, 138 - 157. Thavaneswaran, A. and Abraham, B. (1988). Estimation for nonlinear time - series using estimating equations. J. Time Ser. Anal. 9, 99

247 Institute of Mathematical Statistics

LECTURE NOTES — MONOGRAPH SERIES

OPTIMAL ESTIMATING EQUATIONS FOR MIXED EFFECTS MODELS WITH DEPENDENT OBSERVATIONS Jeong-gun Park and I.V. Basawa University of Georgia Abstract Optimal joint estimating equations for fixed and random parameters are derived via an extension of the Godambe criterion. Applications to autoregressive processes and generalized linear mixed models for Markov processes are discussed. Marginal optimal estimating functions for fixed parameters are also discussed. Key Words: Optimal Estimating Functions; Generalized Linear Mixed Models; Autoregressive Processes; Markov Processes.

1

Introduction

Mixed effects models containing both fixed and random parameters are used extensively in both applied and methodological literature. A review of linear mixed models and their applications is given by Robinson(1991). Generalized linear mixed models are discussed by Breslow and Clayton(1993) among others. Work on the extension of mixed effects models to dependent observations appears relatively scarce. Our main goal in this paper is to develop optimal estimating equations for mixed effects models with dependent observations. See Basawa et al.(1997), Heyde(1997), and Godambe(1991) for recent literature on optimal estimating functions. Desmond(1997) gives an overview of estimating functions. Suppose Yt is a vector of observations on n individuals at time £, and Y(t — 1) = (YΊ,..., Yt-ι). Conditional on Y(t — 1) and a random parameter 7, the density of Yt is denoted by p(yt\y{t — l),/3,7) where β is a fixed parameter. Let π(7|α) denote the (prior) density of 7 which may depend on a parameter a. Suppose, for simplicity, a is known, and we wish to estimate β and 7 from a sample Y{T) = (Yi,..., Yτ). The likelihood function, conditional on 7, is given by ) =P(yo)TiJ=lP(yt\y(t-

l),β,j).

(1.1)

248

PARK AND BASAWA

The joint density of 7 and Y(T) is p(7,v(T)|α,/?)=LGS,7)π(7|α).

(1-2)

An intuitive way, often used in practice, to estimate the "mixed effects" β and 7, is to maximize p(7,y(T)|α,/3) with respect to β and 7 and obtain formally the estimating equations : of

dlogL(β,j) 97

°y=0, +

(i.3)

d/ogπ(7|α) ^7

= Q

Example 1. Linear mixed models Take T = 1, and YQ = yo fixed(given). Suppose, conditional on 7, Y\ is normal with mean vector X β + Z 7 and covariance matrix Σ, where X and Z are known covariate matrices. Further, assume that 7 is a normal vector with mean zero and covariance matrix Γ. It is assumed, for simplicity, that Σ and Γ are known. Equations (1.3) and (1.4) then lead to the well known mixed linear model equations, see, for instance, Robinson(1991) for a review. Example 2. Generalized Linear Mixed Models Again, set T = 1, and suppose (1-5)

), and Var\Yλ\β,η] = V(β,Ί) = U(μ(β,Ί)).

(1.6)

Without further assumptions regarding the conditional density of Y\ given 7, one may be interested in estimating β and 7. If we choose (1.7)

h(μ(β,Ί))=X'β + Z'Ί,

for an appropriate link function /ι( ), and retain the normality assumption regarding 7 (i.e. 7 ~ iVίo, Γj), a penalized quasi-likelihood approach (see Breslow and Clayton( 1993)) yields the estimating equations : (1.8)

)-1(Y1-μ)-Γ-1Ί

where μ = μ(β,j)

= 0,

(1.9)

= hΓι{X'β + Z'j) = g(X'β + Z'j), say. Equations

(1.8) and (1.9) correspond to (1.3) and (1.4) if the conditional density of

249

OPTIMAL ESTIMATING EQUATIONS

Y\ given 7 is a member of an exponential family. See also Sutradhar and Godambe(1997), and Waclawiw and Liang(1993). Analogous to (1.8) and (1.9), we now propose a general method based on the penalized quasi-likelihood equations for dependent data. Let (1.10)

E[Yt\Y(t-l),βΠ]=μt(β,Ί): and Var[Yt\Y(t-l),β,Ί] = Vt(β,Ί).

(1.11)

Suppose further that the prior density of 7 is known to be π(7|α). Consider the estimating equations: (1.12)

( L 1 3 )

If the conditional density of Yt given Y (t — 1) and 7 belongs to an exponential family, the equations in (1.12) and (1.13) correpond to (1.3) and (1.4). Suppose that 7 has a prior density τr(y\a) with mean 0 and variance Γ(α). Consider a modified equation for 7 : E

[iaJ^^\

Γ(α)"'7 = 0. (1.14)

If 7 - N(θ, Γ(α)), we have

_

dlog-κ(η\ά) dη

Hence, in this special case, (1.14) reduces to (1.13). We show in this paper that the equations in (1.12) and (1.14) are optimal estimating equations in the sense of a generalized Godambe criterion. The question of the performance (sampling properties) of the estimates obtained from (1.12) and (1.14) needs to be addressed via asymptotics. In the special case of linear mixed models (example 1), the estimates are known to be best linear unbiased predictors(BLUP), see Robinson(1991). In the general case, we may use an extension of the asymptotic optimality criterion discussed by Wefelmeyer(1996), at least for the estimation of the fixed effects parameter β. See also Heyde(1997). This topic will not be considered in this paper.

250

PARK AND BASAWA

In general, the information matrix(or its estimate) corresponding to the optimal estimating functions can be used to compute the standard errors of the estimates. The paper is organized as follows. The general optimality criterion for the estimation of mixed effects is introduced in Section 2. The optimal estimating equations are derived in Section 3. Section 4 discusses applications to autoregressive processes and generalized linear mixed models for Markov processes. Section 5 considers an alternative method based on a marginal model specification. Finally, Section 6 contains some concluding remarks on work in progress.

2

Optimality Criterion

Let y be the sample space, (Θi x Θ2) C Rk the parameter space, where Θi C Rm,m < k. Assume that 71^(7) is a prior density of 7 for a fixed /?, where 7 € Θi and β e Θ2. Let C be the set of all functions g : y x (Θi x Θ 2 ) -> Rk such that E\g(Y,j,β)\β] = 0, V/3 E Θ 2 , where the expectation is with respect to the joint distribution of Y and 7, for a fixed β. Assume that E[g{Y,η,β)g(Y,η,β)'] is finite for any g £ £. We shall use the notation < gug2

>β= E[gι{Y,Ί,β)g2{Y,Ί,β)'\β], V J l j f t 6 A

(2.1)

where the expectation is with respect to the joint distribution of Y and 7, for a fixed β. Let p(Y, 7, β) denote the joint density of Y and 7. We assume throughout that the following conditions are satisfied : C.I. For any g € £, g is differentiate with respect to both 7 and β\ Both E[%] and E[%) exist. C.2. The joint density p(y, 7, β) is differentiate with respect to both 7 and β. The support of p(y,7,/3) does not depend on the parameters. C.3. For any g G £, E[g] is differentiate with respect to β under the integral sign. CΛ. The support of conditional density Pβ(y\j) does not depend on the parameters 7 and β. For any g G £, i?[#|7] is differentiate with respect to 7 under the integral sign; E[^E{g\Ί)] and E[E(g\j)dl°9^h)] exist. The above conditions ensure the existence of various quantities in (2.4) and the validity of the derivation of the information function defined in (2.6) below.

OPTIMAL ESTIMATING EQUATIONS

251

Let s = ( s i , 5 2 , . . •, Sfc), where

,

dlogp(Y,y,β)

s

,

dlogp{Y,η,

sm+ι=

j =

dβi

For any g € £ 0 , A) C £,

^M = E

dlogp(Y,Ί,β)' 1

, j = l,...,m.

Note that p(Y,j, β) = Pβ{Y\Ί)κβ{Ί)i where pβ{Y\^) is the conditional density of Y given 7 with respect to an appropriate measure μ(y) and πβ(j) is the density of 7 with respect to a measure 1^(7). We then have E[gSj]

= E

dlogpβ(Y\-y)

dlogπβ(j)

dΊj

dΊj

f

= E = E

dE\g\Ί]\

dg

\dlogπβ(j)

E[g\l] • (2.2)

Now,

E[gsm+ι]

= E^g

dlogp(Y,Ί,β)

f dp{y, Ί, -dμ(y)dυ(j), I = 1,..., k - m.

J

Oui

(2 3

>

Combining (2.2) and (2.3), we obtain dg

/ J

^ Γ ^ 1Λ (2.4)

Note that if E\g\η] = 0, (2.4) reduces to

(2.5) LC/p J /

We define the information function by Ig(β) = E[gs]Έ[gg]~ιE[gs],

(2.6)

for any g E £o5 where E[gs] is given by (2.4). The criterion of optimality is to maximize the information function Ig{β) over J G £ Q

252

PARK AND BASAWA

Definition 2.1 A function g* is an optimal estimating function in Co if I9.(β)-Ig(β) is nonnegative definite, for all g E Co and all β. Case I (Fixed Effects) : m = 0, i.e. the model involves only the parameters of fixed effects. When m = 0, E[gs'} — -£?[||]. So the information function is given by

Let g* be the optimal estimating function for β in £o Then

It is easily verified that the nonnegative definiteness of Ig*{β) — Ig{β) is equivalent to that of E(gg) -

H{H*)-'E[g*g*']{H*')-lH'.

The latter turns out to be the optimality criterion given by Godambe(1985) when only fixed parameters are present with H = E \-ga\ and H* = E -A- . See also Godambe(1960) and Durbin(1960). Case II (Random Effects) : ra = fc, i.e. the model involves only the parameters of random effects.

When m = k, E\gs) = -E [f}] + E [ ^ 1 + ^ | 7 ] « 2 ^ 2 i ] . Thus the information function is given by Ig(β) = j'E[gg'}J,

where J = -E [^] + E [ ^ M

+

E[g\Ί]dl°9^{Ί)] . Let g* be the optimal

estimating function for 7 in Co- Then we require Ig*{β) - I9(β) = J*'E[g*g*'}J* - JE[gg]J to be nonnegative definite. This turns out to be Chan and Ghosh(1998)'s optimality criterion when only the parameters of random effects are present and J* is J with g replaced by g*. See also Ferreira(1981, 1982) and Godambe(1998).

253

OPTIMAL ESTIMATING EQUATIONS

3

Optimal Estimating Equations

Here we derive optimal estimating functions for parameters of mixed effects. Let y be the sample space and F? the σ-field generated by a specified partition of y. Let hi be a real-valued function of Y, 7, and β such that = 0, < = l , 2 , . . . , n .

Let u( ,β) :Θι^Rm

with E[u( ,β)] = 0, forfixedβ.

We consider the linear estimating space given by

B(β)u(Ί,β)\ , where 0^(7,/3) is a k x 1 vector which is measurable with respect to the σ-field Flf and B(β) is a, k x m non-random matrix. Let

α*(7,/?) = -E[(^ί, ?jjf) l*f ,7,/{ ^αr(/^ > 7 ,/3)]^

(3.1)

and

(3.2)

Theorem 3.1 Let 9* = Suppose that hi's are mutually orthogonal in the sense that E[h*h*j \F^, 7, /3] = 0 for alii φ j , ij = 1,2, . . . , n ; wΛere ΛJ =a*(η,β)hi. Then g* is an optimal estimating function in £Q Proof: From C h a n a n d Ghosh(1998), it is enough t o show t h a t g* is a n orthogonal projection of s into £o> i-e., < g,s — g* >= 0 for any # G CQ. We have = - . (3.3) Note that, for the simplicity of notation, we use α^ and B for 0^(7,/3) and B(β), respectively in this proof. Now the first term on the righthand side of (3.3) can be expressed as n

=

}

E[ajhjS ] + BE[us }

\Fί,Ί,β]\ +BE[us'].

254

PARK AND BASAWA

From (2.4), since E[hi\FY^,β]

= 0 for all i and any β,

Also, from (2.4), we have

-L - \u Since the function u is only a function of 7 and 5, the conditional expectation of ti, conditional on 7 and 5, is u itself. Thus we have

Next the second term on the righthand side of (3.3)can be found as

2=1.7=1

[[aM^\F^Ί,β)] +BE[uu]B*'. Since the functions aihi and a^hj are orthogonal for all i

*=1 j=l

Substituting α^ and J5*, we get

(3 5 )

Hence, from 3.4 and 3.5, we have < g,s > — < g,g* >= 0. This complete the proof.

Independent Observations If the elementary estimating functions h{, i = 1,..., n are independent with £7[/ii(y, 7, /?)|7, /?] = 0, Vi = 1,2, . . . , n , it's not necessary to partition y i.e. Fj is y itself. For an example, if Y = (y\,y2, 52/n) and y^'s are

255

OPTIMAL ESTIMATING EQUATIONS

independent, we take hi to be a function of yι, 7 and β such that E[hi\^, β] = 0, Vi. Here h^s are independent. Hence, the optimal estimating function in the space Co is given by g* in Theorem 3.1 with α*(7,/3) replaced by

a?(Ί,β) = -E [ ( | ^ , 0 ) |7,/3J [Fαr^lT,/?)]-1.

(3.6)

Application to discrete time stochastic processes As an example of dependent data we develop an optimal estimating function for a discrete time stochastic process. Let {Yi, Y2,..., YT} be a discrete time stochastic process. Let ht be a real-valued function of YΊ,..., Yj, 7, and /? such that

where i* 1 ^ is the σ-field generated by the past observations Yi, Y2,..., Yt-ι Let u(Ίβ) : Θi -> i ϊ m with E[tz( ,/3)] = 0, for fixed β. We consider the estimating space

where αί_ 1(7, β) is a A; x 1 vector measurable with respect to F£_ι and is a k x m non-random matrix. Let

and B*(β)=E\[u(Ί,l (3.8)

Theorem 3.2 Let T α

Then g* is an optimal estimating function in CoThe proof follows from Theorem 3.1 with Fj = F^_x.

PARK AND BASAWA

256

4

Applications

In this section, we will discuss two applications and demonstrate the derivation of optimal estimating functions.

4.1

Autoregressive Processes

Suppose that yt(j), t = 1,2,..., T, j — 1,2,..., n are observed at time t on the j t h subject from the first order autoregressive process(AR(l) process), (4.1) and (4.2)

φ(j) = x'jβ + zjΊ,

where 7 ~ JVu),Γj and εt = (εί(l), .. ,εt(n))',ί = 1,2,... ,T, are iid random vectors with E[εt] = 0 and Vαr[εί] = Vn(a). Assume that 7 and εt are independent for any t. Here β is a (ra x 1) vector of fixed parameters and 7 is a (g x 1) vector of random parameters, and Xj and Zj are vectors of known covariates. Note that we are not making distributional assumptions regarding εt other than the mean and variance assumptions. Let yt = (y t (l),...,y t (n))\

φ=

and

0 0 .

0

0

yt-i(n)

Then (4.1) can be written in a vector form

We consider optimal estimation for (7, β) assuming a known. We choose elementary estimating functions ht and u such that £J(/ιί|F^_1,7,/3) = 0 and

E(u\β) = 0. Let ht = yt-Yt-ι{X'β

+ ZΊ)

and u = 7,

where X = ( x i , . . . , x n ) and Z = (z\,... , z n ) . Consider the estimating space £0 = {ff : ff = Σ)^=i At-ιht we let

+ i?u}. Now

257

OPTIMAL ESTIMATING EQUATIONS and

where π(η) is the prior distribution of 7. Then A^_x and B* are computed as A*_i =

( χ)

t-Ma)-' YM)-'

and d B* B*=

11

ί "£*« \ Γ" Γ".

Thus, the optimal estimating function for (7, β) in the space Co is given by g* = T

-ιVn{a)-1 (yt - Yt-ι(X'β + Z'Ί)) - T~ιΊ,

g\ =

(4.3)

and

= ΣXYt-iVn{aΓι (yt - Yt-ι(X'β + Z'Ί)).

(4.4)

ti t=i

Prom (4.3) and (4.4), the optimal estimates {y*,β*) of (η,β) are given by

ί-itί.ία)-1^* _ γt_lX'β*) I

7* = and

-1

β* = t=l

t-1

ί-l

t-l

where Δ = Z (ΣΪ=I Yt-iVnicή^Yt-i) Z' + Γ" 1 . Prom (2.4), the information function corresponding to the optimal estimating function g* is obtained as ί ZHnZ' + Γ" 1

\

XHnZ'

ZHnX' \

XHnX' J '

where Hn = ΣΪ=i ^ - i F ^ α ) - 1 ^ - ! ] . As a special case, if the subjects are uncorrelated, i.e. Vn(a) = diag(aj,j = 1,..., n), then the optimal estimating equations from (4.3) and (4.4) turn out to be n T j=l t = l

PARK AND BASAWA

258 and

n T V^V^

-1

/ ' Δ^ί 3=1 t=l

3

=°

/ \

y ι

L VJ

'

Thus the optimal estimates n

J

(TJ*,/3J

) of (7,/?) are obtained as

T

Ίi* = Δ j=l

t=l

and n

T

n

T

3=1

t=l

3=1

t=l

' n

T

1

n

Γ

(Σ Σ j=i t=i

LJ=I t=i

where Δ = Σ?=i Σ£=i ^fvliU)^

+ Γ"1.

We extend the method of obtaining optimal estimating functions to the p order autoregressive processes. Suppose that yt{j),t — 1,2, . . . , Γ , j = 1,2, . . . , n , are observed at time t on the j < / ι subject from the pth order autoregressive process(AR(p) process), th

(4.5)

Vtϋ) = ^2Φk(J)yt-k(J)+εt(j), k=l

and ΦU) = foiίi), • 1 ^ 0 ' ) ) = -Xj/Ϊ + ZjΊi

(4-6)

where 7 ~ Nq ίo, Γj and εt = (εt(l),..., ε*(n))', t = 1,..., T, are ΐid random vectors with E[εt] = 0 and Vαr[εt] = Vn(a). Assume that 7 and £* are independent for any t. Here β is a (m x 1) vector of fixed parameters, 7 is a (q x 1) vector of random parameters, and Xj and Zj are matrices of known covariates. Let

άι)

°

0

ypt(2)

0

0

••• ° ^

...

0

γ

tp=

V

••• ypt(n),

)

and ^ ( j ) = {yt-ι{j),yt-2U)i - ">yt-p{j))'- Elementary estimating functions ht and u for 7 and /? are chosen to be

259

OPTIMAL ESTIMATING EQUATIONS and

where X = (X(l),... ,X(n)) and Z = (Z(l),... ,Z(n)). These elementary estimating functions satisfy the required condition of unbiasedness, i.e, ElhtlF^^β] = 0 and #M/?] = 0. In the estimating space Co — {g : g = Σ*Li M-\ht + Bu}, the optimal estimating function for (7, β) is given by g** = (ffί*,^*) where T

Σ

gl* = £ ZY;tVn(a)-1 (yt - Ypt(X'β + Z'Ί)) - Γ ' S , t=ι

and gl* = ΣXY'vtVn{ά)-1 fa - Ypt(X'β + Z;7)) From gl* and 32*5 the optimal estimates of 7 and /? are found as Y* = A~ιZ {γ;tVn(a)-ι(yt

- YptX'β**)}

and Γ

+Vn(θl)'~lYnt \ X

n

t=l T

(

t=i

{

where Δ = Z(ΣLI 4.2

—ZA~~l. t=l T

K

t=i

J

YptV'1 (a)Ypt)Z' + Γ " 1 .

Generalized Linear Mixed Models for Markov Processes

Suppose that {yt(j),t = 0,1,..., Γ, j = 1,2,..., n} is a markov process with a transition density f(Vt(j)\Vt-ιU)>Φ(J)) = c exp{φ{j)mt{yt{j),yt-i{j))

~ qt(yt-i{J)>ΦU))} (4.7)

and xjβ + z'jΎ,

(4.8)

where 7 ~ JV(O,Γ) and c is a function of yt{j) and yt-ι(j). Here /3 is a (mxl) vector of fixed parameters, 7 is a (r x 1) vector of random parameters, and Xj and Zj are known vectors of covariates. It is assumed for simplicity that conditional on 7, the processes {yt{j)} are independent for different j = 1,2,...,n.

PARK AND BASAWA

260 Let X = (xι,x2,...,xn)

and Z = (zχ,Z2,. • • ,zn). Then (4.8) can be

written as

/ φ{\) \ φ(2)

Φ=

= X β + Zj.

We consider joint optimal estimation for (7, β), assuming that Γ is known. We choose elementary estimating functions ht and u for 7 and β such that Let

= rπt — qt and u = 7, where \

m t (y t (2),y ί _ 1 (2))

\

and qt =

™>t(yt(n),yt-i(n))

/ Consider the estimating space £o = {9 '• 9 = Σt=i -A-tht + Bu}. Let

A; = ~E[φ,§f Then we have

and

r-1, where ςft = diag {-Q^^qt{yt-ι(j),Φ{j))J = 1,... , n j . Thus, in the space £o the joint optimal estimating function for (7,/?) is given by g* = (^,^2)5 where n T

= ΣΣzj{ j z

and

m

t(ytU),yt-i(j))-qt(j))-r-1Ί,

(4.9)

n T

(4.10) The optimal estimates (7*,/?*) for (7,/3) can be obtained by solving the equation g* = 0 for 7 and /? simultaneously.

261

OPTIMAL ESTIMATING EQUATIONS

Prom (2.4), the information function corresponding to the optimal estimating function 3* is obtained as

/ ZHnZ' + Γ- 1 ZHnX' \ XHnZ XHnX

\ I

Where Hn = ΣΪ=ι E[qt] We now illustrate the model by two examples. Example 1. Consider an AR(1) process with normal errors, i.e. yt(j) = Φ(j)yt-Λj) + εt(j)

(4.11)

φ[j) = χ.β + z'jΊ,

(4.12)

and

where εt(j) ~ indep iVίO,σ|J and 7 ~ iVίO,Γj. The conditional density is given by -ιϋ)iΦ(j)) =ceχp\-

^2(yt{j)-Φ{j)yt-Aj))

>,

(4.13)

where c = -Ί=^=—. Since (4.13) can be written as f(ytϋ)\yt-i(j),Φϋ))=c* we have mt{ytϋ),yt-ι{j)) = and

This gives μ t (j) = σ^y^ifiφij) and Viϋ) = σ^y^ij). Hence the optimal estimating function for (7,/?) is given by g* = (31,32)? where n T

and n 9*2 =

T

262

PARK AND BASAWA

The optimal estimates of 7 and β are obtained by solving the equations gt = 0 and g\ = 0 which yields the same estimates as ήfj and β\ in section 4.1. Example 2. Let {yt{j)},t = 0,1,2,..., be a Markov chain, for each j = 1,2,..., n, defined on the binary state space {0,1}. Denote πtj = P(yt(j) = l|yt-i(j)), θtj = logit{πtj) = log ί γ ^ j — ) • We then have expjθtj} exp{θtj}' Consider the model θtj=βo + Φjyt-ι(j),

(4.14)

where φj = x'jβ + zfl- Conditionally on 7, the Markov chains {yt{j)}>J = 1,..., n, are assumed to be independent. Suppose 7 ~ Λf if), ΓJ. Conditional on 7, the transition densities are given by p(vtU)\Vt-i(j),Ί)

=

7rjJu)(l-^)(1-ytu))

=

exp{θtjyt(j)

=

exp{βoyt(j) + Φjyt-i(j)yt{j)

- logfl + ea?p{β

- log(l + exp{β0 + Φjyt The optimal estimating functions for (7,^,/?o) then reduce to n

Γ

n

Γ

and

The third estimating function g% here corresponds to the common intercept βo in the model for θtj

263

OPTIMAL ESTIMATING EQUATIONS

5

Marginal Quasi-likelihood Estimation

Let fit(y{t - l),β) = E\yt\yt-ι, • • ,yi,β], and Vt{y(t - l),β) = Var[yt\yt-ι,...,yi,β]. The optimal estimating equation for β is τ

Σ

/

0jH(v(t-l),β)\ Vt-\y(t-l),β)(yt--μt(y(t-l),β))=O.

(5.1)

Let β denote a solution of (5.1). Define μt{y(t - l),j,β) = E[yt\yt-u • ,2/i,7,/?L and t _i,...,yi 1 7,i9]. For fixed /3, let

denote an elementary estimating function for 7. Suppose 7 has a prior density π(7|α) with mean 0 and variance Γ. Consider the estimating equation T

Σ t=i

r - X 7 = 0.

(5.2)

For fixed j9, (5.2) is an optimal estimating equation when only the random parameter 7 is present in the model. We now substitute the marginal quasilikelihood estimate β in (5.2), when β is unknown. Denote the resulting estimate of 7(obtained from (5.2) after replacing β by β) by 7. The estimates β and 7 will be referred to as the marginal quasi-likelihood estimates(MQL). Note, however, that for fixed /3, (5.2) is the same as the estimating equation (1.14) for 7 corresponding to the joint optimal estimation of β and 7. It must be noted that (5.1) and (5.2) are not jointly optimal; on the other hand, (5.1) is optimal for β with respect to the marginal density of y = (y*,..., y\) and (5.2) is optimal for 7(when β is known) with respect to the joint density of y and 7. Application to Autoregressive Processes : Suppose that yt{j), t = 1,2,..., T, j = 1,2,..., n are observed at time t on the jth subject from the first order autoregressive process(AR(l) process): yt(j) = Φ(J)yt-i(j)+εt(j),

(5.3)

and = Xjβ + Zjj, where 7 ~ 7VρΓθ,r) and ε t = ( ε t ( l ) , . . . ,εt{n))\t

(5.4) = 1,2,...,T, are iϊd

random vectors with E[εΐ\ = 0 and Vαr[εt] = Vn(a). Assume that 7 and εt

264

PARK AND BASAWA

are independent for any t. Here β is a (m x 1) vector of fixed parameters, 7 is a (q x 1) vector of random parameters, and Xj and Zj are vectors of known covariates. Let (yt(l),---,»*(")) 1

Vt=

and ϊ t _ i = diag(yt-ι(j),j = 1,... ,n), i.e., lt_i is a diagonal matrix with tΛ O'? j ) diagonal element yt-i(j), j = 1,. . ,n. We consider optimal estimation for ^9, treating 7 as a nuisance parameter when a is known. We assume that ε* has a normal distribution for our illustration. Now

and Vt(y(t - 1),/?) = Ή.xZVαφlift-i,..

.,yi]ZYt-i

where X = (zi,..., x n ) and Z = (zi,..., z n ). To find the conditional expectation and variance, we derive the posterior density π(7|j/ί_i,... ,yi) of φ, conditional on the past observations

oc

p(yti,

r—\

(yr - Yr-^X'β + Z'Ί)) I exp j-Ifr'r-S - i (7 - ΔΓΛ^-ij'Δt-iίT - Δt-_\A-i) t-1 r=l XY

ΣU r-lVn{<x)-Xyr

r=l

- Σy'rVn{<*)-lYr-lX'β r=l

where ί-1

i K ί a ) - 1 ^ ! ^ + Γ"1 r=l

OPTIMAL ESTIMATING EQUATIONS and

265

ί-1 r=l

We then have βt(y(t — ί),β) and Vt(y(t — l),β) given by μt(y(t - 1),/3) = Yt-i{X'β + Z ' A ^ - i } , and - 1),/?) = Yt^

Vn(a).

Thus the optimal estimating equation for β is given as (5.1) and the optimal estimating function is computed as <Γ = t-\

where ί-1

Σ

—

A* Λ

t-ι —

r=l

and

BUi = Yt-iZ'A^ZYt-! + Vn(a). Hence, from the above equation, the optimal estimate of β is found as -.-1

τ ΓT

β =

Σ ,t=ι

Λ*

Ft*-1 A*

J

r=l

U=i

From (2.4), the information function corresponding to the optimal estimating function g* is obtained as

t=l

Suppose that, for fixed /?, we wish to estimate 7. We choose an elementary estimating function ht for 7 as

266

PARK AND BASAWA

Then the optimal estimating equation for 7 is given as (5.2) : 1

ι

Σ(r t _iZ')V n (α)- (yt - Yt-i(X'β + Z'Ί)) - T~ Ί = 0. t=l

Thus the above equation yields an estimate 7* of 7 when β is known. -1

T

= .t=i

•Σ The estimate 7* is optimal for 7 when β is known. When β is unknown, we replace it by β in 7* to obtain 7.

6

Concluding Remarks

In this paper, we have derived optimal estimating equations for estimating fixed and random parameters in mixed effects models with dependent observations. Among the issues that are not addressed in this paper are : (ϊ) asymptotic distribution theory, (ii) estimation of variance components, and (in) computational aspects. We hope to return to these topics in future work. Here, we content ourselves by offering the following remarks on these important issues. (i) Asymptotic Distribution Theory The consistency and asymptotic normality of the marginal quasi-likelihood estimate β in Section 5 can be established, under regularity conditions, from the general theory discussed by Heyde(1997). Appropriate extensions to develop asymptotic distribution theory for the joint estimation of β and 7 are needed. Moreover, work on asymptotic efficiency of the estimates will be useful. (ii) Estimation of Variance Components Assuming 7 ~ ΛΓ(0, Γ), one can estimate Γ from the marginal quasilikelihood in Section 5. An extension of the REML approach discussed by Breslow and Clayton(1993) can be used in practice. (in) Computational Aspects Extensions of Fisher scoring method discussed by Breslow and Clayton(1993) need to be developed. For most of the examples in our paper, however, we have obtained explicit solutions of the estimating equations. Consistent parameter estimates of the information matrices can easily be obtained and hence the estimates of the standard errors of the estimates can be computed, in principle.

OPTIMAL ESTIMATING EQUATIONS

267

References Basawa, I.V., Godambe, V.P., and Taylor, R.L., (Eds.) (1997). Selected Proceedings of the Symposium on Estimating Functions. IMS Lecture Notes-Monograph Series, Vol. 32. Breslow, N.E. and Clayton, D.G. (1993). Approximate Inference in Generalized Linear Mixed Models. Journal of the American Statistical Association 88, 421, 9-25. Chan, S. and Ghosh, M. (1998). Orthogonal Projections and the Geometry of Estimating functions. Journal of Statistical Planning and Inference 67, 227-245. Desmond, A.F. (1997). Optimal Estimating Functions, Quasi-likelihood and Statistical Modelling. Journal of Statistical Planning and Inference 60, 77-121. Durbin, J. (1960). Estimation of Parameters in Time-series Regression Models. Journal of Royal Statistical Soc. B22, 139-153. Ferreira, P.E. (1981). Extending Fisher's Measure of Information. Biometrika 68, 3, 695-698. Ferreira, P.E. (1982). Estimating Equations in the Presence of Prior Knowledge. Biometrika 69, 3, 667-669. Godambe, V.P. (1960). An Optimum Property of Regular Maximum Likelihood Estimation. Annals of Mathematical Statistics 31, 1208-1211. Godambe, V.P. (1985). The Foundations of Finite Sample Estimation in Stochastic Processes. Biometrika 72, 2, 419-428. Godambe, V.P. (Ed.) (1991). Estimating Functions. Oxford University Press, Oxford. Godambe, V.P. (1998). Linear Bayes and Optimal Estimation. To appear in Annals of the Institute of Statistical Mathematics. Heyde, C.C (1997). Quasi-Likelihood and Its Application, A General Approach to Optimal Parameter Estimation. Springer-Verlag, New York. Robinson, G.K. (1991). That BLUP is a Good Thing: The Estimation of Random Effects. Statistical Science 6, 1, 15-51.

268

PARK AND

BASAWA

Sutradhar, B.C. and Godambe, V.P. (1997). On Estimating Function Approach in the Generalized Linear Mixed Model. IMS Selected Proceedings of the Symposium on Estimating Functions(Basawa I. V, Godame, V.P., and Taylor, R.L. Eds.) 32, 193-213. Waclawiw, M.A. and Liang, K.Y. (1993) Prediction of Random Effects in the Generalized Linear Model. Journal of the American Statistical Association 88, 421, 171-178. Wefelmeyer, W; (1996). Quasi-likelihood Models and Optimal Inference. Annals of Statistics 24, 1, 405-422.

271 Institute of Mathematical Statistics LECTURE NOTES — MONOGRAPH SERIES

RECONSTRUCTION OF A STATIONARY SPATIAL PROCESS FROM A SYSTEMATIC