Applied Semi-Markov Processes

APPLIED SEMI-MARKOV PROCESSES

APPLIED SEMI-MARKOV PROCESSES

By JACQUES JANS SEN Solvay Business School, Brussels, Belgium RAIMONDO MANCA Universita di Roma "La Sapienza," Italy

Springer

Library of Congress Control Number:

ISBN-10: 0-387-29547-X

2005933791

e-ISBN 0-387-29548-8

ISBN-13: 978-0387-29547-3

Printed on acid-free paper.

AMS Subject Classifications: 60K05, 60J10, 60K15, 60K10, 60K20, 62P05, 65C40, 65C99, 91B30, 91B7Q

© 2006 Springer Science+Business Media, Inc.

All rights reserved. This work may not be translated or copied in whole or in part without the written permission of the publisher (Springer Science-t-Business Media, Inc., 233 Spring Street, New York, NY 10013, USA), except for brief excerpts m connection with reviews or scholarly analysis. Use in connection with any form of information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed is forbidden. The use in this publication of trade names, trademarks, service marks, and similar terms, even if they are not identified as such, is not to be taken as an expression of opinion as to whether or not they are subject to proprietary rights. Printed in the United States of America. 987654321 springeronline.com

Contents Preface 1 Probability tools for stochastic modelling 1 The sample space 2 Probability space 3 Random variables 4 Integrability, Expectation and Independence 5 Main distribution probabilities 5.1 The binomial distribution 5.2 The Poisson distribution 5.3 The normal (or Laplace-Gauss) distribution 5.4 The log-normal distribution 5.5 The negative exponential distribution 5.6 The multidimensional normal distribution 6 Conditioning (From independence to dependence) 6.1 Conditioning: introductory case 6.2 Conditioning: general case 6.3 Regular conditional probability 7 Stochastic processes 8 Martingales 9 Brownian motion

1 1 3 7 10 16 16 17 18 20 21 22 24 24 28 32 36 40 43

2 Renewal theory 1 Purpose 2 Main definitions 3 Classification of renewal processes 4 The renewal equation 5 The use of Laplace transform 5.1 The Laplace transform 5.2 The Laplace-Stieltjes transform 5.3 An application to the renewal function 6 Application of Wald' s identity 6.1 Wald's identity 6.2 A lower bound for renewal function R 7 Asymptotical behaviour of the N(t)-pYOCQSS 8 Recurrence times 8.1 Definitions 8.2 Distributions of recurrence times 8.3 Asymptotic behaviour

45 45 46 47 52 59 59 64 66 67 67 68 69 72 72 73 79

Contents

VI

8.4 Example: the Poisson process 8.5 Some parameters of the limit distributions 8.6 A characterization of the Poisson process 9 Delayed and stationary renewal processes 10 Numerical aspects 10.1 General quadrature method 10.2 Some particular formulas 10.3 Numerical approximation of the Poisson process 10.4 A real life example in motor-car accidents 10.4.1 Data description 10.4.2 The result distribution 3 Markov chains 1 Definitions 2 Markov chain state classification 2.1 Periodic and aperiodic states 2.2 Essential and inessential states - Irreducibility 2.3 Transient and recurrent states 3 Occupation times 4 Computation of absorption probabilities 5 Asymptotic behaviour 6 Examples 7 A case study in a social insurance problem (Janssen (1966)) 8 Asymptotic numerical treatment 8.1 An algorithm for MC Asymptotical Study 8.2 Irreducible real data example in motor-car insurance 8.3 Reducible and uni-irreducible examples-Canonical form connection 4 Markov renewal theory, Markov random w^alks and semiMarkov processes. 1 Positive (J-X) processes 2 Semi-Markov and extended semi-Markov chains 3 Primary properties 4 Examples 5 Markov renewal processes, semi-Markov processes and associated counting processes 6 Markov renewal functions 7 Classification of the state of a MRP 8 The Markov renewal equation 9 Asymptotic behaviour of a MRP 9.1 Asymptotic behaviour of Markov renewal functions 9.2 Asymptotic behaviour of solutions of Markov renewal

82 83 86 87 92 93 95 97 98 98 101 105 105 108 108 109 110 113 113 115 119 122 125 125 129 133

145 145 145 147 151 154 156 161 163 165 165

Contents

Vll

equations 10 Asymptotic behaviour of SMP 10.1 Irreducible case 10.2 Non-irreducible case 10.2.1 Uni-reducible case 10.2.2 General case 11 Recurrence times 11.1 Definitions 11.2 Transient results 11.3 Asymptotic results 12 Delayed and stationary MRP 13 Particular cases of MRP 13.1 Renewal processes and Markov chains 13.2 SMP an MRP of zero order (Pyke (1962)) 13.2.1 First type of zero order MRP 13.2.2 Second type of zero order MRP 13.3 Continuous Markov processes 14 A case study in social insurance (Janssen (1966)) 14.1 The semi-Markov model 14.2 Comparison of Markov and semi-Markov models 15 Asymptotical numerical examples

168 169 169 171 172 173 173 173 174 175 178 185 185 185 186 186 187 187 187 188 189

5 Functionals of (J-X) processes 1 (J-X) processes 2 Functionals of (J-X) processes 3 Functionals of positive (J-X) processes 4 Classical random walks and risk theory 4.1 Purpose 4.2 Basic notions on random walks 4.3 Renewal results on ladder variables 4.4 Classification of random walks 5 Defective positive (J-X) processes 6 Semi-Markov random walks 7 Indice variables for semi-Markov random walks 8 Distribution of the supremum for semi-Markov random walks

193 193 195 201 202 202 202 207 210 214 219 222

6 Non-homogeneous Markov and semi-Markov processes. 1 General definitions 1.1 Completely non homogeneous semi-Markov processes 1.2 Special cases 1.2.1 Non-homgeneous Markov additive process and semi-Markov process

224 227 227 227 231 231

Contents

Vlll

1.2.2 Non-homogeneous MC 1.2.3 Homogeneous Markov additive process 1.2.4 Non-homogeneous renewal process 1.3 Intensities for CNHSMP 1.3.1 Definition 1.3.2 Generalized Polya processes 2 Non-homogeneous Markov chains 3 Non-homogeneous Markov processes 3.1 Classical definition 3.2 The Chapman Kolmogorov equations 3.3 Intensity functions 3.4 The special case of constant intensity functions 3.5 NHMP as a special case of NHSMP 4 An NHMC transportation example

232 232 232 233 233 234 235 236 237 237 238 241 243 243

7 Markov and semi-Markov rew^ard processes. 1 Reward structure 1.1 Classification and notation 1.2 Discrete time fixed interest rate 1.3 Discrete time variable interest rate 1.4 Continuous time fixed interest rate intensity 1.5 Continuous time variable interest rate intensity 2 Discrete time Markov reward processes 2.1 Undiscounted case 2.1.1 First model 2.1.2 Second model 2.1.3 Third model 2.1.4 Fourth model 2.1.5 Fifth model 2.1.6 Sixth model 2.1.7 Seventh model 2.1.8 Eighth model 2.2 Discounted case 2.2.1 Immediate cases 2.2.2 Due cases 2.3 General algorithm for the DTMRWP 2.3.1 Homogeneous MRWP 2.3.2 Non-homogeneous MRWP 2.4 A DTMRWP applied example 3 Semi-Markov reward processes 3.1 Undiscounted CTSMRWP 3.1.1 First model 3.1.2 Second model

247 247 247 249 249 250 251 252 252 252 253 253 254 254 255 255 256 256 256 261 264 264 265 265 268 268 268 269

Contents 3.1.3 Third model 3.1.4 Fourth model 3.1.5 Fifth model 3.1.6 Sixth model 3.1.7 Seventh model 3.2 Discounted CTSMRWP 3.2.1 First model 3.2.2 Second model 3.2.3 Third model 3.2.4 Fourth model 3.2.5 Fifth model 3.2.6 Sixth model 3.2.7 Seventh model 3.2.8 Eighth model 3.2.9 Ninth model 4 Insurance applications of CTSMRWP 4.1 Two state examples 4.2 Three state examples 4.3 Four state examples

ix 270 271 272 272 273 273 273 275 275 276 277 277 278 278 279 280 282 286 290

References

295

Author index Subject index

303 305

PREFACE This book aims to give a complete and self-contained presentation of semi-Markov processes with finitely many states, divided into three major topics: theory, computation, and application in a correct approach for building applied stochastic models. To help assure the self-containedness of the book. Chapter 1 summarizes the basic probability tools that the reader will need to understand our presentation. The authors argue that semi-Markov processes furnish a framework for the construction of useful applied models, some of them giving fruitful alternative approaches to the classical models, as shown in finance with mixed semi-Markov diffusion models, for example. We strongly hope that practitioners with a variety of interests will have the opportunity to use these new types of models in their own fields. Another important feature of this book is its presentation of both homogeneous and non-homogeneous models. It is well known that the fundamental structure of many real-life problems is non-homogeneous in time and the application of homogeneous models to such problems gives, in the best case, only approximated results or, in the worst case, nonsensical results. This book addresses a very large public that includes undergraduate and graduate students in mathematics, applied mathematics, economics and business studies; actuaries and financial intermediaries; engineers and operations researchers; and also researchers in universities and RD departments in banking, insurance and industry. Readers who have mastered the material in this book will have a firm grasp of semi-Markov processes and be able to use them in the solution of their problems. They will indeed have a new approach giving a more competitive model related to the complexity of real-life problems. Let us now give some comments on the contents of the book. As we start from the fact that the semi-Markov processes are the children of a successful marriage between renewal theory and Markov chains, these two topics are presented in Chapters 2 and 3. The full presentation of Markov renewal theory, Markov random walks and semi-Markov processes is given in Chapter 4; functionals of (JX) processes and semi-Markov random walks are presented in Chapter 5 in the homogeneous case and are followed by a presentation of non-

xii

Preface

homogeneous Markov and semi-Markov processes in Chapter 6. Finally, continuous time semi-Markov reward processes are presented in Chapter 7, and in the last part of this chapter applications to insurance problems are given. Many parts of this book have been tought by the authors at several universities: Universite Libre de Bruxelles, Vrije Universiteit Brussel, Universite de Bretagne Occidentale (EURIA), Universites de Paris 1 (La Sorbonne) and Paris VI (ISUP), ENST-Bretagne, Universite de Strasbourg, and the Universities of Roma ("La Sapienza"), Firenze and Pescara. Our common experience has joined to create this book, taking into account the remarks of students in our various lectures and hoping to convince potential readers to enter an attractive mathematical field with new frontiers for real-life applications.

Jacques Janssen

Raimondo Manca

Chapter 1 PROBABILITY MODELLING

TOOLS

FOR

STOCHASTIC

In this chapter, the reader will find a short summary of the basic probability tools useful for understanding of the following chapters. Although there currently exist some very good presentations of stochastic processes from a "modem" point of view, that is to say with emphasis on sample paths or trajectories without using measure theory (see for example Cinlar (1975b) or Ross (2000), a deep understanding of stochastic processes requires the notion of integrability on a measurable space, well presented in Kolmogorov and Fomin (1961), Halmos (1974), Chung (2000) and many other standard textbooks. We will focus our attention on stochastic processes in discrete time and continuous time defined by sequences of random variables.

1 THE SAMPLE SPACE The basic concrete notion in probability theory is that of the random experiment, that is to say an experiment for which we cannot predict in advance the outcome. With each random experiment, we can associate the so-called elementary events CO, and the set of all these events Q is called the sample space. Some other subsets of Q will represent possible events, Let us consider the following examples. Example 1.1 If the random experiment consists in the flipping of a coin {H means that the outcome of the toss is heads and 7 that it is tails.), then we have as sample space ^ = {H,T]. (1.1) The family of all possible events is the set 3 = {0,{//},{r,}{//,r,}},

(1.2)

that is the family of all subsets of Q. Example 1.2 If the experiment consists in the throwing of two distinguishable dice, then the sample space consists of the 36 elementary events (1.3) ^ = {{Uj)JJ = \.....6], where the outcome (zj) is said to occur if / appears on the first die andy on the second one.

2

Chapter 1

Example 1.3 If the experiment consists in the throwing of two indistinguishable dice, then the sample space consists of the 21 elementary events = l,„.,6}, (1.4) Q=^{(Uj)J<jJ,j where the outcome (ij) is said to occur if / is the result of one of the two dice, andy the result of the second. Example 1.4 If the experiment consists in the measurement of the lifetime of an integrated circuit, then the sample space is the set of all non-negative real numbers R"^. Possible events are [a,6],(a,6),[a,Z?),(a,6] where for example the event [a,b) means that the lifetime is at least a and strictly inferior to b. Example 1.5 An insurance company is interested in the number of claims per year for its portfolio. In this case, the sample space is the set of natural numbers N. Example 1.6 A bank is to invest in some shares; so the bank looks to the history of the value of different shares. In this case, the sample space is the set of all nonnegative real numbers R"^. To be useful, the set of all possible events must have some properties of stability so that we can generate new events such as: (i)

the complement A"": A'' -{coeO.:co^

A],

(ii) the union A[jB : A[jB = {co:coe A orcoeB ],

(1.5) (1.6)

(1.7) (iii) the intersection AClB : AClB = {CO:COG A,COGB] . More generally, if {A^,n>V) represents a sequence of events, we can also consider the following events: n>\

n>\

representing respectively the union and the intersection of all the events of the given sequence. The first of these two events occurs iff at least one of these events occurs and the second iff all the events of the given sequence occur. The set Q is called the certain event and the set 0 the empty event. Two events A and B are said to be disjoint or mutually exclusive iff ^ n 5 =0. (1.9) Event A implies event B iff AczB. (1.10) Example 1.7 (a) In Example 1.2, the event "the sum of the dice points equals 7" is the set

Probability Tools

3

{a6),(2,5),(3,4),(4,3),(5,2),(6,l)}. (1.11) (b) In Example 1.3, the event "the sum of the dice points is seven" is the set {(1,6),(2,5X(3,4)}. (1.12) (c) In Example 1.6, the event "the value of the share is between "50$ and 75$" is given by the set [50,75] (1.13)

2 PROBABILITY SPACE Given a sample space Q , the set of all possible events will be noted by 3 , supposed to have the structure of a a-field or a a-algebra. Definition 2.1 The family 3 of subsets of Q is called a cr-field or a a algebra iff the following conditions are satisfied: (i) Q , 0 belong Xo 3 , (ii) Q is stable under denumerable intersection: ^„G3,V«>l=^f|^„€5, (2.1) n>\

(iii) 3 is stable for the complement set operation: ^G3=>^'e3,(2.2) --a-A). {with A' Then, using the well-known de Morgan's laws saying that

'u^"l -n<' fn^«l =u<' n>\

)

n>\

\n>\

)

(2'3)

n>\

it is easy to prove that a G -algebra 3 is also stable under denumerable union: (2.4) ^„ G 3, V/? > 1 =:> IJ^^ G 3. Any couple (Q, 3) where 3 is a (T -algebra is called a measurable space, The next definition concerning the concept of probability measure or simply probability is an idealization of the concept of X\\Q frequency of an event. Let us consider a random experiment called E with which is associated the couple (Q, 3 ) ; if the set A belongs to 3 and if we can repeat the experiment E n times, under the same conditions of environment, we can count how many times A occurs. If n{A) represents this number of occurrences, the frequency of the event A is defined as f{A)^^^.

(2.5)

Chapter 1 In general, this number tends to become stable for large values of«. The notion of frequency satisfies the following elementary properties: (i) {A,Bs:5,AnB = 0=>fiA[jB)^f(A) + fiB), (2.6) (ii) / ( n ) - l , (2.7) (iii) A,Be3,:::>f(A[jB) = f(A) + f(B)-f(AnB), (2.8) (iv)A€3^f(A') = l-f(A). (2.9) To have a useful mathematical model for the theoretical idealization of the notion of frequency, we now introduce the following definition. Definition 2.2 a) The triplet (Q,3,P) is called a probability space if Q is a non-void set of elements, 3 a or-algebra of subsets of Cl and P an application from 3 /a[0,l] such that: (A„,n > l),A„ e 3,« > 1: (/ ^ y ^ A. fl Aj = (f) ( \ „ (2.10) (i)

^P\ U^" hS-^^^") {\

J «=1

(ii) P(Q) = 1. (2.11) b) The application P satisfying conditions (2.10) and (2.11) is called a probability measure or simply probability. Remark 2.1 1) The sequence of events {A^,n> 1) satisfying the condition that (A^,n>llA,e3,n>l:i^j=>A^nAj=0 (2.12) are called mutually exclusive, 2) The relation (2.11) assigns the value 1 for the probability of the entire sample space Q. There may exist events A' strictly subsets of Q such that (2.13) P{A') = l. In this case, we say that A is almost sure or that the statement defining A is true almost surely (in short a.s.) or holds for almost all CD . From axioms (2.10) and (2.11), we can deduce the following properties: Property 2.1 (i)If ^ , 5 e 3 , t h e n , P(A[jB) = P(A) + P(B)-P(AnB), (ii) If ^ E 5, then, P(A') = l-P(A).

(2.14) (2.15)

Probability Tools

5

(2.16) (iii) P(0) = O. (iv) If (B^,n>l) is a sequence of disjoint elements of 3 forming a partition of Q, then for all A belonging to 5 , P(A)^f^P(AnB„).

(2.17)

(v) Continuity property ofP: if (A^,n> 1) is an increasing (decreasing) sequence of elements of 3 , then

P\ \JA„ UlimP(^J; P\ H A UlimP(^J . \n>]

J

\

\n>\

J

(2.18)

J

Remark 2.2 a) Boole's inequality asserts that if {A^,n> 1) is a sequence of events, then

(2.19)

P\ U^ A U E ^ K ) ,A7>1

J

n>\

b) From (2.14), it is clear that we also have A^B=>P(A)
(2.20)

Example 2.1 a) The discrete case When the sample space Q is finite or denumerable, we can set Q = {co,,„„cDj,..]

(2.21)

and select for 3 the set of all the subsets of Q, represented by 2" . Any probability measure P can be defined with the following sequence: ip.,j>l\p.>0,j>l,^Pj=l (2.22) so that P([wj}) = pjj>l.

(2.23)

On the probability space (Q,2",P), the probability assigned for an arbitrary event A = \^CDj^^,...,co^^^,kj>l,j = \,...,l,k.^kj

P(A) = ZP,^,

if i^j

is given by

(2.24)

Example 1.1 is a particular case. For a "fair" coin, the probability measure P is defined as follows:

Chapter 1

P({//}) = P({r}) = i .

(2.25)

For Example 1.2, we have for two "fair" dice: ^({^i}) = ^ -

(2-26)

For Example 1.3, we get: P({''J}) = J^''<J'

^ ( { M - } ) - ^ , / = 1,...,6.

(2.27)

b) The continuous case Let Q be the real set R; It can be proven (Halmos (1974)) that there exists a minimal cr-algebra generated by the set of intervals: b}. (2.28) It is called the Borel a-algebra represented by>5 and the elements of J3 are called Borel sets. Given a probability measure P on (Q,>5), we can define the real function F, called the distribution function related to P, as follows. Definition 2.3 The function Ffrom R to [0,l] defined by: P((-oo,x]) = F(x),x G R

(2.29)

is called the distribution function related to the probability measure P. From this definition and the basic properties of P, we easily deduce that: P({a,b]) = F(b)-F(a), P({a,b)) = F(b-)-F(a), ^ ^ ^ ^ (2.30) P{[a,b]) = F(b)-F(a-). P{[a,b)) = Fib-)-F(a-l Moreover, from (2.29), any function F from R to [0,l] is a distribution function (in short d.f) iff it is a non-decreasing function satisfying the following conditions: F is right continuous at every point XQ, l\mF(x) = F(x,l (2.31) XTXQ

and moreover lim F(x) = 1, lim F(x) = 0 .

(2.32)

If the function F is derivable on R with/as derivative, we have F(x)=r

f(y)dy,xeR.

(2.33)

J-oo

The function/is called the density function associated with the d.f F and in the case of the existence of such a Lebesgue integrable function on R, F is called absolutely continuous.

Probability Tools

7

From the definition of the concept of integral, we can give the intuitive interpretation of/as follows; given the small positive real number Ax, we have: (2.34) P[{x,x + Ax})« fix)Ax . Using the Lebesgue Stieltjes integral, it can be seen that it is possible to define a probability measure P on (R,/?) starting from a d.f F on R by the following definition of P: P(A)=jdF(x)yAe^, (2.35) A

In the absolutely continuous case, we get P(A)^jf(y)dy.

(2.36)

A

Remark 2,3 In fact, it is also possible to define the concept of d.f in the discrete case if we set, without loss of generality, on (NQ,2^'), the measure P defined from the sequence (2.22). Indeed, if for every positive integer k, we set F(k) = f^Pj

(2.37)

7=1

and generally, for any real x,

then, for any positive integer k, we can write P{{l.,.,k]) = F{k)

(2.39)

and so calculate the probability of any event.

3 RANDOM VARIABLES Let us suppose the probability space (Q,3,P) and the measurable space (E,i//) are given. Definition 3.1 A random variable (in short r.v.) with values in E is an application Xfrom Q to E such that \/BGI/:X-'(B)G^,

(3.1)

where X\B) is called the inverse image of the set B defined by X-\B) = {co:X(cD)eB],X-\B)e3.

(3.2)

Particular cases a) If (£*, I//) = (R, J3), X is called a real random variable.

g

b) If(E,i//) = (R,j3),

Chapter 1

where

R

is the extended real line defined by

RU{+oo}U{-oo} and P the extended Borel cr-field of M, that is the minimal (J -field containing all the elements of /? and the extended intervals [-oo,a),(-oo,a],[-oo,a],(-oo,a), [a,+00),(a,+00],[a,+00],(a,+00), a e M , Xis called a rea/ extended value random variable. c) If £' = R"(«>l)with the product cr-field y5^"^of J3, X is called an «dimensional real random variable. d) If ^ = R^"^(n>l) with the product a-field yS^'^^of ^ , Z is called a real extended n-dimensional real random variable. A random variable Xis called discrete or continuous according as Stakes at most a denumerable or a non-denumerable infinite set of values. Remark 3.1 In measure theory, the only difference is that condition (2.11) is no longer required and in this case the definition of a r.v. given above gives the notion of measurable function. In particular a measurable function from (R,y5) to (R,/?) IS Qdi\\Qd?i Borel function. Let Jf be a real r.v. and let us consider, for any real x, the following subset of Q: [(D:X{CO)<X].

As, from relation (3.2), [co: X{(o) <x] = X-' ((-oo,x]), it is clear from relation (3.1) that this set belongs to the a -algebra 3 . Conversely, it can be proved that the condition {^:X(6;)<X}G3,

(3.4)

(3.5)

valid for every x belonging to a dense subset of R , is sufficient for X being a real random variable defined on Q. The probability measure P on (Q,3) induces a probability measure // on (R,y5) defined as V5 e y5: i^{B) = P{{o): X(co) e B}).

(3.6)

We say that ju is the induced probability measure on (R,>5), called the probability distribution of the r.v. X. Introducing the distribution function related to //, we get the next definition. Definition 3.2 The distribution function of the r.v. X, represented by F^, is the function from R ^ ' [ 0 , l ] defined by

Probability Tools

F^(X) = JU({-^,X])

9

= P({CD:X(CO)<X]),

In short, we write F,{x) = P{X<x),

(3.7)

(3.8)

This last definition can be extended to the multi-dimensional case with a r.v. X being an n-dimensional real vector: X = {X^,,,.,X^), a measurable appHcation from (Q,3,P) to (M%>5"). Definition 3.3 The distribution function of the r.v. X ~{X^,...,X^),

represented

by F^, is the function from W to [0,l] defined by F,{x„...,x„) = P({co:X,{co)<x„...,X„{co)<x„}).

(3.9)

In short, we write (3.10) FM^...,x„) = P{X,<x„...,X„<x„). Each component Xi (i=l,...,n) is itself a one-dimensional real r.v. whose d.f, called the marginal d.f, is given by F^(x.) = F^(+oo,...,+oo,x.,+oo,...,+oo). (3.11) The concept of random variable is stable under a lot of mathematical operations; so any Borel function of a r.v. Xis also a r.v. Moreover, if Xand 7 are two r.v., so are mf{X,Y},sup{Xj},X + Y,X-Y,X'Y,y,

(3.12)

provided, in the last case, that 7 does not vanish. Concerning the convergence properties, we must mention the property that, if (X^,n> 1) is a convergent sequence of r.v. - that is, for aWcoeQ, the sequence (A^^ (^y)) converges to X(co) - , then the limit X is also a r.v. on Q. This convergence, that may be called the sure convergence, can be weakened to give the concept of almost sure (in short a.s.) convergence of the given sequence. Definition 3.4 The sequence (X^(a))) converges a.s. to X(CD) if P[{cD:\imX^(co) = X(co)}) = l.

(3.13)

This last notion means that the possible set where the given sequence does not converge is a null set, that is a set A^ belonging to 3 such that P(N) = 0. (3.14)

IQ

Chapter 1

In general, let us remark that, given a null set, it is not true that every subset of it belongs to 5 but of course if it belongs to 3 , it is clearly a null set (see relation (2.20)). To avoid unnecessary complications, we will suppose from now on that any considered probability space is complete. This means that all the subsets of a null set also belong to 3 and thus that their probability is zero.

4 INTEGRABILITY, EXPECTATION AND INDEPENDENCE Let us consider a complete measurable space (Q, 5 , //) and a real measurable variable X defined on Q . To any set A belonging to 3 , we associate the r.v. I^ , called the indicator of ^4, defined as

If there exists partition {A^,n> 1) with all its sets measurable such that (D^A^:=^ X{co) = a^(a^ e R),n > 1,

(4.2)

then X is called a discrete variable. If moreover, the partition is finite, it is said to bQjinite. It follows that we can write JL"under the following form: 00

X(a)) = ^a„I,^(co).

(4.3)

Definition 4.1 The integral of the discrete variable X is defined by jXdM = Za„MiA„), n

(4.4)

n=\

provided that this series is absolutely

convergent,

Of course, i f X i s integrable, we have the integrability of \x\ too and

^X\d^i = Y}a„\^{A„).

(4.5)

To define in general the integral of a measurable function X, we first restrict ourselves to the case of a non-negative measurable variable X for which we can construct a monotone sequence {X^,n > 1) of discrete variables converging to X as follows:

^«(^) = ET7^f . , , . . . ! •

(4-6)

Probability Tools

11

Since for each n, XSco)<X^,,{co\ 1

(4.7)

0<X{cD)-XSco)<—. the sequence {X^ ,n>\) of discrete variables converges well monotonically to X on Q . Deflnition 4.2 The non-negative measurable variable X is integrable on Q iff the elements of the sequence (X^,n > 1) of discrete variables defined by relation

\x„dP converges,

(4.6) are integrable and if the sequence

Vn

J

From this last definition, it follows that E{X) = \imE{XJ, where - k

(

(4.8)

^

(4.9)

(o\—<X<-

To extend the last definition without the non-negativity condition on X, let us introduce for an arbitrary variable X, the variables X^ and X~ defined by ^ " ( ^ ) = sup{X(^),0},Z-(^) = -inf{X(^),0}, (4.10) so that (4.11) X^X^ -X-. Definition 4.3 The measurable variable X is integrable on Q iff the nonnegative variables X^ and X~ defined by relation (4.10) are integrable and in this case (4.12) jXdju = jX^d/j - jX'dju. Remark 4.1 a) If the integral of X does not exist, it may however happen that

^^

f jX^djUKoo n

f

jZ-^//
J Q

\Q

^ =00 .

(4.13)

J

In these two cases, we say that the integral o f X i s infinite; more precisely, we have \Xdju =-col \xdjU = +co

(4.14)

If A is an element of the a -algebra 3 , the integral on A is simply defined by

J2

Chapter 1

jXdju=jXI,dju, A

(4.15)

A

Of course, X being a non-negative measurable variable with an infinite integral, it means that the approximation sequence (4.6) diverges to +oo for almost all o). Now let us consider a probability space ( Q , 3 , P ) and a real random variable X defined on Q . In this case, the concept of integrability is designed by expectation represented by

E{X) = Jx^P(= jXdP),

(4.16)

Q

provided that this integral exists. The computation of the integral ^ jXdPl = jXdP

(4.17)

can be done using the induced measure // on (R,/?),defined by relation (3.6) and then using the distribution function F of X Indeed, we can write E(X)\

= jXdp]=jXdjU,

and ifFx is the d.f of X, it can be shown that E{X)^\xdF,{x),

(4.18)

(4.19)

R

this last integral being a Lebesgue Stieltes integral. Moreover, if F ^ i s absolutely continuous with/i-as density, we get +00

E{X)= \xf^{x)dx.

(4.20)

-00

If g is a Borel function, we also have (see for example Chung (2000), Royden (1963), Loeve (1963)) +00

E{g{X))=\g{x)dF,

(4.21)

-00

and with a density for X +00

E{g{X))=\g{x)f,{x)dx.

(4.22)

—00

The most important properties of the expectation are given in the next proposition. Proposition 4.1 (i) Linearity property of the expectation :IfX and Y are two integrable r.v. and a,b two real numbers, then the r.v. aX+bYis also integrable and

Probability Tools

13

E{aX + bY) = aE(X) + bE(Y), (ii) If (A^,n>V) is a partition of Q, then

(4.23)

00

E{X) = Y,\XdP.

(4.24)

(iii) The expectation of a non-negative r.v. is non-negative. (iv) IfXand Yare integrable r.v., then X
(4.25)

(v) IfXis integrable, so is \x\ and \E{X)\<E\X\.

(4.26)

(vi) Dominated convergence theorem (Lebesgue: Let {X^,n>\) be a sequence of r.v. converging a.s. to the r.v. X integrable, then all the r.v. X^ are integrable and moreover (4.27) \xmE{X„) = E{\imX„) {^E{X)). (vii) Monotone convergence theorem (Lebesgue): Let {X^,n>\) be a nondecreasing sequence of non-negative r.v; then relation {A.21) is still true provided that + oo is a possible value for each member. {wm) If the sequence of integrable r.v. {X^,n>\) is such that f,E(\X„\)<^.

(4.28) 00

then the random series ^^X^ converges absolutely a.s. and moreover n=\

YE{X„) = E\J^X„ {-E{X)), n=\

\n=\

(4.29)

J

where the r.v. is defined as the sum of the convergent series. Given a r.v. X, moments are special cases of expectation. Deflnition 4.4 Let a be a real number and r a positive real number, then the expectation

£(|X-4)

(4.30)

is called the absolute moment ofX, of order r, centred on a. The moments are said to be centred moments of order r \i a=E(X). In particular, for r=2, we get the variance of Xrepresented by a^ (var(X)), c7'=J?(|X-mh.

(4.31)

14

Chapter 1

Remark 4.2 From the linearity of the expectation (see relation (4.23)), it is easy to prove that C7^=E{X^)-{E{X))\ (4.32) and so CT^<E{X^), (4.33) and more generally, it can be proven that the variance is the smallest moment of order 2 whatever the number a is. The next property recalls inequalities for moments. Proposition 4.2 {Inequalities of Holder and Minkowski) (i) Let Xand Ybe two r.v. such that \X\ ,|7| are integrable with l
(4.34)

\E{XY)\<[E(\x^\jf''\Ei\Y^\)f.

(4.35)

(ii) LetXand Ybe two r.v. such that \x\^ ,|7|^ ,1 < /? < oo, are integrable, then E[\X^Y\'f

<[E[\X\^]f

+(^(|7r)f.

(4.36)

If/7=2 in the first part of this last proposition, then relation (4.36) gives the Cauchy Schwarz inequality \EiXY)\<[E[\xff

\Ei^Y\')f.

(431)

The last fundamental concept we will now introduce in this section is that of stochastic independence, or more simply independence. Definition 4.5 The events A^,...,A^,(n>l)are independent iff f m

stochastically independent or \

m

Vm^2,...,nyn,=l,...,n:n,^n,^-^n,:P\f]A„^ h l l ^ K ) '

(4.38)

For n=2, relation (4.38) reduces to PiA,nA,) = PliA,)P(A,). (4.39) Let us remark that piecewise independence of the events A^,...,A^,(n>l) does not necessarily imply the independence of these sets and thus not the stochastic independence of these n events. As a counter-example, let us suppose we drew a

Probability Tools

15

ball from an urn containing four balls called b\, bi, 63, 64 and let us consider the three following events: A, ={b,,b,},A, ={b,,b,},A, ={b,,b,}. (4.40) Then assuming that the probability of having one ball is K ' ^ ^ 8^^ P(A,nA,)

= P(A,nA,)

= P(A,nA,)

= ^,

(4.41)

but as P{A,nA,nA,)

=^

(4.42)

too, we do not have the relation P(A, nA,f]A,) = P(A,)P(A,)P(A,) , (4.43) and so we have proved that independence in pairs does no imply the independence of these three events. We will now extend the concept of independence to random variables. Definition 4.6 (i) The n real r.v. X\,X2,...,Xn defined on the probability space [Q,^,P^are said to be stochastically independent, or simply independent, iff for any Borel sets B\,B2, ...,5„, we have f]{co:X,(a))eB,}]

= YlP{{^-^d^)^B,}).

(4.44)

(ii) For an infinite family of r.v., independence means that the members of every finite subfamily are independent. It is clear that if X\,X2,...,Xn are independent, so are the r.v. X.,...,X.^ with i^^-'-^ij^, ij^ =\,...,n,k = 2,...,n . From relation (4.43), we find that P(X,<x,,...,X^<x„) = P(X,<x,y-P(X^<xJMx,,^-,xJeW. (4.45) If the functions F^,F^ ,...,F^ are the distribution functions of the r.v. X = (X^,...,X^),X^,...,X^,WQ /V(x„....,xj=:/V_(x0

can write the preceding relation under the form

F,^(XJ,V(X„...,XJER".

(4.46)

It can be shown that this last condition is also sufficient for the independence ofX = {X^,...,XJ,X^,...,X^. If these d.f have densities fx,fx V J A ? relation (4.46) is equivalent to / , ( x , , . . . , x J = /,_(x,).../^^(xJ,V(xp...,xjER\

(4.47)

In case of the integrability of the n real r.y Xi,X2,...,X„, a direct consequence of relation (4.46) is that we have a very important property for the expectation of the product ofn independent r.v.:

j^

Chapter 1

^fri^*VrT^(^*)-

(4.48)

The notion of independence gives the possibihty to prove the result called the strong law of large numbers which says that if (X^,n>l)is a sequence of integrable independent and identically distributed r.v., then

-t,X,-^^E(X),

(4.49)

n k=\

The next section will present the most useful distribution functions for stochastic modelling.

5 MAIN DISTRIBUTION PROBABILITIES Here we shall restrict ourselves to presenting the principal distribution probabilities related to real random variables.

5.1 The Binomial Distribution Let us consider a random experiment E such that only two results are possible: a "success"(5) with probability/? and a "failure (F) with probability q=l'-p. If n independent trials are made in exactly the same experimental environment, the total number of trials in which the event S occurs may be represented by a random variable^whose distribution (p.J = 0,...,n) with p,=P(X = ili = l,...,n (5.1) is called a binomial distribution with parameters (n,p). From basic axioms of probability theory seen before, it is easy to prove that Pi

U^p^q"-\i

= 0,.,.,n,

(5.2)

\'J

a result from which we get (5.3) E(X) = np,yar(X) - npq. The characteristic function and the generating function, when it exists, of X respectively defined by

are given by

gAt) = {pe'+qy.

Probability Tools

17

Example 5.1 {The Cox and Rubinstein financial model) Let us consider a financial asset observed on n successive discrete time periods so that at the beginning of the first period, from time 0 to time 1, the asset starts from value SQ and has at the end of this period only two possible values, uS{) and dSo ( 0\) respectively with probabilities/? and q=\-p. The asset has the same type of evolution on each period and independently of the past. In period /, from time i-\ to time /, let us associate the r.v. ^.J = l,...,n defined as follows: [1, with probabihty/?, [0, with probability q. The value of the asset at the end of period n is given by the r.v. ¥„ defined as Y^=u''"d"-''" (5.7) with ^ „ = ^ , + - + ^„. (5.8) It is clear that the r.v. X„ has a binomial distribution of parameters (n,p) and consequently, we get the distribution probability of Y„: P(Y„^u^d"-^S,) = \ \p'q"-',i^O,...,n.

(5.9)

This distribution is currently used in the financial model of Cox, Ross and Rubinstein (1979).

5.2 The Poisson Distribution IfXis a r.v. with values in N so that the probability distribution is given by P(X = i) = e-'^J

= 0,\,.,.

(5.10)

where /I is a strictly positive constant, X is called a Poisson variable of parameter X. This is one of the most important distributions for all applications. For example if we consider an insurance company looking at the total number of claims in one year, this variable often may be considered as a Poisson variable. Basic parameters of this Poisson distribution are given here: E{X) = X, var(X) = A, A remarkable result is that the Poisson distribution is the limit of a binomial distribution of parameters (n,p) if n tends to +oo and/7 to 0 so that np converges to A. The Poisson distribution is often used for the occurrence of rare events. For example if an insurance company wants to hedge the hurricane risk in the States and if we know that the mean number of hurricanes per year is 3, the adjustment

jg

Chapter 1

of the r.v. X defined as the number of hurricanes per year with a Poisson distribution of parameter /I = 3 gives the following results: P(X=0)-0.0498, P(X-1)=0.1494, P(X-2)-0.2240, P(Z=3)-0.2240, P(X=4)-0.1680, P(X=5)=0.1008,P(X=6)=0.0504, P(X>6)=0.0336. So the probability that the company has to hedge two or three hurricanes per year is 0.4480.

5.3 The Normal (or Laplace Gauss) Distribution The real r.v. X has a normal (or Laplace Gauss) distribution of parameters {/u,(j^),/u G R,cr^ > 0 , if its density function is given by {x-/uY 1 ,XG. AW = yfln From now on, we will use the notation X -< N(jU,a^). The main parameters of this distribution are E(X) = ju, yar(X) = c7\ CT t

(p^(t) = Qxp i/ut - -

>n-2/2^

, g^(0 = exp /// + -

(5.12)

(5.13)

If // = 0, cr^ = 1, the distribution of X is called a reduced or standard normal distribution. In fact, if Xhas a normal distribution {iu,(j^),ju G i?,cr^ > 0, then the so-called reduced r.v. 7 defined by X-u Y= ^ (5.14) (J

has a standard normal distribution, thus from (5.13) with mean 0 and variance 1. Let O be the distribution function of the standard normal distribution; it is possible to express the distribution function of any normal r.v. X of parameters {ju,(7^),jU G R,cr^ > 0 as follows: X - ju X- /u FAx) = P{X<x) = P O (5.15) G ~ a ) Also from the numerical point of view, it suffices to know numerical values for the standard distribution. From relation (5.15), we also deduce that

AW=-^

(5.16) o- J

where of course from (5.12)

Probability Tools

(D'(x) = - ^ e " 2

19

(5 17)

From the definition of O , we have ^ (t)(x) = - ;1^ ^r \e -^dy,x&R

(5.18)

V2;ri and so 0 , (5.19) and consequently, for X normally distributed with parameters (0,1), we get P ( | ^ | < ;c) = (D(x) - cD(-x) = 2(D(x) - 1 , x > 0. (5.20) In particular, let us mention the following numerical results: p [ | X - w | < - c r ] = 0.4972(«50%), P{\X - wl < cr) - 0.6826(» 68%),

(5 2i)

Pi\X -m\< 2cr) = 0.9544(« 95%), ^ ( 1 ^ -m\< 3cr) = 0.9974(« 99%). Remark 5.1: Numerical computation of the d.f. O For the applications in finance, for example the Black Scholes (1973) model for option pricing, we will need the following numerical approximation method for computing O with seven exact decimals instead of the four given by the standard statistical tables. \)x>0

:

1 -^ C > ( x ) « l — 7 = e ^ (b^c + --- + b^c^), 1 1 + OX

^ /7 = 0,2316419,6,-0,319381530,

(5.22) ^ ^

b^ =-0,356563782,^3 =1,781477937, Z), =-1,821255978,^5 =1,330274429, 2)x<0 : (D(x) = l-cD(-x). The normal distribution is one of the most often used distributions, by virtue of the Central Limit Theorem which says that if (X„,«>1) is a sequence of

2Q

Chapter 1

independent identically distributed (in short i.i.d.) r.v. with mean m and variance <j^, then the sequence of r.v. defined by ^ ^ ^

(5.23)

(j\]n

with S^=X,+-" + X^, n>0 (5.24) converges in law to a standard normal distribution. This means that the sequence of the distribution functions of the variables defined by (5.21) converges to O . This theorem was used by the Nobel Prize winner H. Markowitz (1959) to justify that the return of a diversified portfolio of assets has a normal distribution. As a particular case of the Central limit Theorem, let us mention the de Moivre 's theorem starting with f 1, with prob. p , X^=\ ^ ^ (5.25) [0, with prob. 1-/?, so that, for each n, the r.v. defined by relation (5.22) has a binomial distribution with parameters {n,p). By applying now the Central Limit Theorem, w e get the following result: S,-np

la^

>iV(0,l), ^np(\-p) called de Moivre's result.

(5.26)

5.4 The Log-Normal Distribution If the normal distribution is the most frequently used, it is nevertheless true that it could not be used for example to model the time evolution of a financial asset like a share or a bond, as the minimal value of these assets is 0 and so the support of their d.f is the half-real line [0,+oo). One possible solution is to consider the truncated normal distribution to be defined by in setting all the probability mass of the normal distribution on the negative half-real line on the positive one, but then all the interesting properties of the normal distribution are lost. Also, in order to have a better approach to some financial market data, w e have to introduce the log-normal distribution. The real non-negative random variable X has a lognormal distribution of parameters ju,cr - and w e will write X < LN(ju,cr) - if the r.v. l o g Z h a s a normal distribution with parameters Consequently, the density function of X i s given by

ju,a^.

21

Probability Tools 0,x<0,

/vW =

1

{hgx-fif

,x>0. (5.27)

Indeed, we can write P{X <x)^ P{logX
-O

(5.28) logx-//^

(5.29)

and after the change of variable /=logx:, we get the relation (5.27). Let us remark that the relation (5.29) is the most useflil one for the computation of the d.f of X with the help of the normal d.f For the density function, we can also write 1 log X - //

AW =

(D

(5.30)

GX

The basic parameters of this distribution are given by

^(x)=/'^ var(X) = e'^*"'

K-.),

(5.31)

E[ry. Let us say that the lognormal distribution has no generating function and that the characteristic function has no expHcit form. When o'< 0.3, some authors recommend a normal approximation with parameters (//,cr^). The normal distribution is stable under the addition of independent random variables; this property means that the sum of n independent normal r.v. is still normal. That is no longer the case with the lognormal distribution which is stable under multiplication, which means that for two independent lognormal r.v. X\,X2, we have X, < LN{iu,,c7^\i = \,2:^X,xX^<

LNl^ju,ju^,^cTf + CJ2' ) ,

(5.32)

5,5 The Negative Exponential Distribution The non-negative r.v. X has a negative exponential distribution (or simply exponential distribution) of parameter A if its density function is given by f^(x) = Ae-'\x>0, (5.33)

22

Chapter 1

where >^ is a strictly positive real number. By integration, we get the explicit form of the exponential distribution function (5.34) F^(x) = l-e-^\x>0. Of course, Fx is null for negative values of ;r. The basic parameters are E(X) = —, varZ ^,yavX=^-. = —r 1

1 -, gx(0= -.t
(5.35)

(Px(0 =

5.6 The Multidimensional Normal Distribution Let us consider an n-dimensional real r.v. X represented as a column vector of its n componentsX = (X,,...,X^)'. Its d.f is given by: (5.36) F,(x,,...,xJ = P(X,<x„..„X„<xJ. If the density function of A" exists, the relations between the d.f and the density function are: d"F dxy,.dx„ Fxixx.'".\)=

(5.37)

J... i/v(^p...,^J^^p...,^^„. —00

—00

For the principal parameters we will use the following notation: E{X^) = ILi^,k = \,...,n, E{{X,-fi,)f=c7l,k

= \,...,n,

E{{X,-^i,){X,-^i,))

(5.38) 'kl

\ , /C, / — I , . . . , fl,

\ ^k^lj ^E{(X,-M,f)E((X,-M,)y The parameters CTI^I are called the covariances between the r.v. Xk and Xi and the parameters Pj^^, the correlation coefficients between the r.v. Xk and Xi.

It is well known that the correlation coefficient /7^^ measures a certain linear dependence between the two r.v. Xk and Xi. More precisely, if it is equal to 0, there is no such dependence and the two variables are called uncorrelated; for the values +1 and -1 this dependence is certain.

Probability Tools

23

With matrix notation, the following

nxnmatrix

2x=h]

(5-39)

is called the variance -covariance matrix ofX. The characteristic function of Xis defined as: ^^(?„...,/„) = £'(e'<"^'^-"^"'')(=£(."'^)).

(5.40)

Let |ui,L be respectively an /7-dimensional real vector and an ny^n positive definite matrix. The ^-dimensional real r.v. X has a non-degenerated ndimensional normal distribution with parameters fi, E if its density function is given by: 1

--(X-H)T~'(X-H)

A(x) =

e 2

,xeM\

(5.41)

(2;r)2VdetL Then, it can be shown by integration that parameters |ii, 2 are indeed respectively the mean vector and the variance-covariance matrix of X As usual, we will use the notation: X -< A^„(fi,L). The characteristic function of Xis given by: ;>'t--t'z:t

(p,{t) = e 2 . (5.42) The main fundamental properties of the ^-dimensional normal distribution are: -every subset of A: r.v. of the set {Xi,...,Xn} has also a A:-dimensional distribution which is also normal; -the multi-dimensional normal distribution is stable under linear transformations ofX; -the multi-dimensional normal distribution is stable for addition of random variables, which means that if X^ ^ A^^(fi^,2^),A: = l,...,w and if these m random vectors are independent, then (5.43) X, +- + X^ -< N„iii, + - + Ji„,i:, + - + £ J . Particular case: the two-dimensional normal distribution In this case, we have: (7^ cr, 1 E->

P

(5.44)

J_

,detL = (Tj(T2 7 l '

2

2

From the first main fundamental properties of the ^-dimensional normal distribution given above, we have:

24

Chapter 1

X,
= \,2,

(5.45)

For the special degenerated case of Ipl = 1, it can be proved that: p = \:

X^-H^

_X,-//, (5.46)

.1: ^2->"2^

^i-/"i

relations meaning that in this case, all the probability mass in the plan lies on a straight line so the two random variables X\^2 are perfectly dependent with probability 1. To finish this section, let us recall the well-known property saying that two independent r.v. are uncorrected but the converse is not true except for the normal distribution.

6 CONDITIONING DEPENDENCE)

(FROM

INDEPENDENCE TO

6.1 Conditioning: Introductory Case Let us begin to recall briefly the concept oi^ conditional probability. Let (Q,3,P) be a probability space and let ^, 5 be elements of 3 and look at the number of occurrences of the event A whenever B has already been observed in a sequence of 17 trials of our experiment. We shall call this number/7(^|5). In terms of frequency of events defined by relation (2.5), we have:

n(A\BU!MlB.,

(6.1)

provided that n{B) is different from 0. Dividing by n the two members of relation (6.1), we get: njA^B) n{A\B) n{B) n n In terms of frequencies, we get:

fU\BUlidm, MI; ^(£,

(6.3)

From the experimental interpretation of the concept of probability of an event seen in section 2, we can now define the conditional probability of A given B as:

Probability Tools

25

P(A\B) = ^^f^,PiB)>^.

(6.4)

If the events A and B are independent, from relation (4.39), we get: P(A\B)

= P{A),

(6.5)

a relation meaning that, in case of independence, the conditional probability of set A does not depend on the given set B, As the independence of sets A and B is equivalent to the independence of sets A and 5^, we also have: P{A\B'\

= P{A),

(6.6)

The notion of conditional probability is very useful for computing probabilities of a product of dependent events A and B not satisfying relation (4.39). .Indeed from relations (6.4) and (6.6), we can write: P{A {^B) = P{A)P[A\B)

= P{B)P(B\A)

.

More generally, for n events A\,..,Am we get compound probability":

^|hA|=^(4)^(4M.)-^(^„M.n^...nA-,)>

(6.7)

the so-called "theorem of

(6.8)

a relation expanding relation (6.7). P\C\AA

= P{A,),,,P{A^)

(6.9)

is true in the case the of independence of the n considered events. If the event B is fixed and of strictly positive probability, relation (6.4) gives the way to define a new probability measure on (Q, 3) denoted P^ as follows: P ( ^ ) = J^(iIL?),V^G5.

(6.10)

PB is in fact a probability measure as it is easy to verify that it satisfies conditions (2.10) and (2.11) and so P^is called the conditional probability measure given B. The integral with respect to this measure is called the conditional expectation EB relative to PB> From relation (6.10) and since PB{B)=^\, we thus obtain for any integrable r.v. Y\ E,(Y)=\Y{co)dP, n

=-L^JY(co)dP,

(6.11)

P\B) Q

For our next step, we shall now consider a countable event partition {B^,n>\) of the sample space Q. That is: Q = Q 5 „ 5 , n 5 , = 0 , V/,7:/^7. Then, for every event A, we have:

(6.12)

2^

Chapter 1

P(A) = Y.P(B„f]A)

(6.13)

and by relation (6.10): PiA) = XPiB„)P(A\B„).

(6.14)

n>\

Now, for any integrable r.v. Y, we can write: E(Y) = j;^JY(co)dP

(6.15)

and from relation (6.11): E{Y) = Y,P{B„)E,^{Y).

(6.16)

As the partition B^,n>\ generates a sub-a -algebra of 3 denoted 3, obtained as the minimal sub- a -algebra containing all the events of the given partition, we can write the relation (6.16) as E^ (7) called the conditional expectation of Y given 3 , . It is very important to understand that this conditional expectation is a function of CO and so a new random variable as: E^^(Y){co) = YEsSy)\{co), (6.17) n>\

So, the random variable E^ (7) assumes on each set B the value of E^ (7) that is constant and defined by relation (6.11) with B=Bn. Now, let us compute the expectation of this new random variable E^ (Y); from relation (6.17), we can deduce that: E(E^^ (Y)(CO)) - E\Y,ES„

(y)\

(CO) I

^Y,E(E,^(Y)\,^(CO)), (6.18)

= Y,E,^(Y)E(\(m)), n>\

^^E,^iY)P(B„), and finally from relation (6.16), we get: E(E^^(Y)(co)) = E(Y).

(6.19)

Furthermore, since for any set B belonging to 3 , , B is the union of a certain number of events J?„, finite or at the most denumerable, we obtain by integrating both members of relation (6.17):

Probability Tools

27

1^3, {Y){co)dP =X K (^)^«. (^)^^' ==Z^.„(^)jl«„(^)^^'

(6.20)

Using now relation (6.11), we get:

K ( r ) ( . , . P . [ i : ^ ^ J >•(.>,.]p(Br\B„), 5n5„

=S

J YicoW,

(6.21)

= JY((o)dP. In conclusion, we get: JE^^ {Y){co)dP = JY(co)dP,B e 3,. B

(6.22)

B

Of course, for5 = Q, this last relation is identical to (6.19). We shall focus our attention on the meaning of result (6.22) which equates two integrals on every set B belonging to 3i but presenting an essential difference: in the left member, the integrand E^ (Y) is 3, -measurable but in the right member, the integrand 7 is 3 -measurable and so not necessarily 3, -measurable since 3, cz3. Furthermore, the function E^ (Y) is a.s. unique; indeed, let us suppose that there exists another function 3j -measurable/so that relation (6.22) is still true. Consequently, we have: jf(co)dP = JY(o))dP,B e 3,. (6.23) B

B

From relations (6.22) and (6.23), we obtain: \f{co)dP - j^3, {co)dP,B e 3, B

(6.24)

B

SO that: ff((u)dP = j(f(co) - E^^ (co))dP, 5 e 3 , .

(6.25)

23

Chapter 1

As this relation holds for all B belonging to 3 , , it follows that E^ (Y) = / , a . s . ; otherwise, there would exist a set B belonging to 3, so that the difference E^ (7) - / would be different from 0 and so also the integral \{f{co)-E^^{co))dP

(6.26)

B

in contradiction with property (6.25).

6.2 Conditioning: General Case We can now extend the definition (6.17) to arbitrary sub-cr-algebras using property (6.22) as a definition with the help of the Radon Nikodym theorem, Halmos (1974). Definition 6.1 If 5, is a sub-a-algebra of 3 , the conditional expectation of the integrable r,v. Y given 3j, denoted by E^ (Y) or E(Y\'3A,

is any one r.v. of the

equivalence class such that: (i) E^ (Y) is 3, -measurable, (ii)

1^3^ {Y){co)dP = \Y{co)dP,B e 3,. B

(6.27)

B

In fact, the class of equivalence contains all the random variables a.s. equally satisfying relation (6.27). Remark 6.1 Taking B = Q in relation (6.27), we get: E(E^Y)) = E(Y),

(6.28)

a relation extending relation (6.17) to the general case. Particular cases (i) 3 , is generated by one r,v. X. This case means that 3, is the sub- cr -algebra of 3 generated by all the inverse images of X and we will use as notation: E,^(Y) = E(Y\X), (6.29) and this conditional expectation is called the conditional expectation of Y given X, (ii) 3, is generated by n r.v. X^,...,X^. This case means that 3, is the sub- cr -algebra of 3 generated by all the inverse images of X^,...,X^ and we will use as notation:

Probability Tools

29

E^^{Y) = E{Y\X„...,X„),

(6.30)

and this conditional expectation is called the conditional expectation of Y given In this latter case, it can be shown (Loeve (1977)) that there exists a version q){X^,..,,X^) of the conditional expectation so that ^ is a Borel function from R" to E and as such it follows that E{Y\X^,,..,X^)

is constant on each set

belonging to 3j for which X^{co) = X^,...,X^(CD) = x^, for instance. This justifies the abuse of notation E(Y\X,((o)^x„...,X„(co) = x„) = ,p{x„...,x„)

(6.31)

representing the value of this conditional expectation on all the co's belonging to the set [(D:X^(CD) = X^,...,X^(CO) = X^] .

Taking B = Q in relation (6.28), we get: E(Y)= j ^ ( 7 | X , ( ^ ) = x„...,Z,(^) = x„)^P(X,(^)<x„...,X,(^)<xJ, (6.32) a result often used in the sequel to evaluate the mean of a random variable using its conditional expectation with respect to some given event. (iii) If 5 i = { 0 , Q } , we get £(7|3,) = £(7)and if 3,={0,B,B\Q] , then on Band E(Y\:5,) = E(Y\B") on B', (iv) Taking as r.v. 7 the indicator of the event A, that is to say: (hcoe A, [0,co^A, the conditional expectation becomes the conditional probability of A given 3, denoted as follows: E(Y\:5,)

= E(Y\B)

P(A\3,) = E(l,(cop,) and then relation (6.27) becomes: jp{Ap,(co))dP = P(Af]BlBe:5,.

(6.34) (6.35)

B

Letting 5 = Q in this final relation, we get: = P{Al E(p(Ap,))

(6.36)

a property extending the theorem of total probability (6.14). If moreover, A is independent of 3 , , that is to say, if for all B belonging to 3 , : P{Ai^B) = P{A)P{B), (6.37) then we see from relation (6.34) that: (6.38) P[A\S,){o)) = P{A\coea.

30

Chapter 1

Similarly, if the r.v. Y is independent of 3 , , that is to say if for each event B belonging to 3, and each set A belonging to the cr-algebra generated by the inverse images of Y, denoted by
= E(Y)P(B),

(6.40)

= JE(Y)dP, B

and so, relation (6.39) is proved. In particular, if 3, is generated by the r.v. X\,...,Xn, then the independence between 7 and 3, implies, that: E(Y\X,,„.,X^)

= E{Y),

(6.41)

Relations (6.39) and (6.41) allow us to have a better understanding of the intuitive meaning of conditioning. Under independence assumptions, conditioning has absolutely no impact, for example, on the expectation or the probability, and on the contrary dependence implies that the results with or without conditioning will be different, this fact meaning that we can interpret conditioning as given additional information useful to get more precise results in the case of dependence of course. The properties of expectation, quoted in section 4, are also properties of conditional expectation, true now a.s., but there are supplementary properties which are very important in stochastic modelling. They are given in the next proposition. Proposition 6.1 (Supplementary properties of conditional expectation) On the probability space (Q,3,P), we have the following properties: (i) If the r.v. Xis 5, -measurable, then E{Xp,)

= X,a.s..

(6.42)

(ii) LetXbe a r.v. and Y 3, -measurable, then E{XYp,)

= YE{Xp,\a.s..

(6.43)

This property means in fact that 3, -measurable random variables are like constants for the classical expectation.

Probability Tools (iii)

31

Since from relation {6.27), we have that E^(Y) = 7, taking Y = E^ (7), we

see that: E,(E,^(Y)) = E,^(Y)

(6.44)

and of course since: E,^(E,(Y)) = E,^(Y),

(6.45)

putting these last two relations together, we get: E, (£,_ (7)) - £3, E^ ((7)) = E,^ (7).

(6.46)

This last result may be generalised as follows. Proposition 6.2 (Smoothing property of conditional expectation) Let 3j, 32 be two sub- a -algebras of 3 such that 3j c 32' ^^^^ ^^ ^^ ^^^^ ^^^^' E^^ (£3, (7)) = ^3, (E^^ (7)) = ^3, (7), (6.47) a property called the smoothing property in Loeve {1977), Proof To prove it, it suffices to start from the probability space (0,32,^3 ) where P^ is the restriction of the initial probability measure P to 32. This simply means that we only fix our attention on the events belonging to 32 and of course that for any such event A, we define its probability P^ as: P.,^{A) = P{A), Applying now result (6.46) with 3 = 3 2 , we get result (6.47). A particular case of relation (6.47) is for example that: E[E{Y\X„...,X„)\x) = E(E{Y\X,)%,...,X„yE[Y%).

(6.48) n (6.49)

This type of property is very useful for computing probabilities using conditioning and will often be used in the following chapters. Here is an example illustrating this interest for sums of a random number of random variables with the so-called Wald identities, Example 6.1 {Wald's identities) Let {X^, ^ > 1) be a sequence of i.i.d. real random variables and IS! a non-negative r.v. with integer values independent of the given sequence. The random variable defined by:

S,=Y.X„

(6.50)

n=\

is called a sum of a random number of random variables and the problem to be solved is the computation of the mean and the variance of this sum supposing that the r.v. X^ have a variance.

Chapter 1

32

From relation (6.28), we have that:

E(S,) = E(E(S,\N))

(6.51)

and as, from the independence assumptions: E(S,\N)^NEiX),

(6.52)

we also have: E(S,) = E(NE(X)l and finally the so-called the first Wald's identity. For the variance of IS'A^ , as ynx{S,)^E{Sl)-{E{S,))\

(6.53)

(6.54)

it suffices to evaluate E(Sl) as we did above. From relation (6.28), we can write that:

E(SI)=E(E(SI\N)).

(6.55)

As

E(SI\N)=E\

f^

\N

(6.56)

we obtain on the set [co: N{co) = n):

E{sl\N = n) = E(Y,X, = var

Z^,- +k Z^. V V '=1

v/=i

\\

(6.57)

vy

= nvar(X) + (nE(X))\ Therefore, from relation (6.55): E(Sl) = E(N var(X) + N' {E{Xf ),

(6.58) - E{N) var(X) + E{N^ ){E{Xy), and thus, by relations (6.54) and the first Wald's identity (6.53), we get: var(5^ ) = E{N) var(X) + E{N' ){E{X^)) - {E{X)f {E{N)f (6.59) and finally we obtain the second Wald's identity under the form: var(5^) = E{N)wax{X) + \w{N){E{X)f . (6.60)

6.3 Regular Conditional Probability The general definition (6.27) gives the conditional expectation of a r.v. Y given 3, by an implicit relation. Now the question is: can we define the conditional expectation with an explicit relation?

Probability Tools

33

To give the answer, let us begin with the conditional probability defined by relation (6.34); We know that the conditional probability P ( ^ | 3 i ) i s a r.v., not unique but a.s., unique and this impHes that Pf ^ | 3 , j as a set function on 3 has a.s. the properties of probability measures: (i) P(Qp,)(co) = l, (ii) (ill)

>0VAe3,

P{A\^,)(CD)

K^i

J

Zt

(6.61) (6.62)

^

'

'

(6.63)

It is important to note here that the null events A^pA^2'^3' ^n which respectively these last three properties are not true, are generally not identical, so that for each CO, the random set function P(.|3,)((^)from 3 to [0,l] is not necessarily a probability measure since, to be so, these three sets must be identical. That is why we must introduce the concept of regular conditional probability (see Loeve (1977) or Gikhman and Skorokhod (1980)). Definition 6.2 The conditional probability PU3A(cD)is a regular conditional probability if there exists a function p(.,.) from 3 x Q to [0,l] so that: (i) for almost all co of Cl, p(.,co), as a set function on 3 , is a probability measure, (ii) for every fixed event A belonging to 3 , p(A,.) is "5-measurable and is a version of the given conditional probability, that is a.s., we have: (6.64) p{A,co) = P[Ap){co). The interest of such regular conditional probabilities is that we can express the related conditional expectation of an integrable r.v. X a.s. as an integral with respect to the measure j!?(.,(2;): E[Xp,){(D)=

\x{co')p{dco\co).

(6.65)

In many applications, it is sufficient to restrict the attention to all events of the sub- (J -algebra generated by a r.v. X, with values in the measurable space {E^ii/), and denoted by (J{X) . This means that we are only interested in the following conditional probabilities: P[Ap,),AscT{X). (6.66)

34

Chapter 1

If the conditional probability given 3i is regular, we can then define the function Cfrom o-(X)xQ to [0,l] as CiA,co) = P(A\3,)icD),Aea(X),cDeQ

(6.67)

satisfying (i) for almost all CD of Q, C(.,co), as a set function on or(X), is a probabihty measure, (ii) for every fixed events belonging to cr(X), C(A,.) is 3 -measurable, (iii) for every event A belonging to (j(X) and for every event B belonging to 3 , we have: \C(A, co)P{dco) = P{A n B). (6.68) B

C is called the conditional distribution of X given 3j and the mixed conditional distribution ofXgiven 5,is defined as the function Q{.,.) from defined by: Q{S,(D) = p[{co': X(co') e 5|3, ))(^),5 e y/.

y/xQ. Xo [0,l] (6.69)

The problem of the existence of regular conditional probability was solved by Loeve (1977) or Gikhman and Skorokhod (1980). For our goal, let us just say that this is the case if 3 is generated by a finite or countable family of random variables or if the space £" is a complete separable metric space. In the particular case of an /7-dimensional real r.v. X=(Xi, ...,X„),we can now introduce the very useful definition of the conditional distribution function ofX given 3j defined as follows:

= e({^'LY,(^')<x„...,X,(^')<xJ,^). Another useful definition concerns an extension of the concept of the independence of random variables to the definition of conditional independence of the n variables X^,,..,X^. For all (xi,...,x„) belonging toR"", we have the following identity: n

where of course we have: F^{x,M-P(X,<x,p,)

(6.72)

according to the definition (6.70) with n=\. Example 6.2 On the probability space(Q,3,P), let {X,Y)hQ di two-dimensional real r.v. whose d.f is given by:

Probability Tools

35

F{x,y) = P{X<xJ
(6.73)

As R^ is a complete separable metric space, it is said above that there exist regular conditional probabilities given the sub- a -algebras (J{X) or cr(7) and so the related conditional d.f. denoted by: = x]) (6.74) F,\y[4co:Y^y]\F,^,[y\{co:X also exists. If moreover, the d.f. F has a density / we can also introduce the concept of conditional density for both functions F^ij,, Fys^.F^, giving at the same time an intuitive interpretation of conditioning in this special case. We know that for every fixed {x,y)\ f{x,y)^xl^y + o{x,y,lSx,^y) = P{x <X <x-\-lsx,y 0 for (Ax:, Ay)-> (0,0) and similarly for the marginal density function of X: (6.76) / ^ (x)Ac + o(x, Ax) = P{x<X<x + lSx\ where o(x,Ax:)->0 forAx:->0 with of course: fxi^)-\fix.y)dy.

(6.77)

R

Using formula (6.4), we thus obtain: P{y
(6.78)

\imP[y
(6.79)

+ ^y\x<X<x^^^x)

= ^^^^^y, Jx v-^)

^

This relation shows that the function fy^^ defined by:

is the conditional density of Y, given X, Similarly the conditional density of X, given 7 is given by:

Consequently, for any Borel subsets A and B of R, we have: P(X eA\Y(co) = y)^ J/^,, (x\y)dx = - i A

P{{X,Y)^A[\B)^

J f{x,y)dxdyA Ar\B

lf(x,y)dx,

JY\y) A

J / , | , ( x W ^ x fY^y)dy, v^ J

(6.82)

36

Chapter 1

The last equalities show that the density of {X, Y) can also be characterised by one marginal d.f. and the associated conditional density, as from relations (6.79) and (6.81): f = fx^fy\x=fy^fx\y' (6.83) It is possible that conditional means exist; if so they are given by the following relations: E(x\Y = y)=\f(x\y)dx, E(Y\X = xy \f(y\x)dy , (6.84) The conditional mean of X (resp.Y) given Y^y (resp. X=x)can be seen as a function of the real variable j^ (resp. x) called the regression curve of X {resp.Y) given Y(resp. X). The two regression curves will generally not coincide and not be straight lines except if the two r.v. Xemd 7 are independent because, in this case, we have from relations (6.79) and (6.81) that: (6.85) fx\Y = fx^frlx ^ fy and so: E(X\Y) = E(X), E[Y\X) = E(Y), (6.86) proving thus that the two regression curves are straight lines parallel to the axes passing through the "centre of gravity" (E(X), E(Y)) of the probability mass inR^ In the special case of a non-degenerated normal distribution for (X, Y) with vector mean (m\,m2) and variance covariance matrix: (6.87) it can be shown that the two conditional distributions are also normal with parameters: f Y\X
7 STOCHASTIC PROCESSES In this section, we shall always consider a complete probabiUty space with ?ifiltrationF,

(Q,3,P)

Probability Tools

37

Let us recall that a probability space (Q,3,P) is complete if every subsets of an event of probability 0 is measurable, i.e. in the cr-algebra 3 , and so also of probability 0. Definition 7.1 F being a filtration on the considered basic probability space means that F is a family (^3^,t ST) of sub- a -algebras of 3, the index set T being either the natural set {0,1,...,«,...} or the positive half real /me[0,oo), such that: (i) ^3^ c 3 ^ , u>t

(iii) 3o contains all subsets with probability 0. Assumption (ii) is called the right continuity property of the filtration F. Any filtration satisfying these three assumptions is called a filtration satisfying the usual assumptions. The concept of filtration can be interpreted as a family of amounts of information so that 3^ gives all the observable events at time t. Definition 7.2 The quadruplet (( Q , 3 , P , ( 3 ^ , / G T ) )

is called a filtered

probability space. Definition 7.3 A random variable r : Q t-> T is a stopping time if: yteT: {co:T(co)
(7.2)

The interpretation is the following: the available information at time / gives the possibility to observe the event given in (7.2) and to decide for example if one stops the future observations after time /, or not. We have the following proposition: Proposition 7.1 The random variable r is a stopping time if and only if {co:T(Q))0: {co\T{co)
C^'^)

t + E>U>t

n2.=3„

(7.5)

3g

Chapter 1

and so by property (7.1), (ii) applied above, r is indeed a stopping time. 2) Let us suppose that r is a stopping time. We have: [co:T{CO)£>0

Since: {co:T(co)
(7.7) (7.8)

Definition 7.4 A stochastic process (or simply process) with values in the measurable space {E,i^) is a family of random variables: {X„teT} where for all t: X^:Q\-^ E, (3,X) - measurable.

(7.9)

This means, in particular, that for every subsets B of the a -algebra K, the set X;'{B) = {cD:XXco)eB} (7.10) belongs to the a -algebra 3 . Remark 7.1 If(£',X) = (M,y5), the process is called a real stochastic process with values i n R ; if(£,K) = (M",y5"), it is called a real multidimensional process with values in R''. If T is the natural set{0,l,...,^,...}, the process X is called a discrete time stochastic process or a random sequence; if T is the positive half real line [0,oo), the process X is called a continuous time stochastic process. Definition 7.5 The stochastic process X is adapted to the filtration F if for all t, the r.v. Xt is 3^ -measurable. This means that, for all t ^T:

x;'{B) = {(D\XXco)eB]e:5^yB^^.

(7.ii)

Definition 7.6 Two processes X and Y are indistinguishable ifa.s.,for all t GT : X,=Y,. (7.12) This means that: ?{X,^Y„\ft^T)^\.

(7.13)

Probability Tools

39

Definition 7.7 The process X (or Y) is a modification of the process Y (or X) if a.s.,for all t eT: X^=Y^, as. (7.14) This means that: P{X^=Y^.yteT) for alU e r .

= l.

(7.15)

Definition 7.8 For every stochastic process X, the function from T to E, t\-^XXco) (7.16) definedfor each coeQ, is called a trajectory or sample path of the process. It must be clearly understood that the so-called "modem" study of stochastic processes is concerned with the study of the properties of these trajectories. For example, we can affirm that if two processes X and Y are indistinguishable, then there exists a set A/^ belonging to 3 of probability 0 such that: '^co^N:XXco) = Y^(cD)yteT. (7.17) In other words, for each co element of the set Cl-N, the two functions t h^ X^ (co) and / h-> }^ (o)) are equal. Let us remark that from Property 7.1 (iii), the neglected set //belongs to 3^, for all teT. Definition 7.9 A real stochastic process X is cadlag (continu a droite, limite a gauche) if a.s. the trajectories of X are right continuous and have left limits at every point t. Definition 7.10 IfX is a real stochastic process and a set A G ;5, then the r.v. defined by: T{co) = inf {/ > 0: X^co) e A] (7.18) is called the hitting time of A by the process X. It is easily shown that the properties of stopping and hitting times are (see Protter(1990)): (i) If X is cadlag, adapted and A e / ? , then the hitting time related to A is a stopping time. (ii) Let S and Tbe two stopping times, then the following r.v.: SAT{=mm{Sj}), SvT(=rmix{S,T]), S+ T, aS(a>l) (7.19) are also stopping times.

4Q

Chapter 1

Definition 7.11 If T is a stopping time, the a-algebra 5^. defined by: (7.20)

^^={Ke^:K^{co\T{co)Q] is called the stopping time a -algebra.

In fact, the a -algebra 3^^ represents the information of all observable sets up to the stopping time T. W e can also say that S^. is the smallest stopping time containing all the events related to the r.v. Xj^(^^^(co) for all the adapted cadlag processes X or generated by these r.v. W e also have for two stopping times S and T: (i) S
C7-21)

( i i ) 3 , n 3 , , = 3,,,.

(7.22)

8 MARTINGALES In this paragraph, we shall briefly present some topics related to the most wellknown category of stochastic processes called martingales. Let Jf be a real stochastic process defined on the filtered complete probability space ( Q , 5 , P , ( 3 , , / e r ) ) . Definition 8.1 The process X is called a martingale if: (i)Vt>0,3E{X,),

(8.1)

= X^,a.s.

(8.2)

(n)sE{X,\X^)

The latter equality is called the martingale property or the martingale Definition 8.2 The process X is called a super-martingale if: (i) V/>0,3£(X,), (ii)5^(XJ5j<(>)X„a.s.

(resp.

equality. sub-martingale) (8.3) (8.4)

The martingale concept is interesting; indeed, as the best estimator at time s (s>t) for the value of X^, as given by the conditional expectation appearing in relation (8.2), the martingale equality means that the best predicted value simply is the observed value of the process at the time of predicting s. The use of martingale in finance is frequently (see Janssen and Skiadas (1995)) to model the concept of an efficient financial market.

Probability Tools

41

Definition 8.3 The martingale X is closed if: 37 : (i)E(|7|)
(8.5)

(ii)Vr e [0,oo): E ( 7 | 3 J = X,, a.s.. It is possible to prove the following result (see for example Protter (1990)). Proposition 8.1 (i) IfXis a supermartingale, then the function ^ h-> J?(X^) is right continuous iff there exists a unique modification Y ofX such that Y is cddldg. (ii) IfXis a martingale then, up to a modification, the function ^h->£'(X^) is right continuous. It follows that every martingale such that the function t\-^E[X^) is right continuous is cadlag. The two most important results about martingales are the martingale convergence theorem and the optional sampling (or Doob's) theorem. Before giving these results, we still need a final technical definition. Definition 8.4 {Meyer (1966)) A family (^^,u G A \ where A is an infinite index set, is uniformly integrable if limsup "

j

\^^(cD)\d?(co) = 0.

(8.6)

{co%(co)\>n}

Proposition 8.2 Let X be a supermartingale in such a way that the function t (-^ E{^X^ ) is right continuous such that: sup£(|xJ)
(8.7)

then, there exists a r.v. Y such that:

^HA).

(8.8,

( i i ) 7 - l i m Z , a.s. Moreover, ifXis a martingale closed by the r.v. Z, then the r.v. Y also closes X and: Y^E(Zp^), (8.9) where

42

Chapter 1

(8.10)

3^=0V0
j

With the aid of the concept of uniform integrabihty, we can obtain the following corollary.

Corollary 8.1 (i) Let X be a right continuous martingale and uniformly integrable; then the following limit: r = limX (8.11) exists a.s.; moreover Y eL^ and the r.v. Y closes the martingale X. (ii) Let X be a right continuous martingale; then X = {X^J > 0) is uniformly integrable if and only if (8.12) Y = \\mX, /->oo

exists a.s., Y eL\ and (-^/ >^ ^ [0,oojj /^ ^ martingale with, a.s.: ^.=Y, (8.13) Now, an interesting question is: what happens if we observe a martingale X at two stopping times S,T (S
Probability Tools

withtAT

43

= mf{t,T].

From this definition, it follows that if the process X is adapted and cadlag, then so is the stopped process X^. This is due to the fact that / A T is also a stopping time and moreover: This leads to the last result we want to mention. Proposition 8.4 Let X be a right continuous uniformly integrable martingale: then the stopped process ^

=(^/Ar'^^[0'^Jj has the same properties with

respect to the filtration (3^,^ ^ [^'°^])-

9 BROWNIAN MOTION There are a lot of particular stochastic processes and some of them will be extensively studied in the sequel, such as renewal processes, random walks, Markov chains, semi-Markov and Markov processes and their main extensions. However, to finish this introduction to probability theory, we want to introduce briefly the concept of Brownian motion or Brownian process. We will work on a basic filtered probability space satisfying the usual assumptions and noted fQ,3,jP,(5^,/ e [OJQ^)))Definition 9.1 The real stochastic process 5 = (j?^,/e[0,oo)) will be called a Brownian motion or Brownian or Wiener process with trend /^ and variance CF^ provided that: (i) B is adapted to the basic filtration, (ii) B has independent increments, i. e, that: = P{B,-B,sA), ^^ ^^ \/s,t{0<s
(9.2)

(iv) P{B,=x) = l, ( x e R ) .

(9.3)

If moreover, we have:

44

Chapter 1

^ = 0, a^=lx = 0, (9.4) then the Brownian motion is called standard. Let us now give the most important properties of the standard Brownian motion. Property 9.1 IfB is a Brownian motion, then there exists a modification ofB, the process 5 * such that B"^ has, a.s., continuous trajectories. Property 9.2 IfB is a standard Brownian motion, then B is a martingale. Proof It suffices to prove that B is a martingale with the so-called minimal filtration that is (cr(5p...,5j,/ G[0,OO)). Indeed, we successively have: E(B,\B„...,B,) =

E(B,-B^+B^\B„...,B^),

^E(B,-B^\B„...,B^)+E(B^\B„...,B^). Using the property of independence increments and the smoothing property of conditional expectation, we get: E[B,\B„...,B,)

= E{B,-B^)

+ B^,

and finally, as B is standard: E(B,\B„...,B^) = B^.

(9.6)

u

(9.7)

In a similar way, the following property can also be proved. Property 9.3 IfB is a standard Brownian motion, then the process Q where Q = [B^-tJ^[0,^)) (9.8) is a martingale. Remark 9.1 // can also be proved that both Properties 9.2 and 9.3 characterise a standard Brownian motion. Property 9.4 If B is a standard Brownian motion, then for almost allco, the trajectory co\-^ B^ (co) is not of bounded variation on every closed interval

[a,b]. This explains why it is necessary for models in finance and in insurance to define a new type of integral, called the ltd or stochastic integral, if we want to intregrate with respect to B (see for example Protter (1990)).

Chapter 2 RENEWAL THEORY 1 PURPOSE Let us consider the following reliability problem: at time 0, the given system starts with a new component which fails at random time 7]. At this time, a new component immediately enters the system to replace the first one and fails at time c. There is another immediate replacement by a new component inserted in the system, still of the same type, and so on. We will note (r„, /7 > 0) the successive replacement times, setting, of course: 7;=0. (1.1) The lifetimes of the successive components entering the system are given by n>\, (1.2) X„ —T„—T^ n-l5

n n-1 n-2

N(t)

Ti1 T^2

•n-1

Figure 1.1:trajectory of N{t) From an operational point of view, an important characteristic of the considered system at time t is the total number of replacements occurring on the interval [ 0 , / ] . Let us remark that, for the moment we do not take into account the initial component. If N{t) represents the random variable we have just defined, we have, for ^ > 1:

46

Chapter 2

N{t)>n-\<^T„ 0 ) , as shown in Figure 1,1. The first moment of A^(/) will give the mean number of replacements on (0,/]. In particular, if at time 0, the manager must be able to have an inventory large enough to perform all replacements, the level of the inventory will be, on average, the expectation E(N{t)). Of course, the manager must add buffer stock to prevent random evolution. This problem will be solved in section 7. The area of probability studying such processes is called Renewal Theory. It is, at least for applied probability, one of the most important topics encountered in real life problems.

2 MAIN DEFINITIONS Let {X„, « > 1) be a sequence of non-negative, independent and identically distributed random variables defined on the probability space ( Q , 3 , P ) . Definition 2.1 The random sequence {T^,n>0), where: To=0 a.s., r, = X , + . . . + Jr„ n>l, is called a renewal sequence or renewal process. The r.v. T„,n>0 are called renewal times and the r.v. X„,n>l interarrival times.

(2.1) (2.2) are called

Example 2.1 1) In the first section, we give an example in reliability theory. 2) Another important example is queueing theory. Let us consider a queueing system composed of a server, a process of customer arrivals, a process of service times and a discipline rule of the type "first in, first out" (FIFO), which means the first customer present in the system is the first served. In many models of queueing theory, the arrival process is assumed to be a renewal process. In this case, the r.v. T„ is the arrival time of the nth customer, assuming that a customer number 0 is immediately served at time 0, and the r.v. X^ represents the interarrival time between the (n-l)th and the nth customer. 3) An arrival process is also considered in risk theory. Let us consider an insurance company starting at time 0 with a capital amount u (u>0) called the Initial res&Fve. T h e clistomers p a y pfemilimgj and t h f ingurancf c o m p a n y has t o

pay for the accidents claimed by the customers. In this case, the r.v. T^ represents

Renewal Theory

47

the arrival of the ^th claim, assuming that the company just starts at the arrival of a claim called claim 0, and the r.v. X„ is the interarrival time between the («-l)th and the /7th claims {n>\). 4) In counter theory, we consider particles arriving at times r„, n>0 with TQ =0 so that here too, the r.v. X„ are interpreted in terms of interarrival time between two successive particles. Definition 2.2 With each renewal sequence, we can associate the following stochastic process, continuous in time, with values in N : {N{t\ t>0), (2.3) where N(t)>n-l<::>T„
(2.4)

provided the expectation is finite.

3 CLASSIFICATION OF RENEWAL PROCESSES Let us suppose that the random variables are defined on R with distribution function F such that, to avoid trivialities: F(0)<\. (3.1) If F(+^) = U (3.2) we have the usual case of real random variables. From relation (2.2), we get n>\, (3.3) P(7V(0 >n-l) = F^"\t\ F^"^ being the ^-fold convolution product of F with itself Since, fox n>\\ P{N{t) = n) = P{N(t) >n-l)-P{N(t) > n) , (3.4) using relation (3.3), we obtain: n>l. (3.5) P{N(t) = n) = F^"\t) - F^"^'\t), F^^^ is defined as being the Heaviside distribution with a unit mass at the origin, i.e. F''^=Uo; (3.6)

48

Chapter 2

the relation (3.5) is still vaHd for /7 = 0, since ?{N(t) = 0) = l-F(t), (3.7) Using Stein's lemma (1946), the following very important result can be proved: Proposition 3.1 If F(0) < 1, then, for all t, N(t) has moments of any order, In particular, this proposition implies that the renewal function is finite for all finite t. Consequently, we can write successively: 00

E{N(t)) = Y,n[F^"\t)- F^"^'^(t)] = F(t) - F^^\t) + 2F''^(0 - IF^'^t) + •••

(3.8)

^Fit) + F^'\t) + F^'\t) + ---, so that using relation (2.4):

i/(0-XFW(0.

(3.9)

In several cases, it is useful to consider the initial renewal and to define at time t the random variable N^t) as being the total number of renewals on [ 0 , / ] . Clearly, we have, for all / > 0 : (3.10) NXt) = N(t) + l, and consequently: (3.11) E{N\t)) = H(t) + l, Setting (3.12) R(t) = E{N\t)), we get from relations (3.11), (3.9) and (3.6): R(t) = f^F^"\t).

(3.13)

Of course, we have (3.14) R(t) = Uo(t)-^Hit). The classification of a renewal process is based on three concepts: recurrence, transience ^nd periodicity. Definition 3.1 (i) A renewal process (T^, n>l) is recurrent if X^ < oo for all n; otherwise it is called transient. (ii) A renewal process {T„, n>\) is periodic with period 5 if the possible values of the random variables X^, n>\ form the denumerable set {0,5,2S,...}, and 5 is the largest such number. Otherwise {that is, if there is no such strictly positive S ), the renewal process is aperiodic.

Renewal Theory

49

A direct consequence of this definition is the characterization of the type of a renewal process, with the help of distribution function F, Proposition 3.2 A renewal process of distribution function F is (i) recurrent iff F(co) = 1, (ii) transient iff F(oo) < 1, (iii) periodic with period 5 (^5" > 0) iff F is constant over intervals [nS, (n +1)^), ^ G N , and all its jumps occur at points nS, n GN . If / tends toward+00, relation (3.9) gives: if F(+oo) = 1, +00 H(+oo) = I if F(+oo)
il-F(+a))

Or, equivalently by relation (3.13): if F(+oo) = 1, +00 7?(+oo) = ^ if F(+oo)
The interest of the classification given above will be clearer with the concept of lifetime of a renewal process. Definition 3.2 The lifetime of a renewal process (T^,n>l) variable L, possibly defective, defined by: L = sMT„:T„<^}.

is the random (3.18)

So, if Z, = ^ , this means that there is only a finite number of renewals on [0,oo). Also, we define a new random variable A'^ giving the total number of renewals on [0,L). Definition 3.3 The total number of renewals on (0,oo), possibly infinite, is given by ^ = sup{A^(0, ?>0}. (3.19)

50

Chapter 2

In reliability theory, the event {N = k} would mean that the (A:+l)th component introduced in the system would have an infinite lifetime! Also the probability distribution of A^is given by p(A^ = 0) = l-F(+oo), (3.20) P{N = l) = F(+oo){l-F(+oo)), (3.21) and in general, for A: G N : P{N = k) = (F(+oo))' (1 -F(+oo)). (3.22) Of course if F(+oo) = 1, we have, a.s., A^ = +oo.

(3.23)

In the case of a transient renewal process, we can write, using relation (3.22): E{N) = Y,k[F(+^)f{l-F(-^^)), As the function :; l-x

(3.24)

can be written for Ixl < 1 as a power series: '

A = Z^" >

(3-25)

which is analytical on (-1,+1) and thus derivable, we have

Writing relation (3.24) under the form 00

E{N) = F(+oo){l-F(+oo)).Y,k[F(+oD)f-'

(3.27)

we get, using relation (3.26):

So, it is possible to compute the mean of the total number of renewals very easily in the transient case. We can also give the distribution function of L. Indeed, we have: P{L
= j;^P{T,
X,,,=+^).

(3.29)

A7=0

From the independence of T„ and X^+j, we may deduce that: 00

P{L < t) = 1-F(+oo) + Y,F^"\t){l-F(+oo)).

(3.30)

11=1

And finally, by equality (3.13): P{L
(3.31)

Renewal Theory

51

To compute the mean of the Hfetime of the process, we use the following trick based on the independence of the random variablesX„, n>\; we can successively write: (3.32) EiL) = E{TrI^r,<^^) + EiL)-E{l^r,<^^), = F(+^)][l-^^]dt

+ E{L)-F(+^)

(3.33)

0

SO that

E{L)=

j{Fi+oo) - F(t))dt + E{L)' F{+oo).

(3.34)

0

And finally 1 ^ ^(^) = l-^(+oc) ji^(^^)-^(0)dt.

(3.35)

So, for a transient renewal process, the lifetime is always a.s. finite and has a finite mean given by relation (3.35). Example 3.1: The Poisson process In queueing and risk theory presented in Example 2,1, the classical assumption for the arrival process is that it constitutes a renewal process where the r.v. X„ has as common distribution function: 0 , x<0, ^ W = i. -.. ' . / (3.36) l-e'^' , x>0, with /I being a fixed positive constant. AsF(+oo) = 1, the arrival process is recurrent. For (3.3), it is possible to have the analytical expression of the successive /7-fold convolutions. Indeed, we can successively write: F(2)(/) = ;L^(l-e-^(^-))e-'Vx

(3.37)

/

^^{6''" -e-^')dx

(3.38)

0

= l-e'^'-Ate-" ^\-e-"i\ + At), and in general:

(3.39) (3.40)

F^"\t) = \-e-"f^^.

(3.41)

k=0

^'

Applying result (3.5), we have

52

Chapter 2

k=Q

f^'

k=o ^'

r^ 42)

That is, for all fixed t, the process {N(t)) is a Poisson process of parameter A/. The value of the renewal function /f follows from relations (3.9) and (3.8): HiD^f^ne-"^-^, n=\

n\

- - ^ Z T ^ .

tr(«-i)!'

(3.43)

(3.44)

= e-"

Or H(t) = At,

(3.46)

It follows that for a Poisson renewal process, the renewal function is linear. As we shall see in section 5, a renewal process may also be characterized by its renewal function. The Poisson renewal process is the one which has a linear renewal function.

4 THE RENEWAL EQUATION Coming back to relation (3.9), we get, using the associative property of the convolution product: H(t) = F(t) + F^'^ (t) + F^'^ (/) + •••, = F + F^[F-\-F^^^+"•](tl

(4.1)

= F(t) + F^H(t). This relation is called the integral equation of renewal theory, or simply the renewal equation. It can also be written as follows: Hit) = F{t) + \F{t - x)dH{x).

(4.2)

0

Or: F • H{t) = H • Fit),

(4.3)

t

Hit) = Fit) + JHit - x)dFix).

(4.4)

Renewal Theory

53

In the particular case where the density function f of F exists, the last integral equation becomes H{t) = F{t) + j / / ( / - x)f(x)dx.

(4.5)

0

In this case, we can apply the dominated convergence to series (3.9), showing the existence of the density function of//, say h, such that:

K0 = Z/"1(0>

(4.6)

with

/"l(0 = / ( 0 ,

(4.7)

f^'\t)=jfit-x)f(x)dx,

(4.8)

fl"\t)=jf^"-'\t-x)f(x)dx.

(4.9)

0

From relation (4.5) or (4.6) we obtain the integral equation for h: h(t) = f(t) + f®h(tl with

(4.10)

f®h{t)=

(4.11)

jf(t - x)h{x)dx. 0

Or: f®h{t)^h®f{t), (4.12) (4.13) h{t) = f{t) + h®f{t). In fact, the renewal equation (4.2) is one particular case of the type of integral equations: (4.14) X{t) = G{t) + X^F{t\ where X is the unknown function, F and G being known measurable functions bounded on finite intervals and • the convolution product. Such an integral equation is said to be of renewal type. When G = F , we get the renewal equation. The study of these integral equations has a long history which includes contributions from Lotka (1940), Feller (1941), Smith (1954) and Cinlar (1969). Cinlar gave the two following propositions. Proposition 4,1 {Existence andunicity) The integral equation of renewal type (4.14) has one and only one solution, given by X(t)^R»G(t), (4.15) R being defined by relation (3.13).

54

Chapter 2

Proof (1) Existence In the second member of equation (4.14), we replace Xby expression (4.15): (4.16) G(t) + R^G^F(t), Using the commutative property of the convolution product, we get (4.17) G(t) + 7? • G • F(t) = (C/o(0 + R • F ( 0 ) • G(t), And by (3.14): (4.18) G(t) + i? • G • F(t) = R • G(t). So, function R • G(t) is a solution of the renewal-type equation (4.14). (2) Unicity Let X, and X2 be two solutions of equation (4.14), and let 7be defined by: r = X,-^2. (4.19) We then have 7 = 7#F(0, (4.20) and by induction, we get: Y^yF^''\ for all n>0. (4.21) As renewal function R can be defined by the function series (3.13), converging for all positive t, we know that: (4.22) limF^"^(0 = 0, for alW > 0 . n

Hence: lim Y • F^"> (0 = 0, for all / > 0,

(4.23)

n

and so by (4.21): 7(0 = 0 for all / > 0 .

(4.24)

It is also possible to study the asymptotical behaviour of solutions to renewaltype equations. The basic result is the so-called "key renewal theorem", proven by W.L. Smith (1954), and which is in fact mathematically equivalent to Blackwell's theorem (1948), given here as Corollary 4.2. A proof of the key renewal theorem using Blackwell's theorem can be found in Cinlar (1975b). Proposition 4.2 {Asymptotical behaviour. Key renewal theorem) (i) In the transient case, we have: limX(0 = i?(oo)G(oo)

(4.25)

provided the limit G(oo) = limG(0

(4.26)

exists.

Renewal Theory

55

(ii) In the case of recurrence, we have: \\mX{t) = - \G{x)dx,

(4.27)

provided that G is directly Riemann integrable on [0,oo), that F is not arithmetic, and supposing: m • E{X„)^\{\-F{x))dx.

(4.28)

Corollary 4.1 In the case of a recurrent renewal process with finite variance a^, we have: t \

lim ^(0

m^ -\-(J^

= \

2

'

(4-29)

Proof In the integral equation of renewal-type (4.14), let us choose X(t) = R(t)--^, m We shall compute G such that this integral equation is valid. We get: G(t) = X(t)-X^F(t) = R(t) - -^ - /? • F(t) + - \F(t - x)dx , m m^ From (3.14), we have for all / > 0: R(t) = Uo+H(t), so that:

(4.30)

(4.31) (4.32)

(4.33)

H(t) + -i- \F(t-x)dx. (4.34) G(t) = 1 + H(t)---F(t)-F• m m^ Using renewal equation (4.2) and posing x' = t-x in the integral of the second member, we get: G(t) = \

t m

1 V + - \F(x')dx\ mJ

(4.35)

And thus: G(t) = l

_1 \{l-F(x))dx. m mJ

(4.36)

Since 00

m= j{\-F(x))dx,

(4.37)

56

Chapter 2

we can also write: 1 °°

G(t) = ^ {(l-F(x))dx. (4.38) mf As a conclusion, the function G given by this last relation is the only function for which the integral equation of renewal-type has (4.30) as solution. Clearly, function G is monotonic non-increasing on [0,+oo) and:

JG(0^/ = -1 |

(4.39) j{l-F(x))dx dt, m, By permuting the order of integration (with the help of Fubini's theorem), we get: 1^ JG{t)dt = i . j x ( l - F{x))dx. (4.40) 0

^

0

But we have: 00

a'-^m'

= jx'dF{x)

(4.41)

0 00

=:-jxV(l-F(x)).

(4.42)

0

And integrating by parts, we find 00

a'+m'

=2Jx{l-F(x))dx.

(4.43)

0

Coming back to relation (4.40), we finally have

ym = ^ .

(4.44)

Corollary 4.1 is thus a direct consequence of result (ii) of Proposition 4,2. n Remark 4.1 1) From result (4.29), we immediately get an analogous result for the renewal function H, Indeed, we know, from relafion (3.14), that R(t) = H(t) + Uo(tl t>0. (4.45) Applying result (4.29), we get:

liJH(t)-L] = ^!^-l,

(4.46)

or

limfi/(0-^l = 4 ^ '

(4.47)

/-^ool^ ^' mj 2m^ 2) The two results (4.29) and (4.47) are often written under the following forms:

Renewal Theory 2

57 2

^(0 = - + ^ V - r - + o(i),

(4.48)

H(t)^-

+ ^ ^ - ^ + 0(l), (4.49) m 2m where 0(1) represents a function of ^ approaching zero as t approaches infinity. Corollary 4.2 In the case of a recurrent renewal process with finite mean m, we have: lim^^^l. r->cxD f rn

(4.50)

Proof Of course, if o"^ < oo the result is trivial by Corollary 4.1 if we divide both members of the relation by lim-. The result is also true if cr^ = oo .

n

Corollary 4.3 {BlackwelVs theorem) In the case of a recurrent process with finite mean m, we have, for every positive number T : \im( R(t)-R(t-T))

= ^.

(4.51)

Proof Let us consider the renewal-type equation (4.14) with function G defined as follows: G(/) = . '

^^'^^'

0,

(4.52)

T
where ris a fixed positive real number. From the key renewal theorem (Proposition 4.2, part (ii)), we know that the unique solution, given by Proposition 4.1: X(t) = R • G(/), (4.53) is such that \imX(t) = - \G(x)dx. m,

(4.54)

By definition (4.52) of function G, we have X(t) = ^R(t) --R(t T

- T)

(4.55)

T

and 00

JG(x)dx = 1.

(4.56)

58

Chapter 2

Inserting results (4.55) and (4.56) in relation (4.54), we get result (4.51). n Remarks 4.2 1) Probabilistic interpretation of renewal density, Let k{t)dt represent the probability that there is a renewal in the time interval (/,^ + (i/). It must satisfy the following relation, obtained by a simple probabilistic argument using the independence property of the successive "lifetimes": i

k(t)dt = f{t)dt + \f{x)k{t - x)dt,

(4.57)

0

From the unicity part of Proposition 4.1, we get, for all / > 0 : k(t) = h(t). (4.58) So, the probability defined above is given by h(t)dt and more generally by dH(t) with a precision error ofO(dt). This interpretation often simplifies the search for relations useful in renewal theory. 2) The variance of N(t) From Stein's lemma, we know that N(t) has, for all t, moments of any order. Also, let 6^2 (0 be the centred moment of order 2 of N(t): a,(t) = E{(N(t)y).

(4.59)

From results (3.5), we can write successively: a,(t)^j;^k'{U,-F)'F^^\t) 00

(4.60)

00

= ^k'F^'\t)-Y,k'F^'''\t) ^=1

k=\

00

00

(4.61)

= f,[v'-(y-iy]F^''\t)

(4.63)

= X(2v-l)F('-)(0

(4.64)

= 2|;(v-l)F"''(0 + i;F(^)(0.

(4.65)

Now if we compute H^^\t) by means of the relation: /

00

H^'\t) = \ Z^^'^t)

^

/^ °°

• ^F^''\t) V^''=i

we easily find:

^

,

(4.66)

Renewal Theory

59

if(2)(/)^£(K-l)F<'''(0.

(4.67)

Using (3.9) and substituting this last result in relation (4.65), we finally obtain: (4.68) a\t) = H(t) + 2H^'\ty This last result shows that Var (iV(0) = H(t) + 2H^'^ (t) - (H(t) f

(4.69)

5 THE USE OF LAPLACE TRANSFORM 5.1 The Laplace Transform To show the power of the Laplace transform in solving the renewal equation, let us suppose that d.f F characterizing the recurrent renewal process considered has / as density. Let us use the following general notation: for any function a on [0,oo), d will represent its Laplace transform, provided it exists: CO

d(s) = je-''a(x)dx,(=

£{a(x))),

(5.1)

0

With this convention and using the well-known property of the Laplace transform £{(a®Mx)) = d(s)J(s), (5.2) we get from the renewal equation: (5.3) h(s) = m + h(s)-f(s). The Laplace transform of the unique solution is thus given by:

h(s) = J ^ .

(5.4)

l-/(.) Using the inverse Laplace transform, we can then have the value of h(t). Remark 5.1 From the algebraic equation (5.3), we can deduce the expression of the density / as a function of the renewal density h. In Laplace transforms, we get: /W =- ^ . The inverse Laplace transform gives us a function of h.

(5.5)

60

Chapter 2

This leads to the important result that every renewal process is characterized by its renewal density, if it exists, or by its renewal function. Thus there is a one to one correspondence between the d.f. F of a renewal function and its renewal function H. Example 5.1 The Poisson process Let us consider Example 3.1. again. From relation (3.36), we obtain x>0. f(x) = Ae-'\

(5.6)

In this case, we easily have the Laplace transform / :

f(s) = AJe-^'''^'dx,

(5,7)

0

so that

m=-^-

(5-8)

S -\- A

Equality (5.4) gives in this case: Ks)--.

(5.9) s Since, for every constant K : £(K) = - , (5.10) s and using the unicity of the inverse Laplace transform (see for example Widder (1945)), we deduce that: h(t) = ;i, (5.11) Since: / / ( / ) = jh(x)dx,

(5.12)

0

we also have: H(t) = At,

(5.13)

It follows that the Poisson process is the only one for which the renewal function is linear. This result was already announced in Example 3.1. In this case, results of Corollaries 4.2 and 4.3 are exact results. Example 5.2 A gamma distribution in reliability theory (Erlang process) Let us recall that the density function of type gamma, with parameter /I and of order A:(/I > 0, A: e NQ ), is given by:

Renewal Theory

61

\k-\

/M(0 = ^ ^ e - ^

(5.14)

It is easy to see that /^^ is the density function of the sum of k independent, identically distributed, random variables, with the negative exponential distribution of parameter A. This meaning of the gamma distribution of parameters (A, A:) eases the search for the distribution of N(t). Indeed, the event {N(t) = n} is equivalent to the event of having nkov nk + 1, or ..., or nk + k-l occurrences of a Poisson process with parameter A on [0,^]. Let Np(t) represent the number of renewals for such a Poisson process. We have: ?(N(t) = n) = j;^?{N^(t)

= nk-^j)

.

(5.15)

j=0

Using result (3.41), we can now write:

^im->.)-e-'f^, =e

(5.16) y

^-^

In particular, for A: = 2, we get At (Atf ^i^N(t) = n) = e-''^^^^^\\ + ^^^\. (2n)\ 2n + \ To compute h{t) for A: = 2, we use result (5.4) with f{s)^X^\te-^^^'^'dt

(5 17)

(5.18)

(5.19)

0

{s + xy

(5.20)

This gives us: h{s) = ^_ , , , ^, . {s + Xf-X Since

*« 4-2(7721) •

(5.21)

P'22)

we can easily invert h{s) so that hit)-\-\e-'". As //(O) = 0, by integration we immediately get:

(5.23)

62

Chapter 2

This example shows that the difference H(t)

is here given by m

\{e-'"-l)

.

(5.25)

Thus, the error made in approximating H(t) by — does not approach zero as m « —> 00 . Indeed, we have At \ . (5.26) Hm H(ty 2 /->oo

By (4.49), this result means that, for a gamma distribution of parameters (/1,2), we have:

or CT^=^,

(5.28)

which is true, since from relations (5.38) of Chapter 1: m:^j,

(5.29)

^'=JT-

(5.30)

More generally, for the general T distribution of parameters (A,,n), we have that, if F^„ isthed.f:

^?i)(0 = ^...(0 ,

(5.31)

F(^^^j^) being the d.f of the sum of nk independent r.v. of negative exponential distribution with parameter A. So, we have for density function /^^/j^:

/(*/j)(0 = A„*(0 .

(5.32)

Or, by (5.14)

fi'll(t)-^^^e-- . This last result gives an explicit form of the renewal density:

which gives result (5.23) in the particular case of A: = 2. It can be shown (see Barlow & Proschan 1965) that:

(5.33)

Renewal Theory

63

ACO^IZ^-e^''^'-'),

(5.35)

where 6: = cos--- + zsin—- . k k And it can be shown by integrating (see Parzen (1962)) that:

(5.36)

Example 5.3 The "normal" distribution The support for any normal distribution is the entire real axis; this same support is not suitable in renewal theory. However, if / is the density function of a normal distribution with mean m and variance <j^, and if a is small with respect to m, then the truncated normal distribution on [0,oo], is very close to the true normal distribution. Indeed, the density function for the truncated normal distribution on [0,oo) of parameters {m,(j) is given by: 1 X

{u-mf

- [e~ 2-^ du , a •' " 0

(5.38)

where {u-mf

<»

a = j / 2a2 ^^ ,

(539)

0

With the change of variables: 4=

'-^,

(5.40)

the last integral becomes: 00

e2

a^(r\e~d^ .

(5.41)

As m « (J, We have: m =0 ,

(5.42) (5.43)

(J

SO that, approximately: 00

e2

az(7\e"^d^,

(5.44)

0

or a^yflK^

.

(5.45)

64

Chapter 2

This shows that under the assumption (5.42), we have as a "good" approximation: - {i '-' du = -j=L= \e 2-^ du ,

(5.46)

since: 1 f -^^^r 4lnG-a , -— \e 2o- ciu = — . 27r(7 _£ V2;rcr - 1 - 7 ^ ,

tc An\ (5.47) (5.48)

=0 . To conclude, let us say that under assumption (5.42), the usual normal distribution is a good approximation of the truncated one. In other terms, we can use the normal tables to compute the functions we need in renewal theory. Let F be the d.f of a truncated normal distribution, with parameters satisfying (5.42). From the stability property of the normal law, we have F(")(/) = 0 ^ ^ j

,

(5.49)

where O is the d.f. of the reduced normal distribution: 1 'r -"1 ^{x) = -== \e ^du , V2;7-i so that P[iV(0-«] = O

^

(m,a)

-<1> — — r ^ i

v. crv« J

\

n>0,

(5.50)

(5.51)

CT-Jn )

//(Os^of^l,

(5.52)

(t-nmf

hit)=^==^^-j=- .

(5.53)

5.2 The Laplace Stieltjes (L-S) Transform The L-S transform is a bit more general than the Laplace transform. Indeed, for any function a on [0,oo), a will represent its L-S transform, provided it exists, under the form:

Renewal Theory

65

a(s) = je-'^'daix)

{=£(a(x)))

.

(5.54)

0

For a function a such that \ime-''a(x) = 0 ,

(5.55)

Af->00

an integration by parts gives the relation between d and d: d(s) = -a(0) + sd(s). As £ satisfies the following property: £i{a./?)(x)) = £{aix))-£{j3{x)) , we get from the renewal equation (4.4): H(s) = F(s) + His) • F(s). Or:

(5.56) (5.57) (5.58)

\-Fis) which is equivalent to the expression (5.4) if we do not assume the existence of a density for F. Of course, if such a density exists, we have from (5.54): Fis) = fisl (5.60) H(s) = h(s), and consequently, the relations (5.59) and (5.4) are identical.

(5.61)

Example 5.4 Consider the deterministic model for which F = 1/^ . Since 00

F(s)--

(5.62)

0

(5.63)

(5.59) gives us: e-' His)-. ^l-e-'

(5.64) • )

(5.65)

00

(5.66) But as 00

le-dU,ix) = l{{Ul'\x))}

(5.67)

0

=

e-\

we can invert H to find:

(5.68)

66

Chapter 2

H(t) = '£ui'\t) .

(5.69)

Or: 00

H(t) = Y,U,(t) .

(5.70)

5.3 An Application To The Renewal Function From Example 5.1.1, we know that / / i s linear iff the renewal process is Poisson: x>0 , (5.71) H(t) = At<:>F(x) = l-e-''' , If we multiply //(/) by a constant//, then the function H,(t) = ^At (5.72) is still a renewal function. More precisely, //j corresponds to the d.f /^ given by: . (5.73) F,(x) = l-e-^''' More generally, Daley (1965) considers the problem of multiplying a renewal function / / by a constant a to characterize, if possible, a new renewal process. We have the following proposition, Proposition 5.1 (Daley (1965)) If H is a renewal function, then the function aH, with constant a belonging to [ 0,1 ] is still a renewal function corresponding to the following df, F^: 00

F^{x) = a^(l-ay-'F^"\xl

x > 0.

(5.74)

Proof Let us suppose that F^ is the d.f corresponding to the renewal function aH . From the relation (5.58) pertinent to aH, we have: 1 + aH{s) Since, by (5.59): His) - - ^ , 1 - F{s) we can write:

F„W = - 2 ^ \-F{s)

(5.76)

L^ , 1 + - aF(s) l-F(s)

(5.77)

Renewal Theory

67

Or: FAs) = ^ .f%., • 1 - (1 - a)F{s) Since we assume that a < 1, we get: Q<(l-a)F{s)<{\-a)<\ .

(5-78)

(5.79)

We can thus use the power expansion of the function (1 - x)" to express F^ {s): F„{s)^F{s)Z{^-ocr[F{s)]". Or:

(5.80)

00

FM

= aY,{\-ar'[F{s)]\

(5.81)

or (5.74) by inversion. Clearly F^ is a non-negative and non-decreasing function, defined on [0,a), such that: 00

\\mF,{x)^aY{\-a)"'\

(5.82)

n=\

We can now say that F^ is a d.f, and consequently the unique d.f having aH as renewal function. n

6 APPLICATION OF WALD'S IDENTITY 6.1 Wald's Identity We already know that E{S„) = nm, as S„^X,++ X„ and E{N{t)) = R{t). Now let us consider the time of the first "replacement" after time /; it ^y^NV)' Wald's lemma computes the mean of this random variable. Proposition 6.1 (Wald's lemma) E{S,,,,) = mR(t).

(6.1) (6.2) (6.3) is given

(6.4)

Proof From the general definitions, we can write that if A^(/) = /? , we have NXt) = /7 + 1 , and consequently:

68

Chapter 2

as, in this case: S„
(6.6)

n

n=Q k=Q

Interchanging the orders of summation, we get 00 k ^N'(t) ~ 2^2^^k+\^[s„
(6.8)

k=On=0

= |;x,„X/[.,.,<5„.] k=0

(6.9)

n=0

£^..,/[...,].

(6.10)

k=0

As random variables Xj^^^ and S,^ are independent, we use Lebesgue's theorem on monotone convergence to find: E{S,,„)

= Y^EiX,,,)-E{l^s,<_,^)

(6.11)

k=0 00

^J^mF^'^t)

(6.12)

it=0

= mR{t), using result (3.13). Remark 6.1 As, by (3.10): NXt) = N{t) + \ , we can use relation (3.14) to write result (6.4) as: E{S,,,,,) = m[H(t) + l] .

(6.13) n

(6.14) (6.15)

6.2 A Lower Bound For Renewal Function R Corollary 6.1 For all positive t, R(t)>-.

(6.16)

m Proof By definition of NXt), we can write: t<S^yy Taking the expectation of both members, we get: t<E{S,,,^).

(6.17) (6.18)

Renewal Theory

69

Or, from Proposition 6.1: t<mR{t) , i.e., inequality (6.16)

(6.19) D

Remark 6.2 Later, we shalll see that the notion of the stationary renewal process will lead to an upper bound of R{t) for a large class of renewal processes.

7 ASYMPTOTICAL BEHAVIOUR OF THE iV(0-PROCESS This section will yield two important results concerning the counting process {N(t),t > 0) associated with a recurrent renewal process characterized by the d.f F. Proposition 7.1 (Strong law of large numbers). If m < CO, then, almost surely: hm—^ = ~. ^^°° t m

(7.1)

Proof For any sample path of a renewal process, we have: 5^(,)<^<5^(,),,. It follows that:

As the renewal process is recurrent, we use (3.23) to see that, a.s.: lim7V(/) = oo.

(7.2)

(7.4)

/->oo

And by the assumption of i.i.d. for theX„, we use the strong law of large numbers to arrive at: ^ . ^ Z , + - + Jr„^^ (7.5) Or ,. n I hm— = - .

,_ ^, (7.6)

As the range of ——,/ > 0 V ^f^O)

h m - — = —. And as

J

is a subsequence of —, « > 0 , we also have: S„ (7.7)

70

Chapter 2

N(t) ^N(t) + l *^yv(/)+i

^;v(/)+i

N(t)

^^g^

^ ( 0 +1

we also get, by similar reasoning, that: hm—^-^ = —,

(7.9)

using result (7.4) again. Thus the proposition results from the double inequality (7.5), and from a classical a result on limits of sequences. Proposition 7.2 {Central limit theorem). If G^ < 00, then for ally E R : limP

^ N{f) -t/ ^ M
/->00

/m^ Vy,

\y~

,

(7.10)

J

where O represents the standard normal df. Proof We know that successive hfetimes X^,X2,...,X„,.., of the renewal process are i.i.d. and with finite variance. We can thus apply the classical central limit theorem to the sequence of partial sums {T„,n>0), saying that:

limpf^^;^<7J = (D(7).

n->QO

(7.11)

By Definition 2.2, we can pass from the T„ -sequence to the N(t) -process: P{Nit)>n) = PiT„
= P(ILZ^00

(Jy/n

(7.13) .„ . .. (7.14)

As t -nm

t

y/n

(jyfn

Gyfn

cj

m,

(7.15)

we must have from (7.14) lim—pr = +oo. L-lZcJ-in In a similar way, as

(7.16)

71

Renewal Theory

izJJ^JV ' ( ' - • /t>^

(7.17)

an application of the result (7.16) shows that the limit theory imposes that necessarily: l i m — = 1.

(7.18)

Consider now the probability in the first member of equality (7.12). We have successively: P{N(t)>n)

=P

N(t) - t/ m

m

FZ'~Ff.

(7.19)

mr' J

But • - t/ m

f

2 // mr'

nm-t •>Jm,

(7.20)

nm-t

(7.21)

cr^y7

\nm

If « -^ 00 and / -> 00, relations (7.14) and (7.18) imply that the quantity above approaches -y : limP

7^(0- t/ m > -y ^limP(Nit)>n), V\ /m

f

(7.22) (7.23)

This last result implies that limP t-^co

N(t) - t/ m

f

-2 / /

y

=

\-<^{-y),

m = <^{y),

(7.24) (7.25)

by a well-known property of the standard normal d.f. Remark 7.1 From Proposition 7.2, we have the following approximation for large t: ^2

var(7V(/))

-t,

m which simplifies the exact result (4.69).

(7.26)

72

Chapter 2

Example 7.1 {Application to Inventory Theory) The following inventory problem will show that Proposition 7.2 can lead to practical results. A technical component has a random lifetime with mean m = 100 hours and standard deviation cr = 60 hours. For security reasons, the successive replacements must be done without any interruption for 8000 hours. What is the number of replacement components to have in stock at time 0 so that there will be no break with a probability of .95? From (7.10), we get

'W000)-80^^_^; 60^/8000; 1000000

= 0.95.

Or P( A^(8000) < 80 -f 165 • 0.6 • VOOOS) = 0.95 .

(7.27)

(7.28)

Thus, we get P{ A^(8000) < 80 + 9) = 0.95. (7.29) Consequently the level of the security stock at time 0 must be at least 89. It is amusing to note that the number 89 can be obtained with the aid of the very , thus 80, we heuristic but often-used rule saying that to the ratio y = / M 100 add 10% for the security and we get 88, nearly the optimal number!

8 RECURRENCE TIMES 8.1 Definitions Let us consider the renewal process (r„, n>Q) characterized by the d.f F of mean m. At a positive time/, we know that the lifetime is that of A'yv(/)+i ^^^NU) » ^s: ^N{t) ^ t < Sj^f^t^^^. (8.1) In terms of a replacement process, the component in effect at time t is of age d{t) given by S{t)^t-S,^,^, (8.2) and the time that this component has to wait until its failure, say Y{t), is given by r ( 0 = 5;v(0.i-'-

(8.3)

Of course we have that the total lifetime of the component, X;v(,)+,, satisfies X,u).^=S{t) + r{t).

(8.4)

73

Renewal Theory

We shall use the following terminology: the r.v. 5{t), y{t) and Xy^(/)+i will be called the age (or backward recurrence time), the excess (or forward recurrence, or deficit time, or residual life) and the spread (or recurrence time) at time t, respectively. ^. VCO+1

^

X

, ^(^) ,

^Nit)


Figure 8.1: age, excess and spread These random variables, or more precisely these stochastic processes, are useful for applications. For example, if we want to compute the probability that the component in Hfe at time t will not fail during the interval (/, / + r ) , we must be able to compute the following probability: p{r{t)>T), (8,5)

8.2 Distribution Of Recurrence Times We shall first give the distribution of r.v. d{t), y{t) and Xj^^^^^^ for all /. Secondly, we will study the asymptotical behaviour of these d.f., i.e. when t approaches +oo. Proposition S.l(Distribution of the age) If F^ft) is the df of age S(t), we have: (8.6) F,,,,(x)-Fs,,,(t-0)

F,,,=

j

=

l-F(t),

(8.7)

[l-F(t-u)]dH(u).

(8.8)

Proof Results (8.6) and (8.7) are trivial. To prove the last result, it suffices to apply the proposition giving the probabilistic interpretation of renewal function H and the independence of successive steps X„,n>l. n

Chapter 2

74

Proposition 8.2 {Distribution of the excess) If F^(^) is the df of the excess y{t), we have, for all positive x: F^^^^ (x) = ^[F{t-u

+ x)- F(t - u) ] dR(u).

(8.9)

Proof

% Figure 8.2: age events analysis If we examine Figure 8.2, we can deduce that event /(t) < x is reaUzed iff S^^,^
As n can take any integer value, we have 00

P{r(t)<x)^^

j[F(t + x-u)-F(t-u)]dF^"\u),

or (8.9) by relation (3.13).

(8.16)

Renewal Theory

75

Remark 8.1 By (3.14), we can also write that + x)-F(t-u)]dH{u). P ( K O ^ ^ ) = ^(^ + ^ ) ~ ^ ( 0 + l[F(t-u

(8.17)

[0./]

Remark 8.2 For the study of the asymptotical behaviour of Z^^^), relation (8.17) is useful under another form. This form is obtained by replacing F by 1 - i^"" in the integral of (8.9), where F^(x) represents F^(x) = P(X,>x), We get P{r(t)<x) = F(t + x)-F(t)+

(8.18) j [F'(t-u)-F'(t-u

+ x)]dH(u).

(8.19)

[0./]

Or equivalently: P{r(0 <x) = F{t + x)-

j F'(t-u

+ x)dH(u)

[0./]

N

(8.20)

F(t)- JF'(t-u)dH(u) . 0

J

By the renewal equation (4.2), the term in brackets is equal to zero. This yields the following result: = F(t-hx)j F'(t-u + x)dH(u). (8.21) P(rit)<x) [0./]

Remark 8.3 (Expectation of y{t)) Using result (6.15) together with (8.3), we get E{r{t)) = m[H(t) + \]-t.

(8.22)

This result shows that the exact value of this expectation is known as soon as we know H{t), Applying the result (8.22) to the Poisson process, we find: E{r{t)) = m, (8.23) which is a quite natural result if we remember the memoryless property of the exponential distribution. Proposition 8.3 {Distribution of the spread) If Fx^ ^ , is the df of the spread XA^(^) , we have

Chapter 2

76

j[F(x)-F(t-u)]dH(u),

ifx
t-x

(8.24)

t

F(x)-F(t)+

j[F(x)-F(t-u)]dH(ul

ifx>t.

0

Proof Let us consider the following pictures. X

x>t

^N(t)+l

^.^(0

t

^N(,t)+l

X
^JV(o

^N(t)+l

Figure 8.3: spread events analysis We are considering the event ^Ar(^)+i < x . Let us suppose first that x > /. It is thus possible that the component in effect at time t is still the first one; in this case, the probabihty of the considered event is

F(x)-FiO

.

The other possibility is that there exists at least one renewal before or at t, If we suppose that the component in effect at time / was introduced into the system between u and u + du, we have to compute the probability that this component has its lifetime in interval(/-w, x). This result, combined with the probabilistic interpretation of the renewal function gives the integral of (8.24). Of course, if x < /, then there is necessarily at least one renewal before or at t, and n the probability of event ^A^(/)+I<;, is simply given by the integral. Remark 8.4 Using renewal function R defined by relation (3.13), we can write result (8.22) as

Renewal Theory

77

^ W , W = j[F(x)-F(t-u)]dR(u),

(8.25)

t-x

assuming of course that H and R are 0 for strictly negative values of the argument. Corollary 8,1 If F has fas derivative, then Fxj^^,) ^^^^ has fx^^^^, and: ^r f{x)[M{t)-M{t-x)] ^''^' \f{x)[M{t)-M{t-x)]

+ f{x)

ifxt ,

Proof The corollary is a simple consequence of the formula giving the derivative with respect to x of function A(x), defined by: X{x)=

j K(u,x)du;

(8.27)

a(x)

A'(x) = f -^K(u,x)du

+ j3\x)K(j3(x%x) -aXx)K(a(x%x).

a (8.28)

a(x)

Example 8.1: the Poisson process We shall consider the case of a Poisson process for which F is an exponential distribution of parameter A. Since t>0, (8.29) F{t) = l-e-^\ and H(t) = At, (8.30) (8.31) R(t) = ;it + l, results (8.6), (8.7) and (8.8) of Proposition 8.1 yield f A f e-^^'-'^^du, x
(8.34)

Chapter 2

78

(8.35)

= Xe-^' y" (1 - e-"-" )du + e"^' (1 - e"^") 0

(8.36) = e-^^ -e-^\^

-1)+ e-^il -e'^)

(8.37)

= (1 - e-'")(1 - e-" ) + e-" (1 - e-'"),

(8.38)

and finally 7V,)(x) = ( l - e - ^ ^ ) (8.39) (8.40) = F(x). This last result proves that the distributions of the age and the excess are independent of t and that it is the original d.f., which is in fact equivalent to the memoryless property of the exponential distribution. For the d.f of the spread, we have from result (8.24) of Proposition 8.3 that:

^n

^^«,„.. W =

e-^^'-"^ -e-^'']du,

g-^i _ g-'i^ +

ifx
Aj[.

-A(t-u)

•e~^'']du,

ifx>t.

As \-e

X j[e-^(^-")-e-^-]^w = A

-Ax

• e'^^'x

(8.42) (8.43) (8.44)

and l-eX we finally get that: _ ^^ {\-{\ + Xx)e-'\

-/•V«(,)., W

X^xe-^", o^-Ax X^te'^'+ . Xe-^\

I -,i.-Ax

•e-^'t

ifxt.

(8.45)

(8.46) (8.47)

Note that for x
Renewal Theory

79

8.3 Asymptotic Behaviour In many practical situations, we can suppose that the observed renewal process has in fact been going on for a long time, so that we are in the right conditions to observe the "equilibrium process", that is the stationary behaviour given by letting t —> 00. The next proposition gives the asymptotical behaviour of the d.f iV(^), F^(^) and

Proposition 8.4 If m is finite, then for all positive x: (i) limF,(,)(x) = limF,(,)(x) = - \[\-F{u)]du, /->oo

^ ''

/->oo

'^ '

ffl

(8.48)

J

w„

1 -^

(ii) limiv...M (x) = — LdF(u).

(8.49)

Proof (i) From Figure 8.1, we can write that < x), P{Sit) <x) = P{r(t-x) or Fs,,(x) = F^,_,,(x). As for any function g , we have, for fixed x: l i m g ( x , 0 - lim g(x,t),

(8.50) (8.51) (8.52)

SO that, from relation (8.50), we can deduce that F^^^) and F^(^) have the same limit, provided it exists. From (8.17), we get that: P{r(t)>x)=

JF'{t-u

+ x)dH{u) + F'{t-\-x)

.

(8.53)

0

As by (3.14), we know that R(u) = Uo(u) + H(u), ueR, we also have: P{r(t) >x)= JF'(t -u + x)dR{u) -F'(t

(8.54) + x) + F'(t + x)

(8.55)

0

= JF'it-u + x)dR(u), 0

We can now use Proposition 4.2 to show that

(8.56)

80

Chapter 2

limP(r(t)>x)

= - \F'(z + x)dz,

(8.57)

Using the change of variable u = z + x ,WQ obtain: = - \F'{u)du, m

limP(r(t)>x)

(8.58)

X

As, from relation (8.18), we have F' =l-F, we can also write:

(8.59)

l i m P ( / ( 0 <x) = l-— {\l-F'{u)]du. m t->oo ^ '^ m J

(8.60)

X

The desired result then follows from the fact that: m= j[l-F(u)]du.

(8.61)

0

(ii) Let us consider F,^^_^Jx) = K(xj).

(8.62)

In setting a condition for the value of X^ (say j^), we get: I , if>^ > max{x,/}, P(Z^(,),, >1, X,=y) = K{x,t-y ,ify
(8,63)

K{x,t) = l~F{rmix{x,t})+

(8.64)

JK{x,t - y)dF(y). 0

Using Proposition 4.1, we now have: K{x,t) = {\-F{m^x{x,.})^R(,))it). (8.65) Now, applying result (4.27) of Proposition 4.2 we have the asymptotical behaviour of K(x,t) for ^ -> oo : 1 °°

limi^(x,0 = ~ \[l-F(m^x{x,u})]du.

(8.66)

This last integral can be transformed using the following development and with the aid of integration by parts: 00

j[l-F{rmx{x,u})]du= 0

00

A;

j[l-F{x)]du+

j[\-F{u)]du

O

(8.67)

x 00

= [l-Fix)]x

+ [(l-F{u))uY^

+ judF(u). X

The fmiteness of the mean m implies that:

(8.68)

Renewal Theory

81

lim[l-F(w)]M = 0

(8.69)

(see for example Tucker (1962)). Consequently, the first two terms are opposite, and we get that: 00

CX3

j[l-F(m^ix{x,u})]du=

judF(u).

0

(8.70)

X

So, result (8.49) follows from (8.66). An immediate consequence is given by the following result.

n

Corollary 8.2 Ifm is finite, then (i) the limit distribution F^ of F^^^) and F^(^) has a density function f^ given by /.(") = ^ — ^ , m (ii) the limit distribution of F^^^^^^^ has a density distribution: uf{u) m provided F has f as density,

(8.71)

(8.72)

Proposition 8.5 (i) For all positive t,x,y,y < t, t-y

P{r{t)>x,

S(t)>y)=

JF'(t-u

+ x)dH(u) + F'(t + x) .

(8.73)

0

(ii) Ifm is finite, then l i m P ( K O > ^ , ^(0>y) /—>00

1 °°

=-

ff^

{[l-F(u)]du.

(8.74)

J

x+y

Proof (i) The figure below clearly shows the equivalence of the events {cD : r(^^^) > x,S(t,a)) > y} and {co: r(t-y)>x + y}.

(8.75) (8.76)

Chapter 2

82

y .

f-y

,

G^)

X

. r(t)

S(t)

^NOHl

rit-y) x+y Figure 8.4: excess and age events analysis Consequently, + y). P{r(0>x, S(t)>y) = P{rit-y)>x Using the relation (8.17), we obtain P{r(t)>x,

S(t)>y)

+ x)-

= l-F(t

j [\-F{t-u

+ x)]dHiu)

, (8.78)

[0./-;^]

That is, (8.73), since: F' =l-F. (ii) From the limit properties, we have > x + y) = \imP{y(t) > x + y), limP{/(t-y)

(8.79) (8.80)

An application of (8.77) and of result (8.48) of Proposition 8.4 yields: 1

\imP(r(t)>x,

S{t)>y)

x+y

= l--^ 1 °°

m

{[1-F(u)]du

(8.81)

\[l-F(u)]du,

(8.82)

since m=

\[\-F{u)]du.

D (8.83)

8.4 Example: The Poisson Process From (8.29), using (8.48) and (8.51), we easily get: X

/->00

^^

J

= \-e-

(8.84) (8.85)

Renewal Theory

83

limTv^^^^^^ (x) = A' jue-^"du

(8.86)

^-Au

= A'

2

(8.87)

i-Au-l)

(8.88) = l-e-'^(l + A;c). So, we see that F^, and consequently F^, have the same value as F . On the other hand, F^^ has a gamma distribution of second order. Proposition 8.6 explains this fact by showing that ^(0 asymptotically independent. Indeed result (8.74) of this proposition yields:

and /(t)

are

00

limP{r(t)>x,

S(t)>y)

= A \e-^"du

/—>oo

(8.89)

J

^e-'^'^'K Using (8.85) and results (8.48) of Proposition 8.4, we get \\mP{y{t) >x) = \\mP{S{t) >x) = e-^\ And so, from (8.90): limP(;'(0 > x,d{t) >y) = limP(KO > x)-\\mP{S{t) /—>Q0

^

/->00

> y).

(8.90) (8.91) (8.92)

r->00

From this result and from the fact that, for all /, ^;v(o.i=^(0 + KO, (8.93) Jf;y(^)+, is asymptotically the sum of two independent random variables having the same exponential distribution of parameter X. The distribution of X^ is indeed of type gamma and of second order. But more can be said by using results (8.74) and (8.39), which lead to: 00

P{ r(t) > X, S{t) >y)^X

\ e-^"du

(8.94)

x+y

= e-'^'^'\ (8.95) which is exactly the result (8.90). Consequently (8.92) is true for every t, and so S(t) and /(t) are independent for all t. This explains the fact that ^Ar(/)+i has a gamma distribution, as shown by (8.47)

8.5 Some Parameters Of The Limit Distributions Let us first consider the limiting of S(t) or /(t), say F^ . From Proposition 8.4, we know that for all % > 0 :

Chapter 2

84

Fs(x) = — m

(8.96)

{[\-F(u)]du.

The mean S^ of this distribution can be computed in the following way: C»

^

j[l-F^(x)]dx

00

00

=—

0

^

jdxj[l-F(u)]du. 0

(8.97)

X

Using Fubini's theorem, we have U

^ 0 0

S,

(8.98)

=-\[\-F{u)]du,\dx ^

0

0 1

00

= — Jw[l - F(u)]du,

(8.99)

We can compute this last integral by parts, so that -{\-F{u))

m

0

+

^ju'dF(u).

2m „

(8.100)

If we now suppose the finiteness of cr^, the variance related to d.f F, we know that the first term is zero, so that: ^ ^ ^ T2m. !;;!^'^^^^)

(8.101)

or (j^ +m^

(8.102)

2m that is: m

(J

2 2m' This last result shows, surprisingly, that S^ may be larger than m\ Indeed r, (J^ S^ > m <^-^> 2m

m —

(8.103)

(8.104)

or S, >

m<^-'>l. m Let us denote by cv the coefficient of variation: a cv = • m The general result is S^ > m <^ cv >l. Let us also note from (8.103) that we always have

(8.105)

(8.106) (8.107)

Renewal Theory

85

(8.108)

Ss>j,

which is, at least at first view, a paradox. For the mean X^ of the limiting distribution of the spread, we get from (8.49):

X^ = J[l - - \udF{u)]dx, 0

^

(8.109)

Q

or: ^

CO

00

X,= — \[ judF(u)]dx.

(8.110)

0 X

Using Fubini's theorem again, we get: 1 °°

X, =-L'dF(u),

(8.111)

so that: (8.112) X^=m + —. m Once again, we have a paradoxical result: asymptotically, the mean X^ is always larger than m ! This fact that, for large t, the mean lifetime of a component in current operation exceeds the mean lifetime, can be explained as follows: / being arbitrarily fixed, its value will fall, in all likelihood, in a renewal interval {T^((), Ti^(t) + -^yv(/)+i) having a lot of duration. This phenomenon is called the length-biased sampling. For example, in the case of a Poisson process of parameter A, we know that: ^ ^ 2 m = -,c7'=-j,

1

so that, from equality (8.112): 1 X,=j.

(8.113)

(8.114)

Or, equivalently: X,^2m. (8.115) Of course as the coefficient of variation is equal to w , we also have from (8.107): S,=m (8.116) Remark 8.5 Result (8.112) is a direct consequence of result (8.93). Indeed, we have for all t: E{X,,„,,) = E{S(t)) + E{a(t)). (8.117) And taking the limit of both members, we get

86

Chapter 2

lim^(X,,,,) = 2 | 2^ +2m ^ JJ .

(8.118)

That is, (8.112).

8.6 A Characterization Of The Poisson Process Having as renewal process a Poisson process of parameter A, relation (8.73) yields: t-y

P{r{t)>x,

S(t)>y)

= A je-'^^-^'^'^^du + e-'^^''^;

(8.119)

0

the computation of this integral gives as a final result: (8.120) P{r{t)>x, S(t)>y) = e-'^''^'\ Coming back to results (8.33) and (8.37), we can write that, for all positive t, X, y with y x, S(t)>y) = P{r(t)>x)-P{S(t)>y) . (8.121) As both members are equal to zero if ;^ > /, this last relation is true for all nonnegative x,y,t and shows that for all t, the r.v, /(t) and 5{t) are independent. The next proposition will show that this property of independence characterizes the Poisson process; that is, it is not only a necessary condition but also a sufficient one. Proposition 8.7 Suppose that for all t, the recurrence times y{t) and 5{t) are independent; then the renewal process is a Poisson process. Proof Taking the limits of both members of equality (8.121) with / - > o o , Propositions 8.4 and 8.6 yield, for all positive x and y: 00

. 0 0

00

- \[\-F{u)]du=-^\[\-F{u)]du'-\[\-F{u)]du. m ^ m^ m^ x-\-y

X

(8.122)

y

This relation shows that function K defined by K{x) = - \[\-F{u)]du

(8.123)

mJ X

must satisfy the following functional relation: K{x-\-y) = K{x)'K{y). Since, from (8.123): i^(0) = l, and since K is continuous on IR"^, AT must be exponential, i.e.

(8.124) (8.125)

Renewal Theory

87

K(x) = e-^\ Now, (8.123) can be rewritten as

(8.126)

e-'^ =-{[l-F(u)]du. mJ m

(8.127)

By derivation we get ^e-'' =l[l-F{x)l m so that F(x) = l-mAe-'\ It follows that F has a density f(x) = mA'e-'\ And as

(8.128)

(8.129) (8.130)

00

jf(x)dx

= l,

(8.131)

0

we must necessarily have: m=j , so that finally, from (8.129): F(x) = l-e-^'' .

(8.132) D (8.133)

9 DELAYED AND STATIONARY RENEWAL PROCESSES The notion of stationary renewal process is a particular case of a delayed renewal process, A delayed renewal process is a renewal process with the difference that the first r.v. X,, though still independent of the others, does not have the same distribution. More precisely, let (X„, /7 > 1) be a sequence of non-negative independent variables, G being the d.f of all other r.v. The corresponding sequence (r„, n>0), where T,=0 a.s., (9.1)

r„=x,+... + x „

(9.2)

is called a delayed renewal sequence or delayed renewal process, Clearly the basic definition of the "classical" renewal processes can be extended to the case of a delayed renewal process. For example if H^{t) represents the renewal function for a delayed renewal process, and if we pose the condition X, = X, then we have:

Chapter 2

HAt\X,=x) = \^ ' '\) (9.3) {\ + H{t -x) , x
(9.6)

0

Or: (9.7) H,=G(t) + H.G(t). So, knowing renewal function H, we are just a convolution product away from knowing H^. This notion of delayed renewal process is introduced because it has been remarked that if one begins to observe a renewal process which has been running for a long time, the first variable X, observed is in fact the excess y at the starting time of the observations. We may suppose that the limit distribution of y is given by (8.48). Of course, the remaining r.v. X„, n>2 have F as d.f We thus have a particular delayed renewal process for which 1

^

G(x) = -{[\-F(u)]du,

x>0,

(9.8)

Such a process is called a stationary renewal process. The two main results concerning stationary renewal processes are related to renewal function H and to the distribution of the excess. Proposition 9.1 For every stationary renewal process characterized by d.f. (G,F) with a finite mean mforfi we have for all t\ HM--,

(9.9)

m H^ being the renewal function of the stationary renewal process. Proof If we take the Laplace Stieltjes transform of both members of (9.7), we get: (9.10) HAs)^G{s) + H{s)'G{sl with the convention that:

89

Renewal Theory

K(s) = je-^'dKix).

(9.11)

From the "classical" renewal equation (4.4), we deduce that: His) = F(s) + F(s)-H(s), or

(9.12)

(9.13) From (9.8) and the properties of the L-S transform, we obtain: G(s) =

1m J

-\e-'"[l-Fiu)]du

(9.14)

m,

1

-

je-'"F(u)du

(9.15)

An integration by parts leads to: (9.16) G(.) = . l ^ - ^ ( ^ ) m s Replacing H{s) and G(s) by expression (9.13) and (9.16) in equality (9.10), we find: 1 1 - F(s) (9.17) His) = 1 \-F{s), m Or, after simplification: ms But we know that [e-'^dx = -

(9.18)

(9.19)

It follows thus that the inverse Laplace Sfieltjes transform of (9.18) yields: m

u (9.20)

Proposition 9.1 has an interesting meaning. Indeed the value of H^ is asymptotically true for every renewal process, but here in the case of a stationary renewal process, the asymptotical expression is true for every t. Now, let /,(t) be excess at time t for the stationary renewal process and: F;^,„(X) = 1-FJX).

(9.21)

90

Chapter 2

Proposition 9.2 (i) For every delayed renewal process characterized by the distribution functions (G,F), with a finite mean mfor F, we have for all t: P{r(t)<x)

= G(t + x)-

j[l-F(t

+ x-u)]dH,(u)

.

(9.22)

[0,/]

(ii) If moreover, the renewal process is stationary, then, for all t: P{r(t) <x) = -j[l-F(u)]du

.

(9.23)

m Q

Proof (i) Conditioning with respect to the value of X,, (X.-t, t<X., {y{t-X,), t>X,. Since, for / > x: P(X,>y\X,>t)^^j^,

(9.25)

we deduce from (9.24) that: P{rs(0>x) = [ 1 - G ( 0 ] P ( ^ , -t>x\X,>t)+

' jP{r(t-y)

(9.26) > x)dG(y)

0

= [l-G(t)]-^-=^^^^+]pir(t-y)>x)dG(y).

(9.27)

Or: /V!,) (x) - 1 - G(t + x)+ | F ; , . , ) (x)dG(y).

(9.28)

0

This equahty expresses /Y^(^) as a function of F/(/). This last function is known from Proposition 8.2, and can be written (by (8.17)) under the form: 77f(,)(x) = l - F ( / + x)+ j[\-F(t-u-x)]dH(u). (9.29) [0./]

To simplify notation, let us write l-Fit + x) = FM (9.29) thus takes the form: . F;^,„(x) = F,(t) + F,(t-u)*H(t) Coming back now to (9.28), we obtain: F,:„(x) ^\-G(t + x) + F,» G(t) + F,»H* G(/) or

(9.30) (9.31) (9.32)

Renewal Theory

91

F^^^,)(x) = l - G ( / + x) + F, • [ G + / / # G ] ( 0

.

(9.33)

The last term may be written using renewal function H, if we use relation (9.7): F;^,,,{X)

= 1 - G ( / + X) + F , • HAtl

(9.34)

which is what we needed to prove. (ii) By Proposition 9.1, (9.34) becomes, in the stationary case: F^',,„{x) ^\-G{t

+ x)+ \FXt-u)— 0

.

(9.35)

^

The change of variable u^ = t -u yields: F;^^,^{X) =

\-G{t + x) + - \F,{u)du,

(9.36)

As function G is now defined by (9.8), we also have: \[\-F{u)]du

F;(.(X) = 1 - -

+ '^ \[\-F{u)]du,

0

(9.37)

X

By the additive property of the integral related to the integration domain, we get: X

F^^^,.,{x) = \-\[\-F{u)]du,

(9.38)

0

or (9.23).

D

Part (ii) of Proposition 9.2 leads to the same conclusion as the preceding theorem: in the stationary case, the asymptotical distribution of the excess y{t) is the exact distribution for all t. This result implies some interesting corollaries. Corollary 9.1 For every stationary renewal process, we have, for all t: (i)Fs\„(x)::^-][\-Fiu)]du, (ii)P{rit)>x,S(t)>y)^-

(9.39) \[\-F{u)]du,

(9.40)

x+y

(iii) P{X^,,^,, <x) = - judFiu),

(9.41)

Proof Results (i) and (ii) follow directly from relations (8.50), (8.77), and result (9.22) from Proposition 9.2. For (iii), we use relation (8.4) showing that X^^,>^+] is the sum of the two r.v. S(t) and y(t). From Proposition 9.2 and from (9.39), we know that the

92

Chapter 2

distribution of the two-dimensional r.v. {y,S) is independent of /, and consequently that the same is true for the sum y{t) + S{t). For r > X,, we have: '^N{t)+\

— ^N{t-X^)+\'>

(9.42)

so that: P{X^(,)„ <x\t>X,) = P{Z;v(,-^,)^i ^x\t>X,). Letting / tend to +00, we get: ?{X,,„,,<x)

(9.43)

= -\[\-F{u)]du,

(9.44)

because the event {co',t>X^{co)] is asymptotically of probability 1. As the distribution of Xj^^^.^^^ is independent of t, result (9.41)is equivalent to (9.44). n Remark 9.1 As for any two-dimensional random variable {X,Y), we have (9.45) P{X<x, Yx)'P{Y>y)^-P{X>x, Y > y), We can apply this elementary result to the joined distribution of {y{t),d{t)), given by (9.40). Doing so, it is easy to see that

\\r.j[l-F(w)]Jw „ , . . , + - r ^^[\-F{u)]du, ' P{r{t)<x,d{t)
ifx>y,

=

(9.46) -\{\-F{u)\du^-

\\\-F{u)'\du,

\ix
10 NUMERICAL ASPECTS The renewal equation (4.4) can be solved, as shown in the previous sections, directly in some very special cases or by means of the Laplace or Laplace Stieltjes transforms in other cases. In this way the analytical solution of the integral equation (4.4) can be obtained. But in the largest part of real life applications the equation that is obtained by the model can't be solved by analytical methods. In these cases it is necessary to use a numerical approach to get the solution to the general renewal equation (4.4) in a bounded horizon time. One way to face the problem is to use the Laplace transform but it is then necessary to invert Laplace transforms to obtain the solution of the equation and unfortunately, it is well known that this inverse transform, is not stable numerically (see Bellman et al (1966), Csenki (1994)). So, the idea is to solve (4.4) applying numerical methods for integral equations without using the Laplace transform.

Renewal Theory

93

lO^l General Quadrature Methods We assume that the density function o f / o f F exists. So we have to solve the integral equation given by (4.5). Each generic quadrature formula can be written as (see Evans (1993)): kh

k

\f{t)dt = Y.^,,f(ih), 0

(10.1)

/=0

where h is the step length, k < N, k,N sN, w^ / are the weights related to the quadrature formula (10.1); they are functions of both the end point and the point in which we compute the function value. Furthermore, the number N is such that hk = t, hN = Y and 0
H{kh) = F{kh) + ^w,jH{kh-lh)f{lh\ k = l,,..,N, /=o where H represents an approximate value of the function H . In this way the following linear system is obtained: H(h)

-w,o//(//)/(0)

=

w,fi{0)fih) + F{h)

-w,,H(h)f(h)

=

w,^,H(0)f(2h) + F{2h)

H(kh} -w,^,H{kh)f{0) -w,,H((k-mf(h)

•••

-y^u-iff(h)f((k-m

H{2h) -w,,o^(2A)/(0)

(10.2)

(10.3) = w,,H(0)f(kh) +F(kh)

It is trivial to say that the system (10.3) admits solution if (10.4) l-w,^ofiO)^0, k = l,.„N. There is a theorem established in Baker (1977) giving conditions such that if h-^0, then H ^H . Before proposing the theorem, it is necessary to provide two lemmas that hold for real numbers and are proved in Baker (1977). Lemma 10.1 If \4,\\ /=0

+ B,

r = q, q + \,...

q>l,

(10.5)

94

Chapter 2 q-\

where A>0, B > 0, and ^\<^i\^^,

then:

\1\<(A^ + B)(l + Ay-', r = q,q + l,... q>l, Lemma 10.2 Suppose that: A = hL<0 md ph = x>0, then (l + Ay-"^ <e^ if p>q.

(10.7) (10.8)

Remark 10.1 From Lemmas 10.1 and 10.2, it results that: \^,\<(A^ + B)(l + Ay-"^ < (A^ + B)e^' = {hl^ + B)e^''. Theorem 10.1 Let F : [0,7] ^ M, H : [0,7] -> R and q e{l,,,,, N},N eN, such that Nh
0
(10.6)

(10.9)

(10.10)

(10.11)

(10.12) (10.13)

h

then kh

k

t' (h) = \H(kh - T)f(T)dT - ^ w,,H(kh - uh)f(uh), 0

(10.14)

"=0

(7''ih) = \t''(h)\,

(10.15)

T(h) = TAh)-maxy(h),

(10.16)

q
^(h) = X?j"(h).

(10.17)

«=0

Moreover, if we assume that \f{t)\ < c, for t e [0,7], then ^^^^^
mhWj^C]

given that mhw^c^ < 1 .

, )t = ^, ^ + 1,..., A^

(10.18)

Renewal Theory

95

For the proof of the theorem see Baker (1977). A generahsation of this theorem was proved in Corradi et al (2004). Remark 10.2 It follows that if Tj^(h) = o(¥),

h ^ co with fixed Nh

(soN -^co), (E(h) = o(h^~^) and hm Wj.^
10.2 Some Particular Formulas In this part some formulas of numerical solution of the renewal equation will be given. The formula (10.2) will be related to particular generahzed Newton-Cotes formulas. The equations obtained using the Simpson quadrature are: H(kh) = F(kh) + ^H(kh)f(0) + •fY.^ikh '='

- {IT - \)h)f{{2T - \)h) (10.20)

--I

+flj;^H{kh-2Th)f{2Th)

+ !^H(0)f(kh) .

Using the Bezout's formula we obtain: H{kh) = F{kh) + ^H{kh)f{0) + hY,H{kh-Th)f{Th^

(10.21)

Finally, by applying the most simple quadrature method (rectangle formulas), it is then possible to obtain two different formulas, one getting the value of the integrating function at the beginning of the interval and the other getting it at the end of the interval. In this way, we obtained the following relations: k

H(kh) = F{kh) + hY,H{kh - Th)f{Th),

(10.22)

r=l

H(kh) = F(kh) + hJ^H{kh-Th)f{Th).

(10.23)

r=0

Substituting the differential by means of the difference it respectively results that: k

H(kh) = F(kh)-\-Y,H{kh-Th){F{Th)-F((T-l)h)),

(10.24)

r=l

H(kh)^Fikh) + Y,f^{kh-Th){F{(T + l)h)-F((T)h))

.

(10.25)

96

Chapter 2

Supposing h = l in (10.24), we get: H(k)^F(k)

+ f^H{k-T){F{T)-F(T-l)).

(10.26)

r=l

Furthermore putting: y(^) = l

^^^^

^"^'

[F(T)-F(T-1)

(10 27)

T>0,

we get k

H(k) = F(k) + Y,H{k

- T)V(T),

(10.28)

r=l

that is the discrete time renewal equation see Feller ( 1957 vol 1 page 330). In the book by Freiberger and Grenander (1971), there is a similar approach to the numerical solution of the renewal equation but without any justification of the method. Now let if be a continuous time renewal function and {T„} the renewal times. If we set: T h (10.29) h and N'{t) = n if T'„
H'{kh) = F'{kh) + Y.^'i^^-^)h)v'{Th).

(10.31)

The renewal process T^^ is defined in the same probability space {Q,F,P)

of

T . Given co GQ the following result holds a.s. Theorem 10.2 The Tj^process converges to T„ \/co for /? -> 0. Proof By the definitions given in section 10.2 and (10.31) it results that: p[N\t)

= n\ = P{N(t) = n)

The formula (10.32) impHes: r; '''• >T^,

\fn,

(10.32) D (10.33)

//-»0

The obtained results show that it is possible to obtain the discrete time renewal equation by means of the discretization of the related continuous time, and that starting from the discrete one it is also possible to get the continuous one.

97

Renewal Theory

10.3 Numerical Approximation of the Poisson Process For the Poisson process for which F is an exponential distribution of parameter A we know that the renewal equation is: H(t) = ;it. (10.34) To evaluate our numerical method, we use the two rectangle formulas:

H(kh) = 1 - e-'"" + hXY, H{kh - rh) -Arhe

(10.35)

k-\

H{kh) = \-e-'''

+hXY,H{kh-Th)

e

Xrh

(10.36)

Taking into account the Mc Laurin expansion: (10.37) in both cases we get: H{kh) = ?ikh, that is the real value in our case. Here, relations (10.24) and (10.25) respectively become:

(10.38)

k

H(kh) = 1 - e - ' ' ' + J^H(kh -rh)

(1 - e - ' ' ' -1 + e-'^'-'^'),

(10.39)

H{kh) - 1 - e-^"' + ^7/(/t/? -vh)

(1 - e"^*^""' - 1 + e'^''

(10.40)

j-=0

And so k

H(kh) = 1 - e-^** + (e^* - 1 ) 2 ] H(kh - rh) f

(10.41)

e-''',

k-\

H{kh) = 1 - e-^'' + (1 - e-^'' )J^H(kh - Th)e-

(10.42)

Table 10.1 gives the results obtained by relations (10.35), (10.36), (10.41) and (10.42) respectively in columns 2, 4, 5, 6 with h = A, /I = 2; in column 3, we give values obtained by computing the value of the density function in the centre of the discretization intervals. (10.36) (10.41) (10.42) real value

beriods (10.35)

0 1 2 3 4 5 6

0

0

0

0

0

0.18127 0.35936 0.53433 0.70624 0.87513 1.04107

0.18127 0.36248 0.54364 0.72475 0.9058 1.0868

0.22659 0.45848 0.6958 0.93868 1.18724 1.44163

0.18127 0.36254 0.54381 0.72508 0.90635 1.08762

0.2214 0.44281 0.66421 0.88561 1.10701 1.32842

0 0.2 0.4 0.6 0.8 1 1.2

Chapter 2

98 1.20409 1.26774 1.70197 1 8 ^1.36426 ' 1.44862 1.9684

1.4 1.6 1.8 9 1.52163 2 10 1.67623 2.2 11 1.82813 2.4 12 1.97736 2.6 13 2.12398 14 2.26803 2.8 15 2.40956 3 16 2.54861 3.2 3.4 17 2.68522 18 2.81943 3.6 3.8 19 2.9513 4 20 3.08085 21 3.20813 4.2 4.4 22 3.33319 23 3.45605 4.6 ' 24 3.57676 J 4.8 3.69535 5 25 1 Table 10.1: comparison between numerical and process data 1.26888 1.45015 1.62946 2.24107 1.63142 1.81023 2.52013 1.81269 1.99096 2.80572 1.99396 2.17162 3.098 2.17523 2.35224 3.39712 2.3565 2.5328 3.70325 2.53777 2.7133 4.01654 2.71904 2.89375 4.33717 2.90031 3.07415 4.6653 3.08158 3.25449 5.00112 3.26285 3.43478 5.3448 3.44412 3.61501 5.69653 3.62538 3.79519 6.05649 3.80665 3.97531 6.42488 3.98792 4.15538 6.80189 4.16919 4.3354 j7.18774, 4.35046 4.51536 7.58261 4.53173

1.54982 I 1.77122 1.99262 2.21403 2.43543 2.65683 2.87824 3.09964 3.32104 3.54244 3.76385 3.98525 4.20665 4.42806 4.64946 4.87086 5.09226 5.31367 5.53507

10.4 A Real Life Example in Motor Car Accidents. 10.4.1 Data Description. In this paragraph the renewal equation is applied to an actuarial real data application. We wish to outline that the renewal process can be applied in a very general case utilising data from statistical observations. The formula that we shall use in this case is given by relation (10.28). We shall apply the renewal theory in the case of motorcar accidents. In a motorcar insurance contract each time that the insured causes an accident, the insurance company will pay the damage done. It is possible to think that once the insured car is repaired all the conditions are renewed as at the beginning of the contract and the renewal theory can be applied. We have the raw data of accidents that an insurance company observed over roughly a period of 50 years. From these data it is possible to construct the discrete time increasing d.f of the renewal time of the motorcar accident. As a whole we had the data of 156,428 insured persons, and among them 22,395 had at least one accident during the time of subscription. In the file there were no data concerning the insurance subscription dates. We construct two vectors. In the first one, we count the number of insured persons that had the second accident in 1 year, in 2 years,... from the first. In the

99

Renewal Theory

second we count the number of insured persons that had the third accident in 1 year, in 2 years,... from the second. More precisely, at the beginning, we set all the elements of the vectors equal to 0. We add 1 at the nth element of the first vector when the second accident is verified after n-1 and before n years by the first one. In the same way, we add 1 at the nth element of the second vector when the third accident is verified after n-l and before n years by the second one. At the end of this procedure, the nth element of vectors will give the number of accidents during the nth years from the previous one, respectively the second after the first and the third after the second. The results obtained in this way are reported in Table 10.2. For example, the number 1,576 in the third row of this table we have represented the number of the second accidents that took place after two years and before the third year by the first accident and similarly, the number 226 represents the number of the third accidents that took place after two years and before the third year by the second accident.

1 # years

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26

1153^"^288-3 695 1576 1549 1344

928 619 386 278 189 139 101 77 40 36 15 11 11 10 7 14 6 8 7 7 2

126 226 224 172 176 138 107 86 69 61 33 21 15 16 7 3 4 3 2 1

1-2 + 2-3

241 821 1 802 1773 1516 1 104

757 493 364 258 200 134 98 55 52 22 14 15 13 9 15 6 8 7 7

2

1

Chapter 2

100 27 28 29 30 31 32 33 34 35 36 37

4 3 5 3 2 1 1 1

total

8228

4 3 5 3 2 1 1 1 1578

1

9 806

Table 10.2: number of accidents taking place in one year In Table 10.3, we report the yearly mean number and the variance related to the three distributions of Table 10.2.

1-2 + 2-3 1-2 2-3 mean 5.8676 5.4293 5.3453 1 variance 10.8680 11.7207 11.0421 Tab le 10.3: yeairly m e a n and variainee of accid

In Table 10.4, we report the distributions of the frequencies of the three vectors given in Table 10.2. The columns are obtained by dividing the number of each column by the total number of considered accidents in the vector (last element). Though mean and variance are unequal, the three histograms that can be obtained by Table 10.4 have a shape similar to that of the Poisson process. # years

1-2

2-3

1 2 3 4 5 6 7 8 9 10 11 12 13 14

0.018595 0.084468 0.191541 0.188260 0.163345 0.112786 0.075231 0.046913 0.033787 0.022970 0.016894 0.012275 0.009358 0.004861

0.055767 0.079848 0.143219 0.141952 0.108999 0.111534 0.087452 0.067807 0.054499 0.043726 0.038657 0,020913 0.013308 0.009506

1-2 + 2-3 0.02457679 0.08372425 0.183765042 0.180807669 0.154599225 0.112584132 0.077197634 0.050275342 0.037120131 0.026310422 0.020395676 0.013665103 0.009993881 0.005608811 1

Renewal Theory

101

1 16 ^^0.004375 0.001823 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33

0.010139 0.004436 0.001337 0.001901 0.001337 0.002535 0.001215 0.001901 0.000851 0.001267 0.001702 0.000634 0.000729 0.000972 0.000851 0.000851 0.000243

0 0.000486 0.000365 0.000608 0.000365 1 0.000243 0.000122

0 0.000407914 0.000305935 0.000509892 0.000305935 0.000203957 0.000101978

0

0

0.000122

0.000101978

0

0

1 34

35 36

1 0.005302876 0.002243524 0.001427697 0.001529676 0.001325719 0.000917805 0.001529676 0.00061187 0.000815827 0.000713849 0.000713849 0.000203957

0.000101978 1 37 1 0.000122 1 TabI e 10.4: frequency dis tribution

The observation of data leads to the following considerations: i) all three distributions have the same shape, ii) these distributions have the Poisson shape but the variance is about the double of the mean, iii) the mean and the variance of the first two columns of Table 10.2 are similar. In this light, we can say that the renewal hypothesis is acceptable because of the similar shape and the parameters of the first two distributions. It is possible to suppose that after an accident the process is renewed and the behaviour of the insured people will be the same. To get more reliable data we decided to put together the observations of the first and second distributions, obtaining the third column of each table. 10.4.2 The Result Distribution Now we shall consider the frequencies of the last column of Table 10.4 as the "probability" that a new accident will happen after n-l and before n years from the previous one. The nth result can be considered as an evaluation of the "probability" that there will be an accident between the (nA)th and the ^7th years,

Chapter 2

102

given that at least one accident took place, as already said above, we do not have data related to the dates of first contract subscription. We use the time interval between two subsequent accidents. Values shown in the second column are "conditional probabilities" to the event of having at least one accident as described before. To apply the renewal model we need to construct the d.f that gives the probability of having an accident within n years. The "probability" of having at least one accident is obtained from the raw data. It is obtained by dividing the number of persons that had at least one accident (22,395) by the total number of the insured persons (156,428). 9,806 accidents are considered. They represent the number of accidents that were caused after another accident. We define the following events for an insured person: J?: cause at least two accidents during the observed period, (By{, cause another accident after the {nA)\h and before the nth year from the previous one, C cause at least one accident over the full subscription time (probability:22,395/156,428 (=0.14316)). We wish to estimate the following probabilities for the interarrival of successive accidents: (Dyi'. cause an accident after the («-l)th and before the ^th yQaxfrom the contract subscription or from the previous one. The nXh element of the second column Table 10.4 that is equal to the fourth of Table 10.3 gives the following value: n^lM (10.43) but we are interested in: P[0„/C]. (10.44) In the data, as already specified, we don't have the contract subscription date. We are interested to know the probabihties P[©^]. Under the renewal assumption, we can say that: Then by means of Bayes' theorem, it is possible to reconstruct the "probabilities" P[(D^] to have the first accident after one year, two years and so on. These results are reported in the third column of Table 10.5. year Cond. Freq Frequencies Cum. Freq 0.003519 1 0.024577 0.003519 2 0.083724 0.011986 0.015505 0.041814 0.183765 0.026309 3

mean numb. 0.003519 0.01551738 0.04191078

Renewal Theory

103

0.067699 1 0.06812506 4 1 0.180808 0.025885 5 0.09107341 0.089832 0.154599 0.022133 0.10866922 6 0.10595 0.112584 0.016118 0.12175334 0.117002 7 0.077198 0.011052 8 0.13130704 0.1242 0.050275 0.007198 9 0.13903198 0.03712 0.005314 0.129514 0.14506757 0.133281 10 0.02631 0.003767 0.00292 0.15000748 11 0.020396 0.136201 0.15371498 17 0.013665 0.001956 0.138157 0.15663962 0.009994 0.001431 0.139588 n 0.15870604 0.005609 0.000803 14 0.140391 0.16051876 15 0.005303 0.000759 0.14115 16 0.16170875 0.002244 0.000321 0.141471 0.16261899 17 0.001428 0.000204 0.141676 0.16340732 18 0.00153 0.000219 0.141895 19 0.16404851 0.001326 0.00019 0.142085 20 0.16453603 0.142216 0.000918 0.000131 21 0.16503802 0.00153 0.000219 0.142435 22 0.000612 8.76E-05 0.16535298 0.142523 23 0.16565263 0.000816 0.000117 0.142639 24 0.142742 0.16590744 0.000714 0.000102 25 0.142844 0.16613383 0.000714 0.000102 0.16626641 0.000204 2.92E-05 26 0.142873 27 0 0.0 0.142873 0.16635287 28 0.142931 0.000408 5.84E-05 0.16648298 29 0.000306 4.38E-05 0.16658543 0.142975 30 7.3E-05 0.16670596 0.00051 0.143048 31 0.143092 0.000306 4.38E-05 0.16679003 32 0.143121 0.000204 2.92E-05 0.16685396 33 0.000102 1.46E-05 0.16690042 0.143136 34 0 0.0 0.16692826 0.143136 0.000102 1.46E-05 35 0.14315 0.16696609 36 0.0 0 0.14315 0.16698588 37 1 0.000102 1 1.46E-05 1 0.143165 1 0.16701685 1 Table 10.5: the accidents renewal results In the fourth column of this last table the ' 'increasing d.f." obtained by the elements of the third column is reported. In this case the nth element represents the "probability" of having the first accident within n years. These data represent the d.f that we need in the discrete time renewal equation to evaluate the mean number of accidents that an insured person will have within 1, 2,..., n, years. These results are obtained by solving the discrete time renewal equation (36) and are reported in the last column of Table 10.4.

104

Chapter 2

Here our results show that the mean number of accidents is low. Let us recall that we used real data obtained by an insurance company and moreover this is in keeping with the value of the probability of having at least one accident during the full subscription time being, as we give above, 0.14316. If anyone thinks that the results are low in the light of his experience we will be glad to have other data and evaluate them by means of our model.

Chapter 3 MARKOV CHAINS This chapter presents briefly some fundamental results concerning the theory of Markov chains with a finite number of states. These results will be used in the following chapter. We will use the usual terminology introduced by Chung (1960) and Parzen (1962).

1 DEFINITIONS Let us consider an economic or physical system S with m possible states, represented by the set / : (1.1) I = {1 2,..., m } . Let the system S evolve randomly in discrete time (^ = 0, 1, 2,..., n,...), and let /„ be the r.v. representing the state of the system S at time n. Definition 1.1 The random sequence (J„,

/? G N ) w a Markov chain iff for all

7o J 7i J • • • J 7« ^ -^ •

P{Jn

= Jn I "^0 = Jo,

A = JW",

Jn-\ = Jn-\ ) = P{Jn=

Jn I ^n-\

= Jn-l ) ( 1 - 2 )

(provided this probability has meaning) Definition 1.2 A Markov chain {J„, n>0) is homogeneous iff the probabilities (1.2) do not depend on n and is non-homogeneous in the other cases. For the moment, we will only consider the homogeneous case for which we write: P{Jn=j\Jn-^-i)-P,, (1.3) and we introduce the matrix P defined as: The elements of the matrix P have the following properties: (i) p^. >0, for all/, y e / ,

(1.5)

( i i ) ^ ^ . , =1, for all / G / .

(1.6)

ye/

A matrix P satisfying these two conditions is called a Markov matrix or a transition matrix.

106

Chapters

To every transition matrix, we can associate a transition graph where vertices represent states. There exists an arc between vertices / andy iff p^j > 0.

Figure 1.1: py > 0 To fully define the evolution of a Markov chain, it is also necessary to fix an initial distribution for state jQ,i.Q. a vector P = iPl,'",Pm), such that: Pi>0, iel,

(1-7)

ZA=1.

(1.9)

(1.8)

iel

For all /,/?, represents the m///a//^roZ?aZ>///(y of starting from /: p,=P{J,=i), (1.10) For the rest of this chapter we will consider homogeneous Markov chains as being characterized by the couple ( p , P ) . If J„ = i a.s., that is if the system starts with probabihty equal to 1 from state /, then the components of vector p will be: Pj=S,. (1.11) We now introduce the transition probabilities of order pjf^, defined as: Plr'=PU.n=J\J.=i)' (1.12) From the Markov property (1.2), it is clear that conditioning with respect to J^^^, we get PIP=TP!^P'9-

0-13)

k

Using the following matrix notation:

p^^'-[pr]>

(1-14)

we find that relation (1.13) is equivalent to P^'^ = P ^ Using induction, it is easy to prove that if

(1.15)

P«=[/7W],

(1.16)

Markov Chain

107

then we obtain for all « > 1: P^") = P " . (1.17) Note that (1.17) implies that the transition probability matrix in n steps is equal to the nih power of the matrix P. For the marginal distributions related to J„, we define for / G / and n>0: p,{n) = P{J„=i). (1.18) These probabilities may be computed as follows:

Piin) = Y^Pjpf,

iel.

(1.19)

J

If we write: p^J^^=Sj,or¥<'^=l, then relation (1.19) is true for all « > 0. If: p(n) = {p(n),...,p„in)), then relation (1.19) can be expressed, using matrix notation, as: p(«) = pP".

(1.20)

(1.21) (1.22)

Definition 1.3 A Markov matrix P is regular if there exists a positive integer k , such that all the elements of the matrix P^^^ are strictly positive. From relation (1.17), P is regular iff there exists an integer A: > 0 such that all the elements of the kXh power of P are strictly positive. Example 1.1 (i)If: .5 .5' P= 1 0_ we have: •.75 .25" .5 .5 _ so that P is regular. The transition graph associated to P is given in Figure 1.2. (ii)If: 1 0• P= .75 .25. P is not regular, because for any integer k , ;7«=0. The transition graph in this case is depicted in Figure 1.3.

(1.23)

(1.24)

(1.25)

(1.26)

Chapter 3

108

Figure 1.2 The same is true for the matrix: P=

0 1 1 0

(1.27)

.25 .75

Figure 1.3 (iii) Any matrix P For example: "1 2" 3 3 1 3 .4 4.

whose elements are all strictly positive is regular. .7 .2 . r .6 .2 .2 .4 .1 .5

(1.28)

2 MARKOV CHAIN STATE CLASSIFICATION 2.1 Periodic And Aperiodic States Let / e / , and let d(i) be the greatest common divisor of the set of integers n, such that pi"'>0. (2.1) Definition 2.1 If d{i) > 1, the state i is said to be periodic with period d(i). If d{i) = 1, then state i is aperiodic.

Markov Chain

109

Clearly, if p^^ > 0, then / is aperiodic. However, the converse is not necessarily true. Remark 2.1 If P is regular, then all the states are aperiodic. Definition 2.2 A Markov chain, whose states are all aperiodic is called an aperiodic Markov chain, From now on, we will have only Markov chains of this type.

2.2 Essential And Inessential States - Irreducibility Definition 13 A state i is said to lead to state j (written i > j) iff there exists a positive integer n such that A^>0. (2.2) /1^ j means that i does not lead toj. Definition 2.4 States i and j are said to communicate iff i > j and j > i, or if j = i. We write i O j . Definition 2.5 A state i is said to be essential iff it communicates with every state it leads to; otherwise it is called inessential. The relation <> defines an equivalence relation over the state space / resulting in a partition of / . The equivalence class containing state / is represented by

C(0. Definition 2.6 A Markov chain is said to be irreducible iff there exists only one equivalence class. Clearly, if P is regular, the Markov chain is both irreducible and aperiodic. Such a Markov chain is said to be ergodic. It is easy to show that if the state / is essential (inessential) then all the elements of the class C(/) are essential (inessential) (see Chung (I960)). We can thus speak of essential and inessential classes. Definition 2.7 A subset E of the state space I is said to be closed iff ^Pij = I, for all i e E.

(2.3)

JeE

It can be shown that every essential class is minimally closed. See Chung (1960).

no

Chapters

2.3 Transient And Recurrent States Definition 2.8 For given states i and j , with JQ = i, we can define the r.v. r^j called the first passage time to state j as follows: ^ ^{ri ifJ.^j, 0 0. Tij is said to be the hitting time of the singleton {j}, starting from state / at time 0. Supposing: fi"^=P{Ty=n\Jo=i), neNo (2.5) and f, =P{T,j 0: f<"^=P{J„=j, J^^j, 0
(2.7)

where the summation set S^^jis defined as: S\^ij ={{aQ,a^,...,a,): We also have:

a^ =i,a, = j,a,, el,ak

^ j , k = l,,..,n-l},

(2.9)

/7 = 1

l-fy=P{T,=<^\J,=i).

(2.11)

The elements fj"^ can readily be computed by induction, using the following relations: P,-f^'\ (2.12)

pr*=Z#vr^+y;^"'>

«^2.

Let: m,=E{T,\J,=i), with the possibility of an infinite mean. The value of my is given by:

(2.13) (2.14)

Markov Chain

111

m,=Y.nf^''-^{\-f,).^'

(2.15)

If / = 7 , then w,y is called the first passage time mean or the mean recurrence time of state /. For every 7, we define the sequence of successive return times to state j {^n^\ « > a ) as follows: ro^^)=0, rJ^)=sup{^ENo,

(2.16) /:>ri^?, / . ^ 7 , r i ^ ? < v < ^ } ,

n>0,

(2.17)

k

Using the Markov property and supposing JQ = j , the sequence of return times to state j is a renewal sequence with the r.v. ry^-r^l}, n>l (2.18) distributed according to TJJ . If JQ = /, / ^ j , then (/;^^\ n>0) is a general renewal sequence. In this case: ry'=T,, (2.19) and ry^-r^ll-T^, n>L (2.20) This shows that a Markov chain contains many embedded renewal processes. These processes are used to define the next classification of states. Definition 2.9 A state i is said to be transient (recurrent) if the renewal process associated with its successive return times to i is transient (recurrent). A direct consequence of this definition is that: / transient <=> fa / , = 1 . A recurrent state / is said to be null (positive) if m^ = oo (m,, < oo). It can be shown that if m^, < oo, then we can only have positive recurrent states. This classification leads to the decomposition theorem (see Chung (I960)). Proposition 2.1 (Decomposition theorem) The state space I of any Markov chain can be decomposed into r (r >\) subsets C,,...,Q forming a partition, such that each subset Q is one and only one of the following types: (i) an essential recurrent positive closed set, ^*^ Using the following conventions: 00 + a = 00, a G R, oo • a = oo, (a> 0), and in this particular case, oo • 0 = 0 ,

112

Chapters

(ii) an inessential transient non-closed set. Remark 2.2 (1) If an inessential class reduces to a singleton {/}, there are two possibilities: a) There exists a positive integer N such that: 0
Y^pr<^.

(2.25)

In this case, for all k e I:

f^pif<^,

(2.26)

and in particular: \impif=0, \/keL

(2.27)

«-^oo

(ii) State i is recurrent iff

In this case: 00

koi^

Y,pif = «^,

(2.29)

and 00

k^i=>Y.P'^'-Q. «=1

(2.30)

Markov Chain

113

3 OCCUPATION TIMES For any state y, and for^ e No, we define the r.v. Nj{n) as the number of times the state j is occupied in the first n transitions: Nj{n) = #{ke{\,,„,n]'.J,=j], (3.1) By definition, the r.v. Nj{ri) is called the occupation time of state j in the first n transitions. The r.v. NX<^) = \imNAn) (3.2) is called the total occupation time of state j . For any state j and /? G NQ let US define:

^^(">=ll 'ftI'-

(^-^^

We may write:

^;(«) = Z^yM-

(3-4)

We have from relation (2.6): P(A^/<x>)>0|/o=0 = / r (3-5) Let g-^j be the conditional probability of an infinite number of visits to the state j , starting with JQ =i\ that is: g^j=p{Nj{^) = ^\J,=i), It can be shown that: g«=limy;/"\

(3.6) (3.7)

A7->00

§,=frSjj.

(3.8)

g.. = 1 <^ /:. = 1 <=> / is recurrent, (3.9) g.. = 0 <^ y;, < 1 <::> / is transient. (3.10) Result (3.9) can be interpreted as showing that the system S will visit a recurrent state an infinite number of times, and that it will visit a transient state a finite number of times.

4 COMPUTATION OF ABSORPTION PROBABILITIES Proposition 4.1 (i) Ifi is recurrent and if j ^ C(i), then fj = 1. (ii) Ifi is recurrent and if j € C(i), then fj = 0 .

114

Chapter 3

Proof (i) From the definition of the probabilities gy (see relation (3.6)), we have, for all positive integers n: go=Zpi"'g,>

(4.1)

kel

or kel

As / <> j and state / is recurrent by assumption, we know by Proposition 2.1 that state y is also recurrent. Thus, by (3.9) gjj = 1. From (4.2), it follows that for all ^ > 0 : P^l\l-g,) =0, (4.3) As j > i, there exists a positive integer N such that: PI/"^ > 0. (4.4) Using relation (4.3) for k = i ,WQ get: ^,=1. (4.5) Hence, using relation (3.8): fy=l. (4.6) (ii) Since C(i) is a recurrent class, it is closed. Thus if j ^ C(i), then / does not lead toy and so: fij=0. D (4.7) Proposition 4.2 Let T be the set of all transient states of I, and let C be a recurrent class. For all 7, k e C, fo=ALabeling this common value asfc, linear system: f,c-t.P*fKc+Y,P.,, i^T. keT

the probabilities {f^c

i^T)

(4.8) satisfy the (4.9)

keC

Proof From relation (3.5), we have: fy=j;^P^,P{N,(^)>0\J,=k),

(4.10)

kel

or f.-ZP'^fv kel

Using the preceding proposition, we obtain:

(4-11)

Markov Chain

115

A C = Z A . A C + I;^.*, keT

i^T.

D(4.12)

keC

Remark 4.1 Parzen (1962) proved that under the assumption of Proposition 4.2, the linear system (4.9) has a unique solution. This shows, in particular, that if there is only one irreducible class C , then for all i eT : f,c=l(4.13) Definition 4.1 The probability f^c introduced in Proposition 4.2 is called absorption probability in class C, starting from state i. If class C is recurrent: ri i f / e C , fic=\ [0 if / is recurrent, / ^ C.

(4.14)

5 ASYMPTOTIC BEHAVIOUR Consider an irreducible aperiodic Markov chain which is positive recurrent. Suppose that the following limit exists: \impj(n) = /rj, jel (5.1) Starting with JQ = i. The relation (5.2) kel

becomes: Pij

~ ZjPik Pkj^

(5.3)

because Pji.n)-Pt'Since the state space / is finite, we obtain from (5.1) and (5.3):

(5.4) (5.5)

kel

and from (5.4): (5.6) iel

The result: lim;.«=;r,

(5.7)

is called an ergodic result, since the value of the limit in (5.7) is independent of the initial state /. From result (5.7) and (1.19), we see that for any initial distribution p:

116

Chapter 3

\imp,{n) = \^rnY^PjPf,

(5.8)

J

= Z/'y^-

(5.9)

SO that: lim/?,(f7) = ;r,.

(5.10)

/7->00

This shows that the asymptotic behaviour of a Markov chain is given by the existence (or non-existence) of the Umit of the matrix P". A standard result concerning the asymptotic behaviour of P" is given in the next proposition. The proof can be found in Chung (1960), Parzen (1962) or Feller (1957). Proposition 5.1 For any aperiodic Markov chain of transition matrix P and having a finite number of states, we have: a) if state j is recurrent (necessarily positive), then (i)

i^C{j)^\imp\p=—,

(5.11) "JJ

(ii) / recurrent and ^ C(j) => \impj"^ =0,

(5.12)

f

(iii) i transient limpj"^ ^:ii£Ul^

(5.13)

b) Ifj is transient, then for all i e I: limpj"^ =0.

(5.14)

Remark 5.1 1) Result (ii) of part a) is trivial since in this case: p^f' - 0 for all positive n, 2) Result (5.14) in part b) is identical to result (2.27) of Proposition 2.2. From Proposition 5.1, the following corollaries can be deduced. Corollary 5.1 {Irreducible case) If the Markov chain of transition matrix P is irreducible, then for all i,j e / : ]hnpf=7tj,

(5.15)

with 7tj= — .

(5.16)

It follows that for all j : 7tj>^.

(5.17)

Markov Chain

117

If we use Remark 4.1 in the particular case where we have only one recurrent class and where the states are transient (the so called uni-reducible case), then we have the following corollary: Corollary 5.2 (Uni-reducible case) If the Markov chain of transition matrix P has one essential class C {necessarily recurrent positive) and T as transient set, then we have: (5.18) (i) for all /, 7 e C : \\mp\"^ = TT:, with [TTJ,] e C] being the unique solution of the system: (5.19) ieC

(5.20) (ii) For all j

eT\

\\mp\f^ = O f o r a l l / G / .

(5.21)

(iii) For all j eC : limpjf ^TTj for a l l / s T .

(5.22)

Remark 5.2 Relations (5.19) and (5.20) are true because the set C of recurrent states can be seen as a Markov sub-chain of the initial chain. If the £ transient states belong to the set {!,.,.,£}, using a permutation of the set / , if necessary, then the matrix P takes the following form: 1 ••• £ £ + 1 ••• m Pn

Pi2

(5.23) £+ 1 O m

This proves that the sub-matrix P22 is itself a Markov transition matrix. Let us now consider a Markov chain of matrix P. The general case is given by a partition of/:

/ = ruc,U---UQ,

(5.24)

where T is the set of transient states and Ci,...,Q the r positive recurrent classes.

Chapter 3

118

By reorganizing the order of the elements of / , we can always suppose that T = {\,...,1), (5.25) (5.26) C,={£ + \,...,i + v,}, C,={l + v,+\,...,l + v,+v,}, (5.27) r-\

(5.28)

c = U+Zv,+i,-.., m j=\

where Vj is the number of elements in Cy, (y = 1,..., r ) and

^+Z-. y=i

(5.29)

m.

This results from the following block partition of matrix P: 0

0

0

P=

0

0 P

0

(5.30)

V2XV2

0 0 0 where, for j = 1,..., r : Vf^i is the transition sub-matrix for T, P<x^^ is the transition sub-matrix from T to Cj, Pyjxvj is the transition sub-matrix for the class Cj. From Proposition 5.1, we have the following corollary: Corollary 5.3 For a general Markov chain of matrix P, given by (5.30), we have: (5.31) (i) For all i e / and all j eT: limpf"^ = 0. in) For all j ^ C^

(v = l,..., r ) :

{KJ

if/eQ,

\imp\^ =\Q

if/GC,.

A7->00

v'^v,

(5.32)

J

[fi,cXj ifi^T. Moreover, for all v = 1,..., r : Z<=1-

(5-33)

This last result allows us to calculate the limit values quite simply. For (;rj, 7 e Q ), v = 1,..., r , it suffices to solve the linear systems for each fixed V :

Markov Chain

119

r^^

(5.34)

Indeed, since each Q is itself a space set of an irreducible Markov chain of matrix P^,^^, the above relations are none other than (5.5) and (5.6). v = 1,..., r , it suffices to solve For the absorption probabilities(/^ ? i ^T), the following linear system for each fixed v . Using Proposition 4.2, we have: keC^

keT

6 EXAMPLES Markov chains appear in many practical problems in such fields as operations research, business, social sciences, etc. To give an idea of this potential, we will present some simple examples followed by a fully developed case study in the domain of social insurance. (i) Let P be given by

".5 .5"

p=

(6.1)

.1 0 . Since r.75 .25" P^L -5 •5 .

(6.2)

the matrix P is regular and hence irreducible and aperiodic. The limiting distribution (TT], 712) satisfies the system (5.5), (5.6): .5;r, ^1

+

^2

=

712

~

1 •

(6.3)

The unique solution is: 2 1 ^1 = ^

.

^2=3'

(6.4)

Using (5.16), this result implies that the values of the mean return states are: 3 ^11

=:T

'

^^22 = 3 .

(6.5)

(ii) A transportation problem, (Anton & Kolman (1978)). Let us consider a taxicab company of a city V, subdivided into three sectors V,,

V2 a n d K 3 .

120

Chapters

A taxicab picks up a passenger in any sector and drops her or him off in any sector. We can view a taxicab as a physical system S which can be in one of three states: the sectors Fj, F2 or F3. The observation of taxicabs leads to the construction of a Markov chain with three states. This Markov chain might have the following matrix P, for example: 0.5 0.4 0.1] 0.3 0.6 0.1 . (6.6)

0.2 0.1 O.7J This matrix is regular, hence irreducible and aperiodic since all its elements are strictly positive. (iii) A management problem in an insurance company A car insurance company classifies its customers in three groups: GQ : Those having no accidents during the year, G|: Those having one accident during the year, G2 : Those having more than one accident during the year. The statistics department of the company observes that the annual transition between the three groups can be represented by a Markov chain with state space {Go, Gj, G2} and transition matrix P: r.85 .10 .05] P = 0 .80 .20 . (6.7)

[0

0

1 J

We suppose that the company produces 50,000 new contracts per year and wants to know the distribution of these contracts for the next four years. After one year, one has, on average: in group GQ : 50,000 x .85 = 42,500, in group G, : 50,000 x .10 = 5,000 , in group G^ : 50,000 x .05 = 2,500. These results are simply the elements of the first row of P, multiplied by 50,000. After two years, multiplying the elements of the first row of P^^^ by 50,000, we get in group GQ : 36,125, in group Gj : 8,250, in group G2 : 5,625. A similar computation gives:

Markov Chain

121 After three years 30,706 Go 10,213 G, 9,081 G3

After four years 26,100 11,241 12,659

To find the type of the Markov chain with transition matrix (6.7), the simple graph of possible transitions given in Figure 6.1 shows that the class {1, 2} is transient and class {3} is absorbing. Thus, using Corollary 5.2 we obtain the limit matrix

ro 0 11 A=

0 0 1 .

(6.8)

[0 0 ij The limit matrix can be interpreted as showing that regardless of the initial composition of the group; the customers will finish by having at least two accidents.

Figure 6.1. (iv) A heating system- a home management problem A survey on heating systems for family homes (fuel oil, gas, electricity) shows that the transfers from one system to another can be represented by a Markov chain with three states:

122

Chapters

1: fuel oil, 2: gas, 3: electricity and can be characterized by the following matrix: .825 .175 0 P = .60 .919 .021

(6.9)

.049 0 .951 The problem to be solved is the following: if the current distribution of heating systems is 26% for fuel oil, 60% for gas and 14% for electricity, what will be the final distribution of the market? To solve this problem we only need to compute the asymptotical value of P. The transition graph related to P shows that the matrix is irreducible and aperiodic. Solving the system (5.5), (5.6), we get: ;ri-.244, 7^2 = .529, 71,^221. (6.10) That is, approximately, 24% for fuel oil, 53% for gas and 23% for electricity. Remark 6.1 If one wants to know the situation after one or two changes, one can use relation (1.19) with /7 = 1,2,3 and with p given by: p = (.26,.60,.14). (6.11) One obtains the following results: p\'^ =.257 p^^ =,591 pi'^ =.146 PI'^=255

/?f^=.594

/7f^=.151

;?P^=.254

p^2'^=,590 pi'^ = A56.

These results show that the convergence of p^"^ to TT is relatively fast.

7 A CASE STUDY IN SOCIAL INSURANCE (Janssen (1966)) To compute insurance or pension premiums for professional diseases such as siUcosis, we need to compute the average (mean) degree of disability at preassigned time periods. Let us suppose we retain m degrees of disability: ^1 3 • • • J

S^,

the last being 100% and including the pension paid out at death. Let us suppose, as Yntema (1962) did, that an insurance policy holder can go from degree Si to degree Sj with a probability p^ . This strong assumption leads to the construction of a Markov chain model in which the mxm matrix: P = [p,j] is the transition matrix related to the degree of disability.

(v.i)

Markov Chain

123

For individuals starting at time 0 with 5, as the degree of disability, the mean degree of disability after the nth transition is: m

To study the financial equilibrium of the funds, we must compute the limiting value of S^ (n): S, = ]imS,(n),

(7.3)

/7->00

or 5,-lim£;.WS,.

(7.4)

This value can be found by applying Corollary 5.3 for / = 1,..., m . Numerical example Using real-Hfe data for sihcosis, Yntema (1962) began with the following intermediate degrees of disability:

s,----10% Sj = 30% ^3 - 50% ^4 = 70% ^5 = 100%

Using real observations recorded in the Netherlands, he considered the following transition matrix P: '.90 .10 0 0 0^ 0 .95 .05 0 0 P- 0 0 .90 .05 .05 ; (7.5) 0 0 0 .90 .10 0 0 .05 .05 .90 the transition graph associated with the matrix (7.5) is given in Figure 7.1: This immediately shows that: (i) all states are aperiodic, (ii) the set {5'3,iS'4,5'5} is an essential class (positive recurrent), (iii) the singleton {1} and {2} are two inessential transient classes. Hence a uni-reducible Markov chain can be associated with matrix P. We can thus apply Corollary 5.2. It follows from relation (7.4) that:

Chapter 3

124

(7.6) 7=3

where {TT^^.TT^.TI^) is the unique solution of the linear system: K-i

=

TC.

-

.05 • TT^ +

TTA

=

.05 • K^ +

1

.9'K2,

==

^3

+

O'TT^

+

.05-/Ts,

.9-7r,

+

.05-;r5,

.05 • K^ +

.9-7t„

+

n^.

+

^4

(7.7)

Figure 7.1 The solution is: 2 3 n.^-. Therefore:

(7.8)

Markov Chain

125

'^'^(|50 + |^<^ + ^100J%

(7.9)

or ^.=79% (7.10) which is the result obtained by Yntema. The relation (7.10) proves that the mean degree of disability is, at the limit, independent of the initial state /.

8 ASYMPTOTIC NUMERICAL TREATMENT 8.1 An Algorithm For MC Asymptotic Study In this part we present an algorithm useful for the full treatment of asymptotic behaviour of a Markov chain. It will be described following the social insurance example given in the previous paragraph. The algorithm, given in De Dominicis, Manca (1984b), is useful for the classification of the states of a Markov chain. It works on the digraph representing the process transitions from one state to another. To show the algorithm steps we will follow the example given in the previous section. Firstly, we will describe briefly the graph theory aspects following Christophidies(1975). Let r be the graph associated with the states space: (8.1) r = {x,,x^,..,,x^} where the node x. represents the /th state. r(x.) represents the set of nodes that can be reached in one single step from x., namely, we say that Xj e r(x.) if there is a single arc path direct from x, to Xj. T^(x.) represents the set of nodes that can be reached from x. via a path of length k. Finally R(x.) represents the set of all the nodes that can be reached from x^, i.e., supposing r^(x.) = [x^] it results that:

R(x,) = \Jr'(x,), where p<m-\

(8.2)

is the least integer such that

r''^'(x,)cUr*(x,). k=\

Now let A be the adjacency matrix of the graph. As known it results that:

(8.3)

Chapter 3

126

a..=\

fl

if

XieT(Xi), ' '

(8.4)

The adjacency matrix related to the graph in figure Figure 7.1 is given in the Table 8.1. states 1 2 3 4 5 1 1 0 0 0 1 2 0 1 1 0 0 0 0 1 1 1 3 4 0 0 0 1 1 0 0 1 1 jj 5 Table 8.1: adjacency matrix As can be seen in Table 8.1, the elements of value 1 are in the position of the elements that are not equal to 0 in the transition matrix (7.5). A can be treated as a Boolean matrix. Putting: A^=A^-^A (8.5) it results that: fl if XjeF'ix^), (8.6) 0 if XjiT'ix.). . « , = •

Then, denoting by R the reachability matrix of the graph defined as follows ^fl

if

x.ei?(x,),

'«'~[0

if

x.€R(x,),

(8.7)

we find that: R = A''vA'vA'v-vA'"-'. (8.8) The reachability matrix related to the adjacency matrix of Table 8.1 is given in Table 8.2. states 1

0 0 1 Table 8.2: Reachability matrix

Markov Chain

127

With the reachability matrix, it is possible to subdivide the states set of our process in equivalence classes. In fact given an equivalence class Q , it results that: x^.,x.EC,«^.=r.^.=l. (8.9) Relation (8.9) means that the nodes of a class form a circuit, and that it is the only circuit to which they belong to. Conventionally we assume that, if a class consists of a single node, then such a node is itself a circuit. In our example we have the following three classes: q={l}, Q - { 2 } , C3 ={3,4,5}. (8.10) Now it is possible to construct a partial order relation between classes C^. The following definition is introduced: def

Cp
,

.

or3|x,^,...,x,Jpathofstates,x,^ ^C^,x,^ GC^.

(8.11)

That means that C^ < C^ if it is possible to go from the class C^ to the class C^. In our case we have a total order because all the classes are comparable and moreover the natural order corresponds to the reverse of the order relation defined in (8.11). In fact, we obtain: C,
128

Chapters

The canonical form is written permuting the rows and the columns of the matrix taking into account the order of the classes. The rows and columns corresponding to the states of the maximal classes are put in the north-west comer of the matrix. In the successive step the rows and columns of the states corresponding to strictly transient classes are posed in the canonical form matrix. In this step the order between the strictly transient classes is preserved. This means that if a strictly transient class goes to another strictly transient class, then the first is put in the matrix before the second. In the last step the rows and the columns corresponding to the states of the absorbing classes are posed in the south-east comer of the matrix. In the irreducible case the canonical form will always be equal to the starting matrix. Once the matrix is written in canonical form, the algorithm studies the asymptotic behaviour of the MC matrix. It will take into consideration the classification of the process. If the Markov chain is irreducible the steady vector n will be computed and if the process is uni-reducible the limit vector of the unique absorbing class will be computed. If the process is reducible, then for each sub-matrix corresponding to the states of an absorbing class the limit vector will be found. Further the absorbing probabilities(/c,? i ^T), v = 1,..., r , will be found, where r is the number of absorbing classes. Once the absorbing probabilities have been obtained, the algorithm will compute the pl""^ n->co . In our example the MC described by the transition matrix (7.5) is uni-reducible and the limit vector of the only absorbing class is given in Table 8.3. states values

3 4 5 0.222222 0.333333 0.444445 Table 8.3: limit vector.

And we have found once more the results given in the previous paragraph. Now we summarize the main algorithm steps. Input - read the states number, the error for limit vector and the matrix P Compute - the adjacency matrix Compute - the reachability matrix Divide - the states in equivalence classes Find - the maximal classes Find - the absorbing classes Classify - the strictly transient classes Construct - the partial order between the classes Classify - the MC transition matrix

Markov Chain

129

Construct - the canonical form of the MC transition matrix Study - the MC asymptotical behaviour Irreducible case -findsthe transition matrix limit vector Uni-reducible case -findsthe limit vector of the only absorbing class sub-matrix Reducible case - for each absorbing classfindsthe limit vector the •00 . absorbing probabilities and the p\p We don't enter too much into the description of the algorithm because we used, mainly for the classification of the states, computer science tools that are beyond the scope of this book.

8.2 Irreducible Real Data Example In Motor Car Insurance This example will use a transition matrix related to the motorcar bonus mains insurance rules in force in Italy. In this case the Markov model fits quite well because: 1) the state of each insured person is given at the beginning of each year, 2) there are precise rules that give changes of states in function of the behaviour of the insured person during the year, 3) the future state depends only upon the present one. The number of states is 18 and we will find the limit vector of the transition matrix obtained from the data. Table 8.4 gives the evolution rules in force in Italy for bonus malus insurance contract. Arriving state in fimction of claims 1 Starting state 0 claims 1 claims 2 claims 3 claims 4 or more 12 9 6 3 1 1 4 1 2 13 10 7 14 11 8 5 2 3 12 4 15 9 6 3 16 13 10 7 4 5 17 14 11 8 6 5 12 18 15 9 6 7 18 16 13 10 7 8 18 17 14 11 8 9 12 18 18 15 9 10 11 18 18 16 13 10 14 11 12 18 18 17

Chapter 3

130 13 14 15 16 17 18

12 15 18 18 18 13 16 18 14 17 18 18 18 18 15 18 16 18 18 18 18 17 18 18 Table 8.4: Italian bonus malus evolution rules

18 18 18 18 18 18

We have three years historical data concerning 105627 insured persons. This means that we could consider 316881 real or virtual transitions. The Markov transition matrix that was obtained from the available data and taking into account the bonus malus Italian rules is given in tables 8.5.1, 8.5.2, 8.5.3. i 4 2 5 states 1 1 3 0 0 0.056264 0 1 0.941655 0 2 0.935097 0.062379 0 0 0.056611 0 0 0.941646 0 3 4 0 0 0 0.948892 0 0.945231 0 0 5 0 0 6 0 0 0 0 0.949204 0 0 0 0 0 7 0 0 0 8 0 0 9 0 0 0 0 0 0 0 10 0 0 0 11 0 0 0 0 0 12 0 0 0 0 0 0 0 0 0 13 0 0 14 0 0 0 0 15 0 0 0 0 0 0 0 0 0 16 0 0 0 0 17 0 0 0 0 0 0 0 1 18 Table 8.5.1: motor car transition matrix 1

states 7 1 0 2 0.002427 3 0 4 0 5 0.052354 6 0

8 9 0 0.000081 0 0 0.001574 0 0 0.001744 0 0 0.04908 0

10 0 0.000097 0 0 0.002314 0

11 0 0 0.000169 0 0 0.00157

6 0.001973 0 0 0.049364 0 0 0.934685 0 0 0 0 0 0 0 0 0 0

0

1

12 0.000027 0 0 0 0 0

Markov Chain

131

1^ 1^

1 8 ''10.92227 ^ 1 °0

1 0.00339 1 0.061856 0.073137 0 0 0 0 0.082621 0 0.914103 0 0.071989 0 0.923854 0 0 0 0.92933 0 0 0 0.930156 0 0 0 0 0.937854 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 1 0 1 0

0 0 0 0 0 0 0 0 0 0

9 10 11 12 13 14 15 16 17 18

Table 8.5.2 : motor car transition matrix 2

13 0 0 0 0

1 states

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17

14 0 0 0 0 0

0.000067

0 0

0.000146

0 0

0.004246

0.000069

16 0 0 0 0 0.000034

0 0

0 0

0.00026

0

0

17 0 0 0 0 0 0 0 0 0 0 0

18 1 0 0 0 0 0 0 0

0.000087 0.000091 0 0.003827 0.00033 0 0.003696 0 0.000251 0.066697 0 0.002994 0.000153 0 0 0.059651 0 0.002495 0 0 0.074704 0 0 0.004615 0.885204 0 0.107143 0.007653 0 0.222432 0 0.777568 0 0 0 0.123267 0 0.876733 0

0 0

0.003185

0.066723

0 0 0.920681

0 0 0

18 1

15 0 0 0 0 0 0

0 1

0 1

0 0

0.888614 10.111386 1

Table 8.5.3: motor car transition matrix 3 The related adjacency matrix is given in the Table 8.6. The first column contains the starting bonus malus class (starting state) and the first row the arriving state.

1 ' 1 1 2 1 3 0

2 0 0 1

3 1 0 0

4 0 1 0

5 0 0 1

6 1 0 0

7 0 1 0

8 0 0 1

9 10 11 12 13 14 15 16 17 18 1 0 0 1 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0

Chapter 3

132

1 4"["o" 0 5 6 7 8 9 10 11 12 13 14 15 16 17 18

0 0 0 0 0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0 0 0 0 0

1 0 0 0 0 0 0 0 0 0 0 0 0 0 0

1 fo" ^1 fo"0 ["0d" 01

0 I'd'11 1 o" 0 ' f" pb ^0~ 1 0 0 1 01 0 1 0 0 0 1 0 0 1 0 0 1 0 0 0 1 0 0 1 0 0 1 0 0 0 1 0 0 1 0 0 0 0 0 0 1 0 0 1 0 0 0 0 0 0 1 0 0 1 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

0 0 1 0 0 1 0 0 1 0 0 0 0

1 0 0 1 0 0 1 0 0 1 0 0 0

0 1 0 0 1 0 0 1 0 0 1 0 0

0 0 1 0 0 1 0 0 1 0 0 1 0

0 0 0 0 0 0 0 0 1 0 0 1 0 0 1

0 0 0 0

Table 8.6: motor car adjacency matrix The reachability matrix is given in Table 8.7. In this case the transition matrix is irreducible. This time from each state is possible to get in any other state. We report it only to show this event once.

1 2 3 4 5 6 7" ,_.,."9 1 1 1 1 2 1 1 3 1 1 4 1 1 5 1 1 6 1 1 7 1 1 8 1 1 9 1 10 1 1 11 1 1 12 1 1 13 1 1 14 1 1 15 1 1 16 1 1 17 1 1 1 jsj 1

'io 11

'12" ..__,., _^., ""15"" 16"'"i'7" '"i8

Table 8.7: reachability matrix

133

Markov Chain

The canonical matrix is the same for the transition matrix and we don't report it. In Table 8.8 is reported the limit vector.

1 state 1 2 3 4 5 6

State value state value 0.0008914 13 0.8681376 7 0.0541671 0.0004776 14 8 0.0575239 0.0002625 15 9 0.0091463 10 0.0001310 16 0.0061016 17 11 0.0000826 0.0029973 12 0.0000500 18 Table 8.8: limit vector

value 0.0000150 0.0000076 0.0000034 0.0000025 0.0000015 0.0000011

From this last result, we see that state 1 has a greater relevance in the evolution of the bonus malus system and the states from the 7th have very little importance. This fact is common to different national bonus malus systems and is called ''soif du bonus".

8.3 Reducible And Uni-Reducible Examples, Canonical Form Connection In this part we will show how starting from an example of a reducible transition matrix and changing one of its elements, it is possible to get different situations of class order, of different canonical form transition matrix and of asymptotic behaviour. We give in Table 8.9 the transition matrix from which we start. As usual in the first column the starting state is reported and in the first row the arriving state. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

1 .5 0 .1 0 0 0 0 0 0 0 0 0 0 0 0

2 0 .6 0 .11 .7 0 .02 .31 .01 .19 0 .22 .2 .1 .01

3 .05 0 .4 0 0 0 0 0 0 0 0 0 0 0 0

4 0 0 .05 .07 0 0 .2 0 .11 0 0 0 .14 0 .19

5 6 '• 7 8 9 10 .07 .1 0 0 .08 .03 0 .3 0 0 0 .1 0 0 0 0 0 .09 0 0 .5 .09 0 .1 0 0 .3 0 0 0 0 .6 0 0 0 0 .1 0 .3 .06 .07 .05 .34 0 0 .35 0 0 .21 0 .31 .04 .06 .07 .21 .07 .21 0 0 0 0 0 .45 0 0 0 0 0 .3 0 0 .18 .16 .11 0 0 0 0 .18 .21 0 0 .11 .13 .21 .31 .15 0 0 0 TJ^ble<8.9: i nitia 1 mat:rix

11 \""\2 13 .04 .05 0 0 0 0 .08 .11 .1 .04 0 .05 0 0 0 .4 0 0 .04 .1 0 0 0 0 .1 .06 0 .12 .07 0 .55 0 0 0 0 .15 .19 0 .07 .1 0 .17 .06 .07 0

14 .01 0 0 0 0 0 .06 0 0 .13 0 .15 0 0 0

15 .07 0 .07 .04 0 0 0 0 .03 0 0 0 .13 0

01

Chapter 3

134

Table 8.10 reports the adjacency matrix related to the previous transition matrix. 1 ? 3 4 5 6 7 8 9 10 11 12 13 14 15

1 1 0 1 0 0 0 0 0 0 0 0 0 0 0 0

? ^ 4 0 1 0 1 0 0 0 1 1 1 0 1 1 0 0 0 0 0 1 0 1 1 0 0 1 0 1 1 0 0 0 0 0 1 0 0 1 0 1 1 0 0 1 0 1

1 5 1 1 0 1 0 0 1 1 1 1 0 0 1 1 1

6 1 0 0 1 0 1 0 0 0 1 1 0 1 1 1

r?"rrr9 0 0 0 0 0 0 1 0 1 0 0 0 0 0 0

1 1 1 1 1 0 1 1 1 0 0 1 0 1 1

0 0 0 0 0 0 1 0 1 0 0 0 0 0 0

lO" 1 0 0 0 0 0 1 0 1 1 0 1 0 1 0

...,. 1 0 1 1 0 1 1 0 0 1 1 0 1 1 1

12 1 0 1 0 0 0 0 0 1 1 0 1 0 1 0

1 13 14 0 0 1 1 0 0 1 0 1 0 0 0 1 0 1

1 0 0 0 0 0 1 0 0 1 0 1 0 0 0

Tal\)le 8.1():I adj ace ncy mat rix

15 1 0 1 1 0 0 0 0 1 0 0 0 1 0

01

In the Table 8.11 the reachability matrix connected with this example is given. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

1 1 0 1 0 0 0 0 0 0 0 0 0 0 0 0

2 1 1 1 1 1 0 1 1 1 1 0 1 1 1 1

3 1 0 1 0 0 0 0 0 0 0 0 0 0 0 0

4 1 0 1 1 0 0 1 0 1 0 0 0 1 0 1

5 1 1 1 1 1 0 1 1 1 1 0 1 1 1 1

6 1 0

0

0

7 0 0 0 0 0 0 1 0 1 0 0 0 0 0 0

8 9 1 0 1 0 1 0 1 0 1 0 0 0 1 1 1 0 1 1 1 0 0 0 1 0 1 0 1 0 1 0

10 1 0 1 0 0 0 1 0 1 1 0 1 0 1 0

11

0

12 1 0 1 0 0 0 1 0 1 1 0 1 0 1 0

1 ab Ie8 .11 I r eac halt>ilit)^ma trix The states are divided into the following six classes:

13 1 0 1 1 0 0 1 0 1 0 0 0 1 0 1

14 1 0 1 0 0 0 1 0 1 1 0 1 0 0 0

15 1 0 1 1 0 0 1 0 1 0 0 0 1 0

01

135

Markov Chain

c,= {1,3}, C2-

{2,5,8}, {4,13,15},

(8.14)

{6,11}, Q = {7,9}, {10,12,14}. The cllasses are classified in the following way: C|, Cj - maximal, (8.15) C j , ^ - strictly transient, Cj, C4 - absorbing. The partial relation order can be described by means of the graph depicted in Figure 8.1. Arcs connect classes that are comparable. The maximal classes can't be comparable as the minimal ones. In this case also the strictly transient classes are not comparable.

Figure 8.1:1 case relation order After the class ordering process, it is possible to get the canonical form of the transition matrix that is given in Table 8.12. In the first column the starting states are given and in the first row the arriving states. 1 3 7 9

1 .5 .1 0 0

3 .05 .4 0 0

7 0 0 .3 .31

9 0 0 .07 .06

4 0 .05 .2 .11

13 0 .11 .04 .06

15 .07 .07 0 .03

10 .03 0 .05 .07

12 .05 .08 0 .1

14 .01 0 .06 0

2 0 0 .02 .01

5 .07 0 .1 .21

8 .08 .09 .06 .04

6 .1 0 0 0

11 .04 .1 j .1 0

Chapter 3

136

113 ^00 15 10 12 14 2 5 8 6 11

0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0 0

u .07 .05 .04 0 0 0 .11 .2 0 0 .14 .07 .13 0 0 0 0 .01 0 0 .19 .06 0 0 0 0 .21 .07 .13 .19 0 0 0 .3 .15 .15 .22 0 0 0 .13 0 0 0 .1 0 .1 0 0 0 0 0 .6 0 0 0 0 0 0 0 0 0 .7 0 0 0 0 .31 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 TJibleJ8.12: first canonica I matrix

.5 .16 .31 .21 0 .18 .3 0 .34 0 0

.1 0 .21 0 .18 .11 .1 .3 .35 0 0

.09 .11 .15 .07 0 .21 0 0 0 .6 .45

.04 .19 .07 .12 0

.17 1 0 0 0 .4

.55 1

The states follow the classes ordering: states corresponding to the maximal classes are reported in the North West matrix positions, followed by the states that are in the strictly transient classes and in the South East matrix positions there are the states corresponding to the absorbing classes. The successive algorithm step is the computation of absorbing probabilities that are given in Table 8.13. 2 4 1 0.595442 0.404558 2 0 0 3 ! 0.622395 0.377605 4 0.827531 0.172469 5 0 0 6 0 0 7 0.704561 0.295439 8 0 0 9 0.803532 0.196468 10 0.669429 0.330571 0 0 11 12 0.805249 0.194751 13 0.612876 0.387124 14 0.557551 0.442449 15 0.724003 0.275997 1 Tab e 8.13: absorbing probabilities The first row contains the absorbing class index and the first column the states. In Table 8.14, non-zero p'f^ are reported. The first row contains the states of absorbing classes and the first column the states.

137

Markov Chain

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

2

5

6

8

11

0.337437 0.566701 0.352712 0.468963 0.566701

0.141009 0.236815 0.147392 0.195972 0.236815

0.214178

0.116995 0.196484 0.122291 0.162597 0.196484

0.19038

0

0

0.399276 0.566701 0.455362 0.379366

0.166851 0.236815 0.190288 0.158531

0 0.456335 0.347318 0.315965 0.410294

0 0.199908 0.091307

0 0.529412 0.156409

0

0 0.138435 0.196484 0.157881 0.131532

0.104012 0.175008 0.529412 0 0 0.190695 0.103104 0.158218 0.145138 0.204948 0.12042 0.132036 0.234238 0.10955 0.171455 0.146116 0.142255 Table 8.14: limit behaviour

0 0.177696 0.081162

0 0.470588 0.13903

0 0.092456 0.155563 0.470588 0.091648 0.182176 0.208211

0.129881 1

For the second example the transition matrix was obtained from the previous one putting a non-zero element in the position (10, 4). The new transition matrix is the one in Table 8.15.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

1 .5 0 .1 0 0 0 0 0 0 0 0 0 0 0 0

2 0 .6 0 .11 .7 0 .02 .31 .01 .19 0 .22 .2 .1 .01

3 4 .05 0 0 0 .4 .05 0 .07 0 0 0 0 .2 0 0 0 0 .11 0 .12 0 0 0 0 0 .14 0 0 0 .19

10 11 12 6 8 9 7 .07 .1 0 .08 0 .03 .04 .05 .1 0 0 .3 0 0 0 0 0 0 0 .09 0 0 .1 .08 .1 0 .04 0 .5 .09 0 0 0 .3 0 0 0 0 0 0 0 0 0 .6 .4 0 0 0 0 .1 .3 .06 .07 .05 .1 0 .34 0 0 .35 0 0 0 0 .21 0 .31 .04 .06 .07 0 .1 0 .21 .12 .07 .09 .07 0 0 0 .45 0 0 0 .55 0 0 0 .3 0 0 .18 0 0 .15 0 0 .19 0 0 .16 .11 0 .18 .21 0 .11 0 .13 .17 .1 0 .07 0 .31 .15 0 .21 0 fable 8.15 :IIt ransiition mati ix

13 0 0 .11 .05 0 0 .04 0 .06 0 0 0 .07 0 .06

14 .01 0 0 0 0 0 .06 0 0 .13 0 .15 0 0 0

15 .07 0 .07 .04 0 0 0 0 .03 0 0 0 .13 0 0

The adjacency and the reachability matrices are reported respectively in Tables 8.16 and 8.17.

Chapter 3

138

1 2 3 4 5 6 7 8 9 10 11 12 13 14

1 121 ^ pT" 11 ^^ 1'^1^ " 1^ 1^ ^ TT"TT"f ^^ "1^1 ^^ 1 1 0 0 1 0 1 1 0 1 0 1

1 11

1 0 1 1 0 0 1 0 1 0 0 0 1 0 1 0 1 0 1 1 0 0 0 1 0 1 0 1 1 15 0 1

.„...,

0 1

u

0 0 0 0 0 0 0 0 0 0 0

0 1 0 1 0 u 1 1 1 0 0 0 0 0 1 1 1 0 0 1 0 1 1 0 1 1 1 0 0 1 0 0 0 1 1 1 0 1 1 1 1 1

0 0 0 0 0 1 0 1 0 0 0 0 0 0

1 1 1 1 0 1 1 1 0 0 1 0 1 1

0 0 0 0 0 1 0 1 0 0 0 0 0 0

0 0 0 0 0 1 0 1 1 0 1 0 1 0

0 1 1 0 1 1 0 0 1 1 0 1 1 1

0 1 0 0 0 0 0 1 1 0 1 0 1 0

0 1 1 0 0 1 0 1 0 0 0 1 0 1

0 0 0 0 0 1 0 0 1 0 1 0 0 0

0 1 1 0 0 0 0 1 0 0 0 1 0 0

13 1 0 1 1 0 0 1 0 1 1 0 1 1 1

14 1 0 1 0 0 0 1 0 1 1 0 1 0 1 10

15 1 0 1 1 0 0 1 0 1 1 0 1 1 1

Table {J.16: II adjacency matrix 1 2 1 1 0 1 1 1 0 1 0 1 0 0 0 1 0 1 0 1 0 1 0 0 0 1 0 1 0 1 15 1 0 1 1 2 3 4 5 6 7 8 9 10 11 12 13 14

3 1 0 1 0 0 0 0 0 0 0 0 0 0 0

10

4 1 0 1 1 0 0 1 0 1 1 0 1 1 1 11

5

6 1 0

0

0

0

7 0 0 0 0 0 0 1 0 1 0 0 0 0 0

8 1 1 1 1 1 0 1 1 1 1 0 1 1 1

9 0 0 0 0 0 0 1 0 1 0 0 0 0 0

10 1 0 1 0 0 0 1 0 1 1 0 1 0 1

11 1 0 1 1 0 1 1 0 1 1 1 1 1 1

0 1 1 10 1 0 1 1

12 1 0 1 0 0 0 1 0 1 1 0 1 0 1 10

Table 8.17: II reachability matrix

1

1 1

Here too, the subdivision in classes is also given by (8.14) and so result (8.15) holds. But now in the order relation, two strictly transient classes are comparable. The graph of Figure 8.1 becomes the one depicted in Figure 8.2.

139

Markov Chain

Figure 8.2: II order relation The canonical matrix is given in Table 8.18. 1 3 7 9 10 12 14 4 13 15 2 5 8 6 11

1 .5 .1 0 0 0 0 0 0 0 0 0 0 0 0 0

3 .05 .4 0 0 0 0 0 0 0 0 0 0 0 0 0

7 0 0 .3 .31 0 0 0 0 0 0 0 0 0 0 0

9 0 0 .07 .06 0 0 0 0 0 0 0 0 0 0 0

10 .03 0 .05 .07 .21 .3 .13 0 0 0 0 0 0 0 0

12 .05 .08 0 .1 .07 .15 .1 0 0 0 0 0 0 0 0

14 .01 0 .06 0 .13 .15 0 0 0 0 0 0 0 0 0

4 0 .05 .2 .11 .12 0 0 .07 .14 .19 0 0 0 0 0

13 0 .11 .04 .06 0 0 0 .05 .07 .06 0 0 0 0 0

15 .07 .07 0 .03 0 0 0 .04 .13 0 0 0 0 0 0

2 0 0 .02 .01 .19 .22 .1 .11 .2 .01 .6 .7 .31 0 0

Tal>le8. 18:1][can onicjilfor m matrix The absorbing probabilities are presented in Table 8.19.

1 2 3 4 5

2 4 0.592412 0.407588 0 0 0.620466 0.379534 0.827531 0.172469 0 0

5 .07 0 .1 .21 .09 0 .18 .5 .16 .31 .3 0 .34 0 0

8 .08 .09 .06 .04 0 .18 .11 .1 0 .21 .1 .3 .35 0 0

6 .1 0 0 0 .07 0 .21 .09 .11 .15 0 0 0 .6 .45

11 .04 .1 .1 0 .12 0 .17 .04 .19 .07 0 0 0 .4

.55 1

Chapter 3

140 .....

7 8 9 10 11 12 13 14 15

0

0

0.70175

0.29825

0

0

0.79939 0.641512

0.20061 0.358488

0

0

0.794566 0.205434 0.612876 0.387124 0.552853 0.447147 0.724003 0.275997 1 Table 8.19: II absorbing probabilities At least, as in Table 8.14 the non-zero py

5

2 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

are given in Table 8.20

6

8

0.335720 0.566701 0.351619 0.468963 0.566701 0.000000 0.397683 0.566701 0.453015 0.363545 0.000000 0.450282 0.347318 0.313303 0.410294

0.140292 0.215782 0.116399 0.236815 0.000000 0.196484 0.146936 0.200930 0.121912 0.195972 0.091307 0.162597 0.236815 0.000000 0.196484 0.000000 0.529412 0.000000 0.166185 0.157897 0.137883 0.236815 0.000000 0.196484 0.189307 0.106206 0.157067 0.151920 0.189788 0.126047 0.000000 0.529412 0.000000 0.188165 0.108759 0.156120 0.145138 0.204948 0.120420 0.130924 0.236725 0.108627 0.171455 0.146116 0.142255 Ta ble 8.20: IIcase limit blehaviour

11 0.191806 0.000000 0.178604 0.081162 0.000000 0.470588 0.140353 0.000000 0.094405 0.168700 0.470588 0.096675 0.182176 0.210422 0.129881

The last step of this example will be taken by putting a non-zero element in the position (11,2) of the matrix given in Table 8.15. The transition matrix obtained in this way is shown in Table 8.21.

1 2 3 4 5 6

4 1 2 3 0 0 .05 .5 0 0 0 .6 .1 0 .4 .05 0 .11 0 .07 0 .7 0 0 0 0 0 0

6 5 .07 .1 0 .3 0 0 .5 .09 0 0 0 .6

8 7 0 .08 0 .1 0 .09 0 .1 0 .3 0 0

9 0 0 0 0 0 0

10 11 12 13 .03 .04 .05 0 0 0 0 0 0 .1 .08 .11 0 .04 0 .05 0 0 0 0 0 .4 0 0

14 15 .01 .07 0 0 0 .07 0 .04 0 0 0 o|

Markov Chain 7 8 9 10 11 12 13 14 15

0 0 0 0 0 0 0 0 0

.02 .31 .01 .19 ,1 .22 .2 .1 .01

141 0 0 0 0 0 0 0 0 0

.2 0 .11 .12 0 0 .14 0 .19

.1 .34 .21 .09 0 0 .16 .18 .31

0 0 0 .07 .42 0 .11 .21 .15

.3 0 .31 0 0 0 0 0 0

.06 .35 .04 0 0 .18 0 .11 .21

.07 0 .06 0 0 0 0 0 0

.05 0 .07 .21 0 .3 0 .13 0

.1 0 0 .12 .48 0 .19 .17 .07

.04 0 .06 0 0 0 .07 0 .06

0 0 .1 .07 0 .15 0 .1 0

.06 0 0 .13 0 .15 0 0 0

0 0 .03 0 0 0 .13 0 0

Table 8.21: III transition matrix The related adjacency and reachability matrices are given respectively in Tables 8.22 and 8.23.

IT" 1 2 3 4 5 6 7 8 9 10 11 12 13 14

1 0 1 0 0 0 0 0 0 0 0 0 0 0 1 15 0

I ^I ^ '"51

0 1 0 1 0 0 0 1 1 1 0 1 1 0 0 0 0 0 1 0 1 1 0 0 1 0 1 1 0 1 1 0 0 1 0 0 1 0 1 1 0 0 1 0 1

1 0 1 0 0 1 1 1 1 0 0 1 1 1

1 8 TV

|""6 •

1 0 1 0 0 1 0 0 1 1 0 1 0 0 1 1 0 0 0 1 1 0 0 1 0 1 1 1 1 0 0 1 0 0 0 0 1 1 0 0 1 0 1 1 0 1

0 0 0 0 0 0 1 0

ri"2 I^ ^11 ^ ^1 10 ^1^141 ^y 1 1

0 0 0 0 0 1 0 !1 ' 1 1 0 0 0 1 0 0 0 1 0 0 0

0 1 1 0 1 1 0 0 1 1 0 1 1 1

0 1 0 0 0 0 0 1 1 0 1 0 1 0

0 0 1 0 0 1 0 0 0 0 1 1 0 0 1 ' 0 0 1 0 0 0 1 1 0 0 0 1 0

0 1 1 0 0 0 0 1 0 0 0 1 0 0

14 1 0 1 0 0 0 1 0 1 1 0 1 0 1 0

15 1 0 1 1 0 0 1 0 1 1 0 1 1 1 1

Tab le 8.22: III ac Ijacency matrix 1 2 3 4 5 6 7 8 9 10 11 12 13 14

I5J

1 1 0 1 0 0 0 0 0 0 0 0 0 0 0 0

2 3 1 1 1 0 1 1 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0

4 1 0 1 1 0 0 1 0 1 1 0 1 1 1

5

11 1

6 1 0

0

0

7 0 0 0 0 0 0 1 0 1 0 0 0 0 0

0

8 9 10 1 1 0 1 0 0 1 0 1 1 0 0 1 0 0 1 0 0 1 1 1 1 0 0 1 1 1 1 0 1 1 0 10 1 0 1 1 0 0 1 0 1 1 1 10 0

11 1 0 1 1 0 1 1 0 1 1 1 1 1 1 1

12 1 0 1 0 0 0 1 0 1 1 0 1 0 1

01

Table 8.23: III reachability matrix

13 1 0 1 1 0 0 1 0 1 1 0 1 1 1 1

Chapter 3

142

The subdivision in classes is the same as in the two previous cases. But here, the new non-zero element gives only one absorbing state. More precisely we have the following situation: CpC5 -maximal, C3, C4, Q - strictly transient, C2 - absorbing. The new order relation is reported in Figure 8.3.

Figure 8.3: III relation order We see that we have a unique absorbing class and a uni-reducible case. The three strictly transient classes are comparable and the canonical form of the transient matrix is given in Table 8.24. 1 3 7 9 10 12 14 4 13 15 6

7 1 3 9 10 0 .03 .5 .05 0 .1 .4 0 0 0 .3 .07 .05 0 0 0 0 .31 .06 .07 0 0 0 0 .21 0 0 0 0 .3 0 .13 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

12 .05 .08 0 .1 .07 .15 .1 0 0 0 0

14 .01 0 .06 0 .13 .15 0 0 0 0 0

4 0 .05 .2 .11 .12 0 0 .07 .14 .19 0

13 0 .11 .04 .06 0 0 0 .05 .07 .06 0

15 .07 .07 0 .03 0 0 0 .04 .13 0 0

6 .1 0 0 0 .07 0 .21 .09 .11 .15 .6

11 .04 .1 .1 0 .12 0 .17 .04 .19 .07 .4

2 0 0 .02 .01 .19 .22 .1 .11 .2 .01 0

5 .07 0 .1 .21 .09 0 .18 .5 .16 .31 0

8 .08 .09 .06 .04 0 .18 .11 .1 0 .21 0

Markov Chain 11 2 5 8

0 0 0 0

0 0 0 0

143 0 0 0 0

0 0 0 0

0 0 0 0

0 0 0 0

0 0 0 0

0 0 0 0

0 0 0 0

0 0 0 0

.42 .48 .1 0 0 .6 .3 .1 0 0 0 0 .7 0 .3 0 0 .31 .34 .35 J

Table 8.24: III canonical form matrix

2

5

8

0.566701

0.236815

0.196484

Table 8.25: limit vector At last the limit vector of the only absorbing class is reported in Table 8.25. As it was shown, small changes in the transition matrix can give changes to the process classification and different results.

Chapter 4 MARKOV RENEWAL PROCESSES, MARKOV RANDOM WALKS AND SEMI-MARKOV PROCESSES 1 POSITIVE (J-X) PROCESSES Let us consider a physical or economical system called S with m possible states, m being a finite natural number. For simplicity, we will note by /the set of all possible states: / = {l,...,m} (1.1) as we already did in Chapter 3 devoted to Markov chains. At time 0, the system S starts from an initial state represented by the r.v. Jo, stays a non-negative random length of time X\ in this state, and then goes into another state J\ for a non-negative length of time Xi before going into Ji, and so on. So we have a two-dimensional stochastic process in discrete time called a positive {J-X) process: {J-X) = {{J^,X^\n>0) (1.2) supposing X,=Q,a.s. (1.3) where the sequence (J^,/7>0) gives the successive states oi S in time and the sequence {X^,n> 0) gives the successive sojourn times. More precisely, X^ is the time spent by S in state J^-x (n>0). Times at which transitions occur are given by the sequence (T^,n > 0) where: T,=0,T,^X„...,T„^f^X^

(1.4)

and so X„=T„-T„_„n>l.

(1.5)

2 SEMI-MARKOV AND EXTENDED SEMI-MARKOV CHAINS On the complete probability space (Q,3,P), the stochastic dynamic evolution of the considered (J-X) process will be determined by the following assumptions: P(Xo==0)=l,a.s.,

146

P(Jo=i)=Ph /=l...,m with ^p.

Chapter 4

=1,

for all n>Oj'=l, ..,,m, WQ have: P(J^ =J,X^ <x\(J,,X,),k = 0,„.,n-l) = Q,^Jx),a.s.

(2.1) (2.2)

where any function Qy (ij=l, ...,w) is a non-decreasing real function null on M such that if j9 = limQ (x), ij'el, (2.3) then: m

t.p,-\.iel.

(2.4)

7=1

With matrix notation, we will write: Q = ta]' P = h ] ( = Q ( ^ ) ) . V = (P^,-,PJ• This leads to the following definitions.

(2.5)

Definition 2.1 Every matrix mxmQ of non-decreasing functions null on M satisfying properties (2.3) and (2.4) is called a semi-Markov matrix or a semiMarkov kernel Definition 2.2 Every couple (p,Q) where Q is a semi-Markov kernel and p a vector of initial probabilities defines a positive {J,X) process {J,X) ={{Jn.Xn), n>0) with / x R ^ as state space, also called a semi-Markov chain (in short a SMC), Sometimes, it is useful that the random variables X^,n>Q take their values in M instead of M , in this case, we need the next two definitions. Definition 2.3 Every matrixmxm Q of non-decreasing functions satisfying properties (2.3) and (2.4) is called an extended semi-Markov matrix or an extended semi-Markov kernel. Definition 2.4 Every couple (p,Q) where Q is an extended semi-Markov kernel andp a vector of initial probabilities defines a (J,X) process (J,X) =((Jn,X,),n>0)with IxR as state space, also called an extended semi-Markov chain (in short an ESMC). Let us come back to the main condition (2.2); its meaning is clear. For example let us suppose that we observe for a certain fixed n that Jn.\=i, then the basic relation (2.2) gives us the value of the following conditional probability: P(J„=j,X„<x\(J,,X,),k = 0,...,n-U„,,^i):^Q,jix). (2.6)

Markov Renewal Processes

147

That is, the knowledge of the value of Jn.\ suffices to give the conditional probabilistic evolution of the future of the process whatever the values of the other past variables might be. According to Kingman (1972), the event [co \ J^_^{co) = i] i^ regenerative in the sense that the observation of this event gives the complete evolution of the process in the future as it could evolve from n^O as / the initial state. Remark 2.1 The second member of the semi-Markov characterisation property (2.2) does not depend explicitly on n\ also we can be precise that we are now studying homogeneous semi-Markov chains in opposition with the nonhomogeneous case where this dependence with respect to n is valid.

3 PRIMARY PROPERTIES We will start by studying the marginal stochastic processes (J^,«>0), {X^,n > 0) called respectively the J-process and the X-process, (i) The y-process From the semi-Markov relation (2.2) and Lebesgue's theorem (see Chapter 1), we deduce that a.s.: p ( y „ = 7 | ( j , , x , ) , ) t = o,...,«-i) = a„_,y(+^). (3.1) Using the smoothing property (see Chapter 1) of conditional expectation, we get F(y„-7|(y,),/: = 0,...,«-l) = £(a„_,,(+^)|(A),^ = 0,...,^-l), (3.2) and as the r.v. Qj^ ,y("^^) ^^ (A^=0,...,«-l)-measurable, we finally get from relation (2.3) that: = 0,,„,n-\) = Pj^^j. (3.3) P{J„=j\{J,U Since relation (2.4) implies that the matrix P is a Markov matrix, we have thus proved the following result. Proposition 3.1 The J-process is a homogeneous Markov chain with P as its transition matrix. That is the reason why this J-process is called the imbedded Markov chain of the considered SMC in which the r.v. Jn represents the state of the system iSjust after the n\h transition. From results of Chapter 3, it follows that in the ergodic case there exists one and only one stationary distribution of probability n = (7r^,...,7r^) satisfying:

148

Chapter 4

(3.4) m

such that limP(J„ =j\j,^i)(^l\mpl;')^7rj,i,JBl,

(3.5)

where we know from Chapter 3 that

[4"^] = P".

(3.6)

(ii) The X-process Here, the situation is entirely different for the fact that the distribution of X^ depends on J^A- Nevertheless, we have an interesting property of conditional independence but before proving this property we must introduce some definitions. Definition 3.1 The two following conditional probability distributions: F,^,,„(x) = P(X„<x|j„_,,/„),

H,Jx) = P{X„<x\J„_,) are respectively called the conditional and unconditional distributions of the sojourn time X„. From the general properties of conditional recalled in Chapter 1, we successively get F,^^j^{x) = E(P{X„<x\{J,,X,\k
Pj„.Jn

provided that pj j is strictly positive. If not, we can arbitrarily give to (3.8) for example the value U\{x) defined as

ro,x
^,W=

' ^ '

Moreover, from the smoothing property, we also have: Hj^^ (x)(= P{X„ <x\j„_,)) = ^ ( F , _,^(x)|j„_,),

(3.9)

Markov Renewal Processes

149

We have thus proved the following proposition. Proposition 3.2 As a function of the semi-kernel Q, the expressions conditional and unconditional distributions of the sojourn time X^ are given by:

'Q>M) F,j{x){=P{X„<x\j„_,^i,J„^j))

=\

Pij

U,{x),p,j=Q,

(3.11)

Remark 3.1 (i) From the last relation (3.11), we can also express the kernel Q as a function of theFy, i,j=\,...,m: (3.12) Q,j(x) = p,F,j(xUjeI,xeR\ So, every SMC can also be characterised by the triple (p,P,F) instead of the couple (p,Q) where the mxm matrix F is defined as ^ = [7^^.] where the functions F.j, i,j = l,,.,,m are distribution functions on support M"^. (ii) We can also introduce the means related to these conditional and unconditional distribution functions. When they exist we will note: = l„.,m, ^y = jxdF.jixyj ; 77. = \xdH.(x)J = l,..,,m

(3.13)

R

and the last relation (3.11) leading to the relation: m

^i-t,P,P,' the quantities y&^., ij=l,..,,m

(3.14) and 7j.,i=l,...,m are respectively called the

conditional and unconditional means of the sojourn times. We can now give the property of conditional independence. Proposition 3.3 For each integer k, let ni,n2, ...^n^be kpositive integers such that ni
^ K ^ ^ > - . ^ . <xJ/„,.„J„,,...,J„,.„yJ=:F,^^__,„_(x„,)...F,„^,,^^(x„^), (3.15) that is the k random variables X^ ...-.X^ are conditionally independent given

150

Chapter 4

Proof For each n and for all real numbers Xi, ...,x„, it suffices to prove that: P(X,<X,,...,X„<X„\J„J„...J„_„J„) = F,^,^{X,)...FJ^^JSX„). (3.16) This is an immediate consequence of the fact that entrance times in states J<^,,.,Jn are regenerative events. a (i) The T-process By relation (1.4), the sequence {T^,n> 0) represents successive renewal epochs, that is times at which transitions occur. By analogy with renewal theory, we have the following definition. Definition 3.2 The two-dimensional process ((J^,T^),n> 0) is called the Markov renewal process of kernel Q. Before giving the expression of the marginal distribution of the random vector {Jn,Tr) with values in / x R ^ , given that J^^i, let us define the marginal distributions of the (J, 7) process {(Jn,Tn), n>0): (3.17) Q;(t) = P(J„=j\T„0,t>0, With A = r4y] and B = r5,yJ, two mxm matrices of integrable functions, we associate a new matrix A • B whose general element (A • B)/, is the function of t defined by: m

(A.B),(0-£j^^.(/->')c/5,,(>;).

(3.18)

It can be easily seen that this type of product, called convolution product for matrices, is associative but not always commutative. In the particular case of yl =B, we set: A . A - A < ^ \ . . . , A . - - - . A = A<">(=r4"n), ^ ^ ' •^> A^»'=(^,t/o),A<>^=A. If all the functions A.J,B^J,iJ-\,.,,,m,wdim^h

(3.19) at - o o , we can also use an

integration by parts to express (3.18) as follows: m

(A.B),(0 = £j5,,(?-;.)«'4(>;) and moreover if A==B, we get:

(3.20)

m

{X.Bl.{t) = Y.\Mt-y)dA,{y).

(3.21)

Markov Renewal Processes

151

Proposition 3.4 For all n>0, we have: Q;=Q^.

(3-22)

Moreover, we also have: \imQ^"\t) = P".

(3.23)

/->oo

Proof From the definition (3.17), it is clear that = Qff{t),i,j = \,...,m,t>Q. Q^it) = ^{t),Q^{t)

(3.24)

Conditioning with respect to the first transition and using the regenerative property of the SMC, we have: Q}j{t) = P{J,=j,X,+X,
(3.25)

By induction, we get: m

m)=lL\Qlr''it-y)dQi^iy)

(3-26)

and so the matrix relation (3.22) is proved.

n

Relation (3.23) results immediately from relation (3.17) letting t tend to + oo and using the monotone convergence of Lebesgue (see Chapter 1). Remark 3.2 It is clear that the above properties proved for SMC are also valid for ESMC. The only difference is that the T-process can no longer be interpreted as an extension of a classical renewal process but in fact as a random walk, as the r.v. Tn then take their values in R and no longer in R"^ (see section 14).

4 EXAMPLES Semi-Markov theory is one of the most productive subjects of stochastic processes to generate applications in real-life problems. Here we cannot give here a complete view of such applications particularly in the fields of Economics, Manpower models. Insurance, Finance (more recently), Reliability, Simulation, Queuing, Branching processes. Medicine (including survival data). Social Sciences, Language Modelling, Seismic Risk Analysis, Biology, Computer Science, Chromatography and Fluid mechanics. Important results in such fields may be found in Janssen (1986) and Janssen and Limnios(1999). Just to give an idea of such a potential, we briefly present short examples of such appHcations.

152

Chapter 4

Example 4.1: The claim process in insurance Let us consider an insurance company covering m types of risks or having m different types of customers for the same risk forming the set /=={!, ...,m}. For example, in automobile insurance, we can distinguish three types of drivers: good, average and bad and so / is a space consisting of three states: 1 for good, 2 for average and 3 for bad. Now, let (X^,^>1)represent the sequence of successive observed claim amounts, (Y^,n> 1) the sequence of interarrivals between two successive claims and (J^,n> 1) successive types of observed risks. In the classical model of risk theory called the Cramer Lundberg model (1909, 1955), it is only supposed that there is only one type of risk and the claim arrival process is a Poisson process of parameter /I; later, Andersen (1967) extends this model to an arbitrary renewal process and moreover in these two classical models, the process of claim amounts is a renewal process independent of the claim arrival process. The consideration of an SMC for the two-dimensional processes ((J^,XJ,n > 0) or/and ((J^,YJ,n>0) gives the possibility to introduce a certain dependence between the successive claim amounts. This model was first developed by Janssen (1969b, 1970, 1977) along the lines of Miller's work (1962) and since then has lead to a lot of extensions, see for example Asmussen (2000). Example 4.2: Queuing theory Consider a queuing system where customers wait to be served by a single server. Successive customers are served according to the FIFO (first in first out) discipline rule. We also assume that the capacity of the waiting room is infinite, which means that every customer waits until he is served. Moreover, we designate time 0 as the arrival of the first customer, who is immediately served. As in the preceding example, we suppose that each customer has as characteristic a type / belonging to the set /={l,...,m}, type influencing service times and/or interarrival times. More precisely, let the r.v. J^ be the type of the (n+\)th customer, B^ (n>l) the service time of the ^th customer and A^ (n>\) the interarrival time between the nth and (n+\)th customers. One of the main problems in queuing theory is the study of successive waiting times (W^,n>0) where W^ represents the time that the (^+l)th customer has to wait before being served with, as we already know, Wo=0. Every queuing model starts with assumptions on the sequences (J„,n> 0),(B^,n> 1),(A^,n> 1) and their eventual stochastic dependence. For example, in the so-called M/SM/1 model, the three basic assumptions are:

Markov Renewal Processes

153

(i) successive interarrival times {A^,n> 1) form a Poisson process of parameter (ii) successive service times (B^,n> 1) form an SMC of kernel ^Q , (iii) stochastic processes (A^,n>l) and (B^,n>V) are independent. On the other hand, of the SM/M/1 model, the three basic assumptions are the following: (i) successive interarrival times (Ann>l) form a Poisson process of parameter M^

(ii) stochastic processes (A^, /? > 1) and (B^, « > 1) are independent. These two models were introduced respectively by Neuts (1966), Cinlar (1967) and extended by Janssen (1979). For m=l, that is when the customers are all of the same type, these two models give respectively the M/G/1 and G/M/1 classical queuing models. Example 4.3: Occupational illness insurance This problem is related to occupational illness insurance with the possibility to lead to partial or permanent disability. In this case the amount of the incapacitation allowance depends on the degree of disability recognised for the policyholder by the occupational health doctor, in general on a yearly basis, because this degree is a function of a professional illness which can take its course. Considering as in the example of section 7 in Chapter 3 this invalidity degree as a stochastic process (J^, n>0) where Jn represents the value of this degree when the illness really takes its course, we must then introduce the r.v. X„ representing the time between two successive transitions from Jn.\ to /„. In practice, these transitions can be observed with periodic medical inspections. The assumption that the J-X process is an SMC extends the Markov model of Chapter 3 and will be fully treated below. Example 4.4: Reliability There are a lot of semi-Markov models in reliability theory see for example Osaki (1985) and more recently in Limnios and Oprisan (2001), (2003). Let us consider a so-called reliability system S that can be at any time t in one of the m states of/={l,...,m}. The stochastic process of the successive states of S is represented by ^ = (^,,^>0). The state space / is partitioned into two sets U and D so that (4.1) I = U[jD, UnD = 0, U^0, D^0, The interpretation of these two sets is the following : the subset U contains all "good" states, in which the system is working and the subset D of all "bad " states in which the system is not working well or has failed.

154

Chapter 4

The indicators used in rehabihty theory are the following ones: (i) the reliability function R giving the probability that the system was always working from time 0 to time /: (4.2) R{t) = P(S^eUyue[0,t]), (ii) the pointwise availability function A giving the probability that the system is working at time / whatever happens on (0,/]: A{t) = P{S,GU), (4.3) (iii) the maintainability function M giving the probabihty that the system, being in D on [0,0? will leave the set D at time t\ M{t) = P(S^^D,u^[Oj),S,^U), (4.4)

5 MARKOV RENEWAL PROCESSES, SEMI-MARKOV AND ASSOCIATED COUNTING Let us consider an SMC of kernel Q; we then have the following definitions. Definition 5.1 The two-dimensional process (J,T)=((Jn,Tn\n>0) where r„ is given by relation (1.4) is called a Markov renewal sequence or Markov renewal process, Cinlar (1969) also gives the term Markov additive process. It is justified by the fact that, using relation (1.5), we get: PiJ„.^ =jj„,, <x\{J„T,),k^Q,...,n)::^ P(J„,,-j,X„,,<x-T„\(J,,T,),k

= 0,...,n) =

Q,Jx-T„).

This last equahty shows that the (J, T) process is a Markov process with / x R^ as state space and having the "additive property": T„.^=T„+X„^,. (5.2) Let us say that according to the main definitions of Chapter 2, section 2, always in the case of positive (J,X) chains the random variables T^,(n> 0) are from now on called Markov renewal times or simply renewal times, the random variables X^,(n> 1) interarrival or sojourn times and the random variables /„,(/7 > 0) the state variables. We will now define the counting processes associated with any Markov renewal process (in short MRP) as we already did in the special case of renewal theory. For any fixed time t, the r.v. N(t) represents the total number of Jumps or transitions of the {J,X) process on (0,/], including possible transitions from any state towards itself (virtual transitions), transitions supposed to be observable. As in renewal theory, we have: Nit)>t<^T^
Markov Renewal Processes

155

But here, we can be more precise and only count the total number of passages in a fixed state / always in (0,/] represented by the r.v. Ni(t). Clearly, we can write: m

N(t) =

'£N,(0,t>0.

(5.4)

Definition 5.2 To each Markov renewal process, the following m+1 stochastic processes are associated respectively with values in N: (i) the N-process (N(tl t > 0), (ii) theNrprocess (Ni(t),t>0),i=l,...,m, respectively called the associated total counting process and the associated partial counting processes with of course: (5.5) A^(0)=0,iV/(0)-0,/=l, ..„m. It is now easy to introduce the notion oi di semi-Markov process by considering at time /, the state entered at the last transition before or at /, that is JN{ty Definition 5.3 With each Markov renewal process, we associate the following stochastic Z-process with values in I: Z=(Z(t),t>0), (5.6) with: Z{t)=JM(,,). (5.7) This process will be called the associated semi-Markov process or simply the semi-Markov process (in short SMP) of kernel Q. Remarks 5.1 1) As in renewal theory, we will often use counting variables including the initial renewal, that is: N\t) = N{t)-\-\, N\{t) = N,{t)^d,^,

^^'^^

Chapter 4

156

j

Zt

i

^2 To

Tri

1h

1

1^3

R"-

Xn Tn T n-1

Figure 5.1: a trajectory of an SMP 2) Figures 5.1 gives a typical trajectory of MRP and SMP. 3) It is now clear that we can immediately consider an MRP defined by kernel Q without speaking explicitly of the basic (J,X) process with the same kernel Q, because the basic property (2.6) is equivalent to (5.1).

6 MARKOV RENEWAL FUNCTIONS Let us consider an MRP of kernel Q and to avoid trivialities, we will assume that: supa,.(0) 1, as the times (possibly infinite) of successive returns to state /, also called successive entrance times into From the regenerative property of RRP, whenever the process enters into state /, say at time t, the evolution of the process on [^oo) is probabilistically the same as if we had started at time 0 in the same state /. It follows that the process fr^(/|/),^>Oj with: 7;(/|/) = 0

(6.2)

is a renewal process that could be possibly defective. From now on, the r.v. T^ (/|/) will be called the nth return time to state i. More generally, let us also fix statey, different from state / already fixed; we can also define the nth return or entrance time to state j , but starting from i as initial

Markov Renewal Processes

157

state. This time, possibly infinite too, will be represented by fr^(7|/),«>0j, using here too the convention that 7;(y|/) = 0.

(6.3)

Now, the sequence (r„(y|/),«>Oj is a delayed renewal process with values in K^ It is thus defined by two d.f.: Gy being that of 7;(y|/) and Gjj that of r2(y|/)-7](y|/),sothat: G,i,t) =

P(T\j\f)2. Gjj{t) = Of course, the d.f. Gjj suffices to define the renewal process (r„ \j\j),

(6.4) n>Q\.

Remark 6.1 From the preceding definitions, we can also write that:

p[T,(j\i) =

' +^)^\-G,{+^)

and for the mean of the T(i\i\,n>\,

(6.5) possibly infinite, we get:

lu,j=E(T\j\i)y]tdG..{,t),

(6.6)

0

with the usual convention that 0-(+oo) = 0.

(6.7)

The means /u^j.ij e / are called ihQfirst entrance or average return times. Lemma 6.1 Thefiinctions GyJ,jsI

satisfy the following relationships:

m

^=1

G,j{t) = Y,G^'»Q,,it) + {\-Gjj)*Q,j{t),iJsI,t>Q.

(6.8)

Proof By the regenerative property, we can decompose the event "first entrance intoy" starting from / following the state occupied at the first transition so that: m

G,(0-e,(0+

£

G,j*Q,,{t),iJ^I,t>().

(6.9)

By adding and subtracting the missing term for k^j in the sum of the second member of relation (6.9), we get relation (6.8). n

158

Chapter 4

For each possibly delayed renewal process defined by the couple (Gij,Gjj), ij belonging to /, we will represent by Ay and Ry the associated renewal functions defined by relations 2(2.4) and 2(3.12) so that: R,j{t) =

(6.10)

E{N]{t)\j,=i)

and by relations (5.8): + A,j{t). R,{t)^S,p,{t)

(6.11)

From relations 2(9.7), 2(3.9) and 2(3.14), we get: 00

R,{t)-G,.R..{t). Or equivalently, we have: R,(t)-dp,{t)

+ G^*fGf{t),i,j^I.

(6.13)

A7=0

Proposition 6.1 Assumption w < oo implies that: (i)

at least one of the renewal processes [T^ \j\j\n

^ 0j,7 e / is not defective,

(ii) for all i belonging to I, there exists a state s such that lim7;(^|/) = +oo, a.^.,

(6.14)

(iii) for the r.v. T^ defined by relation (1.4), given that Jo=i whatever i is, we have a.s. that lim7;=+oo. (6.15) n

Proof (i) As m is finite, there exists for the embedded MC (J^,n>0) at least one recurrent state, say k, reached by the chain in a finite time so that the state k will be visited infinitely often. It follows that the renewal process is not defective. (ii) Let us now consider / as initial state. We can associate a recurrent state, say s, depending on /, which will be hit starting from / so that the delayed renewal process defined by (G.^,G^J will not be defective. (iii) This is an immediate consequence of result (ii). n The following relations will express the renewal functions R.j, i,j el in function of the kernel Q instead of the m^ functions Gy. Proposition 6.2 For every i andj of I, we have that:

Markov Renewal Processes

R>j(t)-t.Qlf'(0.

159

(6.16)

Proof Counting variables N\{t)J e I, can also be defined as series of indicators as follows: ^ ; ( 0 = i:^K=y,r„.}-

(6.17)

From Proposition 6,1, we know that the series in this last relation has a.s. a finite number of terms, whatever 7 and / are. Consequently, we can integrate term by term to obtain:

^(A^;(0|Jo =0 = Z^(^K=.r„.,}ko =0'

(6-18)

Using now relation (3.16), we get that: 00

^(iv;.(o|Jo=0-£er(o>

(6.19)

A7=0

that is (6.15).

n

Using matrix notation with: R = [i?,],

(6.20)

relation (6.15) takes the form: 00

R = ^Q('^).

(6.21)

n=0

Let us now introduce the LS transform of matrices. For any matrix of suitable functions Ay from R'' to R represented by A=[4]

(6.22)

we will represent its L-S transform by: A=[ 4 ]

(6.23)

with A,j(s) = ]e-"dA,j(t).

(6.24)

0

Doing so for the matrix R, we get the matrix form of relation (6.16), 00

RW = Z(QW)"-

(6'25)

From this last relation, a simple algebraic argument shows that, for any s>0, relations R(^)(I - Q(^)) = (I - Q(s)R(s) = I (6.26)

160

Chapter 4

hold and so, we also have that: R(s) = (I-Q(s)r\ We have thus proved the following proposition.

(6.27)

Proposition 6.3 The Markov renewal matrix R is given by (6.28)

R = |;Q('",

the series being convergent in R"^. Moreover, the L-S transform of the matrix R has the form: R = (I-Qr', the inverse existing for all positive s.

(6.29)

The knowledge of the Markov renewal matrix R or its L-S transform R leads to useful expressions for d.f of the first entrance times. Proposition 6.4 For the L-S transforms of the first entrance time distributions, we have:

Inversely, we have: GAs) R>j(s) =

l-G,(s)

-,i^J,

(6.31)

Proof From relation (6.12), we have: Rj,.(s) = (l-Gj^.(s)r, so that:

(6.32)

G..(s) = l - ^ ^ .

(6.33)

Relation (6.11) allows us to write: R,(s) = G,.(s)R.j(s),i^j,

(6.34)

and consequently RJs) GJs) = ^^^—,i^j.

(6.35)

'

Rjjis)

Results (6.33) and (6.35) give (6.30). Relations (6.31) are immediate from relations (6.11) and (6.8).

n

Markov Renewal Processes

161

7 CLASSIFICATION OF THE STATES OF AN MRP To give the classification of the states here, we will proceed as we did in the case of Markov chains: that is, by considering the embedded renewal processes or delayed renewal processes of return times in the different states of/. This gives the following definition. Definition 7.1 The state J of I is said to be recurrent, transient or aperiodic or periodic with period d, Moreover,] is positive {or non-null) recurrent iff jUjj is finite. The next proposition establishes the interaction between classification of the states of an MRP and that of the same states but for the imbedded MC (J^,n>0), Proposition 7.1 (i) j is recurrent for the MRP iff j is recurrent, necessarily positive in the imbedded MC, (ii) // sup 6^. oo in relation (6.16) with i=j, we get: 00

^,(+«^) = I p r

(7.1)

because ^Q^{t)

(7.2)

=j ^ .

But we know (see relation (2.28) of Chapter 3) that in the imbedded MC, statey is recurrent iff the series in relation (7.1) diverges. Result (i) follows then from Proposition 3.3 of Chapter 2 since R,,(oo) = +00 <=> G..(oo) = 1 (7.3) or equivalently that the renewal process \T^ ^i\j\n

^ Oj is recurrent.

(ii) Since sup 6^ < 00 '•J

we can write, for the mean ij,.. related to the d.f G-.:

(7.4)

162

Chapter 4

and moreover, we know from results of Chapter 3, that the mean recurrence time of statey in the imbedded MC is given by: m'^=Z"ZUP...'

(7.6)

!/

n=\

S„ , , A:=0

where ^njj ={{aQ,a^,...,a^),aQ = i,a^ = j,aj^ el,k = l„.,n-l,a,^

^ j].

Assuming now that i=j, we can write from relations (7.5) and (7.6): {rmjiP,,)mjj < ^i.. < irmx/3,,)mjj.

(7.7) (7.8)

The number of states m being finite, every recurrent state in the embedded MC is necessarily positive recurrent, so that mjj is finite and consequently (ii) is true, by relation (7.8), for jUjj . (iii)

From relation (3.16) of Chapter 2, we know that (7.9)

SO that, from relation (7.1):

This last relation proves that

Zp]f<^^G,jM
(7.11)

that is (iii).

n

Definition 7.2 State] of an MRP is accessible from state i, or j can be reached from state i, if there exists a strictly positive t such that: P(Z(t) = j\Z(0) = i)>0. (7.12) Remark 7.1 A statey is periodic for the MRP iff the d.f Gjj is arithmetic. It is clear that the periodicity of a state j for the MRP bears no relation to the periodicity of this state in the embedded MC. For the periodicity in the MRP, Cinlar (1975b) had nevertheless proved the following result: Ify can be reached from / and if/ can be reached fromy, then state / andy are both aperiodic or both periodic and, in this latter case, have the same period.

Markov Renewal Processes

163

Definition 7.3 An MRP will be called irreducible if every state can be reached from any state. From Remark 7.1, it follows that in the irreducible case, all the states are both periodic with the same period or both aperiodic. From Proposition 7.1, we can deduce the following result. Proposition 7.2 An MRP is irreducible iff the imbedded MC is also irreducible, Definition 7.4 An MRP will be called ergodic iff it is irreducible aperiodic, and if the imbedded MC is also aperiodic, The semi'Markov kernel Q corresponding to an ergodic MRP is also called an ergodic kernel. From Remark 7.1, the ergodicity of an MRP implies that of the imbedded MC, but the ergodicity of the imbedded MC only implies the irreducibility of the MRP.

8 THE MARKOV RENEWAL EQUATION This paragraph will extend the basic results related to the renewal equation developed in section 4 of Chapter 2 to the Markov renewal case. Let us consider an MRP of kernel Q. From relation (6.16), we get:

i?,(o=^i/o(o+i:er(o = c5,.(7o(0 + ( e - i ? ) , coUsing matrix notation with: I(0 = [ ^ , ^ o ( 0 ] , (8.2) relations (8.1) take the form: R(r):.I(r) + Q#R(0. (8.3) This integral matrix equation is called the Markov renewal equation for R. To obtain the corresponding matrix integral equation for the matrix

H = [//,], we know, from relation (6.11) that R(0 = I(0 + H(/). Inserting this expression of R(0 in relation (8.3), one obtains: H(0 = Q(0 + Q*H(0 which is the Markov renewal equation for H.

(8.4) (8.5) (8.6)

164

Chapter 4

Of course, for m=l, this last equation gives the classical renewal equation (4.1) of Chapter 2. In fact, the Markov renewal equation (8.3) is a particular case of the matrix integral equation of the type: f = g + Q*f, (8.7) called an integral equation of Markov renewal type (in short MRT), where f-(yi,-,/j',g=(gp->^j' (8.8) are two column vectors of functions having all their components in 5, the set of single-variable measurable functions, bounded on finite intervals or to B^ if all their components are non-negative. Proposition 8.1 The Markov integral equation of MRT, f = g + Q*f with f,g belonging to B^, has the unique solution: f = R*g.

(8.9) (8.10)

Proof a) Existence of the solution Replacing f of the second member of the equation (8.9) by (8.10), we get: g + Q«f = g + Q # R . g , = (I + Q . R ) . g , where the function matrix I is defined as in relation (8.2). Using relation (8.3), we obtain: (I + Q # R ) . g = R . g (8.12) or R.g = g+Q.(R.g). (8.13) This relation proves that the vector given by relation (8.10) is a solution of the given equation (8.9). b) Unicity of the solution Let fpf2 be two solutions of the equation (8.9) and let y be defined as: (8.14) y = f,-f,. Of course, the vector y satisfies the following equation: y = Q*y (8.15) and by iteration, we get for all positive n\ y = Q^"^*y. (8.16) As, from Proposition 6.2, the renewal matrix R can be defined by the function series R = |^Q^'^ which converges for all positive t, we must have

(8.17)

Markov Renewal Processes

165

limQ<"'(0 = 0,/>0.

(8.18)

«->oo

Consequently, as y belongs to 5, we also have: limQ^"^#y(0-0,/>0.

(8.19)

And so by relations (8.17) and (8.14), we get: f\t) = f\tlt>Q, proving so the unicity of the considered solution.

(8.20) n

9 ASYMPTOTIC BEHAVIOUR OF AN MRP We will give asymptotical results, first for the Markov renewal functions and then for solutions to integral equations of an MRT. To finish, we will apply these results to transition probabilities of an SMP.

9.1 Asymptotic Behaviour Of Markov Renewal Functions We know that the renewal function Ry , ij belonging to /, is associated with the delayed renewal process, possibly transient, characterized by the couple {Gij,Gjj)A,f.on R\ Let us recall that ju^j represents the mean, possibly infinite, of the d.f Gij, Proposition 9.1 For all iJ of I, we have:

(i) iiinM^ = -L, RM)-RM-T)

(ii) lim^^

(9.1) T

= — , for every fixed r .

Proof (i) From relation (6.12), we have, for / ^ 7 : R,{t)_G,^R,{t) t

(9.2)

(9.3)

t

Ifj is transient, we know from Proposition 4.2 of Chapter 2, relation (4.25), that: \imG,j»Rjjit) = G,j(c^)-Rjj(oo) (9.4) SO that from relation (9.3): l i m - - ^ ^ = 0. The same proposition also gives us immediately that for ally transient:

(9.5)

166

Chapter 4

l i m ^ ^ = 0. /->oo

(9.6)

f

If y is recurrent, necessarily positive, we have from Corollary 4.2 of Chapter 2 that:

limM)=_L. '^»

t

(9.7)

Mjj

Moreover, from relation (6.11), we have in this case that: G,j.Rjj{t) = G,j>{U,+Hjj){t)

(9.8)

so that by relation (6.12), we get: lim—^^— = lim—^^— + /->00 lim—^^ /->Q0 f /->00 ^

f

— (9.9)

t

'->«•

Now, the key renewal theorem (Proposition 4.2 of Chapter 2) applied to the function G=l - Gy gives as a result that: \imG»Hjj{t)=~\{\-Gjj{x))dx, Mjj 0

(9.10)

_Mij ^jj

Hence l i m ( 7 / , ( 0 - ( ^ . . / / , ( 0 ) = 0.

(9.11)

From the renewal theory, we know that: l i m ^ ^ =—

(9.12)

and thus relation (9.11) gives:

lim^^M)=i t

(9.13)

lUjj

and so, from relation (9.9), we obtain result (9.1). (ii) The proof of (9.2) is based on the same argument as the one used for proving (i) and on Blackwell's theorem (see Chapter 2, Corollary 4.3). a The next proposition, due to Barlow (1962), is a useful complement to the last proposition as it gives a method for computing the values of the mean return times lUjjJ G / , in the ergodic case. Proposition 9.2 For an ergodic MRP, the mean return times satisfy the following linear system:

Markov Renewal Processes

167

In particular, for i=j, we have: jUjj=—Y,7r,7j„j = l,.,.,m, ^j

(9.15)

k

where the Tj^Jel are defined by relation (3.14), and where 7r = (;7r,,...,;7r^) is the unique stationary distribution of the imbedded Markov chain. Proof Relation (6.9) immediately gives: l^ij=Pijbij-^T.Pik^^ik^^kj)

(9.16)

k^j

or using relation (3.14) /^ij=Y,Pikf^kj-^ri.,

(9.17)

k^j

that is (9.14). Now, let ^i be the matrix \l^A , and let ^^,Ho b^ respectively the two matrices obtained from |LI as follows; for \i^, in setting all diagonal elements equal to 0 and for n^ in setting all non-diagonal elements of \i equal to 0. It follows that: fi = fi^+jio(9.18) Now, under matrix notation, the linear system (9.14) takes the form: ji = Pfio+il, (9.19) where ii is the mxmmatrix with all columns identical to the vector [T]],'",r/^) . From relations (9.18) and (9.19), we get ^,=(P-I)jio+il. (9.20) Premultiplying both sides of this equality by the vector n, we obtain : nii^=nr\, (9.21) since nP = n (9.22) by relation (5.5) of Chapter 3. Moreover, as for all j , TTJ is positive (see relation (5.17) of Chapter 3), relation (9.21) gives relation (9.15). Remark 9.1 In a similar manner. Barlow (1962) proved that if iuf\ij

n G / is the

second order moment related to the d.f Gtj , then:

//r=^/''+ZA.(/^,f+26,,//,.) k^j

and in particular for i=j\

(9.23)

168

Chapter 4

^ =^{Y.^,rif^ ^2YZ^,P,AM,) ^j

k

(9.24)

k^j I

with rif^= \x^dH,{x\keI,

(9.25)

[O.oo)

provided that these quantities are finite.

9.2 Asymptotic Behaviour Of Solutions Of Markov Renewal Equations Under the assumptions of Proposition 8.1, we know that the integral system (8.9), that is y;(0-g,(0 + Z \fj{t-s)dQ,j{s\ieI, (9.26) has the unique solution ^W = E \gj{t-s)dR,j{s\ieL

(9.27)

J [0./]

We will now study the asymptotical behaviour of this solution for t tending toward +oo, and we will establish the analogue of Proposition 4.2 of Chapter 2, i.e. the key renewal theorem. Proposition 9.3 {Key renewal theorem) For any ergodic MRP, we have:

l i m ^ \g(t-s)dR,(s)^^^ '"°° i [0,]

,

(9.28)

L^jm J

provided that the functions gu i belonging to /, are directly Riemann

integrable.

Proof From relation (6.12), we have for i^j\ \gj{t-s)dR,j{s) = gj*G,.*Rj.{t) [0,'] (9.29) =(g,«G,)./?,.(/). From properties of direct integrabihty (see Cinlar (1975b)), since G/, is a d.f, gj • G^j is also directly integrable. Consequently, we can apply the key renewal theorem (Proposition 4.2 of Chapter 2) saying that:

Markov Renewal Processes

169

= —]gj.G,j(T)dT,

\im(gj.G,j).Rjj(t)

(9.30)

Mjj 0

Permuting the order of integration in the double integral of the second member, and using after the change of variable T-U = y ,WQ get: 00

00

CO

jgj • G^j (T)dT = jdGy (u) jgj (T - u)dr °

°

"

00

00

=

(9.31)

jdG,j(u)lgj(y)dy. 0

0

But the assumption of irreducibility of the MRP implies that Gy is a d.f onR"^, and thus: ]gj'G,j(T)dT 0

= ]gj(y)dy.

(9.32)

0

Going back to equality (9.30) and using relation (9.15), we finally get (9.28).

10 ASYMPTOTIC BEHAVIOUR OF SMP 10.1 Irreducible Case Let us consider the SMP (Z(/), / > 0 ) associated with the MRP of kernel Q and defined by relation (5.8). Starting with Z(0) = /, it is important for the applications to know the probability of being in statey at time /, that is: (10.1) ^.(0 = P ( Z ( 0 = ;|Z(0) = /). A simple probabilistic argument using the regenerative property of the MRP gives the system satisfied by these probabilities as a function of the kernel Q:

,(t) = s,j(\-HXt))+z]M^-y)dQ,(y), k

Uj^i.

(10.2)

0

It is also possible to express the transition probabilities of the SMP with the aid of the first passage time distributions G^jJJ G / : ^.(0 = ^ ^ . G , ( 0 + ^.(l-//,(0)> hj^l(10.3) If we fix the valuey in relations (10.2), we see that the m relations for i=\, ..,,m form a Markov renewal type equation (in short MRE) of form (8.9). Applying Proposition 8.1, we immediately get the following proposition. Proposition 10.1 The matrix of transition probabilities

170

Chapter 4

<^ = [
(10.4)

is given by 0 = R#(I-H) with H = [^.//,].

(10.6)

So, instead of relation (10.3), we can now write: (2^.(0= \{\-Hj{t-y))dR,j{y\

(10.7)

(10.5)

Remark 10.1 Probabilistic interpretation of relation (10.7) This interpretation is analogous to that of the renewal density given in Chapter 2, Remark 10.2: the "infinitesimal" quantity dRyiy) (=rij(y)cly, if ryiy) is the density of the function Ry, if it exists) represents the probability that there is a Markov renewal into statey in the time interval (y,y+dy), starting at time 0 in state /. Of course, the factor (1 -Hj{t-y)) represents the probability of not leaving statey before a time interval of length t-y, The behaviour of transition probabilities of matrix (10.4) will be given in the next proposition. Proposition 10.2 Let Z = [Z(t),t>0)be MRP of kernel Q; then:

\im(t>,(t)=^A-,

i,jel,

the SMP associated with an ergodic

(10.8)

k

Proof For a fixed value of7, we know that the ^^., 7 =l,...,m satisfy the MRT integral equation (10.2). As the basic MRP is supposed to be ergodic, we can apply the key Markov renewal theorem (Proposition 9.3) to get: CO

lim4.(0 = ^

,

(10.9)

k

and since 00

\{\-Hj{y))dy

= VjJ^I,

(10.10)

0

relation (10.8) follows.

n

Markov Renewal Processes

171

Remark 10.2 (i) As the limit in relation (10.8) does not depend on /, Proposition 10.2 establishes an ergodic property saying that: lim<2^.(0 = n , ,

^^J^jVj_

(10.11)

k

(ii)

As the number m of states is finite, it is clear that (llj, y e / j is a probability

distribution. Moreover, as TTJ > 0 for ally (see relation (5.17) of Chapter 3), we also have n,>0,yG/.

(10.12)

So, asymptotically, every state is reachable with a strictly positive probability. (iii) In general, we have: \\mpf^\\m(j)M) (10.13) /7->oo

-^

/->00

•^

since of course 71J ^Uj J el,

(10.14)

This shows that the limiting probabilities for the imbedded Markov chain are not, in general, the same as taking limiting probabilities for the SMP. From Propositions 10.2 and 9.2, we inmiediately get the following corollary. Corollary 10.1 For an ergodic MRP, we have: n.=^.

(10.15)

This result says that the limiting probability of being in state y for the SMP is the ratio of the mean sojourn time in statey to the mean return time ofy. This intuitive result also shows how the different return times and sojourn times have a crucial role in explaining why we have relation (10.14) as, indeed, for the imbedded MC, these times have no influence.

10.2 Non-Irreducible Case It happens very often that stochastic models used for apphcations need nonirreducible MRP, as for example, in presence of an absorbing state, i.e. a statey such that Pjj=L (10.16)

172

Chapter 4

We will now see that the asymptotical behaviour is easily deduced from the irreducible case studied above. 10.2.1 Uni-Reducible Case As for Markov chains, this is simply the case in which the imbedded MC is unireducible so that there exist / {l<m) transient states, and so that the other m-l states form a recurrent class C. We always suppose aperiodicity both for the imbedded MC and the considered MRP. Let T= {l,...,/} be the set of transient states {T = 1 -C) .From Proposition 5.1 of Chapter 3, we know that: \im<j>,j{t) = 0,iJsT. (10.17) Moreover, from Proposition 10.2 and relation (10.3): lim(Z^(0-G,(a)) ^^"^^

, /,7GC,

(10.18)

where (;r/^,,...,;r^) represents the unique stationary probability distribution of the sub-Markov chain with C as state space. Since (10.19) G,M = f,, we get (10.20) Gy(^) = f,c^ whereyjc is the probability that the system, starting in state / will be absorbing by the recurrent class C. As there is only one essential class, we know that for all states / of/: fi.c=h (10.21) proving so the following proposition. Proposition 10.3 For any periodic uni-reducible MRP, we have: ]im(/>^j(t) = n\,jel,

(10.22)

where '0, 7 E r ,

n; = -IJH^J^C,

(10.23)

Here too, as the limit in (10.23) is independent of the initial state /, this result gives an ergodic property.

Markov Renewal Processes

173

10.2.2. General Case For any aperiodical MRP, there exists a unique partition of the state space /: / = rUC,U---UC,, r < m , (10.24) where T represents the set of transient states and C^,v = l,...,r represents the V th essential class necessarily formed of positive recurrent. From Chapter 3, we know that the system will finally enter one of the essential classes and will then stay in it forever. So a slight modification of the last proposition leads to the next result. Proposition 10.4 For any aperiodic MRP, we have :

lim^.(0 = n , , / , 7 e / ,

(10.25)

with, for any j eC^,v = l,...,r : f Yij,

n,=

0,

ieC^,v = l,..,,r,

/ E C , . , v ^ v > ' = l,...,r,

[f.cn'j,

(10.26)

ieT

where (TIJ ,j e C^)is the only stationary distribution of the sub-SMP whith C^as State space, that is :H. = ^

—,

(10.27)

where yTT^^k e C^ j is the unique stationary distribution of the sub-Markov chain with C^ as state space and [f^c,^^ eTjis the unique solution of the linear system

yi-lLp,yj-lLp,^i^T. J^T

00.28)

jeC,

Note that, in this proposition, the ergodic property is lost; this is due to the presence of the quantities fc ^^ relation (10.26).

11 RECURRENCE TIMES 11.1 Definitions For an MRP [(J^,TJ,n>0),

the random sequence of renewal epochs

((r^),^>0) still represents as in renewal theory, and the random sequence ((X^),/7>0) represents still successive Hfetimes and so the supplementary

174

Chapter 4

information with respect to renewal processes is given by the random sequence [(J^),n>0), the r.v. J^ giving the state in which the HfetimeX„+i was spent. Also, we can define recurrence times exactly as we did for renewal processes in section 8.1 of Chapter 2. Thus, r.v. 5{t) will give the age of the "component" actually living at time / and r.v. y{t) the time that this "component" has still to wait until its failure, so that we get the following definitions. Definition 11.1 For each t>0, the r.v. 5{t) and /(t) defined by the following relations:

are respectively called the age {or backward recurrence time) and the excess {or forward recurrence time or residual time) at time t.

11.2 Transient Results First we will give some transient results related to the two stochastic processes d = {5{t)j>Q), (11.2) r^{y{t\t>Q), respectively called the age and the excess processes. Proposition 11.1 {Distribution of the age) Let 7^.,(,)(x) = P(cJ(0<x,Z(0 = 7|Z(0) = /),

(11.3)

then \

P,j,x>t,

Pij,s(,)ix) = \ j (l-Hj(t-u))dH.j(u),x
01'4)

and

/^.,,)(0 -limi^,,(o(^) = 4 0 -^-(0)-

(11-5)

Proof The first result of relation (11.4) and relation (11.5) is obvious. To prove the second result of relation (11.4), it suffices to remember the probabilistic interpretation of Markov renewal functions given in Remark 10.1 and to apply the regenerative property of MRP. n

Markov Renewal Processes

175

Proposition 11.2 {Distribution of the excess) Let Fy,riO W = Hr(0 > ^,Z(t) = j\Z(0) = i), then F,,,,„(x)=](l-Hj(t

(11.6)

+ x-s))dR,j(s).

(11.7)

0

Proof Given that there is a renewal into state j between s and s+ds, the probability of the event {/(t)>x]is that of the event \^Xj^^^>^>t + x-s^(sQQ Figure 8.2 of Chapter 2). Then using the regenerative property of MRP, we get relation (11.7). n Proposition 11.3 (Bivariate distribution of both the age and the excess) Let

F,,so),m (^^y) = ^(^(0 > ^.r(0 > yMt) = j\z(0) = /), then \s,),r,My)-

\ (l-Hj(t-i-y-s))dR,j(s).

(11.8) (11.9)

[0,i-x]

Proof On looking at Figures 8.1 and 8.2 of Chapter 2, and in referring to the n preceding proof, it is seen by analogy that (11.9) is correct. A similar argument leads to the following proposition taking into account the next state occupied aftery. Proposition 11.4 (Trivariate distribution of both the age, the excess and the successive state) Let

^ijk,sit),yit)i^^y) = P(^(0 > x,rit) > y,z(t) = j\j,,,,,,

=k\z(0) = /),

then ^ijuolrio(^^y)=

j Pj,{l-FAt

+ y-s))dR,j(s),

(11.11)

11.3 Asymptotic Results The asymptotical behaviour of d.f obtained in the last propositions will easily be deduced from the Markov renewal theorem (Proposition 9.3).

(i i.io)

176

Chapter 4

Proposition 11.5 For any ergodic MRP, we have Zu^klk k

[;c+>',oo)

Proof From result (11.9) of Proposition 11.3, we can write the value of the function ^ijk,s(t),r(t) ^^ point (x,y) as the convolution product of the renewal function Rjj and the function defined by sh^(l-Hj(x + y + t-x-s)),

(11.13)

the convolution product being so computed at point t-x. Using the key Markov renewal theorem with G,(.) = S,j{l-H,(x + y + .)),

(11.14)

we find _

00

\imF,j,,,^^^,^(x,y)^^^^{{l-H.(x + y + s))ds.

(11,15)

k

Now, to the last integral, it suffices to apply the change of variable s' = x + y-\-s n and relation (11.12) is easily obtained. Starting with the result of Proposition 11.4, a similar argument gives the next result. Proposition 11.6 For any ergodic MRP, we have:

{x^y) = .^^i— ]Y,p.,(\-Fj,{s))ds, limF, ,^00 ^•'^(o./co

(11.16)

Corollary 11.1 For any ergodic MRP, we have: lim^,,,,(x) = lim^,,,,(x) = =^]X;.,,(l-iV.(^))^^.

(11.17)

Proof If wefix,for example, x in both members and take the limit for j^ -> oo in relation (11.12), we get relation (11.17). n A similar argument gives the other result. Let us now define the function //and the number 77 as follows:

Markov Renewal Processes

177

'

(11.18)

Using Proposition 11.5 and Corollary 11.1, and bearing in mind that 77 is the mean related to the d.f. of//, the next result is immediate. Remark 11.1 As in renewal theory recurrence times ^(Oand y{t) have the same limit distribution, and moreover, this limiting distribution is independent of the initial state (ergodic property). Corollary 11.2 For any ergodic MRP, we have: .

00

limP(^(0>x,;'(0>>'|Z(0) = i) = - \{\-H{s))ds

(11.19)

and lim/^(cJ(0 < x|Z(0) = /) = limP(;^(0 < x|Z(0) = /)) = - j ( l " H{s)Ys.

(11.20)

Particular cases This last result shows that the common limiting distribution of the r.v. d{t) and y{t) has as density the function defined by: -{\-H{x))

(11.21)

and thus the mean of the limit distribution is given by 2

2/7

where cr^ is the variance related to the d.f //. To compute this variance a^ in terms of kernel Q or of doublet (P,F), we use the following notation : cc2jj--\x^dF^j{x),iJsI, 0 CXD

a,. = jx^dH.(x),i,eI,

(11.23)

0

^2 = fxV//(x), 0

so that from the second equality of relation (3.11), relations (11.18) and (11.23), we clearly have:

Chapter 4

178

J

(11.24) J

Using now relations (11.24) and (11.18), we can finally give the expression of (j^ as follows:

^_

V

v /

y

(11.25) /•

J

In the special case of F.j=F., i,JGl,

(11.26)

the preceding relations give for all / andy: ^2,ij

~

^2,i'

(11.27)

a^ = Z^/^2,/Similarly, for the second special case of

F.j=F.,i,jeI,

(11.28)

we get i

J

(11.29) So, in both particular cases (11.26) and (11.28), formula (11.25) becomes:

J

(11.30)

V J

12 DELAYED AND STATIONARY MRP Let us suppose we begin to observe the evolution of an economical or physical system 5* at time TQ and that the probabilistic evolution of this system is like a semi-Markov process. There is absolutely no reason that we should observe a transition of the system at time 0. In fact, we observe the state Z(0) while waiting for the first observed transition occurring at random time T\. The first lifetime Xi is a residual time and may have its own distribution function. This leads to the concept of delayed MRP. Definition 12.1 The bidimensionalprocess

(^(J^,T^),n>0^with

Markov Renewal Processes

179

T =0 ' ' (12.1) T„=X,+- + X„, n>\, is called a delayed Markov renewal sequence or delayed Markov renewal process {in short DMRP) of triplet (p,Q,Q) if (i)P(Jo=i) =

Pi,i^L

(ii)P(j, =j\X,<x\j,

=i) = aj(x), ij'el,

(12.2)

This definition is clearly based upon the supposition that the m-dimensional vector p = (/?j,...,j[?^)represents a probabihty distribution on / and that matrices Q,Q are two semi-Markov kernels. As in the case of renewal theory, there exist simple relations between renewal functions, marginal distributions, etc... of a DMRP of triplet (p,Q,Q) and the corresponding functions of the classical associated MRP of kernel Q. So, with the convention of adding a tilde to the functions related to the DMR, let Ry(t) be the Markov renewal functions of the DMRP, that is: R,j(t) = E(Nj(t)\j,^i).

(12.3)

We know that: R,j(t)^±P{j„=j,T„
=Ia.-er'(o,

(12.4)

n=\ 00

=Ia.-^,(o. If we now consider the transition probabilities for the delayed semi-Markov process (Z(t),t > O), associated with the DMRP of triplet ( P , Q , Q ) ,that is: ^,.(t) = p(z(t) = j\z(0) = i),

(12.5)

using as usual, a simple probabihstic argument, we get:

^.(o=c5;,(i-//,(0)+ZK(^->^)^a.w.

(12.6)

k=\ 0

where of course, the ^^^. are the transition probabilities of the SMP of kernel Q and with m

Hjit) = Y,Qj,{t)J^I,t>Q. k=\

(12.7)

180

Chapter 4

Using Proposition 10.1, relation (12.7) gives m

},j{t) = 5,j{\-Hj{t)) + Y.{\-Hj)*R,j*Q,M

(12.8)

and from relation (12.4): ^,j{t) = 5y{\-Hj{t)) + {\-Hj)>R,j{t).

(12.9)

Relation (12.6) also shows that the limiting distributions of the transition probabilities ^^ ,/,7 e / exist and are known provided the limiting distributions of the transition probabilities (Z>^ ,/,7 e I exist and are known too. Indeed, let us suppose that lim^.(0 = n , , ijel, /->oo

-^

(12.10)

-^

then, from relation (12.7), the limits lim^.(0 = n , , / , 7 e /

(12.11)

also exist and moreover they are given by m

%=t.~PAjJ,j^I,

(12.12)

k=\

where P,-Qyi+^(12.13) Using now Proposition 10.2 and relation (12.12), we get the following result. Proposition 12.1 If the MRP associated with the DMRP of triplet (p,Q,Q] is irreducible, then lim^.(0 = n , ; / , y 6 /

(12.14)

with Yl.^^^-^J^L

(12.15)

k

It follows that in the ergodic case, both DSMP and associated SMP have the same asymptotical behaviour. In renewal theory, a very special but very interesting case of the notion of DMRP, is the case of the so-called stationary MRP (in short SMRP).This type of process appears when one begins to observe an MRP which has been running a long time so that the first observed r.v. X\ is in fact the excess y . If the kernel Q is ergodic, it is thus natural to suppose that this excess y has its limit distribution.

Markov Renewal Processes

181

More precisely, let us define the vector p^ whoseyth component (j=l,...,m) is given by Ps,=^j (12.16) and let us define the semi-Markov kernel Q^ as follows:

x<0. 0, We can now give the following definition. Definition 12.2 The DMRP of triplet (pjjQ.Qs) yvith kernel Q ergodic, is called a stationary MRP {in short SMRP) of kernel Q. To verify that this definition corresponds well to what we want to observe, it suffices, from Proposition 11.6, to show that, for an SMRP (fJ„,X^),n> O), we have the following equality: P{J, ^k,X, <x)^^^^7rj\pj,i\-Fj,{y))dy.

(12.18)

V

Indeed, we have:

P(/, ^KX, <x) = X/'wa,,.W-

(12.19)

So, by relations (12.16) and (12.17): P{J, =KX, <x) = =^—Y^7rj\{pj,-Qj,{y))dy,

(12.20)

V

which is equivalent to (12.18). The next propositions give specific properties of SMRP. Of course, r.v. Nsjif) represents the total number of passages in statey on [0,/] for the considered SMRP. From this point on, we will systematically use the subscript "S" for parameters related to the SMRP. Proposition 12.2 For every SMRP of triplet (p^jQsQ^) with kernel Q ergodic, all renewal functions are linear. More precisely: E(N,^j(t)) = ^t,jeI.

(12.21)

182

Chapter 4

Proof If R,j(t) = E(N,j(t)),

(12.22)

we have, with the smoothing property of conditional expectation (see Chapter 1): E(E(N,j(t)\zm, R,j(t) = =E(R,_,,,,j(t)),

(12.23)

i

this last equality coming from relation (12.16). By relation (12.4), this last equality gives us: ^.,y«-SZn.a„.*^.y(0. /

(12.24)

k

Taking L-S transforms of both members, we get: ^^,y(^) = E Z n - a , « W ^ . y W /•

(12.25)

k

But the L-S transform of kernel Q^ is given from relation (12.17) by

•—- k " a . w ^ I 0

(12.26)

J

and by integration by parts: srj. So, going back to relation (12.25), this last result leads to the following relations:

^.,;(^)=7ZZ—(A. -Qi,im,{s) ^ • ' ^'

(12.28)

=-ZZ^AA(^)--I-(^(^)-^)From relations (11.18) and (10.11), we have that:

n. ^ -L = ^,iel,

(12.29)

and using moreover relation (9.22) in relation (12.27), we finally get: T

fJi

STJT

STJ

^..;(^)=-ZZ—A*^^.(^)--Z^-^(^) + —

(12.30)

_n, STJ

Using inverse L-S transforms, we finally get the result to be proved, that is relation (12.21). n The next result is an immediate consequence of the last proposition.

Markov Renewal Processes

183

Corollary 12.1 For every SMRP of triplet (p5,Q,Q5) w/V/^ kernel Q ergodic, we have: ( "

-.

(12.31)

This corollary shows that in looking at the total number of transitions, every SMRP is, on the average, equivalent to a stationary renewal process defined by the d. f //given by: m

H^J^TTjHj.

(12.32)

The linearity of Markov renewal functions related to ergodic associated SMRP will give, as the main consequence, the stationarity of the basic stochastic processes related to it. This means that all marginal distributions will no longer depend on t, and will thus have values given in subsection 11.3. To be comprehensive, we will show how these results could be proven for simple basic processes related to excess and age processes and then extrapolate for the general result. Proposition 12.3 For any ergodic SMRP defined by kernel Q, the stochastic process (^Z(t),y(t),t > O) is stationary and P{Z(t) = j\r(t)<x)

= ^]{l-H.(y))dy, ^J

(12.33)

0

Proof As the probabilistic interpretation of Markov renewal functions remains unchanged for SMRP, we have

p(z(t)^j,r(t)<x\z(0) = i) V

(12.34)

0

Using now relation (12.16) the value of the unconditional probability of the first member of relation (12.33) becomes: p{Z(t) = j\r(t)<x)

= ^^^(H,j(t '

1 ^

V

7 /=1

0

By Proposition 12.2, we know that

+ x)-H,^j(t)) (12.35)

184

Chapter 4 Yl

1 m

-YmRsAy)=—''

(12.36)

and by relations (12.17) and (3.17), we also have that: H,,{t) = — \{\-H,{y))dy. ^j

(12.37)

0

Inserting these last two results into result (12.35) and recalling from (12.28) that:

-Z^,'7,=In,.=l,

(12.38)

we obtain: P{Z(t) = j,r(t)<x) +-'£-iM.njU-(l-Hj(t ^U. ^

j(^i-H.(y))dy t

=

^l(l-Hj(y))dy + x-y)) + (l-Hj(t-y)))dy

+ -^\ j(l-Hj(z))dz+ ^J \0

So that finally, we get result (12.33).

(12.39)

J(1 -//,(z))c/z t+x

n

As an immediate consequence of this final proposition, we find that the marginal distributions of processes (Z(t),t>0),(y(t),t^O) are stationary, getting thus the next corollary. Corollary 12.2 Under the assumption of Proposition 12.3, we have for all positive t: njJeI, P{Z(t) = j) = 1 "r

(12.40)

P{r(t)<x)^-j(l-H(y))dy,x>0. V 0

Here too, we see that, in observing the excess process y , everything happens as though we had a stationary renewal process characterized by the d.f //, defined by relation (12.32). Remark 12.1 From relations (12.2), (12.16) and (12.17) and (12.29), we get:

Markov Renewal Processes

i

185

J

'•

j

'li 0

=-YLA^Pij-QiM))dy ^

i

7

/

J

(12.41)

0

0

^ 0

This is a result to be expected as in a stationary SMP, the first observed variable X\ is actually the excess of the running process at the origin of the observations. Nevertheless, as p^ ^TT , the process (J^,n>0), and consequently the process (X^,n > 0), are not usually stationary.

13 PARTICULAR CASES OF MRP We will devote this paragraph to particular cases of MRP having the advantage to lead to some explicit results.

13.1 Renewal Processes And Markov Chains For the sake of completeness, let us first say that with m=l, that is that the observed system has only one possible state, the kernel Q has only one element, say the d.f. F, and the process (Xn,n>0) is then a renewal process. Secondly, to obtain Markov chains studied in Chapter 3, it suffices to choose for the matrix F the following special degenerating case: F,j=U,yij'eI (13.1) and of course an arbitrary Markov matrix P. This means that all r.v. X„ have a.s. the value one, and so the single random component is the (/„) process, which is, from relation (3.4) a homogeneous MC of transition matrix P.

13.2 MRP Of Zero Order (PYKE (1962)) There are two types of such processes

186

Chapter 4

13.2.1 First Type Of Zero Order MRP This type is defined by the following semi-Markov kernel Q-[p>F^,

(13.2)

SO that:

p^=p^,F^=F,JeL

(13.3)

Naturally, we suppose that for every i belonging to I,piis strictly positive. In this present case, we have that the r.v. J^,^ > 0 are independent and identically distributed and moreover that the conditional interarrival distributions do not depend on the state to be reached, so that, by relation (3.11), H^=F,,isL (13.4) Moreover, since: P(X„ <x\(J„X,),k
P(X„<x\(X,),k
(13.6)

Introducing the d.f. F defined as m

(13.7)

F = 1LPJFJ, y=i

the preceding equality shows that, for an MRP of zero order of the first type, the sequence {X^,n> 1) is a renewal process characterized by the d.f. F, 13.2.2 Second Type Of Zero Order MRP This type is defined by the following semi-Markov kernel Q = [PiFj\

(13.8)

so that: Py=p,,F^=Fj,iJ^L

(13.9)

Here too, we suppose that for every / belonging to I,piis strictly positive. Once again, the r.v. y^,«>Oare independent and equidistributed and moreover the conditional interarrival distributions do not depend on the state to be left, so that, by relation (3.11) m

H,=YpjFj{=F\i^L

(13.10)

y=i

Moreover, since: P{X„<x\{J,,X,\k
= Fj^{xl

(13.11)

Markov Renewal Processes

P{X„<x\{X,lk
187

= Y^PjFj{x) = F{x),

(13.12)

7=1

The preceding equality shows that, for a MRP of zero order of the second type, the sequence {X^,n > 1) is a renewal process characterized by the d.f F as in the first type; The basic reason for these similar results is that these two types of MRP are the reverses (timewise) of each other (See next chapter)

13.3 Continuous Markov Processes These processes are defined by the following particular semi-Markov kernel Q(x) = [j.,(l-e-^'^)],x>0, (13.13) where P = r/?^y] is a stochastic matrix and where the parameters

A-JelavQ

strictly positive. The standard case corresponds to that in which p.. =0,iel (see Chung (I960)). From relation (13.13), we get: F,j(x) = l-e-''\ (13.14) Thus the d.f of sojourn time in state / has an exponential distribution depending uniquely upon the occupied state /, such that both the excess and age processes also have the same distribution. For m=l, we get the usual Poisson process of parameter A.

14 A CASE STUDY IN SOCIAL INSURANCE (JANSSEN (1966)) 14.1 The Semi-Markov Model We will now return to the problem presented in section 7 of Chapter 3 which was solved using a Markov chain model. Here, we will extend the model to a semiMarkov one giving the possibility to take into account the duration of passage from invalidity degree Sj to invalidity degree Sk. In this case, relation (7.2) of Chapter 3 is replaced by the following one:

s;(t)='£^,it)s^..

(14.1)

The study of the financial equilibrium of the funds thus depends on: S;=lhnS:(t),

(14.2)

/->oo

or

188

Chapter 4

S;=Y,^im
(14.3)

7=1

It suffices here to apply Proposition 10.4 in order to get the result in the general case. For the case of uni-reducibility, with P represented by relation (7.5) of Chapter 3, we get: S;=t^jS.,

(14.4)

7=3

with

n,=V^.

(14-5)

ifc=3

(7r^,7r^,7r^) being given by relation 3(7.9) and with

Pjj^ being the mean of the d.f Fjk related to the duration of passage from invalidity degree Sj to invalidity degree Sk> Here too, although we have uni-reducibility, the asymptotical state S. is independent of the initial state /. To pass from the Markov chain model to the semi-Markov one, the additional information needed is the knowledge of the matrix B = \Pij\ -

14.2 Comparison of Markov And Semi-Markov Models To compare both models in the same application for the obtained asymptotical results, let us introduce the following inequalities resulting from relation (14.6): inin^., <77. <max;5.,, min min y5.^ < min r/^ < min max J3.^, /

k

i

i

(14.7)

k

max min J3.^ < max TJ. < max max /?.^. /

k

i

Consequently, relation min min /?.^ ^ ^ TV. < n . max max y5.^ i

k

k

(14.5) gives us: max max p.,^ <-~^ ^ ,7=3,4,5. min min y5.^ i

Supposing max max /?.^ A=^ ', min min /?.^ /•

i

k

(14.8)

k

(14.9)

Markov Renewal Processes

189

relation (14.8) takes the form: (14.10) —;r, < n , < / i ; r , . A ' ' ' Now if 5,-S"represent respectively limit invalidity degrees, then result (14.4) implies that:

A-'S<S'
(14.11)

These last two double inequalities lead to the following conclusions: (i) if ^ =1 (iff all the J3y are equal), then S =S\ and so both models give the same result, (ii) if ^ wl, (iff all the J3.j are slightly different), then S ^S\axid

so both

models give the same result, (iii) if ^ » 1, (iff all the /?^. are largely different), then 5,-5" will be in general quite different and so these models give largely different asymptotical results. This last case shows that the passage from a classical Markov chain model to a semi-Markov one can really bring important improvements as indeed, here from the practical point of view, it is clear that we are in this last case.

15. ASYMPTOTIC NUMERICAL EXAMPLES In this paragraph we return to the examples mentioned at the end of Chapter 3 to give the results obtained in the Markov case and also in a semi-Markov environment. First we have the asymptotic result related to the disability example shown in section 7 of Chapter 3. The means r/.J e I were computed from real data from Campania; that is to say an Italian region with more than 4 million inhabitants and were applied to the M.C. used for the example. The semi-Markov asymptotic limit vector is given in Table 15.1. states

7,

n,

1 2 3 4 5

2.00822 3.35343 3.34247 3.46575 3.32603

0.00000 0.00000 0.22000 0.34217 0.43783

Table 15.1: disability n,

Chapter 4

190

The SMP version of motorcar insurance will not be presented. The Bonus-Malus environment is a typical Markovian example and the state changes are made at a fixed time, each year. No randomness is present on the transition time. The three examples given in section 8.3 of Chapter 3 are presented. These examples do not descend from real data but they show, as specified in the previous chapter, a way to understand how a slight change in the embedded M.C. can change the results in the asymptotic behaviour of the studied SMP. Before these three examples, we present in Table 15.2 the uni-reducible case also with the 77.,/ e / that were given to all three cases. 1 states

1 2 3 4 5 6 7 8 9 10 11 1 12 13 14 15

7/ n, 1.8 0 1.5 0.48079 2 0 0 2.3 2.3 0.30807 0 2.1 0 1.8 1.9 0.21115 0 1.7 0 2.3 2 0 0 1.6 0 2.6 2.4 0 1.9 0 1

Table 15.2: uni-reducible case Tables 15.3 and 15.4 report the asymptotic results related respectively to Tables 8.13 and 8.19 of Chapter 3.

1 2 3 4 5 6 7 8 9 10

2

5

6

8

0.28628 0.48079 0.29924 0.39787 0.48079 0.00000 0.33874 0.48079 0.38633 0.32185

0.18344 0.30807 0.19174 0.25493 0.30807 0.00000 0.21705 0.30807 0.24754 0.20623

0.21909 0.00000 0.20449 0.09340 0.00000 0.54155 0.15999 0.00000 0.10640 0.17902

0.12573 0.21115 0.13142 0.17473 0.21115 0.00000 0.14877 0.21115 0.16966 0.14135

11 0.18547 0.00000 0.17311 1 0.07907 0.00000 0.45845 0.13544 0.00000 0.09007

0.15155 1

Markov Renewal Processes

rrr 0.00000

191 0.00000

0.54155

0.00000 0.45845

12 0.38715 0.24807 0.10547 0.17003 0.08928 13 0.29466 0.18881 0.20965 0.12941 0.17748 14 0.26806 0.17176 0.23961 0.11773 0.20284

1 15 0.34809

0.22304 0.14947 0.15287 0.12653 Table 15.3:1 reducible case

1 2 3 4 5 6 7 8 9 10 11 12 13 14

2

5

6

0.28482 0.48078 0.29831 0.39787 0.48078 0.00000 0.33739 0.48078 0.38434 0.30843 0.00000 0.38202 0.29465 0.26580

0.18250 0.30807 0.19114 0.25493 0.30807 0.00000 0.21619 0.30807 0.24626 0.19763 0.00000 0.24478 0.18881 0.17032

0.22073 0.00000 0.20554 0.09340 0.00000 0.54155 0.16152 0.00000 0.10864 0.19414 0.54155 0.11125 0.20965 0.24215

15 10.34809 10.22304 10.14947

8 0.12509 0.21115 0.13101 0.17473 0.21115 0.00000 0.14817 0.21115 0.16879 0.13545 0.00000 0.16777 0.12941 0.11673 10.15287

Table 15.3: II reducible case

11 0.18686 0.00000 0.17400 0.07907 0.00000 0.45845 0.13673 0.00000 0.09197 0.16435 1 0.45845 0.09418 0.17748 0.20500

0.12653 1

Chapter 5 FUNCTIONALS OF (J-X) PROCESSES 1 (J-X) PROCESSES In Chapter 4, we introduced the concept of {J,X) process with Definition 2.4 for which the r.v. X^, ^7=1,2,...take their values in the whole real line R instead of in M^ for what we called in Definition 2.2 di positive {J,X) process. The whole semi-Markov theory presented in Chapter 4 starts from a positive {J^ process. In fact, in 1969, Janssen showed that the consideration of (J,X) processes leads to a very interesting generahsation of the classical concept of random walk with a lot of applications in stochastic modelling. Let us begin this chapter by recalling the basic definition. Definition 1.1 Let p=(pi,'.',Pm) be an m-dimensional vector of initial probabilities and Q an extended semi-Markov kernel as defined by Definition 2.3 of Chapter 4, then every two-dimensional process ((J^,X^),n = 0,1,...)with values in / x R and satisfying the conditions P{Xo^O)=\,a.s., m

P(Jo=i)=pu i=\,.,.,mwith

^p^ =1,

for alln>0,j=l,...,m, we have: P(J„ =j,X„ <x\iJ,,X,),k = 0,...,n-l)^Qj^^jix),a.s.,

(1.1) (1.2)

is called a (J,X) process or an extended semi-Markov chain (in short, ESMC). From this definition, it follows that we can no longer represent the sample paths of such a process with step functions as the r.v. X^ can be positive or negative but we can see the 5'-process defined by: S„=X,+X,+-+ X^,n = 0,l,... (1.3) as the successive positions of a particle moving on a real line and starting from the origin if the r.v. Xn n=0,\,... represents the successive steps of this random movement exactly as the interpretation of a classical random walk corresponding to the case of m=l. This leads to the following definition. Definition 1.2 The S-process defined by relation (1.3) is called a semi-Markov random walk (in short an SMRW).

194

Chapters

It is clear that basic results on positive {J^ processes given in the preceding chapter are still valid here provided that these properties do not involve the nonnegativity of the Xn, The following proposition summarises the basic properties. Proposition 1.1 {Basic properties of {J,X) processes) (i) The process [(J^,S^),n>0) is a Markov process with / x M a^ state space; more precisely, we have a.s. for allj of I and all real x: P(j„=j,S„<x\{J„S,),k = QX...,n-\) = Qj^^j{x-S„). (1.4) (ii) The process [(J^),n>0)is a homogeneous Markov chain with I as state space; more precisely, we have a.s. for allj of I: P{J„ =j,\j„k = 0,l,...,n-l)^pj^^j. (1.5) (iii) For all strictly positive n andfor all real x, we have: P(X„ <x,\Ji^,k = 0,l,...,«-l) = Hj^^(x),n>0,xeR, P(X„<x,\j„k^O,l,...,n) P(X„, <x„...,X„^

= Fj ^j (x),«>0,xeR, * <x„\j„k^QX...,n,yY{^j^,.J„M^

(1.6)

(0<^, <---<^^,x. G1R,/ = 1,...,A:). Of course, probabilities jf?/,, i,j=l,...,m, and functions///,y==l,...,m, Fy, i,j=l,...,m are defined exactly as in Chapter 4 from relations (2.3) and (3.11). The last relation in (1.6) shows the conditional independence of X^ ,...,X^ given From the relation (3.11) of Chapter 4 saying that: •,Pij>0,

F,j(x)(^P(X„<x\J„_,=i,J„^j)):

HXx){-P{X„<x\j„_,

P>J [L/,(x),/7,=0,

(1.7)

=i)) = Y^p^.F,.{x)

it is clear that a ( J ^ process is completely defined by either the doublet (p,Q) or by the triple (p,P,F) where, as in Chapter 4, P = (A>-/^J,Q = fe]>P = h ] > F - [ ^ ] -

(1-8)

Functionals of (J-X) Processes

195

2 FUNCTIONALS OF (J-X) PROCESSES This paragraph introduces the concept of functional ff of a given {J^K) process, fundamental for a lot of applications not only in finance and insurance but also in operations research. To define the functional W, we introduce a real and Lebesgue measurable function/of three variables defined on the set / x / x R . When they exist, we will use the following notation for the expectations with i,kel: 4 = jf(i,k,x)dQ,,(x),

^f> =

jf\iXx)dQ,,(x), (2.1)

k=\

k=\

Definition 2.1 Given a (J,X) process ((J^,X^),«>0) defined by (p,Q), and a Lebesgue measurable real function f on IxIxM., the functional W/ is defined as the stochastic process W^={W^{n),n = QX...) (2.2) where 0,« = 0, WAn):

Y,fiJ,-,,J,,X,\n>Q.

(2.3)

Janssen (1969b) extended for functionals of{Jy>C) processes key results obtained by Pyke and. Schaufele (1964) for functionals of positive {J^ processes: the strong law of large numbers (in short SLLN) and the Central Limit Theorem (in short CLT). The basic idea of the proofs is to decompose the sum W/ji) by introducing return times of return indices for the embedded Markov chain supposed to be ergodic (j^,/7>0) defined by relations (2.16) and (2.17) in Chapter 3, that is for ally belonging to /:

We know that the assumption of ergodicity implies that all states are positive recurrent so that, for ally, (2.5) ^-^oo=>rJ-^"^ ^ 0 0 . Moreover, if for ally we introduce the stochastic process in discrete time:

196

Chapters

{ul''\n>0)

(2.6)

with

^i''= Z

/(^„-P^„>^J>

(2.7)

then it is clear that this process (2.6) is a sequence of independent and identically r.v. with values in R , that is a random walk on the real line. The following propositions give some results concerning the moments of r.v. u(jeI,J^=j, Proposition 2.1 If the embedded Markov chain of the considered (J,X) process is ergodic and if the consideredfunctional is such that the expectations <^p<^/^^ exist for all i belonging to /, then Eiu\^^)->^\i^\^^Y) ^^^^^ ^^^^^^ given by: 1

m

(2.8) 1

m

9

m

mean return times in the embedded MC mi^,l ^sj,s (2.15) of Chapter 3. Proof see Janssen (1969b).

el being given by relations

n

Coming back to the random functional W/ defined by relation (2.3), we now introduce the following decomposition: W^ (N) = R, (N) + V{N) + i?2 {N) (2.9) where

R,{N) =

Z

))

A7=l

z z^ A'

R,{N) =

(2.10)

1

'..

where /„=/(•/

u=uiJ\

n-l^'^n-

•x„),

(2.11)

Functionals of (J-X) Processes

197

To prove the strong law of large numbers for functionals of {JJC) processes, we need the following lemma. Lemma 2,1 If the expectations ^,. exist for all i belonging to I, we have in the ergodic case that:

V{N) N

m -^—^{^n.mf),a,s,, nijj

^^'^^^

where rUj = E{u\^^ \JQ = j) is given by relation (2.8). Proof From relation (2.10), we have that:

|^,WNZ|/„|,

(2.13)

/7=1

and so, the ergodicity of the embedded MC implies that r\ is a.s. finite and of course independent of the value of A^so that the first relation of (2.12) is proved. To prove the second relation of (2.12), let us recall that from the classical strong law of large numbers (see for example Doob (1953), we know that for n-^oo : 1 " — Vw^ -^m.,a.s. (2.14) Moreover, from renewal theory, we also know that for N.(N) 1 -^->—,a,s., N

n-^co: (2.15)

ffijj

where Nj(N) represents the number of passages in statey from 0 to N. As all the states are positive recurrent we have that l/mjj is strictly positive and finite so that, from relation (2.15), Nj(N) diverges a.s. to ooand consequently from (2.13), as every subsequence of a convergent sequence converges to the same number, we have that: 1

Nj(N)-\

y

u^ -^m.M,s.

Then we can write V{N) in the form: W/ZV)-! Nj{N)-\ F(AO 1 7 Z «.' N N ]N.{N)

(2.16)

(2-17)

and finally results (2.15) and (2.16) really imply that M

N->co yyi 'jj

this result completing the proof.

(2.18)

Chapter 5

198

Proposition 2.2 (Strong law of large numbers for functionals of {J^X) processes) If the expectations ^. exist for all i belonging to I, we have in the ergodic case that: WAN) m.. (2.19) -^ -,a,s. N

^-^°° m

Proof From decomposition (2.9) and Lemma 2.1, it suffices to prove that a.s.

nmMO

0.

(2.20)

This result comes from the fact that a.s.: Nj(N) Nj(N)\ R,(N) < N N Nj(N) and as moreover a.s:

(2.21)

>0,a.s. D (2.22)

Nj(N) N

m,

Remark 2.1 This latter proposition shows that the quantity mjlmjj{= Aj-) is independent of the state 7 {solidarity result), Let us now consider the particular case where /O*, y, x) = X, V/, 7 G /, Vx e R. Then, from relation (2.3), we have: 0,^ = 0, j 0,^ = 0, WAn)-

X^,,«>o"K,^>i,

(2.23)

(2.24)

and from relation (2.7): (2.25) From relations (2.1), we get: ^jk=PjkbjkJ^keI, and finally, from relation (2.8):

(2.26)

Functionals of (J-X) Processes 1

199

m

Eiu^^')^—Xmi-

Mjj)J 6 /•

(2.27)

Now applying Proposition 2.2 to the considered special case (2.23), we obtain the next fundamental result in particular for the asymptotical behaviour of semiMarkov random walks studied in the following section. Proposition 2.3 (Strong law of large numbers for (J,X) processes) For any ergodic {J,X) process so that the conditional means b.j, i,j e I are finite, we have the following result: O

m

-^^;^^llm,ci.s. n

(2.28)

i=]

The next result is related to a central limit theorem for functionals of (J^ processes. Starting still from the decomposition (2.9), Janssen (1969b) proved the following two propositions. Proposition 2.4 (Central limit theorem for functionals of (J^X) processes) If the expectations ^. exist for all i belonging to I, we have in the ergodic case andfor the convergence in law that: m. Wf(n)-n--^ r r W m.. m. (2.29) 0,vart/i \[n m V V JJ J Moreover, if jj... defined by relation (221) is non-null, then Wj-(n)-nAj.ju yjn where

->iV(0,//5)

(2.30)

Chapter 5

200

m

4=-^

T^^iVi 1

jPkr =—

(2.31) f

'

.hj.kel.

Remark 2.2 From Remark 2.1, we know that A/is independent of state 7, as of course jUjj is also independent ofy; the same is true for Bf. Proposition 2.5 {Central limit theorem for the two-dimensional process (j„,W^{n),n>0)) If the expectations ^. exist for all i belonging to I, we have in the ergodic case andfor the convergence in law that: f

Mir / , . \

,.A..

\

= ;r,0(x), (2.32) ^n where the function O is the distribution function of the normal law N(0,juB ) . Remark 2.3 An immediate consequence of this last proposition is that processes (J^,n> 0) and {Wf{n),n > 0) are asymptotically independent. These last two propositions immediately give the following one for the special case of {JyK) processes Proposition 2.6 {Central limit theorems for the two-dimensional {J^ processes) For any ergodic {J,X) process such that the conditional variances a^. related to the conditional df F.jJJ e / are finite, we have the following results: S^-n^

-Jn

T"

->iV(0,//5J,

' ^n -

••n^^{x\

(2.33) (2.34)

Functionals of (J-X) Processes

201

where the function O is the distribution function related to the normal law NiO,JjuBg jwith here, for all i and k belonging to I: g{i,k,x) = x-J^7r.ri., (2.35)

^ik=PikbikA=Vi^ Yu^iVi J

Pik^

V/=l

G\ {i,k G /) being the conditional variance related to the conditional distribution

3 FUNCTIONALS OF POSITIVE (J-X) PROCESSES It is clear that all the results of the preceding paragraph are valid for the special case of positive {J^ processes for which the r.v. X^ are a.s. non-negative. But moreover instead of considering the sum of the first n transitions to define Wj{n) in relation (2.3) we can reinsert the time with a sum up to A^(/), that is the total number of transitions in the semi-Markov process related to the considered semi-Markov kernel Q. In fact, this was the case originally considered by Pyke and Schaufele (1964) so that now, relation (2.3) takes the form: 0,A^(0 = 0, N{t) WAt) = (3.1)

X/(^._P^«,^J,A^(0>0.

Using the same decomposition as the one defined by relation (2.9), these authors proved the next three propositions corresponding to the strong law of large numbers and central limit theorem of the preceding sections. Proposition 3.1 {Strong law of large numbers for functionals of positive {Jy)C) processes) If the expectations ^. exist for all i belonging to I, we have in the ergodic case that: Wf{t) m.. M.S., (3.2) t mjj with:

202

Chapter 5 m

'='

(3.3)

m

the limit ratio rrij I rrijj being still independent of j . The propositions related to the central limit theorem have similar extensions for n replaced by N{t) to the numerator and by / to the denominator.

4 CLASSICAL RANDOM WALKS AND RISK THEORY 4.1 Purpose. In Chapter 4, we focused our attention on semi-Markov chains defined by a positive {J^ process. The case of an extended semi-Markov chain is considered in this chapter starting from a general ( J ^ process and having a very different interpretation, directly related to the classical notion of random walk, In next subsections, we will recall some basic notions concerning random walks that will be extended to the main results of what will be called Markov random walks in the next section. After that, we will develop the main classical models in risk theory, which is very useful for insurance companies.

4.2 Basic Notions On Random Walks Let [X^,n> l) be i.i.d. random variables, with F as common d.f, such that: i^(0) 0. (4.2) These two relations imply that for all n, the events {o):X„>Q],{co:X„
(4.4)

S„-t,X,.

(4.5)

We can now give the following basic definition:

Functionals of (J-X) Processes

203

Definition 4.1 The random sequence (^S^,n> O) is called a random walk starting at xo, whose {X^,n > l) are the successive steps. If xo=0, the random walk is said to start at the origin. Example 4.1 If the distribution of r.v. Xn is concentrated on a two-point set

{-1,1} with p = P(X„ = 1), q(= l-p) = P(X^ ^ 1), (4.6) then the associated random walk is called the simple random walk or the Bernoulli random walk. The interpretation is quite simple: let us consider for instance a physical particle moving on a straight line starting at the origin. This particle takes a first unit step to the right with probability p or to the left with probabihty q and so on. Clearly, the r.v. S^ will give the position of the particle on the line after the nth step. Though very particular, the notion of a simple random walk has a lot of important applications in insurance, finance and operations research. A very classical application is the so-cMQd gambler's ruin problem. Let us consider a game with two players such that at each trial, each gambler wins 1 monetary unit with probability p and loses -1 monetary unit with probability q(= l-p), If u is the initial "fortune" of one player, he will be ruined at trial n iff, for the first time, his fortune just after this trial becomes strictly negative. He will be ruined before or at trial n iff he is ruined at one time k, k < n. The probability of this last event will be noted by ^(u,n) and the probability to be ruined precisely at time n will be noted by u(u,n) . Clearly, we have: '¥(u,n) = J^u(u,k)

(4.7)

and u(u,n) = '¥(u,n)-'¥(u,n-l).

(4.8)

The probability of not being ruined on [O,^], that is to say after any trial on [0,n], will be represented by y{u,n), and of course, we have: (4.9) y{u,n) = \-^{u,n). Probabilities y{u,n) and ^(u,n) WCQ cMQd VQspQctiyQly thQ non-ruin probability and the ruin probability on [O,^] starting at time 0 with an initial fortune - also called reserve or equities for insurance companies - of amount u.

204

Chapter 5

Now we will see how to express these two probabihties with the aid of events as functions of the variables X„, ^7=0,1,.. representing the "gain" (positive or negative) of the considered player just after the nth trial. Starting with xo=u, we can write: (4.10) u(u,n) = P{S,>0,k = l,..,,n-l,S„<0), If we introduce now the discrete r.v. T defined as follows: (4.11) T = mf{n:S^<0}, we get: u(u,n) = P{T = n),

(4.12)

'¥(u,n) = P{T
(4.13)

r(u,n) = P{T>n). (4.14) One of the major problems in the so-called risk theory is that of giving explicit results about these three probabilities. This part of risk theory is called the ruin problem and it is equivalent to studying the distribution of the stopping time T, Example 4.2 The cash balance model (Heyman and Sobel (1982), vol.1) also constitutes an application of the simple random walk in the field of finance. Let us consider a firm for which the r.v. S^ will represent the cash balance at the end of the nth day, starting with XQ as the initial balance. This firm has, on day n, the random amount A^ of receipts and the random amount 5„ of disbursements. If X.-A-B,. (4.15) then formula (4.5) does indeed represent the cash balance at the end of the nXh day. In this model, Sn should be negative and if so, it is interpreted as a loan. Let us now suppose that the firm does not use any line of credit, but covers negative balances with a transfer from another account. If M„ represents the cash balance for the nth day in this second model, we have: M, =max{Mrt,Mo + X ) , M,=max{M^.„M,_,+Xj. These relations will be developed later, when devoted to duality theory between queuing and risk models in discrete time. The main concepts in random walk theory are those of ladder variables, Graphically, ladder epochs and ladder heights are clearly seen on trajectories given in a two-dimensional graph where points are designated by the coordinates

{(^,5J,^>0}.

Functionals of (J-X) Processes

205

In the graph of Figure 4.1, we have joined the points of coordinates (k, Sj^), (k +1, iS^^i) in order to clearly show the evolution of the process. For example, in the trajectory of Figure 4.1, we have the strictly ascending ladder points {l,S^),{n-3,S^_^),{n,S^)eind the strictly descending ladder points

Figure 4.1: ladder trajectory The next definitions formalise these concepts of ladder variables. We follow the presentation of Feller (1971) but without the assumption that xo=0. Definition 4.2 The first (strict) ascending ladder point {r^,H^) is the first term of the sequence (^[n,S^),n > O) for which Sn is strictly superior to So- That is: {co:T,=n]={co:S,<S,,„„S^_,<S,.S^>S,],

(4.17)

The r.v. Fj is called thefirst strict ladder epoch and the r.v. defined by //,=5^_,^,=r,

(4.18)

is called the first strict ladder height. The possibly defective bi-dimensional distribution of (T^,H^) will be noted by H^{x) ^P{T,= nM, < x),n > 0,XG D ^ U{+--}. Consequently, we obtain:

(4.19)

206

Chapter 5

P ( r , =«) = //„(+«),

(4.20)

P{H,<x)^f^H„(x)(^M(x)),

(4.21)

SO that both r.v. F, ,//j have the same defect; that is: (4.22) p ( r , = oo) = P{H, = oo) = 1 - M(oo). Together with relations (2.5) and (2.6), Definition 4.2 shows that the event (2.23) is independent of the value of XQ . Consequently, without loss of generality, we may suppose for results concerning r.v. Fj and //, that x^ = 0 . Of course, the value of the ladder height 77, is nevertheless dependent on XQ . The definition of the second strict ascending ladder point is very simple. Indeed, it suffices to consider the random walk: ("^r,.„>«^l) (4.23) for which the first ascending strict ladder point will be defined as the second strict ascending ladder point of the random walk [S^,n>0). We continue in this fashion to define the 3rd, 4th,..., nth,... strict ascending ladder point, denoted in general by ( F ^ , / / , ) . Now, let us consider the event {co:r2 =n] meaning that we must have: {^:r, =«} = {^:5,,„ <5,_,...,5r,_, <5p,,5p,,„ > 5 , j

(4.24)

Using relation (4.5), we obtain: X,,,, <0,...,X,,„ + - + X , , _ , <0,Z,,,, + - + X,,,„ >0

(4.25)

and from the independence property of the r.v. X„, we see that the probability of the event defined by relation (2.33) has the same value as that of the event (2.25), that is: (4.26) p{r, =n) = P{r, =n),n>0. In the same manner, we define the successive strict ascending ladder heights Let us define F^ = 0 and let us suppose now that F^ is finite; if so, we can introduce the following r.v. defined by: 7„=5,^-5,„_, (4.27) SO that

S^.=t(\-S,J>

(4.28)

S,„-t^,-

(4.29)

Then, by definition,

Functionals of (J-X) Processes

207

H„=5'^„ (4.30) is the nth strict ascending ladder height. As the r.v. {Y^,n> O) are i.i.d, the sequence [H^,n> O) is still a random walk but eventually stopped at a certain step as it may be that p(7^=oo)>0. (4.31) The two-dimensional r.v. on N x R^ {(r„,H„),n>0) (4.32) are mutually independent and identically distributed with, as common d.f., P{r„
= f^HXx)

(4.33)

v=l

where the function H^ is given by relation (4.19). In reversing the inequalities in (4.24), we get the definition of the first strict descending ladder point, denoted by f Pf ,//i^ j This means that: {co:r^=n} = {co:S,>S„...,S„_,>S,,S^<S,}

(4.34)

and that: H^=S^^.

(4.35)

As in the ascending case, we define by induction the construction of the following sequence of possibly defective r.v.: (r^,H^,n>0). (4.36) For the sake of completeness, let us mention too that we can also define weak ascending and descending ladder variables by replacing the strict inequality by a non-strict inequality in relations (4.25) and (4.34). These two sequences will be represented by:

(r;;,//;),(rf-,//f-).

(4.37)

4.3 Renewal Results On Ladder Variables Let us consider a process of strict ladder variables ((r^,H^),n > 0). The marginal process ((HJ,n > 0) is a possibly defective renewal process whose d.f. H is given by relation (4.21). If g is the renewal function associated with H, we have by (3.13) of Chapter 2 that: g(x) = f^M'"\x), By relation (3.16) of Chapter 2, we also have:

(4.38)

208

Chapter 5

1 - M(oo) a relation showing that the mean number of strict ladder heights on [0,oo) is finite iff//(oo)S,,k = 0,l„.„n-l and that it is a weak ascending ladder point iff: (4.41) S„>S„k = OX...,n-l. This distinction between strict and weak ladder points has no interest if the d.f H is continuous - but such is not the case for example for the Bernoulli random walk! Let M~ be analogue to M, defined by (4.19), in case of the weak ladder process. We have the following result: Proposition 4.1 If C-f,P(S,

<0,...,S„_, <0,S„ =0),

then, for all positive x: M-(x) = CU.ix) + (\-OM(x).

(4.42) (4.43)

Proof Without loss of generality, we can suppose that 5*0=0. We observe that the event {CO:H;=0} (4.44) happens iff the random walk, starting from 0, has all strict negative values before having a first return to 0. Otherwise, that is if the random walk has a positive value, then the first weak ascending ladder point and the first ascending ladder point are identical. Thus: P(H;=0), (4.45) Since the events {co:H~=0} and {co:X^>0} are disjoint, we have from relations (4.1) and (4.2) that 0<^<1. (4.46) So, with probability 1 - <^, the first strict and weak ladder points coincide and consequently relation (4.43) is proved. a Proposition 4.2 If g~ is the renewal function related to the df M~, then for all non-negative x: <;-i.x)^^g{x).

(4.47)

Functionals of (J-X) Processes

209

Proof Using notation (5.57) of Chapter 2 or the Laplace Stieltjes transform, relation (3.13) of this same chapter gives us: l-M (s) So, by relation (4.43), we get:

~(\-0[\-M(s)] Using relation (3.13) of Chapter 2 once more, we obtain result (4.47) using the n inverse Laplace transform. We will now introduce the fundamental concept of duality based on the "trick" of "time reversing". Given here the special case of random walk, the definition will be extended to the extended SMC later. For the random walk generated by the sequence (JL'^,A:> O^where XQ is still equal to 0, let usfixa positive natural number n and consider the reversed process of the section of order n (X\, ..,,Xn) defined as the random /7-uple

(<,...,X).

(4.50)

called the reversed process of order n and_defined as:

x;^X„X=X„^„...X=^^,

'

(4.51))

so that, in general: Xl = X„.(,.,,X, = X„V,,A: =!,...,«.

(4.52)

It is then clear that: iXlr = X„k = l,..,n, (4.53) a relation showing that the notion of reversed process is involutive or self-dual. The following property is an immediate consequence of relations (4.52): Proposition 4.3 For every positive integer i inferior or equal to n, we have: S.=S* -S* .,S* =S -S .. (4.54) It is always possible to express an event related to the initial process {X\,..,,Xf^ as an event related to the reversed process (Jf *,..., X*) and vice versa. For example, the event: {Q)\S^>SjJ {(D\Sl>Sl-Sl_jJ

= 0,..,,n-\]

= 0,...,n-\]

can be expressed as

or {co\Sl_j>QJ = 0,,„,n-\]

and finally by a

change of indices {co\S* >0,/ = l,...,/?}. We can now present the so-called duality lemma, first given by Feller (1971) and then by Heyman and.Sobel (1982).

210

Chapter 5

Proposition 4.4 {Duality lemma) If for the random walk generated by the sequence {Xj^,k > O) with XQ^O, we set: n(t) = card{5'^ >0,k = l,.,.,n,0<S^ then: E(n(t)) = g(t),


(4.55) (4.56)

g being the renewal function defined by relation (4.38). Proof By definition of the expectation, we have: 00

E{n(t)) = j;^{S,>0,k^l,...,n,0<S„0,k

(4.57) and (Sl,S*,,..,Sl) are

= l„..,n,0<S„
(4.58)

Using relation (4.54), we can write: 00

E{n(t)) = Y.P{S^>S„k

= l„,„n,0<S^
= ko by definition of the renewal function as a sum of indicators.

(4.59)

(4.60) n

The duality lemma gives a supplementary interpretation of the renewal function g related to the ladder height process. This function g is not only equal to the expected number of ladder points with heights belonging to (0,^], but it is also equal to the expected number of visits made by the random walk to the interval (0,/] without leaving the positive axis.

4.4 Classification Of Random Walks This paragraph is devoted to a very important result known as the classification of random walks. Briefly, this result states that only two possibilities exist for the asymptotical behaviour of the random walk(iS'^,^ > O). Either: (4.61) P(limsup^„ =oo) = P(liminf ^„ =:-oo) = l or: (4.62) P{limS^ = 00 or lim5'„ = -oo) = 1. In the first case, the random walk is called oscillating; in the second, it is said to drift to +00 or -oo . In this last possibiHty, we have a.s.

Functionals of (J-X) Processes lim5'„=+oo or lim5„=-oo .

211 (4.63) (4.64)

Proposition 4.5 There exist only two types of random walks: (1) the oscillating type: both ascending and descending renewal processes of ladder heights are persistent. In this case, the process (^S^, /7 = 0,1,...) oscillates with probability 1 between-(:o and +oo, and:

£(r,) = ^{rf) = «);

(4.65)

(2) drift toward ±oo; in the case of -oo, the ascending renewal process is terminated and the descending renewal process is proper with probability 1. The process [S^, ^ = 0,1,...) drifts with probability 1 toward -oo and reaches a finite non-negative maximum. M; moreover: E(V^ ) = - i - f («.) = - l - - ± — 1 - <^ 1 - M(oo) ^ ^ 1-^ And if Mis the r.v. defined as M = max{S,,S„„.,S^,.,.), then P(M <x) = (l- M(oo))g(x). The results are analogous in the case of a drift toward +oo.

(4.66)

(4.67) (4.68)

Proof Using the reversed process and relation (4.55), we have for all natural numbers n: (4.69) P(S„>S„k = 0,l,...,n) = PiS:>S:-Sl„k^O,l,...,n) or (4.70) P(S^>S,,k = OX..,,n) = P(S,>0,k = OX...,n). The first member of equality (4.69) represents the probability that (n,S^) is a weak ascending ladder point; also: f^P(S„>S„k

= OX...,n) = g-(<x,).

(4.71)

77=0

Moreover, from relation (4.47), we get: f^P(S„>S„k^O,\,...,n)

= -l-g(^)

and on the other hand P(S^>S„k = OX„.,n) = P(r^>ny Equalities (4.70) and (4.72) lead to the following relation: f^P(r^>n)^-^gi^).

(4.72)

(4.73) (4.74)

212

Chapters

By definition of the expectation of a non-negative discrete r.v., this last inequality becomes: -l-g(«,)^E(rf)

(4.75)

and finally, using relation (4.39), we get:

^(rf) = - ^ - — ^ . 1-^

(4.76)

I-M(OO)

From the preceding relations, we can now prove the above mentioned classification and their consequences. Indeed, let us suppose first that: P ( r f
p(rf £(rf) = oo<=>(/^(oo) = oo<=>p(r,
(4.8i)

We can of course exchange ascending and descending processes, which is in fact equivalent to replacing the initial steps (X^, n>0) by (-X^, n>0). This leads, by a similar development, to a result which is analogous to (4.81):

p(ri£(rj) = oo<=>{^^(oo) = oo<=>p(rf
(4.82)

Relations (4.81) and (4.82) show that it is impossible to have simultaneously: (4.83) P(r, < 00) = P ( r f < 00) = 1. Thus, we have three possibilities: (i) i^(r,
(4.85)

(iii) P(r,
(4.86) (4.87) (4.88)

for the subsequence (n^,k = l,2,...) defined by: n,=f^R„k /=1

= h....

(4.89)

Functionals of (J-X) Processes and lim^„^ --00

213

(4.90)

for the subsequence (^^,A: = 1,2,...) defined by:

n,=f^}if,k = l,....

(4.91)

Results (4.88) and (4.90) prove the oscillating case. In case (ii), we have result (4.66) from relations (4.75) and (4.76). To obtain the result (4.68), we know that the condition: P(rf
\mi^

(4.94)

These last two results prove that, with probability 1: lim5„=-oo. (4.95) In fact, when the mean /u of the r.v. X„, n>\ exists, the strong law of large number asserts that, a.s.: lim^-//, /7->00

(4.96)

^

SO that we inmiediately get the following results: lu>0^\\mS^ =+00, lu<0^=>\\mS^ =-00.

(4.97)

W->oo

The next theorem gives the complete relation, including the case // = 0 which is more difficult to treat. Proposition 4.6 If the mean ju of the r.v. Xn, n>l exists, then: (i) // =0 implies that the random walk is oscillating,

214

Chapters

(ii) // >0 implies that the random walk drifts toward + oo, (iii) ju <0 implies that the random walk drifts toward - oo . Proof As said above, results (ii) and (iii) are immediate consequences of the strong law of large numbers. For case (i), we know from Proposition 4.5 that at least one of the ascending or the descending renewal processes of ladder heights and ladder epochs is persistent. Let us suppose that it is the case with the ascending one. So, if k is supposed to be the value of theyth ascending ladder epoch, we have: • ^ = -^

.

(4.98)

From (4.95), A: -> oo implies that the quantity (4.98) tends to 0. It follows then that the denominator in (4.98) tends to +oo and thus F, is proper with: (4.99) i?(r,) = +oo. But by Proposition 4.5, this only happens in the oscillating case. n Remark 4.2 In examining Propositions 4.5 and 4.6, we note that the sufficient conditions in 4.5 are also necessary conditions. In short, this gives us: (i)

//-O <=>P limsup5'„=+oo,liminf5'^=-oo = 1 ,

(4.100)

(ii) //>0 « p f l i m ^ =+ool = l,

(4.101)

(iii) //>0 <»pflim5 - - o o ) - l .

(4.102)

5 DEFECTIVE POSITIVE (J-X) PROCESSES To extend the main results of classical random walks on semi-Markov random walks, it is necessary to introduce now the concepts of defective and terminated positive (y,X) processes. In fact we will now consider now a new type of two-dimensional process always defined by a kernel Q but for which the matrix P of the embedded Markov chain (J^,n> 0) could be substochastic, i.e. such that at least the sum of the elements of at least one line is strictly inferior to 1. To do this, we formally introduce a supplementary state to the set / called state 0 which is absorbing so that instead of/={l,...,m} as state space, we work now with the new state space

Functionals of (J-X) Processes

215

7 = /U{0} (5.1) and with a new matrix (m + l)x(m +1) represented by Pdefined from the given substochastic matrix P as follows: P.,=S„„keT,

(5.2)

m

Pjo=^-^pjk(=^-^jy

For the sojourn times, when the embedded MC reaches the absorbing state 0, we define the sojourn time as being infinite and so the associated T process terminates. This leads to the following definition. Definition 5.1 ^ defective positive {J,X) process ((J^,X^),n>0)

with state

space / X R"^ is defined by a vector of initial distributions p = {pQ,P],"',Pm) ^^^ a kernel (m +1) x (m +1) Q = [Qij of mass functions on the positive half real line such that the mxm sub-qmatrix (5.3)

2 , - 2 is a positive semi-Markov kernel and

. , . Jo,

xeR,

[Pm^ x = +oo, fO,

X 6 R,

eoo(^)= / [1,

Qoiix)^0, Moreover, if p,j=limaj(x), •^

.x:->oo

(5.4) X- +00,

ieI,xeR. (5.5)

-^

the matrix

P = [/;,]

(5.6)

is a sub-stochastic matrix satisfying conditions (5.2). The defective positive (J,X) process satisfies the conditions P(Xo=0)=l,a.s.,

(5.7)

216

Chapters

PiJo = 0 = P>i^\,...,m

with X A = 1 >

and for all n>OJ=\, .,,,m, we have: = 0,,„,n-\)^Q,^Jx\a,s, P{J^^j,X^<x\{J,,X,lk

(5-8) (5.9)

Let us recall that when the process enters for the first time state 0, it will stay forever and so we say that the process terminates or is terminated at the first entrance time in state 0; so, we can define the unconditional waiting time distributions as follows: e /, H,{x) = PiX„ < x\j„_, = i) = ZQIJMJ ye/

, H,(x) = P(X„<xJ„_,^0)

fO, xeM, =\ '

[I, X = +00,

and consequently: = P(J„=0\j„_,=i) = l-u,JeI, P(X^=^^\j„_,=i) and of course: \imH.(x) = u.,i€l. Now, let T.j(x,n)(i,jel)

(5.10)

(5.11) (5.12)

be the probability that the (J,X) process terminates at

step n with Jn^j\ before or at time x given that Jo'^iIt is clear that: T,j(x;n) = P(T„ <x,J„ =j,J„,, =0\J,=i), =(^-Oj)(Ql;\x)). The value of the probability 7]^. («)(/, y G / ) that the (JJ^ process terminates at step n with Jn=j, given that jQ=i, is (5.14) T^(n) = \imT^(x;n) = pl;^(l-Uj), the value of the probability T. {n)(i G 7) that the (J^X) process terminates at step n given that Jo==^*j is

T,(n) = Y,T,.(n) = Y^pl;\l-u.), Jel

(5.15)

jel

and finally the probability 7),(/ e I), that the (J^X) process terminates given that Jo=i, is given by

Replacing the Uj,j e 7 by (5.2), we find the result that:

TM)'?',:"'.

Functionals of (J-X) Processes

217

Using a result of Chung (1960, p.227), we have that the (7;,/e/) satisfy the following linear system: T,=ZP>^T,+p,„ieT, (5.18) keT

where 7 is the set of transient states in the imbedded Markov chain. Definition 5.2 (i) A defective positive {J,X) process is i-terminated iff, starting from state i, it terminates a.s., (ii) A defective positive {J,X) process defined by the triple (m,p,Q) is pterminatediff starting from initial distribution p, it terminates a.s., (iii) A class of defective positive (J,X) process defined by the doublet (m,Q) is terminated iff starting with any initial distribution p, // is p-terminated. The next proposition is now obvious. Proposition 5.1 (i) A defective positive (J,X) process is i-terminated iffTi=\, (ii) The defective positive {J,X) process (m, p, Q)is p-terminated

iff^p.T.=l, iel

(iii) The class defective positive {J,X) process is terminated iff Ti=\, for all i belonging to I. Remark 5.1 If state 0 is the only recurrent state, then Ti=l for all / of/and so we have a terminal class. In particular, this is the case when for all / of/, ^ < 1. For defective positive (J,X) processes, we can introduce the concept of lifetime of the process defined by the r.v. Mas follows Definition 5.3 The lifetime of a defective positive (J,X) process is the r.v. M defined as: M= sup r„. (5.19)

It is clear that the conditional distributions of M, having M as support, are given by: M, W = P(M < x\J, = i) = j;^f^T,j(x;n). Using relation (5.13), we get:

(5.20)

218

Chapters

(5.21) a result giving the theoretical explicit form of the lifetime conditional distributions. A simple probabilistic reasoning proves that the functions Mi, i=\,.,.,m satisfy the following integral equation system of renewal type: + X]Mj{x-y)dF,j(y),isI,

M,(x)^(l-o,)

(5.22)

ye/ 0

whose (5.21) is the unique solution. In the case of all the d.f Mf, being proper (i.e. limM.(x) = 1,/ = 1,...,w) and if, moreover, the following mean lifetimes exist: 00

M. = jxdM.(x)
(5.23)

0

then, from relation (5.22), it follows that they satisfy the system: m

M.='^p^jMj-^?jj,i

= l,.,.,m.

(5.24)

7=1

Example 5.1 Let us consider the class of defective positive (J,X) processes for which the matrix P = Ipyj satisfies: Pij=Pj>OJ,jeI, ^

,

(5.25)

y=i

In this case, state 0 is the only recurrent state for the Markov matrix P and so, the considered class of defective positive (J,X) processes is terminated with: u. = a, (5.26)

pl;'=l-a\

It follows then that all the d.f A^are proper and, from (5.21) given by m

M,.(x)=:(l-a)£(i^(x)),, 7=1

a result giving a Feller 's theorem Feller (1971). Here, the Hnear system (5.24) has the following explicit form:

(5.27)

Functionals of (J-X) Processes

219

(p,

-7i|

Pn,

iPi- -5n) (Pr -1) M, =

'—^

(P„ ---^

^ ^ ^ , / = !,...,;„.

(5.28)

If moreover, the considered class here is of order 0, that will say we have as supplementary condition on sojourn conditional distributions: F.^Fj,iJ^\,...,m, (5.29) then from (5.28) the M functions do not depend on / and the common value M of the mean lifetimes M,. is given by

M=

J—,

(5.30)

^-HPJ

where n^Y^PA'^j = ]tdFj(t)J^l,...,m. y=i

(5.31)

0

The special case m=l is already given by Feller (1966).

6. SEMI-MARKOV RANDOM WALKS Let us consider a (J,X) process ((J^,X^),n > 0). It can be seen that the process (X^) defines a random walk on the real line starting at Xo=0, a.s. but contrary to the classical random walks, the successive steps X„ are no longer independent but satisfy a semi-Markovian dependence, useful for a lot of applications for example in risk theory, queues and so on. The position of the particle at the nth step is given by the r.v. S„=ZX„n = l,2,...

(6.1)

From now on, we will suppose that the imbedded Markov chain (y^,^>0)is irreducible and so recurrent. Fory belonging to /fixed, let us define the process (ry\n> 0) called the process of recurrence indices as follows: ry^ = sup{yt>0:yt>rji,',J, ^ j,r^P, 0. k

^^'^^

220

Chapter 5

From Chapter 3, relation (2.15), we know that the mean recurrence times given by: mjj=E(r!;il-ry^),n>Oj'eI (6.3) and are here finite. With the renewal process (rl;^} -rj,^\n > 0), we can associate a classical random walk, i.e. a sequence of i.i.d. random variables: (Uj'\l>0) (6.4) with

(6.5)

Uj^'= I X„.

From Proposition 2.1 and the Remark 2.1, it follows that these random variables have a mean ju given by 1 '" l^jj=—Y.^iVi^ (6.6) so that they are positive or negative for all j , depending on the sign of ju defined by m

M = Yj^ir]i-

(6-7)

i+\

Following Spitzer (1957) and Feller (1971) we will say that a random walk drifts to +00 (resp. -co) iff P(limsup{ft;: S„{co) < 0}) = 0, (6.8) (P(lim inf {6;: S„ {co) > 0}) = 0) n

and that it is oscillating iff P(limsup{^: S^{co) <0)) = P(liminf {^: S^{Q)) > 0}) = 1. n

(6.9)

"

We then get the following theorem concerning the asymptotical behaviour of the semi-Markov random walk {Sn)> Theorem 6.1 If the semi-Markov random walk (Sn) has an irreducible M.C, and all the unconditional means 7]. J el are finite and, then if ju is null and if for one h P(Ul'^=0)
Functionals of (J-X) Processes

221

Proof (i) If // =0, let us suppose that J{)=j and introduce the following r.v.:

2„-Y.U['\n>Q.

(6.11)

As r.v. {Ul^\l > 0) are independent and identically distributed, the sequence (Z„, ^7>0) defines a classical random walk in which Z„ represents the position of the particle moving on the semi-Markov random walk at the ^?th return of the embedded M.C. to statey. Now, as ju=0, it results from Feller results ((1966), theorem 4, p.203 and theorem l,p.379)that P(limsup{^: Z^(CD) < 0}\j, = j) = P(liminf {^: S^(co) > 0}\j, =j) = l. (6.12) As limsup{Z^ < 0} (= limsupl^^ < 0}, n

n

limsup{Z^ > 0} c: limsupli"^ > 0}, n

(6.13)

n

and as moreover m

P(limsup{5'„ < 0} - j;^pjP(\imsup{S„ < 0}\J, = j), y=i

n

n

m

(6.14)

P(limsup{5„ > 0} = X;'y^(limsup{5„ > 0} \J, = j), y=i

n

n

the first part of the theorem is proved. (ii) If // >0, let us introduce the following events: E^^io):

S„ n

Mjj ^Mjj >,n>Q. nijj Inijj

It is clear that for every positive integer n: {oy:S„{w)
(6.15)

(6.16) (6.17)

But applying now the strong law of large numbers for functionals of {J^ processes (Proposition 2.2), we get: P(limsup£'„) = 0 (6.18) n

and consequently applying relation (2.20) of Chapter 1 to relation (6.17), we finally get the result. n

222

Chapter 5

7 INDICE VARIABLES FOR SEMI-MARKOV RANDOM WALKS Let us consider a semi-Markov random walk defined by the {J^ process of triple (m,p,Q). Following Feller (1966), Janssen (1970) extends the concept of indice variables as follows. By definition, the first strict ascending indice is given by: fmi„{„:^.>0}, On / (= / U {0}), we can now introduce the state occupied in the embedded M.C. at step F\ : [0,

7^=00.

On E , we can also define the position of the semi-Markov random walk noted //•: \Sr,

^,=

' [oo,

F,< 00,

'

(7.3)

i^^zco.

We can now introduce the three-dimensional distribution of the triple (i^,//,,f,)with: Hl{x) = P{F,=n,H,<x,g,=j\j,=ili,jeJ,x>0,n = QX2,... (7.4) For the marginal distributions, we have: H'„{x) = P{F,=n,H, <x,\j,^i) = Y,H'i{x),i,^J,x>0,n^Q,\X:; (7.5) J

H\x)

= P(H,

< x,\j,

= i) = YJK(^\

^'^ ^ ' ^ ^ 0,^ = 0,

(7.6)

= 0,l,2,...,

(7.7)

n>\

Hl=P(F,=n\j,=i)

= \imHlix),i,eT,n

H'^ix) = P(H, <x,J,^ = j\j, = 0 = X^5(^)' '>J^7,x>0,

(7.8)

n>l

and so P(H, = oo,y^_ ^y|y„ = / ) ^ j.„p(y^^ = oly^ = /) = Sj,P(F, = a,\j^ = i) = Sj,(l - limH'ix)).

^^'^^

From relations (7.8) and (7.9) we also deduce immediately that: P(j,-jJo-i)-\'^ „,, . . „ (7.10) ^ ^ \\-hmH (x), 7 = 0. We define the second strict ascending indice as tho first strict ascending indice

Functionals of (J-X) Processes

223

of the semi-Markov random walk induced by the following sequence: {^fl.,>^^,.2v..}, (7.11) or, equivalently, as the second indice n, if it exists, otherwise, we take oo, such that Sn>Sk,k=0,\,„„nA. (7.12) Of course, if the first indice F\ is infinite, the process of indices is considered as teminated. Otherwise, the second strict ascending indice is characterised by the couple {F^+F^Mx+H^) and for the occupied state, we write: ^I^JF,.F,' (7.13) Proceeding so for the following indices, we obtain the process of strict ladder indices

((//„,fj,^>0),

^^^^^

//o=0,fo=0, a.s. By construction, it is clear that this new process is a positive {J^ process eventually defective and characterised by the triple (m,p,H) where the matrix H is the matrix (7.15) H(x) = [//^(x)] with functions If^ defined by relation (7.8). The next proposition characterises i]\Q process of strict ladder indices, Proposition 7.1 If the semi-Markov random walk (Sn) has an irreducible M.C, and all the unconditional means r/.,ielare finite, then the positive {JyX) process: ((//„fj,«>0), ^^^^^ //o=0,^o=0.a.s. characterised by the triple (m,p,H) is defective or not following that iu<0 orju>0. Proof With the notation introduced in section 5, we have that: =/'(F„=+co|^„_,=/)

(7.17)

Now, from Proposition 7.1, we have that: F(F^ = +oo\j^ = 0 = 0 <» // > 0, P(i^=+oo|Jo=0>Oo//<0,

(7.18)

and so the proposition is proved.

a

224

Chapter 5

When the process (7.16) is defective, the problem is now to see when it is terminated or not. The reply is given with the next proposition. Proposition 7.2 If the semi-Markov random walk (Sr) has an irreducible M. C and all the unconditional means r/.,i G I are finite, then the positive (J,X) process:

((H„,gJ,n>Ol

^^^^^

//o=0,^o=0,a.s. characterised by the triple (w,p,H) is terminated for every initial distribution piffju<0. Proof From relation (7.17), we know that: 1 - u, -P(i^ = +00|y, = /) > 0 <=> // < 0. (7.20) So, from relation (5.16), this implies that,for every state i, the probability Tj that the process ((H^,gJ,n > 0) terminates, is strictly positive. From this last result, we can deduce that on / , for the imbedded MC of the positive (J,X) process ((H^,g^),n > 0), all the states of the set / are transient and so that the only absorbing class is the set {0}. But from Chapter 3, we know that in this case, the absorption probability for this class is 1 and so: T.=iyieL D (7.21)

8 DISTRIBUTION OF THE SUPREMUM FOR SEMIMARKOV RANDOM WALKS Let us consider a semi-Markov random walk (SJ with an irreducible M.C. and all the unconditional means 77.,/e/finite. We are now interested in the distribution of the following supremum: (8.1) M = sup{S,,S,,„.}, Forju>0, under the assumptions of Proposition 7.2, it follows from this proposition that for all / of/ and all real x: = 0, (8.2) P(M<x\j,=i) This is also true for ju =0, as the positive (J,X) process ((H^,gJ,n > 0) is regular (see Fyke (1961a)) meaning that it has only a finite number of transitions on any time interval. Nor for /u <0, from relation (5.21), we get:

225

Functionals of (J-X) Processes M,{x)^P{M<x\J,^i) where

=

Y,{^-Vj)M,j{x),

M = rM^.]is the matrix of renewal functions

((//„,?„),« > 0 ) . From Proposition 7.2, we know that: hmM,.(x) = l , V / e / .

(8.3) for the process

(8.4)

X->oo

We also see that M.(0) = l-t;.,V/G/. (8.5) Nevertheless, to be useful, the "explicit" form (8.3) requires us to know the kernel of the positive {J,X) process ((H^,g^),n > 0) or the functions H"^ given by relation (7.8). Unfortunately, this is very difficult except in very particular cases. To avoid that, we can start from the following integral equations system of Wiener Hopf type given from an immediate probabilistic reasoning: ,^r ^

\l,]Mj(x-s)dQ,j(x),

x>0,

(8.6)

[ 0, X < 0. For m-l, we get the classical Wiener Hopf equation: M(X):

JM(x-s)dQ(x),

x>0,

(8.7)

0, x<0. Janssen (1970) proved that this integral equations system of Wiener Hopf type has a unique P-solution, meaning a vector (Af,,...,M„) of distribution functions satisfying system (8.6).

Chapter 6 NON-HOMOGENEOUS MARKOV AND SEMIMARKOV PROCESSES In this chapter, we extend some definitions and results of Chapters 3 and 4 to the non-homogeneous case for which time itself has influence on the transition probabilities. Due to the importance of applications, in particular in insurance, we carefully develop some special cases such as non-homogeneous Markov processes.

1 GENERAL DEFINITIONS To begin with, we present the general definition of non-homogeneous semiMarkov processes (in short NHSMP) including as particular cases, nonhomogeneous Markov processes (in short NHMP) in continuous time, nonhomogeneous Markov chains (in short NHMC) in discrete time and nonhomogeneous renewal processes (in short NHRP). We follow the original presentation given by Janssen and De Dominicis (1984).

1.1 Completely Non-Homogeneous Semi-Markov Processes As usual, let us consider a system S having m possible states constituting the set /={1, .,.,m} defined on the probability space ( Q , 3 , P ) . Definition 1.1 The two-dimensional process in discrete time ((/^,X^),/7>0) with values in / x M"^ such that: JQ =/,XQ P(J„

=0,a.s.JeI, =j,X„<x\(J„X,),k
T,^0,T„^±X„a.s. is called a completely non homogeneous semi-Markov chain {in short CNHSMC) of kernel Q(sJ)= (^"-'^Q(s,t),n> l) Consequently, the past influences the evolution of the process by the presence of r^_j andn in (1.1).

228

Chapter 6

Definition 1.2 The sequence Q=[^"~^^Q{sJ),n>\\

of mxm

matrices of

measurable functions of NQ x R"^ x R"^ h-> [0,l] where:

<"-"Q(^,o=[<"-'^e,(^^,o]

(1.2)

and satisfying the following conditions: (i) V« > 0, V/,y e /, V/,5 G R* : ? < ^ ^ ^"-'^Qy(s,t) = 0, (ii) V n > 0 , V / e / , V 5 € R ^ J<"""ey(5,°o) = l, with'"-%(s,<^) •>

=

(1-3)

lim'"-'%(s,t), /-»oo

•^

/^ called a completely non-homogeneous semi-Markov (in short CNHSM) kernel, Clearly, for all fixed s, ^"~'^2^ (^,.) is a mass function, null for^ < 6*. Definition

1.3 For all

i,j eI,nGNQ,s,t

eR'^

the functions

^"^^Pyis),

^"-'^H.j(s,tl ^"-'^F.j(s,t) are defined as follows: '"~''Py(s)=^"-''Q,j(s,^l

^^-'^//,(.,0 = Z'^''e,(^,0,

(1.4)

U,(s)U,(t\'"-'^pJs) (n-\)

= 0,

Fy(s,t) =

V,(^) Working as in Chapter 4, section 4, it is easy to prove that we still have the following probabilistic meaning: ^"-'%(s,t) = P(X„
=/,r„., =s)(=P(T„
^"-''F,.{s,t) = P{X„
^j,T„_,

^^

^s)

(=P(T„
Non-Homogeneous Semi-Markov Processes

229

A-B = U.bA we will write:

<"-'>F(5,o = ['"-"i^.(^,0], <"-"P(5) = [<''>,(5)],

(1.7)

We can now give the following definitions similar to the classical or homogenous semi-Markov theory presented in Chapter 4. Definition 1.4 The counting process (N(t),t >0) defined as N(t):=sup{n:T„
(1.8)

n

is called the associated counting process with the CNHSM kernel Q. Definition 1.5 The process [(J^,TJ,n> O) is called a completely nonhomogeneous Markov additive process or Markov renewal process (in short CNHMAP or CNHMRP). Definition 1.6 The process Z = (Z(t),t > 0) defined as

[ 0,N(t)<'^, where 9 is a new state added to /, is called the completely non-homogeneous semi-Markov process {in short CNHSMP) of kernel Q. Definition 1.7 The random variable L defined as L = mf{t:Z{t) = e] is called the lifetime of the CNHSMP Z.

(1.10)

Definition 1.8 The associated counting process (N(t),t > 0) or the CNHSMP Z= (Z(t),t > 0) of kernel Q is explosive iff L = co,a.s. (1-11) and non-explosive iff L
230

Chapter 6

For the two-dimensional process {{J„,T„),n> 0), we have the following result: P(J, =j,T,


P(J, =j,T,
^^'^^^

k

and in general

P{J„ =j,T„
'

k^

(1.14)

-(e^"'(0),«>i). Using matrix notation, we may write for two my.m matrices of mass functions

A(0, B(0: {A(OCJ«(0=

Xj[5,.(z)J4,(z)

(1.15)

L ^=1

and so relations (1.14) can be written under the matrix form: with (1.16) ^"'\t) =

\_Q^"\t)\n>\,

Q<"(o=[<'"e,(o,o]. In the particular class of classical SMP, the relation (1.16) gives the ^-fold convolution of the SM kernel Q. Another very important distribution is the marginal distribution of the Z process as it gives the state occupied by the system S at time t, Let us introduce the following probabilities: ^%(5,0 = i^(Z(0 = 7lZ(0) = /,7V(^-)0. (1.17) The conditioning means that T^=s and that there exists a transition at time s such that the new state occupied after the transition is /. Clearly, these probabilities satisfy the following relations:

<%(5,o=^.(i-<"^/f,(^,o)+lf*%(",o*""''a*(^>^").^;e/.

(1.18)

kei

From relation (1.1), it is clear that we have: P{J„= j\{J„T,),k
(1.19)

It follows that the process (y^,«>0) can be viewed as a conditional multiple Markov chain\ this means that, given the sequence(r^,«>0), each transition from y^_, -^ J^ obeys a non-homogeneous Markov chain of kernel in-Dp^j^,^ (see Definition 1.7).

Non-Homogeneous Semi-Markov Processes

231

Definition 1.9 The conditional multiple Markov chain (J^,/7>0) is called the imbedded multiple MC.

1.2 Special Cases Let us point out that Definition 1.2 is quite general as indeed it is nonhomogeneous both for the time s and for the number of transitions n, this last one giving the possibility to model epidemiological phenomena such as AIDS for example (see below the example of Polya processes and semi-Markov extensions). This extreme generality gives importance to the following particular cases. 1.2.1 Non-Homogeneous Markov Additive Process And Semi-Markov Process If in the sequel Q=(^""'^Q(^,0.^ > l ) , we have: (^-')Q(^,/) = Q(^,/),^ > 1,^ < t, (1.20) that is Q independent of/?, then the kernel Q is called a non-homogeneous semiMarkov kernel (in short NHSMK) defining a non-homogeneous Markov additive process (in short NHMAP) ((J„,7;),«>0) and a non-homogeneous semiMarkov process (in short NHSMP) Z = {Z{t\t > 0). This family was introduced in a different way by Hoem (1972). It is clear that the relation (1.20) means that the sequences (1.21) ^"-^^F(^,0 = ^is,t\ ^'-'^V{s) = F(s) V« > 0 are independent of ^ or equivalently that Q(s,t) = F(s)-F(s,t). (1.22) Let us point out that, in this case, relations (1.18) become: (/>,j{s,t) = S,j(l-H,isj))^^l^,j{uj)Q,,^^^^^

(1.23)

If moreover, we have P(^) = P,^>0, (1.24) then the kernel Q is called a partially non-homogeneous semi-Markov kernel (in short PNHSMK) defining a partially non-homogeneous Markov additive process (in short PNHMAP) [(J^,TJ,n>0) and a partially non-homogeneous semiMarkov process (in short PNHSMP) Z = (Z(t),t > 0). This family was introduced in a different way by Hoem (1972).

232

Chapter 6

1.2.2 Non-Homogeneous MC If the sequences ^''~'^P(^),V^>0 are independent of ^, then (y^,^>0)is a classical non-homogeneous MC (in short NHMC) 1.2.3 Homogeneous Markov Additive Process A PNHSMK Q such that F{s,t) = F(t-s),s,t >0,t-s>0, (1.25) is of course a classical homogeneous SM kernel as in Chapter 4. Let us say that the case of non-homogeneous Markov processes will be treated in the next section. 1.2.4 Non-Homogeneous Renewal Process For m=l, The CNHMAP of kernel Q is given by Q(s,t) = C"-'Ms,t)),s,t >0,t-s>0 and characterizes the sequence (X^,n> 0) with, as in (1.1), XQ =0,a.s., P(X^<x\x„k
(1.26)

(1.27)

To=0,T„=±X,,a,s, In this case, the process (X^,n>0) is called a completely non-homogeneous dependent renewal process (in short CNHDRP) of kernel Q. If moreover, ^"-^^F(5,0 = ^'-'^F{t -s\s,t>0,t-s>0,n>\, (1.28) it follows that XQ =0,a.s., .(„-,)... ^. ., 0-29) P(X^<x\x„kl and so the process (X^,n>0) is a sequence of independent t r.v. called a completely non-homogeneous renewal process (in short CNHRP) of kernel F.

Non-Homogeneous Semi-Markov Processes

233

1.3 Intensities for CNHSMP 1.3.1 Definition From now on, we will assume that the mass flinctions of the kernel Q are absolutely continuous and so have densities denoted by the corresponding small letters. Defining as usual the intensity random function /ij(t,co)dt,jeI,t>0,coeQ represents the probability that there is a transition in (t,t+dt) to state7 given all the past related to the considered CHMMAP of kernel Q up to time /, (T

^^'^Q

^•(^^^) = ,_NioT

rr^

t)

^^JeI,t>0,coeQ

(1.30)

or by relation (1.4) ^(tM

=

\_NiOrr

rr

A

jeI,t>0,cveQ.

(1.31)

If we do not ask for a transition toy but only for a transition, we have as intensity function: m

or A(t,co) =

^^'^h (T t) '"'"''"'

,t>0,coeQ.

(1.33)

Let us remark that for the classical homogeneous SMP, relations (1.31) and (1.28) become:

X.{t,co) = — ^ " '

"

,?>0,6;€Q.

(1.35)

More particularly, for homogeneous continuous Markov processes for which we know that: y:.(0 = ^ ^ - ^ ' , / > o , / E / (1.36) with /li,/l2,...,/l^ are strictly positive known constants. Consequently, in this particular case, relations (1.34) become: A,(/,^) = / 7 , ^ ^ / , , y E / , / > 0 and

(1.37)

234

Chapter 6

A(^dy) = I^.,7G/,^>0.

(1.38)

For m=l, we get the classical (homogeneous) Poisson process for which: A(t,co) = I,t>0, (1.39) 1.3.2 Generalised Polya Processes Let us come back to CNHDRP and CNHRP defined in section 1.2.4. We know that a CNHDRP is defined by the kernel (1.31): Q(s,t) = C"-'^F(sMs,t >0,t-s>0

(1.40)

and that the sequence {X^,n > O) satisfies relations (1.27). Assuming that there exists the density sequence C"-''f(sMs,t>Oj-s>0, "-'q(s,t) = fa \

(1.41)

^-V(^,o=^^"-'^^(^,oL \dt

J

the intensity function related to the kernel Q is given by formula (1.34) giving here: ^(^) f(T t)

Ut,co)=. . j y Z

.,^>Q,^eQ.

(1.42)

Inversely, from this last relation, we get: <"-"F(/„,0-l-exp{-J]A(«|iV(0-«,r^(,)-r„^

(1.43)

where we have denoted X{u\N{t) = nJ,,,^ =Q = X{uM

(1.44)

on the set [co\N{t) = nJj^^^^=t^],

(1.45)

Now, if we choose X{u\N{t) = nJ,^,^=0=^-^,a>Q,b,s>Q,

(1.46)

we get the family of Polya processes defined by De Vylder (1977) for which by (1.43): /

(«)F ( ? „ , 0 - l - e x p

N-(o+i«)/i

\ + st \ + St„j

(1.47)

The special case ^==0 has also a sense and gives the particular Polya process defined for example in Buhlmann (1970) for which then, by relation (1.31): X{t,(D) = a + bN{t\ (1.48) and by relation (1.43):

Non-Homogeneous Semi-Markov Processes

235

SO that, in this case, the sequence {X^,n > O) is a CNHRP giving for the special case b=0, a classical Poisson process. Let us point out that the Polya processes are used for modelling epidemiological phenomena in which of course the number of cases already observed, say n, has strong influence in the structure of the future cases.

2 NON-HOMOGENEOUS MARKOV CHAIN In section 1.2.2, we define the particular case of non-homogeneous Markov chains (NHMC) as a particular completely non-homogeneous Markov chain (CNHMC) for which the kernel (^''"^^P(^)) is independent of ^ so that we have:

'"-"? = ["->,]

(2.1)

with, from relation (1.5) '"-''p,=P(J„-j\j„.,-i),i,J^I or with Doob 's (1953) notations:

(2.2)

<"->, = P(J„ = 7V„-, = 0 = '""V, - ,„-^,P,,iJ e/.

(2.3)

The computation of the following probabihties pf'-P(J„=j\Jo=iUJ^I is simple. Indeed, using the Markov property, we get successively:

(2.4)

m

(2)^y(0) m

(1)

(2.5)

m

pf=YL'''p.p'lhT'id^i^ k=\ l=\

and in general we get: Wp^(0)p^(i)p^...^(«-

'^P,^>1,

<"'P=[4">].

(2.6)

More generally, if

Y'"\k)^[l^\k)], p\^ik)^P{J„=j\j,=i)JJeI, we get:

(2.7)

236

Chapter 6

F^"\k) = V^^\k)¥^"-^\k) for every / such that ll

(2.8)

and

if

the column vector of marginal distributions of

y„,«>l,weget: 7=1

or in matrix form: p^"^ =p¥^"\n>l. (2.10) Though it must be clear that the use of non-homogeneous models is above all used for describing short or middle time evolution, it may be that there exists a stationary behaviour for the marginal distributions of 7^,^ ^ oo. As we will not use stationary non-homogeneous models in the sequel, we will not develop these results here but we will mention that there exist the concepts of ergodic and delta coefficients (see for example Isaacson and Madsen (1976) used to prove necessary and sufficient conditions for strong ergodicity of NHMC defined as follows: Definition 2.1 An NHMC of kernel (^"'^^P(^)) is strongly ergodic iff there exists a vector q such that q = (?p-..,
,

(2.11)

such that for all n: limsup||p.^"^P^"^^^P...^"^'^P-q|| = 0, ^-^00

p

II

(2.12)

II

where p is a starting distribution for J^. Then Isaacson and Madsen (1976) proved that strong ergodicity is equivalent to the existence of a constant matrix Q such that for each n\ lij^||(«)p(^;+i)p ("^^)p_Q|l 0. (2.13)

3 NON-HOMOGENEOUS MARKOV PROCESSES The NHMPs, also called time continuous Markov processes, are often used for modelling the stochastic dynamics of system evolution involved in a lot of fields of applications like insurance for example.

Non-Homogeneous Semi-Markov Processes

237

We begin with a classical presentation of these processes and then we show how there are particular NHSMPs.

3.1 Classical Definition Let (Q,3,(3^),P)be the basic filtered probability space on which the adapted stochastic system S= (S(t),t e [0,T]) will be defined. Definition 3.1 The S-process is an NHMP with I as state space iff P{S{t)^j\S{t,) = j,,k = \,,,,,n) = P{S{t) = j\S{t„) = j„\

^^^^

\/j\,...,feiyO
(3.3)

We also define: (^^(t,t) = S^,iJeL (3.4) If the matrix (s,t) depends only on t-s, the Markov process is called homogeneous and non-homogeneous in the other case. In the homogeneous case, it suffices to consider the following probabilities (3.5) ^,j(t) = P(S(t) = j\S(0) = i),i,jeI as indeed: (/>^j(t-s) = P(S(t) = j\S(s) = i),i,jeI,s
(3.6)

We also define with Haberman and Pitacco (1999) the so-called occupancy probabilities as follows: (3.7) ^,(5,0 - PiS(z) = i,s < z
3.2 The Chapman Kolmogorov Equations We have the following proposition : Proposition 3.1 The transition probabilities ^^.(s,t),i,jel,0<s the following relations called the Chapman Kolmogorov equations:


238

Chapter 6

.j{s,t) = Y,ik {s,u)
(3.8)
(/f..(s,t)=^X^,u)^X^,tUjeI,0<s
(3.9)

Proof Relation (3.3) gives that: (^,j(s,t) = P(S(t) = j\Sis) = iUjeI,0<s
(3.10)

and so, we also have: ^,j(s,t)^j;.^(S(t)

= j,S(u) = k\Sis)^i).

(3.11)

k=\

Using conditional probability rules given in Chapter 1, we get: m

,f>,j(s,t) = Y,P{S(u) ^k\S(s) - i)P(S(t) = j,\S(s) = i,S(u) = k).

(3.12)

Applying now the Markov property, we get: m

,j(s,t)==Y.PiS(u)^k\S(s)^i)PiSit)

= j,\S(u) = k),

(3.13)

or with relation (2.3): k=]

that are indeed the Chapman Kolmogorov relations. To prove relations (3.9), we know from relations (3.7) that: ^,(^,0 = P(S(z) = i,s
(3.15)

and using once more the conditional probability rules given in Chapter 1, we get: *<'•"= , , P(S(z) = i,s,(s,t) = ^Xs,u)^X^^,tl

(3.16)

that are in fact the relations (3.9) to be proved.

n

(3.17)

3.3 Intensity Functions For a general non-homogeneous Markov process, we have introduced the transition probabilities defined by relation (3.3). Taking s=t and t'=t+dt in this relation leads to the following relation: (^,j(t,t + dt) = P(S(t + dt) = j\S(t) = i),i:^jeI,0<s
Non-Homogeneous Semi-Markov Processes

239

for s=t and for all t belonging to [0,7], we can introduce for all ij with / ^ j the functions fi,{t)^^{t,t),i^j, (3.19) ot called the intensity functions related to the considered Markov process. From the definition of partial derivatives, we have the following approximated result: (l),j{t,t + dt)^lu.j{t)dt,i^j,te[OJl (3.20) of course for dt small enough. So the intensity function ju.j at time t represents the instantaneous rate of a transition from state / to statey at t, given that / is the state occupied at time t. Let us now define the next intensity function //p/ G / as follows:

A-Z/^r

(3.21)

From relation (3.19), we get:

/'^(O-Z^C^O

(3.22)

and as of course: m

£ ^ . ( 0 = i;/,y6/,rG[o,r],

(3.23)

we obtain from relation (3.22) that Ot

or finally:

^Xt)^J-Ml,i.I.

(3.25)

Ot

From relation (3.24) the interpretation of this last intensity function is just the instantaneous rate of leaving state i over the infinitesimal interval [t,t+dt), given that / is the state occupied at time t, From now on, we always assume that these transition functions are Lebesgue integrable on the compact subsets of the real line. We can now prove the following proposition concerning the Kolmogorov forward and backward differential equations. Proposition 3.2 {Kolmogorov forward dmd backward differential equations) The transition probabilitiy functions satisfy the two following sets of differential equations

240

Chapter 6

^ 1

at

k^j

(l).j{z,z) =

(3.26)

d,j,iJeL

(Kolmogorov forward differential equations) The functions ^y(z,t) satisfy for all fixed z in [0,7] the following differential system of equations: dt

t7!

(3.27)

^..(z,z) = SyJ,jeI. (Kolmogorov backward differential equations) Proof Let us only prove result (i); the proof of result (ii) being similar. From the Chapman Kolmogorov equations (3.8), we can write for all / andy belonging to /.• (3.28) 4(z,? + AO = J^,,(z,t)P,jit,t + At) + ,j(z,t)(^jjit,t + At), k*j

and so ^^j(Z,t + At)-(^y(Z,t)

At J>,j(t,t + At) ^ ^^ J>jj(t,t + At)-l tfj At ' At Now, from relation (3.23), we get: <^jj(t,t + At) = l-'^<^j,(t,t + At),

(3.29)

(3.30)

k*j

and so coming back to relation (3.29), we obtain: J>,jit,t + At) ^y(z,t + At)-^y(z,t)

It

(l>j,{t,t + At)

^If'^^^'^^t—^^^^"^Ir^r- ^'-''

Letting A^ -> 0 , we get the result (3.26).

n

It can be shown here that the fact that the possible number of states is finite implies that with the integrability assumption of the intensity functions, both systems (3.26) and (3.27) together with their initial conditions uniquely determine the probability transition functions ^^.. The next proposition is related to the occupancy probabilities defined by relations (3.7). Proposition 3.3 (explicit form of occupancy probabilities) For every i belonging to I, we have:

Non-Homogeneous Semi-Markov Processes

P,Xs,t) = e ^

241

,0<s
(3.32)

Proof From relation (3.9) of Proposition 3.1, we get: ^^(s,t + At) = ^X^MXt,t + At)

(3.33)

and so (/>,(s,t + At)-(/>,(s,t) d,(t,t + At)-l = —-^ = (^/(^,0= 7. (3.34) At At As the probability of having more than one transition in the infinitesimal interval (t;t+dt) is a o(t), we have that: ^..(/,/ -f- At) + o(t) = (/>ii(t,t + At), (3.35) so that

M^j + ^t)-^,(s,t) ~A^

^^i-^,(^/ + AO + o(0

= -(piAz^t)

At

.

(3.36)

As we know that l i m ^ ^ = 0,

(3.37)

we get: -^ll-L = -^^X^,t)juM ot By integration and using as initial condition: M^,s) = l, we get the result to be proved, that is relation (3.32).

(3.38)

(3.39) n

These last results show that it is possible to define a continuous time Markov process by means of the matrix of transitions functions //==(//,), (3.40) where: JUu(t) = -MMi^LO
3.4 The Special Case Of Constant Intensity Functions From the preceding subsection, we know that the knowledge of all intensity functions characterize the Markov process. Usually, these intensity functions, here supposed to be constant, are given under matrix form by the mxm matrix M defined by

Chapter 6

242

(3.42) and of course this special case corresponds to a homogeneous Markov process, I.e.

P(S(t) - j\S(s) = i) = (f>.j (t-s),i,j

€l,0<s
(3.43)

Moreover P represents the matrix of transitions functions, so that (3.44) (3.45) 0(0) = I. Here the Chapman Kolmogorov equations and the forward and backward Chapman Kolmogorov differential equations respectively take the following matrix forms: 0(/) = 0 ( ^ ) 0 ( / - ^ ) , O'(0 = O(0M, (3.46) O'(0 = - M O ( 0 . From the matrix differential calculus, it follows that the solutions of the forward and backward Chapman Kolmogorov differential equations are given under the form: (3.47) 2! Unfortunately, this matrix series is in general slowly convergent so that this result is not very useful for numerical computation. Another and more useful model was given by Cox and Miller (1965) assuming matrix M has m distinct eigenvalues A^,...,/l^ with as corresponding eigenvectors

0(0:

If A represents the mxm matrix whose column / is the eigenvector 1);,/ = l,...,m; it is known in matrix algebra that M can be factorized as follows: M = ADA-',

r;i, 0 D= 0 0

...

0 0

(3.48)

... A

and so from result (3.46) it can be shown that 0... ...0

0(0 = A 0 0

...0 . ...0 e"'

or equivalently under scalar form:

(3.49)

Non-Homogeneous Semi-Markov Processes

i=\

243

{5.50)

A = [a,],A-'=[6,]. An extension of the result of Cox and Miller has been given for the more general case ofpiecewise constant intensities.

3.5 NHMP As A Special Case Of NHSMP For NHSMP, relation (1.31) gives the form of intensity functions: l.(/,^)=^^-"'-^^""'^'^-^-"-^^"'"''\yE/,/>0,^eQ.

(3.51)

Ur, It Jfjf,-) —j,If^^,^ —s:

P,jis)xf..{s,t) _ Xj(t\j,,,,=iJ,,,,=s)='^\ ' ' ; \ \i,jel,s,t>0j-s>0,

(3.52)

and so clearly the intensity functions depend generally on the time of the last transition and in some particular cases only on t-s. On comparing with relation (3.19), we see that for NHMP, the intensity functions only depend on t and not on 7^^^^ = s and so we can see an NHMP as a special NHSMP for which this last condition is satisfied. Moreover, the classical theory immediately works with the transition probabilities P.j{t)JJ Gl,t>0and not with the corresponding kernel Q(s,t).

4 AN NHMC TRANSPORTATION EXAMPLE The following example of a non-homogeneous Markov process concerns the taxi cab transportation problem that we developed in Chapter 3. As in the homogeneous semi-Markov case, we consider that a taxicab driver will work for 8 hours with a time interval of 15 minutes. So we will follow the taxi cab in the transient case within 32 time periods. In this example the non-homogeneous transition chain and the related evolution equation are given. The evolution equation has the following really simple expression: O(5,0 = O ( ^ , / - l ) * P ( / - l ) , (4.1) where Q>{h,k) and V{k) are m-order matrices. ^^.(/?,A:) represents the probability that a cab, that is in the state / at time s will be in the state y at time t, Pij(h) the transition probability from the state / to the statey at time hofa cab.

Chapter 6

244

The inputs are: m = 3, T=32, and the matrix P that is partially reported in Table 4.1. The data of Table 4.1 were given as an example and are not real data. In real application they could be constructed by observation in the following way. We suppose we know all the runs that were done in one month by all taxicab drivers, so we know for each run the starting time, the starting zone and the arriving zone. P(0)

0.3 0.4

0.4 0.2

0.32

0.38 P(20) 0.35 0.62 0.18

0.54

0.3 0.23

0.3 0.4 0.3

0.39 0.35 0.28

P(5) 0.35 0.32 0.33 P(25)

P(10) 0.26 0.33 0.39

0.49

0.3

0.3

0.42 0.28 P(29)

0.23

0.11 0.5 0.4 0.1 0.5 0.08 0.3 0.6 0.1 0.3 0.24 0.63 0.2 0.59 0.13 Table 4.1: taxi cab non-homogeneous MC

0.4 0.6 0.1

0.21 0.28 0.49

0.1 0.1 0.7

The evolution equation result matrix is reported in Table 4.2. 0(0,1)

0.3 0.4

0.4 0.2

0.32

0.38 0(5,6)

0(0,10)

0(0,20)

0.3 0.4 0.3

0.3419 0.3335 0.3246 0.3646 0.3939 0.2415 0.3419 0.3335 0.3246 0.3646 0.3939 0.2415 0.3419 0.3335 0.3246 0.3646 0.3939 0.2415 0(5,15) 0(5,25)

0.39 0.35 0.28

0.35 0.26 0.32 0.33 0.33 0.39 0(10,11)

0.3438 0.3336 0.3226 0.3659 0.4528 0.1813 0.3438 0.3336 0.3226 0.3659 0.4528 0.1813 0.3438 0.3336 0.3226 0.3659 0.4528 0.1813 0(10,21) 0(10,31)

0.49

0.21 0.3 0.42 0.28 0.28 0.49 0(20,21)

0.3706 0.4153 0.2141 0.3706 0.4152 0.2141 0.3706 0.4152 0.2141 0(20,26)

0.35 0.11 0.62 0.08 0.18 0.59 {J)(25,26)

0.3630 0.4443 0.1927 0.3458 0.4120 0.2422 0.3632 0.4463 0.1905 0.3459 0.4121 0.2420 0.3598 0.4300 0.2102 0.3455 0.4112 0.2433 0(25,32) 0(25,28)

0.3 0.23 0.54

0.3 0.23

0.5 0.3

0.4 0.6 1

0.1

0.1 1

0.3662 0.4455 0.1883 0.3582 0.4535 0.1883

0.3457 0.4119 0.2424 0.3457 0.4119 0.2424 0.3457 0.4119 0.2424 0(20,31)

0.3459 0.4125 0.3459 0.4125

1

0.2416 1 0.2416 1

Non-Homogeneous Semi-Markov Processes 0.24 0.5 0.3 0.2

245

0.13 0.63 0.3251 0.3204 0.3545 0.3406 0.3966 0.2628
We should construct the transition matrix P from the known data. We could construct a matrix A where it results that: Gij {s) = number of runs that in the month went from the zone / to the zone j starting at time s (if s is equal to 9 it means that the run began 2 hours and a quarter after the beginning of the shift). Once we construct this matrix it will result that: (4.2) k

where m represents the number of states, which in our case is 3.

Chapter 7 MARKOV AND PROCESSES

SEMI-MARKOV

REWARD

1 REWARD STRUCTURE 1.1 Classification And Notation For applicative purposes the association of a sum of money to a state of the system and to a state transition assumes great relevance. In general, this can be done attaching a reward structure to the process. This structure can be seen as a random variable associated with the state occupancies and transitions (Howard (1971) vol. 2). The rewards can be of different kinds (amounts of money, basis points, time,..), but in the book we will consider only amounts of money. These amounts can be positive, if for the system they can be seen as a benefit and negative if they can be considered as a cost. In this chapter we will describe reward structures for discrete time Markov processes and continuous time semi-Markov processes, in both the homogeneous and non-homogeneous cases. For this reason, we now discuss how to manage discrete time and continuous time reward structures. More precisely, first, we will introduce the Discrete Time Markov ReWard Processes (DTMRWP) and then the Continuous Time Semi-Markov ReWard Processes (CTSMRWP) both in homogeneous and non-homogeneous cases. The reward processes can be seen as classes of stochastic processes that we can classify in different cases. First of all, the distinction of reward processes in discounted and not discounted or undiscounted cdiSQS is necessary. If it is clear that in finance and insurance problems the reward processes without discount is nonsense, nevertheless, in some rehability problems it could have some meaning. Furthermore it is clear that the absence of interest rates simplify the model. The rewards can be studied by a continuous time process or a discrete time process. The rewards can be given for a state transition (impulse rewards) or during the permanence in a state (rate reward) (see Qureshi and Sanders (1994)). The rewards that depend on permanence in the state could be considered as a generalisation of annuity. So there are immediate permanence rewards that are

248

Chapter 7

paid at the end of the period and due permanence rewards that are paid at the beginning of the period. The rewards can hQ fixed in time or can change during time, The rewards given for permanence in a state can be independent of the next transition or can depend on the next state that will be visited. In the discounted case it is possible to have a fixed interest rate or a term structure of interest rates. Furthermore, in non-homogeneous cases, the rewards and the interest rates can be non-homogeneous. All these hypotheses imply different formulas of the system evolution equation. We will first give the simplest ones, generalising step by step up to writing the more difficult ones. We will use the following notation: y/i: reward given for permanence in state /; this reward doesn't change with time and the future transition. y/^ (t): reward given for permanence in state / at time /. This reward changes with time and doesn't change because of the future transition; / denotes the payment time. y/i{s,t) : reward given for permanence in state / given that the system enters this state at time s and goes to the statey at time t (non-homogeneity). Yij'- reward given for permanence in the state /. This reward doesn't change with time and changes because of the future transition. y/ij (t): reward given for permanence in state / paid or received at time /. This reward changes with time and future transition. i//ij(s,t): non-homogeneous permanence reward. It changes because of the future transition. /ij: reward given for the transition from state / to statey (impulse reward). This reward doesn't change with time. yij{t): reward given for transition from state / to statey. This reward changes because of time. Yij^s.t): impulse reward given for transition from state / to state 7 given that the system entered in / at time s and went to the statey at time t (non-homogeneity). The different kinds of y/ rewards represent different types of annuities paid or received for the stay in a state. In the discrete time case, the immediate case means that the reward is paid at the end of each period; in the due case the reward will be paid at the beginning of the

Reward Processes

249

period. The impulse rewards y represent lump sums that are paid at the transition instant.

1.2 Discrete Time Fixed Interest Rate To study the process with discounting, let us recall some basic results for computing present values of amounts of money, annuities and also the related notation. For more details, refer to Volpe di Prignano (1985) or KeUison (1991). We begin with the classical case of an interest rate per period that is constant for all periods and represented by r, So, we get the following results: V = (1 + r)"': represents the one period discount factor, where r is the interest rate for one period, supposed to be constant in time, 1-v' a^ = : represents the present value of an unitary annuity-immediate for the payment of one monetary unit at each end of the t periods, a-^ =-

- represents the present value of an infinite discrete time unitary

annuity-immediate, also called perpetuity, giving successive payments for all time, : represents the present value of 1+r a unitary annuity-due of length / giving payments of one monetary unit paid at each beginning of the t periods and dis the so-called discount rate, ^^=1 +

= (1 + r)a^ =

;d =

d-^ = 1 + - = (1 + J")ci^ ~ 77 • represents the present value of an infinite discrete time unitary annuity-due,

1.3 Discrete Time Variable Interest Rate In this section, we suppose that the interest rates for period k (k=l,2,...) may depend on the period and so will be represented by r^,A: = 1,2,.... Now, results from the preceding subsection become: k

v(k) = f 7 ( l + f^hT^'•represents the k-period discount factor at time 0, giving, at h=\

time 0, the discounted value of one monetary unit to be paid at the end of period k

250

Chapter 7

^ v{k){i/i (k): represents the present value of an annuity-immediate with variable k=]

rewards in the time and variable interest rate on a time horizon /. The infinite rates case is given by the limit to oo of the formula. This series can converge or not depending on the values of y/'fik) and of v(k). t-\

^y(k)y/i(k):

represents the present value of an annuity-due with variable

rewards in the time and variable interest rate; clearly v(0) = 1, k

v(s,k)= ]~J(l + r/;)~^ represents the k period discount factor at time s, with h=s+\

homogeneous interest rates giving, at time s, the discounted value of one monetary unit to be paid at the end of period k, v(s,k) = J~[ (1+ ^^z;) ' represents the k period discount factor at time s, with h=s+]

non-homogeneous interest rates, ^ v(s,k)y/i(s,k)

represents the present value at time s of an annuity-immediate

paid on the time interval (^,/]with non-homogeneous rewards and variable interest rate. ^ v(s,k)y/j(s,k)

represents the present value at time s of an annuity-immediate

paid on the time interval (^,/]with non-homogeneous rewards and nonhomogeneous interest rate. /-I

^v(s,k)y/i(s,k)

represents the present value at time s, of a annuity-due paid on

k=s

the time interval [s,t] with non-homogeneous rewards and variable interest rate; clearly v(s,s) = 1, /-I

^v(s,k)i//i(s,k)

represents the present value at time s, of a annuity-due paid on

k=s

the time interval [s,t] with non-homogeneous rewards and non-homogeneous interest rate.

1.4 Continuous Time Fixed Interest Rate Intensity For our models continuous in time, we need to introduce thQ force of interest or interest rate intensity function S such that S (t) gives the value of the so-called

Reward Processes

251

"spot rate "at time /; this means that on a small interval (t,t + At), the interest given for one monetary unit is well approximated by S(t)At. A very special case is given for S constant. In this special case, it can easily be proved that we have the following results: e~^': represents the discount factor for a time t, giving the present value of one monetary unit payable at time t, 1 - e~^' ^lls ~ — 7 — * represents the present value of a continuous time unitary annuity of a time length /, a^ = —: represents the present value of a continuous time unitary perpetuity that is of infinite length.

1.5 Continuous Time Variable Interest Rate Intensity When the function S is time dependent, the preceding results become: v(&) = e "^ : represents the continuous time discount factor given the present value at time 0 of one monetary unit available at time & , \y/i{9)v{3)d9:

represents the value of a continuous time annuity of length /.

0

The annuity length / can be posed also equal to oo . If the instantaneous intensity rate is supposed constant, this last result becomes 00

\y/,{9)e-''d&,

(1.1)

0

that is the Laplace transform of the reward function. This is an observation that outlines the strict correlation between the Laplace transform and financial Mathematics (for more details see (Levi 1964)). Computing present values at time t, it often happens that the fimction d also depends on s\ in that case, we say that the intensity is non-homogeneous and homogeneous in the other case. So, we have the following definitions: 9

v{s, 9) = e ' : represents the continuous time discount factor for the time interval {s,9] with a homogeneous intensity factor given the present value at time s of one monetary unit available at time 9 ,

252

Chapter 7

v(s,&) = e " : represents the continuous time discount factor for the time interval [s,&] with a non-homogeneous intensity factor given the present value at time 0 of one monetary unit available at time i9, \i//i(s,S)v(s,&)dS:

represents the present value of a continuous time annuity of

s

length t-s and with homogeneous intensity interest rate. The annuity length can go from ^ to oo ; the interest rate intensity could be also non-homogeneous. In the following sections, we will show that reward structures are strongly related to Markov and semi-Markov processes. Reward structure can be considered a very general structure given a financial and economic meaning very useful in stochastic modelling. For example, this behaviour is particularly efficient to construct models useful to follow the dynamic evolution for example of insurance problems. In this case, the permanence in a state usually involves the payment of a premium or the receipt of a benefit. Furthermore often the transition from one state to another induces some other cost or benefit. We will come back later to all these aspects of insurance stochastic models with rewards.

2 DISCRETE TIME MARKOV REWARD PROCESSES. 2.1 Undiscounted Case. We suppose we have a system that evolves randomly within the Markov property having / = {l,...,m} as state space. That means, as we already said, that it moves in such a way that the future is a function only of the present. Moreover, a reward structure is attached to the evolution of the system. 2.1.1 First Model In our first model, we suppose that: a) rewards are fixed in time, b) rewards are given only for permanence in the state, c) rewards are paid at the end of each period. Under these assumptions, if J^^"^ represents the mean total rewards paid or received after n transitions, given that at time 0 the system was in the state / ,then in the homogeneous case, the following results are obtained: V^'^ = 0 , (2.1)

Reward Processes

253

Vr'-y^;- Vr+Wn m

(2.2) m

Vr = 0 + ^. + XA^V. - Vr + ZP'M ,

(2.3)

and in general: m

n-'^ vr'^Y.pt\.-

(2.4)

k=\

For the non-homogeneous case, the preceding results become: m

V,'"Ks) = VrKs)

+ Y.P'^,-'\s)Y, .

(2.5)

Relations (2.4) and (2.5) give the Mean Total Rewards (MTR) that will be expected at time n if the system is in state / at time s or at time s=0 for the homogeneous case. In both cases these results are obtained recursively adding the MTR expected at time (^+^-1) to the mean reward that will be paid at time (s+n), 2.1.2 Second Model For our second model, we assume that: a) rewards are variable in time, b) rewards are given only for permanence in the state, c) rewards are paid at the end of the period. In this case, for the homogeneous case, we get: ^<"=?^,(l),

Vr'= f^<''+2:^.'V.(2),

(2.6)

(2.7)

m

n"'= vr'+Y,pt\M-

(2.8)

k=\

For the non-homogeneous case, this general result becomes: m

f^W(5)= V,^"-'\s) + Y.P^r\s)y^,{s

+ n).

(2.9)

Recursive relations (2.8) and (2.9) for MTR have the same meaning as relations (2.4) and (2.5). 2.1.3 Third Model Now we suppose that: a) rewards are variable in time, b) rewards are given for permanence in the state and at a given transition, c) rewards are paid at the end of the period. For the homogeneous case, the following formulas are obtained:

Chapter 7

254

v^''-Wii^)

(2.10)

+ t.P>''U(X), m

f

m

\

(2.11)

and for the non-homogeneous one: m

f

m

\

^=1

V

y=i

)

(2.12)

Recursive relations (2.11) and (2.12) outline the presence in the MTR of the lump sums that are given or taken at the instant of the change of state. 2.1.4 Fourth Model For this model, the assumptions are: a) rewards are variable in time, b) permanence rewards are functions of the next transition, c) rewards are given also at the transitions, d) rewards are paid at the end of the period. Here, respectively for homogeneous and non-homogeneous cases we obtain: m

yin) ^ vr''+Zpt''\

f m

\

ZP^j{nj(n) + n(^)) I

^

r rn

\

^=1

V y=i

J

(2.13)

(2.14)

Relation (2.13) is the most general formula in the homogeneous cases given above. MTR takes into account that the permanence rewards are a function of the transition. 2.1.5 Fifth Model The last immediate case considers non-homogeneous rewards. We suppose that: a) rewards vary because of the starting time and the time spent after, b) permanence rewards are functions of the next transition, c) rewards are given also at the transitions, d) rewards are paid at the end of the period. In this case, the general formula is:

Reward Processes

255

m

rm

\

k=\

\ y=i

J

(2.15)

In the undiscounted case the difference between the due and immediate cases is not of great relevance. Anyway in the following the formulas for the MTR computation of three due cases are reported. 2.1.6 Sixth Model For this model, the assumptions are: a) rewards are variable in time, b) rewards are given only for permanence in the state, c) rewards are paid at the beginning of the period. And so, the following formulas hold respectively for homogeneous and nonhomogeneous cases: '^<"=!^,(0), (2.16) m

Vr=

(2.17)

J^<" + E A 5 V . ( 1 ) , k=\ m

VM^ Vt''+Y^pt\,{n-\),

(2.18)

and for the non-homogeneous case: m

V^"\s)= ^("-•)(^) + ^ ; , r ) ( 5 ) 5 ^ , ( , + „ _ l ) .

(2.19)

The differences between (2.8) (2.9) and (2.18) (2.19) are given only by the different moments of the payment of the permanence rewards. 2.1.7 Seventh Model Basic assumptions are: a) rewards are variable in time, b) permanence rewards are functions of the next transition, c) rewards are given also at the transitions, d) rewards are paid at the beginning of the period. In the homogeneous case we obtain the following: m

v^''=¥iM

+ Y,P>kyAi), m

m

m

yin) ^ vr' + Y.P>r''Y,P,¥,in-\) k=\

(2.20)

7=1

and in the non-homogeneous case:

m

+ Y.Pt''Y.P''jy^ji"')^ /t=l

j=\

(2-21)

256

Chapter 7

*=' m

'•='

(2.22)

m

The last proposed due case is the one considering non-homogeneous rewards. The related hypotheses are the following: 2.1.8 Eighth Model a) rewards vary with the arriving time and the time spent after, b) permanence rewards are functions of the next transition, c) rewards are given also at the transitions, d) rewards are paid at the beginning of the period. Here, we get:

V,(^\s) = Vt'\s)

+ Y^pt'\s)Y^p,{s k=\

k=\

j=\

+ n - \)y/,j{s,s + n-\) (2.23)

j=\

We would like to outline once more the recursive nature of these formulas that greatly simplifies the construction of the related algorithms.

2.2 Discounted Case. 2.2.1 Immediate Cases In our system, any time that a reward is paid it will be discounted at time 0 in the homogeneous case and at time s in the non-homogeneous one. 2.2.1.1 First Model In the first formulation of this case we suppose that: a) rewards are fixed in time, b) rewards are given only for permanence in the state, c) rewards are paid at the end of the period, d) the interest rate per period r is fixed. In this case Vl""^ represents the Reward Mean Present Value (RWMPV) of all the rewards paid or received after n transitions, given that at time 0 the system was in the state / and Vl^"^ {s) represents the RWMPV of all the rewards paid or received after n transitions, given that at time s the system was in the state /.

Reward Processes

257

Under these assumptions, we obtain for the homogeneous case: V,^'^= (l + r)-V, =(l + r)-V,, m

(2.24)

m

^<^'= (l + r)-V,.+ v^X/'«V = f^*"+ V ^ Z A ^ V ,

(2.25)

m

yM ^ f/(«-.)+

v-^^rV* ,

(2.26)

and for the non-homogeneous one: m

V<"^(s)=V,^"-^\s)+ v"Ypt\s)¥,-

(2.27)

k=\

Formulas (2.26) and (2.27) give the RWMPV in the simplest homogeneous and non-homogeneous cases, i.e. with a fixed interest rate and fixed rewards. We see that the introduction of interest rates preserves the recursivity of the formulas. 2.2.1.2 Second Model Now we introduce the variable interest rates and we will use the formulas given in the first paragraph of this chapter for discounting. In this first formulation with variable interest rate we have that: a) rewards are fixed in time, b) rewards are given only for permanence in the state, c) rewards are paid at the end of the period, d) the interest rate r is deterministic but variable with time. In this case, the values of Vl^"^ and Vi^"\s) (RWMPV) will be obtained taking into account the deterministic interest rate structure. Under these hypotheses we get the following formulas for the homogeneous case: m

Vr = v(W, + V{2)Y,PTW,

m

- V^U

K(2)XAiV*,

k=\ m

^C) = f^(-)+

v{n)Y,p\r\,

(2.28)

k=\

,

(2.29)

k=\

and for the non-homogeneous one: m

^ W ( ^ ) ^ V,^"-^\s) + v{s,s + n)Y^pt'\s)if,,

.

(2.30)

k=\

Relations (2.29) and (2.30) give the RWMPV in homogeneous and nonhomogeneous cases with fixed rewards and variable interest rates. Here too, the recursivity is preserved. 2.2.1.3 Third Model Now we introduce the variability of rewards with the following hypotheses: a) rewards are variable in time,

258

Chapter 7

b) rewards are given only for permanence in the state, c) rewards are paid at the end of the period, d) the interest rate r is fixed. In the homogeneous case, the following formulas, still recursive, hold: m

f^(^):. f ^ ' ' ' + v ^ £ / 7 < V ( 2 ) ,

(2.31)

m

V("^= vr'

+ v"J,p\r\,i.n),

(2.32)

and in the non-homogeneous one: m

V,i^\s) = Vt\s)

+ v"Y^pt\s)y/,{s

+ n).

(2.33)

2.2.1.4 Fourth Model Now we consider the case of variable interest rates with variable rewards under the following assumptions: a) rewards are variable in time, b) rewards are given only for permanence in the state, c) rewards are paid at the end of the period, d) the interest rates are time dependent. For the homogeneous case, related recursive formulas for RWMPV are: m

VP^^

f^<'>+K2)ZA'V.(2),

f^W = vr'

(2.34)

+ y{n)Zpt\ki.n)

,

(2.35)

and for the non-homogeneous one: m

V,^"\s) = V,^"-'\s) + v(s,s + n)Y,pir\s)v^ds

+ n).

(2.36)

2.2.1.5 Fifth Model Now, we will introduce the y rewards under the following assumptions: a) rewards are variable in time, b) rewards are given for permanence in the state and at a given transition, c) rewards are paid at the end of the period, d) the interest rate r is fixed. Under these hypotheses, respectively for homogeneous and non-homogeneous cases, we obtain the following recursive relations: m

y>'"' -

^'"-" + v - ^ / ^ r "

/

m

Vdn) + XP'^Mn)

\

I

(2.37)

259

Reward Processes

V,^"\s)= Vr'\s)

+ v"Y^pjr\s)\

Ms + n) + j;^p,(s

+ n)nj(s + n) .(2.38)

Relations (2.37) and (2.38) outline the presence in the RWMPV of the lump sums given or taken on the instant of the state change.

2.2.1.6 Sixth Model Now we will present the formulas in the same previous assumptions but taking into account the variability of interest rates. More precisely we assume that: a) rewards are variable in time, b) rewards are given for permanence in the state and at a given transition, c) rewards are paid at the end of the period, d) the interest rates are time dependent. In this case, we get the following homogeneous and non-homogeneous formulas; m

f

m

\

(2.39)

m

/

m

\

(2.40)

k=\

2.2.1.7 Seventh Model This model introduces the permanence rewards that depend on the future state to be visited under the following assumptions: a) rewards are variable in time, b) permanence rewards are functions of the transition, c) rewards are given also at the transitions, d) rewards are paid at the end of the period, e) the interest rate is fixed. Then, we obtain, respectively in homogeneous and non-homogeneous cases, the following formulas: (2.41)

m

n

V" Z P'"^''^ (^)Z Pkj (^ + «) (Ykj {s + n) + y/,^ {s + n)). ^=1

y=i

(2.42)

260

Chapter 7

2.2.1.8 Eighth Model Now the following assumptions are made: a) rewards are variable in time, b) permanence rewards are functions of the transition, c) rewards are given also at the transitions, d) rewards are paid at the end of the period, e) the interest rate is variable. The following formulas give the evolution equations of the RWMPV for homogeneous and non-homogeneous cases: m

n

V^M = f^.c-') + K « ) ^ / ^ r ' E A ; ( r , ( « ) + ?^,(«)),

m n v(s,s + n)Y, pf^ {s)Y, Pkj {s + n){ y,j {s + n) + ij/^j {s + n)).

(2.43)

(2.44)

7=1

k=\

2.2.1.9 Ninth Model We now consider the case with non-homogeneous interest rates and rewards under the following assumptions: a) rewards are variable because of the arriving time in the state and the time spent after, b) permanence rewards are functions of the transition, c) rewards are given also at the transitions, d) rewards are paid at the end of the period, e) interest rate is variable . The preceding formula becomes ^

"

^=1

j=\

(2.45)

Remark 2.1 The following formulas can consider all the possible immediate discrete time Markov reward processes (DTMRWP) in homogeneous and nonhomogeneous cases. yin) ^ yin-X) ^ ^(^^^

"" ( '" ^ • S P^''^ (^ ~ ^^P^)y^k in) + Y, Pkj (Ykj {n) + swpe • y/^j (n)) ^=1

V

y=i

(2.46)

Reward Processes

261

(2.47) m

Y^Pkj^^ "^ ^){ykji.s^s + n) + swpe' i//kj(s,s + n))), y=i

where: - the variable swpe will be equal to 0 if the permanence rewards do not depend on the next transition and equal to 1 in the other case, - the discount factors will be equal 1 in the not discounted case, - Yij^{ri),Y^k{s,s + n) will be equal 0 if there are no transition rewards (impulse rewards). So, by giving the right values to all the variables in relations (2.46) and (2.47) it is possible to get the evolution equation of any homogeneous or nonhomogeneous MRWP. 2.2.2 Due Cases In the discounted case the difference between the due and immediate cases is of great relevance. But, in our opinion, it is enough to present some formulas for the due case because of their resemblance to the immediate case. 2.2.2.1 First Model a) rewards are fixed in time, b) rewards are given only for permanence in the state, c) rewards are paid at the beginning of the period, d) the interest rate r is fixed. In this case Vl^"^ represents the RWMPV given that at time 0 the system was in the state / and the rewards were paid at the beginning of the period. Under these hypotheses, for the homogeneous case, the following formulas are obtained: V,''^ = ¥i. (2.48) m

m

n''=¥i + (^+rrT,p,'/^,-vr'+ o+^r'lA^v.,

(2.49)

m

J^w^J/(''-»+ (1 + rr^'XAt'V*,

(2.50)

k=]

and in the non-homogeneous case: V<"\s) = J^<''-"(.)+ (1 + rr^''Zpt'\s)v^,

.

(2.51)

^=1

Relations (2.50) and (2.51) give the RWMPV in the simplest due cases (fixed interest rate and fixed rewards). Also in these cases the recursive nature of the formulas is preserved.

262

Chapter 7

2.2.2.2 Second Model Here, basic assumptions are: a) rewards are variable in time, b) rewards are given only for permanence in the state, c) rewards are paid at the beginning of the period, d) the interest rates are time dependent. For the homogeneous case, related formulas are: V^''=¥i{^) .

(2.52)

m

V^'^^

(2.53)

J^("+V(1)XA^V.(1), m

y(n) ^ ^(„-,) ^ ^(„ _ l)^^^<>-0^^^(„ _i).

(2.54)

k=\

For the non-homogeneous case, we get in general: V<"\s) = Vr'Ks) + v(s,s + n- l)Y^pir\s)v^,(s

+ n-l).

(2.55)

k=\

The discount factors and indices of the reward are given by the moment of payment and so one period before. Also in this case RWMPV can be obtained recursively. 2.2.2.3 Third Model Now, we introduce the / rewards. We make the following assumptions: a) rewards are variable in time, b) rewards are given for permanence in the state and at a given transition, c) rewards are paid at the beginning of the period, d) the interest rate r is fixed. Under these hypotheses, the general recursive formulas for homogeneous and non-homogeneous cases are the following: m

f

V^'"' = Vr''+y"-'T.Pir''\ ^=1

m

¥k{n-\)

7=1

^

\

+ vY^p,y,{n)

,

m / m \ y"~' Z Pi^"~'^ (^) \v^k(s + n-l) + vj] Pkj {s + n)y,j {s + n)\ k=\

\

j=\

(2.56)

J

(2 57)

J

In the RWMPV the lump sums given or taken in the instant of the state changes are present. That is why the recursivity holds. 2.2.2.4 Fourth Model

Reward Processes

263

Let us now consider the permanence rewards function of the next state that will be visited. For this case the following assumptions are made: a) rewards are variable in time, b) permanence rewards are functions of the transition, c) rewards are given also at the transitions, d) rewards are paid at the beginning of the period, e) the interest rate is variable. The following homogeneous and non-homogeneous formulas give the related evolution equations: m

m

m

yM = y^Jn(" y=l

k=\

.

*°'

m

m

k==\

7=1

m

-1) +K«)Z A ^ ' S A;n;(«) . (2-58) k=\

7=1

'"'

(2.59)

2.2.25 Fifth Model The next case, which has sense only in a non-homogeneous environment, takes into account the non-homogeneity in interest rate and in rewards. In this case the following hypotheses hold: a) rewards are variable because of the starting time and the time spent after, b) rewards are given also at the transitions, d) rewards are paid at the beginning of the period, e) the interest rate is variable because of the starting time and the time spent after. We get: yin) ^ yi.-n

m m + ^ ( ^ ^ ^ + „ _ l)Y^pi"-'\s)Y,P,j(s

+ n - W,j(s,S

+ « - 1)

(2.60) m

m

+ v(s,s + n)J^pi"-'\s)Y,Pkj(s

+ n)ykj{s,s + n).

7=1

k=\

Remark 2.2 Using notation given in Remark 2.1, the general homogeneous and non-homogeneous formulas for the due cases are the following:

"' '-' v{n -1) {\-swpe)^p]r\An-\) k=\

.

+

.

swpeY,p\r'Y.Pv¥.M-\) k=\

7=1

.

(2.61)

264

Chapter 7

p^W = J^C-') + v(s,s + n)'^pir^Y.Pvnjis,s

+ n) + v{s,s + n-l) , (2.62)

f

m

(1 - swpe)Y,pt\As,s k=\

m

m

\

+ « - ! ) + swpeY^pV^YjPi^J^v^^^^ k=\

+ " " 1)

j=\

2.3 General Algorithms For The DTMRWP. In this part we will briefly show two algorithms useful for the solution respectively of homogeneous and non homogeneous cases. Each of them can be used for all the different homogeneous and non homogeneous cases. This fact means that with a unique algorithm, it is possible to treat all the possible different homogeneous cases and by means of another algorithm all nonhomogeneous cases can be approached. The algorithms will consider all these different cases: 1.1) Immediate 1.2) Due 2.1) Fixed time permanence rewards 2.2) Variable permanence rewards 2.3) Variable non-homogeneous permanence rewards 2.3) Fixed time transition permanence rewards 2.4) Variable transition permanence rewards 2.5) Variable non-homogeneous transition permanence rewards 3.1) Fixed time transition impulse rewards (lump sums) 3.2) Variable transition impulse rewards 3.3) Variable non-homogeneous transition impulse rewards 4.1) No discounting case (interest rater = 0) 4.2) Fixed interest rate r > 0 4.3) Variable interest rate 4.4) Non-homogeneous variable interest rate The algorithms will work for a time horizon of T periods and will be able to tackle any combinations of the cases considered above. The algorithms are constructed following structured programming rules (Dahl and Hoare (1972)) and we will only summarize the steps:

Reward Processes

265

2.3.1 Homogeneous MRWP Input-YQdids the selectors of the examined case, number of states and the number of periods, the permanence and transition rewards, the fixed or variable interest rate, the transition matrix P. Compute - matrix for all kinds of permanence rewards. Compute - matrix for all kinds of transition rewards. Compute - vector of discount factors (in the non-discounted case is the sum vector). Compute - reward matrix. The sum vector is the vector whose elements are all equal to 1. 2.3.2 Non Homogeneous MRWP Input - reads the selectors of the examined case, the number of states and the number of periods, the permanence and transition rewards, the fixed or variable interest rate, the transition matrix V{s). Compute - matrix for all kind of permanence rewards. Compute - matrix for all kind of transition rewards. Compute - matrix of discount factors (in the non discounted case is the 1 matrix). Compute - reward matrix. More details are available upon request to the authors.

2.4. A DTMRWP Applied Example. To give an example of the application of DTMRWP we return to the taxicab example that was presented in Chapter 3. Let us recall the data of the example: The taxicab company of a city V subdivides the city into three sectors V^, V2 and F3 and any taxicab picks up a passenger in any sector and drops her or him off in any sector. The homogeneous Markov chain is the same given in Chapter 3. In the example we suppose that 1) the time period is 15 minutes, 2) the taxicab is followed for one day, 3) there are 3 shifts in 24 hours: the first shift from 6 to 14, the second shift from 14 to 22 and the third shift from 22 to 6, 4) the rewards are supposed to be the same in the two first shifts and more expansive in the night shift.

Chapter 7

266 Taxicab rewards are given by the two following matrices day rewards r 6 8 12" 8 6 9 12 9 7

night rewards 8 10 14] 10 8 11 14 11 9

(2.63)

where element /, j of these two matrices represents the fare to be paid for a trip from the / zone to they zone. As we suppose that there is an overcharge at night, nightly rewards are bigger than the daily ones and so we have only transition rewards. The taxi driver case considered is modelled with a discrete time Markov model, so the travel times are supposed to be constant and randomness is given by the run destination. In general, in a discrete time Markov case the distinction between permanent and impulse is not important because the waiting time in a state before another transition is constant. The only relevance in MC is that in the permanence rewards the premiums or the benefits will be paid at the beginning or at the end of the period and taking into account the state in which the system is. The transition rewards are paid always at the transition moment and because of the transition. The distinction is important in the semi-Markov case because of the waiting time in the states. Results of this example are given in Table 2.1. The table is divided in three columns. Each column represents a working shift. The rows of each column represent the mean earnings that taxicabs get after the related time period. For example the row 39 represents the mean earnings that the taxicabs get after 39 periods. The second element of the row 39 represents the mean earnings got starting at time 0 in the zone 1. The third element of the row 39 is the earnings value that the taxicab gets starting from the zone 2 and the fourth the mean earnings starting from the zone 3. rotal rewards earned in a working day II Shift

I Shift 1 time per. 1 2 3 4 5 6 7 8 9 10

1 7.40 14.70 22.06 29.45 36.84 44.23 51.63 59.03 66.42 73.82

2 6.90 14.18 21.54 28.93 36.32 43.71 51.11 58.50 65.90 73.30

time per. 3 33 8.20 34 15.91 35 23.42 36 30.86 37 38.27 38 45.67 39 53.07 40 60.47 41 67.87 42 75.26

1 243.95 251.34 258.74 266.14 273.53 280.93 288.33 295.73 303.12 310.52

2 243.43 250.82 258.22 265.62 273.01 280.41 287.81 295.20 302.60 310.00

III Shift 2 1 time per. 3 482.65 482.13 65 245.39 492.04 491.52 66 252.79 501.44 500.92 260.19 67 510.84 510.32 68 267.58 520.23 519.71 69 274.98 529.63 529.11 70 282.38 539.03 538.51 289.77 71 72 548.43 547.90 297.17 557.82 557.30 73 304.57 567.22 566.70 311.96 74

3 484.09 493.49 502.89 512.28 521.68 531.08 540.47 549.87 559.27 568.66

Reward Processes

1 12^^ 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

81.22 88.61 96.01 103.41 110.80 118.20 125.60 132.99 140.39 147.79 155.18 162.58 169.98 177.38 184.77 192.17 199.57 206.96 214.36 221.76 229.15 236.55

80.70 88.09 95.49 102.89 110.28 117.68 125.08 132.47 139.87 147.27 154.66 162.06 169.46 176.85 184.25 191.65 199.05 206.44 213.84 221.24 228.63 236.03

267 82.66 90.06 97.45 104.85 112.25 119.65 127.04 134.44 141.84 149.23 156.63 164.03 171.42 178.82 186.22 193.61 201.01 208.41 215.80 223.20 230.60 238.00

43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64

317.92 325.31 332.71 340.11 347.50 354.90 362.30 369.69 377.09 384.49 391.88 399.28 406.68 414.08 421.47 428.87 436.27 443.66 451.06 458.46 465.85 473.25

317.40 324.79 332.19 339.59 346.98 354.38 361.78 369.17 376.57 383.97 391.36 398.76 406.16 413.55 420.95 428.35 435.75 443.14 450.54 457.94 465.33 472.73

319.36 326.76 334.15 341.55 348.95 356.35 363.74 371.14 378.54 385.93 393.33 400.73 408.12 415.52 422.92 430.31 437.71 445.11 452.50 459.90 467.30 474.70

75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96

576.62 586.01 595.41 604.81 614.20 623.60 633.00 642.39 651.79 661.19 670.58 679.98 689.38 698.78 708.17 717.57 726.97 736.36 745.76 755.16 764.55 773.95

576.10 585.49 594.89 604.29 613.68 623.08 632.48 641.87 651.27 660.67 670.06 679.46 688.86 698.25 707.65 717.05 726.45 735.84 745.24 754.64 764.03 773.43

578.06 587.46 596.85 606.25 615.65 625.05 634.44 643.84 653.24 662.63 672.03 681.43 690.82 700.22 709.62 719.01 728.41 737.81 747.20 756.60 766.00 775.401

Tab le2.1: MTRo fHM]RWPt axicat> example. We also treat this example for the non-homogeneous case. The non-homogeneous Markov chain was already given in the non-homogeneous Markov chain example in section 3.6 of Chapter 6. Table 2.2 gives non-homogeneous Markov reward results Total rewards earned in a working day times 0-1 0-10 0-20 0-30 0-40 0-50 0-60 0-70 0-80 0-90 0-95 0-96 10-11 10-20 10-30 10-40 10-50 10-60 10-70 10-80 10-90 10-95 10-96 20-21

1 8.60 86.38 172.71 257.64 343.70 430.16 515.37 615.02 721.49 827.24 879.65 890.17 7.86 79.56 158.05 237.63 317.58 396.34 489.53 589.49 688.75 737.94 747.82 7.36

2 1 times 1 1 2 3 1 times 1 3 8.00 9.36 20-40 148.48 147.65 151.02 50-51 85.95 87.15 20-50 224.07 223.24 226.62 50-60 172.28 173.48 20-60 298.07 297.24 300.62 50-70 257.21 258.41 20-70 386.66 385.83 389.21 50-80 343.28 344.47 20-80 482.26 481.43 484.81 50-90 429.74 430.93 20-90 576.94 576.11 579.49 50-95 514.95 516.14 20-95 623.71 622.88 626.25 50-96 614.60 615.79 20-96 633.11 632.28 635.66 60-61 6.90 721.07 722.26 30-31 7.40 8.20 60-70 826.81 828.01 30-40 74.31 73.69 76.38 60-80 879.22 880.42 30-50 149.24 148.61 151.30 60-90 889.74 890.94 30-60 222.95 222.33 225.01 60-95 7.44 8.71 30-70 311.10 310.47 313.16 60-96 79.18 80.77 30-80 406.03 405.40 408.09 70-71 157.67 159.26 30-90 500.25 499.63 502.32 70-80 237.25 238.84 30-95 546.92 546.30 548.99 70-90 317.21 318.80 30-96 556.30 555.67 558.36 70-95 395.96 397.55 40-41 7.59 8.92 70-96 8.10 489.15 490.74 4 0 - 5 0 81.77 81.32 82.80 80-81 589.11 590.70 4 0 - 6 0 162.48 162.03 163.52 80-90 688.37 689.96 40-70 257.64 257.20 258.68 80-95 737.57 739.16 4 0 - 8 0 359.61 359.16 360.64 80-96 747.45 749.04 4 0 - 9 0 460.85 460.40 461.88 90-91 6.84 511.02 510.57 512.05 90-95 8.51 40-95

1 7.42 74.44 163.82 260.04 355.48 402.74 412.23 7.40 88.20 183.52 278.06 324.85 334.25 10.34 103.84 207.06 258.20 268.47 9.54 96.21 144.11 153.72 9.40 46.81

2 7.05 74.00 163.38 259.59 355.04 402.29 411.79 6.90 87.57 182.89 277.44 324.23 333.63 9.74 103.34 206.56 257.70 267.97 9.24 95.96 143.86 153.47 8.87 46.11

3 8.42 76.40 165.78 261.99 357.44 404.69 414.18 8.32 90.77 186.09 280.64 327.42 336.82 11.13 104.74 207.96 259.10 269.37 10.43 97.75 145.65 155.27 10.53 49.41 1

268

Chapter 7

I 20-30 I 73.371

72.55|

75.92| 4 0 - 9 6 | 521.09| 520.64| 522.12| 90 - 96 | 56.23 | 55.53 | 58.91 |

Table 2.2: MTR of NHMRWP earned between times s-t In this last example it is not possible to report all the results. The table gives three columns of results. Each row of these columns is composed of 4 elements. The starting and arriving time is reported in the first element. The second, the third and the fourth data report the mean total reward earned starting respectively from the first, the second or the third zone during the two periods given in the first element.

3 SEMI-MARKOV REWARD PROCESSES. In this part, we introduce both homogeneous and non-homogeneous continuous time semi-Markov reward processes (CTSMRWP). Our approach will be different from the one given in Howard (1971, vol. 2) and will follow the way in which we introduce discrete time Markov reward processes. We will go from the simplest case to the most general case, taking all the possible aspects into consideration. Working in continuous time implies that immediate or due cases are meaningless but of course the classification of processes with or without discounting continues to hold. In CTSMRWP too, rewards can be related to the permanence in a state or can be linked to a state transition and moreover, rewards can be fixed in time or can be variable. Permanence rewards, as in the Markov case, can also be related to the next future transition or only to the belonging state. Clearly it is also possible to consider a fixed interest rate or variable interest rate and in the non-homogeneous case, the rate of interest and the rewards can be non-homogeneous in time. In other words all the cases that we outlined in the previous paragraph can be considered in the CTSMRWP environment. As we already said, the difference between immediate or due cases in a continuous environment loses significance, but this distinction should be made in discrete time SMRWP.

3.1. Undiscounted CTSMRWP. As usual now, let us consider a system S with {l,...,m} as states space and an SMP continuous in time of kernel Q and we attach to the evolution of 5* a reward structure. Here too, as in the preceding section 2, we will present some simple basic models.

Reward Processes

269

3.1.1 First Model Here, we assume that: a) rewards are a constant time function, b) rewards are given only for permanence in the state. Vi (t) represents the mean total rewards, MTR, paid or received in a time /, given that at time 0 the system was in state /. Under our assumptions, the following formulas hold respectively for the homogeneous and non-homogeneous cases: m

m

p^(o = (i-//,(0)^i^,+i^.Z[a.w«9^^+Zra^^ V,{s,t) =

{\-H,is,t))-{t-s)ii^,+y^,Y,[QAs,3)-{&-s)d3 *=' '

(3.1) (3.2)

m

For the explanation of (3.1) and (3.2) the following observations are useful: • the terms {\-Hi{t))ty/i and {\-Hi{s,t))'{t -s)y/i%\\Q the amount of money obtained remaining in the state I In fact ty/i and {t - s)i//i represent the flow of money paid respectively, in a time / for the homogeneous case and from the time s up to the time t for the non-homogeneous case. These sums are paid in the case of no transition. There i//i is the constant reward function related to the state /. • the terms Y \Q,j,{&)y/,3d& and Y \ Qk{s,&)y/r{3-s)d&

whereQ,(i9) and

Q/^{s,&) are the derivatives of the 0^(i9)and Q^(^,i9), supposed to exist to simplify our presentation. The elements Q,^(&)d3 and Q,^(s,&)d& are the differential of the Q^ (i9) and Q^ (s, &). They approximate the probabilities that once the system arrived respectively at time 0 or at time s in the state /, to go into the state k just at time &. They could be seen, in a non-formal way, as the instantaneous probabilities to change the state /just at time i9 . In this light, these two terms represent the sum of all the money amounts that were received in state / up to the first transition towards state k. m

• the terms YjlQik^^^kit

m

- &)d3 and Yj\Qikis^Wk{&J)dS

represent the

sum of all the rewards that were paid after the state change at time 3 . This time the evolution equation process is not recursive and to get a solution it is necessary to solve the Volterra integral equations of the second type (3.1) in the homogeneous case and (3.2) in the non-homogeneous case.

270

Chapter 7

Their resolution is not a simple task and we think that it is why the SM models are still not widely used for solving real life problems. But now the fact that it is possible to solve these problems numerically, Corradi et al (2004) and Janssen Manca (2001a), encourages this type of models. 3.1.2 Second Model The next step is to introduce the variable rewards under the following hypotheses: a) rewards are variable in time, b) rewards are given only for permanence in the state. Evolution equations become now: k=\

(3.3)

k=\ Q

m

V,{s,t) = {\-H,{s,t))

[ii^MdT + X

la,(s,S)l>/^XT)dTd&

k=\ m

+

(3.4)

Y^\ak{s,9)V,{3,t)d3. k=\

The meaning of the parts of (3.3) and (3.4) is the same as those of (3.1) and (3.2), the only difference being the fact that this time the total rewards received time after time should be computed by means of an integral because the {//^(t) (i=l,,..,m), are no longer constant functions of time, but may be any continuous function. This fact complicates further the analytical solution equation. 3.1.3 Third Model Let us assume now that the rewards are paid as lump sums at transition times with as assumptions: a) rewards are variable in time, b) rewards are given for permanence in the state and at a given transition. Then, the system evolution equations take the following form: VXt) - (1 - / / , ( 0 ) i>F,(T)dT + X *=' m

k=\

lQ,(&)^l/X,(T)dTd& (3.5)

Reward Processes

271

*=' '

'

(3.6)

The first two parts that compose the evolution equation (3.5) and (3.6) are the same as those for (3.3) and (3.4).The meaning of the third part:

Z\Q'>'('Mn(^,0 + rd^))dS k=l

(3.8)

'

is the following: as we already said, Qij,{&)d& and Q^j^{s,&)d3 approximate the probabilities to change the state /just at time 3 . Moreover, V,^{t-&) and,F^(i9,/) as before, represent the MTR received by the system respectively in a time / -19 or from the time s up to the time / starting from state k in which the system just enters at time & . Yij^ (i9) represents the reward paid or received at transition time 9 . We thus see that relations (3.7) and (3.8) give the algebraic sum of the money amount received or paid at the transition moment and the sum of all MTR received after the transition in the state k at time & . 3.1.4 Fourth Model This model is related to permanence rewards depending on the next state that is visited. With the following assumptions: a) rewards are variable in time, b) permanence rewards are function of the next transition, c) rewards are given also at the transitions. Then, the system evolution equations take the following form:

(3.9)

V^(SJ) = {1- //, (5,0 ) X (Pi, (S,t)\ y/i, (T)dT (3.10) W

n

"^

fit

272

Chapter 7

In this last expression, the third part is unchanged with respect to relations (3.5) and (3.6). The two first two parts of the evolution equations need some explaining. The following terms {\-H,{t))Y^cp,,{t)U,,{T)dT, h=\

(3.11)

^

{\-H.Xs,t))Y,(pAsMWi>,<
(3.12)

represent rewards received when the system did not change state up to time t, After time / the system can go to anyone of the other states and the following probabilities (Pijit) =P\Jn - j^^n > ^ I Jn-\ = U"^^ transitions up to the time / ] and (Pij(s,t) =P[y„+i = j.Tn^x > t\Jn- /,no transitions from the time s up to the time t\ can be expressed as

nin-^^.

(3..3)

The second parts Y.{QA&)l¥ii.(j)drd9,

(3.15)

Z \Qi,{s,&)tx,,,,{s,x)dxd&,

(3.16)

represent the sum of all the money amounts received in state / up to the first change of state towards state A: just at time 9 . The only change regarding equations (3.5) and (3.6) is that now the reward is a function of the next transition. 3.1.5 Fifth Model In the non-homogeneous case is necessary to consider cases for which rewards and/or interest rates are a function of the time of entrances in the successive states. Let us describe one of the most significant models in the non-discounted case under the following assumptions: a) rewards are non-homogeneous, b) rewards are given only for permanence in the state. Clearly, the system evolution equations can be written as:

Reward Processes

273

'=' '

'

(3.17)

+ 2lQAs,Wd^J)d&^ 3.1.6 Sixth Model The hypotheses are the following: a) rewards are non-homogeneous, b) rewards are given for permanence in the state and at a given transition with as system evolution equations: (3.18) ^=1

3.1.7 Seventh Model For our last model in non discounted case, we assume that: a) rewards are non-homogeneous, b) permanence rewards are functions of the next transition, c) rewards are given also at the transitions, with as system evolution equations: ^ (5,0 = ( 1 - //. (^, 0 ) Z "Pih (^' 0 r ¥ih {S, T)dT (3.19)

Remark 3.1 We will not describe all possible models for CTSMRWP, for example the cases with Yij^ and also the permanence y/ij, independent of the time. Indeed, the reader, if necessary, could easily construct, starting from the above formulas, the related evolution equations.

3,2. Discounted CTSMRWP. In this part we will make explicit the case of CTSMRWP with discounting and give the formulas in the same cases that we gave for the undiscounted case, but taking into account of both fixed and variable interest rate intensities.

274

Chapter 7

As for Markov reward models, any time that a reward is paid it will be discounted at time 0 in the homogeneous case or at time s in the nonhomogeneous case. 3.2,1 First Model For our first model, we assume that: a) rewards are fixed in time, b) rewards are given only for permanence in the state, c) the intensity of the interest rate S is fixed. In this case V^t) and Vi(s,t) represent the mean present value (RWMPV) of all the rewards paid or received respectively in a time / or from the time s up to the time t, given that at time 0 the system was in the state / or at time s the system arrived in the state /. Under these hypotheses the following formulas are obtained: 1 _

-dt

m

1

-di^

(3.20)

V,(s,t) = (1 - H,(s,t))

^, + X l&(s,^)

^

M^ (3.21)

+ The meaning of the parts of the evolution equations (3.20) and (3.21) is the same asthatof(3.1)and(3.2). To see the influence of the introduction of discounting, let us point out that the first parts 1 - e'^' 1 - e~^^'-'^ {\-HM)—g-¥i and {\-H,{sJ)) y/, (3.22) give the present value of the amount of money obtained by the system being at time 0 or at time s in the state / without any change of state up to the time /. y/i represents the constant reward flow (continuous annuity) related to the state /. 1 - e~^^ 1 - e~^^^~'^ Quantities — - — and give present values of a unitary continuous o o annuity and can be seen as coefficients that transform the annuity flow in the capital giving the present value of the annuity. In this way the present value of the continuous annuity of instalment y/^ is computed. Multiplying this present value with the probability to remain in the state /, we obtain the two formulas given in (3.22). The second parts

Reward Processes

275

Z [Q,{&)—f-Wid&

and X \Qik{s.3)

^

y/^dS

(3.23)

represent the present value of all the money amounts received in the state / up to the first change of state. The system changes state at any time 3 with the probabilities Q,^(3)d& or Q,^(s,&)d& and up to this moment the system got the continuous instalment 1//,. The "transformation coefficients" give at any instant i9 the present value of these annuities. In both terms of (3.23), all these present values are summed up instant after instant. The third parts ZiQ.<^(^)e''XO-&)dS k=\ '^

and X \Q>,{s,&)e-'''-'%{&,t)dS

(3.24)

k^\ ^

represent the present values of all the rewards that were paid after the state change at time & . In fact Vj^{t-3) and Vj^{&j) are the present values of rewards paid respectively in a time (/ - 9) or from 9 to / after the transition in the state k at time 9 with the instantaneous probabilities Q^ {9)d9 or Q^ {s, 9)d9 . We need its value respectively at time 0 or at time s computed by means of the discount factors e~^^ or e~^^^~'^. The formulas given in (3.24) sum up all these present values instant after instant. 3.2.2 Second Model This model is based on the following assumptions: a) rewards are fixed in time, b) rewards are given only for permanence in the state, c) the intensity function 5{T) of the interest rate is time dependent. In this case, the evolution equations of the CTSMRWP are given by the following formulas: V,{t) = (1 -//.(O)^^, ie^''''''de^t[Q,{3W, ^e-^'^'^'-dedS ,

V,{s,t) =

"

(3-25)

{\-HXs,t))y/,[e'^''''''"de (3.26)

The difference between formulas (3.20) and (3.21) and formulas (3.25) and (3.26) is only given by the variable interest rate which implies greater difficulty in the computation of the results.

276

Chapter 7

The meaning of the three parts is the same as in the first model. 3.2.3 Third Model Now we will introduce variable permanence rewards with the following assumptions: a) rewards are variable in time, b) rewards are given only for permanence in the state, c) the intensity of the interest rate d is fixed. Under these assumptions, homogeneous and non-homogeneous evolution equations become

v.,{t) = {\-H.,{t))\wi{e)e-''de 0 m

^

(3.27)

^

m f

+ Z J0* (-9) V' {0)e-''ded3 + X j 4 {&)e-'X (^ - &)dS, k=\ 0

0

k=\ Q t

(3.28) m

f

>9

m

^

K—\

_y

5

K—\

g

The meaning of the three parts of relations (3.27) and (3.28) is always the same. The only difference is that now the rewards and so the instalments are variable and to get the present values of the annuities that appear in the formula, it is necessary to sum up the values discounted respectively at time 0 or at time s instant after instant. This is done by means of the integrals generating the difference between the formulas (3.27) and (3.28) and the formulas (3.20) and (3.21). 3.2.4 Fourth Model The next case to be presented considers variable interest rates and rewards. The assumptions of this case are the following: a) rewards are variable in time, b) rewards are given only for permanence in the state, c) the intensity function S(T) of the interest rate is time dependent. Under these assumptions, homogeneous and non-homogeneous evolution equations become

Reward Processes

k=\ '

277

k=\

'

Here too, the meaning of the evolution equations is the same as in the previous ones. It can be seen as an enhanced mixture of (3.25) and (3.27) in the homogeneous case and of (3.26) and (3.28) in the non-homogeneous one. Let us just observe that the analytical resolution of the integral equations of both systems becomes more and more difficult. 3.2.5 Fifth Model In this model we introduce transition rewards with rewards of permanence and transition, that change in the time under the following assumptions: a) rewards are variable in time, b) rewards are given for permanence in the state and at a given transition, c) the intensity of the interest rate 5 is fixed. For this model, homogeneous and non-homogeneous evolution equations become k=\ 0

(3.31)

k=\ 0

(3.32)

9

"^=1 S

S

S

Let us just point out here that the transition reward is given at time & and has to be discounted as for V,,(t - &) and V^^Sj). 3.2.6 Sixth Model The assumptions are: a) rewards are variable in time, b) rewards are given for permanence in the state and at a given transition,

278

Chapter 7

c) the intensity fiinction 5(j) of the interest rate is time dependent. Under these assumptions, we get for the evolution equations:

V,{t) = Y^[Q.,{&)e'^'''""{V,{t-&) + y,,{&))d3 *='

(3.33) k=\ p9

m

(3.34)

3.2.7 Seventh Model The case of permanence rewards depending on the next state that will be visited together with fixed and variable interest rate intensities is now presented with as assumptions: a) rewards are variable in time, b) permanence rewards are functions of the next transition, c) rewards are given also at the transitions, d) the intensity of the interest rate S is fixed. The evolution equations are given by: m

f

m

(

^

m ^

k=\ 0 m

t

(3.36)

Let us just say here that the corresponding non-discounted case formulas hold also for the discounted environment and that the meanings of the three parts that compose the formulas are always the same. 3.2.8 Eighth Model This model generalises the preceding by taking variable interest rate; so, its basic assumptions are: a) rewards are variable in time,

Reward Processes

279

b) permanence rewards are flinctions of the next transition, c) rewards are given also at the transitions, d) the intensity function cJ(r) of the interest is time dependent Under these hypotheses the evolution equations are: (3.37)

^(^,0 = E f a(s,&)e'^'''''" {V,(&,t} + h,{9))d9 + {\-H:{s,t)) *" '

(3.38)

.|:^,,(.,0/;^. We"^''^''V^ + X / a ( . , 5 ) f i^,,(^)e-^'>^V^./5. There is no comment to explain the last relation as it is similar to the others. 3.2.9 Ninth Model Now, we will consider non-homogeneous interest rate intensity and nonhomogeneous rewards given under the following assumptions: a) rewards are non-homogeneous, b) permanence rewards are functions of the next transition, c) rewards are given also at the transitions, d) the intensity function S(s, r) of the interest rate is time dependent both from time T of getting the money flows and from time s, the time of discounting. In the related formula the evolution equations are the following: '-' '

(3.39)

Remark 3.2 As in the previous paragraph, we present general formulas that can take account of all particular cases.

k=\ *^

+(1 - swpe) (1 - //, it)) {i^,. i0)e~ ^''''''de + X 1 4 i&) f V^,. (d)e' ^'''""ddd9 +swpeil-H,{t))f^
(3.40)

280

Chapter 7 m

r^

k=\

il-H^{s,t))lv^,is,9)e-^''''''''dd^±l&(s,&)^V^,,(^^^^^^

(3.41)

The only thing to say is that the analytical solution of this kind of evolution equation is in our opinion really a difficult task. Furthermore, the fact that the increasing distribution functions Fy(t) and Fyisj) could be of any type increases the difficulties to get the solution. Also, we think that the problem could be faced only by means of a numerical approach.

4. INSURANCE APPLICATIONS OF CTSMRWP. In this section, we will present some possible real life applications of semiMarkov reward processes. As we know that the solution of evolution equations of continuous time SMP is really difficult, we develop applications of CTSMRWP only from a theoretical point of view. But, let us already say here that the problem of real applications of SMP is numerically possible with results on discrete time semi-Markov processes and the numerical solution of continuous time SMP see Corradi et al (2004) and Janssen Manca(2001). In Haberman & Pitacco (1999), Figure 4.1 illustrates a trajectory of the stochastic process under which hangs a general insurance contract. The proposed model has four states. It is evident that the transition time is naturally random. Moreover, it is clear that Figure 5.1 of Chapter 4 and Figure 4.1 describe both trajectories of the same stochastic type, but here we will assume that a general insurance contract can be considered naturally to evolve in a semi-Markov environment. More precisely, we suppose that trajectories are governed by a semi-Markov kernel Q mostly non-homogeneous. In Figure 4.1, the states of the systems stand on the j^ axis and time on the x axis. Premiums and benefits of the insurance contract are considered besides and clearly, can be seen as rewards.

281

Reward Processes


i

CjjOj)

4-\ 3

1

I

I

J

L

*i

4

y

Y

h

U

s(l)

Vs

Figure 4.1: time evolution of a general insurance contract More precisely in a CTSMRWP environment, we have the following definitions: • P\ ( 0 - V^\ ( 0 is a permanence reward that can be constant or variable in time depending on the insurance contract; • Z?2 ( 0 = Vi ( 0 is a constant or variable permanence reward; ^3 (/)

• d,{Q

(the homogeneous case),

^3 (^3, t^) (the non-homogeneous case), where in the homogenous case, it results t^t^-t^, 'k(t) if <^3(0 = '

b

if

t = t^-ty

Clearly we could also have: k{t) = 0 V/; •^13(^3) = 7^13(0 ^iid ^34(^5) = ^^34(0 are transition rewards. In this light we can say that any insurance contract can be modelled by means of SMRWP of kernel Q to be estimated if we suppose that future depends only on the present. In some cases the homogeneous environment is enough for modelling the insurance phenomenon. In other cases it is necessary to introduce nonhomogeneity that in an insurance problems can be useful to consider the age of

282

Chapter 7

the insured person. Furthermore in more composite cases it is necessary to generalize also the non-homogeneous environment to model the phenomenon, but in general it is possible to affirm that SMRWP environment can naturally model insurance problems. In this first approach, we will consider the first examples reported in Haberman & Pitacco (1999) pages 4-11 to show how to give the related rewards evolution equations, in both homogeneous and non-homogeneous cases. Values of premiums and benefits will have opposite algebraic signs. In these examples we will apply discounted SMRWP because in an insurance environment non-discounting doesn't make sense and furthermore we will suppose that the interest rate intensity 5 is constant as it is usual for insurance activities.

4.1 Two States Examples

Figure 4.2:1 graph of two states example 1 = alive state 2 = dead state Figure 4.2 can be used to depict three different cases of person insurance. i) temporary assurance, ii) endowment assurance, iii) deferred annuity. i) For temporary assurance, in case of death, a constant sum c is assured and a continuous premium at a constant rate p is paid. The policy ends at time T. So we have that:

[p if

'^^^^^"jo if

o
t>T '

^^^"''' ° ^ ^ - ^ '

^^-^^

In all three cases, state 2 is an absorbing state and after time T the insurance contract is extinguished.

Reward Processes

283

Furthermore, in this case

y/^^Q.

y/^{t) can be considered as a constant permanence reward and the evolution equations will follow the system for a time T, The evolution equation is the one with a fixed interest rate, fixed time permanence rewards and a fixed time transition reward that is the first model of section 3.2. Under these hypotheses we can write, using relation (3.20), the following evolution equations: 0
^ V2(t) = 0, yt, in the homogeneous environment and

(4.2) (4.3)

V,(s,t) = (l- H,(s,t))aj::^i//, + ^ Q,2(s,&)aj^i//,d3 ^' 0<s
(4.4)

^2(^,0 = 0. V^.^ (4.5) in the non-homogeneous environment. Fj (t) and V^ (s^ t) represent the present value of the temporary assurance at time t (backward reserve). The non-homogeneous case gives more information because as the relations hold for interval [5,/],^ < / and should be used in life insurance. ii) In the endowment assurance, a sum c is insured in both cases of death and of survival to maturity so we can define the following reward:

[p if \f/^{t) = \c [0 and here too

o
if

t =T

if

t>T

,

y^^=c,

0
(4.6)

y/^^O.

y/^{t) can be considered as a variable permanence reward and the evolution equations will follow the system for a time T, The evolution equation is the one with fixed interest rate, variable time permanence rewards and fixed time transition reward for which we have V,it) =

{\-H,{t))\iixme'''dd 0
+ JQn (^) Jt^i (0)e-''d0d3 + 6

jaA^)e-'y,,d3

(4.7)

284

Chapter 7

for the homogeneous case and V,(s,t) =

{l-H,(s,t))ji/^,(d)e-'^'-^>d0 0<s
t

(4.8)

for the non-homogeneous case. As before, )^(Oand V^{s,t) represent the backward reserve at time t and relations (4.3) and (4.5) hold iii) In the third case of deferred annuity, premiums are paid continuously over [0,7^) when the insured person is in the state 1 and similarly, benefits are paid continuously after 7] until the death of the insured. In this case we have: \p if Q
y/^{t) can be considered as a variable permanence reward and the evolution equations will follow the system for a time co — x. The evolution equations (3.27) and (3.28) for the case of fixed interest rate, variable time permanence rewards and no transition rewards which is the third model of section 3.2, become here: V,{t) = {\-H,(0)

\y/^(0)e-''d0 + JQ2(^) j^i{0)e-''ded& 0

0

(4.10)

0

for the homogeneous case and t

t

V,{s,t) = {\-H,{s,t))\y/,{9)e-'^'-^^d9+ s

$

\Q,,{s,&)\ii/,{e)e-'^'-'^ddd& s

(4.11)

s

for the non-homogeneous one. As in the two preceding examples, F, {t) and V^ {s, t) represent backward reserve at time t that is discounted at time 0 in the homogeneous case and at time s in the non-homogeneous case and here too, results (4.3) and (4.5) hold.. In all these three cases the death state doesn't give any permanence reward. It ends the contract and, in the first two cases, before the natural maturity. Another two states example given in Haberman & Pitacco is the following:

Reward Processes

285

Figure 4.3: II graph of two states example 1 = employed 2 = unemployed. In this case the model can be used to study the annuity benefit in the case of unemploymenX, The death state, in this two states model, is not considered because, as specified in Haberman & Pitacco, the age range covered by such insurance contracts is characterized by low probabilities of death compared to the probabilities of moving from state 1 to state 2 or from state 2 to state 1, or because the financial effects of death may be small compared to that of unemployment. We will suppose that the premiums and benefits are fixed in time, but it is also possible to consider them variable without any difficulty. Under these hypotheses it results that:

\p

if

Q
{b if

0
and y/j{t) = { r^(0 = U : .^rr, ' andi/^,(t)^\^ ^^ _ ^ ' (4.12) 0 if t>T, ^'^' [0 if t>T, where p is the premium paid by the insured, b the benefit that he receives in the unemployment case and T = W -x where H^is the maximum working age and x the insured age. y/^ (t) and ^2 ( 0 could be considered as constant permanence rewards and so, following the first model of section 3.2, the evolution equations (3.20) and (3.21) on [0,r]give: (4.13)

(4.14)

(4.15) V^{s,t) = {\-H^{s,t))aj^y/^

+

\Q2\{s,9)aj^y/^d& (4.16)

+ lQAs,&)e-'^'-%(&,t)d3.

286

Chapter 7

Relations (4.13) and (4,14) represent the mean present value that an insured person has in a time t if it starts at time 0 in the state 1 in the homogeneous case and at time s in the non homogeneous one. Relations (4.15) and (4.16) have the same meaning but this time starting from the state 2. For all the cases in which we have not considered the possibility of virtual transitions, it is clear that means that p.. = 0 and (2// = 0 ^^^ so in the evolution equation, only (2i2 ^^ 221 ^^^ considered.

4.2 Three States Examples

Figure 4.4: (I) graph of three states example Figure 4.4 can be used for the description of two cases: i) a temporary assurance with a rider benefit in the case of accidental death, ii) a lump sum benefit in the case of permanent and total disability. In the first case the three states will have the following meaning: 1 = alive 2 = dead (other causes) 3 = dead (accident) Two different causes of death are considered and lump sums are functions of the cause of death. It results that:

[p ?^i(0 = L

if .^

0
(4.17)

The evolution equations are similar to (4.2) and (4.3). Vi(0 = (1 -H,{t))aT^Y, + X {Su{m•^¥^d1^ *=' 3

•Ef<2u(^^K'>u^^. 'k=2 '

Q
(4.18)

287

Reward Processes

7,(5,0 = ( l - / / , U 0 ) « 7 z ^ t ^ i 0<s
+ Y.

'

k^l

(4.19)

^Qu(s,^)e-'"'-''rudt^, ^

^2(0 = 0, ^3(0 = 0,72(^,0 = 0,^3 (^,0 = 0, V^,/. (4.20) In the other example related to Figure 4.4, a lump sum will be paid in the case of permanent and total disability. The states are: 1 = active 2 = disabled (permanent disability) 3 = dead. The considered rewards are: 0T ^''^''' 0
(4.22)

la2We-''n2d^, 3

yi(^,0 = (i-//i(^,0)«F=^i^i+Xl2u(^'^)^^^^i^^

+ [Qnis.^)^

-SiJ^-s)

Tnd^-

The conditions of (4.20) hold also in this case. Another three states case is illustrated in the following figure:

Figure 4.5: (II) graph three states example 1 = active 2 = disabled (permanent disability) 3 = dead

(4.23)

288

Chapter 7

In this case, an annuity benefit in the case of permanent and total disability is represented. The annuity is payable if the disability happens in the time interval [0,7). The annuity will be paid up to the time co-x, that has the same meaning as (4.9). The rewards are the following: [b if QT, [0 if t>co-x. The time between 0 and co — x should be considered. For this reason the used formulas are the ones with variable permanence rewards and without transition rewards. Related formulas are: V,{t) =

{\-H,{t))\y/,{e)e-''d9 0

(4.25)

Z Je.* (-9) Jf^i {0)e-''ded& H- J Q , {,9)e-'% {t - &)d3, )t=2o ^=2 0

0

0

/ s 3

t

&

(4.26)

t

+t,\Q,ds,&)\¥me-'''-''ded&+\Q,,{s,9)e-'''-'%{3,t)d9, ^-2 s

s

s

t

t

S

V,{t) = {\- H,(0) \y/,{6)6-''dd + j43{&) 1^2{e)e-''ddd&, 0

0

V,{s,t) ^{\-H^{s,t))\y^,{e)e-'^'-^^de+ s

V,(t)^0,V,(s,t)

(4.27)

0

\Q,,{s,9)\y/,{9)e-'^o-'^d9d9, s

= 0.

(4.28)

s

(4.29)

V^(t) and Vi(Syt) give the mean present value of all premiums and benefits that were paid or received from the time 0 up to the time / in the homogeneous case, and from 6' to / in the non-homogeneous one. Let us remark that from state 1, two transitions are possible so in the second part of (4.25) and (4.26) are considered Q2 ^^^ Q3. Instead, from relations (4.29), the third part of (4.25) and (4.26) considers only ^ 2 ^2(0 and ^2(^,0 giv^ the mean present value of the benefits that were received by the insured in a time / in the homogeneous case and from s to t in the nonhomogeneous one. The next model is a generalization of the model shown in Figure 4.3.

Reward Processes

289

In this case the death state is adjoined. The related graph is depicted in the following figure:

Figure 4.6: (III) graph of three states model 1 = employed 2 = unemployed 3 = dead. The rewards are given in (4.12). The related evolution equations will be the following: {A30) k=2 "^

3

V2(t)=

lQnWe-'%(t-7})dt} (4.32)

1.3

^2(^,0 =

[Qn{s,^)e-''^-'\{At)di} 1.3

H^-

(4.31)

(4.33)

H^{s,t))aJ^y/^^•Y,[Q2k{^)clJz^¥2d^' k^2

Let us remark that the virtual transitions are not allowed; this is the reason why the third part of the formulas excludes the Qj^ .

Chapter 7

290

The next model in the book by Haberman & Pitacco book is illustrated by means of the graph given in Figure 4.7 and is called the annuity benefit in the case of total disability. It looks like our case study in social insurance presented in Chapter 3, section 7. The permanent character of disability is not required. Hence, we must consider the possibility of recovery. This fact means that an evaluation (perhaps a doctor's visit) is required to go from state 1 to state 2, and another evaluation to go from state 2 to state 1. After an evaluation, a change may occur, or if the doctor so decides, it may not. From the point of view of the model this means that the virtual transitions are allowed. The graph of this model becomes the following:

Figure 4.7: example of graph with virtual transitions 1 = active 2 = disabled 3 = dead. There are no transition rewards and the permanence rewards can be considered the same as defined in (4.12). The related evolution equations are given by: (4.34) A:=l

Reward Processes

291

3

(4.35)

k=i ^

(4.36)

+(1 - //j (0)a^Wi + Z 14* {-m^WidA k=\ ^ 3

(4.37)

The fact that virtual transitions are allowed in this model implies that G i , 222^0.

4.3. Four States Examples Now the two models with four states are to be given. The first is a multiple decrement model. The example could be, as described in the Haberman & Pitacco, a simplified pension scheme in which are considered active, retired, dead and withdrawn states.

Figure 4.8: simple pension scheme graph The model supposes that: a) retirement can always be taken, b) withdrawal generates no expenses to the fund, c) there is no impact if death occurs after retirement or withdrawal.

Chapter 7

292

Under these hypotheses, the rewards will be given for the premiums that the active will pay and for pension that the retired will receive. Rewards correspond to the ones given in (4.24). The evolution equations will take the form: V,(t) =

{l-H,(t))jy^,(d)e-''d0 0

4

t

k=2o

1}

(4.38)

t

0

0 t

V, (s,t) = {l- H, (s, t)) \iir, {6)6-'''-''dd s

(4.39)

+ S \Qu{s,i})\Yme-'''-'^dddi}+ ^-^ s

s

\Qnis,me-''^-'%{At)di}, s

t

V^{t) =

{\-H^{t))\y/^{e)e-''dd,

(4.40)

0 t

V^{s,t) = (1 -

H^{s,t))\\lf^{d)e-'''-''de,

(4.41)

(4.42) y, (0 = 0, V, (t) = 0, V, (s, t) = 0, V, (s,t) = 0. The last example is the case of a reversionary annuity (widow'spension). Two lives are considered, (x) and (y) and we suppose that (x) and (y) cannot die simultaneously. Figure 4.9 depicts this case. The related states are given, as usual, under the figure.

Figure 4.9: graph of reversionary annuity 1 = (x) alive, (y) alive 2 = (x) dead, (y) alive 3 = (x) alive, (y) dead 4 = (x) dead, (y) dead

Reward Processes

293

The policy gives a continuous life annuity to (y) in the case of the death of (x). Premiums are assumed to be paid continuously over [0, 7), while both (x) and (y) are alive. The rewards will be the following: {p if 0T [0 if t>co-y y being the (y) age. Under these hypotheses, the evolution equations take the form: /

V,(t)^{l-H,(0)

3

/

i9

1?^,(0)6-''d0 + X JQu(&) j?^,(d)e-''d0d&, A:=2 0

0

V,(s,t) = il-H,(s,t))

\y/me-'''-''de s

3

'

(4.44)

0

(4.45)

S

t

V^{t) = {\-H,(0)

t

3

\y/,{9)e-">d9 + JQ,,(&)\y/,{e)e-''ded9, 0

0

V,{s,t) ^{\-H,{s,t))\^^{e)e-'^'"->de+ s

(4.46)

0

\Q,,{s,9)\y/,{e)e-'''-'^ded&. s

(4.47)

s

The last two examples concern the pension scheme environment. They are really simplified models of the most general problem of pension scheme management.

REFERENCES

Andersen E. S., (1967). An algebraic treatment of fluctuations of sums of random variables. Proc. Fifth Berkeley Sympos. Math. Statist, and Probability (Berkeley, Calif., 1965/66), Vol. II. Univ. California Press, Berkeley, Calif Anton H., Kolman B. (1978) Applied finite mathematics with calculus. Academic Press. Asmussen S. (2000) Ruin probabilities. Advanced Series on Statistical Science & Applied Probability, 2. World Scientific Publishing Co., Inc., River Edge, NJ. Baker C.T.H. (1977), The Numerical Treatment of Integral Equations. Clarendon Press, New York. Barlow, R. E. (1962). Applications of semi-Markov processes to counter problems. Studies in applied probability and management science pp. 34-62 Stanford Univ. Press, Stanford, Calif. Barlow R. E., Proschan F. (1964) Comparison of replacement policies, and renewal theory implications. Ann. Math. Statist. 35, 577-589. Barlow R E., Proschan F. (1965). Mathematical theory of reliability. The SLAM Series in Applied Mathematics John Wiley & Sons, Inc., New York-LondonSydney. Bellman R., Kalaba R. E., Lockett J. A. (1966), Numerical Inversion of the Laplace Transform: applications to Biology, Economics, Engineering and Physics. Elsevier New York. Billingsley P., (1968). Convergence of probability measures. John Wiley & Sons, Inc., New York-London-Sydney. Black, F. and Scholes, M. (1973). The Pricing of Options and Corporate Liabilities, Journal of Political Economy, 81, 637-659. Blackwell D. (1948). A renewal theorem. Duke Math. J. 15. 145-150. Buhlmann H. (1970). Mathematical methods in risk theory. Springer-Verlag, New York-Berlin. Carravetta M., de Dominicis R., Manca (1989). Using semi-Markov models in studying transportation mass systems with aging: an algorithmic approach. System Science 15 , 7-27. Christofidles N. (1975). Graph theory. An algorithmic approach. Academic Press, New York-London. Chung K. L. (1960). Markov chain with stationary transition probabilities. Springer. Chung, K. L. (2000) A course in probability theory. Third edition. Academic Press, Inc., San Diego. (Jinlar E. (1967). Queues with semi-Markovian arrivals. /. Appl. Probability 4 365-379.

296

References

^inlar E.,(1969) Markov renewal theory. Advances in AppL Probability 1, 123187. ^inlar, E. (1975a). Markov renewal theory: a survey. Management Science, 21, 727-752. ^inlar, E. (1975b). Introduction to stochastic processes. Prentice Hall, New York. CMIR12 (1991) (Continuous Mortality Investigation Report, number 12). The analysis of permanent health insurance data. The Institute of Actuaries and the Faculty of Actuaries. Corradi, G., Janssen, J., Manca, R., (2004). Numerical treatment of homogeneous semi-Markov processes in transient case. Methodology and Computing in Applied Probability. 6, 233-246. Cox, D. R.; Miller, H. D. (1965). The theory of stochastic processes. John Wiley & Sons, Inc., New York. Cox J. C , Ross S. A., Rubinstein M. (1979). Option pricing: a simplified approach. Journal of Financial Economics. 7, 229-263. Cox J. C , Rubinstein M. (1985). Option Market. Prentice Hall, New Jersey. Cramer H. (1955). Collective risk theory: A survey of the theory from the point of view of the theory of stochastic processes. Skandia Insurance Company, Stockholm,. 92. Csenki A. (1994), Dependability for Systems with a Partitioned State Space (Markov and Semi-Markov Theory and Computational Implementation). Springer Verlag, Berlin. Daley, D. J. (1965) On a class of renewal functions. Proc. Cambridge Philos, Soc. 61 519-526. De Dominicis R., Manca R. (1984a), An algorithmic approach to nonhomogeneous semi-Markov processes, Communications in Statistics, Simulation and Computation. 13, 823-838. De Dominicis R., Manca R. (1984b), A computational procedure for the asymptotic analysis of a homogeneous semimarkov process. Statistics & Probability Letters. 2, 249-253. De Dominicis, R, Manca, R (1986) Some new results on the transient behaviour of semi-Markov reward processes. Methods of Operations Research, 53, 387397. De Dominicis R., Manca R., Granata L., (1991). The dynamics of pension funds in a stochastic environment. Scandinavian Actuarial Journal. 118-128. De Medici, G., Janssen, J., Manca, R. (1995). Financial operation evaluation: a semi-Markov approach, Proc. VAFIR Symposium Brussels De Vylder Fl. (1977). Martingales and ruin in a dynamical risk process Scand. Actuar. /., 217-225. De Vylder FL, Haezendonck J. (1980). Explosions in random point processes. Scand. Actuar. 7., 195, 202.

References

297

Di Biase, G., Manca, R. and Janssen, J., (2005) Future Pricing trough Homogeneous semi-Markov Processes, accepted on Applied Stochastic Models in Business and Industry. Doob J. L. (1953). Stochastic processes. Wiley, New York. Evans G., (1993) Practical Numerical Integration. Wiley, London. Ethier S. N., Kurtz T. G. (1986). Markov processes. Characterization and convergence. Wiley Series in Probability and Mathematical Statistics. John Wiley & Sons, Inc., New York. Feller W., (1941). On the integral equation of renewal theory. Annals of Mathematical Statististcs. 12.243-267. Feller W., (1957) An Introduction to Probability Theory and Its Applications^ Volume I Second Edition, Wiley, New York. XV, 461. Feller W. (1966). An introduction to probability theory and its applications. Vol. II. John Wiley & Sons, Inc., New York. Feller W. (1971). An introduction to probability theory and its applications. Vol. II. Second edition John Wiley & Sons, Inc., New York-London-Sydney. Freiberger W., Grenander U. (1971), A Short Course in Computational Probability and Statistics. Springer Verlag, Berlin. Gantmacher F. R.(1959). The theory of matrices. Vols. 1, 2. Translated by K. A. Hirsch Chelsea Publishing Co., New York. Gikhman I. I., Skorohod, A. V. (1980). The theory of stochastic processes. I. Translated from the Russian by Samuel Kotz. Springer-Verlag, Berlin-New York. Haberman, S., Pitacco, E. (1999). Actuarial models for disability insurance. Chapman & Hall. Halmos, P. R. ( 1974) Measure Theory. Springer., New York. Heyman D. P., Sobel M. J. (1982). Stochastic models in operations research. Vol. I Stochastic processes and operating characteristics. McGraw-Hill Series in Quantitative Methods for Management. McGraw-Hill Book Co., New York. Hoem, J.M. (1972). Inhomogeneous semi-Markov processes, select actuarial tables, and duration-dependence in demography, in T.N.E. Greville, Population, Dynamics, Academic Press, 251-296. Howard, R. (1971). Dynamic probabilistic systems. Wiley, New York.. losifescu Manu A.(1972), Non homogeneous semi-Markov processes. Stud. Lere. Mat. 24, 529-533. Isaacson D. L, Madsen R. W. (1976) Markov chains. Theory and Applications. Wiley Series in Probability and Mathematical Statistics. John Wiley & Sons, New York-London-Sydney. Ito K. (1944) Stochastic integral. Proc. Imp. Acad. Tokyo 20, 519-524 Janssen, J. (1966). Application des processus semi-markoviens a un probleme d'invalidite. Bulletin de I'Association Roy ale des Actuaries Beiges, 63, 35-52. Janssen J. (1969a) Dualite en Theorie Collective du Risque, Cahier du C.E.R.O., 11, 15M61

298

References

Janssen J. (1969b) Les Processus (J-X), Cahiers du C.E.R,0., 11, 1969, 181-214 Janssen J. (1970). Sur une Generalisation du Concept de Promenade Aleatoire sur la Droite Reelle, Annates de I'Institut H. Poincare, 6, 249-269 Janssen J. (1977) The Semi-Markov Model in Risk Theory, Advances in Operations Research edited by M. Roubens, North-Holland, Amsterdam, 613-621. Janssen J., (1979). Some Explicit Results for Semi-Markov Models in Risk Theory and in Queueing Theory, Operations Research Verfahren, 33, 217231 Janssen, J. (1983). Semimarkov processes and applications in risk and queueing theories, Quademi delVIstituto di Matematica della Facoltd di Economia e Commercio delVUniversitd di Napoli, n. 35. Janssen J. (ed) (1986). Semi-Markov Models. Theory and Applications. Plenum Press New York. Janssen, J., De Dominicis R. (1984), Finite non-homogeneous semi-Markov processes. Insurance: Mathematics and Economics, 2>y 157-165.. Janssen J., Limnios N. (eds) (1999). Semi-Markov Models and Applications. Kluwer Academic New York. Janssen J., Manca R. (1994). Semi-Markov modelization for the financial management of pension funds. Proceedings of the TV AFIR Colloquium. Orlando Janssen J., Manca R. (1997a), A realistic non-homogeneous stochastic pension funds model on scenario basis. Scandinavian Actuarial!ournal 113-137 Janssen J., Manca R. (1998), Non-homogeneous stochastic pension fund model: an algorithmic approach. ICA Transactions vol. V. Janssen J., Manca R., (2001a). Numerical solution of non homogeneous semiMarkov processes in transient case. Methodology and Computing in Applied Probability. Kluwer, 3, 271-293,. Janssen J., R. Manca, (2001b). Non-homogeneous semi-Markov reward process for the management of health insurance models. Proceedings ASTIN Washing ton. Bl 74 Janssen J., Manca R. (2002). Salary cost evaluation by means of nonhomogeneous semi-Markov processes. Stochastic Models. 18, 7-23. Janssen, J., Manca, R. (2003). Multi-state insurance model description by means of continuous time homogeneous semi-markov reward processes, mimeo. Janssen, J., Manca, R. (2004a). Stochastic annuities and discrete time Markov reward processes a motor car insurance application.. Mimeo. Janssen, J., Manca, R. (2004b). Non homogeneous discrete time Markov reward processes, stochastic annuities and bonus malus age dependent motor car insurance application. Proc.IME 2004. Janssen J., Manca R., De Medici G., (1995) Financial operation evaluation: a semi-Markov approach. Proc. VAFIR Symposium Brussels

References

299

Janssen, J., Manca, R., Di Biase, G. (1998). Non homogeneous Markov and semiMarkov models for pricing derivative products. In Proc. of ASTIN Colloquium, Glasgow, 455-478. Janssen J., Manca R., Di Biase G. (2001), Option pricing a straightforward procedure. Accepted on Chaos & Complexity Letters. Janssen J., Manca R., Volpe di Prignano E. ( 1997). Semi-Markov modelization for salary line evolution. In Proc. of the VIII Applied Stochastic Models and Data Analysis Anacapri (Napoli), Italy, Janssen, J.; Lauro, N. C., Eds; Napoli. Janssen J., Manca R., Volpe dil Prignano E. (1999), Option pricing with semiMarkov volatility. In Semi-Markov models and applications. Janssen J. and Limnios N. eds. Kluwer. Janssen, J., Manca, R., Volpe di Prignano E. (2004). Continuous time non homogeneous semi-Markov reward processes and multi-state insurance application. Proc. IME 2004. Janssen J., Reinhard J.M. (1982). Some Duality Results for a Class of Multivariate Semi-Markov Processes, Journal of Applied Probability, 30,. 9098 Janssen J., Reinhard J.M. (1985). Probabilites de Ruine pour une Classe de Risques SQmi-M^rkoviQns, ASTIN Bulletin 15, 1985, 123-133. Janssen J., Skiadas C.H.(1995). Dynamic Modelization of Human Life Table Data, Applied Stochastic Models and Data Analysis, 11, 35-49. Kellison S. G. (1991), The theory of interest, Irwin. Kingman, J. F. C. (1972) Regenerative phenomena. Wiley Series in Probability and Mathematical Statistics. John Wiley Sons Ltd., London-New YorkSydney. Korolyuk V.S., Swishchuk A. (1995). Semi-Markov random evolution. Kluwer Academic. Kolmogorov A. N.; Fomin S. V. (1961). Measure, Lebesgue integrals, and Hilbert space. Academic Press, New York-London. Levi, E. (1964), Corso diMatematica Finanziaria e Attuariale. Giuffre, Milano. Levy, P. (1954). Processus semi-Markoviens. Proc. of International Congress of Mathematics, Amsterdam. Limnios, N., Oprisan, G. (2001). Semi-Markov Processes and Reliability. Birkhauser, Boston. Limnios, N., Oprisan, G. (2003). An Introduction to Semi-Markov Processes with Application to Reliability. In Handbook of Statistics vol 21, Shanbhag D.N., Rao C.R. eds. Loeve, M ( 1963). Probability theory. Third edition Princeton,New York. Lotka, A. J. (1940) Sur une equation integrale de I'analyse demographique et industrielle. Mitt. Verein. Schweiz. Versick-Math. 40,. 1-16 Lundberg P. (1909). On the theory of reinsurance. In History of Actuarial Science.Wol VII Habermann S., Sibbett T. A. (eds) (1995) 291-244.

300

References

Markowitz, H. M. (1959 ). Portfolio selection: Efficient diversification of investments. Cowles Foundation for Research in Economics at Yale University, Monograph 16 John Wiley & Sons, Inc., New York; Chapman & Hall, Ltd., London. McClean,S.I. (1993). Semi-Markov models for human resource modelling. IMA Journal of Mathematics Applied in Business and Industry. 4, 307-315. McClean,S.I.; Montgomery, E., Ugwuowo, F. (1998). Non-homogeneous Markov and semi-Markov models. Applied Stochastic Models and Data Analysis. 13, 191-198. Meyer, P. A. (1966). Probability and potentials. Blaisdell Publishing Co. Ginn and Co., Waltham, Mass.-Toronto, Ont.-London. Miller, H. D. (1962). Absorption probabilities for sums of random variables defined on a finite Markov chain. Proc. Cambridge Philos. Soc. 58 286-298. Miller, R. G. (1963). Stationary equations in continuous time Markov chains. Trans. Amer. Math. Soc. 109 35-44. Mine H., Osaki S.( 1970). Markovian decision processes, Ehtvitr. Neuts M. F. (1966) The single server queue with Poisson input and semi-Markov service times. /. Appl. Probability 3 202-230. Norberg R. (1993), Identities for present values of life insurance benefits. Scandinavian Actuarial Journal, 100-106. Olivieri A.M., Pitacco E. (1997). Introduzione alia teoria attuariale delle assicurazioni di persone, Quademi deH'UMI, 42, Pitagora Editrice, Bologna. V, 238. Osaki, S. (1985). Stochastic system reliability modelling. Series in Modem Applied Mathematics, 5. World Scientific Publishing Co., Singapore. Pareto V. ( 1989). Statistique et economic mathematique. Oeuvres Completes de Vilfredo Pareto Librairie Droz, Geneva. Parzen E., (1962). Stochastic processes. Holden-Day Series in Probability and Statistics Holden-Day, Inc., San Francisco. Protter, P. (1990). Stochastic integration and differential equations. A new approach. Applications of Mathematics (New York), 21. Springer-Verlag, Berlin. Pyke, R. (1961a). Markow Renewal Processes: Definitions and Preliminary, Ann. Math. Stat., 32, 1231-1242. Pyke, R. (1961b). Markow Renewal Processes with Finitely Many States, Ann. Math. Stat., 32, 1243-1259, Pyke R. (1962) Markov renewal processes of zero order and their application to counter theory. Studies in applied probability and management science 173183 Stanford Univ. Press, Stanford, Calif. Pyke R., Schaufele R (1964) Limit theorems for Markov renewal processes. Ann. Math. Statist. 35 1746-1764.

References

301

Qureshi M.A., Sanders H.W. (1994). Reward model solution methods with impulse and rate rewards: an algorithmic and numerical results,. Performance Evaluation, 20. 413-436. Reinhard J.M., Snoussi M., (2002). The severity of ruin in a discrete semiMarkov risk model. Stochastic Models, 18, 85-107. Ross, S. M.(1996) Stochastic processes. Second edition. Wiley,New York. Ross, S. M.(2000) Introduction to probability models. Seventh edition. Harcourt/Academic Press, Burlington, MA. Royden, H. L.(1963) Real analysis. The Macmillan Co., New York; CollierMacmillan Ltd., London. Sahin I. (1993), A class of non-stationary reward processes with applications to pension accumulation, Scandinavian Actuarial!ournal, 73-95. Silvestrov D. S. (1980). Semi-Markov processes with a discrete state space, Sovetske Radio, Moskow. Silvestrov D. S., Stenberg F. (2004). A pricing process with stochastic volatility controlled by a semi-Markov process. Communications in Statistics, Theory andMethod. 33, 591-608. Smith, W. L., (1954). Asymptotic renewal theorems. Proc. Roy, Soc. Edinburgh, Sect. A, 64. 9-48. Smith, W.L. (1955). Regenerative stochastic processes. Pro, Roy, Soc. London Ser.A.232,6-31. Spitzer F. (1957). The Wiener-Hopf equation whose kernel is a probability density. Duke Math, J, 24, 327-343. Stein C. (1946) A note on cumulative sums. Ann. Math. Statistics 17. 498-499. Svishchuk A.V. (1995). Hedging of options under mean-square criterion and semi-Markov volatility. Ukrainian Mathematical!oumal, 7. 1119-1127. Svishchuk A.V., Burdeinyi, A.G. (1996). Stability of semi-Markov evolution systems and its application in financial mathematics. Ukrainian Mathematical Journal 10. 1386-1401. Tsantas N. (1993),Recent developments in stochastic modeling of nonhomogeneous Markov system in Applied Stochastic Models and Data Analysis. Janssen J. andSkiadas C.eds, World Scientific Pybl.Cy, p.990-996. Tsantas N., Vassiliou P.C.G. (1993). Non-homogeneous Markov system in a stochastic environment. Journal of Applied Probability, 30, 285-301. Vassiliou, P.C.G.; Papadopolou, A. A., (1992). Non-homogeneous semi-Markov systems and mantainability of the state sizes. Journal of Applied Probability, 29, 519-534. Volpe di Prignano, E. (1985). Manuale di Matematica Finanziaria, ESI Napoli. Widder, D. V. (1945). What is the Laplace transform. Amer. Math. Monthly 52. 419-425. Wolthuis H, (1994). Actuarial equivalence. Insurance: Mathematics and Economics. 15, p 163-179.

302

References

Xie M., (1989).A292 On the solution of renewal-type integral equations, Commun. Statist, -Simula., 18, 281-293 . Yntema L., (1962). A markovian treatment of silicosis. Acta III Conferencia Int. De Actuaries y Estadisticos de la Seguridad Social. Madrid

AUTHOR INDEX

Andersen 152 Anton 119 Asmussen 152 Baker 93, 95 Barlow62, 166, 167 Bellman 92 Black 19 Blackwell 54 Buhlmann 234 Christofidies 125 Chung 1, 12,105, 109, 111, 116, 217 ginlar 1, 53, 54, 153, 154, 162, 168 Corradi 95, 269, 280 Cox D. R. 242 CoxJ. C. 16,17 Cramer 152 Csenki 92 Daley 66 De Dominicis 125, 227 De Vylder 229, 234 Dhal 264 Doobl97,235 Evans 93 Feller53, 96, 116,205,209,218, 220,221,222 Fomin 1 Freiberger 96 Gantmacher 127 Gikhman 33, 34 Grenander 96

Haberman 237, 280, 282, 284, 290, 291 Haezendonck 229 Halmos 1,28 Heyman 204, 209 Hoare 264 Howard 247, 268 Isaacson 236 Janssen40, 95, 122, 151, 152, 153, 187, 193, 195, 196, 199,222,225, 227, 269, 280 Kalaba 92 Kellison 249 Kolman 119 Kingman 146 Kolmogorov 1 Levi 251 Lockett 92 Loevel2,28, 31,33, 34 Limnios 151, 153 Lotka 53 Lundberg 152 Madsen 236 Manca 95, 125, 269, 280 Markowitz 20 Meyer 41 Miller 152, 242 Neuts 153 Oprisan 153 Osaki 153 Parzenl05, 115, 116

304 Pitacco 237, 280, 282, 284, 290, 291 Proschan 62 Protter 39, 41,44 Pyke 185, 195,201,224 Qureshi 247 Ross S. A. 17 RossS. M. 1, Royden 12 Rubinstein 16, 17 Sanders 247 Schaufelel95,201 Scholes 19 Skiadas 40 Skorokhod 33, 34 Smith 53, 54 Sobel 204, 209 Spitzer 220 Stein 48 Volpe di Prignano 249 Widder 60 Yntema 122, 123

Author Index

SUBJECT INDEX

Absolute continuity, 12 Absolute monent of order r, 13 Absorbing (recurrent) class of states, 127 Absorbing state, 112 Absorption probability in a class given a state, 115 Adiacency matrix of a graph, 125 Age distribution, 73, 174 Almost sure convergence, 9 Aperiodic Markov chain, 109 Aperiodic renewal process, 48 Aperiodic state of a Markov chain, 108 Associated counting process with the CNHSM kemel,229 Average return time, 157 Backward recurrence time (age), 73, 174 Bernoulli random walk, 203 Binomial distribution, 16 Blackwell theorem, 57 Boole inequality, 5 Borel a-algebra, 6 Borel set, 6 Brownian motion, 43 Cadlag stochastic process, 39 Central limit theorem for functionals of (J,X) processes, 199 Central limit theorem for the two dimensional (J,X) processes, 199

Central limit theorem, 19, 70 Chapman Kolmogorov equation, 238 Characteristic function, 16 Class of states of a Markov chain, 109 Closed martingale, 40 Complement, 2 Complete probability space, 36 Completely non-homogeneous dependent renewal process, 232 Completely non-homogeneous Markov renewal process, 229 Completely non-homogeneous semi-Markov chain, 227 Completely non-homogeneous semi-Markov kernel, 228 Completely non-homogeneous semi-Markov process, 229 Conditional density, 35 Conditional distribution of a random variable, 34 Conditional distribution of sojourn time, 148 Conditional expectation, 25 Conditional independence of n variables, 34 Conditional probability, 24 Convergence sequence of random variable, 9 Convolution product, 47

306 Correlation coefficient, 22 Counting processes associated with a Markov renewal process, 155 Covariance, 22 de Moivre theorem, 20 de Morgan law, 3 Decomposition theorem, 111 Defective positive (J,X) process, 215 Deferred annuity, 284 Delayed Markov renewal process, 179 Delayed renewal process, 87 Density function, 6 Discount factor, 249 Discrete time stochastic process, 38 Distribution function, 6 Dominated convergence theorem, 13 Drift random walk, 211 Endowment assurance, 283 Ergodic Markov chain, 109 Ergodic Markov renewal process, 163 Ergodic property of a semi-Markov process, 171, 172 Erlang process, 60 Espectation, 12 Essential class of states, 109 Essential state of a Markov chain, 109 Event almost sure, 4 Event certain, 2 Event disjoint, 2 Event elementary, 1 Event empty, 2 Event mutually exclusive, 4 Event, 1 Excess distribution, 74, 175 Excess expectation, 75 Explosive associated counting process with the CNHSMP, 229

Subject Index Extended semi-Markov chain, 146, 193 Extended semi-Markov kernel, 146 Filtered probability space, 37 Filtration right continuity property, 37 Filtration, 37 Forward recurrence time (excess), 73, 174 Frequency, 4 Gambler ruin problem, 203 Generating function, 16 Hitting time, 39, 110 Homogeneous Markov chain, 105 Imbedded multiple Markov chain, 231 Impulse reward (transition reward), 247 Independence, 14, 15 Indistinguishable stochastic processes, 38 Inequality of Cauchy Schwarz, 14 Inequality of Holder and Minkowski, 14 Inessential class of states, 109 Inessential state of a Markov chain, 109 Initial distribution of a Markov chain, 106 Integrable random variable, 11 Integral equation of Markov renewal type, 164 Integral equation of renewal type, 53 Integral of a discrete random variable, 10 Intensity random function, 233, 238 Interarrival time, 46 Interest rate intensity, 250 Intersection, 2 Involutive reversed process, 209 Irreducible Markov chain, 109

Subject Index Irreducible Markov renewal process, 163 J-process, 147 Key renewal theorem, 54, 168 Kolmogorov forward and backward differential equations, 239 Ladder epoch, 205 Ladder height, 205 Ladder points, 205 Ladder variables, 204 Laplace Stieltjes transform of matrices, 159 Laplace Stieltjes transform, 64 Laplace transform, 59 Length-biased sampling, 85 Lifetime of a defective positive (J,X) process, 217 Lifetime of a renewal process, 49 Log-normal distribution, 20 M/SM/1 model, 153 Markov chain, 105 Markov renewal equation, 163 Markov renewal matrix, 160 Markov renewal process, 150, 154 Markov renewal time, 154 Martingale property, 40 Maximal class of states, 127 Mean recurrence time of a state, 111 Mean total rewards, 253 Measurable space, 3 Mixed conditional distribution of a random variable, 34 Modification of a stochastic process, 38 Monotone convergence theorem, 13 Multidimensional normal distribution, 22 Negatine exponential distribution, 21 Non-homogeneous Markov chain, 105,232,235 Non-homogeneous semi-Markov kernel, 231

307 Non-homogeneous semi-Markov process, 231 Non-return state, 112 Non-ruin probabihty, 203 Normal distribution, 18 Occupancy probability, 237 Occupation time of a state in the first n transtions, 113 Optional sampling (Doob's) theorem, 42 Oscillating random walk, 210 Partially non-homogeneous Markov additive process, 231 Partially non-homogeneous semiMarkov kernel, 231 Partially non-homogeneous semiMarkov process,231 Periodic renewal process, 48 Periodic state of a Markov chain, 108 Poisson distribution, 17 Polya process family, 234 Porcess of strict ladder indices, 223 Positive (J-X) process, 145 Probability continuity property, 5 Probability measure, 4 Probability space complete, 10 Probability space, 4 Random experiment, 1 Random variable continuous, 8 Random variable discrete, 8 Random variable extended ndimensional real, 9 Random variable n-dimensional real, 9 Random variable real extended value, 9 Random variable real, 8 Random variable, 7 Random walk duality lemma, 210 Random walk, 203 Rate reward (permanence reward), 247

308 Reachability matrix of a graph, 126 Real multidimensional stochastic process, 38 Real stochastic process, 38 Recurrence time (spread), 73 Recurrent renewal process, 48 Recurrent state, 111 Regular conditional probability, 33 Regular Markov matrix, 107 Renewal counting process, 47 Renewal density, 58 Renewal equation,52 Renewal function, 47 Renewal process, 46 Renewal time, 46 Reversed process of order n, 209 Reversionary annuity, 292 Reward mean present value, 256 Reward process classification, 247 Reward processes, 247 Reward, 247 Ruin probabihty, 203 Ruin problem, 204 a-algebra, 3 Sample space denumerable, 5 Sample space finite, 5 Sample space, 1 Semi-Markov chain, 146 Semi-Markov kernel, 146 Semi-Markov process, 155 Semi-Markov random walk, 193 Set indicator, 10 SM/M/1 model, 153 Smoothing property of conditional expectation, 31 Solidarity result, 198 Spread distribution, 75 Standard normal distribution, 18 Stationary Markov renewal process, 181 Stationary renewal process, 88 Stochastic process adapted to a filtration, 38

Subject Index Stochastic process, 38 Stopped stochastic process, 42 Stopping time a-algebra, 39 Stopping time, 37 Strict ascending indices, 222 Strict ascending ladder point, 208 Strictly transient class of states, 127 Strong law of large numbers for (J,X) processes, 199 Strong law of large numbers for functional of (J,X) processes, 198 Strong law of large numbers for functionals of positive (J,X) processes, 198 Strong law of large numbers, 69 Strongly ergodic non-homogeneous Markov chain, 236 Sub-martingale, 40 Successive entrance time into a state, 156 Sum of random number of random variables, 31 Super-martingale, 40 Sure convergence, 9 Temporary assurance, 282 Terminated defective positive (J,X) process, 217 Times continuous Markov process, 237 Total disability annuity benefit, 290 Total number of renewals, 49 Total occupation time of a state, 113 T-process, 150 Trajectory of a stochastic process, 39 Transient renewal process, 48 Transient state,lll Transition graph, 106 Transition matrix, 105 Transition probabilities of a semiMarkov process, 169 Transition probability of a delayed semi-Markov process, 179

Subject Index Transition probability, 106 Truncated normal distribution, 63 Two dimensional normal distribution, 23 Unconditional distribution of sojourn time, 148 Unconditional waiting time distribution, 216 Uniformly integrable family, 41 Union, 2 Uni-reducible Markov chain, 116

309 Variance covariance matrix, 23 Variance, 13 Wald identities, 31 Wald lemma, 67 Weak ascending ladder point, 208 Wiener Hopf integral equation system, 225 X-process, 148