Handbook of Matrices
H. Liitkepohl
Humholdt-Universitat zu Uer/in, Cienua11y
JOHN WILEY & SONS Chichester· New York ·Brisbane· Toronlo Singapore ·
Copyright(!) 1996 by John Wil�y & Sons Ltd,
Baftins l�� ne , Chichester, West Sussex POI9 I U D. Nation al
lntemational
Engla nd
01243 779777
( +44) 1243
779777
e-mail (lor orders and customer service emtuiries): �-s-books(f�wiley.co.uk. Visit our l lome Page
All
on
http://www.wiley.co.uk
or http://www. wiley.com
Rights Reserved.No part
retrieval system,
or
of this hook may he reproduced. stored in a
tmnsmitted, in any fi>nn or hy any means, eledronic,
mechanical, photocopying. recording or otherwis�. except under the tenns of the C opyright.
D�signs and Patent.' Act 1988 or under the tenns of a lic�-nce issued hy the Copyright Licensing Agency , 90 Tottenham Court Road, London, UK WI P 91IE. without the pennis.,ion in "Tiling of the puhlisher.
Other Wiley EJ1tonal Offices
Jolm Wiley & Son.'. Inc.. 605 Third Awnue, New York, NY I 0 158·00 12. liSA
Jacaranda Wiky Ltd,
:n
Park Road. Milhlll,
Q ueensland 4064. Australia
John Wile�· & Sons (Canada) Ltd, 22 W o rcester Road, Rexdalc. ( lntario 1\19W 11.1. Cana
Jolm Wiley & Sons (A.,ia)
Jin
X ing Distripark.
Pte
l.td, 2 C kme nti l..<><>p 1102-0 I.
Singapore 0512
British Library Ciltalo�:r�iu�: iu l'r�blicatiorr Data A catalogue r�cord t<•r this hook is ava ila bl e from the British Lihmry
ISBN
0 471 9661!8 6; 0 471 97015!! (phk.)
Produced from camcra-rcmly
copy supplkd hy the ••uthors using l.�TeX
Printed ;uld hound in ( ircat Britain hy Biddies Ltd. (iuildt<>rd and King's Lynn
·n1is h<><•k is printed on acid-free paper re,ponsihly manul<•ctnrcd trom ;ustainablc c>r whi ch at least l\\o trees :�rc planted tl>r c·;�ch <•nc used t<•r pap�r produc tion.
for�st:�tion.
Contents
Preface . . . . .
.
XI
.
X Ill
List of Symbols
1 Definitions, Notation, Terminology . 1.1
1.2 1.3 1.4 1.5 1.6
Basic Notation and Terminology . . Operations Relating M atrices . . . Inequality Relations Between M atrices Operations Related to Individual M atrices Some Special M atrices . . . . . . . . . . Some Terms and Quantities Related to Mat rices .
.
2 Rules for Matrix Operations . . . . . . . . . . . 2.1 Rules Related to M atrix Sums and Differences 2.2 Rules Related to M atrix M u ltiplication . . . 2.3 Rules Related to M u ltiplication by a Scal ar 2.4 Rules for t he Kronecker Product 2.5 Rules for t he H adamard Product 2.6 Rules for Direct Sums . . . . . .
3 Matrix Valued Functions of a Matrix 3.1 The Transpose . . . . . . 3.2 The Conjugate . . . . . . . . . 3.3 The Conjugate Transpose . . . 3.4 The Adj oint of a Square M atrix 3.5 The I nverse of a Square M atrix 3.5.1 General Results . . . . . 3.5.2 Inverses Involving Sums and Differen ces 3.5.3 Part i tioned Inverses . . . . . . . . . . . 3.5.4 Inverses Involving Commut ation, Du plication and Elimination M at rices . . . . . . . . . . . . . . . . . .
l 1 3 4
4 9 II 1;j 15 16 18 1\) 20 22 n
'2 ;) '24 2J 27 '27 '27
'28 '2V 31
VI
CONTENTS Generalized Inverses . . . . . . . . 3.6.1 G eneral Results . .. . .. . 3.6.2 The Moore - Penrose Inverse 3.7 M atrix Powers . . . 3.8 The Absolute Value
3.6
4 Trace, Determinant and Rank of a Matrix 4.1 The Trace . . . ... . . .. . . . . . . 4.1.1 G eneral Results . .. . . . . .. . . . 4.1.2 Inequalities Involving the Trace .. . 4 .1.3 Optimization of Functions I nvol ving the Trace 4.2 The Determinant . . . . . .. . . .. . . . . . 4.2.1 G eneral Results . . . . . . .......... . 4.2.2 Determinants of Partitioned M atrices . . . . 4.2.3 Determinants Involving Duplication M atrices 4.2.4 Determinants Involving Elimination M atrices 4.2.5 Determinants Involving Both Duplication and Elimination M at rices . . . . . . . . . . . . . . . . . . . . . . 4 .2.6 Inequalities Related to Determinants .... . ... . 4.2.7 Optimization of Functions I nvolving a Determinant. 4.3 The Rank of a M atrix . . . . . . . . . . . . . . . . . 4.3.1 G eneral Results . . . . .. . . . . ... . . . . 4.3.2 M atrix Decompositions Related to the Rank 4.3.3 Inequalities Related to the Rank
32 32 34 37 39 41 41 41 13 45 47 47 49 51 .52 53 .')4 56 58 58 60 61
5 Eigenvalues and Singular Values ... .
63 63 5.1 Defini tions . . . . . . ..... . . . . . 5 . 2 Properties of Eigenvalues and Eigenvectors 64 5.2.1 G eneral Results . . . . . . . . . . . . 64 5.2.2 Optimization Properties of Eigenvalues 67 69 5.2.3 M atrix Decompositions Invol ving Eigenvalues 72 5 . 3 Eigenvalue Inequalities . . . . . . . . . . . . . . . . . 72 5.3.1 Inequalities for the Eigenvalues of a Single M atrix 5.3.2 Relations Between Eigenvalues of More Than One M atrix 74 76 5 . 4 Results for the S pectral Radius 78 5 . 5 Singul ar Values . . . . . 5.5.1 General Results . 78 5.5.2 Inequ alities . . . 80
6 Matrix Decompositions and Canonical Forms 6.1 Complex M atrix Decompositions . . 6.1.1 Jordan Type Decompositions 6.1.2 Diagon al Decompositions . .
83 83 83 85
..
VII
CONTENTS 601 .3 Other Triangular Decompositions and Factorizations 601.4 Miscellaneous Decompositions 0 602 Real M atrix Decomposition s 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 60201 Jordan Decompositions 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 60202 Other Real B lock Diagonal and Diagonal Decompositions 60203 Other Triangular and M iscellaneous Reductions 0
0
7 Vectorization O perators 0 0 0 701 Defi nitions 0 0 0 0 0 0 0 0 0 0 7 0 2 Rules for the vee Operator 0 703 Rules for the vech Operator
95 95 97 99
8 Vector and Matrix Norms 0 801 General Defi n itions 0 0 0 0 0 802 Specific Norms and I n ner Products 803 Results for General Norms and Inner Products 8.4 Results for M atrix Norms 0 0 0 8.401 General M atrix Norms 0 8.402 Induced M atrix Norms 0 805 Properties of Special Norms 80501 G eneral Results 0 80502 Inequalities 0 0 0 0 0
9 Properties of Special Matrices
86 88 89 89 90 92
101 101 103 1 04 106 106 108 109 109 111
1 13 901 Circulant M atrices 0 0 0 0 113 115 902 Com mutation M atrices 0 0 116 90 201 General Properties 0 117 90202 Kronecker Products 118 90203 Relations With Duplication and Elimination M atrices 903 Convergent M atrices 119 1 20 904 Diagonal M atrices 1 22 905 Duplication M atrices 1 22 90501 General Properties 90502 Relations With Commutation and Elimination M atrices 123 90503 Expressions With vee and vech Operators 0 0 0 0 0 0 0 123 905.4 Duplication M atrices and Kronecker Products 0 0 0 0 0 0 124 90505 Duplication M atrices, Eliminat ion M atrices and Kronecker Products 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 126 906 Elim i n ation Matrices 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 127 90601 General Properties 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 27 90602 Relations With Commutation and Duplication M atrices 127 128 90603 Expressions With vee and vech Operators 0 0 128 906.4 Elimin ation Matrices and Kronecker Products 0 0 0 0
0
0
0
0
.
0
Vlll
CONTE NTS
Elim i nation M atrices, Duplication M atrices and Kro. . . . . necker Products . Hermitian M atrices . . . . . . . . . . . . . 9.7. 1 General Results . . . . . . . . . . 9.7.2 Eigenvalues of H ermitian M atrices 9.7.3 Eigenvalue Inequalities . . . . . . 9.7.4 Decompositions of Hermitian Mat rices Idempotent M atrices . . . . . . . . . . . . . \'onHegative, Positivt> and Stochas t ic Matrices . 9.9.1 Definitions . . . . . . . . . . . . . . . . 9.9.2 Geueral Results . . . . . . . . . . . . . 9.9.3 Results Rel ated to the Spectral Radius . Orthogonal M atrices . . . . . . . . . . . . . . 9. 10.1 General Results . . . . . . . . . . . . . 9 .I0.2 Deco111positious of Orthogonal M atrices Partitioned Matrices . . . . . . . . . . . . . 9.11.1 General Resul ts . . . . . . . . . . . . 9.11.2 Determ inants of Partitioned M atrices 9.1 1.3 Partitioned Inverses . . . . . . . . . 9.11.4 Partitioned Generalized Inverses . . 9. 11.5 Partitioned Matrices Related to Duplic ation M atrices Positive Definite , :"Jegative Defini te and Semidefinite M atrices 9.12.1 General Propnties . . . . . . . . . . . . . . . 9.12.2 Eigenvalue Rt>s u l t s . . . . . . . . . . . . . . . 9.1 '2 .:3 Decomposit ion Tht'orerns for Defini.te M at rices Symmetric M atrices . . . . . . . . . . . . 9. 13.1 General Propert ies . . . . . . . . . 9.13.2 Symmetry and Duplication M atrices 9. 13.:3 Eigenval ues of Symmetric M atrices . 9.1:3.4 Eigenvalue Inequalities . . �).1:).5 Decompositious of Symmetric and Skew-Symmetric Matrices . . . . . . . . . . . . . . . . . . . . . . . . Triangular M atrices . . . . . . . . . . . . . . . . . . . . . . 9. 14.1 Properties of General Tri angul ar M atrices . . . . . . 9. 14.2 Triangularity, Elimination and D u plication M atrices 9. 14.3 Properties of S trictly Triangu lar M atrices · 11 \.'Hilary 'M atnces . . . . . . .
9.6.5
.
9.7
.
.
.
.
.
.
.
.
.
.
.
.
130
.
I :ll
1 :n 133 134 137 I :38 1 39 I :39 110 111 142 143 111 144 145 146 147 148 149 150 15 1
.
.
9.8 9.9
.
.
9.10
.
.
9. 11
.
.
.
.
9.12
.
1:i3
.
9.13
l :)4
156 156 157 158 159
.
.
.
.
.
.
.
.
163 161 164 165 IG7 167
.
9. 11
9 . 15.
·
.
10 Vector and Matrix Derivatives
171 I7 1
10.1 Not ation . . . . . . . . . . . . 10.2 G radients aud Hessian M atrices of Real Valued Functions with .
Vt>ctor Arguments 10.'2. 1 Gradients .
. . . . . . . . .. . . . . . . . . . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
174 174
.
IX
CONTENTS 10.3
10.4
1 0 . 2 . 2 H essian M atrices . . . . . . . . . . . . . . . . . . . . . Derivatives of Real Valued Functions with M atrix A rguments 1 0 .3 . 1 General and Miscell aneous Rules 1 0. 3 . 2 Derivatives of the Trace . . . . 1 0. 3 . 3 Derivatives of Determinants . . . J ac obian M atrices of Linear Functions . 1 0 .4 . 1 Linear Functions with General M atrix Argumen t s 1 0 . 4 . 2 Linear Functions w i t h Symmetric M atrix A rguments 1 0 . 4 . 3 Linear Functions with Triangular M atrix Arguments 1 0 . 4 . 4 Li near Functions of Vector and M atrix Valued Funrtions with Vector Arguments . . . . . . . . Product Rules . . . . . . . . . . . . . . . . . 1 0.5 . 1 M atrix Products . . . . . . . . . . . . . . . . 1 0. 5 .2 Kronecker and Hadamard Prod u cts . . . . . 1 0 . 5 .3 Funct ions with Symmetric M atrix Arguments 1 0. 5 .1 Functions with Lower Triangular M atrix Arguments 1 0 . 5 .5 Products of M atrix Val ued Funct ions wit h Vector Arguments . . . . . . . . . . . . . . . . . . . . . . . Jacobian Matrices of Functions Involving Inverse Matrices 1 0 .6 . 1 M atrix Products . . . . . . . . . . . . . . . . . . 1 0 . 6 . 2 Kronecker and H ad amard Products . . . . . . . 1 0 . 6 .3 M atrix Valued Functions with Vector Arguments Chain Rules and Miscellaneous Jacobian M atrices J ac obian Determinants . . . . . . . 1 0 .8 . 1 Linear Transform ations . . . . . . . . 1 0. 8 . 2 Nonlinear Transformations M atrix Valued Functions of a S calar Variable .
1 0 .5
10.6
10.7 10. 8 10.9
1 1 Polynomials, Power Series and Matrices . .
1 1 . 1 Definitions and N ot ations . . . . . . . . . . . 1 1 . 1 . 1 Definitions and N otation Related to Polynom i als 1 1 . 1 .2 M atrices Related to Polynomials . . . . . . . . . 1 1 . 1 . 3 Polynomials and Power Series Related to M atrices 1 1 .2 Results Relating Polynom ials and Matrices 1 1 .3 Polynom ial M atrices . . . . . . . . . . . 1 1 .3 . 1 Definitions . . . . . . . . . . . . 1 1 .3 .2 Results for Polynomial M atrices
Appendix
A
Dictionary of Matrices and Related Terms
References . Index . . . .
1 75 1 76 1 76 1 77 1 81 183 183 1 85 187 1 88 189 189 192 19 3 195 1 96 1 98 1 98 1 99 200 202 204 204 206 208 211 211 211 213 214 215 217 217 22 1 225 287
•
•
•
0
0
•
•
•
•
•
•
29 1
Preface Nowadays m atrices are used in m any fields of science. A ccordingly they h ave become standard tools in statistics, econometrics, m at hematics, engineering and natural sciences textbooks. I n fact, many textbooks from these fields h ave chapters , appendices or sections on matrices. Often collections of those results used in a book are included . Of course, there are also numerous books and even journals on m atrices. For my own work I h ave found i t useful , however, to have a collection of important matrix results h andy for quick reference in one source. Therefore I h ave started collecting matrix results which are important for my own work . Over the years, this collection has grown to an extent that it may now b e a valuable sou rce of results for other researchers and students as wel l . To m ake it useful for less advanced students I h ave also included m any elementary resul ts. The idea is to provide a collection where special matrix results are easy to locate. T herefore there is some repetition because some results fit u nder different headings and are consequently listed more than once. For example, for suitable matrices A , B and C, A ( B + C) A B + AC is a result on m atrix multiplication as well as on matrix sums . I t is therefore listed u nder both headings. Although reducing search costs has been an i mportant objective in putting together this collection of results, it is stil l possible that a specific result is listed in a place where I would look for i t but where not everyone else would, b ecause too much repetition turned out to be counterproductive. Of course, this collection very much reflects my own personal preferences in this respect. A lso, it is, of course, not a complete collec tion of results. In fact , in this respect it is again very subjective. Therefore, to make the vol u me more useful to others in the future, I would like to hear of any readers ' pet results that I h ave l eft out. Moreover, I wou l d be grateful to learn about errors that readers discover. I t seems unavoidable that flaws sneak in somewhere. In this book, defi nitions and results are given only and no proofs. At the end of most sections there is a note regarding proofs. Often a reference is made to one or more textbooks where proofs can be found . No attempt has been made to reference the origins of the results. A lso, computational algorithms =
..
Xll
arC' not given . These m ay again be found in the references. As mentioned earlier, it is hoped t h at this book is useful as a reference for researchers and st udents. I should perhaps give a warning, however, that it requires some basic knowledge and understanding of m atrix algebra. It is not meant to be a tool for self teaching at an introd uctory level . Before searching for specific matrix results readers m ay want to go over Chapter l t o familiarize themselves with my notation and terminology. Definitions and brief explanations of m any terms related to m atrices are given in an appendix. Generally the resul ts are stated for complex m atrices ( m atrices of complex muubers) . Of cou rse, they also hold for real matrices because the l atter may bt> regarded as special complex m atrices. In some sections and chapters many results are formulated for real m atrices only. In those instances a note to t hat effect appears at the beginning of the section or chapter. To make sure t hat a specific result holds indeed for complex matrices it is t herefore recommended t o check the beginning of the section and chapter where the result is found to determine possible qualifications . Finally, I would l ike to acknowledge the help of many people who com mented on t his volume and helped me avoid ing errors. In partic u lar, I ant grateful to A lexander Benkw itz , Jorg Breitung, M aike B u rda, Helmu t Her wartz , Kirsti n H ubrich , Martin Moryson , and Rolf Tschernig for their care fu l scrutiny. Of course , none of them bears responsibility for any remaining errors. Financial support was provided by the Deutsche Forsch ungsgeruein srhaft . Sonderforschungsbt>rt>ich 373. Bt>rlin. April 1 996
Helmut Lutkepohl
List of Symbols General Notation
L n
s u m m ation product equivalent to, by definition equal to implies that, only if if and only if 3 0 1 4 1 59 v'-1, imaginary unit complex conjugate of a complex n umber n atural logarith m exponential function sine function cosine function m axrmum mm1mum supremum , least upper bound infimu m , greatest lower bound 0
I 0
-
c
In exp 0 Sill
cos max mrn sup inf df dx of ox n! 0
0 0
derivative of the function f(x) with respect to
=1°2
.
.
0
11
bin omial coefficient
Sets and Spaces ffi
.
x
partial derivative of the function f(-) with respect to x
n! k!(n - k)!
{0:
c
.
o}. { 0} .
.
set real numbers complex numbers
.
XIV
IN 7L.
IR
Q
positive i ntegers integers m-dimensional Euclidean space, space of real (m x I ) vectors real ( m x n ) matrices m-d imensional complex space, space of complex (m x I ) vectors com plex ( m x n ) matrices
m
_'nl. X n.
General M at rices A= [a;j] x n) A A= [A1 : A2] A = [A 1 : . . : An ]
matrix matrix m atrix matrix
A=
matrix consisting of submatrices A1 and A2
(m
.
with typical element a;j with m rows and n columns consisting of submatrices A1 and A2 consisting of sub matrices A 1, ... An ,
AI
matrix consisting of submatrices A 1, .
A=
A=
[
An A II A21
.412 An
]
A=
. . ,
p artitioned matrix consisting of submatrices A11,A12,A2I,An
partitioned m atrix consisting of submatrices A;j
Special Matrices
Drn
( m 2 �m(m + I ) ) duplication m atrix , 9 ( x m) identity matrix, IO (mn x ) commutation matri x , 9 ( �m(m + I ) x m 2 ) elimination matrix , 9 (m x ) zero or null matrix, 2 x
lm 1\mn or l\m n Lm Omxn 0 '
m
mn
n
zero, null matrix or zero vector
Matrix O p erations A+B .4-B AB
An
sum of matrices A and B, 3 d ifference between matrices A and B, 3 product of matrices A and B, 3
XV
cA, Ac A0B A0B
product of a scalar c and a matrix A, 3 Kronecker product of matrices A and B. 3 H ad amard or elementwise product of mat rices A and B, 3 direct sum of matrices A and B . 4
Matrix Transformations and Functions det A , det ( A ) t r A , tr(A) II A l l
IIA\h
rk A , rk( A ) l A Iabs A' A AH
A a dj A-1 A
A+
A' A 1 /2
determ inant of a matrix A, 6 trace of a matrix A , 4 norm of a matrix A , 10 1 Euclidean norm of a m atrix A, 1 03 rank of a matrix A , 12 absolute value or modulus of a mat rix A. 5 transpose of a m atrix A, 5 conj ugate of a matrix A, 5 conj ugate transpose or H ermitian adjoint of a matrix A , 5 adjoint of a matrix A , 6 inverse of a matrix A , 7 generalized i nverse of a mat rix A, 7 Moore - Penrose inverse of a m at rix A, 7 ith power of a matrix A, 7 square root of a matrix A , 8 0 a11
dg( [a;j ] ) 0
amm
a1 1
0
0
a mm
diag( a l l , . . . , amm) vee , col rvec row vech
col u m n stacking operator , 8 row stacking operator , 8 operator that stacks the rows of a m atrix in a column vector, 8 half- vectorization operator, 8
Matrix Inequalities A >0
all elements of A are posit ive real numbers. 4
.
XVI
A> 0 A< 0 A< 0 A> B A> B
all elements of A are nonnegative real numbers, 4 all elements of A are negative real numbers, 4 all elements of A are nonpositive real numbers, 4 each element of A is greater than the corresponding element of B, 4 each element of A is greater than or equal to the correspondi ng element of B, 4
O ther Symbols Related to Mat rices minor ( a;1 ) cof( a,j )
,\(A) Ama.r or Ama.r(A) Amin or Amin(A) cr( A ) CT mar or CTma.r (A) CTmin or CTmin(A) p( A ) PA ( · ) QA ( )
minor of the element a;j of a matrix A , 6 cofactor of the element a;j of a matrix A, 6 eigenvalue of a matrix A, 63 maximu m eigenvalue of a matrix A, 63 mi nimum eigenvalue of a matrix A, 63 singular value of a matrix A, 64 maximu m singular value of a matrix A, 64 minimum singular value of a matrix A, 64 spectral radius of the m atrix A, 64 characteristic polynomial of a matrix A, 63 minimal polynomial of a matrix A, 215
1 Definitions, Notation, Terminology 1.1
Basic Notat ion and Terminology
A n ( m x n) matrix A is an array of numbers : ...
A=
A lternative notations are: A(mx n ),
A ( m x n)
A = [a ;j ] ( m x n ) .
and n are posit i ve integers denoting the row dimension and t h (' column dimension, respectively. ( m x n) is the dimension or order of A. Thf' a;j, i = 1 , .. . , m , j 1 , . . . , n, are real (elements of IR) or complex numbers (elements of
=
•
A=
A11 .421
AI2 .422
A1q A2q
= [.4;1 ] •='·· }= 1'
A p1
A p2
••
·•
= [A;J].
A pq
Here the A;j are (m; x n j ) submatrices of A with 2'..:�'=1 m; = m aud 2'..:)=1 n1 = n. A matrix written in terms of submatrices rather than individual
) -
H A N D BOOK O F M ATRICES
'
<:>lements is often called a partitioned matrix or a block matrix. S pecial cases are A= [At . . . . , An] =(At: . . . : An], where the A; are blocks of colum ns, and A=
where the A; are blocks of rows of A . A matrix A ( m xm )
having the same number of rows and columns i s a square or quadratic matrix . The elements a11,a22, ... ,amm (a;;, i l , . . . , m) const itute its principal diagonal. =
Im ( m x m)
1
0
0
1
with a;; = 1 for i = 1, . . . , m , and a;j = 0 for i -:f. j is an (m x m) identity or unit matrix and 0 0 Omxn =
= [a;j] 0 0 ( m x n)
with a;1 = 0 for all i, j is an ( m x n) zero or null matrix. It is someti mes simply denoted by 0 . A ( 1 x n ) matrix a=[at, ... ,an] is an n-dimensional row vector or ( 1 x n ) vector. An ( m x 1 ) matrix
b= is an m-vector, m-dimensional vector or m-dimensional colun1n vector. The set of all real m-vectors is often denoted by IRm and t he set of all complex m-vectors i s denoted by a:;m. M any more special matrices are listed in Section 1.5 and the Appendix.
3
D E F I N ITIONS, N OTATION, TER M I NOLOGY
1.2
Operat ions Relating Matrices
Addition: A= [a;i] ( m x n), B= [b;i] (m x n)
A+ B:::: [a;j+ b;j] ( m x n)
(for the r ules see Section 2 . 1) .
Subtraction: A= [a;j] ( m x n), B = [b;j] ( m x n)
A - B= [a;i - b;i] ( m x n)
(for the rules see Section 2 . 1) .
Multiplication by a scalar: A= [a;j] (m x n), c a number ( a scalar) cA:::: [ca;j] (m x n)
Ac
=
( for the r ules see Section 2 . 3 ) .
[ca;j] ( m x n)
Matrix multiplication or matrix product : A= [a;iJ (m x n), B= [b;j] (n x p) AB = [ta;kbkil ( m x p) k=l (for the r u les see Section 2 . 2 ) . Kronecker product, tensor product or direct product: A= [a;j] ( m x n), B = [b;j] (p x q) A®B=
a11B
am!B
a1nB
" ·
amnB
(mp x nq)
( for t h e rules see Section 2 . 4 ) . Hadamard product, Schur product or elementwise product: A= [a;j] ( m x n) , B = [b;i] (m x n) A 0 B = [a;ibij] (m x n)
( for the rules see Section 2 . 5 ) .
4
H A N D BOOK OF M ATRICES
Direct sum: A (m x m), B ( n x n ) A q, B
=
[ � � ] ((
+ n) x (m + n ))
m
( for the rules see Section 2.6).
Hierarchy of operations: If more than two matrices are related by the foregoing operations. they are performed in the following order :
(i) operations in parentheses or brackets, ( i i ) m atrix multiplication and multiplication by a scalar , ( i i i ) Kronecker product and H adamard product, ( i v ) addition and subtraction, ( v ) d i rect s u m . Op erations of the same hierarchical level are p erformed from left to righ t .
1.3
Inequality Relations Between Mat rices
A = (a;j] , B = (b;j] ( m x A A A A
>0 > - 0 > B > B
n
-¢=:::}
-¢=:::}
¢::::::;> ¢::::::;>
) real :
Qjj > a;j -> a;i > a;i >-
0, l 0, l b;j, b;j,
- 1, . . - 1, . . l - 1, . l - I, . .
.
- 1, . , n , , m , J - 1, . . , n, . . . , m, J - 1 , . , n, . - I, . . . , n . . , m, J J
.
.
, m,
.
.
.
.
.
.
.
.
Warning: In ot her literature i nequality signs between matrices are sometimes used to denote different rel ations between posi tive definit<:> and st'midefinite matrices.
1.4
Operat ions Related to Individual Matrices
Trace: A =
[a,iJ
(m x m) m
tr A
=
tr( A )
_
a11 + · · · + amm =La;;
( for its properties see Section 4 . I) .
i= I
5
D E F I N ITIONS, NOTATION, T E R M I N O LOGY
Absolute value or modulus: A= [a,j] (m x n)
IAiabs = [ la;jlabs] =
Ia I I Iabs la21labs
la12labs lanlabs
lainlabs la2nlabs
lam I labs lam2labs
lamn labs
(m x n )
where lclabs denotes the modulus of a complex number r = c1 + ir" which is defined as lclabs = Jci + c� = JCC. Here c is the complex conj ugate of c. Properties of the absolu te value of a m atrix are given in Section 3 . 8 . Warning: In other literature IAI sometimes denotes the determinant of the mat rix A. Transpose: A = [a,;] ( m x n ) I
A' =
that is, the rows of A are the columns of A' (see Sec tion 3.1 for its properties ) . In some other l iterature the transpose of a matrix A is denoted by AT. Conjugate: A = [a;j] ( m x n)
where ii;i is t he complex conj ugate of a;j (see Sect ion p ropert ies ) .
3.2
Conjugate t ranspose o r Hermitian adjoint: A= [a;j] (m x n ) AH _A'= [ii;;]' (n x m) (see Section 3 . 3 for its properties ) . Diagonal matrices: A= [a;jJ ( m x m) 0
dg (A) =
for i t s
6
H A N D BOOK O F M ATRICES
0
diag( a 1 1, . . . , amm) = d i ag
0
( see Section 9 . 4 for further details ) . Determinant: A =
[a;j] (
m
x
det A = det ( A )
m
=
)
l:)-1)Pat;1a2i,
X ·· · X
am;m,
where the sum is t aken over all products consisting of precisely onf' element from each row and each column of A multiplied by -1 or 1, if t he permutation it , . . . , im is odd or even , respectively ( see Section 4 . 2 for the p roperties of determinants) . Minor of a matrix: The determi nant of a submatrix of an ( m x matrix A is a m inor of A.
m
)
Minor of an element of a matrix: A = [a;j] ( m x m ) . The m i nor of a,i is the determi nant of the matrix obtained by deleting the ith row and jth column from A ,
min or( a;i)
=
I
a;- ,ja; j
det
+t , - t
am,jPrincipal minor: A =
I
[a;j]
(m x
m
I
a;- ,j a; + j
+I
ai-l m '
t , +t
1
)
det is a principal minor of A for k = 1 ,
. . . , m-
1.
Cofactor of an element of a matrix: A= [a;j] ( m x of a;j is cof( a ; j ) 8 ( - 1 / +i mi nor(a ;j) . Adjoint: A =
[a;1]
(m x
m
), m
>
m
) . The cofactor
1, I
cof( ami)
cof( amm)
= [cof( a;i )]'.
I
�
DEFI NITIONS, NOTATIO N , T E R M I NOLOGY AadJ = 1 if m
=
1 (see Section 3 . 4 for the properties of the adjoint ) .
Inverse: A = [a;j J ( m x m ) with det ( A) -::j:. 0 . The inverse of A is t he u nique ( m x m) matri x A-1 satisfying AA-1 =A-1A = Im (see Sect ion 3.5 for its propert ies ) . Generalized inverse: An ( n x m ) matrix A- is a generalized inverse of the ( m x n) matrix A i f it satisfies AA-A= A (see Section 3 . 6 for i t s properties ) . Moore -Penrose ( generalized) inverse: The ( n x m) matrix A+ is the Moore - Pen rose ( generalized ) inverse of the ( m x n) mat rix A. if it satisfies ( i ) AA+A= A, ( i i ) A+AA+ =A+, ( iii ) (AA+)H = AA+, ( i v ) (A+A)11 =A+A (see Section 3 . 6 . 2 for its properties) . Power of a matrix: A ( m x m )
n;=I A= A X . .. X A i t imes
A'=
for positive integers i for i = 0 for negative integers i, if det ( A. ) -::j:. 0
I f A can be wri tten
as
0 A=U
0 for some unitary matrix U (see Section 1.5 and Chap ter 6 ) t hen the power of A is defined for any a E IR, a > 0, as follows: Aa
=
U
,\"I
0
0
,\0' m
This definition applies for instance for Hermitian and real symmet ric matri ces (see Section 1.5 for the defini tions of Hermi tian and sym metric m atrices and Section 3 . 7 for the properties of powers of matrices ) .
8
H A N DBOOK OF M ATRICES
Square root of a matrix: The ( m x m) matrix A 1 12 is a square root of t he (m x m ) m atrix A if A112A112 A Elsewhere in the l iterature a matrix B satisfying B' B A or B B' A is sometimes regarded as a square root of A. =
=
.
==
Vectorization: A = [a;j ] ( m x n )
vee A = vec ( A )
=
( mn
col( A ) =
x 1)
t hat ts, ve e stacks the columns of A in a column vector. rvec(A.)
=
[vec( A' )]'
that is, rvec stacks the rows of A in a row vector and row( A ) = vec( A ' ) = rvec ( A)'
t hat is, row stacks the rows of A in a column vector. (See Chapter 7 for the properties of vectorization operators) . Half-vectorization: A= [a ;j] (m x m )
vech A= vech( A )
n m( m + 1 )
=
X 1)
Urnm
that is, vech stacks the columns of A from the principal di agonal downwards in a column vec t or ( see Chapter 7 for the propertirs of the half-vectorization operator and more details) .
9
D E F INITIO NS, NOTATION, T E R M I N O LOGY 1.5
Some S pecial Mat rices
Commutation matrix: The (mn x mn) matrix 1\,n is a commutation matrix if vec ( A ' ) = 1\mn vec ( A ) for any (m x n) matrix A. I t is sometimes denoted by f-(m,n . For example,
/\32 = /\3 2 '
=
1 0 0 0 0 0
0 0 1 0 0 0
0 0 0 1 0 0 0 0 1 0 0 0
0 0 0 1 0 0
0 0 0 0 0 1
is a com mutation matrix . ( For t he properties of commutation matrices see Section 9. 2 . ) D iagonal matrix: An ( m x m) matrix UJ!
0
with a;1 = 0 for i #- j is a diagonal matrix . ( For the propert ies of diagonal matrices see Section 9 . 4 . ) Duplication matrix: An (m 2 x �m(m + 1 ) ) matrix Dm is a duplication matrix if vec ( A ) Dm vech ( A ) for any symmetric ( m x m) matrix .4 . For example, 1 0 0 0 0 0 0 1 0 0 0 0 0 0 1 0 0 0 0 1 0 0 0 0 0 0 0 1 0 0 0 0 0 0 1 0 0 0 1 0 0 0 0 0 0 0 1 0 0 0 0 0 0 1 =
is a duplication matrix. (For the properties of duplication matrices see Section 9 . 5 . ) Elimination matrix: A ( �m( m + 1 ) x m 2) elimination matrix L , is defined such that vech ( A ) L,vec( A ) for any (m x m) matrix A. =
10
H A N DBOOK O F MATRICES
For exampl e, 1 0 0 0 0 0
L3 =
0 l
0 0 0 0
0 0
0 0 1 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 1 0 0 0 0 1 0 0 0 0 0 0
0 0 0
0 0 0 0 0 1
is an elimination matrix. ( For the properties of elim ination m atrices see Section 9 . 6 . ) Hermitian matrix: A n ( m x m) matrix A i s Hermitian if A' = AH =A . ( For its properties see Section 9. 7.) Idempotent matrix: An ( m x m) m at rix A is idempotent if A2 = thP properties see Section 9 . 8 . )
A.
( For
Identity matrix: Au ( m x m ) m atrix 1
0
0
I
lm =
=
l for with a;; unit m atrix.
1
=
I,
. . . , m and a;j = 0 for i f. j is an identity or
Nonnegative matrix: A real ( m x n) m atrix A= [a;j] is nonnegatiw if a;j > 0 fori = 1 , . . . , m, j = 1 , . . . , n. ( For the properties see Section 9.9. ) Nons in gular matrix: An ( m x m) matrix A is said to be nonsingular or i uverti ble or regular if det (A) f. 0 and thus A- 1 exists. ( For the rules for mat rix inversion see Section 3 .5 . ) Normal matrix: An ( m x m ) m atrix A i s normal i f AH A= AAH .
Null matrix: Au ( m x n ) matrix is a null m atrix or zero matrix, deno!t'd by Omxn or simp ly by 0, if all its elements are zero. Orthogonal mat rix: A n ( m x m) m atrix A is orthogonal if A nonsingular and A'= A-1• ( For the properties see Section 9 . 1 0 . )
Is
Positive an d negative definite and semidefinite matrices: A Hermitian or real symmetric ( m x m) m atrix A is positive definite if xH Ax > 0 for all ( m x 1 ) vectors x f. 0; it is positive semidefinite
D E F I N ITIO N S, NOTATIO N, TERM I N O LOGY
II
if x H A x > 0 for all ( m x 1 ) vectors x; it is negative definite if xH Ax < 0 for all ( m x 1 ) vectors x :f. 0; it is negative semidefinite if xH Ax < 0 for all (m x 1 ) vectors x; it is indefinite if ( m x 1) vectors x and y exist such that xH Ax > 0 and yH Ay < 0 (see Section 9. 12). Positive matrix: A real ( m x n ) m atrix A = [a;j] is positive if a;j > 0 for i = 1 , . . . , m, j = 1, . . . , n ( see Section 9.9 ) . Symmetric matrix: A n ( m x m ) m atrix A = [a;i] with a;j = aj;, !, J = 1 , . . . , m is symmetric. I n other words, A is symmetric if A' = A (see Section 9 . 1 3 ) . Triangular matrices: A n ( m
x
m ) m atrix
0
0
0 with a,i = 0 for j > i is lower t riangular. An ( m
x m
) m at rix
0 0
0
amm
with a;j = 0 for i > j is upper t riangular (see Section 9. 1 4 ) . Unitary matrix: A n ( m x m ) m atrix A is unitary if it is n onsingular and A H = A - 1 (see Section 9. 1 5 ) . Note: M any more special m atrices are listed i n the Appendix. 1.6
Some Terms and Quantities Related to Mat rices
Linear independence of vectors: The m-dimensional row or column , Xk are linearly independent if, for complex numbers vectors x1, Ct, . . . , Ck, CtX1 + · · + CkXk = 0 implies c1 = · · = ck = 0. They are linearly dependent if c1x1 + · · + CkXk = 0 holds with at least one c; :f. 0. I n other words, x 1 , . . . , Xk are linearly dependent if aj E If exist such that for some i E {1 , . .. , k } ,x; = o:1 x 1 + · · · + o:;_ 1 x;_1 + Cl:i + tXi+ l + · · · + Cl:kXk. .
.
•
·
·
·
H A N D BOOK O F M ATRIC ES
Rank: A = [aij] ( m x
n)
=
rk A rk( A ) = maximum number of linearly independent rows o r colum ns of A row rk A row r k ( A ) = maximum number of linearly i ndependent rows of A =
=
col r k ( A ) col rk A maximum nu mber of l inearly independent columns of A _
( for rules related to t he rank of a m atrix see Section 4 . 3 ) . Elementary operations: The following changes to a matrix are cal led t>lenwntary operations: ( i ) interchanging two rows or two columns, ( ii )
mu
l t ip l y i n g any row or column by a nonzero number,
( iii) add ing a multiple of one row to another row , ( i v ) adding a multiple of one colum n to another colu m n . Quadratic form: Given a real symmetric (m x m) matrix A , the funct ion IR defined by Q( .r ) = .r' Ax is called a quadratic form. The Q : IRm quadratic form is called positive ( semi ) definite if A is posit ive (semi ) definite. I t is callf'd negative ( semi) definite if A is negativP ( sem i ) defi nite. I t is indefinite if A is indefinite. �
Hermitian form: Given a Hermitian ( m x m) matrix A , the function Q : If"' IR defined by Q ( .r ) = .rH Ax is called a HPrmitian form. The Hermit ian form is called positive ( sem i) definite if A is p os i t i ve (se m i ) definite. I t is cal l ed negative ( semi) definite if A is rwgatiw (st rl l i ) definite. I t is indefinite if A is indefi n ite . �
'
Characteristic p olynomial: The polynomial in A given by det(U m - A) is the charact erist ic poly nomial of the (m x m) matrix A ( see Section 5. l ) . Characteristic determinant: The determinant det(U m characteristic determi nant of the (m x m) matrix A . Characteristic equation: The equation det(Alm - A ) characteristic equation of the ( m x m) matrix A .
-
A ) is t he
0
IS
t he
Eigenvalue, characteristic value, characteristic root or latent root: The roots of t he characteristic polynomial of an ( 111 x m ) matrix A
DEFINITION S . N OTATIO N, TERMINO L OGY
are the eigenvalues, the characteristic values, t.he charact eristic roots or the l atent roots of A (see Chapter 5 ) . Eigenvector or characteristic vector: A n ( mx 1 ) vector v f. 0 satisfy i ng Av = A v, w here A is an eigenvalue of the ( m x m ) matrix A, is an eigenvector or characteristic vector of A corresponding to or associated with the eigenvalue A (see Chapter 5 ) . Singular value: The singular val ues o f an ( m x n ) matrix A are the nonnegative square roots of the eigenval ues of AAH if m< n , and of AH A, if m > n (see Chapter 5 ) .
13
2
Rules for Matrix Operations In the following all m atrices are assumed to be complex matrices u nless otherwise stated. All rules for complex matrices also hold for real mat rices because the latter may be regarded as special complex matrices. 2.1
Rules Related to Matrix S u ms and D ifferences
(1) A , B ( m xn),c E ([;: ( a ) A ±B = B ±A. ( b ) c(A ±B) = ( A ± B)c = cA ±cB . ( 2 ) A,B,C ( mxn): ( A±B)±C = A±( B±C) = A±B±C. ( 3 ) A ( m xn) , B ,C (nxr ) : A ( B ±C) = A B ± AC. ( 4 ) A, B ( m xn), C ( nxr ) : (A± B)C = AC±BC. ( 5) A ( m xn) , B , C (P xq) : ( a ) A 0( B ±C) = A0B±A0C. ( b ) ( B±C) 0A = B0A±C0A . ( 6 ) A, B, C ( mxn): (a ) A 8( B ±C) = A 8B±A8C. ( b ) ( A ±B )I:;)C = A8C±B8C. ( 7) A, C ( mxm ) , B, D (nxn): (AEBB)±(CEBD) = ( A±C) $( B±D ) . ( 8 ) A, B ( m xn) : ( a ) l A + B labs < I A iabs + ! B l abs· ( b ) ( A ±B)'== A'±B' . ( c ) ( A ± B )H = AH ± BH . - ( d ) A ± B = A± B .
HANDHOOK OF MATRWES
16 (9) A, B (m x m) : ( a ) tr(A ± B) = tr( A ) ± tr( B ) . ( b)
(c) ( 10)
(11)
.4,
A
d g (A ±B)= dg(A) ± dg(B). vech ( A ±B)= vech ( A ) ± vech( B ) .
B ( x n): ( a) rk(A ±B)< rk ( A ) + rk ( B) . (b) ( A ± B)= vec ( A ) ± vec(B). m
vee
(m x m):
(a)
.-1.
+ A' is a symmetric ( m x m)
matrix .
(b) A - .-1.' is a skew-symmetric ( m x m ) matrix . ( c ) A+ A H
is
a Hermit i an ( m x
m) matrix . (d ) A- AH is a skew- Hermitian ( m x m ) matrix . Note: All t he rules of this section fol low easily from basic principles by writing down the typical elements of the matrices invol ved. For further details consult introductory matrix books such as Bronson ( 1 989), Barnett ( Hl90). Horn & Johnson ( 1 98.5) and Searle ( 1 982). 2.2
Rules Related to Matrix Multiplication
( 1 ) A (m x
n
) , B ( n x p),C
( 2 ) A, H ( m x m ) (3) (4)
:l
.-1.,
(m x
11
x
11
n ) ,C
(b)
(9)
(c)
.4,
(AB)C =
x
(n
=
=
p) :
) , B (p x q) , C ( 11
11
lJ
x
(n x n) :
) R (m X n) :
X m .
A( BC) = ABC.
.-l.(B ±C') AR ± AC. (.4 ± B)C AC ± BC. ) D (q x s ) :
1· ,
(.-I.:-;;. B)(C 0 D)=
( 8 ) A (m x
( a)
q) :
AB i= RA in general.
:
( 6 ) A, C (m x m ) , B,
( 7) A (m
x
) , B, C (n x p):
B (m x
( 5) A ( m
(p
(.4 <£! B)(C r:p D)= AC tB RD.
l ABiabs<
) , B ( n x p) :
(AB)' = B'A'. ( AB ) H BHA11. AB = A B .
AC 0 B D.
IAiabsiBiabs·
=
B( 1
1 1
x 111) nonsingular :
(AR)-1 = B-1 A-1.
17
Rl'LES FOR M ATRIX O P E RATIO N S ( 10 ) A = [A;j) (m x B ;i ( n, x Pi) :
n)
with A;j ( m; x
AB = ( 1 1 ) A (m x
n), B
( 12) A, B (m x
m
(n
)
x m):
:
(13) A (rn x n ) , B (n
[�
nj). B
A;kBki
[B,1) ( n
-
x
p) with
]·
tr( A B) = tr( B A).
det(AB) = d et( A ) det(B).
x p):
rk( A B )< min{rk( A), rk( B)}.
n) : rk( A A') = rk( A' A)= rk(A). A (m x n), B ( n x p) :
(14) A (m x (15)
vec( A B ) = (Jp 0 A)vec( B) = ( B' (16) A (m x 11), B (1 1 x p) , C (p x q) : (17) A (m x n), B ( n x m) :
lm)vec(A) = (B' C A)vec(ln).
vee( ABC) = (C' 0 A )vec(B).
tr( A B)= vec(A')'vec(B) = vec(B')'vec(.-\).
(18) A (m x n ), B ( n x p),C (p x m) :
(19) A ( m x n ) . B ( n
0
tr(AB C) = vec(A')'(C':�'J)vec(H).
x p ) . C (p x q) , D (q x m) :
tr(ABCD) = vec(D')'(C' 0 A)vec(B) =we ( D)'(..\::' C')vec(B'). (20) A (m x m)./3 (m x n): (21) A (m x n ) . B (11
x
n ):
A n onsing u l ar
=>
rk( t B)
.4
(m
x
m) nonsingu)a;:
A A-1 = A-1.4 =
(24) A ( m x n ) :
(a ) A ln = lm A = A .
( b ) AOnxp = Omxp. (25) A (m x n ) real:
=
rk( B ).
B nonsingular => rk( A B) = rk(.4)
(22) A ( m x m ) , B ( m x n ) , C ( n x n ) : A, C nonsingular => rk(A BC) = rk(B).
(23)
..
!111.
Opxm A = Opxn·
(a) .4A' and A' A ar e symmetric positive semidefinite.
( b ) rk( A ) = m => A A ' is p ositive definite. (c) rk(A) = n => .-\'A is positive definite.
( 26) A ( 111
X
1!)
:
(a) AAH and Alf A are He r m i t i an positive sPmidPfinitP.
( b ) rk(A) = m => AA H is positive definite. (c) rk( A.)
= n
=> A H A is positive definite.
18
H A N D BOOK OF M ATRICES
Note: M ost of the results of this section can be proven by considering typical elements of the matrices involved or follow d irectly from the definitions (see also introductory books such as Bronson ( 1 989 ) , Barnett ( 1 990 ) , Horn & J ohnson ( 1 985 ) , L ancaster & Tismenetsky ( 1 985) and Searle ( 1 982 ) ) . The rules involving the vee operator can be found, e.g. , in M agnus & Neudecker (I988 ) .
2.3
Rules Related to M u ltip lication by a S calar
( 1 ) A ( m x n ), c 1 , c2 E ([;:
( a) c 1 ( c2 A ) = c.( Ac 2 ) = (c 1 c 2 )A = A ( c 1 c2 ) . ( b ) ( c1 ± c2)A c 1 A ±c2 A . =
A, B ( m x n ) , c E ([; c( A ± B ) = c A ± c B . (3) A ( m x n ) , B ( n x p ) , c1, c 2 E ([;: ( c1A ) ( c2 B ) = c 1 c2 A B . ( 4) A ( m x n ) , B ( p x q ) , c 1, c2 E ([; : ( 2)
:
( a) c.( A 0 B) = (c1 A ) ® B = A 0 ( c1B). ( b ) c 1 A 0 c2 B = ( c1c2 ) ( A 0 B) .
( 5 ) .4, B ( m x n ) , c1, c2 E
( a ) c 1 ( A 8 B) = ( c1A ) 8 B = A
8
( c 1 B).
( b ) c 1 A ,:;:: c2B = ( c1c2) ( A \-)B).
(6) A (m x m ) , B ( n x n ) , c E ([;: (7) A ( m X n ) :
c( A EB B ) = cA EB cB.
OA = A O = Dmxn·
(8) .4 ( m x n ) , c E ([':
(a) / cA/abs = / c / abs / A / abs · ( b ) ( c A )' = cA'. (c ) ( cA ) H = cA H .
(9) A (m x m), c E ([;:
(a) tr( cA) = c tr( A ) . ( b ) dg( c A ) = c dg( A ) . ( c ) A is nonsingu l ar, c i= 0 ( d ) det ( c A ) = c m det( A ) .
=>
( cA ) - 1
=
� A- 1 •
(e) vech(cA ) = c vech ( A ) . ( f ) >. is eigenvalue of A
=>
c>. is eigenvalue of c A .
19
RU L ES FOR M ATRIX O P ERATIO N S
( 10 ) A ( m x n ) , c E
( b ) c :j:. 0
=? =;.
( c ) vec( c A )
=
rk( cA ) = rk(A ) . (cA) + = � A + . c vec(A ) .
Note: A l l these rules are elementary and can be found in introductory matrix books such as Bronson ( 1989 ) , Barnett ( 1 990) , Horn & J oh nson ( 1 985 ) , Lancaster & Tismenetsky ( 1 985) or follow directly from definitions. 2.4
Rules for t he K ronecker P roduct
( 1 ) A ( m x n ) , B (p x q) :
A @ B :j:. B @ A in general.
( 2 ) A (m x n) , B, C ( p x q) :
A @ ( B ± C) = A @ B ± A @ C.
( 3 ) A ( m x n ) , B (p x q ) , C ( r x s)
:
A @ ( B @ C ) = ( A @ B) @ C = A @ B 0 C.
( 4) A , B ( m x n ) , C, D ( p x q ) : ( A + B ) @ ( C + D) = A @ C + A @ D + B @ C + B 0 D. ( 5) A ( m x n ) , B (p x q ) , C ( n x
r ,
) D (q x s)
:
( A @ B ) ( C @ D) = A C @ BD. (6) A (m x n), B
(P x q ) , C ( r x s ) , D ( n x k ) , E (q x 1), F ( s x t ) :
( A @ B @ C ) ( D @ E @ F) = A D @ B E @ C F. (7) A ( m x n ) , c E
c @ A = cA = A @ c.
( 8) A ( m x n ) , B ( p x q ) , c, d E
cA @ dB = cd( A @ B ) = ( cdA) @ B = A @ ( cdB) . (9) A ( m x m) , B ( n x n) , C (p x p ) :
( 1 0 ) A ( m x n) , B (p x q ) : ( a) ( A @ B)' = A' @ B1 • ( b ) ( A @ B ) H = A H @ BH . -
-
( c ) A @ B = A @ B. ( d ) ( A @ B )+ = A+ @ B+ .
( A EB B) @ C = ( A @ C) EB ( B 0 C ) .
20
H A N D BOOK OF M AT R IC ES
(e) l A
0
(f) rk( A
B l abs = I A i abs
c:;
0
I H i abs
B ) = rk( A ) rk( B ) .
( 1 1 ) A ( m x m ). B ( 11 X 11 ) : ( a) A, B nonsingular ( b ) tr( A
0
=?
(A
0
B)- 1 = A- 1 0
B ) = t r ( A ) tr( B ) .
B- 1 .
( c ) dg( A 0 B ) = dg( A ) 0 dg( B) .
( d ) det ( A 0 B ) = ( det A t (det B )m . (e) >. ( A ) and >.( B ) are eigenvalues of A and B . respectively, w ith associated eigenvectors v ( A ) and v ( B ) =? >. ( A ) · >. ( B ) is eigenvalue of A 0 B with eigenvector v ( A ) 0 v( B ) . ( 1 2 ) .r . y ( m x 1 ) : ( 1 3)
.r
.r ' 0 y = yx' = y 0 x' .
(m x 1 ) , y ( n x 1 ) :
vec(.ry' ) = y 0 x .
( 1 4) A (m x 11 ) , B ( 11 x p). C ( p x q ) :
( C' 0 A ) vec ( B) = vee( A BC ) .
( 1 5 ) A ( m x 11 ) , B ( n x p ) , C ( p x q ) , D ( q x m ) : tr( A BCD)
vee ( D' )' ( C'
:�!
A ) vee ( B )
vec ( D' ) ' (C' 0 A ) vec ( B ) vee( D ) ' ( A 0 C' )vec ( B' ) , vec ( A ' ) '( D' 0 B ) vec ( C ) vee( H' ) ( A' 0 C)vec( D) vee( C' )' ( B' 0 D)vec ( A ) . '
Note: A substantial collection of results on K ronecker products incl uding m any of those given here can be fou nd in M agnus ( 1988 ) . Some resu lts are also given in M agnus & Neudecker ( 1 988) and Lancaster & Tismenetsky ( 1 985 ) . There are m any more rules for Kronecker products related to spt>cial matrices, in particular to com mutation , duplicat ion and el iminat ion matrices (see Sections 9.2. 9.5 and 9.6 ) .
2.5
Rules for t he Hadamard P rod uct
( 1 ) A . B ( 1 1! X
11
)
•
c
E
Q�
:
(a) A ,_;:. B = B '�' A . ( h ) c( A :· B ) = ( c A ) ·?• B = A ,_;:, (cB).
(2) A. B. C ( m x 11 ) : ( a ) A ·.·: ( B , : C ) = ( A ; B ) ,_;:· C = A · � B c ; . c_
21
RULES FOR M ATRIX O PERATI O N S
( b ) ( A ± B ) 0 C = A 0 C ± B 0 C. ( 3 ) A , B, C, D ( m x
n.
)
:
( A + B ) 0 ( C + D) = A 0 C + A 0 D + B 0 C + B 0 D . ( 4 ) A , C ( m x m ) , B , D ( n. x n ) :
( A EB B) 0 (C EiJ D) = ( A �1 C) 6.1 ( B C:: 1 D).
(5) A, B ( m x n ) :
( a) ( A C:, B)' = A' 0 B' . ( b ) ( A '::' B ) H = A H 8 B H . (c ) A
0
-
-
B = A 0 B.
( d ) l A (:) B l abs = l A I a bs 8 I B i abs ·
( 6 ) A ( m X n) :
A
8 Omxn
= O m xn 0 A = Omxn ·
A 0 Im = Im 8 A = dg( A ) . 1 1 ( m x m) : ( 8) A ( m x n ) , J = 1 1 (7) A ( m x m ) :
(9) A , B , C ( m x n ) :
tr[A'( B 0 C)] = tr[(A' 1::1 B' )C'] .
( 10 ) A , B ( m x m ) , j = ( 1 , . . . , 1 )' ( m x 1 ) :
tr ( A B ' ) = j'(A
��
B)j.
( 1 1 ) A, B, D ( m x m), j = ( 1 , . . . , 1 )' (m x 1 ) : D d i agonal ( 12 ) A , B , D ( m x m ) : ( 13) A. B (m x
n
( a ) vec( A
)
r:;'
=>
tr( AD B' D) = j ' D( A
D diagonal
=>
8
B ) Dj.
( DA ) C:' ( BD) = D( A ,::; B ) D.
:
B ) = diag( vec A )vec ( B ) = diag( vec B )vec ( A ) .
( b ) vec( A :;; B ) = ( vee A )
r::J
vec( B ) .
( 14 ) A , B ( m x m ) : ( a) vech ( A 8 B ) = diag( vech A )vech( B ) = diag( vech B ) vech ( A ) .
( b ) vech ( A 1::1 B ) = ( vech A )
0
vech ( B) .
( 1 5 ) ( Schur product theorem) A , B ( rn x m ) : A , B positive (semi ) definite (semi) definite.
=>
A ·�.· B posi t i ve
Note: Most results of this section fol low directly fro m definitions. ThP remaining ones can be found in M agnus & Neudecker ( 1988) . The Schur product theorem is, for example, given in Horn & Johnson ( 1 985 ) .
22 2 .6
H A N DBOOK O F M ATRICES Rules for D irect S ums
( 1 ) A ( m x m ) , B ( n x n ) , c E
( a ) c ( A EB B) = cA EB cB. ( b ) A :j:. B <==> A EB B :j:. B EB A .
( 2 ) A , B ( m x m ) , C, D ( n x n ) : ( a ) ( A ± B) EB (C ± D) = ( A EB C ) ± ( B $ D) . ( b ) ( A EB C) ( B EB D) = A B EB C D . (3) A ( m x m ) , B ( n x n ) , C ( p x p) :
(a) A EB ( B $ C) = ( A EB B) EB C = A EB B EB C. ( b ) ( A $ B) 0 C = ( A ® C) EB ( B® C) . (4) A , D ( m x m ) , B , E ( n x n ) , C, F ( p x p) :
( A EB B EB C ) ( D EB E EB F) = A D EB B E EB C F. (5) A ( m x m ) , B ( n x n ) :
( a ) l A EB Blabs = l A Iabs EB ! B labs · ( b ) ( A EB B)' = A' EB B' . ( c ) ( A EB B) H = A H EB B H . (d) A
87
-
-
B = A EB B . ( e ) ( A EB B ) - 1 = A - 1 EB B - 1 , if A and B are nonsingul ar . ( f ) ( A EB B) + = A + EB B + .
( 6) A ( m x m ) , B ( n x n ) :
( a) t r ( A EB B) = tr(A)
+ tr( B) .
( b) dg( A EB B) = dg( A ) EB dg( B ) . ( c ) rk( A EB B) = rk ( A ) + r k ( B ) .
( d ) det( A EB B) = det( A )det ( B ) .
( e ) ..\ ( A ) and ..\ ( B) are eigenvalues of A and B , respectively, with associated eigenvectors v ( A ) and v( B ) => ..\ ( A ) and ..\ ( B) are eigenvalues of A EB B with eigenvectors
respecti vely.
[
v(A ) O
]
and
[ 0 ] v
( B)
'
Note: All results of this section fol low easily from the defi n itions (see also Horn & J oh nson ( 1 985) ) .
3
Matrix Valued Functions of a Matrix In this chapter all matrices are assu med to
be complex matrices unless
otherwise stated . A l l rules for complex matrices also hold for real matrict>s as the latter may be regarded as special complex matrices. 3.1
T he Transpose
Definition: The ( n x
m) matrix
I
A' =
(n x
is the transpose of the ( m x n ) matrix A = (a ;j] . ( 1 ) A , B (m x n ) :
( A ± B)' = A' ± B' .
( m x n ) , c E
( 5) A, B (m x n) :
( A 8 B )' = A' 1::J B' .
( m x m) , B ( n x n ) : ( 7) A ( m x n ) : ( a ) I A' I abs = I A I�bs · (b) rk(A') = rk( A ) . (c) (A')' = A . (6) A
(A
E£l
B)' = A' E£l B' .
m)
24
H A N DBOOK O F M ATRIC eS
(d) (A')H
( A H )'
=
=
(e) A ' = A' = A H .
(8) A ( m x m)
A
.
: =
(a) dg(A' )
dg( A ) .
( b ) t r ( A' ) = tr( A ) . ( c ) d et ( A ' ) = det ( A ) . ( d ) ( A' ) - 1 = ( A - l ) 1 , if A is nonsingular. ( A adi ) ' . (e) ( A' ) ad; =
(f) >. is eigenvalue of A
(9) A ( m x
>. i s eigenvalues of A' . ( A ' ) + = ( A+ )'.
) rea l : n ) , 1\mn ( m n n
x ( 1 0 ) A (m x vee ( A' ) = 1\mn vee( A ) .
(11) A, B (m x n) : ( 12 ) A ( m x
rnn
) B ( m x p) , C
:
=>
( b ) A is syrn metric
x
n
) , D (q
:
x p) :
A' B'
C' D'
].
A' = A .
¢::::::}
( 1 4 ) A ( m x m ) nonsingular:
)
(q
= � r [� [
( a ) A is d iagonal
n
) commutation matrix:
vec ( B' )' vec ( A ) = tr( A B ) .
n ,
( 1 3 ) A ( m x m)
( 15) A (m x
=>
A' = A . A orthogonal
¢::::::}
A' = A - l .
A'A and AA' are symmetric m atrices .
Note: All these results are elementary. They fol low either directly from definitions or can be obtained by considering the individual elements of I he 111atrices involved (see, e.g., Lancaster & Tismenetsky ( 1 985) ) . 3.2
The Conj ugate
Definition: The ( m x n ) matrix A = [a;1] is the conjugate of the ( m x n ) matrix A = [ aij ] - Here aij denotes the complex conjugate of a; j . ( 1) A , B ( m x
(2) A (m x
n
)
-
:
A ± B = A ± B.
) B ( 11
n ,
( 3 ) A ( m x n ), c
E
-
x p) :
([' :
c
AB = AB.
-
A = cA. ·-
-
25
M ATRIX VA L U E D F U N CTIONS OF A M ATRIX
(4) (5) (6) ( 7)
B (P q) : A @- B =- A @ B. ) : A B = A B. - A EB B = A EB B. m), B ( ) A (m ) ( a ) I A i abs = I A i abs· ( b ) rk( A ) = rk( A ) . (c) A = A. ( d ) A' = A' = A H . (e) AH = A H = A'(f) vec ( A ) = vec ( A ) . A = A. (g) A is real
A (m A, B ( A (m
(8) A
(m
x
m
)
x
n ,
x
8
n
x
x
x
-
n
n
x
n
8
:
:
m) :
=;.
= det A .
(a) tr(A) = tr(A).
-
=A A = A - 1 , if A is nonsingular.
( b ) d et ( A )
adj
(C) A adj (d) - 1
.
(e) vech ( A ) = vech ( A) . ( f ) dg( A ) = dg( A) .
(g)
(9}
A(
m
A
A
is eigen value of with eigenvector with eigenvector v . x
n
),
B (m
( 1 0) a 1 , . . . , a m E Q:: :
x p) , C ( q x
n
) , D (q
-
v =? A
is eigenvalue of
A -
x p) :
[� �] =[� �]-
diag(a l , - - - , a m ) = diag( a l , - - - , a m ) -
Note: The resu lts in this section are easily obtained from basic definitions or element wise considerations and the fact that for c 1 , c2 E Q:: , c 1 c2 = i'1 c2 .
3.3
The Conj ugate Transp ose
AH _ A' = [a;jJ' ) ( A ± B)H = AH BH .
Definition: The conjugate t ranspose of the ( n x m ) matrix (1)
A, B ( m
x
n
:
±
(m
x
n
) matrix
A = [a;j ] is the
26
H A N D BOOK OF M ATRICES
( 2 ) A (m x 11 ) , B ( 11 x p) : ( 3 ) A ( m x n ) , c E
( A B) H = B H A H .
( cA ) H = cA H .
( 4 ) A ( m x n ), B (p x q) : (5) A, B (m x n) :
(A
�� '
(A 0 B ) H = A H 0 B H . B) H
=
A H (-;-) B H .
(.4 EB B ) H
(6) A ( m x m ) , B ( 11 x n ) :
(7) A (m x n ) : ( a) / A H /abs = / A /�bs · ( b ) rk ( A H ) = rk( A ) .
= AH
tfi
BH.
( c ) ( A H ) H = A. .
( d ) ( A.' ) H = A. = ( A H )1 • (e) :� H = A' = ( A ll ) . ( f) ( A. H ) + = ( A + ) H . (g) vec ( A 11 ) = vee( . ' ) A
( h ) A is real
=>
.
All =
.4 1 .
(8) A ( m x m) : ( a) t r ( A 11 ) = t. r(A: ) = tr( A ) . ( b ) det( A H ) = det ( "·l ) = det A . ( c ) ( A H ) a dj = ( JlOdi ) H .
( d ) ( A H ) - 1 = ( A - 1 ) H , if A is nonsingular. _H (e) dg( A ) = dg( .4 ) = ( dg A ) H
(9) A (m x n), B
( m x p) , C ( q x
[ (' D ] [ A A
( 10 )
G] ,
B
H
H BH
. . . , am E
(11) A (m x
m) :
[diag( a 1 , . . . , am )] H = diag( a 1 , . . . , ilm ) · .4 Hermitian ¢:=:;> A H = A .
( 12 ) A ( m x m ) nonsingular: (13) A
n ) . D ( q x p) :
(m x n ) :
A unitary
¢:=:;>
AH = A-1
(a) A H A and A A H are Hermitian positive semidefinite. ( b) rk( A ) = m ( c ) rk( A ) = n
=>
=>
A A H is Hermitian positive definite.
A H A is Hermitian positive definite.
Note: The results in this section are easily obtained from basic definit ions or clementwise considerations (see, e.g., Barnett ( 1 990 ) ) .
27
M ATRIX VA L U E D F U N CTIONS O F A M ATRIX
3.4
The Adjoint of a Sq uare Mat rix
Definition: For m > 1 , the ( m x m) m atrix A adj = (cof( a;1 )]' is the adjoint of the ( m x m) matrix A = [ a ;j ] . Here cof( a ;j ) is the cofactor of a ;j . For m- 1 A a dj = 1 . For i nstance, for m = 3 , '
[ aa3n2 a, - det [ 2 a3 2 al det [ 2 an det
A a dj =
( 1 ) A , B (m
X
] a!J ] a 33 a13 ] a 23
a23 a33
[ aa32 11 a1 det [ 1 a3 1 al l - det [ a2 1 - det
] a 13 ] a 33 a,J ]
a 23 a33
a 23
[ aa32 I1 [ aa3l i -det 1 a det [ 1 1 a2 1 det
an ] a3 2 a12 ] a 32 a12 ]
I
an
( A B) a dj = a adj A a dj . m ( 2 ) A ( m X m ) , c E
x
m) : ( a) A a dj = det( A ) A - 1 , if A is nonsingular. ( b ) A A a di = A a di A = det( A ) lm .
(3) A ( m
( c ) ( A ' ) a dj ( d ) ( A H ) a dj
=
( A adi ) ' . ( A a dj ) H .
=
(e) det(A a dj ) = ( det A) m - 1 . ( f) rk( A ) < m - 1 =:} A a dj = 0. ( 4) A ( m
x
m), m > 2 : m 1 0
if if if
rk( A ) = m rk( A ) = m - 1 rk( A ) < m - 1
Note: These results follow easily from basic principles and definitions ( see, e.g . , L ancaster & Tismenetsky ( 1 985) and M agnus & Neudecker ( 1 988 ) ) . 3.5
The I nverse of a Sq uare Mat rix
Definition: An ( m x m ) matrix A - 1 is the inverse of the ( m x m) matrix A i f A A - 1 = A - 1 A Im . =
3.5.1
General Result s
( 1 ) A , B ( m x m) n onsingular :
(AB)- 1 = s- 1 A- 1 .
28
H A N DBOOK O F M AT RI C ES c
( 2 ) A ( m x m) nonsingular,
E
( 3 ) A ( m x m ) nonsingular, B ( n x
n
(a) ( A 0 B ) - 1 = A- 1 0 8 - 1 •
( b ) ( A ifl B ) - 1 = A - 1
ifl
(cA ) - 1 = c- 1 A - 1 .
) nonsingular:
s- 1 .
(4) A ( m x m ) nonsingular: l
(a) .4 - 1 =
A adj .
det ( A ) ( b ) ( A ' ) - I = ( A - I )' . ( c) ( A ) - 1 = A - 1 . ( d ) ( AH )- 1 = ( A - I )H . (e) ( A.adj ) - I =
l
det ( A )
A.
l
( f ) det ( A - I ) =
det ( A )
(g) >. is eigenvalue of A with eigenvector .4 - 1 with eigenvector v . ( 5 ) .4 ( m x m ) :
¢:=:;>
( a) rk( A ) = m
(b) rk(A) =
( c ) A is diagonal dominant
(6) a1 ,
.
.
•
, am
"4
E Q' . a;
. d = J ag ( a , , . . . , am )
( a) A ort hogonal A unitary
� is eigenvalue of
=?
A is nonsingular.
=j:. 0 for i = 1 , . . . m :
( 7 ) Irn ( rn x m ) identity matrix: ( 8 ) A ( m x m ) nonsingular:
(b)
=?
A - 1 exists .
A+ = A-1.
=?
m
v
{=::::::>
¢:=:;>
=:} .4 - 1
= d'Jag ( a -1 1 ,
- 1 ).
. . . , am
Im ' = Im .
A - 1 = A'.
A - 1 = AH .
( 9 ) A ( m x m) positive definite:
( .4 1 1 2 ) - 1 is a square root of A - 1 .
Note: M any of these results are standard rules which can be found 1 1 1 introductory text books such as Barnett ( 1 990) and Horn & J ohnson ( 1 985 ) or follow immediately from definitions. 3.5.2
Inverses Involving Sums and D ifferences
( 1 ) .4 ( m
x
m ) with eigenvalues >. , , . . . A ,
m ,
j .>.; /abs < 1 , i =
l,
... m ,
:
M ATRIX VA L U E D F U N CTIONS OF A M ATRIX
(a) Um + A ) - 1 =
29
00
:2:) - A ( i=O 00
i=O
00
i=O 00
{2) A ( m x m), B ( m x n) , C (n x m ) , D (n x n) :
if all involved i nverses exist . (3) A (m x n) :
(4) A ( m x m ) nonsingular, B (m x m) : ( A + B B H ) . Um + B H A - 1 8 ) nonsingular => ( A + B B H ) - 1 B = A - 1 B ( /m + B H A - 1 B ) - 1 . m
(5) A, B ( m x
) nonsingular:
(a) A - 1 + B - 1 = A- 1 ( A + B ) B - 1 . ( b ) A - 1 + B - 1 nonsingular => ( A - 1 + B - ' ) - ' = A( A + B ) - 1 B = B ( A + B ) - 1 A . ( c ) A - 1 + B - 1 = ( A + B ) - 1 => A B - 1 A = B A - 1 B .
(6) A . B ( m x m ) :
(a) Im + A B nonsingular => ( /m + A B ) - 1 A = A ( Im + B A ) - 1 . ( b ) A + B nonsingular => A - A ( A + B ) - 1 A = B - B ( A + B ) - 1 B . Note: Most resul ts of this subsection may be found in Searl ( 1 982, Chapter 5 ) or follow from results given there. For ( 1 ) see Section 5.4. 3.5.3
Partitioned Inverses
(1) A ( m x
)
m ,
H
(m x
n
), C (n x m). D (n x n) :
30
H A N D BO O K O F !>J AT R ! C E S
[ (_� � ] A and ( D - CA - I B) nonsingular � [ � � rl ( D - CA - 1 B ) - 1 CA- 1 - A - 1 B ( D - CA- 1 B ) - 1 ] [ A - 1 +-A( D- 1-B CA. 1 B ) - 1 CA- 1 ( D - CA - 1 8 ) - 1 ,
( 2 ) A ( m x m) , B ( m x n ) . C ( n x m ) , D ( n x n ) :
- ( A - BD- ' C ) - ' B D - 1 D- 1 + D- 1 C( A - BD- 1 C ) - 1 B D - 1
].
( 3 ) A ( m x m ) symmetric, B ( m x n ) , C ( m x p ) , D ( n x n ) symmetric. f�' (p x p) symmetric : A B' C'
B D 0
C 0 E
-I
-FBD- 1 - FCE- 1 D- 1 B' FC E - 1 D- I + D - I B' F B D- 1 E - 1 C' F B D - 1 E - 1 + E - 1 C' FCE- 1 i f all inverses exist and F = ( A - B D- 1 B' - C E- 1 C') - 1 . F - D - 1 B' F - E - 1 C' F
( 4) A; ( m; x m; ) nonsingular, i
=
1, .
-I
0
m,
0
n E IN , m > n , A ( m x n ) , B ( m x ( m - n ) ) : rk( A ) n , rk( B ) = m - n , A H B =
� [A : B) - 1 (6)
m,
, r :
0
0 (5)
. .
=
[ ((BAHH AB )) -- !! ABHH ] . =
0
n E IN , m < n , A ( m x n ) , B ( ( n - m) x n ) : rk ( A) = m, rk( B ) n - m, A B H = 0
�
[ � r'
=
=
[A H ( A A H ) - ' = B H ( B B H ) - ' ] .
31
M ATRIX VA L U E D F U N CTIONS OF A M ATRIX
Note: The inverses of the partitioned matrices in ( 1 ) - ( 3 ) are given in M agnus & N eudecker ( 1 988, Chapter 1 ) . The other result s are straightforward consequences of the definition of t he inverse. 3.5.4
Inverses Involving Commutation, Duplication and Elimination Matrices
Reminder: •
•
•
l\mn or l\m,n denotes an ( mn x mn ) commutation matrix ( for det ails, see Section 9 . 2 ) . 2 Dm denotes the ( m x � m ( m + 1 ) ) duplication matrix (see Section 9.5). Lm denotes the ( � m ( m + 1 ) x m 2 ) elimination matrix (see Sectiou 9.6 ) .
- ' = r\ nm · ( 1 ) r\ mn ' - D+ ( 2 ) ( D'm Dm ) - 1 n m D+ r · ( 3 ) ( D:n 1 Dm + l ) - 1 + ( 4 ) ( Lrn L'rr, ) - 1
=
=
1 0 0
0 � lm 0
0 0 ( D'm Dm )- 1
Lrn L'rr, .
( 5 ) A ( m x m ) nonsingular :
[D'm (A ® A ) Dm ] - '
=
' Dt, ( A - 1 ;) A - 1 ) Dt, .
(6) A ( m x m ) , lm 0 A + A ® lm nonsingular: (a ) [ D;t', ( A ® lm ) Dm J - 1 = 2D;t', ( lm 0 A + A 0 lm ) - 1 Dm .
( b ) [ D;t', ( lm 0 A + A 0 lm ) DmJ - 1
=
D;t', ( lm
(7) A , B ( m x m ) , A @ B + B 0 A nonsi ngular:
[D;!; ( A 0 B ) Dm t '
=
0
A + A 0 lm ) - 1 Dm .
2D;!; ( A 0 B + B 0 A ) - 1 Dm .
(8) A ( m x m ) real symmetric nonsingular ,
c
E 1R :
{ D;t', [A 0 A + c vec ( A ) vec ( A )']Dm } - 1
=
[
D;!; A - 1 0 A - 1 -
c
1+
]
ern vec ( A - 1 ) vec ( A - 1 )' Drn.
( 9 ) A , B ( m x m ) real , lower triangular, i n vertible : [Lrn ( A' 0 B ) L'rr, t '
=
Lm ( ( A' ) - 1
( 1 0 ) A ( m x m ) real , lower triangular, invertible,
0
c
B- 1 ) L '.r, .
E 1R :
{ L m [A ' 0 A + c vec ( A ) vec(A' )'] L;n } - 1
,'}•))
H A N D BOOK O F
M AT R I C ES
Note: The results of this subsection may be found in M agnus ( 1 988 ) .
3.6
Generalized Inverses
Definition: A n ( 71 x m ) matrix A- is a generalized inverse of the ( m x n ) matrix A if i t satisfies A A - A = A . 3.6.1
General Result s
(1) A ( m x m ) :
rk ( A )
=
m � A- = A- t .
(2) A ( m x n) : ( a) A - is not unique in general. ( b ) A A - and A - A are idempotent. ( c ) !,. - A A - and In - A - A are idempotent.
( d ) A ( A H .·\ ) - A_H is idempotent.
(3) A ( m x n ) :
(a) rk ( A ) = rk( A - A ) = rk( A A - ) . ( b ) rk ( A ) = tr ( A - A ) = tr(A A - ) . ( c ) r k ( A A - ) = tr( AA - ) .
( d ) rk(A - ) > rk( A ) . ( e ) rk( A - ) = rk ( A ) � A - A A - = A - . (f) rk ( A ) = n ¢:::=> :1 - A = ln . ( g ) rk ( A )
=
m � A A - = lm .
(4) A ( m x n) : (a) A ( AH A ) - A H A = A . ( b ) A H A (A /{ A ) - A H = AH . ( c ) A (A H A ) - A H is Hermitian. ( d ) ( A_ - ) H is a generalized inverse of A H . ( 5 ) On x m is a generalized inverse of Om x n · ( 6) A ( m x m )
:
A is i dem potent
=>
A is a generalized inverse of itself.
( 7 ) A ( m x n ) , B ( n x m) : A - is a generalized inverse of A :1 - + B - A - A R A A - is a generalized inverse of A .
=>
33
M ATRL : VA L U E D F U N CTIONS OF A M AT RI X A - is a generalized inverse of ( 8) A ( m x n ) , B , C ( n x m) : A => A - + B ( lm - A A - ) + ( In - A - A )C is a general ized i nverse of A.
( 9 ) A (m x n ) , B (m x m) , C ( n x n ) : B , C nonsi ngular => c- 1 A - s - 1 is a generalized inverse of BAC.
( 10) A (m
X
n ) , B (P X q ) :
A - ® s - is a generalized inverse of A l:) B .
( 1 1 ) A (m x n ) , B ( m x r ) , C ( m x r ) : Generalized inverse matrices A - and c- of A and C, respectively, exist such that .4 .4 - Be- C = B can be solved for X and B => the system of equations AXC X = A - sc- + y - A - A Ycc- is a solution for any ( n X m ) matrix =
Y.
Partitioned Matrices ( 12 ) A ( m x m ) nonsingular, B ( m x n) , C ( r x m ) , D ( r x n ) :
= m ' D = CA - 1 B ] D ] a genera 1zed .mverse of [ C [ On �]. B
=>
4-I xm
Dm x r On x r
( 13 ) A ( m x n ) , rk( A ) = r : such that BAC =
=>
.
IS
1.
B (m x m ) , C (n x
[6 �] C[ � � ]
A
11
) are nonsingular and
B is a generalized inverse of .4 ,
for any D ( r x ( n - r )) , E ( ( m - r ) x r ) , F ( ( m - r ) x ( n - r ) ) .
( 1 4 ) A ( m; x n, ) , i = 1 , . . . , r :
0
0 is a general ized inverse of
0
0
Note: A number of books on generalized inverses exist which contain the foregoing results and more on generalized inverses ( e .g . , Rao & �I i tra ( 1 91 1 ) . Pringle & Rayner ( 1 97 1 ) , Boullion & Odell ( 1 97 1 ) , Ben- Israel & GreYille ( 1 974 ) ) . Some of these books contain also extensive lists of references .
34
H A N D BOOK OF M AT R I C ES
3.6.2
The Moore -Penrose Inverse
Definition: The ( n x m ) matrix A+ is t he Moore -Penrose (generalized ) i nverse of the ( m x n ) matrix A i f it satisfies the following four conditions: (i) AA+A = A ,
( ii ) A + A A + = A+ , ( ii i ) ( A A + ) H = A A + , (iv)
(A + A ) H = A + A .
Properties ( 1 ) A (m x n ) :
A + exists and is unique.
( 2 ) .-1 ( m x n ) : A = cH
[ g � ] v is the singular value decomposi tion of A => [ � ] U. A+ = VH
( 3) 1 ( m x .
-
11
),
U
( m x m), U,
F
�·
o- 1 0
(n x n) :
unitary => ( U A V ) + = V H A + U H .
( 4 ) A ( m x n ) , rk( A ) = r :
B
(m X r ) , c ( r X n ) such that A = BC => A+ = c+ B + .
if c # 0 if c = 0
( 5 ) c E Q' : ( 6 ) A ( m x n ) , c E
( cA ) +
( 7 ) A ( m x n ) , B ( t• x s ) :
( 8)
A ( m x m) :
=
c+ A + .
(A 0 B)+ = A+
0
B+
(a) A is nonsingular => A + = A - 1 .
( b ) A is Hermit ian => A + is Hermitian .
(c )
A
is Hermit ian and idempotent => A+ = A .
(9) A ( m x n ) :
(a) ( A + ) + = A . (b) ( A H )+ = ( A + ) H .
35
M ATRIX VA L U ED F U N CTIONS O F A M ATRIX
(c) A H A A + = A H .
(d) A + AAH = A H . (e) A H ( A + ) H A + = A + . (f) A + ( A + ) H A H = A + . (g) ( A H A ) + = A + ( A + ) H . (h) (AAH )+
==
(A+)HA+.
{i) A ( A H A ) + A H A == A . (j ) A A H ( A A H ) + A == A . (k) A + = ( A H A ) + A H = A H ( A A H ) + . { 10) A ( m
x
n) :
(a) rk( A ) =
m
( b ) rk( A ) = n
{=:::::}
A A + = lm .
{=:::::}
A + A = ln .
( c ) rk( A ) = n => A + = ( A H A ) - 1 A H .
( d ) rk( A ) = m (e) r k ( A ) = n
=>
A+ = A H (AAH )- 1
=>
(AAH )+ = A (AHA ) -2A H . (f) rk( A ) = 1 => A + = [tr( A A H ) ] - 1 A H .
(g) A = Om x n { 1 1 ) A (m
x
{=:::::}
A + = On x m ·
n) :
(a) rk( A + ) = rk ( A ) .
( b ) rk(A A + ) = rk(A+ A ) == rk ( A ) . ( c ) t r ( A A + ) == rk( A ) .
{ 1 2 ) A (m
x
n) :
(a) A A + and A + A are idempoten t .
(b) lm - A A + and In - A + A are idempotent. ( 13 ) A ( m
{ 14 ) A ( m
X
n) , B (n
X
n) , B
( 15 ) A , B ( m
x
(m
X r
)
X r
:
)
A B = Om x r :
{=:::::}
A H B == On x r
n ) , A BH = 0 :
B + A + = Or x m ·
{=:::::}
A + B = On x r ·
( A + B )+ = A + + ( in - A+ B ) [C+ + Un - c+c ) M B H ( A + ) H A + ( lm - BC + ) j ,
where C == ( /m - A A + ) B and
:l6 ( 16 )
H A N D BOOK O F M AT R I C ES A
(m x
n
), B ( n x
r
), C ( m x
r
AH A B = A H C
(17)
A
(m x
B (n x
)
n ,
r
)
:
)
:
<===>
AB = AA+c.
det( B B H ) # 0
=>
A B ( A B )+ = A A + .
(18 ) .4 ( m x m ) Hermitian idempotent, B ( m x n) :
=> A - B B+ is Hermitian idempotent with rk ( A - B B+ ) = rk( A ) - rk( B ) .
(a) A B = B
( b ) A B = 0 and r k ( A ) + rk( B ) = m => A = lm - B B + . (19) n ( m
X
m ) Hermitian
p os i t i ve
defi n ite, A ( m X n ) :
AHn- ' A(AH n- ' A ) + AH
= AH.
( 20 ) :1 ( m x m ) Hermitian : ,\ # 0 is eigenvalue of A with associated I is Pigenvalue of A+ w i th associated eigenwct or e i gen v ec t or ! ' => ,\ !• .
Partitioned Matrices
( 2 1 ) A; ( m; x n, ) ,
i
=
1, . . . . r
:
+
0 0 ( 22 ) A ( m x
n
)
Ar
:
[� r [� 0
,
·
Bj+ -
_
A r+
. � [ [ ]' � �r � ] [ + D) ] D
A+
( 2 3 ) A. ( m x n ) , B ( m x p) :
( 4. .
0
0
=
A
0
+
=
A + - A + B( c+ c+ +
•
where C = Um - A A + ) B an d D ( 24)
:1
= (m x
( lp - c + C ) [ lp + ( lp - c + C) B H ( A + ) H A + B ( lp - c+C) J - 1 x BH ( A + ) H A + ( lm - BC+ ) .
n
). B
(p x
n):
[�r
= [A + - TBA+ : T] ,
where T = £+ + ( !,.. - £ + B ) A + ( A + ) H BH I\ ( lp - E E + ) with E = B( l,. - A + A ) and I\ = [lr + ( lp - E E+ ) B A + ( A+ ) H BH ( lp - E E + )J - 1 •
37
M ATRIX VA L U ED F U NCTIO NS OF A M ATRIX
Note: These results on Moore - Penrose generalized inverses are also contained i n books on generalized inverses such as Rao & M itra ( 1 97 1 ) , Pri ngle & Rayner ( 1 97 1 ) Boullion & Odell ( 1 97 1 ) , Ben-Israel & Greville ( 1974 ) . A good collecti on of resul ts is also contai ned in M agnus & N eudecker ( 1 988), including many of t hose given here. M any of the results fol low easily by verifying the defining properties of a Moore -Penrose inverse. ,
3.7
Matrix Powers
Definition: For i E ll. , the ith power of the A' , is defined as follows:
(m
x m
) m at rix A, denoted by
1
for positive integers i
A
'
=
for i = 0
-I
-I
IT A
for negative integers i , if det( A ) # 0.
j=l If
A can
be written
as
0
A=U
0
for some unitary m atrix U (see Chapter 6 ) then the power of A is defined for any n E IR, o > 0 , as fol lows: 0
A" = U 0 This definition app l ies, for instance, for Hermitian matrices . The definit ions are equivalent for integer values of o . Properties
(1)
A(
m x m
), c E
(C , i E IN :
( c A ); = c1 A i .
( 2 ) (Binomial formul a) A, B ( m x m ) , i E IN , i > 1 :
( A + B) i
I
=
L L A k , BA k 2 B . . . A k, B AkJ+l .
j =O
38
H A N DBOOK OF M ATRICES
where the second sum is taken over all k 1 , . . . , kj + l E { 0 , . . . , i } with kl + · · · + ki + l = i - j . ( 3 ) ( Binomial formula) A , B ( m x m ) : ( A + B) 2 = A 2 + A B + BA + B 2 . ( 4 ) ( Binomial formula for commuting matrices) A , B ( m x m ) , i E IN , i > 1 : AB = BA => ( A + B) ;
::::
t (�) Ai s i-i J
.
; =0
.
( 5 ) A , B ( m x m ) , i E IN , i > 1 :
i- 1 j Aj - Bj = LA ( A - B ) B i - 1 - j . j= O (6) A (m
x
m ) , B ( n x n ) , i E IN :
(a) ( A 0 B) ; = A ; 0 B ; . (b) ( A
ffi
B) ; =
[ � � r :::: [ �;
�;
] :::: A;
ffi
Bi .
(7) A ( m x m) , i E IN :
(a) J Ai l abs < J A I �b · ( b) ( A ; )' :::: (A'( ( c ) ( A i ) H :::: ( A H ( ( d ) Ai :::: A ; . (e) ( A i ) - 1 = ( A - I ) ; . (f) rk(A' ) < rk (A). (g) det ( A ' ) :::: ( det A ) ' . s
.
.
tr( A i ) = vec ( Ai f2 1 ) 1 vec( A i f2 ) . lm . m ) identity matrix , i E IN : r:,
( 8 ) A ( m x m ) , i E IN even : ( 9 ) lm ( m
X
( 10 ) i E IN , i -f; O :
=
O ;, x m = Om x m ·
39
M ATRIX VA L U E D F U N CTIONS OF A M ATRIX
( 1 1 ) i E IN : A;
>. 0
>.
0 0
0 0
1
0 1
'
0 0
>.
1
>.
(m x n )
-1 ) >.i (�
0
>. i
0
0
0
0
m+l ( m i- 1 ) >. • m+ 2 ( i ) >.i m- 2
( 12 ) A ( m x m ) : ( a) A is idempotent => Ai = A for i = 1 , 2 , . . . ( b ) A is nilpotent ( c ) A is symmetric ( d ) A is Hermitian ( 13 ) A ( m x m ) , i E IN : vec ( A' )
( 14 ) A ( m x m) : A ' - i -oo 0
<==>
A' = Om x m for some i > 0.
=>
=>
Ai is symmetric for i = 1 , 2 , . . . Ai is H ermitian for i = 1 , 2 , . . .
( /m 0 A' ) vec ( lm ) ( ( A i ) ' 0 lm ) vec( Im ) ( ( A i f 2 ) 0 Ai f 2 ) vec ( Im ) , if A is posit i vi' definite ( ( A i/2 ) ' 0 Ai f 2 ) vec( Im ) , if i i s even ( A' 0 A )vec( A i - 2 ) , if i > 2 .
<==>
all eigenvalues of A h ave modulus less than 1 .
Note: M ost results of this section are basic and follow immediately from definitions. The binomial formulae m ay be found in Johansen ( 1 995. Sec t ion A .2) and ( 1 3 ) is a consequence of an important rel ation bet ween Kroneckt>r products and the vee operator ( see Section 2 . 4 ) .
3.8
T he A bsolute Value
Definition: G iven an ( m x n ) matrix A = (a;j] its absolute value or modulus . IS
! a l l labs
la2 1labs
Ia dabs Ian labs
la1n labs la2n labs
(m x n )
H A N DBOOK OF M AT RICES
40
where t he modulus of a complex number c = c1 + i c 2 is defined as lclabs Jci + c� = .jCC . Here c is the complex conj ugate of c.
=
( 1 ) A (m x n ) : ( a) I A iabs > Om xn · ( b ) I A i abs = Om xn {::::::} A = Om xn · ( 2 ) A ( m X n ) , c E ([, : lcAiabs = l c l abs I Ai abs · ( 3 ) A , B ( m X n ) : l A + B labs < I A i abs + ! B l abs · ( 4 ) A ( m X n ) , B ( n X p) : I AB i abs < I A i abs I B labs · ( 5 ) .4 ( m X m ) , i E IN : lA; Iabs < I A I� bs · l A 0 B l abs = I A i abs ® ! B l abs · (6) A (m X n) , B (p X q) (7) A ( m X m ) , B ( n X n ) : l A 81 B l abs = I A i abs 81 ! B l abs · (8) A ( m x n) : ( a) I A' I abs = I A I�bs · (b) I Alabs = I A l abs · ( c ) I A H iabs = I A I�bs · ( 9 ) A ( m x n ) , B ( m x p ) , C ( q x n ) , D (q x p) : :
[ CA DB ]
_
abs -
[ I CAiabs
! B labs I i abs I D iabs
].
These results may be found in Horn & Johnson ( 1985, Chapter 8 ) or follow easily from the definition of the absolute value. Note:
4
Trace, Determinant and Rank of a Matrix 4.1
The Trace
Definition: The trace of an ( m x m) matrix A = [a ;j ] is defined
trA = t r ( A ) 4. 1 . 1
(3) (4)
=
(6)
(7) (8)
=
L a;; . i= l
tr( A ± B ) = tr(A ) ± tr( B ) . A ( m x m), c E
tr(A) = rk( A ) . A ( m x m) with eigenvalues A 1 , . . . , Am : tr( A ) = A 1 + . . . + Am A (m x n ) , B ( n x m) : (a) tr( A B ) = tr( BA). ( b ) tr(AB) = vec (A')'vec(B) = vec(B') ' vec(A ) . =
(5)
a 1 1 + · · · + amm
m
General Results
( 1 ) A , B ( m x m) : (2)
=
as
42
H A N DBOOK O F M AT R IC ES
( 9 ) A ( m x n ) , B (n x p) , C (p x q ) , D ( q x m) :
vec( D' )' (C' 0 A ) vec( B ) vec(A' )' ( D ' 0 B)vec(C) vec( B' )' ( A' 0 C)vec( D) vec(C' )' ( B ' 0 D)vec ( A ) .
t r( A BC D)
B nonsingular => tr ( BAB- 1 ) = tr( A ) . ( 1 1 ) A , B ( m x n ) , j k = ( 1 , . . . , 1 )' (k x 1 ) : tr( AB') = j:r, ( A 0 B)jn . ( 1 2 ) A , B, D ( m x m) , j = ( 1 , . . . , 1 )' ( m x 1 ) :
( 10 ) A , B ( m x m) :
tr( A D B' D) = j ' D' ( A :�.' B ) Dj.
D diagonal ( 1 3 ) A , B, C ( m x n ) :
tr[A' ( B (-:) C)] = tr((A' >-:1 B' )C] . ( 14 ) A ( m x m) : tr(A ��� lm ) = tr(A) . ( 15 ) A (m x m ) , B ( n x n) : ( a) tr(A 0 B) = tr( A )tr( B). ( b ) tr(A oil B ) = tr( A ) + tr( B). ( 16 ) A (m x m ). B ( m x n ) , C ( n x m ) , D ( n x n ) : tr
[ � � ] = tr( A)
+
tr( D ) .
tr( /\mm ) = m. ( 18 ) Drn (m2 x � m ( m x 1 ) ) duplication matri x : ( a) tr( D:r, Dm ) = tr( Dm o:r, ) = m2. ( b ) tr( D:,. Dm ) - 1 = m( m + 3 )/4. ( 19) Lrn ( � m( m + 1 ) x m2 ) elimination matrix :
( 1 7 ) l\.mm ( m2 x m 2 ) commu tation matri x :
tr( Lm L:r, ) = tr( L:r, L m ) = � m( m + I ) .
( 20 ) A =
[a;1], B ( rn x m) real positive semidefinite, tr( A 0 ) =
m
L a� i= I
<===>
a
E Dl, a > 0 , a f. I :
A is diagonaL
( 2 1 ) A , B f. 0 ( m x m) real positive semidefinite, a E Dl, 0 < a < I :
tr( A 0 B 1 - o )
=
(tr A t (tr B ) 1 - o
<===>
B = cA for some c E
Dl, c > 0 .
43
TRACE, D ET ER M I N A NT A N D R A N K OF A M ATRIX
( 22 )
A, B (m
x m
) real positive semidefinite, a E IR, a > 1 :
[tr (A + B) a ] i f a = (tr A " ) i f a + (tr B a ) i f a {::::::} B = cA for some c E IR , c > 0.
The rules involving the vee operator, the commutation , duplication and elimination matrices are given in Magnus & Neudecker ( 1 988) and Magnus ( 1 98 8 ) . (20) - (22) are given in Magnus & Neudecker ( 1 988, Chapter 1 1 ) . The other rules follow from basic principles. Note:
4.1.2
Inequalities Involving the Trace
In this subsection all matrices are real unless otherwise stated. A ( m x n) complex : (a) l tr A labs < tr lA I abs · (b) tr(A H A ) = tr(AA H ) > 0 . (2) A, B (m x n) : (a) (Cauchy - Schwarz inequ ality) tr(A' B ) 2 < tr(A' A)tr(B' B ) . ( b ) tr(A' B ) 2 < tr(A' A B' B ) . (c) tr(A' B ) 2 < tr ( A A' BB' ) . ( 3 ) (Schur's inequal ity) A ( m x m ) : tr(A 2 ) < tr(A' A ) . ( 4 ) A ( m x m ) positive semidefinite: (det A ) 1 f m < ,!. tr(A ) . (5) A ( m x m ) : All eigenvalues of A are real => I ,!. tr( A) Iabs < [,!. tr( A 2 )] 1 12 . (6) A ( m x m ) symmetric with eigenvalues ) q < · · < Am , X ( m with X'X = In : n n LA; < tr( X' AX) < LAm-n+i . i=l (1)
·
(7) A = [a;j ] ( m
x m
) positive semidefinite:
tr( Aa ) (8) A, B =f:. 0 ( m
x m
{
>
- 'C""' L.... ':l a =l a� for (} > 1 < 2:: ;" a � for O < a < l . 1 u
) positive semidefinite, a E IR :
x
n)
H A N DBOOK OF M ATRICES
44
( a) ( Holder's inequality ) 0 < a < 1 => tr(A0 B 1 - 0 ) < (tr A)" ( tr B ) 1 - <> . ( b ) a > 1 , /3 a/( a - 1 ) => tr( AB) < ( tr A0 ) 1 1 ° ( tr BP ) 1 /i3 . ( c ) ( Minkowski 's inequality) a > 1 => [tr( A + B) 0 j l /<> < (tr A 0 ) L/ o + (tr B 0 ) 1 1 <> . (9) A , B ( m x m ) positive semidefinite: ( a) E IR. 0 < a < 1 => tr( A0 B 1 - 0 ) < tr( a A + ( 1 - a ) B ) . ( b) ,� tr( A B ) > ( det A ) 1 1"' ( det B ) 1 f m . ( 10 ) A ( m x m ) positive definite: ln det( A) < tr( A ) ( 1 1 ) A ( m x m) posit ive definite, B ( n x m), C ( m x n ) : BC = In => tr(C ' AC) > t r( BA - 1 B' ) - 1 . =
n
m.
( 1 2 ) A ( m x m ) posit ive semidefinite with maximum eigenvalue Ama .r ( A ) . B (n x m) : ( 13) A ( m x m)
tr( BAB ' ) < A m ax ( A)tr( B B ' ) . positive semidefinite with maximum eigenvalue A m ax ( A )
:
?
tr( k ) < A m a x (A)tr( A). ( 14) ..\ . B ( m x m ) : t r[( A + R ) ( A + B)'] < 2 [tr( AA' ) + tr( BB')]. ( 1 5 ) A ( 1 11 x m ) posit ive definite, B ( m x m) positive semidefinite: ex p (tr( A - 1 B)] > ( 16)
(17)
( 18 ) ( 19) ( 20 )
det( A + B) . det( A )
A,
B ( m x m ) posit i ve semidefinite: ( a) tr( A B ) < tr( A c_:::; B ) . ( b ) t r( A B ) < � ( tr A + tr B) 2 . (c) t r( A ·2 B ) < � ( t.r A + tr B j'l . ( d ) tr( A , ;, B) > 0 . ( e ) tr( A • ·.· B ) < tr(A 0 B ) . A ( m x m ) , B ( 11 x n ) positive semidefinite : tr( A. 0 B) > 0 . A , lJ , C, D ( m x m ) positive semidefinite: C - A , 0 - B positive semidefin ite => tr( A B ) < tr(C D). A . B ( m x m ) symmetric: tr( A B ) < � tr( A2 + B2 ) . A, B ( m x m ) : tr( A 0 B) < � tr( A 0 A + B 1;;) B ) , -
-
45
TRACE, D ET ERM I N A N T A N D R A N K OF A M ATRIX
tr(A �� B) < � tr(A C:' A + B r:;:: B). Note: Inequalities ( 1 6 ) - (20) are from Neudecker & Shuangzhe ( 1 993) (see also Neudecker & Shuangzhe ( 1 995) ) . The other inequalities can be found in Chapter 1 1 of Magnus & Neudecker ( 1 988 ) . 4. 1 . 3
Optimization of Functions Involving the Trace
In this subsection again all matrices are real if not otherwise stated . ( 1 ) n ( m X m ) real sym metric with eigenvalues ) q < . . . < A m and associated orthonormal ( m x 1 ) eigenvectors v1 , . . . , 1'm , n E { l , . . . , m} :
min{ tr(B'fl B) : B ( m x n ) real , B ' B = /, } = A 1 + · · · + A, . The m inimizing matrix is B
=
[v1 , . . . , v, ] .
max{tr( B'flB) : B ( m x n ) real, B ' B = /, } = A m + · · · + A m - n + I · The maximizing matrix is B [ vm , . . . , Vm - n + d · ( 2 ) n ( m X m ) complex Hermitian with eigenvalues A t < . . . and associated orthonormal ( m x I ) eigenvectors r 1 , . . . { 1 , . . . , m} : =
< Am
, l'm .
min{tr( B H Q B ) : B ( m x n ) complex, B H B The m inimizing matrix is B
=
=
/, }
=
n E
A 1 + · · · + A, .
( v1 , . . . , v,] .
max{ tr( B H flB) : B ( m x n ) complex, R H H = /11 } =
Am + · · · + A m - n + l ·
The maximizing matrix is B [vm , . . . , t'm - n + t l · ( 3 ) n ( m X m ) positive definite with eigenvalues A t < . . . < Anl and associated orthono.-mal eigenvectors v1 , . . . , Vm . 0 < r < m : =
min{t r(fl - A ) 2
:
A ( m x m) positive semidefinite, rk( A ) = r } A T + · · + A �, -r · =
The m inimum is attained for A
=
m
L
i= - r+ l m
A, v, v: .
H A N D BOO K OF M ATRICES
46
( 4 ) X (m x n ) ,
)q
· · ·
eigenvalues of X' X with associated orthonormal eigenvectors V t , . . . , v, , 0 < r < n : <
< >.,
min{ tr[( X - A B ) ( X - A B )'] : A ( m x r ), B ( r x n ) with BB' = Ir } = A t + A 2 + · · · + A m -r . The minimum is attained for B (5)
Y
(m
=
[v, ,
.
.
. , Vn - r + d ' and A = X B'.
x n ) , X (p x n) with rk( X ) = p, Q ( m x m) positive definite, A t < · · · < A m eigenvalues of Q t f 2 Y X'(X X')- t X Y'nt / 2 ' with associated orthonormal eigenvectors V t , . . . , Vm , I < r < m < p : min{ tr[( Y - A B X )' fl(Y - ABX )] : A ( m x r ) , B ( r x p), rk( A ) = rk( B ) = r} = tr[( Y - A B X )'fl(Y - A BX )] , where
A = Q - t / 2 [v m , . . . , V m - r + l ]
and (6)
y
(m
X
B = [v m , . . . , V m - r + ! ]'Q t f 2YX'(X X ' ) - 1 . n ) , X (p X n ) , rk( X ) = p, n ( m X m ) positive definite:
min{ t r[( Y - AX )'fl(Y - AX )] : A ( m x p ) } = tr(QYY' - flY X'(X X' )- t XY' ) . The minimum is attained for A = YX'(XX' ) - t . ( 7 ) y ( m X n ) , X ( P X n ) with rk ( X ) = p, n ( m X m ) positive definite, R (q x m ) with rk( R) = q , C ( q x p ): rnin{ tr[(Y - AX )'fl(Y - A X )] : A ( m x p), RA = C} = tr[(Y - AX )'fl(Y - A X )] , -
.
where ( 8 ) B , C ( m x m ) pos itive definite, B diagonal, A t < · · · < Am eigenvalues of s - t f2C s - t / 2 with associated orthonormal eigenvec tors V t , . . . , V m , I < r < m : min{ln det(AA' + B) + tr [(AA' + B ) - t C] : A ( m x r ) } = m + In det (C) + L:;r'� r ( .A; - In A ; - I ) .
47
T RACE, D ET ERM I N A NT A N D R A N K OF A M ATRIX
The m inimum is attained for 0
1 /2
0 (9) B ( m x m)
positive definite: m
in { tr( B - 1 A) - In ldet( B - 1 A) l abs : A ( m x m) positive semidefinite} = m.
The m inimum is attained for A = B . The results of this subsection are partly given in Magnus & Neudecker ( 1 988, Chapter 1 7), Horn & Johnson ( 1 985, Chapter 4 ) and Liitkepohl ( 199 1 , Section A . 1 4 ) . Result (9) is from Johansen ( 1995, Section Note:
A. ! ).
4.2
The Determinant
Definition:
The determinant of the ( m x m ) matrix A = (a ;j ] is defined
as
where the sum is taken over all products consisting of precisely one element from each row and each column of A multiplied by - I or 1 , if the permutation i 1 , . . . , i m is odd or even , respectively. 4 .2. 1
General Results
(1) ( 2 ) A = [a;j ] ( m x m) , m > 2 :
det ( A )
a ; 1 cof(ai i ) + a 1 j cof(a 1 j ) +
· · · · ·
+ a ; m cof( a i m ) · + a mj cof(a mj )
for any i, j E { 1 , . . . , m } . Here cof( a;i ) denotes the cofacor of a ;j (see Section 1 . 4). ( 3 ) A (m x m), c E
18
H A N DBOOK O F M ATRICES
( a) det ( A B ) = det ( A )det ( B ) .
( h ) B is n o n si ng u l ar
=>
( 5 ) !, ( m x m ) i d e n t i ty matrix: (6) A (m
x m
det ( /111 )
x m
=
)
det. ( A ) . l.
) with eigenvalues ) q , . . . , Am : det ( A ) =
(7) A (m
=
det ( B A B- 1 )
m
>.1 · · · Am = II >.; . z
=I
:
(a) det ( A ' )
=
det.( A ) .
( b) det ( A H ) = det ( A ) .
( c ) det( A ) = det( A ) . ( d ) det ( .-\ - 1 ) = ( det A ) - 1 . if A is nonsingu lar . ( e ) d et ( Aadj ) = ( d e t .-\ ) m - l . ( f ) det ( A ) lm
( 8 ) .-1 ( m
=
.4 ad1 A = A A a dj .
) B (n x n) :
x m .
( a ) det ( A C B ) = ( d e t A )" ( det B )"' .
(b)
d t ( A ·I· H ) = det ( A )det ( B ) . e
( 9 ) .·\ = [a;1 ] ( m x m ) :
( a ) A = diag( a l l •
(b)
( 1 0 ) :1 ( m
A
. . . , a711,
is t ria n g u l a r
x n
) real :
=>
)
=>
det( A )
det ( A ) =
de t ( lm + .4 .41 )
a1 1
=
=
a11 · · · a
· · · Um m
, ,.
m
= II i= I
=
m
II
i= I
a; ; .
a;; .
d et( l + A' A ) . ,.
( 1 1 ) .-\ ( m x m ) : ( a ) rk( A )
<
( b ) rk ( .-1 ) =
m -¢:::::::} det ( A )
m -¢:::::::} det( A )
(c ) A is singular
-¢:::::::}
= 0.
f. 0 .
det( A ) = 0 .
( d ) The rows of .·\ are linearly i ndependent
(e)
The
( f) :! (g) .· l
-¢:::::::}
d r t ( A ) f.
0.
col u mns of .-\ are linearly independent. -¢:::::::} d e t ( A ) f. 0 .
has
has
a
row or col umn of zeros
=>
det ( A )
t wo ident ical rows o r columns
( 1 2 ) A . IJ ( r n x m ) :
=>
=
0.
d et ( A )
=
0.
49
TRACE, DETERM I N A NT A N D R A N K O F A M ATRIX
(a) B is obtained from A by adding to one row (column) a scalar multiple of another row (column) => det(A) = det ( B ) . (b) B is obtained from A by interchanging two rows or columns => det ( B ) = -det ( A ) . ( 1 3) ( B ine t Cauchy formula) A , B ( m x n ) , m < n : det ( A B' ) = 2::: det(A, )det ( B�1 ) , where the su m is taken owr all ( m x m) subm at.rices A m of A and B n are the corresponding submatrices of B . ( 14 ) A (m x m ) p osi t ive definite, B ( m x m ) positive semidefinite: r
det( A + B) = det ( A ) ( 15)
B = 0.
A = [ a ;J ] (m x m ) positive definite: m
d('t( A ) = II a;; i =I
( 16 )
{::::::}
{::::::}
A = diag( a I I ·
.
.
. , a,rn ) .
( Vandcrmondc determinant)
AI
,
.
.
.
•
Am E
=
det
0 0
0 0
1 0
0 1
II(,\,
)
-
Aj ) .
= ( - 1 ) m po .
Note: Most results of this subsection are straightforward implications of
th(' defi nitions and elementary matrix rules. They can be found in t ext books su h as Lancaster & T ismenet s k y ( 1 985) and Barnett ( 1 990) . c
4.2.2 (1) A
Determinant s of Partitioned Matrices
( m x n ) , B ( m x p) , C = [A : B) (m x ( n + p) ) : (a) det (CH C ) = det ( A H A )det[B H ( lm - AA+ )B] = det( BH B)det [A H ( l, - BH + )A] .
50
H A N D B O O K O F M ATRICES
(b) rk( A) = n :::} det(CH C ) = det( A H A)det [BH B - BH A(AH A) - 1 AH B] . ( c ) rk( B) = p :::} det(CH C) = det( B H B)det[A H A - A H B(BH B) - 1 BH A] . ( 2 ) det
[ 1�
10n
]
( - l )m .
=
( 3 ) A ( m x m), B (m x n ) , C ( n x n ) :
det (4)
A
det ( A )det(C).
=
det(A)det(C').
B, C ( m x m ) : det
:1
=
(m x m), B ( n x m), C ( n x n ) :
( 5 ) A.
(6)
[ �nxm � ]
[
A C
B
Orn x tn
]
=
( - l ) m det( B)det(C).
(m x m). B ( m x n ) , C ( n x m), D ( n x n ) : A nonsingu lar
:::}
det
D nonsingular
:::}
det
[ � � ] = det ( A )det( D - CA - 1 B ) . [ �· � ]
(7) A ( m x m ) , a, b ( m x 1 ) : A nonsingular ( 8 ) A ( m x m), a , b ( m x 1 ) ,
:::} c
det
E
=
det(D)det ( A - BD - 1 C).
[ 6't � ]
==
-b H A a d1 a .
A nonsingular
Note: Some of these results on partitioned matrices are given in M agnus ( 1988). Most results are straightforward implications of the definitions and
t'lementary matrix rules, however.
51
TRACE, D ETERM I N A NT A N D RA N K O F A M ATRIX
4.2.3
Determinants Involving Duplication M atrices
Reminder: Dm denotes the ( m 2 x � m ( m + 1 ) ) duplication matrix (see Section
9.5 for more details).
det( D:r, Dm ) = 2 m ( m - l ) / 2 . ( 2 ) det(Dm D:r, ) = 0. ( 3 ) A (m x m ) : ( a) det ( D:r, (A ® A)Dm ) = 2 m ( m - l )/2 det ( A ) m + I . (b ) det ( D;t", ( A ® A) D;t", ' ) = 2 - m ( m - l )/2 det ( A ) m + l . (c ) det ( D;t", ( A ® A ) Dm ) = (det A ) m + l . (4) A, B ( m x m ) : (a) det ( B ) 0 => det(D;t", ( A ® B)Dm ) = 0. (b ) det(B) :;f 0 and A 1 , . . . , A m are the eigenvalues of AB - 1 => det[D;t", ( A ® B )Dm ] = T m ( m - l )/2 det ( A ) (de t B) m IT ( A , + Aj ) · (1)
=
i >j
( c ) det( A ) '# 0 and At , . . . , Am are the eigenvalues of B A - l => det[D;t", (A ® A ± B ® B)Dm ] = (det A) m + t IT ( 1 ± A; Ai ) .
i>j
(5) A (
) with eigenvalues )q , . . . A m det[D� ( A ® lm )Dm ] = T m ( m - l ) f2 det( A ) IT (.\; + Aj ) ,
m x m
,
:
i >j
det[D;t", ( /m ® A + A ® lm )Dm] = 2m det ( A ) IT (A; + A1 ) . (6) A =
[ a ;j] , B = [b;i J (
m x m
) lower triangular:
i >j
det[D� (A ® A ± B ® B)Dm ] = IT(a ;; ai i ± b;;bjj ) .
i>j
( 7 ) A (m
x m
) real , symmetric, nonsingular, c E IR :
( 8 ) A (m
x m
) with eigenvalues At , . . . A m , i E IN , i > 1 :
det ( D! [A ® A + c vec (A)vec (A)']Dm ) = ( 1 + cm)det (A) m +l . ,
i- 1
det D� L ) A i - t - i ® Ai )Dm i =O
= im (det A ) i - l IT /l k l
k >1
I I A � D B O O K O F M AT R I C E S
where �k1 =
{
·\i I
l/lk-
Note: The rules p resented in this ( l 988).
if .\k =F .\1 if .\k = .\1 �u bsection may be fo und
i n M agnus
Determinants Involving Elimination Matrices
4.2.4
Reminder:
Lm
f3
=
[b ; j ] ( m x m )
upper ( lower) t ri angu l ar :
,· . B ) L d et ( L rn ( .'"11 \'� rn ) = I
(4 )
:l
= [a 1 ] , B ,
=
[ b; j ] ( m x m ) det ( L, ( A'
det ( J.111 ( .4
whert' Al i i I
(6) A
-
0
·2
rn
II
i
=I
m biiiaii - ' + I .
lower t riangular: m
B ) L�, ) =
II
i= I
b:; a :7- ' + 1 .
rn
B ) L:n )
aII
a1;
a; I
a ;;
= II det ( A ( • I )det ( B1 i 1 ) , i= I
B( ' I -
b; m
b; i bm;
...
l . . . . , m. ( m x rn ) real . lower t ri angu l ar n o u s i ngu l ar , c E
b mm
-
,
det( L, [.-1' : ; :1 + r vec( A)vec ( A' )'] L�, ) ( 7) A . B ( m x m ) lower triangu lar: :
IR :
= ( 1 + r m )d t> t ( .4 )"' + 1 .
det( Lm ( A B' 0 A ' B ) L :ll ) d et ( L m ( A B
c:;
B' A ) U,. )
det( L m ( A B ' 0 A ' B' ) Dm ) ( det .4 ) "' + 1 ( det B ) "' + 1 .
53
TRACE, D ET E RM I N A NT A N D R A N K O F A M ATRIX
(8)
[b;i ] , C
A = [a,j ] , B =
= [ cij ] , D
=
[ d;j ] ( m
d et ( L ( AB ® C' D ) L', ) = '
m
x m
) lower triangular:
m m II (c; ; d;, ) i ( a ; ; b;; ) - i + l , i= I
d t ( L m ( A' ® B + C' ® D ) L ', ] = II ( b;; aii + d;; cii ) . e
(9) A = [a,j] (
i>j
m x m
) lower triangular, n E IN :
n-1
l:) A' )" - I - i @ Ai
det L m
j =O
L�,
= n m ( det A )" - 1 II JLkf · k>l
where if akk
= au
Note: The rules presented in t hi s subsection may be found in Magnus
( 1 988). 4.2.5 (1)
Determinants Involving Both Duplication and Elimination M at rices
A = [a ;j] , B =
[ b ;i J ( m
x m
) lower triangular:
d et( L m ( A' 0 B' ) Dm ) d et ( L m ( A 0 B' ) Dm )
m II i= I
(2)
A, B ( d et (
m x m
Lm(
where
)
:
A® B ) Dm ) =
0
det ( A ) ( det B) m
m- 1 II n=l
C1 1
L]n
Cn t
·.·
if det ( B) = 0 det(C(n l ) if det ( B ) :f; 0
Cnn
denotes the nth principal submatrix of AB - 1 = [c;1 ] .
54
H A N D BOOK OF M AT R I C ES
(3)
A =
[a; j ] , B = [b ;j ] , C = [c;j] ( m x m) det [L m ( A ® B' C)Dm] det [ L m ( A B' ® C' ) D m]
=
lower triangular :
m
II (b; ; c;d a;:' - i + l ,
i= 1 m
= II ci ; ( a;; b;; ) m - i + 1 .
(4) A , B ( m x m) :
i= 1
det( L m ( A ® B) D! ) '
2- m ( m - 1 ll 2d et ( A )( det
where
if det(B)
=
0
B) m II det (C(n) ) if det ( B ) /:.
0
0
m- 1 n=1
C1 1
C1n
are the principal submatrices of C = hi]
= AB- 1 .
Note: The rules presented in this subsection are given in Magnus ( 1 98 8 ) . 4.2.6
Inequalities Related to Determinants
( 1 ) ( Cauchy-Schwarz inequality) A , B (m x n ) : ( det A 8 B) 2 < det ( A8 A )det ( B8 B).
( 2 ) ( H adamard 's inequality) A = [a ij] ( m x m) :
(3)
( det A ) 2 <
m
m
i=1
j=1
II L i a;i l;bs
(Hadamard's inequality) A = [a ;j] ( m x m) positive semidefinite:
det ( A ) <
m
II a;; .
•=1
( 4 ) A = [a;j] ( m x m) : ( 5)
( Fischer's i nequality) A (m x m), B (m x n), C (n D
=
x n) :
[ ,4BH B ] pos1t1ve. d fi mte. C
.
e
=>
det( D)
<
det ( A )det ( C ) .
55
TRACE, DET E RM I N A NT A N D R A N K O F A M ATRIX
( 6 ) A = [a;j ] (m x m ) positive definite:
det ( 7 ) A = [a;j ]
a; 1
(m x m)
>
a;;
0,
i = 1 , . . . , m.
positive semidefinite:
det .
.
.
>
a;;
0,
i
= 1 , . . . , m.
( 8 ) A = [a;j] (m x m) positive definite with eigenvalues k
II .\ ;
i == I
<
)q
<
·
·
· < .\111
k
det
<
II .\m - k + • ·
i =I
In det ( A ) < tr(A ) - m positive semidefinite: ( det A) 1 f m < � tr(A). positive defin i te, B ( m x m ) positive semidefinite:
( 9 ) A (m x m) positive definite:
A (11) A
(10)
(m x m) ( m x m)
det ( A + B ) > det ( A ) . ( 1 2 ) A , B ( m x m ) positive semidefinite:
det ( A + B) > det ( A ) + det ( B ) . ( 13 ) A ( m x m ) positive definite, B ( m x m ) negative semidefinite:
det(A + B )
<
det ( A ) .
( 14 ) ( M inkowski 's inequality ) A '# 0 , B '# 0 ( m x m) positive semidefinite:
[det ( A + B)p! m > ( det A) 1 f m + ( det B) l /m .
( 1 5 ) A , B ( m x m) positive semidefinite , et E IR , 0 < et < 1 :
( det A) " (det B) 1 - o
<
det(etA + ( 1 - et)B).
( 1 6 ) A, B (m x m ) positive semidefinite:
(det A ) l /m (det B) 1 f m
<
1 m
-t r(A B ) .
H A i'<' DBOOK OF � ATRICES
( 1 7) A ( m
x m
)
posi t ive definite, B ( m x m) positive semidefinite:
�
de ( A + B )
< e x p [tr ( A- 1 B )] .
et( A )
( 1 8 ) ( Oppenhe i m 's i nequal i ty ) A . B = [b;j] ( rn x m ) p ositive se m i d efi n i te : det ( A
( 1 9) .-l . B ( rn
x m
A
)
)
m
B ) > det( A ) IT b;; .
8
i=1
p ositive definite:
det( A
:;: ,
,
B ) > det ( A )det( B ) .
( 20 ) ( Ost rowsk i Taussky inequal ity)
(m
x m
:
det � ( A + A 11 ) < l det ( A ) I aLs · det ( A B ) < � [( det A ) 2 + ( det B ) 2 ] .
� ( A + Aff ) p osi tive de fi n i t e
(21 ) .·l . B
(22 ) .·l .
(m B (m
x
m ) re a l :
x m
)
re a l :
d et ( A
:::}
0
B ) < Hdet( A
8
A ) + dt>t( H :.:; B ) ] .
Note: Inequalit ies ( 2 1 ) and ( 2 2 ) are from Neudecker & S huangzlw ( 1 99:� .
1 99 5 ) . The other i nequalit ies m ay b e found in H orn & J oh nson ( 1 98 fi ) an d \hg nus & !\ e u d ecke r ( 1 98 8 ) .
Optimizat ion of Functions Involving a Determinant
4 .2. 7
I n t h is subsection all matrices are real if not stated oth er w i sf' .
(1)
S1
( m x m)
positive de fi ni te with e i ge n val u es ) q < A2 < · · · < A,, an d associated ort honon H al ( m x l ) e i ge n vec t ors u 1 , . . . , l'm :
min{ det ( B'S1 B )
:
max { det( B' n B ) : B ( m m
A 1 · A 2 · · · A11 .
= [v1 , . . . , v ,., ] .
The m i nimizing matrix is B
The
) B' B = /,., } =
x n ,
B (m x
axi m iz i n g matrix is B
n ) , B' B = In } = Am · Am - 1 · · · Am - n + 1 · =
[ vm , . . . , Vm - n + d ·
(2 ) n ( m X 111 ) complex Hermit ian positive defi n i t t• w i t h e i gP n \· alu es ,\ 1 < .\ 2 < · · · < Am and associated orthonormal ( m x l ) eig em·e c t o r s 1' 1 · · · · · 1'm :
m i n { det( B H D B ) : B ( m
x n
Tlw m in imizing matrix is B m
ax { d e t ( B H S1 B )
:
)
com p l e x , B H B
= h , . . . , vn]
B (m
The n t axi rnizing m a t r i x is B
=
=
x
/,. } = A 1 · A2 · · · A, .
n ) co m pl e x B H B
=
..\,.
.
/,., }
. A ,, 1 . . . Am - + 1 . -
[ um , . . . t"rn - n + d .
=
rl
57
TRACE, DETE R M I N A NT A N D RA N K OF A M ATRIX
(3)
Y(
m
x n ) , X (p x n ) , rk( X )
=
p:
A X ) ( Y - AX )'] : A ( x p) } = det [Y(/n - X' (X X' )- 1 X )Y'J .
m i n { det [( Y -
m
A = Y X'(X X' ) - 1 . rk ( X ) = p, R (q x )
The minimum is attained for
(4) Y ( m x n ) , X (p x n ) with
C (q x p) :
min { det[(Y -
m
A X ) ( Y - AX)'] : A (
m
with
x p), RA
= det[( Y - A X ) ( Y
rk( R ) = q ,
C} - A X ) '] , =
YX'(XX' )- 1 + R'( RR' ) - 1 (C - RYX' ( X X ' ) - 1 ). )q < x n ) , rk( Y ) = rk( X ) = < .\ m t he eigenvalues of
w here A =
(5)
Y, X (
m
· · ·
m,
( X X' ) - 1 /2 X Y ' ( Y Y') - 1 Y X' ( X X' ) - 1 1 2 '
with associated ort honormal eigen vectors
v1 ,
. . . , Vm :
BCX ) ( Y - BCX )'] : B ( x r ) , C ( r x ) rk(B ) rk(C) = r } = det( YY ' ) ( l - A m ) · · · ( 1 - A m -r+ I ) .
m i n { det[(Y m
m
=
,
The m i n imum is attained for
- [V m , · · · Cand
,
v vl ) - 1 / 2
Vm - r + l J ' ( ...-\
./\
YX'C' (CXX ' C' ) - 1 . x ) p osi t i ve definite, B diago n al , .\ 1 ( 6 ) B, C ( .\m the eigenval ues of B - 1 1 2 CB - 1 1 2 with associated eigenvectors v 1 , Vm , 1 < r < B
m
=
m
•
.
.
m :
.
m i n { l n det(AA'
< < ort honormal
+ B) + tr[( AA' + B)- 1 CJ =
m
+ I n det ( C) +
m -r
L ( .\ i= I
:
;
A (m x r ) }
- I n .\; -
1 ).
The min imum is attained for
0
1 /2
0
Note: The resu l ts of this subsection m ay be obtained from :\I agnus & Neudecker ( 1 988 ) and Liitkepohl ( 1 991 , Section A . 1 4 ) .
H A N D B O O K O F M AT R I C ES
'i 8
4.3
The Rank of a Matrix
Definitions :
The
rank of a m atrix A , denoted by rk A or rk( A ) , is t he maximum number of l inearly i ndependent rows or col u m ns of A . The col u m n rank of a matrix A , denoted by col rk A or col rk( A ) , is the n u mber of l i rlf'arly independent columns of A . The row rank of a matrix A , denoted by row rk A or row rk( A ) . is t lw nu mber of linearly independent rows of A .
General Result s
4.3.1
( 1 ) :1 ( m x n ) :
row rk( A ) = col rk( A ) = r k( A ) .
(2) :1 ( m x n ) . r E <1' :
r
(3) A (m x n) :
=>
-::j:. 0
rk(rA ) = rk( A ) .
( a ) rk( A' ) = r k ( A ) .
( h ) r k( A ) = r k ( ..l ) .
( c ) r k ( :\ 11 ) = rk( A ) .
( d ) r k ( A + ) = rk( .4 ) .
(4)
:1
( e ) rk ( A - ) = r k ( A. ) (m x m) :
{:=::}
A- AA- = A - .
m 1 0
rk( :l a dJ ) = (5) A (m x n) :
( a) rk( A A ' ) = rk( A ' A )
=
if if if
rk( A ) = m rk( A ) = m - 1 rk( A ) < m - I
rk( A ) .
( b ) rk( A .-\11 ) = rk( A 11 A ) = rk( A ) .
( 6 ) a . b ( nr x l ) : (7 )
:1
(m
a , b -::j:. O => rk( a b' ) = l
11 ) : rk( A ) = r· => ( ( r· x n ) such t hat ..1 = BC.
.
t h ere exist
x
rna
t rices B ( m x 1' ) a n d
'
( 8 ) :l ( m x n ) . B ( n x n ) :
( 9 ) .-\ ( m x n ) . B ( m x
m
)
:
H
nonsi ngu l ar
=>
rk( A B ) = rk( .·1 ) .
B nonsingular => rk ( BA )
=
( 1 0 ) A ( m x n ) , lJ ( m x m ) , C ( n x n ) : H, C nonsi ngular => rk( B A C ) ( 1 1 ) A ( m x n ) , B (p x q) :
( 1 2 ) :1 ( m x m ) . B ( n x n ) :
rk( A
·:.Y
rk( A
=
rk( A ) .
B ) = rk( A )rk ( B ) .
tf'
B ) = rk( A ) + rk( B ) .
rk ( .4 ) .
TRACE, D ET E RM I N A N T A N D R A N K
(m A (m
B (r
( 13 } A
x
n),
( 14 }
x
n) :
x
OF
m) : rk(B) =
r,
59
A M ATRIX
rk(A) = m
=>
rk( BA) =
r.
(a) rk(A) = m => A + = A H ( AA H ) - 1 . (b) rk(A) n => A+ = ( A H A ) - 1 A H . =
( 15 ) A
(m m) : x
( a) rk( A ) < m - 1 => A a dj = 0 . (b) rk( A ) = m {::::::} det(A) ::j:. 0 . (c) rk(A) = {::::::} A is nonsingular. A (m n) : rk(A) = 0 {::::::} A = 0 . A ( m m) i dempotent : (a) rk(A) = m => A = Im . (b) rk(A) = tr(A ) . (c) rk(/m - A) = m - rk(A ) . rk(/m ) = m. Kmn (mn mn) commutation matrix: rk ( Kmn ) = m n . Dm ( m 2 x � m (m + 1 ) ) duplication matrix: rk(Dm ) = � m ( m + 1 ) . L m ( � m(m + 1 ) m 2 ) elimination matrix: rk( Lm ) = � m(m + 1 ) . A (m x m) : ( a) rk( . Um - A) < m {::::::} � is an eigenvalue of A . ( b ) 0 is a simple eigenvalue of A => rk(A) = m - 1 . A ( m n) : rk (A) = r => rk( I<mn (A' 0 A)) = r 2 . A (m m), A;j ( n n ) submatrix of A : rk(A) = r, n > r => det( A;j ) = 0 . A ( m m) : ( a) rk(A) = r => there exist (m m) matrices B; , i = l , . . . , r, with rk( B; ) = 1 and A = B 1 + + Br . (b) rk(A i ) = rk( A i + l ) for some i E IN => rk(A ; ) = rk ( Ai ) for all j > i. (c) There exists an i E IN such that rk(A i ) rk( A i +l ) . ) A (m m) nonsingular, B (m n ) , C ( n x m), D ( n m
( 16 } ( 17}
( 18} ( 19 ) (20) (21) (22)
(23) ( 24} ( 25)
x
x
x
x
x
x
x
x
x ·
·
·
=
(26)
x
x
x n
:
60
H A N DBOOK O F M ATRICES
Note: M any of these results are elementary and fol low immediately from ddi nit ions ( see. e.g . , text books such as Horn & Johnson ( 1 985) and Lancaster i< Tismenetsky ( I 985 ) ) . See also 1\I agnus ( I 988 ) for some of t he results. Matrix Decompositions Related to the Rank
4.3.2
( 1 ) (Singular value decomposition) A ( m x n ) , rk( A ) = 7', a 1 , . , 0"r "# 0 singular values of A : exist u ni tary m at rices U ( m x m ) , V ( n x n ) such that .
.
o
0
J
vn
There
'
where D = d iag(a 1 ar ) and some or all of t he zero submatrices d isappear when 7' = m and/or 7' = n . •
(2)
.4
.
.
.
•
( m x m ) symllletric :
(m x
111
) mat rix
T
rk( A ) = such that A =
T
7'
¢::::::>
[6 �]
there exis t s a nonsingular T' . 7'
where the zero submat rices d isappear when
= m.
( 3) ( Di agonal reduction ) .-1 ( m x 11 ) : rk( A ) = 7' ¢::::::> there exists a nonsingular ( m x m ) Juat rix S and a nonsingular ( n x n ) m atrix T such that A=S
[6 �]
T,
w lwrt> some or all of the zero submatrices disappear when 7' 111 and/or r = 11 . ( 4 ) A ( 111 x m ) : rk( A ) = rk( A 2 ) ¢::::::> there exist nonsingular matricPs P and D such that
( 5 ) ( Rank fac torization ) A ( m x n ) : rk( A ) = 7' ¢::::::> there exists an ( m x r ) matrix B and an ( r x u ) matrix C w i t h rk( B ) = rk(C') = 7' such t h at A = BC. ( 6 ) ( Triangular factorization) A = [a,1) ( m x m ) . rk( A ) = det
7' :
=f. O ,
i = l,
.
.
.
.
7'
TRAC E, D ET ER M I N A N T A N D R A N K
OF
A M ATRIX
61
t here exists a lower triangular ( m x m) m atrix L and an upper tri angular ( m x m ) m atrix U one of w hich is nons i ngular such that A = LU. =>
( 7 ) ( Polar decompositio n ) A ( m x m ) : rk( A ) = r ¢::::::> there exists a positive semidefin ite ( m x m ) m atrix P with rk( P ) = r and a u n i t ary ( m x m ) m atrix U such that A = PU . Note: For these and further decomposition theorems see Chapter 6. 4.3.3
Inequalities Related to the Rank
( 1 ) A ( m x n) :
( a) r k ( A ) < m i n ( m , n ) . ( b ) r k ( A - ) > rk( A ) .
( 2 ) A ( m x n) , B ( n x r) :
( a) r k ( A B ) < m i n { rk( A ) , rk(B) } .
( b ) r k ( A ) + r k ( B ) < rk( A B ) + n . ( c ) A B = 0 => rk(A) + rk( B ) < n . (3) A, B ( m x n) :
rk( A + B ) < rk ( A ) + rk( B ) .
(4) A (m x n) , B ( m x r) : (5) A (m x n) :
rk( [A : B] ) < rk( A ) + rk( B ) .
Ax = 0 for some ( n x 1 ) vector
( 6 ) ( Sy lvester's law) A, B (m x m) :
x
f. 0 => rk( A ) < n - 1 .
rk( A ) = r. rk( B ) = s => r k ( A B ) > r + s - m .
( 7 ) A , B ( m x m ) H ermitian : ( 8 ) A ( m x m ) Hermitian :
rk( A 8 B ) < rk( A ) rk ( B ) . rk( A ) > [tr( A )] 2 /tr( A 2 ) .
( 9 ) ( Frobenius inequality ) A ( m x n) , B ( n x r), C (r x s) : rk( A B ) + rk( BC) < r k ( B ) + rk(ABC). Note: These results c an be found i n m any m atrix textbooks ( see, e .g., Horn & Johnson ( 1 985 ) or Rao ( I 973 ) ) or fol low eas i ly from rules given t here.
5
Eigenvalues and Singular Values In this chapter all m atrices are complex unless otherwise stated .
5.1
Defi n itions
For an ( m x m) matrix A the polynomial in � . PA ( � ) det ( Mm - A ) . is c alled the characteristic polynomial of A. The roots of PA ( � ) are said to be the eigenvalues , characteristic values, characteristic roots or latent roots of A. Let � 1 , . . . , � n be the distinct eigenvalues of the ( m x m ) matrix A . Then the characteristic polynomial can be represented as =
where the m; are positive integers with L � 1 m , = m . The number m ; is the multiplicity or algebraic multiplicity of the eigenvalue �i , i = 1 , . . . . 11 . A n eigenvalue is called simple, if its multiplicity is 1 . The geometric multiplicity of an eigenvalue � of an ( m x m ) m atrix A is t he number of blocks 0 � 1 0 0
�
0
0
0
0
0
1 �
with � on the principal d i agonal, i n the Jordan decomposition of A which is given in the next section (see also Section 6. 1 ) . For an ( m x m ) m atrix A . p ( A ) = max { l � l abs : � is an eigenvalue of A }
H A N D BO O I\ OF �t. �'I' R i f.ES
64
is t he spectral radius of A . The set { � : A is eigenvalue of ..4 } is t hP spectrtiiJl of .A . For an ( 111 x 1n ) n1atrix ..4 with eigenvalue A any ( 1 n x 1 ) vector r' f. 0 satisfying ..4 r' = A r is said to be an eigenvector or characteristic vector of .·\ corresponding to or associated with the eigenvalue A. For t\vo ( 1 n x 1 n ) 1natrices ..4 and B a root of the polynon1ial PA .n(A) = det ( A B ..4) is sotu eti tnes called eigea1value or characteristic value of .4 i11 tl1e 111etric of B. A ccordingly for an eigenvalue A of A in the n1etric of lJ an ( 111 x 1 ) vector r :j:. 0 sati sfy ing ..4t' = ABv, is called an eigenvector or el1aracteristic vector of .4 in the 1netri(� of B corresponding to or associated with A. For a n ( 111 x 11 ) 1nat rix .4 thP nonnegat i ve squarP roots of t he eigenvalut�s of 1 11 1 if 111 > 11 , and of .4 ..4 H , if 111 < 11 , are said to be t h e singtiiar val11es of :\ . -
·
-
.
.
.
,
.
P roperties of Eigenvalues a nd Eigenvectors
5.2
Notatio11:
denotes an eigenvalue of tht., tnatrix .4 .
A( .4 ) A
( .A )
nu n
An,a.r ( ..4 ) ..
is t.he sJnallest eigenvalue of thP rnatrix .4 if all eigen values are real . is t he l argest eigenvalue of t he n1atrix ..4 if all eigenvalues are real .
General Results
5.2 1
( 1 ) .4 ( 11l x 17l ) : 1 has ex ac tly 11l eigenvalues if the ntult.iplicit.it.,s of t h{"' roots are taken i nto account. .-
( 2 ) .4 . B ( 111 x 1n ) : .4 , B h ave the san1e eigenvalues � tr( ..4 k ) t r( B k ) k = 1 . . , 1n . =
( 3)
(4)
.-1
4
..
( 1u
x 111
.
.
) \V it h t")igenvalue
( a ) ,\ is eigenvalue of
.4
A
.
and associated eigeHV<.,ctor
\vith eigenvector
r- :
v.
( b ) ,\ i is eigt>nval ue of .4 r with eigenvect or 1 ' for i E Lr\J . ( c ) A - 1 is eigenvalue of ..4 - 1 wit.h eigenvt,.ctor r: , if ..4 is nonsingular. ( 11l
X Ill
)
:
( a ) A is eigenvalue of 4 � ( b ) ,\ is Pigenval ue of
( c ) ,\ is f'igenvalue of
..
.4
A
is eigenvalue of A'.
� ,\ is eigenvalu(;) of ..4 H .
.A =>
A+
T
is eigenvalue of
.4 +
.
T
l,n .
EIG ENVA L U ES A N D S I N G U LA R VA L U ES
( 5 ) A = [a;j ] ( m x m) : ( a ) A = diag( a , , , . . . , a mm ) => a l l , · · · · a mm are the eigenvalues of A. (b) A is triangular => a 1 1 , . . . , am m are the eigenvalues of A . ( c ) A is H ermitian => all eigenvalues of A are real numbers. ( d ) A is real symmetric => al l eigenvalues of A are real numbers . (e) A is idempotent => all eigenvalues of A are 0 or 1 . (f) A is nilpotent => all eigenvalues of A are 0 . (g) A is real orthogo n al => all eigenvalues of A are 1 or - 1 . ( h ) A is singular {::::::} 0 is an eigenvalue of A . ( i ) A is nonsingular {::::::} all eigenvalues o f A are nonzero . ( 6 ) A (m x m ) Herm i ti an : ( a ) A is pos i tive definite greater t h an 0 .
{::::::}
all eigenvalues of A are real and
( b ) A is positive semidefinite {::::::} all eigenvalues of A are real and greater than or equal to 0 .
( 7 ) A ( m x m) real sy mmetric : ( a) A is pos i tive definite greater t h an 0 .
{::::::}
all eigenvalues of A are real and
( b ) A is positive sem idefi nite {::::::} all eigenvalues o f A arc real and greater t h an or equal to 0 .
( 8 ) A ( m x m ) , B ( m x m) nonsingular: >. is eigenvalue of A {::::::} >. is eigenvalue of s- 1 A B .
( 9 ) A ( m x m ) , B ( m x m) u nitary : >. is eigenvalue of A {::::::} A is eigenvalue of BH AB.
( 10 ) A ( m x m ) , B ( m x m) orthogonal : >. is eigenvalue of A {::::::} >. is eigenvalue of B' A B . ( 1 1 ) A ( m x m ) , B ( m x m ) positive definite: >. is eigenvalue of BA {::::::} A is eigenvalue of B 1 1 2 A B 1 1 2 . (12) A (m x n), B (n x m), n > m : ( a) A is eigenvalue of A B => A is eigenvalue of BA.
( b ) >. -:f 0 is eigenvalue of BA => >. is eigenvalue o f A B . ( c ) >. 1 . . . . , Am are the eigenvalues of A B => .>. , . . . . . A 111 • 0 . . . . . 0 are the f.'igenvalues of BA .
65
H A N DBOOK O F M AT R I C E S
66 ( 13 ) A ( m x m) ,
c E
( a) >. is eigenvalue of A with eigenvector c A with eigenvector v.
(b)
v
::::}
c>.
is eigenvalue of
1'
is eigenvector of A corresponding to eigenvalue >. eigenvector of A corresponding to eigenvalue >. .
( 14 ) A ( m
x m ) , c 1 , c2 E
c2 # 0
correspondi ng to an eigenvalue >. ::::} A corresponding to eigenvalue >. .
::::}
111
) . ') o . ') 1
.
v 1 , v2 eigenvectors of A c1 t•1 + c2 t'2 is eigenn"C'tor of
( 1 5 ) A (m x m ) w i t h eigenvalues A; , Aj and associated eigenvectors 0 : A; -:f Aj ::::} v; and Vj are linearly independent.
( 1 6 ) ..\ ( 111 x
cr IS
r, . r-1
#
'/p
E . is eigenvalue of A with eigenvt•ctor r ::::} 1o + ') 1 A + · · · + ')p AP i:; eigenvalue of 1 o lm + 1 1 A + · · · + 1 r ··V' w i t h eigenvect or v . •
.
.
.
.
(m x m ) ,
B ( n x n ) , >. ( A ) , >. ( B ) eigenvalues of A and B , respectively, with assoc iated eigenvect ors t' ( A ) and r( H ) . respectively:
( 17) A
( a ) A( A ) · A( B ) is eigenvalue of A 0 B with eigenvector v ( A ) C r( H ) .
( b ) A ( .-1 ) aud A ( B ) are eigenvalues of A
[ t'\)4) ]
and
,
m
( 19)
=
>. 1 · · · A
m
=
B with eigenvectors
[ v(�) ] , respectively .
( 1 8 ) A ( 111 x m ) wit h eigenvalues >. 1 , . . . A m ( a) det ( A )
t3
:
IJ A; .
i=I
m
( b ) t r( A ) = >. 1 + · · · + A m = L A; . i= I
A (m x m ) :
nonzero eigenvalues ::::} rk( A ) > r. ( b ) rk( >. lm - .4 ) < m - 1 ¢::::::> >. is eigenvalue of A. . ( a) A h as
1'
( c ) 0 is a simple eigenvalue of A
::::}
rk(A. ) = m - 1 .
( 20 ) A ( m x m ) with eigenvalues >. 1 , . . . , Am : ( a) I A; I abs < 1 , i = 1 , . . . , m ¢::::::> det ( lm - A z ) # 0 for l z l abs < I , z E
(c)
i
= 1, .
I A ; Iabs < l . i = l ,
.
. .
. .
,m
::::}
Im ± A is nonsingular.
, m ::::} Im, ± A 0 A is nonsingu l ar .
67
EIGENVA L U ES A N D S I N G U L A R VA L U E S
( 2 1 ) (Cayley - Hamilton t heorem) A (m x m)
:
)q, . . .
, A m are eigenvalues of A
m
=>
IJ (>. Im - A ) ;
=
O.
i= I ( 22 ) A ( m x m ) : The geometric multiplicity of an eigenvalue >. of A , that is, the number of Jordan blocks with >. on the principal diagonal, is not greater than the algebraic multiplicity of .>.. (See Section 5.2.3 for the Jordan decomposition . ) Note: Most of the results of this section can be found in Horn & Johnson ( 1 985 ). The others are easy to derive from results given there . Chatelin ( 1993) is a reference for further results on eigenvalues and vectors including computational algorithms. 5.2.2
Optimization P roperties of Eigenvalues
( 1 ) ( Rayleigh - Ritz theorem) A ( m x m ) Hermitian :
{ X H Ax XH X { x H Ax A m ar ( A ) = max A m i n ( A ) = min
XH X
: x (m
( 2 ) ( Rayleigh - Ritz theorem for real matrices) A ( m x m ) real symmetric:
x'Ax { A m i n (A) = min x'x x'Ax Am ax (A ) = max { xx
} }
x ( m x 1 ) real, x f. 0 , x ( m x 1 ) real , x f. 0 .
'
(3)
} x 1 ) , x =f. O } .
x ( m x 1), x =f. O ,
( Courant - F ischer theorem) A ( m x m ) Hermitian with eigenvalues A1 < ···< Am 1 < i < m : .>. ; =
y,,
min
, y ., _ , ( m x I )
max
{ xHAx
'
:
H
X X
X ( m X 1), X f. 0 , X H Yj = O , j = 1 , . . . , m - i
and A; =
max min l y 1 . . . . , y , _ 1 (m x )
{ x H Ax H
X X
:
}
}
X ( m X 1 ) , X f. 0 , X H Yi = 0 , j = 1 , . . . , i - 1 .
68
H A N DBOOK O F M ATRIC ES
( 4 ) ( Courant - F ischer t heorem for real m atrices ) .-\ ( 111
m :
x
m ) real >;yrumetric with eigenvalues A 1 < · · · < Am . 1 <
>.; =
Yl ·
(m x
.
ll1lll
x
· · Y111 - 1
l)
real
m ax
{ x' Ax,
max
!!,q ,
(m J.'
x
t)
. Yt - 1
real
Jll \ll
{ x' Ax x
'x
<
:
X X
( m x 1 ) real. x -:f 0, x'yi = O, j = 1 , . . . . m
and
i
-
i}
:
}
( 1 11 x 1 ) real . x -:f 0 , x'Yi = 0 , j = 1 , . . . . i - 1 .
m ) H e r m i t i a n with e i ge n valu es >. 1 < · · · < Am and associated ort honormal e i ge n vect o rs t ' t , . . . , t>m . n E { 1 . . . . . Ill } :
( 5 ) .-\ ( m
x
rnin{ tr( X H A X ) : X The
m
( m X n) ,
x H X = In } = A t + . . . + >., .
i n i m iz i ng m atrix is X = [vt , . . . . t•nl ·
axi m i z i n g m at rix is X = [vm . . . . . t 'm - n + d · ( 6 ) .-\ ( m x m ) real symmetric with eigenvalues A t < · · · < >.,, and as so c i a t e d ort honormal eigenvec tors l' t , . . . . l'm · 1 1 E { 1 . . . . . m } : The
n
m
t i n { t r ( X ' A X ) : X ( m x n ) real, X ' X = In } = A 1 + · · · + >." .
The m i n i m iz i n g matrix
is X = (
11 1 ,
m ax{ t r ( X ' .-\ X ) : X ( Ill x n ) real ,
( 7)
T he !ll aximizing matrix is
. . . , l'n ] . X ' X = I11 } = >.,, + · · · + Am - n + t ·
X = (1•,. , . . . , l'm - n + d ·
A ( m x m ) Hermitian with eigenvalues 0 < )' 1 < { 1 . . . . . m} : >. t . >. .,- . . . >.
< >., .
"
min{ ( J.· f1 A x t ) · · (x� Axn ) ( m X 1 ) . X = [J.' t . . . . , J.'n ] ( m ·
J.',
X n),
x /{ x
!, } .
11
E
69
EIGENVA LU E S A N D S I N G U LAR VA L U ES
( 8 ) A (m x m) Hermitian positive definite with eigenvalues ) q < A 2 < · · · < A m and associated orthonormal ( m x l ) eigenvect ors v 1 , Vm : .
.
.
•
min{det( B H A B ) : B ( m x n) , B H B = ln } = A t · A 2 · · · A, . The minimizing matrix is B = [ v, , . . . , Vn ] · max{det ( B H A B ) : B (m x n) , B H B = ln } = Am · A m - l · · · A m - n + l · {9)
The m aximizing matrix is B = [v m , . . . , V m - n + d · A ( m x m) real symmetric positive definite with eigenvalues A 1 < A 2 < · · · < Am and associated orthonormal ( m x l ) eigenvectors min{det( B' AB ) : B ( m x n ) real, B ' B = ln } = A , · A2 · · · An . The minimizing matrix is B = [v, , . . . , v n ] . max{ det ( B' A B ) : B ( m x n ) real, B ' B = /, } = Am ·A rn - 1 · · · Arn - , + 1 · The maximizing m atrix is B = [vrn , . . . , Vm- n + d ·
Note: The results of this section may be found in Horn & Johnson ( l 98.') ) and partly i n M agnus & Neudecker ( 1 988 ) . Matrix Decompositions Involving Eigenvalues
5.2.3
( 1 ) (Schur decomposition)
( m x m) : There exists a unitary ( m x m) matrix U and an upper triangular matrix A with the eigenvalues of A on t he principal diagonal such that A = U AU H . (2) (Schur decomposition of a real matrix) A ( m x m) real with real eigenvalues : There exists a real orthogonal ( m x m) matrix U and a real upper triangular matrix A with the eigenvalues of A on the principal diagonal such t hat A = F A C ' . ( 3 ) ( Spectral decomposition of a normal matrix) A ( m x m) normal with eigenvalues A 1 , . . . , A m A = U AU H , w here A = diag( A 1 , . . . , A m ) and U is the unitary ( m x m) matrix whose columns are the orthonormal eigenvectors v1 , . . . , Vm of A associated with A 1 , . . . , A m . In other words, m A = L A; vfl . A
:
i =I
v,
70
H A N D BOOK OF M AT R IC E S
( 4) ( Spectral decomposition of a Hermitian m atrix) A = U AC H . .4 ( m x m) Hermitian with eigenvalues A 1 , . . . , Am where A = diag{ A 1 , . . . , Am ) and U is the unitary ( m x m ) matrix whose columns are the orthonormal eigenvectors V t , . . . , V m of A associated with A t , . . . , Am . In other words,
m
A=
L A; v; vfl . i= t
( 5) ( Spectral decomposi tion of a real symmetric matrix) A ( m x m ) real sy mmetric with eigenval ues A t , . . . , Am : A = { ' A [ 0 where A = diag( A t , . . . , Am ) and U i s the real orthogonal ( m x m ) matrix whose colu m ns are the orthonormal eigenvectors v 1 , . . . , t:m of A associated with A t , . . . , Am . In other words,
m
A=
L A; v, v; . i= t
J ordan Decompositions ( 6 ) A ( m x m ) with distinct eigenvalues A t , . . . , A n : There exists a nonsi ngular ( m x m ) matrix T such that A = TAr - t , where
A=
is block diagonal with blocks 0 1 A; =
0
0
0
0
0 0
i = l , . . . , k > n.
on the diagonal and { n t , . . . , nk } = { l , . . . , n } , that is, the same eigenvalue m ay appear on the diagonal of more than one :\. , . ( For more details see Chapter 6 . ) ( 7 ) A ( m x m ) with m distinct eigenvalues A t , . . . , Am : A = TAT - t , where A = diag( A t , . . . , Am ) and T is a nonsingular ( m x m ) matrix whose columns are eigenvectors of A associated with A 1 , . . . , Am .
il
EIGENVA L U ES A N D S I N G U L A R VA LUES
( 8 ) ( Near-diagonal Jordan decomposition ) A ( m x m) with distinct eigenvalues A1 , . . . , An , t > 0 a given real number: There exists a nonsingular ( m x m) matrix T such that A = TAT- • , w here
A = 0 is block diagonal with An , 0
An ,
0
0
A; ( t ) =
0
.
{
i = I, . . . , k > n, {
0
and { n 1 , . . . , nk } = { I ,
0 0
0
{
.
.
.
.
An ,
. , n} .
( 9 ) ( Real Jordan decomposition) A ( m x m) real with distinct real eigenvalues A1 , . . . , Ap and dist inct There exists a real complex eigenvalues a 1 ± i/11 , . . . , a q ± i/1q : nonsingular ( m x m) matrix T such that A = TAr- • , where 0 A =
0 is block diagon al with
A; =
Ap , 0
I Ap ,
0
0
0
0
0 I
0 0 i = I , . . . , k > p. I Ap ,
-.
) , _
H A t' D BOOK O F �1 AT R IC ES
{Pt , . . . p k } = { l . . . . . p } and .
[
i3
0: q , - 3q ,
q, 0: q , 0
r;
=
]
[
0
[2 O: q ,
- ;3q,
0 Jq ,
Ct
0
]
q,
0
0
0
0
l _ . . . , { > q, { q 1 , . - . , qt } = { 1 , . - . , q } .
I =
Note: The results of this section can be found in Horn & J ohnson ( 1 9Ki> ) .
Set•
Chaptt•r
5.3
for furt her matrix decomposition results.
Eigenvalue Inequalities
Inequalit ies for t he Eigenvalues of a S ingle Matrix
5.3.1
(1)
6
(
.-l
m x m
)
H � · r mit ian,
.l'
:f. 0
(
m x
.
A 7l
)
x
{ �
( 111 x
111
:
\· II \"
'
'
( ..\ m ( A ) ,
(4) A
X
}
m x
{
}.
x
X
X
I ' . . . . 17!.
I =
{3)
:
11 A x x < Ama.r ( A ) . Amin ( A ) < H m ) Hermitian with eigenvalues ..\ 1 ( A ) , . . . , .Arn ( A ) : xH x : x ( I) , x :f. 0 uin t H x < ..\, ( A ) < max x11Ax 11 : x ( 111 l ) . x :f- 0 X
(2) 4 (m
I)
m
x
)
Hermit ian with eigenvalues ..\ I ( A ) < · - - < .Am ( A ) .
- I
"
m
)
X (m
X ' X = l,
i=I
i= I
..\ J ( A )
real symmetric with eigenvalues n ) real:
x
n
i=I
n
<
X (m x
<
73
EIG ENVALU ES A N D S I N G U L A R VA L U E S
( 5 ) A = [a ; j] ( m
x m
) Hermitian: i=
Amin ( A) < a;; < ..\max ( A ) , ( 6 ) A = [ a;j] ( m x
m
[a;j ] (
m x
Am (A) :
.
.
. , m.
) Hermitian with eigenvalues ..\ 1 ( A ) <
n n n L ..\; ( A ) < L a;; < L ..\m -n+i ( A ) , i= I i= 1 i= I
( 7) A =
1,
m
n
=
·
1, .
·
·
< ..\m ( A ) :
. . , m.
) Hermitian with eigenvalues 0 < ..\ 1 ( A) <
n n II ..\; ( A ) < II a;; , i=l i=l
n = 1,
<
. . . , m.
( 8 ) A ( m x m ) Hermitian positive definite with eigenvalues ..\ J ( A) < · · · < Am ( A) , X ( m x n ) :
(9) A (m
n
n
i= 1
i= I
) real pos 1 t 1 ve definite with eigenvalues ..\! ( A ) < Am ( A ) , X ( m x n ) real : x
m
<
n n II ..\ ; ( A) < det(X ' A X ) < II Am - n + i ( A ) . i=l i=l
X' X = In
( 1 0 ) A = [ a;j] ( m x m ) Hermitian positive definite with eigenvahws ..\ 1 ( A ) < · · · < Am ( A ) :
A(n) =
(11)
n
n
i= I
i=l
(Incl usion principle) ) Hermitian with eigenvalues ..\ 1 ( A ) < < A ( ..\m (A) , A( n ) ( n n ) a principal submatrix of A with eigenvalues ..\ 1 ( A ( n J ) < · · · < ..\n (A(n) ) : m
x
m
x
..\ ; ( A ) < ..\ ; ( A( n ) ) < Am -n+i ( A ) ,
i
=
1,
.
. .
, n,
14
H A N DBOOK OF M AT R I C ES
Relations Between Eigenvalues of More Than One Matrix
5.3.2
( 1 ) A ( m x 111 ) Hermi t i an. B ( 1 1 1 x m ) positive semidefinite:
R) > Amin ( A ) . Amar ( A + R ) > Amar ( A ) .
( a ) Am•n ( A +
(b)
( 2 ) A . R ( m x m ) Herm itian:
( a) ( b)
{ 3 ) ..1 .
l3
Amin ( A + B ) > Amin ( A )
+
Ami n { B ) .
Amar ( A + B ) < Ama.r ( A ) + Amax ( B ) .
( rn x m ) H ermitian , 0 < a , b E IR :
( a ) ..\"'"' ( a A
( b ) Am a .r ( a A
+
+
b B ) > a ..\ m;,. ( A ) + 6 ..\m; ,. { B ) . b R ) < a A mar ( A ) + b..\mar ( B ) .
( 4 ) A ( m x m ) Hermitiau w ith eigenvalues A t ( A ) < · · · < A m ( A ) . B ( m x 111 ) Ilermitiau positive sem idefinite . ..\ d A + B ) < · · · < Am ( A + H ) eigeuvalues of A + B : ..\; ( A + B ) > ..\; ( A ) ,
(5)
i
=
:\
1 , . . . , m.
( 1n x m ) Hermitian with eigenvalues ..\ 1 ( A ) .::; · · < A m ( A ) . J' ( 111 x I ) . A , ( A ± J'J' 11 ) < · · · < ..\m ( A ± J.' .I' 11 ) eigenvalues of A ± .u11 : ·
A; ( A. ±
n:11 )
< A;+ I ( A ) < A; + 2 ( A ±
.r.IJ1 ) .
..\, ( A ) < A , + I ( A ± x.r 11 ) < A;+2 ( A ) ,
i
i
=
1 , 2, . . . .
1 , 2, . . . , m
=
m -
-
:! :
:! .
< Am ( A ) { 6 ) .4 . B ( m x 111 ) Hermitian , r k ( B ) < r , A I ( A ) < · · eigenval ues of A . A I ( A + B ) < · · < Am ( A + B ) eigenval ues of A + B : ·
·
A; ( A + B ) < A;+,. ( A ) < A, + 2r ( A + B ) . A, ( A ) < A; + r ( A + B ) < ..\ ; + 2r ( A ) ,
i i
=
( 7) ( Poincare's separation theorem)
=
1 . 2 , . . . . m - 'b· :
1 , 2 , . . . . m - 2r.
A ( m x 1 1 1 ) Hermitian with eigenvalues ..\ ! ( A ) < · · · < ..\,. ( A ) , X ( m x n ) such t hat X 11 X = /,. , 11 < m, A 1 ( X 11 A X ) < . . . < A, ( X H A X ) eigenvalues of X 11 A X : A; ( A ) < A; ( X 11 A X ) < Arn- n +i ( A ) ,
i
=
1 . . . . . n.
( 8 ) A . B ( 1 1 1 x m ) Hermitian with eigenvalues A d A ) < · · · < ..\m ( A ) aud A I ( B ) < · · · < Am ( B ) . eigenvalues of A + B :
( a)
respectively ; n
..\ ! ( A + B ) <
L [ A i ( A ) + ..\; ( B ) ] < L ..\ i ( A + B ) , i= l
·
· · < ..\m ( :l + B )
n = 1, . . . . m.
75
EIG ENVA L U ES A N D S I N G U LAR VA L U ES
n = 1, . . . , m.
( b ) ) q ( B ) < ..\n ( A + B) - ..\n ( A ) < ..\m ( B ) ,
( c) I .An ( A + B ) - An ( A ) I abs (d ) (9)
< max { l..\j ( B ) I abs : j = 1 , . . . , m } = p( B) , n = l , ..\n ( A + B) < min{ ..\;( A ) + ..\i ( B) : i + j = n + m } , n = 1 , . . . , m.
. .
. , m.
(Weyl 's theorem ) A, B ( x ) Hermitian with eigenvalues ..\ 1 ( A ) < · · < ..\m ( A ) and ..\ 1 (B) < · · · < ..\m ( B ) , respectively ; ..\ ! ( A + B) < · · < ..\m ( A + B ) eigenvalues of A + B : m
m
·
·
..\; ( A + B) >
and ..\; ( A + B) <
i= 1
..\; ( A ) + ..\! ( B) ..\; - 1 ( A ) + ..\ 2 ( B)
..\; ( A ) + .Am ( B) ..\;+ I ( A) + ..\m - ! ( B)
, . . . , m.
(m x
) Hermitian with eigenvalues ..\ 1 ( A ) < · · · < .Am ( A) and ..\ 1 ( B ) < · · · < ..\m ( B), respectively ; ..\ ! (A - B) < · · · < Am ( A - B ) eigenvalues of A - B :
( 10 ) A , B
m
..\ 1 ( A - B) > 0
=>
..\; ( A ) > ..\; ( B) , i
=
1, . . .,
m.
( 1 1 ) A ( m x m ) Hermitian with eigenvalues ..\ I ( A) < · · · < ..\m ( A ) , 1 ) , c E IR, A B=
x [ xH ]
x(
m
x
c
with eigenvalues ..\ 1 ( B) < · · · < ..\m +l ( B) : ..\; ( B) < ..\; (A) < ..\;+ • ( B ) ,
i = 1,
. . . , m.
( 1 2 ) A ( n x n ) Hermitian with eigenvalues ..\ I ( A) < · · · < ..\n ( A ) , D (p x p) Hermitian, C (n x p) and B
=
[ �H � ] (
m X m
)
with eigenvalues ..\ ! ( B) < · · · < ..\m (B) : ..\; ( B ) < ..\ ; ( A ) < Am - n+i ( B)
for i = 1 , . . . , n.
i'6
H A N D B O O K O F M AT R I C ES
The results of this section are given in Horn & Johnson ( 1 98 5 . C 'hapt ers 1 and 7 ) or follow easily from results given there. Some results. in part icular t hose on real sy nunetric matrices, may also be found in �Iagnus l'i.: :'ieudecker ( 1 988 ) . :\-I any of these and further results are also rontained in C 'hatelin ( I 993 ) . Note:
5 .4
Results for t he Spectral Radius
aij
.4 > ( > ) 0 A > (>) B
a,j
.
( 1 ) .4 ( 111 X 111 ) . i E N (2)
> ( >) 0. i = 1, . . > ( > ) b;1 , i = 1 , .
.
.
, m.
1 , . . . . 11 . j = 1, . ..
j =
. , m,
.
n .
.
: p( A ) l = p( A l ) . A . R ( 111 x m ) Hermitian with eigenvalues .\ ! { A ) < · · · < A m ( A ) and .\ t ( H ) < · · · < A, ( B ) . respectively: A t ( A. + B) < · · < Arn ( --1 + B ) ••igemalu•·s of :l + B : ·
f .\ , ( .4 + H ) -
(3)
A
( 4 ) :1
( 111 x m ) :
( a)
p( A )
<
1 ¢::=:} A" � 0 :::>
( h ) p( A )
<1
( C ) p( A )
<
( d ) p( .·1 )
=>
(e) p( A ) < l
=>
= [a,1] ( 111
( a)
1 :::>
x m)
i
-X i ( A. ) l abs < p( B ) .
L>-\' t =O
�
•X ,
i.e
.
. A
1
=
( Im
+ A)- 1 .
L Ai 0 A i = { lm2 - .4 0 A. )- 1 . i=O L( -A)i 0 A i = ( lm2 + A C A )- 1 . i=O
re-al :
.4 > 0 => p( A ) > O .
( b ) A > 0.
L a;1 j=1
> 0.
i = l, . .,m .
( c ) A stochast ic => p( A ) > 0. ( d ) .·l dou bly stochil.Stic => p( A )
=> p( A ) > 0 .
> 0.
m.
is ronvergent .
( lm - A ) - 1
=
L( -A )
as n
= I.. . ..
77
EIG E N VA L U ES A N D S I N G U L A R VA L U E S
(e) A > 0, A i > 0 for some i > 1 => p( A) > 0. ( 5 ) A, B ( m x m ) real: (a) 0 < A < B => p( A ) < p( B) . ( b ) 0 < A < B => p( A) < p( B) . ( 6 ) A = [ a;j] ( m x m ) real,
principal submatrix of A : A > 0 => p ( A ( n ) ) < p( A ) , n = 1 , .
( 7 ) A = [ a;j] > 0 ( m x m ) real : (a) . max a ;; < p( A ) . 1 = I , . . . ,m m
m
; =I
;=1
. . , m.
(b) _min L aij < p( A ) < 1. _-max L a•j · 1 , . ,m . 1 - l , . . . ,m .
(d)
0
( 8 ) A = [ a;j] >
(a)
(m
x m
m
) real, x = ( x 1 , . . . , x, )' > 0 m
1 """' 1 """' min a; ) x) < < . max - L.., a I ) x) . - p( A ) t = 1 , , m X i L.., t = l , . . . , m Xi . . ·
·
; =1
(b) (9) A (m
(m x 1 )
m
min Xj L 1 m
. ;= , x
,
rPal :
.
;=1
a;i
X; . z=l
< p( A ) < . max x1 = i , . . . ,m
m ) real, A > O , x ( m
;
x
1)
m
a ;i L . . Xj •=I
real , x > O . a , b E IR, a , b > 0 :
(a) ax < Ax < bx => a < p( A ) < b. ( b ) ax < Ax < bx => a < p( A ) < b. ( c ) Ax = a x => a = p( A ) . ( 1 0) A ( m x m ) real, ..\ eigenvalue of A with real eigenvector x :f. 0 : A > 0, I .A I? bs = p( A) => I .A i abs is eigenvalue of A with eigenvector l x labs ·
78
H A N DBOOK O F �1 AT R I C ' E S .-1
( m x m) real , A > 0 : (a) p( A. ) is a simple eigenvalue of A . ( b ) p( A) is eigenvalue of A with eigenvector x > 0 . ( 1 2 ) A ( m x m ) real , A > 0 : p( A ) is eigenvalue of A with eigenvector x > 0 , x :f. 0 . ( 1 3 ) A ( m x m ) real , A > 0 , ..\ eigenvalue of A : ..\ :f. p( A ) => I ..\ I abs < p( A) · { 14) ( Hopf's theorem) .·1 [a;j ] ( m x m ) real , A > 0 , A m - I eigenvalue of A with second largest modulus, Af = rnax{ a;j : i , j = 1 , . , m } , Jl min{ a;1 ! , J = 1. . . . . m} : I .Am - dabs < M - jl < 1 . (11)
..
=
{ 15) A (m
x m ) real ,
wrresponding corresponding
A
- M
:
+ 11
> 0 , x > 0 ( m x 1 ) real eigenvec tor of A to eigenvalue p( A ) , y > 0 ( m x 1 ) real eigenvector of A' to eigenvalue p( A' ) , x'y 1 : Iirn [p( A ) - 1 Ar = x y' . =
( 1 6 ) \ ( m x m ) rea l A > O .
p(A)
=
·
,
a - oc
:
Note: Most results of t his subsection are, for instance, given in Horn 8.: Johnson ( 1 9 85 ) . 5.5
S ingular Values
Notation: O'( A ) O'mi n ( A ) O'max ( A ) 5.5.1
denotes a singular value of the matrix A . denotes the smallest singular value of the matrix A . denotes t he largest singular val ue of the matrix A.
General Results
( 1 ) A (m x n), m > H :
. {( mm
xH AH Ax XH X
)
l/2
X
( 11 X
1),
X
:j:. 0
}
79
EIGENVA L U ES A N D S I N G U L A R VA L U ES
(2) A ( m x n)
( { max
:
O"max ( A )
xH A H Ax
) 1 12
X(
n X
1),
X -::j; 0
}
XH X 1 m ax { ( x H A H A x ) 12 : x ( x 1 ) , x H x = 1 } . n
(3) A (m x ) r A, 1 < i < r : n
O";
,
=
min{ m, n } ,
.mm
y1 , . . , y , _ 1 (m x l )
u1 >
· · · > O"r
singular values of
( x H A H Ax ) 1 12 { max X
HX
x ( m x 1 ) , x -::j; O, x H yj = O, j = 1 , . . , i - 1
and u; = max y , , . . . ,Y m - • (m x l ) X (m X 1), (4) A (m
x
m)
X
.mm { (
-::j; 0,
xH A H Ax
XHX
X H Yj = 0, j =
}
) /2 l
1, . . . , m
-
i}.
nonsingular: : x ( m x 1 ) , x -:j; O
( 5 ) A ( m x n ) : u 1 , . . . , 0"r singular values of A values of A H . (6) A ( m x
n
<==:::>
.
u 1 , . . . , 0"r singular
) , r = min{ m, n } , O" J , . . . , O"r singular values of A : r tr(A H A) = L u? , i= I
.
( 7) (Singular value decomposition) A ( m x ) rk( A ) = r, u1 , . . , O"r -::f 0 singular values of A : exist unitary matrices U ( m x m ) , V ( x ) such that n
}
,
n
There
n
where D = diag( u1 , . . . , O"r ) and some or all of the zero sub mat ricE's disappear if r = m and/or r = n.
80
H A l\' D BO O K OF \1 AT R I C E S
( 8 ) ( Singular va]llt> decomposition of a real matrix ) ..! ( 111 x n ) rea l . r k ( A ) = r , t7 1 , . . . , t1"r =f 0 s i n gula r values of A : There exist real orthogonal matri c es [ ! ( m x 111 ) , V ( 11 x 11 ) such t hat
C'AV
[� �].
=
where D = d iag( t7 1 , . . . , t7, ) and sonw or all of t he zero sub 1 uat.rict's d isappear if r = m aud/or r = 11 .
(9) A ( m x n) : :\
=
('
] V 11 is t he singular value decomposit ion o [ �: ()) ] 0
0
( 1 0 ) A ( 111 x 11 ) r· = •
1 1 1 ill { m . n } ,
singular val ues of A -¢::::::} Zl'ros arc t lw eigeuvalues of
5.5.2 (1)
:\ . ·
·
(
0 < t7 t , . . . , t7r E ffi : t1" 1 ,
.
.
.
•
t1"r , - t7 1
•
.
.
.
•
.
t7 t , . . . . t7r i HP t lw - G"r an d !m - n ! ,�,,
Inequalities ·
fJ ( 1 11 x n ) . r = t u i u { m , n } . t7! { .4 ) > · · · > t7r ( .4 ) > 0 , t7! { B ) > > t7r ( H ) > 0 . t7! { A + fJ ) > · · · > t7r ( A + B ) > 0 singtllar values
of .4. .
H and A + B . rt>spectivrly :
(7'• +; - d A + fJ ) < t7, ( A ) + t7j ( B ) , ( 2 ) . \ . fJ ( m X n ) : (3)
{·H
..
f \
I < i, j <
r
with i + j < r + ! .
G"mar ( A + B ) < G"max ( A ) + G"mar ( B ) .
:1 . }] ·
·
.4. .
·
( m x n ) . r = m i n { m , n } . t7l { A ) > · · · > t7r ( A ) > 0 . t7 t ( B ) > > t7r ( H ) > 0 , t7 1 ( A B11 ) > · · · > t7r ( .4 B 11 ) > 0 singular val ues of lJ and A IJ11 • respectively :
[a 1 . . . . . a11,] ( 111 x 111 ) . Uj ( 111 x l ) , t7 t ( A ) values o f A :
( 4 ) :\
=
"
f1
"
<
> · · · > am ( .·\ ) si t tgular
L t7J ( A ) " .
j=l
11 = 1 . 2 . .
.
.
.
m.
81
EIG E N VA L U ES A N D S I N G U L A R VA L U ES x n ,
) B ( m x ( n - 1 ) ) is obtained from A by deleting any one column of A , r = min { m, n } , · · · > · · · > O" min( m ,n - l ) ( B ) singular values of A and B, respectively :
(5) A (m
if m
>
n
and
if m < ( 6 ) A ( m x ) B ( ( m - 1 ) x ) is obtained from A by deleting any one row of A . r = m in { m , } , · · · > O"r ( A ) and 0"1 ( B) > · · · > O"m1n ( m - l , n ) ( B) singular values of A and B , respectively : n.
n
,
n
n
· · · > 0
if m < n and if m >
n.
(7) A (m X n ) :
1
O"m a .r ( A ) > C t r ( A H .4 ) ) 12 .
Note: The results of this section can be found in Horn & Johnson ( 1 985 ) .
Other books discussing singular values include Barnet t ( I 990 ) , Lancaster & Tismenetsky ( 1 985 ) and , in particular, Chatelin ( 1 993) where further results may be found.
6
Matrix Decompositions and Canonical Forms All matrices in this chapter are complex unless otherwise specified. 6.1
Complex Matrix Decompositions
6. 1 . 1
Jordan Type Decompositions
( 1 ) (Jordan decomposition) A
There exists a ( m x m ) with distinct eigenvalues A 1 , . . . , A n nonsingular ( m x m ) matrix T such that A = TA T- 1 , where A is a Jordan fo rm, that is, :
A =
is block diagonal with blocks
A; =
An , 0 0 0
1 An ,
0 1
0 0
...
...
0 0
i = l,
. . . , k > n,
1 An ,
on the diagonal and { n 1 , . . . , nk } { 1 , . . . , n} , that is, the same eigenvalue may appear on the principal diagonal of more than one =
H A N D B O O K O F !\L\T R IC ES A; .
For example, 2 0 0
:\ =
I
0
2
I
()
0
2 [2]
0
is a Jordan form of a matrix with the two distinct eigenvalu('s A 1 = 2 , A 2 = 3 . 2 and 2 0 0
I
0
2 0
I
t\
-
, 3 -
A z = [2] .
2
[
3.2 0
( 2 ) ( J ordan decomposition of a matrix whose eigenvalues are all distinct ) A ( m x m ) with m distinct eigenval ues .\ 1 , . . . , .\m : A = T.t\1'- 1 , where A = diag( A 1 , . . . , A m ) and T is a nonsingular ( m x m ) mat rix whose colu mns are eigenvectors of A corresponding to A 1 , . . . , A711 •
( 3 ) ( Near-d iagonal J ordan decomposition) A
( m x m ) wit h d istinct eigenval ues A 1 , . . . , A,.. , f > 0 a given real nu mber: There exists a nonsingular ( m x m ) matrix T such t h at 1 A = TAT- • w here A I (f)
0
0
Ak(f)
A =
is block diagonal with An , A;(f)
=
(
0
An '
0
0
0
0
0 (
0 0
i = l , . . . , k > n.
and { n 1 , . . . nk } = { l , . . . , n } , t h at is, the same eigenvalue appear on t he principal d iagonal of more t h an one A ; ( f ) . .
n 1 a \'
.
For proofs see, e.g. , Horn & Johnson ( 1 985, Chapter 3 ) or Bamet t ( I 990 , Chapter 8 ) . Note:
85
M ATRIX DECO M POSITIONS A N D C A N O NICAL FO R MS 6.1.2
Diagonal Decompositions
( 1 ) ( Spectral decomposition of a normal m atri x ) A ( m x m) normal w i t h eigenvalues A1 , . . . , Am : A = U AU11 , where A = d i ag( A 1 , . . . , Am ) and U is the unitary ( m x m ) m atrix whose columns are the orthonormal eigenvectors v 1 , . . . , Vm of A a...<;sociatrd with A 1 , . . , Am . I n other words, .
m A = L A;V;V� . i= I
( 2 ) ( Spectral decomposition of a H ermitian m atrix) A ( m x m ) H ermitian with eigenvalues A 1 . . . , Am
A = U A U 11 .
where A = d iag( A 1 , . . . , Am ) and U is the unitary ( m x m ) matrix whose columns are the orthonormal eigenvectors v1 , . . . , t'm of A associated with A 1 , . . . , Am . In other words, ,
m A = L A; ";vf . i= I
( 3 ) ( Singular value decomposition) A ( m x n ) , rk( A ) = r , <11 , . . . ,
A-u
There
[ D0 0 ] V II , 0
where D = diag( <11 , . . . ,
( 4 ) ( Diagon al reduction) A ( m x n ) . rk( A ) = r : 5 and a nonsingular ( n
There exists a nonsingular ( m x m ) mat rix x n ) matrix T such t h at
where some or all of the zero su b m at rices disappear when and/or r = n .
r·
m
( 5 ) (Congruent canonical form) A ( m x m) H ermi tian : There exists a nonsingular ( m x m) mat rix T such that A = TAT11 , w here A = diag(d1 , , dm ) anci tlw d, are + 1 , - 1 or 0 corresponding to the positive, negative and zero eigenvalues of A , respectively. .
•
.
86
H A N DBOOK O F M ATRICES
( 6 ) A ( m x m) symmetric, rk( A ) ( m x m ) matrix T such that
r
There exists a nonsingular 0 0
] T'
where the zero submatrices disappear when
' r
= m.
( 7 ) A ( m x m) : There exists a nonsingular ( m x m) matrix T, a unitary ( m x m ) matrix U and an ( m x m) d iagonal matrix A with nonnegative elements on the principal diagonal such that A = TU AU'T - 1 . S imultaneous D iagonalization of Two Matrices ( 8 ) ( Simultaneous d i agonalization of two Hermitian matrices) A , B ( m x m ) Hermitian , A B = BA : There exists a unitary ( m x m) matrix U such that A = U DU H and B = U AU H , where D and A are d i agonal matri ces. ( 9 ) ( Simultaneous d i agonalization of a positi ve definite and a H ermitian
matrix ) A ( m x m) Hermitian positive definite, B ( m x m) Hermitian : There exists a nonsingular ( m x m) matrix T such that A = TTH and B = T ATH , where A is a diagonal matrix. ( 1 0 ) ( Simul tan eo us d i agonalization of a positi ve definite and a symmetric matri x ) A ( m x m ) positive definite, B ( m x m ) sym metric: There exists a nonsingular ( m x m ) m atrix T such that A = TTH and B = TAT' , w here A is a real diagonal matrix with nonnegative diagonal elements. Note: The results of this subsection and further diagonal decompositions may be found in Horn & J oh nson ( 1985 , Chap ters 3 , 4 and 7 ) and Rao ( 1 973, Chapter 1 ) . 6.1.3
O t her Triangular Decompositions and Factorizations
( 1 ) ( Schur decomposition) There exists a unitary ( m x m ) matrix U and an A (m x m) : upper triangular matrix A with the eigenvalues of A on the principal diagonal such that A = U AU H . ( 2 ) ( Choleski decomposition) A ( m x m) positive definite: There exists a unique lower ( upper) triangular ( m x m) matrix B with real positive principal diagonal elements such that A = B B H .
87
M AT RIX DECOM POSITI O N S A N D C A N O N ICA L FO R M S
( 3 ) (Triangular factorization) A = [a;j] ( m x m ) , rk( A ) =
r,
# 0,
det
n = l, . . . , r :
There exists a lower triangular ( m x m ) matrix L and an upper triangular ( m x m) m atrix U , one of which is nonsingular, such that A = LU. ( 4 ) (Triangular factorization of a nonsingular matrix) A = [a;j] ( m x m) nonsingular, det
# 0,
n = l,
. . .
,m:
There exists a unique lower triangular ( m x m ) matrix L and a unique upper t ri angul ar ( m x m) m atrix U both having unit principal dm ) with diagonal such that A = L DU, where D = diag( d 1 , .
a1 1
n
II dj
i= l
•
•
,
aln n = 1 , . . . , m.
= det an i
. .
.
an n
( 5 ) (G ram - Schmidt triangular redu ction) There exists an ( m x n ) m atrix U with A ( m x n) , m > n : orthogonal colu m ns and an upper triangu lar ( n x n) matrix R such t h at A = UR. ( 6 ) ( G ram- Schmidt triangular reduction of a square matrix)
A ( m x m) : There exists a unitary ( m x m) m atrix [T and an upper triangular ( m x m ) m atrix R such that A = U R.
( 7 ) ( Gram - Schmidt triangul ar reduction of a nonsingular matrix) A ( m x m) nonsingular : There exists a unique u nitary ( m x m ) m atrix U and a unique upper triangular ( m x m) matrix R with positive principal diagonal such that A = U R. (8) A (m x m) : There exist (m x m) permutation m at rices P and Q, a lower triangular ( m x m) matrix L and an upper triangu lar ( m x m ) m atrix [T so that A = P LUQ. ( 9 ) A (m x m ) nonsingul ar :
There exists an (m x m ) permutation m atrix P , a lower triangular ( m x m) matrix L and an upper triangular (m x m) matrix U such that A = P L U .
88
H A N DBOOK O F M ATRICES
Note: For proofs and further details see, e.g. , Horn & J ohnson ( 1 985, Chapters 2 and 3 ) . 6.1.4
Miscellaneous Decompositions
( 1 ) ( Rank factorization) A ( m x n ) , rk( A ) = r : There exists an ( m x r) matrix B and an ( r x n ) m at rix C with rk( B) = rk(C) = r such that A = BC. ( 2 ) ( Polar decomposition )
A ( rn x m ) , rk( A ) r : There exists a p ositive semi definite ( m x m ) matrix P with rk( P) = r and a unitary ( m x m ) mat rix U such t h at A = PU. =
(3) A (
) nonsingular : There exists a symmetric ( m x m ) matrix ( complex) orthogonal ( m x m ) m atrix Q such that A = PQ.
m x m
P and
a
( 4 ) ( Square root decomposition ) A ( m x m ) p ositive (semi ) definite: There exists a positive ( sem i ) definite ( m x m ) matrix B such that A = B B , that is, B i s a square root of A . ( 5 ) A ( m x m ) unitary : There exists a real orthogonal ( m x m ) matrix Q and a real sy mmet ric ( m x m ) matrix S such that A = Qexp( iS). ( 6 ) A ( m x m ) orthogonal : There exists a real orthogonal ( m x m ) matrix Q and a real skew�symmetric ( m x m ) m atrix S such t hat A = Qexp( iS).
( 7 ) ( Echelon for m ) A ( m x m ) : There exists a nonsingular ( m x m ) matrix B such that BA = [c;j] is in echelon form, that is , for each row i , either c;j O , j = l , . . . , m , or there exists a k E { 1 , . . , m } such t h at Cik I and Cij = 0 for j < k and Cf k = 0 for I ,P i . ( For more details on the echelon form see the Append ix . ) =
.
=
( 8 ) ( Hermite canonical for m ) A ( m x m ) : There exists a nonsingular ( m x m ) m atrix B such that BA is an idempotent H ermite canonical form. ( For the definition of a Hermite canonical form see the Append i x . ) ( 9 ) A ( m x m ) : There exists a d i agonalizable ( m x m ) matrix D and a nilpotent ( m x m ) m at ri x N such that A = D + N and DN = N D. Note: For results ( 1 ) , ( 7 ) and (8) see Rao ( 1 973, Chapter 1 ) . The other results may be found i n Horn & Johnson ( 1 985, Chapter 3 ) . Both referellet'S contain a number of further decomposition results.
89
M ATRIX D EC O M POSITIONS A N D C A N O N ICAL FO RMS
6.2 6.2. 1
Real Matrix Decompositions
Jordan Decompositions
( 1 ) (Jordan decomposition of a real matrix with real eigenvalues) A ( m x m ) real with distinct eigenvalues A 1 , . . . , An which are all real : There exists a real nonsingular ( m x m ) matrix T such that A = TAr- • , w here
At
0
0
Ak
A= is block diagonal w i th
A; =
An, 0
1 An ,
0
0
0
0 1
. . .
0 0 .
i = 1, . . , k > n, 1 An ,
0
{ 1 . . . , n } , that is, the same eigenvalue may and { n 1 , . . . , n k } appear on the principal d iagonal of more than one A ; . ,
( 2 ) ( Real Jordan decomposition) A ( m x m ) real with distinct real eigenval ues A 1 . . . , Ap and distin c t There exists a real complex eigenval ues o 1 ± i/31 , . . , O q ± i /3q : nonsingular ( m x m ) matrix T such that A = TAT- 1 , where ,
.
0
A =
r. r,
0 is block d iagonal with
A; =
A p, 0
1 .\p ,
0
0
0
0
0 1
. .
.
0 0 .
i = 1 , . . , k > p, 1 A p,
H A N D BOOK OF M AT R ICES
90
{P I · . . , pk } = { I , . . . , p} , and .
[
/3q, O: q,
O: q , - i3q ,
f; =
0 0
]
[
/2
]
0
[2
0
0
0
i = 1' . . . ' I
/3q, O: q,
O:q, - /3q,
0
[
0
> q ' { q I ' . . . ' ql } = { 1 ' . . . ' q } .
[2
O: q , -/3q,
i3q , O: q,
]
Note: For proofs and further details see Horn & J ohnson ( 1985, Chapter 3) or Barnett ( 1 990. Chapter 8 ) . 6.2.2
O ther Real B lock Diagonal and Diagonal Decompositions
( 1 ) A ( m x m) real normal ( A' A = AA') : ( m x m) m atrix Q such that
There exists a real orthogonal
0 Q' .
A=Q 0
where the A, are real numbers or real ( 2 x 2 ) matrices of the form A; =
[ �: !: ] . -
( 2 ) (Spectral decomposition of a real symmetric matrix ) A ( m x m ) real sym metric with eigenvalues A 1 , . . . , Am : A = Q AQ' , where A = di ag( .\ 1 , . . . , .\m ) and Q is the real orthogonal ( m x m ) m atrix whose columns are the orthonormal eigenvectors v 1 , Vm of A associated with A 1 , . . . , Am . In other words, .
m
A=
L A;v; v: . I=
I
.
.
•
91
M AT RIX D ECOM POSITI ONS A N D CANONICAL FO R M S (3) A
(m
(m
x
x m
m
) real skew -symmetric:
There exists a real orthogonal
) m atrix Q such that 0
0 0
A =Q
Q' ,
0
where t he A ; are real ( 2
x
2 ) m atrices of the form b; 0
(4) A (m
x m
) real orthogonal :
m atrix Q such t h at
]
.
There exists a real orthogonal ( m x m ) 0
,\ I
Q' ,
A = Q 0
where t he
,\ ;
= ± 1 and t he A ; are real ( 2 _ A t. -
[ -Sincos B; •
()i
2 ) matrices of the form
x
sin B; cos B;
].
( 5 ) ( Singu l ar value decomposition of a real m atrix) A ( m x n ) real , rk( A ) = r, O"J , . . . , O'r ::j:. 0 singular values of A There exist real orthogonal m atrices U ( m x m ) , V ( n x n ) such that A=U
[D ] O
0 O
1
V ,
where D = diag( u1 , . . . , O'r ) and some or all of the zero submatrices disappear when r = m and/or r = n . Note: The results ( 1 ) - (4) are taken from Horn & Johnson ( 1 985, Chapter 2 ) and for ( 5 ) see Chapter 7 of the same reference.
H A N DBOOK O F M AT RICES
92
6.2.3
O t her Triangular and Miscellaneous Reductions
( 1 ) ( Schur decomposition of a real m atrix w ith real eigenvalues) A ( m x m) real with real eigenvalues : There exists a real orthogonal ( m x m ) matrix Q and a real upper triangular matrix A with the eigenvalues of A o n the principal d iagonal such that A = Q A Q' . ( 2 ) ( Choleski decomposition of a real matrix) A ( m x m) real positive definite : There exists a u n ique real lower ( upper) triangular ( m x m) m atrix B with positive diagonal elements such that A = B B' . ( 3 ) ( G ram-Sch m idt t riangular reductio n of a real matri x ) There exists a real ( m x n) matrix Q A (m x n) real , m > n : with orthogonal columns and a real upper triangular ( n x n ) matrix R such that A = Q R .
(4)
.4 ( m x n ) real , m >
: There exists a real orthogonal (m x m) matrix Q and a real upper triangular ( n x n ) m atrix R such that n
QA -
[
R O(m - n ) x n
where the zero m atrix disappears i f n
=
]'
m.
( 5) ( G ram -Schmidt triangular reduction of a real square matri x ) .4 ( m x m) real : There exists a real orthogonal (m x m) matrix Q and a real upper triangular ( m x m) mat rix R such that A = Q R . ( 6 ) ( G ram - Schmidt triangular reduction o f a real nonsingular m atri x ) A ( m x m ) real nonsingular : There exists a unique real orthogonal ( m x m) matrix Q and a unique u pper triangular real ( m x m) m atrix R with positive principal d i agonal such that A = Q R. (7)
A
(m
x m ) real :
Therr exists a real orthogonal ( m x m) matrix Q and au upper H esseuberg matrix R such that A = QRQ' .
( 8 ) (Square root decomposition) A ( m x m) real p ositive ( se m i ) defi n i te: There exists a real positive ( semi ) definite ( m x m ) matrix B such that A = B B , that is, B is a square root of 4 .
.
( 9 ) ( Simul taneous d iagonalization of a real positive defin ite and a real
sy mmetric matrix ) There A ( m x m) rral positive definite, B (m x m) real symmetric: exists a real nonsingular ( m x m) matrix T such that A = TT' and B = TAT' , where l\. is a real d iagonal matri x . Note: These decomposition and factorization theorems can be fou n d , for instance, in Horn & Johnson ( 1 98 5 ) and Rao ( 1 973 ) A lgorithms for comput i ng .
M AT RIX D ECO M POSITIONS A N D CANONICAL FO R M S
93
m any of the m atrix factors d iscussed in this chapter are described in Golub & Van Loan ( 1 989) .
7 Vectorization Operators All matrices i n this chapter are complex unless otherwise specified.
7.1
Defi nit ions
vee denotes the col u m n vectorizi ng operator which stacks t he columns of matrix i n a colu m n vector , that is, for an (m x n) matrix A = [a;j ] ,
vee A = vee( A) = col( A ) _
Related operators are
rvec( A) [vee( A')]'
( mn x 1 ) .
a
H A N DBOOK OF M ATRICES
96
which stacks the rows of A i n a row vector and
row( A ) = vec ( A ' )
=
( mn x 1 )
rvec ( A ) ' =
which stacks the rows of A in a column vector. For example,
vee
aII a2 1 a3 1
a12 a 22 a32
aI I a2 1 a31 a12 a 22 a32
and
aII a12 aI I a12 a2 1 row a 2 1 a 22 a 22 a3 1 a 32 a31 a32 The half- vectorization operator, vech , stacks only the columns from the principal diagonal of a square m atrix downwards in a column vector, that is, for an ( m x m) matrix A = (a ;j ] ,
vech A = vech( A )
=
q m(m + 1 ) x 1 ) .
Umrn
97
V ECTO RI Z ATION O P E RATO RS
For example,
vech
aI I a2 1 a3 1
a12 an a32
a13 a 23 a 33
a1 1 a2 1 a3 1 an a 32 a 33
I n the fol lowing sections results are given for vee and vech only. Similar results can be obtained for other vectorization operators from the results given. The following matrices are closely related to the vee and vech operators : •
•
•
7.2
Kmn or Km.n is an ( m n x m n ) commutation matrix defined such that for any ( m x n ) m atrix A , Kmn vec ( A ) = vec ( A' ) ( see Section 9 . 2 for its pro perties) . Dm denotes the ( m 2 x b m( m + 1 ) ) duplication m at rix defined such that vec ( A ) = Dm vech ( A ) for any symmetric ( m x m ) matrix A (see Section 9 . 5 ) . Lm denotes the ( b m ( m + 1 ) x m 2 ) elimination matrix defined such that vech( A ) = L m vec ( A ) for any (m x m) m at rix A (see Section 9.6). Rules for t he vee O perator
( m x n) : (2) A (m x n), c E (3) A (m x n) : ( 1 ) A, B
vec ( A ± B ) = vec ( A ) ± vee( B ) .
vec (cA) = c vec ( A ) .
(a) vec ( A ' ) = A"mn vee( A ) .
( b ) vec ( A ) = /\"n m vec ( A ' ) .
x m) : Lm vec ( A ) = vech ( A ) . ( 5 ) A ( m x n ) , B ( n x p) : (4) A (m
vee( A B) = ( /p (6) a ( m
0
A )vec ( B ) = ( B ' 0 Im )vec ( A ) = ( B ' 0 A ) \·ec ( ln ) .
x 1 ) , b ( n x 1 ) : vee ( ab' ) = b 0 a. ( 7) A ( m x n ) , B ( n x r ) , C ( r x s ) : vec( A BC) = (C' ® A )vec ( H ) . ( 8 ) A ( m x n ) , B ( r x s) : ( a) vee( A G B ) = Un 0 1\,m 0 lr ) ( vee( A ) 0 vee( B ) ) . ( b ) vec ( A ' ·Sl B ) = ( /\m• . n 0 lr ) ( vec( A ) 0 vec ( B ) ) . ( c ) vec( A 0 B' ) = ( /n 0 Kr, m • ) ( vec ( A ) 0 vee( B ) ).
H A N D BOOK OF M ATRICES
98
(d) vee( A' 0 B ) = ( Km , , n ( In 0 Km • ) 0 lr )vec(A 0 B ) . (e) vec( A 0 B') = ( In 0 Kr, m • ( Km • 0 lr ))vec( A 0 B ) . (f) vee( A ) 0 vee( B ) = ( /n 0 Km • 0 lr )vec(A 0 B ) . ( 9 ) A , B ( m x n) : ( a) vee ( A 8 B) = diag(vec A)vec(B) = diag(vec B)vec ( A ) . ( b ) vee( A 8 B ) = (vee A ) 8 vec( B) . (c) vec( A )'vec ( B ) = vec( B)' vec ( A ) . ( 10 ) A ( m x n ), B ( n x m) ( a) vec (A')'vec( B ) = tr(A B ) = tr( BA) . ( b ) vec(A' )'vec( B) = vec( B' ) ' vec( A ) . ( 1 1 ) A ( m x n ) , B ( n x p ) , C (P x q ) , D ( q x m ) :
:
vec( D')'(C' 0 A)vec ( B ) - tr(A BC D) - tr(DABC) - tr(CDA B ) - tr( BC DA).
( 12 ) A ( m x n) , B ( n x p) , C ( P x q ) , D (q x m ) :
vec( D' ) ' ( C' 0 A )vec( B) - vec(A' )'( D' 0 B)vec (C) vec ( B' )' ( A' 0 C)vec(D) vee( C' )' ( B' 0 D)vec( A) vec(D)'( A 0 C')vec ( B' ) .
( 13 ) A ( m x n) , B ( n x p) , C (p x q ) , D ( q x m ) :
vec( D' ) ' ( C' 0 A)vec(B) - vec( D' )'vec( ABC) - vee( A' D' )'vec ( BC) - vee( A' D'C' )'vec ( B ) .
( 14 ) A (m x m) :
( a) A lower triangular => vee( A ) = L;, vech ( A ) . ( b) A symmetric => Dt vec(A) = vech(A ) . (c) A sym metric => vec(A) = Dm vech(A). ( 15 ) A ( m x m ) : ( a) n;, vec ( A ) = vech( A + A ' - dg( A ) ) . ( b ) Dt vee( A ) = ! vech ( A + A').
99
VECTO RIZATION OPE RATO RS
( 16 )
(c) Dm v;, vee( A) == vee( A + A' - dg( A ) ) . ( d ) vec(dg( A ) ) = L;, L m Kmm L;, L m vec(A). A ( m x m ) , a, b (m x 1 ) , c E
[a A v;,+1 vee c b' ] = + Dm + l vee
( 1 7)
[ ac Ab' ] --
c a+b v;, vee( A) c � ( a + b) D;!; vec ( A )
A , S, V ( m x m) : S = ASA' + V and lm > - A 0 A nonsingular vec(S) = Urn> - A 0 A) - 1 vec( V ) .
Note: A very rich source for results on the vee operator is Magnus ( 1 988 ) . M agnus & Neudecker ( 1 988) also contains many of the results given here. 7.3 (1)
(2) (3)
(4)
(5)
Rules for t he vech Operator
A , B ( m x m) : vech(A ± B) = vech ( A ) ± vech ( B) . A ( m x m), c E vech(cA) = c vech(A). A, B ( m x m) : ( a) vech (A 8 B ) = diag(vech A)vech ( B) = diag(vech B)vech ( A ) . ( b ) vech ( A 8 B) = (vech A ) 8 vech( B). A ( m x m) : ( a) vech ( A ) = L m vec(A). (b) vech( A + A') = 2 D;!; vec(A). (c) vech ( A + A' - dg(A)) = D;, vee( A). (d) D;, Dm vech( A ) = 2vech(A) - vech(dg( A ) ) . (e) vech (dg(A ) ) = Lm Km m L� vech(A). A ( m x m) : (a) A lower triangular => L;, vech( A ) = vee( A ) . ( b ) A symmetric => vech( A ) = D;!; vec( A ) .
H A N D BOOK O F M AT RI C ES
1 00
(c) A symmetric => Dm vech ( A ) vec ( A ) . ( 6 ) A ( m x n ) , B ( n x p ) , C ( p x m ) : vech ( A BC) = L m (C' 0 A )vec ( B ) . ( 7 ) A , B ( m x m ) : vech( A ) ' vech ( B ) = vech( B)' vech ( A ) . =
M any results for the vech operator including the nontrivial ones given here are col lected in Magnus ( 1 988). Note:
8
Vector and Matrix Norms 8.1
General Defi nitions
function I I · / I attaching a nonnegative real number / l A / I to an ( m x n ) matrix is a norm if the fol lowing three conditions are satisfied for all complex ( m x ) matrices A , B and complex numbers c : A A
n
(i) (ii) (iii)
/ IA II
>
0
if A ::f 0 ,
/ /c A / / = / c / abs / I A / 1 ,
( triangle inequality). + Bll < IIAII + IIBII Here /c/abs denotes the modulus of c , that is, / c/ abs = y'CC with complex conjugate of c. If (i) is replaced by I IA
c
being the
( i )' /\A\\ > 0 . is said to be a seminorm. I n this case 1 / A / 1 = 0 is possible even if A ::f 0 . Instead of defining a norm for all complex ( m x n ) matrices, it may be defined for real matrices only. If (i), ( ii) and (iii) hold for all real ( m x n ) mat rices and real numbers c, 1 / · I I is called a norm over the field of real numbers ( ffi ) . In that case / c / abs is, of course, simply the absol ute value of c. Since vectors are ( m x 1 ) or ( 1 x ) matrices, norms are defined for vectors as well. Let l l · l l a and 1 / · 1 / b be norms for ( m x 1 ) and ( x 1 ) vectors , respect ively. The norm I I · I I for ( m x ) matrices is compatible wit h /1 · / I a and I I · l i b if II · II
n
n
n
1/A x//a < I IA II I I x/l b
for all ( m x ) matrices A and ( x 1 ) vectors x . A norm \ \ · / I a for ( m vectors is compatible with a norm for ( m x m ) matrices I I · I I if n
n
x
1)
1 / Ax l l a < I I A / I l l x \ / a
for all ( m x m) matrices A and (m x 1 ) vectors x . An (m x m) matrix A is called an isometry for a norm 1 1 · 1 1 for ( m x 1 ) vectors if 1 / A x / 1 = 1/ x /1 for all ( m x 1 ) vectors x .
H A N DBOOK O F M ATRICES
1 02
A norm I I · II for ( m x m ) matrices is called multiplicative or submultiplicative if for all ( m x m ) norm .
I I AB I I < I I A II I I B I I matrices A , B . A multiplicative norm is said to be a matrix
{
Ax l l : II sup A x II IIIub = l l x ll
(m x 1 ) ,
x =f.
0
}
is t he matrix norm induced by a norm II · II for ( m x 1 ) vectors .r . Alternatively i t is called sup norm o r operator norm or lub ( least upper bound) norm. A norm for ( m x n) matrices A is unitarily invariant if !IU A V II = II A ll for any two unitary matrices U (m x m) and V (n x n) . A norm 1 1 · 1 1 for ( m x n ) matrices is monotone if for any two (m x n ) matrices A = [aij ] . B = [bij] the fol lowing holds: l a ij l abs < lbij labs , i = 1 , . . , m , j = 1 , . . , n ·
.
::}
I I A II < I I B I! .
A norm for ( m x n) matrices is absolute if for all (m x n ) matrices A = [a,1 ] ,
II A ll = II [ l a ii l ab J I I · s
A matrix norm I I · II is said to be a minimal matrix norm , if for any matrix norm II · !I a the fol lowing holds:
I I A I! a < II A II for all
(m x m ) matrices
A
::}
II A II = !l A l l a for all A .
A m at rix no rm 1 1 · 1 1 is self-a djoint if I I AH I I = II A I I for all ( m x m ) matrices
A.
Related to norms are inner products. A function < · , · > attaching a complex number < A , B > to any two (m x n) matrices A , B is called an inner product if for any arbi trary (m x n ) matrices A , B, C and c E
< A , A > > 0 if A =f. 0, < A + B, C > = < A, C > < cA, B > = c < A , B > , < A , B > = < B, A > .
+
< B, C > ,
function < · , > is an inner product over the field of real numbers if i t attaches a real number to a pair of real matrices in such a way that for any real (m x n) matrices A , B, C and c E lR the fol lowi ng conditions are satisfied: ·
( i) < A, A > > 0 if A =f. 0,
103
VECTO R A N D M ATRIX N O R M S
(ii) < A + B , C > = < A , C > + < B, C > , (iii) < cA , B > = c < A , B > , (iv) < A , B > = < B, A > .
If < · > is an inner product for ( ) matrices, IIAII = < A , A > 1 / 2 is a norm for ( ) matrices. II I I is said to be a norm derived from the ·,
m x n
m x n
inner product
<
·
·
,
>.
·
Note: More information on vector and matrix norms can be obtained from many books on matrices and li near algebra. Almost all results of this <:hapter can be found in H orn & Johnson ( 1985). 8.2
S pecific Norms and Inner P roducts
Special norms for
(m
x n
) matrices A
(a ,i ) arP :
=
Euclidean norm, /2-norm, Frobenius norm, Hilbert-Schmidt norm or Schur norm: 1/2 m
n
for complex matrices.
L L l aij l ;bs i=l j = l m
n
I I A I I 2 = )t r(A ' A ) = L L l aij l �bs
1/ 2
i=l j=1
for real matrices.
lp-norm or M inkowski p-norm:
I I A II P Mahalanobis norm:
definite (
m x m
Maximum norm:
=
1/p
n
for 1 < p <
L L l aij ��bs i=l j=1
I IAI I n
=
[ tr( A H QA) ] 1 1 2 , where
) matrix. I I A II oo = max { l a ij l abs : i = 1, . . , m , .
Column-sum norm:
Row-sum norm:
m
II A I I col _ max
{f ·= 1
n
j a;j l abs : j
I I A II row - max L j a;j labs j=1
:i
==
=
1,
=·
r2
j
is a positive ==
1, . .
. .
. . . , m
1 , . . , n} .
n
}
.
.
1 04
H A N DBOOK O F M ATRICES
Spectral norm or max1n1um singular value norm or operator norm: •
II A II•pec = O"max ( A ) = max{ J:\ : ..\ is eigenvalue of A H A} , that is, ma x ( A) is the maximum singular value of A . Special inner products for ( m x n) matrices A : < A , B > _ tr(AH B ) for complex matrices A , B . < A , B > = tr( A' B ) for real matrices A, B. u
8.3
Res u lts for General Norms and Inner P roducts
( 1 ) (Cauchy-Schwarz inequality)
< · , · > an inner product for (m x n) matrices (over A, B (m x n) :
(2) (3)
ffi
or <[;) ,
I < A, B > 1 ;bs < < A, A >< B, B > . < · , · > an inner product for (m x n ) matrices, A , B ( m x n ) : I < A, B > 1;bs = < A , A > < B, B > -<===> A = cB for some c E an inner product over ffi for real (m x n ) matrices, A, B (m x n ) real:
I < A , B > l � bs = < A , A >< B, B >
-<===>
A = cB for some c E ffi .
(4) ( Parallelogram i dentity) II · II a norm for ( m x n ) mat rices derived from an inner product, A, B (m x n ) :
II A + B ll 2
+
I I A - B ll 2 = 2 ( II A II 2 + II B II2 l·
( 5 ) ( Pythagoras theorem ) 1 1 · 1 1 a norm for ( m x n ) matrices derived from an inner product < · . > , A, B (m x n ) : ·
< A , B > = 0 => II A + B l l2 = II A II 2 + II B W . ( 6 ) ( Polarization identity) 1 1 · 1 1 a norm for ( m x n ) matrices derived from an inner product < · , · A, B (m x n) : Re < A. B > ! ( I I A + B ll 2 - II A - B W ) . Re < A . B > � ( II A + B ll 2 - IIAII 2 - II B II 2 ) . where Re denotes the real part of a complex number.
>.
105
V ECTO R A N D M AT R I X N O R M S
( 7 } I I · I I a norm for (m x n) matrices derived from an inner product, A , B (m x n) :
I I A + BII I IA - Bll
< I I A I I2 + I I B I I2 -
( 8 ) 1 1 · 1 1 a seminorm for (m x n) matrices, A , B (m x n ) : I I I A I I - II B I I I abs
< I!A - Bll.
( 9 } l l · l l a , I I · l i b norms for ( m x n) matrices A , c E IR, c > 0 : (a) (b) (c)
I I A I I = c j J A I I a is a norm for (m x n ) matrices. I I A I I - I I A I I a + I I A I Ib defines a norm for ( m x n ) matrices. I I A l l = max( JI A IIa , I I A I I b ) defines a norm for ( m x n ) matrices.
( 10) 1 1 · 1 1 a norm for (m x 1 ) vectors x, T ( m x m) nonsingular: l lxiiT - I JTxJ I is a norm for ( m x 1 ) vectors . ( 1 1 ) 1 1 · 1 1 a seminorm for ( m x 1 ) vectors x, T (m x m ) : l lxiiT = I I Txl l is a seminorm for ( m x 1 ) vectors. ( 12) 1 1 · 1 1 a norm for ( m x m ) matrices, y (m x 1 ) , y :f. 0 : l lxJjy = I JxyH J I is a norm for ( m x 1 ) vectors. ( 13} 1 1 · 1 1 a norm for ( m x m) matrices , S, T ( m x m) nonsingular: I I A I Is,T = I I SATI I is a norm for (m x m) matrices. ( 14 ) 1 1 · 1 1 a norm for ( m x m) matrices, S = [s;j] (m x m ) , s;J :f. 0, i , j = 1 , . . . , m : I I A I I 0 = l i S �> A l l is a norm for ( m x m ) matrices . ( 15 ) l l · l l a , I I · l i b norms for (m x n ) matrices : There exist positive constants Ct , c2 E IR s uch that c t J I A I I a < I I A I I b < c2 I I A I I a for all ( m x n ) matrices A.
( 16 ) 1 1 · 1 1 a norm for (m x n ) matrices: II · I I is monotone {::::::} I I · I I is absolute. ( 1 7) 1 1 · 1 1 an absolute norm for (m x n ) matrices, A , B (m x n ) : I A i abs < ! B la bs ::} II A I I < I I B I I There exists a norm for ( 18 ) I I · I I a norm for (m x m ) matrices: ( m x 1 ) vecto rs which is comp atible with I I · I I ::} I I A t l l · · · I J A ; I I > p(A t · · · A ; ) = max{ J.A i abs ..\ is an eigenvalue of A 1 · · · A ; } for all , A ; and i = 1 , 2, . . . (m x m) matrices A 1 , :
•
•
•
Note: The results of this section may be found in Horn & Johnson ( 1985, Chapter 5 ) .
H A N DBOOK O F M ATRICES
1 06
8.4 8 .4. 1 (1)
(2)
Results for Mat rix Norms
General Matrix Norms
1 1 · 1 1 a matrix norm for (m x m) matrices, c E IR, c > 1 : II A II c = ciiAI I is a matrix norm for (m x m) matrices. I I · I I a matrix norm for (m x m) matrices, S (m x m) nonsingular: I I A I I s = I I S 1 A SI I is a matrix norm for (m x m) matrices . 11 · 11 ( 1 ) • . . . . l l · ll(n) matrix norms for (m x m) matrices: I I A II = max{ I I A I\( l ) • · . . , II A \\(n ) } is a matrix norm for (m x m ) matrices. 1 1 · \\ a matrix norm for (m x m) matrices: I I A I \ H = II A H \ I is a matri x norm for (m x m) matrices . \ \ · \\ a norm for (m x m) matrices: There exists c E IR, c > 0 , such that 1 \ A \ I c = c\\A\\ is a matrix norm for (m x m) matrices. \\ · \ \ a norm for (m x m) matrices: -
(3)
(4)
(5) (6)
\\ A\\m ax 1
=
max{ I I AB\\ : B (m x m) , 1\B\ 1 = 1 }
and
1 1 A IImax 2 = max{ I I BA I I : B ( m X m) , II B II = 1 } are matrix norms for (m x m) matrices. ( 7 ) 1 1 · 11 a matrix norm for (m x m) matrices, S, T ( m x m) nonsingular: II A I I s,T = I I S AT II is a norm for (m x m) matrices which may not be a matrix norm , that is, II · ll s,T may not be submultiplicative. ( 8 ) 11 · 11 a matrix norm for (m x m) matrices , S = [s;j ] (m x m) , sij -# 0. i, j 1 , . . . , m : I I A I I 0 = I I S 0 A I I is a norm for (m x m) matrices which may not be a matrix norm . ( 9 ) 11 · 1 1 a matrix norm for (m x m) matrices, A (m x m) : ( a) l l lm l l > 1 . ( b ) 1 \ A k l l < I I A I \ k for k = 1 , 2 , . . . (c ) I I A I I > p( A ) = max { \ .A iabs : ..\ is eigenv alue of A } . (d) lim I I A i l l 1 / i = p ( A ) = max { \.Aiabs : ..\ is eigenvalue of A } . (e) A -# 0 idempotent => II A II > 1 . (f) A singular => l l lm - A l l > 1 . (g) I I Im - A\1 < 1 => A nonsingular. =
• - oo
(h) A non sing ular => I I A - 1 1 1 >- \ I Im l l . 1\A\1
107
V ECTO R A N D M ATRIX N O R M S
(i) ! l im ! ! = 1 , ! ! A l l < 1 ::}
1
1 < + I!A \1
II ( Im -
A) - 1 , I <
1 1 - I! AI I
(j ) I I A I I < 1
l l lm \ 1 < l l lm l l + I I A I I -
::}
I I ( !m
_
A} - 1 1 1 < -
lllm l l - (lllm ll 1 - IIAII
( 10} 1 1 · 11 a m atrix norm for ( ) matrices , A, B ( (a) I I A B - Im l l < 1 ::} A , B nonsingular. m x m
(b) A nonsingular, B singular ::} ! l A - B ll ( 1 1 ) 1 1 · 1 1 a m atrix norm for ( A + B nonsingular:
m x m
1 )11AII
>
m x m
)
.
:
�
' !lA I I!
) matrices, A , B ( m x m ) , A nonsingular,
(a) II A - 1 - ( A + B) - 1 1 1 < ! I A - 1 I I I I ( A + B) - 1 I I II BII · (b) (c)
I I A - 1 B ll I I A - 1 - ( A + B) - 1 11 1 < I I A - B l l < 1 ::} 1 IIA- 11 1 - I I A - 1 Bll 1 I I A - 1 B ll < 1 and I ! BII < IIA- 1 1\ I I A - 1 - ( A + B) - 1 1 1 < ::} 1 I!A- 11
1
IIA- 1II IIBII - /IA - 1/I I I B/ 1 .
) matrices: 1 1 · 1 1 is unitarily invariant { 12} 1 1 · 1 1 a m atrix norm for ( ) matr ices A . ::} ! I AI \ .pec < ! ! A ll for all ( ) : There exists a matrix norm 1 1 · 1 1 such that I I A I I < 1 ::} ( 13 ) A ( l im;_00 A i 0, i.e., A is convergent. ) p(x ) = :Lr' 0 p; xi a power series: There exists a matrix ( 14 ) A ( norm 1 1 · 1 1 such that :L 7 0 IPdabs i ! A W converges for ::} p( A) = :L:"' 0 p; Ai exists. ) f E IR, f > 0, p(A) = max{ ! .A \abs : ..\ is eigenvalue of A } : ( 15 ) A ( There exists a matrix norm 1 1 · 1 1 such that p( A) < I I A I I < p(A ) + f . ) ( 16} A ( m x m
m x m
m x m
=
m x m
,
n --+ oo
m x m
m x m
,
:
inf { I I A l l : I I · I I is a matrix norm for ( ) matrices} = p( A) - max{ I .Aiabs : ..\ is eigenvalue of A } . m x m
Note: The foregoing results may be found in Horn & Johnson ( 1 985,
Chapter 5).
H A N DBOOK O F
1 08
8.4.2
M AT RI C ES
Induced Matrix Norms
( 1 ) : 1 · 1 1 a norm for ( m x 1 ) vectors: I I A I I tu b = m ax _
{
I I A x l l : x ( m x 1 ) , x 'f. O
l l .r l l
}
is a matrix norm for ( m x m) ma t ri ces . (2)
11 · 11 a norm for ( m x 1 ) vectors with induced matrix norm
A (m x m) :
max { I I A .rl l : .r ( m x 1 ) , J l x l l = 1 } max { I I A. x l l : .r ( m x 1 ) , ! l x l l < 1 } ma x
{
I I Axll
l l · l l tu b .
}
I x (m x 1 ) . I '1 .r I :I a l l l .r l fo r any no rm I I · I I for ( m x 1 ) vectors . =
:
a
( 3 ) ! l · l l lub an indu ced matrix norm for ( m x m) ( 4 ) il · IIL�i . l l · lll�� ind ll(' ed ma trix nor ms :
{
m
at r ices :
I I) :i A I I (b) : A (m x m ) , max J I A I I ,u b (b) .4 = max I I J I Iu b : .4 ( m x m ) . A 'f. 0 I I A I I\: b)
{
l l lm l l t ub =
}
1
.
( 5 ) 11 · 11)�� . 11 · 11).� � mat rix norms induced by norms l l · l l a . l l · l l b . respectively. I I A I I ��� for every ( m x m ) matrix J A I I i :i for ( m x 1 ) vect ors: .-l. � t here exists a c E IR . c > 0, such that l l x l l a = r l l x l l b for a l l ( m x 1 ) vectors x . ( 6 ) I1 · !1L�i . l 1 · ll\,� � ind u ce d matrix norms : JIAiiL�i < liAi l )�� for all ( m x m ) matrices .4 � I I A I I i :i 1 I A I I \ �� for all ( m x m ) m a t ri c es .4 . ( 7 ) I J · I I a matrix norm for ( m x m) matrices: There exists an induced mat rix norm 1: · l i 1 u b such that : J A i l > l i A : II ub for every ( m x m ) matrix =
=
.4 .
( 8 ) 1 1 · i l a mat rix norm for ( m x m) matrices . I I · l l tub an induced mat rix norm for ( m x m ) matrices: 1 1 .4 1 1 < J j .4 i J iub for all ( m x m ) matrices A � 1 1 .4 1 1 I I A I I I ub for all ( m x m ) matric es A. =
( 9 ) I I · I I a matrix norm for ( m x m) mat r ices : 1 1 · 11 is a minimal matrix norm � i l · l l is an induced matrix norm .
109
V ECTO R A N D M ATRIX N O R M S
( 10} II · II a norm for ( m x 1 ) vectors with induced matrix norm l l · l l 1ub : 1 1 · 11 is an absolute norm -<==> I I D I I Iub = max, = t , . . . , m l dd ab for every diagonal matrix D = diag(dt , . . . , dm ) · ( 1 1 } I I · I I a norm for ( m x 1) vectors with induced matrix norm II · l i M . s ( m m) nonsin gular, llxlls- • � n s- t x l \ , II A II \ �l = IlS- I A SIIM : 1 1 · 11\ �l is the matrix norm induced by l l · lls - t . s
X
Note: The foregoing results on induced matrix norms may be found in
Chapter 5 of Horn 8.5
8.5.1
&
Johnson ( 1 985).
P roperties of Special Norms
General Results
It -norm
(1) (2} (3) (4)
11 · 1 \ t is a matrix norm. I I · l i t is not derived from an inner product. x (m x 1 ) : ! l x l l t = max{ l x H Y\abs : Y (m x 1 ) , \I Y\ I oo = 1 } . A ( m x m) : I I A H \ \ t = II A I\ t ·
Euclidean norm
(5} l l · ll 2 is a matrix norm for ( m x m) matrices. ( 6 ) l l · l l 2 is derived from the inner product tr( A H B) for complex ( m x n ) matrices A , B or tr(A' B) for real (m x n ) matrices A , B. (7) A (m x n) : I I A H II 2 = I I A II 2 · ( 8 ) U ( m x m) unitary, A ( m x m) : I I U A ll = 11A I I 2 · 2 (9} U ( m x m) unitary, V ( n x n) unitary, A (m x n) II U A V\ \ 2 = II A \ \ 2 . ( 10 ) A ( m x m) with eigenvalues A t , . . . , Am : :
m A normal ::} I I A I I � = L \ A ; I;bs· •=
I
Maximum norm
( 1 1 ) II · l l oo is not a matrix norm. ( 12} 1 \AII = m i i A I I x; is a matrix norm for ( m x m) matrices.
H A N DBOOK O F M ATRICES
1 10
( 13} A ( m
x
m) :
limoo IIAIIr · I I A I I oo = p-
S pectral norm
( 1 4 ) l l · l l . pe c is a matrix norm for ( m x m) matrices induced by the vector norm ll · l l 2 · { 15 ) U ( m x m ) unitary, V ( n x n ) unitary, A ( m x n ) : II U A Vl!,pec = I I A II•pec· ( 16) I I · l l•re c is the only induced unitarily invariant matrix norm. ( 1 7 ) 11 · 11 a matrix norm for ( m x m ) matrices: 1 1 · 1 1 is unitarily invariant => I I A II•p ec < I I A I I for every ( m x m ) matr ix A . ( 1 8 ) A ( m X m ) : I I A H II•rec = I I A I I•pec · ( 1 9 } II · ll •p e c is the only induced , self-adjoint matrix norm . {20) A ( m x m ) : \ I AI \ p 6
ec
max { \ \Ax\1 2 : x ( m 1 ) , \\x\! 2 1 } max { I I Axlb : x ( m 1 ) , \ \ x l b < 1 } x m ax IIA l l 2 : x ( m 1 ) , x :f. 0 l l xl l 2 max { IYH Ax l abs : x , y ( m 1 ) , l l x l\2 = II YII 2 = 1 } ma x { IYH Ax\abs : x , y ( m 1 ) , l\ x l\2 = \\Y\\2 < 1 } · I IAA H I I • p ec = 1\A H A l \ • p ec = I! A \ I;pe c · x
{
=
x
}
x
x
x
( 21 ) A ( m
x
m) :
Column-sum norm
(23) \ l · l l c o l is a matrix norm for ( m m ) matrices. (24) l l · l lcol is induced by 1 1 · \\ t , that is, I I Ax l l t x ( m 1 ) , x :f. 0 . = max c ol 1\AII l lx \ l t x
{
:
x
}
Row-sum norm
( 25 ) I I · l l row is a matrix norm for ( m m ) matrices. ( 26 ) \ 1 · \ \ r ow is induced by 1\ · l l oo , that is, \ I A x lloo m ax ow X ( m 1 ) , x :f. 0 . I I A \ Ir : l \ x l loo x
=
{
X
}
Ill
V ECTO R A N D M ATRIX N O R M S
Note: The foregoing properties of special norms may be found in Horn & Johnson ( 1985, Chapter 5) . 8.5.2
Inequalities
( 1 ) A (m x n) :
(a) I I A l i i < jnin II A II2· (b) 1 \A\\t < m n \ \ A\\oo . ( c ) !l A IIt < n I I A I Icol · (d) 1 \ A \ \ 1 < m \ \ A I \ row A ( m x m ) : !l AI I t < m3 1 2 1\ Ail.pec · ( Holder's inequality) A , B ( m m) , 1 < p, q < oo, ! + � = 1 : A (m x n ) : (a) \ I A \ 1 2 < \ l A I I t . (b) \I A\1 2 < jnin i\AIIoo · (c) 1 \ A\ b < vn \ I A\Icol · (d ) \ I A \ 1 2 < vm ii AIIrow . ( e) 1 \ A I \ 2 < jmin(m, ) 1 \ A I \. pec · (Cauchy�Schwarz inequality) A , B ( m n) : (a) \ I A H B \ 1 2 < IIA I\ 2 \ I B \ b(b) \ tr( A H B ) \abs < II A II2 \ I B \ 1 2 · A, B (m m) : ( a) 1 \ A B \ b < \ I A I \ . p ec \ I B \ 1 2 · ( b ) \ I A B I \ 2 < \ I A\ 1 2 \I B \ I,pec· ·
( 2)
(3)
x
(4 )
IIA B \ 1 1 < 1 \ A I\p I I B \Iq .
n
( 5)
x
( 6)
x
( 7 ) A ( m x m)
(8} A (m x n) :
(a) (b) (c) (d) (e)
\ I A \ Ioo 1 \ A IIoo 1 \ A \ \ oo 1 \ A \Ioo 1 \ A \ \ oo
with eigenvalues < \ lA I I t · < IIAII2 · < 1 \ A I\col · < IIAII row
·
< 1 \ A I \. p ec·
)q ,
. . . , Am :
\ IA\1� >
m
L \ >.i \ �bs· i= 1
H A N DBOOK O F M ATRICES
112
(9) A ( n) : (a) I I A IIcol < I I A i h · ( b ) II AI leo/ < JTi I I A I I 2 · ( c ) I I AIIcol < m iiAIIoo · { d) II AIIcol < m IIAIIrow · ( 10 ) A ( m ) I I A I Icol < Jffi i i AII•pec· ( 1 1 ) A (m n ) : (a ) II A I Irow < I I A I I l · ( b) I I A IIrow < Jffi !I A I I 2 · ( c } I I A IIrow < T! I I A I I = · ( d ) I I A IIrow < n I I A IIcol · ( 1 2) A ( m X m ) : I I A IIrow < Jm iiAII.pec· ( 13) A ( m n ) : (a) II A II.pec < IIAII l · ( b ) I I A I I ,pec < I I A I I 2 · ( c ) II A II,pec < Jmi1 11AII = · ( d ) II AII,pec < JTi ii A IIcol · (e) II A II.pec < rm I I A IIrow . ( 14 ) A ( m) : I I A II;p ec < I I A IIcoi i i A I I r ow • m x
X m
:
x
x
m X
Note: The foregoing inequalities can be derived from results presented in Horn & J ohnson ( 1985, Chapter 5), where also further references are givf'n .
9
Properties of Special Matrices 9.1
Circulant Mat rices
Definition: A n ( m x m) matrix
am - l
a2 a1 am
a3 a2 a!
a rn - l Um - 2 U rn - 3
U rn am - l a fn - 2
a3 a2
a4 a3
a5 a4
a1
a2 at
bm c 1 c2
E
a1 am eire( a !
1
•
•
•
,
am) =
am
is a circulant matrix .
Properties
(1)
a1
1
•
•
•
,
am
1
b1
1
•
•
•
,
1
1
r 1 e i r e ( a I • . . . , am )
= circ ( c l a l ( 2 ) A = circ(a J , . . . a m )
1
•
•
•
1
b, )
c2b1 . . . . , r1 am ± c2bm ) .
:
,
( a)
±
± c2 c i r r ( b1
( b ) A = c i r r ( a 1 , . . , ilm ) . -
.
( 3) A = circ( a J , . . , am ) , a( .r ) = a 1 + a 2 .r + 0 = exp( 27ri/ 1 1 1 ) : .
· · · +
a
m
'n l.
J r
( a ) tr( A ) = ma 1 . ( b ) det( A )
(c) Aj
=
111 -
1
= 0 a ( Oi ) . j =D
a(Bi ) .
j = 0,
.
.
.,
m -
I , are the eigenvalues of .4 .
H A N DBOOK O F M ATRIC ES
1 14
(4) A ( m
x
m ) circulant:
(a) A H is circulant. ( b ) A ; is circulant for i = 0, 1 , 2, . . . (c) A - 1 is circulant if A is nonsingular. (d) A+ is circulant. (e) rk(Ai ) = rk( A ) for i = 1 , 2, . . . ( f ) A is normal, that is, A H A = A A H . (g) A is simple. ( h ) A. is a Toeplitz matrix . ( i ) A. is unitarily similar to a diagonal matrix. ( 5 ) A., B (
m
x
m
) c irculant:
(a) A B is circulant . (b) AB
(c) (6)
A. �)
=
B
BA.
is circulant.
A,
B ( m x m) circulant with eigenvalues A 1 ( A ) , . . . , Am ( A ) . . . , Am ( B ) , respectively:
and A t ( B ) ,
(a) A t ( A ) + A t ( B) , . . . , Am ( A ) + Am ( B ) are the eigenvalues of A + B . ( b ) A 1 ( A ) - A 1 ( B ) , . . . , Am ( A ) - Am ( B ) are the eigenvalues of A - B . ( c ) A t ( A) A t ( B ) , . . . , Am ( A ) Am ( B ) are the eigenvalues of A B . (7) A (m x
m ) circulant with eigenvalues A 1 , . . . , Am :
(a) A is H ermitian {::::::} A; E IR, i = l , . . . ( b ) . is Hermitian positive definite {::::::} A; E IR, A; > 0, i I , . . . , m. ( c ) A is unitary {::::::} [ A; \abs = I , i = I , . . . ( d ) A = F H diag(A 1 , . . . , Am ) F , where F is a Fourier matrix (see the A ppendix for a definition) . (e) nl-imoo A n exists {::::::} A1 = I or [A; \ abs < 1 , i = I , . . . , m. , m.
A
, m.
l (f) nlim ( lm + A + · · · + A n - l ) exists {::::::} I A; I abs < I , i n -oo I , . . . , m.
115
PROPERTIES O F S PECIAL M ATRICES
0 0
( 8 ) C = c i rc ( 0 , I , 0 , . . . , 0 )
(b) (c)
(d) (e) (f) (9)
0 0
0 1
(m
= 0 I
( a)
1 0 0 0
0 0
1 0
x m
)
:
em = lm · C' = c m - l _ c - 1 = cm 1 c- l = C' . A = circ( a 1 , . . . , am ) -¢::::::> A = a 1 lm + a 2C + · · · + am em ) is circulant -¢::::::> AC = CA. A( -
_
I _
m x m
lm = eire( ! , 0 , . . . , 0 ) .
Note: For proofs and more on circulant matrices see Davis ( 1 979).
9.2
Commutation Matrices
Definition: The ( m n x m n ) commutation matrix Kmn or Km . n is defined such that, for any ( m x n ) matrix A , Kmn vec( A ) = vec ( A ' ) . For instance,
1{32 =
so that, for a ( 3
/{32 vee
x
2) m atrix
a1 1 a2 1 a3 1
a12 an a32
I 0 0 0 0 0
0 0 1 0 0 0
0 0 0 0 I 0
0 I 0 0 0 0
0 0 0 1 0 0
0 0 0 0 0 I
A = [ aij ] ,
= 1{32
al l a2 1 a 31 a1 2 an a 32
a1 1 a12 a2 1 an a 31 a32
= vec ( A ') .
H A N D BO O K O F M ATRIC ES
1 16
9.2.1
General P roperties
( 1 ) Hkt = [h;j ] ( m x
n
) with h k t
= 1 and h;j = 0 if i i: k or j =f. l : m
L L H kl c:> H�� ·
1\·,, =
(2) (3) (4) (5)
n
k = 1 1= 1
Kmn is a real m atrix. Km n is a permutation matrix. 1\m l = Em . f\ 1 n = !,. . ==
( 6 ) 1\:r, n l\n m · (7) I\·,-;.. ;. = l\nm ·
( 8 ) f\rn n .p
= f\m,n p l\n ,p m
= l\·n ,p · m l\m , n
p
= ( Em 0 !\,. ,p ){ f\m,p ® In ) = ( fn 0 f\m,p ) { f\n ,p
(9) (10) (11) (12)
0
fm ) .
1\m rl ,p f\,. p ,rn f\p m , n = Em n p · r k { l\mn ) =
mn.
t r { I\,.11 ) = 1 + greatest comm on divisor of m t r( Km
m
)=
m.
-
1 and
n
- 1.
( 1 3) de t( l\m n ) ( - l ) m n ( m - 1 ) ( n - 1 )/4 . ( 1 4 ) de t ( Km m ) = ( - l ) m ( m - l )/ 2_ ( 15 ) The eigenvalues of 1\m m are 1 and - 1 with multiplicit ies 1 m ( m + I ) and � m( m - 1 ) , respectively. =
( 16 ) ( !,.2 + Kmm )1 = fm 2 + f\mm · ( 1 7) � ( 1m 2 + 1\m m ) is idem potent. 111 2
( 1 8 ) rk { l
(19) ( 20) (21) ( 22 ) (23) ( 24 )
+ [\.mrn ) =
�m(m + 1 ) .
t r ( f,., + J\,,1 ) = m( m + I ) .
1\rnm { lrn2 + Kmm ) = f,., + K mm · ( lm 2 + f\mm ) l\mm = Iml + K mm · { lm 2 - f\rnm )1 = fm2 - Kmm · � ( 1m2 - A' mm ) is idempotent . rk{ lm > - 1\mrn ) = � m(m - 1 ) .
117
P RO P E RTIES O F SPECIAL M ATRICES
( 25) t r ( Im2 - A"mm ) = m ( m - 1 ) . 9.2.2
(1)
Kronecker P roducts A (m x n) :
(a) 1\mn ( A '
(b)
0
[ /\mn ( A '
0
A ) is symmetri c .
A J ] 2 = AA' 0 A' A .
( c ) rk ( A ) == r ::::} r k [ l\mn (A' 0 A )] = r 2 .
(2) A ( m x m) :
(a ) ( 1m2 + A'mm ) ( A 0 A ) ( /m> + Kmm ) = 2( fm 2 + 1\mm ) ( A 0 A ) = 2 ( A 0 A ) ( !m2
(b)
+
1\mm ) .
( /m2 - 1\mm ) ( A 0 A ) ( fm2 - 1\mm ) = 2( fm 2 - 1\mm } ( A 0 A ) = 2( A 0 A ) ( fm2 - 1\mm ) .
( 3 ) A ( m x n ) , a (p x l ) , b ( 1 x p) :
1\p m ( A ® a ) = a @ A , 1\mp (a ® A ) = A ® a , ( A 0 b ) Knp = b 0 A . ( b 0 A ) 1\pn = A r:Y b.
(4) A ( m x m ) . a (m
x
1) :
(a )
( 1m 2 + Km m ) ( A 0 a ) = ( Im2 + 1\mm ) ( a C A ) = A 0 a + a 0 A.
(b)
( 1m2 - A'm m ) ( A. 0 a ) = - ( 1m2 - 1\"mm ) ( a 0 A ) = A 0 a - a O A.
( 5 ) A ( m x n ) , B (p x q ) : (a ) 1\pm ( A 0 B ) = ( B 0 A ) 1\qn · ( b ) 1\pm ( A 0 8 ) 1\nq B 0 A . =
( c ) vec ( A 0 B ) = Un 0 1\qm 0 lp ) [vee( A ) 0 vee{ B )} .
(6) A, B (m x n) :
tr [1\mn ( A' 0 B)} = tr(A' B ) .
(7) A, B (m x m) :
( a) ( Jm 2 + A"mm ) ( A ® B ) ( fm 2 + Kmm ) =
( Jm 2 + A"mm ) ( B 0 A ) ( !m2 + A"mm ) ·
{ b ) ( 1m2 - 1\·mm ) ( A 0 B ) { lm 2 - h' m m ) ::::: ( 1m2 - l\.mm ) ( B 8 A ) ( Im2 - 1\ , , ) .
( c ) ( 1m 2 + l\.mm ) ( A 0 B + B 0 .4 )( 1"1 2 + 1\.m m ) ::::: 2( !,2 + 1\mm ) ( A 0 B ) ( !m2 + 1\.mrn ) = 2 ( /m 2 + 1\mm ) ( A 0 B + B 0 A )
H A N D BOOK O F M ATRICES
1 18
= 2(A 0 B + B 0 A)(lm> + Kmm ) . ( d ) Um• - Kmm ) ( A 0 B + B 0 A)( lm• - Kmm ) = 2( /m> - Kmm ) ( A 0 B ) ( lm> - Kmm ) = 2(/m> - Kmm )(A 0 B + B 0 A) 2(A 0 B + B 0 A ) ( lm • - l\mm ) . ( 8 ) A ( m x n ) , B (p x q ) , C ( q x s ) , D ( n x r ) : =
1\.mp ( BC 0 AD) = ( A 0 B ) Kn q (C 0 D). (9) A (m
x
n) ,
B (p
x
q), C (r
x
s) : Km p,r (C 0 A 0 B ) K, , q n Km ,p r ( B 0 C 0 A ) K, q,n ·
( 10 ) A ( m
n ) real , rk( A )
= r : A ) , . . . , Ar > 0 are eigenvalues of A' A ::::} -\ 1 , . . , Ar and ± �, i < j, are the nonzero eigenvalues of Km n ( A' 0 A). x
.
9.2.3
Relations With Duplication and Elimination Matrices
( 1 ) Kmm = 2Dm Dt - lm> . ( 2 } l\.mm Dm = Dm . ( 3 ) D'!;, Km m = Dt .
( 4 ) lm> + Kmm = Dm Lm Um> + Kmm ) . ( 5 ) D'!;, = � Lm ( /m> + Kmm ) . ( 6 ) Lm Km m L'm = 2/ ! m( m + I ) - D'm Dm . ( 7) Dm = ( /m> + Kmm ) L'm ( Lm ( lm > + Kmm ) L'm ) - 1 . (8) L m A. mm L 'm is idempotent diagonal. ( 9 ) rk( Lm l\.mm L'm ) = m . ( 10 ) tr( Lm Kmm L'm ) = m. ( 1 1 } L'm Lm K mm L'm Lm is idempotent diagonal. ( 1 2 ) rk(L'm L m Kmm L'm Lm ) = m. ( 1 3 ) tr( L'm Lm Kmm L'm Lm ) = m. ( 1 4 ) L'm Lm Kmm L'm = Kmm L'm Lm Kmm L'm . ( 15 ) Kmm = Dm D'm + L'm Lm Kmm L'm Lm - lm> . ( 1 6 ) A ( m X m ) : L'm Lm Kmm L'mLm vec (A) = vec(dg ( A ) ) .
PROPERTIES O F S P ECIAL M ATRICES
1 19
Note: A substantial collection of results for commutation matrices including the ones given here may b e fou nd in M agnus ( 1 98 8 ) .
9.3
Convergent Matrices
Definition : A (m x m) is said to be convergent or stable if A n ___. 0 as n ( 1 ) A (m x m), B ( n x n ) , ( a) A is convergent,
c E (C : l c l abs <
I
=> cA is convergent .
( b ) A , B are convergent => A 0 B is convergent .
(2)
( c ) A , B are convergent => A EB B is convergent.
A
(m x m) with eigenvalues
.\ 1 , . . . , Am : <==>
A is convergent
(3)
A ::: diag(a , . . 1 A is convergent .
,
am )
<==>
:
max I .\ ; l abs < 1 . m
i= l �
I a ; I abs < 1 , i = 1 ,
,
. . . , m.
( 4 ) A ::: [a;j ] ( m x m) triangular: A is convergent <==> l a;; l abs < I , i = 1 , . . . , m.
(5)
A
(m x m) : (a) A is convergent
<==>
n
L Ai i= I
converges for n ___.
oo .
( b ) I I A I I < I for some matrix norm 1 1 · 1 1 => A is convergent. ( c ) A is convergent => Im ± A is nonsingul ar.
( 6 ) A (m x m) convergent: 00
(a) Urn + A ) - 1 =
L(-A( 00
( b ) ( /m - A ) - 1 =
L A; . i=O
( c ) Um � + A ® A ) - l =
00
L::: ( - A ) i 0 A' . 00
( d ) ( /m� - A 0 A ) - 1 =
L Ai 0 Ai . i=O
___. DC .
H A N DBOOK OF M ATRICES
1 20
( 7 ) (Stein 's theorem )
A ( m x m) : A is convergent {::::=} given a positive defin ite ( m x m ) matrix V , there exists a positive definite matrix Q such that Q - A H Q A = V.
Note: These results fol low straightforwardly from definitions or m ay be found in Lancaster & Tismenetsky ( 1 985 ) or H orn & J ohnson ( 1 985) . 9.4
Diagonal Matrices
Definit ion: A = ( a ij ] ( m x m ) is a d iagonal matrix or , briefly, A is d iagonal . if aij = 0 for i f- j, that is, .•1 -
Properties
( 1 ) A B ( m x m) d iagonal, c E ( a ) cA is d iagonal.
(f, :
( b ) A ± B is d iagonal . ( c ) A B is d iagonal.
(d) AB = BA.
( e ) A '�' B is diagon al .
( 2 ) A ( m x m ) diagonal . B (a) A
(b)
C::·
A .fl
( n x n ) d iagonal :
B is d iagonal. B is d iagonal.
( 3 ) A ( m x m ) d iagonal:
( a) r k ( A ) = number of nonzero d iagonal e lements.
( b ) A' = A .
(c ) AY = A . if A is real.
(4) A
= d iag( a l l • . . . , a ,nm ) :
( a) t r ( A ) = a 1 1 + m
( b ) det ( A. ) =
· · ·
IT a ii ·
i= l
+ amm ·
121
PROP ERTIES O F S PECI A L M ATRICES
( c ) a;;
r -t
0,
. = 1 , . . . , m => A - 1 = d t. ag( a - 1 , amm 1 1 . . . , - 1 ). -, 1 if a;; ::j; 0 a . where a:!: = 0 if a;; 0
2
II
.
d.
a
{
=
for i # j
0
m II Unn for l. = J.
(e) Aa dJ = [a�/ ] , where a I]. . =
n=l n ;t '
( 5 ) A ( m x m) diagonal :
(a) A is idempotent => all diagonal elements of A are either 0 or 1 . ( b ) A is n ilpotent => A = Om x m · (c) A is orthogonal => A = Im . (d ) A is unitary => A = Im . ( 6 ) A = diag( a . . . , amm ) : ( a) A is positive definite => 0 < a;, E IR, i = 1 , . . . , m. ( b ) A is positive semidefinite => 0 < a;; E IR. i = 1 , . . . , m . ( c ) A is negative definite => 0 > a;; E IR , i = 1 . . . , m. (d) A is negative semidefinite => 0 > a;; E IR , i = 1 , . . . , m. a;; is an eigenvalue of A and the it h ( 7) A = d iag( a . . . , amm ) : col umn of Im is a corresponding eigenvector for i = 1 , . . . . m. , bmm ) ( 8 ) A = diag(a l 1 • . . . , amm ) , B = diag(b t t 1 (a) D� ( A 0 B)Dm is a n m( m + l ) x �m(m + l ) ) diagonal matrix with diagonal elements � (a;;bjj + ajj b;; ), 1 < j < i < m . ( b ) Lm ( A 0 B)L;.,.. is a ( � m( m + 1 ) x � m( m + 1 ) ) diagonal matrix with diagonal elements a;; bjj , i < j. (c) L m ( A 0 B)Dm is a ( � m(m + 1 ) x � m( m + 1 ) ) diagonal matrix with diagonal elements a;; bjj , i < j. 1 1 ,
.
11,
•
•
:
•
. . . , amm ), B = diag(b t l , bmm) : m (a) d et[Lm( A 0 B)L:nJ = II b: ; a;:' - i + l . m + ( b ) d et[Lm ( A 0 B)Dm ] = II b:;a7,' - i 1 . 1
( 9 ) A = diag( a ! l ,
•
i= I
i=
.
.
.
H A N DBOOK O F M AT R I C E S
1 22
Many of the results of this section follow easily by considering the individual elements of the matrices involved. The others may be found in \lagnus ( 1988 ) or in Horn & Johnson ( 1985 ) . Note that a diagonal matrix is a special case of a triangular matrix. For transforming matrices to diagonal form see Chapter 6 . Note:
9.5
D uplication Mat rices
Definition: The ( m 2 x �m( m + I ) ) duplication matrix Dm is defin ed su'h m) matrix A . For example. that vec ( A ) = Dm vech( A ) for any symnwtric ( •
m x
D2 =
1
0
0 0
1 1
0 0 0
0
0
I
so t hat
[ vee a 1 1 a
a1 1 a12 a12 an
12
9.5.1
= D2
General Properties
( 1 ) Dm is a real matrix. ( 2 ) rk ( D, ) = � m ( m + 1 ) . ( 3 ) Dt, = ( D:n Dm ) - 1 D:n . (4 ) (5) (6) (7) (8) (9)
-1 + + Dm Dm - ( D' D ) det ( D:n Dm ) 2 m ( m - 1 ) / 2 . I
n1
n1
·
=
d et ( Dm D:n ) = 0 .
tr ( D:n Dm ) = m 2 . tr( D�, Dm ) - 1 = t m ( m + 3 ) . 1 0
0
0
0 0
0
D:n Dm
2 /m
all a12 a 22
= D2 vech
[
123
P RO P E RTIES O F S PECI A L M AT RICES
1 0 0
0 0 (D:n Dm ) - 1
0 � lm 0
( 12 ) D:n Dm is a n m ( m+ 1 ) x � m(m+ l ) ) diagonal matrix with 1 (m times) and
2 n m( m - 1 ) tim
e
s) on the diagonal .
( 1 3} Dm D:n is (m2 x m2 ) w i th eigenvalues 0 ( � m( m - 1 ) times ) , 1 (m times) and
9.5.2 (1} (2} (3) (4}
2
n m( m - 1 ) times) .
Relations With Commutation and Elimination Matrices Dm D"/;. = � ( 1m 2 + Kmm ) . Kmm Dm = Dm . + l\'mm -- Dm+ Dm · Lm Dm = f �m ( m +l ) ·
(5} Dm Lm ( fm2 + Kmm ) = 1m2 + Kmm · ( 6 } D"/;. = � Lm ( /m2 + Kmm ) · ( 7 } D:n Dm = 2 /�m(m+l ) - Lm f{m m L:n . { 8 } Dm = ( 1m2 + Kmm ) L:n [Lm ( lm> + Kmm ) L:n] - 1 . 9.5.3 Expressions With vee and vech Operators ( 1 ) A ( m x m) : D"/;. vec ( A ) = � vech(A + A' ) , D"/;.. vee( A ) = Lm vec (A) = vech( A ) , if A is symmetric D:n Dm vech( A ) = 2vech ( A ) - vech(dg( A)) , D:n vee( A ) = vech (A + A' - dg( A))
,
Dm D:n vec(A) = vec (A + A' - dg( A )] .
(2 ) A ( m x m ) , a, b ( m x 1 ) ,
c
E
] Dm +l vee [ C 1
a
�
b' ] [ D� + vec 1 a A = c
( 3 ) A ( m x m ) sym m et r ic nonsingular , c E
c
a+b D:n vee( A ) c
�(a + b) D"/;.. vee ( A )
(a) det(D"/;. [A 0 A + c vec(A)vec ( A )') Dm )
=
( 1 + em)( det A) m +l .
,
H A N D BOOK O F M ATRICES
1 24
( b ) ( D;!", [A 0 A + c vec(A )vec ( A )'] Dm ) - 1
[
= D! A- 1
0
A- 1
]
cc vec(A - 1 )vec( A - 1 ) ' Dm . l + m
-
Duplication Matrices and Kronecker Products
9.5.4
(1} A
( m x m) :
det[D;, ( A 0 A ) Dm] = 2m ( m - l )/ 2 ( det A) m + l . [D� ( A 0 A ) Dm] - 1 = D;!", ( A - 1 0 A- 1 )D;!",' , if det ( A ) f- 0. det[D;!", ( A 0 A )Dm] (det A) m + l . Dm D;t", ( A 0 A ) Dm = (A 0 A ) D m . D;t", ( A 0 A ) Dm D;!", = D;t", ( A 0 A ). [ D;!", ( A 0 A ) Dm ] ' = D! ( A ' 0 A ' )Dm , s = 0, 1 , 2, . . . [D;t", ( A 0 A ) Dm] - • = D! ( A - • 0 A ' )Dm , s = 1 , 2, . , if dt>t ( A ) f- 0 . ( h ) [D;!", ( A C A ) Dm j l / 2 D;!", ( A 1 12 0 A 1 1 2 ) Dm , if A 1 1 2 exists. A ( m x m ) with eigenvalues .\ 1 , . . , A m : .\ ; .\i , 1 < j < i < m , art> the eigenvalues of D!( A 0 A ) Dm . A , B ( m x m) : D! ( A 0 B)Dm = D;t", ( B 0 A )Dm . D;t; ( A 0 B )Dm = � D;t", ( A 0 B + B 0 A)D m tr [D;!", ( A 0 B)Dm ] = � [tr(A )tr( B) + tr(A B)], det ( B) = 0 => det[D;!", ( A 0 B)Dm ] = 0. diag (b J J , . . . , bmm) : D;t; ( A 0 B ) Dm A = diag (a! J , . . . , am m ) , B is a diagonal matrix with diagonal elements � ( a ; ; bii + aii b;; ) , 1 < j < i < m. A = [a;j] . B = [b; i] ( m x m ) upper (lower) triangular: D;!", ( A ·2 B)Dm is upper ( lower ) triangular with diagonal elements � ( a;, b11 + aj 1 b;; ) , 1 < j < i < m . A ( m x m). H ( m x m ) nonsingular, .\ 1 , . . . , .\ m eigenvalues of A H - 1 : m det[D! ( A •0 B) Dm] = T m( m- l )/2det(A)(det B) II(.\ ; + Aj ) . (a) (b) (c) (d) (e) ( f) (g)
=
-
.
=
(2}
(3)
.
,
(4) (5}
(6)
=
i >j
(7)
A,
H ( m x m)
with
A
0
B + B 0 A nonsingular:
[D! ( A 2· H )Dm ] - 1 = 2 D;t", ( A 0 B + B 0 A ) - 1 Dm .
(8} A (m x m)
with eigenvalues .\ 1
•
.
.
.
, Am
:
.
125
P RO P E RTIES O F S PECIAL M AT RICES
(a) Dm Dt, ( Im 0 A + A 0 lm )Dm = ( lm 0 A + A 0 lm )Dm . ( b ) tr[D;t ( Im 0 A + A 0 lm )Dm] ( + l )tr(A). ( ) [ D;t ( lm 0 A + A @ lm ) Dm]-1 = Dt, (Im 0 A + A 0 lm ) - 1 Dm , if Im 0 A + A 0 Im is nonsingular. (d) det[D;t ( Im 0 A + A 0 lm )Dm] = 2 m det(A) II (A; + Aj ). i>i ( e) A , + Aj , 1 < i < j < are the eigenvalues of D;t ( Im 0 A + A 0 lm )Dm . ( 9 ) A (m x m ) with eigenvalues AJ , . . . , A m , i E IN , i > 1 : =
m
c
m,
i- 1 det D;;. L(Ai- 1 -j 0 Ai )Dm
=
j =O
im (det Ar - 1 IT il k l k>l
where
Jlkl = ( 10 )
A = [ a ;1 ] ,
B
==
{
[b;j] ( m x
m
) lower triangular :
det[Dt, (A @ A ± B 0 B)Dm] = IT ( a aj j ± b;; bj j ) · i> j
(11)
;;
A (m x m) nonsingular, B ( m x m), A 1 , . . . . A m eigenvaluPs of BA - 1 : det[Dt, ( A 0 A ± B 0 B)Dm]
( 12 ) A, B ( m x
m
,
(a) D m + 1
)
=
(det A) m + l II( l ± A;Ai ). •>i
symmetric, a , b ( m x 1 ) , o , /3 E
] [
([
])
ob' + f]a' o/3 ( a' 0 b1 )Dm ob + f]a o B + f]A + ab' + ba' ( a' 0 B + b' 0 A ) Dm D:n ( a 0 b) D:n (a 0 B + b 0 A ) D:n (A @ B)Dm (b)
D! + 1
( [ o a'A ] 0 [ b b'B ] ) D!+ ' = a
(3
'
� (ob' + ;3a' ) o/3 ( a ' 0 b' ) D;t � (ob + Ba) �(o B + f]A + ab' + ba' ) �- ( a1 0 B + b' 0 A)D;t,' D;t", ( a 0 b) � D;t ( a 0 B + b @ A ) Dm+ ( A c;. B)Dm+ ' '
H A N DBOOK O F M ATRICES
1 26
9 .5.5
Duplication Matrices, Elimination Matrices and Kronecker Products
(1) A (m
x m
)
(a) L m ( A
:
0
A ) Dm = D;!; ( A 0 A ) Dm
0
( b ) L m Um 0 A + A 0 lm ) Dm = D;!; ( lm 0 A + A 0 lm ) Dm o
( 2 ) A = diag(a l l , o . . , am m ) , B = diag( b u , . . o , b m m ) :
L m ( A 0 B ) Dm
is a diagonal matrix with diagonal elements a;;bjj , 1 < i < j < m o
( 3 ) A = [a;1 ] , B
=
[b;j ] ( m
x m
) upper ( lower) t riangular:
(a) L,. ( A 0 B ) Dm is upper ( lower) triangular with diagonal elements a ; ; bi i , 1 < i < j < m o m ( b ) det [Lm ( A 0 B ) D m ] = II bt a;'; - i + l o i=l ( 4 ) A = [ a ;j ] , B = [ b;j ] , C = [ c;J] ( m x m ) lower triangular: ( a ) L m ( A 0 B' )Dm Lm ( A 0 B' ) L'r,. o =
( b ) det [Lm ( A B'
0
B' A ) L ; ] n
det[Lm ( A B 0 B' A ) Dm ] = det [Lm ( A B' 0 A' B' )Dm ] m 1 = ( det A ) + (d et B ) m+ l o m
( d ) det [ Lm ( A B ' 0 C ' ) Dm ] =
(5) A . B ( m
) det ( B ) f- 0 :
x m ,
=
i=I m
II C� ; ( a;;b;; ) m -i + l
0
i=I
det[Lm ( A 0 B ) Dm ] = det (A)(det B)
m
m- 1
II det(C(n))
n=l
and mm m det [ L m ( A 0 B ) D:, ' J = T ( - l ) / 2 det ( A)(det B) w here
C11
Cln
Cn 1
Cnn
are the principal su bmatrices of C = [cij]
=
AB- 1
0
m- 1
II det(C(n))•
n=l
127
PROPERTIES O F SPEC I A L M ATRICES
Note: M agnus ( 1 988) provides a very extensive collection of resul t s on duplication matrices. The results of this section are taken from that source.
Elimination Matrices
9.6
Definition: The ( � m ( m + 1 ) x m 2 ) elimination matrix Lm is defi ned such that vech( A ) = Lm vec ( A ) for any ( m x m ) matrix A. For example, L2 =
1 0 0 0 0 1 0 0 0 0 0 1
so that vech
9.6.1
[
a1 1
a2 1
a,2
an
a1 1
]
a21
an
a1 1
= L? •
a2 1
a12 an
=
L 2 vee
[
al l
a21
a12
an
].
General Properties
( 1 ) Lm is a real matrix. { 2 ) rk( Lm )
( 3 ) L �,
=
=
L�,
tm(m + 1 ) .
( 4 ) L m L 'm = /! m ( m + l ) · ( 5 ) L 'm Lm is idempotent diagonal with diagonal elements 1 and 0. ( 6 ) r k ( L 'r,. L m ) = � m( m + 1 ).
!m( m + 1 ). { 8 ) det ( Lm L'r,. ) = 1 . ( 9 ) det( L;, Lm ) 0. ( 7 ) tr(L'm Lm )
=
=
9.6.2
Relations With Commutation and Duplication Matrices
( 1 ) Lm Kmm L'm is idempotent diagonal. ( 2 ) rk( Lm h."mm L;11 )
=
m. m.
( 3 ) tr( L m l\m, L ;11 ) = ( 4 ) L;, L m Kmm L;n L m i s idempotent diagonal . ( 5 ) rk( L;, Lm l\mm L;n L m ) m . =
H A N D BOOK O F M ATRICES
128
( 6 ) t.r( L'm Lm l\mm L'm Lm ) == m. ( 7 ) L'm Lm Kmm L'm Kmrn L'm Lm Kmm L'm . ( 8 ) L m Dm = f t m ( m + ) · ==
l
. � L m ( /rn > + /\ mm ) . ( 1 0 ) Dm Lm ( /m2 + Kmm ) = /m2 + 1\mm · ( 1 1 ) Lm Kmm L'm + D;11 Dm = 2/tm(m + l ) · ( 1 2) Dm Lm ( /m2 + K mm ) L:11 = ( 1m2 + 1\rnm ) L'm . (9) D;t",
9.6.3 (1)
=
Expressions With vee and vech Operators A
(m
x m) :
Dt, vec ( A ) = vech ( A ) , if A is sy m m etr i c . ( b ) vec ( A ) = L;11vech( A ) , if A is lower triangular. ( c ) L m l\.mm L'mvech( A ) vech[dg( A )] . (d) L;n Lm l\mm L'm L,.vec( A ) vec [dg( A)] . ,
( a) L vec ( A )
=
=
=
(2) A ( m x m ) nonsingular, lower triangular, E {[' : c
(a) dct ( L .,. [A' 8 A + c vec(A)vec( A' ) '] L'm ) = ( I + cm)( det. 4 ) + 1 . ( b ) { L m [A' C:) A + vec( A )vec( A' )') L'm } - 1 , .C vec ( A - 1 )vec ( A · L m . - Lm A - 1 0 A - 1 I + em _
9.6. 4
[
r
-
1 - l \1
]
.
"'
I
Elimination Matrices and Kronecker Products
A = diag( a l l · . . . , Omm ) . B = diag(bl l , . . . , bmm ) : L.,. ( A 0 B) L:n d i agon a l with d i agonal elements a;;b1 j . i < j. ( 2 ) A = [a;j ] , lJ [b;i J ( m x m ) upper (lower) triangular: (1)
IS
=
( a) L m ( A ·8 B ) L'm is upper ( lower) triangular w i t h elements a;;bi1 , 1 < i < j < m. m ( b ) det[Lm ( A C! B ) L ;11 ] = II bj; a;:' - i + l . 1
(3)
A =
[a;j ] .
=I
B = [b;j ] ( m x m ) lower triangular :
diagonal
P RO P E RTIES O F S P EC I A L �f ATRICES
(a ) d et [L ( 4' rn
·
l
"' 0
B )L 'm
1 29
m _ J II b i a m - i+ l -
ii u
i= I
·
m . are the eigenvalues of L m ( A' 2 B) L�,. . ( c ) [L m ( .4 ' � B)L ;,, ]' ::::: Lm[(A') ' :S· B']L �, for s 0. 1 . 2 . . . . s . . . - 2 , - l , if .4 . B nonsingular � . if lower t riangular .4 1 1 2 • B 1 12 exist s ( d ) det [Lrr. ( A B ' 7:· B' A )L;n J det[Lm ( A B' C A' B)L ;,] = (d et A ) m + l (det B ) m + l . ( e) L;, L m ( A ' 2 B ) L;, = ( A' 8 B ) L ;, . ( 4 ) A . B (m x m ) : A . B are strictly lower triangular =? L m ( A ' Z B) L;,. is nilpotent. ( 5 ) A ::::: [ a;j ]. B ::::: [b ;j ] . C = [c;j] . D = [d;j ] ( m x m ) lower triangular : m det[Lm ( A B' :& C' D)L ;,] = IT( r; , d;; )' ( a1 1 b1 1 )m - • - : , (b)
a ; ; bj j ·
< i <j <
:::=
i= I
det [Lm ( A ' ·2 B
�
C'
•7:
D)L � J =
IT ( b " aj j + d;; r1 ; )
• ?. J
( 6 ) A. B, C. D ( m x m ) nonsingular, lower triangular : -:- .- ( C ' ) - 1 j1 L' · r ( ·1 B' -.'. · C' D) L 'm .� - 1 = L m [( B' ) - 1 '7'· D - 1 ]L' L [ .. • - 1 .... ( 7 ) A = � a ;j ] ( m x m ) lower t riangular. n E [\ : l det Lm L (A' t - l - j 8 AJ L'm = n rr. ( det A t - 1 IT Jlu . L
m
•
......_.
-·
f'l1
rn
""1
··
Tl -
J=C
where Jl k l
=
{
( 8 ) A = [aij J · B = [ b ;J ]
k>I
( a )\ - a /1 )/ ( au - au ) n a nk k- 1
(m
x
m) :
m n_ ,
-.
where a1 n := l . . . . . m .
if a H if a H
a ,.. ..
'1
Un n
b,., ,
f.
a1 1
= a11
n:
·
H A N D BOOK O F M ATRICES
130
9.6.5
Elimination Matrices, Duplication Matrices and Kronecker Products
( 1 } A (m x m) :
(a) L m ( A ·Sl A ) Dm = D� ( A 0 A)Dm . ( b ) Lm ( /m 0 A + A 0 lm )Dm = D� ( /m 0 A + A 0 lm )Dm . {2} .4 = diag(a1 1 , . . . , a mm ) , B = diag (bl l , . . . , bmm ) : Lm ( A 0 B ) Drn is diagonal matrix with diagonal elements a ;;bjj . 1 < i < j < (3} A = [a;j] . B = [b ,j] ( m x ) upper (lower) triangular: (a) L m ( A 0 B ) Dm is upper ( lower) triangular . m (b) det[Lm ( A 0 B ) Dm ] = II b:;a;'.' - i + l . a
m.
m
[ c;j ] ( m x m ) lower triangular: (a ) Lm ( A c;;, Lm ( A 0 B ' ) L 'rr, . ( b ) det[Lm ( A B' ;-:;. B' A ) L:n J = det[L m ( A B 0 B' A ) Dm ] = det[Lm ( A B' 0 A' B' ) Dm] = ( de t A ) m + 1 ( de t B) m + 1 . m ( c ) det[Lm( A 8 B' C ) Dm ] = II (b;; c;; ); a;'.'- i+ t .
(4} A = [a ;j] , B
=
[b;j ] , C = B ' ) Dm =
i=l
m
( d ) det[Lm ( A B' 0 C' ) Dm ] = II c;; ( a;; b;; )m - i + l . i=l ( 5 ) A. B ( m x m ) , det ( B) ::f. O :
det[L111 ( A 0 B )D, ] and
=
m- 1
det ( A )(det B ) m II det(q n )) n=l m- 1
det [ L m ( A 8 B ) /J�, ' ] = 2- m ( m - 1 J/2det( A ) ( det B ) m II det( Cr " ) ) , n= where l
are the principal su bmatrices of c = [ Cjj l = A s - l . Note: The results of this section are taken from Magnus ( 1 988) where also further results on elimination matrices may be found.
131
PROPERTIES O F SPECI A L M ATRIC ES 9.7
Hermitian Mat rices
Definitions: An ( m x m ) matrix A is Hermitian if AH = A, that is, the ijth element a;j is the complex conjugate of the jith element., iij i = a;i , t h at is,
w here the a;; , i I , . . . , m , are real . An ( m x m ) m atrix A is skew-Hermitian if A H = - A , that is , the ijth element a;j is - 1 times the complex conjugate of the jith element, - iii ; = a;i , that is, =
9. 7 . 1
General Results
( 1 } A (m
) real : A is Hermitian x m ) Hermitian , c E IR :
x m
( 2 } A, B ( m
<==>
A is symmetric.
(a) cA is Hermitian.
( b ) A ± B is Hermitian . (c) A B
=
BA
=?
A B is H ermitian.
( d ) A 8 B is H ermitian.
(3) A (m
x m
) Hermitian , B ( n
x n
(a) A 0 B is H ermitian .
) Hermitian:
( b ) A EB B is H ermitian . ( 4 ) A = [a;j) ( m
x m
) Hermi tian :
(a) A' is Hermitian . ( b ) A H is Hermitian .
( c ) A- 1 is Hermitian , if A is nonsingular.
( d ) Ai is Hermitian for i = 1 , 2, .
-
(e) A is Hermitian .
..
1 32
H A N DBOOK O F M ATRICES
(f)
I A i abs
is Hermitian . is Hermitian for i
(g ) a; 1
a;;
( h ) .4 11 A = (5) A (m x m) :
A A 11 ,
that is,
A
=
1 , 2, . . . , m .
is normal .
(a) A + A 11 is Hermitian . ( b ) A - A 11 is skew- Hermitian . ( c ) AAlf is Hermitian . ( d ) .4. 11 A is Hermitian . ( ) There exist unique Hermitian ( m x m ) matrices B , C such that A = B + iC. ( 6 ) A (m x m) : e
(a) A is Herm itian (b) A is Hermitian ( c ) A is Hermitian B.
<:=:}
<:=:}
¢::::::}
is skew- Hermitian. x 11 Ax E IR. for all ( m x 1 ) vee tors x . B 11 A B Hermitian for all ( m x m ) matrices iA
( d ) A is Hermitian {::::::} A 11 A = A A 11 and all eigenvalues of A are real numbers. (e) A is Hermitian {::::::} there exists a unitary ( m x m ) matrix l ' and a real diagonal ( m x m) matrix A such that A U !I.. U 11 . ( Householder transformation ) 2 x x 11 x ( m x 1 ) : lm - 11 is Hermitian . X J: A is Hermitian => B11 A B is Hermitian. A ( m x m ) . H ( 111 x ) A, B ( m x m ) Hermitian: A H HA {::::::} there exists a unitary ( m x m ) matrix C such that [.' A U 11 and U BU 11 are diagonal matrices. A is similar to a real ( m x m) matrix B {::::::} A ( m x m) there exist Hermitian ( m x m ) matrices H and C', one of which is nonsingular, such that A II C. .4. , B ( m x m) Hermitian : =
( 7) (8) (9)
( 10)
n
:
=
:
=
(11)
(a) [tr( AB }f < t r( A B 2 ) . ( tr A )2 . ( b ) rk( A ) > , f tr( A 2 ) :f 0 . t r( ) 1 2
�� /t�
1 33
P RO P E RTIES O F SPECIAL M ATRICES
( ) rk ( A 8 B) < rk ( A ) rk ( B ) . ( 12 ) A (m x m) , b (m x 1 ) , c E IR : . . c bH t. H B= ermtttan => det ( B) s b A c
]
[
=
c det( A) - bH Aadj b.
Note: M any of these results are elementary. The others can be found 1 n Horn & Johnson ( 1985 ) . 9. 7.2
Eigenvalues of Hermitian Matrices
Notation:
-\( A )
denotes an eigenvalue of the matrix A .
Amin ( A )
is the smallest eigenvalue of t he matrix A .
Amax ( .4 )
is the largest eigenvalue of t he matrix A .
( 1 ) A ( m x m) H ermitian : ( a) A l l eigenvalues of A are real numbers. <:::=?
( b ) A is positive definite greater than 0 .
all eigenvalues of A are real and
( c ) A is posit i ve semidefinite <:::=? al l eigenval ues of A are real and greater than or equal to 0 . ( 2 ) ( Rayleigh-Ritz theorem ) A ( m x m) Hermitian : A
min
xH Ax { ( A ) = min : X (m X 1 ) , :f. 0 } , XH X { xH Ax x (m x 1 ) , x :f. O } . = max X
Am ax ( A )
XH X
:
( 3 ) ( Courant-Fischer theorem ) A ( m x m) Hermitian with eigenvalues -\ 1 < · · · < A m , 1 < i < xH A x -\; min max : !J J . · · · Ym - • x X H (m X I)
{
=
X ( m X 1 ) , X :f. 0 , X H Yj and -\;
=
YJ
m ax
, y, _ l
(mX 1) . . . .
min
= 0, j =
1, . . . , m
{ xHAx :
- i}
XHX
X ( m X 1 ) , X :f. 0 , XH Yj = 0 ,
j = l, . .
.,i- 1
}·
m :
H A N D BO O K O F M AT RI C ES
1 34
( 4 ) A ( m x m) Hermitian with eigenvalues A 1 < · orthonormal eigenvectors VJ , . . . , Vm n E { 1 1
· · < Am and associated 1 m} : min{tr(XH AX) : X ( m x n ) XH X = In } = ) q + · · · + An . The minimizi ng matrix is X = [V 1 1 1 vn] · max{tr(XH AX) : X ( m x n ) 1 XH X = In } = Am + · · · + Am - n+l · Vm -n+d · The maximizing matrix is X = [vm1 1
•
•
•
1
•
•
•
9. 7.3
(1)
(2)
A ( m x m) Hermitian,
A (m
•
•
•
x
(6)
1
:f. 0 ( m x 1 ) :
:
1 m.
x
m ) Hermitian with eigenvalues X ( m x n) : xn x
(5)
•
x HAx Ami n ( A) < X H X < Am ax( A). A ( m x m ) Hermitian with eigenvalues A 1 (A), . . . , Am (A) : { xHAx : x (m x 1 ) , x :f 0 min xHx } { x H Ax x ( m x 1 ) 1 x :f 0 < A; ( A ) < max i = 11
(4)
•
Eigenvalue Inequalities
xHx
(3)
•
= In
}
1
A 1 (A) < · · · < Am(A),
n n :�::::.x i (A) < tr(XH AX) < :�:::> m - n+; ( A). i= I i=I
A = [a;j] ( m x m ) Hermitian: Am;n (A) < a;; < Amax (A), i = 1 , . . . , m . A = [a;j ] ( m x m ) Hermitian with eigenvalues A 1 (A) < · · · < Am (A) : n n n i=l i= l A = [a;j ] ( m x m ) Hermitian with eigenvalues 0 < A 1 (A) < · · · < Am (A) : n II A ; (A) < II a ii, n = 1 , . . . , m. i=l i= l n
1 35
P RO P E RTIES OF S P EC I A L M ATRICES
( 7 ) ( Inclusion principle ) < A ( m x m ) Hermitian with eigenvalues .X I ( A) < Am ( A ) , A( n J ( n x n ) a principal submatrix of A with eigenvalues
A J ( A (n) ) < · · · < A n ( A ( n ) ) :
< Am- n + i ( A ) ,
.X ; ( A ) < .X ; ( A c n J )
i = 1 , . . . , n,
Aman ( A ) < Amin (A (n) ) < Amax ( A (n) ) < Amax (A) . (8) A (m
x m
(9) A, B (m
) Hermitian , B ( m
x m
x m
Am;n (A + B)
>
Amin ( A ) ,
Amax ( A + B)
>
Amax (A) .
) H ermitian :
Amin ( A + B) ( 10 )
(11)
A, B (
m x m
) p ositive semidefi nite:
>
Amin(A) + Am;n (B).
Amax (A + B ) < Amax( A ) + Amax ( B ) . ) H ermitian , 0 < a, b E IR. : Am ; n ( aA + bB)
>
Amax ( aA + bB)
<
a.Xmin ( A ) + b.Xmin ( B) ,
aAmax ( A) + b.Xmar ( B ) . A( ) H ermitian with eigenval ues .X 1 ( A ) < · · · < .X, ( A ) , B ( ) H ermitian posit i ve semidefinite, .X I ( A + B ) < · · · < Am ( A + B ) m x m
m x
m
eigenvalues of A + B :
.X ; ( A + B ) > .X ; ( A ) ,
i = 1 , . . . , m.
( 12 ) A ( m x m ) Hermitian with eigenvalues .X 1 ( A ) < · · · < Am (A), x (m x 1 ) , A 1 ( A ± xxH ) < · · · < A m ( A ± xxH ) eigenvalues of A ± xxH : .X ; ( A
±
x xH ) < .X ; + 1 ( A ) < A; + 2 ( A ± xxH ) ,
.X ; ( A ) < .X; + 1 ( A ± xx H )
< .X;+ 2 ( A ) ,
i = 1 , 2, . . . , m - 2;
i = 1 , 2,
...
, m -
2.
( 13 ) A , B ( m x m ) Hermitian , rk ( B ) < r, .X 1 ( A) < · · · < Am ( A ) eigenvalues of A , .X 1 ( A + B) < · · · < A m ( A + B ) eigenvalues of A + B : .X ; ( A + B ) <
.X; (A)
<
Ai+ r ( A)
<
Ai+ r (A + B )
A;+ 2 r ( A + B ) , <
.X;+ 2 r(A),
i
= 1 . 2,
. . . , m - 2r;
i = 1 . 2 , . . . , m - 2r·.
1 36
H A N DBOOK OF M ATRICES
( 14) ( Poincare's separation theorem )
A(
m x m
.X 1 ( A) < · < A m(A), X ( .X , ( XHA X ) < . . . < An (XH AX )
) Hermitian with eigenvalues
n ) such that X H X = eigenvalues of XH AX
·
m x
·
ln , n < : .X ; (A) < A;( X H AX) < Am- n+ i (A), i = 1 , , n. ) H er mitian with eigenvalues .X 1 (A) < · · :S Am(A) and ( 15 ) A, B ( .Xt (B) < · · · < Am ( B ) , respectively ; .X , ( A + B ) < · · · < Am ( A + B) eigenvalues of A + B : n n ( a) 2:: [ -X; (A) + .X ; ( B ) ] < 2:: .X ; ( A + B), n = 1 , m,
. . .
m x m
·
.
i=l
. . , m.
.X , ( B ) < An ( A + B) - An ( A ) < Arn ( B), = I , (c ) I .Xn (A + B ) - An ( A ) Iabs } n = 1, < max{ I .Xj ( B)Iabs : j = 1 , ( d ) An(A + B ) < min{.x ; ( A ) + Aj (B) i + j = n + } = 1, ( b)
. . . , m
n
. .
.
m
,
, m.
·
and
.X; ( A + B) <
.X t (A) < · · · < Am( A) and + B ) < · · · :S A171 ( A + B )
.X; (A) + .A t ( B) .X ;_ 1 (A) + .X 2 ( B)
.X;(A + B ) >
(17)
. . . , m.
,
:
( 16 ) ( Weyl 's theorem ) A , B ( m x m ) H ermitian with eigenval ues .X , ( B) < · · < Am(B), respectively ; .X , ( A t>igenvalues of A + B :
i
. . . , m.
n
.X; (A) + A m ( B ) .X i + t ( A) + Am- t ( B)
= 1, .4 , B ( m ) Hermitian with eigenval ues .X 1 ( A ) < · · < Am ( A ) an d .X , ( B) < · · < Am( B), respectively : .X , ( A - B) < · · · :S Am ( A - B ) eigenval ues of A - B : .X i ( A - B) > 0 => .X; (A ) > .X ; ( B), i 1 , A( ) Hermitian with eigenvalues .Xt (A) < · · · < Am (A), ( .
. . , m.
x m
·
·
=
(18)
m x m
1 ) , c c IR,
[
A B = XH
x] C
. . . , m.
x
m x
1 37
P RO P ERTI ES O F S P EC I A L M ATRICES
with eigenvalues .A 1 ( B ) <
·
·
·
< Am+ I ( B ) :
.A ; ( B ) < .A ; ( A ) < Ai+ I ( B ) ,
i
= 1, . . . , m.
( 19 ) A ( m x m ) Hermitian w ith eigenvalues .A 1 ( A ) < p) Hermitian , C ( m x p) and B
·
·
·
·
< Am(A),
D (p x
< An ( B ) :
.A ; ( B ) < .A; ( A ) < An -m+i ( B ) , 9. 7.4
·
[ �H � ] ( n x n )
=
with eigenvalues .A 1 ( B ) <
·
i
= 1 , . . . , m.
Decompositions of Hermitian Matrices
( 1 ) (Spectral decomposition) A = U A UH . A ( m x m ) Hermitian with eigenvalues .A 1 , . . , .Am w here A = diag( .A 1 , . . . , A m ) an d U is the unitary (m x m) matrix w hose columns are the orthonormal eigenvectors v 1 , . . . , Vm of A associated with .A 1 , . . . , A m . In other words, m A = L .A ; v; v f . i= I .
( 2 ) ( Congruent canonical form) A ( m x m ) Hermitian : There exists a nonsingular ( m x m ) matrix T such that A = TATH , w here A = diag( d 1 , . . . , dm ) and t he d; are + 1 , - 1 or 0 corresponding to the positive, negative and zero eigenvalues of A , respectively. ( 3 ) ( Si multaneous diagonalization of two Hermitian matrices) A , B ( m x m ) H ermitian , AB = BA : There exists a unitary ( m x m ) matrix U such t hat A = U D U H and B = U A U H , w here D and A are d iagonal m atrices. ( 4 ) ( Simultaneous diagonalization of a p ositive definite and a Herm itian matrix) A ( m x m ) positive definite , B ( m x m) Hermitian : There exist s a nonsingular ( m x m ) matrix T such that A = TTH and B = TATH , w here A is a diagonal matri x . Note: M ost results of this section are given i n Horn & J ohnson ( 1 985. Chapter 4 ) or follow easily from results given there. For the results on eigenvalues see also Chap ter 5 and for the decomposition theorems see C'haptt>r 6.
H A N DBOOK OF M ATRICES
1 38
9.8
Idempotent Matrices
Definition: An ( m x m ) matrix A is idempotent if A2 = A . ( 1 ) A ( m x m ) idempotent , B ( n x n ) idempoten t : ( a ) A 0 B is idempotent.
( b ) A EB B is idem potent. (2)
.4 ( m x m ) idempotent : ( a) .4' = A ,
(b)
.4 '
i
= 1,2 . . . .
is idem potent . ( c ) .4 H is idempotent.
( d ) Im - A is idempotent. (e) A l l eigenvalues of A are 0 or 1 . (f) r k ( .4 ) = t r( A ) .
( g) A is simple.
( 3 ) A ( m x m ) idempotent:
( a ) A is uonsingular => A = lm .
( b ) A is Hermitian => A+ = A . ( c ) A is Hermitian
=>
A is positive semidefinite.
( d ) A is diagonal => t he diagonal elements of A are 0 or 1 .
( e ) rk( A ) = r· => t here exists a nonsingular matrix Q such t hat
w here the zero submatrices disappear if r = m. (4) A ( m x n) : (a) A A+ and A+ A are idempotent. ( b ) Im - AA+ and In - A+ A are idempotent . ( c ) A ( A H A ) + A H is idempotent. ( d ) A H ( AA H ) + A is idempotent . ( e ) Im - A ( A H A )+ A H is idempotent. (f)
In -
A H ( A A H )+ A is idempotent.
( 5 ) A ( m x m) :
A idempotent
{::::::}
rk( A ) + rk( Im - A ) = m .
( 6 ) A; ( m x m ) idempotent , i = 1 , . . . , n :
139
P RO P E RTIES O F S P E C I A L M ATRICES
(a) A ; Aj = O m x m . i :f j => A 1 + · · · + A n is idempotent. (b) A 1 + · · · + A n is idem potent ::::> rk ( A 1 + . . . + An ) = rk(A l ) + · · · + rk( A n ) .
( 7 ) A ; ( m; x mi ) idempotent for i = 1 , . . , r : .
is idem potent.
Note: Most of t hese results follow immediately from t he defining property of idempotency. A number of results is given in Lancaster & Tismenetsky ( 1 985) and other matrix books.
9.9
Non negative, Positive and Stochastic Matrices
In this section all matrices are real unless otherwise noted .
9. 9 . 1
Definitions
A = [a;j ] . B = [b;j ] ( m x
n
) real :
A is nonnegative ( A > 0 or 0 < A) if a;j > 0, i = 1 , . . . , m, j = 1 , . . . , n . A is positive ( A > 0 or 0 < A ) if a;j > 0, i = 1 , o o . , m , j = 1 , o o . , n . A<
B (or B > A ) if a;j < b;i , i = 1 ,
A<
B (or B > A ) i f a;i < b;i , i = 1
A = [a;j ] ( m x m) real : A is stochastic if 0 < a;j < 1 , 1, . . . , m.
l,J
,
0
0
•
0
0
•
, m, j = 1 , ,
= 1,
m, j = 1 ,
0
0
.
0
0
•
0
0
•
, m, and
, n. ,
n.
r:; I a;j
A is doubly stochastic if 0 < a;j < 1 , i , j = 1 , . . . , m , 1 , i = 1 , o o . , m , and I:7' 1 a;j = 1 , j = 1 , o o . , m. p ( A) - max { I .X I abs :
A
= 1, i =
I:;'
1
a;j
is eigenvalue of A } is the spectral radius of A.
140
H A N DBOOK OF M ATRICES
9.9.2
General Results
A l l results for non negative matrices are also valid for stochastic and doubly
stochastic matrices because they are special nonnegati ve matrices. ( 1 ) A , B ( m x n) , a , b E lR : ( a) A > 0 , B > 0 , a , b > 0 ::::} a A + bB > 0.
( b ) A > O ::::} A' > O. ( c ) A > 0 ::::} A' > 0. ( 2 ) A, B ( m x
m
)
:
( a ) A > 0 ::::} A' > 0 , i = 0 , 1 , 2 . . . ( b ) A > O ::::} Ai > O , i = 1 , 2 . . . (c) 0 < A < B
::::}
.
.
0 < A' < B' , i = 0 , 1 , 2 . . .
( 3 ) A, B ( m x m ) , C . D ( m x n ) : 0 < A < B, 0 < C < D
::::}
0 < AC < B D .
( 4 ) A ( m x n ) , x ( 11 x 1 ) :
(a) A > 0, x > 0, x :f. 0 ::::} Ax > 0 .
( b ) A > 0 , x > 0 , Ax = 0 ::::} A = 0 . ( 5 ) A ( m x m ) , U = diag( dJ , . . . , dm ) : A > 0 , d; > 0 . i 1 , . . . , m ::::} D- 1 A D > 0. 1 ( 6 ) .-1 ( m x m ) nonsingular, A > 0 : A - > 0 {::::::} A is a general ized permutation matrix . =
(7) A (m x m), A > 0 :
( a ) A is t ridiagonal ::::} all eigenvalues of A are real .
(b) (8) A, B
(a)
(b) (9)
A
A
is stochastic
{::::::}
1
1
1
1
A
( m x m) : A,
A.
H are s tochastic ::::} A B is stochastic.
B are doubly stochastic
::::}
A B is doubly stochasti c .
( 111 x 1T1 ) : A is doubly stochastic <:==:} for some 11 E IN , t here exist ( m x 111 ) permut ation 11 1atrices P1 . . . , Pn and c 1 , c,. E U{ with C J + · · · + c,. = 1 such that A = c 1 P1 + + cn Pn . ,
.
·
·
•
.
,
·
Note: More 0 11 non 1 1egati ve watrices can be found in Mine ( 1 988 ) , Sen eta ( 1 97 3 ) . Berman , �euman n & Stern ( 1 989) , Berman & Plemmons ( 1 97 9 ) .
P RO P E RTIES O F S P ECI A L M ATRICES
141
M any of the results of t his section can also be found in Horn & Johnson ( 1985, C hapter 8 ) . 9.9.3 (1 )
Results Related to the Spectral Radius
A = [a;j ] ( m x ) (a) A > 0 ::::} p(A) > 0 . m
:
m
(b) (c) (d) (e) (2)
(3)
A > 0 , 2::::a ii > 0 , i = 1 , . . . , m ::::} p(A) > 0 . j =I A is stochastic ::::} p( A) > 0 . A i s doubly stochastic ::::} p(A) > 0 . A > 0 , A' > 0 for some i > 1 ::::} p(A) > 0 .
A, B ( m x m ) (a) 0 < A < B ::::} p( A) < p(B). ( b ) 0 < A < B ::::} p(A) < p(B) . A = [a;j] ( m ) , A > 0 : p(A (n) ) < p(A ) , :
m x
are principal submatrices of (4)
n = 1,
.
..
, m,
A.
A = [a;j] ( m x ) A > 0 : ( a) :=max ,l . . . , m a;; < p(A) . m ,
(b) (c )
m
m
m
m
· L: a;i < p(A) < . max L a • i m t = l , . . . , z =min , l J=l J=l . . .
,
rn
_ -l, m i n L a ,J < p(A) < _max L a ;j . _ . J - 1, . , m •=I J . . . , m •=I . .
(d) (5)
A = [a ;j] > O ( m . min, m
•=1,
x m), m
-1 '""
X;
x (x 1 , . . . , .rm ) ' > 0 ( m x =
1
1) :
m
· ; a . maxm - '"" J � a · xJ· .J = l xJ· -< p(A) -< •=1, , J=l �
Xj
.
I)
l
where
142
H A N DBOOK O F M ATRIC ES
. min J=l, , .
rn
Xj
m
a ·
L . _!_!... Xi
•=I
m
< p( A ) <
a··
. =ml a.x, m Xj L . _!_!..., i
J
,
.
•=I
X
( 6 ) A ( m x m ) , A > O , x ( m x l ) , x > O , a, b E IR, a , b > O : ( a ) ax < Ax < bx ( b ) ax < A x < bx ( c ) A x = ax
=}
=}
=}
a < p( A ) < b. a < p( A ) < b.
a = p( A ) .
( 7 ) A ( m x m ) , A eigenvalue of A with eigenvector x :f- 0 : A > 0 . I A i abs = p( A ) =} I A i abs is eigenvalue of A with eigenvector l x l abs · ( 8 ) A ( m x m ) , .4 > 0 : p( A) is eigenvalue of A with eigenvector x > 0 , x :f- 0 . (9) A (m x m), A > 0 :
( a) p( A ) is a simple eigenvalue of A . ( b ) p( A ) is eigenval ue of A with eigenvector x > 0 . ( c ) A is eigenvalue o f A , A :f- p( A )
=}
I A i abs < p( A ) .
( 1 0 ) ( H opf's theore m ) A [a,j J ( m x m ) , A > 0 , A m - 1 eigenvalue o f A w i t h second largest max { a;J : i, j = l , . . . , m } , J.l. = min{ a;i : i , j = modulus, M l , . . . , m} : l Am - I labs < M - J.1. < 1. p( A ) - M + J.1. =
( 1 1 ) A ( m x m ) , A > O , x > 0 ( m x 1 ) eigenvector of A corresponding to eigenvalue p( A ) , y > 0 ( m x 1 ) eigenvector of A' corresponding to eigenvalue p( A ) , x'y = 1 : lim [p(A ) - 1 A ]; = xy' . • - oo
( 12 ) A ( m x m) , A > 0 : ( a) l im [p( A ) - 1 A]i > 0 .
Note: The results o f this subsection are given i n Horn & J oh nson ( 1 985, Chapter 8).
9.10
Ort hogonal Matrices
Definition: A n ( m x m) m at rix A is orthogonal if it is nonsingul ar and A - 1 = A' .
143
P RO P E RTIES OF S P ECIAL M ATRICES
9.10.1
General Results
( 1 ) A (m x m) : ( a) A is orthogonal
-¢::::::>
AA' = Im
-¢::::::>
A' is orthogonal . A H is orthogonal .
( b) A is orthogonal (c ) A is orthogonal
(d ) A is orthogonal (e)
A
is orthogonal
-¢::::::> -¢::::::> -¢::::::>
-¢::::::>
A' A = Im .
-
A is orthogonal . A - I is orthogo nal .
( 2 ) A ( m x m ) orthogonal : ( a)
Ai
is orthogonal for i = 1 , 2 , . . .
( b ) jdet ( A ) I abs
=
1.
( c ) rk( A ) = m . ( 3 ) A ( m x m ) real : (a)
A
is orthogonal => A is normal .
( b ) A is orthogonal (c)
A
-¢::::::>
A is u nitary.
is orthogonal -¢::::::> x' A' Ay = x'y for all real ( m x 1 ) vectors x , y. (d) A is skew-symmetric => ( Im - A ) ( Im + A ) - 1 is orthogonal. (4)
( a ) The ( m x m ) u n i t m atrix Im is orthogonal . ( b ) The ( m n x mn) commutation matrix l\m n is orthogonal .
[ cosSln. B
] 1. s orthogonal.
( c ) A permutation m atrix is orthogonal .
(d) For B E IR ,
8
-sin B COS 8
( 5 ) A; ( m x m) real i = 1 , 2, . . . : => l i m A; is real orthogonal . ( 6 ) .4 , B ( m x m ) :
A ; is orthogon al and lim A; exists
A , B are orthogonal => AB is orthogonal .
( 7 ) A ( m x m ) orthogonal , B ( n x n ) orthogonal : ( a) A 0 B is orthogonal. ( b ) A EB B is orthogonal.
( 8 ) x (m x 1 ) , A (m x m ) : (9) A (m x n), B (n x n) : ( 10 ) A ( m x m ) , B ( m x n ) :
A is rea l ort hog on al => I ! Ax ll2 = l l x l h ·
B is real orth ogo nal => I I A B I I 2 = I I A i h .
A is real ort hog ona l => I I A B I I 2 = I I B i h ·
( 1 1 ) A ( m x m ) real orthogonal:
H A N DBOOK OF M ATRICES
1 44
(a) A is eigenvalue of A => l -\ l abs = 1 . ( b ) ,\ is eigenvalue of A => 1 / ,\ is eigenvalue of A .
( 1 2 ) A ( m x m ) real with eigenvalues -\ 1 , . . . , Am : A is normal . I -\; l a bs = 1 , i = 1 , . . . , m -¢::::::> A is orthogonal. ( 1 3 ) A ( m x m ) real : is sim ilar to A ' .
A is similar to a real orthogonal matrix
=>
A-1
Note: The results of this subsection may b e found i n Horn & Joh nson ( 1 98,5 ) or follow easily from (:� ) . 9. 10.2
Decompositions of Orthogonal Matrices There exists a real orthogonal ( m x m )
( 1 ) ..1 ( m x m ) real ort hogonal: matrix Q such t hat
0
Q ''
A = Q
0
where t he -\; = ± 1 and the A; are real ( 2 x 2) matrices of the form A; =
cos 0, [ -sin B;
sin B; cos B;
].
( 2) A ( 111 x 111 ) ort hogonal : There exists a real ort hogonal ( m x m ) matrix Q and a rPal skew-symmetric ( m x m ) matrix 5 such that A = Qexp ( i S ) . Note: For t h e n·sn l ts of t his subsect ion see Chapter 6 .
9.11
Partitioned Mat rices
Definition:
An
( m x 11 ) matrix
145
P RO P E RTIES OF S P E C I A L M ATRICES
consisting of ( m ; x n1 ) sub matrices A;j , i = 1 , . . . , p, j 1 , . . . , q , is said to be a part itioned matrix or a block matrix. Special cases are =
A = [A 1 , . . . , An] = [A t : . . . : An] ,
where the A; are blocks of columns, and A=
where the A1 are blocks of rows of A . 9.11.1
General Results
( 1 ) A ( m x n ) , B ( m x p) , C (q x n ) , D ( q x p ) :
(a)
[ �· � r [ �: g: ] . -
[ � � ] [ 2. � ] . A B ]H A ll CH ] [ (c) = [ BH DH C D [ (d) [ A B I A i abs ! B l abs ] C D ] abs - I C iabs I D iabs .
(b)
=
·
_
( 2 ) A ( m x n ) . B ( m x p ) , C ( q x n ) , D (q x p) . E ( n x r ) . F ( n x s ) , G (p x r ) . H (p x s ) :
[ AC DB ] [ GE HF ] [ A EE -
( 3 ) A (m x m) , B ( m
x
n ),
tr
] . DH
A F + BH
CF +
x m), D (n x n) :
[� �]
( 4 ) A ( m x n) , B (p x q) :
rk
C (n
C
+ BG + DC
[� �]
=
=
tr ( A ) + tr( D).
rk ( A ) + rk( B ) .
-\(A ) and -\ ( B ) are eigenvalues of A and B. respectively, with associated eigenvectors t•( A ) and t• ( B) => -\ ( A )
( 5 ) .4 ( m x m ) , B ( n x n ) :
146
H A N D BOOK OF M ATRICES
and }.. ( B ) are eigenvalues of
with eigenvectors
[
v( A ) O
]
[ v (0B ) ] '
an d
respectively. ( 6 ) A ( n x n ) Hermitian with eigenvalues A J ( A ) < · · · < A ( A ) D ( p x p ) Hermitian , C ( n x p) and n
,
with eigenvalues A t ( B) < · · < Am ( B ) : ·
A ; ( B ) < A ; ( A ) < Am - n +; ( B )
( 7 ) A ; ( m; x m; ) , i = l , At
. . . , r :
0
I
-
(a) 0
for i = 1 , . . . , n .
Ar
A't
0 i = 0, 1 , 2, . . .
A'r
0
( b ) A; is idempotent , i = l , . . . , r
is idempotent.
9 . 1 1.2 ( 1 ) det
Determinants of Partitioned Matrices
[ I� 10 ]
=(
-
1 )m .
( 2 ) A ( m x m) , B ( m x n) , C ( n x n ) :
[� �] det [ � g ]
det
nxm
mxn
=
det ( A )det ( C ) ,
= det ( A )det( C ) .
147
P RO P E RTI ES O F SPECIAL M ATRI CES
( 3 ) A , B, C ( m
x
m) :
[ CA OBm x m ] ( - l )m det( B)det (C), mxm det [ g ! ] = ( - l )m det( B)det ( C ) . det
( 4 ) A ( m x m) , B ( m
=
) C (n
x n ,
(a) A nonsingular
=>
( b ) D nonsingular
=>
x
m), D (n
[ :,
�]
9. 1 1 .3
:
[ � � ] = det ( A )det ( D - CA - 1 B ) . det( D)det ( A - B D - 1 C). det [ � � ] =
1),
x
E (C :
c
= c det( A ) - b' Aa
( 6 ) ( Fischer's inequality ) A ( m x m) , B ( m x n ) , C ( n D=
)
det
(5) A (m x m ) nonsingular, a , b ( m det
x n
x n
)
:
[ �H � ] positive definite
=>
det ( D) < det(A )det (C) .
Partitioned Inverses
( 1 ) A; ( m; X m; ) nonsingular, i = l , . 0
-
0 (2) A (m x m), B (m
( a)
.
.
, r :
I
0 0
) C (n
x n ,
x
m) , D ( n
x n
)
:
[ � � ] , A and ( D - CA- 1 B) nonsingular =>
[
[ � � rl A - 1 + A - 1 B ( D - C A - 1 B) - 1 CA - 1 - ( D - CA - 1 B) - 1 C A - 1
- A - 1 B( D - CA - 1 B) - 1 ( D - CA- 1 B) - 1
]
.
148
(b)
[� �] r� �
�
[
(3)
H A N D BOOK O F M ATRICES
,
D and (A - BD- 1 C) nonsingular
l r
( A - BD- 1 C) - 1 - ( A - BD- 1 C') - 1 BD- 1 - D- 1 C(A - BD- 1 C)- 1 D- 1 + D - 1 C( A - BD - 1 C) - 1 BD - 1 A (m x m) sym metric, B ( m x n), C ( m x p) , D (n x n ) symmetric, E ( J > x p) sym met ric : A B C -I B' D 0
C'
0
E
-FBD- 1 - FCE- 1 F D- 1 B' FCE - 1 - D- 1 B'F D - 1 + D- 1 B' F BD- 1 E- 1 + E- 1 C' FCE-1 E- 1 C' FBD- 1 - E- 1 C'F if all inverses exist and
F = (A - BD- 1 B' - CE- 1 C' ) - 1 . n , A ( m x n), B ( m x ( m - n)) :
(4)
m, n E
(5)
m. n E IN , m < n , A ( m x n ) , B ( (n - m) x n ) :
IN , m >
rk( A )
�
9 . 1 1.4 (1) A
rk( B ) = n - m, AB11 = 0 � [A H ( A A H ) - I : BH ( B B H ) - 1 ] .
= m,
[ rl
Partitioned Generalized Inverses
(m x
m
) nonsingular .
B ( m x n), C ( r x m ) . D ( x n ) : r·
rk [ � � ] = m. D = cA - 1 B [ OAn-x 1m OOnmxxrr ] t·s a genera1.tzed .mverse or [ CA � ] . __..... -.r
(2) A
=
( m, x u, ), i =
1,
.
...
r :
]
.
PROPERTIES O F SPECIAL M ATRICES
AI
( a)
149
0
0
is a generalized inverse of 0
0
AI
+
0
0
(b) 0
(3)
(4)
A ( m x n) :
A ( m x n) , B ( m x p) :
[
A 0
0 0
] + = [ A+ ]
_[
D A+ A+ ) B (C+ + . ] [A B + c+ + D ·
w here D
(5)
0
]
0 0
.
·
C = ( /m - AA+ ) B and
= ( /p - C + C )[Ip + ( /p - C+ C )BH (A + )H A+B(lp - C+c)t 1 x BH ( A + )H A + ( /m - BC + ) .
A ( m x n) , B ( p x n) :
[ � r = [A + - TsA + : T] ,
= E+ + ( In - E+ B)A + ( A + )H BH 1\(lp - EE+ ) with E = B(ln - A+ A) and /\. = (Ip + (lp - EE+ ) BA + ( A + )H BH ( lp - EE+ W 1 .
w here T
9. 1 1 .5 ( 1 ) Dn
Partitioned Matrices Related to Duplication M atrices
(n2 x �n(n + 1 ) )
duplication m atrix : 1 0 0 1 0 0
0
2/m 0
0
� fm 0
0 0 D:r. Dm
H A N DBOOK OF M ATRICES
! 50
(2)
A(
)
m x m ,
a, b (
m x
1 ) , c E (C :
[
1
C
Dm+ l vee a
[
�
]
c
]-
+ l vee c b' Dm+ a A (3)
A, B ( (a)
m x m
D� + l
) symmetri c , a, b ( m
x
a+b D:n vee( A) c
;(a + b) D;!:. vee( A) 1 ) , a, {3 E (C :
( [ : � ] 0 [ � � ] ) Dm+ l =
ab' + (3a' (a' ® b')Dm a B + {3A + ab' + ba' (a' 0 B + b' ® A)Dm D:n (A ® B)Dm D� (a 0 B + b ® A)
(a' 0 b')D;!:. a(3 ; ( ab' + (3a') ; (ab + (3a) haB + {3A + ab' + ba' ) ; (a' 0 B + b' 0 A ) D;t.' D;!:. (a ® b) ; D;t. (a 0 B + b 0 A) D;t", ( A 0 B)D;t.' '
Note: Results on partitioned matrices including a number of the foregoing ones are given , for i nstance, in M agnus ( 1 988) and Magnus & Neudecker ( 1 98 8 ) . Many of the foregoing results are immediate consequences of definitions.
9.12
Positive Defi nite, Negative Definite and Semidefinite Matrices
Definitions: A Hermitian ( m • • •
•
x m
) matrix
A is
positive defin ite if xH Ax > 0 for all ( m x 1 ) vectors x # 0; positive semidefinite if xH Ax > 0 for all ( m x 1 ) vectors x; negative definite i f xH Ax < 0 for all ( m x 1 ) vectors x # 0 ; negative semidefinite i f xH Ax < 0 for all ( m x 1 ) vectors x .
Note that a definite m atrix is always Hermitian accord ing to t his definition. A real sym metric ( m x m ) matrix A is •
positive definite if x' Ax
> 0 for all real (
m x
1 ) vectors
x # 0;
151
P RO P E RTIES OF S P E C I A L M ATRICES •
• •
positive sem idefinite if x' Ax > 0 for all real ( m x 1 ) vectors x ; negative definite if x' A x < 0 for all real ( m x 1 ) vectors x :f- 0; negative sem idefinite if x'Ax < 0 for all real ( m x 1 ) vectors x .
Without special notice, a real definite matrix is always understood to be symmetric. 9.12.1
General Properties
( 1 } A , B ( m x m) :
( a) A positive definite, B positive semidefinite => A + definite.
(b) A negative definite, B negative sem idefinite definite. (c) A , B positive sem i definite (d) A, B negative sem idefinite ( 2 } A ( m x m ) , c E ffi :
=>
=>
A + B negative semidefinite. cA is positive (sem i )
( b ) A negative (semi ) defi nite, c > 0 definite.
A positive (sem i ) definite, c < 0
=>
=>
( d ) A negative (semi) definite, c < 0 -¢:::::::>
(e) A positive (semi) definite (f) A =
[a;j] -¢:::::::>
(g) A =
[a;j]
A + B negative
A + B positive semidefinite.
( a) A posi tive (semi) definite, c > 0 definite.
(c)
=>
B positive
cA negative (semi ) cA negative (sem i ) definite. cA positive (sem i ) definite.
-A negative (sem i ) definite.
positive (semi) definite d et
> ( > ) 0 for i = 1 , . . . , m.
a;; negat ive (sem i ) definite
-¢:::::::>
a
II
.
..
a li
det
a;
1
for i = 1 , . . . , m. ( h ) A positive (sem i ) definite i = 1 , 2, . . .
a;; :::}
{
> ( > ) 0,
z
< (<) 0,
i odd
.
even
A' positive ( semi ) definite for
1 52
H A N DBOOK O F M ATRICES
( 3 ) A . B ( m x m)
:
( a) (Schur product theorem) A . B positive (semi ) defi n ite ( b ) A , B positive sem idefinite (4) A (m x m), B (n x n) : positive (sem i ) definite.
'*
'*
A (� ' B positive (sem i ) definite.
rk ( A C::> B ) < rk( A ) rk ( B ) .
A , B positive ( semi ) definite
'*
A
4
B
( 5 ) A ( m x m ) positive (sem i ) definite: ( a) det ( A ) > ( > ) 0 . ( b ) t r ( A ) > ( > ) 0. (6) A ( m x m) negative ( semi) definite :
tr( A) < ( < ) 0.
(7) A (m x m ) positive ( negative) definite: (a) r k ( A )
= m.
( b ) A is nonsingular. ( c ) .4 - t is positive ( negative) definite. (8) A ( m x m) Hermitian : A is idempotent '* A is positive semidefinite. ( 9 ) A = d i ag( a t J , . . . , amm ) : ( a) A positive (semi ) definite ( b ) A negative (sem i) definite ( 10 ) A ( m x m ) , B ( m
x
<::::=:}
<::::=:}
aii > ( > ) 0, i aii < ( < ) 0 , i '*
( b ) A positive ( negative) definite, rk( B ) ( negative) definite. ( 1 1 ) ( Fischer 's inequality) A (m x m). B (m x n), C (n
[ ABH B ] c
( 12 ) ( A itken's integral )
=
I , . . . , m.
1 , . . . , m.
n) :
( a) A positive ( negative) semidefini te ( n egative) semidefinite.
D=
=
x
n
=
B H A B positive '*
B H A B positive
n) :
positive definite
.4 ( m x m ) real positive definite, x
=
'*
det ( D ) < det ( A )det ( C ) .
(x 1 , . . . x m ) (m ,
'
x
1 ) real:
1 53
PROPERTIES O F S P EC I A L M ATRICES
Eigenvalue Results
9 . 1 2. 2
( 1 ) A ( m x m ) Hermitian (or real symmetri c ) : ( a ) A is positive ( semi ) definite positive ( nonnegative) .
¢::::::>
all eigenvalues of A are
( b ) A is negative (semi ) definite negative ( nonpositive ) .
¢::::::>
all eigenvalues of
( 2 ) A = [a;j ) ( m x m ) positive definite with eigenvalues A 1 n
n
i=l
i=l
<
···
A
are m
:
n = 1 , . . . , m.
II A ; < II a ;; ,
( 3 ) A ( m x m ) positive definite with eigenvalues A 1 < · · · < Am , X (m x n): n
n
II A ; < det( X H AX ) < II Am - n + i · i= I
( 4 ) A ( m x m ) real positive definite with eigenvalues ) q < Am , X ( m x n ) real : X' X = In
=>
<
n
n
II A ; < det ( X ' A X ) < II Am -n +i · i=l
( 5 ) A = [a;J) ( m x m ) positive definite with eigenvalues A 1 < · · · < Am n
''
i= I
i= I
II A; < det ( A ( n ) ) < II Am - n + i ·
(6)
m ) Hermitian (or real sym metric ) , B ( m x m ) posit ive semidefinite: Am in ( A + B) > Amin ( A ) ,
A (m
x
Am ax ( A + B) > Am a x ( A ) . ( 7 ) A ( m x m ) Hermitian (or real symmetric) with eigenvalues A 1 ( A ) < < · · < A m ( A ) , B ( m x m ) positive semidefinite, A 1 ( A + B ) < A m ( A + B ) eigenvalues of A + B : ·
>., ( A + B) > >.; ( A ) ,
i = l, .
. . , m.
! 54
(8)
H A N DBOOK O F M ATRICES A
( m x m ) positive definite with eigenvalues ) q { 1 , . . . , m} : ..\ 1 · ..\ 2 · · A ·
<
···
<
..\ 771 ,
n
E
n=
min { ( .r{f A.r l ) · · · ( x :; A xn ) : Xj ( m X 1 ) , X = [ x i · · . . , xn ] (m X ) x H x n
,
( 9 ) :1 (m x m ) positive definite with eigenvalues ..\ 1 < ..\ 2 < · · assoc iated orthonormal ( m x 1 ) eigenvectors t> 1 , . . . , Vm : ·
=
In } -
<
>.rn
aud
Illin{ det ( B H A B ) : B ( m X ) B H B = In } = ..\ 1 · ..\ 2 · · · An . n
,
The minimizing matrix is B = [ v1 , . . . , vn ) -
The maximizing matrix is B = [ v m . . . . , Vm - n + d · ( 10 ) A ( m x m ) real positive definite with eigenvalues ..\ 1 < ..\ 2 < · · · < and associated orthonormal ( m x 1 ) eigenvectors v 1 , . . . , Vm : min{ det ( B ' A B ) : B ( m x The minimizing matrix is B
=
n
)
real, B ' B
= In }
= ..\ 1 · ..\ 2 · · · A n .
[ v 1 , . . . , vn ] -
max{ det ( B ' A B ) : B ( m x ) real , B ' B = In } n
>. ,.
=
>. , . · Am - I · A m - n + 1 . ·
·
The Illaximizing matrix is B = [ vm , . . . , Vm - n + J 9. 12.3 (1)
Decomposition Theorems for Definite Matrices
A (m x m ) ( a) A is positive definite '* A is similar to Im . ( b ) A is positive semidefinite, rk ( A) -'* A IS. similar to :
[ �n 00 ] .
n
A is negative definite '* A is similar to - I771 • ( d ) A is negative semidefinite, rk (A ) = A I. S similar to '* (c)
(2)
[ -�n
A (m x m) :
�].
n
( a) ( Choleski decom position) A is positive definite ¢::::::> there ex ists a lower triangular ( m x m ) matrix B with real positive elements on the principal diagonal such that A = B B H .
P RO P E RTIES O F SPECIAL M ATRICES
1 55
( b ) ( Choleski decomposition of a real m atrix) A is real positive definite ¢::::::> there exists a real lower triangular ( m x m ) matrix B w i th positive elements on the p rincipal diagonal such that A = B B' . (c) (Square root decomposition) A is positive (semi ) definite ¢::::::> there exists a positive (semi) definite ( m x m) matrix B such that A = BB, that is , B is a square root of A . ( d ) ( Square root decomposition of a real matrix) A is real positive (semi) definite ¢::::::> there exists a real positive (semi) definite ( m x m ) matrix B such that A = B B , that is, B is a real square root of A . (e) A is positive definite ¢::::::> there exists a nonsingular ( m x m ) m atrix B s uch that A = B H B .
( f) A is real positive definite ¢::::::> there exists a real nonsingular ( m x m ) m atrix B such that A = B' B.
( 3 ) A ( m x m) : A is positive semidefin i te, rk ( A ) = r =} there exists an ( m x r) matrix B w i th rk ( B ) = r such that A = BB H and A+ = B ( B H B ) - 2 B H .
( 4 ) (Simultaneous diagonalization of a positive definite and a Hermitian matri x ) A ( m x m) positive definite, B ( m x m ) Hermitian : There exists a nonsingular ( m x m ) matrix T such that A = TT H and B = TAT H , w here A is a diagonal matrix. ( 5 ) (Simultaneous diagonalization of a positive definite and a symmetric matrix ) A ( m x m ) positive definite, B ( m x m ) symmetric: There exists a nonsingular ( m x m ) matrix T such that A = TT H and B = TAT' , where A is a real diagonal matrix with nonnegative diagonal elenwnts.
Note: M any results on positive and negative (semi ) definite matrices including proofs can be found in Horn & Johnson ( I 985, Chapter 7 ) and other books on matrices. See also Chap ter 4 for results related to traces and determinants of defin i te matrices. For Aitken 's integral see Searle ( 1 982, p . 340) and for t h e decomposition results see Chapter 6. Since positive ( negati ve) definite and semidefini te matrices are Hermitian , all results for the l atter also hold for the former matrices (see Section 9 . 7 ) .
H A N DBOOK O F M ATRICES
1 56
9.13
Symmetric Matrices
Definitions: An (m x m ) m atrix A is symmetric if A' = A , that is, the ijt h element a ;j i s equal to the j ith element, aji = a ;j , so that
a1 1 a1 2
A=
a1m a2m
a12 a 22
An ( m x m) m atrix A is skew-symmetric if A' = - A , that is, the ijth element a;j is - I times the jith element, a ;j = - a ;j , so that 0 -a12
A=
a 1m a 2m
a12 0
- a i m - a 2m
9.13.1
0
General P roperties
( 1 ) A ( m x m) real:
A is symmetric
( 2 ) A , B ( m x m) sym metric, c E (I; :
¢::::::>
A is Hermitian .
(a) cA is sym metric.
(b) A ± B is symmetric. ( c ) .4 8 = BA ��
:::}
A B is symmetric .
B is symmetric. ( 3 ) A ( m x m ) sym metri c , B ( n x n ) symmetric: (d) A
(a) A 0 B is symmetric.
(b)
A if' B is symmetric.
( 4) A = [ a ;j) (m x m) sy m metric: ( a ) A' is sym metric.
(b)
A H is sym metric . ( c ) A - ! is sym metric , if A is nonsingular.
( d ) A a dj is sym metri c . ( e ) A' i s sym n 1 etric for i = 1 , 2 , . . . (f)
A ..
is symmetric .
PROP ERTIES O F S P E C I A L M ATRICES
157
(g) I A iats is sym metric .
( 5 ) A (m x m ) :
...
1 , 2 , . . . , m.
=
is symmetric for i
(h) a;;
(a) A + A ' is sym metric . ( b ) A - A ' i s skew-symmetric. (c) A A' is sym metric. ( d ) A' A is symmetric. ( 6 ) A (m x m) : {:::=}
( a) A is symmetric m atrices B .
B'AB is sym metric for all (m x m)
( b ) A is symmetric {:::=} there exists a real orthogonal ( m x m) matrix Q and a real diagonal ( m x m) m atrix A such that A = QAQ' . ( 7) A ( m x m ) , B ( m x n ) :
A is sym metric
( 8 ) A, B ( rn x m) real sym metric: (a) [tr( A B )F < tr ( A 2 B 2 ) .
( b ) rk(A) >
B' A B is symmetric.
( tr A )2 if tr( A2 ) =f. 0 . , tr ( A 2 )
( 9 ) A ( m x m ) real , b ( m x I ) real ,
B=
'*
[ b b'A ] 1. s symmetnc.
c
c
E IR : '*
det( B)
=
c
det ( A ) - b'Aa dj b
( 10) A ( m x m) , B ( n x n ) , C ( m x n) :
A , B are symmetric
'*
[ �� � ]
is symmetric.
Note: M any of these resu lts are elementary. Most others can be found in Horn & Johnson ( 1985 ) . More rules for sym metric matrices are given in t he section on positive defi nite and semidefinite matrices which are symmetric by definition if they are real (see Section 9 . 1 2 ) . 9.13.2
Symmetry and Duplication Matrices
( 1 ) A ( m x m) sym metric ,
Dm ( m 2 x � m( m + 1 ) ) duplication matrix:
1 58
H A N D BOOK O F M ATRICES
(a) vec(A) = Dm vech ( A ) . ( b ) D;t",vec(A) = vech ( A ) . ( 2 ) A ( m x m) sym metric nonsingular, c E (I; : ( a) det ( D;t". [A 0 A + c vec(A)vec( A )'] Dm ) = ( 1 + cm)det ( A ) m + l . ( b ) ( D;t", [A 0 A + c vec (A )vec ( A )'] Dm ) - 1 = D + A - 1 0 A - 1 - c vec(A - 1 ) vec( A - 1 )' Dm · m
[
1 + em
( 3 ) A , B ( m x m) symmetric, a , b (m x 1 ) , a , (3 E (I; :
( [ a a' ] [ /3 b' ] ) Dm + l = (a) Dm + l a A b B ,
]
0
ab' + f]a' a;J (a' 0 b' ) Dm ab + ;Ja a B + f]A + ab' + ba' (a' 0 B + b' 0 A ) Dm D:-r, ( A 0 B)Dm D:-r, (a 0 b) D:n (a 0 B + b 0 A )
'
hab' + f]a' ) (a' 0 b') D;t", a/3 � (ab + f]a ) � (aB + f]A + ab' + ba' ) � ( a' 0 B + b' 0 A ) D;t' D;t", ( A 0 B) D;t' D;t". ( a 0 b) � D;t", (a 0 B + b 0 A ) Note: These results are given in Magnus ( 1988) (see also Section 9 . 5 ) . 9.13.3
Eigenvalues of Symmetric Matrices
denotes an eigenvalue of the matrix A . ..\(A) >. m in ( A ) is the smallest eigenvalue of the matrix A . >. m a.r ( A ) is the largest eigenvalue of the matrix A . ( 1 ) A ( m x m ) real symmetric: (a) All eigenvalues of A are real numbers. (b) A is positive definite ¢::::::> all eigenvalues of A are real and greater than 0 . (c) A positive semidefinite ¢::::::> all eigenvalues of A are real and greater than or equal to 0 . ( 2 ) A ( m x m) real symmetric with eigenvalues ..\ 1 , . . . , >.m and associated orthonormal eigenvectors V 1 , . . . , Vm : m A = 2: ..\; v; v: .
Notation:
i=I
159
P RO P E RTIES O F S P ECI AL M ATRICES
( 3 ) ( Rayleigh-Ritz theorem) A ( m x m) real symmetri c : Am;,. ( A ) Amax ( A)
Ax { = min x' x { x'xAxx = m ax x
'
( 4 ) (Courant-Fischer theorem) A (m x m ) real symmetric w ith eigenvalues ..\ 1
A; =
Yt
(m X
x
1 I
Ym-
I
1)
and ..\ ; =
11 1
(m X
l
.
··
<
Am ,
real, x # 0, x'Yj = 0 , j
=
1 . . ., m ,
-
x'Ax { min : xx
max
. . .
·
#0 .
'X
)real
(m x
<
} }
#0 ,
x' Ax { : m ax x
m in
•
x
: x ( m x 1 ) real,
'
1
x
x ( m x 1 ) real,
Y, - t
'
)rea)
x ( m x 1 ) real, x # 0, x' YJ = 0 , j = 1 , .
..,i
-
1
i} }.
( 5 ) A (m x m) real symmetric with eigenvalues ..\ 1 < · · · < A m and associated orthonormal eigenvectors v 1 , . . , Vm , n E { 1 , . , m } : .
min{ tr( X ' A X ) : X
(m x n )
(m x n)
. . , v,. .
]
real , X' X = I,. } = A m + · · · + A m - n + l ·
The maximizing matrix is X = [ vm , . . . , Vm - n + d · Note: For these results see Chapter 5 . 9 . 1 3 .4 (1) A
Eigenvalue Inequalities
(m x m ) real symmetric, x # 0 ( m Am;n ( A )
.
real , X ' X = I,. } = ..\ 1 + · · · + ..\,. .
The m i n imizing matrix is X = [ v 1 , . max{ tr(X ' A X ) : X
.
<
x' Ax x' x
--
x
1 ) real:
< Amax ( A ) .
1 60
H A N DBOOK O F M ATRICES
( 2 } A ( m x m ) real symmetric with eigenvalues ) q ( A ) , . . . , >. m ( A ) :
. { x' A x
mm
:
'
x x
x ( m x 1 ) real ,
{ x'Ax,
< ..\; ( .4 ) < max i
=
X X
x
( m x 1 ) real ,
1 , . . . , m.
(m
x m ) real symmetric with eigenvalues ..\ 1 ( A ) Am ( A ) , X ( m x n ) real:
(3) A
n
L ..\; ( A ) < tr ( X' A X )
X ' X = In (4) A =
[a;j] ( m
i=l
[a ;j] ( m
Am ( A )
:
n
L ..\; ( A ) < L a;; i= I
i=I
[a;j] ( m x m )
>.rn ( A ) :
· <
n
L Am-n+i( A ) . i=I
i
=
1 , . . . , m.
x m ) real symmetric with eigenvalues ..\ 1 ( A ) <
n
(6) A =
· .
x m ) real symmetric:
Amin ( A ) < a;; < >.max ( A ) , (5) A =
<
<
n
L Am - n +; ( A )
<
i= 1
<
n = 1 , . . , m. .
,
real symmetric with eigenvalues 0 < ..\ 1 ( A ) <
n
n
II >., ( A ) < II a; ; , •= I
i=l
n
=
1,
.. .
.
·
·
·
<
m.
( 7 ) ( Inclusion princip le ) A ( m x m ) real symmetric with eigenvalues ..\ 1 ( A) < · < >.m ( A ) , A( n l ( n x n ) principal submatrix of A with eigenvalues ..\ i ( A r n d < · · · < A n ( A ( n ) ) : ·
·
Amin ( A ) < Amin ( A( n j ) < Ama.r ( A ( n ) ) < Amax ( A ) . (8) A (m x
rn
) real symmetric,
B ( m x m)
>.m i n ( A + B) Ama.r ( A
real p ositive semidefinite:
> Ami n ( A ) ,
+ B) > Amax ( A ) .
161
P RO P E RTIES O F S PECIAL M ATRICES ( 9 ) A, B ( m x m) real symmetric:
Amon ( A + B ) > Amin ( A ) + Amin ( B ) , Amax ( A + B) < Amax ( A ) + Amar ( B ) . ( 10 ) A, B ( m x m) real symmetric, 0 < a, b E IR : >. min ( a A + b B ) > a>. min ( A ) + b >.m;n ( B ) , Amax ( a A + bB) < a>. max ( A ) + b >. max ( B ) . ( 1 1 ) A ( m x m) real symmetric with eigenvalues >. ! ( A ) < · · · < < >.m ( A ) , B ( m x m) real positive semidefinite, >. ! ( A + B ) < Am ( A + B ) eigenvalues of A + B : >.; ( A + B ) > >.; ( A ) ,
i = 1 . . . , m. ,
< ( 12 ) A ( m x m) real symmetric with eigenvalues >. ! ( A ) < >.m ( A ) , x ( m x 1 ) real, .>.1 ( A ± xx' ) < · · · < >.m ( A ± x.r ' ) eigenvalues of A ± x x ' : ·
>.; ( A ± x x ' ) < >.;+ I ( A) < >.; + 2 ( A ±
.r .r
'
),
>.; ( A ) < >. i + ! (A ± xx ' ) < >.;+ 2 ( A ),
·
·
i = 1 , 2, . . . , m - 2;
i = l , 2, . . , m - 2.
( 13 ) A , B ( m x m ) real symmetric, rk ( B ) < r·, >.! ( A) < · · < >.m ( A ) eigenvalues of A , >.! ( A + B ) < · · · < >.m ( A + B ) eigenvalues of A + B : ·
>.; ( A + B ) < >.;+ r ( A ) < >.;+ 2r ( A + B ) , >., ( A ) < >. i +r ( A + B ) < .>., + 2 r ( A ) ,
i i
=
= 1 , 2 . . . . , m - 2r; 1 , 2,
. . . , m -
2r.
( 1 4 ) ( Poincare's separation theorem ) A ( m x m) real symmetric w ith eigenvalues .>.! ( .4 ) < · · · < Am ( A ) , X ( m x n ) real such that X ' X = In , Tl < m , >. I ( X ' A X ) < · · · < An ( X' A X ) eigenvalues of X' AX :
>.; ( A ) < A;( X ' A X ) < Am - n + ; ( A ) .
i = 1 , . . . , n.
( 1 5 ) A , B ( m x m ) real symmetric with eigenvalues >. ! ( A ) < · · · < >.m ( A ) and >. 1 (B ) < · · < Am( B ) , respectively; >. ! ( A + B ) < · · · < >. ( A + B ) eigenvalues of A + B : n n ( a ) �:_) >.; ( A ) + >.; ( B)] < L >.; ( A + B ) , n = 1 , . . . , m . m
·
i= I
(b) ..\ 1 ( B ) <
i= I
An ( .4 + B ) - An ( A ) < Am ( B ) ,
n = 1 , . . . , m.
162
H A N DB OOK O F M ATRICES
( c) I .:X n ( A + B) - An ( A ) Iabs < max{ l .:\j ( B ) I abs n = 1,
.
.
:
. , m.
j = 1, .
( d ) A n ( A + B ) < min{.:\; ( A ) + Aj ( B ) : i + j = n + n = 1 , . . , m.
.
.
, m
},
},
m
.
( 16 ) ( Weyl 's theorem ) A , B ( m x m ) real symmetric with eigenvalues .X 1 ( A ) < · < Am ( A ) and A 1 ( B ) < · · < Am ( B ) , respectively; .:\ J ( A + B) < · < Am ( A + B ) eigenvalues of A + B : ·
·
·
.:\; ( A + B) >
and
·
.X; ( A ) + .:\ 1 ( B ) Ai - I ( A) + .:\ 2 ( B )
.X i ( A ) + .Xm ( B ) A i + I ( A ) + Am _ I ( B )
.:\; ( A + B) < i
·
Am ( A ) + .X; ( B )
1 , . , m. ( 1 7 ) A, B ( m x m ) real sym metric with eigenvalues .:\ 1 ( A ) < · · < Am ( A ) =
.
.
·
and .:\ I ( B ) < · · · < Am ( B ) , respectively; A 1 ( A - B) < eigenvalues of A - B :
.X J ( A - B) > O
=>
·
.:\, ( A ) > .:\ ; ( B ) , i = l ,
·
.
·
.
< Am ( A - B )
.
, m.
( 1 8 ) A ( m x m ) real symmetric with eigenvalues .:\ J ( A ) < Am ( A) , .r ( m x 1 ) real , c E IR. ,
B=
<
X ] [ x' A
c
with eigenvalues .X I ( B) < · < Am+ ! ( B) : ·
·
.X; ( B ) < .X; ( A ) < A;+ 1 ( B) ,
i = 1, .
. . , m.
( 19 ) A ( m x m ) real sy mmetric with eigenvalues .:\ 1 (A ) < A m ( A ) , D (p x p ) real sym metric , C ( m x p) real and
B=
[ �1 � ] ( n X n)
with eigenvalues .:\ 1 ( B ) < · · · < An ( B) :
.:\; ( B ) < .:\; ( A ) < An-m+i ( B ) Note: For these results see Chapter 5 .
for i = 1 , .
.
. , m.
<
163
PROPERTIES O F SPECIAL M ATRICES
9.13.5
Decompositions of Symmetric and Skew-Symmetric M atrices
( 1 ) (Spectral decomp osition ) A ( m x m ) real symmetric with eigenvalues A 1 , . . . , Am : A = Q AQ', w here A = d i ag(A 1 , . . . , Am ) and Q is the real orthogonal ( m x m ) matrix w hose columns are the orthonormal eigenvectors v 1 , . . . , Vm of A associated with A 1 , . . . , Am . I n other words, A =
m
L A ; v; v; . i=l
( 2 ) A ( m x m ) symmetric, rk ( A ) = ( m x m ) matrix T such that A =T
[
r
lr
O
There exists a nonsingular
� ] T' ,
where the zero submatrices disappear if r = m. ( 3 ) A ( m x m ) real skew-symmetric: ( m x m ) matrix Q such t hat
There exists a real orthogonal
0
0 0
A=Q
Q',
0
where the A; are real (2 x 2 ) matrices of the form b; 0
]
.
( 4 ) ( Simultaneous d i agonalization of a real positive definite and a real symmetric matrix) A ( m x m ) real p ositive definite, B ( m x m ) real symmetric: There exists a real nonsingular ( m x m ) matrix T such that A = TT' and B = TAT' , w here A is a real diagonal matrix . Note: For more details o n matrix decomposition results see Chapter 6 .
H A N DBOOK OF M ATRICES
164
9.14
Triangular Matrices
Definitions: A n ( m x m) matrix aI I
0
0
= [aij ] ,
A= ami with Uij
=
0 amm
...
0 for j > i , is a lower triangular matrix . A n ( m x m) matrix ... .
. .t
-
0 0
0
Umm
w ith a;j = 0 for i > j, is upper triangular . A is triangular if it is upper t riangular or lower triangular. A is strictly upper ( lower) triangular if a;j = 0 for i > j ( i < j ) , that is , all elements on the principal diagonal in addition to the elements below or above that diagonal are zero. A is strictly triangular if it is strictly upper or strictly lower triangular. 9.14.1
Properties of General Triangular Matrices
( 1 ) A , B ( m x m) upper ( lower) triangular, c E
(b)
c
A is upper ( lower) triangular.
( c ) A B is upper ( lower) triangular. ( d ) A (�) B is upper ( lower) triangul ar. ( 2 ) A (m x m ) upper ( lower) triangular, B ( n x n ) upper ( l ower ) triangul ar:
( a) A (b) A
r:9
cl•
B is upper ( lower) triangular. B is upper ( lower) triangular.
( 3 ) A ( m x m) upper (lower) triangular: ( a) A' is lower ( upper) triangul ar. ( b ) A H is lower (upper) triangular . ( c ) A is upper (lower) triangular. ( d ) A - 1 is upper ( lower) triangular, if A is nonsingular.
1 65
P RO P E RTIES O F SPECI A L M ATRICES
(e) A a d] is upper ( lower) triangular. (4)
A = [ a;; ] ( (a) (b)
m x m
) triangular:
m t r ( A ) = a 1 1 + · · · + a mm = L a;; . i=l m d et ( A ) = a 1 1 · · · amm = II a;; . i=l
( c ) rk( A ) > num ber of nonzero diagonal elements. (d) a 1 1 , . .
.
, am m
are the eigenvalues of A.
(e) A is symmetric => (f) A is Hermitian =>
A = d iag( a 1 1 , . . . , amm ) · A is real diagonal .
Note: Most of these results are simple consequences of t he definitions. See Magnus ( 1 988) for more details. 9.14.2 (1) (2)
Triangularity, Elimination and Duplication Matrices
A( ): A = [a 1 ] , B (a) D;!-, (A
A lower triangular => L'm vech( A )
m x m
= vec ( A ) .
[b;; ] ( m x m ) upper ( lower) triangular: 0 B ) Dm is upper ( lower) triangular with diagonal elements � ( a ;;bj] + aj jb;; ) , 1 < j < i < m . ;
(b)
=
det [ D;!", ( A 0 B ) Dm ] = � IT (a; ; bjj + ajj b;; ) i>j
(c) L m ( A 0 B)L'm is upper (lower) triangular wit h diagonal elements a ;,bjj , 1 < i < j < m .
m
i= 1
(e) L m ( A 0 B ) Dm is upper ( lower) triangular w i t h diagonal elements a;; b]J , 1 < i < j < m .
m
(3)
A = [a ; 1 ]
i= I ,
B = [b ;;] ( m
x m
) lower triangular:
m
i= I
( b ) a ;;b;i • 1 < i < j < ( c ) [L ( A m
'
0
m,
are the eigenvalues of
B ) L �, ] ' = Lm [( A ' ) ' 0 B ' ] L 'r, for
L m ( A' 0 B) L:71 •
H A N DBOOK O F M ATRICF:S
166 s
0 , 1 , 2, . . . . . - 2, - 1 , if A , B nonsingular
s
.
.!.
if lower triangular A 1 12 B112 exist , 2 (d) det[D;!; ( A ® A ± B ® B ) Dm ] = II (a;;aii ± b;;bj j ) . i>j (e) det [Lm ( A B' 0 B' A ) L :r,] = det[Lm ( A B' 0 A' B ) L:r,] = (de t A ) m + 1 ( de t B ) m + 1 . ( f ) L :r, Lm ( A' ® B ) L :r, = (A' ® B ) L :r, . s
,
( 4 ) A = [a;i ] , B = [b,i ] , C = [c;j ) , D = [d;i J ( m x m) lower triangular : m m det [Lm ( A B' @ C' D) L:n J = II (c;;d;d (a;;b; ; ) -i+ l , i= I
det[Lm ( A' @ B + C ' @ D)L:nJ =
II (b ;;aj j + d;; cj j ) ·
i>j ( 5 ) A, B , C, D (m x m) nonsingular, lower triangular:
[L m ( A B' 0 C' D)L:n J - 1 = Lm [ ( B' ) - 1 ® D- 1 ) L:n L m [A - I ® ( C ' ) - 1 ] L:r, . ( 6 ) A = [a ;j ] ( m x m ) lower triangular, n E det
Lm
n-1
L ( A' t - l - j ® Aj
j =O
where
J.l k l =
{
IN :
L'm
(ak k - a;; ) / ( a u - a l l ) n a nk k- 1
= n m ( det A t - l
II J.l k l ,
k>l
if a u =/:- a ll
if a u = a ll ( 7) A = [a;i ] , B = [b ;j ] , C = [c;j ] ( m x m) lower triangular: ( a) Lm ( A 0 B ' ) Dm Lm ( A ® B' ) L :r, . ( b ) det[Lm ( A B 0 B' A ) Dm ] = det [L m ( AB' 0 A' B' ) Dm ] = det[Lm ( A B' ® B' A ) L:r, ] = ( de t A ) m + l ( de t B ) m + 1 . m ( c ) det [Lm ( A ® B'C) Dm ] = II (b;;c; d a7,' - i + 1 • i=l m m (d ) det [L m (A B' ® C' ) Dm ] = II c:; (a;;b;; ) - i + l . =
i=l
Note: The foregoing results may b e found in M agnus ( 1 988) which is a rich source of results for t ri angular matrices and their relationship to d uplication and elimination matrices.
P RO P E RTIES O F SPECI A L M AT RICES
9.14.3
167
Properties of S trictly Triangular Matrices
( 1 ) A ( m x m) strictly triangular: ( a) A is si ngular. ( b ) A m = Om x m · ( c ) tr( A ) = 0 . ( d ) det ( A ) = 0 . ( 2 ) A , B ( m x m ) strictly upper ( lower) triangular, c E
( b ) cA is strictly upper ( lower) triangul ar. (c) A B is strictly upper ( lower) triangular. (d) A
0
B is strictly upper ( lower) triangular.
(e) Lm ( A' 0 B ) L'm is nilpotent.
( 3 ) A ( m x m ) strictly upper ( lower) triangular, B (n x n ) strictly upper ( lower) triangular: (a) A 0 B is strictly upper ( lower) triangular. (b) A
EEl
B is stri ctly upper (lower) triangular.
( 4 ) A ( m x m ) strictly upper (lower) triangul ar:
( a) A' is strictly lower ( upper) triangular. ( b ) A H is strictly lower ( upper) triangular.
( c ) A is strictly upper ( lower) triangular. ( d ) A a dJ is strictly upper ( lower) triangular. Note: These results are easy implications of t he defi nitions. 9.15
Unitary Matrices
Definition: A n ( m x m ) matrix A is unitary if it IS nonsingular and A - I = AH . ( 1 ) A ( m x m) :
( a) A is unitary -<==> the colu m ns of A are orthonormal vectors.
( b ) A is unitary -<==> t he rows of A are orthonormal vectors. ( c ) A is u n itary -<==> x H A H Ay = x H y for all ( m x l ) vectors x , y. ( 2 ) A ( m x m ) : A is unitary -<==> A A H = A H A = Im .
H A N DBOOK OF M AT RICES
1 68 -¢::=>
¢::::::}
¢::::::}
¢::::::}
¢::::::}
( 3 ) A ( m x m) real :
A' is unitary. A H is unitary. A is unitary. A - l is unitary. A' is unitary for i = I , 2, . . . ¢::::::}
A is orthogonal
( 4 ) A ( m x m) unitary: ( a) jdet( A ) Iabs = ( b ) r k ( A ) = m.
A is unitary.
l.
( c ) A is norm al , that is, A H A = A A H . ( d ) A is eigenvalue of A => j .A ! ab = l . s
1 ) => I ! A x l ! 2 = ! ! x l l 2 · (f) B ( n x m ) => I ! B A I ! 2 = I ! B I I 2 ·
(e)
x
(m x
(g) B ( m x n ) => ! I A B I ! 2 = I ! B I I 2 ·
( 5 ) A; ( m x m ) , i = 1 , 2 , . . . : A; is u n itary for i = 1 , 2 , . . . and Em A; exists => Em A ; is unitary. s - oo
( 6 ) A , B ( m x m) :
A , B are unitary
1 - oo
=> A B is unitary.
( 7) A ( m x m) unitary, B ( n x n) unitary : (a) A ® B is unitary. (b) A
ffi
B is unitary.
( 8 ) A ( m x m) with eigenvalues A J , . . . , A m : A is normal, j .A; l ab s = 1 , i = 1 , . . . , m ¢::::::} A is unitary.
(9)
(a) The (m x m ) unit m atrix Im is unitary. ( b ) The ( mn x mn ) commutation matrix l\mn is unitary.
[ cossin BB
].
( c ) A permutation m atrix is unitary. ( d ) For () E IR ,
-sin B cos B
IS
. umtary.
(e) ( Householder transformatio n ) 2xx H . . s umtary. For x ( m x 1 ) , lm xHx 1 ( 10) A , B ( m x m) : ( a) A is similar to a unitary matrix => A - 1 is similar to A H .
( b ) A , B are unitarily equivalent values.
-¢::=>
A , B have the same singular
1 69
P ROPERTIES O F S PECI A L M ATRICES
( c ) A , B are unitarily equivalent
{::=:}
A H A and B H B are similar.
( 1 1 ) A ( m x m) unitary : There exists a real orthogonal ( m x m) m atrix Q and a real sym metric ( m x m ) matrix S such that A = Qexp(i5). Note: The results of this subsection follow either directly from the definitions or c an be found in Horn & J ohnson ( 1 985 ) . A number of results on unitary m atrices are also given in Lancaster & Tismenetsky ( 1 985).
10 Vector and Matrix Derivatives In this chapter all vectors and matrices are assumed to be real unless otherwise stated . Differentiable always means continuously differentiable of sufficiently high order so that all expressions are well-defined .
10.1
Notation
f( ( x)
(
Let be a differentiable real valued function of the real m x 1 ) vector X = X1 , . . . , X m )' .
fO X)( x)
of
8
of ax of OX
OX )
-
or
af(x) ax fOX( X ) {)
m
(m
x 1)
m
g;, ( :� . . . ' :�) or a���) ( a£;�) , . . . , a;;:) ) ( 1 m ) are vectors o f first order partial derivatives. f( x)/ ax is sometimes called the gradient vector of f ( x). and
=
=
'
x
{)
oQf X and
of ax
Xo
of' ax
xo
x=x o
of'l f' x
u
- afa(xxo )
x =x o
=
=
(m
aff'l(x'o ) - [af(f'l xo )] ' ( 1 x x =
u
u
x 1)
x m)
H A N DBOOK O F M ATRICES
1 7"2
are vectors of first order partial derivatives evaluated at the ( m x 1 ) vector xa .
[)2 f
{)x{)x '
=
[) 2 f( X ) {)x{)x'
=
[
[)2 f( X ) ax; aXj
[) 2 f
]
82 !
ax l ax )
.
[) 2 f OX m OX t
.
.
.
8x t 8Xm
.
.
-
(m x m)
[)2 J OXmOXm
is the Hessian matrix of second order partial derivativeo. of /( x ) and
.r o
=
82/( xo ) axox'
=
[
02/
ax 1· ox].
x=xo
]
(m x m)
is the Hessian matrix of second order partial derivatives of f(x) evaluated at :r = xa . Let f( X ) be a differentiable real valued function o f the real ( m x n ) matrix X = [ :r ,i l ·
af ax
af( X ) ax
(m x n)
at
of
is t he II I atrix of first order partial derivatives of /( X ) and a!
(m x n)
{)X
is the matri x of first order partial derivati ves of f( X ) evaluated at the ( m x n ) mat ri x X o . avec( X ) avec ( X ) '
avec( X )avec ( X )'
( mn x mn )
is the Hessian matrix of second order partial derivatives of f( X ) . Here it is i n 1 p ortant to note the order in which the partial derivatives are arranged . They have the same order as for the function f( vee( X ) ) . Accordingly, avec ( X ) a vec ( X ) I X = X o
is the Hessian mat rix evaluated at X = Xo .
avec ( X )avec ( X )'
1 73
V ECTOR A N D M ATRIX DERIVATI VES
Let y( x ) = [y1 ( x ) , . . . , Yn ( x )] ' be a real ( n x 1 ) vector of differentiable functions of the real ( m x 1 ) vector x ( x 1 , . . . , .r m ) that is, y is a function mapping a subset of m.m on a subset of IRn . =
and
,
OY 1
oy1 oy o.r '
'
oy( x ) ox'
( n x m)
( )
, oy (m x n) ox' OX OX are matrices of first order partial derivatives of y(x ) . oyfox' is sometimes called the Jacobian matrix of y and oy' /ox is sometimes called the gradient of y. oy'
oy ox '
r = .r o
oy ox '
oy( x ) '
-
oy(xo ) ox'
xo
..
.
is the J acobian m atrix evaluated at x = x0 and its transpose is the gradient of y evaluated at .r o . For m n , =
det
( 0���) )
= det
( ::, )
is the Jacobian or J acobian determinant of y( x ) . The Hessian matrix of y( x ) is ovec(oyfox ' ) ox ' m.mxn , For a matrix valued function of a single variable, A : S C IR x ...... A(x) (a ;j (x ) ] , the matrix of first order derivatives is �
=
dA(x) d .r
dA dx
-
d a mI d .r and the corresponding matrix of first order derivatives evaluated at .ro is d A (xo ) dx
dA dx
x = xo
H A N DB O O K O F M ATRICES
1 74
In the remainder of this chapter all functions are assumed to be differentiable.
Gradients and Hessian Matrices of Real Valued Functio ns wit h Vector Arguments
10.2
10.2. 1
Gradients 1R
( 1 ) x (m x 1 ), c E
oc
constant:
OX
= Om x l ·
( 2 ) ( Linearity ) x ( m x 1 ) , l ( x ) , g(x) real valued functions, c 1 , c2 E IR :
o [c. J( x ) + c2 g(x)] c ol( x ) = 1 ox ox
og( x) + c2 ox ·
( 3 ) ( P roduct rule) x ( m x 1 ) , 1 ( x ) , g ( x ) real valued functions:
a1 ( x ) g ( x)
= I( X )
OX
og ( x ) ax
+ g(X )
o1 ( x ) ox
( 4 ) x ( m x 1 ) , 1( x ) , g( x) , h ( x) real valued functions:
ol( x ) g( x)h ( x ) ox
=
o h ( x)
l(x )g( x)
OX
+ l( x)h ( x )
og ( x ) ( x) ol( x ) . x)h + g ( ax ox
( 5 ) ( Ratio rule) x ( m x 1 ) , l( x ), g ( x ) :/; 0 real valued functions:
[
]
o [l( x )f g( x) ] 1 x) . ol( x) _ o ( g = x ) I( x ox g( x) 2 g ( ) o x ox ( 6 ) (Chai n rule) x ( m x 1 ) , y( x ) ( n x 1 ) , I( Y) a real valued function :
ol ( y(x )) ax (7) x, a (m x 1) :
oy( x )' oi (Y) o x oy ·
Oa1 X
a. = x o
( 8 ) X ( m X 1 ) , a , y( X ) ( n X 1 ) (9) x (m x 1 ) , A (m x m) :
:
Oa1 y( X ) OX
=
o y( X)1 a. OX
ox ' Ax A + A )x . ( = ox I
1 75
V ECTOR A N D M ATRIX D E RI VATIV ES
( 10 ) x ( m x 1 ) , a ( n x 1 ) , A ( n x m ) , B ( n x n ) symmetric:
�
( a - Ax) (a - Ax)
= -2A ' B( a - Ax).
( 1 1 ) X ( m X 1 ) , y( X ) ( n X 1 ) , A ( n X n ) :
ay( x )' Ay(x) ay(x)' (A + A' ) y( X ) . ax ax x 1 ) , y(x) ( n x 1 ) , z(x) (p x 1 ) , A ( n x p) : ay( x )' A z(x) ay( x)' az( x ) ' , AZ( X ) + A y( X ) . = ax ax ax _
( 1 2) x ( m
Note: ( 1 ) - ( 6 ) are standard rules of calculus for functions with vector arguments. For the chain rule see, e.g., M agnus & Neudecker ( 1 988, Chapter 5 , Sec. 1 2 ) . ( 7 ) and ( 8 ) follow from ( 2 ) . ( 9 ) is obtained from stan dard rules for derivatives by writing x ' Ax i n summation notation and computing the derivative for each individual component of x . ( 1 0 ) and ( 1 1 ) follow from ( 9 ) and the chain rule ( 6 ) . ( 1 2) i s a consequence of the product and chain rules. 10.2.2 (1) ( 2) (3)
Hessian Matrices
a2a' x = Om x m · X, a ( m X 1 ) : axax ' 2 x ' Ax a X ( m X 1 ) , A ( m X m) : = A + A' . ax ax' x ( m x 1 ) , a ( n x 1 ) , A ( n x m ) , B ( n x n ) symmetric : a2 (a - Ax)' B(a - Ax) = 2 A' B .4 . uxux ' x ( m x 1 ) , y( x) ( n x 1 ) , A ( n x n ) : ay( x ) a2 y(x )' Ay( x ) ay(x)' ' (A + A ) axax ' ax ax' a + [y (x ) ' ( A + A' ) 0 Im ] a ' ve e n
(4)
n
( 5 ) X ( m X 1 ) , y( X ) ( n X 1 ) , z ( X ) (p X 1 ) , A ( n X p)
a2 y(x) ' Az(x) ax ax '
� [ ( y�; ) ] . :
ay(x) ' az( x ) az( x )' ' ay( x ) A A + ax ax' ax' ax a ay( X ) + [z(x) A 0 Im l vee ax ax' a )' + [y(x )' A 0 Im ] a ' vee 1
1
)'
[ ( 1)] � [ ( �; ) ] .
H A N DBOOK O F M ATRICES
1 76
Note: A l l resu lts are simple consequences of the resu lts in the previous subsection and the rules for vector valued functions with vector arguments given in the fol lowing sections.
Derivatives of Real Valued Functions with Mat rix Arguments
10.3
1 0.3. 1 (1)
\
.
General and Miscellaneous Rules
ac = Om x n · aX
( m x n ) , c E IR constant :
( 2 ) ( Linearity ) X ( m x n ) , f( X ) , g ( X ) real val ued fu nctions, c 1 , cz E IR : oh f( X ) + c2 g ( X )] {)X
.\ ) = CJ af( aX
o g(X ) + c2 aX .
( 3 ) ( Product rule) X (m
x
n ) , f( X ) , g ( X ) real valued functions: af( X )g ( X l ax
1
=
( .Y )
af( X ) ag(X ) + ( Y) Y ax ax ·
( 4 ) .\ ( m x n ) , f( X ) , g ( .\ ) , h ( X ) real valued functions: 8/( X ) g ( X ) h ( X ) ax
( 5 ) ( Ratio rule) X ( m x n ) , f( X ) , g ( X ) -:/; 0 real valued functions: a [ f( X ) / g ( X ) ] ax
=
1
g( X )2
[
9
(X)
af( X ) ax
_
ag( X ) f( X ) ax
( 6 ) ( C h ai n rule) .\ ( m x 11 ) , y = /( X ) , g ( y ) real valued functions: og ( f ( X ) ) ax
_
dg( y ) af( X ) ax · dy
( 7 ) .\ ( m x n ) , f( X ) a real valued function : vee
(
af( X ) aX
)
=
af( X ) avec ( X ) .
]
.
V ECTOR A N D M ATRIX DERIVATI V ES
1 77
(8) X (m x n), a (m x 1 ) , b (n x 1 ) : aa' X b
ax
( 9 ) X ( m x m) , a , b ( m x 1 ) : 0�:�;
( 10 )
X
b
= ab' .
= ) : ( Xi ) ' ab ' ( x i - 1 -i ) ' ,
i
) =0
= 1 , 2, . . .
( m x m ) P.onsingular, a , b ( m x 1 ) :
aa
X 1b
�
:
.\
= - ( X - 1 ) ' ab' ( X - 1 ) ' .
( 1 1 ) X (m x n). a, b (n x 1) :
aa' X ' X b
(12) X (m x n), a, b ( m x 1 ) :
{)X
aa' XX ' b {)X
=
V ( ba l + a b' ) .
....�
b
=( a +a I
b' ) .
.\ .
( 1 3 ) X ( m x m ) symmetric with simple eigenvalue _\ and corresponding eigenvec tor v satisfying v ' v = 1 : {)
_\ -= aX
I
vv .
Note: ( 1 ) - (7) are standard results from calculus. ( 9 ) and ( 1 0 ) are given in M agnus & N eudecker ( 1 988, Chapter 9, Sec . 1 3 ) and ( 8 ) is a special c asr of ( 9 ) . ( 1 1 ) and ( 1 2) follow from results given in M agnus & Neudecker ( 1 98 8 . Chapter 9, Sec. 9) by noting t hat a' Ab = tr ( Aba ) . Result ( 1 3 ) i s from :vl agnus & Neudecker ( 1988, C hapter 9 , Sec . 1 1 ) . '
10.3.2
Derivatives of the Trace
First Order Derivatives atr( X) a tr( X ' ) ( 1 ) X ( m x m) : = = lm . {) X aX (2) X ( m x n ), A (n x m) :
(3) X (m x n). A (m x n ) :
atr(AX) atr( X A ) = ' . = A aX aX atr ( X ' A ) atr( AX' ) . = =A aX aX
H A N DBOOK O F M AT RICES
1 78
(4)
X ( m x n), A (p x m ) , B ( n x p) :
(5)
X (m x n) , A ( p x n), B (m x p) : otr(X 2 ) = 2X , X (m x m) : ox otr(X; ) = ; i - t (X' ) ox
(6)
(7) X,
(11) ( 12) (13)
·
,
i = l , 2, . . .
A , B (m x m) :
ot (X'X v otr(XX' ) ) r 2 = = ( x n) : 0X 0X ot r ( X ) = ( A + A' ) X . X ( m n), A ( m X m ) : otr( X) = 2 AX. X (m n), A ( m X m) symmetric : otr( x' ) = X(A + A' X (m n), A (n X n ) : ). otr( x' ) = 2 XA. X ( m n). A ( n X n) symmet r ic : X ot ) = X ' A' + A' X' . r X , A (m x m) :
(8) X
( 10 )
·
·
•
(9)
otr( AX B) = A' B' ox otr(AX' B ) = BA ox
,'\. .
m
:�A
X
:�A
X
::
X
X
(:/
::
:
x' A' otr(A ) = 2 A' AX. (14) '( ( 1 5 ) X ( m x n ) , A (p x n ) : otr(AX ' X A' ) 2 XA'A. ox Y. X ' B ) otr(A ) ( X ) A B 1 6 : m x n , (p x m) , ( ( BA + A' B' ) -'-" . m x p) ( �u X otr(A X B ) = X(BA + A ( 1 7 ) X (m x n), A (p x n), B (n x p) : ' B' ). o otr( ' B ) = B'XA ' + BXA. ( 18 ) X (m x n), A (n x n), B (m x m) : otr( X, AX B ) -- B' X ' A' + A' X' B ( 1 9 ) X (m x n), A , B (n x ) . uX X (m n ) , A (p X m ) : X
_
_
_
_
� ��X
m
:
I
V ECTOR A N D M ATRIX DERIVATI V ES
( 20) X ( m
x
n) ,
A (p x
m
1 79
) , B ( n x n ) , C ( m x p) :
atr(AX BX ' C ) = ' ' ' A C X B + C AX B . ax (21) X (m
x
n),
A (P x at r(
m
), B (n x
m
) , C ( n x p) :
A:X: C ) = A' C' X' B'
( 22) X ( m
x m
) nonsingular :
( 23 ) X ( m
x m
) nonsingular, A , B ( m x
( 24 ) X ( m
X
(25) X ( m
x
m
)
+
B ' X ' A'C ' .
:
, avec F(X) . atr[F(X)] = vee( !p ) avec(X)' avec(X)' n ) , F ( X ) ( p x q) , G( X ) ( r x s ) , A (q x r ) , B ( s x p) :
n),
F(X) (P
X
P) :
atr[F(X)AG(X ) B] _ avec( X)'
[
avec F(X) vee(!P ) ' [B'G(X) ' A ' Q? IP ] avec( X)'
+
[B' Q? F(X)A]
]
avec G( X ) . avec( X) '
Hessian Matrices
)
(1) X (m
x m
(2) X (m
x
n) ,
A (P x
(3) X (m
X
n) ,
A (P
(4) X (m
X m
)
:
:
m
) , B ( n x p) :
X n),
B (m
X p) :
a2 tr ( AXB) ----'------'---avec(X)avec (X)' = Omn x mn a2 tr(AX' B) · avec(X )avec (X)' = Om n x mn ·
a2tr(X2 ) 2 uvec( X )uvec(X)' = Kmm , n
•
n
i-2
l:) X ' ) i - 2 -j Q? xi
j =O (5) X (m
x
n) :
a2tr(X ' X) ...,..--- �-..!..-- = 2 Imn avec( X)avec ( X )'
·
,
i
=
2, 3, . . .
H A N D BOOK O F M ATRICES
1 80 X
(m
A(
a2 tr(X1 A X ) = In avec ( X ) avec ( X )1
x
n),
x
n), A (m
(m
x
n), A (n
(9) X (m
X
n ) . A ( n X n ) symmetric:
(6)
(7) X (m (8)
( 10)
X
X,
A (m
x m
)
m x m
n ) . A (p
( 12 ) X ( m
X
n), A
(m
x n
:
x
( A + AI ) .
a2tr(X1AX ) = 2( In O A ) . avec( X ) avec ( X )1
) symmetric:
a2tr(X A X' ) = ( A + A ) 0 Im . avec ( X ) avec ( X )1 I
n) :
(P
) . A (p
x
m)
a2 tr(X A X ' ) = 2 ( A 0 1, ) . avec( X )avec ( X ) '
a2tr( AX X ' A ' ) -2 ( In 0 A' A ) . avec ( X )Ovec ( X ) '
:
a2 t r ( A X ' X A' ) = 2 ( A ' A 0 Im ) · avec ( X ) avec (X )' m ) , B ( m x p) :
X n) : x
a2 tr( A X X ' B ) avec ( X )avec ( X ) ' ( 14 ) X ( m
°
a2 tr(X A X ) I • l\ ) l A ( mm . m + I A m 0 0 = a ve c( X ) a vec ( X ) 1
x
X
x
:
( 11) X (m
(13)
x m
)
n ), A (P
x
n), B (n
x
-
-
In
°
( B A + AI BI ) .
p) :
a2t r( AX 1 X B) I I B ) 0 Im . ( BA A + av ec ( X )Ovec ( X ) ' = ( 15)
X
(m
x
n ), A (n
n), B (m
x
x m
)
:
a2 t r ( X A X' B ) _ . - A !I "' !I uvec ( ·"- )uvec ( X )' ( 16 ) X ( m
x
n) , A , B (n
x m
)
0
B' + A'
0
R.
:
a2tr(X AX B) _ ' -'-:.._ - ( A 0 B1 + B 0 A ) l\.mn avec( X )Ovec ( X ) '
-
( 1 7) X ( m
x
n ). A (p
-
x m
), B (n
x
n ), C (m
x
·
p) :
a 2 t r ( A X BX' C ) _ 'C'' + B 1 0 C' A . __,_ .:.._ - B 0 A avec ( X )avec ( X )'
_
( 18) X (m
x
n ), A (p
_ _ _
) B (n
x m ,
x m
), C (n
x
p) :
a2 tr( AX BXC) = ( B 0 A'C 1 + C A 0 B' ) l\'m n . X c av ec ( X )a ve ( ) '
181
V ECTO R A N D M AT RI X DERIVATI V ES
( 19) X (m
x m
) nonsingular :
iP tr(X - 1 ) = Kmm (X ' - 2 Q? x - t + x '- t Q? x - 2 ) . ovec ( X ) 8vec ( X ) ' (20) X ( m
x m
) nonsingular, A , B ( m x
m
)
:
o2 tr( A X - 1 B ) ovec( X )ovec ( X )' . = l\ mm ( X ' - 1 A' B' X' - 1 Q? X - 1 + X' - 1 @ X - 1 B A X - 1 ) . Note: The results of this subsection are partly given i n M agnus & Neudecker ( 1 988, C h apter 9 , Sec. 9 and 1 3 ) and partly fol low from their results by straightforward application of the rules for the trace and matrix derivatives, notably the chain rule and the product rules of Section 1 0 . 5 . 10.3.3
D erivatives of Determinants ad ) ( 1 ) X ( m X m) nonsingular :
(2) X (m
x
n), A (p
x
m
), B (n
�iX
=
det ( X ) ( X ' ) - 1
=
( X a dj ) ' .
x p) , det( A X B ) =f. 0 :
��X B ) = det ( A X B ) A' ( B' X ' A' ) - 1 B' . odei: X ' ) ) rk( X = = 2det ( X X' ) ( XX' ) - 1 X . ode�;' X ) 2det ( X ' X ) X ( X ' rk( X ) = a de t
( 3 ) X (m X n) , (4) X ( m
(5) X (m
x
n) ,
x m
m :
=
n :
X )- 1 .
) nonsingu l ar:
2) �1\ 2( det X ) 2 ( X ' ) - 1 . ( a) ode�,� - t ) - ( de t X ) - 1 ( X ' ) - 1 . ( b) ') od ��\ = i ( det X ); ( X ' ) - 1 , i = ± l , ± 2 , . . . (c) od
=
=
( 6 ) X, A , B (m x
m
) , det ( A X i B ) =f. 0 :
i- t . 8det ( AX ; B ) . . = det ( A X ' B ) 2:) (x' - 1 - 1 ) B ( A X ' B ) - 1 A X J ] ' . ax =l
J
H A ND BOOK OF M ATRICES
182
(7)
X
(m
x
n ) , A (p x
m
B ( n x n) , C
),
(m
x p), det ( A X B X ' C )
:j; 0 :
odet ( A X B X ' C ) ox det ( A X B X ' C ) {C[AX BX'G] - 1 A X B + A ' [C' X B' X ' A'] - 1 C' X B' } .
(8) X (m
x
(9) X (m
x
n ) , A (p
x
n), B
(m x
m
),
C (n
x
)
C (n
x
p), det ( AX' BXC) :j; 0 :
odet ( A X' B X C ) ox det(AX' BXC) { BXC[A X ' B X G] - 1 A + B' X A' [C' X' B' X A't 1 C' } .
n ) , A (p x
m
B (n x
),
m ,
p) , det ( A X B X C) :j; 0 :
odet ( A X BXC) _ ox det( A X B X C ) { C[AX BXC]- 1 AX B + B X C[AX BXG] - 1 A } ' .
( 10 )
X
(m
(11) X (m ( 12 ) X ( m
)
x m , x
x
n), A
(m x
o ln d
)
e��
m
=
(X' ) _ 1
·
ox positive definite, det ( X ' A X ) > 0 : 'AX)
=
2 A X ( X ' AX ) - 1 •
n ) , A ( n x n ) positive definite, det ( X A X ' ) > 0 : o l n det ( X A X ' )
=
ox
n ) , A ( n x p) , B det ( A ' X ' B X A ) > 0 :
( 13) X (m
o ln det( X )
det ( X ) > 0 :
x
o l n det(
:�X
( m x m)
' BXA)
=
2(XAX') - I XA
·
positive defi nite,
2 B X A ( A' X' BXA) - I A' .
n ) , A (p x m ) , B ( n x n ) positive definite, det ( A X BX' A') > 0 :
( 14 ) X ( m
x
o l n det(
( 15 ) X ( m
)
x m ,
::
BX'A')
= 2A'( A X B X ' A' ) - 1 A X B .
det ( X ) > 0 : o2 l n det ( X )
ovec ( X )ovec ( X ) '
-
-
Kmm
( ( X ' ) - I 0 x- I ] .
Note: ( 1 ) - (5) and ( 7) - ( 9 ) are either given in or follow straightforwardly from Magnus & N eudecker ( 1 988, Chapter 9, Sec. 10). Rules ( 10) - ( 15 ) follow from the chain rule and t he derivative of the logarithm . I n ( 1 5 ) , the derivat i ve of vec ( X - 1 ) from Sec. 1 0 . 6 is used i n addition . ( 6 ) results from ( 2 ) , a chain rule and a prod uct rule.
183
V ECTOR A N D M ATRIX D ERIVATI V ES 1 0 .4
10.4.1
Jacobian Matrices of Linear Functions
Linear Functions with General Matrix Arguments
( 1 ) X (m x n) :
avec( X) avec(X) '
=
lmn ,
avec(X') _ 1\. - mn ' avec( X)'
( 2 ) X (m x
( 3) X (m
x
(4) X (m x ( 5 ) X (m x
avec( X ) = Knm · avec( X ' ) ' avec( eX ) c = n ) , E IR : avec( X) ' clmn · avec(AX B ) __ "' B' A. n ) , A ( p x m ) , B ( n x q) : avec( X ) ' avec( AX' B ) n ) , A (p x n ) , B ( m x q) : = ( B 0 A)l\ mn . avec(X) ' n ) , A ( p x m ) , B ( n x q ) , C (p x q) : v
I
avec( AX B + C) = B' 0 A. avec(X)' ( 6) X ( m
x n) ,
A, C (p
x m) ,
B, D (n
x q) :
avec(AX B ± CX D) = B' 0 A ± D' 0 C · avec( X ) ' ( 7) X ( m x n ) , A, C (p x n ) , B , D ( m x q) :
avec( AX' B ± CX' D) = ( B 0 A ± D 0 C) l\ mn · avec(X )' I
I
•
( 8 ) X ( m x n ) , A (p x m ) , B ( n x q ) , C (p x n ) , D ( m x q) :
avec( AX B + CX' D) avec ( X ) '
avec (CX' D + AX B) avec ( X ) ' - B ' 0 A + ( D' 0 C) I\mn ·
_
( 9 ) X ( m x n ) , A ; (p x m ) , B; ( n x q),
i=
1,
.
.
.,r
:
.
H A N DBOOK O F M ATRICES
1 84
( 1 0 ) X ( m x n) , Ai (p x n ) , Bi ( m x q ) ,
8vee ( L �- 1 A ; X 1 B; ) = 8vee ( X ) 1
i = l, . .
.
,r:
(� B'1 0 A'- ) \ mn · y
�
•=1
Kronecker and Hadamard Products (11)
X
( m x n ) , A (P x q) :
8vec (A) 0 vee ( X ) - vee( A ) 0 Imn · Bvee( X )1 ( b ) 8vec( X ) 0 vee( A) = Imn 0 vee( A ) . 1 ovec( X ) 8vee(A 0 X ) . = 0 { l (c) I\ q np 0 Im ) [vec(A) 0 Imn ] . 8vee( X ) 1 8vec(X 0 A ) . (d) = ( I 0 I\ qm 0 Ip )(Imn 0 vec(A)]. n 8vee ( X ) 1 X ( m x n ) . A (p x q ) , B ( r x m ) , C ( n x s ) (a)
( 12)
_
:
.
avec( A 0 BXC) = ( Iq 0 I\ •p 0 Ir ) [vee( A ) 0 C 0 B] , Clvec ( X )1 1
avec ( BXC 0 A ) = ( 1, 0 I\ qr 0 Ip )[C 0 B 0 vec(A)] . ovee( X )1 x n), A (P x q ) , B (r x s ) : I
"
( 13 ) X (m
8vee( A 0 X 0 B ) 8vec( X ) ' = ( Iq 0 I\n , ,p (?) Imr ) [vee ( A ) 0 ( In @ 1\,m 0 Ir )(Imn
( 1 4 ) X ( m x n) , A (p x q ) , B ( 1· x s ) , C (k x m) ,
D (n
x l)
(2)
vee( B ) ) ] .
:
8vec( A 0 CX D 0 B ) 8vee( X )' = ( Iq 0 I\, l,p 0 h r ) [ vec(A) 0 ( 11 0 K,k 0 Ir ) ( D1 0 C 0 ver( B))]. ( 15 ) X ( m x n ) , A (p x q ) , B ( mp x nq ) :
.
8vee [(A 0 X ) 8 B] = d 1. ag( vee B)( lq 0 I\ np 0 Im )[vee(A) 0 Imn J , 8 vee ( X ) 1 8vec [( X ·S> A" ) J
V ECTO R A N D M ATRIX DERIVATIV ES
1 85
( 1 6 } X, A ( m x n ) : avec( X 0 A ) avec( A 0 X ) = = diag(vec A ) . avec( X ) ' avec( X ) ' ( 17} X ( m x n ) , A ( P x q ) , B ( p x
) C (n x q) :
m ,
avec( A 0 BXC) _ _ avec( B X C 0 A) _ ' - d t ag( vee A ) ( C' 0 B ) . -----::-avec( X ) ' avec ( X ) ' �
( 1 8 ) X ( m x n ) , A ( p x q ) , B ( p x n ) , C ( m x q) : avec ( B X ' C 0 A ) ' avec( A 0 B X ' C) = = d t ag( vee A )(C' 0 B ) J(mn . 8vec( X ) ' 8vec( X )' ( 19 } X, A , B ( m x n ) : avec( A 0 X 0 B ) = diag[vec( A 0 B ) ) . avec ( X ) ' ( 2 0 } X ( m x n) , A , B ( n x
m
)
:
avec( A 0 X ' 0 B ) . . = dt ag[vec ( A 0 B )] A mn · 0 vee ( X ) ' ( 2 1 } X , A ( m x n) , B ( P x q) : avec [ ( A 0 X ) 0 B) . . = ( In 0 1\ q m 0 Jp ) [d tag(vec A ) 0 vee( B ) ) , avec( X )' avec[B 0 ( A 0 X )) . . = ( /q 0 l\ np 0 Im ) [vec ( B ) 0 d t ag(vec A ) ) . avec ( X )' Note: ( 1 ) - ( 10) fol low from basic properties of the derivative and t he rule vee( AX B) = ( B' 0 A ) vec ( X ) for the vee operator (see Section 7.2). ( 1 1 ) ( 2 1 ) m ay be derived using vec( A 0 B ) = ( I 0 1\ 0 / ) [vee( A ) 0 vee( B )) and vee ( A 8 B) = diag(vec A ) vec ( B) and other standard rules for Kronecker and H adamard p roducts (see Chapter 2 and M agnus & Neudecker ( 1 988, Chapter 9, Sec. 14 ) ) . 10.4.2
Linear Functions with Symmetric Matrix Arguments
Reminder: Dm denotes a duplication m atrix and D1;, its M oore -Penrose inverse (see Section 9 . 5 ) . Lm is an elimination matrix ( see Section 9.6) .
H A N DBOOK O F M ATRICES
1 86
( 1 ) X ( m x m)
(2) X (m x (3} X (m x
sym metric :
8vech( X ) avech( X ) ' = Im ( m + l l /2 '
avec(X ) = Dm . avech( X ) ' avec(cX ) m ) symmetric, c E IR : = cDm . avech( X )' m ) symmetric, A ( m x n ) : avech( A'X A) L = n ( A' ° A' ) D"' avech( X ) '
(4) X ( m x m)
symmetric, A ( n
x m),
B (m
avec ( AX B ) = (B avech( X )'
I
0
D + ( A' ° A' ) D"' .
=
n
x p) :
A )D, .
( 5 ) X ( m x m ) symmetric , A, B ( m x m ) :
avech (AXA' ± BX B' ) = D+ (A 0 A ± B 0 B ) Dm . avech( X ) ' "'
(6)
X (m x m)
symmetric , A, B ( m
x m) :
avech(AX B' + BX A' ) = D + ( B 0 A + A 0 B ) Dm · avech( X) ' "'
( 7) X ( m x m )
symmetric, A, C ( n
B, D ( m
x m),
x p) :
avec ( AX B ± CX D) = ( B ' 0 A ± D' 0 C ) Dm · avech( X ) ' (8}
X (m
x m)
symmetric, A; ( n
x m) ,
B; ( m
x p) ,
i = 1, . . .,r :
avec ( L:�= l A; X B; ) - � ( B ; 0 A. ) Dm · � avech( X ) ' I
( 9 } X ( m x m)
symmetric, A; ( m
x n) , i = 1 ,
avech( L ;- 1 A ; X A i ) � = � L n ( A'I 0Vech ( X ) ' •=I
. . . , r :
� 0 A' ) Dm = � D + ( A ' 0 A ' )D m . I
•= 1
n
I
I
Note: These results fol low from basic properties of the derivative and the
vee and vech operators. Notably, the rule vec( A X B ) ( B' 0 A)vec( X ) is useful (see Chapter 7 ) . For proofs see also Magnus ( 1 988, Capter 8, Sec . 8 . 2 ) . =
187
V ECTO R A N D M AT RI X DERIVATIV ES
L inear Functions with Triangular Matrix Arguments
10.4.3
Reminder: Lm denotes an elimination m atrix (see Section 9 . 6 ) .
( 1 ) X ( m x m ) lower triangular:
avech( X ) = J/2 ' 8vech(X)1 lm ( m+ I
avec(X) 1 = Lm 8vech(X)1 avec( eX) cL:., . m ) lower triangular, c E IR. : 8vech( X )1 m ) lower triangular, A ( n x m ) , B ( m x p) •
(2)
X (m
x
(3)
X (m
x
=
:
avec( AX B) 8vech (X)1 (4} X ( m
x m)
1 ( B 1 0 A)L m
lower triangular, A, e ( n avec ( AX B ± ex D) 8vech(X)1
( 5 ) X ( m x m)
=
=
lower triangular, Ai ( n
•
B, D ( m
x m) ,
( B 1 0 A ± D1 0 e) L1m · x m) ,
Bi ( m
x p) , i
avech( AXB) avech( X) I m ) lower triangular:
(6) X, A, B ( m x m ) lower triangular: ( 7 ) X, A, B , e, D ( m x
avech(AX B ± ex D) 8vech (X}1 (8)
X, A i , Bi ( m
x m)
=
=
=
=
1,. . .,r
:
I I L m ( B 0 A) L m .
Lm ( B1 0 A ± D 1 0 e)L1m ·
lower triangular, i
avech (L�- 1 A i X B i ) 8vech( X)1
x p) :
Lm
=
1,
.
.
.,r:
(�I B'� 0 A'- ) Lm1 · � •=
Note: The results of this subsection fol low from basic properties of
derivatives and the vee and vech operators. In particular, the rules vec ( A) L:., vech( A) for lower triangular matrices A (see Section 9 . 1 4 ) and vee( AX B ) = ( B1 0 A)vec ( X ) ( see Section 7.2) are useful. For proofs see also Magnus ( 1 988, Chapter 8, Sec. 8.3 ) . =
H A N DBOOK O F M ATRICES
1 88
10.4.4
Linear Functions of Vector and Matrix Valued Functions with Vector Arguments
ac = On x m · ax' ( 2 ) X ( m X 1 ) , y( X ) , Z ( X ) ( n X 1 ) , C I , C2 E ffi : ( 1 ) x (m x 1),
c
(n x 1)
constant :
az( x ) ay( x ) a[cl y (x) ± c2 z(x)] C C ± 2 £l = 1 £l £l 1 ux' ux' ux
( 3 ) x ( m x 1 ) , yi ( x ) ( n
x 1 ) , ci E
ffi,
•
i = 1, . . ., r :
a [L: �- 1 c;yi (x)] � ay; ( x ) c; . = ax' x' a L.I ..., i= ( 4 ) x ( m x 1 ) . y( x ) ( n x 1 ) , z ( x ) ( p x 1 ) , A ( q x n ) , B ( q x p ) :
a [ A y(x) ± B z ( x)] az ( x ) ay( x ) ± B !'l 1 . =A ax ' ux ax' avec( c Y ) avec ( Y ) =c ax' ax ' ( 6 ) x ( m x 1 ) , Y ( x) ( n x p ) . A ( q x n ) , B (p x r ) : ( 5 ) x ( m x 1 ) , Y( x ) ( n x p ) , c E IR :
avec( A Y B) ax'
=
I
(B 0 A)
avec ( Y ) . ax'
( 7 ) x ( m x 1 ) , Y; ( x ) ( n ; x pi ) , A ; ( q x n; ) , B; (p; x r ) , i
=
avec ( }i ) avec (L: := I A ; l( B; ) - � I L... ., A ) ( Bl 0 !) I £l I l!X X u I= I I
1,
. . . , s :
•
.
( 8 ) x ( m x 1 ) . Y ( x ) ( n x p ) , A ( q x n ) , B (p x r ) ,
C (q
avec ( A Y B + C) _ avec ( Y ) ( - B ® .4 ) £l 1 £l 1 ux ux I
( 9 ) x ( m x 1 ) , Y ( x ) ( n x p ) , A ( r x s) :
x r)
:
.
avec( Y ) ) ( vec ( A ) ( !, , ax' ( avec ( Y ) ) avec ( Y ® A ) . c(A) . = ( lp ® l\ • n 0 lr ) 0 vc ' avec( A 0 Y ) ax' ax
=
0
. l\p r 0 In )
0
ax'
VECTOR A N D M ATRIX D E RI VATI V E S ( 10}
x (m
x 1),
Y(x ) ( n
x p) ,
A (r
x
1 89
s ,
) B ( k x /) :
avec( A ® Y ® B) ox' = ( I, ® f\p l,r ® Ink )
(11)
x (m
x
[vec( A )
0 ( lp 0 K 1n
x 1),
Y(x) (n
x p) ,
A (r
x
( ov��('Y ) 0 vee( B) )] .
0 h) s ,
) B ( k x n ) , C ( p x /) :
( [(c' B ) ovecox('Y ) ] ® vee ( A) ) ovec ( Y ) ' ( c ( [ ) ) . ] 0 Ik ) vee (A 0 0 B) ox'
ovec ( BYC ® A) _ y ® ( I - 1 \ ,k ® I ) ox '
0
r
ovec( A ® BYC) _ ( I 0 1\. l r ox' x ( m x 1 ) , Y(x ) ( n x p ) , A ( n -
( 12}
•
'
x p) :
ovec(Y 8 A ) _ ovec(A 8 Y ) _ . ovec( Y ) ( A ag vee ;::, ) 0 - dt ;::, ' ux' ux x' Note: The results of this subsection fol low from the linearity of the functions considered, the rules for the vee operator and basic matrix operations. .
10.5
P rod uct R u les
10.5. 1 (1)
Matrix Products
X (m
x m) :
avec( X 2 ) (a) = X 0 Im + Im 0 X . avec ( X )' . i- 1 ve X ) e( ( b) = " (X' ) i - t - j ® xi , L... vec ( X ; =0
�
r/
•
�
1
i = 1 , 2, . . .
8vec ( X ' 2 ) (c) ( X ® Im + Im 0 X ) I\m m · = X c( ) ve o ' X (m x n) : avec ( X ' X ) = ( In > + A nn ) ( ln ® X ) . (a) ;::, uvec( X ) avec ( X X') = Urn > + A.m m )( X ® Im ) · (b) o v ec ( X ) ' •1
( 2)
1
•
•
I
H A N DBOOK O F M AT RICES
1 90
(3) X (m x n), A (n x m) :
avec( X AX) A 0 Im = X c( )' fJ ve X I
(4) X ( m
x n) ,
A (m
I
x m) :
avec( X' AX) = ( X A 0 In ) A m n ) X c( ' a ve ,
( 5 ) X ( m x n),
A (m
,
+ In 0 X A .
x m)
1
•
+ ( In
0
,
X A).
symmetric:
avec( X' AX) a ve c( X )' =
.
0 )\
Un > + l\ n n ) U
,
n
,
A).
(6) X (m x n) , A ( n x n) :
avec( X AX' ) = ( X A 0 Im ) + ( Im 0 X A) l\ mn . f)vee ( X ) ' .
I
( 7 ) X ( m x n ) , A ( n x n ) symmetric:
avec( X AX') avec( X ) '
,
,
= ( 1m 2 + l\ mm )( X A
0
Im ) ·
( 8 ) X ( m x n ) , A ( P x m ) , B ( n x m ) , C ( n x q) :
avec(AX BXC) = C' X' B' 0 A + C' 0 AXB . ) avec ( X ' (9) X (m
( 10 )
x n),
A (p
x n) ,
avec( AX ' BXC) fJvec ( X )'
B (m =
x m ) , C ( n x q) :
( C' X ' B' 0 A ) F \ mn + (C' 0 AX' B) .
X ( m x n ) , A ( p x n ) , B ( m x m ) symmetric:
avec( AX' BXA' ) = ( lp > avec( X ) '
( 1 1 ) X ( m x n ) , A (p x m ) , B ( n x n ) ,
+
I
I\pp ) (A 0 AX B). '
C ( m x q) :
avec ( A X B X ' C ) = ( C ' X B ' 0 A ) + ( C ' 0 A X B ) Km n . avec( X ) '
( 12) X ( m x n ) , A ( p x m ) , B ( n x n ) symmetric:
avec( AX BX' A ' ) = Ur > avec( X ) '
, + I'-rr H A X B 0 A ) .
191
V ECTO R A N D M AT RIX DERIVATIVES
(13)
X ( m x m ) , A ( n x m ) , B ( m x p) : .
i- 1
avec ( AX' B ) = L B '(X ' ) i- 1 - j 0 AX i ' avec(X)' = J O
(14)
.
X, A ( m x m ) : avec( A + X ) 2 avec(X)' avec( A + X ) ; avec( X)'
( 16)
= 1, 2, . . .
X ( m x n ) , A (p x m ) , B ( n x p) : avec (A X B )' L ' ' A ') i - 1 -i ' 0 (A )i A (B X XB , i = B avec(X)' j O = =
( 15)
i
i- 1
1 , 2, . . .
(A ' + X ' ) 0 lm + lm 0 (A + X ), i- 1
L(A ' + X' ) i - 1 -j 0 ( A + ; qi '
j =O
i = 1, 2. . . .
X, A ( m x n ) : avec[( A + X ) '( A + X ) ] = ( I F\ nn )[ /n 0 ( A' + -\ ' ) ] , + n (X avec ) ' avec [(A + X ) ( A + X ) ' ] ( I m2 + 1\.mm )[( 4 + "' ) "' Im ] . avec (X.) ' X ( m x n ) , A (P x p) , B (p x m ) , C ( n x p) avec( A + BX C )2 = (A ' + C ' X ' B' ) C' 0 B + C' 0 (A + BXC)B, avec(X)' avec (A + BX C ) ; ' + C' X ' B ' ) i - 1 -i C' 0 ( A + B X C )J B , (A = avec( X )' j =O 2
_
•
(17)
-�
v
:
�
i = 1, 2, . . .
A ( p x q) , B ( p x m ) , C ( n x q) : avec[(A + B XC) ' (A + BXC) ] -_ Iq, + 1\.. qq )[C. ' (A ' + c�' '\'' B' )B 0 ]. ( avec(X)' avec [(A + B X C )( A + B X C )' ] = /p > 1\pp )[( A + B X C )C' 8 B ] . ( + avec( X )' Note: ( 1 ) - ( 2 ) are gi ve n in Magnus & Neudecker ( 1 988, Chapter 9, Sec .
( 18) X (m x n),
The remaining results follow via the product and chain rules of different ial calculus. 13).
H A N DBOOK OF M ATRICES
1 92
Kronecker and Hadamard Products
10.5.2
( 1 } X (m x n) :
avec ( X ) 0 vec( X ) £l = lm n 0 vec( X ) + vec ( X ) 0 lmn · 1 uvec ( X )
( a)
avec( X 0 X ) £l (b) \" ) = ( ln 0 l\.nm 0 lm ) [ lmn 0 vec(,\. ) + vec ( ,\. ) 0 lmn ] . uvec ( , 1 avec( X 0 X') (c) avec ( X ) I = ( In \5 1\mm 0 ln ) [ lmn 0 vec ( X ' ) + vee( X ) 0 Kmn ] · avec(X' 0 X ) ( d ) avec ( X )' = Um 0 1\nn 0 lm ) [ l\.mn 0 vec ( X ) + vec( X ' ) 0 lmn l · avec( X' 0 X ' ) (e) avec ( X )' = Urn 0 1\mn 0 In ) [ 1\mn 0 vec ( X ' ) + vec ( X ' ) 0 Km n ] ·
(2}
X (m x n ) , A ( P x m ) , B ( n x q ) , C ( r x
avec ( A X B 0 C X D) avec ( X )' = ( lq 0 r.;, P c� lr ) [B'
(3)
X
0
m
) , D (n x s) :
A •:V vec ( CX D) + vec ( A X B ) 0 D'
( m X n ) , A (p X m ) , B ( n X q ) , avec( A X B 0 CX' D) avec ( X ) ' = { lq 0 1\", p 0 lr )
r (r
X n), D ( m X s)
0
C] .
:
X [B' 0 A S vec ( CX' D) + vec ( A X B ) 0 ( D' 0 C ) l\m n ] .
( 4 ) X ( m x n ) , A ( P x n ) , B (m x q ) , C ( r x m ) , D ( n x s ) :
avec ( A X ' B 0 CX D) avec ( X )' = ( fq (� f\, p 0 fr ) x [( B'
(5) X ( m
0
A ) l\mn
x n ), A (p x n),
avec( AX' B 0 C X ' D) avec ( X )'
= ( lq
•2 1\",P (:)
l )[( B' r
0
vec ( CX D) + vec ( A X ' B ) 0 D'
B (m x q),
0
A ) l\mn
0
C ( r x n) , D ( m x s) :
ve c ( CX' D)+
8
C] .
193
V ECTO R A N D M ATRIX DERIVATIV ES
+ vec(AX ' B) 0 (D' 0 C ) Kmn ] ·
X ( m x n) : 8 X) 2 " ( ag vec .'\. d ). ( a) avec(X 1 avec( X ) X 8 X 8 X ) 3 d " ag [vec (X 8. X )] . ( b ) avec(avec(X)' I ' X 8 X' ) avec( . 2 d (c) uvec( X) ' 1 ag(vec X )h mn · ( 7 ) X (m x m) : avec( X 8 X') = avec(X 1 8 X) d . ag(vec ):, )1\. n + d . ag(vec ,\. 1 ) . m 1 1 uvec(X)1 uvec(X ) 1 ( 8 ) X ( m n ) , A, C ( P x m ) , B, D ( n x q) : avec( AX B C::1 CX D) avec( X )' diag[vec(AX B)](D' 0 C) + diag[vec(CX D)](B' 0 A). (9) X ( m x n ) , A (p x n ) , B ( m x q) , C (p x m ) , D ( n x q ) : avec( C X D 0 AX ' B) avec(AX' B 0 CX D) avec( X)' avec( X)' diag[vec(AX' B)](D1 0 C) + diag[vec(CX D)](B' 0 A ) h. m . ( 10) X ( m x n ) , A , C (p x n ) , B , D ( m x q ) : avec(AX1 B (.:_) CX ' D) ' 0 C) {diag[vec(AX' B)](D avec( X )1 + diag[vec( CX ' D) ] ( B' 0 A)}/\·,, . Note: The results of this subsection follow from the b asic product rule for differentiation and the rules for Kronecker and Hadamard products . See also Magnus & Neudecker ( 1 988, Chapter 9, Sec. 1 4 ) . (6)
I
v
=
=
!.l
!.l
!.l
I
=
=
"
,
x
=
n
10.5.3
Functions with Symmetric Matrix Arguments
Dm denotes a duplication matrix and D�, its Moore -Penrose inverse (see Section 9.5) . L m is an elimination matrix (see Section 9.6) . (1) X (m ) symmetric: Reminder:
x m
H A N DBOOK O F M ATRIC ES
1 94
8vec(X2) = ( X 0 Im + lm 0 X ) Dm · ( a ) a v ech (X )' ' -1 i ' av ec ) (X b ( ) = """ ( \" i - 1 -j 0 Xi ) Dm ' i = l , 2, . . . .. avech(X)' JL... =O . i- 1 av (X ) ec h ' (c ) avech(X)' = D� L x i - 1 -j 0 xi Dm , i = I , 2 , i =D X , A ( m m ) symmetric: avech(X AX) - D;!; (X A rg; Im + Im 0 X A) Dm avech(X)' Lrn(XA 0 Im + lm 0 XA)Dm . X ( m m) symmetric, A ( p m ) , B ( m x m ) , C ( m q ) : avec ( A X B X C ) = (C' X B' A + C' 0 A \" B )Dm . 0 a vech(X) ' X ( m m ) symmetric, A (p x m ) , B ( m q ) : . i- 1 avec(AX' B ) = L ' x i j 0 AXi )Dm , i = 1 , 2 , . . . (B avech(X)' i = D m ) symmetric, A (p m ) , B ( m p ) : X (m iavec( AX B)' ' X A ' ) i - 1 - j B' 0 (AX B)j A B L ( avech(X) ' i=O '
.
(2)
(3)
.
.
x
x
x
x
'
(4)
x
x
-
(5)
x
1
-
x
x
1
i = l , 2, . . . (6)
X ( m m ) symmetric, A ( p x m ) , B ( m p ) , C (p p) : avec( AX B + C)i avech(X) ' iL ( B' XA' + C' ) i - 1 -j B' C?; (AXB + C)i A Dm , x
x
x
1
j =O
i = l,2, . . . ( 7)
X (m m ) symmetric, A (p m), B ( q), C (p q) : avec [(A X B + � ) (A X B + C) ] (/ ) + K q ) [B' 0( B' X A ' + C' ) A] Dm . q q avec ( X )' m x
x
x
'
=
x
195
V ECTOR A N D M ATRIX DERIVATI V ES
X (m x m) symmetric, A (p x ) B (m x q ), C ( p x q) : avec[(AX B + C)( AX B + C) '] ( /p > + K )[(AX B + C)B' 0 A]Dm . pp avech(X)' ( 9 ) X ( m x m) symmetric: avec(X 0 X) = U 0 mm 0 Im )[lm> 0 vec(X) + vec(X) ® lm>]Dm . rn l\ avech(X) ' ( 1 0 ) X (m x m) symmetric, A (p x m ) , B (m x q), C ( r x m), D (m x ) avec( AX B 0 C X D) avech(X )' ( /q 0 K$p 0 lr )[B' 0 A 0 vec(C X D) + vec(AX B) 0 D' 0 C]Dm . ( 1 1 ) X (m x m) symmetric: X) 8 avec(X ( a) fJvech( X )' -_ 2 d'1 ag ( vee X)Dm . av 8 X X X e ) c l 8 · 3 di ag [vec (X 8 X )] Dm b ( ) . uvech ( X ) 1 ( 1 2 ) X (m x m) symmetric, A ( p x m), B (m x q), C (p x m), D (m x q) : avec( AX B 8 CX D) ' 0 C) { diag[vec(AX B)]( D fJvech(X)' + diag[vec ( CX D)](B' 0 A)}Dm . Note: These results follow from those of the previous su b section and the chain rule for matrix differentiation. See also Magnus ( 1 988, Chapter 8 , Sec. (8)
m
,
=
..:. __ .. _:: ____:..._ ..._ .::.�
•
s
:
=
=
8.2) .
1 0.5.4
Functions with Lower Triangular M atrix Arguments
D;t; is the Moore -Penrose inverse of the duplication matrix Dm (see Section 9 .5 ) and Lm denotes an elimination matrix (see Section 9.6). ( 1 ) X (m x m) lower triangular: avech(X i ) Lm L (X')i-1-i 0 xi L'm , i 1 , 2, . . . fJvech(X)' i=O ( 2 ) X, A (m x m) lower triangular: fJvech(X AX) Lm (X ' A ' 0 Im + Im 0 X A)L ' . m fJvech (X)'
Reminder:
=
i- 1
=
=
H A N DBOOK O F M AT RICES
1 96
(m
x
lower triangular: avech(X ' X ) = 2 D+m ( /m ® X ' )L'm • avech(X)' avech ( XX') = 2D+ ( X 0 Im )L' m m· avech(X)' Note: For proofs see Magnus ( 1 988, Chapter 8 , Sec. 8.3 - 8.4).
( 3)
X
10.5.5
m)
P ro ducts of M at rix Valued Functions with Vector A rguments
( 1 ) x (m x 1 ) , Y ( .r ) ( n x n) :
avec(Y i ) - � [( ) i - 1 -j i ] avec(Y) ®Y .t Y uX a X ' - L... . O x ( m x 1 ) , Y( x) ( n x p) : avec(Y ' Y) _ ( I + r\ ) ( I Y ' ) avec(Y) , ® ax' ax ' avec(YY ' ) = ( /n2 + A.n n ) ( Y ® In ) avec( Y) ux' ux' X (m J ) , Y ( X ) ( n p), Z ( X ) (p q) avec ( Y Z ) - ( I 0 Y) avec ( Z) + ( Z ' I avec ( Y) . ® n ) ax' ax' ax' x ( m x 1 ) , Y ( x ) ( n x p) , Z ( x) ( q x r) , A ( s x n ) , B (p x q), C ( r x k ) : avec ( AYBZC) = ( C' 0 A Y B) avec(Z) + ( C' Z' B 0 A ) avec ( Y) ax' ax' ax' . x ( m x 1 ) , Y( x ) ( n x p) , A (q x n) , B (p x q ) , C ( q x q) : avec(AY B + C)' ax' i-1 L( B ' Y ' A ' + C')i - l - j B' 0 (AY B + C)i A avec(Y) ax' j =O I
!.l
)=
(2)
-
p>
pp
p
_
(4)
X
X
X
.
!.l
!.l
(3)
.
t
:
q
I
(5)
i = 1 , 2, . . .
( 6 ) x ( m x 1 ) , Y ( ) ( n x p), .r
A ( q x n ) , B (p x r ) , C ( q x r)
:
V E CTO R A N D M AT R I X D E R I VATI V ES
1 97
iJvec [(A Y 13 + C)'( A Y R + C)] ( a) ax' avec ( Y ) . = ( /r> + l\rr ) [H ' 0 ( H' Y' A ' + C' ) A ] 0 x' avec[( A Y lJ + C ) ( A Y lJ + C)'] (b) ax' OVPc ( Y) . = ( lq> + l\ qq ) [( M R + () B G) A] ' c') x ( 7 ) x ( m x I ) , Y ( x ) ( n x p) , Z ( x ) ( q x r· ) : , cJVPr ( Z ) 8vec ( Y) 8(ver ( Y ) 0 vec ( Z )] . 0 + ver 0 vec ( Z ) (} ) = ( a) >l ' u:r ax ' ox' CJVt'C ( Y (;) Z ) ,
(h)
.
,
I
[
(h:'
]
.
CJvPc ( Z ) ovec ( Y) . O ve r ( X ) + ve c ( } ) O lh:' = ( lp (·) l\ r n O lq ) . i.Jx' ( 8 ) x ( m x 1 ) , Y ( .r ) ( n x p) , Z (.r ) (q x r ) , A ( s x n ) , lJ ( p x k ) . (' ( / X q) , ]) ( r X lz ) :
CJvec ( A Y 13 c;;J (.'ZD) o;r'
( h 16> X
1\'h, 0
[( H'
0
A)
/J)
ow r ( y ) ax'
()
VPr ( ( 'ZD)
+ VPc( A Y H ) @ ( D' @ C)
owe ( Z ) (,1J' '
(9) x ( m x 1 ) , Y ( x ) , Z(x) ( n x p) :
].
avec ( Z 0 Y) ' ;r ox ' c} . , avec ( Y ) Hve r ( Z ) . . dtag( vee }' ) dtag( vee Z) + .1 ']r ;r ' ( .r' ( 1 0 ) J' ( m x 1 ) , Y ( .r ) ( n x p) , Z(x) ( q x r· ) , A ( s x n ) , U ( p x /; ) , Dve c ( Y
<·1
Z)
.
('
( .'i X
q),
[)
( r· X
CJve<· ( i\ Y R
( )
ox'
.
k) :
( 'ZD)
ovec ( (.' Z lJ (�) ax' .
'
A Y H)
dtag[w r( ( Z IJ ) ] ( H
I
C:)
A)
aw r ( Y )
().r 1 . . , , ()vpr ( Z) . + dtag[ ver ( A } R ) ] ( f) C:l ( ) ) ' '
'
( J' Not(�: The rPsult.s of t h is sn bsertion follow frotu t.lw ch a i n r u l e and t lw r u ks for basic matrix operations. Sf'(' also the rPsttl!.s of the p revious su bsed.ioll S , i n particular Sections 1 0 . 5 . 1 and 1 0 .5 . 2 .
H A N DBOOK O F M AT RICES
1 98
Jacobian Matrices of Functions I nvolving I nverse Matrices
10.6
X (m
x
m)
nonsingular: avec(X- 1 ) x' - l ® x - � . 8vec(X)' 8vec(X'- 1 ) 8vec(X) ' ( 2 ) X ( m m ) symmetric nonsingular: a vech(X- 1 ) Dm+ (X _ 1 ® X _ 1 )Dm , 8vech(X)' - where Dm denotes a duplication matrix and D;t. its Moore - Penrose inverse (see Section 9.5) . Note: For proofs of these results see Magnus & Neudecker ( 1 988, Chapter 9, Sec . 1 3 ) and Magnus ( 1 988, Chapter 8, Sec. 8.2 ) . (1)
x
_
1 0 . 6. 1
Matrix Products
(1) X (m
x
nonsingular, A ( m ) , B ( m p) : avec(AX- 1 B) - B' X t - 1 0 AX- I 8vec(X)' m ) nonsingular : i- 1 a vec [ (X - l ) '' j = - "' i - i 0 x- i - 1 ' i = 1 , 2, . . . ' ) (X L... 8vec (X)' ; =0 m)
n x
x
=
X (m
x
(3) X (m
x
(4)
x
(2)
·
n),
rk(X) = n : 8vec[(X ' X )- 1 ] = - (In, + Knn)[(X' X) - 1 0 (X' X) - 1 X ' ] . ovec(X)' X ( m n ) , rk(X) = m : ave;�� ��( 'r 1 l = - ( 1m, + Kmm )[(X X' ) - 1 x 0 (X X' ) - 1 ] .
(5) X ( m
x
m)
nonsingular, A ( n
x
m) ,
B (m
x
m) ,
C (m
x
p) :
1 99
V ECTOR A N D M ATRI X D ERIVATIVES
' BXC) = C' 0 A x- ' B - C' X' B ' X ' - ' 0 A X - ' . avec(Ax(a) avec(X)' ' C) = C' X ' - ' B' 0 A - C' X' - ' 0 AX E X - ' . avec(AX BXb ( ) avec(X)' 1 BX1 C) avec(AX(c) avec(X)' = - C ' x' - ' B ' x'- ' 0 A x - ' - C' x' - ' 1:9 A x - ' Ex - ' . Note: The results of this su b section follow from the chain rule and the results of the previous sections . 10.6.2 (1}
(2}
Kronecker and Hadamard Products
X (m m ) nonsingular, A ( p x q ) : avec(A 0 X - 1 ) - ( / .mp 0 Im ) [vec (A ) 0 X·t- J @ X· - I ] , avec( X ) ' = q <9 A avec(X- 1 0 A) - ( 0 . 0 Ip)[X ,_ 1 0 X, _ 1 0 vee( A)] . ovec( X )' = /m A qm X ( m m ) nonsingular: avec(.\ ... - I ) 0 vee (X ) I) I >I " '61vec ( " ) . (X vee (a) m uvec( v ) X ) 0 vec(X - 1 ) = I , 0vec(X - 1 ) - vec(X )0X '- 1 0X- 1 . avec( (b) m a vee ( ,\' )' ' avec( Xvec(X 1 ) 0 ) (c) avec(X)' = - [X ' - 1 0 x-' 0 vec(X - 1 ) + vec(X - 1 ) 0 X ' - 1 0 X - ' ] . 1 avec (X(d) avec(X)'0 X ) = ( /m 0 J\mm 0 Im )[vec(X- 1 ) 0 Im, - X ' - 1 1:9 x- 1 0 vee( X)]. avec( X 0 x- 1 ) (e) 8vec( X )' = ( /m 0 f{m m 0 Im) x [Im 0 vee (X- 1 ) - vee (X) 0 X , _ 1 0 X- 1 ] . X
x
.. \
-'l
2
1
KA '61
v 1-�
KA '61 ,1.
'
- I ,0,
,1.
'
H A N D BOOK OF M AT RICES
200
(3)
(4)
l 0 x- l ) x fJvec( (f) fJvec(X)' = -( Im 0 f\mm 0 Im ) x [X ' - 1 0 X - 1 0 vec(X-1 ) + vec(X- 1 ) 0 X ' - 1 0 X - 1 ] . X (m m) nonsingular, A (m x m) : fJvec(A 0 X- 1 ) _- fJ vec(X - 1 0 A) -_ _ d mg · ( vee A)(X ' - 1 0 x - 1 ) . 8vec(X )' 8vec(X )' X ( m x m ) nonsingular: 1 fJvec(X0 X ) _ ovec(X 8 x - 1 ) (a) 8vec(X)' 8vec(X)' x
diag(vec X - 1 ) - diag(vec X)(X ' - 1 0 X- 1 ) . I I X ()vee ) - 2 di ag(vec x - I )( X ' - 1 0 x - 1 ). 0 � \"" b ( ) vec(X )' Note: The results of this su b sec tion follow from the chain rule for derivatives and the results of the previous sections . =
=
10.6.3 (1)
Matrix Valued Functions with Vector A rguments
x (m x
Y(.r) ( n x n ) nonsingular: fJvec(Y - 1 ) _ ( yt - l 0 y- 1 /vec(Y) . fJ x' fJ x' ( x 1 ) , Y(x) ( n x n ) nonsingular, Z(x) ( n x p) : fJvec(Y - 1 Z) = ( I 0 y - l ) fJ vec(Z) _ ( z' y ' - l 0 y - l ) fJvec(Y) P fJ x' fJ x' fJx' _ ( m x 1 ) , Y(.r) ( n x n ) nonsingular, Z(.r) (p x n ) : fJvec( Z }" - 1 ) - (Y ' _ 1 0 I fJ vec( Z ) ( y ' - l 0 zy - 1 ) fJvec(Y ) P) fJ x' fJ x' fJ x' (m x 1 ) , Y(.r) , Z(x) ( n x n ) nonsingular: fJvec(Y- 1 z - 1 ) fJ x' ( y ) - ( zt - I 0 y - I Z- I ) fJvec ( Z) . ( zt - y 1 - I 0 y - ) fJvec fJ x' fJ x' 1),
=
(2)
x
(3)
x
m
_
_
·
( 4)
x
=
-
I
I
20 1
V ECTO R A N D M AT RI X DERIVATI V ES (5)
x (x)( ( 1 ) , Y(x)(x) (( U q )V avec( U Y-1 V z-1 W ) ax' m x
x n ,
n x n) n x p) ,
nonsingular,
W (x) (p
Z(x) (p
x r) :
x
p)
nonsingular,
Y(x) ( q) , Z(x) ( p p) nonsingular: a[vec(Y) Q9 vec(Z-1 )] avec (Y1 ) vee( z - l) ax' ux ) (Z ec av ' - vee( y) 0 ( z - l z - l ) ux' ( 7) x ( Y(x) ( ) nonsingular, Z(x) (p q) : a[vec(Y-1) 0 vec(Z)] ax' (6)
x(
m x 1),
x
n x
(9
£l
(9
m x 1),
( 8)
---!l :-: -' -'-
x
n xn
x ( 1 ) , Y(x) ( ) nonsingular, Z(x) (p p) nonsingular: a[vec(Y-1) Q9 vec(Z-1 )] - vee ( y - I ) 0 [( z'- I 0 Z - I ) _a__ve,-_, c -'-(- Z_:--_)] ax' - [(Y '- I C9 y- 1) 8vecax'(Y) ] (S: vec (z -1 ). x ( 1 ) , Y(x) ( q), Z(x) (p p) nonsingular: avec(Y 0 z-1) 0 0 ax' avec(Y) [ vec(z - l ) X ) v . �:�Z 0 ) ] ' ( 1 1 0 Q9 c( - ve Y ) (z Z ) m
x
n x n
x
O J:'
(9)
mx
n x
x
-
( fq
I\pn
OX '
fp )
(9
H A N DBOOK OF M ATRICES
202
( 10) x
( m x 1 ) , Y ( x ) ( n x n ) nonsingular, Z ( x ) ( p x q ) : ovec(Y- 1 <9 Z) ( /n @ I\qn @ fp ) ox'
ovec(Z) _ 1 (Y ) vee @ [ 0 x' - ( (Y ' - t @ Y - t ) ov��( Y ) ) @ vec (Z ) ] . ' nonsingular, Z(x ) (p x p) nonsingular:
x
(11)
x ( m x 1 ) , Y( x ) ( n x n) ovec(Y- 1 @ z - t ) ox'
- ( /n @ 1\pn @ fp )
[vec( Y - 1 ) <9 (( z'- 1 @ Z - 1 /v��Z� ) ) ov Y ) ( l I ' ) @ vec( z- t )] . ��� y Y ) + ( @
x
( 12)
x ( m x 1 ) , Y (x) ( n x n) , Z(x) ( n x n) nonsingular: ovec ( Y 0 Z- 1 ) ovec( z - t 8 Y) ox' ox' . ( vee z _ 1 ) ovec ( Y ) d tag ::::. x' u - diag(vec Y ) (Z' - 1 <9 z- t ) ov�c(Z) . x'
( 13)
x (m x 1 ) , Y ( x ) ( n x n ) nonsingular, Z ( x ) ( n x n ) nonsingular: ovec(Y) ovec(Y- 1 <� z - t ) t t ' 1 <9 z[diag(vec )(Y y - ) ox' ox' . ] + dtag(vec Y - 1 )(Z'- 1 <9 Z - 1 ) ovec(Z) . ox'
The results of this sub section follow from the chain rule for derivatives and the rules of the previous sections. Note:
10.7
C hain Rules and M iscellaneous Jacobian Matrices
denotes a duplication matrix and D:;, its Moore -Penrose inverse (see Section 9 . 5 ) . Lm is an elimination matrix (see Section 9.6).
Reminder: Dm
V ECTO R A N D M ATRIX DERIVATIV ES
203
( 1 ) x ( m x 1 ) , y(x ) ( n x 1 ) , z ( y) (p x 1 ) : oz(y(x)) ox'
(2) X ( m
x n),
Y(X) ( p
x
q) ,
oz(y) oy(x) oy' ox'
Z(Y) ( r x s )
:
8vec Z(Y(X)) 8vec Z(Y) 8vec Y(X) . ovec(Y)' ovec(X) ' 8vec(X)' (3) X ( m x n ) , S open subset of IRm x n : rk (X) = r for all X E S => x + is a differentiable fun ction of X on S and + x
+ x + 0 Un - x + X )] Kmn . (4) X ( m x n ) , Y ( X ) (p x q), S open subset of IRm x n : rk[Y ( X )) r for all X E S => Y ( X ) + is differentiable on S and =
(- y + ' 0 y +
8vec Y(X) + 8vec( X )'
+
+ y+' y + 0 ( I q
(5) X ( m
x m)
:c --::----.:.--::-:,-:...
X (m
x
m)
[(Ip - yy + ) 0 y + y + ' _
y + Y)]Kp
8vec(Y) . )q ovec(X)'
nonsingular:
ovec( X a q; ) ovec(X)' (6}
'
=
det(X) [(vec x - 1 )(ve e X ' - 1 ) ' - x' - 1 0 x - 1 ) .
symmetric nonsingular:
8vech( X a
+ [(vee X - 1 ) (vee X - 1 ) , - X - 1 0 X - 1 ] Dm . ) t(X de Dm ovech ( X )' _
( 7 ) X ( m x m ) lower triangular, nonsingular:
0;::�;���; )
(8}
X
(m x n ) ,
=
det(X)L m [(vec X - 1 )(vec X' - 1 ) ' - X '- 1 0 X - 1 ] L :., .
Y(X) (p
8vec[Y ( X ) a
=
x
p) nonsingular:
8vec( Y ) 1 1 I I 'y )[( Y c ' )'c Y ) ve )( ve Y Y 0 det( . 8vec( X )'
H A N DBOOK OF M ATRIC ES
204
(9)
X
( m x m ) positive definite, Y ( X ) = [Yii ] ( m x m ) lower triangular with Yii > 0 , i = 1 , . . , m , such that X = Y Y ' , that is, Y is obtained by a C holeski decomposition of X : 8vech ( Y ) 8vech ( X )' .
� [D;t; ( Y 0 lm ) L:., t 1 . Note: ( 1 ) and ( 2 ) are just chain rules. (3) is given in �hgnus & Neudecker ( 1 988 , Chapter 8, Sec . 5 ) . ( 4 ) follows from (3) and a chain rule. (5) is established in Magnus & Neudecker ( 1 988, Chapter 8, Sec. 6 ) . (6), ( 7 ) and ( 8 ) follow from ( 5 ) and a chain rule ( see also Magnus ( 1988, Chapter 8, Sec. 8 . 2 , 8 . 3 ) ) . ( 9 ) is, e.g., given in Liit kepohl ( 199 1 , Appendix A , Sec. A . 1 3 ) . 1 0 .8
10.8 . 1
Jacobian Determinant s
Linear Transformations
8vec ( X ) ) ( ( a) det = 1. 8vec( X )' ' (b) det ( 8vec( .": ) ) = ( - l ) mn ( m - 1 )( n - 1 )/4 . 8vec ( ) 8ve c( ,: ) ) ( = ( - l ) mn ( m - l ) ( n - 1 ) /4 . (c) det 8vec( .\ ' ) ' 8vech(X) ) ( ( ( 1. et d 8vech( X )' mn · IR : det ( 8vec(cX ) ) (3) X 8vec(X) ' 8vech(cX ) ) det ( ( ) cm( m + 1 ) / 2 . 8vech( X )
(1) X (m x n) :
,\
2) X m x m)
(m x n),
(4) X m x m ,
'
=
:
c
E
_
-
c
(5) X (m x n), A
E IR :
( 111
x
'
m),
c
=
B (n x n) : o ec ( - B) (de t A) "(det B) "' . de t vee ,\ ) ' (6) X (m x n), A (n x n), B (m x m) : 8vec ( A ,":' B) ( - l ) m n ( m - l )( n - 1 ) / 4 ( de t A ) m ( de t B t . det 8ve c( .\ )'
(
( � �: ) )
=
=
205
V ECTO R A N D M ATRIX DERIVATIVES
( 7 ) X, A =
[a ;j ), B = [b;j ) ( m x m ) lower triangular : ( 8vec h( A X B ) ) m ; rn- i + l = rr a . b d et · 8vech(X)' 1::1 , "
.
(8)
X (m
x
m)
symmetric, A ( m
x m) :
( 8vech ( A' X A) ) = (det A ) m + l . det 8vech( X )' (9)
X ( m x m ) symmetric, A , B ( m eigenvalues of BA - 1 : det
( 10) X ( m
x
x m) ,
det(A)
:/= 0, A 1 ,
.
•
( 8vech (AX A' ± BX B' ) ) 1 = (det A ) m + l rr( ± A; A1 ) > 8vech( X )' I. _ }. m)
symmetric, A =
[a;j ), B = [b;i] (
m
x m)
.
, Am
.
lower triangular:
( 8vech(AX A' ± BX B') ) = IT det > (a;;ajj ± b ii bjj ). 8vech( X ) ' J _} ( 1 1 ) X ( m x m ) symmetric, A , B ( m x eigenvalues of A B - 1 :
m) ,
det(B)
:/= 0, A 1 , . . . , Am
( ov ec h ( A X B ' + B X A ') ) = 2 m d et ( A )( d et B ) m ii d et ( + 8vech( X ) ' •. > "• "; ) ; ( 12 } X , A = [a;j], B = [b;j], C = [c;j], D = [d;j) ( m x m ) lower triangular: \ .
(8vech(AX B + C X D) det 0 vech ( X ) '
( 13 ) X ( m
x
m)
\ .
.
) = II ( aii bii + c;; d11 ) . i?. i
lower triangular, A , B ( m x m) nonsingular:
mII1 ( 8vech( B' X A + A' X' B) ) det(C(il ), det = 2 m det(A)(det B ) m ovech( X ) ' I :: I -
where i
c( i) =
Cj1
...
=1
'
...'m
Cii
are the principal submatrices of C =
[c;j ) = A B - 1 .
-
1'
H A N D BOOK O F M ATRICES
206 ( 14 )
X,
A=
[a ;j ] ( m x n ) , B ( m x m ) , C ( n x n ) :
m 8vec ( A 8 BXC) ) ( n aij · det = ( det B) n (det C) rn ii II o vec ( X )'
( 1 5 ) X ( m x n) ,
det
( 16)
A=
i=l j=l
[a;j] ( n x m ) , B ( n x n ) , C ( m x m ) :
( 8vec(A 8 BX' C) ) 8vec ( X ) '
n m l = ( ) mn( m - 1 )( n - 1 )/ 4 ( de t B ) m ( de t C t II II a;j . i= I j =I X , A = [a ;j], B = [b;j] ( m x n ) : m n 8vec ( A 8 X 8 B) ·b· · · d et II II - i= a , 8vec(X) ' i=J l ;
)
(
( 17) X (m x n),
A=
8vec ( A ( det
_
•; .
[a;j], B = [bij] ( n x m ) :
0
X' 8 B) 8vec( X)'
) = ( - 1 )mn( m- l )(n - 1 )/4 IIn IIrn a · · b · · . i=l i=l
•;
•;
Note: The results of this subsection follow from the rules of the previous
sections of this chapter and the rules for determinants (see also Magnus ( 1988, Chapter 8)). 10.8.2 (1)
Nonlinear Transformations
( Chain rule)
x,
y( x ) , z (y) ( m x
(
1) :
x) d et 8z (y( ) (2)
( Chain rule) X, Y ( X ) , Z ( Y ) ( m
>�
ux'
) = det ( oz (y) ) det ( oy( x ) ) >�
>�
uy'
ux'
.
x n) :
( ( 8vec Y(X) ) ( 8vec Z( Y ) ) 8vec Z ( Y ( X ) ) ) . det det = det 8vec ( X )'
( 3 ) X (m
x m)
8vec( Y )'
symmetric with eigenvalues .\ 1 , .
8vec( X)'
. , A rn :
( 8vech(X i ) ) ·m det = z (det X ) i - 1 II ovech( X ) '
.
"'L > I
J.l k t ,
207
V ECTO R A N D M ATRIX D E RI VATIV ES
wherP
{ -
/-l k l
( 4 ) X = [x1:1]
A�: # At A�:
=
At
(m x m) lower triangular:
( 8vech( X i ) ) det 8vech( X) '
where /-l k l
(5) X, A
if if
( A � - A I }/( A �: - At ) i A �- 1
_
=
{
=
m
· z ( det X ) i - l
( x �k - x i1 ) / ( xu - x u ) . i 1 z x u-
if if
II /-l k l ,
k>l
XU
= X I/
(m x m) symmetric, A 1 , . . . , A m eigenvalues of XA :
( 8vech(XAX) ) det = 2 m det(A )det (,\ ) II ( A; + Aj ) . 8vech( X )'
(6) X = [x;i ] , A = [a;j]
,
.
• >J
(m x m) lower triangular:
8vech(X AX) ) ( det = 2 m det(A)det(X) II (a; ; X;; + ajjXjj ) . 8 vech ( X ) ' .. • >J
(7)
X = [x;j ]
(m x m) lower triangular:
( 8vech (X'X) ) det 8vech( X ) '
( 8vech(X X' ) ) det 8vech(X)'
(8) X (m
X
2 m II xi; , i=l
m
- .
2 m II x 7;' i + l i=
1
x m) nonsingular: det
(9)
m
(8;�:��;):)) = ( - l ) m (det X) - 2m .
(m x m) symmetric nonsingular:
8vech ( X - l ) ) ( det = ( - l ) m ( m + l )/2 (de t X ) - ( m +l ) . 8vech( X )'
( 10)
X
( m x m) nonsingul ar, lower triangular:
8vech ( X - l ) ) ( = ( - l ) m ( m + l ) /2 (det X ) - ( m + l ) . de t 8vech( X)'
H A N D BOO K O F M ATRICES
208
(11) X
(m
X
(m
x
m) symmetric nonsingular: 8vech( X a dj ) = - m ( m + 1 ) /2 ( 1 - m)( det X ) ( m + 1 )( m - 2)/2 . ( 1) det 8vech(X)'
(12)
(
)
x
m) nonsingular, lower triangular: 8vech( X a � ) ( - 1 ) m (m + 1 ) f 2 ( 1 - m)(d et X ) ( m + l ) ( m - 2)/2 . det 8vech( X ) '
(
)
=
Note: Most results of this subsection are given in Magnus ( 1988, Chapter 8 ) . They fol low from rules of the previous sections and t he results for
determinants .
Matrix Val ued Functions of a S calar Variable
10.9
( 1 ) x E IR., A ( x ) = A (2)
(m x
n)
dA = dx
constant:
'( Linearity) x E IR., A ( x ) , B ( x ) ( m x n ) , c 1 , c 2 E
Om x n ·
IR. :
+
d [c 1 A ( x ) c2 B( x )] dA(x) =c 1 dx dx
+ c2
dB(x) dx ·
( 3 ) ( Product rule) .r
E
IR. ,
A ( .r ) (m x n ) , B( x ) ( n x p ) : dA(x) B(x) dX
( 4 ) .r E IR ,
_ 4 ( ) d B( x )
dA(x) B( + dx dx X). A ( x ) ( m x n ) , B( .r ) ( n x p) , C ( x ) (p x q ) : -
'
X
d A ( x ) B( .t· )C( x ) dx d C( x ) dB(x) = A ( x ) H(x) C( x ) + A( x ) dX dX
(5) x E
IR., A ( .r ) ( m x m )
+
dA(x) B ( x )C( x ) . dX
nonsingular:
d.4( x ) - 1 dx (6)
( Ratio rule)
.r E IR.. A ( x ) (m x d B ( x ) A ( .r ) - 1 d .r
m) nonsingular, =
B ( x ) (n
x
m) :
dA(x) dB(x) _ 1 _ _ 1 A(x)- l . A(x ) B( x ) A ( x ) dx dx
V ECTOR A N D M AT R I X D E RI VATIV E S
209
( 7 ) (Generalized i nverse rule) x E IR, A ( x ) ( m x n ) , A ( x ) - some generalized inverse of A ( x ) : A(x)
dA(x)A(x) dx
=
-A(x)A ( x ) -
dA ( x ) x )- A ( x ) . A( dx
Note: The first fou r ru les follow from t he corresponding rules for real valued fu nctions by considering typical elements of t h e matrices i nvolved. Rule ( 5 ) is obtained by apply ing t he product ru le to A A - 1 lm , ( 6 ) follows from ( 5 ) and the pro duct rule and the resu l t in ( 7 ) follows by applying t llf' product rule to A A - A = A and multiplying by A A - from the left . =
11 Polynomials, Power Series and Matrices 1 1. 1
Definitions and Notations
11.1.1
Definitions and Notation Related to Polynomials
Polynomial: For given Po , PI , . . . , Pn E
·
r given P o , PI , . . . , Pn E IR, a function p : lR defined by p( x ) = Po + PI x + · · + Pn x n is a real polynomial .
Real polynomial:
F
o
-
IR
·
Degree of a polynomial: A polynomial p(x ) Pn =/= 0 is said to have degree n .
=
Po +P 1 X + · · +Pn X n with ·
+ p 1 ,\ + · · · +
Root of a polynomial: A number ,\ for which p(,\ ) = po Pn ,\ n = 0 is called a root of the polynomial p( · ) .
Multiplicity of t he root of a polynomial: A polynomial p( x ) = Po + P I X + · · · + Pn X n with distinct roots .\ 1 . . . , A r can be written in the ,
form
p(x )
=
Pn ( X - .\ 1 ) 1: 1 · · · ( x - A r ) k •
=
r
Pn
IJ ( x - .\ ; ) 1: ,
i= I
where the k; are positive integers. ki is called the multiplicity of the root .\; , i = 1 , . . . , r.
Convergent polynomial: A polynomial p( x) = po + p 1 x + · · + pn x n with roots ,\ 1 , . . . , Ar is said to be convergent if j.\; labs < 1 , i = 1 , . . . , r . Divergent polynomial: A polynomial p(x) = po + P I X + · · · + PnX n with ·
roots .\ 1 , . . . , A r is said to be divergent if j .\; labs > i E { l , . . . , r} .
1
for at least
one
·) l "l �
H A N DBOOK O F M ATRJ< ' ES
�
Asymptotically stable polynomial: A real polynomial p( x ) = p0 + PI X + · + p71 x" with roots o 1 + i/31 , . . . , O r + iBr is said to he asymptotically stable if o , < 0, i = 1 , . , r. ·
·
. .
Unstable polynomial: A real polynomial p( x ) = po + p1 x + with roots o 1 + i8 1 , . . . , O r + i/3r is said to be u nstable if at least one i E { 1 . . . r} . .
.
11
11
·+p X o; > 0 for
·
·
L inear polynomial: A polynomial of degree 1 is linear. Monic polynomial: A polynom ial p( .r ) = Po + PI x + Pn = 1 is called monic.
· ·
· + p11x" with
Factor or divisor of a polynomial: If the polynomial p(x) can be written as the product of two polynomials q ( x ) and r(x ) , where q ( x ) has degree > 0 . p( .r ) = q ( x ) r( .r ) , the polynomial q ( x ) is said to be a factor or a divisor of the polynomial p( x ) . Common factor or divisor of polynomials: I f the polynomials p( x ) and q ( x ) both have a factor r ( x ) , that is, t here exist polynomials p l ( x ) and ql ( x ) such that p( x ) = r( x )pl ( x ) and q ( .r ) r( x )q1 ( x ) , r ( x ) is said to be a common factor or divisor of the polynomials p(x ) an d q( x ) . =
Greatest common divisor of two polynomials: Among all monic polynom ials which are common divisors of t he polynomials p( x ) and q( .r ) t he one with the largest degree is said to be the greatest common divisor or factor of p( x ) and q( x ) . ,
Relatively prime o r coprime polynomials: The polynomials p( x ) and q ( x ) are relat i vely prime or coprime if they do not have a common
factor.
Dividing polynomial: A polynomial q ( x ) divides a polynomial p( x ) if q ( x ) is a factor of p( x ) . Least common multiple of two polynomials: The least common mul tiple of the polynomials p(x) and q ( x ) is the polynomial r( x ) with smallest degree such that p(x) and q( x ) divide r·( x ) . Resultant: A function f : (f,m + n + l (f, is called a resultant if f (po , p J , · · . Pm - 1 • Qo , ql , · · . , qn ) i= 0 ¢::::::> p(x ) = Po + P i X + · · · + l + x m and q( x ) = qo + x + · · + q,. x are relatively prime m x 1 Q1 " Pm--+
,
·
polynomials.
POLY N O M I A LS , POWER S E R I ES A N D M ATRICES
11.1.2
213
Matrices Related to Polynomials
Companion matrix: The companion matrix of a monic polynomial p ( x ) = po + PI X + · · · + Pm - I X m - l + x m is the (m x m) matrix 1
or
or -Pm - 1 -Pm - 2
- po
-P I
- Pm - 2
-Pm - 1
0
0
1
0 0
0
0
1
.
.
.
0 0 0
0
1
0
0
0 0
0 0
1
0
0
1
-po
-pi
-Pm - 2
-Pm - 1
1
0
0
1
.
.
.
0 0
0
- po -pl
0 0 0 0
1
or -pi -po
1
0 0 0 0
1
0 0
0
0 -Pm - 2
0 1
-Pm - 1
Sylvester matrix: For the polynomials p ( x ) = po +PI x+ · · · + Pm - l x m- 1 + x m and q(x) = qo + Q1 x + · · · + Qn - t X n - l + Qn X n , the corresponding
Sylvester matrix is 1
0 5=
Pm - 1 1
0
0
Qn
Qn - 1 Qn
0
Pm - 2 Pm- 1
Po PI 1
Qn - 2 qn - 1
0 Po
Pm - 1 Qo Ql
0 0 PI
0
Po
0 0
qo
( (m + n ) x ( m + n ) )
where the upper part consists of n rows and t h e low r p art consi s t s of m r ows For example, for p ( x ) = p0 + p 1 .r + p2 .r 2 + .r 3 and e
.
214
H A N DBOOK O F M ATRICES
q ( .r ) = q 0 + q 1 .r + q2 .r 2 , the corresponding Sylvester matrix is 1
P2 1 Ql Q2
0
S=
92
0 0
Pt P2 qo Ql Q2
0
0
Po PI
Po
Qo Ql
Qo
0
0 0
Hurwitz matrix: The H urwitz matrix corresponding to the polynomial p( X ) = Po + Pt .r + · · · + Pm X m .IS Pm - 1 Pm
H=
Pm- 3 Pm- 5 Pm- 2 Pm- 4 Pm - 1 Pm- 3 Pm Pm- 2 0 Pm- 1
0 0 0
0
..
.
0
P-m+l P-m+2 P-m+3 P-m+4 P- m+5
(m x m)
Po
where Pi = 0 for i < 0. For example, for m = 4 , H=
P3 P4
0 0
Pt P2 P3 P4
0 Po Pt P2
0 0 0 Po
is the Hurwitz matrix corresponding to p( x ) . Derogatory matrix: An ( m x m) matrix A is derogatory if a polynomial p( x ) = Po + p1 x + · + Pr .rr of degree r < m exists such that ·
·
p( A ) = Pa lm + Pt A + · · · + Pr Ar 1 1 . 1.3
=
Om x m ·
Polynomials and Power Series Related to Matrices
Characteristic polynomial: The polynomial defined by p(x ) = det ( x lm - A )
is t he characteristic polynomial of the ( m x m) matrix A . It is denoted by PA C ) . Annihilating polynomial: A polynomial p(x ) = Po + Pt X + · · · + Pn x" is said to annihilate the ( m x m) matrix A if p( A ) = pa lm + Pt A + · · · + PnA"
=
Om x m ·
POLYN O M I A LS, POW ER SERIES A N D M ATRICES
•) �
Minimal polynomial of a matrix: The monic polynomial with minimal degree that annihilates an ( m x m ) matrix A is called the mini mal polynomial of A . It is denoted by QA ( · ) . Matrix polynomial: Given a polynomial p(.r ) = Po + PI X + · · · + Pn X" . th e fu nc ti o n p : (fm x m --> IRm x m ) defined by
p(A ) = Pa lm + P I A + · · · + Pn A " for
(m x m )
matrices A, is a matrix polynomial .
Matrix power series: Given a power series p(x) = 2:::::: � 0 pn .r " . the function p : ([' m x m (or p : IRm x m --> IRrn x m ) defined by p(A )
=
L Pn A"
n=O
for ( m x m) matrices A , is a matrix power series. provided t he infinite sum exists. Exponential function of a matrix: A ( m x m)
exp ( A )
=
1 L -1 A" . co
n=O
Tl .
Sine of a matrix: A ( m x m )
sin(A) = L ( - 1 )" co
n=O
(2n + 1 ) !
A2 n + l .
Cosine of a matrix: A ( m x m)
cos( A )
1 1 .2
=
co
( - 1 )"
n=O
2n.
L
1
A 2" .
Resu lt s Relating Polynomials and Matrices
Reminder: For an ( m x m) matrix A , • •
PA ( x ) = det(xlm - A ) denotes the characteristic polynomial . Q A ( x ) denotes the minimal polynomial.
10 "
216
(1)
H A N D BO O K O F M ATRICES
A ( x ) ,\ E (f : (a) QA (x) is unique . (b) degree[qA (.r)] < (c) p(x) is a polynomial with p(A) = 0 => QA (x) divides p(x). (d) PA (A) = 0 (Cayley-Hamilton theorem). (e) QA (x) divides PA (x). (f) PA ( ,\ ) 0 {::::=> ,\ is eigenvalue of A. (g) QA (,\) 0 {::::=> ,\ is eigenvalue of A . p(.z: ) a polynomial with companion matrix C: The eigenvalues of C are the roots of p( x) . A ( m x m) with distinct eigenvalues .\ 1 , . . . , ,\" having multiplicities m1 , respectively: m
m
,
m.
=
=
(2)
(3)
. . . , mn ,
(4)
and r; is the order of the largest block of the Jordan decomposition of A corresponding to the eigenvalue .\, (see Chapter 6 for the Jordan decomposition) . (.z: A ( m x m ) with distinc t eigenvalues .\ 1 , . . . , .\, , q(x) (.r-.\ 1 ) .\71 ) : A is diagoualizable {::::=> q(A) = 0. =
( 5 ) A ( rn x rn :
) ( a) A is diagonalizable {::::=> all roots of QA(x) have multiplicity 1 . (b) A is diagonalizable {::::=> all linear factors of QA (x) are distinct . (c) A is diagonalizable {::::=> [qA(x) 0 => dqA (x)/dx -:f. 0] ( where dqA(J· )/dx denotes the derivative of QA(.r)). p(x) a monic polynomial: C is a companion matrix of p(x) => p(x) qc(x) pc(x). A ( m x m ) . C companion matrix of PA (x) : (a) A is similar to C {::::=> QA(x) = PA (x). (b) A is similar to C' {::::=> every eigenvalue of A has geometric multiplicity 1 . A , B (rn x rn ) : (a) A , B similar => QA (x) QB(x). ( b) A , B similar => PA (x) = PB (x). (c) P .4. (x) = P B (x) QA (x) = QB (x) => A, B similar . =
(6) (7)
(8)
· · ·
=
=
=
=
P OLYN O M I A LS, POW E R SERIES A N D M ATRICES
217
similar, p(x ) a polynomial: p( A ) = 0 ¢:::::} p ( B ) = 0. QA$B ( x ) is the least common mu ltiple of ( 10 ) A ( m x m ) , B ( n x n ) q A ( X) and q B ( X ) . ( 1 1 ) ( Hurwitz theorem) A real polynomial p( x ) is asymptotically stable if and only if all principal minors of its Hurwitz m atrix are positive. ( 9 ) A, B (m x m )
Note: Most results of this section can be found in Horn & Johnson ( 1 985 ) . The last resu l t is given in Barnett { 1 990) . 1 1 .3
Polynomial Matrices
1 1. 3 . 1
Definitions
Polynomial matrix: An ( m x n ) matrix P ( x ) = [p,j ( .r )] whose typical elements are polynomials Pij ( x ) = Pij ,O + Pij , ! X + · · · + Pij, r ,1 X r '1 , is a
polynomial matrix. A lternative notation :
where r
=
max; , j r;j and
with Pij , k = 0 for k
P1 1 ,k
Pl n , k
Pm l ,k
Pm n ,l:
k = 0, . . . , r,
> r;j .
Real polynomial matrix: A polynomial matrix whose elements are real
polynomials is sai d to be a real polynomial matrix.
Degree of a polynomial matrix: The degree of an ( m x n ) polynomial matrix P ( x ) [p;j ( x)] with typical elements p;; ( x ) = Pij ,O + PlJ. . . 1 x + · . · + p IJ· · ,r,1 x r • is max &. ,J. rIJ· ' where rlJ· is the degree of Pii ( x ) , i = 1 , . . . , m , j = 1 , . . . , n . Equivalently, the degree of P(x) = Po + P1 x + + Pr x r is r if Pr 'f. 0. =
1
·
·
!
·
Polynomial matrix operations: All matrix operations specified in Chapter 1 are defined analogously for polynomial matrices except
where noted otherwise in the following.
Rank of a polynomial matrix: The rank of a polynomial matrix is the
number of columns of the largest submatrix whose determi nant is not identically zero.
H A N DBOOK OF M AT R I C ' ES
218
Eigenvalues or latent roots of a polynomial matrix: The eigenvalues or latent roots of the polynomial matrix P( x ) are the roots of t.lw polynomial det P ( x ) . Latent vector of a polynomial matrix: Let P ( x ) be a polynom ial m atrix w i t h eigenvalue .\ . A vector u such t hat P( .\ ) v = 0 is called a l atent vector of P ( x ) . Regular polynomial matrix: The ( rn x rn) polynomial matrix P ( x ) Po + P1 .1· + · · + P, .I. r is c alled regular i f Pr is nonsingu lar.
=
·
Monic polynomial matrix: The ( m x m) polynomial matrix P( x ) Po + P1 x + · · · + P xr is monic if Pr = lm . r
Unimodular or invertible polynomial
matrix: The
( m x m)
polynomial mat rix P ( .1· ) i s said to be u n i mod u l ar or invertible if det P ( 1.· ) = const ant :f. 0. t h at is. det P ( .l' ) is a const ant fu nc tion .
Left an d right multiples of polynomial matrices: The ( m x r1 ) polynom ial mat rix P ( J' ) sat isfy i ng P ( x ) = Q ( x )T( x ) for polynomial
matrices Q ( J.' ) ( 111 x h ) and T(.l' ) ( h x n ) is called a left m u l t i ple of T( .l' ) and a right m u l tiple of Q ( x ) .
Left quotient and left divisor: Consi der polynomial matri ces P ( .I' )
=
Po + P1 x
+
· · ·
+
Pr .l'r
(m X n).
P, ::j::. 0.
and Q ( x ) = Qo
+
Q1x
+
·
· ·
+
Q.x•
(m x h),
det( Q. ) :f. 0 and
s
< r.
That is, t he latter polynomial is regular. An (h x n) polynomial matrix T( x ) is a left quotient of P ( x ) and Q ( x ) if an ( m x n ) poly nomial matrix R ( x ) w i t h degree less than s exists such t hat P ( x ) = Q ( J.· )T ( J: ) + R ( x ) . The polynomial matrix R(x) is said to be a left remainder . Q ( J.: ) is called a left divisor of P( x ) if R( .l' ) 0 and i n t his c a.se P ( x ) is said to be left divisible by Q ( x ) . _
Right quotient and right divisor: Consider polynomial matrices
and Q(x)
=
Q0 + Q 1 x
+
· · ·
+ Q . x ·'
(m x h),
d e t(Q. ) :f.
0 and
s
< r.
T hat is, the l atter polynomial is regular. An ( m x h ) polynom ial matrix T(.1· ) is a right quot ient of P ( x ) and Q ( x ) if an ( m x n )
219
POLYN O M I A LS, POWER S E RI ES A N D M ATRICES
R( x ) with degree less than s exists such that P ( x ) = T ( x ) Q ( x ) + R( x ) . The polynomial matrix R(x) is said to be a right remainder. Q ( x ) is called a right divisor of P ( x ) if R ( x ) El 0 and in this case P ( x ) is said to be right divisible by Q ( x ) .
polynomial matrix
Common left divisor: A polynomial matrix D(x) is a common left divisor of the polynomial matrices P ( x ) and Q ( x ) if D ( x ) is a left
divisor of both P ( x ) an d Q ( x ) , that is, if polynomial matrices P1 ( x ) and Q 1 ( x ) exist such that P ( x ) = D ( x ) P1 ( x ) and Q ( x ) = D( x ) Q I ( x ) . D(x) is called a greatest common left divisor of P ( x ) and Q ( x ) if for any other common left divisor F( x ) there exists a polynomial matrix B ( x ) such that D ( x ) = F ( x ) B ( x ) . The polynomial mat rices P ( x ) and Q ( x ) are said to be relatively left prime or left coprime if their greatest common left divisors are unimodular.
Common right divisor: A polynomi al matrix D(x) is a common right divisor of the polynom ial matrices P ( x ) and Q ( x ) if D( x ) is a right
divisor of both P ( x ) and Q ( x ) , that is, if polynomial matrices PI ( x ) and Q ! ( x ) exist such that P ( x ) P1 ( x ) D( x ) and Q(x) Q 1 ( x ) D( x ) . D(x) is cal led a greatest common right divisor of P ( x ) and Q ( x ) if for any other common right divisor F ( x ) there exists a polynomial matrix B ( x ) such that D(x) = B ( x ) F ( x ) . The polynomial matrices P ( x ) and Q ( x ) are said to be relatively right prime or right coprime if thei r greatest common right divisors are unimodular. =
=
Skew prime polynomial matrices: The polynomial matrices P ( x ) and
are said to be skew prime if det P ( x ) and det Q(x) are relatively prime polynomials. Q(x)
Elementary operations for polynomial matrices: The following
modifications of a polynomial matrix are called elementary operations: ( i ) i nterchangi ng two rows or two columns, ( ii ) multiplying any row or column by a nonzero number, ( iii ) adding to one row another row multiplied by an arbitrary polynomial , ( iv) adding to one column another column multiplied by an arbitrary polynomial.
Elementary polynomial matrix: An (m X m) polynomial matrix is
elementary if it may be obtained by applying a single elementary operation to lm .
H A N DBOOK OF M ATRICES
220
Equivalent polynomial matrices: Two polynom ial matrices P ( x ) and Q(x) U(x)
are said to be equivalent if unimodular polynom ial matrices and V ( x ) exist such that P ( x ) = U (x ) Q (x ) V (x ) .
Characteristic matrix: The ( m x m) polynomial matrix P ( x ) = x lm - A is the characteristic m atrix of an (m x m) matrix A . Invariant factors or invariant polynomials of a polynomial mat rix: The invariant factors or polynomials of an ( m x m) polynomi al matrix P( x ) of rank r are defined as . lk ( x )
=
dk ( X ) , dk - L ( X )
k = 1 , . . . , r,
w here d0 ( x ) = 1 and dk ( x ) is the monic greatest common div isor of all minors of order k of P ( x ) for k = 1 , . . . , r. Elementary divisors of a polynomial matrix: Let ,\ be a root of an
i nvariant factor of a polynomial matrix P(x ) . Then the linear polynomial p( x ) x - ,\ is an elementary divisor of P(x ) . =
Block companion matrix of a polynomial matrix: The (block ) companion matrix of an ( m x rn ) polynomial matrix P( x ) Po + P1 .r + + Pr _ 1 .r r - L + lm .r r is the ( rrn x tm ) matrix =
·
·
·
...
0
or
0 0 - Pt
or - Pr - L - Pr - 2
fm 0
0 lm
- Po
0 0
0 0
0
0
0 0 0 - Po
- PL
... ...
0 0
0
0
lm 0 - Pr - 2
0 fm - Pr - L
0 lm
0 0
0 0
- Po - PL
0 0
lm 0
0 lm
- Pr - 2 - Pr L
or - Pt - Po
0 0
0 0
lm 0
) ·) 1 ·--
POLYNO M I A LS, POWER SERIES A N D M ATRICES 1 1.3.2
Results for Polynomial Matrices
. valid for The results for matrix operations listed in Chapter 2 rem am polynomial matrices. Here a few special results are given. ( 1 ) P( x) ( m x m )
( a) (b) (c) (d)
(e)
polynomial matrix of degree r : The degree of det P ( x ) < mr. Elementary operations on P ( x ) do not alter rk P ( x ) . Elementary operations on P ( x ) do not alter its invariant factors. P(x) is an elementary polynomial mat-r ix => P(x) is unimodular. P ( x ) is unimodular 1
1 P ( x ) - = --- P(x)a d'� det P (x)
is a polynomial matrix of finite degree. ( 2 ) r > 1 , P(x) Po + P1 x + · · · + Pr xr ( m x m ) polynomial matrix. Pr :f. 0 : P(x) is unimodular :::} det Pr = 0. ( 3 ) P(x) (m x m ) monic polynomial matrix: det P ( x ) iJ ( x ) · i111 ( x ) . where the ik ( x ) are the invariant fac tors of P(x ) . ( 4 ) P ( x ) ( m x m ) monic polynomial matrix of degree r and corresponding block companion matrix C: det ( x lr m - C) = det P ( x ) . (5) P(x) = Po + P1 x + · · · + Pr xr , Q ( x ) = Q o + Q ! x + · · · + Q , x ' ( m x m ) polynomial matrices, r > s : ( a) Q ( x ) is regular => the left ( right) quotients and remainders of P ( x ) and Q(x ) are unique. ( b) A greatest common left (right) divisor of P ( x ) and Q ( x ) is unique up to postmultiplication (premultiplication) by an arbitrary unimodular polynomial matrix. ( 6 ) P ( x ) , Q(x) ( m x m ) polynomial matrices: ( a) P ( x ) , Q ( x ) are unimodular => P(x ) Q ( x ) and Q ( x ) P ( x ) are unimodular. (b) P ( x ). , Q ( x ) are left coprime => P(x)' and Q ( x)' are right. copr i me. (c) P(x ). , Q ( x ) are right coprime => P(x)' and Q ( x ) ' are left copnme. (d) P(x ) , Q ( x ) are left coprime => there exist ( m x m ) polynomial m atrices T(x ) , S(x) such that P(x)T( x ) + Q ( x) S( x) = Im . =
=
·
·
·)•)•) -�-
H A N DBOOK O F M ATRICES
coprime =} there exist ( m x m ) polynomial such that T(x)P(x) + S( x ) Q ( x ) = !., . are left coprime =} rk [ P ( x ) Q ( x )] = m . are right coprime
(e)
P ( .r ) , Q ( .r ) are right matrices T( .r ) , S(.r)
(f) (g)
P ( x ) , Q ( .r ) P ( .r ) . Q ( .r )
:
=}
rk
[ ���� ]
=
m.
are left coprime =} there exists a unimodular polynomial matrix T(x) such that [ P ( x ) : Q ( x )] T( x ) = [Im 0] . ( i ) P ( .r ) , Q ( x ) are right coprime =} there exists a unimodular polynomial matrix T(x) such that
(h)
P(.r ) , Q ( .r )
:
•
T(x )
[
P(x) Q(x) =}
(j )
P ( .r ) , Q ( x ) [ Im : OJ .
are left coprime
(k)
P ( .r ) . Q ( x )
are right coprime =}
(I)
P(x), Q(x) .
copnme.
[ ���\ ]
] = [ lm0 ] .
[P(x) : Q ( x )]
is equi valent to
are skew prime
=}
is equivalent to
[ I� ] .
P(x), Q(x)
are left and right
( 7 ) P(.r ) , Q ( x ) ( m x m ) monic polynomial matrices of degrees r and s. respectively: P(.r ) , Q ( .r ) are skew prime -¢:::::::> the equation X ( .r ) P( x ) + Q ( x ) Y ( x ) Im has a unique solution for polynomial matrices X (.r ) and Y (.z: ) with degree X ( x ) < s and degree Y ( x ) < r . =
( 8 ) P ( .r ) , Q ( x ) ( m x m ) monic polynom ial matrices of respecti vely, C block companion matrix of P ( x ) :
skew prime Qo
-¢:::::::>
0
lm r + Q 1 0 C +
· · ·
+ Q, - 1
0
degrees r and s. P ( x ) , Q ( .r ) are
c• - 1 + Q , 0
C'
is nonsingular. ( 9 ) P(x) = Po + P1 .r , Q ( x ) = Q 0 + Q 1 x ( m x m ) polynomial matrices: ?1 and Q 1 are nonsi ngular =} P ( x ) and Q ( x ) are equivalent. ( 1 0 ) P ( x ) , Q ( x ) ( m x m ) polynomial matrices: P ( x ) , Q ( x ) are equ ivalent -¢:::::::> P( x ) and Q ( x ) have the same invariant factors. ( 1 1 ) (Smith normal ( canonical ) form) P(x) ( m x m ) polynomial matrix, rk P(.r) = r : There exist
POLYNO M I AL S , POW E R SERIES A N D MATRICES
unimodular polynomial mat.rices
U(x)
and
V(x)
223
such that. 0
P(x) = U(x) 0
ir ( X ) 0
where the
i!: ( x )
V(x)
are the invariant factors of
0 P(x).
Note: The results listed in this subsection can be found, e.g. , in Barnet t ( 1 990 ) . For the Smith normal form see also Gantmacher ( 1 959a ) .
App endix A Dictionary of Matrices and Related Terms !!!
Warning
" ' • • •
Man y t erms gzven in t h e following a re defined differe n tly in some of th e m a t rix lzt e rature. The preci5e mean ing of a given t erm in other litera t u re sh ould be checked ca refu lly in e a ch individual case .
Absolute value of a matrix: The absolute value of an ( m x n ) mat rix A = [ a ,1] is defi ned as
l A Iabs
=
[ la;j labs ] =
l a 1 1 l abs la2 1 l abs
l a 1 2 labs l a22 \abs
la in labs la2n labs
l am I l abs
\am2 l abs
\amn \abs
(m xn)
where \ c labs denotes the modulus of the complex number c = c 1 + ic2 defined as \c\ abs = Jci + c� = vee. Here c is the complex conj ugat e of c . ( For the properties of the absolut e value of a matrix see Sf'rt.ion 3.8. ) Addition of matrices: A = [a ;j] ( m x n ) , B = [b;j] ( m x n )
A+B
=
[a ;1 + b;j ] ( m x n ) .
( For the rulf's see Section 2 . 1 ) . Adjoint of a matrix: For m > 2 , the ( m x m ) matri x A a dj = [cof( a,1 )]' is t he adjoint of the (