Induced Representations of Locally Compact Groups

more information - www.cambridge.org/9780521762267

CAMBRIDGE TRACTS IN MATHEMATICS General Editors

´ S, W. FULTON, A. KATOK, B . BOL L OB A F. KIRWAN, P. SARNAK, B. SIMON, B. TOTARO 197 Induced Representations of Locally Compact Groups

CAMBRIDGE TRACTS IN MATHEMATICS GENERAL EDITORS ´ B. BOLLOBAS, W. FULTON, A. KATOK, F. KIRWAN, P. SARNAK, B. SIMON, B. TOTARO A complete list of books in the series can be found at www.cambridge.org/mathematics. Recent titles include the following: 167. Poincar´e Duality Algebras, Macaulay’s Dual Systems, and Steenrod Operations. By D. Meyer and L. Smith 168. The Cube-A Window to Convex and Discrete Geometry. By C. Zong 169. Quantum Stochastic Processes and Noncommutative Geometry. By K. B. Sinha and D. Goswami ˘ 170. Polynomials and Vanishing Cycles. By M. Tibar 171. Orbifolds and Stringy Topology. By A. Adem, J. Leida, and Y. Ruan 172. Rigid Cohomology. By B. Le Stum 173. Enumeration of Finite Groups. By S. R. Blackburn, P. M. Neumann, and G. Venkataraman 174. Forcing Idealized. By J. Zapletal 175. The Large Sieve and its Applications. By E. Kowalski 176. The Monster Group and Majorana Involutions. By A. A. Ivanov 177. A Higher-Dimensional Sieve Method. By H. G. Diamond, H. Halberstam, and W. F. Galway 178. Analysis in Positive Characteristic. By A. N. Kochubei ´ Matheron 179. Dynamics of Linear Operators. By F. Bayart and E. 180. Synthetic Geometry of Manifolds. By A. Kock 181. Totally Positive Matrices. By A. Pinkus 182. Nonlinear Markov Processes and Kinetic Equations. By V. N. Kolokoltsov 183. Period Domains over Finite and p-adic Fields. By J.-F. Dat, S. Orlik, and M. Rapoport ´ ´ and E. M. Vitale 184. Algebraic Theories. By J. Adamek, J. Rosicky, 185. Rigidity in Higher Rank Abelian Group Actions I: Introduction and Cocycle Problem. By A. Katok and V. Nit¸ica˘ 186. Dimensions, Embeddings, and Attractors. By J. C. Robinson 187. Convexity: An Analytic Viewpoint. By B. Simon 188. Modern Approaches to the Invariant Subspace Problem. By I. Chalendar and J. R. Partington 189. Nonlinear Perron–Frobenius Theory. By B. Lemmens and R. Nussbaum 190. Jordan Structures in Geometry and Analysis. By C.-H. Chu 191. Malliavin Calculus for L´evy Processes and Infinite-Dimensional Brownian Motion. By H. Osswald 192. Normal Approximations with Malliavin Calculus. By I. Nourdin and G. Peccati 193. Distribution Modulo One and Diophantine Approximation. By Y. Bugeaud 194. Mathematics of Two-Dimensional Turbulence. By S. Kuksin and A. Shirikyan ¨ 195. A Universal Construction for R-free Groups. By I. Chiswell and T. Muller 196. The Theory of Hardy’s Z-Function. By A. Ivi´c 197. Induced Representations of Locally Compact Groups. By E. Kaniuth and K. F. Taylor 198. Topics in Critical Point Theory. By K. Perera and M. Schechter 199. Combinatorics of Minuscule Representations. By R. M. Green ´ 200. Singularities of the Minimal Model Program. By J. Kollar

Induced Representations of Locally Compact Groups EBERHARD KANIUTH University of Paderborn, Germany

KEITH F. TAYLOR Dalhousie University, Nova Scotia

cambridge university press Cambridge, New York, Melbourne, Madrid, Cape Town, Singapore, S˜ao Paulo, Delhi, Mexico City Cambridge University Press The Edinburgh Building, Cambridge CB2 8RU, UK Published in the United States of America by Cambridge University Press, New York www.cambridge.org Information on this title: www.cambridge.org/9780521762267  C

Eberhard Kaniuth and Keith F. Taylor 2013

This publication is in copyright. Subject to statutory exception and to the provisions of relevant collective licensing agreements, no reproduction of any part may take place without the written permission of Cambridge University Press. First published 2013 Printed and bound in the United Kingdom by the MPG Books Group A catalogue record for this publication is available from the British Library Library of Congress Cataloguing in Publication data Kaniuth, Eberhard. Induced representations of locally compact groups / Eberhard Kaniuth, University of Paderborn, Germany, Keith F. Taylor, Dalhousie University, Nova Scotia. pages cm. – (Cambridge tracts in mathematics ; 197) Includes bibliographical references and index. ISBN 978-0-521-76226-7 (hardback) 1. Locally compact groups. 2. Topological spaces. 3. Representations of groups. I. Taylor, Keith F., 1950– II. Title. QA387.K356 2013 2012027124 512 .25 – dc23 ISBN 978-0-521-76226-7 Hardback

Cambridge University Press has no responsibility for the persistence or accuracy of URLs for external or third-party internet websites referred to in this publication, and does not guarantee that any content on such websites is, or will remain, accurate or appropriate.

We dedicate this book to our wives for their lifetime of support and exceptional patience during the preparation of the manuscript.

Contents

Preface

page ix

1

Basics 1.1 Locally compact groups 1.2 Examples 1.3 Coset spaces and quasi-invariant measures 1.4 Representations 1.5 Representations of L1 (G) and functions of positive type 1.6 C ∗ -algebras and weak containment of representations 1.7 Abelian locally compact groups 1.8 Notes and references

1 1 6 12 21 27 35 39 44

2

Induced representations 2.1 Inducing from an open subgroup 2.2 Conditions for irreducibility of induced representations 2.3 The induced representation in general 2.4 Other realizations and positive definite measures 2.5 The affine group and SL(2, R) 2.6 Some basic properties of induced representations 2.7 Induction in stages 2.8 Tensor products of induced representations 2.9 Frobenius reciprocity 2.10 Notes and references

45 46 51 61 70 80 87 96 101 107 112

3

The imprimitivity theorem 3.1 Systems of imprimitivity 3.2 Induced systems of imprimitivity 3.3 The imprimitivity theorem 3.4 Proof of the imprimitivity theorem: the general case 3.5 Notes and references

114 114 120 125 128 138

vii

viii

Contents

4

Mackey analysis 4.1 Mackey analysis for almost abelian groups 4.2 Orbits in the dual of an abelian normal subgroup 4.3 Mackey analysis for abelian normal subgroups 4.4 Examples: some solvable groups 4.5 Examples: action by compact groups 4.6 Limitations on Mackey’s theory 4.7 Cocycles and cocycle representations 4.8 Mackey’s theory for a nonabelian normal subgroup 4.9 Notes and references

140 141 145 154 162 169 173 177 184 201

5

Topologies on dual spaces 5.1 The inner hull–kernel topology 5.2 The subgroup C ∗ -algebra 5.3 Subgroup representation topology and functions of positive type 5.4 Continuity of inducing and restricting representations 5.5 Examples: nilpotent and solvable groups 5.6 The topology on the dual of a motion group 5.7 Examples: motion groups 5.8 The primitive ideal space of a two-step nilpotent group 5.9 Notes and references

203 204 214 223 230 235 244 253 258 266

6

Topological Frobenius properties 6.1 Amenability and induced representations 6.2 Basic definitions and inheritance properties 6.3 Motion groups 6.4 Property (FP) for discrete groups 6.5 Nilpotent groups 6.6 Notes and references

269 270 277 282 287 294 303

7

Further applications 7.1 Asymptotic properties of irreducible representations of motion groups 7.2 Projections in L1 (G) 7.3 Generalizations of the wavelet transform 7.4 Notes and references

305 305 310 329 332

Bibliography Index

333 340

Preface

Locally compact groups arise in many diverse areas of mathematics, the physical sciences, and engineering and the presence of the group is usually felt through unitary representations of the group. This observation underlies the importance of understanding such representations and how they may be constructed, combined, or decomposed. Of particular importance are the irreducible unitary representations. In the middle of the last century, G. W. Mackey initiated a program to develop a systematic method for identifying all the irreducible unitary representations of a given locally compact group G. We denote the set of all unitary equivalence classes of irreducible unitary representations  Mackey’s methods are only effective when G has certain restrictive of G by G. structural characteristics; nevertheless, time has shown that many of the groups that arise in important problems are appropriate for Mackey’s approach. The program Mackey initiated received contributions from many researchers with some of the most substantial advances made by R. J. Blattner and J. M. G. Fell.  as a topological space. At the Fell’s work is particularly important in studying G core of this program is the inducing construction, which is a method of building a unitary representation of a group from a representation of a subgroup. The main goal of this book is to make the theory of induced representations accessible to a wider audience. As the book progresses, we provide a large number of examples to illustrate the theory. A few particular groups reappear at various stages in the development of the material as more and more can be said about them. We have written the book with the assumption that the reader will be familiar with the basics of harmonic analysis, the theory of unitary representations, and C ∗ -algebras. In the first chapter, we have gathered together the components most necessary for the main body of the book. We present these basic results, largely without proof, to orient the reader and establish notation. We recognize that not all readers will be completely familiar with all that is quickly covered in ix

x

Preface

Chapter 1, but we believe that the book can still serve as a useful reference for such readers mainly due to the variety of worked examples. Graduate students learning about induced representations here will want to have the standard references we mention in the first chapter at hand as they work through the later chapters. If H is a closed subgroup of a locally compact group G and π is a representation of H , then the induced representation indG H π is a representation of G. (Throughout this book, all representations of groups are unitary representations so we typically drop the word unitary.) In Chapter 2, we first define indG H π in the case where H is an open subgroup of G. In that case, the construction of the Hilbert space on which indG H π acts is particularly easy and the reader can concentrate on the algebraic manipulations to develop an intuitive feel for the inducing construction. Moreover, this enables us to quickly get to results of substance applicable to discrete groups such as the free group on two generators where, in Example 2.15, we construct a family of irreducible representations. After defining the induced representation in general, we provide several of the commonly used realizations and simplifications that occur in special cases such as inducing from the normal factor in a semidirect product group. Much of the rest of Chapter 2 is devoted to establishing the basic computational properties of the inducing construction, such as the vital induction in stages theorem. The pivotal theorem of this book is the imprimitivity theorem, which is established in Chapter 3. Again, we prove it first in the open subgroup case, where the analytical details are straightforward, to illustrate the main strategy of the proof. Our proof in the general case is an elaboration of a proof given by Ørstedt [120]. With the imprimitivity theorem available, in Chapter 4 we turn to developing  for a the systematic procedure, known as Mackey analysis, for constructing G given locally compact group G. In order for this procedure to work, G must  is understood, the orbit structure have a closed normal subgroup N such that N  must be well behaved, and stability subgroups under the action of G on N (considered as subgroups of G/N) arising in this action must have a wellunderstood representation theory. The concepts involved simplify when N is abelian and simplify even more when G is the semidirect product of an abelian N and a group H acting on N . We begin Chapter 4 by developing Mackey analysis for groups having an abelian subgroup of finite index; without loss of generality, we can take the abelian subgroup to be normal. The value in looking at this elementary case is that the role of the orbit structure in the dual of N becomes clear. With that in mind, we turn to the general situation of a closed normal abelian subgroup

Preface

xi

 There N of G and carefully study the orbit space formed by G acting on N. is a technical concept called Mackey compatibility for the subgroup N within G. It is actually fairly easy to recognize whether a particular N is a Mackeycompatible subgroup of a given G. When this happens, Theorem 4.27 provides a  The objects appearing in this parametrization are easiest parametrization of G. to deal with when G splits as a semidirect product of the normal abelian subgroup N and another locally compact group H acting on N . Indeed, we are now able to provide two sections of examples where the analysis works perfectly. We believe that these worked out examples will be one of the valuable aspects for many readers. We also introduce some examples to illustrate the limitations when the abelian factor is not Mackey compatible in a semidirect product. If G has a substantial closed normal abelian subgroup N but does not split as a semidirect product, then so-called cocycle representations must be used in the analysis. We briefly present the details and an illustrative example. The final section of Chapter 4 deals with Mackey analysis in the case that the relevant closed normal subgroup N is not abelian. The treatment necessarily requires greater sophistication, with C ∗ -algebraic techniques and the orbit structure of actions on non-Hausdorff topological spaces. Nevertheless, Theorem 4.65 is established as the generalization of Theorem 4.27 to the case of a nonabelian N .  in those Tools for studying the topological structure of the dual space G, cases where Mackey analysis is successful, are developed in Chapter 5. The presentation of the main theorems follows the original proofs due to Fell. L. W. Baggett made significant contributions to understanding the topology of dual spaces and our treatment of generalized motion groups follows [2]. We hope that the extensive number of examples in Chapter 5 will contribute to more researchers taking advantage of the topological tools available in studying dual spaces. Chapters 6 and 7 illustrate some of the different ways in which the theory  can be used to of induced representations and knowledge of the topology of G investigate other mathematical phenomena. We have included some topics from areas in which we have been involved personally, so these chapters certainly do not represent even a major sampling of the varied implications of the content of the earlier chapters. Chapter 6 is devoted to an exploration of topological versions of Frobenius properties generalizing the Frobenius reciprocity theorems of finite and compact groups. Chapter 7 explores the asymptotic behavior of the coefficient functions of the irreducible representations of motion groups and methods for constructing projections in the Banach ∗-algebra L1 (G). In both these applications, we exploit the explicit structure of induced representations.

xii

Preface

Chapters 1 to 4 are most useful for the researcher wishing to learn the basic techniques of induced representations and applying them to construct the duals of particular groups, while later chapters are intended more for specialists and to give an indication of the varied applications. For this reason, the exposition is more expansive in the first part of the book. For a graduate course in representation theory, a portion dedicated to induced representations could be supported by Chapters 2, 3, and 4. Some of the topics in this book have, of course, been covered in other monographs. For example, Mackey’s [105] provides the core of his theory while [104] and [106] are overviews which draw deep connections between the theory of induced representations and other areas of science. The books by Gaal [57], Barut and Raczka [15], and Fabec [39] each introduce some of the basic theory of induced representations and each has its own areas of focus. We have been very much influenced by the well-paced book by Folland [55] and the monumental volumes of Fell and Doran [53, 54]. In terms of level, this book lies between [55] and [53, 54]. The reader who is familiar with [55] can move quickly through our first three chapters, perhaps picking out some topics of Chapter 2 that are not touched on in [55]. However, much of the rest of our book is beyond the scope of [55]. Fell and Doran [53, 54] develop the general theory of Banach ∗-algebraic bundles from which the core theorems of this book can be extracted, but the task can be daunting for those new to the area. We have chosen to keep the focus clearly on the representation theory of locally compact groups and there are significant parts of this book which have not appeared in any monograph. There are important classes of locally compact groups where either Mackey analysis is not effective or other methods provide more detailed information. Harish-Chandra, working in parallel with Mackey, developed a comprehensive approach to the representation theory of semisimple Lie groups. An excellent introduction to this theory is Knapp [93]. Kirillov showed that there is a bijective correspondence between the coadjoint orbits in the vector space dual of the Lie  This algebra of a connected and simply connected nilpotent Lie group G and G. forms the basis for a detailed harmonic analysis on nilpotent Lie groups. Often the Kirillov construction and Mackey analysis can be used together in the study of a nilpotent Lie group. Corwin and Greenleaf [33] provides an introduction to the representation theory of nilpotent Lie groups. Portions of the material in this book have been used by one or the other of us in graduate courses or seminars at the University of Paderborn, Technical University of Munich, the University of Saskatchewan, or Dalhousie University. We are grateful to those who attended these lectures for their feedback. The desire to formulate a more comprehensive manuscript on induced

Preface

xiii

representations grew out of our long-time collaborations. Our visits back and forth across the Atlantic for research purposes and the writing of this book have been supported by grants from NSERC Canada, the University of Paderborn, and Dalhousie University We thank the staff of Cambridge University Press for their support during the process of bringing this work to completion. Eberhard Kaniuth Keith F. Taylor

1 Basics

This chapter provides the definitions and many of the basic properties of the objects of abstract harmonic analysis, including locally compact groups, their representations, and various algebras associated with both the groups and their representations. Besides providing the notational conventions used throughout the book, the necessary concepts are organized in a manner useful for the development of the theory of induced representations. For most of the propositions and theorems, we do not provide proofs as these are generally known and accessible in existing monographs. In Section 1.8, we provide a brief guide to the existing literature for the reader who seeks a more comprehensive treatment of a topic. However, we do provide full proofs in Section 1.3, which contains the tools for analysis on coset spaces that may not be so well known but are essential in defining induced representations and proving many of the key theorems.

1.1 Locally compact groups A topological group G is a set with the structure of both a group and a topological space such that the group product is a continuous map from G × G into G and the group inverse is continuous on G. The group product of x and y in G will be denoted multiplicatively as xy and the inverse of x is x −1 except in a few specific cases such as the group of integers or the real numbers. In general, the identity element is denoted by e. If y ∈ G is fixed, then each of the maps Ry : x → xy, Ly : x → y −1 x, and x → x −1 are homeomorphisms of G. When the topology on G is Hausdorff and locally compact, we call G a locally compact group. The general theory of abstract harmonic analysis is 1

2

Basics

highly developed for locally compact groups and this is the class of groups we are interested in here. We will use uppercase Latin letters such as G, H , N, K, or A to denote locally compact groups and lowercase Latin letters to denote their elements with exceptions such as f and g which will denote functions as necessary. Fix a locally compact group G. For A, B ⊆ G, let AB = {xy : x ∈ A, y ∈ B} and A−1 = {x −1 : x ∈ A}. The set A is called symmetric if A−1 = A. Also, for a natural number k, let Ak = {x1 · · · xk : xj ∈ A, 1 ≤ j ≤ k}. If A and B are compact, then so is AB, and if one of the sets A or B is open, then so is AB. If A is compact and B is closed, then AB and BA are closed. If X is a locally compact Hausdorff space, a positive Borel measure μ on X is called regular if, for any Borel subset E of X, μ(E) = inf{μ(U ) : E ⊆ U, U open} = sup{μ(K) : K ⊆ E, K compact}. A complex Borel measure ν on X is regular if its total variation |ν| is regular. A Radon measure on X is a positive Borel measure on X such that μ(K) < ∞, for any compact set K ⊆ X, μ(E) = inf{μ(U ) : E ⊆ U, U open}, for any Borel subset E of X, and, for every open U ⊆ X, μ(U ) = sup{μ(K) : K ⊆ U, K compact}. If μ is a σ -finite Radon measure, then μ is regular. Returning to a locally compact group G, the existence of a translationinvariant Radon measure on G is of fundamental importance. A Borel measure μ on G is called left invariant (respectively, right invariant) if μ(xE) = μ(E) (respectively, μ(Ex) = μ(E)), for any x ∈ G and Borel subset E of G. The existence and uniqueness of a nonzero left-invariant Radon measure for G is sufficiently significant that we formulate the statement carefully. Theorem 1.1 Let G be a locally compact group. Then there exists a nonzero left-invariant Radon measure μG . It satisfies μG (U ) > 0 for any nonempty open subset U of G. If ν is any nonzero left-invariant Radon measure on G, then there is a constant c > 0 such that ν = cμG . Such a measure μG is called a left Haar measure on G. It is understood that a choice has been made out of the family {cμG : c > 0}. Usually this choice is not made explicit, but if there is a distinguished compact neighborhood V of e, we may assume that μ(V ) = 1. For example, if G is compact itself, we assume

1.1 Locally compact groups

3

μG (G) = 1. If G is an infinite group equipped with the discrete topology, we may assume that μG ({e}) = 1. Then μG is simply a counting measure (use the left invariance). Of course there also exists a right-invariant Radon measure on G with the same kind of uniqueness, a right Haar measure. In fact, the homeomorphism x → x −1 interchanges left and right Haar measures (ν(E) = μG (E −1 ), for all Borel subsets E of G, defines a right Haar measure on G). We have chosen to use consistently left Haar measures. Whenever we are working with a locally compact group G, we will assume without mentioning it that a left Haar measure is chosen. Actually, we will rarely use the notation μG . If A is a measurable subset of G, then |A| will denote μG (A). If f ∈ Cc (G), the space of continuous complex-valued functions on G with compact support, then f isintegrable with respect to μG and we usually write  f (x)dμ G (x) simply as G f (x)dx. Indeed, we use the same simplification G for any kind of function f on G (non-negative measurable, Haar integrable, or even vector-valued versions of integrability) for which G f (x)dx makes sense. The left invariance of μG implies that G f (yx)dx = G f (x)dx, for + any  y ∈ G. Note that if f ∈ Cc (G) = {g ∈ Cc (G) : g ≥ 0} and f = 0, then G f (x)dx > 0. On the other hand, if we fix a y ∈ G and define a new measure ν by ν(E) = μ(Ey), for all Borel subsets E of G, then ν is a left-invariant Radon measure on G that is positive on nonempty open sets. Thus, there is a positive constant G (y) so that μ(Ey) = ν(E) = G (y)μG (E), for every Borel subset E of G. This gives a change of variables formula to use in integrals:   f (x)dx = G (y) f (xy)dx, G

G

for any function f where the integral makes sense and for any y ∈ G. Letting y vary, y → G (y) is a continuous homomorphism of G into R+ , the multiplicative group of positive real numbers. It is called the modular function of G. The modular function enables a change of variables by inversion:   −1 f (x )dx = f (x)G (x −1 )dx. G

G

If G ≡ 1 on G, that is, if every left Haar measure is also right invariant, then G is called unimodular. Of course, if G is abelian, then right and left translations are the same, and if G has the discrete topology, then the counting measure is both left and right invariant. Also, if G is compact, then G (G) is a compact subgroup of R+ , so it must be trivial. Thus, each of these classes of

4

Basics

groups, abelian, discrete, or compact, is contained in the class of all unimodular groups. Nevertheless, one frequently encounters nonunimodular groups, and the modular function and functions related to it play an important role in later chapters. For unimodular groups, the left Haar measure is also right invariant and we usually just refer to the Haar measure rather than the left Haar measure. The Lebesgue space, Lp (G, μG ), is denoted simply Lp (G) for 1 ≤ p ≤ ∞. Left and right translation of Lp -functions is continuous: given f ∈ Lp (G) and  > 0, there exists a neighborhood U of e in G such that Ly f − f p <  and 1 p p <  for all y ∈ U . For f ∈ L (G) and g ∈ L (G), the integral  Ry f − f −1 G f (y)g(y x) dy exists for almost all x. Therefore one can define  f ∗ g(x) = f (y)g(y −1 x) dy, (1.1) G

for any x ∈ G such that the right-hand side of (1.1) exists. The resulting measurable function, f ∗ g, is called the convolution of f and g. Then f ∗ g ∈ Lp (G) and ||f ∗ g||p ≤ ||f ||1 ||g||p . When p = 1, this implies that L1 (G), equipped with convolution as multiplication, is a Banach algebra. For f ∈ L1 (G) and x ∈ G, let f ∗ (x) = G (x −1 )f (x −1 ).

(1.2)

Then ||f ∗ ||1 = ||f ||1 and f → f ∗ is an involution on L1 (G). Proposition 1.2 Let G be a locally compact group. Then L1 (G) is a Banach ∗-algebra when equipped with the convolution (1.1) and involution (1.2). If G is nondiscrete, L1 (G) does not have an identity. However, it always possesses a two-sided approximate identity in Cc (G), which can be constructed as follows. Let U be a neighborhood basis at e in G, and for  each U ∈ U choose fU ∈ Cc+ (G) such that f (x −1 ) = f (x) for all x ∈ G, G f (x)dx = 1, and with compact support contained in U . Then, for any g ∈ Lp (G), 1 ≤ p < ∞, fU ∗ g − g p → 0 and g ∗ fU − g p → 0 as U → {e}. The Hilbert space structure of L2 (G) is also fundamental. The inner product on L2 (G) is given by  f, g = f (x)g(x)dx, G

for f, g ∈ L (G). If X is a locally compact Hausdorff space, then M(X) denotes the space of regular complex Borel measures on X equipped with the total variation  norm and the pairing (g, μ) = X g(t) dμ(t) identifies M(X) with C0 (X)∗ , the Banach space dual of C0 (X). If X = G, a locally compact group, then M(G) 2

1.1 Locally compact groups

5

can be equipped with a convolution product. For μ, ν ∈ M(G), there is a unique μ ∗ ν ∈ M(G) such that    ϕ(x) d(μ ∗ ν)(x) = ϕ(xy) dμ(x) dν(y), G

G

G

for all ϕ ∈ C0 (G), and we have  μ ∗ ν ≤ μ · ν .  If δx denotes the point mass at x ∈ G, then ϕ(y)d(μ ∗ δ )(y) = x G G Rx ϕ(y)dμ(y) and   −1 ϕ(y)d(δ ∗ μ)(y) = L ϕ(y)dμ(y). Moreover, for μ ∈ M(G), define x G G x ∗ ∗ −1 μ ∈ M(G) such that μ (E) = μ(E ), for any Borel E ⊆ G. With this structure M(G) is a Banach ∗-algebra with identity δe , called the measure algebra of G. For each f ∈ L1 (G), there is a measure μf ∈ M(G) such that dμf (x) = f (x) dx. This embeds L1 (G) as a closed two-sided ideal in M(G). Indeed, if ν ∈ M(G) and f ∈ Lp (G), then the function  f (x −1 y)dν(x) ν∗f :y → G

belongs to Lp (G) and, when p = 1, satisfies   ϕ(y)d(ν ∗ μf )(y) = ϕ(y)(ν ∗ f )(y)dy, G

G

for all ϕ ∈ Cc (G). Similarly, f ∗ ν is defined and μf ∗ ν = f ∗ ν, for f ∈ L1 (G) and ν ∈ M(G). A closed subgroup H of G is locally compact with the relative topology. If H is an open subgroup of G, then H is automatically closed in G since G \ H is the union of the open cosets xH, x ∈ H . If H is a closed subgroup of G, then μG (H ) > 0 if and only if H is open in G. Thus, when H is open, μH (E) = μG (E), for Borel subsets E of H , defines a Haar measure on H . However, if H is not open in G, then the relationship between the left Haar measure of H and that of G is not straightforward. Let H be a closed subgroup of G and let G/H = {xH : x ∈ G}, the space of left H -cosets. Let q : G → G/H, q(x) = xH, be the natural mapping. Then G/H is a locally compact Hausdorff space when given the quotient topology. Thus, q is an open as well as a continuous mapping. If H is a normal closed subgroup of G, then G/H carries the structure of a group and hence is a locally compact group, the quotient group modulo H . The map q is then the quotient homomorphism. If M is some topological group with identity element eM and ψ : G → M is a continuous homomorphism, then N = {x ∈ G : ψ(x) = eM } is a closed normal  : G/N → M subgroup of G. There is a unique injective homomorphism ψ

6

Basics

 ◦ q = ψ, and ψ  is continuous since q is open. Suppose that ψ is such that ψ  is a topological surjective, M is locally compact, and G is σ -compact. Then ψ  isomorphism. However, ψ need not be open when G is not σ -compact. It is sometimes useful that every compact subset C of G is contained in an open σ -compact subgroup of G. In fact, choose a compact symmetric n neighborhood V of e in G and let A = C ∪ C −1 ∪ V . Then H = ∪∞ n=1 A is an open σ -compact subgroup. It is common for a locally compact group to be naturally acting on some other topological space. If is a topological space and G is a locally compact group, a left action of G on is a continuous map, (x, ω) → x · ω, of G × → that satisfies e · ω = ω and (xy) · ω = x · (y · ω), for all x, y ∈ G, ω ∈ . When we have such an action, for ω ∈ , the G-orbit of ω is G(ω) = {x · ω : x ∈ G} and the stabilizer of ω is Gω = {x ∈ G : x · ω = ω}. Since Gω is clearly a subgroup of G, we often refer to such a Gω as the stability subgroup associated with ω. If is at least a T1 space, then stability subgroups are closed. Note that yGω → y · ω is a bijection of G/Gω with G(ω). Moreover, for y ∈ G and ω ∈ , Gy·ω = yGω y −1 , so stability subgroups associated with two points in the same G-orbit are conjugate in G.

1.2 Examples The examples in this section serve the dual purpose of not only providing an idea of the nature of locally compact groups and their variety, but also introducing some of the particular groups or classes of groups that will be used later in the text. Example 1.3 The set R of real numbers with addition and the usual topology has already been mentioned as a locally compact group. The integers Z form a closed subgroup of R. Indeed, for every a ∈ R, Za = {ka : k ∈ Z} is a closed subgroup of R, and any proper closed subgroup of R is of this form. Let R∗ = {a ∈ R : a = 0} and R+ = {a ∈ R : a > 0}. Equipped with multiplication of real numbers, both R∗ and R+ are locally compact groups with 1 as identity and a −1 = 1/a as the inverse of a generic element a. Note that R+ is an open subgroup of R∗ . If C is the field of complex numbers, then the circle group, T = {z ∈ C : |z| = 1}, is a compact group under multiplication. It is an exceptionally important group for harmonic analysis. If a > 0, then ψa (t) = exp(2π it/a), for t ∈ R, defines a continuous homomorphism of R onto T. Since Za = ker ψa , a : t + Za → exp(2π it/a) identifies R/Za with T. ψ

1.2 Examples

7

The Haar measure μR of R with μR ([0, 1]) = 1 is Lebesgue measure and the Haar measure on T is given by μT (E) = μR (ψ1−1 (E) ∩ [0, 1)), for any Borel subset E of T. That is, for any non-negative measurable function f on T,   f (z)dz = f (exp(2π it))dt, T

[0,1)

where the integral on the right is a Lebesgue integral. It is easiest to describe the Haar measure on R∗ by showing the formula for invariant integration. If f is a non-negative measurable function on R∗ , then  da R∗ f (a) |a| satisfies   da da f (ba) f (a) = , ∗ ∗ |a| |a| R R ∞ is integration with respect to the Haar for all b ∈ R∗ . Therefore, 0 f (a) da a measure on R+ . We will often specify a left Haar measure on a given locally compact group G by writing out the expression for integration with respect to the measure in a convenient parametrization of the elements of G. Example 1.4 Let n ∈ N and G1 , G2 , . . . , Gn be locally compact groups. Then the Cartesian product, G1 × G2 × . . . × Gn , also denoted nj=1 Gj , is a group n  Gj when given the coordinatewise operations. With the product topology, j =1

is a locally compact group called the product group. The left Haar measure on n  Gj is the product of the left Haar measures on the groups Gj . Thus, if f is j =1

a non-negative measurable function on G = 







f (x)dx = G

G2

j =1

Gj ,

f (x1 , x2 , . . . , xn )dxn . . . dx2 dx1 .

... G1

n 

Gn

If Gj = H, a fixed locally compact group, for j = 1, . . . , n, then

n  j =1

Gj is

denoted H n . Thus, we have Rn , Zn , Tn , and combinations Rk × Zl × Tm , for any non-negative integers k, l, and m. Example 1.5 We can equip R × R+ with a different multiplication. For (b1 , a1 ), (b2 , a2 ) ∈ R × R+ let (b1 , a1 )(b2 , a2 ) = (b1 + a1 b2 , a1 a2 ). Notice that (0, 1)(b, a) = (b, a)(0, 1) = (b, a) and (b, a)(−a −1 b, a −1 ) = (−a −1 b, a −1 )(b, a) = (0, 1). Thus, this multiplication endows R × R+ with

8

Basics

a group structure, and the resulting group will be denoted Gaff . The operations of multiplication and inversion are clearly continuous for the product topology. Thus Gaff is a locally compact group. For (b, a) ∈ Gaff , define a transformation of R by (b, a) · x = ax + b, for all x ∈ R. This is an affine transformation of R, and every orientation-preserving affine transformation of R arises this way. Moreover, this action is consistent with the group product in Gaff . The group Gaff is called the affine group or, often, the ax + b group. We identify the left Haar measure on Gaff by providing a left-invariant integration formula. For a non-negative measurable function f on Gaff ,    f (z)dz = f (b, a)a −2 da db. R

Gaff 

R+



One can check that, for any (b , a ) ∈ Gaff ,     f ((b , a  )(b, a))a −2 da db = f (b, a)a −2 da db. R

R

R

R

On the other hand,       −2 f ((b, a)(b , a ))a da db = f (b + ab , aa  )a −2 da db R

R

R

= a

R

  R

f (b, a)a −2 da db. R

Thus, Gaff is nonunimodular and Gaff (b, a) = a −1 . The previous example is a special case of a fruitful technique for constructing new locally compact groups from given ones. This is the semidirect product construction, which we present in detail because many of our examples dealt with later in the book arise in this manner. Let N and H be locally compact groups. Let Aut(N ) denote the group of automorphisms of N . An automorphism of N is a topological group isomorphism of N with itself. Suppose that there is a homomorphism α : h → αh of H into Aut(N) such that (n, h) → αh (n) is continuous from N × H to N. We use these data to form a locally compact group denoted N α H, or simply N  H if the homomorphism α is understood. As a set and topological space, N  H = N × H. The group product of (n1 , h1 ) with (n2 , h2 ) in N  H is given by (n1 , h1 )(n2 , h2 ) = (n1 αh1 (n2 ), h1 h2 ). One checks easily that this product is associative, that (eN , eH ) serves as the identity, where eN and eH are the identities of N and H , respectively, that (n, h)−1 = (αh−1 (n−1 ), h−1 ), and that the group operations are continuous. In

1.2 Examples

9

short, N  H is a locally compact group, called the semidirect product of N and H . To find the left Haar integral on N  H, one uses those on N and H together with a factor which records the amount by which αh scales the left Haar integral of N. More precisely, fix h ∈ H and define a measure μhN on N by μhN (E) = μN (αh (E)), for Borel subsets E of N . Since μhN (nE) = μN (αh (nE)) = μN (αh (n)αh (E)) = μN (αh (E)) = μhN (E), μhN is a left Haar measure on N . Thus, there exists δ(h) > 0 so that μhN (E) = δ(h)μN (E). Then δ : H → R+ is a continuous homomorphism and   f (x)dx = δ(h) f (αh (x))dx, N

N

for any non-negative measurable function f on N . Now the left Haar integral of any non-negative measurable function f on N  H is given by    f (n, h)d(n, h) = f (n, h)δ(h)−1 dndh. NH

H

N

The reader should check the left invariance. We will compute the modular function NH . Let (m, k) ∈ N  H. Then    f ((n, h)(m, k))d(n, h) = f (nαh (m), hk)δ(h)−1 dndh NH H N   f (αh (αh−1 (n)m), hk)δ(h)−1 dndh = H N   = f (αh (nm), hk)dndh H N = f (αhk−1 (n), h)N (m−1 )H (k −1 )dndh H N f (n, h)δ(kh−1 )N (m−1 )H (k −1 )dndh = H N  −1 = δ(k)(N (m)H (k)) f (n, h)d(n, h). NH

This shows that, for (m, k) ∈ N  H, NH (m, k) = N (m)H (k)δ(k)−1 .  = {(eN , h) : h ∈ H }, then N  and H   = {(n, eH ) : n ∈ N} and H If we define N     are closed subgroups of N  H that satisfy N ∩ H = {e} and N H = N  H,  is normal in G. Moreover, for h ∈ H and n ∈ N, and N (eH , h)(n, eH )(eH , h)−1 = (αh (n), eH ).

10

Basics

In general, if G is a locally compact group, we are sometimes able to find two closed subgroups N and H such that N is normal in G, N ∩ H = {e} and N H = G. In such a case we can view G as the semidirect product of N and H as follows. For h ∈ H, define αh on N by αh (n) = hnh−1 , for all n ∈ N. Since N is normal, αh is an automorphism of N and h → αh is a homomorphism of H into Aut(N ). Also (n, h) → αh (n) is continuous from N × H to N. Thus, we can form the semidirect product N α H, and the map (n, h) → nh is a topological isomorphism between N α H and G. Example 1.6 Let A ∈ M(n, R), the algebra of real n × n-matrices. Then etA is defined, for all t ∈ R, and t → etA is a continuous homomorphism of R into GL(n, R), the group of nonsingular matrices in M(n, R). Consider Rn as an additive group and write the elements of Rn as column vectors. If we interpret etA as the automorphism x → etA x of Rn , then we can form Rn A R. That is Rn A R = {(x, t) : x ∈ Rn , t ∈ R} with the group product given by (x, t)(y, s) = (x + etA y, t + s), for all (x, t), (y, s) ∈ Rn A R. If δ = |det A|, then etA scales the Lebesgue (Haar) measure on Rn by a factor t δ , for each t ∈ R. Therefore, left Haar integration on Rn A R is given by    f (x, t)d(x, t) = f (x, t)δ −t dxdt. Rn A R

R

Rn

Moreover, the modular function is given by Rn A R (x, t) = δ −t , for all (x, t) ∈ Rn A R. Recall Example 1.5. If n = 1 and A = 1, the map ψ : R A R → Gaff given by ψ(x, t) = (x, et ), for (x, t) ∈ R A R, is an isomorphism of topological groups. In higher dimensions, the nature of Rn A R and its representation theory varies greatly as the matrix A varies, providing a valuable family of examples. Example 1.7 We already mentioned the general linear group GL(n, R) = 2 {A ∈ M(n, R) : det A = 0}. Since M(n, R) can be identified with Rn via its entries as coordinates and det A is a polynomial in the entries of A, GL(n, R) 2 is an open subset of Rn . Hence, it is locally compact in the topology inherited 2 from Rn . The entries of A−1 are rational functions of the entries of A with only factors of det A in the denominators. Thus A → A−1 is continuous, and multiplication of matrices is also easily seen to be continuous. So GL(n, R) is a locally compact group.

1.2 Examples

11

If we let X = (xij )ni,j =1 be a generic of GL(n, R), then   f (x)dx = f (X)(det X)−n dx11 · · · dxnn GL(n,R)

GL(n,R)

is both left- and right-invariant integration on GL(n, R). Thus G is unimodular. Closed subgroups of GL(n, R) are now all available as examples of locally compact groups, and they all give rise to semidirect product groups by acting on Rn through matrix multiplication. Specifically, let SO(n) denote the compact subgroup consisting of all orthogonal matrices with determinant one. Then we can form the semidirect product Gn = Rn  SO(n), the group of rigid motions of Rn . Of course, in this context, there is nothing special about the real numbers. If K is any locally compact topological field such as C, the p-adic numbers, or a finite field, then GL(n, K) and its closed subgroups form a rich source of examples. Example 1.8 This example is known as the Heisenberg group of dimension 2n + 1 (n ∈ N). For x, y ∈ Rn , x = (x1 , . . . , xn ) and y = (y1 , . . . , yn ), and z ∈ R, let [x, y, z] denote the matrix ⎞ ⎛ 1 x1 · · · xn z ⎜ 0 1 . . . 0 y1 ⎟ ⎟ ⎜ ⎜ .. .. .. .. ⎟ . ⎜. . . . ⎟ ⎟ ⎜ ⎝ 0 0 . . . 1 yn ⎠ 0 0 ... 0 1

Let Hn = {[x, y, z] : x, y ∈ Rn , z ∈ R}. If x · y = ni=1 xi yi , the matrix product of [x, y, z] and [x  , y  , z ] is given by [x, y, z][x  , y  , z ] = [x + x  , y + y  , z + z + x · y  ]. Clearly, Hn is a closed subgroup of GL(n + 2, R). Easy calculations show that Hn is unimodular, with Haar integral given by     f ([x, y, z])d[x, y, z] = f ([x, y, z]) dz dy dx. Hn

Rn

Rn

R

We invite the reader to check that N = {[0, y, z] : y ∈ Rn , z ∈ R} is a closed normal subgroup of Hn , isomorphic to Rn+1 , and that A = {[x, 0, 0] : x ∈ Rn } is a closed subgroup of Hn , isomorphic to Rn . Since Hn = N A and N ∩ A = {[0, 0, 0]}, Hn is isomorphic to the semidirect product Rn+1 α Rn , where α is defined by αx (y, z) = (y, z + x · y), for (y, z) ∈ Rn × R and x ∈ Rn .

12

Basics

Let Z = {[0, 0, z] : z ∈ R}, a closed subgroup of Hn which is isomorphic to R. It is obvious that [0, 0, z] commutes with every element of Hn . Conversely, if [x, y, z] commutes with every element of Hn , then a quick look at the group product shows that x = y = 0. Thus Z is exactly the center of Hn . The map ψ : Hn → R2n defined by ψ([x, y, z]) = (x, y), for [x, y, z] ∈ Hn , is a continuous homomorphism onto R2n with ker ψ = Z. So the quotient group Hn /Z is isomorphic to R2n . We will often work with H1 , the three-dimensional Heisenberg group for simplicity of notation. But almost everything we present for it has an obvious generalization to Hn . There is an important discrete subgroup of H1 . Let D = {[k, l, m] : k, l, m ∈ Z}. Clearly, D is discrete in the relative topology. It is called the discrete or integer Heisenberg group.

1.3 Coset spaces and quasi-invariant measures We will now present a collection of tools for analysis on quotient spaces that are not as widely known as the material presented in most other parts of this chapter, but are essential in the theory of induced representations. Therefore, we provide proofs of the results in this section. Let G be a locally compact group, H a closed subgroup of G, G/H the left coset space, and q : G → G/H the quotient mapping. There is a useful linear map  from Cc (G) to Cc (G/H ). For f ∈ Cc (G), form f H on G by f H (x) = H f (xh) dh, for x ∈ G. Since f is uniformly continuous, f H is a continuous function on G. Moreover, left invariance of the Haar measure on H gives f H (xh) = f H (x), for all x ∈ G and h ∈ H . Therefore, there is a unique function in Cc (G/H ), which we will denote f # ,  # H such that f (xH ) = f (x) = H f (xh) dh, for all x ∈ G. Proposition 1.9 For every ϕ ∈ Cc (G/H ), there exists an f ∈ Cc (G) such that f # = ϕ. If ϕ ∈ Cc+ (G/H ), then f can be chosen in Cc+ (G). Proof Select a compact subset K of G so that supp ϕ ⊆ q(K) and choose g ∈ Cc+ (G) so that g(x) > 0, for all x ∈ K. Clearly g # (ω) > 0, for all ω ∈ supp ϕ. Define ψ on G/H by  ϕ(ω)/g # (ω) if ω ∈ supp ϕ ψ(ω) = 0 if ω ∈ supp ϕ.

1.3 Coset spaces and quasi-invariant measures

13

Then ψ ∈ Cc (G/H ). For x ∈ G, let f (x) = ψ(q(x))g(x). Then f ∈ Cc (G) and f ≥ 0 if ϕ ≥ 0. Moreover, f # (ω) = ψ(ω)g # (ω) = ϕ(ω), for all ω ∈ G/H .  We will need the following observation elsewhere in the book. For a compact subset K of G with nonempty interior, let CK (G) = {f ∈ Cc (G) : supp f ⊆ K}. Proposition 1.10 For a compact subset K of G with nonempty interior, there exists a positive constant cK such that f # ∞ ≤ cK f ∞ , for all f ∈ CK (G). Proof Fix f ∈ Cc (G) with supp f ⊆ K. Note that f # (xH ) = 0 unless x ∈ KH . Thus, to estimate |f # (xH )| we may, and do, assume x ∈ K. For h ∈ H , xh ∈ supp f implies h ∈ (K −1 K) ∩ H . But (K −1 K) ∩ H is a compact subset of H , so it has finite Haar measure in H . Let cK denote the H -Haar measure of (K −1 K) ∩ H . Then, for x ∈ K,  # |f (xH )| ≤ |f (xh| dh ≤ cK f ∞ . H

Thus, f ∞ ≤ cK f ∞ .



#

Note that left translation commutes with the # -map. That is, for x ∈ G and f ∈ Cc (G), (Lx f )# = Lx (f # ). Definition 1.11 A rho-function for (G, H ), or simply a rho-function, when G and H are understood, is a non-negative locally integrable function ρ on G which satisfies ρ(xh) =

H (h) ρ(x), G (h)

for x ∈ G, h ∈ H . Proposition 1.12 If f ∈ Cc+ (G), then the function ρf defined by  G (h)H (h)−1 f (xh)dh (x ∈ G) ρf (x) = H

is a rho-function for (G, H ). Moreover, ρf is continuous. Proof Just change variables. Continuity of ρf follows from the fact that f has compact support.  Rho-functions are used to transfer integrations between G and G/H . For this purpose, the following proposition is essential.

14

Basics

Proposition 1.13 Let ρ be a rho-function for (G, H ) and let f ∈ Cc (G) such that f # ≡ 0 on G/H . Then  f (x)ρ(x)dx = 0. G

Proof Using Proposition 1.9, there exists g ∈ Cc (G) such that g # (q(x)) = 1 for all x ∈ supp f . That is, g H (x) = 1 for all x ∈ supp f . Therefore   0= ρ(x)g(x) f (xh−1 )H (h−1 )dhdx G H    −1 f (xh )ρ(x)g(x)dx H (h−1 )dh = G  H G (h)H (h−1 ) f (x)ρ(xh)g(xh)dxdh = G H   = f (x)ρ(x) g(xh)dhdx = f (x)ρ(x)dx, G

H

G



as was to be shown. Thus, we can define a linear functional λρ on Cc (G/H ) by  λρ (f # ) = f (x)ρ(x)dx, G

for f ∈ Cc (G). In light of Proposition 1.13, λρ is well defined and a linear map of Cc (G/H ) into C. Proposition 1.14 Let ρ be a rho-function for (G, H ). Then there exists a regular Borel measure μρ on G/H such that   # f (ω)dμρ (ω) = f (x)ρ(x)dx, G/H

G

for all f ∈ Cc (G). Proof For any α ∈ Cc+ (G/H ), there is an f ∈ Cc+ (G) with f # = α. Since  ρ(x) ≥ 0, for all x ∈ G, λρ (α) = G f (x)ρ(x)dx ≥ 0. Therefore, λρ is a positive linear functional and, as such, is given by a regular Borel measure μρ on G/H by the Riesz representation theorem.  The next proposition contains three simple facts that we leave to the reader to verify. For a Borel measure μ on G/H , let μx denote its translate by x. That is, μx (E) = μ(x · E), for any Borel set E ⊆ G/H .

1.3 Coset spaces and quasi-invariant measures

15

Proposition 1.15 Let ρ be a rho-function for (G, H ). (i) If A is a closed subset of G/H such that ρ(x) = 0, for all x ∈ G \ q −1 (A), then supp μρ ⊆ A. (ii) For x ∈ G, Lx ρ is also a rho-function for (G, H ) and μLx ρ = (μρ )x −1 . (iii) If f ∈ Cc+ (G) and ρ = ρf , then for any α ∈ Cc (G/H ),   α(ω)dμρ (ω) = α(q(x))f (x)dx. G/H

G

A regular Borel measure μ on G/H is called an invariant measure if μx = μ, for all x ∈ G. Theorem 1.16 Let G be a locally compact group and H a closed subgroup of G. There exists a nonzero positive invariant regular Borel measure on G/H if and only if G (h) = H (h), for all h ∈ H . When this is the case, the invariant measure is unique up to multiplication by a positive constant. Proof Suppose that G (h) = H (h), for all h ∈ H . Then ρ ≡ 1 on G is a rhofunction for (G, H ). By Proposition 1.14, there exists a regular Borel measure μ on G/H such that   # f (ω)dμ(ω) = f (x)dx, G/H

G

for all f ∈ Cc (G). Then, for x ∈ G,    f # (ω)d(μx )(ω) = f # (x −1 · ω)dμ(ω) = (Lx f )# (ω)dμ(ω) G/H G/H G/H    −1 = f (x y)dy = f (y)dy = f # (ω)dμ(ω), G

G

G/H

for any f # ∈ Cc (G/H ). Thus, μx = μ. Conversely, suppose that μ is a nonzero positive invariantmeasure on G/H . Define a positive linear functional λ on Cc (G) by λ(f ) = G/H f # (ω)dμ(ω). Then, for x ∈ G, λ(Lx f ) = λ(f ) by a calculation similar to the one above. So λ is a nonzero positive left-invariant linear functional on Cc (G). By uniqueness of the left Haar integral, there exists a c > 0 such that   # f (ω)dμ(ω) = c f (x)dx G/H

G

 for all f ∈ Cc (G). Now, pick f ∈ Cc (G) such that G f (x)dx = 1. For any fixed h ∈ H , let g(x) = f (xh)H (h), for x ∈ G. Then g # = f # , as is easily

16

Basics

checked. Thus H (h)G (h−1 ) − 1 =



 f (x)H (h)G (h−1 ) − f (x) dx



G =

(f (xh)H (h) − f (x))dx  = g(x)dx − f (x)dx G G  1 (g # (ω) − f # (ω))dμ(ω) = 0. = c G/H G

Hence, H (h) = G (h), for all h ∈ H .



As a consequence of Theorem 1.16, there exists a nonzero positive invariant regular Borel measure on G/H whenever both G and H are unimodular. However, if G is unimodular while H is not, then there is no nonzero positive invariant regular Borel measure on G/H . If H is a closed normal subgroup of G, then the left Haar measure on G/H is invariant under the action of G and the modular function of G, when restricted to H , agrees with the modular function of H . A regular Borel measure μ on G/H is called quasi-invariant if μ = 0 and μx ∼ μ, for every x ∈ G, where ∼ denotes mutual absolute continuity of measures. Proposition 1.17 Let μ be a quasi-invariant measure on G/H . Then (i) Either supp μ = G/H or μ = 0. (ii) If ν is another regular Borel measure on G/H such that ν ∼ μ, then ν is quasi-invariant. Proof (i) Suppose U is a nonempty open subset of G/H such that μ(U ) = 0. Since μx −1 ∼ μ, μ(xU ) = 0, for any x ∈ G. If K is any compact subset of G/H , we can cover K by finitely many sets of the form x · U with x ∈ G. Thus μ(K) = 0 for every compact subset K of G/H . This implies μ = 0. (ii) is trivial.  Theorem 1.18 Let G be a locally compact group and H a proper closed subgroup. There exists a quasi-invariant regular Borel measure μ on G/H such that the Radon–Nikodym derivative,   dμx (ω), σ (x, ω) = dμ

1.3 Coset spaces and quasi-invariant measures

17

for x ∈ G, ω ∈ G/H , is a continuous function on G × G/H . Moreover, σ satisfies the identity σ (z, ω)σ (x, zω) = σ (xz, ω),

(1.3)

for ω ∈ G/H, x, z ∈ G. We begin the proof with two lemmas. Lemma 1.19 Let U be an open relatively compact symmetric neighborhood of e in G. Then there exists a subset A of G with the following properties: (i) For any x ∈ G, xH ∩ Uy = ∅, for some y ∈ A. (ii) If K is a compact subset of G, then {y ∈ A : KH ∩ Uy = ∅} is finite. Proof If G is compact and U = G, we could take A to be any singleton. Otherwise let A be the family of all subsets A of G that satisfy y, z ∈ A, y = z implies z ∈ UyH . The collection A is nonempty, directed by upward inclusion and, for any chain in A, the union is also in A. Thus, by Zorn’s lemma, there exists a maximal A ∈ A. If x ∈ G \ A were such that xH ∩ Uy = ∅, for all y ∈ A, then we could add x to A and make A strictly larger. So (i) holds for A. Let K be a compact subset of G and let AK = {y ∈ A : KH ∩ Uy = ∅}. For each y ∈ AK , KH ∩ Uy = ∅ implies U K ∩ yH = ∅. For y ∈ AK , pick xy ∈ U K ∩ yH . If AK was infinite, then {xy : y ∈ Ak } would have a cluster point x, say, in the compact set U K. Let V be a neighborhood of e such that V V −1 ⊆ U . Since the xy cluster at x, there are distinct y, z ∈ AK such that xy , xz ∈ V x. This implies that xy xz−1 ∈ V V −1 ⊆ U . But xy ∈ H and xz ∈ ZH , so xy ∈ U xz ⊆ U zH which forces y ∈ U zH , in contradiction to A ∈ A. Thus  AK is finite and (ii) holds. Lemma 1.20 There exists a rho-function ρ for (G, H ) which is continuous and everywhere strictly positive on G. Proof Let f ∈ Cc+ (G) be such that f (e) > 0 and f (x −1 ) = f (x), for all x ∈ G. Let U = {x ∈ G : f (x) > 0}, and let A be a subset of G with properties (i) and (ii) of Lemma 1.19. For each y ∈ A define f y (x) = f (xy −1 ), for x ∈ G, and let ρf y be the continuous rho-function defined in Proposition 1.12. That is,  ρf y (x) = G (h)H (h−1 )f (xhy −1 )dh. H

Notice that ρf y (x) = 0 if x ∈ UyH . So, by property (ii) for A, if K is any compact subset of G, then ρf y is 0 on K, for all but finitely many y ∈ A. Thus

ρ = y∈A ρf y is a continuous function on G and is clearly a rho-function.

18

Basics

For every x ∈ G, by property (i) for A, there exists a y ∈ A such that xH ∩ Uy = ∅. Then f y (xh) > 0, for some h ∈ H . Therefore, ρf y (x) > 0 and hence ρ(x) > 0.  Continuing with the proof of Theorem 1.18, we fix a rho-function ρ that is continuous and everywhere strictly positive on G, as guaranteed by Lemma 1.20, and let μ = μρ . By Proposition 1.15, x · μ = μL−1 for every x ∈ G. The x ρ function y → ρ(xy)/ρ(y) is constant on H -cosets in G, so it defines a function on G/H . Recall that q : G → G/H denotes the quotient map. Let ρ(xy) , x, y ∈ G. σ (x, q(y)) = ρ(y) Then (x, ω) → σ (x, ω) is a continuous function from G × G/H into R+ . The identity (1.3) clearly holds. For any f ∈ Cc (G) and x ∈ G,    # f (ω)d(x · μ)(ω) = f (y)Lx −1 ρ(y)dy = f (y)σ (x, q(y))ρ(y)dy G/H G G  f # (ω)σ (x, ω)dμ(ω). = G/H

Since Cc (G/H ) = {f , f ∈ Cc (G)}, we conclude  that x · μ is absolutely con (ω), for (x, ω) ∈ G × G/H . tinuous with respect to μ and σ (x, ω) = d(x·μ) dμ Finally, since σ (x, ω) is strictly positive, x · μ ∼ μ. So μ is quasi-invariant, and the theorem is proved. #

Corollary 1.21 (Weil’s integration formula) Let μ be a quasi-invariant regular Borel measure on G/H with corresponding rho-function ρ, continuous and strictly positive on G. Then, for any f ∈ Cc (G),    f (x)ρ(x) dx = f (xh) dh dμ(xH ). (1.4) G

G/H

H

Proof This follows immediately from the relationship between μ and ρ as given in Proposition 1.14.  Lemma 1.22 Let ν be any quasi-invariant measure on G/H . Then a Borel subset A of G/H is locally ν-null if and only if q −1 (A) is locally null in G. Proof Let A ⊆ G/H be a Borel set. By intersecting A with an arbitrary compact subset of G/H , we may assume, without loss of generality, that A is relatively compact. Let f ∈ C + (G) be such that f = 0. Then Fubini’s Theorem implies     f (x)1A (x · ω)dxdν(ω) = f (x)1A (x · ω)dν(ω)dx. (1.5) G/H

G

G

G/H

1.3 Coset spaces and quasi-invariant measures

19

Suppose that ν(A) = 0. Then ν(x −1 · A) = 0, for all x ∈ G. Thus the right-hand side of (1.5) is 0, and hence so is the left-hand side. Therefore  f (x)1A (x · ω)dx = 0, G

for almost all ω ∈ G/H . Let C be any compact subset of G and U a compact neighborhood of e. Select f ∈ Cc+ (G) so that f (x) ≥ 1 for x ∈ CU −1 . Since ν(q(U )) > 0 (recall that the support  of any quasi-invariant measure is all of G/H ), there exists y ∈ U so that G f (x)1A (x · q(y))dx = 0. So   f (xy −1 )1q −1 (A) (x)dx. 0 = G (y) f (x)1A (q(xy))dx = G

G

 Now, for x ∈ C, f (xy ) ≥ 1 which implies that G 1q −1 (A)∩C (x)dx = 0. Thus q −1 (A) ∩ C is a null set for any compact C ⊆ G. That is, q −1 (A) is locally null. Conversely, suppose q −1 (A) is locally null. Again, let f ∈ C + (G) be such that f = 0. For any fixed ω ∈ G/H , choose y ∈ G so that q(y) = ω. Then x → f (xy −1 ) is continuous with compact support, so  0= f (xy −1 )G (y −1 )1q −1 (A) (x)dx G   f (x)1q −1 (A) (xy)dx = f (x)1A (x · ω)dx. = −1

G

G

Then the left-hand side of (1.5) is 0, so the right-hand side is 0 as well. Hence, for almost all x ∈ G,  f (x)1A (x · ω)dν(ω) = f (x)ν(x −1 · A). 0= G/H

Since f = 0, there exists x ∈ G so that ν(x −1 · A) = 0, which implies ν(A) = 0 by quasi-invariance.  As an immediate consequence of Lemma 1.22 we obtain the following corollary. Corollary 1.23 Let ν and μ be any two quasi-invariant measures on G/H . Then ν ∼ μ. Remark 1.24 The function σ (x, ω) defined from a strictly positive continuous rho-function ρ by σ (x, q(y)) = ρ(xy)/ρ(y) trivially satisfies the identity σ (xy, ω) = σ (x, y · ω)σ (y, ω),

20

Basics

for all x, y ∈ G, ω ∈ G/H . However, if ν is any quasi-invariant measure on )(ω), then for any ϕ ∈ Cc (G/H ) and x, y ∈ G, G/H and if σ  (x, ω) = ( d(x·ν) dν   ϕ(ω)σ  (xy, ω)dν(ω) = ϕ(ω)d(xy · ν)(ω) G/H G/H  = ϕ(y −1 · ω)d(x · ν)(ω) G/H  = ϕ(y −1 · ω)σ  (x, ω)dν(ω) G/H  = ϕ(ω)σ  (x, y · ω)d(y · ν)(ω) G/H  = ϕ(ω)σ  (x, y · ω)σ  (y, ω)dν(ω). G/H

It follows that σ  (xy, ω) = σ  (x, y · ω)σ  (y, ω), for locally almost all ω ∈ G/H . The action of G on G/H can be transferred to continuous actions of G on key function spaces. For any function ϕ from G/H into any range space and x ∈ G, define Lx ϕ(ω) = ϕ(x −1 · ω), for all ω ∈ G/H . At this time, we present the continuity result for Cc (G/H ) equipped with the supremum norm. Proposition 1.25 Let G be a locally compact group and H a closed subgroup of G. For ϕ ∈ Cc (G/H ), the map x → Lx ϕ is a continuous map of G into (Cc (G/H ), · ∞ ). Proof It is clear that x → Lx is a homomorphism of G into the group of isometries of (Cc (G/H ), · ∞ ). So it suffices to show continuity of Lx ϕ, as a function of x, at x = e, the identity in G. Let K be a compact subset of G/H supporting ϕ. Let  > 0 be given. For each ω ∈ G/H , there exists a neighborhood Wω of ω such that |ϕ(ω ) − ϕ(ω)| < /2 if ω ∈ Wω . Then Uω = {x ∈ G : x −1 · ω ∈ Wω } is a neighborhood of e. Let Vω be a symmetric neighborhood of e such that Vω2 ⊆ Uω . Then Vω · ω is a neighborhood of ω, for each ω ∈ K. Since K is compact, there exist ω1 , . . . , ωn ∈ K such that K ⊆ ∪nj=1 Vωj · ωj . Let V = ∩nj=1 Vωj , a neighborhood of e. For x ∈ V and any ω ∈ K, there exists a j, 1 ≤ j ≤ n, such that ω ∈ Vωj · ωj . This implies x −1 · ω ∈ Uωj · ωj . Thus, |Lx ϕ(ω) − ϕ(ωj )| < /2. Therefore, |Lx ϕ(ω) − ϕ(ω)| ≤ |Lx ϕ(ω) − ϕ(ωj )| + |ϕ(ωj ) − Le ϕ(ω)| < .

1.4 Representations

21

Note that if neither ω nor x −1 · ω is in K, then both Lx ϕ(ω) and ϕ(ω) are 0. Moreover, if ω ∈ G/H \ K is such that x −1 · ω ∈ K, then simply applying the above with x acting on x −1 · ω leads to |Lx ϕ(ω) − ϕ(ω)| <  as well. So this holds for all ω ∈ G/H . Thus, Lx ϕ − ϕ ∞ <  if x ∈ V . Therefore, x → Lx ϕ is a continuous map of G into Cc (G) when the latter is given the supremum norm. 

1.4 Representations An overarching theme of this book is the development of methods for constructing and analyzing representations and their inter-relationships for various classes of locally compact groups. In this section, we review the basic language of representation theory as it is commonly used in the study of general locally compact groups. The reader is directed to the monographs of Hewitt and Ross [73], Folland [55], and chapter 13 of Dixmier [37] for detailed proofs. If H is a Hilbert space, U(H) denotes the group of unitary operators on H. If G is a locally compact group, a continuous unitary representation of G is a pair (π, H(π)), where H(π) is a Hilbert space and π is a homomorphism of G into U(H(π )) such that, for any ξ, η ∈ H(π), the function x → π(x)ξ, η is continuous on G. That is, we require that the map π : G → U(H(π )) be continuous when U(H(π)) is equipped with the weak operator topology. Throughout this book, we will simply use the word representation to mean a continuous unitary representation. Moreover, we will usually simply use π to refer to the representation with the Hilbert space H(π ) left understood. The dimension of π , dπ , is defined to be the Hilbert space dimension of H(π ). Example 1.26 For any group G, define the trivial representation τ by setting H(τ ) = C and τ (x) = 1, for all x ∈ G. (Note that U(C) can be identified with T, the unit circle group.) It is immediate that τ is an irreducible representation since dτ = 1. Example 1.27 Let G be any locally compact group and H a closed subgroup of G. There is always a representation of G associated with natural action of G on the quotient space G/H . Let μ be a quasi-invariant measure on G/H with the function σ : G × G/H → R+ as in Theorem 1.18. Define a representation π of G as follows: H(π) = L2 (G/H, μ). For x ∈ G and f ∈ L2 (G/H, μ), define  π(x)f (ω) = σ (x −1 , ω)f (x −1 · ω),

22

Basics

for all ω ∈ G/H . To see that π is a representation, note first that π (x)f ∈ L2 (G/H, μ) and  2

π (x)f 2 = |f (x −1 · ω)|2 σ (x −1 , ω) dμ(ω) G/H  = |f (ω)|2 dμ(ω) = f 22 . G/H

Using the identity in Theorem 1.18, one checks that π (x)π(y) = π(xy), for any x, y ∈ G. Clearly, π(e) is the identity operator. Thus, π is a homomorphism of G into the group of unitary operators on L2 (G/H, μ). We will show that π is strong operator topology continuous. Fix f ∈ L2 (G/H, μ). We need to show that, for any  > 0, there exists a neighborhood U of e such that x ∈ U implies π (x)f − f 2 < . Fix  > 0. Since μ is a regular Borel measure, there exists a g ∈ Cc (G/H ) such that f − g 2 < /3. Fix a compact symmetric neighborhood V of e. Let K be a compact subset of G/H supporting g. Then V · K = {z · ω : z ∈ V , ω ∈ K} is a compact subset of G/H and π (x)g(ω) = 0 implies x ∈ V and 1/2 . Now π(x)g(ω) = ω  ∈ K. Thus, π(x)g − g 2 ≤ μ(V · K) π(x)g − g ∞ −1 σ (x , ω)Lx g(ω), so Proposition 1.25 and continuity of σ (x −1 , ω) jointly in x and ω mean there exists a neighborhood U of e with U ⊆ V and 

π(x)g − g ∞ < , 3μ(V · K)1/2 for all x ∈ U . Thus, for x ∈ U ,

π (x)f − f 2 ≤ π(x)f − π (x)g 2 + π(x)g − g 2 + g − f 2 = 2 f − g 2 + π(x)g − g 2 μ(V · K)1/2  2 + < = . 3 3μ(V · K)1/2 Thus, x → π(x)f is continuous for every f ∈ L2 (G/H, μ). This is strong operator topology continuity. Weak operator topology continuity follows by an application of the Cauchy–Schwarz inequality. Example 1.28 When H = {e}, the trivial subgroup of G in the previous example, we obtain the left-regular representation, λG , of G which has L2 (G) as its Hilbert space. That is, for f ∈ L2 (G), λG (x)f (y) = f (x −1 y), for almost all y ∈ G. Example 1.29 For any locally compact group G and x ∈ G, define ρG (x) on L2 (G) by, for any f ∈ L2 (G), ρG (x)f (y) = G (x)1/2 f (yx), for almost all y ∈ G.

1.4 Representations

23

Note that ρG (x)f is measurable and    2 2 |ρG (x)f (y)| dy = G (x)|f (yx)| dy = |f (y)|2 dy. G

G

G

From this, it follows that ρG (x) is a unitary operator on L (G), for each x ∈ G. Checking that ρG satisfies the rest of the conditions to be a representation is now similar to what was done in Example 1.27. We call ρG the right-regular representation of G. 2

Fix a representation π for the following discussion. Since π is a homomorphism into the group of unitary operators, we have π(e) = I , the identity operator on H(π), π(x)∗ = π(x −1 ), for any x ∈ G (so ξ, π (x)η = π(x −1 )ξ, η for ξ, η ∈ H(π)), and π (xy) = π (x)π (y) for all x, y ∈ G. A subspace V of H(π) is called π-invariant if π(x)ξ ∈ V , for all x ∈ G and ξ ∈ V . If V is π -invariant and η ∈ V ⊥ , the orthogonal complement of V , then, for all ξ ∈ V and x ∈ G, π (x)η, ξ  = η, π(x −1 )ξ  = 0, whence π(x)η ∈ V ⊥ . Thus V ⊥ and V = (V ⊥ )⊥ are π -invariant. If K is a closed π-invariant subspace of H(π), define π K (x) = π (x)|K , for each x ∈ G. Then π K is a representation of G with H(π K ) = K and is called a subrepresentation of π. If PK is the orthogonal projection of H(π ) onto K, then any ξ ∈ H(π ) is decomposed uniquely as ξ = ξK + ξK⊥ , where ξK = PK ξ ∈ K and ξK⊥ = (I − PK )ξ ∈ K⊥ . Thus, for any x ∈ G, ⊥

π(x)ξ = π K (x)ξ + π K (x)ξK⊥ . We then say that π is decomposed as the direct sum of the two subrepresen⊥ ⊥ tations π K and π K and write π = π K ⊗ π K . More generally, we make the following definition. Definition 1.30 Let  be a nonempty set. For each α ∈ , let Hα be a Hilbert space and let πα be a representation of G with H(πα ) = Hα . For (ξα )α∈ ∈ ⊕α∈ Hα , the Hilbert space direct sum, let (⊕α∈ πα )(x) ((ξα )α∈ ) = (πα (x)(ξα ))α∈ , for all x ∈ G. Then ⊕α∈ πα is called the direct sum of the πα , α ∈ . If all the πα are identical, say πα = σ , for all α ∈  and m = card(), then ⊕α∈ σ is denoted simply as mσ . Definition 1.31 We say that π is irreducible if {0} and H(π) are the only π-invariant closed subspaces of H(π).

24

Basics

The coefficient functions x → π(x)ξ, η provide a criterion for irreducibility that is easy to verify and sometimes easy to apply. Define ϕξ,η (x) = π(x)ξ, η. Then ϕξ,η is a bounded continuous function on G, for each ξ, η ∈ H(π). Proposition 1.32 Let π be a representation of a locally compact group G. The following two conditions are equivalent. (i) π is irreducible. (ii) For ξ, η ∈ H(π ), ξ = 0 and η = 0 implies that ϕξ,η = 0. Two representations π and σ of G are called (unitarily) equivalent if there exists a unitary map U : H(π) → H(σ ) such that U π(x) = σ (x)U, for all x ∈ G. We write π  σ to indicate that π and σ are equivalent. The unitary map U that satisfies U π(x) = σ (x)U , x ∈ G, is said to intertwine π and σ . If we think of H(π ) and H(σ ) as G-modules (under the actions via π and σ , respectively), then U is a G-module isomorphism. Equivalences of the type arising in the following proposition are common. Proposition 1.33 Let π be a representation of G and let {Kα : α ∈ } be a family of closed π-invariant subspaces of H(π) such that Kα ⊥ Kβ , for α = β, α, β ∈ , and ∪α∈ Kα is total in H(π ). Then π is equivalent to ⊕α∈ π Kα . The intertwining unitary promised in Proposition 1.33 is defined as follows. For each α ∈ , let Pα denote the orthogonal projection of H(π ) onto Kα . Define U : H(π ) → ⊕α∈ Kα by U ξ = (Pα ξ )α∈ , for each ξ ∈ H(π ). It is a routine application of the definitions to check that U is a unitary map intertwining π with ⊕α∈ π Kα . For any two representations π and σ of G, let HomG (π, σ ) = {T ∈ B(H(π ), H(σ )) : T π (x) = σ (x)T , for all x ∈ G}, where B(H(π), H(σ )) denotes the space of bounded linear operators from H(π ) to H(σ ). The elements of HomG (π, σ ) are called intertwining operators of π and σ . The next proposition is routine. Proposition 1.34 Let π and σ be representations of G. Then HomG (π, σ ) is a linear subspace of B(H(π), H(σ )) that is closed in the following sense: if (Tα )α is a net in HomG (π, σ ) and T ∈ B(H(π ), H(σ )) satisfies T ξ, η = limTα ξ, η, α

for all ξ ∈ H(π) and η ∈ H(σ ), then T ∈ HomG (π, σ ).

1.4 Representations

25

In the special case when π = σ , HomG (π, π) = {T ∈ B(H(π )) : T π (x) = π (x)T , for all x ∈ G}. This is the commutant of π (G) = {π(x) : x ∈ G}. To emphasize this we will usually write π (G) for HomG (π, π). Since π(x)∗ = π(x −1 ) for all x ∈ G, π(G) is a von Neumann algebra on H(π). Irreducibility of the representation π is reflected in a striking manner in π (G) as the next proposition shows. Proposition 1.35 Let π be a representation of a locally compact group G. Then π is irreducible if and only if π(G) = CI , where I is the identity operator on H(π ). Remark 1.36 If G is not the trivial one-element group, then λG is not irreducible. In fact, ρG (G) ⊆ π(G) , where ρG is the right regular representation of G as introduced in Example 1.29. Moreover, if z ∈ G, z = e, then ρG (z) is not a multiple of the identity operator. To see this, let V be a relatively compact neighborhood of e with V ∩ V z−1 = ∅, and let f = χV . Then f, f  = |V | > 0 while ρG (z)f, f  = 0. Therefore ρG (z) ∈ CI and λG (G) = CI . Hence λG fails to be irreducible. If π is a representation of G and P is a projection in B(H(π )) (that is, P 2 = P = P ∗ ), then P H(π) is π -invariant if and only if P ∈ π (G) . In fact, the map P → P H(π ) is a bijection between the set of projections in π (G) and the set of closed π-invariant subspaces of H(π ). If P is a projection in π(G) for some representation π of G, let π P denote the subrepresentation formed by restricting each π(x) to P H(π). If P and Q are projections in π (G) , we can form the linear space Qπ(G) P = {QAP : A ∈ π (G) }. The map T → T |P H(π) identifies Qπ(G) P with HomG (π P , π Q ). This is formalized as the following proposition which allows us to view spaces of intertwining operators as a “part” of a commutant algebra. Proposition 1.37 Let π be a representation of a locally compact group G. Let P and Q be two projections in π (G) . Then HomG (π P , π Q ) is isomorphic to Qπ(G) P . We collect together a number of properties of intertwining spaces in Proposition 1.38 below. We use ∼ = to denote an isomorphism of vector spaces. Proposition 1.38 Let π, π1 , π2 and σ, σ1 , σ2 be representations of G. (i) π1  σ1 and π2  σ2 imply HomG (π1 , π2 ) ∼ = HomG (σ1 , σ2 ). (ii) HomG (π, σ1 ⊕ σ2 ) ∼ = HomG (π, σ1 ) ⊕ HomG (π, σ2 ), the direct sum of vector spaces.

26

Basics

(iii) If σ is irreducible and m ∈ N, then the dimension of HomG (σ, mσ ) equals m. (iv) If σ is irreducible and m is an infinite cardinal, then the dimension of HomG (σ, mσ ) is infinite. (v) If σ is irreducible, then HomG (σ, π ) = {0} if and only if there is a projection P ∈ π(G) such that σ  π P . If the situation in 1.38(v) holds, then σ occurs as a subrepresentation of π. It may happen that σ  π P for some P ∈ π(G) , a projection, and that σ still occurs as a subrepresentation of π I −P . Simple arguments show that there exists a projection P ∈ π(G) such that π P  mσ , for some cardinality m, and σ does not occur as a subrepresentation of π I −P . Then we say that σ occurs in π (or π contains σ ) with multiplicity m. Using Proposition 1.38, we then have that HomG (σ, π ) ∼ = HomG (σ, mσ ). This leads to the following useful result. Proposition 1.39 Let π and σ be representations of G. If σ is irreducible, then (i) HomG (σ, π ) is infinite dimensional if and only if the multiplicity with which σ occurs in π is infinite. (ii) If HomG (σ, π ) is finite dimensional, then dim HomG (σ, π ) equals the multiplicity with which σ occurs in π. Definition 1.40 If π is a representation of G, an element ξ ∈ H(π ) is called a cyclic vector for π if the set {π (x)ξ : x ∈ G} is total in H(π ). If there exists a cyclic vector for π, then π is called a cyclic representation. Now Proposition 1.32 can be phrased as follows: π is irreducible if and only if every nonzero vector in H(π ) is cyclic. The next proposition can be a useful characterization of cyclic vectors. Proposition 1.41 Let π be a representation of G and let ξ ∈ H(π ). The following are equivalent: (i) ξ is a cyclic vector for π. (ii) T ∈ π(G) and T ξ = 0 imply T = 0. A vector ξ that satisfies property (ii) of Proposition 1.41 is often referred to as a separating vector for the algebra π (G) . When ξ is a separating vector for π(G) , the map T → T ξ is an injective linear map from π(G) into H(π ). So π(G) cannot be too large if π is a cyclic representation. A general representation can be decomposed into a sum of cyclic representations. Proposition 1.42 Let π be a representation of G. Then π is equivalent to the direct sum of cyclic representations of G.

1.5 Representations of L1 (G) and functions of positive type

27

We saw that one can add representations, in a sense, by taking the direct sum which acts on the direct sum of the respective Hilbert spaces. To multiply representations, one uses the tensor product of the Hilbert spaces. It is useful to start by considering two groups and representations of each of them. Definition 1.43 If G1 and G2 are locally compact groups with representations π1 and π2 , the outer tensor product π1 × π2 of G1 × G2 acts on H(π1 ) ⊗ H(π2 ) by (π1 × π2 )(x1 , x2 )(ξ1 ⊗ ξ2 ) = π1 (x1 )ξ1 ⊗ π2 (x2 )ξ2 , for all ξi ∈ H(πi ) and xi ∈ Gi , i = 1, 2. For two representations of the same group, one can restrict the outer tensor product to the diagonal of the product of the group with itself to get a representation of the original group. Definition 1.44 If G is a locally compact group and π1 and π2 are representations of G, then the (inner) tensor product of π1 and π2 is the representation π1 ⊗ π2 given by, for x ∈ G, (π1 ⊗ π2 )(x) = (π1 × π2 )(x, x). Remark 1.45 If χ is a one-dimensional representation of G and π is any representation, then H(χ ) = C and H(χ ) ⊗ H(π ) = H(π ). Thus, (χ ⊗ π)(x) = χ(x)π (x), for all x ∈ G. We end this section by formally introducing the dual space.  denote the set of Definition 1.46 For a locally compact group G, let G  the dual equivalence classes of irreducible representations of G. We call G space of G. If σ is an irreducible representation of G, let [σ ] denote its equivalence  Although we will occasionally use this notation, we will class, so [σ ] ∈ G. not distinguish between an irreducible representation and its equivalence class  will be when no confusion should result. Thus, depending on the context, σ ∈ G read as either an equivalence class of irreducible representations or a particular element of that class.

1.5 Representations of L 1 (G) and functions of positive type Recall that a normed ∗-algebra is an algebra A over C with a norm · that satisfies ab ≤ a · b , for all a, b ∈ A, and an involution a → a ∗ which

28

Basics

satisfies a ∗ = a , for all a ∈ A. If A is complete in the norm, A is a Banach ∗-algebra. Our prototypical example of a Banach ∗-algebra is L1 (G) and Cc (G), as a ∗-subalgebra of L1 (G), is an important example of a normed ∗-algebra that is not complete. If A is a normed ∗-algebra, a ∗-representation or simply a representation of A is a pair (π, H(π )), where H(π) is a Hilbert space and π is a homomorphism of A into B(H(π )) such that π(a ∗ ) = π(a)∗ , for all a ∈ A. It is one of the many remarkable consequences of spectral theory that, if A is a Banach ∗-algebra, then any representation π of A is automatically continuous. Indeed,

π (a) ≤ a , for all a ∈ A. Notice that, just as with representations of groups, we refer to a representation (π, H(π)) of A as simply π, leaving the Hilbert space as understood. Let A be a normed ∗-algebra and π a continuous representation of A, then π extends to a representation of A, the completion of A. Thus π (a) ≤ a , for all a ∈ A, even when A is not complete if we assume π is continuous. A subspace V of H(π ) is called π-invariant if π (a)ξ ∈ V , for all ξ ∈ V and a ∈ A. If V is π-invariant, then so is V ⊥ and V = V ⊥⊥ . If V is a closed π-invariant subspace of H(π ), then we can form a representation π V of A by π V (a) = π (a)|V , for all a ∈ A. We call π V a subrepresentation of π . If V is a closed subspace of H(π ) and PV is the orthogonal projection of H(π ) onto V , then V is π-invariant if and only if PV ∈ π (A) = {T ∈ B(H(π )) : T π (a) = π(a)T , for all a ∈ A}. A representation π of A is called irreducible if H(π ) and {0} are the only closed π-invariant subspaces of H(π ). This is equivalent to π(A) = CI as in the case of a representation of a group. The analogy with group representations continues through definitions of equivalence, direct sums, and intertwining spaces. In particular, if π and σ are two representations of a normed ∗-algebra A, let HomA (π, σ ) = {T ∈ B(H(π), H(σ )) : T π (a) = σ (a)T , for all a ∈ A}, the intertwining space of π and σ . There is one phenomenon that arises for representations of normed ∗-algebras that has no analogue for group representations, and that is degeneracy. If A is a normed ∗-algebra and H is any Hilbert space, the representation π(a) = 0, for all a ∈ A, is the totally degenerate representation of A on H. If π is a continuous representation of A, let N = {ξ ∈ H(π ) : π(a)ξ = 0, for all a ∈ A}.

1.5 Representations of L1 (G) and functions of positive type

29

Then N is a closed π-invariant subspace of H(π ). Moreover, N ⊥ is the closed linear span of {π(a)η : a ∈ A, η ∈ H(π )}. We say that π is nondegenerate if N = {0}. In this case, π(A)H(π ) = {π(a)η : a ∈ A, η ∈ H(π )} is a total set in H(π ). We usually restrict attention to nondegenerate representations of normed ∗-algebras. Let us turn to the Banach ∗-algebra L1 (G), where G is a locally compact group. For a representation π of G, we construct a representation  π of L1 (G) 1 as follows. For f ∈ L (G), define bf on H(π) × H(π ) by  bf (ξ, η) = f (x)π (x)ξ, η dx, G

for all ξ, η ∈ H(π). Then bf is linear in the first argument and conjugate linear in the second argument and satisfies |bf (ξ, η)| ≤ f 1 ξ · η , for all ξ, η ∈ H(π ). That is, bf is a bounded sesquilinear form on H(π ). Thus there exists a unique bounded linear operator, denoted  π (f ), on H(π ) such that  f (x)π (x)ξ, η dx,  π (f )ξ, η = G

for all ξ, η ∈ H(π ). It is easily verified that  π : L1 (G) → B(H(π )) is a linear mapping. Moreover, for f, g ∈ L1 (G) and ξ, η ∈ H(π ), a short computation shows that  π (f ∗ g)ξ, η =  π (f ) π (g)ξ, η and  π (f ∗ ) =  π (f )∗ . π is nondegenerate, let ξ ∈ So  π is a representation of L1 (G). To see that  H(π), ξ = 0. The function x → π(x)ξ, ξ  is bounded and continuous and takes the value ξ 2 = 0 at x = e. Therefore, there exists an f ∈ L1 (G) such that  f (x)π (x)ξ, ξ dx = 0.  π (f )ξ, ξ  = G

This shows that  π is nondegenerate. We collect properties of this relationship between representations of G and nondegenerate representations of L1 (G) in a proposition. Proposition 1.47 For each representation π of G, there exists a nondegenerate representation  π of L1 (G) such that, for all ξ, η ∈ H(π) and f ∈ L1 (G),  f (x)π (x)ξ, η dx.  π (f )ξ, η = G 1

Let σ be a representation of L (G) and π1 and π2 representations of G. Then the following hold:

30

(i) (ii) (iii) (iv) (v)

Basics There is a unique representation π of G with  π = σ. HomG (π1 , π2 ) = HomL1 (G) (π1 , π2 ). π1  π2 if and only if π1  π2 . π is irreducible if and only if  π is irreducible. π is cyclic if and only if  π is cyclic.

Now we drop the notational distinction between a representation π of G and the corresponding representation  π of L1 (G) and simply write π (f ) for  π (f ). Example 1.48 If λG is the left regular representation of G, then an elementary calculation shows that, for f ∈ L1 (G), g, h ∈ L2 (G),  f (x)λG (x)g, h dx = f ∗ g, h. λG (f )g, h = G

Thus, λG (f ) is simply the operator of left convolution by f on L2 (G). For f ∈ L1 (G), if f ∗ g = 0, for all g ∈ L2 (G), then f = 0. Thus, λG is faithful (has trivial kernel) when considered as a representation of L1 (G). The next proposition is useful and easy to prove. Proposition 1.49 Let π be a representation of G. For x ∈ G and f ∈ L1 (G), π (x)π (f ) = π(Lx f ). We now turn to a fundamental method for constructing cyclic representations of L1 (G), and hence of G. First observe that if π is a representation of G with cyclic vector ξ , then ξ is also a cyclic vector for π(L1 (G)). The associated coefficient function, ϕξ,ξ (·) = π (·)ξ, ξ  can be used to define a continuous linear functional on L1 (G) by f → G f (x)ϕξ,ξ (x) dx. Note that, since |ϕξ (x)| ≤ ξ 2 , for all x ∈ G, ϕξ ∈ L∞ (G). For any f ∈ L1 (G),   ∗ (f ∗ f )(x)ϕξ,ξ (x)dx = (f ∗ ∗ f )(x)π (x)ξ, ξ dx G

G

= π(f ∗ ∗ f )ξ, ξ  = π(f )ξ, π (f )ξ  ≥ 0.

A linear functional  on L1 (G) is said to be positive if (f ∗ ∗ f ) ≥ 0, for all f ∈ L1 (G). Identifying L∞ (G) with the dual space of L1 (G), a function ϕ in L∞ (G) is said to be of positive type if the associated linear functional of L1 (G) is positive. Thus, diagonal coefficient functions such as ϕξ,ξ are functions of positive type.

1.5 Representations of L1 (G) and functions of positive type

31

Notice that, for any ϕ ∈ L∞ (G), f ∈ L1 (G),    (f ∗ ∗ f )(x)ϕ(x)dx = f ∗ (y)f (y −1 x)ϕ(x) dy dx G  G G = f (y −1 )G (y −1 )f (y −1 x))ϕ(x) dy dx G G   = f (y)f (yx)ϕ(x) dy dx G G = f (x)f (y)ϕ(y −1 x) dy dx. G

G

∞

Thus ϕ ∈ L (G) is of positive type if and only if   f (x)f (y)ϕ(y −1 x) dy dx ≥ 0, G

G

for all f ∈ L1 (G). What follows is a fundamental construction, starting with a function of positive type, of a cyclic representation whose corresponding diagonal coefficient function is the original function of positive type. This construction is a special case of the GNS construction named after Gelfand, Naimark, and Segal. If ϕ ∈ L∞ (G) is of positive type, define    f (x)g(y)ϕ(y −1 x) dx dy = (g ∗ ∗ f )(x)ϕ(x)dx, (f, g)ϕ = G

G

G

for f, g ∈ L (G). Then (·, ·)ϕ is a positive semidefinite sesquilinear form on L1 (G). The Cauchy–Schwarz inequality implies that 1

|(f, g)ϕ |2 ≤ (f, f )ϕ (g, g)ϕ , for all f, g ∈ L1 (G). Then, for f ∈ L1 (G), (f, f )ϕ = 0 if and only if (f, g)ϕ = 0, for all g ∈ L1 (G). Let Nϕ = {f ∈ L1 (G) : (f, f )ϕ = 0} and Vϕ = L1 (G)/Nϕ . For f + Nϕ , g + Nϕ ∈ Vϕ , let f + Nϕ , g + Nϕ ϕ = (f, g)ϕ . Then ·, ·ϕ is well defined on Vϕ and is an inner product. Let Hϕ be the Hilbert space completion of Vϕ . For f ∈ L1 (G), let [f ] = f + Nϕ in Hϕ . Since |(f, g)ϕ | ≤ ϕ ∞ f 1 g 1 for f, g ∈ L1 (G),

[f ] ϕ = [f ], [f ]ϕ ≤ ϕ 1/2 ∞ f 1 ,

32

Basics

for f ∈ L1 (G). For x ∈ G and f, g ∈ L1 (G),   (Lx f, Lx g)ϕ = f (x −1 z)g(x −1 y)ϕ(y −1 z)dzdy G G   f (z)g(y)ϕ(y −1 z)dzdy = G

G

= (f, g)ϕ . Thus Lx Nϕ ⊆ Nϕ , and so, for x ∈ G, a map πϕ (x) can be defined on Vϕ by πϕ (x)[f ] = [Lx f ], for all f ∈ Vϕ . The calculation above shows that πϕ (x)[f ], πϕ (x)[g]ϕ = [f ], [g]ϕ , for all [f ], [g] ∈ Vϕ . Hence πϕ (x) extends uniquely to a unitary operator, also denoted πϕ (x), on Hϕ . Clearly, πϕ (xy) = πϕ (x)πϕ (y), for all x, y ∈ G, and

πϕ (x)[f ] − πϕ (y)[f ] ϕ ≤ ϕ 1/2 ∞ Lx f − Ly f 1 , for each f ∈ L1 (G). Thus x → πϕ (x)[f ] is continuous, for any [f ] ∈ Vϕ . The usual ε/3-argument shows that x → πϕ (x)ξ is continuous, for all ξ ∈ Hϕ . So πϕ is a representation of G. Proposition 1.50 Let ϕ be a function of positive type in L∞ (G), and let πϕ be the representation of G constructed above. There exists a cyclic vector ξϕ ∈ Hϕ for πϕ such that πϕ (x)ξϕ , ξϕ ϕ = ϕ(x), for locally almost all x ∈ G. Remark 1.51 If G is a discrete group, then the delta function δe is an identity in 1 (G) and [δe ] can serve as the cyclic vector ξϕ referred to in Proposition 1.50. When G is not discrete, ξϕ is selected as a cluster point of the image in Hϕ of an approximate identity in L1 (G). There are three immediate consequences of Proposition 1.50. First, if ϕ ∈ L∞ (G) is of positive type, then ϕ agrees locally almost everywhere with a continuous function. From now on, we shall therefore assume all functions of positive type to be continuous. The second consequence is that, for any x ∈ G, |ϕ(x)| = |πϕ (x)ξϕ , ξϕ ϕ | ≤ ξϕ 2ϕ = ϕ(e) and hence ϕ ∞ = ϕ(e). Thirdly, for x ∈ G, ϕ(x −1 ) = πϕ (x −1 )ξϕ , ξϕ ϕ = ξϕ , πϕ (x)ξϕ ϕ = ϕ(x).

1.5 Representations of L1 (G) and functions of positive type

33

For a given representation π with dim H(π) > 1 and ξ, η ∈ H(π ), one does not expect π (x)ξ, ξ  = π(x)η, η, for all x ∈ G, even when ξ and η are cyclic vectors and ξ = η . But there is some uniqueness in the other direction. Proposition 1.52 Let π and σ be cyclic representations of G with cyclic vectors ξ ∈ H(π ) and η ∈ H(σ ), respectively. If π(x)ξ, ξ  = σ (x)η, η, for all x ∈ G, then there exists a unitary map U ∈ HomG (π, σ ) such that U ξ = η. In particular, π and σ are equivalent. Corollary 1.53 If π is a cyclic representation of G with cyclic vector ξ and ϕ(x) = π(x)ξ, ξ , for all x ∈ G, then πϕ  π. Definition 1.54 Let π be a representation of G and ξ ∈ H(π ), ξ = 0. Define ϕ(x) = π(x)ξ, ξ , for x ∈ G. Then ϕ is called a function of positive type associated with π. If S is a set of representations of G and ϕ is a function of positive type, we may say ϕ is associated with S if ϕ is associated with σ for some σ ∈ S. Note that equivalent representations have the same functions of positive type associated with them. Thus, we can unambiguously refer to the functions of  positive type associated with π ∈ G. For a locally compact group G, the set P (G) of all continuous functions of positive type on G carries much of the representation theory of G in its structure. If P1 (G) = {ϕ ∈ P (G) : ϕ ∞ = 1} = {ϕ ∈ P (G) : ϕ(e) = 1}, then P (G) = {αϕ : α ≥ 0, ϕ ∈ P1 (G)}, so all the information is already in P1 (G). In general, the structure of P1 (G) is extremely complex. However, its study has revealed important and useful properties and led to the Gelfand– Raikov theorem, which states that any locally compact group has sufficiently many irreducible representations to separate points. The set P1 (G) is a convex subset of the unit sphere in L∞ (G). Moreover, Q = {αϕ : 0 ≤ α ≤ 1, ϕ ∈ P1 (G)} is a weak∗ -closed subset of the unit ball of L∞ (G), hence weak∗ -compact. The Krein–Milman theorem implies that Q is the weak∗ -closure of the convex hull of its extreme points. These extreme points consist of {0} ∪ E(G), where E(G) denotes the set of extreme points of P1 (G). This implies that E(G) has a convex hull that is weak∗ -dense in P1 (G). The significance of this rests on the following fundamental characterization of E(G). Theorem 1.55 Let G be a locally compact group and let ϕ ∈ P1 (G). Then ϕ ∈ E(G) if and only if πϕ is irreducible.

34

Basics

The w ∗ -topology on P1 (G) as a subset of L∞ (G) coincides with the topology of uniform convergence on compact sets. That is, a net (ϕα )α in  P1 (G) converges uniformly on compacta to 1ϕ ∈ P1 (G) if and only if G f (x)ϕα (x)dx → G f (x)ϕ(x)dx, for all f ∈ L (G). We can use the regular representation λG of G to construct elements of P1 (G) with desirable properties. For example, if z ∈ G, z = e, let V be a relatively compact neighborhood of e such that z ∈ V 2 and V −1 = V . Let h = |V |−1/2 1V . Then h ∈ L2 (G) and h 2 = 1. Let ϕ(x) = λG (x)h, h, for all x ∈ G. Then ϕ ∈ P1 (G) and ϕ(z) = 0, while ϕ(e) = 1. Since there is a net consisting of convex combinations of elements of E(G) that converges uniformly on compact sets, hence pointwise, to ϕ, there must exist ψ ∈ E(G)) such that ψ(z) = ψ(e) = 1. By Theorem 1.55, πψ is an irreducible representation of G. Moreover, πψ (z)ξψ , ξψ ψ = ψ(z) = ψ(e) = πψ (e)ξψ , ξψ ψ . Thus, πψ (z) = πψ (e). Since representations are homomorphisms, it is clear that any two distinct points in G can be separated by an irreducible representation of G. Theorem 1.56 (Gelfand–Raikov) Let G be a locally compact group, and let x, y ∈ G, x = y. Then there exists an irreducible representation π of G such that π (x) = π (y). The collection of irreducible representations of G is sufficiently rich to faithfully represent all the details of the structure of G. Also, the irreducible representations of G, when considered as ∗-representations of L1 (G), are collectively faithful on L1 (G). We conclude this section by introducing the notion of a positive definite function and pointing out that the bounded continuous positive definite functions are nothing but the continuous functions of positive type. A function ϕ : G → C is called positive definite if n 

αi αj ϕ(xj−1 xi ) ≥ 0,

i,j =1

for any x1 , . . . , xn ∈ G, α1 , . . . , αn ∈ C, and n ∈ N. Proposition 1.57 Let ϕ be a bounded continuous function on G. Then the following are equivalent. (i) ϕ is positive definite.  (ii) ϕ is of positive type, that is, G (f ∗ ∗ f )(x)ϕ(x)dx ≥ 0 for all f ∈ L1 (G).

1.6 C ∗ -algebras

35

1.6 C ∗ -algebras and weak containment of representations It will turn out that C ∗ -algebras play a major role in later chapters of this book, not only because associated with every locally compact group is its group C ∗ -algebra (see below). Therefore, even though the reader is expected to be somewhat familiar with C ∗ -algebra theory, we first recall some basic definitions and results. All these can be found, for instance, in Dixmier’s monograph [37]. A C ∗ -algebra is a Banach ∗-algebra A whose norm satisfies the so-called C ∗ -condition a ∗ a = a 2 for all a ∈ A. For the basic notions in representation theory, we can therefore refer to the preceding section. Definition 1.58 Let A be a C ∗ -algebra. (i) For a representation π of A, let ker(π ) = {a ∈ A : π (a) = 0} denote the kernel of π . (ii) A closed ideal I of A is called a primitive ideal if I = ker(π) for some irreducible representation π of A, and the collection Prim(A) of all primitive ideals of A is called the primitive ideal space of A. The primitive ideal space is equipped with the hull-kernel or Jacobson topology, which is most easily defined by describing the closure operation. For J ⊆ Prim(A), let J = {I ∈ Prim(A) : I ⊇ ∩J ∈J J }. With this topology, Prim(A) is always a T0 -space, but in general not a T1 -space.  denote the set of equivalence classes of irreducible representations of Let A A. If π and σ are equivalent representations, then ker(π ) = ker(σ ). Therefore  and consider the we may abuse notation safely and write ker(π ) for π ∈ A  → Prim(A), π → ker(π ). canonical map k : A  of A is equipped with the pull-back of the Definition 1.59 The dual space A  is open in A  if and hull-kernel topology on Prim(A). Thus a subset U of A −1  only if U = k (V ) for some open subset V of Prim(A). This topology on A is called the dual space topology or Fell topology. Next we have to introduce the notion of weak containment of representations. Definition 1.60 (i) Let S and T be two sets of representations of A. Then S is weakly contained in T if ∩τ ∈T ker(τ ) ⊆ ∩σ ∈S ker(σ ). We then write S ≺ T , and if S is a singleton, say {π}, simply π ≺ T . We say S and T are weakly equivalent (S ∼ T ) if S ≺ T and T ≺ S. (ii) For an arbitrary representation π of A, the support of π , supp π, is  such that σ ≺ π . defined to be the set of all σ ∈ A

36

Basics

Recall that a linear functional ϕ on A is positive if ϕ(a ∗ a) ≥ 0 for all a ∈ A. Such a ϕ is called a state if ϕ = 1 and a state ϕ is called a pure state if ψ and ϕ − ψ both positive imply that ψ = λϕ, for some 0 ≤ λ ≤ 1. If π is a representation of A in the Hilbert space H(π), then for any vector ξ in H(π ), ϕ defined by ϕ(a) = π(a)ξ, ξ , for a ∈ A, is a positive functional associated with π. We also say that a positive functional ϕ is associated with a set S of representations of A if ϕ is associated with some π ∈ S. It is very important to characterize weak containment in terms of positive functionals. Theorem 1.61 Let A be a C ∗ -algebra, π a representation of A, and S a set of representations of A. Then the following three conditions are equivalent. (i) π is weakly contained in S. (ii) Every nonzero positive functional on A associated with π is a w∗ -limit of finite linear combinations of positive functionals associated with S. (iii) Every nonzero positive functional ϕ on A associated with π is a w∗ -limit of finite sums ψ of positive functionals associated with S which satisfy

ψ ≤ ϕ . It follows from Theorem 1.61 and the definition of the dual space topology  that the support of a representation π of A is a closed subset of A. When all the representations involved are irreducible, Theorem 1.61 can be sharpened in two respects. Firstly, in (ii) and (iii) one does not need linear combinations and sums, respectively, anymore. Secondly, condition (ii) is only required for some nonzero functional. Moreover, (i) simply reflects the closure  operation on A.  and S ⊆ A.  Then the following conditions are equivTheorem 1.62 Let π ∈ A alent. (i) π is weakly contained in S.  (ii) π ∈ S, the closure of S in A. (iii) Some nonzero positive functional associated with π is a w∗ -limit of positive functionals associated with S. (iv) Every nonzero positive functional ϕ associated with π is the w∗ -limit of positive functionals ψ associated with S for which ψ ≤ ϕ . Both Theorems 1.61 and 1.62 can be found in Fell [46], [49], and Dixmier [37]. Now let G be a locally compact group. Although L1 (G), equipped with convolution, involution, and the norm · 1 , is a Banach ∗-algebra, it does not satisfy the C ∗ -condition. We are going to define a C ∗ -algebra associated with  G and use it to introduce an important topology on G.

1.6 C ∗ -algebras

37

There are five at first glance different seminorms we now define on L1 (G). For f ∈ L1 (G), let p1 (f ) = sup{ π(f ) : π is a representation of G}, p2 (f ) = sup{ π(f ) : π is a cyclic representation of G},  : π ∈ G}, p3 (f ) = sup{ π(f  ) ∗ p4 (f ) = sup{(G (f ∗ f )(x)ϕ(x)dx)1/2 : ϕ ∈ P1 (G)}, p5 (f ) = sup{( G (f ∗ ∗ f )(x)ϕ(x)dx)1/2 : ϕ ∈ E(G)}. Evidently, each pj is a seminorm on L1 (G). Proposition 1.63 Let G be a locally compact group. Then p1 (f ) = p2 (f ) = p3 (f ) = p4 (f ) = p5 (f ), for all f ∈ L1 (G). If f ∈ L1 (G), f = 0, then p1 (f ) ≥ λG (f ) > 0. It is then clear that any one of p1 to p5 can be used to define a new norm on L1 (G).  Definition 1.64 For f ∈ L1 (G), let f ∗ = sup{ π (f ) : π ∈ G}. Since we always have π(f ) ≤ f 1 for any representation π , f ∗ ≤

f 1 , for all f ∈ L1 (G). Also, for f ∈ L1 (G), 

f ∗ ∗ f ∗ = sup{ π(f ∗ ∗ f ) : π ∈ G}  = f 2∗ . = sup{ π(f ) 2 : π ∈ G} The other properties presented in the following proposition are even easier to check. Proposition 1.65 Let G be a locally compact group. For f, g ∈ L1 (G), (i) (ii) (iii) (iv)

f ∗ ≤ f 1 ,

f ∗ g ∗ ≤ f ∗ g ∗ ,

f ∗ ∗ = f ∗ ,

f ∗ ∗ f ∗ = f 2∗ .

Equipped with the new norm · ∗ , L1 (G) is a normed ∗-algebra satisfying Proposition 1.65(iv), the extra condition on the norm required of a C ∗ -algebra. However, (L1 (G), · ∗ ) is not complete unless G is a finite group. Definition 1.66 The group C ∗ -algebra of G, denoted C ∗ (G), is the completion of (L1 (G), · ∗ ) as a normed ∗-algebra. The norm · ∗ is designed so that any representation π of L1 (G) extends uniquely to a representation of C ∗ (G). We retain the notation π for this representation of C ∗ (G). A critical fact, easily verified, is that for two representations

38

Basics

π and σ of G, HomG (π, σ ) = HomC ∗ (G) (π, σ ).

(1.6)

Thus, just as in Proposition 1.47 for L1 (G), the concepts of equivalence and irreducibility of representations mean the same whether one is considering the representations as being of G or of C ∗ (G). For a representation π of G, π(C ∗ (G)) is a norm closed ∗-subalgebra of B(H(π )). Indeed, π (C ∗ (G)) = π(L1 (G)), where the closure is in the operator norm. Since there is a bijection between equivalence classes of unitary representations of G and representations of C ∗ (G) and since, due to (1.6), this bijection preserves irreducibility, we can simply transfer the dual space topology and the notion of weak containment from the C ∗ -algebra context to the group context. In the sequel, for a representation π of G, ker(π) will always denote the kernel of π considered as a representation of C ∗ (G). Definition 1.67 Let G be a locally compact group.  is the topology which makes the (i) The dual space or Fell topology on G ∗ (G) a homeomorphism.  →C bijection G (ii) If S and T are sets of unitary representations of G, then S is weakly contained in T if ∩τ ∈T ker(τ ) ⊆ ∩σ ∈S ker(σ ).  which are weakly (iii) The support of a representation π is the set of all ρ ∈ G contained in π . (iv) Prim(C ∗ (G)) will also be referred to as the primitive ideal space of G and denoted Prim(G). Positive linear functionals on C ∗ (G) can be represented by continuous func∗ tions of positive type. In fact, for any positive  linear functional φ on C (G), 1there is a unique ϕ ∈ P (G) such that φ(f ) = G f (x)ϕ(x) dx, for all f ∈ L (G). Moreover, φ is a state if and only if ϕ ∈ P1 (G) and φ is a pure state if and only if ϕ ∈ E(G). Conversely, if ϕ ∈ P (G), there is a unique positive linear functional φ on C ∗ (G) such that, for all f ∈ L1 (G),  φ(f ) = f (x)ϕ(x) dx. (1.7) G ∗

Since the w -topology on P (G) agrees with the topology of uniform convergence on compact sets, weak containment can be expressed as follows. Proposition 1.68 Let S be a set of representations of G and π a representation of G. Then π ≺ S if and only if, for any ϕ ∈ P (G) associated with π , there

1.7 Abelian locally compact groups

39

exists a net (ψα )α , where each ψα is a linear combination of functions of positive type associated with S, such that ψα → ϕ uniformly on compacta in G. For convenience we also reformulate Theorem 1.62 in the group context.  and π ∈ G.  The following are equivalent. Proposition 1.69 Let S ⊆ G (i) π ∈ S. (ii) π ≺ S. (iii) At least one nonzero function of positive type associated with π is the uniform on compacta limit of a net of functions of positive type associated with S. (iv) Every function of positive type associated with π is the uniform on compacta limit of a net of functions of positive type associated with S.  is discrete. If G is discrete, then G  Proposition 1.70 If G is compact, then G is compact.  It can be a difficult problem to completely describe the topology of G for an arbitrary G even when a satisfactory description of the members of  is available. The overarching theme of Chapters 5, 6, and 7 is developing G  when circumstances are favorable techniques for working out the topology of G and using knowledge of the topology in applications. We end this section by  naturally arise. pointing out two ways in which closed subsets of G If N is a closed normal subgroup of G and q : G → G/N is the quotient homomorphism, then, for any representation π of G/N, π ◦ q is a representation of G which annihilates N . Clearly HomG/N (π, σ ) = HomG (π ◦ q, σ ◦ q), for any two representations π and σ of G/N. Proposition 1.71 Let N be a closed normal subgroup of G. The map π →  consisting of  with the closed subset of G π ◦ q is a homeomorphism of G/N  those elements of G which annihilate N .   with its image in G When no confusion can result, we will identify G/N under the homeomorphism of Proposition 1.71 and write π ◦ q as simply π .

1.7 Abelian locally compact groups We will now specialize to abelian groups where a rich theory generalizing classical Fourier analysis emerges. A basic reference for harmonic analysis in abelian groups is Rudin [138]. We will mainly only present the notation and basic results that are necessary for our later chapters.

40

Basics

For this section, suppose G is an abelian locally compact group. It follows from Proposition 1.35 that any irreducible representation of G is onedimensional. For one-dimensional representations, equivalence is equality as  consists of all the continuous homomorphisms of G into T, functions. Thus, G the characters of G. The tensor product of two characters χ and ψ of G is simply the pointwise product χ ψ, which is again a character (see Remark 1.45). Moreover, χ is also a character and χ χ = 1, the constant 1 homomorphism.  is the topology of uniform convergence on compacta (comThe topology on G pare with Proposition 1.68) and pointwise product and complex conjugation are continuous with respect to this topology.  carries the Definition 1.72 Let G be a locally compact abelian group. Then G  structure of a locally compact abelian group as well. We refer to G as the dual group of G. Example 1.73 Let G = R. For each γ ∈ R define χγ : R → T by χγ (t) = eiγ t , for all t ∈ R. Then γ → χγ is a topological group isomorphism of R with  R. Note that it will often be convenient for us to parametrize  R by scaling by 2πiγ t , t ∈ R, for each γ ∈ R. 2π. That is, by considering χγ (t) = e Example 1.74 Let G = T. For each n ∈ Z define ψn (z) = zn , for all z ∈ T. Then n → ψn identifies Z with  T. Let q(t) = e2πit , for t ∈ R. Then q is a continuous homomorphism of R onto T with kernel Z. This identifies T with  note that ψn ◦ q(t) = e2πint , for t ∈ R. This embeds T = R/Z, R/Z. For ψn ∈   R/Z as a closed subset of  R as in Proposition 1.70. Example 1.75 Let G = Z. For each z ∈ T, let σz (n) = zn , for all n ∈ Z. Then Z. z → σz identifies T with  Example 1.76 Let n ∈ N, n > 1 and G = Z/nZ. By Proposition 1.71,  = {σz : z ∈ Tn }, where Tn = {z ∈ T : zn = 1}, the nth roots of unity. Z/nZ Many of the frequently occurring abelian locally compact groups are products of finitely many of the above examples, so the following identification is helpful to quickly determine dual groups for such products. Example 1.77 Let G1 , . . . , Gn be abelian locally compact groups and form n n n   i=1 Gi and  i=1 Gi , the product groups. For (χ1 , . . . , χn ) ∈ i=1 Gi and (x1 , . . . , xn ) ∈ ni=1 Gi , let (χ1 , . . . , χn )(x1 , . . . , xn ) = χ1 (x1 ) · . . . · χn (xn ).  Thus, we can naturally consider (χ1 , . . . , χn ) as a character of ni=1 Gi . All n n   characters of ni=1 Gi arise this way. Therefore,  i=1 Gi = i=1 Gi .

1.7 Abelian locally compact groups

41

For a locally compact abelian group G, L1 (G) is a commutative Banach  f → χ(f ) is a nonzero multiplicative linear functional ∗-algebra. For χ ∈ G,  can be identified with the on L1 (G) and they are all of this form. Thus, G 1 spectrum of L (G). Indeed, the Fourier transform defined below, and its properties, are a major motivation for the general Gelfand transform for general commutative Banach algebras. Definition 1.78 For f ∈ L1 (G), the Fourier transform of f is the function  → C defined by f : G   f (χ) = χ(f ) = f (x)χ(x) dx, G

 for all χ ∈ G. Many of the properties of the Fourier transform are summed up in the follow as a commutative C ∗ -algebra under pointwise ing statement. We view C0 (G) operations and equipped with the norm · ∞ . Proposition 1.79 Let G be a locally compact abelian group. The map f → f  and the image of L1 (G) is an injective ∗-homomorphism of L1 (G) into C0 (G)  under the Fourier transform is dense in C0 (G). In light of Definition 1.64, f ∗ = f ∞ , for all f ∈ L1 (G). Thus, the  This Fourier transform extends to a C ∗ -isomorphism of C ∗ (G) with C0 (G). sums up one aspect of the Fourier transform, but there is also its important role as a unitary map between L2 -spaces. Theorem 1.80 (Plancherel theorem) Let G be a locally compact abelian group.  can be simultaneously chosen so that f 2 = Then Haar measures on G and G 1

f 2 , for any f ∈ L (G) ∩ L2 (G). Moreover, {f : f ∈ L1 (G) ∩ L2 (G)} is  and so there is a unitary map F : L2 (G) → L2 (G)  such that dense in L2 (G) 1 2 F (f ) = f, for all f ∈ L (G) ∩ L (G). Unless otherwise indicated, we will always assume that Haar measures are scaled so that the Plancherel equality holds. The unitary map F in Theorem 1.80 is called the Plancherel transform. Example 1.81 When G = R, suppose  R is parametrized by {χγ : γ ∈ R}, 2πiγ t , for t ∈ R. Then we can write where χγ (t) = e  f(γ ) = f (t)e2πiγ t dt, R

  2 where we are writing γ in place of χγ . Then  R |f (γ )| dγ = for γ ∈ R, 2 1 2  R |f (t)| dt, for all f ∈ L (R) ∩ L (R). Therefore, both R and R can be equipped with unscaled Lebesgue measure and the Plancherel equality holds.

42

Basics

Remark 1.82 If G is a locally compact abelian group and λG denotes its left regular representation, then a straightforward calculation shows, for any  and χ ∈ G,  that x ∈ G, ξ ∈ L2 (G) FλG (x)F −1 ξ (χ) = χ(x)ξ (χ ), where F is the unitary map in Theorem 1.80. Thus, if we define the modulation  by MG (x)ξ (χ) = χ (x)ξ (χ ), for all x ∈ representation MG of G on L2 (G) 2   then λG  MG . G, ξ ∈ L (G), and almost all χ ∈ G, In Remark 1.36 it was observed that λG is reducible if G is not the trivial group. Now, for abelian G at least, we can determine many closed λG -invariant subspaces of L2 (G). From the simple multiplicative nature of MG , one sees that the support of MG (x)ξ will be the same as the support of ξ . If is any Haar-measurable  of positive Haar measure, let subset of G  : ξ (χ) = 0, for almost all χ ∈ G  \ }. L2 ( ) = {ξ ∈ L2 (G) 2  that is MG -invariant. Thus, H

Then L2 ( ) is a closed subspace of L2 (G) = −1 2 F L ( ) is λG -invariant. 1  ∨ Definition 1.83 For  ξ ∈ L (G), define the inverse Fourier transform ξ of ξ ∨ on G by ξ (x) = G  ξ (χ)χ(x) dχ , for x ∈ G. Let A(G) denote the image of  under the inverse Fourier transform. Then A(G) is a dense subalgebra L1 (G) of C0 (G).

The inverse Fourier transform terminology is justified by the following inversion result. Theorem 1.84 (Inversion theorem) Let G be a locally compact abelian group.  and Let f ∈ L1 (G) ∩ A(G). Then f ∈ L1 (G)  f(χ)χ(x) dχ , f (x) =  G

for all x ∈ G. Elements of a locally compact abelian group may be considered as characters on the dual group. That is, for a locally compact group G and x ∈ G, the map  into T. χ → χ (x) is a continuous homomorphism of G Theorem 1.85 (Pontryagin duality theorem) Let G be a locally compact  The map x → αx abelian group. For each x ∈ G, let αx (χ) = χ(x), for χ ∈ G.  is a topological group isomorphism of G with the dual group of G.

1.7 Abelian locally compact groups

43

The Pontryagin duality theorem allows us to derive a number of further important results. The first one is a duality between quotient groups of a locally  If H is a closed subgroup of G, compact abelian group G and subgroups of G. ⊥   is a closed subgroup then H = {χ ∈ G : χ|H ≡ 1}, the annihilator of H in G,   → H⊥ of G. We already know from Proposition 1.71 that the map ϕ : G/H defined by ϕ(ω) = ω ◦ q, where q : G → G/H is the quotient homomorphism, is a homeomorphism. In this case, it is a group isomorphism as well. Moreover,  we have (H ⊥ )⊥ = H . Using this, it is identifying G with the dual group of G, easy to show the following. Proposition 1.86 Let H be a closed subgroup of G and define a map ψ :  ⊥→H  by ψ(χ H ⊥ ) = χ |H . Then ψ is a topological group isomorphism. G/H In particular, every character of H extends to a character of G. The second application of the duality theorem concerns the structure of compactly generated abelian groups. Proposition 1.87 Every compactly generated locally compact abelian group is topologically isomorphic to a direct product Rn × Zm × C, where n, m ∈ N0 and C is a compact group. Finally, Proposition 1.70 together with Theorem 1.85 yield Proposition 1.88 Let G be an abelian locally compact group. Then  is compact. (i) G is discrete if and only if G  is discrete. (ii) G is compact if and only if G  denotes the measure algebra of G.  For μ ∈ M(G),  define Recall that M(G)  ∨ μ on G by μ (x) = G  χ(x) dμ(χ), for all x ∈ G. In light of the facts  and the Banach space dual of C0 (G)  is that C ∗ (G) identifies with C0 (G)  M(G), we can build representations of the various important functions of pos = {μ ∈ M(G)  : μ(E) ≥ itive type in the case of abelian groups. Let M + (G) + +  = {μ ∈ M (G)  : μ(G)  = 1}, the proba and M1 (G) 0, for all Borel E ⊆ G}  For each χ ∈ G,  δχ ∈ M1+ (G), where δχ is the one-point bility measures on G. mass at χ . ∨

Theorem 1.89 (Bochner’s theorem) Let G be a locally compact abelian group. Then  (i) P (G) = {μ∨ : μ ∈ M + (G)}. +  ∨ (ii) P1 (G) = {μ : μ ∈ M1 (G)}.  (iii) E(G) = {δχ∨ : χ ∈ G}.

44

Basics

When we develop Mackey’s theory for locally compact groups with an appropriate closed abelian normal subgroup in Chapter 4, we will draw on many of the facts of abelian harmonic analysis that were presented in this section.

1.8 Notes and references The concept of a locally compact group, and the realization that such groups were the most natural environment for a common generalization of Fourier analysis and the representation theory of finite groups, evolved over the first half of the twentieth century. The books of Weil [150] and Pontryagin [125] represent notable contributions to the articulation of this abstract harmonic analysis. Other important milestones were Theorem 1.56 of Gelfand and Raikov [59], the Gelfand–Naimark–Segal construction [58], [143], and the theory of functions of positive type developed by Godement [63]. A comprehensive treatment of the theory of locally compact groups, including a proof of the existence of a left Haar measure and an introduction to representation theory, can be found in Hewitt and Ross [73]. Our presentation of quasi-invariant measures on coset spaces in Section 1.3 draws upon the treatment of similar material in Fell and Doran [53] and Folland [55]. A good source for full proofs of the results in Sections 1.4, 1.5, and 1.6 is Dixmier [37], where representation theory is developed in the C ∗ -algebra context first and then applied to groups through the group C ∗ -algebra. The basic theory of abelian harmonic analysis is available in Rudin [138], with other results in Section 1.7 being interpretations of Section 1.6 in the abelian situation.

2 Induced representations

The single most important method for producing representations of a locally compact group is the procedure of inducing representations from subgroups. If H is a closed subgroup of a locally compact group G and π is a unitary representation of H , then indG H π is a unitary representation of G that is constructed by combining the action of π with the algebraic and measure-theoretic inter-relation of G, H , and G/H . The definition of indG H π in full generality is technical and somewhat complex. As a result, several approaches to the induction procedure have been developed that lead to unitarily equivalent realizations. Since any one of these realizations is convenient for some purpose, we introduce each of the various approaches and establish the relevant equivalences in Sections 2.3 and 2.4. Much of the complexity of the definition of indG H π in general is due to measure-theoretic delicacy in the action of G on the quotient space G/H . These issues all disappear when G/H is discrete. Therefore, there is some value in first developing the definition of indG H π and exploring basic questions of irreducibility and equivalence when H is an open subgroup. This is done in Sections 2.1 and 2.2. The results presented in Section 2.2 are particularly valuable in the study of discrete groups. Several examples of induction from an open subgroup are presented in Section 2.2. After the general definition is available, we devote all of Section 2.5 to two important examples of induced representations. First, we study a representation of the affine group of the real line that is formed by inducing the trivial representation from the subgroup of dilations up to the whole group. The resulting representation of the affine group is the classical representation that arises from its natural action on R by translations and dilations. Also in

45

46

Induced representations

Section 2.5, we formulate the so-called principle series of representations of the special linear group, SL(2, R). These representations are formed by inducing characters from the subgroup of SL(2, R) consisting of upper-triangular matrices. We draw a connection between SL(2, R) and the affine group as we investigate the reducibility properties of the induced representations we have formed. Discovering whether a given representation is irreducible or not can be difficult and it is part of the purpose of this section to illustrate some techniques. In the actual practice of using induced representations there are a variety of basic calculation rules that enable one to combine the inducing procedure with other operations on representations. These rules are presented in Sections 2.6 and 2.7. For instance, the induction process commutes with the processes of forming the conjugate representation, pulling back representations of quotient groups, and forming direct sums. Note that an important consequence of this latter property is that a necessary condition for indG H π to be irreducible is that π be irreducible. Moreover, a representation π of H and its associated representation on aH a −1 , a ∈ G, lead to equivalent induced representations. Probably the most important among all these rules is the theorem on induction in stages. In Section 2.8 we discuss the question of how the induction process behaves in relation to building inner and outer tensor products of representations. As an application, we present in Section 2.9 a version of the Frobenius reciprocity theorem that generalizes the classical reciprocity theorem for compact groups.

2.1 Inducing from an open subgroup We first introduce the construction of induced representations in a setting that is useful in its own right and where the algebraic ingredients become quite transparent. Let G be a locally compact group and suppose that H is an open subgroup of G. Then H is automatically closed, G/H is a discrete space, and the counting measure on G/H is an invariant measure. Definition 2.1 Let π be a representation of H on the Hilbert space H(π). Construct a new space of H(π )-valued functions on G: H(G, π ) = {ξ : G → H(π) : ξ (xh) = π (h−1 )ξ (x), for x ∈ G, h ∈ H, 

ξ (x) 2 < ∞}. and xH ∈G/H

2.1 Inducing from an open subgroup

47

Note that xH ∈G/H ξ (x) 2 is well defined because of the covariance property ξ (xh) = π (h−1 )ξ (x) and the fact that π(h−1 ) is unitary for all h. Likewise, the sum in the following definition of an inner product is well defined. For ξ, η ∈ H(G, π ), let  ξ (x), η(x). ξ, η = xH ∈G/H

The induced representation indG H π is the representation of G on H(G, π ) defined by, for x, y ∈ G, ξ ∈ H(G, π ), −1 indG H π(x)ξ (y) = ξ (x y).

In order to justify calling indG H π a representation, we have to show that: (i) H(G, π ) is a Hilbert space; (ii) indG H π is a homomorphism of G into the group of unitaries of H(G, π ); (iii) x → indG H π (x)ξ, η is continuous, for ξ, η ∈ H(G, π ). Clearly H(G, π ) is a vector space and · , · is an inner product. To see completeness note that, since G/H is discrete, any choice γ : G/H → G of a section of the H -cosets is a regular section. Fix a section γ . Then the inner product in H(G, π ) can be written  ξ, η = ξ (γ (ω)), η(γ (ω)). ω∈G/H

There is a map (depending on the choice of γ ) W of l 2 (G/H, H(π )) into H(G, π ) defined by, for f ∈ l 2 (G/H, H(π )), ω ∈ G/H, h ∈ H , Wf (γ (ω)h) = π (h−1 )f (ω). It is easy to see that W is a surjective isometry. Since l 2 (G/H, H(π )) is complete, so is H(G, π ). Thus property (i) holds. Because the counting measure on G/H is invariant under the action of G, each indG H π (x) is a unitary operator. Clearly G G indG H π(x1 ) indH π(x2 ) = indH π (x1 x2 ),

for x1 , x2 ∈ G, so property (ii) holds. It takes more work to show the continuity required for (iii). In proceeding to do this, we introduce a technique that can be generalized to the situation where H is not necessarily open. Let F(G, π ) denote the dense subspace of H(G, π ) consisting of elements that are nonzero on only a finite number of H -cosets.

48

Induced representations

Proposition 2.2 Let f : G → H(π) be a continuous function with compact support. Define ξf : G → H(π ) by  ξf (x) = π(h)f (xh)dh H

for all x ∈ G. Then: (i) ξf ∈ F(G, π ); (ii) for any ξ ∈ F(G, π ) there exists a continuous function g : G → H(π ) with compact support such that ξ = ξg ; (iii) every ξ ∈ F(G, π ) is a left uniformly continuous function on G. Proof (i) For x ∈ G and k ∈ H ,   ξf (xk) = π(h)f (xkh)dh = π(k −1 h)f (xh)dh = π (k −1 )ξf (x) H

H

and q(supp f ) is a compact, hence finite subset of G/H , where q : G → G/H is the quotient map. Thus ξf ∈ F(G, π ). (ii) Let ξ ∈ F(G, π ). Note first that ξ is continuous. To see this, let x ∈ G and ε > 0. Since π is continuous there exists a neighborhood U of e, U ⊆ H , such that π (h)ξ (x) − ξ (x) < ε for all h ∈ U . Then, if y ∈ xU , say y = xh for some h ∈ U ,

ξ (x) − ξ (y) = ξ (x) − π(h−1 )ξ (x) = π(h)ξ (x) − ξ (x) < ε. Let F ⊆ G/H be a finite subset such that ξ (x) = 0 for x ∈ q −1 (F ). Let ψ ∈ Cc+ (H ) such that H ψ(h)dh = 1. Define ϕ on G by, for ω ∈ G/H and h ∈ H ,  ψ(h) if ω ∈ F, ϕ(γ (ω)h) = 0 if ω ∈ F. Let g = ϕξ . Then g is an H(π )-valued continuous function with compact support on G. Moreover, for any x ∈ G,    ξg (x) = ϕ(xh)π(h)ξ (xh)dh = ϕ(xh)dh ξ (x) = ξ (x). H

H

(iii) Let ξ ∈ F(G, π ) and express ξ = ξg for some g as in (ii). Let F ⊆ G/H be finite such that ξ (x) = 0 for x ∈ G \ q −1 (F ). Fix a compact neighborhood V of e and let M = V · supp g and K = (M −1 M) ∩ H. Then M and K are compact. Let ε > 0 be arbitrary. Since g is continuous with compact support, it is uniformly continuous. Let U be a neighborhood of e such that U ⊆ V and xy −1 ∈ U implies g(x) − g(y) < ε|K|−1 (where |K| denotes the Haar measure of K). Suppose x, y ∈ G satisfy xy −1 ∈ U . If x ∈ MH , then y ∈ (supp g)H , so

2.1 Inducing from an open subgroup

49

g(xh) = g(yh) = 0 for all h ∈ H . If x ∈ MH , let h ∈ H such that xh ∈ M. For any k, if yhk ∈ supp g, then xhk = (xy −1 )yhk ∈ V supp g = M, so k ∈ (M −1 M) ∩ H = K. Likewise, xhk ∈ supp g implies k ∈ K. Thus, k ∈ H \ K implies g(xhk) = g(yhk) = 0. Now      π(k)(g(xk) − g(yk))dk  ≤

g(xhk) − g(yhk) dk

ξg (x) − ξg (y) =  H  H =

g(xhk) − g(yhk) dk < ε. K



Thus ξ = ξg is left uniformly continuous.

For any x ∈ G and ξ ∈ H(G, π), 1xH ξ ∈ F(G, π). So, by Proposition 2.2(iii), each ξ ∈ H(G, π) is left uniformly continuous on G. Let s, t ∈ G. Then  G 2

indG

ξ (s −1 · γ (ω)) − ξ (t −1 · γ (ω)) 2 . H π (s)ξ − indH π(t)ξ = ω∈G/H

π (x)ξ is continuous from G into H(G, π). The weaker conThus x → tinuity in property (iii) follows. This completes the verification that indG H π is indeed a (continuous unitary) representation of G. We can now use H(indG H π) to denote H(G, π). There is an alternate description of indG H π, an equivalent representation actually, that has the advantage of acting on a more concretely presented Hilbert space. Recall that    l 2 (G/H, H(π)) = η : G/H → H(π ) :

η(ω) 2 < ∞ indG H

ω∈G/H

is a Hilbert space with inner product η1 , η2  = ω∈G/H η1 (ω), η2 (ω), for η1 , η2 ∈ l 2 (G/H, H(π)). We can also think of l 2 (G/H, H(π )) as the Hilbert space direct sum ⊕ω∈G/H H(π ) or the tensor product l 2 (G/H ) ⊗ H(π ). For a given choice of section γ : G/H → G, recall that W : l 2 (G/H, H(π )) → H(indG H π) is defined by, for f ∈ l 2 (G/H, H(π )), ω ∈ G/H, h ∈ H , Wf (γ (ω)h) = π (h−1 )f (ω). As we saw, W is a unitary map between Hilbert spaces. The inverse map 2 −1 ξ = ξ ◦ γ , that W −1 : H(indG H π ) → l (G/H, H(π )) is given simply by W is, W −1 ξ (ω) = ξ (γ (ω))

(ω ∈ G/H, ξ ∈ H(indG H π )).

50

Induced representations

For x ∈ G, x = γ (q(x))[γ (q(x))−1 x] is the unique way of writing x in the form γ (ω)h with ω ∈ G/H and h ∈ H , where q : G → G/H is the quotient map. Given η ∈ l 2 (G/H, H(π )), let ξ = W η. Then ξ (x) = π (x −1 γ (q(x)))η(q(x))

(x ∈ G).

Now define a representation of G on l 2 (G/H, H(π )) by using W to transfer the action. That is, for x ∈ G let Uγπ (x) = W −1 indG H π (x)W. Proposition 2.3 For η ∈ l 2 (G/H, H(π )) and x ∈ G, Uγπ (x)η(ω) = π [γ (ω)−1 xγ (x −1 · ω)]η(x −1 · ω) for all ω ∈ G/H . Proof First note that, for x ∈ G and ω ∈ G/H , x −1 γ (ω) = γ (x −1 · ω)[γ (x −1 · ω)−1 x −1 γ (ω)] and γ (x −1 · ω)−1 x −1 γ (ω) ∈ H . Now we simply compute G −1 W −1 indG H π(x)W η(ω) = indH π(x)W η(γ (ω)) = W η(x γ (ω))

= W η(γ (x −1 · ω)[γ (x −1 · ω)−1 x −1 γ (ω)]) = π[γ (ω)−1 xγ (x −1 · ω)]η(x −1 · ω).



Different choices of the section γ give, of course, different formulae for Uγπ , but they are each equivalent to indG H π. Let π1 and π2 be two representations of H . The reader can easily verify that G π1  π2 implies indG H π1  indH π2 . Example 2.4 Let G = Z and H = {0}. Let τ be the only irreducible representation of H ; that is, τ is given by τ (0) = 1 on the one-dimensional Hilbert space. Then H(indZ{0} τ ) = l 2 (Z) and indZ{0} τ (n)f (m) = f (m − n) = λ(n)f (m), for m, n ∈ Z and f ∈ l 2 (Z). Thus indZ{0} τ is just the left regular representation λ of Z. The easiest way to describe nontrivial invariant subspaces for the regular representation of an abelian group is to take the Fourier transform. Let F : l 2 (Z) → L2 (T) be the unitary map that restricts to the Fourier transform on l 1 (Z). That is, Ff = f for f ∈ l 1 (Z), where  f (n)zn (z ∈ T). f(z) = n∈Z

Then Fλ(n)F −1 g(z) = zn g(z) for any g ∈ L2 (T), n ∈ T, z ∈ Z. From this we see that, for every Borel subset E of T, lE2 = {f ∈ l 2 (Z) : Ff (z) = 0 for almost all z ∈ T \ E}

2.2 Conditions for irreducibility

51

is a closed indZ{0} τ -invariant subspace of l 2 (Z). Thus, in particular, indZ{0} τ is reducible.

2.2 Conditions for irreducibility of induced representations In general, giving useful conditions that imply indG H π is irreducible can be difficult. In this section, we take some small steps in understanding this issue. Recall that if π1 and π2 are two representations of H , then the intertwining space of π1 and π2 is HomH (π1 , π2 ) = {T ∈ B(H(π1 ), H(π2 )) : T π1 (h) = π2 (h)T , for all h ∈ H }. A representation π of H is irreducible if and only if HomH (π, π ) = HomH (π ) = CI. As one might hope, there is a connection between the spaces G HomH (π1 , π2 ) and HomG (indG H π1 , indH π2 ). We shall see that the first embeds onto a special subspace of the second. In the case of an open subgroup it is easier to work with Uγπ1 and Uγπ2 where γ is a fixed section from G/H into G, which we choose so that γ (q(e)) = e. For i = 1, 2 and ω ∈ G/H let Kωπi = {η ∈ l 2 (G/H, H(πi )) : η(ω ) = 0

if ω = ω}.

Then each Kωπi is naturally identifiable with H(πi ) and l 2 (G/H, H(πi )) is the orthogonal sum over ω ∈ G/H of the subspaces Kωπi . Moreover, let C = {S ∈ HomG (Uγπ1 , Uγπ2 ) : S(Kωπ1 ) ⊆ Kωπ2

for all ω ∈ G/H }.

Then C is a WOT-closed subspace of HomG (Uγπ1 , Uγπ2 ). For any T ∈ HomH (π1 , π2 ) and η ∈ l 2 (G/H, H(π1 )), define Tη : G/H → H(π2 ) by Tη(ω) = T (η(ω)), ω ∈ G/H. Then Tη ∈ l 2 (G/H, H(π2 )), T(Kωπ1 ) ⊆ Kωπ2 for every ω ∈ G/H and for any x ∈ G and η ∈ l 2 (G/H, H(π1 )), TUγπ1 (x)η(ω) = T [Uγπ1 (x)η(ω)] = T [π1 (γ (ω)−1 xγ (x −1 · ω)]η(x −1 · ω) = π2 [γ (ω)−1 xγ (x −1 · ω)]T [η(x −1 · ω)] = Uγπ2 (x)Tη(ω), for all ω ∈ G/H . Therefore T ∈ C. Proposition 2.5 The map T → T is a vector space isomorphism of HomH (π1 , π2 ) onto C. Proof It is clear that T → T is linear and one-to-one. To see that it is onto, fix an element S ∈ C.

52

Induced representations Let ω0 = q(e), so γ (ω0 ) = e. For each i = 1, 2 and v ∈ H(πi ), define  v if ω = ω0 , v δω0 (ω) = 0 if ω = ω0 ,

ω ∈ G/H . Then Kωπi0 = {δωv 0 : v ∈ H(πi )} for i = 1, 2. Define T from H(π1 ) to H(π2 ) by, for v ∈ H(π1 ), T v = (Sδωv 0 )(ω0 ). Then T is a linear operator and T ≤ S . Note that γ (ω0 )−1 hγ (h−1 · ω0 ) = h, for h ∈ H . Thus, for any v ∈ H(π1 ) and h ∈ H, Uγπ1 (h)δωv 0 (ω) = π1 [γ (ω)−1 hγ (h−1 · ω)]δωv 0 (h−1 · ω) = δωπ10 (h)v (ω) for ω ∈ G/H . Therefore T π1 (h)v = (SUγπ1 (h)v )(ω0 ) = (SUγπ1 (h)δωv 0 )(ω0 ) = (Uγπ2 (h)Sδωv 0 )(ω0 ) = π2 (h)Sδωv 0 (h−1 · ω0 ) = π2 (h)T v, for all h ∈ H and v ∈ H(π1 ); whence T ∈ HomH (π1 , π2 ). Moreover, for any η ∈ l 2 (G/H, H(π2 )) and all ω ∈ G/H , (T˜ η)(ω) = T [η(ω)] = (Sδωη(ω) )(ω0 ) = (Sη)(ω). 0 Thus T = S.



Let us investigate a special situation to see how Proposition 2.5 can be used to establish irreducibility of an induced representation. Let H be a locally compact abelian group and let D be a discrete group of automorphisms of H . We write the action of d ∈ D on h ∈ H as d · h. Form the semidirect product G = H  D. That is, G = {(h, d) : h ∈ H, d ∈ D} with group product (h, d)(h , d  ) = (h(d · h ), dd  ). The group D also acts on the  of H by d · χ(h) = χ(d −1 · h), for h ∈ H, χ ∈ H , and d ∈ D. dual group H We identify H and D with the obvious subgroups of G. Choose the section γ : D = G/H → G to be the monomorphism γ (d) = (eH , d), where eH is the identity of H . We simply write U χ for Uγχ . Note that H(U χ ) = l 2 (D). The reader can easily verify that the formula of Proposition 2.3 now becomes, for , η ∈ l 2 (D), and (h, d) ∈ G, χ ∈H U χ (h, d)η(c) = c · χ (h)η(d −1 c)

(c ∈ D).

With this preparation, we can formulate a satisfying condition that implies irreducibility of some induced representations.

2.2 Conditions for irreducibility

53

Theorem 2.6 Suppose that G = H  D, where H is an abelian locally com. Then indG pact group and D is a discrete group. Let χ ∈ H H χ is an irreducible representation of G if and only if d · χ = χ, for all d ∈ D, d = eD .  and d · χ = χ for all d = eD . Thus, d → d · χ is a Proof Suppose χ ∈ H bijection of D with the orbit of χ . We show that the representation U χ (which is equivalent to indG H χ) is irreducible. For each c ∈ D, let Kc = {η ∈ l 2 (D) : η(d) = 0 for d ∈ D, d = c}, and let Pc denote the orthogonal projection of l 2 (D) onto Kc . Recall that, for any f ∈ L1 (G), the operator U χ (f ) ∈ B(l 2 (D)) is given by  U χ (f )η(c) = f (x)U χ (x)η(c)dx, G

η ∈ l 2 (D), c ∈ D. The Haar integral on G is given by  g(h, d)dh H

d∈D

for any integrable function g on G. For f ∈ L1 (G) and d ∈ D, let fd ∈ L1 (H ) be defined by fd (h) = f (h, d), for all h ∈ H . Let fd denote the Fourier trans. Then, for η ∈ l 2 (D) and c ∈ D, form of fd on H  U χ (f )η(c) = f (h, d)U χ (h, d)η(c)dh H

d∈D

=

  H

d∈D

=



 fd (h)(c · χ)(h)dh η(d −1 c)

fd (c · χ)η(d −1 c).

d∈D

For a fixed c ∈ D, there exists a net, {f A }A∈A , in L1 (G) such that U χ (f A ) → Pc in the strong operator topology. To see this, let A denote the set of all finite subsets of D \ {c}, partially ordered by inclusion. For each A ∈ A, {d · χ : d ∈  that does not contain the point c · χ. By A} is a finite, hence closed, subset of H A ≤ 1, regularity of L1 (H ), there exists a function g A ∈ L1 (H ) such that 0 ≤ g   A A g (c · χ ) = 1, and g (d · χ) = 0, for all d ∈ A. Define f A ∈ L1 (G) by  f (h, d) = A

for all (h, d) ∈ G.

g A (h) 0

if if

d = eD , d = eD ,

54

Induced representations Fix η ∈ l 2 (D). For any b ∈ D,  −1 A  A U χ (f A )η(b) = f d (b · χ)η(d b) = g (b · χ )η(b). d∈D

For any ε > 0, let F ∈ A be such that 

|η(b)|2 < ε.

b∈D\(F ∪{c})

Then, for any A ∈ A with F ⊆ A, denoting D \ (A ∪ {c}) by X, we have   A (b · χ)η(b)|2 ≤

U χ (f A )η − Pc η 22 = |g |η(b)|2 < ε. b∈X

b∈X

Therefore, limA∈A U χ (f A ) = Pc in the strong operator topology. If S ∈ HomG (U χ ), then SU χ (f A ) = U χ (f A )S, for all A ∈ A, and multiplication of operators is SOT-continuous. Thus, SPc = Pc S, for all c ∈ D. Hence the algebra C of Proposition 2.5 coincides with HomG (U χ ). Therefore, applying Proposition 2.5 with π1 = π2 = χ, we have HomG (U χ ) is isomorphic as a vector space with HomH (χ) = C. This shows that U χ , and hence indG H χ , is irreducible. To prove the converse direction, let Dχ = {b ∈ D : b · χ = χ } and let ρ denote the right regular representation of D on l 2 (D). If b ∈ Dχ and (h, d) ∈ G, then, for any η ∈ l 2 (D), c ∈ D, ρ(b)U χ (h, d)η(c) = U χ (h, d)η(cb) = ((cb) · χ)(h)η(d −1 cb) = (c · χ)(h)η(d −1 cb) = (c · χ)(h)ρ(b)η(d −1 c) = U χ (h, d)ρ(b)η(c). Therefore, if b ∈ Dχ , then ρ(b) ∈ HomG (U χ ). Moreover, if b = eD , then ρ(b) is not a constant multiple of the identity operator on l 2 (D). Thus, Dχ nontrivial implies U χ , and hence indG  H χ, is reducible. . The question Let G = H  D be as in Theorem 2.6, and let χ1 , χ2 ∈ H χ1 χ2 arises of when U and U might be equivalent. Our first observation is that if χ1 and χ2 lie in the same D-orbit, then U χ1  U χ2 .  and b ∈ D. Then Proposition 2.7 Let G = H  D be as above, and let χ ∈ H b·χ χ U U .

2.2 Conditions for irreducibility

55

Proof As in the proof of Theorem 2.6, ρ is the right regular representation of D. Then, for any (h, d) ∈ G, ρ(b)U χ (h, d)ρ(b)∗ η(c) = U χ (h, d)ρ(b)∗ η(cb) = [(cb) · χ](h)ρ(b)∗ η(d −1 cb) = [c · (b · χ)](h)η(d −1 c) = U b·χ (h, d)η(c). This shows U b·χ  U χ .



It turns out that Proposition 2.7 describes the only equivalences that arise for U χ1 and U χ2 . Proposition 2.8 Suppose that G = H  D, where H is an abelian locally  and assume that χ1 compact group and D is a discrete group. Let χ1 , χ2 ∈ H χ1 χ2 is not an element of {d · χ2 : d ∈ D}. Then U and U are not equivalent. Proof As in the proof of Theorem 2.6, we use the regularity of L1 (H ). We construct a net, {f A }A∈A , in L1 (G) such that U χ2 (f A ) → 0 in the strong operator topology but there exists a nonzero η ∈ l 2 (D) with U χ1 (f A )η = η, for all A ∈ A. This will show that U χ1 and U χ2 cannot be equivalent. Let A denote the set of all finite subsets of D, partially ordered by inclusion.  that For each A ∈ A, {b · χ2 : b ∈ A} is a finite, hence closed, subset of H 1 does not contain the point χ1 . Again, by regularity of L (H ), there exists a   A ≤ 1, g A (χ ) = 1, and g A (b · χ ) = 0, function g A ∈ L1 (H ) such that 0 ≤ g 1 2 for all b ∈ A. Define f A ∈ L1 (G) by  A g (h) if d = eD A f (h, d) = 0 if d = eD , for all (h, d) ∈ G. As in the proof of Theorem 2.6, for i = 1, 2 and any η ∈ l 2 (D), A (b · χ )η(b). U χi (f A )η(b) = g i A (b · χ ) = 0, for all b ∈ A and any A ∈ A, Since g 2   A (b · χ )η(b)|2 ≤

U χ2 (f A )η 22 = |g |η(b)|2 . 2 b∈D\A

b∈D\A

Thus, limA∈A U χ2 (f A ) = 0 in the strong operator topology. However, if we define η ∈ l 2 (D) by η(eD ) = 1 and η(b) = 0, for all b = eD , then U χ1 (f A )η =  η, for all A ∈ A. Therefore, U χ1 is not equivalent to U χ2 .

56

Induced representations

We can now construct examples of infinite-dimensional irreducible representations of certain kinds of groups. We will look at one particular group, but the reader is encouraged to explore variations on this example. Example 2.9 Let G = {(t, n) : t ∈ R, n ∈ Z} with group product given by (t, n)(s, m) = (t + 2n s, n + m). Thus, G is of the form G = H  D with  as {χr : r ∈ R}, where χr (t) = H = R, D = Z, and n · t = 2n t. We identify H 2πirt e , t ∈ R. Then, for n ∈ Z and r ∈ R, n · χr = χ2−n r . Therefore, the representation of G induced from χr can be realized in the form U χr acting on l 2 (Z) by −m

U χr (t, n)η(m) = e2πi(2

r)t

η(m − n),

for m ∈ Z, η ∈ l 2 (Z), and (t, n) ∈ G. When r = 0 it is clear that r is not in {2−n r : n ∈ Z, n = 0}. Therefore Theorem 2.6 applies to show that U χr is irreducible if r = 0. Proposition 2.7 shows that U χ2n r  U χr for all n ∈ Z, whereas Proposition 2.8 tells us that U χr is not equivalent to U χs if r and s are from different Z-orbits. Therefore {U χr : 1 ≤ |r| < 2} is a family of mutually inequivalent irreducible representations of G. We can find some more irreducible representations of G by simply lifting the characters of Z = G/H . That is, for z ∈ T, let ψz (t, n) = zn

((t, n) ∈ G).

In Chapter 3, we shall present the deep and powerful imprimitivity theorem of Mackey. This theorem yields that every irreducible representation of G = R  Z is equivalent to one of the collection {U χr : 1 ≤ |r| < 2} ∪ {ψz : z ∈ T}. We now turn our attention to the situation where we have an open subgroup H of G which is very far from being normal in G. That is, we consider H such that [H : (H ∩ xH x −1 )] = ∞, for every x ∈ G \ H . An important example is to take G to be F2 , the free group on two generators a and b, say. Let H be the infinite cyclic subgroup generated by a. Then H ∩ xH x −1 = {e}, for any x ∈ G \ H. The goal is to show that this condition on H and its conjugates implies that the C appearing in Proposition 2.5 again coincides with HomG (Uγπ1 , Uγπ2 ). This can be achieved when we assume in addition that π1 and π2 are finitedimensional representations of H . The first step is to interpret the condition on H in terms of orbits in G/H .

2.2 Conditions for irreducibility

57

Fix a section γ : G/H → G such that γ (eH ) = e. If q : G → G/H denotes the quotient map as usual, let ω0 = q(H ) and note that γ (ω) · ω0 = ω, for any ω ∈ G/H , and that H = {x ∈ G : x · ω0 = ω0 }. For ω ∈ G/H , let Hω = {x ∈ G : x · ω = ω}. −1

Then Hω = γ (ω)H γ (ω) , ω ∈ G/H . Lemma 2.10 Suppose H is an open subgroup of G such that [H : (H ∩ xH x −1 )] = ∞, for every x ∈ G \ H . Then |Hω (ν)| = ∞, for any two distinct ω, ν ∈ G/H . Proof The Hω -orbit Hω (ν) is a left Hω -space. The stabilizer of ν in Hω equals Hω ∩ Hν . Thus |Hω (ν)| = [Hω : (Hω ∩ Hν )] = [γ (ω)H γ (ω)−1 : (γ (ω)H γ (ω)−1 ∩ γ (ν)H γ (ν)−1 )] = [H : (H ∩ (γ (ω)−1 γ (ν)H (γ (ω)−1 γ (ν))−1 )] = ∞, since ω = ν implies γ (ω)−1 γ (ν) ∈ H .



If π is a representation of H on H(π), we can move it to a representation of Hω . Let π ω (x) = π (γ (ω)−1 xγ (ω)), x ∈ Hω . Since {π ω (x) : x ∈ Hω } = {π(h) : h ∈ H }, π ω is irreducible if and only if π is irreducible. Since we have fixed a section γ , let us write U π for Uγπ , the realization of the induced representation that acts on l 2 (G/H, H(π )) via U π (x)η(ω) = π(γ (ω)−1 xγ (x −1 · ω))η(x −1 · ω) for ω ∈ G/H, x ∈ G, and η ∈ l 2 (G/H, H(π )). Let π1 and π2 be two representations of H on H(π1 ) and H(π2 ), respectively. We recall the notation Kωπi = {η ∈ l 2 (G/H, H(πi )) : η(ω ) = 0

for all ω = ω},

for i = 1, 2 and ω ∈ G/H . In order to apply Proposition 2.5 usefully, we want to show that for any S ∈ HomG (U π1 , U π2 ), S(Kωπ1 ) ⊆ Kωπ2 , for all ω ∈ G/H . This can be done when one of π1 and π2 is finite-dimensional. Thus, assume that π1 is of finite dimension n and let {ei : 1 ≤ i ≤ n} be a fixed orthonormal basis of H(π1 ). For ω ∈ G/H and 1 ≤ i ≤ n, let  e if ν = ω, ei δω (ν) = i 0 if ν = ω,

58

Induced representations

for ν ∈ G/H . Then {δωei : 1 ≤ i ≤ n} is an orthonormal basis of Kωπ1 , and {δωei : 1 ≤ i ≤ n, ω ∈ G/H } is an orthonormal basis of l 2 (G/H, H(π1 )). For any S ∈ B(l 2 (G/H, H(π1 )), l 2 (G/H, H(π2 ))) define its “partial” matrix coefficients as follows. For ω, ν ∈ G/H and 1 ≤ i ≤ n, let s(ω, i; ν) = Sδωei (ν). Then each s(ω, i; ν) ∈ H(π2 ), and they satisfy the properties of the following lemma.

Lemma 2.11 (i) ν∈G/H s(ω, i; ν) 2 < ∞ (ω ∈ G/H, 1 ≤ i ≤ n). (ii) For any ξ ∈ l 2 (G/H, H(π1 )) and ν ∈ G/H , n   ξ, δωei s(ω, i; ν). Sξ (ν) = ω∈G/H i=1

If S ∈ HomG (U π1 , U π2 ), then the intertwining property of S translates into a key technical relation on its coefficients. Lemma 2.12 Let S ∈ HomG (U π1 , U π2 ), ω, ν ∈ G/H, and x ∈ G. Then, for 1 ≤ i ≤ n, π2 (γ (x · ν)−1 xγ (ν))s(ω, i; ν) n  π1 (γ (x · ω)−1 xγ (ω))ei , ej s(x · ω, j ; x · ν). = j =1

Proof For any x ∈ G, ω, ω ∈ G/H, and 1 ≤ i, j ≤ n we have  e e π1 (γ (ν)−1 xγ (x −1 · ν))δωei (x −1 · ν), δωj (ν) U π1 (x)δωei , δωj  = ν∈G/H

= π1 (γ (ω )−1 xγ (x −1 · ω ))δωei (x −1 · ω ), ej   π1 (γ (x · ω)−1 xγ (ω))ei , ej  if ω = x · ω, = 0 if ω = x · ω. Therefore, by Lemma 2.11(ii), SU π1 (x)δωei (ν) =

n  

U π1 (x)δωei , δωj s(ω , j ; ν) e

ω ∈G/H j =1

=

n 

π1 (γ (x · ω)−1 xγ (ω))ei , ej s(x · ω, j ; ν).

j =1

On the other hand, U π2 (x)Sδωei (ν) = π2 (γ (ν)−1 xγ (x −1 · ν))Sδωei (x −1 · ν) = π2 (γ (ν)−1 xγ (x −1 · ν))s(ω, i; x −1 · ν).

2.2 Conditions for irreducibility

59

Using the equality SU π1 (x) = U π2 (x)S and replacing ν by x · ν gives the relation claimed in the lemma.  Lemma 2.13 Let S ∈ HomG (U π1 , U π2 ) and ω, ν ∈ G/H . If |Hω (ν)| = ∞, then s(ω, i; ν) = 0 for 1 ≤ i ≤ n. Proof The following calculation uses Lemma 2.12 and the Cauchy–Schwarz inequality. For 1 ≤ i ≤ n, x ∈ G, and μ ∈ G/H ,

s(ω, i; μ) 2 = π2 (γ (x · μ)−1 xγ (μ))s(ω, i; μ) 2 n  2   = π1 (γ (x · ω)−1 xγ (ω))ei , ej s(x · ω, j ; x · μ) j =1

≤

n 

2

|π1 (γ (x · ω))−1 xγ (ω))ei , ej | · s(x · ω, j ; x · μ)

j =1

≤

n 

−1

|π1 (γ (x ·ω) xγ (ω))ei , ej |

2

j =1

n 

s(x ·ω, j ; x · μ) 2



j =1

= π1 (γ (x · ω)−1 xγ (ω))ei 2

n 

s(x · ω, j ; x · μ) 2

j =1

=

n 

s(x · ω, j ; x · μ) 2 .

j =1

Thus n 

s(ω, i; μ) 2 ≤ n

n 

s(x · ω, j ; x · μ) 2 .

j =1

i=1

Now suppose that μ = x · ν for some x ∈ Hω . Then, since x · ω = ω, it follows that n 

n 

1

s(ω, j ; μ =

s(x · ω, j ; x · ν) ≥

s(ω, i; ν) 2 . n j =1 j =1 i=1 2

n

2

From Lemma 2.11(i) we get ∞>

n  

s(ω, j ; μ) 2 ≥

=

μ∈Hω ·ν

j =1

s(ω, j ; μ) 2

j =1 μ∈Hω ·ν

j =1 μ∈G/H n  

n  

s(ω, j ; μ)

2



n  1   ≥

s(ω, j ; ν) 2 . n μ∈H ·ν j =1 ω

60

Induced representations

Since |Hω (ν)| = ∞, we must have s(ω, j ; ν) = 0 for 1 ≤ j ≤ n.

n

j =1

s(ω, j ; ν) 2 = 0, whence 

Lemma 2.13 contains the key information that is necessary to strengthen Proposition 2.5 in the situation at hand. As a result, we can draw some strong conclusions. Theorem 2.14 Let G be a locally compact group and let H be an open subgroup of G such that [H : (H ∩ xH x −1 )] = ∞, for all x ∈ G \ H . If π is a finite-dimensional irreducible representation of H , then indG H π is irreducible. Moreover, if π1 and π2 are finite-dimensional irreducible representations of H , G then indG H π1 and indH π2 are equivalent if and only if π1 and π2 are equivalent. Proof If π1 and π2 are any representations of H with π1 finite-dimensional, Lemma 2.13 tells us that for S ∈ HomG (U π1 , U π2 ), we have Sδωei ∈ Kωπ2 , for 1 ≤ i ≤ n; hence SKωπ1 ⊆ Kωπ2 , for all ω ∈ G/H . Proposition 2.5 then shows that HomG (U π1 , U π2 ) is isomorphic as a vector space with HomH (π1 , π2 ). But π is irreducible if and only if HomH (π ) = HomH (π, π) is onedimensional, so if and only if HomG (U π ) is one-dimensional, which in turn is equivalent to U π being irreducible. Now, let π1 and π2 be both finite-dimensional and irreducible. Then combining the preceding two paragraphs yields that π1 and π2 are equivalent if and  only if U π1 and U π2 are equivalent. Example 2.15 If F2 = a, b, the free group on two generators, and H = a, then, as we noted earlier, the hypotheses of Theorem 2.14 are satisfied. Now the irreducible representations of H are one-dimensional of the form χz , z ∈ T, where χz (a n ) = zn for n ∈ Z. From Theorem 2.14, we conclude that {indIHF2 χz : z ∈ T} is a family of mutually inequivalent irreducible representations of F2 . We now present an example to show that, in Theorem 2.14, the hypothesis that π be finite-dimensional cannot be dropped. The example uses the following lemma, the proof of which is an easy application of Theorem 2.6 and the induction in stages theorem (Theorem 2.47). Because of the relevance to Theorem 2.14, we prefer to present this lemma and example before proving the induction in stages theorem. Lemma 2.16 Let N be an abelian locally compact group and D a discrete group acting on N. Let G = N  D. Let T be a subgroup of D and H = N   and Dχ = {d ∈ D : d · χ = χ}. T . Consider H as a subgroup of G. Let χ ∈ N H Let π = indN χ . Suppose Dχ is nontrivial and Dχ ∩ T = {eD }. Then π is irreducible, but indG H π is reducible.

2.3 The induced representation in general

61

Proof Both conclusions follow from Theorem 2.6. Since H acts freely on G χ , π is irreducible. By induction in stages, indG H π is equivalent to indN χ. G Since Dχ is nontrivial, indN χ is reducible by the converse direction of Theorem 2.6.  Example 2.17 Let G be the semidirect product G = Z2  SL(2, Z), where D = SL(2, Z) acts on Z2 in the natural way. Let N = Z2 , and let H = Z2  T , where  the subgroup T of SL(2, Z) consists of all matrices of the form 1 n ± , n ∈ Z. Moreover, let α be an irrational number and define a charac0 1   a11 a12 ter χ of N by χ (n, m) = exp(2π iαn), n, m ∈ Z. Then, for A = ∈ a21 a22 SL(2, Z), A · χ (n, m) = χ(a11 n + a12 m, a21 n + a22 m) = exp(2π iα(a11 n + a12 m)). Thus A ∈ Dχ if and only if (a11 − 1)n + a12 m ∈ α1 Z for all n, m ∈ Z. It follows   1 0 that ∈ Dχ for all q ∈ Z, and hence Dχ is nontrivial. On the other q 1 hand, if A ∈ T then A · χ = χ only if A = I . Indeed, choosing one of n and m to be zero and the other to be one, we obtain that a11 = 1 and a12 = 0. Thus Dχ ∩ T = {eD }. Finally, H ∩ gH g −1 is of infinite index in H for every g ∈ G \ H . In fact, it is easily verified that if A ∈ SL(2, Z) \ T , that is, a21 = 0, then AT A−1 ∩ T = {E, −E} and hence H ∩ gH g−1 = Z2  {I, −I }. G With π = indH N χ, Lemma 2.16 tells us that π is irreducible but indH π −1 is not irreducible even though [H : (H ∩ xH x )] = ∞, for all x ∈ G \ H . This demonstrates that the hypothesis that π be finite-dimensional cannot be dropped from Theorem 2.14.

2.3 The induced representation in general Let G be a locally compact group, H a closed subgroup of G, and π a representation of H on the Hilbert space H(π ). A representation of H that is “induced by π ” can be defined with different approaches. We will begin by following a method due to Blattner that is very useful in theoretical situations. In Section 2.4 we introduce two other formulations that are useful in describing the irreducible representations of particular groups. Let q : G → G/H, q(x) = xH , be the quotient map. Let G and H be the modular functions of G and H , respectively. In our discussion of rhofunctions in Section 1.3, the homomorphism h → H (h)1/2 G (h)−1/2 played a significant role. This role intensifies with the study of induced representations.

62

Induced representations

To reduce clutter in formulae, let us set G (h) = H (h)1/2 G (h)−1/2 , for h ∈ H, δH G and even write δ for δH when G and H are understood. Note that δ is continuous. For a function ξ : G → H(π), there are three conditions we wish to impose.

(1) ξ is continuous with H(π ) given the norm topology. (2) q(supp ξ ) is compact in G/H . (3) ξ (xh) = δ(h)π(h−1 )ξ (x), for x ∈ G, h ∈ H . Let F (G, π) = {ξ : G → H(π) : (1), (2), and (3) hold for ξ }. Functions which satisfy (1), (2), and (3) are fairly easy to come by, as the next proposition shows. Compare this with Proposition 2.2 in the special case where H is open in G. Proposition 2.18 Let f ∈ Cc (G, H(π)) and define ξf : G → H(π ) by  ξf (x) = δ(h−1 )π(h)f (xh)dh, H

for x ∈ G. Then ξf ∈ F(G, π). Conversely, for any ξ ∈ F (G, π), there exists g ∈ Cc (G, H(π )) such that ξg = ξ . Proof First note that q(supp ξf ) ⊆ q(supp f ), so ξf satisfies condition (2). Also, for any x ∈ G and h ∈ H ,  ξf (xh) = δ(k −1 )π(k)f (xhk)dk H



δ(k −1 )π(h−1 k)f (xk)dk

= δ(h) H

= δ(h)π(h−1 )ξf (x). Thus property (3) holds for ξf . To see that ξf is continuous, fix a compact neighborhood V of e in G. Let M = V supp f and K = (M −1 M) ∩ H . Both M and K are compact sets. Let d = max{δ(k −1 ) : k ∈ K}, and let |K| denote the H -Haar measure of K. Let x ∈ G and ε > 0 be arbitrary. Since f ∈ Cc (G, H(π )), it is uniformly continuous on G, so there exists a neighborhood U of e such that U −1 = U ⊆ V and z−1 w ∈ U implies f (z) − f (w) < ε. If x ∈ G \ MH and y ∈ U x, then f (xk) = f (yk) = 0 for all k ∈ H , so ξf (x) = ξf (y) = 0. If x ∈ MH and y ∈ U x, let h ∈ H be such that xh ∈ M. If yhk ∈ supp f , then xhk = (xy −1 )yhk ∈ V supp f = M. So k ∈ (M −1 M) ∩ H = K. Likewise, if xhk ∈ supp f , then k ∈ K. Thus, k ∈ H \ K implies

2.3 The induced representation in general

63

f (xhk) = f (yhk) = 0. Now we simply estimate     δ(k −1 )π(k)[f (xk) − f (yk)]dk 

ξf (x) − ξf (y) =  H  ≤ δ(k −1 ) f (xk) − f (yk) dk H  = δ(k −1 )δ(h−1 ) f (xhk) − f (yhk) dk H 

f (xhk) − f (yhk) dk ≤ δ(h−1 )d K −1

< δ(h )d|K|ε. Thus ξf is continuous. Since (1), (2), and (3) hold for ξf , we have shown ξf ∈ F (G, π). Conversely, let ξ ∈ F (G, π). Since q(supp ξ ) is compact, by Proposition 1.9, there exists ψ ∈ Cc (G) such that H ψ(xh)dh = 1 for all x ∈ supp ξ . Let  g = ψξ ∈ Cc (G, H(π)) and check that ξg = ξ . Particularly simple elements of F(G, π) can be constructed as follows. Let g ∈ Cc (G) and v ∈ H(π), and define f ∈ Cc (G, H(π)) by f (x) = g(x)v, x ∈ G. Then  ξf (x) = δ(h−1 )g(xh)π(h)vdh. H

We shall henceforth denote this element of F (G, π) by (g, v). The next task is to define an inner product on the linear space F (G, π). Recall, from Section 1.3, that for f ∈ Cc (G), f # ∈ Cc (G/H ) is defined by f # (xH ) =  # + H f (xh)dh. The map f → f is linear, surjective, and maps Cc (G) onto formula (1.4) that, for ξ ∈ Cc+ (G/H ). It follows from the Weil integration  F(G, π) and ϕ ∈ Cc (G), ϕ # = 0 implies G ϕ(x) ξ (x) 2 dx = 0. Define λξ , a map from Cc (G/H ) into C, by  ϕ(x) ξ (x) 2 dx, λξ (ϕ # ) = G

for ϕ ∈ Cc (G). Then λξ is a well-defined positive linear functional. Thus, there exists a positive Radon measure μξ on G/H such that   ϕ # (ω)dμξ (ω) = ϕ(x) ξ (x) 2 dx, G/H

G

for all ϕ ∈ Cc (G). Proposition 2.19 Let ξ ∈ F (G, π), and let μξ be defined as above. Then: (i) supp μξ = q(supp ξ ); (ii) μξ is a finite measure.

64

Induced representations

Proof (i) Let α ∈ Cc+ (G/H ) be such that α(ω) = 0 for all ω ∈ q(supp ξ ). Let ϕ ∈ Cc+ (G) be such that ϕ # = α. Since ϕ ≥ 0, we must have ϕ(x) = 0 for all x ∈ supp ξ . Then   α(ω)dμξ (ω) = ϕ(x) ξ (x) 2 dx = 0. G/H

G

This proves (i). Now, (ii) follows by the regularity of μξ .



For any ξ, η ∈ F(G, π ), use polarization to define a complex Radon measure μξ,η on G/H . That is, 1 μξ,η = (μξ +η − μξ −η + iμξ +iη − iμξ −iη ). 4 By the polarization identity applied to ξ (x), η(x), for each x ∈ G, we have, for any ϕ ∈ Cc (G),   ϕ(x)ξ (x), η(x)dx = ϕ # (ω)dμξ,η (ω). G

G/H

This shows that the form (ξ, η) → μξ,η (G/H ) is a conjugate bilinear form on F(G, π ). Moreover, μξ,ξ (G/H ) = μξ (G/H ) > 0 whenever ξ = 0. Thus ξ, η = μξ,η (G/H ) defines an inner product on F(G, π ) and

ξ 2 = ξ, ξ  = μξ (G/H ), for ξ ∈ F(G, π ). Proposition 2.20 Let ξ, η ∈ F(G, π ), and let ψ be any element of Cc (G) such that ψ # (ω) = 1 for all ω ∈ q(supp ξ ) ∪ q(supp η). Then  ξ, η = ψ(x)ξ (x), η(x)dx. G

Proof This is simply because supp μξ,η ⊆ q(supp ξ ) ∪ q(supp η) and μξ,η (G/H ) = μξ,η (supp μξ,η ).  As usual, for y ∈ G, we let Ly denote left translation by y of any function on G or even on G/H . Thus, for instance, for α ∈ Cc (G/H ) and ω ∈ G/H , Ly α(ω) = α(y −1 · ω). If ϕ ∈ Cc (G), note that   Ly ϕ(xh)dh = ϕ(y −1 xh)dh = Ly ϕ # (q(x)), (Ly ϕ)# (q(x)) = H

H

for all x ∈ G. Thus, (Ly ϕ) = Ly ϕ . #

#

Lemma 2.21 For ξ ∈ F(G, π ) and y ∈ G, Ly ξ ∈ F(G, π ) and Ly ξ = ξ . Proof First note that Ly ξ is a continuous function on G and q(supp Ly ξ ) = y · q(supp ξ ), a compact subset of G/H . For x ∈ G and h ∈ H , Ly ξ (xh) = ξ (y −1 xh) = δ(h)π (h−1 )ξ (y −1 x) = δ(h)π (h−1 )Ly ξ (x). So Ly ξ ∈ F(G, π ).

2.3 The induced representation in general

65

If ψ ∈ Cc (G) is such that ψ # (ω) = 1, for all ω ∈ q(supp ξ ), then (Ly ψ)# (ω) = 1 for all ω ∈ q(supp Ly ξ ). By Proposition 2.20, we have   2 2 Ly ψ(x) Ly ξ (x) dx = ψ(y −1 x) ξ (y −1 x) 2 dx

Ly ξ = G G  = ψ(x) ξ (x) 2 dx = ξ 2 .  G

For the moment, let F denote the completion of (F (G, π), ·, ·). Then F is a Hilbert space and, for each y ∈ G, Ly extends to an isometry on F. Moreover, Ly −1 Ly is clearly the identity, so Ly is a unitary operator. It is also immediately clear that Lx Ly = Lxy for all x, y ∈ G. Lemma 2.22 For ξ, η ∈ F(G, π), the function x → Lx ξ, η is continuous on G. Proof It suffices to prove continuity at e. Let V be a relatively compact neighborhood of e. Let ψ ∈ Cc (G) be such that ψ # (ω) = 1, for all ω ∈ q(V · supp ξ ∪ supp η). Then, by Proposition 2.20, for x ∈ V ,  ψ(y)ξ (x −1 y), η(y)dy. Lx ξ, η = G

Let K = supp ψ, and let M = sup{|ψ(y)| · η(y) : y ∈ K}. For any ε > 0, let U ⊆ V be a neighborhood of e such that x ∈ U implies that

ξ (x −1 y) − ξ (y) < ε for all y ∈ K. Then, for x ∈ U ,     −1  |Lx ξ, η − ξ, η| =  ψ(y)ξ (x y) − ξ (y), η(y)dy  G |ψ(y)| · ξ (x −1 y) − ξ (y) · η(y) dy ≤ K

< εM|K|. Thus x → Lx ξ, η is continuous at e.



Since F(G, π) is dense in F, for any ξ, η ∈ F, x → Lx ξ, η is seen to be continuous by an ε/3 argument. Thus, x → Lx satisfies the requirement for a representation of G. This representation is called the representation of G induced from π or simply the induced representation and is denoted by indG H π. Consistent with this notation, F will be denoted H(indG π). H To summarize, H(indG H π) is the completion of F (G, π) and −1 (indG H π(x)ξ )(y) = ξ (x y)

for x, y ∈ G and ξ ∈ F (G, π).

66

Induced representations

Example 2.23 Consider the special case when G is arbitrary, H = {e}, and π = τ , the trivial representation of H . Then G/H = G and F(G, τ ) = Cc (G). For ξ ∈ F(G, τ ), the corresponding Radon measure μξ is defined such 2 2 that G ϕ(ω)dμξ (ω) = G ϕ(x)|ξ (x)| dx, for all ϕ ∈ Cc (G). Thus, ||ξ || =  G 2 G |ξ (x)| dx and the completion, H(indH τ ), of F(G, τ ) = Cc (G) is sim−1 ply L2 (G) and indG H τ (x)ξ (y) = ξ (x y), for x, y ∈ G and ξ ∈ Cc (G). Thus, G indH τ = λG , the left regular representation of G on L2 (G). More examples will be presented in the next sections when alternate realizations of the induced representation, which are more convenient for explicit calculations, are available. For later use, we insert here several facts concerning the special elements (f, v), f ∈ Cc (G), v ∈ H(π ), of F(G, π ). Lemma 2.24 Let H be a closed subgroup of the locally compact group G and π a representation of H . Let D be a total subset of H(π ) and put (Cc (G), D) = {(f, v), f ∈ Cc (G), v ∈ D}. Then: (i) (ii) (iii) (iv)

(Cc (G), D) is total in H(indG H π); (Cc (G), D)(x) is total in H(π) for every x ∈ G. if f, g ∈ Cc (G) and v ∈ H(π ), indG H π(g)(f, v) ∈ F(G, π ); indG H π (Cc (G))(Cc (G), D)(x) is total in H(π) for every x ∈ G.

Proof (i) Let f ∈ Cc (G), v ∈ H(π ), and ξ ∈ F(G, π ), and choose ψ ∈ Cc (G) such that ψ # (ω) = 1, for all ω ∈ q(supp ξ ) ∪ q(supp f ). Then  ψ(x)(f, v)(x), ξ (x)dx (f, v), ξ  = G



 G (h) 1/2 f (xh)π (h)v, ξ (x)dhdx H (h) G H   ψ(xh−1 )f (x) = π(h)v, ξ (xh−1 )dxdh 1/2 (h) (h)] [ H G G H   ψ(xh−1 )f (x) π(h)v, δ(h−1 )π (h)ξ (x)dxdh = 1/2 (h) (h)] [ H H  G G −1 = ψ(xh )H (h)−1 f (x)v, ξ (x)dhdx G H    ψ(xh)dh f (x)v, ξ (x)dx = G H f (x)v, ξ (x)dx. =  

=

ψ(x)

G

2.3 The induced representation in general

67

 Thus, if ξ ∈ F(G, π ) is orthogonal to (Cc (G), D), then G f (x)v, ξ (x)dx = 0, for all f ∈ Cc (G) and v ∈ D, and hence, since ξ is continuous, v, ξ (x) = 0, for all v ∈ D and x ∈ G. Since D is total in H(π ), it follows that ξ (x) = 0, for all x ∈ G. This shows that the elements (f, v), f ∈ Cc (G), v ∈ D, span a dense linear subspace of F(G, π ). Since F(G, π ) is dense in H(indG H π ), the proof of (i) is complete. (ii) Since (f, v)(x) = (Lx −1 f, v)(e) for every f ∈ Cc (G) and v ∈ H(π ), we can assume that x = e. Now, let v ∈ H(π )and  > 0 be given,  and choose   v1 , . . . , vn ∈ D and λ1 , . . . , λn ∈ C such that  nj=1 λj vj − v  ≤ /2. There exists a neighborhood V of e in H such that ⎞−1 ⎛ n   

π (h)v − v ≤ |λj |⎠ , and π(h)vj − vj ≤ ⎝ 2 2 j =1 for all h∈ V and j = 1, . . . , n. Select a function g ∈ Cc+ (H ) such that supp g ⊆ V and H g(h)δ(h−1 )dh = 1. By Tietze’s extension theorem there exists f ∈ Cc+ (G) such that f |H = g. It follows that          n n       −1 v −    λ (f, v ) λ g(h)δ(h) π (h)v dh = v − j j  j j    H     j =1 j =1     n     λ v ≤ v − j j    j =1    n    −1  |λj |  g(h)δ(h) [π (h)v − v ]dh + j j   H

j =1

  |λj | · sup π (h)vj − vj ≤ , + 2 j =1 h∈V n

≤

as required. (iii) It suffices to show that indG H π(g)(f, v)(y) is a continuous function of y and that q(supp(indG π(g)(f, v))) is compact in G/H . For any y ∈ G, H  g(x)(f, v)(x −1 y)dx indG H π (g)(f, v)(y) = G   = g(x) δ(t −1 )f (x −1 yt)π (t)vdtdx G H    g(x)f (x −1 yt)δ(t −1 )dx π (t)vdt = G H = (g ∗ ft )(y)π(t)vdt, H

68

Induced representations

where we have set ft (s) = f (st)δ(t −1 ), s ∈ G, t ∈ H . Note that ft ∈ Cc (G) for every t ∈ H . Let Kg and Kf denote the (compact) support of g and f , respectively. One easily checks that supp(indG H π(g)(f, v)) ⊆ Kg Kf H . Thus G q(supp(indH π (g)(f, v))) ⊆ q(Kg Kf ), a compact subset of G/H . Let U be a neighborhood of e such that U is compact. Then, for any z ∈ G, (zU Kg Kf ) ∩ H is a compact subset of H . For measurable subsets A of H and B of G, let |A|H and |B|G denote the Haar measure in H and G, respectively. Fix y ∈ G and  > 0, and let M = sup{δ(t −1 ) : t ∈ (y −1 U Kg Kf ) ∩ H } < ∞. Let η > 0 be such that η g ∞ |Kg |G |(y −1 U Kg Kf ) ∩ H |H v M < . Since f is uniformly continuous, there exists a neighborhood W of e in G such that |f (a) − f (b)| < η for all a, b ∈ G with ab−1 ∈ W . Let V be a neighborhood of e in G satisfying V −1 = V , V ⊆ U , and x −1 V x ⊆ W for all x ∈ Kg . For any z ∈ V y, we have yz−1 ∈ V and hence (x −1 yt)(x −1 zt)−1 ∈ x −1 V x ⊆ W for all t ∈ H and x ∈ Kg . Thus, |f (x −1 yt) − f (x −1 zt)| < η for every z ∈ V y, x ∈ Kg , and t ∈ H . Note that  (g ∗ ft )(z) = g(x)f (x −1 zt)δ(t −1 )dx = 0, G −1

unless t ∈ (z Kg Kf ) ∩ H ⊆ (y −1 U Kg Kf ) ∩ H . On the other hand, when t ∈ (y −1 U Kg Kf ) ∩ H ,  |g(x)| · |f (x −1 yt) − f (x −1 zt)|δ(t −1 )dx |(g ∗ ft )(y) − (g ∗ ft )(z)| ≤ Kg

≤ g ∞ |Kg |G Mη. Let F : G → H(π ) be defined by F (z) = indG H π (g)(f, v)(z) for all z ∈ G. Then z ∈ V y implies 

F (y) − F (z) ≤ |(g ∗ ft )(y) − (g ∗ ft )(z)|dt · v H

≤ |(y −1 U Kg Kf ) ∩ H |H g ∞ |Kg |G M v η < .

2.3 The induced representation in general

69

This proves that indG H π(g)(f, v) is a continuous map from G into H(π ) and that indG π (g)(f, v) ∈ F(G, π ). H (iv) It is enough to show that indG H π(Cc (G))(Cc (G), D)(e) is total in H(π ). In light of (ii), it suffices to show that, given f ∈ Cc (G), v ∈ D, and  > 0, there exists g ∈ Cc (G) so that

indG H π(g)(f, v)(e) − (f, v)(e) < . Fix a relatively compact neighborhood U of e in G. Let M = sup{δ(t −1 ) : t ∈ (U Kf ) ∩ H }, and choose η > 0 such that ηM v · |(U Kf ) ∩ H }|H < . If x ∈ U , then |f (x −1 t) − f (t)| = 0 unless t ∈ (U Kf ) ∩ H . Let W be a neighborhood of e such that ab−1 ∈ W implies |f(a) − f (b)| < η, and choose g ∈ Cc (G) such that g ≥ 0, supp g ⊆ W , and G g(x)dx = 1. Then, with F as in the proof of (iii),  

F (e) − (f, v)(e) ≤ v g(x)|f (x −1 t) − f (t)|δ(t −1 )dtdx G H   = v g(x) |f (x −1 t) − f (t)|δ(t −1 )dtdx Kg

(U Kf )∩H

< , 

as was to be shown.

Another fact that will be used later, is the statement of the following lemma. Lemma 2.25 Let H be a closed subgroup of G and π a unitary representation of H . For every compact subset K of G, there exists a constant cK > 0 such that 

ξ (x) dx ≤ cK ξ K

for all ξ ∈

H(indG H

π).

Proof Choose g ∈ Cc+ (G) such that g|K = 1. Then, for every ξ ∈ H(indG H π ),   

ξ (x) 2 dx ≤ g(x) ξ (x) 2 dx = g # (ω)dμξ (ω) K

G

G/H

≤ g # ∞ μξ (G/H ) = g # ∞ ξ 2 .

70

Induced representations

By H¨older’s inequality, this implies  1/2  2 1/2

ξ (x) dx ≤

ξ dx |K|1/2 ≤ g # 1/2

ξ 2 . ∞ |K| K

K

1/2

Now set cK = g # ∞ |K|1/2 < ∞.



2.4 Other realizations and positive definite measures It is often useful to have a realization of indG H π on a Hilbert space of H(π )valued functions where the inner product is given by integration over G/H with respect to a quasi-invariant measure. Recall from Section 1.3 that there exists a continuous function ρ : G → R+ such that, for all x ∈ G and h ∈ H , ρ(xh) = δ(h)2 ρ(x) = H (h)G (h)−1 ρ(x). Given this rho-function ρ, there is a measure μ on G/H such that   # ϕ (ω)dμ(ω) = ϕ(x)ρ(x)dx, G/H

G

for all ϕ ∈ Cc (G). If we set μx (E) = μ(x · E), for x ∈ G and Borel subsets E of G/H , then for any α ∈ Cc (G/H ),   α(ω)dμx (ω) = α(x −1 · ω)dμ(ω). G/H

G/H

For every x ∈ G, μx is equivalent to μ and ρ(xy) dμx (yH ) = , dμ ρ(y) for all x, y ∈ G. That is, μ is a quasi-invariant measure on G/H . In this second approach to induced representations, the twisting due to the homomorphism δ is transferred from the “functions” in the Hilbert space to the action. That is, we replace F(G, π ) by F H (G, π ) = {η : G → H(π) : (a), (b), and (c) hold}, where (a) η is continuous with H(π) given the norm topology, (b) q(supp η) is compact in G/H , (c) η(xh) = π(h−1 )η(x), for all x ∈ G, h ∈ H .

2.4 Other realizations

71

Note that in the case when μ can be chosen as a G-invariant measure, then δ ≡ 1 on H (that is, H = G |H ) by Theorem 1.16, and F H (G, π) = F (G, π). In general, a simple linear transformation maps F H (G, π) onto F (G, π). The next lemma is easily established from the properties of ρ. √ Lemma 2.26 The map M√ρ defined by M√ρ η = ρ η, for η ∈ F H (G, π), is a linear isomorphism of F H (G, π) onto F (G, π). We will now transfer the inner product structure of F (G, π) to F H (G, π) using M√ρ . By the polarization identity we only have to transfer the norm. For η ∈ F H (G, π), select ψ ∈ Cc (G) such that ψ # ≡ 1 on q(supp η). Then, by Proposition 2.20,    √ M√ρ η2 = √ρη 2 = ψ(x) ρ(x)η(x) 2 dx G   2 ˙ ˙ = ψ(x) η(x) 2 ρ(x)dx = ψ # (x) η(x) dμ(x) G G/H  ˙

η(x) 2 dμ(x). = G/H

Note that η(x) depends only on the coset x˙ = xH by property (c) of elements of F H (G, π). Therefore, we can define an inner product on F H (G, π) so that M√ρ is an isomorphism of inner product spaces. For η, ν ∈ F H (G, π), property (c) again implies that x → η(x), ν(x) is constant on H -cosets. Thus, define  ˙ η, νμ = η(x), ν(x)dμ(x). G/H

Let K denote the completion of (F H (G, π), ·, ·μ ) and extend M√ρ to a unitary π transformation between H(indG H π) and K. Define the representation Uμ by −1 G √ Uμπ (x) = M√ ρ indH π(x)M ρ ,

for all x ∈ G. Proposition 2.27 For x, y ∈ G and η ∈ F H (G, π),  1/2 ρ(x −1 y) Uμπ (x)η(y) = η(x −1 y). ρ(y) √ Proof Let η ∈ F H (G, π) and let ξ = ρ η. Then  1 −1 G Uμπ (x)η(y) = M√ ξ (x −1 y) ρ indH π(x)ξ (y) = √ ρ(y)  1/2 ρ(x −1 y) = η(x −1 y). ρ(y)



72

Induced representations

 −1 1/2 Recall that ρ(xρ(y)y) is the square root of the Radon–Nikodym derivative of μx −1 with respect to μ. This is the natural factor required to convert the action of G on G/H into a unitary representation on some space of functions when the inner product is defined by integration with respect to μ. By definition, Uμπ is equivalent to indG H π and clearly a different choice of a quasi-invariant measure would result in yet another equivalent representation. Sometimes it is important to distinguish which realization of the induced representation is being used, but often there is no confusion in simply using U π to denote the induced representation with G, H , and μ being understood. There are times when it is useful to realize all the elements of H(Uμπ ) as equivalence classes of measurable H(π)-valued functions on G. Proposition 2.28 Fix a quasi-invariant measure μ on G/H and let η ∈ H(Uμπ ). Then there exists a measurable function  η : G → H, defined up to an H -saturated locally null set, such that −1 (i)  η η(x), for all h ∈ H and almost all x ∈ G.  (xh) = π(h2 ) ˙ < ∞. η(x) dμ(x) (ii) G/H   H ˙ η(x) 2 dμ(x). (iii) For any ξ ∈ F (G, π ), ξ − η 2 = G/H ξ (x) − 

Moreover, if ν is any other H(π)-valued function on G satisfying (i), (ii), and (iii), then ν(x) =  η(x), for almost all x ∈ G. H Proof Let (ηn )∞ n=1 be any sequence in F (G, π ) such that ηn → η. Pass to a subsequence, if necessary, also denoted (ηn )∞ , such that M =

Nn=1

∞

η − η

< ∞. Let g (x) =

η (x) +

η n+1 n N 1 n+1 (x) − ηn (x) , n=1 n=1 for all x ∈ G, N ∈ N. Then  1/2  1/2 ˙ ˙ gN (x)2 dμ(x) ≤

η1 (x) 2 dμ(x) G/H

G/H

+

N   n=1

≤ η1 +

1/2 ˙

ηn+1 (x) − ηn (x) dμ(x) 2

G/H N 

ηn+1 − ηn ≤ η1 + M.

n=1

Let g(x) = limN→∞ gN (x). Then it follows that  ˙ ≤ ( η1 + M)2 < ∞. g(x)2 dμ(x) G/H

Thus there exists a measurable subset of G/H such that (G/H ) \ is locally μ-null and g(x) < ∞ whenever x˙ ∈ . That is,

η1 (x) +

∞  n=1

ηn+1 (x) − ηn (x) < ∞,

2.4 Other realizations

73

for any x ∈ G with x˙ ∈ . This implies that (ηn (x))∞ n=1 converges in H(π ), η(x) = limn→∞ ηn (x), for all x ∈ q −1 ( ). Clearly, for any x ∈ q −1 ( ). Define   η(xh) = π(h−1 ) η(x), for all x ∈ q −1 ( ) and h ∈ H . The usual arguments of integration theory show that (ii) holds and that, except for locally null sets, any η. sequence from F H (G, π) converging to η will result in the same  For any ξ ∈ F H (G, π), continuity of the norm in H(Uμπ ) and limit theorems for integrals show that  2 2

ξ − η = lim ξ − ηn = lim ˙

ξ (x) − ηn (x) 2 dμ(x) n→∞ n→∞ G/H  ˙

ξ (x) −  η(x) 2 dμ(x). = G/H

A similar continuity argument shows the last statement of the proposition.  We will follow standard practice and, for any η ∈ H(Uμπ ), use η to denote the equivalence class of all functions from G into H(π ) that agree locally almost everywhere with  η and also to denote an actual representative of the equivalence class. In light of Proposition 2.28, we can consider H(Uμπ ) to be the Hilbert space of all measurable functions η : G → H satisfying η(xh) = π (h−1 )η(x), for all h ∈ H and almost all x ∈ G, and  ˙ < ∞.

η(x) 2 dμ(x) G/H

The representation

Uμπ

is given by, now for any η ∈ H(Uμπ ) and x ∈ G, 

Uμπ (x)η(y)

ρ(x −1 y) = ρ(y)

1/2

η(x −1 y).

When working with a particular locally compact group G and closed subgroup H it is usually possible to select a measurable cross-section γ : G/H → G. That is, q(γ (ω)) = ω, for all ω ∈ G/H . Such a γ is neither unique nor canonical. Nevertheless, let us assume that one has been selected and is fixed. Let π be a representation of H and μ a quasi-invariant measure on G/H . Now consider the Hilbert space L2 (G/H, H(π ), μ) = {F : G/H → H(π) : F is measurable and 

F (ω) 2 dμ(ω) < ∞}, G/H

with the usual identification of functions that agree almost everywhere. We can use the cross-section γ to define a unitary transformation W from H(Uμπ ) onto L2 (G/H, H(π ), μ) by W η(ω) = η(γ (ω)), for η ∈ H(Uμπ ) and ω ∈ G/H . It is

74

Induced representations

clear that W has the announced properties and that W −1 F (x) = π(x −1 γ (q(x)))F (q(x)), for x ∈ G and F ∈ L2 (G/H, H(π ), μ). Note that x −1 γ (q(x)) ∈ H , for any x ∈ G. We will refrain from introducing a new notation for the induced representation when it is transferred, via W , to L2 (G/H, H(π), μ). We simply calculate, for F ∈ L2 (G/H, H(π ), μ) and x ∈ G, W Uμπ (x)W −1 F (ω) = Uμπ (x)W −1 F (γ (ω)) 1/2  ρ(x −1 γ (ω)) W −1 F (x −1 γ (ω)) = ρ(γ (ω))  1/2 dμx −1 = W −1 F (γ (x −1 · ω)γ (x −1 · ω)−1 x −1 γ (ω)) (ω) dμ  1/2 dμx −1 = π(γ (ω)−1 xγ (x −1 · ω))F (x −1 · ω). (ω) dμ Although this last formula appears complicated, because the Hilbert space L (G/H, H(π ), μ) is concrete (especially when π is one-dimensional), it is this form of the induced representation that is usually used when one is constructing explicitly representations of a given group. It takes an even clearer form on elementary tensors. For f ∈ L2 (G/H, μ) and η ∈ H(π ), define the elementary tensor η ⊗ f as the function from G/H into H(π ) such that (η ⊗ f )(ω) = f (ω)η, for all ω ∈ G/H . The set of elementary tensors is total in L2 (G/H, H(π ), μ) and identifying elementary tensors with the obvious elements of H(π) ⊗ L2 (G/H, μ) identifies L2 (G/H, H(π ), μ) with H(π) ⊗ L2 (G/H, μ). On an elementary tensor such as η ⊗ f , W Uμπ (x)W −1 is given by 2

W Uμπ (x)W −1 (η ⊗ f )(ω) 1/2  dμx −1 (ω) = π(γ (ω)−1 xγ (x −1 · ω))f (x −1 · ω)η. dμ Example 2.29 We now specialize this realization of the induced representation to the case where H has a complementary subgroup. Thus suppose that there exists a closed subgroup K satisfying the following conditions: (1) KH = G and K ∩ H = {0}. (2) The bijective map (k, h) → kh between K × H and G is a homeomorphism. Notice that hypothesis (2) can be omitted when one of K and H is σ -compact. Now, in the above situation the equation ρ(kh) = G (h)H (h)−1 ,

2.4 Other realizations

75

for k ∈ K, h ∈ H , defines a continuous, strictly positive ρ-function on G. Let μ be the corresponding quasi-invariant measure on G/H , and identify K with G/H via the homeomorphism k → kH . So   # f (k)dμ(k) = f (x)ρ(x)dx, K

G

for all f ∈ Cc (G), and since ρ is invariant under left translation by elements of K and (Lk f )# = Lk (f # ), we see that μ equals a left Haar measure on K. Take the cross-section γ : K → G to be the identity, and identify H(Uμπ ) with L2 (K, H(π ), μ). Let x ∈ G and l ∈ K, and write x −1 l = kh−1 , where k ∈ K dμx −1 H (h) and h ∈ H . Then γ (l)−1 xγ (x −1 · l) = h and dμ (l) =  , and hence the G (h) realization of U π on L2 (K, H(π), μ) is given by   H (h) 1/2 π(h)f (x −1 · l), U π (x)f (l) = G (h) for f ∈ L2 (K, H(π ), μ)) and l ∈ K, where h is as above. Observe that if x = k ∈ K, then U π (k)f (l) = f (k −1 l). In particular, if π is one-dimensional, this formula shows that indG H π|K is equivalent to the left regular representation of K. Example 2.30 If K and H commute in the previous example, that is, if G = H × K (we interchange the order of H and K just for clarity of the following formulae), then G (h) = H (h), for h ∈ H . Thus, for any representation π of H , if we consider L2 (K, H(π), μ)) as H(π ) ⊗ L2 (K), the induced H ×K representation indH π is unitarily equivalent to the outer tensor product representation π × λK , where (π × λK )(h, k)(η ⊗ f )(l) = (π (h)η ⊗ f )(k −1 l) = f (k −1 l)π (h)η, for all l ∈ K. Outer tensor product and tensor product representations will be studied in some detail in Section 2.8. We will present some deeper examples in the next section. We now turn our attention to representations that are defined by positive definite measures. If one starts with a measure on a subgroup H of a locally compact group G and extends it trivially to G, then the representation defined by the measure on G is equivalent to the induced representation coming from that defined by the measure on H . This will be used in a fundamental manner in Chapter 5 as we study the continuity properties of induction. A Radon measure ν on a locally compact group G is called positive definite, or a measure of positive type, if ν(f ∗ ∗ f ) ≥ 0 for all f ∈ Cc (G). Examples of such measures are the functionals f → G f (x)ϕ(x)dx, for any continuous positive definite function ϕ on G. A positive definite measure ν gives rise to a unitary representation as follows. First, ν defines a pseudo-inner product on

76

Induced representations

Cc (G) by setting f, gν = ν(g ∗ ∗ f ), for all f, g ∈ Cc (G). Let Kν = {f ∈ Cc (G) : f, f ν = 0} = {f ∈ Cc (G) : g, f ν = 0 for all g ∈ Cc (G)}. Then Kν is a closed linear subspace of Cc (G). We denote the elements of the quotient space Cc (G)/Kν by [f ]ν , f ∈ Cc (G), and let Hν be the completion of Cc (G)/Kν with respect to the inner product [f ]ν , [g]ν  = ν(g ∗ ∗ f ), f, g ∈ Cc (G). Since (Lx g)∗ ∗ (Lx f ) = g ∗ ∗ f for all f, g ∈ Cc (G) and x ∈ G, we can define a unitary representation λν of G on Hν by λν (x)[f ]ν = [Lx f ]ν , for f ∈ Cc (G) and x ∈ G. Now let H be a closed subgroup of G and μ a positive definite measure on H . Recall that δ : H → (0, ∞) is defined as δ(h) = H (h)1/2 G (h)−1/2 , for all h ∈ H . We associate with μ a linear functional ν on Cc (G) by setting ν(f ) = μ(δ −1 · f |H ), for all f ∈ Cc (G). Lemma 2.31 ν is a positive definite measure on G. Proof Let f ∈ Cc (G) and let g be any function in Cc+ (G) such that  H g(xh)dh = 1, for all x ∈ supp f . Then   −1 ν(f ∗ ∗ f ) = δ(h ) f ∗ (y)f (y −1 h)dydμ(h) H G −1 = δ(h ) f (y)f (yh)dydμ(h) G  H δ(h−1 ) g(yk)f (y)f (yh)dkdydμ(h) = H    G H = δ(h−1 )g(y)f (yk −1 )f (yk −1 h)G (k −1 )dydkdμ(h)  H H  G G (k) δ(h−1 )g(y)f (yk)f (ykh) = dkdydμ(h) H (k) H G H  g(y) δ(k)−1 Ly −1 f (k)δ(kh)−1 Ly −1 f (kh)dkdμ(h)dy = G H H    = g(y) (δ −1 Ly −1 f )(k)(δ −1 Ly −1 f )(kh)dkdμ(h)dy G  H H = g(y) (δ −1 · Ly −1 f )∗ (k)(δ −1 · Ly −1 f )(k −1 h)dkdμ(h)dy G H H   g(y) ((δ −1 · Ly −1 f |H )∗ ∗ (δ −1 · Ly −1 f |H ))(h)dμ(h)dy = G H  = g(y)μ((δ −1 · Ly −1 f |H )∗ ∗ (δ −1 · Ly −1 f |H ))dy ≥ 0, G

as required.



2.4 Other realizations

77

Theorem 2.32 The representation λν of G is unitarily equivalent to the induced representation indG H λμ , an equivalence being established by the map [f ]ν → ξf from Cc (G)/Kν into F(G, λμ ), where ξf : G → Hμ is defined by ! " ξf (x) = δ −1 · Lx −1 f |H μ . Proof Clearly, for any f ∈ Cc (G), the map ξf is continuous and has compact support modulo H . Moreover, for x ∈ G and h ∈ H , ! ! " " ξf (xh) = δ −1 · L(xh)−1 f |H μ = δ(h) Lh−1 (δ −1 · Lx −1 f |H ) μ ! " = δ(h)λμ (h−1 ) δ −1 · Lx −1 f |H ) μ = δ(h)λμ (h−1 )ξf (x), so that ξf ∈ F (G, λμ ). Let f ∈ Kν . For any y ∈ G such that Ly f |H = 0, select g as in the lemma so that g(y) > 0. Then, the calculation in the proof of Lemma 2.31 shows that μ((δ −1 · Ly f |H )∗ ∗ (δ −1 · Ly f |H )) = 0, for all y ∈ G. Thus ξf = 0. Therefore, the assignment f → ξf defines a linear map [f ]ν → ξf from Cc (G)/Kν into F(G, λμ ). The formula of the proof of the lemma also shows that this map preserves inner products. In fact, if ξ, η ∈ F (G, λμ ) and g ∈ Cc+ (G)  is such that H g(yh)dh = 1 for all y with yH ∈ supp ξ ∪ supp η, then  ξ, η = g(y)ξ (y), η(y)dy G

(Proposition 2.20). Thus the map [f ]ν → ξf extends uniquely to an inner product-preserving map W from Hν into H(indG H λμ ). W intertwines λν and G indH λμ . Indeed, for f ∈ Cc (G) and x, y ∈ G, ! " W (λν (y)[f ]ν )(x) = W [Ly f ]ν (x) = δ −1 · Lx −1 (Ly f )|H μ = (W [f ]ν )(y −1 x) = (indG H λμ (y)W [f ]ν )(x). It remains to show that the range of W is dense in H(indG H λμ ). To that end, let l ∈ Cc (G) and g ∈ Cc (H ). Then   −1 (l, [g]μ )(x) = δ(h )l(xh)λμ (h)[g]μ dh = δ(h−1 )l(xh)[Lh g]μ dh. H

H

! " This latter integral converges in Hμ and hence equals H δ(h−1 )l(xh)Lh gdh μ .  Now define a function f on G by f (y) = H δ(h−1 )l(yh)g(h−1 )dh. Then

78

Induced representations

f ∈ Cc (G) and, for k ∈ H and x ∈ G,

 δ −1 (k)Lx −1 f (k) = δ −1 (k) δ(h−1 )l(xkh)g(h−1 )dh H  −1 δ(h )l(xh)Lh g(k)dh. = H

This proves that

! " ξf (x) = δ −1 · Lx −1 f |H μ = (l, [g]μ )(x).

Since the elements (l, [g]μ ), l ∈ Cc (G), g ∈ Cc (H ), generate a dense linear subspace of H(indG  H λμ ), we conclude that W is surjective.

Summary Let us pause and remind ourselves that in this section we gave various equivalent realizations of the induced representation. We also want to point out how much the expressions simplify when one is inducing from the distinguished normal subgroup in a semidirect product (see Realization III for semidirect products). Let G be a locally compact group and H a closed subgroup. Let π be a representation of H . There is a representation of G called the representation of G induced from the representation π of H , or briefly, the induced representation. When all this data is understood, this representation will be denoted U π . When we want to emphasize the role of G and H , it will be denoted indG H π. When we need to be careful about the measure μ, it will be denoted Uμπ . To be more precise, we have formed three equivalent representations, any of which is called the induced representation. In the rest of this book, we shall use any of the following realizations of U π as is convenient.

Realization I H(U π ) is the completion of the inner product space F(G, π ), which consists of all continuous functions ξ : G → H(π ) with q(supp ξ ) compact in G/H and satisfying ξ (xh) = δ(h)π (h−1 )ξ (x), for x ∈ G and h ∈ H . The inner product of ξ, η ∈ F(G, π ) is formed by choosing a ψ ∈ Cc (G) with ψ # = 1 on q(supp ξ ) ∪ q(supp η) and putting  ξ, η = ψ(x)ξ (x), η(x)dx. G

For x ∈ G, U (x) is defined on H(U π ) by setting π

U π (x)ξ (y) = ξ (x −1 y), for ξ ∈ F(G, π ) and y ∈ G.

2.4 Other realizations

79

Realization II Fix a quasi-invariant measure μ on G/H and associated function ρ. Let dμx −1 μx (E) = μ(x · E), for x ∈ G and measurable subsets E of G/H . Let dμ denote the Radon–Nikodym derivative of μx −1 with respect to μ and recall that ρ(x −1 y) dμx −1 = (yH ), ρ(y) dμ for all x, y ∈ G. Then H(U π ) consists of all measurable functions ξ : G → H(π ) such that ξ (xh) = π(h−1 )ξ (x), for all h ∈ H and almost all x ∈ G, and satisfying 2 ˙ < ∞. The inner product of ξ, η ∈ H(U π ) is given by G/H ξ (x) dμ(x)  ˙ ξ, η = ξ (x), η(x)dμ(x). G/H

The induced representation is given by 1/2  ρ(x −1 y) π U (x)ξ (y) = ξ (x −1 y), ρ(y) for x ∈ G, ξ ∈ H(U π ), and almost all y ∈ G.

Realization III Fix a quasi-invariant measure μ on G/H and a measurable cross-section γ : G/H → G of the H -cosets. Then U π (x) acts on F ∈ L2 (G/H, H(π ), μ) by  1/2 ! " dμx −1 π π γ (ω)−1 xγ (x −1 · ω) F (x −1 · ω), U (x)F (ω) = (ω) dμ for μ-almost all ω ∈ G/H .

Realization III for Semidirect Products There are many useful situations where H is a closed normal subgroup of G and there is another closed subgroup L such that L ∩ H = {e} and G = H L (thus, G can be identified with a semidirect product of L acting on H by conjugation). The map l → lH identifies L with G/H and, since hlkH = lkH for hl ∈ G and k ∈ L, the action of G on G/H reduces to (hl) · k = lk. Clearly the left Haar measure on L identifies with a G-invariant measure on G/H . The inclusion map of L into G is a natural cross-section γ . If ω = k ∈ L, and x = hl ∈ G, then γ (ω)−1 xγ (x −1 · ω) = k −1 hl(l −1 k) = k −1 hk.

80

Induced representations

So if π is a representation of H on H(π ), then we can realize U π on L (L, H(π)) by, for x = hl ∈ G, h ∈ H, l ∈ L, and F ∈ L2 (L, H(π )), 2

U π (hl)F (k) = π (k −1 hk)F (l −1 k), for almost all k ∈ L.

2.5 The affine group and SL(2, R) The construction of an induced representation is algorithmic once all the input data is available. Deciding whether the resulting representation is irreducible and deciding whether two induced representations are equivalent or not is often challenging. Two groups are considered where the explicit construction of particular induced representations have proven, over time, to be extremely important in analysis. The first example is the group of affine linear transformations of the real line. An affine transformation is one of the form x → ax + b, with a = 0. Usually, attention is restricted to orientation-preserving affine transformations, so a > 0. This group was described in Example 1.5 and denoted Gaff . We will start the analysis on the larger group of affine transformations, where a can be both positive and negative, because the induced representation we consider is directly irreducible in that case. The other group is SL(2, R), the group of 2 × 2 real matrices of determinant 1. The full representation theory of SL(2, R) is technically involved and provides a prototype for the representation theory of semisimple Lie groups and algebraic groups in general. Among the irreducible unitary representations of SL(2, R) there is one family, known as the principle series, that consists of representations induced from a subgroup. We will define the principle series representations and prove their irreducibility. In the course of that proof, we make use of the irreducible representations of Gaff that are constructed earlier in the section. Recall that R∗ = R \ {0} with multiplication as the group product. Let a ∈ semidirect product R  R∗ . R∗ act on b ∈ R by a · b = ab. Let G denote  the  As in Example 1.6 (see Example 1.5 also), R∗ R f (b, a)a −2 db da is the left Haar integral on G and G (b, a) = |a|. Let H = {(0, a) : a ∈ R∗ } and let τ denote the trivial representation of H . Since K = {(b, 1) : b ∈ R} is complementary to H as in Example 2.29, we τ can use the formulation of indG H τ given in that example. Let U denote G τ 2 this realization of indH τ . Then H(U ) = L (K), which we identify with L2 (R).

2.5 The affine group and SL(2, R)

81

To connect with the notation of Example 2.29, let x = (b, a) ∈ G and l = (c, 1) ∈ K. Then   c−b −1 −1 −1 −1 −1 x l = (−a b, a )(c, 1) = (−a (c − b), a ) = , 1 (0, a)−1 . a Identifying (c, 1) with c in R, we have, for f ∈ L2 (R),   c−b . U τ (b, a)f (c) = |a|−1/2 f a Proposition 2.33 The representation U τ of G = R  R∗ defined above is irreducible. Proof We begin by noting that a routine calculation with the Fourier transform shows, for f ∈ L2 (R), γ ∈ R, (b, a) ∈ G, (U τ (b, a)f )ˆ(γ ) = |a|1/2 e2πibγ f(aγ ). To show that U τ is irreducible, we use the condition given in Proposition 1.32. So let f, g ∈ L2 (R) and form the coefficient function ϕf,g on G given by ϕf,g (b, a) = U τ (b, a)f, g, for (b, a) ∈ G. We wish to show that ϕf,g = 0 if both f and g are nonzero. This would imply that U τ is irreducible. To this end, we compute    |ϕf,g (b, a)|2 d(b, a) = |U τ (b, a)f, g|2 a −2 db da ∗ G R R |(U τ (b, a)f )ˆ,  g |2 a −2 db da = R∗ R 2       e2πibγ f(aγ )  |a|−1 db da = g (γ )   ∗ R R R |ωa∨ (b)|2 |a|−1 db da, = R∗

R

where, for each a ∈ R, the integrable function ωa is defined by ωa (γ ) = g (γ ), for all γ ∈ R, and ωa∨ is its inverse f(aγ )  Fourier transform. Now Plancherel’s theorem says that R |ωa∨ (b)|2 db = R |ωa (γ )|2 dγ , even if both sides of this equation are infinite. Thus,    |ϕf,g (b, a)|2 d(b, a) = |ωa (γ )|2 dγ |a|−1 da ∗ G R R |f(aγ )|2 | g (γ )|2 dγ |a|−1 da = R∗ R    2 −1  = |f (aγ )| |a| da | g (γ )|2 dγ . R

R∗

82

Induced representations

For γ = 0, as a runs through R∗, aγ varies over all of R \ {0}. Changing variables, R∗ |f(aγ )|2 |a|−1 da = R |f(ν)|ν|−1/2 |2 dν. Define, for 0 = ν ∈ R, (ν) = |ν|−1/2 . If f = 0 in L2 (R), then f is not a null function on R. Thus    |ϕf,g (b, a)|2 d(b, a) = |f(ν)(ν)|2 dν | g (γ )|2 dγ = 0. G

R

R

Therefore, ϕf,g = 0 if f and g are nonzero. This implies that U τ is irreducible.  Remark 2.34 Proposition 2.33 will be generalized in Theorem 7.19 and applied, in Theorem 7.42 and Examples 7.43 and 7.44, to give the continuous wavelet transform and generalizations. Note that Gaff (Example 1.5) is the connected component of the identity aff in G. Let ρ = indG H + τ , where τ is the trivial representation of the subgroup + H = {(0, a) : a > 0} of Gaff . It is interesting to see what change occurs in the irreducibility of the induced representation when R∗ is changed to R+ . Using the same method as above, we realize ρ on L2 (R) following Example 2.29. For (b, a) ∈ Gaff , f ∈ L2 (R), and x ∈ R,   x−b −1/2 ρ(b, a)f (x) = a f . a Let P : L2 (R) → L2 (R) denote the Plancherel transform, P(f ) = f. Define a representation σ , equivalent to ρ, by σ (b, a) = Pρ(b, a)P −1 , for all (b, a) ∈ Gaff . For g = f and γ ∈ R,    x − b −2πixγ −1/2 e σ (b, a)g(γ ) = [ρ(b, a)f ]ˆ(γ ) = a f dx a R = a 1/2 e−2πibγ f(aγ ) = a 1/2 e−2πibγ g(aγ ). If g is supported on (0, ∞), then so is σ (b, a)g since a takes only positive values. Thus, 1(0,∞) L2 (R) is a σ -invariant closed subspace of L2 (R). Like2 wise 1(−∞,0) L2 (R) is σ -invariant. Let H+ = {f ∈ L2 (R) : f(γ ) = 0, γ < 0} 2 2 2 2  and H− = {f ∈ L (R) : f (γ ) = 0, γ > 0}. Then H+ and H− are ρ-invariant 2 2 2 2 + subspaces of L (R) and L (R) = H+ ⊕ H− . Let ρ (b, a) = ρ(b, a)|H2+ and ρ − (b, a) = ρ(b, a)|H2− , for all (b, a) ∈ Gaff . It is now easy to adapt the proof of Proposition 2.33 to show that ρ + and ρ − are irreducible representations of Gaff . 2 can be described as the collection of all functions Remark 2.35 The space H+ 2 f ∈ L (R) such that there exists a holomorphic function F on the domain ∞ {x + iy ∈ C : y > 0} satisfying supy>0 −∞ |F (x + iy)|2 dx < ∞ and f (x) = limy→0 F (x + iy), for almost all x. See Stein and Weiss [146], 1.2 and 3.1 of 2 2 g(γ ) =  g (−γ ), H− = {f ∈ L2 (R) : f ∈ H+ }. chapter III, for the details. Since 

2.5 The affine group and SL(2, R)

83

It remains to show that ρ + and ρ − are not equivalent. We show this by restricting down to R identified with the subgroup {(b, 1) : b ∈ R}. Let ρR+ (b) = ρ + (b, 1), for b ∈ R. Likewise, define ρR− on R. If ρ + and ρ − were equivalent, then ρR+ and ρR− would be equivalent representations of R. We recall that equivalent representations of a locally compact group N have the same kernel when considered as ∗-representations of C ∗ (N). To see that ρR+ and ρR− have different kernels, let 0 = ψ ∈ L1 (R) be such 2 ) ⊆ [1, 2] ⊆ (0, ∞) and ψ  ≥ 0. For any f1 , f2 ∈ H− , that supp(ψ  ρR− (ψ)f1 , f2  = ψ(b)ρR− (b, 1)f1 , f2  db R  = ψ(b)σ (b, 1)f1 , f2  db R   = ψ(b) e−2πibγ f1 (γ )f2 (γ ) dγ db R  R  ψ(b)e−2πibγ db f1 (γ )f2 (γ ) dγ = R R (γ )f1 (γ )f2 (γ ) dγ = 0, = ψ R

. Thus, ψ ∈ ker ρR− . since f1 and f2 are zero on the support of ψ 2 On the other hand, select f ∈ H+ so that f(γ ) = 1, for γ ∈ [1, 2]. Then a calculation exactly as above shows that   (γ )|f(γ )|2 dγ = (γ ) dγ > 0. ρR+ (ψ)f, f  = ψ ψ R

ker ρR+ .

ρR+

R

ρR−

Thus, ψ ∈ / Since and are not equivalent representations of R, ρ + and ρ − are not equivalent as representations of Gaff . In summary, with τ denoting the trivial representation of the subgroup in ∗ RR+ each case, indRR {0}×R∗ τ is irreducible while ind{0}×R+ τ decomposes as the direct sum of two irreducible representations. We now turn to the special linear group   # a b SL(2, R) = : ad − bc = 1, a, b, c, d ∈ R . c d We construct an important family of representations of SL(2, R) via the inducing process and investigate their reducibility. For simplicity, let G = SL(2, R). Let P be the subgroup of G consisting of all upper-triangular matrices of determinant 1. Thus    # a b P = ma,b = : a ∈ R \ {0}, b ∈ R . 0 a −1

84

Induced representations

The commutator subgroup of P consists of all m1,b , b ∈ R. So every character of P is of the form χt+ (ma,b ) = |a|it

or

χt− (ma,b ) = |a|it sgn(a) (t ∈ R).

The so-called principal series representations of G are then defined by + πt+ = indG P χt

and

− πt− = indG P χt .

To obtain explicit realizations of the principal series representations, we recall the third realization presented in Section 2.4 applied to the situation with G = SL(2, R), H = P , and χ one of the characters of P . Fix a quasi-invariant measure μ on G/P and a measurable cross-section γ : G/P → G of the P -cosets. Then, for x ∈ G, U χ (x) acts on f ∈ L2 (G/P , μ) by  1/2 ! " dμx −1 χ U (x)f (ω) = χ γ (ω)−1 xγ (x −1 · ω) f (x −1 · ω), (ω) dμ for μ almost all ω ∈ G/P . A useful parametrization of G/P is obtained by recognizing that for a = 0,      a b 1 0 a b = c d c/a 1 0 a −1 and, for b = 0,



0 −b−1

b d



 =

0 1 −1 0



b−1 0

−d b

 .

So, as Borel sets, we can identify G/P with R ∪ {ω0 } and select the measurable cross-section γ : G/P → G to be given by     0 1 1 0 γ (ω0 ) = and γ (ω) = , −1 0 ω 1 for ω ∈ R. Note that the quotient map q : G → G/P is given by     0 b a b = ω q and q = c/a 0 −b−1 d c d if a = 0. For x ∈ Gand ω ∈ G/P , we need to calculate x −1 · ω = q(x −1 γ (ω)). If a b x= and ω ∈ R, then c d      d −b 1 0 d − bω −b −1 x γ (ω) = = . −c a ω 1 −c + aω a

2.5 The affine group and SL(2, R)

85

aω−c , as long as ω = d/b. The interested reader can check that So x −1 · ω = −bω+d −1 x · (d/b) = ω0 , x −1 · ω0 = −a/b, if b = 0, and x −1 · ω0 = ω0 , if b = 0. If we define a measure μ on G/P so that it coincides with a Lebesgue measure on R and μ(ω0 ) = 0, then μ is a quasi-invariant measure on G/P . Up to measure aω−c , zero, we are only concerned with the transformations ω → x −1 · ω = −bω+d   a b where we continue to let x = . Then one computes the derivative c d   dμx −1 1 . (ω) = dμ (−bω + d)2

The final ingredient we need for the induced representation is the identity     1 0 1 0 −1 −1 x aω−c γ (ω) xγ (x · ω) = 1 −ω 1 −bω+d  1  b = −bω+d . 0 −bω + d Putting all of this together, we get     aω − c a b f (ω) = | − bω + d|−1 χ((−bω + d)−1 )f . Uχ c d −bω + d Letting χ be χt+ and χt− , respectively, we get     aω − c b + a −1−it f (ω) = | − bω + d| f πt c d −bω + d and πt−



a c

b d

 f (ω) = sgn(−bω + d)| − bω + d|

−1−it

 f

aω − c −bω + d



 a b ∈ G, f ∈ L2 (R), and ω ∈ R. c d Consider the closed subgroup L of G defined as  #  a 0 : a, c ∈ R, a > 0 . L= c a −1

for

Note that, if b = 0, then    a b 1 = c d db−1

0 1



0 −1

1 0



b−1 a

0 b

 .

 ,

86

Induced representations 

   0 1 0 −1 Therefore, G = L ∪ L L∪L L and we can study the −1 0 1 0 irreducibility   of a representation of G by checking its behavior on L and 0 1 . −1 0 First, we take advantage of the fact that L is isomorphic to the affine group. Define α : Gaff → L by  −1/2  a 0 α(b, a) = , a −1/2 b a 1/2 for all (b, a) ∈ Gaff . Then one checks that α is an isomorphism of locally compact groups. For each t, ρt = πt+ ◦ α is therefore a representation of Gaff acting on L2 (R). By composing the formula for πt+ with α, and simplifying, we get   ω−b −1/2 −it/2 ρt (b, a)f (ω) = a a f , a for (b, a) ∈ Gaff . Note that ρt (b, a) = a −it/2 ρ(b, a), where ρ is the representation of Gaff studied earlier in this section. Since a −it/2 is a scalar, each ρt has the same invariant subspaces as ρ. 2 = {f ∈ L2 (R) : f(γ ) = 0, γ < 0} has no nontrivial We deduce that H+ 2 . This proves the following closed ρt -invariant subspaces; likewise for H− lemma. Lemma 2.36 Any closed subspace of L2 (R) which is invariant under the set 2 2 , H− , or L2 (R). of unitaries {πt+ (x) : x ∈ L} is one of {0}, H+   0 1 Now, consider the action of . For f ∈ L2 (R) and almost all ω ∈ R, −1 0   0 1 + πt f (ω) = |ω|−1−it f (−1/ω). −1 0 2 We wish to show that this unitary operator on L2 (R) does not leave H+ −1 invariant. To that end, let f (ω) = (ω + i) . Then an application of the criterion 2 . Then in Remark 2.35 shows that f ∈ H+   1 0 1 πt+ f (ω) = −sgn(ω)i|ω|−it −1 0 ω+i   0 1 2 after simplification. We claim that g = πt+ . To see this, f ∈ / H+ −1 0 2 . By Remark 2.35, there exists a holomorassume to the contrary that g ∈ H+ phic function F on {z ∈ C : Imz > 0} so that limy→0 F (ω + iy) = g(ω) almost

2.6 Basic properties

87

everywhere in ω. Define F1 on {z ∈ C : Imz > 0} by F1 (z) = izF (z). Then F1 is holomorphic on {z ∈ C : Imz > 0} and lim F1 (ω + iy) = sgn(ω)|ω|−it ,

y→0

for almost every ω ∈ R. Let Log(z) denote the branch of log(z) defined on C \ {−iy : y ≥ 0} that agrees with the natural logarithm, ln, on the positive real axis. Then Log(z) is holomorphic on C \ {−iy : y ≥ 0} and Log(ω) = ln(|ω|) + iπ if ω < 0. Thus, e−itLog(z) is holomorphic on C \ {−iy : y ≥ 0}, e−itLog(ω) = ω−it if ω > 0, and e−itLog(ω) = |ω|−it eπt if ω < 0. Define F2 (z) = e−itLog(z) − F1 (z), for all z ∈ C, Imz > 0. Then F2 is holomorphic on {z ∈ C : Imz > 0} and lim F2 (ω + iy) = 0,

y→0

if ω > 0. This implies F2 (z) ≡ 0 on {z ∈ C : Imz > 0} (see, for example, theorem 1.9 in [154]). But, if ω < 0,   lim F2 (ω + iy) = lim e−itLog(ω+iy) − lim F1 (ω + iy) = |ω|−it eπt + 1 . y→0

y→0



y→0

 0 1 2 f ∈ / H+ as claimed. −1 0 2 2 Since H+ is not πt+ -invariant, neither is H− . Combining this with Lemma + 2.36 completes the proof that πt is irreducible for all t. This argument can be adapted to show that πt− is irreducible as long as t = 0. In fact, it can be shown that π0− is reducible into two irreducible summands, 2 2 and the other on H− . It also turns out that certain pairs of one acting on H+ principle series representations are mutually equivalent. However, we will not go into the details here. This is a contradiction. Therefore, πt+

2.6 Some basic properties of induced representations In this section a number of very natural properties of induced representations are presented. The proofs are elementary applications of the definitions. It is, however, worthwhile working through the details in order to become familiar with the standard arguments and the notation. This will help in following more complicated proofs later. Moreover, these properties themselves will be used a number of times in the sequel. Let H be a Hilbert space and let H∗ be the dual space of H. For v ∈ H, let fv be the element of H∗ defined by fv (w) = w, v, w ∈ H. Then v → fv is a conjugate linear isomorphism between H and H∗ , and H∗ equipped with

88

Induced representations

the inner product fv , fw  = v, w is a Hilbert space which is denoted H and called the conjugate Hilbert space of H. Suppose that π is a unitary representation of a locally compact group G on H. Then π determines another unitary representation π¯ on H by setting −1 π(x)f ¯ v (w) = fv (π(x )w) = fπ(x)v (w) (v, w ∈ H, x ∈ G).

It is clear from the definition of π¯ that π¯ (x)fv , fw  = π(x −1 )w, v (v, w ∈ H). π¯ is called the conjugate representation of π . When π is a character, then of course π¯ is the complex conjugate character. The proof of the following lemma is fairly straightforward. So we only give a sketch and leave the details to the reader. Lemma 2.37 Let π be a representation of the closed subgroup H of G. Then G indG ¯. H π = indH π

¯ ¯ Proof Let σ = indG H π and, for ξ ∈ F(G, π ), define ξ : G → H by ξ (x) = ¯ and that the map fξ (x) , x ∈ G. Then it is easily verified that ξ¯ ∈ F(G, π) W : fξ → ξ¯ from F(G, π ) ⊆ H(σ ) onto F(G, π¯ ) is linear and inner product preserving. Now, for η ∈ F(G, π ) and x, y ∈ G, ¯ = W σ¯ (x)fη (y) = Wfσ (x)η (y) (W σ¯ (x)W −1 )η(y) = σ (x)η(y) = fσ (x)η(y) = fη(x −1 y) = η(x ¯ −1 y) = indG ¯ (x)η(y). ¯ H π Since F(G, π) ¯ and F(G, π ) are dense in H(indG ¯ ) and H(σ¯ ), respectively, we H π ¯.  conclude that W extends to a unitary equivalence between σ¯ and indG H π The next of these properties states that lifting a representation from a quotient group commutes with inducing. Proposition 2.38 Let H be a closed subgroup of G, let N be a closed normal subgroup of G contained in H , and let q : G → G/N be the quotient homomorphism. Let π be a unitary of H /N and let π˜ = π ◦ q|H .  representation  G/N G Then indH π˜ is equivalent to indH /N π ◦ q. Proof Note that H(π) ˜ = H(π ) and π(n) ˜ = I for all n ∈ N . Moreover, the modular functions G and H , when restricted to N , both agree with N . Thus, for any ξ ∈ F(G, π), ˜ x ∈ G, and n ∈ N, ˜ −1 )ξ (x) = ξ (x). ξ (xn) = H (n)1/2 G (n)−1/2 π(n So we can define ξ (xN) = ξ (x), for any xN ∈ G/N.

2.6 Basic properties

89

We claim that ξ ∈ F(G/N, π). It is clear that ξ is continuous and has compact support modulo H /N . So it remains to check the covariance condition. In preparation for this consider, for each y ∈ G, the automorphism n → yny −1 of N . There is a positive number δy such that   −1 f (yny )dn = δy f (n)dn N

N

 for all f ∈ Cc (N ). For f ∈ Cc (G), let f # (xN) = N f (xn)dn. Then, by Weil’s formula (1.4) with y ∈ G,   G/N (y −1 N ) f # (xN)d(xN) = f # (xyN )d(xN) G/N G/N   f (xyn)dnd(xN) = G/N N   = δy f (xny)dnd(xN ) G/N N  = δy f (zy)dz G   = δy G (y −1 ) f (xn)dnd(xN) G/N N  f # (xN)d(xN). = δy G (y −1 ) G/N

Since f ∈ Cc (G) is arbitrary, G/N (y −1 N ) = δy G (y −1 ). Likewise, H /N (t −1 N ) = δt H (t −1 ) for all t ∈ H. Hence, for t ∈ H, H (t)G (t)−1 = H /N (tN )G/N (tN )−1 . Hence we obtain, for x ∈ G and t ∈ H, ξ (xN tN ) = ξ (xt) = H (t)1/2 G (t)−1/2 π˜ (t −1 )ξ (x) = H /N (tN )1/2 G/N (tN )−1/2 π (t −1 N )ξ (xN). This shows that ξ ∈ F (G/N, π). Clearly,  is a linear map of F (G, π) ˜ into F(G/N, π). Let η ∈ F(G/N, π). Define ξ : G → H(π) ˜ = H(π ) by ξ (x) = η(xN), for x ∈ G. A calculation similar to the one above shows that ξ ∈ F (G, π) ˜ and ξ = η. Thus  is onto. Now, for any fixed ξ ∈ F (G, π), ˜ choose a compact subset C of G such that the support of ξ is contained in CH and an f ∈ Cc (G) such that f H (x) = 1

90

Induced representations

for all x ∈ CH. Then f # ∈ Cc (G/H ) and    # H /N # (f ) (xN) = f (xtN )d(tN ) = f (xtn)dnd(tN ) H /N H /N N  = f (xs)ds = f H (x) = 1, H

for all x ∈ CH and hence all xN ∈ q(CH ). Therefore,  f # (xN) ξ (xN ) 2 d(xN)

ξ 2 = G/N   2 =

ξ (xN ) f (xn)dnd(xN ) G/N N   =

ξ (xn) 2 f (xn)dnd(xN ) G/N N  = f (z) ξ (z) 2 dz = ξ 2 . G G/N

Thus  extends to a unitary map of H(indG ˜ onto H(indH /N π ). H π) Finally, for x, y ∈ G and ξ ∈ F(G, π˜ ), ˜ ](yN ) = [indG ˜ ](y) = ξ (x −1 y) [ indG H π(x)ξ H π(x)ξ = ξ (x −1 yN) = [indH /N π (xN)ξ ](yN ) G/N

G/N

= [((indH /N π) ◦ q)(x)ξ ](yN ). G/N

˜ and (indH /N π) ◦ q, and we have established that Thus  intertwines indG H π these representations are unitarily equivalent.  We show next that conjugating the subgroup H and the representation π accordingly leads to equivalent induced representations. For a ∈ G, let a · π denote the representation of the subgroup aH a −1 on H(π) defined by (a · π )(x) = π (a −1 xa), for all x ∈ aH a −1 . Proposition 2.39 Let H be a closed subgroup of G and π a unitary representation of H . For any a ∈ G, the induced representations indG aH a −1 a · π and indG π are equivalent. H Proof Of course, we can assume that the Haar measure on aH a −1 is the image of the Haar measure on H under the mapping t → ata −1 . Then, in particular, aH a −1 (t) = H (a −1 ta), for all t ∈ aH a −1 . Let ξ ∈ F(G, π ), and define ξ : G → H(a · π ) = H(π ) by ξ (x) = G (a)1/2 ξ (xa). Clearly, ξ is continuous. Next, choose a compact subset C of

2.6 Basic properties

91

G such that supp ξ ⊆ CH. Then supp ξ ⊆ CH a −1 = (Ca −1 )(aH a −1 ). Thus ξ has compact support modulo aH a −1 . Moreover, for x ∈ G and t ∈ aH a −1 , ξ (xt) = G (a)1/2 ξ (xta) = G (a)1/2 ξ (xa(a −1 ta)) = G (a)1/2 H (a −1 ta)1/2 G (a −1 ta)−1/2 π (a −1 t −1 a)ξ (xa) = aH a−1 (t)1/2 G (t)−1/2 G (a)1/2 (a · π)(t −1 )ξ (xa) = aH a−1 (t)1/2 G (t)−1/2 (a · π )(t −1 )ξ (x). This shows that ξ ∈ F(G, a · π). Clearly,  is a linear map of F(G, π ) into F (G, a · π ). Conversely, given η ∈ F(G, a · π ), let ξ (x) = G (a −1 )1/2 η(xa −1 ), for all x ∈ G. A calculation similar to the one above shows that ξ ∈ F(G, π ) and ξ = η. Thus  is onto. Now, fix any ξ ∈ F(G, π ) and choose a compact subset C of G such that the support of ξ is contained in CH and an f ∈ Cc (G) such that f (xt)dt = 1, H

for all x ∈ C. Let g(x) = f (xa), for x ∈ G. Then, for x ∈ (Ca −1 )(aH a −1 ),    g(xt)dt = f (x(asa −1 )a)ds = f (xas)ds = 1, aH a −1

H

H

since xa ∈ CH. Moreover, since the support of ξ is contained in (Ca −1 )(aH a −1 ), we have   2 2 g(x) ξ (x) dx = G (a) f (xa) ξ (xa) 2 dx

ξ = G G  2 2 = f (x) ξ (x) dx = ξ . G G Thus  extends to a unitary map of H(indG H π) onto H(indaH a −1 a · π). Finally, for x ∈ G and ξ ∈ F(G, π ), −1 1/2 (indG ξ (x −1 ya) aH a −1 (a · π)(x)ξ )(y) = ξ (x y) = G (a)

= G (a)1/2 (indG H π (x)ξ )(ya) = (indG H π(x)ξ )(y). G Thus  intertwines indG H π and indaH a −1 a · π.



The third of the properties alluded to at the outset of this section is a formula for the restriction of an induced representation to a normal subgroup. First, we establish this formula in a special case. Lemma 2.40 Let H be a closed subgroup of G and N a closed normal subgroup of G contained in H . Let π be a unitary representation of H , and let f ∈ Cc (N )

92

Induced representations

and ξ ∈ F(G, π ). Then, for all x ∈ G, [(indG H π)|N (f )ξ ](x) = (x · (π |N ))(f )(ξ (x)). Proof Define a map η : G → H(π ) by  f (n)π (x −1 nx)(ξ (x))dn η(x) = (x · (π|N ))(f )(ξ (x)) = N  = f (n)G (x −1 nx)1/2 H (x −1 nx)−1/2 ξ (x(x −1 nx)−1 )dn. N

Since N is normal in G, the modular functions G and H , when restricted to N, agree. Thus   f (n) indG η(x) = f (n)ξ (n−1 x)dn = H π (n)ξ (x)dn. N

N

Since f ∈ Cc (N ) and ξ has compact support modulo H , it follows that η is continuous and η has compact support modulo H . Moreover,   η(xt) = f (n)ξ (n−1 xt)dn = H (t)1/2 G (t)−1/2 π (t −1 ) f (n)ξ (n−1 x)dn N

N

= H (t)1/2 G (t)−1/2 π(t −1 )η(x), for all x ∈ G and t ∈ H. This proves that η ∈ F(G, π ). Now,  let ω be an arbitrary element of F(G, π ) and choose g ∈ Cc (G) such that g(xt)dt = 1 for all x in the support of ω. Then H



 

g(x)η(x), ω(x)dx = g(x)f (n)ξ (n−1 x), ω(x)dndx G N   = f (n) g(x)(indG H π(n)ξ )(x), ω(x)dxdn N G  G = f (n)indG H π(n)ξ, ωdn = (indH π )|N (f )ξ, ω.

η, ω =

G

N

G Since F(G, π ) is dense in H(indG H π), it follows that η(x) = (indH π )|N (f )ξ (x) locally almost everywhere on G. However, both η and (indG H π )|N (f )ξ are continuous on G. This finishes the proof of the lemma. 

Proposition 2.41 Let H be a closed subgroup of G and N a closed normal subgroup of G contained in H . Let π be a unitary representation of H , and let f ∈ C ∗ (N ) and ξ ∈ H(indG H π). Then [(indG H π)|N (f )ξ ](x) = (x · (π|N ))(f )(ξ (x)) for locally almost all x ∈ G.

2.6 Basic properties

93

Proof Suppose first that f ∈ Cc (N). There are ξn ∈ F(G, π), n ∈ N, such that ξn → ξ in H(indG H π). Let K be  any compact subset of G. By Lemma 2.25, there exists cK > 0 such that η(x) dx ≤ cK η , for all η ∈ H(indG H π ). K

Using Lemma 2.40, we get 

((indG H π )|N (f )ξ )(x) − (x · (π|N ))(f )(ξ (x)) dx K 

((indG ≤ H π)|N (f )(ξ − ξn ))(x) dx K  +

(x · (π |N ))(f )[ξn (x) − ξ (x)] dx K  ≤ cK (indG π )| (f )(ξ − ξ ) +

(x · (π |N ))(f ) · ξn (x) − ξ (x) dx N n H K 

ξn (x) − ξ (x) dx ≤ 2cK f 1 ξ − ξn . ≤ cK f 1 ξ − ξn + f 1 K

Since ξn → ξ , it follows that, for every such K, the first integral is zero, and this implies that ((indG H π)|N (f )ξ )(x) = (x · (π|N ))(f )(ξ (x)) locally almost everywhere. Finally, let f ∈ C ∗ (N) and ξ ∈ H(indG H π) be arbitrary. Choose fn ∈ ∗ Cc (N ), n ∈ N, such that fn → f in C (N). Then, for every x ∈ G,

(x · (π|N ))(fn )(ξ (x)) − (x · (π|N ))(f )(ξ (x)) ≤ fn − f C ∗ (N) ξ (x) , (2.1) which converges to zero as n → ∞. Similarly, G

(indG H π )|N (f )ξ − (indH π)|N (fn )ξ ≤ f − fn C ∗ (N) ξ → 0,

(2.2)

as n → ∞. As in the first part of the proof, we now conclude from (2.1) and (2.2) that ((indG H π )|N (f )ξ )(x) = (x · (π |N ))(f )(ξ (x)) holds for locally almost all x ∈ G.  We now show that inducing respects the process of taking direct sums of representations. Recall that δ(h) = H (h)1/2 G (h)−1/2 , for h ∈ H . Proposition 2.42 Let H be a closed subgroup of the locally compact group G, and let πλ , λ ∈ , be any family of unitary representations of H . Then G indG H (⊕λ∈ πλ ) = ⊕λ∈ indH πλ . G Proof Let π = ⊕λ∈ πλ and σ = indG H π, and for each λ ∈ , let σλ = indH πλ and Pλ be the orthogonal projection of H(π ) onto H(πλ ).

94

Induced representations

For ξ ∈ F(G, π) and λ ∈ , define ξλ : G → H(πλ ) by ξλ (x) = Pλ (ξ (x)). Then, for h ∈ H and x ∈ G, ξλ (xh) = Pλ (ξ (xh)) = Pλ (δ(h)π(h−1 )ξ (x)) = δ(h)πλ (h−1 )ξλ (x). Thus ξλ satisfies conditions (1), (2), and (3) at the outset of Section 2.3, that is, ξλ ∈ F(G, πλ ). The linear mapping W : F(G, π) → ⊕λ∈ F(G, πλ ) ⊆ ⊕λ∈ H(πλ ), defined by (W ξ )λ (x) = ξλ (x), x ∈ G, λ ∈ , is an isometry. Indeed, with ψ as in Proposition 2.20 and since supp ξλ ⊆ supp ξ ,  

(W ξ )λ 2 = ψ(x) ξλ (x) 2 dx

W ξ 2 = λ∈

$

 =

ψ(x) G

λ∈

 λ∈

G

Pλ (ξ (x))

% 2

 dx =

ψ(x) ξ (x) 2 dx = ξ 2 . G

It is easily verified that Pλ (W (F(G, π))) is a πλ -invariant subspace of H(πλ ) and that W intertwines the representations σ and ⊕λ∈ σλ . Therefore it only remains to observe that W (F(G, π)) is dense in ⊕λ∈ H(πλ ). To that end, let  be a finite subset of  and for each λ ∈  , let ηλ ∈ F(G, πλ ) be given. Define η : G → H(π) by η(x)λ = 0 for λ ∈  and η(x)λ = ηλ (x) for λ ∈  . Then η is continuous, q(supp η) ⊆ ∪{q(supp ηλ ) : λ ∈  }, and η(xh)λ = ηλ (xh) = δ(h)πλ (h−1 )ηλ (x), for all x ∈ G, h ∈ H , and λ ∈ . Thus η(xh) = δ(h)π (h−1 )η(x). This shows  that η ∈ F(G, π), and by definition of W , W η = ⊕λ∈ ηλ . As an immediate consequence of Proposition 2.42 we obtain Corollary 2.43 If π is a representation of H and indG H π is irreducible, then π is irreducible. The converse of this corollary is, of course, far from being true in general. For a homomorphism  from a group A into a group B, let ker  denote the kernel of . Let H be a closed subgroup of the locally compact group G. Note that since H and G |H are homomorphisms from H into the multiplicative group of positive real numbers, for any representation π of H , ker π ∩ ker(H /G ) is a subgroup of H . Lemma 2.44 Let π be a representation of a closed subgroup H of a locally compact group G and let x and y be elements of G such that x ∈ yH . Then there exists ξ ∈ F(G, π) such that ξ (x) = 0 and ξ (yh) = 0, for all h ∈ H .

2.6 Basic properties

95

Proof Fix any nonzero v ∈ H(π ). Since π is continuous and x ∈ yH , there exists a compact neighborhood V of e in G such that xV ∩ yH = ∅ and Reπ (h)v, v > 0 for all h ∈ V . Next, choose f ∈ Cc+ (G) such that f  (x) > 0 and supp f ⊆ xV , and let ξ = (f, v) ∈ F (G, π). Then ξ (yh) = H δ(k)f (yhk)π (k)vdk = 0, for every h ∈ H since f |yH = 0. On the other hand   δ(h)f (xh)π (h)v, vdh Reξ (x), v = Re H  δ(h)f (xh)Reπ(h)v, vdh > 0, = V

since f (x) > 0 and Reπ(h)v, v > 0, for every h ∈ V . This proves that ξ (x) = 0.  Theorem 2.45 Let H be a closed subgroup of the locally compact group G and let π be a representation of H . Then    & H G −1 ker(indH π) = ker π ∩ ker a. a G a∈G Proof Let σ = indG H π and F = F(G, π). Since F is a σ -invariant dense linear subspace of H(σ ), for x ∈ G we have x ∈ ker σ if and only if σ (x) is the identity operator on F. First, let x be an element of G such that axa −1 ∈ ker π ∩ ker(H /G ) for all a ∈ G. Then, for all ξ ∈ F and y ∈ G, σ (x)ξ (y) = ξ (x −1 y) = ξ (y(y −1 x −1 y)) = δ(y −1 x −1 y)π(y −1 xy)ξ (y) = δ(x −1 )ξ (y) = ξ (y). Thus σ (x) is the identity on F. Conversely, let x ∈ ker σ . Then, for all ξ ∈ F and y ∈ G, ξ (y) = σ (x)ξ (y) = ξ (x −1 y). In particular, ξ (x −1 ) = ξ (e). By Lemma 2.44, this implies that x ∈ H . But then ξ (e) = ξ (x −1 ) = δ(x −1 )π(x)ξ (e), for all ξ ∈ F . Since the set of all ξ (e), ξ ∈ F , is total in H(π) (Lemma 2.24(ii)), it follows that δ(x −1 )π (x) is the identity on H(π ). However, since π(x) is a unitary operator, this implies first that G (x) = H (x) and then that x ∈ ker π . Since ker σ is a normal subgroup of G, we obtain that    & H −1 ker π ∩ ker a, a x∈ G a∈G as was to be shown.



96

Induced representations

Corollary 2.46 If H does not contain any nontrivial normal subgroup of G, then indG H π is a faithful representation of G.

2.7 Induction in stages Let G be a locally compact group with two closed subgroups K and H such that K ⊆ H , and let π be a unitary representation of K. Then we can form the induced representation indH K π of H , which in turn by inducing leads to the H representation ρ = indG (ind H K π) of G. One says that ρ is the representation of G obtained by inducing in stages. On the other hand, we can induce π straight away up to G. Our objective is to show that these two procedures result in equivalent representations. Thus we are going to prove the following. Theorem 2.47 Let K and H be closed subgroups of the locally compact group G such that K ⊆ H , and let π be a unitary representation of K. Then G H the representations indG K π and indH (indK π) are unitarily equivalent. The reader will expect that in order to establish this theorem we shall produce an isometric isomorphism  between certain dense linear subspaces of H G G H H(indG H (indK π )) and H(indK π) which are invariant under indH (indK π ) and G indK π , respectively, such that  intertwines the two representations on those subspaces. For this entire section we fix the following notation. If A is a locally compact group, B a closed subgroup of A, and τ a unitary representation of B in the Hilbert space H(τ ), then we denote by F(A, τ ) the set of all continuous mappings ξ : A → H(τ ) with the following properties: 1. ξ has compact support modulo B. 2. ξ (ab) = A (b)−1/2 B (b)1/2 τ (b−1 )ξ (a), for all a ∈ A, b ∈ B. Recall from Section 2.3 that F(A, τ ) is dense in H(indA B τ ) and invariant under all operators indA τ (a), a ∈ A. In what follows let K, H, G, and π be as in B Theorem 2.47. The conclusion of Theorem 2.47 will be a consequence of the following four lemmas. Lemma 2.48 For each ξ ∈ F (G, π) and x ∈ G, define ξ (x) : H → H(π ) by ξ (x)(h) = H (h)−1/2 G (h)1/2 ξ (xh), for all h ∈ H . Then ξ (x) ∈ F(H, π ).

(2.3)

2.7 Induction in stages

97

Proof It is clear that ξ (x) is a continuous mapping from H into H(π). Notice next that ξ (x) satisfies the covariance condition. Indeed, for every h ∈ H and k ∈ K, ξ (x)(hk) = H (hk)−1/2 G (hk)1/2 ξ (xhk) = H (hk)−1/2 G (hk)1/2 G (k)−1/2 K (k)1/2 π (k −1 )ξ (xh) = H (hk)−1/2 G (h)1/2 K (k)1/2 π (k −1 )ξ (xh)   = H (k)−1/2 K (k)1/2 π(k −1 ) H (h)−1/2 G (h)1/2 ξ (xh)   = H (k)−1/2 K (k)1/2 π(k −1 ) ξ (x)(h) . In addition, ξ (x) has compact support modulo K, which can be seen as follows. If C is a compact subset of G such that supp ξ ⊆ CK, then for each x ∈ G, by definition of ξ (x), supp ξ (x) ⊆ x −1 CK ∩ H = (x −1 C ∩ H )K,

(2.4)

since K ⊆ H . This proves ξ (x) ∈ F(H, π ).



Lemma 2.49 For any ξ ∈ F (G, π) and x ∈ G, let ξ (x) ∈ F (H, π ) be defined as in (2.3). Then the mapping ξ : x → ξ (x) from G into F (H, π ) ⊆ H H(indH K π ) belongs to F(G, indK π). Proof For x ∈ G and h, h1 ∈ H we have, with ρ = indH K π, ξ (xh)(h1 ) = H (h1 )−1/2 G (h1 )1/2 ξ (xhh1 )   = H (h)1/2 G (h)−1/2 H (hh1 )−1/2 G (hh1 )1/2 ξ (xhh1 ) = H (h)1/2 G (h)−1/2 ξ (x)(hh1 ) = H (h)1/2 G (h)−1/2 ρ(h−1 )ξ (x)(h1 ). Thus, for all x ∈ G and h ∈ H , ξ (xh) = G (h)−1/2 H (h)1/2 ρ(h−1 )(ξ (x)), so that ξ satisfies the covariance condition for elements in H(indG H ρ). Let C be a compact subset of G such that supp ξ ⊆ CK. If x ∈ G is such that ξ (x) = 0, then ξ (xh) = 0 for some h ∈ H , and hence x ∈ CKH = CH . Thus, ξ has compact support modulo H . To prove ξ ∈ F (G, ρ), it remains to show that x → ξ (x) is continuous from G into H(ρ).

98

Induced representations

To that end,  G and choose f ∈ Cc (G)  fix a compact neighborhood V of e in such that f (xk)dk = 1, for all x ∈ V C. Then f (yk)dk = 1, for all y ∈ K

V CK. Hence, for any x ∈ G and h ∈ H , we have  f (xhk)dk = 1,

K

(2.5)

K

provided that xh ∈ V CK. That is, h ∈ x −1 V CK ∩ H = (x −1 V C ∩ H )K.

(2.6)

Now consider x, y ∈ G such that y −1 x ∈ V . Then, by (2.4), supp ξ (y) ⊆ (y −1 C ∩ H )K ⊆ (x −1 V C ∩ H )K. It follows from (2.6) that (2.5) holds for all h ∈ supp ξ (y) ∪ supp ξ (x). This implies, for all x, y ∈ G with y ∈ xV −1 , 

ξ (y) − ξ (x) 2 = f (xh) ξ (y)(h) − ξ (x)(h) 2 dh H



=

f (xh)H (h)−1 G (h) ξ (yh) − ξ (xh) 2 dh.

H

Since f is a continuous function with compact support and ξ is uniformly continuous on every compact set, it is now straightforward to deduce that ξ is continuous.  Lemma 2.50 Let K, H , and π be as above. The linear mapping  : ξ → ξ from F (G, π) into F(G, indH K π) is isometric. Proof Let ξ ∈ F(G, π) and  fix C ⊆ G compact such that supp ξ ⊆ CK. Choose f ∈ Cc (G) so that f (xk)dk = 1, for all x ∈ CK. Then (2.5) holds K

for all h ∈ x −1 CK ∩ H = (x −1 C ∩ H )K. Thus, using (2.4), (2.5) holds for all h ∈ supp ξ (x). For each x ∈ G, it follows that 

ξ (x) 2 = f (xh)H (h)−1 G (h) ξ (xh) 2 dh. (2.7) H

Now, choose g ∈ Cc (G) such that

 H

g(yh)dh = 1, for all y ∈ CH . Then, using

(2.7) and supp ξ ⊆ CH , repeated application of Fubini’s theorem and the

2.7 Induction in stages

99

choice of f and g yield  2

ξ = g(x) ξ (x) 2 dx 

G



=

g(x) G



=

H

H (h)−1 G (h)

H



=

H (h) H

−1



−1

G





 H (h)−1 g(xh−1 )dh dx

 H

 g(xh)dh dx

2

f (x) ξ (x) 

G

=

 2

g(xh )f (x) ξ (x) dx dh

f (x) ξ (x) 2

=

 g(x)f (xh) ξ (xh) 2 dx dh

G

 G



=

 f (xh)H (h)−1 G (h) ξ (xh) 2 dh dx

H

f (x) ξ (x) 2 dx = ξ 2 .



G

Lemma 2.51 (F(G, π )) is dense in F(G, indH K π ). Proof To find a total set in F(G, indH K π) which is contained in (F (G, π )), it is most convenient to make use of the (·, ·)-elements in Hilbert spaces of induced representations. Recall first that we have mappings  : Cc (G) × H(π ) → F(G, π ),  (f, v)(x) = K (k)−1/2 G (k)1/2 f (xk)π (k)vdk, K

for f ∈ Cc (G), v ∈ H(π ), and x ∈ G, 1 : Cc (H ) × H(π) → F(H, π ),  1 (g, v)(h) = K (k)−1/2 H (k)1/2 g(hk)π (k)vdk, K

for g ∈ Cc (H ), v ∈ H(π ), and h ∈ H , and H 2 : Cc (G) × H(indH K π) → F(G, indK π ),  2 (f, w)(x) = H (h)−1/2 G (h)1/2 f (xh) indH K π (h)wdh, H

for f ∈ Cc (G), w ∈ H(indH K π), and x ∈ G.

100

Induced representations

Now, let f1 ∈ Cc (H ) and f2 ∈ Cc (G), and define f : G → C by  f (x) = H (h)−1/2 G (h)1/2 f1 (h−1 )f2 (xh)dh. H

It is easily checked that f is a continuous function with compact support. We are going to show that for any v ∈ H(π), ((f, v)) = 2 (f2 , 1 (f1 , v)).

(2.8)

G G Recall that δH (h) = H (h)1/2 G (h)−1/2 , δK (k) = K (k)1/2 G (k)−1/2 , and H 1/2 −1/2 , for all h ∈ H and k ∈ K, respectively. Then, δK (k) = K (k) H (k) for each x ∈ G,  G −1 (f, v)(x) = δK (k )f (xk)π(k)vdk K



=



G −1 δK (k )

K

G −1 δH (h )f1 (h−1 )f2 (xkh)dh

 π (k)vdk.

H

Using this and the definition of , for every h1 ∈ H , G −1 ((f, v))(x)(h1 ) = δH (h )(f, v)(xh1 )  1 G −1 G δK (k )δH (h1 h)−1 f1 (h−1 )f2 (xh1 kh)π (k)vdhdk = K H

 

=

G −1 G −1 −1 δK (k )δH (k h) f1 (h−1 h1 k)f2 (xh)π (k)vdhdk

K H

 

=

H −1 G −1 δK (k )δH (h )f1 (h−1 h1 k)f2 (xh)π (k)vdhdk.

K H

On the other hand, for x ∈ G and h1 ∈ H , with ρ = indH K π,   G −1 2 (f2 , 1 (f1 , v))(x)(h1 ) = δH (h )f2 (xh)ρ(h)1 (f1 , v)dh (h1 ) 

H

H

 G −1 H −1 δH (h )f2 (xh) δK (k )f1 (h−1 h1 k)π (k)vdkdh.

= 

= H

G −1 δH (h )f2 (xh)1 (f1 , v)(h−1 h1 )dh

K

Combining the last two calculations proves (2.8).

2.8 Tensor products

101

Finally, recall that 1 (Cc (H ), H(π )) is total in H(indH K π ), and hence 2 (Cc (G), 1 (Cc (H ), H(π ))) H is a total subset of H(indG H (indK π)). As F(G, π) ⊇ (Cc (G), H(π )), (2.8) implies that (F(G, π)) contains a total set, and hence is dense in H H(indG H (indK π )). Now, the proof of Theorem 2.47 can quickly be given. By the previous lemmas, H  : F(G, π) → H(indG H (indK π ))

is a linear and isometric mapping with dense range. Therefore,  extends in a unique way to a unitary map, also denoted , of H(indG K π ) onto G H H(indH (indK π )). It remains to show that  intertwines indG K π and H indG (ind π ). H K By definition of  and using that F (G, π) is indG K π -invariant, we have ξ (y −1 x)(h) = H (h)−1/2 G (h)1/2 ξ (y −1 xh) = (indG K π(y)ξ )(x)(h), for all ξ ∈ F (G, π), x, y ∈ G, and h ∈ H . With ρ = indH K π , this implies −1 G (indG H ρ(y)ξ )(x) = ξ (y x) = (indK π (y)ξ )(x).

Since F(G, π) is dense in H(indG K π), we get, for each y ∈ G, G indG H ρ(y) =  indK π (y),

which completes the proof.



We conclude this section with a simple application to the left regular representation, which of course can as well be proved directly. Corollary 2.52 Let H be a closed subgroup of the locally compact group G, and let λH and λG denote the left regular representations of H and G, respectively. Then λG is equivalent to indG H λH . Proof In Theorem 2.47, choose K = {e} and let π = 1, the one-dimensional H identity representation of {e}. Since λG = indG {e} 1 and λH = ind{e} 1, the conclusion follows right away from Theorem 2.47. 

2.8 Tensor products of induced representations Since, after inducing, forming tensor products is the next most important tool to produce new representations, it is quite natural to ask how these two processes

102

Induced representations

are related. Recall that if G1 and G2 are two locally compact groups with unitary representations ρ1 and ρ2 , the outer tensor product ρ1 × ρ2 of G1 × G2 acts on H(ρ1 ) ⊗ H(ρ2 ) by (ρ1 × ρ2 )(x1 , x2 )(ξ1 ⊗ ξ2 ) = ρ1 (x1 )ξ1 ⊗ ρ2 (x2 )ξ2 , for all ξi ∈ H(ρi ) and xi ∈ Gi , i = 1, 2. Our aim in this section is to prove the theorem below, which is usually referred to as Mackey’s tensor product theorem. Theorem 2.53 Let G1 and G2 be locally compact groups with closed subgroups H1 and H2 . Then, for any two representations π1 of H1 and π2 of H2 , the G2 G1 ×G2 1 representations indG H1 π1 × indH2 π2 and indH1 ×H2 (π1 × π2 ) are equivalent. To prove this theorem we have to exhibit a dense linear subspace E of the G2 1 Hilbert space H(indG H1 π1 ) ⊗ H(indH2 π2 ) and an isometric linear mapping  1 ×G from E onto a dense linear subspace of H(indG H1 ×H2 (π1 × π2 )) which intertwines the two representations in question. Gi G1 ×G2 by δi , i = 1, 2, and δH by δ. Note that To that end, we will denote δH i 1 ×H2 δ(t1 , t2 ) = δ1 (t1 )δ2 (t2 ), for (t1 , t2 ) ∈ H1 × H2 . For i = 1, 2, let Fi denote the set of all continuous mappings η : Gi → H(πi ) with the following properties: (a) η has compact support modulo Hi . (b) η(xt) = δi (t)πi (t −1 )η(x), for all x ∈ Gi and t ∈ Hi . i Recall that Fi is a dense linear subspace of H(indG Hi πi ), and therefore the algebraic tensor product F1 ⊗ F2 of F1 and F2 evidently is a dense linear G2 1 subspace of H(indG H1 π1 ) ⊗ H(indH2 π2 ). For (ξ1 , ξ2 ) ∈ F1 × F2 the mapping (y1 , y2 ) → ξ1 (y1 ) ⊗ ξ2 (y2 ) from G1 × G2 into H(π1 ) ⊗ H(π2 ) belongs to 1 ×G2 H(indG H1 ×H2 (π1 × π2 )). Indeed, this mapping is continuous and has compact support modulo H1 × H2 , and for (t1 , t2 ) ∈ H1 × H2 ,

ξ1 (y1 t1 ) ⊗ ξ2 (y2 t2 ) = δ1 (t1 )π1 (t1−1 )ξ1 (y1 ) ⊗ δ2 (t2 )π2 (t2−1 )ξ2 (y2 ) = δ(t1 , t2 )(π1 × π2 ((t1 , t2 )−1 ))(ξ1 (y1 ) ⊗ ξ2 (y2 )). Thus, we can define a linear mapping  from E = F1 ⊗ F2 into the Hilbert 1 ×G2 space H(indG H1 ×H2 (π1 × π2 )) by setting (ξ1 ⊗ ξ2 )(y1 , y2 ) = ξ1 (y1 ) ⊗ ξ2 (y2 ), for ξi ∈ Fi and yi ∈ Gi , i = 1, 2. Theorem 2.53 will be an immediate consequence of the sequence of four lemmas that follows. Lemma 2.54 The map  is inner product preserving.

2.8 Tensor products

103

Proof Let ξ1 , η1 ∈ F1 and ξ2 , η2 ∈ F2 . There are compact subsets K1 of G1 and K2 of G2 such that  ξi vanishes outside Ki Hi , i = 1, 2. For i = 1, 2, choose ϕi ∈ Cc+ (Gi ) so that ϕi (yt)dt = 1, for all y ∈ Ki , and define ϕ on G1 × G2 Hi

by ϕ(y1 , y2 ) = ϕ1 (y1 )ϕ2 (y2 ). Then ϕ ∈ Cc+ (G1 × G2 ) and, for (y1 , y2 ) ∈ K1 × K2 ,    ϕ((y1 , y2 )(t1 , t2 ))d(t1 , t2 ) = ϕ1 (y1 t1 )dt1 · ϕ2 (y2 t2 )dt2 = 1. H1 ×H2

H1

H2

Since (ξ1 ⊗ ξ2 ) vanishes outside K1 H1 × K2 H2 = (K1 × K2 )(H1 × H2 ), we have (ξ1 ⊗ ξ2 ), (η1 ⊗ η2 )  = ϕ(y)(ξ1 ⊗ ξ2 )(y), (η1 ⊗ η2 )(y)dy G1 ×G2

 

ϕ1 (y1 )ϕ2 (y2 )ξ1 (y1 ) ⊗ ξ2 (y2 ), η1 (y1 ) ⊗ η2 (y2 )dy2 dy1 ,

= G1 G2





ϕ1 (y1 )ξ1 (y1 ), η1 (y1 )dy1 ·

= G1

ϕ2 (y2 )ξ2 (y2 ), η2 (y2 )dy2 G2

= ξ1 , η1 ξ2 , η2  = ξ1 ⊗ ξ2 , η1 ⊗ η2 .



G1 G2 1 ×G2 Lemma 2.55  intertwines indG H1 ×H2 (π1 × π2 ) and indH1 π1 × indH2 π2 .

Proof For all (x1 , x2 ), (y1 , y2 ) ∈ G1 × G2 and ξ1 ∈ F1 , ξ2 ∈ F2 , we have   1 ×G2 indG (π × π )(x , x ) ◦ (ξ ⊗ ξ ) 1 2 1 2 1 2 (y1 , y2 ) H1 ×H2 = (ξ1 ⊗ ξ2 )(x1−1 y1 , x2−1 y2 ) = ξ1 (x1−1 y1 ) ⊗ ξ2 (x2−1 y2 ), while on the other hand   G2 1 π × ind π )(x , x )(ξ ⊗ ξ ) (y1 , y2 )  (indG 1 2 1 2 1 2 H1 H1   G2 1 =  indG H1 π1 (x1 )ξ1 ⊗ indH2 π2 (x2 )ξ2 (y1 , y2 )     G2 1 (y (y2 ) π (x )ξ ) ⊗ ind π (x )ξ = indG 1 1 1 1 2 2 2 H1 H2 = ξ1 (x1−1 y1 ) ⊗ ξ2 (x2−1 y2 ).



Lemma 2.56 Let fi ∈ Cc (Gi ) and vi ∈ H(πi ), i = 1, 2. Define f ∈ Cc (G1 × G2 ) by f (x1 , x2 ) = f1 (x1 )f2 (x2 ), for (x1 , x2 ) ∈ G1 × G2 . Then (f, v1 ⊗ v2 )(x1 , x2 ) = (f1 , v1 )(x1 ) ⊗ (f2 , v2 )(x2 ).

104

Induced representations

Proof Fix (x1 , x2 ) ∈ G1 × G2 . For all w1 ∈ H(π1 ) and w2 ∈ H(π2 ), (f, v1 ⊗ v2 )(x1 , x2 ), w1 ⊗ w2    δ1 (t1 )δ2 (t2 )f1 (x1 t1 )f2 (x2 t2 )(π1 (t1 )v1 ⊗ π2 (t2 )v2 )dt2 dt1 , w1 ⊗ w2  =  H1 H2

 

δ1 (t1 )δ2 (t2 )f1 (x1 t1 )f2 (x2 t2 )π1 (t1 )v1 , w1 π2 (t2 )v2 , w2 dt2 dt1

= H1 H 2



=



δ1 (t1 )f1 (x1 t1 )π1 (t1 )v1 , w1 dt1 · H1

δ2 (t2 )f2 (x2 t2 )π2 (t2 )v2 , w2 dt2 H2

  =  δ1 (t1 )f1 (x1 t1 )π1 (t1 )v1 dt1 , w1  ·  δ2 (t2 )f2 (x2 t2 )π2 (t2 )v2 dt2 , w2  H1

H2

= (f1 , v1 )(x1 ), w1 (f2 , v2 )(x2 ), w2  = (f1 , v1 )(x1 ) ⊗ (f2 , v2 )(x2 ), w1 ⊗ w2 . The assertion of the lemma follows since linear combinations of elementary  tensors are dense in H(π1 ) ⊗ H(π2 ). Lemma 2.57 The set of all ((f1 , v1 ) ⊗ (f2 , v2 )), where fi ∈ Cc (Gi ) and 1 ×G2 vi ∈ H(πi ), i = 1, 2, is total in H(indG H1 ×H2 (π1 × π2 )). Proof The set of all (f, v), f ∈ Cc (G1 × G2 ), v ∈ H(π1 × π2 ) = H(π1 ) ⊗ 1 ×G2 H(π2 ), is total in H(indG H1 ×H2 (π1 × π2 )), and the set of elementary tensors is total in H(π1 ) ⊗ H(π2 ). Hence the set of functions (f, v1 ⊗ v2 ), where 1 ×G2 f ∈ Cc (G1 × G2 ) and vi ∈ H(πi ), i = 1, 2, is total in H(indG H1 ×H2 (π1 × π2 )). Thus, it suffices to show that any (f, v1 ⊗ v2 ) can be approximated arbitrarily closely by linear combinations of elements of the form ((g1 , v1 ) ⊗ (g2 , v2 )), for gi ∈ Cc (Gi ), i = 1, 2. To that end, choose open, relatively compact subsets V1 of G1 and V2 of G2 such that supp f ⊆ V1 × V2 . Recall that, by Lemma 2.25, associated with the compact set K = V¯1 × V¯2 of G1 × G2 is a constant c > 0 so that

(g, v) ≤ c v · g ∞ ,

(2.9)

for all v ∈ H(π1 × π2 ) and g ∈ Cc (G1 × G2 ) with supp g ⊆ K. Now, given η > 0, by the Stone–Weierstrass theorem there exist fij ∈ Cc (Gi ), i = 1, 2, j = 1, . . . , n, such that supp fij ⊆ Vi and   n    f (x1 , x2 ) − f1j (x1 )f2j (x2 ) ≤ η,  j =1

2.8 Tensor products

105

for all (x1 , x2 ) ∈ G1 × G2 . Let fj (x1 , x2 ) = f1j (x1 )f2j (x2 ), for (x1 , x2 ) ∈ G1 × G2 and j = 1, . . . , n. From Lemma 2.56 we obtain n 

((f1j , v1 ) ⊗ (f2j , v2 ))(x1 , x2 ) =

j =1

n 

(f1j , v1 )(x1 ) ⊗ (f2j , v2 )(x2 )

j =1

=

n 

(fj , v1 ⊗ v2 )(x1 , x2 ).

j =1

This implies, for every (x1 , x2 ) ∈ G1 × G2 , (f, v1 ⊗ v2 )(x1 , x2 ) −

n 

((f1j , v1 ) ⊗ (f2j , v2 ))(x1 , x2 )

j =1

 

δ1 (t1 )δ2 (t2 )f (x1 t1 , x2 t2 )(π1 (t1 )v1 ⊗ π2 (t2 )v2 )dt2 dt1

= H1 H2

 

−

δ1 (t1 )δ2 (t2 )

n 

fj (x1 t1 , x2 t2 )(π1 (t1 )v1 ⊗ π2 (t2 )v2 )dt2 dt1

j =1

H1 H2

  n  = f − fj , v1 ⊗ v2 (x1 , x2 ). j =1

It follows from this and (2.9) that   n    (f, v1 ⊗ v2 ) − ((f1j , v1 ) ⊗ (f2j , v2 ))   j =1

   n     =  f − fj , v1 ⊗ v2   j =1

  n     fj  ≤ c v1 · v2 · f −  j =1

This finishes the proof of the lemma.

∞

≤ ηc v1 · v2 . 

Clearly, combining the preceding four lemmas establishes Theorem 2.53. We shall now derive from Theorem 2.53 the so-called inner tensor product theorem. Let G be a locally compact group and denote by (G) the diagonal subgroup of G × G. That is, (G) = {(x, x) : x ∈ G}. Via the topological isomorphism (x, x) → x, (G) is always identified with G. If ρ1 and ρ2 are unitary representations of G, then the inner tensor product ρ1 ⊗ ρ2 of G acts on H(ρ1 ) ⊗ H(ρ2 )

106

Induced representations

and is defined by restricting the outer tensor product of ρ1 and ρ2 to (G). Thus, for x ∈ G, (ρ1 ⊗ ρ2 )(x) = (ρ1 × ρ2 )(x, x). Theorem 2.58 Let H be a closed subgroup of the locally compact group G. Then, for arbitrary representations ρ of G and π of H , G ρ ⊗ indG H π = indH (ρ|H ⊗ π ).

Proof Identifying, as before, (G) with G and applying Theorem 2.53 to G1 = G2 = G, H1 = G, H2 = H, π1 = ρ, and π2 = π, we get G G G ρ ⊗ indG H π = (ρ × indH π)|(G) = (indG ρ × indH π )|(G)

= indG×G G×H (ρ × π)|(G) . Therefore, it remains to verify that the two representations indG×G G×H (ρ × π )|(G) G and indH (ρ|H ⊗ π ) are equivalent. For that, let F denote the dense linear subspace of H(indG H (ρ|H ⊗ π)) consisting of all continuous mappings ξ : G → H(ρ|H ⊗ π) = H(ρ) ⊗ H(π ) with compact support modulo H and satisfying ξ (xt) = δ(t)(ρ|H ⊗ π)(t −1 )ξ (x) for all x ∈ G and t ∈ H . For ξ ∈ F , define ξ : G × G → H(ρ) ⊗ H(π ) = H(ρ × π) by ξ (y1 , y2 ) = (ρ(y1−1 y2 ) ⊗ I )ξ (y2 ), for all y1 , y2 ∈ G, where I is the identity operator on H(π ). It is routine to show that ξ is continuous and has compact support modulo G × H . Moreover, for x ∈ G and t ∈ H, ξ ((y1 , y2 )(x, t)) = (ρ(x −1 y1−1 y2 t) ⊗ I )ξ (y2 t) = δ(t)[ρ(x −1 y1−1 y2 ) ⊗ π (t −1 )]ξ (y2 ) = δ(t)[ρ(x −1 ) ⊗ π(t −1 )][ρ(y1−1 y2 ) ⊗ I ]ξ (y2 ) = δ(t)(ρ × π )((x, t)−1 )ξ (y1 , y2 ). This proves ξ ∈ H(indG×G G×H (ρ × π)). In addition, for x ∈ G and ξ ∈ F,   (ρ × π )(x, x)ξ (y1 , y2 ) = ξ (x −1 y1 , x −1 y2 ) indG×G G×H = (ρ(y1−1 y2 ) ⊗ I )ξ (x −1 y2 ) = (ρ(y1−1 y2 ) ⊗ I )(indG H (ρ|H ⊗ π)(x)ξ )(y2 ) = (indG H (ρ|H ⊗ π)(x)ξ )(y1 , y2 ). G To establish the equivalence of indG×G G×H (ρ × π )|(G) and indH (ρ|H ⊗ π) it therefore remains to show that (F) contains a total subset of H(indG×G G×H (ρ ×

2.9 Frobenius reciprocity

107

π)). Let v ∈ H(ρ), w ∈ H(π), and g ∈ Cc (G), and define ξ ∈ F by ξ (y) = ρ(y −1 )v ⊗ (g, w)(y). Then, by the definition of , for all y1 , y2 ∈ G, ξ (y1 , y2 ) = ρ(y1−1 )v ⊗ (g, w)(y2 ). Now, obviously the set of all such mappings ξ is total in H(indG×G G×H (ρ × π )). This completes the proof of the theorem.  It is possible to give a more direct proof of Theorem 2.58 which avoids the use of Theorem 2.53. However, Theorem 2.53 is valuable in its own right. Theorem 2.58, in case H = {e} and π is the trivial representation of H , so that indG H π is the left regular representation of G, yields the following. Corollary 2.59 For any representation ρ of G, ρ ⊗ λG = (dim ρ) · λG .

2.9 Frobenius reciprocity Consider a closed subgroup H of a locally compact group G and two representations σ of G and π of H , respectively. The question of whether or not σ is a subrepresentation of indG H π often arises. Alternatively, one could know that σ is a subrepresentation of indG H π and be concerned about the implied relationship between π and σ |H . For finite groups these questions are answered completely by the following Frobenius reciprocity theorem. Theorem 2.60 (Frobenius reciprocity for finite groups) Let H be a subgroup of a finite group G, and let π be a representation of H and σ a representation of G. Then the spaces HomG (indG H π, σ ) and HomH (π, σ |H ) are isomorphic. In particular, if both π and σ are irreducible, from the equality of the dimensions of HomG (indG H π, σ ) and HomH (π, σ |H ) we conclude that σ occurs in indG H π with the same multiplicity that π occurs in σ |H . In this section, we prove a generalization of Theorem 2.60. In doing so, we cannot simply replace “Let H be a subgroup of a finite group G” with “Let H be a closed subgroup of a locally compact group G”. The reader should check the simple example consisting of G = R, H = {0}, and σ and π the respective trivial representations. There are a number of successful ways in which the statement of Theorem 2.60 has been modified.

108

Induced representations

The following version of Frobenius reciprocity can be established at this stage and is particularly useful. Theorem 2.61 Let H be a closed subgroup of a locally compact group G such that G/H has a finite invariant measure. Let π be a representation of H and let σ be a finite-dimensional representation of G. Then the spaces HomG (indG H π, σ ) and HomH (π, σ |H ) are isomorphic. The proof of this theorem begins with a series of lemmas. Lemma 2.62 Let N be a group and let ρ1 , ρ2 , and τ be representations of N with τ finite-dimensional. Then the vector spaces HomN (ρ1 ⊗ τ, ρ2 ) and HomN (ρ1 , ρ2 ⊗ τ ) are isomorphic. Proof We first construct a canonical isomorphism  : B(Hρ1 ⊗ Hτ , Hρ2 ) → B(Hρ1 , Hρ2 ⊗ Hτ∗ ). Fix an orthonormal basis {v1 , . . . , vd } of Hτ , and let {v1∗ , . . . , vd∗ } denote the dual basis of Hτ∗ .

Any element of Hρ1 ⊗ Hτ can be expressed in the form dj=1 ξj ⊗ vj with ξj ∈ Hρ1 , 1 ≤ j ≤ d, and orthonormality of {v1 , . . . , vd } shows that the ξj are uniquely determined in such an expression. Likewise, any element of Hρ2 ⊗ Hτ∗

has a unique expression of the form dj=1 ηj ⊗ vj∗ with η1 , . . . , ηd ∈ Hρ2 . Now define  as follows. For S ∈ B(Hρ1 ⊗ Hτ , Hρ2 ), let (S)ξ =

d 

S(ξ ⊗ vj ) ⊗ vj∗ ,

j =1

for all ξ ∈ Hρ1 . One sees immediately that (S) ∈ B(Hρ1 , Hρ2 ⊗ Hτ∗ ) and that the map S → (S) is bounded and linear. Injectivity of  follows from the fact that, for any S ∈ B(Hρ1 ⊗ Hτ , Hρ2 ) and ξ ∈ Hρ1 , (S)ξ = 0 implies S(ξ ⊗ vj ) = 0 for 1 ≤ j ≤ d. To see that  is onto, let T ∈ B(Hρ1 , Hρ2 ⊗ Hτ∗ ) be given. For any ξ ∈ Hρ1 , T ξ ∈ Hρ2 ⊗ Hτ∗ , so there exist well-defined elements η1 (ξ ), . . . , ηd (ξ ) ∈

Hρ2 such that T ξ = dj=1 ηj (ξ ) ⊗ vj∗ . Each ηj (ξ ) depends linearly on ξ . Define S : Hρ1 ⊗ Hτ → Hρ2 by S(ξ ⊗ vj ) = ηj (ξ ), 1 ≤ j ≤ d, and extend S to all of Hρ1 ⊗ Hτ by additivity. One easily checks that S ∈ B(Hρ1 ⊗ Hτ , Hρ2 ) and (S) = T . It remains to show that  maps HomN (ρ1 ⊗ τ, ρ2 ) onto HomN (ρ1 , ρ2 ⊗ τ ). First, it is easy to check that the definition of  does not depend on the choice of the orthonormal basis of Hτ . Thus, if n ∈ N and S ∈ HomN (ρ1 ⊗ τ, ρ2 ), then since {τ (n)v1 , . . . , τ (n)vd } is an orthonormal basis of Hτ with dual basis

2.9 Frobenius reciprocity

109

{τ (n)v1∗ , . . . , τ (n)vd∗ }, for any ξ ∈ Hρ1 , (S)ρ1 (n)ξ =

d 

S(ρ1 (n)ξ ⊗ τ (n)vj ) ⊗ τ (n)vj∗

j =1

=

d  !

" S(ρ1 (n) ⊗ τ (n))(ξ ⊗ vj ) ⊗ τ (n)vj∗

j =1

=

d 

ρ2 (n)S(ξ ⊗ vj ) ⊗ τ (n)vj∗

j =1

⎤ ⎡ d  = (ρ2 ⊗ τ )(n) ⎣ S(ξ ⊗ vj ) ⊗ vj∗ ⎦ j =1

= (ρ2 ⊗ τ )(n)(S)ξ. Thus (S) ∈ HomN (ρ1 , ρ2 ⊗ τ ). On the other hand, let T ∈ HomN (ρ1 , ρ2 ⊗ τ ) and let n ∈ N. Let S ∈ B(Hρ1 ⊗ Hτ , Hρ2 ) be such that (S) = T and let ηj (ξ ) = S(ξ ⊗ vj ), 1 ≤ j ≤ d, ξ ∈ Hρ1 . Let (ukl )dk,l=1 be the unitary matrix that represents τ (n) with respect to the basis {v1 , . . . , vd }. Then (ulk )dk,l=1 represents τ (n) with respect to {v1∗ , . . . , vd∗ }. For ξ ∈ Hρ1 and 1 ≤ j ≤ d, S(ρ1 ⊗ τ )(n)(ξ ⊗ vj ) = S[ρ1 (n)ξ ⊗ τ (n)vj ] =

d 

ukj S[ρ1 (n)ξ ⊗ vk ]

k=1

=

d 

ukj ηk (ρ1 (n)ξ ).

k=1

On the other hand, we have d 

ηj (ρ1 (n)ξ ) ⊗ vj∗ = T ρ1 (n)ξ = (ρ2 ⊗ τ )(n)T ξ

j =1

= (ρ2 ⊗ τ )(n)

d 

ηl (ξ ) ⊗ vj∗

l=1

=

d 

ρ2 (n)ηl (ξ ) ⊗ τ (n)vj∗ +

l=1

= =

d  l=1 d  k=1

ρ2 (n)ηl (ξ ) ⊗ ρ2 (n)

+ d  l=1

d  k=1

, ulk vk∗ ,

ulk ηl (ξ ) ⊗ vk∗ .

110

Induced representations

By uniqueness of the representations in Hρ2 ⊗ Hτ∗ , we conclude that ηj (ρ1 (n)ξ ) =

n 

ρ2 (n)ulj ηl (ξ ).

l=1

Returning to the calculation of S(ρ1 ⊗ τ )(n)(ξ ⊗ vj ), we have S(ρ1 ⊗ τ )(n)(ξ ⊗ vj ) =

d d  

ukj ulk ρ2 (n)ηl (ξ )

l=1 k=1

= ρ2 (n)ηj (ξ ) = ρ2 (n)S(ξ ⊗ vj ), for 1 ≤ j ≤ d and any ξ ∈ Hρ1 . Thus S ∈ HomN (ρ1 ⊗ τ, ρ2 ). This completes the proof of Lemma 2.62.  It is now convenient to fix a finite invariant measure μ on G/H so that Weil’s formula takes the form    ˙ f (x)dx = f (xh)dhdμ(x) G

G/H

H

π. For ξ, η ∈ F (G, π) their inner product is and use the  notation U for ˙ ξ, η = G/H ξ (x), η(x)dμ(x). Let ρ be an arbitrary representation of G. The next two lemmas show that HomH (π, ρ|H ) can be embedded in HomG (U π , ρ). indG H

π

Lemma 2.63 For S ∈ HomH (π, ρ|H ), define S : F (G, π) → H(ρ) by  ˙ ρ(x)Sξ (x)dμ(x). S(ξ ) = G/H

Then S(ξ ) ≤ S μ(G/H ) ξ , for all ξ ∈ F (G, π) and hence S extends to a bounded linear operator S : H(U π ) → H(ρ). Moreover, S ∈ HomG (U π , ρ). 1/2

Proof For ξ ∈ F(G, π), the map x → ρ(x)Sξ (x) is a bounded continuous function from G into H(ρ) that is constant on H -cosets since, for x ∈ G and h ∈ H, ρ(xh)Sξ (xh) = ρ(x)ρ(h)Sπ (h−1 )ξ (x) = ρ(x)Sξ (x).  Thus S(ξ ) = G/H ρ(x)Sξ (x)dμ(x) is well defined and linear in ξ . Using H¨older’s inequality, we get  

S(ξ ) ≤ ˙ ≤ S ˙

ρ(x)Sξ (x) dμ(x)

ξ (x) dμ(x) G/H

1/2

 ˙

ξ (x) 2 dμ(x)

= S G/H

= S μ(G/H )1/2 ξ .

G/H

μ(G/H )1/2

2.9 Frobenius reciprocity

111

Thus S extends uniquely to all of H(U π ) with S ≤ μ(G/H )1/2 S . Finally, for y ∈ G and ξ ∈ F(G, π),   ˙ = ˙ ρ(y)S(ξ ) = ρ(y)ρ(x)Sξ (x)dμ(x) ρ(x)Sξ (y −1 x)dμ(x) G/H G/H  ˙ = S(U π (y)ξ ). = ρ(x)SU π (y)ξ (x)dμ(x) G/H

This shows that S ∈ HomG (U π , ρ).



Lemma 2.64 The linear map  : HomH (π, ρ|H ) → HomG (U π , ρ) is injective. Proof Let S ∈ HomH (π, ρ|H ) be such that S = 0. Suppose, to the contrary, that S = 0. Let v ∈ H(π) be such that Sv = 0. The map y → ρ(y)Sv is a continuous function  from G into H(ρ) that is not identically zero. Let f ∈ Cc (G) such that G f (y)ρ(y)Svdy = 0. Then  ˙ ρ(x)S(f, v)(x)dμ(x) S((f, v)) = G/H    ˙ = ρ(x)S f (xh)π(h)vdh dμ(x) G/H H    ˙ = ρ(x) f (xh)ρ(h)Svdh dμ(x) G/H H   ˙ = f (xh)ρ(xh)Sv dh dμ(x) G/H H  f (y)ρ(y)Sv dy = 0. = G

This contradicts S = 0 and shows that  is injective.



We are now ready for the proof of Theorem 2.61. The proof begins by showing that there is no loss of generality in assuming that the finite dimensional representation σ is the trivial representation 1G of G. Proof of Theorem 2.61 Two applications of Lemma 2.62 show that HomG (U π , σ ) is isomorphic to HomG (U π ⊗ σ¯ , 1G ) and HomH (π, σ|H ) is isomorphic to HomH (π ⊗ σ¯ |H , 1H ). On the other hand, σ |H = σ¯ |H and U π ⊗ σ¯ is equivalent to U (π⊗σ¯ |H ) by Theorem 2.58. Thus, replacing π with π ⊗ σ¯ |H , it suffices to prove that HomG (U π , 1G ) is isomorphic to HomH (π, 1H ). Let  : HomH (π, 1H ) → HomG (U π , 1G ) be the bounded linear map defined in Lemma 2.63. By Lemma 2.64, we only need to show that  is onto. Let T ∈ HomG (U π , 1G ) with T = 0. Fix a unit vector w ∈ H(1G ), so H(1G ) = Cw. Let ξT = T ∗ w. Then ξT = 0 and T ∗ (H(1G )) = CξT . Now ξT⊥ is the null

112

Induced representations

space of T and, as such, is U π -invariant. Thus CξT is U π -invariant. So, for any  x ∈ G, U π (x)ξT = α(x)ξT , for some α(x) ∈ T. This means that, as a measurable function on G, for every x ∈ G, ξT (xy) = ξT (y), for almost all y ∈ G. Therefore,     ˙ = ˙

ξT (xy) − ξT (y) dxdμ(y)

ξT (xy) − ξT (y) dμ(y)dx G/H

G

G

G/H

= 0.

 Thus, G ξT (xy) − ξT (y) dx = 0, for all y in an H -saturated measurable subset M of G with |G \ M| = 0. It follows that, for each y ∈ M, there is a measurable subset My of G with |G \ My | = 0 and ξT (xy) = ξT (y), for all x ∈ My . We may assume that Myh = My , for each y ∈ M and h ∈ H . Fix y ∈ M and set vT = ξT (y) ∈ H(π ). Let h ∈ H and select x ∈ Myh−1 ∩ (My (yhy −1 )). Then xyh−1 y −1 ∈ My and hence π(h)vT = π(h)ξT (y) = ξT (yh−1 ) = ξT (xyh−1 ) = ξT (xyh−1 y −1 y) = ξT (y) = vT . Thus, π(h)vT = vT for all h ∈ H . Identify H(1G ) with C and define a linear map S : H(π ) → H(1H ) as follows. For v ∈ H(π ), let Sv = v, vT . Then, for any h ∈ H , Sπ (h)v = π(h)v, vT  = v, π (h−1 )vT  = v, vT  = Sv. Thus S ∈ HomH (π, 1H ). In this situation, S takes the form   ˙ = Sξ (x)dμ(x) S(ξ ) = G/H

˙ ξ (x), vT wdμ(x),

G/H

for all ξ ∈ F(G, π). Recalling that vT = ξT (x), for almost all x ∈ G, and ξT = T ∗ w, we obtain  ˙ = ξ, ξT  = T ξ. ξ (x), ξT (x)dμ(x) S(ξ ) = G/H

This shows that  is a linear isomorphism of HomH (π, 1H ) onto HomG (U π , 1G ).

2.10 Notes and references Induced representations for finite groups were introduced in 1898 by Frobenius [56], and he also proved the result which has later been referred to as the Frobenius reciprocity theorem. The theory of induced unitary representations for locally compact groups was to a large extent developed by Mackey in a

2.10 Notes and references

113

series of papers in the 1950s [99]–[101]. Mackey confined himself to second countable groups G and separable Hilbert spaces H. In this situation, Mackey already established most of the basic calculation rules for the inducing process, such as the result on inducing conjugated representations (Proposition 2.39), the theorem on inducing in stages (Theorem 2.47), and the inner and outer tensor product theorems (Theorems 2.53 and 2.58). For arbitrary G and H, induced representations were first defined by Blattner [22]. Our exposition in Section 2.3 follows his method. Blattner also proved the theorem on induction in stages in full generality, and, which explores the connection between induced representations and positive definite measures, is due to him [26]. It is worth pointing out that there is a much more general concept of inducing representations, the Rieffel inducing process, which is formulated in the context of C ∗ -algebras [133]. However, this concept does not meet the intention of accessibility of this book and is beyond its scope. The question of when a representation, that is induced from an open subgroup, is irreducible has been studied by many authors (compare [20], [21], [31], [99], and [119]), and there are several results of the nature of Theorem 2.14. They all require the representation of the subgroup to be finite-dimensional. Lemma 2.16 and Example 2.17, which demonstrate that this hypothesis is essential, are taken from Bekka and Curtis [17]. See Quigg [127] and [128] for results on irreducibility of induced representations under different hypotheses. Our brief treatment of the principle series of SL(2, R) is a meager introduction to a vast theory of harmonic analysis on semisimple Lie groups. The core of this theory was developed by Harish-Chandra and is outside the scope of this book. Our presentation of the irreducibility of the principle series representations follows that of Knapp [93], except for our use of the knowledge of the irreducible representation of the affine group to slightly shorten the argument. The extension to the general situation of Mackey’s tensor product theorem and the result on inducing conjugated representations are due to Fakler [40], [42]. Our presentation is based on his papers. Besides Blattner’s, there are several other approaches to forming induced representations utilizing quasiinvariant measures on coset spaces and Borel cross-sections. We have presented those that turn out to be useful later in the book. All of these approaches lead to equivalent realizations of induced representations. The Frobenius reciprocity theorem for finite groups was extended to compact groups by Weil [150]. Mautner [108] proved a version of the Frobenius reciprocity theorem for noncompact groups, where the subgroup is supposed to be compact, and Mackey [102] established a Frobenius reciprocity theorem involving direct integral decompositions of representations. Both of these results are very technical. In Section 2.9 we have adapted a reciprocity theorem from Moore [110].

3 The imprimitivity theorem

The inducing construction in Chapter 2 gives one the power to create unitary representations of a group G when representations of a closed subgroup H are given. One can also find conditions under which the induced representation is irreducible. However, a key question is often the converse: Is every irreducible representation of G equivalent to one induced from a proper subgroup? The imprimitivity theorem presented in this chapter is an invaluable tool in answering this question for many groups. The full definition and some basic properties of systems of imprimitivity are introduced in Section 3.1. An induced representation is part of what is called, in Section 3.2, an induced system of imprimitivity. In Section 3.2, we show that if two representations are induced from some subgroup, then the intertwining space of these representations can be identified with the intertwining space for the corresponding systems of imprimitivity. In Section 3.3, we state the imprimitivity theorem and provide a proof in the special case when the system of imprimitivity is living over a discrete coset space (that is, the corresponding subgroup is open). This prepares the way to understanding the general proof, which is presented in Section 3.4.

3.1 Systems of imprimitivity We start with the general notion of a system of imprimitivity. Definition 3.1 Let G be a locally compact group and let be a locally compact left topological G-space. A system of imprimitivity for G over is a pair (σ, P ) where σ is a unitary representation of G on a Hilbert space H and P is a ∗-homomorphism of C0 ( ) into B(H) such that 114

3.1 Systems of imprimitivity

115

(1) P [C0 ( )]H is dense in H. (2) σ (x)P (ϕ)σ (x −1 ) = P (Lx ϕ) for all ϕ ∈ C0 ( ) and x ∈ G. Note that C0 ( ) is a commutative C ∗ -algebra and by (1), P is a nondegenerate ∗-representation of C0 ( ) on H. Thus, P carries information about

, while σ carries information about G, and condition (2) means that σ and P together carry information about the action of G on . Moreover, notice that

P (ϕ) ≤ ϕ ∞ for all ϕ ∈ C0 ( ). Since the concept of a system of imprimitivity is so vital to understanding the representation theory of many locally compact groups, we shall study it in some detail beginning with two classes of systems; first, those over finite left G-spaces, followed by those arising from restricting a representation to an abelian closed normal subgroup. Example 3.2 Assume that is a finite left G-space and that (σ, P ) is a system of imprimitivity for G over . Define Pω = P (δω ), for each ω ∈ , where δω denotes the one-point mass at ω. Then

(a) P (ϕ) = ϕ(ω)Pω , for all ϕ ∈ C( ). ω∈

(b) Pω is a projection operator on H, for all ω ∈ . (c) Pω Pω = 0, for ω, ω ∈ , ω = ω .

P (ω) = P (1) = I , the identity operator on H. (d) ω∈

The fact that P (1) = I results from the nondegeneracy condition since 1 is the identity of C( ). Now, for x ∈ G and ω, ω ∈ , Lx δω (ω ) = δω (x −1 · ω ) = δx·ω (ω ). Thus, (2) above becomes (e) σ (x)Pω σ (x −1 ) = Px·ω , for all x ∈ G and ω ∈ . Condition (a) simply tells us that P is completely determined by the map ω → Pω of into the projection operators on H. There is one more step of simplification that can be made here. For ω ∈ , let Hω = Pω H. Then each Hω is a closed subspace of H and (b), (c), and (d) imply that H = ⊕ω∈ Hω , while (e) says that σ (x)Hω = Hx·ω , for all x ∈ G and ω ∈ . This reduces us to the usual definition of a system of imprimitivity for a finite group G as a G-module H and a decomposition of H into a finite direct sum of subspaces which are permuted among each other by G. Example 3.3 For the next example of a naturally arising system of imprimitivity let G be a locally compact group with an abelian closed normal subgroup N . Let σ be a representation of G. Consider the restriction σ |N of σ to N and consider the associated ∗-representation of C ∗ (N).

116

The imprimitivity theorem

Since the Fourier transform extends from L1 (N ) to a C ∗ -isomorphism of ) (Proposition 1.79 and the paragraph following that propoC (N ) with C0 (N sition), there is a ∗-representation P of C ∗ (N) associated with σ |N so that, for all f ∈ L1 (N ),  P (f)ξ = f (n)σ (n)ξ dn, ∗

N

for all ξ ∈ H(σ ).  can be considered as a locally compact left G-space by x · χ (n) = Now N  Thus we have a left G-space N , χ (x −1 nx), for all n ∈ N, x ∈ G, and χ ∈ N.  a representation σ of G on H(σ ), and a ∗-representation P of C0 (N ) on H(σ ). Proposition 3.4 With the above notation, (σ, P ) is a system of imprimitivity  for G over N. Proof It is easy to see that condition (1) in Definition 3.1 is fulfilled. Indeed, given ξ ∈ H(σ ) and ε > 0, there exists a neighborhood V of e in N such that

σ (n)ξ − ξ ≤ , for all n ∈ V , and with f = |V |−1 1V it follows that 

P (f)ξ − ξ = σ |N (f )ξ − ξ = |V |−1 σ (n)ξ − ξ dn ≤ . V

 and It remains to show that σ (x)P (ϕ)σ (x ) = P (Lx ϕ) for all ϕ ∈ C0 (N) 1  it suffices x ∈ G. By continuity and the density of {f : f ∈ L (N )} in C0 (N), to prove this for ϕ = f, for any f ∈ L1 (N). For x ∈ G, there is a δ(x) > 0 so that   −1 f (x nx)dn = δ(x) f (n)dn, −1

N

N

for all f ∈ L (N ). For x ∈ G and f ∈ L (N), define f x ∈ L1 (N ) by f x (n) = δ(x)−1 f (x −1 nx). A simple calculation shows that fx = Lx f. Thus, for ξ, η ∈ H(σ ), 1

1

σ (x)P (f)σ (x −1 )ξ, η = P (f)σ (x −1 )ξ, σ (x −1 )η = σ |N (f )σ (x −1 )ξ, σ (x −1 )η  = f (n)σ (xnx −1 )ξ, ηdn N

= δ(x)−1 



f (x −1 nx)σ (n)ξ, ηdn

N

f (n)σ (n)ξ, ηdn = σ |N (f x )ξ, η

=

x

N

x )ξ, η = P (Lx f)ξ, η. = P (f

3.1 Systems of imprimitivity

117

 and x ∈ G, and Therefore, σ (x)P (ϕ)σ (x −1 ) = P (Lx ϕ), for all ϕ ∈ C0 (N)  (σ, P ) is a system of imprimitivity for G over N.   are quite Since ∗-representations of commutative C ∗ -algebras such as C0 (N)  well understood and the left G-space N is a fairly concrete object, it is natural to ask how much information about σ is carried by these two objects. When N is very large, much of the nature of σ can be reconstructed from the action of  and P . For example, if N = G, the action of G on N  is trivial and σ is G on N completely determined by P . On the other hand, if N is too small, for example, N = {e}, then P contains no information. In Section 3.3 the reader will find an almost complete answer to this question. In fact, Theorem 3.17 provides a very powerful tool for describing the irreducible representations of groups which have a substantial abelian closed normal subgroup. Remark 3.5 In this paragraph we shall outline how the concept of a system of imprimitivity can be formulated in terms of a projection-valued measure. This was already essentially carried out in Example 3.2 for systems of imprimitivity over a finite G-space. If X is a set, A is a σ -algebra of subsets of X, and H is a Hilbert space, then an H-projection-valued measure on A is a map E : A → E(A) from A into the set of projection operators on H satisfying the following conditions: (1) E(X) = I , the identity operator on H. (2) For any sequence An , n ∈ N, of pairwise disjoint sets in A, % $∞ ∞  An = E(An ). E n=1

n=1

The convergence on the right-hand side of the equation in (2) is in the strong operator topology. If ξ, η ∈ H, then (1) and (2) imply that A → E(A)ξ, η is a complex measure on (X, A). Now, suppose is a locally compact Hausdorff space and let B denote the σ -algebra of Borel subsets of . An H-projection-valued measure E on B is called regular if, for any A ∈ B, E(A) = sup{E(C) : C ⊆ A, C compact}. The following is a common generalization of the spectral theorem for normal operators and the Riesz representation theorem for continuous linear functionals on C0 ( ) [79]. Let P be a nondegenerate ∗-representation of C0 ( ) on a Hilbert space H. Then there exists a unique regular H-projection-valued measure E on B such

118

The imprimitivity theorem

that, for every ϕ ∈ C0 ( ), P (ϕ) satisfies  P (ϕ)ξ, η = ϕ(ω)dE(ω)ξ, η,

(3.1)



for all ξ, η ∈ H. In addition, suppose G is a locally compact group acting on so that is a left topological G-space and σ is a representation of G on H so that (σ, P ) is a system of imprimitivity for G over . Let E be the H-projection-valued measure on B so that (3.1) holds. For each x ∈ G, define two further maps x E and E x from B into the projections on H by x E(A)

= E(x · A) and E x (A) = σ (x)E(A)σ (x −1 ),

(3.2)

for all A ∈ B. It is easily verified that both x E and E x are regular H-projectionvalued measures on . Proposition 3.6 If (σ, P ) is a system of imprimitivity for G based on and E is the H-projection-valued measure such that (3.1) holds, then σ (x)E(A)σ (x −1 ) = E(x · A),

(3.3)

for all x ∈ G and A ∈ B. Proof Let x ∈ G and let x E and E x be defined as in (3.2). Let P˜ denote the ∗-representation of C0 ( ) on H given by P˜ (ϕ) = σ (x)P (ϕ)σ (x −1 ), for all ϕ ∈ C0 ( ). For ξ, η ∈ H, we have on the one hand P˜ (ϕ)ξ, η = P (ϕ)σ (x −1 )ξ, σ (x −1 )η  = ϕ(ω)dE(ω)σ (x −1 )ξ, σ (x −1 )η

 =

ϕ(ω)dE x (ω), ξ, η.

(3.4)



On the other hand, P˜ (ϕ)ξ, η = σ (x)P (ϕ)σ (x −1 )ξ, η = P (Lx ϕ)ξ, η   = Lx ϕ(ω)dE(ω)ξ, η = ϕ(x −1 · ω)dE(ω)ξ, η





=

ϕ(ω)dx E(ω)ξ, η.

(3.5)

3.1 Systems of imprimitivity

119

The uniqueness of a spectral decomposition such as in (3.1) applied to the ∗-representation P˜ and (3.4) and (3.5) yield x E = E x . That is, (3.3) holds for all A ∈ B and every x ∈ G.  As a consequence of Proposition 3.6, the concept of a system of imprimitivity can equivalently be defined as a pair (σ, E), where σ is a unitary representation of G on H and E is an H-projection-valued measure on the left G-space

so that (3.3) holds. In this book, we have chosen to emphasize the role of continuous functions as much as possible and, therefore, use Definition 3.1 in order to keep the measure theory to a minimum. Let and  be left G-spaces. Recall that a map  : →  is said to be G-equivariant if (x · ω) = x · (ω) for all ω ∈ and x ∈ G. If and  are both topological G-spaces and  is a homeomorphism of onto  which is G-equivariant, we will call  a G-homeomorphism. With this concept, we are able to introduce the notion of equivalent systems of imprimitivity. Definition 3.7 Let (σ1 , P1 ) and (σ2 , P2 ) be systems of imprimitivity for G over the locally compact left topological G-spaces 1 and 2 , respectively. Suppose there exist a unitary operator U : H(σ1 ) → H(σ2 ) and a G-homeomorphism  : 2 → 1 with the following properties: (1) U σ1 (x) = σ2 (x)U for all x ∈ G. (2) U P1 (ϕ) = P2 (∗ ϕ)U for all ϕ ∈ C0 ( 1 ), where ∗ : C0 ( 1 ) → C0 ( 2 ) is defined by ∗ ϕ = ϕ ◦ . Then (σ1 , P1 ) and (σ2 , P2 ) are called equivalent systems of imprimitivity. If, in this definition, 1 = 2 = and  is the identity map of , then we say that (σ1 , P1 ) and (σ2 , P2 ) are equivalent over . Remark 3.8 An almost trivial version of the equivalence of systems of imprimitivity is useful in studying transitive systems of imprimitivity. We remind the reader that a left G-space is transitive if for one, and hence all, ω ∈ we have G(ω) = . Suppose is a transitive locally compact left G-space and fix a base point ω0 ∈ . Let H = {x ∈ G : x · ω0 = ω0 }, the stabilizer of ω0 . Then H is a closed subgroup of G, and G/H is also a transitive locally compact left G-space. Define  : G/H → by (xH ) = x · ω0 , for xH ∈ G/H . Since H stabilizes ω0 ,  is well defined. It is also easy to see that  is G-equivariant and a continuous bijection between G/H and . It is not true that  is always a homeomorphism. Consider the example of Rd , the real numbers with the discrete topology, acting on (0, ∞), with its usual topology, by t · ω = et ω, for t ∈ Rd and ω ∈ (0, ∞). However, if G and are both second countable, then  is a homeomorphism. This follows from a now classical result of Glimm [60].

120

The imprimitivity theorem

Proposition 3.9 Let G be a locally compact group and (σ, P ) a system of imprimitivity for G over a transitive locally compact left G-space . Fix ω0 ∈ and let H = {x ∈ G : x · ω0 = ω0 }. Assume that  : xH → x · ω0 is a homeomorphism, and let P  = P ◦ ∗ . Then (σ, P  ) is a system of imprimitivity for G over G/H , and (σ, P  ) is equivalent to (σ, P ). Proof The equivalence is instituted by the identity operator on H(σ ) and the homeomorphism . 

3.2 Induced systems of imprimitivity Some of the most important systems of imprimitivity arise from the inducing process. We shall define the system of imprimitivity associated with a representation of a closed subgroup and develop some basic properties. Example 3.10 Let G be a locally compact group, H a closed subgroup of G, and π a representation of H on H(π). For simplicity, let U π = indG H π and H = H(indG π ). Consider the locally compact left topological G-space G/H H and, for any ϕ ∈ C0 (G/H ), define P π (ϕ) on H by [P π (ϕ)ξ ](y) = ϕ(yH )ξ (y), for all ξ ∈ H and locally almost all y ∈ G. It is routine to verify that P π (ϕ)ξ ∈ H for every ξ ∈ H and that P π (ϕ) is a linear operator with P π (ϕ) ∞ ≤ ϕ ∞ . Then P π : ϕ → P π (ϕ) is a ∗-representation of C0 (G/H ). For each ξ ∈ H and  > 0, let η ∈ F(G, π) such that ξ − η < . Let C = q(supp η), and let f ∈ Cc (G) so that f # (xH ) = 1, for all xH ∈ C. Then f # ∈ Cc (G/H ) and P π (f # )(η) = η. Thus, P π (Cc (G/H ))H is dense in H. Now, we show that U π (x)P π (ϕ)U π (x −1 ) = P π (Lx ϕ), for every x ∈ G and ϕ ∈ C0 (G/H ). In fact, for any ξ ∈ H and locally almost all y ∈ G, [U π (x)P π (ϕ)U π (x −1 )]ξ (y) = [P π (ϕ)U π (x −1 )]ξ (x −1 y) = ϕ(x −1 yH )[U π (x −1 )ξ ](x −1 y) = Lx ϕ(yH )ξ (y) = [P π (Lx ϕ)ξ ](y). Thus, (U π , P π ) is a system of imprimitivity of G over G/H . It is called the system of imprimitivity induced by π. The following proposition provides a formula for P π (f # ), if f ∈ Cc (G), that will be convenient later in this section and in Section 3.4. Proposition 3.11 Let π be a representation of a closed subgroup H of a locally compact group G, and let P π be as defined in Example 3.10. Then, for

3.2 Induced systems of imprimitivity

121

all f ∈ Cc (G) and ξ, η ∈ H(indG H π),  π # P (f )ξ, η = f (x)ξ (x), η(x)dx. G

Proof Without loss of generality, assume that ξ, η ∈ F (G, π). Fix f ∈ Cc (G) and note that, for any ϕ ∈ Cc (G/H ) such that supp ϕ ∩ supp f # = ∅, μP π (f # )ξ,η (ϕ) = 0. It follows that, for any ξ, η ∈ H = H(indG H π ), supp μP π (f # )ξ,η ⊆ supp f # .

(3.6)

Let g ∈ Cc (G) be such that g # (xH ) = 1 for all xH ∈ supp f # . Then, by (3.6), P π (f # )ξ, η = μP π (f # )ξ,η (G/H )  = g # (xH )dμP π (f # )ξ,η (xH ) G/H



=

g(x)P π (f # )ξ (x), η(x)dx G



=

g(x)f # (xH )ξ (x), η(x)dx.

(3.7)

G

Recall that ξ ∈ F (G, π) implies that ξ (xt) = δ(t)π (t −1 )ξ (x) and, hence, ξ (xt), η(xt) = ξ (x), η(x)δ(t)2 , for all t ∈ H, x ∈ G. Now, let ρ be a strictly positive continuous rho-function on G associated with H , and let μ be the corresponding quasi-invariant measure on G/H . By Weil’s formula (1.4), (3.7) becomes   π # f # (xH )g(xt)ξ (xt), η(xt)ρ(xt)−1 dt dμ(xH ) P (f )ξ, η = G/H H

 

=

f # (xH )g(xt)ξ (xt), η(xt)ρ(x)−1 δ(t)−2 dt dμ(xH )

G/H H

 

= G/H H

 = G/H

f # (xH )ξ (x), η(x)ρ(x)−1 dt dμ(xH ) ⎛

f # (xH )ρ(x)−1 ξ (x), η(x) ⎝



H

⎞ g(xt)dt ⎠ dμ(xH )

122

The imprimitivity theorem 

f # (xH )ρ(x)−1 ξ (x), η(x)dμ(xH )

= G/H



⎛ ρ(x)−1 ξ (x), η(x) ⎝

= G/H

 

=



⎞ f (xt)dt ⎠ dμ(xH )

H

f (xt)ξ (xt), η(xt)ρ(xt)−1 dt dμ(xH )

G/H H



=

f (x)ξ (x), η(x)dx.



G

Corollary 3.12 With the notation as in Proposition 3.11, for any f ∈ Cc (G) and ξ ∈ H(U π ),  π # 2

P (f )ξ = f (x)f # (xH ) ξ (x) 2 dx. G

Proof The formula of Proposition 3.11 yields   f (x)f # (xH ) ξ (x) 2 dx = f (x)ξ (x), f # (xH )ξ (x)dx G

G



=

f (x)ξ (x), (P π (f # )ξ )(x)dx G

= P π (f # )ξ, P π (f # )ξ  = P π (f # )ξ 2 .



It will be very important to be able to identify when two induced systems of imprimitivity from the same subgroup H are equivalent over G/H in the sense of Definition 3.7. Note that a unitary operator which institutes such an equivalence is an intertwining operator between the two representations of G as well as an intertwining operator between the two ∗-representations of C0 (G/H ) that make up the respective systems of imprimitivity. This motivates the consideration of spaces of intertwining operators for systems of imprimitivity. Definition 3.13 Let (σ1 , P1 ) and (σ2 , P2 ) be systems of imprimitivity for G over the same locally compact left G-space . Let HomG [(σ1 , P1 ), (σ2 , P2 )] denote the set of all T ∈ B(H(σ1 ), H(σ2 )) such that T σ1 (x) = σ2 (x)T , for all x ∈ G, and T P1 (ϕ) = P2 (ϕ)T , for all ϕ ∈ C0 ( ). Now, let H be a closed subgroup of G, and let π and ρ be unitary representations of H on H(π) and H(ρ), respectively. Let (U π , P π ) and (U ρ , P ρ ) denote the induced systems of imprimitivity.

3.2 Induced systems of imprimitivity

123

Lemma 3.14 Let T ∈ HomG [(U π , P π ), (U ρ , P ρ )]. Then    n  n          (U ρ (fk )T ξk )(x) ≤ T  (U π (fk )ξk )(x)     k=1

k=1

for almost all x ∈ G and every choice of finitely many f1 , . . . , fn ∈ Cc (G) and ξ1 , . . . , ξn ∈ H(U π ). Proof We begin by showing that, if h is a real-valued continuous function on G such that  g(x)g # (xH )h(x)dx ≥ 0 (3.8) G

for all g ∈ Cc (G), then h(x) ≥ 0 for all x ∈ G. To see this, suppose that h(x0 ) < 0 for some x0 ∈ G. Then there is a neighborhood V of x0 such that h(x) < 0 for all x ∈ V . Choose g ∈ Cc (G) with g ≥ 0, g(x0 ) >  0, and supp g ⊆ V . Then g # ≥ 0 and g # (x0 H ) > 0 also, and so it is clear that g(x)g # (xH )h(x)dx < 0, G

contradicting the inequality (3.8). Taking for h the function  n 2  n 2        2 π ρ h(x) = T  (U (fk )ξk )(x) −  (U (fk )T ξk )(x) ,     k=1

k=1

the lemma will be established when we prove that, for every g ∈ Cc (G),  h(x)g(x)g # (xH )dx ≥ 0. (3.9) G

By using Corollary 3.12 twice and the intertwining properties of T , we have 

 n 2  2 k          ρ ρ # ρ g(x)g # (xH )  (U (fk )T ξk )(x) dx = P (g ) U (fk )T ξk      G k=1 k=1  2 n      = T P π (g # ) U π (fk )ξk    k=1  2 n      ≤ T 2 P π (g # ) U π (fk )ξk    k=1

and

  n 2 2  n      π #  π    π U (fk )ξk  = g(x)g # (xH )  (U (fk )ξk )(x) dx, P (g )     G k=1

k=1

which proves (3.9) and completes the proof of the lemma.



124

The imprimitivity theorem

Proposition 3.15 Let π and ρ be unitary representations of the closed subgroup H of G. Let T ∈ HomG [(U π , P π ), (U ρ , P ρ )]. Then there exists a unique S ∈ HomH (π, ρ) such that, for every ξ ∈ H(U π ), (T ξ )(x) = S(ξ (x)) for every x ∈ G. Proof Since, for each x ∈ G, the set of all vectors ξ (x), where ξ ∈ H(U π ), is dense in H(π ), the uniqueness of S is clear. Let L denote the linear subspace of H(U π ) spanned by all elements of the form U π (f )ξ, f ∈ Cc (G), ξ ∈ F(G, π). All the elements of L can be considered as continuous H(π)-valued functions on G. Thus, L(e) = {η(e) : η ∈ L} is a subspace of H(π ). Recall that L(e) is dense in H(π) (Lemma 2.24). , . . . , ξn ∈ H(π) are such that By  nLemma 3.14,  if f1 , . . . , fn ∈ nCc (G) and ξ1

π

ρ U (fk )ξk (x) = 0, then U (fk )T ξk (e) = 0 also, for any x ∈ G. k=1

k=1

Thus we can define a linear mapping S from L(e) into H(ρ) by % $ n n   π (U (fk )ξk )(e) = (U ρ (fk )T ξk )(e), S k=1

k=1

for any f1 , . . . , fn ∈ Cc (G) and ξ1 , . . . , ξn ∈ F(G, π). By Lemma 3.14 again, S extends to a bounded linear mapping of H(π ) into H(ρ), which we also denote by S. For f ∈ Cc (G), ξ ∈ F(G, π), and x ∈ G, we have (T U π (f )ξ )(x) = (U ρ (f )T ξ )(x) = (U ρ (f )(Lx f )T ξ )(e) = S(U π (Lx f )ξ (e)) = S(U π (f )ξ (x)). This relation extends by linearity to all of L and by approximation to give, for any η ∈ H(U π ), a locally null set A in G, depending on η, such that (T η)(x) = S(η(x)) for all x ∈ G \ A. To finish the proof, it remains to show that S ∈ HomH (π, ρ). For t ∈ H , let δ(t) = (H (t)/G (t))1/2 as before. Then, for f ∈ Cc (G) and ξ ∈ F (G, π), Sπ (t)[U π (f )ξ (e)] = S[δ(t)U π (f )ξ (t −1 )] = δ(t)S[U π (Lt −1 f )ξ (e)] = δ(t)[U ρ (Lt −1 f )T ξ ](e) = δ(t)[U ρ (f )T ξ ](t −1 ) = ρ(t)[U ρ (f )T ξ ](e) = ρ(t)S[U π (f )ξ (e)]. Since the set {U π (f )ξ (e) : f ∈ Cc (G), ξ ∈ F(G, π)} is total in H(π ), it follows that Sπ (t) = ρ(t)S for all t ∈ H . Thus S ∈ HomH (π, ρ). On the other hand, if S is an arbitrary element of HomH (π, ρ) and ξ ∈ H(U π ) is continuous, then x → S(ξ (x)) is a continuous function on G with values in

3.3 The imprimitivity theorem

125

H(ρ). Define T ξ on G by (T ξ )(x) = S(ξ (x))

(3.10)

for all x ∈ G. Then, for every x ∈ G and t ∈ H , (T ξ )(xt) = S(ξ (xt)) = S(δ(t)π (t −1 )[ξ (x)]) = δ(t)ρ(t −1 )S[ξ (x)] = δ(t)ρ(t −1 )[(T ξ )(x)].

(3.11)

Cc+ (G),

Moreover, for any f ∈   f (x)(T ξ )(x), (T ξ )(x)dx = f (x)S(ξ (x)), S(ξ (x))dx G G  f (x)ξ (x), ξ (x)dx ≤ S 2 G

= S 2 μξ,ξ (f # ) ≤ S 2 ξ 2 f # ∞ .

(3.12) Cc+ (G)

Combining (3.11) and (3.12) and recalling that f → f maps onto Cc+ (G/H ), we see that T ξ defines a finite Radon measure on G/H . Thus T ξ ∈ H(U π ) and T ξ ≤ S ξ . Therefore T extends to a bounded linear map, also denoted T , of H(U π ) into H(U σ ) such that T ≤ S . It is trivial to check that T ∈ HomG [(U π , P π ), (U σ , P σ )].  #

In light of Proposition 3.15, the map S → T as defined in (3.10) is an isometry of HomH (π, σ ) onto HomG [(U π , P π ), (U σ , P σ )]. We state this formally as a theorem. Theorem 3.16 Let G be a locally compact group and H a closed subgroup of G. Let π and σ be unitary representations of H , and let (U π , P π ) and (U σ , P σ ) be the respective induced systems of imprimitivity. Then HomH (π, σ ) and HomG [(U π , P π ), (U σ , P σ )] are isometrically isomorphic as Banach spaces, with the isomorphism  : HomH (π, σ ) → HomG [(U π , P π ), (U σ , P σ )] given by, for S ∈ HomH (π, σ ) and ξ ∈ H(U π ), ξ continuous, (Sξ )(x) = S(ξ (x)) for all x ∈ G.

3.3 The imprimitivity theorem Systems of imprimitivity over coset spaces are always of the induced type, up to equivalence. This is the content of the celebrated imprimitivity theorem

126

The imprimitivity theorem

which will be proved, in full generality, in the next section. The reader who wants to avoid the technical details may confine attention to the case when the subgroup is open. Theorem 3.17 [The imprimitivity theorem] Let G be a locally compact group, H a closed subgroup of G, and (σ, P ) a system of imprimitivity for G over G/H . Then there exists a unitary representation π of H such that (σ, P ) is equivalent over G/H to (U π , P π ), the system of imprimitivity induced by π . Moreover, π is unique up to unitary equivalence. Note that the uniqueness part of the theorem follows from Theorem 3.16. The proof of the existence of the representation π, actually its construction, is a technical achievement. To assist the reader in following the reasoning, we have first provided the proof for the case when H is an open subgroup of G. In that case, all the localization difficulties disappear and the arguments are essentially algebraic. We have formulated this proof in a way which is parallel to the one in the general case. For now suppose that H is an open subgroup of G. For convenience, we write a generic element of G/H as ω and let ω0 = eH , the identity coset. Step 1 (Definition of the Hilbert space of π). Let H(π ) = P (δω0 )H(σ ). Note that Lt δω0 = δω0 for all t ∈ H . Then σ (t)P (δω0 ) = σ (t)P (δω0 )σ (t −1 )σ (t) = P (Lt δω0 )σ (t) = P (δω0 )σ (t), so that the projection onto H(π) commutes with σ (t), for all t ∈ H . This implies that H(π ) is invariant under σ (t) for all t ∈ H . Step 2 (Definition of π). For t ∈ H , let π(t) = σ (t)|H(π ) . It is clear that π is a unitary representation of H . Now we form the induced system of imprimitivity (U π , P π ) as in Section 3.2 and let H = H(U π ), for simplicity. Since H is open in G, H may be identified with the set of all (necessarily continuous) functions ξ : G → H(π ) which satisfy 

ξ (x) 2 < ∞ and ξ (xt) = π(t −1 )ξ (x), xH ∈G/H

for all x ∈ G, t ∈ H . We make this identification. Step 3 (Definition of an isometric mapping  : H(σ ) → H). For v ∈ H(σ ), let v : G → H(π) be defined by v(x) = P (δω0 )(σ (x −1 )v). First, we have to show that v ∈ H. To that end, let x ∈ G and t ∈ H . Then, by the definition

3.3 The imprimitivity theorem

127

of π and since σ (t −1 ) commutes with P (δω0 ), v(xt) = P (δω0 )(σ (t −1 )(σ (x −1 )v)) = σ (t −1 )P (δω0 )(σ (x −1 )v) Since

= π(t −1 )v(x). P (δω ) = I , the identity operator in H(σ ), with convergence in the

ω∈G/H

strong operator topology, we get 

v(x) 2 = xH ∈G/H



P (δω0 )(σ (x −1 )v) 2

xH ∈G/H



=

σ (x −1 )P (δx·ω0 )v 2

xH ∈G/H

=



P (δω )v 2 = v 2 .

ω∈G/H

Thus v ∈ H, and  is an isometric linear mapping of H(σ ) into H. Step 4 ( intertwines P and P π ). Let ϕ ∈ Cc (G/H ) and v, w ∈ H(σ ). Then  ϕ(xH )v(x), w(x) P π (ϕ)v, w = xH ∈G/H

=



ϕ(xH )P (δω0 )(σ (x −1 )v), P (δω0 )(σ (x −1 )w)

xH ∈G/H

=





ϕ(xH )P (δx·ω0 )v, w =

xH ∈G/H

P (ϕ(ω)δω )v, w

ω∈G/H

= P (ϕ)v, w = P (ϕ)v, w. This shows P π (ϕ) = P (ϕ) for all ϕ ∈ Cc (G/H ), and hence for all ϕ ∈ C0 (G/H ). Step 5 ( intertwines σ and U π ). Let v ∈ H(σ ) and y ∈ G. Then, by definition of , we have ((σ (y)v)(x) = P (δω0 )(σ (x −1 y)v) = v(y −1 x) = (U π (y)v)(x), for all x ∈ G. Thus σ (y) = U π (y) for every y ∈ G. Step 6 ( is a unitary map of H(σ ) onto H). By Step 5, the range of  is

invariant under U π . Moreover, P π (δω ) = I and ω∈G/H

U (x)P (δω0 )H = P π (δx·ω0 )H, π

π

for each x ∈ G. Therefore, if we show that H(σ ) ⊇ P π (δω0 )H, then H(σ ) must be all of H. Now, for v ∈ H(π) = P (δω0 )H(σ ), by Step 4, v = P (δω0 )v = P π (δω0 )v.

128

The imprimitivity theorem

So v as a function on G is supported on H and, for t ∈ H , v(t) = P (δω0 )σ (t −1 )v = σ (t −1 )v. But an arbitrary element ξ in P π (δω0 )H is 0 on G \ H and, for t ∈ H , ξ (t) = π(t −1 )ξ (e) = σ (t −1 )ξ (e). Hence (ξ (e)) = ξ . Thus, the range of  contains P π (δω0 )H. This completes the proof that (σ, P ) is equivalent over G/H to (U π , P π ) under the assumption that H is open.

3.4 Proof of the imprimitivity theorem: the general case The proof of the imprimitivity theorem in the general case will follow the same line as that given in the previous section (under the assumption that H be open in G), but several of the steps will be substantially harder to achieve. The difficulties center around the problem of localizing at the identity coset in G/H and the fact that we cannot take the elements of the Hilbert space to be continuous functions. It is instructive to pay attention to the methods that are used to overcome these difficulties. It requires a sequence of five technical lemmas to attain Step 1, the definition of H(π ). Let H be a closed subgroup of G and (σ, P ) a system of imprimitivity over G/H . Let D = span {σ (f )v : f ∈ Cc (G), v ∈ H(σ )}. One may think of D as the “smooth” elements in H(σ ). Lemma 3.18 The subspace D is dense in H(σ ) and invariant under both σ (G) and σ (Cc (G)). Proof The invariance of D under σ (G) and σ (Cc (G)) follows from σ (x)σ (f ) = σ (Lx f ) and σ (f )σ (g) = σ (f ∗ g), for x ∈ G and f, g ∈ Cc (G). To prove that D is dense in H(σ ), let w ∈ H(σ ) and ε > 0 be given. Since σ is strongly continuous, there exists a neighborhood V of e in G such that σ (x)w − w ≤ ε for all x ∈ V . Let g ∈ Cc+ (G) such  that supp g ⊆ V and G g(y)dy = 1. Then, for any v ∈ H(σ ) with v = 1, we have      |σ (g)w − w, v| =  g(y)(σ (y)w, v − w, v)dy   G g(y)|σ (y)w − w, v|dy ≤ ε g(y)dy = ε. ≤ G

Thus σ (g)w − w ≤ ε. Hence D is dense in H(σ ).

G



3.4 Proof in the general case

129

There is another linear mapping of Cc (G) into B(H(σ )), that is, the composition of P with the map f → f # . For f ∈ Cc (G), f # ∈ Cc (G/H ), and for each compact subset K of G, there is a constant cK such that

f # ∞ ≤ cK f ∞ ,

(3.13)

for all f ∈ CK (G) (see Proposition 1.10). For any two vectors v, w ∈ H(σ ), we can define a linear functional νv,w on Cc (G) by setting νv,w (f ) = P (f # )v, w.

(3.14)

Lemma 3.19 For each v, w ∈ H(σ ), the functional νv,w defined in (3.14) is continuous with respect to the inductive limit topology on Cc (G) and thus determines a regular complex Borel measure, also denoted νv,w , on G so that  P (f # )v, w = f (x)dνv,w (x) (3.15) G

for all f ∈ Cc (G). Proof For any compact K ⊆ G, let cK be a constant so that (3.13) holds. Then, for every f ∈ Cc (G), |νv,w (f )| = |P (f # )v, w| ≤ P (f # ) · v · w ≤ f # ∞ v · w ≤ cK v · w · f ∞ . Thus, νv,w is continuous on Cc (G) with the inductive limit topology. Now apply the general Riesz representation theorem.  We need a second regular Borel measure, this time on G × G, associated with any pair v, w ∈ H(σ ). Lemma 3.20 Let v, w ∈ H(σ ). There exists a regular Borel measure λv,w on G × G such that  f (x)g(y)dλv,w (x, y) = P (f # )σ (g)v, w, (3.16) G×G

for all f, g ∈ Cc (G). Proof For F ∈ Cc (G × G) and y ∈ G, let Fy (x) = F (x, y) for all x ∈ G. Then Fy ∈ Cc (G) and since F is uniformly continuous, it is easily verified that y → Fy# is a continuous map from G into (Cc (G/H ), · ∞ ) and has compact support. Thus, y → P (Fy# ) is a continuous mapping with compact support from G into B(H(σ )).

130

The imprimitivity theorem

We claim that the function y → P (Fy# )σ (y)v, w is continuous. For y, z ∈ G, we compute |P (Fy# )σ (y)v, w − P (Fz# )σ (z)v, w| ≤ |P (Fy# )(σ (y) − σ (z))v, w| + |(P (Fy# ) − P (Fz# ))σ (z)v, w| ≤ P (Fy# ) · σ (y)v − σ (z)v · w + P (Fy# ) − P (Fz# ) · v · w . Continuity now follows from boundedness of P (Fy# ) , strong continuity of σ , and continuity of y → P (Fy# ). Therefore, we can define a linear functional λv,w on Cc (G × G) by  λv,w (F ) = P (Fy# )σ (y)v, wdy, (3.17) G

for F ∈ Cc (G × G). To check that λv,w is continuous in the inductive limit topology on Cc (G × G), let K be a compact subset of G × G and choose compact subsets K1 and K2 of G such that K ⊆ K1 × K2 . Let F ∈ CK (G × G). Then, for each y ∈ G, Fy ∈ CK1 (G) and Fy ≡ 0 unless y ∈ K2 . Hence   # |P (Fy )σ (y)v, w|dy ≤

P (Fy# ) · v · w dy |λv,w (F )| ≤ G

K2

≤ cK1 |K2 | · v · w · F ∞ . The constant cK1 |K2 | · v · w being independent of F ∈ CK (G × G), it follows that λv,w is continuous with respect to the inductive limit topology on Cc (G × G). We denote by λv,w also the associated regular Borel measure on G × G. For f, g ∈ Cc (G), the elementary tensor f ⊗ g ∈ Cc (G × G) is defined by f ⊗ g(x, y) = f (x)g(y), for all (x, y) ∈ G × G. Clearly, (f ⊗ g)#y (xH ) = f # (xH )g(y), and hence by definition (3.17),   f (x)g(y)dλv,w (x, y) = λv,w (f ⊗ g) = P (g(y)f # )σ (y)v, wdy G×G G  g(y)σ (y)v, P (f # )∗ wdy = G

= σ (g)v, P (f # )∗ w = P (f # )σ (g)v, w. This establishes formula (3.16).



Our next objective is to show that for ξ, η ∈ D, the measure νξ,η is absolutely continuous with respect to the Haar measure and has a continuous Radon– Nikodym derivative which can be determined explicitly.

3.4 Proof in the general case

131

To start with, let v, w ∈ H(σ ) and g, h ∈ Cc (G). Then, by (3.15) and (3.16), for all f ∈ Cc (G),  f (z)dνσ (g)v,σ (h)w (z) G

= P (f # )σ (g)v, σ (h)w  = h(z)σ (z−1 )P (f # )σ (g)v, wdz G = h(z)P (Lz−1 f # )σ (z−1 )σ (g)v, wdz G  h(z)P ((Lz−1 f )# )σ (Lz−1 g)v, wdz = G   = h(z) f (zx)g(zy)dλv,w (x, y)dz G G×G   = h(zx −1 )f (z)g(zx −1 y)G (x −1 )dz dλv,w (x, y) G×G G   f (z) h(zx −1 )g(zx −1 y)G (x −1 )dλv,w (x, y)dz. = G

G×G

Since g, h ∈ Cc (G), it is straightforward to verify that  Rσ (g)v,σ (h)w : z −→ h(zx −1 )g(zx −1 y)G (x −1 )dλv,w (x, y)

(3.18)

G×G

is a continuous function on G. Now, let ξ and η be arbitrary elements in D, say ξ=

n 

σ (gi )vi and η =

m 

σ (hj )wj ,

j =1

i=1

for certain gi , hj ∈ Cc (G) and vi , wj ∈ H(σ ). It follows that   f (z)dνξ,η (z) = f (z)Rξ,η (z)dz, G

(3.19)

G

where Rξ,η is defined by Rξ,η (z) =

m n  

Rσ (gi )vi ,σ (hj )wj (z).

(3.20)

i=1 j =1

Rξ,η is a continuous function, and (3.19) shows that the definition of Rξ,η does not depend on the particular representations of ξ and η. We summarize the foregoing as follows. Lemma 3.21 For ξ, η ∈ D, let Rξ,η be defined as in (3.20) and (3.19). Then νξ,η is absolutely continuous with respect to the left Haar measure on G and the Radon–Nikodym derivative is the continuous function Rξ,η .

132

The imprimitivity theorem

In passing we mention that for g, h ∈ Cc (G), v, w ∈ H(σ ), and u, z ∈ G,  −1 h(u−1 zx −1 )g(u−1 zx −1 y)G (x −1 )dλv,w (x, y) Rσ (g)v,σ (h)w (u z) = G×G  = Lu h(zx −1 )Lu g(zx −1 y)G (x −1 )dλv,w (x, y) G×G

= Rσ (u)σ (g)v,σ (u)σ (h)w (z). Thus, for any ξ, η ∈ D and u ∈ G, Rξ,η (u−1 ) = Rσ (u)ξ,σ (u)η (e).

(3.21)

We now define a form β : D × D → C by evaluating these Rξ,η at e. That is, for ξ, η ∈ D, let β(ξ, η) = Rξ,η (e).

(3.22)

Lemma 3.22 The map β : D × D → C defined by (3.22) is a sesquilinear form on D × D with the following properties: (i) β(ξ, ξ ) ≥ 0, (ii) β(σ (t)ξ, σ (t)η) = G (t)H (t −1 )β(ξ, η), (iii) P (f # )ξ, η = G f (x)β(σ (x −1 )ξ, σ (x −1 )η)dx, for all ξ, η ∈ D, t ∈ H , and f ∈ Cc (G). Proof The sesquilinearity of β follows from that of the mapping (ξ, η) → νξ,η . If f ∈ Cc+ (G), then f # ≥ 0 on G/H and, since P is a ∗-homomorphism of C0 (G/H ), P (f # ) is a positive operator on H(σ ). Thus for any ξ ∈ D,   f (z)Rξ,ξ (z)dz = f (z)dνξ,ξ (z) = P (f # )ξ, ξ  ≥ 0. G

G

Since f was an arbitrary positive function in Cc (G), this implies that Rξ,ξ (z) ≥ 0 for all z ∈ G. In particular, β(ξ, ξ ) = Rξ,ξ (e) ≥ 0. Thus (i) holds. To prove (ii), we use sesquilinearity to reduce to the case where ξ = σ (g)v and η = σ (h)w for some g, h ∈ Cc (G) and v, w ∈ H(σ ). Then, for any t ∈ H , recalling (3.17), (3.18), and (3.21), we have β(σ (t)σ (g)v, σ (t)σ (h)w) = Rσ (g)v,σ (h)w (t −1 )  h(t −1 x −1 )g(t −1 x −1 y)G (x −1 )dλv,w (x, y) = G×G t = F (x, y)dλv,w (x, y) G×G  (3.23) = P (t F )#y σ (y)v, wdy G

3.4 Proof in the general case

133

where, for x, y ∈ G, t

F (x, y) = h(t −1 x−1)g(t −1 x −1 y)G (x −1 ).

Now, for xH ∈ G/H and y ∈ G,  t t # F (xy, y)ds ( F )y (xH ) = H  = h(t −1 s −1 x −1 )g(t −1 s −1 x −1 y)G (s −1 x −1 )ds H  h(s −1 x −1 )g(s −1 x −1 y)G (s −1 x −1 )ds = G (t)H (t −1 ) H

= G (t)H (t −1 ) (e F )#y (xH ).

(3.24)

Inserting (3.24) into (3.23) and using (3.23) again with e replacing t yields β(σ (t)σ (g)v, σ (t)σ (h)w) = G (t)H (t −1 )β(σ (g)v, σ (h)w). This establishes (ii). Finally, (iii) follows immediately from (3.15), (3.20), (3.21), and (3.22) combined.  We are now ready to complete Step 1 of the proof of the imprimitivity theorem, the definition of the appropriate Hilbert space. Let Kβ denote the null space of β; that is, Kβ = {ξ ∈ D : β(ξ, η) = 0 for all η ∈ D}. For ξ ∈ D, let [ξ ] = ξ + Kβ ∈ D/Kβ . We define an inner product on D/Kβ by [ξ ], [η] = β(ξ, η), for all ξ, η ∈ D. Step 1 (Definition of the Hilbert space of π ). Let H(π ) denote the completion of the inner product space (D/Kβ , ·, ·). Step 2 (Definition of π). Note that Kβ is invariant under σ (t), for t ∈ H . In fact, Lemma 3.22(ii) implies that, if ξ ∈ Kβ , then β(σ (t)ξ, η) = G (t)H (t −1 )β(ξ, σ (t −1 )η) = 0, for all η ∈ D. Thus, [σ (t)ξ ] is independent of the representative ξ of [ξ ]. Also, note that, for [ξ ], [η] ∈ D/Kβ and t ∈ H , [σ (t)ξ ], [σ (t)η] = β(σ (t)ξ, σ (t)η) = G (t)H (t −1 )β(ξ, η) = G (t)H (t)−1 [ξ ], [η],

(3.25)

134

The imprimitivity theorem

by Lemma 3.22(ii) again. For t ∈ H , recall that δ(t) = H (t)1/2 G (t)−1/2 .

(3.26)

For t ∈ H , define π(t) : D/Kβ → D/Kβ by π(t)[ξ ] = δ(t)[σ (t)ξ ],

(3.27)

for all ξ ∈ D. By (3.25) and (3.26), π (t) preserves the inner product on D/Kβ and hence extends to an isometric linear mapping of H(π ), also denoted π (t). Since σ and δ are homomorphisms of G and H , respectively, and π(e) is the identity operator on H(π), π : t → π(t) is a homomorphism of H into the unitary group of H(π). Lemma 3.23 π is a unitary representation of H . Proof It only remains to verify the weak continuity of π , and for that it suffices to show that t → π(t)[ξ ], [η] is continuous for [ξ ], [η] ∈ D/Kβ . By linearity, it is enough to check the case where ξ = σ (g)v and η = σ (h)w with g, h ∈ Cc (G) and v, w ∈ H(σ ). Then π(t)[σ (g)v], [σ (h)w] = δ(t)β(σ (t)σ (g)v, σ (h)w) = δ(t)Rσ (t)σ (g)v,σ (h)w (e)  h(x −1 )g(t −1 x −1 y)G (x −1 )dλv,w (x, y), = δ(t) G×G

which varies continuously with t by continuity of δ, uniform continuity of g, and standard integration arguments.  Now we form the induced system of imprimitivity (U π , P π ) as in Section 3.2 and let H = H(U π ), for simplicity. Step 3 (Definition of an isometric map  : H(σ ) → H). For each ξ ∈ D, define a map ξ : G → H(π) by ξ (x) = [σ (x −1 )ξ ],

(3.28)

for all x ∈ G. We will see in Lemma 3.25 that  can be extended to all of H(σ ) and in Lemma 3.26 that  is actually an isometry. But first we have to prove that ξ is an element of H for each ξ ∈ D. To that end, we show Lemma 3.24 If ξ ∈ D, then ξ : G → H(π ) is continuous.

3.4 Proof in the general case

135

Proof Of course, we can assume that ξ = σ (g)v for some g ∈ Cc (G) and v ∈ H(σ ). For x, y ∈ G, we have

ξ (x) − ξ (y) 2 = σ (x −1 )σ (g)v − σ (y −1 )σ (g)v 2 = β(σ (Lx −1 g − Ly −1 g)v, σ (Lx −1 g − Ly −1 g)v) = Rσ (Lx−1 g−Ly−1 g)v,σ (Lx−1 g−Ly−1 g)v (e)  [g(xu−1 z) − g(yu−1 z)][g(xu−1 ) − g(yu−1 )]G (u−1 )dλv,v (u, z). = G×G

Again, using standard arguments, one sees that this equation implies

ξ (x) − ξ (y) 2 → 0, as x → y in G. Hence ξ is continuous.



Lemma 3.25 If ξ ∈ D, then ξ ∈ H and ξ ≤ ξ . Proof As to ξ ∈ H, according to the previous lemma, it only remains to show that ξ satisfies the covariance condition and that μξ is a finite measure on G/H . For x ∈ G and t ∈ H , ξ (xt) = [σ (t −1 )σ (x −1 )ξ ] = δ(t)π (t −1 )[σ (x −1 )ξ ] = δ(t)π(t −1 )ξ (x), as required. Now, let ϕ ∈ Cc (G/H ) and choose f ∈ Cc (G) such that f # = ϕ. Then   2 f (x) ξ (x) dx = f (x)[σ (x −1 )ξ ], [σ (x −1 )ξ ]dx μξ (ϕ) = G G  = f (x)β(σ (x −1 )ξ, σ (x −1 )ξ )dx = P (ϕ)ξ, ξ , (3.29) G

by Lemma 3.22(iii). In particular, |μξ (ϕ)| ≤ ϕ ∞ ξ 2 , for all ϕ ∈ Cc (G), which implies that

ξ 2 = μξ (G/H ) ≤ ξ 2 < ∞. This proves the lemma.



By Lemma 3.25,  can be extended uniquely to a norm decreasing linear map from H(σ ) into H. Lemma 3.26 For every v ∈ H(σ ), v = v . Proof Notice first that it follows from the definition of a system of imprimitivity that P (Cc (G/H ))H(σ ) is dense in H(σ ). In order to prove the lemma, it therefore suffices to show v ≥ v for v ∈ H(σ ) of the form v = P (ψ)w, where ψ ∈ Cc (G/H ) and w ∈ H(σ ).

136

The imprimitivity theorem

Choose ϕ ∈ Cc (G/H ) so that 0 ≤ ϕ ≤ 1 and ϕ(xH ) = 1 for all xH ∈ supp ψ. Using formula (3.29), which works the same for v, we obtain |μv (ϕ)| = |P (ϕ)v, v| = |P (ϕ)P (ψ)w, v| = |P (ϕψ)w, v| = |P (ψ)w, v| = v 2 . Thus v 2 = μv (G/H ) ≥ μv (ϕ) = v 2 .



This completes Step 3. We have now come to one of the major technical points in establishing the imprimitivity theorem. Let us recall that by Proposition 3.11, for f ∈ Cc (G) and ξ, η ∈ H,  π # P (f )ξ, η = f (x)ξ (x), η(x)dx. (3.30) G

Lemma 3.27 The set P (Cc (G/H ))D is total in H. π

Proof Let χ ∈ H be such that P π (f # )ξ, χ  = 0, for all f ∈ Cc (G) and ξ ∈ D. By (3.28) and (3.30), (3.31) implies  f (x)[σ (x −1 )ξ ], χ (x)dx = 0,

(3.31)

(3.32)

G

for all f ∈ Cc (G) and ξ ∈ D. We have to show that (3.32) yields χ (x) = 0 locally almost everywhere on G. Let K be a compact subset of G, let A = {x ∈ K : χ(x) = 0}, and assume that |A| > 0. By the generalized Lusin’s theorem, there exists a compact subset C of A such that |C| > 0 and χ|C is continuous. Now, for every x ∈ C, χ (x) = 0 and, since D/Kβ is dense in H(π ), there exists ηx ∈ D such that [ηx ], χ (x) = 0. Let ξx = σ (x)ηx ∈ D and define Fx : C → C by Fx (y) = [σ (y −1 )ξx ], χ (y). Then Fx is a continuous function on C and Fx (x) = [ηx ], χ (x) = 0. Hence there exists a neighborhood Vx of x in C such that Fx (y) = 0 for all y ∈ V x . Since C is compact, it is covered by finitely many of the Vx , x ∈ C. As |C| > 0, there is at least one x ∈ C with |Vx | > 0. Replacing C by V x , we may assume that there is a compact subset C of G with |C| > 0 such that χ|C is continuous and that there is a ξ ∈ D so that F (x) = [σ (x −1 )ξ ], χ (x) = 0 for all x ∈ C.

3.4 Proof in the general case

137

Let U be a relatively compact open set in G with C ⊆ U . By the Tietze extension theorem, applied to the real and imaginary parts of F , there is a continuous extension F˜ of F to U¯ such that |F˜ (y)| ≤ 2 F ∞ for all y ∈ U¯ . Since χ is locally square-integrable, 1/2 

χ(y) 2 dy < ∞. (3.33) d= U¯

F being continuous and nonzero on the compact set C, c = inf{|F (y)|2 : y ∈ C} > 0.

(3.34)

Now, by (3.21) and (3.22) and the definition of the inner product on D/Kβ ,

[σ (y −1 )ξ ] = Rξ,ξ (y)1/2 , and Rξ,ξ is continuous by Lemma 3.21. It follows that M = sup{ [σ (y −1 )ξ ] : y ∈ U¯ } < ∞.

(3.35)

Choose an open set V so that C ⊆ V ⊆ U and |V \ C|1/2 <

c|C| . 4dM F ∞

(3.36)

Let g ∈ Cc (G) be such that 1C ≤ g ≤ 1V and define f ∈ Cc (G) by . (y) if y ∈ V , g(y)F f (y) = 0 if y ∈ G \ V . Then, starting with (3.32) and using (3.33)–(3.36), we get  f (y)[σ (y −1 )ξ ], χ (y)dy 0= G  = f (y)F (y)dy + f (y)[σ (y −1 )ξ ], χ (y)dy C V \C   = |F (y)|2 dy + f (y)[σ (y −1 )ξ ], χ (y)dy C V \C  ≥ c|C| − |f (y)| · [σ (y −1 )ξ ] · χ(y) dy V \C

1/2 



≥ c|C| − M

|f (y)| dy

1/2

2

V \C

χ (y) dy 2

V \C

c |C| > 0. 2 This contradiction shows that χ = 0 locally almost everywhere. Therefore, P π (Cc (G/H ))D is a total set in H.  ≥ c|C| − 2Md · F ∞ · |V \ C|1/2 ≥

138

The imprimitivity theorem

Step 4 ( intertwines P and P π ). To prove P (ϕ) = P π (ϕ) for all ϕ ∈ C0 (G/H ), it suffices to consider ϕ ∈ Cc (G/H ). Fix such a ϕ and choose f ∈ Cc (G) so that f # = ϕ. From Proposition 3.11 and Lemma 3.22(iii), we obtain  π f (x)ξ (x), η(x)dx P (ϕ)ξ, η = G = f (x)[σ (x −1 )ξ ], [σ (x −1 )η]dx G = f (x)β(σ (x −1 )ξ, σ (x −1 )η)dx = P (ϕ)ξ, η G

for all ξ, η ∈ D. By continuity, P π (ϕ)v, w = P (ϕ)v, w for all v, w ∈ H(σ ). It follows that, for any ψ ∈ Cc (G/H ) and v, w ∈ H(σ ), ¯ ¯ (ϕ)v, w P (ϕ)v, P π (ψ)w = P π (ψ)P (ϕ)v, w = P (ψ)P π ¯ ¯ = P (ψϕ)v, w = P (ψϕ)v, w = P π (ϕ)v, P π (ψ)w. Since, by Lemma 3.27, P π (Cc (G/H ))D is dense in H, this formula shows P (ϕ)v = P π (ϕ)v, for all v ∈ H(σ ). Step 5 ( intertwines σ and U π ). σ (y) = U π (y) follows at once from the definition of  in (3.28). Indeed, for all x ∈ G and ξ ∈ D, (U π (y)ξ )(x) = ξ (y −1 x) = [σ (x −1 y)ξ ] = (σ (y)ξ )(x). Step 6 ( is a unitary map of H(σ ) onto H). Recall that  is a linear isometry by Lemma 3.26. Moreover, Step 4 implies that the range of  includes P π (Cc (G/H ))H(σ ) which is total in H by Lemma 3.27. Thus, the range of  is all of H and  is a unitary map. Steps 4, 5, and 6 mean that (σ, P ) is equivalent to (U π , P π ) over G/H . This completes the proof of the imprimitivity theorem (Theorem 3.17) in the general case.

3.5 Notes and references The imprimitivity theorem was established by Mackey [99] for second countable locally compact groups. After the definition of induced representations

3.5 Notes and references

139

was extended to general locally compact groups by Loomis [98] and Blattner [22], it was some time before a proof of the imprimitivity theorem in the general situation was devised. In 1974, Rieffel [134] extended the concept of inducing representations to C ∗ -algebras and proved a version of the imprimitivity theorem in that setting. He also showed how to derive the theorem for general locally compact groups from his C ∗ -algebra version. Around the same time, Fell developed an imprimitivity theorem for Banach ∗-algebraic bundles which can be found in Fell and Doran [54, section XI.14]. Each of the approaches of Rieffel and Fell has far-reaching implications, but the presentation of a reasonable account of either the theory for C ∗ -algebras or Banach ∗-algebraic bundles is beyond the scope of this book. The proof we have presented for the imprimitivity theorem follows the ideas of Ørstedt [120]. The essential idea of using the dense linear subspace D is credited to Poulsen [126], where a proof was given for connected Lie groups and D consisted of the C ∞ -vectors in H(σ ).

4 Mackey analysis

Suppose G is a locally compact group and N is a closed normal subgroup of G. The phrase Mackey analysis (or Mackey machine) refers to a set of procedures  as a set from knowledge of N , the action of G on N , and aimed at describing G certain representations of closed subgroups of G which contain N. The goals of this chapter are to introduce Mackey analysis, prove the core results in the situations where the analysis works well, and provide a wide variety of examples to demonstrate its power but also illustrate the limitations of the theory. The theory is developed in stages of increasing generality to help the reader  is described in building understanding. A large number of examples, where G with explicit formulae calculated for the irreducible representations, are provided in the case where N is abelian and G splits as a semidirect product of N with G/N. Explicit calculations are much more difficult when G does not split or when N is not abelian. Thus, there are fewer examples provided to illustrate the more general formulations of Mackey analysis. Nevertheless, the descriptions provided in the later sections of the chapter are useful for establishing general properties of dual spaces, sometimes through inductive arguments. In the remainder of this book, we will make use of Mackey analysis both through explicit calculation of irreducible representations and in the development of general theory. We begin by proving the main theorem in the relatively simple case where G has an abelian normal subgroup N of finite index. Treating this case makes  under the natural action of G play a critical role it clear that the G-orbits in N in the analysis. Section 4.2 is devoted to a study of orbit structures, including a selection of examples which illustrate situations that arise in the application of Mackey analysis.

140

4.1 Almost abelian groups

141

In Section 4.3, we assume that N is an abelian closed normal subgroup of G and introduce the technical condition that N be a Mackey-compatible subgroup. When N is a Mackey-compatible subgroup of G, Theorem 4.27 provides a  as a set. However, the components of this description require description of G further illumination in many cases. The rest of the section is devoted to refining this description in the case where G splits as a semidirect product of N and G/N. An extensive collection of examples are worked out in the next sections. Section 4.4 is devoted to nilpotent and two-step solvable groups. We also provide a proof in this section of the useful fact that any irreducible representation of a connected and simply connected nilpotent Lie group is induced from a character of a closed subgroup. Some examples of groups formed by the action of a compact group on an abelian group are treated in Section 4.5. When the condition that N be a Mackey-compatible subgroup of G fails, Mackey analy sis breaks down and Theorem 4.27 does not give a complete description of G. Examples of this breakdown are given in Section 4.6. When G does not split as a semidirect product of N and G/N, there is still a successful Mackey analysis making use of the cocycle that identifies the particular algebraic extension of G/N by N that G is. The theory is developed in Section 4.7 and applied to a crystal symmetry group as an example, where all the computations can be explicitly worked out. Sections 4.8 is devoted to Mackey analysis when the normal subgroup N is no longer assumed to be abelian. There is no question that this takes us into  in the deeper mathematics. In particular, the relatively comprehensible C0 (N), ∗ case of abelian N, is replaced with C (N). In spite of the difficulties, Theorem 4.65 provides for Mackey analysis in a quite general situation. We present several different sets of conditions under which the hypotheses of Theorem 4.65 are satisfied.

4.1 Mackey analysis for almost abelian groups A group G is called almost abelian if it contains an abelian normal subgroup of finite index. Besides obviously containing all abelian and all finite groups, this class includes the crystal symmetry groups, so an elementary analysis of their irreducible representations may have independent interest. However, our main purpose in studying the irreducible representations of almost abelian groups is to illuminate the main ingredients that are used in Mackey’s general approach to constructing the dual spaces of groups.

142

Mackey analysis

Throughout this section, G is a locally compact group with an abelian closed normal subgroup N such that [G : N ] < ∞. Then N is open in G. Our main goal is to understand what an irreducible representation of G looks like, at least in terms of N , G/N, and the particular way in which G is formed from N and G/N. Let us start by selecting an irreducible representation σ of G on the Hilbert space H(σ ). Let σ |N denote the restriction of σ to N. In Example 3.3 we showed that σ defines a system of imprimitivity (σ, P ) for G over the dual ) on H(σ ) such . Thus, P is a nondegenerate ∗-representation of C0 (N group N that   f (n)σ (n)dn P (f ) = σ |N (f ) = N

 is given by x · χ(n) = for any f ∈ L1 (N ). The action of x ∈ G on χ ∈ N −1 χ (x nx) (n ∈ N), and (σ, P ) satisfies σ (x)P (ϕ)σ (x −1 ) = P (x · ϕ), where  x ∈ G, and χ ∈ N . Now P is far from x · ϕ(χ ) = ϕ(x −1 · χ) for ϕ ∈ C0 (N), being faithful in general. Consider  : P (ϕ) = 0}. ker P = {ϕ ∈ C0 (N)  and it is known that every closed ideal is This is a closed ideal of C0 (N), determined by its zero set. That is, if  : ϕ(χ) = 0 for all ϕ ∈ ker P }, h(ker P ) = {χ ∈ N  and then h(ker P ) is a closed subset of N  : ϕ(χ) = 0 for all χ ∈ h(ker P )} = ker P . {ϕ ∈ C0 (N)  since N is abelian, n · χ = χ for all n ∈ N Returning to the action of G on N, . So for any χ ∈ N , the orbit through χ , and χ ∈ N G(χ) = {x · χ : x ∈ G}, has at most [G : N ] elements. The first proposition of this section illustrates the profound connection between irreducible representations of G and G-orbits in  N. Proposition 4.1 Let G be a locally compact group and N an abelian closed normal subgroup with finite index in G. Let σ be an irreducible representation  associated with σ by P (f) = of G, and let P be the ∗-representation of C0 (N) 1  such σ |N (f ), for all f ∈ L (N). Then there exists a unique G-orbit Oσ in N that h(ker P ) = Oσ .

4.1 Almost abelian groups

143

 and x ∈ G we have ϕ ∈ ker P if and only if x · ϕ ∈ Proof For ϕ ∈ C0 (N) ker P , since P (x · ϕ) = σ (x)P (ϕ)σ (x −1 ). Since ϕ(x · χ ) = (x −1 · ϕ)(χ ), we conclude that x · χ ∈ h(ker P ) whenever χ ∈ h(ker P ). Thus, h(ker P ) is a  G-invariant closed subset of N. Now suppose that h(ker P ) is not a single G-orbit. Then there exist  is Hausdorff and the χ1 , χ2 ∈ h(ker P ) such that G(χ1 ) ∩ G(χ2 ) = ∅. Since N  is essentially an action of the finite group G/N, there exist action of G on N neighborhoods V1 and V2 of χ1 and χ2 , respectively, so that x · V1 ∩ y · V2 = ∅ for all x, y ∈ G.  such that ϕi (χi ) = 1 and supp ϕi ⊆ Vi . Let For i = 1, 2, choose ϕi ∈ C0 (N) Ki be the closed linear span of {P (x · ϕi )ξ : ξ ∈ H(σ ), x ∈ G} (i = 1, 2). Then Ki is a σ -invariant subspace of H(σ ). Indeed, this follows from σ (y)P (x · ϕ)ξ = P ((yx) · ϕ)σ (y)ξ  and ξ ∈ H(σ ). Moreover, by the choice of ϕ1 and ϕ2 , for x, y ∈ G, ϕ ∈ C0 (N), P (x · ϕ1 )ξ, P (y · ϕ2 )η = P (y · ϕ2 x · ϕ1 )ξ, η = 0 for all x, y ∈ G and ξ, η ∈ H(σ ). Thus K1 and K2 are orthogonal. Since σ is irreducible, at least one of the Ki must be {0}. But if Ki = {0} then P (ϕi ) = 0, which implies that ϕi (h(ker P )) = {0}. However, this contradicts χi ∈ h(ker P ) and ϕi (χi ) = 1. Thus h(ker P ) must consist of a single G-orbit.   If σ is an irreducible representation of G and Oσ the unique G-orbit in N such that h(ker P ) = Oσ , then we will call Oσ the G-orbit associated with σ. It is clear that if σ and σ  are equivalent irreducible representations of  is a well-defined G, then Oσ = Oσ  . So, associated with every element of G G-orbit. Continuing with a fixed irreducible representation σ of G, the quotient ∗  ker P is isomorphic to the finite-dimensional algebra C -algebra C0 (N)/ C(Oσ ), since the mapping ϕ → ϕ|Oσ is a surjective homomorphism and  function in C0 (N), Oσ = h(ker P ). Since every function on Oσ extends to some    we can define a mapping P : C(Oσ ) → B(H(σ )) by P χ ∈Oσ ϕ(χ )δχ =   P (ϕ), for ϕ ∈ C0 (N). Then P is a nondegenerate ∗-representation of C(Oσ )

on H(σ ) and (σ, P  ) is a system of imprimitivity for G over Oσ . Since Oσ is a transitive finite left G-space, we can use Proposition 3.9 to move to a homogeneous space. Select some χ0 ∈ Oσ and let H = {x ∈ G : x · χ0 = χ0 }. Let θ : G/H → Oσ be given by θ(xH ) = x · χ0 . If P  : C(G/H ) → B(H(σ )) is defined to be P  = P  ◦ θ∗ , then (σ, P  ) is a system of imprimitivity for G over G/H.

144

Mackey analysis

By the imprimitivity theorem (Theorem 3.17), there exists a representation π of H such that (σ, P  ) is equivalent to (U π , P π ). Since σ is irreducible, U π is irreducible and so π must be irreducible by Corollary 2.43. Now H , containing N, is an open subgroup of G. In this case we can extract more information from the proof of the imprimitivity theorem given in Section 3.4. In Step 1 of that proof, we saw that we can take H(π) to be P  (δω0 )H(ω), where ω0 = eH , the identity coset in G/H . Moreover, for t ∈ H and v ∈ H(π ), π (t)v = σ (t)v. We can actually identify π|N . Let f ∈ L1 (N) such that f(χ0 ) = 1 and f(χ) = 0 for χ ∈ Oσ , χ = χ0 . Since θ (ω0 ) = χ0 , we can compute the projection P  (δω0 ) to be P  (δω0 ) = P  (δχ0 ) = P (f) = σ |N (f ). Then, for any n ∈ N and v ∈ H(π) = P  (δω0 )H(ω), π(n)v = σ (n)P  (δω0 )v = σ (n)σ |N (f )v = σ |N (Ln f )v   = χ ∈Oσ L n f (χ)P (δχ )v = Ln f (χ0 )v = χ0 (n)f(χ0 )v = χ0 (n)v. Thus π(n) = χ0 (n)IH(π) . That is, π|N is a multiple of χ0 . We summarize this in a proposition. Proposition 4.2 Let G be a locally compact group and suppose that N is an  select a abelian closed normal subgroup of G with finite index. Let σ ∈ G, , and let character χ0 ∈ Oσ ⊆ N H = Gχ0 = {x ∈ G : x · χ0 = χ0 }. Then there exists an irreducible representation π of H such that U π is equivalent to σ and π|N is a multiple of χ0 . With χ0 ∈ Oσ fixed, the irreducible representation π of H such that U π is equivalent to σ is unique up to unitary equivalence by Theorem 3.17. If χ1 is another element of Oσ , say χ1 = y · χ0 , then let H1 = {x ∈ G : x · χ1 = χ1 }. One can easily check that H1 = yHy −1 . Then y · π is an irreducible representation of H1 . By Proposition 2.39, U y·π is also equivalent to σ . In addition, (y · π)|N is a multiple of χ1 . So σ can be realized as an induced representation in various ways, with the inducing representations coming one from each stability subgroup associated with the elements of Oσ .  and let H = Gχ0 . For any representation π of Now suppose that χ0 ∈ N H such that π|N is a multiple of χ0 , consider U π = indG H π and the associated nondegenerate ∗-representation P π of C0 (G/H ) on H(U π ). For any β ∈ C0 (G/H ), ξ ∈ F(G, π), and x ∈ G, (P π (β)ξ )(x) = β(xH )ξ (x).

4.2 Orbits in the dual

145

Since G(χ0 ) is finite and θ : G/H → G(χ0 ) is a bijection, there exists f ∈ L1 (N ) such that β = f◦ θ. So (P π (β)ξ )(x) = f(x · χ0 )ξ (x). On the other hand, since π|N is a multiple of χ0 ,   " ! π " ! π f (n) U (n)ξ (x)dn = f (n)ξ (n−1 x)dn U |N (f )ξ (x) = N N   = f (n)π (x −1 nx)ξ (x)dn = f (n)(x · χ0 )(n)ξ (x)dn N

= f(x · χ0 )ξ (x).

N

Hence P π (β) = U π |N (f ) if f ∈ L1 (N) is such that β = f◦ θ . The importance of this is that for representations π1 and π2 of H which restrict to multiples of χ0 on N , if S ∈ HomG (U π1 , U π2 ), then S ∈ HomG ((U π1 , P π1 ), (U π2 , P π2 )) and, by Theorem 3.16, HomG (U π1 , U π2 ) is isomorphic to HomG (π1 , π2 ). In particular, if π is an irreducible representation of H whose restriction to N is a multiple of χ0 , then U π is an irreducible representation of G. It is also clear that when such a U π is restricted back down to N, the associated orbit in  of G.  is G(χ0 ). This leads to the following description of the dual G N Theorem 4.3 Let G be a locally compact group and N an abelian closed  be such that the normal subgroup of G such that [G : N ] < ∞. Let X ⊆ N  is a singleton. For χ ∈ X, let G χ χ intersection of X with each G-orbit in N denote the set of equivalence classes of irreducible representations of Gχ which restrict to a multiple of χ on N. Then  - χ  = indG . π : π ∈ G G χ Gχ χ ∈X

Theorem 4.3 is the prototype for the design of descriptions of the dual for more general groups G when there is a substantial abelian normal subgroup N  is available. This theorem suggests that an understanding of the G-orbits in N essential.

4.2 Orbits in the dual of an abelian normal subgroup In this section, we take a closer look at the topological properties of the collection of orbits in the dual of an abelian normal subgroup under the action of the full group. We continue with the notation of N being a closed normal abelian subgroup  the dual group of N . The action x ∈ G of a locally compact group G and N,  x·χ ∈N  is given by (x · χ)(n) = χ(x −1 nx), for n ∈ N. This gives on χ ∈ N,

146

Mackey analysis

 and the mapping an action of G as automorphisms of the abelian group N  into N  is jointly continuous. (x, χ) → x · χ from G × N  for the moment. Then G(χ) = {x · χ : x ∈ G} is the G-orbit of Fix χ ∈ N χ and Gχ = {x ∈ G : x · χ = χ} is the stabilizer of χ in G. It is easily seen that Gχ is a closed subgroup of G. Define θ : G/Gχ → G(χ ) by θ(xGχ ) = x · χ , for x ∈ G. Of course, θ is well defined and a bijection. With the quotient topology on G/Gχ , which makes G/Gχ a locally compact Hausdorff space,  θ is continuous. When θ is and the relative topology on G(χ) as a subset of N, a homeomorphism, systems of imprimitivity over G(χ ) can be moved to ones on G/Gχ . The next two examples show that this is, unfortunately, not always the case. Example 4.4 Let R+ d denote the multiplicative group of positive real numbers with the discrete topology and let G = R  R+ d , the semidirect product, where a ∈ R+ acts on R by t → at. Let N = {(t, 1) : t ∈ R}, which is an open normal d  subgroup isomorphic to R. So N is also isomorphic to R via y → χy , where χy (t, 1) = eiyt , for (t, 1) ∈ N and y ∈ R. For (s, a) ∈ G and (t, 1) ∈ N , (s, a)−1 (t, 1)(s, a) = (t/a, 1).  Consider the character χ1 . Clearly, Thus (s, a) · χy = χy/a , for all χy ∈ N. Gχ1 = N and G/Gχ1 is homeomorphic to R+ d . On the other hand, G(χ1 ) =  {χ1/a : a ∈ R+ } which, as a subset of N, is homeomorphic to the real interval d (0, ∞). Thus, θ : G/Gχ1 → G(χ1 ), (0, a)N → χ1/a fails to be a homeomorphism. Example 4.5 This example will show that the lack of second countability in Example 4.4 is not the only way that θ may fail to be a homeomorphism. We just have to look at a certain second countable subgroup of the previous example. Let Q+ d denote the group of positive rational numbers with the discrete topology and form G = R  Q+ d . Following the same notation as in Example 4.4, we now have G/Gχ1 a countable discrete set Q+ d and G(χ1 ) = {χ1/r : r ∈ (0, ∞) ∩ Q}, which is homeomorphic to the nondiscrete, nonlocally compact set (0, ∞) ∩ Q. So θ is not a homeomorphism.  Notice that in Example 4.4 the orbits were embedded quite nicely in N, whereas in Example 4.5, except for the trivial orbit, they are hopelessly entangled.

4.2 Orbits in the dual

147

It turns out that Examples 4.4 and 4.5 illustrate the only two ways in which the natural bijection θ can fail to be a homeomorphism. We note that an orbit such as G(χ) with the relative topology is a transitive left topological G-space. Proposition 4.6 Let H be a σ -compact locally compact group and let be a locally compact Hausdorff transitive left topological H -space. Fix any ω ∈

and let Hω = {t ∈ H : t · ω = ω}. Then the map φ : H /Hω → given by φ(tHω ) = t · ω, t ∈ H , is a homeomorphism. Proof Let q : H → H /Hω be the quotient map and θ = φ · q. It suffices to show that θ is an open map. Let U be an open subset of H and let t ∈ U . Select a symmetric compact neighborhood V of e such that tV 2 ⊆ U . Since H is σ -compact, there is a countable set {tn : n ∈ N} in H such that H = ∞ ∪∞ n=1 tn V . Then = ∪n=1 θ(tn V ), and each θ(tn V ) is compact and hence closed in . Since is a locally compact Hausdorff space, it is a Baire space. Thus θ(tn V ) has nonempty interior for some n. Choose v ∈ V such that θ (tn v) is in the interior of θ (tn V ). But then θ(t) = (tv −1 tn−1 ) · θ (tn v) is in the interior of (tv −1 tn−1 ) · θ (tn V ) = θ(tv −1 V ) ⊆ θ(U ). Thus θ(U ) is open, as required.  Both conditions in Proposition 4.6 are usually easy to check. As to local compactness, recall the following basic fact from elementary topology. If X is a locally compact Hausdorff space and E ⊆ X, then E is locally compact in the relative topology if and only if E is open in E. Therefore, the following definition is useful. Let X be any topological space and E a subset of X. Then E is said to be locally closed if E is open in E. This shows one of the reasons why we are particularly interested in closures . There is another important reason. We of orbits under the action of G on N saw in Section 3.1 that when a representation σ of G is restricted to N , it  into B(H(σ )) and (σ, P ) is a system defines a ∗-homomorphism P of C0 (N) of imprimitivity. Later in this chapter, we will carry out a detailed analysis of  and, as such, P . Briefly, ker P is a closed ideal in C0 (N)  : ϕ(γ ) = 0 for all γ ∈ }, ker P = {ϕ ∈ C0 (N)  Because of the relationship between σ where  is some closed subset of N. and P , it turns out that  is invariant under the action of G. Finally, if σ is . irreducible, then  is the closure of a G-orbit in N Definition 4.7 If H is a group and a left H -space, a subset  of is called H -saturated if t ·  =  for all t ∈ H . Proposition 4.8 Let H be a group and let be a left topological H -space. For ω ∈ , the following properties hold.

148

Mackey analysis

(i) H (ω) is H -saturated and hence a union of H -orbits. (ii) If H (ω) is locally closed, then H (ω ) = H (ω) for ω ∈ H (ω) \ H (ω). The proof of Proposition 4.8 is an easy exercise. The main point of the proposition is that a locally closed orbit is distinguished among all the other orbits in the closure by being open and dense. It is now time to present a number of examples so that the reader can develop . a feeling for the orbit structure in N Example 4.9 The affine group of the line is the well-behaved relative of Examples 4.4 and 4.5. Recall from Example 1.5 that G = Gaff = R  R+ ,  where a ∈ R+ acts on R by t → at. Let N = {(t, 1) : t ∈ R} and identify N  as {χy : y ∈ R} as in Example 4.4. There are three orbits in N , G(χ0 ) = {χ0 }, G(χ1 ) = {χy : y > 0}, and G(χ−1 ) = {χy : y < 0}, which we identify topologically with {0}, (0, ∞), and (−∞, 0) as subsets of R. Each orbit is locally closed and G(χ0 ) = {χ0 }, G(χ1 ) = G(χ1 ) ∪ {χ0 }, and G(χ−1 ) = G(χ−1 ) ∪ {χ0 }. The next five examples are all of the form G = R2 A R, where A is a linear transformation of R2 , with the action of t ∈ R on R2 given by t · x = etA x, for x ∈ R2 . By choosing a basis of R2 judiciously, we can ensure that the matrix of A with respect to that basis has one of the following canonical forms:           0 0 1 0 0 1 1 0 1 μ , , , , , 0 0 0 0 0 0 0 −1 −μ 1 

λ 0

1 λ



 with λ = 0, or

0 −μ

μ 0

 with μ = 0.

Let N =  {(x, 0) : x ∈ R2 }. We consider the elements of R2 as column vectors x1 x= . To avoid the transpose notation, we write  R2 for the vector space of x2  = {χy : y ∈  R2 }, where χy (x) = row vectors y = (y1 , y2 ). The dual of N is N iyx 2  e , for x ∈ R . Then G/N can be identified with R and (x, t) ∈ G acts on N solely through t. A simple calculation shows that (x, t) · χy = χye−tA ,  In the different examples, we will simplify notation by (x, t) ∈ G, χy ∈ N.  2 R2 by t · y = ye−tA . looking at the R-orbits  in R, where t ∈ R acts on  y ∈ 0 0 1 0 The case where A = is trivial, and A = gives the picture of 0 0 0 0 Example 4.9 crossed with R.

4.2 Orbits in the dual  Example 4.10 Let A =

149

   0 1 1 t tA , then e = and the action of t ∈ R 0 0 0 1

on y = (y1 , y2 ) ∈ R2 is t · y = (y1 , y2 − ty1 ). There are two types of orbit in  R2 . Points of the form (0, y2 ) are left fixed whereas the other type of orbit is any vertical line through (y1 , 0) with y1 = 0. All orbits are closed.   0 1 It is worthwhile noting that, with A = , G = R2 A R is isomor0 0 phic to the three-dimensional Heisenberg group H1 , which can be defined briefly by ⎫ ⎧⎛ ⎞ ⎬ ⎨ 1 s u H1 = ⎝ 0 1 t ⎠ : s, t, u ∈ R , ⎭ ⎩ 0 0 1 considered as a group with matrix product. Weleave  it as an exercise to verify 0 1 2 . that H1 is isomorphic to R A R when A = 0 0 The description of the irreducible representations of H1 and its higherdimensional versions, is historically tied up with some of the great mathematical and physical discoveries of the twentieth century. We will carry out this description in Section 4.4.   1 −μ Example 4.11 Let A = , with μ ∈ R. One computes that etA = −μ 1 et R(μt), where R(μt) is the rotation matrix   cos(μt) sin(μt) R(μt) = . − sin(μt) cos(μt) When μ = 0, the orbit picture consists of {0} and straight rays out from the origin. When μ = 0, we get spirals similar to those shown in Figure 4.1. In any case, {0} is an orbit and, for y = 0, R(y) = R(y) ∪ {0}. , we have for y = 0 that Or, in N G(χy ) = G(χy ) ∪ {χ0 }.  Example 4.12 Let A =

λ 1 0 λ e

tA

 with λ = 0. Then  =

eλt 0

teλt eλt

 .

150

Mackey analysis

2 for Figure 4.1 Some orbits in R Example 4.11

Figure 4.2 Some orbits in  R2 for Example 4.12

The orbits are pictured in Figure 4.2 for a typical λ = 0, and we see that, for any y = 0, we still have G(χy ) = G(χy ) ∪ {χ0 }.  0 μ Example 4.13 Let A = , so etA = R(μt) as defined in Example −μ 0 4.11. Besides {0}, the orbits are the circles centered at 0. In all cases, G/Gχ is compact and so G(χ) is compact. See Figure 4.3.  t    e 0 1 0 Example 4.14 Let A = , so etA = . Now there are three 0 −1 0 e−t kinds of orbit as shown in Figure 4.4, the trivial orbit {0}, the four half axes, and single branched hyperbolas with horizontal and vertical asymptotes. The hyperbolas are closed, while for y equal to one of (1, 0), (−1, 0), (0, 1), or (0,-1), 

G(χy ) = G(χy ) ∪ {χ0 }. Suppose now that A is a linear transformation of Rn and t ∈ R acts on R via x → etA x. Form the semidirect product G = Rn A R and let N = . As above, this is {(x, 0) : x ∈ Rn }. We are interested in the G-orbits in N n equivalent to studying the R-orbits in  R = {y = (y1 , . . . , yn )}, where t ∈ R acts on  Rn by y → ye−tA . As one might suspect, the nature of the orbits depends on the eigenvalues of A. We leave an investigation of some of the many possibilities as exercises for the reader and content ourselves with just two cases. n

4.2 Orbits in the dual

2 for Figure 4.3 Some orbits in R Example 4.13

151

2 for Figure 4.4 Some orbits in R Example 4.14

Example 4.15 Suppose 0 is the only eigenvalue of A, that is, A is a nilpotent matrix. Then G is actually a nilpotent group, but we won’t pursue this point now. A basis can be chosen for Rn so that A has matrix form with zeros in all entries except for the upper subdiagonal,where the entries are zero or one. Then etA is upper triangular with ones on the diagonal and polynomials in t above the diagonal. This means that t · y has components which are polynomials in t.  is given by a polynomial map from R into N.  It Thus, each G-orbit in N  follows that each orbit in N is closed. This is what we wanted to point out in the nilpotent case. We leave it to the reader to explore some of the lowdimensional possibilities. The next example is known as the Mautner group. It is the lowest-dimensional Lie group (dimension 5) which is not type I. We will not study what it means for a group to be of type I. Suffice it to say that, for nontype I groups the problem of describing all the irreducible representations remains intractible and, in a certain philosophical sense, will always be intractible. ⎛ ⎞ 0 θ 0 0 ⎜ −θ 0 0 0⎟ ⎟ with θ, μ ∈ R, θ = 0, μ = 0, Example 4.16 Let A = ⎜ ⎝ 0 0 0 μ⎠ 0 0 −μ 0   R(θ t) 0 . Let and θ/μ irrational. Then, in 2 × 2-block form, etA = 0 R(μt) G = R4 A R; this is known as the Mautner group. Actually, G depends on

152

Mackey analysis

θ and μ, but the important features only depend on θ/μ being irrational, and so one usually refers to the Mautner group even though it is a whole family of groups. As usual, we have the closed normal subgroup N = {(x, 0) : x ∈ R4 } ∼ = R4 .  with   We identify N R4 = {y = (y1 , y2 , y3 , y4 )}, and then the G-orbits in N 4 −tA are the same as the R-orbits in R under the action y → ye . Consider the two subspaces V1 = {(y1 , y2 , 0, 0) : y1 , y2 ∈ R} and V2 = {(0, 0, y3 , y4 ) : y3 , y4 ∈ R}. If y = 0 and y ∈ V1 or y ∈ V2 , then the orbit through y is centered at 0 lying in V1 or V2 , respectively. However, for any point y = (y1 , y2 , y3 , y4 ) which is in general position, that is, y ∈  R4 \ (V1 ∪ V2 ), let r1 = (y12 + y22 )1/2 and 2 2 1/2 r2 = (y3 + y4 ) . Then r1 > 0 and r2 > 0 and Tr1 ,r2 = {λ = (λ1 , λ2 , λ3 , λ4 ) : λ21 + λ22 = r12 and λ23 + λ24 = r22 } is a torus. The orbit through y lies on this torus. Moreover, since θ/μ is irrational, the action of R on any such torus is the famous Dedekind flow where each orbit is a dense winding line on the torus. Thus, R(y) = Tr1 ,r2 and R(y) is definitely not open in its closure. Notice that there are no distinguished orbits in Tr1 ,r2 since each of the uncountably many orbits is dense in Tr1 ,r2 . After all these examples with the real line acting, let us consider an example with R2 acting on R2 to show an orbit structure with three strata.  Example 4.17 Let st ∈ R2 act on R2 by      t  s x1 e x1 · = . x2 e s x2 t      x1 0 , x ∈ R . Now the G-orbits , : x Let G = R2  R2 and let N = 1 2 x2 s  0 2 2 −t  are the R -orbits in  R , where in N · (y1 , y2 ) = (e y1 , e−s y2 ). There are t

the four open quadrants R2 ((1, 1)), R2 ((−1, 1)), R2 ((1, −1)), and R2 ((−1, 1)), the four half axes, and {(0, 0)}. Notice that, for example, R2 ((1, 1)) = R2 ((1, 1)) ∪ R2 ((1, 0)) ∪ R2 ((0, 1)) ∪ {(0, 0)} and R2 ((1, 0)) = R2 ((1, 0)) ∪ {(0, 0)}. The other orbit closures, except {(0, 0)}, are similar to one of these two.

4.2 Orbits in the dual

153

Our final example is a discrete group, the discrete Heisenberg group. Although it is not as distinguished as its continuous analogue, Example 4.10, it has played an important role in understanding the representation theory of discrete groups and has provided an interesting example in C ∗ -algebra theory. Example 4.18 Let G = {(k, l, m) : k, l, m ∈ Z} with group product (k1 , l1 , m1 )(k2 , l2 , m2 ) = (k1 + k2 , l1 + l2 , m1 + m2 + k1 l2 ). One can easily remember the multiplication rule by viewing the elements of G as matrices of the form ⎛ ⎞ 1 k m ⎝0 1 l ⎠. 0 0 1 Thus G is a discrete subgroup of the continuous Heisenberg group. Let N = {(0, l, m) : l, m ∈ Z}.  is topologically Then N is a normal subgroup of G, isomorphic to Z2 , and N isomorphic to T2 , the 2-torus. For (z, w) ∈ T2 , the corresponding character of N is given by χ(z,w) (0, l, m) = zl wm , for l, m ∈ Z. Now G/N is isomorphic  is essentially the action of Z on T2 defined by to Z and the action of G on N k k · (z, w) = (zw , w), for k ∈ Z and z, w ∈ T. There are two types of Z-orbit in T2 . If w is a root of unity, say w N = 1 and wj = 1 for 1 ≤ j < N, then, for any z ∈ T, Z((z, w)) = {(zw j , w) : 0 ≤ j < N}. So the circle T × {w} is a union of finite orbits, each with N elements. However, if w ∈ T and w is not a root of unity, then for any z ∈ Z, Z((z, w)) = {(zw j , w) : j ∈ Z}, which is dense in the circle T × {w}. Thus T × {w} contains uncountably many distinct orbits each with the same closure T × {w}. Such orbits are definitely not locally compact (equivalently, not locally closed) and not homeomorphic to the appropriate quotient space, which is Z in this case. When we are computing the dual spaces of various examples later, we will visit some of the above examples again as well as several new orbit spaces. The variety of examples presented in this section should be sufficient for the reader to illuminate some of the more complicated arguments in the next section.

154

Mackey analysis

4.3 Mackey analysis for abelian normal subgroups For this section, we assume that G is a locally compact group and that N is an abelian closed normal subgroup of G. We describe a procedure developed by  in terms of N,  the action of G on N , and certain Mackey for constructing G  irreducible representations of the stabilizers of points in N. In order for the procedure to work, certain restrictions must be placed on the way in which N is embedded in G. In spite of these restrictions, the procedure applies in a wide range of situations that will be explored in the examples later in this chapter.  is a left topological G-space with x · χ (n) = χ(x −1 nx), for Recall that N  Let X = N/G  } be the quotient n ∈ N, x ∈ G, and χ ∈ N. = {G(χ) : χ ∈ N space endowed with the quotient topology. The subgroup N is said to be regularly embedded in G if X is almost Hausdorff; that is, every closed subset C of X contains a subset which is Hausdorff, dense in C, and relatively open in C. Remark 4.19 (i) Suppose Y is an almost Hausdorff topological space. Then any singleton {y} is locally closed in Y . Indeed, if V ⊆ {y} is Hausdorff, dense in {y} and relatively open in {y}, then V = {y}. , G(χ ) is locally (ii) If N is regularly embedded in G, then, for any χ ∈ N   closed in N . To see this, note that G(χ)/G ⊆ N/G carries the quotient topology for the action of G on G(χ). By (i), the point {G(χ )} is open in G(χ )/G and so its inverse image, the set G(χ), is open in G(χ ). If σ is a representation of G on the Hilbert space H(σ ), we saw in Example 3.3 and Proposition 3.4 that there is a system of imprimitivity (σ, Pσ ) for G  → B(H(σ )) is the unique ∗-homomorphism such  where Pσ : C0 (N) over N,  that Pσ (f) = N f (n)σ (n)dn, for all f ∈ L1 (N). The kernel of Pσ , ker Pσ = ) : Pσ (ϕ) = 0}, is a closed ideal of C0 (N ). Let {ϕ ∈ C0 (N  : ϕ(χ) = 0 for all ϕ ∈ ker Pσ }. Cσ = h(ker Pσ ) = {χ ∈ N  we say that σ |N lives on E if Cσ ⊆ E. If A is a subset of For a subset E of N,   given by N, recall that k(A) is the closed ideal of C0 (N)  : ϕ(χ) = 0 k(A) = {ϕ ∈ C0 (N)

for all χ ∈ A}.

Then ker Pσ = k(Cσ ). Lemma 4.20 With the above notation . (i) The set Cσ is a closed G-invariant subset of N (ii) If σ is irreducible then, for any proper, closed, G-invariant subset D of Cσ , Pσ (k(D))H(σ ) = {Pσ (ϕ)ξ : ϕ ∈ k(D), ξ ∈ H(σ )} is a total set in H(σ ).

4.3 Mackey analysis for abelian normal subgroups

155

 we have Proof (i) By definition Cσ is closed. For any x ∈ G and ϕ ∈ C0 (N), Pσ (Lx ϕ) = σ (x)Pσ (ϕ)σ (x)−1 , so that Lx ϕ ∈ ker Pσ if and only if ϕ ∈ ker Pσ . ). Thus x −1 · For χ ∈ Cσ and x ∈ G, ϕ(x −1 · χ) = Lx ϕ(χ) for all ϕ ∈ C0 (N χ ∈ Cσ for all x ∈ G. (ii) Since D is a proper closed subset of Cσ , there exist χ0 ∈ Cσ \ D and ψ ∈ k(D) such that ψ(χ0 ) = 1. Then Pσ (ψ) = 0 by definition of Cσ . Let HD denote the closed linear span of {Pσ (ϕ)ξ : ϕ ∈ k(D), ξ ∈ H(σ )}. Since Pσ (ψ) = 0 and ψ ∈ k(D), we have HD = {0}. But HD is a σ -invariant subspace of H(σ ). To see this, let x ∈ G, ϕ ∈ k(D), and ξ ∈ H(σ ). Then simply note that σ (x)Pσ (ϕ)ξ = Pσ (Lx ϕ)σ (x)ξ and Lx ϕ ∈ k(D) because D is G-invariant. Since σ is irreducible, it follows that HD = H(σ ), which shows (ii).  Proposition 4.21 Let G be a locally compact group, and suppose that N is an abelian closed normal subgroup of G that satisfies one of the following two conditions. (a) N is regularly embedded in G.  is second countable and all G-orbits in N  are locally closed. (b) N  such that If σ is an irreducible representation of G, then there exists χ ∈ N σ |N lives on G(χ). Moreover, Pσ (k(G(χ) \ G(χ)))H(σ ) is total in H(σ ). →N /G denote the quotient Proof Suppose first that (a) holds. Let p : N map. Now Cσ is closed and G-invariant, so Cσ = p −1 (p(Cσ )). Thus p(Cσ ) is  a closed subset of N/G. By hypothesis (a) there exists a subset V of p(Cσ ) which is Hausdorff, dense in p(Cσ ), and open in p(Cσ ). If V is not a singleton, then there exist disjoint nonempty open subsets V1 and V2 of p(Cσ ). Then W1 = p −1 (V1 ) and W2 = p −1 (V2 ) are disjoint open G-invariant subsets of Cσ . Let Di = Cσ \ Wi for i = 1, 2. Each Di is a proper, closed, G-invariant subset of Cσ . By Lemma 4.20, Pσ (k(Di ))H(σ ) is total in H(σ ) for i = 1, 2. But D1 ∪ D2 = Cσ . Thus ker Pσ = k(Cσ ) = k(D1 ) ∩ k(D2 ) ⊆ k(D1 )k(D2 ). Hence Pσ (ϕ1 )Pσ (ϕ2 ) = 0, for all ϕ1 ∈ k(D1 ) and ϕ2 ∈ k(D2 ). Since Pσ (k(D2 ))H(σ ) is total in H(σ ), this means that Pσ (ϕ1 )H(σ ) = {0}, for all ϕ ∈ k(D1 ), contradicting its total nature. Therefore, V must be a singleton, say V = {G(χ )}. That is, G(χ) is open and dense in Cσ . Therefore σ |N lives on G(χ ) and G(χ) \ G(χ) is a proper, closed, G-invariant subset of Cσ . Thus Pσ (k(G(χ ) \ G(χ)))H(σ ) is total in H(σ ). . So Now suppose that (b) holds. Then G(χ) is open in G(χ ) for each χ ∈ N it suffices to show that G(χ) = Cσ , for some χ ∈ Cσ , and complete the proof as above.

156

Mackey analysis

To see that there exists a χ ∈ Cσ with G(χ) = Cσ , let U be a countable basis for the topology of Cσ . Of course we can assume that U = ∅ for each U ∈ U. For U ∈ U, let G(U ) = {x · χ : x ∈ G, χ ∈ U }. If G(U ) is not all of Cσ , then W = Cσ \ G(U ) and G(U ) are disjoint, nonempty, open G-invariant subsets of Cσ . Taking complements, D1 = G(U ) and D2 = Cσ \ G(U ), leads to the same contradiction using Lemma 4.20 as in the first part of the proof. Thus G(U ) is a dense open subset of Cσ , for each U ∈ U. Since Cσ is a Baire space and U is countable, ∩{G(U ) : U ∈ U} = ∅. Let χ ∈ ∩{G(U ) : U ∈ U}. Then  G(χ ) ∩ U = ∅, for all U ∈ U. This implies that G(χ) is dense in Cσ .  If either condition (a) or (b) in Proposition 4.21 is satisfied, then G-orbits in N  G(χ) is a locally compact transitive Gare locally closed. Thus, fixing χ ∈ N, space. Suppose that, in addition, G is σ -compact. Then the map θ : G/Gχ → G(χ ), given by θ (xGχ ) = x · χ, is a homeomorphism by Proposition 4.6. This means that, in either of these situations, the technical conditions in the following theorem are automatically satisfied. Theorem 4.22 Let G be a locally compact group and N an abelian closed normal subgroup of G. Let σ be an irreducible unitary representation of G.  such that σ |N lives on G(χ ), and suppose that the Suppose there exists χ ∈ N map θ : xGχ → x · χ from G/Gχ to G(χ) is a homeomorphism. Then there exists an irreducible representation π of Gχ with the following properties. (i) π|N is a multiple of χ. (ii) σ is equivalent to indG Gχ π. Moreover, π is uniquely determined up to equivalence. ) such that Proof Let Pσ be the nondegenerate ∗-representation of C0 (N Pσ (f) = N f (n)σ (n)dn, for all f ∈ L1 (N). Then (σ, Pσ ) is a system of  (Proposition 3.4). Let Cσ = h(ker Pσ ). The assumpimprimitivity for G over N tion that σ |N lives on G(χ) means that G(χ) = Cσ . The assumption that θ is a homeomorphism implies that G(χ) is locally compact and hence open in Cσ . ) onto C0 (Cσ ) with kernel The map ϕ → ϕ|Cσ is a homomorphism of C0 (N equal to ker Pσ . Thus there is a well-defined nondegenerate ∗-representation Pσ ). Now G(χ ) is open and of C0 (Cσ ) such that Pσ (ϕ|Cσ ) = Pσ (ϕ), for ϕ ∈ C0 (N dense in Cσ , so any ψ ∈ C0 (G(χ)) extends trivially to a function, also denoted ψ, in C0 (Cσ ) that vanishes on Cσ \ G(χ). Thus C0 (G(χ )) can be considered as a closed ideal of C0 (Cσ ). Since D = Cσ \ G(χ) is a proper, closed, G-invariant subset of Cσ , Lemma 4.20(ii) tells us, after moving from Pσ to Pσ , that Pσ (C0 (G(χ )))H(σ ) is total in H(σ ). So, by restriction to the ideal C0 (G(χ )), Pσ is a nondegenerate

4.3 Mackey analysis for abelian normal subgroups

157

∗-representation of C0 (G(χ)) and (σ, Pσ ) is a system of imprimitivity for G over G(χ). The map θ∗ : C0 (G/Gχ ) → C0 (G(χ)), given by θ∗ α = α ◦ θ, for α ∈ C0 (G/Gχ ), is a ∗-isomorphism. Let Pσ = Pσ ◦ θ∗ , then by Proposition 3.9, (σ, Pσ ) is a system of imprimitivity for G over G/Gχ . Now we can apply the imprimitivity theorem (Theorem 3.17) to conclude that there exists π a representation π of Gχ such that (σ, Pσ ) is equivalent to (indG Gχ π, P ), the system of imprimitivity induced by π. Let W : H(σ ) → H(indG Gχ π) be the unitary map such that  π W σ (x) = indG Gχ π(x)W and W Pσ (α) = P (α)W,

for all x ∈ G and α ∈ C0 (G/Gχ ). If g ∈ L1 (N ) and ξ ∈ F(G, π) then, by Proposition 2.41,     G G indGχ π |N (g)ξ (x) = g(n) indGξ π(n)ξ (x)dn = g(n)ξ (n−1 x)dn N N  = g(n)π(x −1 n−1 x)(ξ (x))dn N

= (x · π|N )(g)(ξ (x)), g |G(χ ) ). Then for all x ∈ G. If  g vanishes on G(χ) \ G(χ), then let β = θ∗−1 (  Pσ (β) = Pσ ( g ) = σ |N (g). Therefore,   P π (β) = W Pσ (β)W −1 = W σ |N (g)W −1 = indG Gχ π |N (g). By the definition of P π , for all ξ ∈ F(G, π) and x ∈ G, g (x · χ)ξ (x). [P π (β)ξ ](x) = β(xGχ )ξ (x) =  1 g (G(χ ) \ G(χ )) = {0}, and x ∈ Hence, for ξ ∈ F (G,  π), g ∈ L (N)  such that  G G,  g (x · χ )ξ (x) = indGχ π(g)ξ (x) = (x · π |N )(g)[ξ (x)]. Now the set {ξ (e) : ξ ∈ F(G, π)} is dense in H(π ) by Lemma 2.24(ii). Thus, for any v ∈ H(π ) and g vanishes on G(χ) \ G(χ), we have χ (g)v =  g (χ)v = g ∈ L1 (N ) such that  g (χ) = 1 and  g vanishes on G(χ ) \ G(χ ). π|N (g)v. Fix g ∈ L1 (N) such that  Then, for any f ∈ L1 (N) and v ∈ H(π),

χ (f )v = χ (f ∗ g)v = π |N (f ∗ g)v = π |N (f )π |N (g)v = π |N (f )v. Thus π|N is a multiple of χ. This completes the proof of (i) and (ii). Suppose that τ is another irreducible representation of Gχ such that τ |N is G a multiple of χ and σ = indG Gχ τ . Let U : H(σ ) → H(indGχ τ ) be a unitary  τ map that intertwines σ and indG Gχ τ . Then U intertwines Pσ and P . Indeed, let 1 −1 g |G(χ ) ∈ C0 (G(χ)) and let β = θ∗  g |G(χ ) . Then, for g ∈ L (N ) be such that 

158

Mackey analysis

any ξ ∈ F(G, π) and x ∈ G, [U Pσ (β)U −1 ξ ](x) = [U σ |N (g)U −1 ξ ](x) = [(indG Gχ τ )|N (g)ξ ](x) = (x · τ |N )(g)[ξ (x)] =  g (x · χ)ξ (x) = β(x · χ)ξ (x) = [P τ (β)ξ ](x). Notice that the set of all such β is dense in C0 (G/Gχ ). It follows that (τ, P τ ) is equivalent to (σ, Pσ ) which is equivalent to (π, P π ), and this implies that τ is equivalent to π. This establishes the uniqueness part of the theorem.  . If σ and τ are equivalent irreducible representations Let O be a G-orbit in N  we say σ |N lives of G and σ |N lives on O, then τ |N also lives on O. If σ ∈ G, on O if any of its concrete realizations restricted to N lives on O.  : σ |N lives on O}. Select χ ∈ O. We can think of N  as O = {σ ∈ G Let G a Gχ -space, and then {χ } is a Gχ -orbit. So we write χ χ = {π ∈ G χ : π|N lives on {χ }}. G χ , π|N lives on {χ} if and only if π|N is a multiple of Lemma 4.23 For π ∈ G χ.  on H(π ) given Proof Let Pπ be the nondegenerate ∗-representation of C0 (N) g ) = π |N (g), for g ∈ L1 (N). If π|N is a multiple of χ, then clearly by Pπ ( h(ker Pπ ) = {χ} and π|N lives on {χ }.  Pπ (ϕ) = 0 Conversely, suppose π|N lives on {χ}. Then, for ϕ ∈ C0 (N),  if and only if ϕ(χ) = 0. This means that Pπ (C0 (N )) is a one-dimensional  = CIH(π ) . ∗-subalgebra of B(H(π)). Since Pπ is nondegenerate, Pπ (C0 (N))  we have π |N (L1 (N )) = CIH(π ) as Since π|N (L1 (N )) is dense in Pπ (C0 (N)), well. Pick g ∈ L1 (N) such that π |N (g) = IH(π) . Then, for any f ∈ L1 (N ), π|N (f − g ∗ f ) = π |N (f ) − π |N (g)π |N (f ) = 0, ). It follows hence f−  g f = ker Pπ . Thus ϕ −  g ϕ ∈ ker Pπ , for all ϕ ∈ C0 (N  that the linear map from C0 (N) into B(H(π)) given by ϕ → Pπ (ϕ) − ϕ(χ )IH(π )  Thus Pπ (ϕ) = ϕ(χ)IH(π) , for all ϕ ∈ C0 (N ). Hence is 0 on all of C0 (N).  π|N (f ) = χ (f )IH(π) , for all f ∈ L1 (N). So π|N is a multiple of χ .  and χ ∈ O, if θ : xGχ → x · χ is a homeomorphism, For a G-orbit O in N χ χ . This turns out to O into G then Theorem 4.22 gives a one-to-one map of G be onto. Theorem 4.24 Let G be a locally compact group and N an abelian closed  and let χ ∈ O. Suppose normal subgroup of G. Let O be a G-orbit in N that θ : xGχ → x · χ is a homeomorphism of G/Gχ onto O. Then the map χ O π → indG Gχ π defines a bijection between Gχ and G .

4.3 Mackey analysis for abelian normal subgroups

159

Proof Let π be an irreducible representation of Gχ whose restriction to N is a multiple of χ , and let σ = indG Gχ π. We want to show that σ is irreducible and lives on O. As usual, let Pσ be the nondegenerate ∗-homomorphism of  such that Pσ ( g ) = π |N (g) for all g ∈ L1 (N ). If S ∈ B(H(σ )) satisfies C0 (N) ). Sσ (x) = σ (x)S for all x ∈ G, then SPσ (ϕ) = Pσ (ϕ)S for all ϕ ∈ C0 (N With calculations similar to those in the proof of Theorem 4.22 we g )ξ (x) = σ |N (g)ξ (x) = have, for g ∈ L1 (N), ξ ∈ F(G, π), and x ∈ G, Pσ ( g ◦ θ ∈ C0 (G/Gχ ) and (P π ( g◦  g (x · χ )ξ (x). If  g vanishes on O \ O, then  θ )ξ )(x) =  g (x · χ)ξ (x), for all ξ ∈ F(G, π) and x ∈ G. Thus P π ( g ◦ θ) =  that vanishes on O \ O can be approximated in norm g ). Any ϕ ∈ C0 (N) Pσ ( by  g , where g ∈ L1 (N) and  g vanishes on O \ O. Thus P π (ϕ ◦ θ ) = Pσ (ϕ), for all ϕ ∈ h(O \ O). Now, for any α ∈ C0 (G/Gχ ), there is a ϕ ∈ h(O \ O) such that α = ϕ ◦ θ . Thus S ∈ HomG (σ ) implies SP π (α) = SPσ (ϕ) = Pσ (ϕ)S = P π (α)S. G Recalling that σ = indG Gχ π, we have shown that HomG (indGχ π ) = G π ∼ HomG ((indG Gχ π, P )). It follows from Theorem 3.16 that HomG (indGχ π ) = G HomG (π ). So irreducibility of π implies that indGχ π is irreducible.  and ϕ vanishes on O, then Pσ (ϕ) = P π (ϕ ◦ θ ) = 0. Moreover, if ϕ ∈ C0 (N)  \ O and ϕ ∈ C0 (N)  is such that ϕ(χ0 ) = 0 while ϕ vanishes Thus, if χ0 ∈ N on O, then Pσ (ϕ) = 0 or ϕ ∈ ker Pσ . Thus χ0 ∈ Cσ . Therefore Cσ ⊆ O, which means that indG Gχ π = σ lives on O. Combining this with Theorem 4.22, including the uniqueness statement in G Theorem 4.22 and the fact that π1  π2 implies indG Gχ π1  indGχ π2 , we get χ O that the map π → indG  Gχ π defines a bijection between Gχ and G . In order to facilitate the formulation of a global version of Theorems 4.22 and 4.24, we introduce a term for those closed normal abelian subgroups for which the hypotheses of these two theorems hold. Definition 4.25 Let G be a locally compact group. A closed normal abelian subgroup N of G is called a Mackey-compatible subgroup if  and  π |N lives on a G-orbit in N (a) for each π ∈ G,  xGχ → x · χ is a homeomorphism of G/Gχ with G(χ ). (b) for each χ ∈ N, Remark 4.26 Although the condition of being Mackey compatible is not intrinsic to the topological and algebraic relationship between N and G, there are a variety of broadly applicable and easily checked sets of conditions which imply Mackey compatibility. As examples, we point out the following. (1) If N is regularly embedded in G and G/N is σ -compact, then Proposition 4.21, condition (a), and Proposition 4.6 imply that N is Mackey compatible.

160

Mackey analysis

(2) Proposition 4.21, condition (b), and Proposition 4.6 imply that N is  is second countable, all G-orbits in N  are locally Mackey-compatible if N closed, and G/N is σ -compact. (3) If G/N is compact, then N is Mackey-compatible, since a compact group acting on a locally compact Hausdorff space always results in a Hausdorff quotient space. Thus, compactness of G/N implies both conditions in part (1) of this remark. Together, Theorems 4.22 and 4.24 give a picture that is in some sense complete. Theorem 4.27 Let G be a locally compact group. Suppose that N is an abelian closed normal Mackey-compatible subgroup of G. Then 5  is the disjoint union O∈N/G O ; (i) G G  χ O = {indG   (ii) G Gχ π : π ∈ Gχ }, for O ∈ N/G and χ ∈ O.  is a subset X of N  such that X ∩ O is a A cross-section of the G-orbits in N  singleton for each O ∈ N/G. In almost any example with a Mackey-compatible subgroup N, a cross-section is easy to identify. It is therefore convenient to reformulate Theorem 4.27 assuming a cross-section is fixed. Theorem 4.28 Let G be a locally compact group. Suppose that N is an abelian closed normal Mackey-compatible subgroup of G and that X is a cross-section  Then of the G-orbits in N. χ  = {indG G Gχ π : π ∈ Gχ , χ ∈ X}.  when the Although Theorem 4.28 provides a meaningful understanding of G, hypotheses hold, there is still an issue to be resolved. How does one construct χ χ for a given χ ∈ N?  We will answer this for groups which split as semidirect G products now and deal with the nonsplit case in a later section. Let N be an abelian locally compact group and let H be a locally compact group which acts on N . Let G = N  H = {(n, h) : n ∈ N, h ∈ H } with product given by (n1 , h1 )(n2 , h2 ) = (n1 (h1 · n2 ), h1 h2 ), the semidirect product. Without further comment, we identify N and H with the obvious closed subgroups of G. Then N is a normal subgroup of G. In this situation, it is reasonable to formulate all of the G-orbit information . If in terms of H . So h · χ (n) = χ(h−1 · n) for n ∈ N, h ∈ H , and χ ∈ N Hχ = {h ∈ H : h · χ = χ} and H (χ) = {h · χ : h ∈ H },  Therefore N /G = then Gχ = N  Hχ and G(χ) = H (χ ), for all χ ∈ N.   N /H. Moreover, for any χ ∈ N, G/Gχ can be identified with H /Hχ via the mapping (n, h)Gχ → hHχ for h ∈ H and n ∈ N.

4.3 Mackey analysis for abelian normal subgroups

161

. Recall that G χ χ denoted the class of irreducible representations Fix a χ ∈ N of Gχ whose restriction to N is a multiple of χ. For the groups currently under χ χ is easy to describe. In fact, if π is any representation of consideration, G Hχ , define a map χ × π from Gχ into the unitary operators on H(π) by (χ × π )(n, h) = χ(n)π(h), for h ∈ Hχ , n ∈ N. Since k · χ = χ for k ∈ Hχ , if (n, h), (m, k) ∈ N  Hχ , then (χ × π )((n, h)(m, k)) = (χ × π )((nh · m, hk)) = χ (n)(h−1 · χ )(m)π (hk) = χ(n)χ(m)π (h)π(k) = (χ × π )(n, h)(χ × π)(m, k). On the other hand, if σ is a representation of N  Hχ whose restriction to N is a multiple of χ, let π = σ |Hχ . Then σ (n, h) = χ(n)π(h) = (χ × π)(n, h). Since χ (n) is a scalar for all n ∈ N, intertwining spaces depend only on the χ χ . χ with G H -component. Thus, π → χ × π defines a bijection of H Now it is a simple matter to translate Theorem 4.28 and arrive at the following description of the dual of N  H . Theorem 4.29 Let N be a locally compact abelian group and H a locally compact group acting on N such that N is Mackey compatible in N  H . Let  Then X be a cross-section of the H -orbits in N.   NH χ , χ ∈ X . (χ × π) : π ∈ H N  H = indNH χ Theorems 4.27 and 4.29 effectively describe a procedure by which the dual of a group with an appropriate abelian normal subgroup can be completely constructed. This procedure is often called Mackey theory or, more informally, the Mackey machine or Mackey’s little group method for abelian normal subgroups. The next two sections are devoted almost entirely to examples of Mackey theory in action, but we end this section with an example of a semidirect product of two small finite groups to illustrate the theory. Example 4.30 Let N = Z4 = {0, 1, 2, 3}, with addition mod 4. Let H = Z2 = {1, −1}, with (−1)2 = 1. Let H act on N by setting −1 · n = −n mod 4. Then all other actions are determined. Form G = N  H . Let U4 = {z ∈ T : z4 = 1} = {1, i, −1, −i}, the group of fourth roots of unity. For each u ∈ U4 ,  = {χu : u ∈ U4 }. The action of H on define χu (n) = un for n ∈ N . Then N  N is given by −1 · χu = χu , for u ∈ U4 . The set X = {1, i, −1} meets each  exactly once. The stabilizer of i is trivial, while that of both H -orbit in N  consists of two characters, there are two one1 and −1 is H . Since H dimensional representations of G associated with each of 1 and −1 in X. They are τ, σ1,−1 , σ−1,1 , and σ−1,−1 where, for (n, ±1) ∈ G, τ (n, ±1) = 1, σ1,−1 (n, ±1) = ±1, σ−1,1 (n, ±1) = (−1)n , and σ−1,−1 (n, ±1) = (−1)n (±1).

162

Mackey analysis

We also have indG N χi , a two-dimensional irreducible representation of G. We leave it as a small exercise to work out an explicit expression for indG N χi . Then, G  by Theorem 4.29, G = {τ, σ1,−1 , σ−1,1 , σ−1,−1 , indN χi }.

4.4 Examples: some solvable groups For many groups, which are semidirect products, Theorem 4.29 enables us to give an explicit description of the dual. That is, we can give formulae for a parametrized set of irreducible representations that are mutually inequivalent and such that any other irreducible representation is equivalent to one in this set. We will continue the simplification of writing π for either the representation π or its equivalence class [π] when there is no danger of confusion. In this section, we present a variety of examples including the nontrivial three-dimensional connected solvable Lie groups which were introduced in Section 4.2, and a selection of nilpotent (hence solvable) Lie groups including the Heisenberg groups. It will be observed that every irreducible representation of the nilpotent examples can be obtained by inducing a character (a onedimensional representation) of some subgroup. This is actually true for any simply connected nilpotent Lie group and we take the opportunity to present a proof using Mackey theory. We start with a familiar group. Example 4.31 Let G = Gaff = R  R+ , the affine group of the line. So G = {(t, a) : a, t ∈ R, a > 0} with product (t1 , a1 )(t2 , a2 ) = (t1 + a1 t2 , a1 a2 ). With  = {χr : r ∈ R}, where χr (t) = eirt . As in Example 4.9 there are N = R, N  corresponding to the set of positive numbers, the set of three G-orbits in N  is second countable and orbits are locally negative numbers, and {0} in R. So N closed (clearly N is also regularly embedded in G). Since R+ is σ -compact,  that N is Mackey compatible. We can select X = {χ0 , χ−1 , χ1 } as a set in N intersects each orbit once. Then Gχ0 = G and the other stability subgroups −1 and U 1 on are trivial. Let U ±1 = indG N χ±1 . Then we can realize both U L2 (R+ ), where R+ carries its Haar measure da . For (t, a) ∈ G, f ∈ L2 (R+ ), a + and b ∈ R , U ±1 (t, a)f (b) = e±ib

−1

t

f (a −1 b).

 = {ωs : s ∈ R} ∪ {U −1 , U 1 }, where each Theorem 4.29 then tells us that G ωs is the character of R+ pulled back to G defined by ωs (t, a) = a is for all (t, a) ∈ G, s ∈ R. In Section 2.5, the trivial character of the subgroup {(0, a) : a > 0} was induced to a representation ρ of G and it was shown that ρ = ρ + ⊕ ρ − , where

4.4 Examples: some solvable groups

163

ρ + and ρ − are irreducible and inequivalent infinite-dimensional representations. Thus, one must be equivalent to U −1 and one to U 1 . We leave it to the reader to match the equivalent representations. Example 4.32 We return to Example 2.9, where G = R  Z = {(t, n) : t ∈ R, n ∈ Z} with group product given by (t, n)(s, m) = (t + 2n s, n + m). Notice that (t, n) → (t, 2n ) identifies this G with a closed subgroup of Gaff . Let N =  G(χr ) =  = {χr : r ∈ R} as in the previous example. For χr ∈ N, R and N  {χ2−n r : n ∈ Z}. Thus the orbits in N are locally closed. Clearly N is Mackey compatible. If r = 0 then Gχr = G, while if r = 0 then Gχr = N. Any irreducible representation of G = Gχ0 which is a multiple of the trivial character χ0 when restricted to N defines a character of G/N = Z and so must χ0 = {ψz : z ∈ T}. For the be of the form ψz (t, n) = zn for some z ∈ T. Thus G other orbits, we pick one representative from each G-orbit, say {χr : 1 ≤ |r| < χr = {U χr }, where U χr is the 2}. For each χr in this set, Gχr = N and hence G 2 realization of the induced representation on l (Z) given by U χr (t, n)η(m) = −m ei(2 r)t η(m − n), for m ∈ Z, η ∈ l 2 (Z), and (t, n) ∈ G.  = {ψz : z ∈ T} ∪ {U χr : 1 ≤ |r| < 2}. Theorem 4.29 tells us that G In Section 4.2, the orbit spaces were studied for the various three-dimensional solvable groups which arise as a semidirect product of R acting on R2 . Each such group was of the form R2 A R with group product given by (x, t)(y, s) = (x + etA y, t + s). Different interesting examples arose from choices of A as in Examples 4.10–4.14. Example 4.10 is the three-dimensional Heisenberg group, H1 , and will be included below when we discuss two different classes of nilpotent groups, each of which includes H1 . For each of Examples 4.11–4.14, we can use the following notation. We write  2 R = {χγ : γ = (γ1 , γ2 ), γ1 , γ2 ∈ R}, where χγ (x) = exp iγ x = exp i(γ1 x1 +   x1 ∈ R2 . The representation of R2 A R obtained by inducγ2 x2 ), for x = x2 ing χγ from R2 (identified with the obvious normal subgroup) can be realized on L2 (R) via U χγ (x, t)f (s) = eiγ e

−sA

x

f (s − t),

for s ∈ R, f ∈ L2 (R), (x, t) ∈ R2 A R. We will also use a uniform notation through these examples for the characters lifted from the quotient group R. For u ∈ R, let σu (x, t) = exp iut, for all (x, t) ∈ R2 A R. It is clear that the orbits are locally closed in each of these examples and, since R2 is second countable and R is σ -compact, that N = R2 is Mackey compatible as a subgroup of R2 A R. The reader should consult the picture

164

Mackey analysis

of each orbit space in Figures 4.1–4.4 in Section 4.2 to help in verifying our 2 . claims that certain sets form a cross-section of orbits in R   1 −μ Example 4.33 Let A = , with μ ∈ R. Then etA = et R(μt), −μ 1 where R(μt) is the rotation matrix   cos(μt) sin(μt) R(μt) = . − sin(μt) cos(μt) 2 intersects If S 1 = {γ ∈ R2 : γ12 + γ22 = 1}, then every nonzero R-orbit in R 1 S exactly once and with trivial stability subgroup. Thus, 2  R = {σ : u ∈ R} ∪ {U χγ : γ ∈ S 1 }. R A u   λt   teλt e λ 1 tA . Let Example 4.34 Let A = with λ = 0. Then e = 0 λ 0 eλt

= {(0.1), (0, −1), (γ1 , 0) : γ1 = 0}. The reader can check that each nonzero 2 intersects exactly once and the stability subgroups are trivial. R-orbit in R Thus, 2  R = {σ : u ∈ R} ∪ {U χγ : γ ∈ }. R A u     t 1 0 e 0 tA Example 4.35 Let A = , so e = . Let 0 −1 0 e−t

 = {(1, 0), (0, 1), (−1, 0), (0, −1), (γ1 , γ1 ), (γ1 , −γ1 ) : γ1 = 0}. 2 intersects  exactly once and with trivial Again, every nonzero R-orbit in R stability subgroup. Thus 2  R = {σ : u ∈ R} ∪ {U χγ : γ ∈ }. R A u   0 μ Example 4.36 Let A = , with μ = 0. To simplify the arithmetic, −μ 0 we take μ = 2π. So etA = R(2π t) as defined in Example 4.33, where R(2π t) is the matrix which rotates through the angle 2π t. Besides {0}, the R-orbits are the circles centered at 0. So X = {(r, 0) : r ≥ 0} provides a cross-section of the R-orbits; but now the stability subgroups are nontrivial and we have to introduce new notation to describe the dual. Using G = R2 A R, for r > 0, write Gr = Gχ(r,0) . Then Gr = {(x, n) : x ∈ R2 , n ∈ Z}. As discussed before χr (r,0) = {ϕz × χ(r,0) : z ∈ T}, where for z ∈ T, Theorem 4.29, G   ϕz × χ(r,0) (x, n) = zn eirx1 ,

4.4 Examples: some solvable groups

165

  z,r z,r for all (x, n) ∈ Gr . Let us denote indG can be Gr ϕz × χ(r,0) by U . Then U 2 realized on L (T) as follows. Note that, for v ∈ R, v denotes the least integer greater than or equal to v. Define q : G → T by q(x, t) = e2πit . Then q is a continuous homomorphism with ker q = Gr . Thus, we can identify G/Gr with T and q is the quotient homomorphism. Select the cross-section γ : T → G to be γ (w) = (0, s) if 0 ≤ s < 1 and w = e2πis . For Realization III of the induced representation, we need to compute γ (w)−1 (x, t)γ ((x, t)−1 · w) for w ∈ T, (x,  t) ∈ G. Suppose w = e2πis , 0 ≤ s < 1. Then γ ((x, t)−1 · w) = γ e2πi(s−t) = (s − t)(mod 1) and γ (w)−1 (x, t)γ ((x, t)−1 · w) = (R(−2π s)x, (t − s)). Therefore, for (x, t) ∈ G, f ∈ L2 (T), and w ∈ T with w = e2πis , 0 ≤ s < 1,   ϕz × χ(r,0) (R(−2π s)x, (t − s)) = z(t−s) ei(r,0)R(−2πs)x . Thus U z,r (x, t)f (e2πis ) = z(t−s) ei(r,0)R(−2πs)x f (e2πi(s−t) ), where 0 ≤ s < 1. Finally, we get  = {σu : u ∈ R} ∪ {U z,r : z ∈ T, r > 0}. G Example 4.37 We now consider certain semidirect products of Rn with R, n ≥ 2. Let G be the semidirect product G = Rn A R defined by the matrix ⎛ ⎞ 0 1 0 ··· 0 ⎜0 0 1 ... 0⎟ ⎜ ⎟ ⎜ .. ⎟ , A = ⎜ ... ... ... ⎟ . ⎜ ⎟ ⎝0 0 0 ··· 1⎠ 0

0

0

···

0

with group product given by (x, t)(y, s) = (x + etA y, t + s). Note that elements of Rn are considered as column vectors here. It is easily verified that Aj = 0 for j ≥ n and that, for 2 ≤ j ≤ n − 1, Aj is the matrix whose (k, j + k)th entry, 1 ≤ k ≤ n − j , is one and all other entries are zero. Thus ⎛ ⎞ t2 t n−1 1 t 2! · · · (n−1)! ⎜ t n−2 ⎟ · · · (n−2)! ⎜0 1 t ⎟ ⎜ ⎟ .. .. .. .. ⎟ . etA = ⎜ ⎜. . . . ⎟ ⎜ ⎟ ⎝0 0 0 ··· t ⎠ 0 0 0 ··· 1 Notice that if Zj (G), j = 1, 2 . . ., denotes the ascending central series of G, then Zj (G) = {(x, 0) : x = (x1 , . . . , xn ) ∈ R, xk = 0 for k > j }, for 1 ≤ j ≤

166

Mackey analysis

n − 1, and Zn (G) = G. Thus G is an n-step nilpotent group. Also observe that, when n = 2, G is isomorphic to the three-dimensional Heisenberg group. n be the For γ = (γ1 , . . . , γn ) ∈ Rn , written as a row vector, let χγ ∈ R n  character χγ (x) = exp iγ x. Then the action of t ∈ R on R is given by t · χγ = χγ etA . It follows that G(χγ ) = {χγ } and Gχγ = G whenever γ1 = . . . = γn−1 = 0, whereas, if γl = 0 for some 1 ≤ l ≤ n − 1, then G(χγ ) = {γ etA : t ∈ R} is n and Gχγ = Rn . Clearly, in this a one-dimensional closed submanifold of R tA case for each such γ , the map t → γ e is a homeomorphism between R and G(χγ ). The hypotheses of Theorem 4.29 hold for this semidirect product. For any γ of the form (0, . . . , 0, a), G(χγ ) = {χγ } and all of the acting R stabilizes χγ . If we let ψb (t) = eibt , t ∈ R, and σa,b = ψb × χγ for γ = (0, . . . , 0, a), then σa,b (x, t) = ei(axn +bt) , for all (x, t) ∈ G. These are the one-dimensional irreducible representations of G. Let  = {γ ∈ Rn : γl = 0 for some 1 ≤ l ≤ n − 1}. Then, for each γ ∈ , the induced representation U χγ is an infinite-dimensional irreducible representation of G. To select inequivalent U χγ , we need to identify a subset S ⊂  which meets each R-orbit in exactly one point. To that end, observe first that if γ = (γ1 , . . . , γn ) and ν = (ν1 , . . . , νn ) are elements of  such that ν ∈ G(γ ) and k and l are minimal with the property , we see that γk = 0 and νl = 0, then k = l and νk = γk . Choosing t = − γγk+1 k  , . . . , γn ). For that G(γ ) contains an element of the form (0, . . . , 0, γk , 0, γk+2 1 ≤ k ≤ n − 1, let Sk = {(0, . . . , 0, γk , 0, γk+2 , . . . , γn ) : γk , γk+2 , . . . , γn ∈ R, γk = 0}. Now, let γ , ν ∈ Sk for some k and suppose that ν ∈ G(γ ). Then, for some t ∈ R, νl =

l  j =k

t l−j γj (k ≤ l ≤ n). (l − j )!

Then νk = γk , and since νk+1 = γk+1 = 0, it follows that t = 0 and hence ν = γ . Thus the set S = ∪n−1 k=1 Sk is a cross-section for the nontrivial G-orbits in Rn . For each γ ∈ , we can realize U χγ on L2 (R) by, for (x, t) ∈ G and f ∈ 2 L (R), U χγ (x, t)f (s) = eiγ e

−sA

x

f (s − t),

4.4 Examples: some solvable groups

167

for all s ∈ R. Then  = {σa,b : a, b ∈ R} ∪ {U χy : y ∈ S}. G Example 4.38 For n ≥ 1, let Hn be the (2n + 1)-dimensional Heisenberg group. By definition, Hn is the set Rn × Rn × R with multiplication (x, y, t)(x  , y  , t  ) = (x + x  , y + y  , t + t  + x, y  ), x, x  , y, y  ∈ Rn , t, t  ∈ R, where ·, · denotes the Euclidean scalar product on Rn to avoid confusion here. Note that Hn is isomorphic to the multiplicative group of upper-triangular real (n + 1) × (n + 1)-matrices of the form ⎞ ⎛ 1 x1 · · · xn t ⎜ 1 0 · · 0 y1 ⎟ ⎟ ⎜ ⎜ · · · ⎟ ⎟ ⎜ ⎟ ⎜ · · · ⎟, ⎜ ⎟ ⎜ ⎜ · 0 · ⎟ ⎟ ⎜ ⎝ 1 yn ⎠ 1 where t, xj , yj ∈ R, 1 ≤ j ≤ n, and all entries not indicated are zero. Let N = {(x, y, t) ∈ Hn : x = 0} and A = {(x, y, t) ∈ Hn : y = t = 0}. Then N and A are abelian subgroups of Hn , N is normal, and Hn is isomorphic to the semidirect product N  A, where A acts on N by (x, 0, 0) · (0, y, t) = (0, y, t + x, y). We identify N with Rn × R and A with Rn . For ξ ∈ Rn and λ ∈ R, define χξ,λ (y, t) = ei(λt+y,ξ ) , for all (y, t) ∈ N . The action of x ∈ Rn on χξ,λ is computed to be x · χξ,λ (y, t) = χξ,λ (y, t − x, y) = χξ −λx,λ (y, t), for (y, t) ∈ Rn × R. If λ = 0, then x · χξ,0 = χξ,0 and Aχξ,0 = A. This yields a family of one-dimensional representations of Hn . For ξ, η ∈ Rn , define ση,ξ (x, y, t) = ei(x,η+y,ξ ) , for all (x, y, t) ∈ Hn . For each λ ∈ Rn , λ = 0, {χξ,λ : ξ ∈ Rn } is an orbit homeomorphic to A. Let  = {χ0,λ : λ ∈ R \ {0}}. Then  meets each of these free orbits in exactly one n λ point. Let U λ = indH N χ0,λ , for λ = 0. We can realize U on the Hilbert space 2 n 2 n L (R ) as follows. For (x, y, t) ∈ Hn and f ∈ L (R ), U λ (x, y, t)f (z) = eiλ(t−z,y) f (z − x),

168

Mackey analysis

for z ∈ Rn . Then n = {ση,ξ : ξ, η ∈ Rn } ∪ {U λ : λ ∈ R \ {0}}. H In the last two families of examples, the groups were all connected, simply connected, nilpotent Lie groups. For each such G, every irreducible representation of G was obtained by inducing a character from some subgroup. If the subgroup happened to be all of G, then the result is a one-dimensional representation of G. All other irreducible representations of G were infinitedimensional. But the fact that they are induced from characters is convenient for analysis on such groups and it is an important property of connected, simply connected, nilpotent Lie groups that this is true in general. For the basic theory of nilpotent Lie groups, we refer the reader to Corwin and Greenleaf [33]. We will use standard structural facts in the following proof without special reference. The proof of the following theorem shows the power of Mackey’s theory even in situations where one is not trying to explicitly construct the dual of a group. Theorem 4.39 Let G be a connected and simply connected nilpotent Lie group and let π be an irreducible representation of G. Then there exist a closed subgroup H of G and a character χ of H such that π = indG H χ. Proof We argue by induction on the dimension of G, dim G, as a manifold. Nothing has to be shown if dim G = 1, that is, G = R. Now, let dim G = n and suppose that the statement is true for connected and simply connected nilpotent Lie groups of dimension ≤ n − 1. Let Z denote the center of G. Then there exists a character ψ of Z such that π(z) = ψ(z)IH(π) for all z ∈ Z. Suppose that dim Z ≥ 2, so that Z = Rd for some d ≥ 2. Then there exists a one-dimensional subspace V of Z such that ψ|V = 1. Then π is effectively an irreducible representation of G/V . That  such that π = σ ◦ q, where q : G → G/V denotes is, there exists σ ∈ G/V the quotient homomorphism. Since G/V is a connected and simply connected nilpotent Lie group of dimension n − 1, by the inductive hypothesis there exist a G/V closed subgroup L of G/V and a character η of L such that σ = indL η. Now, let H = q −1 (L) and χ = η ◦ q. Then χ is a character of H and Proposition G/N 2.38 implies indG η) ◦ q = σ ◦ q = π. Thus we are left with the H χ = (indL case that dim Z = 1. Assume that dim Z = 1. We can assume that ψ is nontrivial because otherwise π can be viewed as a representation of the factor group G/Z, which is of lower dimension, and we proceed as above. The center of G/Z is a nontrivial vector group W . We choose a one-dimensional subspace of W and let N denote its pullback to G. Then N is a normal subgroup of G, isomorphic to R2 . We will write the group

4.5 Examples: action by compact groups

169

operation in N additively to reflect this. Select z ∈ Z, z = 0, and y ∈ N \ Z. Now, [x, y] = xyx −1 y −1 ∈ Z for every x ∈ G; hence [x, y] = λ(x)z for some unique real number λ(x). Notice next that, for x1 , x2 ∈ G, [x1 , y][x2 , y] = [x1 , y]x1−1 [x1 x2 , y]yx1 y −1 = [x1 , y][x1 x2 , y]x1−1 [x1 , y]−1 x1 = [x1 x2 , y]. Thus the map x → [x, y] is a continuous homomorphism of G into Z, and hence λ : x → λ(x) is continuous and satisfies λ(x1 x2 ) = λ(x1 ) + λ(x2 ) for all x1 , x2 ∈ G. Since y ∈ Z, λ is nonzero and hence λ(G) = R as G is connected. Let K = {x ∈ G : λ(x) = 0}. Then K is a closed normal subgroup of G and G/K is isomorphic to R. In particular, K is a connected and simply connected nilpotent Lie group of dimension n − 1. By applying Mackey’s theory (Section 4.3), we are going to show the existence of some irreducible representation τ of K such that π = indG K τ . Once this has been done, the inductive hypothesis yields that there exist a closed subgroup H of K and a character χ of H such that τ = indK H χ . Then, by the theorem on inducing in stages, π = indG χ , as required. H Any element of N = R2 can be written uniquely as sy + tz, s, t ∈ R, and the characters of N are given by χa,b (sy + tz) = exp[i(as + bt)], for a, b ∈ R. We determine the G-orbit and the stability group of χa,b . Note that a topological automorphism of a vector group is a linear map. Thus x(rv)x −1 = r(xvx −1 ), for x ∈ G, v ∈ N, and r ∈ R. We now compute x −1 · χa,b (sy + tz) = χa,b (x(sy + tz)x −1 ) = χa,b (sxyx −1 + tz) = χa,b (s([x, y] + y) + tz) = χa,b (sy + (λ(x)s + t)z) = exp[i(as + λ(x)bs + bt)] = χa+λ(x)b,b (sy + tz). It follows that G(χa,0 ) = {χa,0 } and Gχa,0 = G, for any a, whereas if b = 0, then G(χa,b ) = {χc,b : c ∈ R} and Gχa,b = {x ∈ G : λ(x) = 0} = K. Thus G and the abelian normal subgroup N of G satisfy all the hypotheses of . Fix χa,b ∈ O. Theorem 4.24. The restriction of π to N lives on an orbit O in N itb Then χa,b (tz) = e = ψ(tz) for t ∈ R and ψ is nontrivial. Thus b = 0. By Theorem 4.24, there exists an irreducible representation τ of K = Gχa,b such that τ |N is a multiple of χa,b and π = indG  K τ. This completes the proof.

4.5 Examples: action by compact groups We turn next to semidirect products formed by a compact group acting on an abelian locally compact group. Thus, let N be a locally compact abelian group

170

Mackey analysis

and K a compact group. Suppose that x → k · x denotes the action of k ∈ K on N and form the semidirect product G = N  K. As usual, K acts on the dual  of N via k · χ(x) = χ(k −1 · x), for all k ∈ K, χ ∈ N,  and x ∈ N . group N  Since K is compact, the K-orbits in N are compact. Hence N is regularly embedded in G, and therefore Theorem 4.29 applies to G.  and τ ∈ K χ , let χ × τ denote the finite-dimensional representaFor χ ∈ N tion of N  Kχ defined by (χ × τ )(x, k) = χ(x)τ (k). Let πχ,τ = indG NKχ (χ × τ ). By Theorem 4.29, each πχ,τ is irreducible and every irreducible representation of G is of this form. Moreover, two such representations πχ,τ and πη,ω , χ , η ∈  τ ∈K χ , ω ∈ K η , are unitarily equivalent if and only if there exists k ∈ K N, such that k · χ = η (hence Kη = kKχ k −1 ) and ω(l) = τ (k −1 lk), for all l ∈ Kη . In particular, πχ,τ = πχ,ω only if ω = τ . We formulate this as Theorem 4.40 Let G = N  K be a semidirect product, where N is an abelian locally compact group and K is a compact group. Choose a subset X  which meets each K-orbit in N  in exactly one point. Then the mapping of N 5 χ ) and G.  (χ , τ ) → πχ,τ is a bijection between the set χ ∈X ({χ } × K Our first example illustrating Theorem 4.40 is a discrete group formed by the action of a finite (hence compact) group on Z2 . Example 4.41 The dihedral 4 group, D4 , is the set of symmetries of a square. It can be represented as D4 = {I, R, R 2 , R 3 , M, MR, MR 2 , MR 3 }, where I is the identity, R is rotation through π/2, and M is a fixed reflection, say 2 through a horizontal axis.Let D4 act  on Z byidentifying  each element with a 0 −1 1 0 2 × 2 matrix. Take R = and M = . Then the rest of the 1 0 0 −1 matrices in D4 are determined. Let G = Z2  D4 , N = Z2 , and K = D4 . 2 by the square S = [−1, 1] × [−1, 1], with the edges We parametrize Z identified, defining characters χ(t,s) , for (t, s) ∈ S, by   n = eπi(tn+sm) , χ(t,s) m for all n, m ∈ Z. For A ∈ D4 , A · χ(t,s) = χ(t,s)A−1 , for all (t, s) ∈ S. Thus, the 2 can be visualized as essentially its action on action of D4 on the dual Z the square S as symmetries. The reader should take a few minutes to verify the following stability subgroups. Remember that the edges of S are identified, so

4.5 Examples: action by compact groups

171

2 . the four corners, (±1, ±1), represent the same point in Z Kχ(0,0) = Kχ(±1,±1) = K = D4 , Kχ(±1,0) = Kχ(0,±1) = {I, R 2 , M, MR 2 }, / {−1, 0, 1}, Kχ(t,0) = Kχ(t,±1) = {I, M} if t ∈ / {−1, 0, 1}, Kχ(0,s) = Kχ(±1,s) = {I, MR 2 } if s ∈ / {−1, 0, 1}, Kχ(t,t) = {I, MR 3 } if t ∈ / {−1, 0, 1}. Kχ(t,−t) = {I, MR} if t ∈ Finally, Kχ(t,s) = {I } if 0 < |t| = |s| < 1. Let X = {(t, s) : 0 ≤ s ≤ t ≤ 1}. Then X meets each D4 -orbit in S exactly once. χ for each of the finite According to Theorem 4.40, we need to calculate K groups listed above. For each one of them, we will use τ for the trivial representation. The two-element group Z/2 appears several times and we will abuse notation by using ε for the nontrivial character each time. The group Kχ(±1,0) is isomorphic to Z/2 × Z/2, so its dual consists of four characters which we 4 in Example 4.30 (the denote as {τ, η1 , η2 , η3 }. We actually worked out D map (1, 1) → R, (0, −1) → M extends to an isomorphism of Z/4  Z/2 with D4 ). There we found that D4 has one two-dimensional irreducible representation, which we will now denote π and four characters, which we will denote τ, σ1 , σ2 , σ3 . We decompose X as the disjoint union of seven sets; the three corners, the three sides without endpoints, and the interior: X = {χ(0,0) } ∪ {χ(1,1) } ∪ {χ(1,0) } ∪ {χ(1,s) : 0 < s < 1} ∪{χ(t,0) : 0 < t < 1} ∪ {χ(t,t) : 0 < t < 1} ∪ {χ(t,s) : 0 < s < t < 1}. For each χ ∈ X, recall that Gχ = N  Kχ is the stabilizer of χ in G. To reduce ψ the space required, we will write G(t,s) for Gχ(t,s) and UH for indG H ψ for various subgroups H and representations ψ of H .  decomposes as a disjoint union of those irreducibles associAccordingly, G ated with members of each of these sets. Thus, 6    7  = χ(0,0) × ρ , χ(1,1) × ρ : ρ ∈ {τ, σ1 , σ2 , σ3 , π } G 5  (χ(1,0) × τ ) (χ(1,0) × η1 ) (χ(1,0) × η2 ) (χ(1,0) × η3 )  , UG(1,0) , UG(1,0) , UG(1,0) UG(1,0)  5  (χ(1,s) × τ ) (χ(1,s) × ε) , UG(1,s) :0<s<1 UG(1,s)  5  (χ(t,o) × τ ) (χ(t,0) × ε) UG(t,0) , UG(t,0) :0
172

Mackey analysis

We now consider some examples of historical significance, the classical motion groups consisting of rigid orientation-preserving maps of Rn . The importance of these groups and their representation theory has resulted in the term generalized motion group sometimes being used to refer to any group of the form N  K where N is abelian and K is compact. Example 4.42 Let G = Rn  SO(n) with the product given by (x, S)(y, T ) = (x + Sy, ST ), x, y ∈ R , S, T ∈ SO(n). We identify the normal subgroup N = Rn × {I } n (row vectors) by the pairing γ (x) = exp iγ x. With  with R with Rn and N  is given by (x, T ) · γ = γ T −1 . Thus this identification, the action of G on N  are spheres Sr = {ξ ∈ Rn : ξ = r}, r ≥ 0. the orbits in N As a representative of Sr , choose re1 = r(1, 0, . . . , 0), r ≥ 0. The stability  let π˜ (x, S) = π (S), for all subgroup for 0 is SO(n). For each π ∈ SO(n), (x, S) ∈ G, the lifting of π to G. For r > 0, the stability group of re1 in SO(n) is SO(n − 1), considered as the subgroup of SO(n) that leaves the first coordinate fixed. Let n

σr,τ = indG Rn SO(n−1) (χr × τ ),  τ ∈ SO(n − 1), r > 0, where χr (x) = exp 2π irx1 for x = (x1 , . . . , xn ) ∈ Rn . Thus,  ∪ {σr,τ : r > 0, τ ∈ SO(n  = {π˜ : π ∈ SO(n)}  G − 1)}. If more detail is needed, one turns to the known descriptions of the irreducible representations of SO(k) for k = n − 1, and n. In particular, if n = 2 then SO(n − 1) = {I } and hence every irreducible representation of G = R2  SO(2) is either equivalent to a character of SO(2) 2 2 lifted to G or to an induced representation U χ = indG R2 χ, χ ∈ R , χ = 1R . 2 From the latter set, only a single ray of characters of R is needed such as {χr : r > 0}, where χr (t, s) = exp 2π irt for all (t, s) ∈ R2 . Let Rθ denote the element of SO(2) which rotates the plane through the angle θ . Since SO(2) can be identified with T, its irreducible representations are parametrized by Z. For n ∈ Z, let πn (x, Rθ ) = exp inθ for all (x, Rθ ) ∈ R2  SO(2). So R2   SO(2) = {πn : n ∈ Z} ∪ {U χr : r > 0}. If K is any compact group, then the Peter–Weyl theorem implies that  ⊕(Hπ ⊕ . . . ⊕ Hπ ), L2 (K) =  π∈K

4.6 Limitations on Mackey’s theory

173

where each Hπ occurs dim Hπ times as a direct summand. If K is second  must be a countable set. It was countable, then L2 (K) is separable so K expected at one time that the property of having a countable dual characterizes compact second countable groups. However, this is not the case. The final example of this section served as the first known example of a noncompact  is countable. In its construction, we use the topological group G such that G properties of the field of p-adic numbers and its group of additive characters. We refer to Hewitt and Ross [73] for a treatment of these properties. Example 4.43 Let p be a prime number, and let N be the additive group of the locally compact field Qp of p-adic numbers. Let K denote the subset of all elements a ∈ Qp of valuation |a|p = 1. Then K is a compact group under multiplication. Form the semidirect product G = N  K, where K acts on N  we observe first that the dual group N  of by multiplication. To determine G, N is topologically isomorphic to N. There exists a character χ of N such that, for any x ∈ N, χ(x) = 1 if and only if x is a p-adic integer. For each  by χy (x) = χ(yx), x ∈ N . Then the map y → χy is a y ∈ N , define χy ∈ N  (see [73]). topological isomorphism between N and N For j ∈ Z, let Nj = {y ∈ N : |y|p = pj }. Then N = {0} ∪j ∈Z Nj , and K acts transitively on each Nj . Since k · χy (x) = χ(k −1 yx) = χk −1 y (x) (x ∈ N),  are {0} and the sets Oj = {χy : y ∈ Nj }, j ∈ Z. Clearly, the K-orbits in N Kχy = {1} if y = 0. For each j ∈ Z, pick some yj ∈ Nj . Then it follows from Example 4.32 that =K  {indG G N χyj : j ∈ Z}.  is countable. The group G is Since K is compact and second countable, G often referred to as the Fell group.

4.6 Limitations on Mackey’s theory In the last two sections we saw how the calculation of the dual of a semidirect product N  H is routine when N is abelian and H is such that the representation theory of any of its closed subgroups is understood, provided N is Mackey compatible as a subgroup of N  H . In this section, we present two examples to illustrate how Mackey theory breaks down when some aspect of Mackey compatibility fails.

174

Mackey analysis

Example 4.44 The reader should first review Example 4.31, where the dual of Gaff = R  R+ was worked out. There we saw that 1 −1  }, G aff = {ωs : s ∈ R} ∪ {U , U

where the ωs are the characters and U 1 and U −1 are infinite-dimensional and associated with the nontrivial orbits in the dual of the abelian subgroup. Now let P denote the multiplicative group of positive real numbers equipped  = {χr : r ∈ R} with the discrete topology. Let G = R  P. With N = R and N . Thus, as in Example 4.31, we still have a · χr = χr/a and just three orbits in N  N is second countable and orbits are locally closed. The way in which N fails to be Mackey compatible in G is that P is not σ -compact. Nevertheless, we can follow the procedure of Section 4.3. The stability subgroup of the trivial character of N is P. Associated with this orbit we have all the elements of  P. For ω ∈  P, let ω (t, a) = ω(a) for all G (t, a) ∈ G. For r = 0, we can form πr = indN χr . Then πr acts on 2 (P), a Hilbert space of uncountable dimension. By Theorem 2.6, πr is irreducible for each r = 0. Moreover, Propositions 2.7 and 2.8 imply that πr is equivalent to πs if and only if χr and χs are in the same P-orbit; that is, if and only if r and  s have the same sign. Thus, we can take π1 and π−1 as representatives. So G contains {ω : ω ∈  P} ∪ {π1 , π−1 }. However, there are irreducible representations of G which are not equivalent to any of those just constructed. Two particular examples are devised easily. Let ϕ : G → Gaff be defined by ϕ(t, a) = (t, a), for (t, a) ∈ G. Then ϕ is clearly a continuous algebraic isomorphism. Thus, U 1 ◦ ϕ and U −1 ◦ ϕ are irreducible representations of G. Note that the Hilbert space of both U 1 ◦ ϕ and U −1 ◦ ϕ is the separable space L2 (R+ ). Thus, neither can be equivalent to either of π1 or π−1 , which act irreducibly on a nonseparable Hilbert space. Thus, the Mackey procedure does not give all the dual of G. We note that we are far from knowing the complete dual (see the notes at the end of this chapter). To get the previous example, we took the somewhat artificial step of using the positive real numbers with the discrete topology. This had the effect of destroying the homeomorphic equivalence of an orbit with the quotient by the stability subgroup – a crucial requirement in the Mackey theory. It is actually not necessary to go to the extreme of making the reals discrete in order to destroy the homeomorphism property. The Mautner group is a two-step solvable Lie group where Mackey theory fails to apply.

4.6 Limitations on Mackey’s theory

175

We introduced the family of Mautner groups in Example 4.16 as being of the form R4 A R, where R acts by rotating in the first two components of R4 at one speed and in the second two components at another speed. When these two speeds are not commensurate there are free orbits which are not homeomorphic to R. Here we will select one convenient pair of incommensurate speeds and write R4 as C2 so we can use elements of the unit circle to rotate. This simplifies calculations and helps the intuition. Example 4.45 The underlying set of G is C2 × R, and multiplication is given by (z, w, t)(z , w , t  ) = (z + eit z , w + e2πit w , t + t  ), for z, w, z , w ∈ C, t, t  ∈ R. Thus G is the semidirect product of C2 with R, where R acts on C2 by t · (z, w) = (eit z, e2πit w). 2 with C2 via the pairing (z, w), (ξ, η) = eiRe(zξ +wη) . This is the We identify C 4 with R4 , written in complex coordinates. The action usual identification of R 2 is then given by t · (ξ, η) = (eit ξ, e2πit η). Therefore, the R-orbits of R on C 2 are as follows: in C ⎧ ⎪ {(0, 0)} if ξ = η = 0, ⎪ ⎪ ⎪ ⎪ ⎨ {(ω, 0) : |ω| = |ξ |} if ξ = 0 and η = 0, R · (ξ, η) = ⎪ {(0, ω) : |ω| = |η|} if ξ = 0 and η = 0, ⎪ ⎪ ⎪ ⎪ ⎩ {(eit ξ, e2πit η) : t ∈ R} if ξ = 0 and η = 0. Thus the first three types of orbit are closed, whereas the last one fails to be locally compact (equivalently, open in its closure). If ξ = 0 and η = 0, then R · (ξ, η) = {(ω, ω ) : |ω| = |ξ |, |ω | = |η|}, a 2-torus. (The reader can compare this with our description in Example 4.16.) Hence the normal subgroup N is not Mackey-compatible. However, by Theorem 4.24, all the irreducible representations σ of G such that σ |C2 lives on any orbit of the first three types, are given by the imprimitivity theorem and Mackey’s theory. We start by calculating the stability groups Gξ,η for 2 , with either component 0. (ξ, η) ∈ C For ξ = 0 we have G(ξ,0) = {(z, w, t) ∈ G : eit = 1} = C2  2π Z, and for η = 0, G(0,η) = {(z, w, t) ∈ G : e2πit = 1} = C2  Z.

176

Mackey analysis

For r > 0 and ω ∈ T, let χr,0,ω be the character of C2  2π Z given by χr,0,ω (z, w, 2π m) = ωm exp i(rRe(z)), for z, w ∈ C, m ∈ Z. Similarly, for s > 0 and ω ∈ T, let χ0,s,ω denote the character of C2  Z given by χ0,s,ω (z, w, m) = ωm exp i(sRe(w)), for z, w ∈ C, m ∈ Z. For simplicity of notation, let H1 = C2  2πZ and H2 = C2  Z. Moreover, for r > 0, let Sr = {ω ∈ C : |ω| = r}, Or1 = Sr × {0} and Or2 = {0} × Sr . Then, for r, s > 0 and ω ∈ T, the representations G πr,0,ω = indG H1 χr,0,ω and π0,s,ω = indH2 χ0,s,ω

are irreducible and, with the notation of Section 4.3, by Theorem 4.24 Or2 = {π0,s,ω : ω ∈ T}. Or1 = {πr,0,ω : ω ∈ T} and G G 2 with ξ = 0 and η = 0. Then Now, let (ξ, η) ∈ C G(ξ,η) = {(z, w, t) ∈ G : eit = 1 = e2πit } = C2 × {0}. It follows that for each such (ξ, η) the induced representation πξ,η = indG C2 (ξ, η) is irreducible. As noted above, if r = |ξ | and s = |η|, then R · (ξ, η) = Sr × Ss . We invite the reader to compute explicit realizations of each of πξ,η , πr,0,ω , and π0,s,ω . Of course, we also get irreducible representations of G by lifting characters  in the same manner as of R, but we are far from being able to write down G we did with the previous examples in this section. Besides the fact that the hypotheses of Theorem 4.29 are not satisfied, there exists no measurable set which meets each R-orbit exactly once. We know this because the intersection of such a set with one of the tori Sr × Ss would be a measurable cross-section of the irrational flow on a torus, but such a cross-section does not exist. To illustrate that we are still missing irreducible representations of G, even up to equivalence, we provide an example of an irreducible representation π of 2 . G such that π|C2 does not live on a single orbit in C 2 , and for each Let μ be a normalized Haar measure on S1 × S1 ⊆ C (z, w, t) ∈ G, define an operator π (z, w, t) : L2 (μ) → L2 (μ) by ¯ π(z, w, t)f (ξ, η) = eiRe(zξ +wη) f (e−it ξ, e−2πit η), ¯

4.7 Cocycles and cocycle representations

177

for f ∈ L2 (μ), ξ, η ∈ S1 . It is easily verified that π (z, w, t)π(z , w , t  ) = π((z, w, t)(z , w  , t  )), and hence, for all f, g ∈ L2 (μ), π (z, w, t)∗ f, g = f, π (z, w, t)g  ¯ g(e−it ξ, e−2πit η)dμ(ξ, η) = f (ξ, η)eiRe(zξ¯ +wη)  −it ¯ −2πit ¯ g(ξ, η)dμ(ξ, η) = f (eit ξ, e2πit η)eiRe(−ze ξ −we η) = π((z, w, t)−1 )f, g = π (z, w, t)−1 f, g. Hence π is a unitary representation of G. We claim that π is irreducible. To see this, let T ∈ HomG (π, π). Then ¯ Tf (ξ, η), T (π(z, w, t)f )(ξ, η) = π (z, w, t)Tf (ξ, η) = eiRe(zξ +wη) ¯

and hence T commutes with multiplication by every function of the form (ξ, η) → exp iRe(zξ¯ + w η), ¯ |z| = |w| = 1. Since the collection of finite linear combinations of such functions forms a dense subalgebra of C(S1 × S1 ), it follows that T commutes with multiplication by all g ∈ C(S1 × S1 ). Then T (g) = T (g · 1) = g · T (1), and hence T is multiplication by f = T (1). Since T also commutes with all operators π (0, 0, t), t ∈ R, and the action of G on 2 is given by C (z, w, t) · (ξ, η) = (eit ξ, e2πit η), we obtain that the function f is invariant under the action of G. This in turn implies that f is constant μ-almost everywhere. In fact, if  f (ξ, η) = cm,n ξ m ηn (ξ, η ∈ S1 ) m,n∈Z

denotes the Fourier series expansion of f , then the invariance of f means that cm,n = exp it(m + 2π n) for all t, and hence cm,n = 0 unless m = n = 0. Thus π is irreducible. Finally, by definition of π , π (z, w, 0)f (ξ, η) = χξ,η (z, w)f (ξ, η). 2 . This shows that π|C2 does not live on any orbit in C

4.7 Cocycles and cocycle representations  when there Theorem 4.29 gives a satisfactory and functional description of G  are well is an abelian closed normal subgroup N such that the G-orbits in N

178

Mackey analysis

behaved and G splits as a semidirect product of G/N with N . However, it is not always true that a subgroup of G exists that is complementary to N (the H in Theorem 4.29, which is isomorphic to G/N). To get a description that is analogous to Theorem 4.29 in the situation where G is not a semidirect product, we need to introduce the concepts of cocycle and cocycle representation. We χ χ , for begin by trying to force the technique of Theorem 4.29 to describe G . We hope this will motivate the introduction of cocycles. χ ∈N Let N be an abelian closed normal subgroup of G, let H = G/N, let q : G → H denote the quotient homomorphism, and suppose there exists a measurable cross-section γ : H → G such that γ (N) = e. The action of G on  is essentially an H -action and, for h ∈ H and ψ ∈ N , N (h · ψ)(n) = ψ(h−1 · n) = ψ(γ (h)−1 nγ (h)), . As usual, define Hχ = {h ∈ H : h · χ = χ }. Then for all n ∈ N. Fix a χ ∈ N −1 χ χ is, in Gχ = q (Hχ ). In the situation of Theorem 4.29, one element of G the notation of that theorem, 1 × χ, where 1 is the trivial representation of Hχ . χ . But χ χ are products of this one with the rest of H All other elements of G 1 × χ is simply an extension of χ to a character of Gχ . We will attempt to duplicate such an extension, but first we need to document how much γ fails to be a homomorphism. Define ν : H × H → G by ν(h, k) = γ (hk)−1 γ (h)γ (k). Note that q(ν(h, k)) = e, so in fact ν(h, k) ∈ N , for all h, k ∈ H . Lemma 4.46 With γ and ν as above, ν is a measurable function of H × H into N that satisfies, for h, k,  ∈ H , (i) γ (h)γ (k) = γ (hk)ν(h, k), (ii) ν(h, eH ) = ν(eH , h) = e, where eH = N , the identity in H , (iii) ν(h, k)ν(k, ) = ν(hk, )(−1 · ν(h, k)). Proof It is clear that ν is measurable and that (i) and (ii) hold from the definition of ν. To see (iii), let h, k,  ∈ H . Then γ (hk)ν(h, k)ν(k, ) = γ (h)γ (k)ν(k, ) = γ (h)(γ (k)γ ()) = (γ (h)γ (k))γ () = γ (hk)ν(h, k)γ () = γ (hk)γ ()γ ()−1 ν(h, k)γ () = γ (hk)ν(hk, )(−1 · ν(h, k)). Now (iii) follows.



4.7 Cocycles and cocycle representations

179

Of course, ν retains its properties when restricted to Hχ × Hχ and we will continue to use the same notation for the restricted ν. We now extend χ to Gχ by using the parametrization (h, n) → γ (h)n from Hχ × N to Gχ . Let χ ∗ (γ (h)n) = χ(n), for all h ∈ Hχ , n ∈ N . Lemma 4.47 With χ , χ ∗ , and ν as above, χ ∗ is a measurable function of Gχ into T that satisfies, for x, y ∈ Gχ and n ∈ N, (i) χ ∗ (n) = χ (n), (ii) χ ∗ (xn) = χ ∗ (x)χ(n), (iii) χ ∗ (x)χ ∗ (y) = χ(ν(q(x), ¯ q(y)))χ ∗ (xy). Proof It is clear that χ ∗ is measurable and that (i) and (ii) hold from the definition of χ ∗ since γ (eH ) = e. Let h = q(x), k = q(y), mx = γ (h)−1 x, and my = γ (k)−1 y. Then mx , my ∈ N and xy = γ (h)mx γ (k)my = γ (h)γ (k)(k −1 · mx )my = γ (hk)ν(h, k)(k −1 · mx )my . Using the fact that k · χ = χ, we have χ ∗ (xy) = χ ∗ (γ (hk)[ν(h, k)(k −1 · mx )my ]) = χ (ν(h, k)(k −1 · mx )my ) = χ(ν(h, k))(k · χ)(mx )χ(my ) = χ(ν(q(x), q(y)))χ ∗ (x)χ ∗ (y), which is (iii).



Let ω(x, y) = χ(ν(q(x), ¯ q(y))), for x, y ∈ Gχ . Then ω records the failure ∗ of χ to be a homomorphism, but has its own fundamental structure. Lemma 4.48 With χ , χ ∗ , and ω as above, ω is a measurable function of Gχ × Gχ into T that satisfies, for x, y, z ∈ Gχ , (i) χ ∗ (x)χ ∗ (y) = ω(x, y)χ ∗ (xy), (ii) ω(x, e) = ω(e, x) = 1, (iii) ω(x, yz)ω(y, z) = ω(xy, z)ω(x, y). Proof Again, it is clear that ω is measurable and that (i) and (ii) hold from the definition of ω and Lemma 4.47. Moreover, (iii) follows from part (iii) of  Lemma 4.46 if one notes that q(z) ∈ Hχ . The properties appearing in the statement of Lemma 4.48 are key to the χχ so we will study functions with these properties in a general description of G setting.

180

Mackey analysis

Definition 4.49 A measurable 2-cocycle or simply a cocycle on a locally compact group L is a measurable function μ : L × L → T which satisfies μ(xy, z)μ(x, y) = μ(x, yz)μ(y, z) and μ(x, e) = μ(e, x) = 1, for all x, y, z ∈ L. Cocycles are also called multipliers in the literature. Definition 4.50 Two cocycles μ1 and μ2 on L are similar, μ1 ∼ μ2 , if there exists a measurable function f : L → T such that μ1 (x, y) = f (x)f (y)f (xy)−1 μ2 (x, y), for all x, y ∈ L. A cocycle which is similar to the constant 1 function on L × L is said to be trivial. Note that μ¯ is a cocycle if μ is and the pointwise product of two cocycles is also a cocycle. Definition 4.51 Let μ be a cocycle on the locally compact group L. A map π from L into U(H(π )), the unitary group of the Hilbert space H(π), is called a μ-representation (or, less precisely, a cocycle representation or projective representation) if π(x)π(y) = μ(x, y)π (xy), for all x, y ∈ L, and if for each ξ ∈ H(π), the map x → π (x)ξ is measurable. The definitions of invariant subspaces, irreducibility, and equivalence of μ-representations are exactly the same as for ordinary representations. If π is a μ-representation and σ is a ν-representation, then π ⊗ σ is a μν-representation. Notice that a cocycle representation necessarily maps the group identity to I , the identity operator on its Hilbert space.  Definition 4.52 Let μ be a cocycle on the locally compact group L. Let (L, μ) denote the set of equivalence classes of irreducible μ-representations of L. With this new terminology, let us return to G, N, χ , γ , χ ∗ , and ω. By Lemma 4.48, ω is a cocycle on Gχ and χ ∗ is a one-dimensional ω-representation. Thus,  χ ∗ ∈ (G χ , ω). Notice also that ω is constant on N -cosets in both arguments. In fact, if we define ω0 (h, k) = χ¯ (ν(h, k)), for h, k ∈ Hχ , then ω0 is a cocycle on Hχ and ω(x, y) = ω0 (q(x), q(y)), for all x, y ∈ Gχ . We call ω the inflation of ω0 . We are now ready to state the main result of this section.

4.7 Cocycles and cocycle representations

181

Theorem 4.53 Let G be a locally compact group and let N be an abelian closed normal subgroup of G such that there exists a measurable cross-section  Then there exist a cocycle ω0 on Hχ and an of H = G/N into G. Let χ ∈ N. ∗ ω-representation χ of Gχ that extends χ, where ω is the inflation of ω0 to  Gχ . Moreover, the map π → χ ∗ ⊗ (π ◦ q) is a bijection between (H χ , ω0 ) and χ χ . G Proof The first claim is immediate from the construction of χ ∗ , ω, and ω0 . For the second claim, let π be an irreducible ω0 -representation of Hχ acting on H(π ). Define σ : Gχ → U(H(π )) by σ (x) = χ ∗ (x)(π ◦ q)(x) = χ ∗ (x)π (q(x)) for x ∈ Gχ . Using Lemma 4.47(iii), one checks that σ is an irreducible (ordinary) representation of Gχ . This map from π to σ clearly preserves equivalence  χ both ways and defines an injective map  : (H χ , ω0 ) → Gχ . Thus, it only remains to show that  is onto. Let σ be an irreducible representation of Gχ such that σ |N is a multiple of χ. For each x ∈ Gχ , let τ (x) = χ ∗ (x)σ (x). This defines a measurable map of Gχ into U(H(σ )). For x, y ∈ Gχ , τ (x)τ (y) = χ ∗ (x)χ ∗ (y)σ (x)σ (y) = ω(x, y)χ ∗ (xy)σ (xy) = ω(x, y)τ (xy). Thus, τ is an ω-representation of Gχ . Moreover, τ (xn) = τ (x), for all  x ∈ Gχ , n ∈ N . Thus, if π(h) = τ (γ (h)), for h ∈ Hχ , then π ∈ (H χ , ω0 ) and (π ) = σ .  Remark 4.54 In extending χ to Gχ , the construction of χ ∗ depended on the choice of the cross-section γ . A different cross-section would lead to a different extension with a different associated cocycle. However, if ψ1 and ψ2 are any two measurable maps of Gχ into T whose restrictions to N agree with χ and if ωi (x, y) = ψi (x)ψi (y)ψi (xy)−1 , for i = 1, 2, then straightforward calculations show that each ωi is a cocycle. Moreover, if f (x) = ψ1 (x)ψ2 (x)−1 , for all x ∈ Gχ , then ω1 (x, y) = f (x)f (y)f (xy)−1 ω2 (x, y), for all x, y ∈ Gχ . Thus, ω1 and ω2 are similar. Therefore, the ω we explicitly constructed was just one member of an equivalence class of cocycles associated with χ. This equivalence class is referred to as the Mackey obstruction for χ . We now present an example of the use of Theorem 4.53 to compute a dual.

182

Mackey analysis

Example 4.55 Let G = {(1, m, n), (−1, m, n) : m, n ∈ Z} equipped with the product given by the following equations: (1, m1 , n1 )(1, m2 , n2 ) = (1, m1 + m2 , n1 + n2 ), (1, m1 , n1 )(−1, m2 , n2 ) = (−1, m1 + m2 , n1 + n2 ), (−1, m1 , n1 )(1, m2 , n2 ) = (−1, m1 + m2 , n1 − n2 ), (−1, m1 , n1 )(−1, m2 , n2 ) = (1, m1 + m2 + 1, n1 − n2 ). Let N = {(1, m, n) : m, n ∈ Z}. Then N is an abelian normal subgroup  as {χz,w : z, w ∈ T}, of G, isomorphic to Z2 . Thus we can parametrize N where χz,w (1, m, n) = zm wn , for (1, m, n) ∈ N and z, w ∈ T. The quotient group H = G/N is isomorphic to Z2 = {1, −1} and we will identify it with Z2 . One checks easily that G has no elements of order two, and hence G does not split as a semidirect product of Z2 with Z2 . It is therefore necessary to use the methods of Theorem 4.53, in combination with Theorem  4.24, to completely describe G. A convenient cross-section γ : Z2 → G is given by γ (1) = (1, 0, 0) and  γ (−1) = (−1, 0, 0). Using (−1, 0, 0)−1 = (−1, −1, 0), the action of Z2 on N  and (1, m, n) ∈ N , is determined by, for χz,w ∈ N (−1) · χz,w (1, m, n) = χz,w ((−1, −1, 0)(1, m, n)(−1, 0, 0)) = χz,w (1, m, −n) = χz,w¯ (1, m, n). Thus (−1) · χz,w = χz,w¯ and, of course, 1 · χz,w = χz,w . G It is illuminating to compute indG N χz,w explicitly. Let πz,w = indN χz,w . Using Proposition 2.3, we realize πz,w on C2 . In the standard basis of C2 , each πz,w is given by the following matrices:    m n  0 z w 0 zm w−n . , π (−1, m, n) = πz,w (1, m, n) = z,w zzm wn 0 0 zm w −n The orbit of χz,w is {χz,w , χz,w¯ } and the stabilizer is trivial if w is not real. Thus, πz,w is equivalent to πz,w¯ and is irreducible when w is not real. For w = 1 or −1, Gχz,w = G and πz,w must be reducible for any z ∈ T. The irreducible components must be one-dimensional since πz,w is only two-dimensional. We could do the computations to diagonalize the matrices, but our main goal in this example is to illustrate the cocycle that arises and determine the cocycle representation of Z2 associated with each χz,1 or χz,−1 .

4.7 Cocycles and cocycle representations

183

Let us start with χz,1 , for some z ∈ T. The associated cocycle on Z2 depends on the point (z, 1), so let us denote it by ωz,1 . Then, for h, k ∈ Z2 , ωz,1 (h, k) = χz,1 (ν(h, k)) = χz,1 (γ (hk)−1 γ (h)γ (k)). Thus ωz,1 (h, k) = 1 for all values of h and k except h = k = −1. Then ωz,1 (−1, −1) = χz,1 ((−1, 0, 0)(−1, 0, 0)) = χz,1 (1, 1, 0) = z¯ . ∗ of χz,1 to G hence is given on G \ N by The extension χz,1 ∗ ∗ χz,1 (−1, m, n) = χz,1 ((−1, 0, 0)(1, m, −n)) = χz,1 (1, m, −n) = zm ,

for all (−1, m, n) ∈ G \ N. Since every irreducible subrepresentation of πz,1 is one-dimensional, every irreducible ωz,1 -representation of Z2 must be one-dimensional. Suppose σ is such an irreducible ωz,1 -representation. Then σ : Z2 → T. Since σ (1) = 1, we only need to determine σ (−1). It must satisfy σ (−1)σ (−1) = ωz,1 (−1, −1)σ (1) = z. That is, σ (−1)2 = z. If z = e2πit for 0 ≤ t < 1, let z1/2 = eπit . Then there are two choices of σ (−1), z1/2 and −z1/2 . Let σz+ and σz− be the two irreducible ωz,1 -representations of Z2 given by σz+ (1) = 1, σz+ (−1) = z1/2

and

σz− (1) = 1, σz− (−1) = z−1/2 .

∗ ∗ Let πz,1,+ = χz,1 ⊗ (σz+ ◦ q) and πz,1,− = χz,1 ⊗ (σz− ◦ q), where q denotes the quotient map G → Z2 . Then

πz,1,+ (1, m, n) = zm , for (1, m, n) ∈ N, πz,1,+ (−1, m, n) = z1/2 zm , for (−1, m, n) ∈ G \ N, and πz,1,− (1, m, n) = zm , for (1, m, n) ∈ N, πz,1,− (−1, m, n) = −z1/2 zm , for (−1, m, n) ∈ G \ N. It is instructive to verify by direct computation that πz,1,+ and πz,1,− are ordinary representations of G. For χz,−1 , one finds that ωz,−1 = ωz,1 , for each z ∈ T. The extension of χz,−1 to G is given on G \ N by ∗ ∗ (−1, m, n) = χz,−1 (1, m, −n) = (−1)n zm , χz,−1

for all (−1, m, n) ∈ G \ N . Since ωz,−1 = ωz,1 , we have the same irreducible ∗ ⊗ (σz+ ◦ q) and cocycle representations, σz+ and σz− . Let πz,−1,+ = χz,−1

184

Mackey analysis

∗ πz,−1,− = χz,−1 ⊗ (σz− ◦ q), as above. Then

πz,−1,+ (1, m, n) = (−1)n zm , for (1, m, n) ∈ N, πz,−1,+ (−1, m, n) = (−1)n z1/2 zm , for (−1, m, n) ∈ G \ N, and πz,−1,− (1, m, n) = (−1)n zm , for (1, m, n) ∈ N, πz,−1,− (−1, m, n) = −(−1)n z1/2 zm , for (−1, m, n) ∈ G \ N.  We now have a complete parametrization of G:  = {πz,w : z, w ∈ T, Re(w) > 0} ∪ {πz,1,+ , πz,1,− , πz,−1,+ , πz,−1,− : z ∈ T}. G Remark 4.56 Actually, the group G in the previous example is one of the classical wallpaper or planar crystal groups and is often denoted pg. It is worth noting that the quotient space R2 /G, under the action which was given, is homeomorphic to the Klein bottle K, and standard results from algebraic topology imply that G is isomorphic to the fundamental group of K.

4.8 Mackey’s theory for a nonabelian normal subgroup The theme of this section is to extend the results of Section 4.3 (Mackey’s theory) to a more general setting in which the normal subgroup N of the locally compact group G need not be abelian anymore. Then the appropriate  but generalization of the dual group of an abelian normal subgroup is not N ∗ ∗ Prim(N ), the primitive ideal space of C (N), the group C -algebra of N. Although the proofs become more demanding, one can recognize the adaptation of many of the arguments of Section 4.3 to this situation. Fix a locally compact group G and a closed normal subgroup N of G for this section. We first identify the action of G on Prim(N) which generalizes the action  when N is abelian. Throughout this section, for x ∈ G, δ(x) denotes on N   the positive real number such that N f (x −1 nx) dn = δ(x) N f (n) dn, for any f ∈ L1 (N ). As usual, δ : G → R+ is a continuous homomorphism. For x ∈ G, f ∈ Cc (N ), let (x · f )(n) = δ(x)−1 f (x −1 nx), for all n ∈ N . Then f → x · f extends to an isometric automorphism of L1 (N ) such that σ (x · f ) =  Thus, f → x · f extends to an automorphism of (x −1 · σ )(f ), for all σ ∈ N. ∗ C (N ). This defines an action of G on C ∗ (N). For any closed ideal J of C ∗ (N) and x ∈ G, x · J = {x · f : f ∈ J } is also a closed ideal. Since, for any representation ρ of N and x ∈ G, x · ρ(x · f ) = ρ(f ), we have that J = ker ρ if and only if x · J = ker(x · ρ). In particular, x · J ∈ Prim(N) for J ∈ Prim(N ).

4.8 Mackey’s theory for a nonabelian normal subgroup

185

Lemma 4.57 With the above notation (i) The map J → x · J is a homeomorphism of Prim(N ) for each x ∈ G. (ii) The map (x, J ) → x · J is a left action of G on the topological space Prim(N). The proof is straightforward. Lemma 4.58 For each J ∈ Prim(N ), the stabilizer GJ = {x ∈ G : x · J = J } is a closed subgroup of G containing N. Proof The stabilizer of a point under a group action is always a subgroup and the map from G × Prim(N) → Prim(N) taking (x, I ) → x · I is continuous but, since Prim(N) need not be T1 , it is not immediately obvious that GJ is closed. To prove that GJ is closed, let (xα )α be a net in GJ converging to some x ∈ G. Then J = xα · J → x · J

and

J = xα−1 · J → x −1 · J

in Prim(N) and hence x · J, x −1 · J ∈ {J }. Assume that x · J = J and hence also x −1 · J = J . Then, since Prim(N ) is a T0 -space and x −1 · J ∈ {J }, there exists a neighborhood U of J such that x −1 · J ∈ U . Then x · U is a neighborhood of x · J with J ∈ x · U . This contradicts x · J ∈ {J }. Thus x · J = J , that is, x ∈ GJ .  Definition 4.59 Let L be a locally compact group, π a unitary representation of L, and E a locally closed subset of Prim(L). We say that π lives on E if h(ker π) ⊆ E

and

h(ker π ) ⊆ E \ E.

Equivalently, π lives on E if k(E) = k(E) ⊆ ker π

and

k(E \ E) ⊆ ker π.

The following theorem is the analogue of Theorem 4.22 in the current general situation. Theorem 4.60 Let N be a closed normal subgroup of a locally compact group G and let J ∈ Prim(N) and suppose that (1) The G-orbit G(J ) is locally closed in Prim(N ). (2) If H denotes the stabilizer of J in G, then the map xH → x −1 · J is a homeomorphism between G/H and G(J ).

186

Mackey analysis

Let π be an irreducible representation of G such that π|N lives on G(J ). Then there exists an irreducible representation τ of H such that (i) π is equivalent to indG H τ. (ii) ker(τ |N ) = J . Moreover, τ is uniquely determined up to unitary equivalence. Note that if N is abelian, or more generally, when Prim(N ) is a Hausdorff space, in Theorem 4.60 condition (2) implies condition (1) since in a locally compact Hausdorff space every locally compact subset is open in its closure. Thus in such a case, condition (1) can be dropped. We split the lengthy proof of Theorem 4.60 into three lemmas. Before stating the first one, we have to pave the ground for introducing a system of imprimitivity. Let N be a closed normal subgroup of G and let J ∈ Prim(N ) such that the conditions (1) and (2) of Theorem 4.60 are satisfied. Let A = k(G(J ) \ G(J ))/k(G(J )). Then A is a C ∗ -algebra, a subquotient of C ∗ (N), and for each x ∈ G, x · J contains k(G(J )) and we can form the primitive ideal A ∩ (x · J /k(G(J ))) of A. Let J = A ∩ (J /k(G(J ))) and observe that G acts on A and that x · J = A ∩ (x · J /k(G(J ))), for all x ∈ G. The map G(J ) → Prim(A), x · J → x · J is a homeomorphism from G(J ) onto Prim(A). Since (2) holds, the map G/H → Prim(A), xH → x −1 · J is a homeomorphism and hence j : C0 (G/H ) → C0 (Prim(A)), j (ϕ)(x · J) = ϕ(x −1 H ), x ∈ G, is an isometric ∗-algebra isomorphism. Now let ρ be a representation of N such that ker ρ = J . Of course ρ can be viewed as a representation of A, which for clarity we momentarily denote by ρA . Then ker ρA = J. Let M(A) denote the multiplier algebra of A and Z(M(A)) the center of M(A). Then the Dauns–Hofmann theorem [34] asserts that there is an isometric ∗-isomorphism T → fT from Z(M(A)) onto C b (Prim(A)) such that T (a) + I = fT (I )a + I , for all a ∈ A and I ∈ Prim(A). Combining this with the map j , for each ϕ ∈ C0 (G/H ) there exists a unique Tϕ ∈ Z(M(A)) such that Tϕ (a) + I = fTϕ (I )a + I , for all a ∈ A and I ∈ Prim(A). Moreover, note that ρA extends uniquely to a ∗-representation of M(A) and that then ρA (T )ρA (a) = ρA (T (a)), for all a ∈ A and T ∈ M(A).

4.8 Mackey’s theory for a nonabelian normal subgroup

187

For x ∈ G, the kernel of x · ρA equals x · J and this implies that x · ρA (Tϕ )x · ρA (a) = x · ρA (Tϕ (a)) = x · ρA (fTϕ (x · J)a) = fTϕ (x · J)x · ρA (a), for all ϕ ∈ C0 (G/H ) and a ∈ A, and therefore x · ρA (Tϕ ) = fTϕ (x · J)IdH(ρ) . Lemma 4.61 Let π be an irreducible representation of G such that π |N lives on G(J ) and let H = GJ . Define P (ϕ) = π |N (Tϕ ) ∈ B(H(π )), for each ϕ ∈ C0 (G/H ). Then (π, P ) is a system of imprimitivity over G/H . Proof It is clear that the map P is a ∗-homomorphism. We show next that P (Lx ϕ) = π(x)P (ϕ)π (x −1 ), for ϕ ∈ C0 (G/H ), x ∈ G. Notice first that for all f ∈ Cc (N ) and x ∈ G we have  π (x)π |N (f )π(x −1 ) = f (n)π (x)π(n)π(x −1 )dn N = f (n)x · π (n)dn = (x · π|N )(f ) N

= π |N (x −1 · f ). Since Cc (N ) is dense in C ∗ (N), this equation holds for all f ∈ C ∗ (N ). Then, viewing π|N as a representation of A and of M(A), we conclude that π(x)π |N (T )π (x −1 ) = π |N (x −1 · T ) holds for all T ∈ M(A) and all x ∈ G. Similarly, it is shown that y · ρA (x −1 · T ) = (yx) · ρA (T ), for all T ∈ M(A) and all x, y ∈ G, where ρA is as above. Let ω ∈ H(ρ) with

ω = 1. It follows that y · ρ(x −1 · Tϕ )ω, ω = (yx) · ρ(Tϕ )ω, ω = j (ϕ)((yx) · J) = ϕ((yx)−1 H ) = ϕ(x −1 y −1 H ) = (Lx ϕ)(y −1 H ) = j (Lx ϕ)(y · J) = y · ρ(TLx ϕ )ω, ω, for all ϕ ∈ C0 (G/H ) and x ∈ G. Therefore, x −1 · Tϕ = TLx ϕ and hence P (Lx ϕ) = π |N (TLx ϕ ) = π |N (x −1 · Tϕ ) = π(x)π |N (Tϕ )π (x −1 ) = π(x)P (ϕ)π (x −1 ), for all ϕ ∈ C0 (G/H ) and x ∈ G.

188

Mackey analysis

It remains to show that P (C0 (G/H ))H(π ) is dense in H(π ). Let H = P (C0 (G/H ))H(π ). Then H is π(G)-invariant since P (Lx ϕ) = π (x)P (ϕ)π (x −1 ) for all x ∈ G and ϕ ∈ C0 (G/H ). Thus either H = {0} or H is dense in H(π ). It therefore suffices to show that there exists ϕ ∈ C0 (G/H ) with P (ϕ) = 0. To that end, let I denote the kernel of π|N in A. Since Prim(A) = {x · J :  and choose a ∈ A such x ∈ G}, there exists x ∈ G with I ⊆ x · J. Let ρ ∈ N −1 that x · ρ(a) = 0. If ϕ ∈ C0 (G/H ) is such that ϕ(x H ) = 0, then x · ρ(Tϕ ) = ϕ(x −1 H )IdH(ρ) = 0 and hence x · ρ(Tϕ (a)) = ϕ(x −1 H )x · ρ(a) = 0. This shows that Tϕ (a) ∈ A \ x · J ⊆ A \ I . Now choose ξ ∈ H(π ) such that π |N (Tϕ (a))ξ = 0 and let η = π|N (a)ξ . Then P (ϕ)η = π |N (Tϕ )η = π |N (Tϕ (a))η = 0. 

This finishes the proof.

Lemma 4.62 Let π be an irreducible representation of G such that π |N lives J such that indG on G(J ). Let σ ∈ G GJ σ = π . Then ker(σ |N ) = J . Proof Again, let H = GJ to reduce some double subscripts. Note that since π|N lives on G(J ), & ker(x · σ |N ). k(G(J )) ⊆ ker(π|N ) = x∈G

Moreover, k(G(J ) \ G(J )) ⊆ ker(x · σ |N ) for every x ∈ G because k(G(J ) \ G(J )) is G-invariant and k(G(J ) \ G(J )) ⊆ ker(π|N ). Let H = span{σ |N (f )(ξ (e)) : f ∈ k(G(J ) \ G(J )), ξ ∈ H(indG H σ ), ξ continuous}. Then H is dense in H(σ ). This will follow once we have shown that H is σ (H )invariant since σ is irreducible and H = {0}. Thus fix h ∈ H and a continuous G element ξ of H(indG H σ ), and define η ∈ H(indH σ ) by η(x) = δ(h)H (h)−1/2 G (h)1/2 ξ (h−1 x). Then η is continuous and η(e) = σ (h)ξ (e). It follows that σ (h)σ |N (f )(ξ (e)) = σ (h)σ |N (f )σ (h−1 )σ (h)(ξ (e)) = h · (σ |N )(f )σ (h)(ξ (e)) = δ(h)σ |N (h−1 · f )(η(e)), which is an element of H. Let now ϕ ∈ C0 (G/H ) and f ∈ k(G(J ) \ G(J )) and put a = f + k(G(J )) ∈ A and b = Tϕ (a) ∈ A. Let  : H(π ) → H(indG H σ ) be the unitary

4.8 Mackey’s theory for a nonabelian normal subgroup

189

σ map satisfying π(x) = indG H σ (x) and P (ϕ) = P (ϕ) for all x ∈ G and ϕ ∈ C0 (G/H ). Then, locally almost everywhere,

(x −1 · σ |N )(Tϕ )(x −1 · σ |N )(a)(ξ (x)) = (x −1 · σ |N )(Tϕ (a))(ξ (x))   = (indG H σ )|N (b)ξ (x) = (π|N (b)−1 )(x) = (π|N (Tϕ )π |N (a)−1 )(x) = (P (ϕ)π|N (a)−1 ξ )(x) = (P σ (ϕ)π |N (a)−1 ξ )(x)   = ϕ(xH ) (indG H σ )|N (a)ξ (x)   = j (ϕ) x −1 · J (x −1 · σ |N )(a)(ξ (x)) = x −1 · ρ(Tϕ )ω, ω(x −1 · σ |N )(a)(ξ (x)). Since ξ , δ, and x → x · f are continuous, the map x → δ(x −1 )σ |N (x · f )(ξ (x)) = (x −1 · σ |N )(f )(ξ (x)) is continuous for every f ∈ C ∗ (N) and hence also for every a ∈ A. But then also the two maps x → (x −1 · σ |N )(b)(ξ (x)) = (x −1 · σ |N )(Tϕ )(x −1 · σ |N )(a)(ξ (x)) and x → x −1 · ρ(Tϕ )ω, ω(x −1 · σ |N )(a)(ξ (x)) are continuous. This shows that the equation (x −1 · σ |N )(Tϕ )(x −1 · σ |N )(a)(ξ (x)) = x −1 · ρ(Tϕ )ω, ω(x −1 · σ |N )(a)(ξ (x)) holds for all x ∈ G. In particular, σ |N (Tϕ )σ |N (a)(ξ (e)) = ρ(Tϕ )ω, ωσ |N (a)(ξ (e)). Since we have seen above that H is dense in H(σ ), we get that σ (Tϕ ) = ρ(Tϕ )ω, ωIdH(σ ) , for all ϕ ∈ C0 (G/H ). Let I be the kernel of σ |N in A. Our next aim is to prove that I = J. Recall that Prim(A) = {x · J : x ∈ G}. Thus let x ∈ G be such that I ⊆ x · J. Then x · Jis the kernel of x · ρ in A. Denote by Iz the kernel of σ |N in Z(M(A)) and by x · Jz the kernel of x · ρ in Z(M(A)). We claim that Iz ⊆ x · Jz . To see this, let T be an arbitrary element of Iz . Then T (a) ∈ I ⊆ x · J for all a ∈ A and

190

Mackey analysis

hence 0 = x · ρ(T (a)) = x · ρ(T )x · ρ(a), for all a ∈ A. Since x · ρ(A)H(ρ) is dense in H(ρ), it follows that x · ρ(T ) = 0 and therefore T ∈ x · Jz . Recall that H = GJ and, toward a contradiction, assume that x ∈ H . Choose ϕ ∈ C0 (G/H ) such that ϕ(x −1 H ) = 0 and ϕ(eH ) = 0. Then x · ρ(Tϕ ) = 0 and ρ(Tϕ ) = 0, and hence also σ |N (Tϕ ) = ρ(Tϕ )ω, ωIdH(σ ) = 0. Thus Tϕ ∈ Iz ⊆ x · Jz , which contradicts x · ρ(Tϕ ) = 0. So x ∈ H and x · J = J. This means that Jis the only element of Prim(A) containing I . Consequently, I = J. It follows that ker(σ |N ) ∩ k(G(J ) \ G(J )) = J ∩ k(G(J ) \ G(J )). Since k(G(J ) \ G(J )) is an ideal in C ∗ (N) and σ |N (k(G(J ) \ G(J )))H(σ ) is dense in H(σ ) and ρ(k(G(J ) \ G(J )))H(ρ) is dense in H(ρ), we obtain that ker(σ |N ) = J .  Lemma 4.63 Let σ and τ be two irreducible representations of H and π an irreducible representation of G such that ker(σ |N ) = J = ker(τ |N )

and

G indG H σ = π = indH τ.

Then σ and τ are equivalent. Proof Let P be the ∗-homomorphism defined in Lemma 4.61. It suffices to show that if τ is any irreducible representation of H such that ker(τ |N ) = J G and if  : H(π ) → H(indG H τ ) is a unitary operator intertwining π and indH τ , τ then P (ϕ) = P (ϕ) for all ϕ ∈ C0 (G/H ). In fact, the imprimitivity theorem τ (Theorem 3.17) then asserts that the two systems of imprimitivity (indG H τ, P ) and (π, P ) are equivalent and hence that τ and σ are equivalent.  such that ker ρ = J = ker(τ |N ) and let Jz denote the kernel of ρ in Let ρ ∈ N Z(M(A)). Then the same arguments as in the proof of Lemma 4.62 show that Jz equals the kernel of τ |N in Z(M(A)). Since ρ is an irreducible representation of M(A), there exists a continuous homomorphism χ : Z(M(A)) → C such that ρ(T ) = χ (T )IdH(ρ) , for all T ∈ Z(M(A)). Now, since ker χ = Jz , each T ∈ Z(M(A)) has a unique representation T = T  + χ (T )I , where T  ∈ Jz . As Jz

is the kernel of τ |N in Z(M(A)), we get τ |N (T ) = τ |N (χ(T )I ) = χ(T )IdH(τ ) . Now, for any ξ ∈ H(indG H τ ), ϕ ∈ Cc (G/H ), and a ∈ A we have locally almost everywhere     −1 G P (ϕ) −1 (indG H τ )|N (a)ξ (x) = π |N (Tϕ ) (indH τ )|N (a)ξ (x)   G = (indG H τ )|N (Tϕ )(indH τ )|N (a)ξ (x)   = (indG H τ )|N (Tϕ (a))ξ (x) = (x −1 · τ |N )(Tϕ (a))(ξ (x)) = (x −1 · τ |N )(Tϕ )(x −1 · τ |N )(a)(ξ (x)) = x −1 · ρ(Tϕ )ω, ω(x −1 · τ |N )(a)(ξ (x))   = ϕ(xH ) (indG H τ )|N (a)ξ (x)   = P τ (ϕ)(indG H τ )|N (a)ξ (x).

4.8 Mackey’s theory for a nonabelian normal subgroup

191

G G Let H = (indG H τ )|N (A)H(indH τ ). Then H = {0} and H is indH τ (G)G invariant, and hence H is dense in H(indH τ ). Thus the above equation implies  that P (ϕ) −1 = P τ (ϕ) for all ϕ ∈ C0 (G/H ), as was to be shown.

Combining the preceding three lemmas, we can now quickly complete the proof of Theorem 4.60. Given π as in Theorem 4.60, by Lemma 4.61 and the imprimitivity theorem there exists an irreducible representation τ of H = τ GJ such that the two systems of imprimitivity (π, P ) and (indG H τ, P ) are G equivalent. In particular, π is equivalent to indH τ . Since π |N lives on G(J ), Lemma 4.62 implies that ker(τ |N ) = J . Thus assertions (i) and (ii) of Theorem 4.60 hold. Finally, if σ is any other irreducible representation of H such that ker(σ |N ) = J and indG H σ = π, then σ = τ by Lemma 4.63. This completes the proof of Theorem 4.60. Retain all the previous notation. Let J ∈ Prim(N) and H = GJ , and set  : ker(τ |N ) = J } J = {τ ∈ H H and G(J ) = {π ∈ G  : π|N lives on G(J )}. G Then, under the conditions there, Theorem 4.60 provides a one-to-one map J . This map can now be shown to be onto. G(J ) into H from G Theorem 4.64 Let N be a closed normal subgroup of a locally compact group G and let J ∈ Prim(N ) and suppose that (1) the G-orbit G(J ) is locally closed in Prim(N ); (2) if H denotes the stabilizer of J in G, then the map xH → x −1 · J is a homeomorphism between G/H and G(J ).   Then the map τ → indG H τ is a bijection between HJ and GG(J ) .  is such that ker(σ |N ) = J , then indG Proof We have to show that if σ ∈ H H σ G is irreducible and (indH σ )|N lives on G(J ). Observe first that (indG H σ )|N lives on G(J ). In fact, by Proposition 2.41 we have & & ker x · (σ |N ) = x · J = k(G(J )), ker((indG H σ )|N ) = x∈G

x∈G

and since G(J ) is locally closed, we also have k(G(J ) \ G(J )) ⊆ k(G(J )) = ker((indG H σ )|N )). It remains to show that indG H σ is irreducible. For that, by Theorem 3.16, it suffices to prove that if T is a bounded linear operator on H(indG H σ ) such that

192

Mackey analysis

G σ σ T indG H σ (x) = indH σ (x)T , for all x ∈ G, then also T P (ϕ) = P (ϕ)T , for all ϕ ∈ C0 (G/H ). To this end, note that, for x ∈ G,   G (P σ (ϕ)(indG H σ )|N (a)ξ )(x) = ϕ(xH ) (indH σ )|N (a)ξ (x)   = x −1 · ρ(Tϕ )ω, ω (indG H σ )|N (a)ξ (x)

= (x −1 · σ |N )(Tϕ )(x −1 · σ |N )(a)(ξ (x)) = (x −1 · σ |N )(Tϕ (a))(ξ (x)) " ! = (indG H σ )|N (Tϕ (a))ξ (x) ! " G = (indG H σ )|N (Tϕ )(indH σ )|N (a)ξ (x). For x ∈ G, let Hx = span{(x −1 · σ |N )(a)(ξ (x)) : a ∈ A, ξ ∈ H(indG H σ ), ξ continuous}. Then Hx = {0} and it is easily verified that Hx is σ (H )-invariant. Thus Hx is dense in H(σ ) for every x ∈ G. On the other hand, we have for all a ∈ A, ξ ∈ H(indG H σ ), and locally almost all x ∈ G, by Proposition 2.41   (x −1 · σ )|N (a)(ξ (x)) = (indG H σ )|N (a)ξ (x). The above equation therefore implies that P σ (ϕ) = (indG H σ )|N (Tϕ ), and hence also T P σ (ϕ) = P σ (ϕ)T for all ϕ ∈ C0 (G/H ). This finishes the proof.  Define an equivalence relation on Prim(N) by setting I ∼ J if G(I ) = G(J ), and let Prim(N )/G denote the quotient space. If J ∈ Prim(N ) is such that J satisfies the hypotheses of Theorem 4.64, then so does every I ∈ G(J ), and G(I ) = G G(J ) . This since Gx·J = x · GJ , it follows from Proposition 2.39 that G leads to formulation of the following theorem. Theorem 4.65 Let N be a closed normal subgroup of the locally compact group G. Suppose that the following conditions are satisfied. (1) For each J ∈ Prim(N ), G(J ) is locally closed in Prim(N ). (2) For each J ∈ Prim(N ), the mapping x −1 GJ → x · J is a homeomorphism between G/GJ and G(J ).  there exists J ∈ Prim(N) such that π |N lives on G(J ). (3) For each π ∈ G,  Then G is the disjoint union = G(J ) : G(J ) ∈ Prim(N )/G}. G {G Proof It follows from Theorem 4.60 that every irreducible representation of G G(J ) for some J ∈ Prim(N ). is contained in G

4.8 Mackey’s theory for a nonabelian normal subgroup

193

G(I ) ∩ Now suppose that I, J ∈ Prim(N) are such that there exists π ∈ G I and τ ∈ G J such that G(J ) . Then there exist σ ∈ G G G indG GI σ = π = indGJ τ,

ker(σ |N ) = I , and ker(τ |N ) = J . Proposition 2.41 implies that & ker(x · σ |N ) = ker(indG k(G(I )) = G I σ |N ) x∈G

= ker(indG GJ τ |N ) =

&

ker(x · τ |N )

x∈G

= k(G(J )). Since G(I ) and G(J ) are locally closed, it follows that G(I ) = G(J ). This completes the proof.  Definition 4.66 Let G be a locally compact group and N a closed normal subgroup of G. Then N is said to be Mackey compatible if the following three conditions are satisfied.  there exists J ∈ Prim(N ) such that π |N lives on the orbit (1) For each π ∈ G, G(J ). (2) The map xGJ → x · J from G/GJ onto G(J ) is a homeomorphism for each J ∈ Prim(N). (3) G(J ) is locally closed in Prim(N ) for each J ∈ Prim(N). We now discuss ways by which one might determine if a normal subgroup N of G is Mackey compatible. Lemma 4.67 If Prim(N) is a Hausdorff space, then the map s : J → GJ from Prim(N) into the space K(G) of all closed subgroups of G (see Sections 5.1 and 5.2) is semicontinuous. Proof Let (Jα )α be a net in Prim(N ) such that Jα → J for some J ∈ Prim(N) and GJα → H in K(G). Given x ∈ H , there exist a subnet (GJλ )λ of (GJα )α and elements xλ ∈ GJλ such that xλ → x (compare with Lemma 5.4). Then Jλ = xλ · Jλ → x · J in Prim(N ) (Lemma 4.57). Since Prim(N ) is Hausdorff, we conclude that x · J = J .  Let N be a closed normal subgroup of G. We say that N is regularly embedded in G if Prim(N )/G, equipped with the quotient topology, is almost Hausdorff. This generalizes the corresponding definition at the outset of Section 4.3 for abelian normal subgroups. The following proposition is the analogue of Proposition 4.21 in the current more general setting. Though similar, the proof is somewhat more complicated and therefore included for convenience.

194

Mackey analysis

Proposition 4.68 Let N be a closed normal subgroup of the locally compact group G and suppose that one of the following two conditions is satisfied. (a) N is regularly embedded in G. (b) Prim(N) is second countable and G(J ) is locally closed in Prim(N) for every J ∈ Prim(N). Then, if π is an irreducible representation of G, there exists J ∈ Prim(N) such that π |N lives on G(J ). Proof We split the proof into three steps. (i) We first show that there exists a smallest closed, G-invariant subset C of Prim(N) such that π|N lives on C. To see this, for any closed, G-invariant subset F of Prim(N ) let HF denote the closed linear span of the set π |N (k(F ))(H(π )) in H(π ). Then HF is π(G)-invariant since π (x)π |N (f )ξ = δ(x)π |N (x −1 · f )π (x)ξ for all ξ ∈ H(π ), f ∈ C ∗ (N), and x ∈ G. Since π is irreducible, either HF = {0} or HF is dense in H(π ). Now let F denote the collection of all closed, G-invariant subsets F of Prim(N ) such that π|N lives on F . Then F = ∅ since Prim(N ) ∈ F. Let C = 9 {F : F ∈ F}. Then C is closed and G-invariant, and π|N lives on C. In fact, since h(ker(π |N )) ⊆ F for all F ∈ F, we also have h(ker(π |N )) ⊆ C. Moreover, if D is any proper, closed, G-invariant subset of C, then HD is dense in H(π ) since otherwise k(D) ⊆ ker(π|N ) and π|N lives on D, which is impossible. (ii) Suppose that N is regularly embedded in G and let q : Prim(N ) → Prim(N )/G denote the quotient map. Let C be as in (i). Then q(C) is closed since C is closed and G-invariant. By hypothesis, there exists a subset V of q(C) which is Hausdorff and dense and open in q(C). We claim that V is a singleton. Assuming the contrary, there exist two nonempty and disjoint open subsets V1 and V2 of q(C). For j = 1, 2, let Dj = q −1 (Vj ). Then Dj is open in C and G-invariant and Dj = ∅ and D1 ∩ D2 = ∅. So C \ Dj = C, C \ Dj is closed and G-invariant, and C = (C \ D1 ) ∪ (C \ D2 ). (i) now implies that HC\Dj is dense in H(π ), and as k(C) = k(C \ D1 ) ∩ k(C \ D2 ) ⊇ k(C \ D1 )k(C \ D2 ), it follows that HC = H(π). However, this is a contradiction since π |N lives on C and therefore k(C) ⊆ ker(π|N ). So we have V = G(J ) for some J ∈ Prim(N). Applying (i) again then yields that k(G(J )) = k(C) ⊆ ker(π|N ) and that π|N (G(J ) \ G(J ))H(π ) is dense in

4.8 Mackey’s theory for a nonabelian normal subgroup

195

H(π). In particular, k(G(J ) \ G(J )) ⊆ ker(π |N ). This shows that π |N lives on G(J ) and hence proves the proposition in case (a). (iii) Now suppose that (b) holds and let C be as in (i). Since G(J ) is open in G(J ) for every J ∈ Prim(N ), by (i) it suffices to show that there exists J ∈ Prim(N) such that C = G(J ). To that end, let U be a countable basis for the topology of C, and for each U ∈ U, let U G = ∩x∈G x · U . Then U G is dense in C. Indeed, otherwise there exists a nonempty, G-invariant, open subset V of C with V ∩ U G = ∅, and then C = (G \ V ) ∪ (G \ U G ), a union of two closed, G-invariant, proper subsets of C. As in the proof of (ii), this implies HC = H(π ), a contradiction. Thus U G is dense in C for each U ∈ U. Since U is countable and C is a Baire space, it follows that & W = {U G : U ∈ U} = ∅. Finally, it is obvious that G(J ) is dense in C for every J ∈ W .



Let (G, X) be a topological transformation group, where X is a T0 -space. Then, for each x ∈ X, the stability group Gx is closed in G (in fact, in the proof of Lemma 4.58, Prim(N) can be replaced by any T0 -space) and the mapping ψx : G/Gx → G(x), tGx → t · x is a continuous bijection. Lemma 4.69 Suppose that G/Gx is σ -compact, G(x) is a Baire space, and G(x) contains a dense open T1 -subset. Then ψx is a homeomorphism. Proof We first show that G(x) is a T1 -space. To see this, let U be a dense open T1 -subset of G(x). Then ψx−1 (U ) is open in G/Gx , and hence the sets tψx−1 (U ), t ∈ G, form an open cover of G/Gx . Since G/Gx is σ -compact, 5 −1 we can choose a sequence (tn )n in G with G/Gx = ∞ n=1 tn ψx (U ). Then 5∞ G(x) = n=1 tn · U , and each tn · U is a T1 -space and open and dense in 9 G(x). Because G(x) is a Baire space, we get that ∞ n=1 tn · U = ∅. Choose 9∞ y ∈ n=1 tn · U and let z be an arbitrary element of G(x). Then, for some n ∈ N, y, z ∈ tn · U and hence either z = y or z ∈ {y} as tn · U is a T1 -space. So {y} is closed in G(x) and since z → t · z is a homeomorphism of G(x), it follows that G(x) = G(y) is T1 . It is well known that every compact subset of a T1 -space is closed. Therefore, ψx (C) is closed in G(x) for every compact subset C of G/Gx . Since G/Gx 5 is σ -compact, G(x) can be written as G(x) = ∞ n=1 ψx (Cn ), where the Cn are compact subsets of G/Gx covering G/Gx . So, since G(x) is a Baire space, at least one of the closed subsets ψx (Cn ) of G(x) has to have nonempty interior, W say. Let V be any open subset of ψx−1 (W ). Then ψx (V ) = W ∩ ψx (Cn \ V ),

196

Mackey analysis

and ψx (Cn \ V ), the image of a compact set in G/Gx , is closed in G(x). So ψx (V ) is open in G(x). Now let U be an arbitrary open subset of G/Gx and let y ∈ ψx (U ). Then t · y ∈ W for some t ∈ G and V = ψx−1 (W ) ∩ (t · U ) is an open subset of ψx−1 (W ). By the preceding paragraph, ψx (V ) is open in G(x) and hence so is t −1 · ψx (V ) ⊆ ψx (U ). Since y ∈ t −1 · ψx (V ) and y ∈ ψx (U ) was arbitrary, it follows that ψx (U ) is open in G(x). Thus ψx is an open mapping and hence a homeomorphism.  It is clear from the proof that if G/Gx is compact, then in the preceding lemma the hypothesis that G(x) be a Baire space is not needed. Proposition 4.70 Let N be a closed normal subgroup of G. (i) Let J ∈ Prim(N) and suppose that (a) G/GJ is σ -compact. (b) G(J ) is locally closed and G(J ) contains a T1 -subset which is open and dense in G(J ). Then the hypotheses of Theorem 4.64 are fulfilled. (ii) If (a) and (b) hold for all J ∈ Prim(N ) and, in addition, N is regularly embedded in G or N is second countable, then the hypotheses of Theorem 4.65 are fulfilled. Proof (i) Since G(J ) is open in G(J ), it is homeomorphic to Prim(A) where A = k(G(J ) \ G(J ))/k(G(J )). Being the primitive ideal space of a C ∗ -algebra, G(J ) is a Baire space. It follows now from Lemma 4.69 that the map xGJ → x −1 · J is a homeomorphism from G/GJ onto G(J ). (ii) If N is second countable, then so is Prim(N). Therefore, if N is second countable or N is regularly embedded in G, and if condition (b) holds, then by  there exists J ∈ Prim(N ) such that π|N lives Proposition 4.68 for each π ∈ G on G(J ). Together with the assertion of (i), this ensures that the hypotheses of Theorem 4.65 are satisfied.  Proposition 4.71 Let N be a closed normal subgroup of the locally compact group G such that G/N is compact and suppose that one of the following two conditions holds. (a) N is type I. (b) N is second countable and Prim(N) is a T1 -space. Then the hypotheses of Theorem 4.65 are satisfied. Proof (i) Suppose first that N is type I. We show that then the hypotheses of Proposition 4.70(ii) are fulfilled. Notice first that every C ∗ -subalgebra and

4.8 Mackey’s theory for a nonabelian normal subgroup

197

every quotient C ∗ -algebra of a type I C ∗ -algebra is of type I [37, proposition 4.3.5]. Moreover, if A is a type I C ∗ -algebra, then Prim(A) contains an open, dense, Hausdorff subset [37, theorem 4.4.5]. Thus Prim(N ) is almost Hausdorff, and this forces Prim(N)/G to be almost Hausdorff because G/N is compact; that is, N is regularly embedded in G. This in turn implies that G(J ) is locally closed and hence G(J ) is homeomorphic to Prim(A) for some C ∗ -algebra of type I. Consequently, G(J ) is almost Hausdorff and therefore the hypotheses of Proposition 4.70(ii) are satisfied.  Lemma 4.72 Let (G, X) be a topological transformation group with G compact. If X is almost Hausdorff, then so is the quotient space X/G. Proof We are going to show that if A is any nonempty closed subset of X/G, then A contains a nonempty subset U which is open in A and Hausdorff. To see that satisfaction of this condition is sufficient, let U denote the collection of all such subsets U of A. Then Zorn’s lemma implies that U has a maximal element U0 , and U0 must be dense in A. Indeed, if A \ U0 has nonempty interior, say W , then there exists an open subset V of A \ U0 such that V = ∅ and V is Hausdorff. Then U0 ∪ V is open in A, Hausdorff, and strictly contains U0 . This contradicts the maximality of U0 . Thus let A be a nonempty closed subset of X/G and let B = q −1 (A). Then B is a closed, G-invariant subset of X. Since X is almost Hausdorff, there exists a subset U of B which is Hausdorff and open and dense in B. Choose 5 x ∈ U ; then G(x) ⊆ G(U ) ⊆ t∈G t · U and since G(x) is compact, we find 5n t1 , . . . , tn ∈ G with G(x) ⊆ j =1 tj · U . All the sets tj · U are Hausdorff and open and dense in B. Let ⎛ ⎞ n n & U1 = tj · U and U2 = B \ G ⎝B \ tj · U ⎠ . j =1

j =1

5n

Then U2 = ∅ since G(x) ⊆ U2 , U2 ⊆ j =1 tj · U , and U2 is open in B and G-invariant. The set U1 is Hausdorff and open in B, and it is easily verified that U1 is also dense in B. Now, let V = U1 ∩ U2 , so that V = ∅. We show that q(V ) is Hausdorff and open in A. To that end, let x, y ∈ V be such that G(x) = G(y). Then x ∈ U1 5 and y ∈ U2 and therefore G(y) ⊆ U2 ⊆ nj=1 tj · U . Since the sets tj · U are Hausdorff, for each t ∈ G there exist open neighborhoods Ut of t · x and Wt of t · y such that Ut ∩ Wt = ∅. Since G(y) is compact, there exist t1 , . . . , tn ∈ G 5 9 such that G(x) ⊆ nj=1 Utj . Then W = nj=1 Wtj is an open neighborhood of 5 5n y with W ∩ j =1 Utj = ∅. Finally, let U = B \ G(B \ nj=1 Utj ). Then q(U ) is an open neighborhood of q(x) in X/G and q(W ) is an open neighborhood

198

Mackey analysis

of G(y) in X/G satisfying q(U ) ∩ q(W ) = ∅. This completes the proof that q(V ) is Hausdorff.  Example 4.73 This example demonstrates an application of Mackey analysis when the closed normal subgroup is no longer abelian. Let ⎧⎛ s ⎫ ⎞ ⎨ e x z ⎬ G = ⎝ 0 1 y ⎠ : s, t, x, y, z ∈ R . ⎩ ⎭ 0 0 et This is a five-dimensional solvable Lie group under the matrix product. If we write (s, t, x, y, z) for the elements of G, then the group product is (s, t, x, y, z)(s  , t  , x  , y  , z ) 



= (s + s  , t + t  , x + es x  , y  + et y, es z + et z + xy  ). The identity in G is (0, 0, 0, 0, 0) and (s, t, x, y, z)−1 = (−s, −t, −e−s x, −e−t y, e−s−t (xy − z)).  To that end let We will apply Mackey theory to compute G. N = {(0, 0, x, y, z) : x, y, z ∈ R}. Then N is a closed normal subgroup of G which is obviously isomorphic to the three-dimensional Heisenberg group, H1 , whose dual was calculated in Example 4.38. Thus,  = {U λ : λ ∈ R \ {0}} ∪ {ση,ξ : (η, ξ ) ∈ R2 }, N

(4.1)

where each U λ is an irreducible representation acting on L2 (R) and each ση,ξ is a character of N. In fact, for (0, 0, x, y, z) ∈ N, U λ (0, 0, x, y, z)f (a) = eiλ(z−ay) f (a − x), for almost all a ∈ R, f ∈ L2 (R), λ = 0, and ση,ξ (0, 0, x, y, z) = ei(xη+yξ ) , for (η, ξ ) ∈ R2 . Let ∞ = {U λ : λ ∈ R \ {0}} and N 1 = {ση,ξ : (η, ξ ) ∈ R2 }. N Notice that {(s, t, 0, 0, 0) : s, t ∈ R} is a closed subgroup of G complementary to N. Thus, H = G/N can naturally be identified with R2 . The action, by conjugation, of (s, t) ∈ H on (0, 0, x, y, z) ∈ N is found to be (s, t) · (0, 0, x, y, z) = (0, 0, es x, e−t y, e(s−t) z). It is convenient to think of G as N  H .

4.8 Mackey’s theory for a nonabelian normal subgroup

199

, note that each of N ∞ and N 1 must be H To find the action of H on N invariant. Calculate   (s, t) · U λ (0, 0, x, y, z)f (a) = U λ ((−s, −t) · (0, 0, x, y, z))f (a) = U λ (0, 0, e−s x, et y, e(t−s z)f (a) (t−s) t = eiλ(e z−e ay) f (a − e−s x). Now the infinite-dimensional irreducible representations of H1 are determined, up to equivalence, by their restriction to the center. Since (s, t) · U λ (0, 0, 0, 0, z) = U e

(t−s)

λ

(0, 0, 0, 0, z), (t−s)

for all z ∈ R, we have that (s, t) · U is equivalent to U e λ . Thus, there are ∞ , corresponding to positive and negative λ, respectively. just two H -orbits in N 1 −1 Select U and U as representatives of these orbits. The stability subgroups are nontrivial in both cases; HU ±1 = {(s, s) : s ∈ R}, so GU ±1 = N  HU ±1 = {(s, s, x, y, z) : s, x, y, z ∈ R}. We need at least one unitary representation of GU λ whose restriction to N is λ U , for λ = 1, −1. We will construct these directly. For each s ∈ R, define T (s) : L2 (R) → L2 (R) by λ

T (s)f (a) = e−s/2 f (e−s a), a ∈ R. Then T is a unitary representation of R. Moreover, a direct calculation shows that T (s)−1 U λ (0, 0, x, y, z)T (s) = (s, s) · U λ (0, 0, x, y, z), for all (x, y, z) ∈ N, s ∈ R. Observing that (s, s, x, y, z) = (s, s, 0, 0, 0)(0, 0, e−s x, y, e−s z), define V λ (s, s, x, y, z) = T (s)U λ (0, 0, e−s x, y, e−s z), for (s, s, x, y, z) ∈ GU λ . Another direct calculation shows that V λ is a unitary representation of GU λ . Clearly V λ extends U λ and must therefore be irreducible. Lemma 4.74 Any irreducible representation of GU λ extending U λ must be of  the form χ ⊗ V λ for some χ ∈ H U λ , where   χ ⊗ V λ (s, s, x, y, z) = χ(s, s, 0, 0, 0)V λ (s, s, x, y, z). Proof To simplify the notation in the proof, set S = HU λ , L = N  S, σ = U λ , and τ = V λ . Note that S is isomorphic to R. We have to show that every irreducible representation of L extending σ is of the form χ ⊗ τ for some character χ of S (lifted to L, of course).

200

Mackey analysis

 and Suppose first that we already know that {χ ⊗ τ : χ ∈  S} is closed in L, let π be an arbitrary irreducible representation of L such that π|N = σ . Then, since 1S ≺ λS = indLN 1N , S ⊗ τ. π = π ⊗ 1S ≺ π ⊗ indLN 1N = indLN (π|N ) = indLN σ ∼   it follows that π = χ ⊗ τ for some character As this latter set is closed in L, χ of S.  let (χα )α be a net in  To show that the set  S ⊗ τ is closed in L, S such that 2  χα ⊗ τ converges to ρ for some ρ ∈ L. Let ξ ∈ H(ρ) [= L (R)] of norm 1. Then there exist ξα ∈ H(τ ) [= L2 (R)] of norm 1 such that χα (x)τ (x)ξα , ξα  → ρ(x)ξ, ξ  uniformly on compact subsets of L. Now let Sd denote the group S with the discrete topology and let Ld = N  Sd . Since Sd is compact, after passing to a subnet if necessary, we may assume that χα → χ for some character χ of Sd . It follows that, uniformly on compact subsets of Ld , χ (x)ρ(x)ξ, ξ  = lim χα (x)ρ(x)ξ, ξ  = limτ (x)ξα , ξα . α

α

This implies that ρ = χ ⊗ τ as representations of Ld . However, since both ρ and τ are continuous on L, χ has to be continuous on S. This completes the proof of the lemma.   We return to our example. Now H U λ = {χv : v ∈ R}, where  χv (s,λ s, 0, 0, 0) = G e , s ∈ R. For each v ∈ R, λ = 1, −1, let πv,λ = indGU λ χv ⊗ U . By Lemma 4.74 and Theorem 4.65, the set of irreducible representations of G whose restrictions to N live on either the H -orbit of U 1 or that of U −1 is exactly {πv,1 , πv,−1 : v ∈ R}. 1 . For (s, t) ∈ H and (η, ξ ) ∈ R2 , a simple We now identify the H -orbits in N computation shows that (s, t) · ση,ξ = σe−s η,et ξ . Thus, there are nine H -orbits 1 . In η–ξ -coordinates, the orbits are the four open quadrants, the four half in N axes, and the origin. The set {σ±1,±1 , σ±1,0 , σ0,±1 , σ0,0 } meets each H -orbit in 1 exactly once. The stabilizers in H are: Hσ0,0 = H and N ivs

Hσ±1,±1 = {(0, 0)}, Hσ±1,0 = {(0, t) : t ∈ R}, Hσ0,±1 = {(s, 0) : s ∈ R}. We just need notation for characters of the two copies of R. Let ψu (s, 0) = eius , ϕw (0, t) = eiwt for s, t, u, w ∈ R. Then we form   i(wt+x) × σ ϕ w 1,0  (0, t, x, y, z) = ϕw (0, t)σ1,0 (0, 0, x, y, z) = ei(wt−x) ,  (0, t, x, y, z) = ϕw (0, t)σ−1,0 (0, 0, x, y, z) = ei(us+y) , ϕw × σ−1,0 × σ , ψ u 0,1 (s, 0, x, y, z) = ψu (s, 0)σ0,1 (0, 0, x, y, z) = e  ψu × σ0,−1 (s, 0, x, y, z) = ψu (s, 0)σ0,−1 (0, 0, x, y, z) = ei(us−y) .

4.9 Notes and references

201

These are the characters of the respective stability subgroups, Gσ1,0 , Gσ−1,0 , Gσ0,1 , or Gσ0,−1 , whose restrictions to N agree with the corresponding σ1,0 , σ−1,0 , σ0,1 , or σ0,−1 . Define     G μ1,w = indG Gσ1,0 ϕw × σ1,0 , μ−1,w = indGσ−1,0 ϕw × σ−1,0 ,     ψu × σ0,1 , νu,−1 = indG ψu × σ0,−1 , νu,1 = indG Gσ Gσ 0,1

0,−1

for u, w ∈ R. The Hilbert space of each of μ1,w , μ−1,w , νu,1 , and νu,−1 is L2 (R). We also have {χa,b : (a, b) ∈ R2 }, where χa,b (s, t, x, y, z) = ei(as+bt) , for all (s, t, x, y, z) ∈ G. These are the characters of H lifted to G and are the representations of G whose restriction to N agree with σ0,0 . Finally, we need to define the four representations associated with the H -orbits of σ±1,±1 . Let ρ1,−1 = indG ρ1,1 = indG N σ1,1 , N σ1,−1 , G ρ−1,1 = indN σ−1,1 , ρ−1,−1 = indG N σ−1,−1 . Then 5 5  = {πv,1 , πv,−1 : v ∈ R} {μ1,w , μ−1,w : w ∈ R} {νu,1 , νu,−1 : u ∈ R} G 5 5 {ρ1,1 , ρ1,−1 , ρ−1,1 , ρ−1,−1 } {χa,b : (a, b) ∈ R2 }.

4.9 Notes and references The major part of the theory developed in this chapter is due to Mackey [103] and consequently is usually referred to as Mackey analysis or the Mackey machine. It allows one, under fairly broad conditions, to analyze the irreducible representations of a locally compact group G in terms of the representations of a closed normal subgroup N of G and certain subgroups of G/N. The reader will have spotted the imprimitivity theorem as the main tool. Throughout his development of the theory, Mackey assumed G to be second countable, firstly because the definition and basic properties of induced representations were only available in this situation and secondly because he employed direct integral decompositions. Blattner [27] showed how the assumption of second countability can be dispensed with entirely. For most of this chapter, we have confined ourselves to the case of an abelian closed normal subgroup N for two reasons. The theory turns out to be considerably easier in this situation and the final results, Theorems 4.27 and 4.28,  for various useful classes of already suffice to determine the dual space G locally compact groups G. This is demonstrated by the collection of examples worked out in Sections 4.4 and 4.5. Of course, the irreducible representations of classical groups, such as the Heisenberg group, the affine group, and the

202

Mackey analysis

group of rigid motions of the plane, were known before the theory of induced representations and the Mackey machine was developed. Theorem 4.39 which, in common terminology, says that connected and simply connected nilpotent Lie groups are monomial, appears to be folklore. It was certainly known to Kirillov and can be shown utilizing his coadjoint orbit theory. The proof given here, as an application of Mackey’s theory, is an adaptation of the proof of Fell [47, theorem 2]. Our discussion of the more general case of a not necessarily abelian normal subgroup in Section 4.8 follows the exposition in Echterhoff [38]. The two examples presented in Section 4.6 show that Mackey’s theory breaks down if one of the conditions of Mackey compatibility fails. Example 4.44, the semidirect product G = R  P, where P denotes the multiplicative group of positive real numbers with the discrete topology, is borrowed from [10]. In this example, condition (b) in Definition 4.25 is not fulfilled. In contrast, the irreducible representation π of the Mautner group M constructed in Example 2 , so that 4.45 has the property that π|C2 does not live on any orbit in C condition (a) in Definition 4.25 is violated. Actually, several series of irreducible representations of the Mautner group have been constructed in Baggett [4] and Baggett and Merrill [7]. It is also worth mentioning that using the positive solution to the Effros–Hahn conjecture, due to Gootman and Rosenberg [66], the primitive ideal space of the Mautner group can be determined in the sense that every irreducible representation of M turns out to be weakly equivalent to one arising from Mackey’s theory. The theory of Mackey and Blattner was further substantially generalized, through work of Blattner, Fell, Green, and others, to the setting of Banach ∗-algebraic bundles. For a full treatise of this general and intricate subject, we refer the interested reader to the monograph by Fell and Doran [54], in particular chapters XI and XII, where induced representations as well as the imprimitivity theorem and the generalized Mackey analysis, respectively, are developed. The figures in Section 4.2 were inspired by those in Rosenberg [137].

5 Topologies on dual spaces

 of a locally compact group G carries the Fell topology The dual space G defined in Definition 1.67 with the closure operation given in Proposition 1.68. This topology is important for investigating the L1 - and C ∗ -algebras of G. As we have seen in Chapter 4, for many groups Mackey’s theory enables us to  as a set. The basic tools are inducing representations of closed determine G subgroups up to G and restricting representations to closed subgroups. However, even when starting with irreducible representations, neither inducing nor restricting lead to irreducible representations in general. It is therefore desirable to have a topology on the collection Rep(G) of “all” representations  but also to express this topology of G, which reduces to the dual topology on G, in terms of pairs (H, π ) as above. Such a theory was developed by Fell [49] and [51] and will be presented in the first four sections of this chapter. The first step is to introduce the so-called inner hull–kernel topology on Rep(G), actually on Rep(A) for a general C ∗ -algebra A (Section 5.1). Using the existence of a smooth choice of Haar measures on the topological space K(G) of all closed subgroups of G and forming a certain C ∗ -algebra, called the subgroup C ∗ -algebra (Section 5.2), a natural topology can be defined on the set S(G) of all subgroup representation pairs (H, π ). This topological space S(G) turns out to be a crucial object in the theory. Since the topology on Rep(G) can conveniently be reformulated in terms of convergence of functions of positive type associated with the representations, the same needs to be done with the topology on S(G) (Section 5.3). The theory culminates in Fell’s theorems on continuity of the processes of inducing and restricting representations. More precisely, the map (K, H, π) → (K, indG H π) from the subset of K(G) × S(G) consisting of all triples with H ⊆ K into

203

204

Topologies on dual spaces

S(G) is continuous (Theorem 5.39). A similar result holds for restricting representations (Theorem 5.37). The next three sections are devoted to investigating the dual space topology for several (classes of) locally compact groups to which Mackey’s theory can be applied to derive a setwise description of the dual space. The basic tools are the subgroup representation topology as well as continuity of the inducing and restricting processes. In Section 5.5 we study a number of two-step solvable and some simply connected nilpotent Lie groups, including the affine group and the Heisenberg groups. A very important class of locally compact groups is formed by the general motion groups, that is, semidirect products G = N  K, where N is an abelian locally compact group and K is a compact group. Special examples are the groups of rigid motions Rn  SO(n) of Rn , n ≥ 2. For general motion groups there exists a neat, though fairly involved description, due to  in terms of data provided by N  and irreducible Baggett [2], of the topology on G representations of subgroups of K. This result (Theorem 5.58) is presented in Section 5.6, followed by some specific examples in Section 5.7. In Section 5.7 we also deduce from Theorem 5.58 a criterion for when the dual space of a general motion group is Hausdorff. Let G be a two-step nilpotent locally compact group. In general, such groups are not of type I. Therefore Mackey’s theory does not apply to provide a  as a set. However, it is possible to method for computing the dual space G parametrize the primitive ideal space Prim(G) through group data. Moreover, employing Fell’s subgroup representation topology and continuity of the processes of inducing and restricting representations, the topology on the parameter space, which makes the bijection with Prim(G) a homeomorphism, can be identified. This is carried out in Section 5.8 and is illustrated by a number of examples.

5.1 The inner hull–kernel topology Let X be an arbitrary topological space and let C(X) denote the family of all closed subsets of X. For every compact subset C of X and every finite family F of nonempty open subsets of X, let U (C, F ) = {Y ∈ C(X) : Y ∩ C = ∅ and Y ∩ V = ∅ for all V ∈ F }. It is obvious that the collection of all such sets U (C, F) has the finite intersection property. Therefore, there is a unique topology on C(X) for which these sets form a basis. This topology is called the compact–open (or Fell) topology on C(X).

5.1 The inner hull–kernel topology

205

The first crucial result about C(X) is that it is compact. This was proved by Fell [48], using the concept of a universal net. Recall that a net (aα )α in A is called universal if given any subset B of A, then either aα ∈ B eventually or aα ∈ A \ B eventually. Then a topological space A is compact if and only if every universal net in A is convergent [122, theorem 1.6.2]. Lemma 5.1 If X is any topological space, then C(X) is compact in the compact–open topology. Proof Let (Yα )α be a universal net in C(X), and let Z be the set of all those x ∈ X such that for every neighborhood W of x, we have W ∩ Yα = ∅ eventually. Clearly, Z is a closed subset of X. It now suffices to show that Yα → Z in C(X). Let U (C, F) be a typical neighborhood of Z in C(X). Then, for every V ∈ F, there exists an element xV ∈ V ∩ Z, and hence, by the definition of Z, Yα ∩ V = ∅ for large α. Toward a contradiction, suppose we don’t have Yα ∩ C = ∅ eventually. Then there exists a subnet (Yαβ )β of (Yα )α such that Yαβ ∩ C = ∅ for all β. For each β, choose xβ ∈ Yαβ ∩ C. Since C is compact, after passing to a further subnet if necessary, we can assume that xβ → x for some x ∈ C. So, for each neighborhood W of x, Yαβ ∩ W = ∅ for large enough β. Then, since the net (Yα )α is universal, we have Yα ∩ W = ∅ eventually. This shows that x ∈ Z ∩ C, contradicting Z ∈ U (C, F). Thus Yα ∩ C = ∅ eventually, and since U (C, F) was arbitrary, we conclude that Yα → Z in C(X).  Lemma 5.2 Suppose that X is a locally compact space in the sense that every point of X has a basis of compact neighborhoods. Then C(X), equipped with the compact–open topology, is a compact Hausdorff space. Proof According to Lemma 5.1, it only remains to show that C(X) is Hausdorff. Let Y1 and Y2 be distinct elements of C(X). If Y1 ∩ Y2 = ∅, then U (∅, X \ Y1 ) ∩ U (∅, X \ Y2 ) = ∅ and Yj ∈ U (∅, X \ Yk ), j = k, 1 ≤ j, k ≤ 2. So, without loss of generality, assume that there exists some x ∈ Y1 \ Y2 . Since X is locally compact, there exists a compact neighborhood C of x such that C ∩ Y2 = ∅. Then Y2 ∈ U (∅, X \ C) and Y1 ∈ U (∅, C ◦ ), where C ◦ denotes the interior of  C. Finally, U (∅, X \ C) ∩ U (∅, C ◦ ) = ∅. Remark 5.3 Let Y be a closed subset of X. It is obvious that the Fell topology on C(Y ) equals the relative topology of the Fell topology of C(X). Lemma 5.4 Let X be a locally compact space and let (Yα )α be a net in C(X). Then Yα → Y in C(X) if and only if the following conditions are satisfied.

206

Topologies on dual spaces

(i) If yα ∈ Yα and yα → y in X, then y ∈ Y . (ii) If y ∈ Y , then there exists a subnet (Yαβ )β of (Yα )α and for each β, an element yβ of Yαβ such that yβ → y. Proof First assume that Yα → Y , and let yα ∈ Yα and y ∈ Y such that yα → y. Toward a contradiction, assume that y ∈ Y . Then there exists a compact neighborhood V of y with V ∩ Y = ∅. As yα → y, yα ∈ V eventually. Since U (V , ∅) is a neighborhood of Y in C(X), we have Yα ∈ U (V , ∅) and Yα ∩ V = ∅ eventually. This is a contradiction, and hence y ∈ Y . So (i) holds. For (ii), let y ∈ Y and consider the set B of all pairs (α, U ), where U is an open neighborhood of y such that U ∩ Yα = ∅. If (α1 , U1 ), (α2 , U2 ) ∈ B, then U (∅, U1 ∩ U2 ) is a neighborhood of Y and hence there exists α such that α ≥ α1 , α2 and Yα ∩ (U1 ∩ U2 ) = ∅. It follows that B can be partially ordered by defining (α, U ) ≥ (α  , U  ) if α ≥ α  and U ⊆ U  . For each (α, U ) ∈ B, choose yα,U ∈ Yα ∩ U . Then (yα,U )(α,U ) is a net converging to y. This proves statement (ii). Now suppose that (i) and (ii) hold and let U (C, F) be a typical neighborhood of Y . If we don’t have Yα ∩ C = ∅ eventually, then since C is compact, after passing to a subnet if necessary, we can assume that there exist yα ∈ Yα ∩ C such that yα → y for some y ∈ C. But then y ∈ Y by (i), which is a contradiction. Let V ∈ F and assume that V ∩ Yαβ = ∅ for some subnet (Yαβ )β of (Yα )α . If y ∈ Y ∩ V , then after passing to a further subnet, by (ii) we can assume that there exist yβ ∈ Yαβ such that yβ → y. Thus yβ ∈ V ∩ Yαβ for large β, and this contradiction completes the proof.  Let A be a C ∗ -algebra and fix a cardinal number κ such that κ ≥ dim H(π )  and κ is greater than or equal to the cardinality of A. Let for all π ∈ A Rep(A) denote the set of all unitary equivalence classes of nondegenerate ∗-representations of dimension ≤ κ of A. We remind the reader that for π ∈ Rep(A), the support of π, supp π, is defined to be  : ker ρ ⊇ ker π}. supp π = {ρ ∈ A  let Definition 5.5 For any finite family F of nonempty open subsets of A, U (F) = {π ∈ Rep(A) : supp π ∩ V = ∅ for all V ∈ F}. Since the collection of all such sets U (F ) is closed under forming finite intersections, there is a unique topology on Rep(A) for which the sets U (F ) form a basis of open sets. This topology is called the inner hull–kernel topology on Rep(A).

5.1 The inner hull–kernel topology

207

 the inner hull–kernel topology coincides with the Of course, relativized to A, hull–kernel topology. Also observe that the inner hull–kernel topology does not distinguish between elements of Rep(A) which are weakly equivalent, since such elements have the same support. Thus every subset of Rep(A) which is open in the inner hull–kernel topology, is closed under the equivalence relation of weak equivalence. Lemma 5.6 Rep(A) is compact in the inner hull–kernel topology.  assigning to π ∈ Rep(A) its supProof Consider the map φ : Rep(A) → C(A)  and taking for π the port. Then φ is surjective since given a closed subset S of A direct sum of all σ ∈ S, we have φ(π) = S. Now, if F is any finite collection of  then φ(U (F)) = U (∅, F). Since C(A)  is compact nonempty open subsets of A, in the compact–open topology, it follows that Rep(A) is compact in the inner hull–kernel topology.  Lemma 5.7 Let (πα )α be a net in Rep(A) and π ∈ Rep(A). (i) Then πα → π in Rep(A) if and only if every subnet of (πα )α weakly contains π. (ii) If πα → π and π weakly contains a representation σ ∈ Rep(A), then πα → σ . Proof (i) Let πα → π and let (παβ )β be a subnet of (πα )α . Let τ ∈ supp π and  Since παβ → π , supp(παβ ) ∩ V = let V be any open neighborhood of τ in A. ∅ eventually. Thus τ ∈ ∪β supp(παβ ), and since τ ∈ supp π was arbitrary, it follows that     ker π = k(supp π) ⊇ k ∪β supp παβ = k ∪β supp παβ & & = k(supp παβ ) = ker παβ . β

β

Thus π is weakly contained in the set (παβ )β . Conversely, suppose that (πα )α does not converge to π in Rep(A). Then  and a subnet (παβ )β of (πα )α such that there exist an open subset V of A  \ V for each V ∩ supp π = ∅, but supp(παβ ) ∩ V = ∅ for all β. Then παβ ≺ A β and since V ∩ supp π = ∅, π cannot be weakly contained in the set (παβ )β . (ii) If (παβ )β is any subnet of (πα )α , then παβ → π and hence σ ≺ π ≺ {παβ } by (i). The “only if” part of (i) shows that πα → σ .  Lemma 5.8 Suppose that A does not have an identity, and let Ae be the C ∗ algebra obtained by adjoining an identity e to A. For σ ∈ Rep(A), let  σ be the σ (e) = IH(σ ) . Then, with representation of Ae on H(σ ) extending σ such that 

208

Topologies on dual spaces

respect to the inner hull–kernel topologies on Rep(A) and Rep(Ae ), the map σ → σ is an embedding of Rep(A) into Rep(Ae ). e = {  ∪ {π∞ }, where π∞ is the one-dimensional Proof Recall that A σ : σ ∈ A} σ representation of Ae with kernel A. It is a standard fact that the map σ →   and the open set A e \ {π∞ } of A e . Identify is a homeomorphism between A Rep(A) with the subset { σ : σ ∈ RepA} of Rep(Ae ). It is straightforward to verify that Rep(A) is open in Rep(Ae ), equipped with the inner hull–kernel e , U (V ) ∩ Rep(A) = U (V ∩ topology. Moreover, for any open subset V of A  and V ∩ A  = ∅ when V = ∅. Therefore, for any finite collection F of A)  : V ∈ F }, U (F ) ∩ Rep(A) = e and F  = {V ∩ A nonempty open subsets of A  U (F ). This proves the lemma.  Let π ∈ Rep(A),  > 0, a1 , . . . , am ∈ A, m ∈ N, and let ϕ1 , . . . , ϕn , n ∈ N, be positive functionals on A associated with π. Given these data, let U = U (π, a1 , . . . , am , ϕ1 , . . . , ϕn , )

(5.1)

denote the set of all τ ∈ Rep(A) satisfying the following condition: there exist φ1 , . . . , φn , each of which is a sum of positive functionals associated with τ , such that |φj (ak ) − ϕj (ak )| <  for all 1 ≤ j ≤ n, 1 ≤ k ≤ m,

(5.2)

| φj − ϕj | <  for 1 ≤ j ≤ n.

(5.3)

Theorem 5.9 For every π ∈ Rep(A), the sets as in (5.1) form a neighborhood basis of π in the inner hull–kernel topology on Rep(A). Proof We first show that A can be assumed to be unital. Indeed, suppose that A does not have an identity and consider Ae . Regarding Rep(A) as a subset of Rep(Ae ), the inner hull–kernel topology on Rep(A) is just the relative topology of the corresponding topology on Rep(Ae ) (Lemma 5.8). On the other hand, every positive functional ψ on A extends uniquely to (e) = ψ . If ψ(a) = σ (a)ξ, ξ ,  on Ae satisfying ψ a positive functional ψ where σ ∈ Rep(A) and ξ ∈ H(σ ), then since A has an approximate identity, (e) = ψ  .

ψ = ξ 2 =  σ (e)ξ, ξ  = ψ Moreover, if ψ1 , . . . , ψr are positive functionals on A, then   r r      ψl  =

ψl .    l=1

l=1

It follows that the set U in (5.1) can be described as the set of all τ ∈ Rep(A) for which there exist positive functionals φ1 , . . . , φn , each of which is a sum

5.1 The inner hull–kernel topology

209

of positive functionals associated with  τ such that |ϕj (ak ) − φj (ak )| <  and   | ϕj − φj | = |ϕj (e) − φj (e)| <  (1 ≤ j ≤ n, 1 ≤ k ≤ m). Thus the sets U in (5.1) are exactly the intersections of Rep(A) with the sets Ue defined analogously using Ae instead of A. Therefore we may assume that A has an identity and then omit condition (5.3) from the definition of U . We prove next that every inner hull–kernel neighborhood of π contains some set U of the form (5.1). For that we have to show that if S is a subset of Rep(A) such that S ∩ U = ∅ for every such U , then π belongs to the inner hull–kernel closure of S. Since the sets U form a directed set (indexed by the ak , ϕj , and ), there exists a net (σα )α ⊆ S such that, for every U , σα ∈ U eventually.  such that Vj ∩ supp π = ∅ for Now let V1 , . . . , Vn be open subsets of A 1 ≤ j ≤ n, and let V = {τ ∈ Rep(A) : supp τ ∩ Vj = ∅ for 1 ≤ j ≤ n}. Since these sets V form a neighborhood basis of π in the inner hull–kernel topology, it suffices to show that S ∩ V = ∅ for any such V . To that end, for every 1 ≤ j ≤ n, choose πj ∈ Vj ∩ supp π . Let ϕ be a nonzero positive functional associated with π1 . Since π1 ≺ π , ϕ can be approximated in the w ∗ -topology by positive functionals ϕ  associated with π . For every α0 , by the definition of the sets U , ϕ  can be w ∗ -approximated by sums of the form ϕ  = ψ1 + . . . + ψr , where all ψl are positive functionals associated with σα for some α ≥ α0 . Next, since σα ≺ supp σα , each ψl can in turn be w ∗ -approximated by a sum of positive functionals associated with supp σα . Combining these facts, we see that, for each α0 , ϕ can be w ∗ -approximated by sums of positive functionals associated with ∪ {supp σα : α ≥ α0 }. Thus π1 belongs to the hull–kernel closure of the set ∪ {supp σα : α ≥ α0 }. It follows that there is a subnet (σαβ )β of (σα )α such that every neighborhood of π1 in  intersects supp(σαβ ) for all large enough β. We now repeat the arguments, A replacing (σα )α by (σαβ )β and π1 by π2 , and so on. Finally, we arrive at a subnet (σαγ )γ with the property that, for every 1 ≤ j ≤ n, every neighborhood of πj  intersects supp(σαγ ) for all large enough γ . In particular, there exists σαγ in A such that Vj ∩ supp(σαγ ) = ∅ for all 1 ≤ j ≤ n. This shows that S ∩ V = ∅, as required. To prove that, conversely, every U as in (5.1) contains an inner hull–kernel neighborhood of π, it suffices to show that if S ⊆ Rep(A) is such that π is in the inner hull–kernel closure of S, then C ∩ U = ∅. Thus, let U be as in (5.1).

rj ψj,l of Since π ∼ supp π , for every 1 ≤ j ≤ n, there exists a sum ψj = l=1 positive functionals ψj,l , 1 ≤ l ≤ rj , associated with supp π such that |ψj (ak ) − ϕj (ak )| < /2, 1 ≤ k ≤ m.

(5.4)

210

Topologies on dual spaces

Let ψj,l be associated with the representation σj,l ∈ supp π. Next, choose δ > 0 such that 4δ · max rj < , 1≤j ≤n

(5.5)

 such that and for each 1 ≤ j ≤ n and 1 ≤ l ≤ rj , let Wj,l be the set of all ρ ∈ A there exists a positive functional φρ associated with ρ satisfying, for 1 ≤ k ≤ m, |φρ (ak ) − ψj,l (ak )| < δ. Then Wj,l is a hull–kernel neighborhood of σj,l . Indeed, if C is any subset of  with σj,l ∈ C, then C ∩ Wj,l = ∅ by Theorem 1.61. Now, let A V = {τ ∈ Rep(A) : supp τ ∩ Wj,l = ∅, for 1 ≤ j ≤ n, 1 ≤ l ≤ rj }. Then, since σj,l ∈ supp π, V is an inner hull–kernel neighborhood of π, and hence there exists some σ ∈ V ∩ S. Choose ρj,l ∈ Wj,l ∩ supp σ , and let φj,l be a positive functional associated with ρj,l satisfying |φj,l (ak ) − ψj,l (ak )| < δ, 1 ≤ k ≤ m.

(5.6)

Since ρj,l is irreducible and ρj,l ≺ σ , there exists a positive functional ωj,l associated with σ such that |ωj,l (ak ) − φj,l (ak )| < δ, 1 ≤ k ≤ m. (5.7)

rj Finally, for 1 ≤ j ≤ n§, let ωj = l=1 ωj,l , a positive functional associated with σ ·. Combining (5.4), (5.5), (5.6), and (5.7), we obtain |ωj (ak ) − ϕj (ak )| ≤ /2 + |ωj (ak ) − ψj (ak )| ≤ /2 +

rj 

|ωj,l (ak ) − ψj,l (ak )|

l=1 rj

≤ /2 +





|ωj,l (ak ) − φj,l (ak )| + |φj,l (ak ) − ψj,l (ak )|

l=1

≤ /2 + 2δ · rj < , for all 1 ≤ k ≤ m. Thus σ ∈ U , and this finishes the proof.



Remark 5.10 Suppose that the representation π in Theorem 5.9 is irreducible. Then a neighborhood basis of π in the inner hull–kernel topology on Rep(A) is given by the sets U  (π, a1 , . . . , am , ϕ1 , . . . , ϕn , ), which are defined in the same way as the sets U (π, a1 , . . . , am , ϕ1 , . . . , ϕn , ) except that each φj must be a positive functional associated with τ , rather than merely a sum of such. Indeed, when π is irreducible, then for the ψj in (5.4) we do not need sums

5.1 The inner hull–kernel topology

211

of more than one term. Otherwise, the proof of Theorem 5.9 goes through as before. Proposition 5.11 For π ∈ Rep(A) and T ⊆ Rep(A), consider the following conditions. (i) π is weakly contained in T .  (ii) supp π is contained in the closure of ∪τ ∈T supp τ in A. (iii) π belongs to the inner hull–kernel closure of T in Rep(A). Then (i) and (ii) are equivalent, (iii) implies (ii), and if π is irreducible, then all three conditions are equivalent. Proof (i) ⇒ (ii) Let σ ∈ supp π. Then, by (i), : . & & ker σ ⊇ ker π ⊇ ker τ ⊇ supp τ . ker ρ : ρ ∈ τ ∈T

5

τ ∈T

 Since σ was an arbitrary supp τ in A.

So σ belongs to the closure of τ ∈T element of supp π, (ii) follows. (ii) ⇒ (i) Since ρ ∼ supp ρ for any representation ρ, (ii) implies that π ≺ supp π ⊆ ∪{supp τ : τ ∈ T } ≺ T .

 (iii) ⇒ (ii) Let σ ∈ supp π and let V be any open neighborhood of σ in A. Then π ∈ U ({V }), and hence there exists τ ∈ T such that V ∩ supp τ = ∅. So σ ∈ ∪τ ∈T supp τ , and since σ was an arbitrary element of supp π , π satisfies (ii). Finally, suppose that π is irreducible and that (ii) holds. To show that π satisfies (iii), let U (F) be a typical inner hull–kernel neighborhood of π in Rep(A), where F = {V1 , . . . , Vn } is a collection of nonempty open subsets of  \ Vj and hence  Then π ∈ Vj for all j , since otherwise supp π ∼ π ≺ A A. n supp π ∩ Vj = ∅ for some j . So V = ∩j =1 Vj is a hull–kernel neighborhood  Then, by (ii), supp τ ∩ V = ∅ for some τ ∈ T . Thus τ ∈ U (F ), as of π in A. required.  Now let G be a locally compact group. It will be useful to express the inner hull–kernel topology on Rep(G) = Rep(C ∗ (G)) in terms of functions on G. For this, we introduce the following terminology. Recall that a function ϕ of positive type on G is associated with a family S of unitary representations of G if there exist σ ∈ S and a vector ξ ∈ H(σ ) such that ϕ(x) = σ (x)ξ, ξ  for all x ∈ G. Let π be a unitary representation of G. Given a compact subset C of G,  > 0, and finitely many functions ϕ1 , . . . , ϕn of positive type associated with π , we

212

Topologies on dual spaces

define W (π, C, ϕ1 , . . . , ϕn , ) to be the set of all τ ∈ Rep(G) = Rep(C ∗ (G)) such that there exist φ1 , . . . , φn , each of which is a sum of functions of positive type associated with τ and for which |φj (x) − ϕj (x)| <  holds for all x ∈ C and 1 ≤ j ≤ n. Theorem 5.12 Let G be a locally compact group and π ∈ Rep(G). Then the sets W = W (π, C, ϕ1 , . . . , ϕn , ) form a neighborhood basis of π in the inner hull–kernel topology of Rep(G). Proof Let U = U (π, a1 , . . . , am , ϕ1 , . . . , ϕn , ), a1 , . . . , am ∈ C ∗ (G), be a typical inner hull–kernel neighborhood of π as given by Theorem 5.9. We are going to show that some set W is contained in U . To that end, without loss of generality, we can assume that  ≤ 1, ϕj ≤ 1, 1 ≤ j ≤ n, and ak ∈ L1 (G),

ak 1 ≤ 1, 1 ≤ k ≤ m (note that the norm of ϕj as an element of C ∗ (G)∗ is the same as the norm as an element of L1 (G)∗ ).  Choose a compact subset C of G containing e such that G\C |ak (x)|dx < /4, for 1 ≤ k ≤ m, and let W = W (π, C, ϕ1 , . . . , ϕn , /4). If σ ∈ W , there are sums φ1 , . . . , φn of functions of positive type associated with σ such that |φj (x) − ϕj (x)| < /4 for all x ∈ C and j = 1, . . . , n. In particular, since e ∈ C and the φj and ϕj are of positive type, | φj − ϕj | = |φj (e) − ϕj (e)| < /4, for 1 ≤ j ≤ n. Moreover, for j = 1, . . . , n and k = 1, . . . , m, the preceding two estimates imply that  |ak (x)| · |φj (x) − ϕj (x)|dx |φj , ak  − ϕj , ak | < /4 · ak 1 + G\C  ≤ /4 + (2 + /4) |ak (x)|dx ≤ . G\C

It follows that σ ∈ U and so, since σ is an arbitrary element of W , W ⊆ U . Conversely, let W = W (π, C, ϕ1 , . . . , ϕn , ) be given, and suppose that W contains none of the sets U . Then, by Theorem 5.9, we can find a net of unitary representations σα in Rep(G) \ W with the following property: for each α and each j = 1, . . . , n, there exists a sum ϕj,α of positive functionals on C ∗ (G) associated with σα such that ϕj = w ∗ − limα ϕj,α and ϕj = limα ϕj,α , for 1 ≤ j ≤ n. In particular, the norms ϕj,α may be assumed to be bounded in α. It follows that ϕj,α (f ) − ϕj (f ) → 0

(5.8)

uniformly on any norm-compact subset of L1 (G). Now, let ξj ∈ H(σ ) such that ϕj (x) = σ (x)ξj , ξj , x ∈ G. Using an approximate identity in L1 (G), we can

5.1 The inner hull–kernel topology

213

approximate ξj by elements of H(σ ) of the form σ (g)ξj . More precisely, we find g ∈ L1 (G) such that, for all x ∈ G and j = 1, . . . , n, |σ (x)σ (g)ξj , σ (g)ξj  − ϕj (x)| < /2.

(5.9)

Since the map x → g ∗ ∗ Lx g from G into L1 (G) is continuous, the functions of the form g ∗ ∗ Lx g, x ∈ C, form a compact subset of L1 (G). Thus, by (5.8), for each 1 ≤ j ≤ n, ϕj,α (g ∗ ∗ Lx g) − ϕj (g ∗ ∗ Lx g) → 0

(5.10)

uniformly for x ∈ C. Since ϕj (g ∗ ∗ Lx g) = σ (x)σ (g)ξj , σ (g)ξj , (5.9) and (5.10) yield the existence of some α such that |ϕj,α (g ∗ ∗ Lx g) − ϕj (x)| < 

(5.11)

for all x ∈ G and j = 1, . . . , n. However, every ϕj,α is a sum of positive functionals associated with σα , and then so is the function x → ϕj,α (g ∗ ∗ Lx g). Thus (5.11) shows that σα ∈ W . This contradiction finishes the proof.  Suppose that the representation π in Theorem 5.12 is irreducible. Then, just as in the case of Theorem 5.9 (compare Remark 5.10), a basis of inner hull– kernel neighborhoods of π is formed by the sets which are defined like the sets W (π, C, ϕ1 , . . . , ϕn , ), except that each φj must be a function of positive type associated with σ (rather than merely a sum of such functions). We conclude this section by showing that the operation of forming tensor products is continuous with respect to the inner hull–kernel topology. Lemma 5.13 Let S ⊆ Rep(G) and τ ∈ Rep(G) and let K denote the set of all τ can be approximated, uniformly ξ ∈ H(τ ) such that the coefficient function ϕξ,ξ on compact sets, by sums of functions of positive type associated with S. Then K is a closed τ -invariant linear subspace of H(τ ). Proof Clearly, K is closed in the norm and under scalar multiplication. Moreover, it is easily verified that if ξ ∈ K, x1 , . . . , xn ∈ G, and λ1 , . . . , λn ∈ C,

n ∈ N, then ni=1 λi τ (xi )ξ ∈ K. In particular, K is τ -invariant. It only remains to show that K is closed under addition. Fix ξ, η ∈ K and let ω = ξ + η. Let L and M be the closed τ -invariant subspaces of H(τ ) generated by ξ and η, respectively, and let N denote the closure of L + M. Then, by the first paragraph, L and M are both contained in K. Let P denote the projection onto L⊥ . Then P (M) is a dense subspace of N ∩ L⊥ and hence by Mackey’s form of Schur’s lemma (theorem 1.2 of [105]), the restriction of τ to the invariant subspace N ∩ L⊥ is equivalent to a subrepresentation of τ M . Since M ⊆ K, it follows that N ∩ L⊥ ⊆ K.

214

Topologies on dual spaces

Now ω can be written as ω = ξ  + η , where ξ  ∈ L and η ∈ N ∩ L⊥ . As L and N ∩ L⊥ are orthogonal and τ -invariant, it follows that π(x)ω, ω = π (x)ξ  , ξ   + π (x)η , η , for all x ∈ G. This equation implies that ω ∈ K because ξ  ∈ L ⊆ K and η ∈ N ∩ L⊥ ⊆ K. Thus K is closed under addition.  Proposition 5.14 The map (π, ρ) → π ⊗ ρ from Rep(G) × Rep(G) into Rep(G) is continuous with respect to the inner hull–kernel topology on Rep(G). Proof Let πα → π and ρα → ρ in Rep(G). By Lemma 5.7 we only have to show that the net (πα ⊗ ρα )α (and hence by the same argument every subnet of it) weakly contains π ⊗ ρ. Let ξ ∈ H(π ) and η ∈ H(ρ). Then Theorem 5.9 shows that the function of positive type associated with ξ ⊗ η can be approximated uniformly on compact sets by sums of functions of positive type associated with the representations πα ⊗ ρα . Since the closed linear span of such vectors ξ ⊗ η equals H(π) ⊗ H(ρ) = H(π ⊗ ρ), Lemma 5.13 applied to τ = π ⊗ ρ and S = (πα ⊗ ρα )α yields that π ⊗ ρ is weakly contained in (πα ⊗ ρα )α . 

5.2 The subgroup C ∗ -algebra Let G be a locally compact group. In this section a subgroup algebra As (G) and a subgroup C ∗ -algebra Cs∗ (G), a completion of As (G) with respect to a C ∗ norm on As (G), are introduced and studied. As (G) is a Banach ∗-algebra with a natural fibering, the base space being the space K(G) of all closed subgroups of G and the fiber at K ∈ K(G) being L1 (K). This naturally leads to a topology on the set of all pairs (K, π), where K ∈ K(G) and π is a representation of K. Lemma 5.15 Let G be a locally compact group. Then the set K(G) of all closed subgroups of G is closed in C(G) and hence a compact Hausdorff space. Proof Let (Hα )α be a net of closed subgroups of G such that Hα → Y for some Y ∈ C(X). Then, since e ∈ Hα for all α, e ∈ Y by Lemma 5.4(i). If x, y ∈ Y then, after consecutively passing to subnets twice and relabeling, by Lemma 5.4(ii) there exist xα , yα ∈ Hα such that xα → x and yα → y. Then xα yα ∈ Hα and xα yα → xy, which implies that xy ∈ Y . Similarly, it is shown that x −1 ∈ Y whenever x ∈ Y .  It might be helpful to know that if the given locally compact group G is second countable, then one can often work with sequences rather than nets in K(G).

5.2 The subgroup C ∗ -algebra

215

Lemma 5.16 Let G be a second countable locally compact group. Then C(G) is second countable. Proof Let (Ui )i∈N be a countable basis for the topology of G such that Ui is compact for every i. It suffices to verify that the sets   U ∪i∈I Ui , Uk1 , . . . , Ukn , where k1 , . . . , kn ∈ N and I is a finite subset of N, form a basis for the compact– open topology. Thus let a compact subset C of G and a finite family F of nonempty open subsets of G be given, and let H ∈ U (C, F). Since H ∩ C = ∅, for each x ∈ C there exists ix ∈ N such that x ∈ Uix and Uix ∩ H = ∅. As C is compact, we find elements x1 , . . . , xm of C such that C ⊆ ∪m l=1 Uixl . Then D = ∪m U is compact and satisfies H ∩ D = ∅. Moreover, for each V ∈ F, i l=1 xl there exists kV ∈ N such that H ∩ UkV = ∅ and UkV ⊆ V . Then H ∈ U (D, {UkV : V ∈ F}) ⊆ U (C, F), and this completes the proof.



The following straightforward lemma will be used in a later section. Lemma 5.17 Let N be a closed normal subgroup of G and let KN (G) = {H ∈ K(G) : H ⊇ N }. Then the map H → H /N is a homeomorphism between KN (G) and K(G/N). Proof Let q : G → G/N denote the quotient homomorphism. Note that, for H ∈ KN (G) and any subset M of G, H ∩ M = ∅ if and only if H ∩ MN = ∅, and hence if and only if q(H ) ∩ q(M) = ∅. It follows that, for any compact subset C of G and any finite family V of nonempty open subsets of G, q(U (C, V) ∩ KN (G)) = U (q(C), {q(V ) : V ∈ V}). Since every compact subset of G/N is of the form q(C) for some compact subset C of G, the statement of the lemma follows.  The main tool in constructing the object in the title of this section is the existence of a family of Haar measures μK , K ∈ K(G), which depend continuously on K in the sense of the following definition. Definition 5.18 Let G be a locally compact group. A smooth choice of Haar measures on K(G) is a mapping K → μK assigning to each K ∈ K(G) a left  Haar measure μK such that, for every f ∈ Cc (G), the function K → K f (x)dμK (x) is continuous on K(G).

216

Topologies on dual spaces

Before recalling the essential technique used to prove the existence of a smooth choice of Haar measures, namely that of a generalized limit, we include a simple lemma. Lemma 5.19 For each compact subset C of G there exists a constant r(C) > 0 such that μK (C ∩ K) ≤ r(C) for all K ∈ K(G). Proof We fix a function g ∈ Cc+ (G) such that g(e) > 0 and a neighborhood U of e such that g(t) ≥ η for some η > 0 and all t ∈ U . For x ∈ C, choose a neighborhood Ux of x such that Ux−1 Ux ⊆ U . A finite number of these neighborhoods, Ux1 , . . . , Uxn say, cover C. Let I = {i ∈ {1, . . . , n} : Uxi ∩ K = ∅}, and for each i ∈ I , choose yi ∈ Uxi ∩ K. Then it is easily verified that   1 n g(yi−1 x)dμK (x) ≤ . μK (C ∩ K) ≤ η K i∈I η Since neither n nor η depends on K, r(C) = n/η has the required property.  Since we cannot expect the reader to be familiar with the notion of a generalized limit, we are now going to discuss it. In doing so, we follow the exposition in Williams [153, pp. 457/8]. Let A be a directed set and let cA be the space of all complex-valued bounded nets (xα )α∈A such that limα xα exists. If A = N, then cN is the space of all convergent sequences, which is usually denoted by c. Note that cA is a subspace of the Banach space ∞ (A) equipped with the norm x = supα∈A |xα |. The linear functional γA sending x ∈ cA to limα xα is of norm one. A generalized limit over A is any extension  of γA to ∞ (A) such that  = 1 and (x) ≥ 0 whenever xα ≥ 0 for all α. To see that generalized limits exist, one can proceed as follows. Let ∞ R (A) be the Banach space of all real valued bounded functions on A. Define cA1 and γA1 in analogy with cA and γA above, and let 1 be any extension of norm 1 one of γA1 to ∞ R (A). If x = (xα )α ∈ cA is such that xα ≥ 0 for all α, we must have 1 (x) ≥ 0. Indeed, assuming that 1 (x) < 0 and taking y = (yα )α with yα = x ∞ for all α, it follows that x − y ∞ ≤ x ∞ and |1 (x − y)| =

x ∞ − 1 (x) > x ∞ . This contradicts 1 = 1. If x = y + iz with y, z ∈ ∞ R (A), let (x) = 1 (y) + i1 (z). It is easy to verify that  defines a linear functional on ∞ (A) extending γA . Clearly,  ≤ 2 and (x) ≥ 0 if xα ≥ 0 for all α. We claim that actually  = 1. To see this, let x be an element in the unit ball of ∞ (A) attaining only finitely many values. Thus, let A1 , . . . , An be a partition of A and c1 , . . . , cn ∈ C such that

5.2 The subgroup C ∗ -algebra |cj | ≤ 1 and x =

n

j =1 cj 1Aj .

(x) = But 1 (1Aj ) ≥ 0 and 1 for all j , we get

n

|(x)| ≤

Then

n 

cj (1Aj ) =

j =1

j =1

217

n 

cj 1 (1Aj ).

j =1

(1Aj ) = 1 (1A ) ≤ 1 since 1 = 1. Since |cj | ≤

n 

|cj | · 1 (Aj ) ≤

j =1

n 

1 (Aj ) ≤ 1.

j =1

If now y is an arbitrary element in the unit ball of ∞ (A) and  > 0, then there exists x in the unit ball of ∞ (A) such that y − x ≤  and x attains only finitely many values. Since  is continuous and  > 0 is arbitrary, it follows that |(y)| ≤ 1. Thus  = 1, and consequently  is a generalized limit over A. Notice that if x and y are nets over A and if xα = yα for sufficiently large α, then (x − y)α = 0 for large α and hence (x) = (y) for every generalized limit . Consequently, for every net x and generalized limit , |(x)| ≤ lim sup |xα | = inf {sup |xβ |}. α

α

β≥α

Now suppose that B is a directed set and that there is a map β → αβ from B into A with the property that for every α0 ∈ A there exists β0 ∈ B such that β ≥ β0 implies αβ ≥ α0 . Then, for each x ∈ ∞ (A), β → xαβ is a subnet of (xα )α . Let cA (B) = {x ∈ ∞ (A) : lim xαβ exists}. β

∞

Then cA (B) is a subspace of  (A) containing cA . Let γB denote the norm one linear functional on cA (B) given by γB (x) = limβ xαβ . Then, replacing A with B, the above discussion shows that there exists a generalized limit over A which restricts to γB on cA (B). In particular, if x ∈ ∞ (A) has a subnet converging to c, then there exists a generalized limit  over A such that (x) = c. It follows that c = limα xα if and only if c = (x) for all generalized limits  over A. Theorem 5.20 Let G be a locally compact group. There exists a smooth choice of Haar measures on K(G). Proof Choose g ∈ Cc (G) such that g ≥ 0 and g(e)  > 0. For each K ∈ K(G), let μK be the left Haar measure on K satisfying K g(x)dμK (x) = 1. We are going to show that K → μK is a smooth choice of Haar measures. To that end, fix K ∈ K(G) and let (Kα )α∈A be a net in K(G) converging to K and let 

218

Topologies on dual spaces

be any generalized limit over A. If f ∈ Cc (G), then choosing C = supp f in Lemma 5.19, it follows that the function  f (x)dμKα (x) α → sα (f ) = Kα

is bounded on A. Set φ(f ) = ((sα (f ))α ). Then φ is a positive linear functional on Cc (G). Using φ, we want to define a positive linear functional on Cc (K). For that, we first observe that if f ∈ Cc (G) is such that f |K = 0, then φ(f ) = 0. This can be seen as follows. For each n ∈ N, we can find fn ∈ Cc (G) such that

fn − f ∞ ≤ 1/n and  # 1 supp fn ⊆ x ∈ G : |f (x)| ≥ . n Then K ∈ U (supp fn , G \ supp fn ) and if Kα ∈ U (supp fn , G \ supp fn ), then Kα fn (x)dμKα (x) = 0. This implies that φ(fn ) = 0 for all n and hence φ(f ) = 0. Since every h ∈ Cc (K) extends to some f ∈ Cc (G), we can now define a linear functional ϕ on Cc (K) by setting ϕ(f |K ) = φ(f ), for f ∈ Cc (G). Then φ is positive since Cc+ (G)|K = Cc+ (K). We proceed to show that ϕ is left invariant. Let f ∈ Cc (G), a ∈ K, and  > 0 be given. Fix a compact neighborhood W of a. Then, for b ∈ W , the function x → f (ax) − f (bx) has support contained in W −1 · supp f . Let r(W −1 · supp f ) be the constant provided by Lemma 5.19. The uniform continuity of f implies that there exists a neighborhood V of a, contained in W , such that  |f (ax) − f (bx)| ≤ r(W −1 · supp f ) for all b ∈ V . Thus, by Lemma 5.19,       f (ax)dμH (x) − f (bx)dμH (x) ≤   H

H

for all b ∈ V and H ∈ K(G). Since K ∈ U (∅, V ), Kα ∈ U (∅, V ) for all α ≥ α0 , and hence for every α ≥ α0 there exists some aα ∈ Kα ∩ V . As        = = ϕ(f |K ), f (aα x)dμKα (x) f (x)dμKα (x) Kα

Kα

α

α

we obtain that |ϕ(L

a −1

       . (f |K )) − ϕ(f |K )| ≤  (f (aα x) − f (ax))dμKα (x)  Kα

α

5.2 The subgroup C ∗ -algebra

219

Now, since  is a generalized limit and      (f (ax) − f (aα x))dμK (x) ≤ , α   Kα

for all α ≥ α0 , we conclude that |ϕ(La −1 (f |K )) − ϕ(f |K )| ≤   . Since  > 0 is arbitrary, it follows that ϕ is a left-invariant Haar integral on Cc (K). Since    ϕ(g|K ) =  = (1) = 1, g(x)dμKα (x) Kα

α

the choice of g and the uniqueness of the left Haar integral up to a multiplicative constant imply that  f (x)dμK (x) φ(f ) = ϕ(f |K ) = K

for all f ∈ Cc (G). The right-hand side of this equation is independent of the choice of , and hence so is the left-hand side. As we have pointed out prior to the theorem, this means that the limit limα→∞ sα (f ) exists and   f (x)dμKα (x) = f (x)dμK (x) lim α→∞ K α

K

for all f ∈ Cc (G). This finishes the proof of the theorem.



Notice that if K → μK and K → νK are two smooth choices of Haar measure, then there exists a continuous function u : K(G) → (0, ∞) such that νK = u(K)μK for all K ∈ K(G). Indeed, if f ∈ Cc+ (G) is such that f (e) > 0, then the function u, defined by −1   f (x)dμK (x) f (x)dνK (x) u(K) = K

K

on K(G), is continuous. In the sequel, we fix once and for all a smooth choice of Haar measures μK , K ∈ K(G). Let Y be the set of all pairs (K, x), where K ∈ K(G) and x ∈ K. It follows from Lemma 5.4 that Y is a closed subset of K(G) × G. Lemma 5.21 For f ∈ Cc (Y), the function  Ff : K → f (K, x)dμK (x) K

is continuous on K(G). Proof Extend f to a continuous function g with compact support on K(G) × G, and choose a compact subset C of G such that supp g ⊆ K(G) × C. Select

220

Topologies on dual spaces

h ∈ Cc+ (G) such that h = 1 on C, and set  # R = sup h(x)dμK (x) : K ∈ K(G) . K

 Then R < ∞ since K(G) is compact and the function K → K h(x)dμK (x) is continuous on K(G). Now, let K0 ∈ K(G) and  > 0 be given. Since g is uniformly continuous on K(G) × G, there exists a neighborhood V1 of K0 in K(G) such that |g(K, x) − g(K0 , x)| ≤ /2R, for all K ∈ V1 and all x ∈ G. Moreover, according to Definition 5.18 there exists a neighborhood V2 of K0 such that       g(K0 , x)dμK (x) −  ≤ /2, g(K , x)dμ (x) 0 K0   K

K0

for all K ∈ V2 . Then, since h = 1 on C, for K ∈ V1 ∩ V2 ,  |Ff (K) − Ff (K0 )| ≤ |g(K, x) − g(K0 , x)| · h(x)dμK (x) K       +  g(K0 , x)dμK (x) − g(K0 , x)dμK0 (x) K

≤R· This shows that K →

K0

sup

{|g(K, x) − g(K0 , x)|} + /2 ≤ .

x∈G,K∈V1

 K

f (K, x)dμK (x) is continuous.



For K ∈ K(G), let K denote the modular function of K. Since the functions   (K, x) → f (yx)dμK (y) and K → f (y)dμK (y) K

K

are continuous on Y and on K(G), respectively, and since   f (yx)dμK (y) = f (y)dμK (y), K (x) K

K

for all f ∈ Cc (G), it follows that the function (K, x) → K (x) is continuous on Y. We now define a convolution and an involution on Cc (Y) by setting  f (K, xy)g(K, y −1 )dμK (y) (f ∗ g)(K, x) = K ∗

and f (x) = K (x) · f (K, x −1 ), for all f, g ∈ Cc (Y), K ∈ K(G), and x ∈ K. Note that every element of Cc (Y) may be thought of as a function on K(G) whose value at K is in Cc (K). Since K(G) is compact, Lemma 5.21 yields that #  |f (K, x)|dμK (x) : K ∈ K(G)

f = sup K

5.2 The subgroup C ∗ -algebra

221

is finite for every f ∈ Cc (Y), and it is easily verified that f → f defines a norm on Cc (Y) which is also submultiplicative and satisfies f ∗ = f . Thus Cc (Y) becomes a normed ∗-algebra. The completion As (G) of Cc (Y) with respect to this norm is a Banach ∗-algebra, called the subgroup algebra of G. For each K ∈ K(G) and f ∈ Cc (Y), define fK : K → C by fK (x) = f (K, x), x ∈ K. Then f → fK is a continuous ∗-homomorphism of Cc (Y) onto Cc (K) ⊆ L1 (K, μK ) and it therefore extends to a continuous ∗homomorphism, denoted φK , from As (G) onto a dense subalgebra of L1 (K, μK ). Note that, by definition of the norm on Cc (Y), φK ≤ 1 for every K ∈ K(G). Lemma 5.22 For each f ∈ As (G), the function K → φK (f ) is continuous on K(G) and f = supK∈K(G) φK (f ) . Proof For f ∈ Cc (Y), the first statement follows from Lemma 5.21 since 

φK (f ) = |f (K, x)|dμK (x), K

and the second is immediate from the definition of the norm on Cc (Y). For arbitrary f ∈ As (G), we only have to observe that, since φK (f ) ≤ 1 for every K ∈ K(G), the function K → φK (f ) is a uniform limit on K(G) of  functions K → φK (g) , where g ∈ Cc (Y). Let K be a closed subgroup of G and π a unitary representation of K. Then π gives rise to a ∗-representation σK,π = π ◦ φK of As (G). Explicitly, for f ∈ Cc (Y),  σK,π (f ) = f (K, x)π(x)dμK (x). K

Clearly, if π is irreducible, then so is σK,π . It follows from Lemma 5.22 that these ∗-representations of As (G) separate the elements of As (G). Therefore, we can associate with As (G) a C ∗ -algebra as follows. Definition 5.23 Let S(G) denote the set of all subgroup representations, that is, pairs (K, π) where K is a closed subgroup of G and π ∈ Rep(K). The completion of As (G) with respect to the C ∗ -norm

f c = sup { σK,π (f ) : (K, π) ∈ S(G)} is a C ∗ -algebra, called the subgroup C ∗ -algebra and denoted Cs∗ (G). We equip S(G) with the topology which makes the one-to-one map (K, π) → σK,π from S(G) into Rep(Cs∗ (G)) a homeomorphism with respect

222

Topologies on dual spaces

to the inner hull–kernel topology on Rep(Cs∗ (G)). We shall also refer to this topology on S(G) as the subgroup representation topology. In the sequel, corresponding representations of As (G) and of Cs∗ (G) will be designated by the same letter. It will be seen later that every irreducible representation of Cs∗ (G) arises in this manner. We continue with some properties of the topological space S(G). Lemma 5.24 The topology on S(G) does not depend on the particular smooth choice of Haar measures. Proof Let {μK } and {μK } be two smooth choices of Haar measure. As we have noticed before Lemma 5.21, there exists a positive continuous function u on  denote K(G) such that μK = u(K)μK for all K ∈ K(G). Let As (G) and σK,π  the subgroup algebra and lifted representations associated with {μK }. Define a mapping F : Cc (Y) → Cc (Y) by F (f )(K, x) = u(K)−1 f (K, x), for (K, x) ∈ Y and f ∈ Cc (Y). It is obvious from the definition of the norms on As (G) and As (G) that F extends to an isometric ∗-isomorphism, also  ◦ F = σK,π for every denoted F , from As (G) onto As (G). Moreover, σK,π (K, π) ∈ S(G). The lemma follows from these facts.  Lemma 5.25 The set {σK,π : (K, π) ∈ S(G)} is closed in Rep(As (G)), equipped with the inner hull–kernel topology. In particular, S(G) is a compact space. Proof Suppose that (Kα , πα )α is a net in S(G) such that (σKα ,πα )α converges to some ρ ∈ Rep(As (G)). We have to show that ρ is lifted from some subgroup K of G. Since K(G) is compact, after passing to a subnet if necessary, we can assume that Kα → K for some K ∈ K(G). Let f ∈ As (G) be such that φK (f ) = 0. Then, by Lemma 5.22,

ρ(f ) ≤ lim inf α σKα ,πα (f ) = lim inf α πα (φKα (f )) ≤ lim inf α φKα (f ) = φK (f ) = 0. So ρ(f ) = 0. This shows that ker φK ⊆ ker ρ, and therefore ρ is lifted from a representation of K.  Since Rep(As (G)) is compact, it follows that S(G) is compact.

5.3 Subgroup representation topology

223

Lemma 5.26 The projection (K, π) → K from S(G) onto K(G) is continuous. Proof Let C be a closed subset of K(G), and let J = {f ∈ Cc (Y) : f (K, x) = 0 for all x ∈ K whenever K ∈ C}. Then J is an ideal of Cc (Y). Let I be its closure in Cs∗ (G) and let  = {σ ∈ Rep(Cs∗ (G)) : ker σ ⊇ I }. Then  is closed in Rep(Cs∗ (G)), and hence the set {(K, π) : σK,π ∈ } is closed in S(G). Now, if K ∈ C then σK,π (J ) = {0} for all representations π of K and hence σK,π ∈ . Conversely, if K ∈ C then there exist f ∈ I and x ∈ K such that f (K, x) = 0 and hence σK,π (f ) = 0 for some representation π of K. Thus σK,π ∈ . This shows that {(K, π) : σK,π ∈ } = {(K, π) : K ∈ C}, 

as required.

Now we are in a position to determine the dual space of the subgroup C ∗ algebra Cs∗ (G). Theorem 5.27 Let G be a locally compact group. (i) Every irreducible ∗-representation of Cs∗ (G) is of the form σK,π for a unique (K, π) ∈ S(G). ∗  (ii) C s (G) is the union of the pairwise disjoint closed subsets  K ∈ K(G). {σK,π : π ∈ K}, Proof (i) Let σ be an irreducible representation of Cs∗ (G). Then there exists K ∈ K(G) such that ker σ ⊇ {f ∈ Cc (Y) : fK = 0} (compare the proof of Lemma 5.25). Hence σ is lifted from some irreducible representation of L1 (K, μK ). The uniqueness of K is evident. ∗   (ii) By (i), C s (G) is the union of the sets K = {σK,π : π ∈ K}, K ∈ K(G), ∗  and these sets are pairwise disjoint. Each K is closed in Cs (G) since the map (K, π) → K from S(G) to K(G) is continuous (Lemma 5.26). 

5.3 The subgroup representation topology and functions of positive type It is a well-known fact that the w∗ -topology on the set of characters of a locally compact abelian group coincides with the topology of uniform convergence

224

Topologies on dual spaces

on compact subsets. We are now going to generalize this in order to describe the inner hull–kernel topology on the set S(G) of subgroup representations. The first step is to define a topology on the set ∪{C(K) : K ∈ K(G)} so that a function which is defined on K may approach a function defined on K0 as K approaches K0 in K(G). To start with, let X be an arbitrary locally compact Hausdorff space and let S = X × C. A subset of S is called semicompact if it is closed and its projection to X is relatively compact. Let D denote the family of all semicompact subsets of S and recall that C(S) is the family of all closed subsets of S. For D ∈ D and any finite family F of nonempty open subsets of S, let U (D, F) = {A ∈ C(S) : A ∩ D = ∅, A ∩ V = ∅ for all V ∈ F}. It is clear that the collection of sets U (D, F) does have the finite intersection property. The semicompact–open topology of C(S) is that topology for which the sets U (D, F) form an open basis. Let E(X) = {C(T ) : T ∈ C(X)}. Thus the elements of E(X) are precisely the complex valued continuous functions f defined on a closed subset of X. Let f ∈ E(X) with domain D(f ) ∈ C(X). Then identifying f with its graph {(x, f (x)) : x ∈ D(f )}, which is a closed subset of S, we can view E(X) as a subset of C(S). We shall always consider E(X) as equipped with the relative semicompact–open topology. The next few results concerning the topological space E(X) will be used in the sequel. Lemma 5.28 The map f → D(f ) from E(X) to C(X) is continuous. Proof Let C be a compact subset of X and F a finite family of open subsets of X. Then the inverse image of U (C, F) under the map f → D(f ) is the set {f ∈ E(X) : D(f ) ∩ C = ∅, D(f ) ∩ V = ∅ for all V ∈ F}. However, this set equals U (C × C, {V × C : V ∈ F}), which is open in E(X). Since the sets U (C, F) form a basis for the topology on C(X), the map f → D(f ) is continuous.  It can actually be shown that the map f → D(f ) is also open. Lemma 5.29 Let f ∈ E(X) and let (fα )α be a net in E(X) such that D(fα ) → D(f ) in C(X). Then the following two conditions are equivalent. (i) fα → f in E(X). (ii) For each subnet (fαβ )β of (fα )α and each choice of elements xβ ∈ D(fαβ ) such that xβ → x, for some x ∈ D(f ), we have fαβ (xβ ) → f (x).

5.3 Subgroup representation topology

225

Proof (i) ⇒ (ii) Let (fαβ )β be a subnet of (fα )α and let xβ ∈ D(fαβ ) and x ∈ D(f ) be such that xβ → x. Given  > 0, there exists a compact neighborhood V of x in X with the following properties. (1) |f (y) − f (x)| <  for all y ∈ V ∩ D(f ). (2) The set D = {(y, λ) ∈ X × C : y ∈ V , |λ − f (x)| ≥ } is semicompact. (3) f ∩ D = ∅, that is, {(y, f (y)) : y ∈ D(f )} ∩ D = ∅. By hypothesis and the definition of the semicompact–open topology on E(X), fαβ ∩ D = ∅ eventually. On the other hand, xβ ∈ V eventually. These two facts imply that |fαβ (xβ ) − f (x)| ≤ |fαβ (xβ ) − f (xβ )| + |f (xβ ) − f (x)| < 2 eventually. This shows that fαβ (xβ ) → f (x). (ii) ⇒ (i) Let D be a semicompact subset of X × C and V an open subset of X × C such that f ∈ U (D, V ), that is, f ∩ D = ∅ and f ∩ V = ∅. To prove that fα → f in E(X), it suffices to show that every subnet (fαβ )β of (fα )α contains in turn a subnet (fαβγ )γ such that fαβγ ∩ D = ∅ and fαβγ ∩ V = ∅ eventually. Thus, let D be any semicompact subset of X × C such that f ∩ D = ∅. We claim that fα ∩ D = ∅ eventually. Toward a contradiction, assume that this is false. Then there exists a subnet (fαβ )β of (fα )α such that fαβ ∩ D = ∅ for all β. For each β, choose xβ ∈ D(fαβ ) such that (xβ , fαβ (xβ ) ∈ D. Since the projection of D to X has compact closure, after passing to a subnet if necessary, we can assume that xαβ → x for some x ∈ X. Since, by hypothesis, D(fα ) → D(f ) in C(X), we have x ∈ D(f ). By (ii), it follows that fαβ (xβ ) → f (x). Now, since (xβ , fαβ (xβ )) ∈ D for all β and D is closed in X × C, we obtain that (x, f (x)) ∈ D. This contradicts f ∩ D = ∅ and establishes the above claim. Now let V be an open subset of X × C intersecting f in some point (x, f (x)). Since x ∈ D(f ) and D(fα ) → D(f ), there is a subnet (fαβ )β of (fα )α with the property that for each β there exists xβ ∈ D(fαβ ) such that xβ → x. Then, by (ii), (xβ , fαβ (xβ )) → (x, f (x)) in X × C. Hence (xβ , fαβ (xβ )) ∈ V eventually. Since this argument applies to any subnet of (fα )α , we conclude that fα ∩ V = ∅ eventually. Since we have already seen that fα ∩ D = ∅ eventually, the proof is complete.  Applying Lemma 5.29 to the case where all fα have the same domain Y ⊆ X, we get immediately Corollary 5.30 Let Y be a closed subset of X. Then on the subset {f ∈ E(X) : D(f ) = Y } of E(X), the relativized semicompact–open topology coincides with the topology of uniform convergence on compact subsets of Y .

226

Topologies on dual spaces

Corollary 5.31 Let W = {(Y, f ) ∈ C(X) × E(X) : Y ⊆ D(f )} and endow W with the relative topology of the product topology on C(X) × E(X). Then the map (Y, f ) → f |Y from W to E(X) is continuous. Proof Let (Yα , fα )α be a net in W converging to (Y, f ) in W. Then Yα → Y in C(X) and fα → f in E(X). Thus D(fα |Yα ) → D(f |Y ), and it therefore suffices to show that the net (fα |Yα )α and f |Y satisfy condition (ii) of Lemma 5.29. To that end, let (fαβ |Yαβ )β be a subnet of (fα |Yα )α and let yβ ∈ Yαβ be such that yβ → y for some y ∈ Y . Since fα → f in E(X), we get that fαβ (yβ ) → f (y) by the implication (i) ⇒ (ii) of Lemma 5.29.  We now return to our locally compact group G with a fixed smooth choice of Haar measures K → μK , K ∈ K(G). Let Es (G) denote the topological subspace of E(G) consisting of all f ∈ E(G) for which D(f ) is a closed subgroup of G. Moreover, as defined before Lemma 5.21, let Y = {(K, x) : K ∈ K(G), x ∈ K}. Proposition 5.32 Let C be a compact subset of Y, and let W = {g ∈ Cc (Y) : supp g ⊆ C}, equipped with the topology of uniform convergence on Y. Then the function  (f, g) → f (x)g(D(f ), x)dμD(f ) (x) (5.12) D(f )

is continuous on Es (G) × W . Proof Let (fα , gα )α be a net in Es (G) × W such that fα → f in Es (G) and gα → g in W . Since C is compact, we find an element F of Cc (Y) such that F (D(f ), x) = f (x) for all x with (D(f ), x) ∈ C. Let K = D(f ) and Kα = D(fα ). Then Kα → K by Lemma 5.28, and hence Lemma 5.21 yields that   F (Kα , x)g(Kα , x)dμKα (x) → f (x)g(K, x)dμK (x). (5.13) Kα

K

We claim that, given  > 0, eventually |fα (x) − F (K, x)| <  whenever (Kα , x) ∈ C.

(5.14)

Otherwise, after passing to a subnet, we could find an  > 0 and, for every α, some xα such that (Kα , xα ) ∈ C and |fα (xα ) − F (Kα , xα )| ≥ . Since C is compact, we can then in addition assume that xα → x, for some x ∈ K. But then fα (xα ) → f (x) by Lemma 5.29. This contradicts the fact that F (Kα , xα ) → F (K, x) = f (x).

5.3 Subgroup representation topology

227

Again since C is compact, by Lemma 5.19 there is a constant r = r(C) > 0 such that μKα ({x ∈ Kα : (Kα , x) ∈ C}) ≤ r for all α. Put

 cα =

fα (x)gα (Kα , x)dμKα (x)

and

Kα

 dα =

F (Kα , x)g(Kα , x)dμKα (x). Kα

Then, for each  > 0, by (5.14) we have eventually  |fα (x) − F (Kα , x)| · |g(Kα , x)|dμKα (x) |cα − dα | ≤ Kα  + |F (Kα , x) · |gα (Kα , x) − g(Kα , x)|dμKα (x) Kα

≤  gα ∞ r + F ∞ gα − g ∞ r. Since gα → g uniformly on Y and  |cα − f (x)g(D(f ), x)dμD(f ) (x)| D(f )  f (x)g(D(f ), x)dμD(f ) (x)|, ≤ |cα − dα | + |dα − D(f )

using (5.13), we obtain that   fα (x)gα (D(fα ), x)dμD(fα ) (x) → D(fα )

f (x)g(D(f ), x)dμD(f ) (x),

D(f )

as required.



Lemma 5.33 Let f ∈ Cc (Y) and F ∈ Cc (K(G) × G) such that F |Y = f . For each x ∈ G, define Fx : K(G) × G → C by Fx (K, y) = F (K, x −1 y), and let fx = Fx |Y . Then fx ∈ Cc (Y) for each x ∈ G, and the map x → fx from G to Cc (Y) is continuous with respect to the As (G)-norm on Cc (Y). Proof It is obvious that fx ∈ Cc (Y). Let x0 ∈ G and  > 0 be given. Choose a compact subset C of G such that the interior of K(G) × C contains the compact support of F . Then there exists a neighborhood V of x0 with the following properties. (1) K(G) × x0 C contains the compact support of Fx for every x ∈ V . (2) Fx − Fx0 ∞ <  for all x ∈ V .

228

Topologies on dual spaces

Since C is compact, by Lemma 5.19 there exists a constant r > 0 such that μK (C ∩ K) ≤ r for all K ∈ K(G). It follows that, for all x ∈ V ,  |F (K, x −1 y) − F (K, x0−1 y)|dμK (y) ≤ r.

fx − fx0 As (G) sup K∈K(G) K



This proves the lemma.

Definition 5.34 We say that an element ϕ of Es (G) is of positive type and associated with (K, π) ∈ S(G) if D(ϕ) = K and if there exists ξ ∈ H(π) such that ϕ(x) = π(x)ξ, ξ  for all x ∈ K. If Q is a subset of S(G), then ϕ is associated with Q if it is associated with some (K, π) ∈ Q. Theorem 5.35 Let (K, π) ∈ S(G). Given functions ϕ1 , . . . , ϕn of positive type on K associated with (K, π) and open neighborhoods Vj of ϕj , 1 ≤ j ≤ n, in the semicompact–open topology of Es (G), we define U (ϕ1 , . . . , ϕn ; V1 , . . . , Vn ) to be the set of all (K  , π  ) ∈ S(G) for which there exist ϕ1 , . . . , ϕn such that, for each 1 ≤ j ≤ n, (i) ϕj ∈ Vj , (ii) D(ϕj ) = K  , (iii) ϕj is a finite sum of functions of positive type associated with π  . Then the family of all such sets U (ϕ1 , . . . , ϕn ; V1 , . . . , Vn ) forms a neighborhood basis of (K, π) in the inner hull–kernel topology of S(G). Proof Let U = U (ϕ1 , . . . , ϕn ; V1 , . . . , Vn ). Without loss of generality we can assume that ϕj (e) = 1 for 1 ≤ j ≤ n. To show that U is an inner hull–kernel neighborhood of (K, π), it suffices to show that if Q is any subset of S(G) such that (K, π) ∈ Q, then Q ∩ U = ∅. Select ξ, . . . , ξn ∈ H(π ) so that ϕj (x) = π(x)ξj , ξj  for all x ∈ K. Since ϕj (e) = 1, ξj = 1. By Theorem 5.9, applied to Cs∗ (G), there is a net (Kα , πα )α of elements of Q and for each α and j = 1, . . . , n a sum φj,α of positive functionals on Cs∗ (G) associated with σKα ,πα such that

φj,α ≤ 1 for all α and j,

(5.15)

φj,α → ϕj pointwise on Cs∗ (G)

(5.16)

and

(here we identify functions of positive type on a closed subgroup of G with the corresponding positive functionals on Cs∗ (G)).

5.3 Subgroup representation topology

229

Now fix an  > 0 with 0 <  < 1 so that, for each j = 1, . . . , n, {ψ ∈ C(K) : |ψ(x) − ϕj (x)| <  for all x ∈ K} ⊆ Vj . Next choose f ∈ Cc (Y) such that π (f )ξj − ξj < /3 for j = 1, . . . , n. Then, since ξj = 1, for all x ∈ K, |π(x)π (f )ξj , π (f )ξj  − ϕj (x)| ≤ π(f )ξj · π (f )ξj − ξj + π (f )ξj − ξj     + < . ≤ 1+ 3 3 3 For x ∈ K, let fx be defined as in Definition 5.34. It is then easily verified that π (x)π(f )ξj , π (f )ξj  = ϕj (f ∗ ∗ fx )

(5.17)

for all x ∈ K. It follows from (5.15) and (5.16) that φj,α → ϕj uniformly on any norm-compact subset of As (G). Since, by Lemma 5.33, the map x → fx from G into Cc (Y) is continuous with respect to the As (G)-norm on Cc (Y), we conclude that, for each j , φj,α (f ∗ ∗ fx ) → ϕ(f ∗ ∗ fx )

(5.18)

uniformly in x on compact subsets of G. Now define elements ψj,α and ψj of Es (G) by D(ψj,α ) = Kα D(ψj ) = K

and and

ψj,α (x) = φj,α (f ∗ ∗ fx ), x ∈ Kα , ψj (x) = ϕj (f ∗ ∗ fx ), x ∈ K.

From the corresponding property of φj,α it is straightforward to verify that ψj,α is a sum of functions of positive type associated with πα . Moreover, by (5.17) and the choice of , ψj ∈ Vj for j = 1, . . . , n.

(5.19)

Now it follows from (5.16) that Kα → K in K(G). We claim that ψj,α → ψj in Es (G) for each j . To see this, let xα ∈ Kα such that xα → x. Then, since |ψj,α (xα ) − ψj (x)| ≤ |ψj,α (xα ) − ψj (xα | + |ψj (xα ) − ψj (x)|, (5.18) implies that ψj,α (xα ) → ψj (x) for each j . Thus ψj,α → ψj in Es (G) by Lemma 5.29. From this, (5.19), and the facts that (Kα , πα ) ∈ Q and each ψj,α is a sum of functions of positive type associated with πα , we obtain that Q intersects U . It remains to prove that these sets U form a neighborhood basis of (K, π) in S(G). For that it suffices to take a net (Kα , πα )α in S(G) which is eventually inside each such U and show that some subnet of it converges to (K, π ).

230

Topologies on dual spaces

Fix functions ϕ1 , . . . , ϕn of positive type associated with π . By the assumption on the net (Kα , πα )α , after passing to a subnet, we can for each α and j = 1, . . . , n find functions ϕj,α with the following properties. (1) ϕj,α is a sum of functions of positive type on Kα associated with πα . (2) ϕj,α → ϕj in Es (G), for every j = 1, . . . , n. Then the functions ϕj,α are bounded in norm uniformly in α and j . For each j = 1, . . . , n and g ∈ Cc (Y), Proposition 5.32 then yields that   ϕj,α (x)g(Kα , x)dμKα (x) → ϕj (x)g(K, x)dμK (x). Kα

K

Since the ϕj,α are uniformly bounded and Cc (Y) is dense in Cs∗ (G), Theorem  5.9 implies that (Kα , πα ) → (K, π) in S(G).

5.4 Continuity of inducing and restricting representations As Mackey’s theory shows, the operations of restricting representations to subgroups and of inducing representations up to bigger groups are extremely  of a locally important tools to determine the elements of the dual space G  compact group G. It is of course desirable to also obtain the topology of G, in terms of the corresponding entities of subgroups of G. To approach this purpose, it is indispensable to establish continuity of the processes of both inducing and restricting representations. We first turn to the restriction operation. Let (K, π) ∈ S(G) and H a closed subgroup of K. We are going to show that the map (H, K, π ) → (H, π |H ) is continuous for the relevant topologies when H , K, and π vary simultaneously. To commence with, we rephrase Theorem 5.35 as follows. Theorem 5.36 Let (Kα , πα )α be a net in S(G) and let (K, π) ∈ S(G). Then the following two conditions are equivalent. (i) (Kα , πα ) → (K, π) in S(G). (ii) For each finite sequence ϕ1 , . . . , ϕn of functions of positive type associated with (K, π) and each subnet (Kαβ , παβ )β of (Kα , πα )α , there exists a further subnet (Kαβγ , παβγ )γ and for each γ and 1 ≤ j ≤ n, a finite sum ϕj,γ of functions of positive type associated with (Kαβγ , παβγ ) such that ϕj,γ → ϕj in Es (G). Recall that Es (G) is the set of all f ∈ E(G) for which D(f ) ∈ K(G). Combining Theorem 5.36 with Corollary 5.31, we already obtain continuity of restriction.

5.4 Continuity of inducing and restricting

231

Theorem 5.37 Let G be a locally compact group and let W1 = {(H, K, π ) : (K, π) ∈ S(G), H ∈ K(G), H ⊆ K}. Endow W1 with the relative topology of the product topology on K(G) × S(G). Then the map (H, K, π ) → (H, π |H ) from W1 to S(G) is continuous. Proof Let (Hα , Kα , πα )α be a net in W1 converging to (H, K, π ) ∈ W1 . Notice first that since Kα → K and Hα → H in K(G) and Hα ⊆ Kα for all α, we have that H ⊆ K. To show that (Hα , πα |Hα ) → (H, π |H ), we verify condition (ii) of Theorem 5.36. Thus, let ϕ1 , . . . , ϕn be a sequence of functions of positive type associated with (H, π |H ), and let (Hαβ , παβ |Hαβ )β be any subnet of (Hα , πα |Hα )α . Let ϕj (h) = π |H (h)ξj , ξj , for h ∈ H , where ξj ∈ H(π|H ) = H(π ), 1 ≤ j ≤ n. For each 1 ≤ j ≤ n, let ψj be the corresponding function of positive type associated with π, that is, ψj (x) = π (x)ξj , ξj , for x ∈ K. Consider the subnet (Kαβ , παβ )β of the net (Kα , πα )α . Since (Kαβ , παβ ) → (K, π), by the implication (i) ⇒ (ii) of Theorem 5.36 there exist a further subnet (Kαβγ , παβγ )γ and, for each γ and 1 ≤ j ≤ n, a finite sum ψj,γ of functions of positive type associated with (Kαβγ , παβγ ) such that ψj,γ → ψj in Es (G), for every 1 ≤ j ≤ n. By Corollary 5.31, the mapping (Y, f ) → f |Y from W to E(G) is continuous. It follows that ϕj,γ = ψj,γ |Hαβγ → ψj |H = ϕj in Es (G), for ≤ j ≤ n. The implication (ii) ⇒ (i) of Theorem 5.36 now shows that the net (Hα , πα |Hα )α converges to (H, π |H ) in S(G).  Corollary 5.38 Let Q be a subset of S(G) and let (K, π) be an element of S(G) that is weakly contained in Q. Let L be a closed subgroup of K such that L ⊆ H for all (H, τ ) ∈ Q. Then π|L is weakly contained in the set of all restrictions τ |L , where (H, τ ) ranges over Q. Proof Let ρ be an irreducible representation of K that is weakly contained in ∗  π, and consider the associated irreducible representation σ = σK,ρ of C s (G). By Proposition 5.11, σ belongs to the closure of Q in the inner hull–kernel topology. It follows from Theorem 5.37 that σ |L belongs to the closure of, and hence is weakly contained in, the set Q|L = {τ |L : (H, τ ) ∈ Q}. On the other hand, π|L is weakly equivalent to the set of all ρ|L , where  ρ ∈ supp π. Thus π|L is weakly contained in Q|L . We now turn to proving continuity of the inducing operation. As the reader will expect, this is somewhat more complicated than showing continuity of restricting representations.

232

Topologies on dual spaces

Theorem 5.39 Let G be a locally compact group and let W2 = {(H, K, π ) : (K, π) ∈ S(G), H ∈ K(G), H ⊇ K}. Equip W2 with the relative topology of the product topology on K(G) × S(G). Then the map (H, K, π ) → (H, indH K π) from W2 into S(G) is continuous. Proof Let (Hα , Kα , πα )α be a net in W2 such that Hα → H in K(G) and (Kα , πα ) → (K, π) in S(G). Since Kα ⊆ Hα for all α and Kα → K in K(G), we have that K ⊆ H . Now recall that a net (σα )α of representations of a C ∗ -algebra A converges to a representation σ of A if (and only if) every subnet α of (σα )α weakly contains σ (Lemma 5.7). Thus, to prove that (Hα , indH Kα πα ) → H (H, indK π ) in S(G), after passing to a subnet and relabeling, it suffices to show α ∗ that the set R of all subgroup representations (Hα , indH Kα πα ) of Cs (G) weakly contains (H, indH K π). Since π is a direct sum of cyclic representations, we can assume that π is cyclic. Let ϕ be the function of positive type associated with a cyclic vector for π. By Theorem 5.36, after replacing (Kα , πα )α by a subnet if necessary, we find for each α a function ϕα which is a sum of functions of positive type associated with πα such that ϕα → ϕ in Es (G). Let μ be the measure on K defined by dμ(x) = K (x)1/2 H (x)−1/2 ϕ(x)dμK (x). By Lemma 2.31, the measure ν defined on Cc (H ) by ν(f ) = μ(f |H ) is of positive type and, by Theorem 2.32, λν = indH K π since ϕ is associated with a cyclic vector of π . Similarly, for each α, let   Kα (x) 1/2 dμα (x) = ϕα (x)dμKα (x) Hα (x) and let να be the corresponding measure of positive type on Hα . Then να has the property that λνα is a subrepresentation of the direct sum of finitely many α copies of indH Kα πα . Let f ∈ Cc (H ) and! let ξf ∈ F(H,"K, λμ ) ⊆ H(indH K π ) as in Theorem 2.32, that is, ξf (x) = δ −1 · (Lx −1 f )|K μ , for all x ∈ H , where we have set δ(k) = [K (k)/H (k)]1/2 for k ∈ K. Since the set of all ξf , f ∈ Cc (H ), is ∗ dense in H(indH K π), it suffices to show that the positive functional on Cs (G) ∗ associated with ξf can be approximated in the w -topology by positive funcα tionals associated with the subgroup representations (Hα , indH Kα πα ), that is, finite sums of positive functionals associated with elements of R. Now fix f ∈ Cc (H ) and let ξ = ξf . Extend f to a continuous function, say F , with compact support on G. For each α, set fα = F |Hα and let ξα = α   ξfα ∈ H(indH Kα πα ). Define F ∈ Cc (Y) by F (L, x) = F (x) for all L ∈ K(G)

5.4 Continuity of inducing and restricting

233

and x ∈ L. We claim that, using the notation of Section 2.3,  ∗ ∗ g ∗ F )(Hα , x)dνα (x), λνα (g)[fα ]να , [fα ]να  = (F Kα

for every g ∈ Cc (Y). Indeed, for each t ∈ Hα we have  ∗ να (fα ∗ Lt fα ) = (fα∗ ∗ Lt fα )(k) dμKα (k) Kα   fα (y −1 )Hα (y −1 )fα (t −1 y −1 k) dμHα (y)dμα (k), = Kα

Hα

(Hα , y), for y ∈ Hα , the preceding equation implies and since fα (y) = F  λνα (g)[fα ]να , [fα ]να  = g(Hα , t)να (fα∗ ∗ Lt fα ) dμHα (t) Hα  ∗ ∗ g ∗ F )(Hα , x)dμα (x). = (F Kα

Substituting for μα , we get    Kα (x) 1/2 ∗   λνα (g)[fα ]να , [fα ]να  = (F ∗ g ∗ F )(Hα , x) ϕα (x)dμKα (x). Hα (x) Kα ∗ ∗ g ∗ F )(H, x) for all Choose a function E in Cc (G) such that E(x) = (F x ∈ H . Given  > 0, we then have eventually )(Hα , x) − E(x)| ≤ , ∗ ∗ g ∗ F |(F

(5.20)

for all x ∈ Hα . Since the function (L, x) → L (x) is continuous on Y, Proposition 5.32 yields that    K (x) 1/2 ∗   ϕ(x)dμK (x) λν (g)[f ]ν , [f ]ν  = (F ∗ g ∗ F )(H, x) H (x) K    Kα (x) 1/2 = lim E(x) ϕα (x)dμKα (x). α Hα (x) Kα The above formulae for λν (g)[f ]ν , [f ]ν  and λνα (g)[fα ]να , [fα ]να  together with (5.20) yield λν (g)[f ]ν , [f ]ν  = lim λνα (g)[fα ]να , [fα ]να . α

This completes the proof of the theorem.



Even though the proof of the next corollary is similar to that of Corollary 5.38, we include it for completeness.

234

Topologies on dual spaces

Corollary 5.40 Let Q be a subset of S(G) and let (K, π) be an element of S(G) that is weakly contained in Q. Let L be a closed subgroup of G such that L ⊇ K and L ⊇ H for all (H, τ ) ∈ Q. Then indLK π is weakly contained in the set of all representations indLH τ , where (H, τ ) ranges over Q. Proof Let σ be an element of the support of π, considered as an element of ∗  C s (G). Then σ belongs to the closure of Q in the inner hull–kernel topology. Theorem 5.39 implies that indLK σ lies in the closure of, and hence is weakly contained in, the family of all induced representations indLH τ , where (H, τ ) ∈ Q. However, indLH π is weakly equivalent to the set of all representations indLH σ , σ ∈ supp π. Therefore indLH π is weakly contained in the set {indLH : (H, τ ) ∈ Q}.  Now, we further specialize Corollary 5.38 and Corollary 5.40, respectively, to the case where the subgroups H and L are fixed. Corollary 5.41 Let H be a closed subgroup of G. (i) If T is a family of unitary representations of G and π is a unitary representation of G which is weakly contained in T , then π |H is weakly contained in the set of representations {τ |H : τ ∈ T }. (ii) If S is a family of unitary representations of H and π is a unitary representation of H which is weakly contained in S, then indG H π is weakly : σ ∈ S}. contained in the set of representations {indG H We conclude this section with a useful application of Corollary 5.41. Proposition 5.42 Let G be a locally compact group, N a closed normal subgroup of G, and H a closed subgroup of G containing N . Let τ be a representation of H and  a family of representations of N such that τ |N ≺ {x · σ : σ ∈ , x ∈ G}. If 1H /N ≺ λH /N , then G indG H τ ≺ {indN σ : σ ∈ }.

Proof Notice first that by Corollary 5.41(ii), H indH N (τ |N ) ≺ {indN (x · σ ) : σ ∈ , x ∈ G}.

From 1H /N ≺ λH /N we obtain that 1H ≺ indH N 1N . Using Theorem 2.58 and the fact that weak containment is preserved under forming tensor products (Proposition 5.14), it follows that H τ = τ ⊗ 1H ≺ τ ⊗ indH N 1N = indN (τ |N ).

5.5 Examples: nilpotent and solvable groups

235

Now, applying Corollary 5.41 again as well as induction in stages and the fact G that indG N (x · σ ) = indN σ (Remark 2.35), we get G H G H indG H τ ≺ indH (indN (τ |N )) ≺ {indH (indN (x · σ )) : σ ∈ , x ∈ G}

= {indG N σ : σ ∈ }, 

as was to be shown.

In the next sections we shall apply these continuity results, in particular the various corollaries, to determine the topology of a number of dual spaces.

5.5 Examples: nilpotent and solvable groups In this section we apply Mackey’s theory and the results of the preceding sections to determine the dual space topology for a number of two-step solvable Lie groups. Except for the first three examples, all others are three-step nilpotent simply connected Lie groups, starting with the 2n + 1-dimensional Heisenberg group, n ≥ 1. Example 5.43 Our first example is Gaff , the affine group. That is, Gaff = {(x, a) : x ∈ R, a > 0} with multiplication (x, a)(y, b) = (x + ay, ab). Let N be the normal subgroup  consisting of all elements (x, 1), x ∈ R. Recall that G aff consists of all chariλ acters χλ , λ ∈ R, defined by χλ (x, a) = a , and two infinite-dimensional representations σ + and σ − , which are obtained by inducing the characters χ + : (x, 1) → eix and χ − : (x, 1) → e−ix from N to Gaff , respectively (Exam contains 1N , but not χ − , ple 4.31). The closure of the orbit of χ + in N − + and similarly for χ . By continuity of inducing, σ + = indG N χ weakly con− − tains all χλ , λ ∈ R, but not σ , and similarly for σ . On the other hand,   G aff /N = {χλ : λ ∈ R} is closed in Gaff and the relative topology of Gaff on  Gaff /N is just the topology which makes the map λ → χλ a homeomorphism.   Consequently, a subset T of G aff is closed in Gaff if and only if it satisfies the following two conditions. (1) If σ + ∈ T or σ − ∈ T , then {χλ : λ ∈ R} ⊆ T . (2) The set {λ ∈ R : χλ ∈ T } is closed in R.  of any unital C ∗ -algebra is We remind the reader that the dual space A  compact and hence G is compact whenever G is discrete. The converse does not hold as is shown by the following example, which is attributed to Fell and therefore usually referred to as the Fell group.

236

Topologies on dual spaces

Example 5.44 Let G be the semidirect product G = R  Z, where n ∈ Z acts on R by n · x = en x, x ∈ R. The dual action of Z on R = {χy : y ∈ R} is given by n · χy = χe−n y . If y = 0, then the stability group of χy in Z is trivial and the orbit of χy is discrete and hence homeomorphic to Z. Thus Mackey’s theory applies and gives = G Z ∪ {πy = indG R χy : y ∈ R, y = 0}. Since each orbit in R =  R intersects one of the intervals [1, e] or [−e, −1]  is the union of the two compact sets  and inducing is continuous, G Z and {πy : y ∈ [1, e] ∪ [−e, −1]}.  The set It is an easy matter to explicitly determine the topology on G. C+ = {πy : y > 0} = {πy : y ∈ [1, e]}  and homeomorphic to the quotient space obtained from [1, e] by is open in G Z since C+ identifying the endpoints. Moreover, for any y > 0, {πy } = {πy } ∪  is Hausdorff and 0 ∈ Z(y) and hence G  Z ∼ indG R χ0 ≺ indR χy = πy .

Similarly, for C− = {πy : y < 0} = {πy : y ∈ [−e, −1]}. Before turning to simply connected nilpotent Lie groups, we treat a threedimensional solvable Lie group with a particularly interesting orbit structure. Example 5.45 Let  G be thegroup discussed in Example 4.14. Thus G = 1 0 . The R-orbit structure in R2 =  R2 was illusR2 A R with A = 0 −1  was determined in trated in Figure 4.4 and, using Mackey’s theory, the set G Example 4.35. Recall that  = {(1, 0), (0, 1), (−1, 0), (0, −1)} ∪ {γ = (γ1 , ±γ1 ) : γ1 ∈ R∗ }  is a cross-section for the nontrivial R-orbits in R2 . For u ∈ R, let χu ∈ G 2 denote the pullback of the character of R = G/R associated with u, and for γ ∈ R2 \ {(0, 0)} let χγ be the associated character of R2 and πγ = indG R 2 χγ . With this notation,  = {χu : u ∈ R} ∪ {πγ : γ ∈ }. G  Clearly, it suffices to consider It is not difficult to work out the topology of G.  a subset  of  and to determine the closure of  = {πγ : γ ∈ } in G. Now, continuity of the processes of inducing and restricting representations  belongs to the closure of  precisely when π |R2 is weakly implies that π ∈ G

5.5 Examples: nilpotent and solvable groups

237

contained in the set ∪{R(χγ ) : γ ∈ }. Thus, χu ∈  for one (and then all) u if and only if either one of the points (1, 0), (0, 1), (−1, 0), and (0, −1) is in  or  contains a sequence (γn )n converging to (0, 0) in R2 . Next, let λ = (λ1 , λ1 ) ∈ ; then πλ ∈  if and only if λ1 ∈ {γ1 : (γ1 , γ1 ) ∈ }, and similarly for λ = (λ1 , −λ1 ). Finally, if (1, 0) ∈ , then π(1,0) ∈  exactly when there exist γ = (γ1 , ±γ1 ) ∈  with γ1 > 0 and arbitrarily small. The analogous statements hold for (0, 1), (−1, 0), and (0, −1). Example 5.46 For n ≥ 1, let Hn be the (2n + 1)-dimensional Heisenberg group. Let the abelian normal subgroup N be defined as in Example 4.38, and n consists of let Z denote the center of Hn . Then H (1) the one-dimensional representations (characters) χu,v , u, v ∈ Rn , given by χu,v (x, y, t) = exp 2π i(x, u + y, v), n (2) the infinite-dimensional induced representations σλ = indH N ωλ , λ ∈ R, λ = 0, where ωλ is the character of N defined by ωλ (0, y, t) = exp(2π iλt). The map (u, v) → χu,v is a homeomorphism between R2n and the closed subset of characters in (1). Furthermore, the map λ → σλ is a homeomorphism n consisting of the infinite dimensional between R \ {0} and the open subset of H  representations in (2). For λ = 0, the Hn -orbit of ωλ equals R × {λ} ⊆ R2 = N, N and hence Hn (ωλ ) is weakly equivalent to the induced representation indZ λ. n It follows that if λ → 0, then σλ → indH Z 1Z , which is weakly equivalent to    H n /Z. Thus a subset T of Hn is closed in Hn if and only if (1) {u, v) ∈ R2n : χu,v ∈ T } and {λ ∈ R \ {0} : σλ ∈ T } is closed in R2n and R \ {0}, respectively, (2) if 0 is a limit point of {λ ∈ R \ {0} : σλ ∈ T }, then all characters χu,v belong to T . The next example is, up to topological isomorphism, the only fourdimensional simply connected nilpotent Lie group which does not split into the direct product of two smaller-dimensional groups. Example 5.47 Let G be the semidirect product G = R3  R, where t ∈ R acts on R3 by   t2 t · (x1 , x2 , x3 ) = x1 + tx2 + x3 , x2 + tx3 , x3 . 2 The group G is three-step nilpotent, and the first two members of the ascending central series of G are given by Z1 (G) = {(x1 , 0, 0) : x1 ∈ R}

and

Z2 (G) = {(x1 , x2 , 0) : x1 , x2 ∈ R}.

238

Topologies on dual spaces

 as a set. The dual action of Employing Mackey’s theory, we first determine G 3 is given by R on R t · χy (x) = χy ((−t) · x)     t2 = exp 2π i y1 x1 − tx2 + x3 + y2 (x2 − tx3 ) + y3 x3 2 = χ(y ,y −ty ,y −ty + t 2 y ) (x). 1

2

1

3

2

2

1

3 with R3 . If y = (y1 , y2 , y3 ) ∈ R3 with To simplify notation, we identify R y1 = 0 or y2 = 0, then the R-orbit of y is closed and homeomorphic to R. So Mackey’s theory applies and tells us that any irreducible representation of G is either a character of G/Z2 (G) or induced from a character of R3 . It is easily verified that a cross-section for the R-orbits in R∗ × R2 is formed by the set of all y = (y1 , 0, y3 ), y1 , y3 ∈ R3 , y1 = 0. For r, s ∈ R with r = 0, s let σr,s = indG R3 (r, 0, r ). Then the assignment φ1 : (r, s) → σr,s is a bijection   \ G/Z between the sets R∗ × R and G 1 (G). It is clear that the R-orbits in ∗ ∗ {0} × R × R are parametrized by R . It follows that the map φ2 : t → πt = ∗   indG R3 (0, t, 0) is a bijection between R and G/Z1 (G) \ G/Z2 (G). Now, using continuity of restriction and induction of representations and of the quotient map R3 → R3 /R, it is not difficult to conclude that the maps φ1 and φ2 are  Since homeomorphisms of their domains onto the corresponding subset of G.   G/Z2 (G), the set of characters of G, is closed in G and homeomorphic to R2 , it remains to consider a closed subset T of the range of φ1 and of φ2 , respectively,  and to determine the closure of T in G. If W ⊆ R \ {0}, then the closure of {πw : w ∈ W } either contains none or  all of G/Z 2 (G) according to whether or not 0 is an accumulation point of W . Indeed, a character χu,v of G belongs to {πw : w ∈ W } if and only if (0, 0, u) belongs to the closure of the set ∪w∈W R · (0, w, 0) = ∪w∈W {(0, w)} × R. Next, 2 belongs to the closure of the set let Q ⊆ R∗ × R. Then an element πw ∈ G {σr,s : (r, s) ∈ Q} precisely when (0, w, 0) lies in the closure of ∪ {R · (r, 0, s) : (r, s) ∈ Q}. This is equivalent to the existence of a sequence of triples (rn , sn , tn ) such that (rn , sn ) ∈ Q and rn → 0, −tn rn → w

and

sn t2 + n rn → 0 rn 2

(n → ∞). These conditions are easily seen to be equivalent to   w2 0, − = lim (rn , sn ) ∈ Q. n→∞ 2 Finally, we have to determine the possible accumulation points of Q in + −  G/Z 2 (G). Let Q = {(r, s) ∈ Q : r > 0} and Q = {(r, s) ∈ Q : r < 0}. A character χu,v lies in the closure of φ1 (Q+ ) if and only if (0, 0, u) ∈ R3 is

5.5 Examples: nilpotent and solvable groups

239

t2

a limit of some sequence (rn , −tn rn , srnn + 2n rn ), where (rn , sn ) ∈ Q+ , tn ∈ R. Convergence of the third component implies that sn + 12 (tn rn)2 → 0 and hence sn → 0 as tn rn → 0. Moreover, since rn > 0, lim inf n→∞ srnn ≤ u. It is obvious that these two conditions are also sufficient. Thus χu,v ∈ φ1 (Q+ ) if and only if (0, 0) ∈ Q+ and s  ≤ u. lim inf(r,s)∈Q+ ,(r,s)→(0,0) r Similarly, it is shown that χu,v ∈ φ1 (Q− ) if and only if (0, 0) ∈ Q− and s  lim sup(r,s)∈Q− ,(r,s)→(0,0) ≥ u. r  is closed in G  if Summarizing the above facts, we obtain that a subset T of G and only if it satisfies the following five conditions. (1) The sets {(u, v) ∈ R2 : χu,v ∈ T }, {w ∈ R \ {0} : πw ∈ T }, and {(r, s) ∈ R \ {0} × R : σr,s ∈ T } are closed in R2 , R \ {0}, and (R \ {0}) × R, respectively.  (2) If 0 is a limit point of {w ∈ R\ {0} : πw ∈ T }, then G/Z 2 (G) ⊆ T . w2 (3) If w ∈ R \ {0} and − 2 , 0 is a limit point of {(r, s) : σr,s ∈ T }, then πw ∈ T . (4) If (0, 0) is a limit point of the set {(r, s) : r > 0, σr,s ∈ T } and s  ≤ u, lim inf(r,s)→(0,0),r>0,σr,s ∈T r then χu,v ∈ T for all v ∈ R. (5) If (0, 0) is a limit point of the set {(r, s) : r < 0, σr,s ∈ T } and s  ≥ u, lim sup(r,s)→(0,0),r<0,σr,s ∈T r then χu,v ∈ T for all v ∈ R. Up to topological isomorphism there exist six five-dimensional simply connected nilpotent Lie groups which do not decompose into a direct product of smaller-dimensional groups. These six groups are commonly labeled G5,j or 5,j , 1 ≤ j ≤ 6 (compare [35] and [117]). In the sequel we are going to determine the dual space topology for two of them, G5,3 and G5,4 , the latter being the most complicated one among all six. Example 5.48 For x = (x1 , . . . , x5 ) and y = (y1 , . . . , y5 ) in R5 define a product by  x2 xy = x1 + y1 + x4 y3 + x5 y2 + 5 y4 , x2 + y2 + x5 y4 , 2  x3 + y3 , x4 + y4 , x5 + y5 .

240

Topologies on dual spaces

Then G = R5 , endowed with this product, forms a group and N = {x ∈ G : x4 = x5 = 0} is an abelian normal subgroup of G with G/N being isomorphic to R2 . However, G cannot be written as a semidirect product of two abelian groups. For x = (x1 , x2 , x3 , 0, 0) ∈ N and y = (0, 0, 0, y4 , y5 ), we have y −1 xy = (x1 − (y4 x3 + y5 x2 ), x2 , x3 , 0, 0).  y ∈ G, and x ∈ N, Thus, for χr,s,t ∈ N, y · χr,s,t (x) = exp 2π i(r(x1 − y4 x3 − y5 x2 ) + sx2 + tx3 ) = χr,s−y5 r,t−y4 r (x).  with R3 . Then the G-orbit of (r, s, t) Now assume that r = 0 and identify N 2 equals {r} × R and the stabilizer of (r, s, t) equals R3 . It follows that G   \ G/Z G 1 (G) = {πr = indN αr : r ∈ R, r = 0},

where αr is the character of N defined by αr (x1 , x2 , x3 ) = exp 2π irx1 . To   \ G/Z identify G 1 (G), note that G/Z1 (G) is the direct product of the three  dimensional Heisenberg group H1 and R. Thus G/Z 1 (G) = H1 × R, and the  topology of H1 is known by Example 5.46. Using continuity of induction and   \ G/Z restriction, it is clear that the map r → πr from R \ {0} onto G 1 (G) is a homeomorphism.  Since, for each r ∈ R \ {0}, πr ∼ indG Z1 (G) αr , it follows that a subset T of G  is closed in G if and only if (1) {r ∈ R \ {0} : πr ∈ T } is closed in R \ {0};    (2) T ∩ G/Z 1 (G) is closed in G/Z1 (G) = H1 × R; (3) if 0 is an accumulation point of the set {r ∈ R \ {0} : πr ∈ T }, then T ⊇  G/Z 1 (G). In all the previous examples there was a small number of normal subgroups N of G such that each irreducible representation of G was either induced from a character of some N or the pullback of some irreducible representation of G/N. Our final example is considerably more complicated in that there occur infinitely many such N. Fortunately, however, these subgroups N vary continuously. Example 5.49 For x = (x1 , . . . , x5 ) and y = (y1 , . . . , y5 ) in R5 define a product xy by  1 1 xy = x1 + y1 + x4 y3 + x4 x5 y4 + x5 y42 , x2 + y2 + x5 y3 + x52 y4 , 2  2 x3 + y3 + x5 y4 , x4 + y4 , x5 + y5 .

5.5 Examples: nilpotent and solvable groups

241

Then G = R5 , equipped with this product, becomes a group and N = {x ∈ G : x4 = x5 = 0} is an abelian normal subgroup which we identify with R3 in what follows. The group G is three-step nilpotent and its center equals {x ∈ G : x3 = x4 = x5 = 0}. Let C = {x ∈ G : xj = 0 for 2 ≤ j ≤ 5}. Then G/C is topologically isomorphic to the group studied in Example 5.47. Thus  namely G  \ G/C,  as our first task is to determine the remaining portion of G, a set. For any x ∈ N and y ∈ G, we have y −1 xy = (x1 − y4 x3 , x2 − y5 x3 , x3 ),  is given by and hence the dual action on N y · χr,s,t (x) = χr,s,t (x1 − y4 x3 , x2 − y5 x3 , x3 ) = χr,s,t−y4 r−y5 s (x),  with R3 as well and assuming that r = 0, r, s, t ∈ R. Thus, identifying N  equals {(r, s)} × R. Therefore, the set R∗ × R, correthe orbit of (r, s, t) ∈ N sponding to the set of characters χr,s,0 , forms a cross-section for the orbits in  \ N/C.  The stability group of (r, s, 0) equals N  s  Gr,s = R3 × R − , 1 . r It is known that, since Gr,s is a simply connected nilpotent Lie group and Gr,s /N is one-dimensional, Gr,s must be a splitting extension of N. Using the data provided in Nielsen [117], it is easy to figure out that   2  # s 3 s 3 s 2 s u , − , − , − v v v, v : v ∈ R Ar,s = a = 3r 2 6r 2r r is a subgroup of Gr,s complementary to N . For (r, s, t, u) ∈ R∗ × R3 , let χr,s,t,u denote the character of Gr,s defined by χr,s,t,u (xa) = exp 2π i[rx1 + sx2 + tx3 + uv], for x = (x1 , x2 , x3 , 0, 0) ∈ N and a ∈ Ar,s as above. Then the representation πr,s,u = indG Gr,s ωr,s,0,u of G is irreducible and, by Mackey’s theory  \ G/C  is unitarily equivalent to some (Theorem 4.27), every element of G

such πr,s,u . Thus the assignment (r, s, u) → πr,s,u defines a bijection between  \ G/C.  We are going to show that this map is a home = R∗ × R2 and G omorphism. Since inducing is continuous, for continuity it suffices to show that the map (r, s, u) → (Gr,s , χr,s,0,u ) from  into S(G) is continuous for the subgroup representation topology on S(G). To that end, let (rn , sn , un ) → (r, s, u) in . Then, since rn → r, r = 0, and sn → s, it is obvious that    s  sn 3 Grn ,sn = R × R − , 1 → Gr,s = R3 × R − , 1 rn r

242

Topologies on dual spaces

in K(G). Now let      s  sn and y = x1 , x2 , x3 , v − , 1 yn = xn,1 , xn,2 , xn,3 , vn − , 1 rn r be elements of Grn ,sn and Gr,s , respectively, such that yn → y. Then xn,j → xj for 1 ≤ j ≤ 3 and vn → v, and therefore χrn ,sn ,0,un (yn ) → χr,s,0,u (y). This shows that the map (r, s, u) → (Gr,s , χr,s,0,u ) is continuous.  \ G/C  to  is continuous as well, let To prove that the inverse map from G  Then (rn , sn , un ) and (r, s, u) in  be such that πrn ,sn ,un → πr,s,u in G. χrn ,sn ∼ πrn ,sn ,un |Z(G) → πr,s,u |Z(G) ∼ χr,s and hence rn → r and sn → s. This implies (see above) that Grn ,sn → Gr,s in K(G). Then, by continuity of restricting representations (Theorem 5.37),     Grn ,sn , πrn ,sn ,un |Grn ,sn −→ Gr,s , πr,s,u |Gr,s in S(G). Notice next that since Grn ,sn is normal in G,   πrn ,sn ,un |Grn ,sn ∼ G χrn ,sn ,0,un , and similarly for πr,s,u and χr,s,0,u . Thus there exist yn = (0, 0, 0, vn , 0) ∈ G such that     Grn ,sn , yn · χrn ,sn ,0,un −→ Gr,s , χr,s,0,u in S(G). Given g = (x1 , x2 , x3 , − sr x5 , x5 ) ∈ Gr,s , there exist   s gn = xn,1 , xn,2 , xn,3 , − xn,5 , xn,5 ∈ Grn ,sn , r n ∈ N, such that gn → g. Then yn · χrn ,sn ,0,un (gn ) = exp 2π i[rn (xn,1 − vn xn,3 ) + sn xn,2 + un xn,5 ] → χr,s,0,u (g) = exp 2π i[rx1 − sx2 + ux5 ]. Taking first x1 = x2 = x3 = x5 = 0, we conclude that vn → 0 since rn → r = 0. Then, choosing xj = 0 for 1 ≤ j ≤ 4, it follows that un → u. This shows  \ G/C  onto R∗ × R2 is continuous. that the map from G  is known (Example 5.47),  of G Since the topology of the closed subset G/C  \ G/C.   it remains to determine the accumulation points in G/C of subsets of G    Let (πn )n be a sequence in G \ G/C such that πn → π in G. Let s ∈ R such that π|Z(G) ∼ χ0,s and let πn = πrn ,sn ,0,un as above. Then, since πn |Z(G) → passing to a subsequence if necessary, π |Z(G) , rn → 0 and sn → s. Moreover,  we can assume that either  srnn  → ∞ or srnn → q for some q ∈ R, and since   K(G) is compact, Grn ,sn → H for some closed subgroup H of G. If  srnn  → ∞

5.5 Examples: nilpotent and solvable groups

243

then H = R3 , identified with R3 × {(0, 0)}, whereas H = R3 × R(−q, 1) if sn → q. Since restricting representations is continuous (Theorem 5.37), rn   (5.21) Grn ,sn , π |Grn ,sn → (H, π |H ) in the subgroup representation topology. Conversely, if (5.21) holds, then πn → π since by continuity of inducing (Theorem 5.39),   πn ∼ indG → indG Grn ,sn πn |Grn ,sn H (π |H ) ! π. Note that πn |Grn ,sn ∼ {χrn ,sn ,t,un : t ∈ R}. Now (5.21) holds if and only if given any σ ∈ supp(π |H ), there exists a sequence of characters χrn ,sn ,tn ,un , n ∈ N, such that if   sn gn = xn,1 , xn,2 , xn,3 , − xn,5 , xn,5 ∈ Grn ,sn rn and g ∈ H are such that gn → g, then χrn ,sn ,tn ,un (gn ) = exp 2π i[rn xn,1 + sn xn,2 + tn xn,3 + un xn,5 ] → σ (g). (5.22) We now distinguish the three possible cases of π. 3 , that is, π is a character of G. Then (5.22) holds Assume first that π ∈ G/R if and only if rn , sn , tn → 0 and, in addition, (1) if H = R3 , then the sequence (un )n is bounded since un xn,5 → 0 for any null sequence (xn,5 )n ; (2) if H = R3 × R(−q, 1), then un → u for some u ∈ R and π(0, 0, 0, −qx5 , x5 ) = exp 2πiux5 for all x5 ∈ R.   Next consider the case where π |Z(G) ∼ χ0,s with s = 0. Then  sn  → ∞ and rn

hence H = R3 and g = (x1 , x2 , x3 , 0, 0). Since x3 is arbitrary and (xn,5 )n is an arbitrary null sequence, (5.22) holds if and only if sn → s, rn , tn → 0, and the 3 sequence (un )n is bounded. This means that πn → π since π ≺ indG R3 (π |R ) G 3 and π|R3 ∼ {χ0,s,v : v ∈ R} = G(χ0,s,0 ) and hence indG (π | ) ∼ ind χ R R3 R3 0,s,0 . 3 The remaining case is that π|R ∼ χ0,0,t with t = 0, that is, π is one of the infinite-dimensional irreducible representations of the Heisenberg group H1 = G/Z(G). Then π ∼ indG R3 χ0,0,t and πn → π implies that rn , sn → 0 and tn → t. Moreover, H must be equal to R3 . To see this, toward a contradiction assume that srnn → q, so that H = R3 × R(−q, 1). Then, with K = R(−q, 1) and Kn = R(− srnn , 1), πn |Kn ∼ χun and     G  3 π|K ∼ indG R3 χ0,0,t |K ∼ indR3 ∩K χ0,0,t |R ∩K ∼ K since K ∩ R3 is trivial. This shows that H = R3 , and (5.22) is satisfied if and only if rn , sn → 0, tn → t and, by the same argument as above, the sequence (un )n is bounded.

244

Topologies on dual spaces

 \ G/C  and π ∈ G/C,  we have π ∈ S if and only Summing up, given S ⊆ G if there exists a sequence of elements πn = πrn ,sn ,tn ,un in S such that (i) if π is a character, then (1) and (2) hold; (ii) if π|Z(G) ∼ χ0,s with s = 0, then rn , tn → 0, sn → s, and the sequence (un )n is bounded;   (iii) if π|R3 ∼ χ0,0,t , t = 0, then rn , sn → 0, tn → t,  srnn  → ∞, and the sequence (un )n is bounded.

5.6 The topology on the dual of a motion group By a (generalized) motion group we mean a locally compact group G of the form G = N  K, where N is a locally compact abelian group and K is a compact group. For such G, the Mackey analysis (Theorem 4.27) provides a  of G as a set in terms of the dual group N  description of the dual space G  of N and irreducible representations of the stabilizer groups Kχ , χ ∈ N. The  explicitly through these purpose of this section is to describe the topology on G data. For any locally compact group H , let Sirr (H ) denote the set of all (L, σ ) ∈ S(H ), where σ is an irreducible representation of L, and endow Sirr (H ) with the subspace topology of S(H ).  and τα ∈ K  Lemma 5.50 Let (χα , τα )α be a net, where χα ∈ N χα , and let πα = G  Suppose that the following conditions are satisfied. indNKχα (χα × τα ) ∈ G.  for some χ ∈ N . (i) χα → χ in N (ii) (Kχα , τα ) → (L, σ ) in Sirr (K) for some (L, σ ) ∈ Sirr (K).  converges to indG Then L ⊆ Kχ , and the net (πα )α in G ) in Rep(G). NL (χ × σ    In particular, πα → π in G for every π ∈ supp indG (χ × σ ) . NL Proof Let Kα = Kχα , Hα = N  Kα , and H = N  L. From (Kχα , τα ) → (L, σ ) in S(K) we obtain that Kα → L in K(G) (Lemma 5.26). Moreover, χα → χ and Kα → L implies that L ⊆ Kχ . In particular, χ × σ is an (irreducible) representation of H . We claim that (Hα , χα × τα ) → (H, χ × σ ) in S(G). Indeed, this follows from hypothesis (ii) and the fact that the functions of positive type associated with (H, χ × σ ) are precisely the functions φ of the form φ(x, k) = χ(x)ϕ(k), x ∈ N, k ∈ K, where ϕ is a function of positive type associated with (L, σ ) (and similarly, for (Hα , χα × τα )). Continuity of

5.6 The topology on the dual of a motion group

245

inducing (Theorem 5.39) now implies that πα → indG NL (χ × σ ) in Rep(G). The last statement of the lemma then follows since every subnet of the net  (πα )α weakly contains π .  and choose χ ∈ N  such that π|N ∼ K(χ). In the sequel, we Let π ∈ G 2 realize the induced representation indG N χ on the Hilbert space L (K). Then π G is a subrepresentation of indN χ, and we can choose a subspace F of L2 (K) such F that (indG N χ )| is equivalent to π. Then F is a left ideal of the Hilbert algebra 2 L (K). Indeed, this follows from the fact that (indG N χ)|K is the left regular representation of K and hence F is stable under left translation. Therefore F can be written as a direct sum of minimal left ideals. We fix a nonzero vector ξ in one of these minimal left ideals of F . Let ϕ be the positive linear functional on C ∗ (G) defined by ϕ(f ) = indG N χ(f )ξ, ξ . Since ξ ∈ F , ϕ is associated with π. Let J be the minimal two-sided ideal of L2 (K) containing ξ . Proposition 5.51 Let π, χ , and ϕ be as above and retain the above notation.  such that π ∈ T . Then there exists a net of Suppose that T is a subset of G triples (πα , χα , ϕα ), consisting of irreducible representations of G, characters on N , and functionals of positive type, with the following properties.  πα ∈ T for every α, and πα → π in G.  χα → χ in N and, for each α, πα |N ∼ K(χα ). The net (Kχα )α converges to some L ∈ K(K) such that L ⊆ Kχ . 2 For each α, there exists an indG N χα -invariant subspace Eα of L (K) such G that (indN χα )|Eα is equivalent to πα . (v) For each α, ϕα is of the form

(i) (ii) (iii) (iv)

∗ ϕα (f ) = indG N χα (f )ξα , ξα , f ∈ C (G),

where ξα ∈ J ∩ Eα . Moreover, ϕα ≤ ϕ and ϕα → ϕ in the w∗ topology. Proof It follows from Theorem 1.62 that there exists a net of tuples (πα , ψα ) such that (i) holds and ψα is associated with πα , ψα ≤ ϕ for all α, and  such that πα |N ∼ ψα → ϕ in the w ∗ -topology. For each α, there exists χα ∈ N K(χα ). Since πα |N → π|N by continuity of restriction and since π|N ∼ K(χ),  Moreover, since K(K) is compact, after we can assume that χα → χ in N. passing to a subnet if necessary, we can assume that Kχα → L for some L ∈ K(K). Notice next that Kχα → L and χα → χ implies that L ⊆ Kχ . Property (iv) follows from the fact that πα is a subrepresentation of indG N χα , realized on L2 (K). It remains to show that one can arrange for the special form of ϕα in (v).

246

Topologies on dual spaces

2 For each α, let Eα be an indG N χα -invariant subspace of L (K) such that G Eα indN χα | is equivalent to πα . Choose a vector ηα in Eα such that

ψα (f ) = indG N χα (f )ηα , ηα , for all f ∈ C ∗ (G). Let g be the identity of the minimal two-sided ideal J of G L2 (K), and let ρ = indG N χ and ρα = indN χα . Then, for each f ∈ Cc (G), ϕ(f ) = ρ(f )ξ, ξ  = ρ(f )(g ∗ ξ ), g ∗ ξ  = g ∗ ρ(f )(g ∗ ξ ), ξ  = ρ(g ∗ f )(g ∗ ξ ), ξ  = g ∗ ξ, ρ((g ∗ f )∗ )ξ  = ξ, g ∗ ρ((g ∗ f )∗ )ξ  = ξ, ρ(g ∗ (g ∗ f )∗ )ξ  = ρ((g ∗ (g ∗ f )∗ )∗ )ξ, ξ  = ϕ((g ∗ (g ∗ f )∗ )∗ ) = lim ψα ((g ∗ (g ∗ f )∗ )∗ ) α

= limρα (f )(g ∗ ηα ), g ∗ ηα , α

where the last equality follows by reversing the calculations for ϕ and ρ. For each α, the assignment f → ρα (f )(g ∗ ηα ), g ∗ ηα  defines a positive linear functional ϕα on Cc (G), and the above equation shows that limα ϕα (f ) = ϕ(f ) for every f ∈ Cc (G). Since Eα is a left ideal in L2 (K), g ∗ ηα ∈ Eα , and therefore ϕα is associated with πα . Also, since J is an ideal, g ∗ ηα ∈ J and hence ξα = g ∗ ηα ∈ J ∩ Eα . Moreover, for every α and f ∈ Cc (G), |ϕα (f )| ≤ f C ∗ (G) ξα 22 , and hence ϕα extends uniquely to a positive linear functional, also denoted ϕα , of C ∗ (G). Finally,

ϕα = g ∗ ηα 22 ≤ ηα 22 = ψα ≤ ϕ . This latter inequality not only establishes the norm bound in (v), but also tells us that the w ∗ -convergence of ϕα to ϕ on Cc (G) extends to w∗ -convergence on  all of C ∗ (G). This finishes the proof of the proposition.  and π ∈ G  such that π ∈ T . Retain the notation of Lemma 5.52 Let T ⊆ G E Proposition 5.51. Then there exists a subspace E of L2 (K), for which indG N χ| is equivalent to π and which has the following property: for each ξ ∈ E, there exists a subnet (Eαβ )β of the net (Eα )α and for each β a vector ηβ ∈ Eαβ such that ηβ → ξ in L2 (K). Proof Let ξα be as in Proposition 5.51. Then, for each α, ξα 2 = ϕα ≤

ϕ = ξ 2 , and hence the net (ξα )α of vectors is bounded in norm. Let J be the ideal associated with ξ prior to Proposition 5.51. Since J is finitedimensional, there exists a subnet (ξαγ )γ of the net (ξα )α such that ξαγ → ω for some ω ∈ J .

5.6 The topology on the dual of a motion group

247

Let E be the closed subspace generated by ω under the representation indG N χ ∗ and let E  = {indG χ(f )ω : f ∈ C (G)}, a dense subspace of E. For each N f ∈ C ∗ (G), the net of vectors indG N χαγ (f )ξαγ has the following properties. (1) indG N χαγ (f )ξαγ ∈ Eαγ for every γ . G (2) indG N χαγ (f )ξαγ → indN χ(f )ω.  and since the map Indeed, (2) follows since ξαγ → ω in J and χαγ → χ in N, G 2  into B(L (K)) is continuous. Now the statement of ρ → indN ρ(f ) from N the lemma follows by a diagonal-type approximation argument, using that E  is dense in E and passing to an appropriate subnet of γ → αγ . E ∗ It remains to verify that indG N χ| is equivalent to π. For f ∈ C (G), we have G indG N χ (f )ω, ω = lim indN χαβ (f )ξαβ , ξαβ  β

= lim ϕαβ (f ) = ϕ(f ) = indG N χ(f )ξ, ξ . β

G J E Thus, since ω ∈ J and ξ ∈ E, indG N χ| and indN χ| are two cyclic represen∗ tations of C (G) whose cyclic vectors define the same positive functional on G G E J J C ∗ (G). Therefore, indG N χ| is equivalent to indN χ| . As π = indN χ| , the proof is complete. 

 with  and π ∈ G  such that π ∈ T . Let χ ∈ N Proposition 5.53 Let T ⊆ G 2 π|N ∼ K(χ ). Then there exists a closed subspace E of L (K) such that E 2 indG N χ | is equivalent to π and, given any minimal two-sided ideal I of L (K), there exists a net of triples (πα , χα , Eα ) with the following properties.  πα ∈ T and πα → π in G.  . χα ∈ N , πα |N ∼ K(χα ), and χα → χ in N The stability groups Kχα converge in K(K) to some subgroup L of Kχ . Eα For each α, Eα is a closed subspace of L2 (K) such that (indG is N χα )| equivalent to πα . (v) For every ξ ∈ E ∩ I , there exist ξα ∈ Eα ∩ I such that the net (ξα )α converges uniformly on K to ξ .

(i) (ii) (iii) (iv)

Proof Let E be a subspace of L2 (K) as provided by Lemma 5.52. Then, by Proposition 5.51, there exists a net of triples (πα , χα , Eα ) satisfying properties (i)–(iv). To prove (v), let η ∈ E ∩ I be given. Then, after passing to a subnet, by Lemma 5.52 there exist ηα ∈ Eα such that ηα → ξ in the L2 -norm. Now let ω denote the identity of the ideal I and let ξα = ω ∗ ηα ∈ Eα ∩ I . Then  ξα = ω ∗ ηα → ω ∗ ξ = ξ uniformly.

248

Topologies on dual spaces

Lemma 5.54 Let K be a compact group and H a closed subgroup of K. Let F be a finite-dimensional left ideal of L2 (K) and let σ = λK |F , i.e., σ (x) = λK (x)|F for x ∈ K. (i) The set F |H = {f |H : f ∈ F } is a left ideal of L2 (H ). (ii) If M is a minimal left ideal of L2 (H ) contained in F |H , then the restriction σ |H contains the representation ρ of H defined by ρ(y) = λH (y)|M , y ∈ H , as a direct summand. Proof (i) follows because F ⊆ C(K) and hence F |H is a closed subspace of L2 (H ) which is invariant under left translation by elements of H . To prove (ii), we define a map φ : F → F |H simply by φ(f ) = f |H . Then φ is continuous because F is finite-dimensional and, for x, y ∈ H and f ∈ F , φ(σ (x)f )(y) = σ (x)f |H (y) = f (x −1 y) = λH (x)(f |H )(y). Thus φ intertwines the representations σ |H and λH , restricted to the subspace F |H . Writing F = F  ⊕ ker φ, it follows that σ |H , restricted to the subspace F  , is equivalent to λH , restricted to the subspace F |H , and this latter representation contains ρ as a subrepresentation.  Corollary 5.55 If the representation σ in Lemma 5.54 is a multiple of a representation τ , then τ |H contains ρ. , Gχ = N  Kχ , and χ  the character of Gχ defined by χ (xh) = Let χ ∈ N χ (x), for x ∈ N and h ∈ Kχ . Then, with q : G → G/N the quotient homomorG  ⊗ (λKχ ◦ q). Let σ = χ  ⊗ (λKχ ◦ q). We know that, by the phism, indN χ χ = χ G theorem on induction in stages, indN χ is equivalent to indG Gχ σ . However, for later use, we will have to determine a unitary operator explicitly furnishing this equivalence. G 2 −1 −1 We realize indG N χ on L (K) by [indN χ(xk)ξ ](l) = χ (l xl)ξ (k l), for x ∈ N, k, l ∈ K, and ξ ∈ L2 (K). Let L denote the space of all continuous mappings η : G → C(Kχ ) satisfying the equation η(g(xk))(h) = [σ ((xk)−1 )η(g)](h), G

g ∈ G, x ∈ N, k, h ∈ Kχ . Then L is indN χ χ-invariant. We now associate with any ξ ∈ C(K) a C(Kχ )-valued function φ(ξ ) by setting [φ(ξ )(xk)](h) = χ(k −1 xk)ξ (kh), for x ∈ N , k ∈ K, and h ∈ Kχ . It is then easily verified that φ(ξ ) ∈ L. Observe next that φ is an isometry. In fact, φ(ξ ) 2 = K |ξ (k)|2 dk = ξ 22 . Moreover, G φ intertwines the representations indG N χ and indGχ σ . Indeed, for any x, y ∈ N ,

5.6 The topology on the dual of a motion group

249

k, l ∈ K, and h ∈ Kχ , we have −1 −1 −1 [indG Gχ σ (yl)φ(ξ )](xk)(h) = [φ(ξ )(l y xl)l k)](h)

= χ(k −1 l(l −1 y −1 xl)kl)ξ (l −1 kh) = χ(k −1 xk)χ(k −1 yk)ξ (l −1 kh) = χ(k −1 xk)χ((kh)−1 y(kh))ξ (l −1 kh) = χ(k −1 xk)[indG N χ(yl)ξ ])(kh) = [φ(indG N χ(yl)ξ )(xh)](h). We show next that φ(C(K)) is a dense subspace of H(indG Gχ σ ). Since Kχ is compact, the representation σ is completely reducible. Thus L2 (Kχ ) = H(σ ) is the Hilbert space direct sum of subspaces Lα , where each Lα is σ -invariant and a finite-dimensional minimal ideal of L2 (Kχ ). Now each Lα , being finitedimensional, has a convolution identity and hence consists entirely of continuous functions. This shows that there exists a dense linear subspace X of H(σ ) consisting of functions η : G → L2 (Kχ ) satisfying (1) η is continuous; (2) for each x ∈ G, η(x) : h → η(x)(h) is continuous on Kχ ; (3) for each η ∈ X, there exists a finite-dimensional subspace Lη of L2 (Kχ ) such that η(x) ∈ Lη for every x ∈ G. Since X ⊆ L and X is dense in H(indG Gχ σ ), φ extends uniquely to a unitary G G 2 map from L (K) onto H(indGχ σ ) intertwining indG N χ and indGχ σ .  and let χ  Now return to the semidirect product G = N  K. Let χ ∈ N  denote the character of Gχ = N  Kχ defined by χ (xk) = χ(x) for x ∈ N and k ∈ Kχ . Moreover, let λ denote the lift of the left regular representation G of Kχ to Gχ . Then indN χ χ is equivalent to χ  ⊗ λ and hence, by induction in G   stages, indG N χ is equivalent to indGχ (χ ⊗ λ ). We need to make this equivalence explicit. Retaining the notation of Chapter 2, let F(G, χ  ⊗ λ ) denote the dense linear   subspace of H = H(indG Gχ (χ ⊗ λ )) consisting of all continuous maps ξ from   2 G into H(χ ⊗ λ ) = L (Kχ ) satisfying the covariance condition ξ ((x, k)(y, l)) = (χ  ⊗ λ )((y, l)−1 )ξ (x.k) = χ(y −1 )λKχ (l −1 )ξ (x, k),  for x, y ∈ N, k ∈ K, l ∈ Kχ . Let ψ ∈ Cc+ (N) such that N ψ(x)dx = 1, and define ϕ ∈ Cc (G) by ϕ(x, k) = ψ(x), x ∈ N , k ∈ K. Then it is easily verified that     ϕ((x, k)(y, l))dydl = ψ(y)dydl = 1. Kχ

N

Kχ

N

250

Topologies on dual spaces

It follows that the norm of ξ ∈ F(G, χ  ⊗ λ ) is given by    2 2

ξ H = ϕ(g)dg ξ (g) L2 (Kχ ) = ϕ(x, k) ξ (x, k) 2L2 (Kχ ) dxdk G K N    ψ(x) ξ (eN , k) 2L2 (Kχ ) dxdk =

ξ (eN , k) 2L2 (Kχ ) dk = K K N 2 = |ξ (eN , k)(l)| dldk, K

Kχ

since ξ (en , k) = ξ ((x, k)(k −1 · x −1 , ek )) = ξ (x, k) . We now set up the equivalence mentioned above. For f ∈ C(K), define a map φ(f ) : G → C(Kχ ) by [φ(f )(x, k)](l) = χ (k −1 · x −1 )f (kl), for (x, k) ∈ G and l ∈ Kχ . It is clear that φ(f ) ∈ F(G, χ  ⊗ λ ) and

φ(f ) 2H = K |f (k)|2 dk, so that φ is an isometry. We claim that the range of φ is dense in H. Since Kχ is compact, the representation χ  ⊗ λ is completely reducible. In fact, L2 (Kχ ) = H(χ  ⊗ λ ) is the direct sum ⊕α Iα , where each subspace Iα of L2 (Kχ ) is (χ  ⊗ λ )-invariant and a finite-dimensional minimal ideal of L2 (Kχ ). Also, being a finite-dimensional ideal of L2 (Kχ ), Iα is contained in C(Kχ ), and hence so is the linear span of all Iα . It follows that there is a dense linear subspace E of F(G, χ  ⊗ λ ) such that, for ξ ∈ E, (1) ξ (x) is a continuous function on Kχ for every x ∈ G; (2) there exists a finite-dimensional subspace Iξ of C(Kχ ) with ξ (x) ∈ Iξ for all x ∈ G. For ξ ∈ E, we define a function ψ(ξ ) on K by ψ(ξ )(k) = ξ (eN , k)(eKχ ). Since Iξ is finite-dimensional and consists of continuous functions, evaluation at the identity eKχ of Kχ is continuous on Iξ . So ψ(ξ ) is a continuous function on K. Notice next that φ(ψ(ξ )) = ξ for all ξ ∈ E. Indeed, for (x, k) ∈ G and l ∈ Kχ , we have [φ(ψ(ξ ))(x, k)](l) = χ (k −1 · x −1 )ψ(ξ )(kl) = χ((kl)−1 · x −1 )ξ (eN , kl)(ekχ ) = ξ ((eN , kl)(kl)−1 · x, eK ))(eKχ ) = ξ (x, kl)(eKχ = ξ (x, k)(l). This shows that φ maps C(K) onto a dense subspace of H. Since φ is an isometry, φ extends uniquely to an isometry, which we also denote φ, of L2 (K) G   onto H. Moreover, φ intertwines the representations indG N χ and indGχ (χ ⊗ λ ).

5.6 The topology on the dual of a motion group

251

Indeed, for f ∈ C(K) and (x, k), (y, h) ∈ G and l ∈ Kχ , we have −1 · x −1 )[indG φ[indG N χ (y, h)f ](x, k)(l) = χ(k N χ(y, h)f ](kl)

= χ (k −1 · x −1 )(l −1 · χ )(k −1 · y)f (h−1 kl) = χ(k −1 · (x −1 y))f (h−1 kl) = φ(f )(h−1 · (y −1 x), h−1 k)(l)   = [indG Gχ (χ ⊗ λ )(y, h)φ(f )](x, k)(l).

Let now Z be a linear subspace of L2 (K) which is invariant and irreducible under the representation indG N χ. There exists an irreducible representation τ Z of Kχ such that, denoting by τ  the representation of Gχ lifted from τ , indG N | G   is equivalent to indGχ (χ ⊗ τ ). Lemma 5.56 Let Y = {f |Kχ : f ∈ Z, f continuous}. Then Y is λKχ -invariant and λKχ |Y is equivalent to nτ for some n ∈ N. Proof Decompose L2 (Kχ ) into a direct sum of its minimal closed two-sided   ideals Iα . Then the Hilbert space H(indG Gχ (χ ⊗ λ )) decomposes into a direct sum of subspaces Mα , where each Mα consists of those functions on G which   are in H(indG Gχ (χ ⊗ λ )) and take values only in the ideal Iα . Let I = Iα0 be the minimal ideal of L2 (Kχ ) associated with τ . Then the subspace associated   with the representation indG Gχ (χ ⊗ τ ) is exactly M = Mα0 .   φ(Z) Z Now indG is equivalent to indG Gχ (χ ⊗ λKχ )| N χ| , which in turn is equivG   alent to indGχ (χ ⊗ τ ). Hence φ(Z) must lie in the subspace M. Therefore, if f is a continuous function in Z, then f |Kχ = φ(f )(eN , eK ) must be contained in the ideal I . This shows that Y ⊆ I , and since Y is a subspace of I which is closed under left translation by elements of Kχ , it follows that λKχ |Y is equivalent to a multiple of τ .  The following corollary is now immediate. Corollary 5.57 Retaining the above notation, let L be a minimal left ideal of Z.Then L = {f |Kχ : f ∈ L} is a nonzero left ideal of L2 (Kχ ), and the  representation λKχ |L is equivalent to nτ . Combining all the preceding, somewhat technical, results in this section, we are now able to prove the main result, which describes explicitly the closure of  for G a general motion group. any subset of G Theorem 5.58 Let G be a semidirect product G = N  K, where N is a  and locally compact abelian group and K is a compact group. Let π ∈ G G    T ⊆ G, and write π = indNKχ (χ × τ ), where χ ∈ N and τ ∈ Kχ . Then π is

252

Topologies on dual spaces

contained in the closure of T if and only if there exists a net (χα , τα )α consisting  of pairs of characters χα of N and elements τα ∈ K χα such that the following conditions are satisfied. (i) πα = indG NKχα (χα × τα ) ∈ T for each α.  and (Kχα , τα ) → (L, σ ) in S(K) for some (L, σ ) ∈ Sirr (K). (ii) χα → χ in N (iii) L ⊆ Kχ and σ ≤ τ |L . Proof Suppose first that a net (χα , τα )α satisfying (i), (ii), and (iii) exists. Then, K by (iii) and the Frobenius reciprocity theorem for compact groups, τ ≤ indL χ σ and hence K

NK

χ × τ ≤ χ × indL χ σ = indNL χ (χ × σ ). Together with the theorem on induction in stages, this implies that   NKχ G π = indG ind (χ × τ ) ≤ ind (χ × σ ) NKχ NKχ NL = indG NL (χ × σ ).  and hence π is contained in the Lemma 5.50 now shows that πα → π in G, closure of T . Conversely, suppose that π ∈ T . Choose a subspace E of L2 (K) as guaranteed by Proposition 5.53, and let M be a minimal left ideal of E and I the minimal two-sided ideal of L2 (K) containing M. Let F = {f |Kχ : f ∈ M}, and let ((πα , χα , Eα ))α be a net of triples as provided by Lemma 5.54. Then the net (Kχα )α converges in K(K) to some subgroup L of Kχ . For each α, let Fα denote the space of all functions f |Kχα , where f ∈ Eα and f is continuous. Then, applying Lemma 5.56, we have χ is such that π = indG (1) λkχ |F is equivalent to nτ , where τ ∈ K NKχ (χ × τ ). Fα (2) for each α, λKχα | is equivalent to nα τα , where nα ∈ N and τα is an irreducible representation of Kχα such that πα = indG NKχα (χα × τα ). Now let A be a minimal left ideal of L2 (L) which is contained in F |L , where F |L is defined as in Proposition 5.51, and let σ denote the representation λL |A . Then, by Corollary 5.55, τ |L ≥ σ and hence τ ≤ indK L σ by the Frobenius reciprocity theorem. This shows (iii). It remains to prove that (Kα , τα ) → (L, σ ) in S(K). To that end, we employ Theorem 5.36. Let η ∈ A, η = 0, and define a function ϕ of positive type associated with σ by setting  ϕ(s) = η(s −1 t)η(t)dνL (t), s ∈ L. L

5.7 Examples: motion groups

253

Then η = ξ |L for some ξ in the left ideal J . Now, by Proposition 5.53, there exists a net (ξα )α in L2 (K) with the following properties. (1) ξα → ξ uniformly on K. (2) For each α, ξα ∈ I ∩ Eα . Let ηα = ξα |Kα , so that ηα ∈ Mα . Define ϕ on Kα by  ηα (s −1 t)ηα (t)dνKα (t), s ∈ Kα . ϕα (s) = Kα

Then ϕα is a function of positive type associated with πα . Thus, in order to show the convergence of the net (Kα , τα )α to (L, σ ), by Theorem 5.36 it suffices to verify that ϕα → ϕ in Es (G). Thus assume, without loss of generality, that the net (ϕα )α has been replaced by a subnet of itself without changing notation. Let sα ∈ Kα and s ∈ L such that sα → s. We have to show that ϕα (sα ) → ϕ(s). Now      −1 −1  |ϕ(s) − ϕα (sα )| =  η(s t)η(t)dνL (t) − ηα (s t)ηα (t)dνKα (t) L K    α   −1 −1  ξα (s t)ξα (t)dνKα (t) =  ξ (s t)ξ (t)dνL (t) − L K    α   −1 −1  ≤  ξ (s t)ξ (t)dνL (t) − ξ (s t)ξ (t)dνKα (t) L Kα     −1 −1  +  [ξ (s t) − ξ (sα t)]ξ (t)dνKα (t) K  α    −1 −1  +  [ξ (sα t) − ξα (sα t)]ξα (t)dνKα (t) Kα

= c1 + c2 + c3 . Given  > 0, c1 is eventually <  due to the smooth choice of Haar measures and continuity of ξ , c2 is eventually <  because of the uniform continuity of ξ , and c3 is eventually <  because ξα converges uniformly on K to ξ . This finishes the proof of the theorem. 

5.7 Examples: motion groups The first purpose of this section is to describe the dual space topology for some motion groups to which Theorem 5.58 applies easily. Afterwards, we use Theorem 5.58 to deduce a necessary and sufficient condition for a general motion group to have a Hausdorff dual space.

254

Topologies on dual spaces

Example 5.59 Let G = Rn  SO(n), the group of rigid motions of Rn , n ≥ 2, and let H = Rn  SO(n − 1), where SO(n − 1) is identified with the stabilizer group of the vector (1, 0, . . . , 0) in SO(n). For r > 0 let χr denote the character  − 1), let of Rn defined by χr (x) = exp(2π irx1 ), and for r > 0 and τ ∈ SO(n G σr,τ = indH (χr × τ ). Recall that, with this notation,  -    = SO(n) − 1) . G σr,τ : r > 0, τ ∈ SO(n n = SO(n)  is closed in G  and carries the discrete topology. Clearly, G/R   if and only if for every We claim that a subset T of G is closed in G  τ ∈ SO(n − 1), the following two conditions are satisfied. (1) The set Rτ = {r ∈ (0, ∞) : σr,τ ∈ T } is closed in (0, ∞). (2) If 0 is a limit point of Rτ in R, then the support of indSO(n) SO(n−1) τ is contained in T .  is continuous, (1) is a Clearly, since the map r → σr,τ from (0, ∞) to G necessary condition. As for (2), observe that if rj → 0 then (χrj × τ )j is a  converging to 1Rn × τ . By continuity of inducing, (2) follows sequence in H whenever T is closed. Conversely, suppose that (1) and (2) hold and that (πj )j is a sequence  for all  Then we can assume πj ∈ SO(n) in T converging to some π ∈ G.  j , so that πj = σrj ,τj for certain rj ∈ (0, ∞) and τj ∈ SO(n − 1). Then, by  then π |Rn ∼ continuity of restricting, SO(n)(χrj ) → π|Rn . Thus, if π ∈ SO(n),  SO(n)(χr ) for some r > 0 and hence rj → r, whereas rj → 0 if π ∈ SO(n). By Theorem 5.58, after passing to a subsequence if necessary, we can assume that (SO(n − 1), τj ) = (Kχrj , τj ) → (L, τ ) in Sirr (SO(n)) for some (L, σ ) ∈ Sirr (K) such that π ≺ indG Rn L (χr × τ ), where r = 0 when rj → 0. Thus L = SO(n − 1) and τj = τ eventually, so that we can assume τj = τ for all j . If  r > 0, then indG Rn L (χr × τ ) = σr,τ ∈ G and hence π = σr,τ , which belongs to T by condition (1). Finally, if r = 0 then π ≤ indG H (1Rn × τ ) and hence π ∈ T by (2). Example 5.60 We return to Example 4.43, the semidirect product G = N  K, where N is the additive group of the p-adic number field and K is the multiplicative group of p-adic numbers x of valuation |x|p = 1 and K acts on N by multiplication. Then =K  ∪ {πj = indG G N χj : j ∈ Z}, where for each j , χj = χyj for a fixed element yj ∈ N with |yj |p = p j (see Example 4.43).

5.7 Examples: motion groups

255

 notice first that K  is closed in G  and K  is To determine the topology of G, discrete since K is compact. The set of all πj , j ∈ Z, is also discrete. In fact, with Oj = K(χj ) = {χy : y ∈ Nj }, we have  : π|N ∼ Oj }, {πj } = {π ∈ G and Oj is open since Nj is open in N and the mapping y → χy is a homeo [73, (10.16)]. morphism between N and its dual group N  if and only if πjn |N → Finally, a sequence (πjn )n converges to some γ ∈ K  This in 1N , that is, there exist elements kn ∈ K such that kn · χjn → 1N in N. turn means that χkn yjn = kn · χyjn = kn · χjn → 1n = χ0 , which in turn is equivalent to kn yjn → 0 and hence equivalent to yjn → 0 since |kn |p = 1. Example 5.61 Let G be the discrete group considered in Example 4.41, that is, the semidirect product G = Z2  D4 , where D4 is the dihedral 4 group.  turned out to be a Retaining the notation of Example 4.41, the dual space G  is determined j , 1 ≤ j ≤ 6, where this partition of G disjoint union of sets G by the different stabilizers of characters of Z2 : 1 = {π(t,s) = UGχ(t,s) : 0 < s < t < 1}; G (t,s) 2 = {UGχ(t,t) ×τ , UGχ(t,t) × : 0 < t < 1}; G (t,t) (t,t) 3 = {UGχ(t,0) ×τ , UGχ(t,0) × : 0 < t < 1}; G (t,0)

(t,0)

4 = {UGχ(1,s) ×τ , UGχ(1,s) × : 0 < s < 1}; G (1,s) (1,s) 5 = {UGχ(1,0) ×τ , UGχ(1,0) ×η1 , UGχ(1,0) ×η2 , UGχ(1,0) ×η2 }; G (1,0) (1,0) (1,0) (1,0) 6 = {χ(0,0) × ρ, χ(1,1) × ρ : ρ ∈ {τ, σ1 , σ2 , σ3 , π }}. G j the parameter space topology coincides It is clear that on each of the sets G 5 and G 6 are finite and that with the relativized dual space topology. Note that G  it suffices  is a T1 space. Therefore, in order to describe the topology on G, G j which to fix j ∈ {1, . . . , 4} and determine the limit set of any sequence in G   converges to some element of G \ Gj . We carry this out for the dense open set 1 , the other cases being similar. Thus, let (π(tn ,sn ) )n∈N be a sequence in G 1 G \G 1 . Then we can assume that (tn , sn ) → (t, s) for converging to some π ∈ G some (t, s) ∈ [0, 1]2 . According to the possible choices of (t, s), using Theorem 5.58 and the fact that the stabilizer in D4 of any character χ(u,v) , 0 < u < v < 1, is trivial, we get 2 ; (1) if (tn , sn ) → (t, t) with 0 < t < 1, then π(tn ,sn ) → G 3 ; (2) if (tn , sn ) → (t, 0) with 0 < t < 1, then π(tn ,sn ) → G 4 ; (3) if (tn , sn ) → (1, s) with 0 < s < 1, then π(tn ,sn ) → G

256

Topologies on dual spaces

4 ; (4) if (tn , sn ) → (1, 1), then π(tn ,sn ) → {χ(1,1) } × D 4 . (5) if (tn , sn ) → (0, 0), then π(tn ,sn ) → {χ(0,0) } × D We close this section by applying the results obtained above to derive a criterion for when the dual space of a motion group is Hausdorff. Proposition 5.62 Let G = N  K, where N is a locally compact abelian  is a Hausdorff space if and only if group and K is a compact group. Then G the stabilizer map χ → Kχ = {a ∈ K : a · χ = χ }  to K(K) is continuous. from N  Proof Suppose first that the map fails to be continuous. Then there exist χ ∈ N  such that χα → χ, but for which the net (Kχα )α does and a net (χα )α in N not converge to Kχ . However, since K(K) is compact, we may assume that Kχα → L for some closed subgroup L of K. It is clear that then L ⊆ Kχ , but by assumption L = Kχ . The net of subgroup representation pairs (N  Kχα , χα × 1Kχα ) converges to (N  L, χ × 1L ) in S(G). Let  πα = indG NKχα (χα × 1Kχα ) ∈ G. Then, by continuity of inducing (Theorem 5.39),   NKχ G πα → indG NL (χ × 1L ) = indNKχ indNL (χ × 1L ) . NK

Now, since L = Kχ , apart from χ × 1Kχ , the representation indNL χ (χ × 1L ) χ . Thus the net (πα )α contains at least one different character χ × σ , σ ∈ K converges to at least the two different irreducible representations indG NKχ (χ × 1Kχ )

and

indG NKχ (χ × σ )

 is not Hausdorff. of G, and hence G  is not Hausdorff. Then there exists a net (πα )α Conversely, assume that G   Let χ ∈ N  and τ ∈ in G converging to two distinct elements ρ and ω in G. G  Kχ such that ρ = indNKχ (χ × τ ). Then there exist (L, σ ) ∈ S(G) and a net ((χβ , (Kχβ , τβ )))β with the following properties. (1) L is a subgroup of Kχ and τ is a subrepresentation of indK L σ. (2) The net formed by the representations ρβ = indNKβ (χβ × τβ ) is a subnet of the net (πα )α . (3) The net (χβ , (Kχβ , τβ )β converges to (χ , (L, σ )). Now the net (ρβ )β converges also to ω and hence ρβ |N → ωN in Rep(N).    It follows that ω = indG NKχ (χ × τ ) for some τ ∈ Kχ . Since ρ and ω are

5.7 Examples: motion groups

257

different, τ and τ  are inequivalent. Moreover, τ and τ  are both subrepresenK tations of indL χ σ . This implies that L must be a proper subgroup of Kχ . This means that the net (Kχβ )β does not converge to Kχ , although (χβ )β converges to χ . This shows that the stabilizer map is not continuous at χ, completing the proof of the proposition.  We remind the reader that a locally compact group is called compactfree if the identity is the only element which generates a relatively compact subgroup. Corollary 5.63 Let G be a semidirect product G = N  K, where K is com pact and N is a compactly generated and compact-free abelian group. Then G is a Hausdorff space if and only if G is the direct product of N and K.  × K,  we only have to show that Hausdorffness of Proof Since N ×K =N  G implies that G = N × K, equivalently, that K acts trivially on N . Because the characters of N separate the elements of N , this will follow once we have  and a ∈ K. shown that a · χ = χ for every χ ∈ N  is Hausdorff, by Proposition 5.62 the stabilizer map χ → Kχ of Now, as G  into K(K) is continuous. Let U = U (∅, F) be a typical neighborhood (see N  Section 5.1) of K = K1N in K(K) and choose a neighborhood V of 1N in N such that Kχ ∈ U for all χ ∈ V . By the structure theorem for compactly generated locally compact abelian groups [73, (9.8) theorem] and since N is compact-free, N has the form N =  = Rk × Tl , which is topologically divisible in the Rk × Zl , k, l ∈ N0 . Thus N  and a neighborhood W of 1N , there exist η ∈ W following sense. Given χ ∈ N n and n ∈ N such that η = χ. Choose W = V . Then Kη ∈ U and Kη ⊆ Kχ . However, this implies that Kχ ∈ U as well. Thus we have seen that Kχ ∈ U  Since this holds for any such neighborhood U of K in K(K), for each χ ∈ N. .  we conclude that Kχ = K for every χ ∈ N One might conjecture that a locally compact group G must be the direct product of a locally compact abelian group and a compact group whenever  is a Hausdorff space. The following example shows that this the dual space G conjecture is not even true for motion groups. However, the group constructed below has a center of finite index. Example 5.64 Let N = Z × T and K = Z2 = {1, −1}, where −1 acts on N by = Z× T with T × Z. (−1) · (n, z) = (n, (−1)n z), for n ∈ Z, z ∈ T. Identify N Then, for (w, m) ∈ T × Z, the stability subgroup K(w,m) equals K if m is even  into K(K) is continuous, and K = {1} if m is odd. So the map χ → Kχ from N  is Hausdorff by Proposition 5.62. and hence G

258

Topologies on dual spaces

Now assume that G is the direct product of an abelian group M and a compact group C, and write elements of G as triples (n, z, a), n ∈ Z, z ∈ T, a ∈ K. The map ϕ : (n, z, a) → n is a continuous homomorphism onto Z. Thus ϕ(C) = {0} and hence n = 0 whenever (n, z, a) ∈ C. Moreover, M is contained in the center of G. However, an element (m, w, b) of G belongs to the center of G if and only if m is even. Thus every element of G is of the form (m, w, b)(0, z, a) = (m, w(−1)m z, ba) = (m, wz, ba) with m even. This contradiction shows that G cannot be written as the direct product of an abelian group and a compact group.

5.8 The primitive ideal space of a two-step nilpotent group Let G be a two-step nilpotent locally compact group. Since such a group need not be of type I, according to general understanding there is no hope of being  However, it is possible to describe, in terms of group data, able to determine G. the primitive ideal space Prim(G) as a topological space. This is the aim of this section. We start by observing that every irreducible representation of G is weakly equivalent to a representation induced from some G-invariant character of a certain normal subgroup of G. It will turn out that, conversely, such induced representations are weakly equivalent to irreducible representations. This then leads to a convenient parametrization of Prim(G). Proposition 5.65 Let G be a two-step nilpotent locally compact group and let Z be a closed subgroup of G such that Z is contained in the center of G and  let G/Z is abelian. For λ ∈ Z, Zλ = {z ∈ Z : λ(z) = 1}

and

Lλ = {x ∈ G : [x, G] ⊆ Zλ },

that is, Lλ /Zλ equals the center of G/Zλ . Let π be an irreducible representation of G such that π|Z is a multiple of λ. Then there exists a G-invariant character α of Lλ such that π ∼ indG Lλ α. Proof We first assume that λ is faithful on Z and later reduce to this case. Then Lλ = Z(G). Let ξ ∈ H(π) with ξ = 1, and let ϕ be the function of positive type associated with ξ . For x ∈ G, define ϕ x ∈ P (G) by ϕ x (y) = ϕ(xyx −1 ), y ∈ G. Then, since [G, G] ⊆ Z and π |Z ∼ λ, ϕ x (y) = ϕ([x, y]y) = π([x, y])π (y)ξ, ξ  = λ([x, y])ϕ(y).

5.8 The primitive ideal space of a two-step nilpotent group

259

Moreover, again since [G, G] ⊆ Z, for all y1 , y2 ∈ G, λ([x, y1 y2 ]) = λ([x, y1 ][x, y2 ]) = λ([x, y1 ])λ([x, y2 ]). Thus the function c(x) : y → λ([x, y]) is a character of G (actually, a character of G/Z(G) lifted back to G) satisfying the equation ϕ x = c(x)ϕ for all x ∈ G. In addition, the map c : x → c(x) is a homomorphism of G into the group of characters of G. If x ∈ G \ Z(G), then [x, y] = e for some y ∈ G and hence λ([x, y]) = 1 since λ is faithful on Z. Therefore c(G), viewed as a subgroup  separates the points of G/Z(G) and hence is dense of the dual group G/Z(G),  in G/Z(G). Now, for any character γ of an arbitrary locally compact group G, the map f → γf is an isometric isomorphism of L1 (G) onto itself. Since π (γf ) = (π ⊗ γ )(f ) for any representation π, f → γf extends uniquely to an isometric isomorphism, denoted u → γ u, of C ∗ (G) to itself. In particular, γ u ∈ ker π if and only if u ∈ ker(π ⊗ γ ). For any function ψ on G and a, b ∈ G, let a ψb (y) = ψ(ayb), y ∈ G. Returning to our situation above, it is easily verified that, for f ∈ L1 (G) and a, b, x ∈ G,   −1 (c(x)f )(y)a ϕb (y)dy = λ([ab, x ]) f (y)ax ϕx −1 b (y)dy. G

G

Since ϕ ∈ P (G), a ψb and ax ϕ are linear combinations of positive definite functions. Therefore, in the preceding equation, both integrals depend continuously on f with respect to the C ∗ -norm on L1 (G). This implies that x −1 b

c(x)u, a ϕb  = λ([ab, x −1 ])u, ax ϕx −1 b  holds for all u ∈ C ∗ (G) and a, b, x ∈ G. Now, since ker π = {u ∈ C ∗ (G) : u, c ϕd  = 0 for all c, d ∈ G}, we obtain that ker(c(x) ⊗ π) = ker π for all x ∈ G. As c(G) is dense in  it follows that π ≺ π ⊗ γ for every γ ∈ G/Z(G)  and hence also G/Z(G),  Then, by Theorem 2.58, and γ ⊗ π ≺ γ ⊗ (γ ⊗ π) = π for all γ ∈ G/Z(G).  since indG 1 ∼ G/Z(G), Z(G) Z(G) G  indG Z(G) (π|Z(G) ) = π ⊗ indZ(G) 1Z(G) ∼ π ⊗ G/Z(G)

 ∼ π. ∼ {π ⊗ γ : γ ∈ G/Z(G)} On the other hand, since π is irreducible, π|Z(G) is a multiple of a character α of Z(G). This shows that π ∼ indG Z(G) α. Now, drop the hypothesis that λ be faithful on Z, let K = Zλ , and let q : G → G/K be the quotient homomorphism. Then λ defines a faithful character λ˙ on ˙ Z/K by λ(zK) = λ(z), z ∈ Z, and π defines an irreducible representation π˙ of ˙ by the first part of the proof G/K by π(xK) ˙ = π(x), x ∈ G. Since π| ˙ Z/K ∼ λ,

260

Topologies on dual spaces

G/K

there exists a G/K-invariant character α˙ of Z(G/K) such that π˙ ∼ indZ(G/K) α. ˙ −1 Then α = α˙ ◦ q is a G-invariant character of q (Z(G/K)) and, by Proposition 2.38,   G/K π = π˙ ◦ q ∼ indZ(G/K) α˙ ◦ q = indG q −1 (Z(G/K)) α. Finally, notice that q −1 (Z(G/K)) = Lλ since for x ∈ G, xK ∈ Z(G/K) if and only if [x, G] ⊆ K, and by definition of K this in turn is equivalent to the condition λ([x, G]) = {1}, that is, to x ∈ Lλ . This completes the proof of the proposition.  We now define the set which will turn out to parametrize Prim(G). For  let Lλ be as before and let Aλ denote the set of all characters α of λ ∈ Z(G), Lλ such that α|Z(G) = λ. Let  α ∈ Aλ }. P = {(λ, α) : λ ∈ Z(G), Theorem 5.66 Let G be a two-step nilpotent locally compact group. Then the map (λ, α) → ker(indG Lλ α) is a one-to-one correspondence between P and Prim(G).  and α ∈ Aλ , and let π be any irreducible representation of Proof Let λ ∈ Z(G) G G that is weakly contained in indG Lλ α. We have to show that in fact π ∼ indLλ α. Since α is G-invariant, π |Lλ ∼ α and hence G G  indG Lλ α ∼ indLλ (π|Lλ ) = π ⊗ indLλ 1Lλ ∼ π ⊗ G/LLλ .

 It therefore suffices to show that π ⊗ γ ≺ π for every γ ∈ G/L Lλ . The following arguments are analogous to those in the proof of Proposition 5.65, they are however included for completeness. For each y ∈ G, define γy : G → T by γy (x) = λ([y, x]), x ∈ G. Then γy is continuous and, since the map x → [y, x] is a homomorphism of G into Z(G), we have γy (x1 x2 ) = λ([y, x1 x2 ]) = λ([y, x1 ][y, x2 ]) = λ([y, x1 ])λ([y, x2 ]) = γy (x1 )γy (x2 ), for all x1 , x2 ∈ G. So γy is a character of G, which by its definition is identically λ for every y ∈ G. Moreover, the map  : y → γy one on Lλ . Thus γy ∈ G/L λ since is a homomorphism from G into G/L γy1 y2 (x) = λ([y1 y2 , x]) = λ([y1 , x])λ([y2 , x]) = γy1 (x)γy2 (x), λ separates for all y1 , y2 , x ∈ G. Notice next that the subgroup (G) of G/L the points of G/Lλ . Indeed, if x ∈ G is such that 1 = γy (x) = λ([y, x]) for all

5.8 The primitive ideal space of a two-step nilpotent group

261

λ . Now y ∈ G, then x ∈ Lλ . It follows that (G) is dense in G/L (π ⊗ γy )(x) = λ([y, x])π (x) = π([y, x] x) = π(yxy −1 ) = y −1 · π (x). Since π and y −1 · π are equivalent, we conclude that π and π ⊗ γy are equivλ , it follows that π ⊗ γ ≺ π alent for all y ∈ G. Since (G) is dense in G/L λ . for each γ ∈ G/L Thus we have shown that ker(indG Lλ α) ∈ Prim(G). Since by Proposition 5.65 every P ∈ Prim(G) is of the form P = ker(indG Lλ α) for some pair (λ, α) ∈ P, it only remains to observe that the map P → Prim(G),

(λ, α) → ker(indG Lλ α)

 and αj ∈ L  is injective. But this is a simple matter. Suppose that λj ∈ Z(G) λj , G G j = 1, 2, are such that αj |Z(G) = λj and indLλ α1 ∼ indLλ α2 . Then clearly 1 2 λ1 = λ2 and, setting L = Lλj ,     G α1 ∼ indG | α ∼ ind α 1 L 2 |L ∼ α2 , Lλ Lλ 1

2

since α1 and α2 are G-invariant. Consequently, α1 = α2 . This finishes the proof of the theorem.  The last part of the proof of Theorem 5.66 in particular shows that ideals in Prim(G) are maximal ideals. The parametrizing set P can be identified with a subset of S(G) and therefore be given the relative subgroup representation topology. However, this relative topology does not in general describe the topology on Prim(G). We next introduce a topology on P which makes the bijection of Theorem 5.66 a homeomorphism. Definition 5.67 A net (λι , αι )ι in P converges to (λ, α) ∈ P if and only if every subnet (λικ , αικ )κ contains in turn a subnet (λικν , αικν )ν such that  (i) λικν → λ in Z(G), (ii) (Lλικν , αικν )ν converges in S(G) to some (L , α  ) where L ⊆ Lλ and α|L = α. It is straightforward to check that this definition of convergence of nets defines a topology on P. We are now able to achieve the main result of this section. Theorem 5.68 Let G be a two-step nilpotent locally compact group. With the topology on P defined in Definition 5.67, the map (λ, α) → ker(indG Lλ α) is a homeomorphism between P and Prim(G).

262

Topologies on dual spaces

Proof Let (λι , αι )ι be a net in P converging to (λ, α) ∈ P. Then, by definition of the topology on P, every subnet (λικ , αικ )κ contains a subnet (λικν , αικν )ν such that (Lικν , αικν )ν converges in S(G) to a pair (L , α  ) with L ⊆ Lλ and α|L = α  . Because both α and α  are trivial on Zλ , we can write α = β ◦ qλ and α  = β  ◦ qλ , where qλ is the quotient homomorphism from G to G/Zλ and β is a character of the abelian group Lλ /Zλ and β  is a character of the subgroup L /Zλ of Lλ /Zλ . Then α|L = α  implies that β|L /Zλ = β  . Since Lλ /Zλ is abelian, the one-dimensional representation β is weakly contained in L /Z the induced representation indLλ /Zλλ β  . Thus   L /Z α = β ◦ qλ ≺ indLλ /Zλλ β  ◦ qλ = indLLλ α  . Therefore, if π is any unitary representation of G that is weakly contained in indG Lλ α, then Lλ  G  π ≺ indG Lλ (indL α ) = indL α .

Now, let Mν = Lλικν and γν = αικν . Since the net (Mν , γν )ν converges to (L , α  ) in S(G), it follows from continuity of inducing that for each ν, there exists an irreducible representation πν which is weakly contained in indG Mν γν such that G  πν → π in G. Notice next that ker πν = ker(indMν γν ) and ker π = ker(indG Lλ α) since the kernels of these induced representations are maximal ideals. Finally,  onto Prim(G) is continuous, we see that since the map σ → ker σ from G G G ker(indMν γν ) → ker(indLλ α). Summarizing we have shown that every subnet G of (ker(indG Lλι αι ))ι contains a subnet which converges to ker(indLλ α). This proves continuity of the map from P to Prim(G). To show that the inverse map is also continuous, let (ker(indG Lλι αι ))ι be a net G converging to ker(indLλ α) in Prim(G). Choose irreducible representations π and πι such that π is weakly equivalent to indG Lλ α and πι is weakly equivalent  πι → π . Now, since π|Z(G) to indG α . By definition of the topology on G, ι Lλι is a multiple of λ and πι |Z(G) is a multiple of λι , it follows from continuity of  Let (λικ , αικ )κ be an arbitrary subnet of (λι , αι )ι . restriction that λι → λ in Z(G). Since K(G) is compact, there is a further subnet (λικν , αικν )ν such that (Lλικν )ν converges in K(G) to some subgroup L of G. Set μν = λικν and βν = αικν . We claim that L ⊆ Lλ . To see this, let x ∈ L . Then, by definition of convergence in K(G), there is a net (xν )ν such that xν ∈ Lμν for each ν and xν → x.  an element y of G belongs to Lη if and only if η([y, t]) = 1 Now, for η ∈ Z(G), for all t ∈ G. Thus we have μν ([xν , t]) = 1 for all ν and all t ∈ G. For every  to C is t ∈ G, the function (y, η) → ([y, t], η) → η([y, t]) from G × Z(G) continuous. Since xν → x and μν → λ, we obtain that λ([x, t]) = 1 for all t ∈ G and hence x ∈ Lλ .

5.8 The primitive ideal space of a two-step nilpotent group

263

The restriction of π to Lλ is a multiple of α, since the restriction of indG Lλ α to Lλ is a multiple of α. Hence the restriction of π to L is a multiple of the character α|L of L . By continuity of restriction, for each ν there exists an irreducible representation ρν of Lμν which is weakly contained in πικν |Lμν , such that the sequence (Lμν , ρν )ν converges to (L , α|L ) in S(G). Now πικν |Lμν is a multiple of βν . This implies that ρν = βν . Thus (Lμν , βν )ν converges in S(G) to (L , α|L ). By definition of convergence in P, this shows that the original net (λι , αι )ι converges in P to (λ, α). Therefore the inverse map from Prim(G) to P is continuous.  For the continuous Heisenberg groups Hn , n ∈ N, in particular the threedimensional real Heisenberg group H1 , we have already described the dual (the primitive ideal space) and its topology in Example 5.46. We conclude this section by applying Theorems 5.66 and 5.68 to determine the set Prim(G) and its topology for three further two-step nilpotent locally compact groups. All three are upper-triangular matrix groups, but they show a fairly different behavior. Example 5.69 Let k be a locally compact field and G the group of uppertriangular matrices ⎛ ⎞ 1 x z ⎝ 0 1 y ⎠ , x, y, z ∈ k. 0 0 1  λ = 1Z(G) . Thus, as for It is easy to check that Lλ = Z(G) for every λ ∈ Z(G), the three-dimensional Heisenberg group,   Prim(G) = {ker(indG Z(G) λ) : λ ∈ Z(G), λ = 1Z(G) } ∪ G/Z(G). Of course, the map λ → Pλ = ker(indG Z(G) λ) is a homeomorphism between   \ Z(G) \ {1Z(G) } and Prim(G) \ G/Z(G). Moreover, if (λα )α is a net in Z(G)  {1Z(G) } converging to 1Z(G) , then Pλα → χ for every χ ∈ G(Z(G)). Example 5.70 Let p be a prime number, p the p-adic number field equipped with the usual topology defined by the p-adic metric, and p the compact open subring of p-adic integers. Let G be the two-step nilpotent group of upper-triangular matrices ⎛ ⎞ 1 x z ⎝ 0 1 y ⎠ , x ∈ p , y, z ∈ p . 0 0 1 In what follows we denote such a matrix by the triple (x, y, z). Let N be the abelian normal subgroup consisting of all elements (0, y, z) and note that

264

Topologies on dual spaces

Z(G) = {(0, 0, z) : z ∈ p }. Since G/N is compact, G is a CCR-group. Our  For the following well-known facts, see first aim is to determine Prim(G) = G. [73, (10.16)]. For k ∈ Z, p k p is a compact open subgroup of the additive group p , and these are the only nontrivial closed subgroups of p . Note that & p k p ⊇ p k+1 p , pk p = p , and p k p = {0}. k∈Z

k∈Z

There exists a character χ of p with kernel equal to p . For each y ∈ p , p by λy (x) = χ(yx), x ∈ p . Then the mapping y → λy is a define λy ∈

p . We now compute Lλy for y = 0. topological isomorphism between p and

k There is a unique k ∈ Z such that y ∈ p p \ p k+1 p . Then Zλy = {(0, 0, z) ∈ G : χ(yz) = 1} = {0} × {0} × y −1 p = {0} × {0} × p −k p , and this implies that Lλy = {(x, y, z) ∈ G : [(x, y, z), G] ⊆ Zλy } = {(x, y, z) ∈ G : (0, 0, xy  − x  y) ∈ Zλy for all x  ∈ p and y  ∈ p } = {(x, y, z) ∈ G : x p ⊆ p−k p and yp ⊆ p−k p } = {(0, y, z) ∈ G : y ∈ p−k p } = {0} × p−k p × p . For k ∈ Z, let k = {λy : y ∈ p k p \ p k+1 p }

and

Gk = {0} × p−k p × p .

p \ {1 p } is the disjoint union of the open and closed sets k . The Then

above computation shows that Lλ = Gk for all λ ∈ k . Moreover, if (λn )n is  \ {1Z(G) } converging to 1Z(G) , then Lλn → N in an arbitrary sequence in Z(G) K(G). Indeed, if λn = λyn with yn ∈ kn , then necessarily kn → ∞ and hence Lλn = {0} × p −kn p × p → N in K(G). These facts allow us to determine P and the topology on it. For k ∈ Z, let −k  }. Pk = {(λ, γ × λ) : λ ∈ k , γ ∈ p p

 Pk is open and closed in P and, by Since k is open and closed in p = Z(G), definition of the topology of P (Definition 5.67), the map (λ, γ ) → (λ, γ × λ) −k  and P . Now P is a disjoint union is a homeomorphism between k × p p k $ %  P= Pk ∪ {(1Z(G) , α) : α ∈ G/Z(G)}, k∈Z

and if (λn , αn ) = (λn , γn × λn ) ∈ Pkn , n ∈ N, then (λn , αn ) → (1Z(G) , α) in P if and only if kn → ∞ and (p−kn p × p , αn ) → (N, α|N ) in S(G).

5.8 The primitive ideal space of a two-step nilpotent group

265

p , (p −kn p , γn ) → ( p , γ ) Equivalently, since α|N = γ × 1Z(G) with γ ∈

in S( p ). This in turn means that if x ∈ p and k is such that x ∈ p−k p , then x ∈ p−kn p for all n with kn ≥ k and γn (x) → γ (x). Our final example is a group which is not of type I, namely the discrete analogue of the three-dimensional continuous Heisenberg group. Example 5.71 Let G be the integer Heisenberg group, that is, G = Z3 with multiplication given by (k, l, m)(k  , l  , m ) = (k + k  , l + l  , m + m + kl  ), (k, l, m), (k  , l  , m ) ∈ Z3 . Then (k, l, m)−1 = (−k, −l, −m + kl) and [(k, l, m), (k  , l  , m )] = (0, 0, kl  − k  l). We identify Z with the center Z(G) = {0} × {0} × Z via m → (0, 0, m) and  via z → χz , where χz (n) = zn , n ∈ Z. Then the circle group T with Z(G) Lz = {(k, l, m) ∈ G : χz ([(k, l, m), g]) = 1 for all g ∈ G} 



= {(k, l, m) ∈ G : zkl −k l = 1 for all k  , l  ∈ Z} = {(k, l, m) ∈ G : zk = zl = 1}. Let z = exp(2π it), t ∈ R. Then Lz = Z(G) when t is irrational, whereas if t is rational, say t = pq with (p, q) = 1, Lz = qZ × qZ × Z. The next step is to calculate Aχz when t is rational. For u, w ∈ T, define the character αu,w,z of Z3 by αu,w,z (k, l, m) = uk wl zm ,

k, l, m ∈ Z.

Then, for z ∈ T, z : α|Z(G) = χz } = {αu,w,z |Lz : u, w ∈ T}. Aχz = {α ∈ L Thus the parametrizing set P is given by P = {(χz , χz ) : z = exp(2π it), t ∈ R, t irrational}  #   p , u, w ∈ T . (χz , αu.w,z |Lz ) : z = exp 2π i q p,q∈N, p≤q, (p,q)=1

266

Topologies on dual spaces

It remains to determine the topology on P. Let (λn , αn )n be a sequence in P and (λ, α) ∈ P. Let λn = χzn and λ = χz , zn , z ∈ T. We first consider three special cases. (1) For all n, zn = exp(2π itn ), where tn is irrational. (2) For all n, zn = exp(2π i pqnn ), where 1 ≤ pn ≤ qn , (pn , qn ) = 1 and qn → ∞. (3) For all n, zn = exp(2π i pqnn ), where 1 ≤ pn ≤ qn , (pn , qn ) = 1, but the sequence (qn )n is bounded. In case (1), Lzn = {0} × {0} × Z for all n and hence (χzn , αn ) → {χz } × Aχz which is a singleton if z ∈ exp(2π iQ) and homeomorphic to {χz } × T2 if z ∈ exp(2π iQ). Case (2) is analogous because Lzn = qn Z × qn Z × Z → {0} × {0} × Z. Assuming (3), since the sequence (zn )n is convergent, it actually has to be finite and hence constant. Finally, it is straightforward to combine the cases whenever the convergent sequence (zn )n has subsequences of either type. The higher-dimensional integer Heisenberg groups Zk+1  Zk , k ≥ 2, where x = (x1 , . . . , xk ) ∈ Zk acts on (y, m) = (y1 , . . . , yk , m) ∈ Zk+1 by ⎞ ⎛ k  xj yk ⎠ , x · (y, m) = ⎝y, m + j =1

can be treated in an analogous manner.

5.9 Notes and references The existence of a smooth choice of Haar measures on the space K(G) of all closed subgroups of G was established by Glimm [62]. Our presentation in Section 5.2 follows closely the exposition in the excellent appendix H.2 of Williams [153]. All other material appearing in Sections 5.1–5.4 is the work of Fell. The compact–open topology, which is also often referred to as the Fell topology, on the collection C(X) of closed subsets of an arbitrary topological space X was introduced in [48]. The need to have available a topology on Rep(G), or more generally on Rep(A) for a C ∗ -algebra A, which coincides  with the hull–kernel topology, arises for instance from the fact that on A the representations obtained from the procedures of inducing or restricting irreducible representations, are in general not irreducible anymore. This inner hull–kernel topology on Rep(A) was defined and studied in [46], and it makes  is compact. The fundamental Theorem 5.9 Rep(A) a compact space since C(A) was proved in [49, theorem 2.1].

5.9 Notes and references

267

Fell’s purpose was to introduce an adequate topology on the set S(G) of all pairs (K, π), where K is a closed subgroup of G and π is a unitary representation of K, and then to prove that the map (H, K, π ) → (H, indH K π ), where H is a closed subgroup of G containing K, is continuous in all three variables K, π and H (and similarly for the restriction process). A main step in this program, is the construction in section 2 of [51] of the subgroup algebra As (G) and the subgroup C ∗ -algebra Cs∗ (G) which, as Fell points out, was inspired by the work of Glimm [62]. According to this construction, every (K, π) ∈ S(G) defines a representation σK,π of Cs∗ (G), and the irreducible representations of Cs∗ (G) are precisely the σK,π , where π is irreducible (Theorem 5.27). The inner hull–kernel topology on S(G) is then naturally defined to be that topology which makes the map (K, π) → σK,π from S(G) into Rep(C ∗ (G)) a homeomorphism. The basic properties of the inner hull–kernel topology (Lemmas 5.24, 5.25, and 5.26) were given in [51, section 2]. The results in Section 5.3, also established in [51], provide a description of the topology on S(G) in terms of functions of positive type on subgroups (Theorem 5.35) and form an essential tool when proving continuity of inducing and restricting representations in Section 5.4. Theorem 5.37 concerning the restriction process and, most notably, Theorem 5.39, stating the continuity of the maps (H, K, π ) → (H, π |H ) and (H, Kπ ) → (H, indH K π ) from the corresponding subsets of K(G) × S(G) into S(G), respectively, form the highlights in Fell’s theory. Special cases of these theorems have previously been shown and are easier to obtain. For instance, for both H and K fixed, Theorem 5.39 is [49, theorem 4.1], and, as Fell points out, for fixed H but varying subgroups K of H , it can also be deduced from results of Glimm [62]. Since its introduction, the hull–kernel (or dual space) topology on the dual  or the primitive ideal space Prim(G) of a locally compact group G has space G consistently been a major topic in representation theory and harmonic analysis. When G is abelian, this topology coincides with the topology of uniform convergence of characters on compact subsets of G. It is also well-known that the dual space of a compact group is discrete. Conversely, it has been shown by Stern [147], and independently by Baggett [3] for second countable groups,  forces G to be compact. The hull–kernel topology on the that discreteness of G dual space of several low-dimensional simply connected nilpotent Lie groups has been analyzed by Dixmier [35], by studying the characters of irreducible representations which are tempered distributions, and subsequently by Fell [49] applying Proposition 5.42. Accordingly, we follow [49] to some extent. Alternatively, denoting by g the Lie algebra of G and by Ad∗ the coadjoint action of G on the vector space dual g∗ of g, the dual space topology could be  and the coadjoint determined by using the Kirillov correspondence between G orbit space g∗ /Ad∗ (G) and the important result, due to I. D. Brown [28], that

268

Topologies on dual spaces

this correspondence is a homeomorphism when g∗ /Ad∗ (G) is endowed with the quotient topology. Those examples in Section 5.5, which are semidirect products of two abelian groups, could also be treated by employing a description of the topology on the primitive ideal space of certain transformation group C ∗ -algebras [151, theorem 5.3]. The dual space topology for the classical motion groups Rn  SO(n), n ≥ 2, has long been known. The description of the dual space topology for general semidirect product groups N  K, where N is a locally compact abelian group and K is a compact group, highlighted in Theorem 5.58, is due to Baggett [2] and our exposition follows [2]. A similar result holds for semidirect products G = N  A of second countable locally compact abelian groups under the additional hypothesis that N is regularly embedded in G [2, theorem 3.3]. In [2], Baggett also proved Proposition 5.62. As all the examples show, the dual space is in general wildly non-Hausdorff. Of course, the important question of which locally compact group G, the primitive ideal space Prim(G) or the  respectively, is Hausdorff, has attracted the attention of several dual space G, authors. For connected and simply connected nilpotent Lie groups G, using the  is homeomorphic to the coadjoint orbit space of the vector space fact that G dual of the Lie algebra of G and applying induction on the nilpotence length,  forces G to be abelian. In it is not difficult to deduce that Hausdorffness of G the reverse direction, if G is a locally compact group with relatively compact conjugacy classes, then Prim(G) is a Hausdorff space (see [81] and [97]). It is reasonable to conjecture that conversely, if G is any locally compact group  then G has to be a group with relatively compact with Hausdorff dual space G, conjugacy classes. This conjecture has been verified for connected groups by Baggett and Sund [11]. Let G be a two-step nilpotent locally compact group. The setwise parametrization P of Prim(G), as given in Theorem 5.66, was completed in Poguntke [124], slightly extending lemma 2.1 of Kaniuth [82], which in turn substantially built on an idea of Howe [76]. The topology on P was defined in Baggett and Packer [8] and shown there to make the bijection between P and Prim(G) a homeomorphism. The proof of Theorem 5.68 presented here follows [8] and uses extensively continuity of induction and restriction. Another topological description of Prim(G), employing results of Packer and Raeburn [121] on decomposing certain group C ∗ -algebras as the C ∗ -algebras of sections of C ∗ -bundles, was given in Baggett and Packer [9]. Of the examples performed in Section 5.8, Example 5.70 is taken from Kaniuth and Moran [87], whereas Example 5.71 (actually, more generally the (2n + 1)-dimensional integer Heisenberg group Zn+1  Zn ) was treated in [8].

6 Topological Frobenius properties

Let G be a locally compact group and H a closed subgroup of G, and suppose that π and τ are irreducible representations of G and H , respectively. If G is compact, then according to the classical Frobenius reciprocity theorem, the restriction π|H contains τ as a subrepresentation if and only if π is contained in the induced representation indG H τ . In Section 2.9 we have presented a somewhat more general result, but also indicated limitations of further extensions. Such severe limitations led to the replacement of containment of representations by weak containment and the subsequent development of a rich and nuanced theory. The resulting properties are usually referred to as topological or weak Frobenius properties. More precisely, a locally compact group G is said to satisfy property (FP1) if for each closed subgroup H of G and irreducible representations τ and π of H and G, respectively, the conG dition τ ≺ π|H implies that π ≺ indG H τ . If, conversely, π ≺ indH τ entails τ ≺ π |H for all such H , τ , and π , then (FP2) holds for G. If G satisfies both (FP1) and (FP2), then G is said to have the topological Frobenius property (FP). The purpose of this chapter is to investigate (FP), (FP1), (FP2), and even weaker versions of these properties for several classes of locally compact groups. These investigations uncover structural details in the representation theory of the various classes of groups and make heavy use of the tools developed in Chapters 4 and 5. One of the eminent, well-known characterizations of amenability is that indG {e} 1{e} , the regular representation, weakly contains every representation of G (see [78]). In Section 6.1 we present Greenleaf’s important theorem [68] stating that an amenable group G has the even stronger property that π ≺ indG H (π |H )

269

270

Topological Frobenius properties

for any representation π of G and any closed subgroup H of G. This result will be used several times in subsequent sections. In Section 6.2 we introduce all the relevant topological Frobenius properties and show some basic inheritance results. It turns out that validity of either (FP1) or (FP2) is in general fairly restrictive. Consequently, it is necessary to confine to special classes of locally compact groups. We begin with the classical motion groups Rn  SO(n), n ≥ 2, and prove that (FP2) fails for all of them and (FP1) only holds for n = 2 (Section 6.3). The largest class of locally compact groups for which (FP) is known to be true is formed by the groups with relatively compact conjugacy classes. We establish this result for discrete groups in Section 6.4 and also the converse that every conjugacy class in a discrete group with property (FP) is finite. Finally, Section 6.5 is devoted to nilpotent groups. To mention just one sample of the results: (FP) holds for a connected nilpotent Lie group G if and only if G modulo its maximal compact normal subgroup is abelian. Our intention in this chapter is not to present the most general results in all cases. Instead, we aim at providing an introduction to the subject and have tried to find a good balance between generality and the degree of difficulty of proofs. Extensions and additional special contributions are outlined in Section 6.6.

6.1 Amenability and induced representations The purpose of this section is to establish Greenleaf’s celebrated theorem, which asserts that if G is an amenable locally compact group and H is any closed subgroup of G, then π ≺ indG H (π |H ) for every unitary representation π of G. This result will play a fundamental role when studying topological Frobenius properties in this chapter. As in Section 1.3, let ν be a quasi-invariant measure on the space G/H of left cosets of H in G. Let σ : G × G/H → [0, ∞) denote the function defined by σ (x −1 , ω) =

dνx (ω), x ∈ G, ω ∈ G/H, dν

x where dν denotes the Radon–Nikodym derivative (compare Section 1.3). We dν need to define actions of L1 (G) and G on both L1 (G/H, ν) and L∞ (G/H, ν) = L1 (G/H, ν)∗ . To start with, for μ ∈ M(G) and ϕ ∈ L1 (G/H, ν) define the measure μ ∗ ϕ on G/H by   ψ(x · ω)ϕ(ω)dν(ω)dμ(x), ψ ∈ C0 (G/H ). μ ∗ ϕ, ψ =

G

G/H

6.1 Amenability and induced representations

271

Then, as is easily checked, (μ1 ∗ μ2 ) ∗ ϕ = μ1 ∗ (μ2 ∗ ϕ) for μ1 , μ2 ∈ M(G). The action of M(G) on L∞ (G/H, ν) is now defined by μ ∗ ϕ, ψ = ϕ, μ ∗ ψ for μ ∈ M(G), ϕ ∈ L∞ (G/H, ν), and ψ ∈ L1 (G/H, ν). Basic properties of these actions are given in the next three lemmas. Lemma 6.1 For ϕ ∈ L1 (G/H, ν) and x ∈ G we have (δx ∗ ϕ)(ω) = σ (x −1 , ω)ϕ(x −1 · ω) ν-almost everywhere. Proof If ψ ∈ Cc (G/H ), then   δx ∗ ϕ, ψ = ψ(y · ω)ϕ(ω)δx (y)dν(ω) G/H G  = ψ(x · ω)ϕ(ω)dν(ω) G/H  ψ(ω)σ (x −1 , ω)ϕ(x −1 · ω)dν(ω), = G/H

which implies that δx ∗ ϕ(ω) = σ (x −1 , ω)ϕ(x −1 · ω) locally ν-almost everywhere. However, if two ν-integrable functions coincide locally ν-almost everywhere, then they coincide ν-almost everywhere.  Lemma 6.2 Let f ∈ L1 (G) and ϕ ∈ L1 (G/H, ν). Then  (i) (f ∗ ϕ)(ω) = G f (x)ϕ(x −1 · ω)σ (x −1 , ω)dx for ν-almost all ω ∈ G/H . (ii) f ∗ ϕ 1,ν ≤ f 1 ϕ 1,ν . Proof (i) For any ψ ∈ Cc (G/H ) we have   f ∗ ϕ, ψ = ψ(x · ω)f (x)ϕ(ω)dν(ω) dx G G/H   = ψ(ω)f (x)ϕ(x −1 · ω)σ (x −1 , ω)dν(ω) dx G G/H   ψ(ω)f (x)ϕ(x −1 · ω)σ (x −1 , ω)dx dν(ω) = G/H G ; < = f (x)ϕ(x −1 · ω)σ (x −1 , ω)dx, ψ . G

Since this holds for all ψ ∈ Cc (G/H ), it follows that  f (x)ϕ(x −1 · ω)σ (x −1 , ω)dx f ∗ ϕ(ω) = G

272

Topological Frobenius properties

locally ν-almost everywhere. Because both functions are in L1 (G/H, ν), (i) follows. (ii) Since σ (ab, ω) = σ (a, b · ω)σ (b, ω) and σ (e, ω) = 1 for all a, b ∈ G and ω ∈ G/H , (i) yields       f (x)ϕ(x −1 · ω)σ (x −1 , ω)dx  dν(ω)

f ∗ ϕ 1,ν =   G/H G   ≤ |f (x)ϕ(ω)|σ (x −1 , x · ω)σ (x, ω)dν(ω) dx G G/H   = |f (x)ϕ(ω)|σ (e, ω)dν(ω) dx = f 1 ϕ 1,ν . G

G/H

So (ii) holds as well.



Lemma 6.3 For ϕ ∈ L1 (G/H, ν), the maps f → f ∗ ϕ from L1 (G) into L1 (G/H, ν) and x → δx ∗ ϕ from G into L1 (G/H, ν) are both continuous. Proof The first statement follows from f ∗ ϕ 1,ν ≤ f 1 ϕ 1,ν for f ∈ L1 (G) (Lemma 6.2). Since L1 (G/H, ν) is a Banach L1 (G)-module and L1 (G) has a bounded approximate identity, ϕ factors as ϕ = g ∗ ψ for some g ∈ L1 (G) and ψ ∈ L1 (G/H, ν) [74, (32.50)]. Then δx ∗ ϕ = (δx ∗ g) ∗ ψ for all x ∈ G. Since the map x → δx ∗ g from G into L1 (G) is continuous, the first assertion  implies that x → δx ∗ ϕ is continuous as well. We now introduce several notions which are relevant for the remainder of this section. Definition 6.4 Let G be a locally compact group, H a closed subgroup of G, and ν a quasi-invariant measure on G/H . A mean on L∞ (G/H, ν) is a linear functional m satisfying m(ϕ) = m(ϕ) for all ϕ ∈ L∞ (G/H, ν), m(ϕ) ≥ 0 whenever ϕ ≥ 0 locally ν-almost everywhere and m(1G/H ) = 1. Such a mean is called left invariant if m(δx ∗ ϕ) = m(ϕ) for all x ∈ G and ϕ ∈ L∞ (G/H, ν), and it is said to be topologically left invariant if m(f ∗ ϕ) = m(ϕ) for all ϕ ∈ L∞ (G/H, ν) and all f ∈ Q(G), where Q(G) = {f ∈ L1 (G) : f ≥ 0, f 1 = 1}. The group G is called amenable if there exists a left-invariant mean on L∞ (G) itself. Definition 6.5 Let G, H , and ν be as in Definition 6.4 and let Q(ν) = {ϕ ∈ L1 (G/H, ν) : ϕ ≥ 0, ϕ 1,ν = 1}.

6.1 Amenability and induced representations

273

A net (ϕα )α in Q(ν) converges weakly to topological left-invariance if f ∗ ϕα − ϕα → 0 for each f ∈ Q(G) in the topology σ (L1 (G/H, ν), L∞ (G/H, ν)), and it converges strongly to topological left invariance if f ∗ ϕα − ϕα → 0 in the norm topology of L1 (G/H, ν), for all f ∈ Q(G). Lemma 6.6 Suppose that there exists a topologically left-invariant mean on L∞ (G/H, ν). Then there exists a net in Q(ν) which is strongly convergent to topological left invariance. Proof Let L∞ (ν) = L∞ (G/H, ν) and L1 (ν) = L1 (G/H, ν), and let m be a topologically left-invariant mean on L∞ (ν). Notice that Q(ν) ⊆ L∞ (ν)∗ is σ (L∞ (ν)∗ , L∞ (ν))-dense in the set of all means on L∞ (ν) and choose a net (ϕα )α in Q(ν) such that ϕα → m in the topology σ (L∞ (ν)∗ , L∞ (ν)). To produce a net in Q(ν) which is strongly convergent to topological left invariance requires only straightforward modifications of the proof in the case H = {e}. For each g ∈ Q(G), take a copy L1 (ν)g of L1 (ν), form the product space  E = {L1 (ν)g : g ∈ Q(G)} equipped with the product of the norm topologies, and define a linear map T : L1 (ν) → E by T ϕ(g) = g ∗ ϕ − ϕ. Then proceed as in the proof of Greenleaf [67, theorem 2.4.2].  The following two propositions are generalizations of the corresponding results in the classical case H = {e} (compare theorems 3.2.1 and 2.4.3 of [67]) and they form the main steps toward the proof of Greenleaf’s theorem (Theorem 6.9 below). Proposition 6.7 Let H be a closed subgroup of a locally compact group G and let ν be a quasi-invariant measure on G/H . Suppose that there is a net in Q(ν) which is strongly convergent to topological left invariance. Then, given a compact subset C of G and  > 0, there exists ϕ ∈ Q(ν) such that 

δx ∗ ϕ − ϕ 1,ν = |ϕ(x −1 · ω)σ (x −1 , ω) − ϕ(ω)|dν(ω) ≤  G/H

for all x ∈ C. In particular, the conclusion holds if G is amenable. Proof Let g be a fixed function in Q(G). There exists a compact neighborhood U of e in G such that

|U |−1 1U ∗ g − g 1 ≤  and δt ∗ g − g 1 ≤ 

(6.1)

for all t ∈ U . Select x1 , . . . , xn ∈ G with C ⊆ ∪nj=1 xj U and set h0 = |U |−1 1U and hj = |xj U |−1 1xj U = δxj ∗ h0 ∈ Q(G), 1 ≤ j ≤ n.

274

Topological Frobenius properties

Let (ϕα )α be a net in Q(ν) converging strongly to topological left invariance, and let η = /5. Then there exists an index α such that

g ∗ ϕα − ϕα 1,ν ≤ η

and

hj ∗ ϕα − ϕα 1,ν ≤ η, for 1 ≤ j ≤ n. (6.2)

We assert that ϕ = g ∗ ϕα ∈ Q(ν) serves as the required element, that is, satisfies

δx ∗ ϕ − ϕ 1,ν ≤ 5η for all x ∈ C. By (6.1) we have for all t ∈ U ,

h0 ∗ ϕ − δt ∗ ϕ 1,ν ≤ h0 ∗ ϕ − ϕ 1,ν + ϕ − δt ∗ ϕ 1,ν ≤ h0 ∗ g − g 1 · ϕα 1,ν + g − δt ∗ g 1 · ϕα 1,ν ≤ 2η. Let x = xj t with t ∈ U , 1 ≤ j ≤ n. It follows that     

hj ∗ ϕ − δx ∗ ϕ 1,ν =  hj (y)(δy ∗ ϕ)dy − δx ∗ ϕ   G 1,ν       = δ ∗ h (y)(δ ∗ ϕ)dy − δ ∗ ϕ 0 y t  xj  G

1,ν

= δxj ∗ (h0 ∗ ϕ − δt ∗ ϕ) 1,ν = h0 ∗ ϕ − δt ∗ ϕ 1,ν ≤ 2η. This in turn implies, using (6.1) and (6.2),

δx ∗ ϕ − ϕ 1,ν ≤ 2η + hj ∗ ϕ − ϕ 1,ν = 2η + hj ∗ (g ∗ ϕα ) − (g ∗ ϕα ) 1,ν ≤ 2η + hj ∗ g ∗ ϕα − hj ∗ ϕα 1,ν + hj ∗ ϕα − ϕα 1,ν + ϕα − g ∗ ϕα 1,ν ≤ 4η + hj 1 · g ∗ ϕα − ϕα 1,ν ≤ 5η. Since this holds for every x ∈ C, the proof is complete.



Proposition 6.8 Let G be an amenable locally compact group, H a closed subgroup of G, and ν a quasi-invariant measure on G/H . Then there exists a topologically left-invariant mean on L∞ (G/H, ν).  Proof Let m be a topologically left-invariant mean on L∞ (G). We define m b on the space C (G/H ) of bounded continuous functions on G/H by m (g) = m(g ◦ q), where q : G → G/H is the quotient map. Clearly, m  is a mean on C b (G/H ), and it is topologically left invariant since m (f ∗ g) = m((f ∗ g) ◦ q) = m(f ∗ (g ◦ q)) = m(g ◦ q) = m (g)

6.1 Amenability and induced representations

275

for all g ∈ C b (G/H ) and f ∈ Q(G). It remains to show that m  extends to a topologically left-invariant mean on L∞ (G/H, ν). Fix a compact neighborhood U of eH in G/H and let ϕU = ν(U )−1 1U . Let ∞ Lc (G/H, ν) be the linear space of all essentially bounded Borel measurable functions on G/H (not identified modulo equivalence, that is, modulo coincidence locally ν-almost everwhere). With each g ∈ L∞ c (G/H ), we associate a function T (g) on G by setting  g(x · ω)ϕU (ω)dν(ω), x ∈ G. T (g)(x) = G/H

Then |T (g)(x)| ≤ g ∞ and, for any s, t ∈ G,     |T (g)(s) − T (g)(t)| =  (g(s · ω) − g(t · ω))ϕU (ω)dν(ω) G/H  ≤ |g(ω)|·|ϕU (s ·ω)σ (s, ω) − ϕU (t ·ω)σ (t, ω)| dν(ω) G/H

≤ g ∞ δs ∗ ϕU − δt ∗ ϕU 1,ν , where we have used that (δx ∗ ψ)(ω) = ψ(x −1 · ω)σ (x, ω) for x ∈ G and ψ ∈ L1 (G/H, ν) (Lemma 6.1). Since the map x → δx ∗ ψ from G into L1 (G/H, ν) is continuous (Lemma 6.3), it follows that T (g) is continuous. Therefore T : b g → T (g) defines a linear map from L∞ c (G/H ) into C (G). Now, if g = 0 locally ν-almost everywhere, then Lx g = 0 locally ν-almost everywhere, and since ϕU has compact support, it follows that T (g) = 0. Moreover, T (1G/H ) = 1G and g ≥ 0 implies that T (g) ≥ 0. Recall that we have a ∞ canonical linear map jν : L∞ c (G/H ) → L (G/H, ν) which identifies functions differing only locally on ν-null sets. Since T (g) = 0 whenever g ∈ ker jν , T factors through jν and then gives rise to a linear, order preserving map T : L∞ (G/H, ν) → C b (G). We show next that T commutes with the action of L1 (G) on L∞ (G/H, ν) and on C b (G). If f ∈ L1 (G) and g ∈ L∞ c (G/H ), then for each x ∈ G,  (f ∗ T (g))(x) = f (y)T (g)(y −1 x)dy G   = f (y) g((y −1 x) · ω)ϕU (ω)dν(ω)dy G G/H   ϕU (ω) f (y)g(y −1 · (x · ω))dydν(ω) = G/H G  ϕU (ω)(f ∗ g)(x · ω)dν(ω) = T (f ∗ g)(x). = G/H

276

Topological Frobenius properties

It follows that T has the same property, and hence commutes with the action of L1 (G) on L∞ (G/H, ν). We conclude that the topological left-invariant mean on C b (G) lifts back via T to such a mean on L∞ (G/H, ν).  Theorem 6.9 Let G be an amenable locally compact group and H a closed subgroup of G. Then, for any unitary representation π of G, π ≺ indG H (π|H ). Proof It suffices to show that 1G ≺ indG H 1H . Indeed, it then follows from Theorem 2.58 and since weak containment is preserved by forming tensor products (Proposition 5.14), that G G π = π ⊗ 1G ≺ π ⊗ indG H 1H = indH (π|H ⊗ 1H ) = indH (π |H ).

To prove that 1G ≺ indG H 1H , given a compact subset C of G and  > 0, we have to find a positive definite function ψ associated with indG H 1H such that |ψ(x) − 1| ≤  for all x ∈ C. Let ν be a quasi-invariant measure on G/H . Since G is amenable, by Proposition 6.8 there exists a topologically left-invariant mean on L∞ (G/H, ν). Then, by Lemma 6.6, there exists a net in Q(ν) which is strongly convergent to topological left invariance. Finally, Proposition 6.7 applies and yields the existence of some ϕ ∈ L1 (G/H, ν) such that ϕ ≥ 0, ϕ 1,ν = 1, and  |ϕ(x · ω) − ϕ(ω)|dν(ω) ≤  2 G/H

for all x ∈ C. Let ξ (x) = ϕ(xH )1/2 for x ∈ G. Then   2 |ξ (x)| dν(xH ) = ϕ(xH )dν(xH ) = 1, G/H

so that ξ ∈

H(indG H

G/H

1H ). Since, by Lemma 6.1, δx ∗ ϕ(ω) = σ (x −1 , ω)ϕ(x −1 · ω)

for all x ∈ G and ω ∈ G/H , it follows that  G indG indH 1H (x)ξ, ξ  = H 1H (x)ξ (y), ξ (y)dν(yH ) G/H  = σ (x −1 , yH )1/2 ξ (x −1 y), ξ (y)dν(yH ) G/H  σ (x −1 , ω)1/2 ϕ(x −1 · ω)1/2 ϕ(ω)1/2 dν(ω) = G/H  (δx ∗ ϕ)1/2 (ω)ϕ 1/2 (ω)dν(ω) = G/H

6.2 Basic definitions and inheritance properties

277

for all x ∈ G. Hence, using that |α − β|2 ≤ |α 2 − β 2 | for α, β ≥ 0, 1/2 |indG − ϕ 1/2 )ϕ 1/2 1,ν H 1H (x)ξ, ξ  − 1| = ((δx ∗ ϕ)

≤ (δx ∗ ϕ)1/2 − ϕ 1/2 2,ν ϕ 1/2 2,ν = (δx ∗ ϕ)1/2 − ϕ 1/2 2,ν  1/2 ≤ |δx ∗ ϕ(ω) − ϕ(ω)|dν(ω) 

G/H

=

1/2 |ϕ(x · ω) − ϕ(ω)|dν(ω)

,

G/H

which is ≤  for all x ∈ C. This proves that 1G ≺ indG H 1H , as required.



6.2 Basic definitions and inheritance properties We start the discussion of the topological or weak Frobenius properties by introducing the various relevant versions. Definition 6.10 Let G be a locally compact group and H a closed subgroup of G. The pair (G, H ) is said to satisfy the topological Frobenius property (FP1)  and τ ∈ H , the condition that τ ≺ π |H (respectively, (FP2)) if for any π ∈ G G implies that π ≺ indG τ (respectively, π ≺ ind H H τ implies that τ ≺ π|H ). We say that G has property (FP1) and (FP2) if for every closed subgroup H of G, the pair (G, H ) has the corresponding property. Definition 6.11 A locally compact group G has the topological Frobenius property (FP) if both (FP1) and (FP2) hold for G. Moreover, G is said to have the restricted topological Frobenius property (RFP) if for each closed subgroup , 1G ≺ indG H of G and σ ∈ H H σ is equivalent to σ = 1H . Definition 6.12 Let G be a locally compact group. Then G satisfies the weak Frobenius property (WF1) (respectively, (WF2)) if for every closed sub  π ≺ indG group H of G and π ∈ G, H (π|H ) (respectively, for every τ ∈ H , G τ ≺ (indH τ )|H ). At this point, the following remarks are in order. Remark 6.13 (1) By Greenleaf’s theorem (Theorem 6.9), (WF1) holds for an arbitrary locally compact group G if and only if G is amenable. (2) Suppose that (FP1) holds for G. Then G satisfies (WF1). To see this, let H be any closed subgroup of G and π an irreducible representation of G. Choose G G any τ ∈ supp(π |H ). Then indG H τ ≺ indH (π|H ) and, by (FP1), π ≺ indH τ . In

278

Topological Frobenius properties

particular, taking H = {e} and π = 1G , 1G ≺ indG {e} 1{e} = λG . This implies that G is amenable. In Section 6.1 we have shown that conversely, amenability of G implies that π ≺ indG H (π|H ) for every closed subgroup H of G and π ∈  G. (3) Property (FP) implies property (RFP). In fact, if H is a closed subgroup  is such that 1G ≺ indG of G and σ ∈ H H σ , then σ ≺ 1G |H = 1H by (FP2) and hence σ = 1H . On the other hand, 1G ≺ indG H 1H by (1). It will turn out later in this chapter that actually (RFP) and (FP) are equivalent for several classes of locally compact groups. The advantage of exploiting (RFP) rather than (FP) is that (RFP) is inherited by subgroups, whereas it seems to be unknown whether the same is true of (FP). When investigating the impact a certain property may have on the structure of a locally compact group, it is always helpful to know that the property in question is inherited by subgroups and by quotient groups. The following lemma ensures that this is true of (RFP). Lemma 6.14 If G has property (RFP), then so does every closed subgroup and every quotient group of G. Proof Since closed subgroups and quotient groups of amenable groups are amenable, in both cases we only have to verify the “only if” assertion in Definition 6.11. Suppose first that K and H are closed subgroups of G with K ⊆ H and let  such that 1H ≺ indK σ ∈K H σ . Then  H  G G 1G ≺ indG H 1H ≺ indH indK σ = indK σ and hence σ = 1K since (RFP) holds for G. So (RFP) holds for H . Secondly, let N be a normal subgroup of G and let q : G → G/N denote the  quotient homomorphism. Let H be a closed subgroup of G/N and let σ ∈ H G/N  −1 such that 1G/N ≺ indH σ . Then σ ◦ q ∈ q (H ) and   G/N indG (σ ◦ q) = ind σ ◦ q ! 1G/N ◦ q = 1G , −1 H q (H ) and therefore σ ◦ q = 1q −1 (H ) by hypothesis. Thus σ = 1H .



Let H be a closed subgroup of an arbitrary locally compact group G and let K be a compact normal subgroup of G. Let σ be a representation of H such that σ (H ∩ K) = {I }. Then σ˙ (xK) = σ (x) for x ∈ H defines a representation σ˙ of the closed subgroup H K/K of G/K. If σ is irreducible, then so is σ˙ . The following proposition turns out to be crucial and will be used several times in this chapter.

6.2 Basic definitions and inheritance properties

279

Proposition 6.15 Let H be a closed subgroup and K a compact normal sub and τ ∈ H  be such that group of the locally compact group G. Let π ∈ G π (K) = {I } and π ≺ indG τ . Then τ (H ∩ K) = {I }, and if representations π˙ H G/K of G/K and τ˙ of H K/K are defined as above, then π˙ ≺ indH K/K τ˙ . Proof To establish the proposition, we employ the realization of indG H τ in terms 1 of positive definite measures (Section 2.4). Let ϕ ∈ P (H ) such that τ = πϕ , the representation of H associated with ϕ through the GNS-construction. Let the Haar measure of K be normalized and for f ∈ Cc (G), define f K ∈ Cc (G) by  f K (x) = f (xk)dk, x ∈ G. K

Since f (xky) = f (xy) for all x, y ∈ G and k ∈ K, for every h ∈ H we have  K ∗ K f K (yh−1 ) f K (x −1 y −1 )dy [(f ) ∗ Lx (f )](h) = G   = f (k −1 yh−1 )f (x −1 y)dkdy G K = f (yh−1 )f (kx −1 y)dydk K G  = (f ∗ ∗ Lxk−1 f )(h)dk. K

K

K

Using the notation of Section 2.4 and denoting by νϕ the positive definite measure on G associated with ϕ, the preceding equation implies    K ∗ K νϕ ((f ) ∗ Lx (f )) = δ(h−1 )ϕ(h) (f K )∗ ∗ Lx (f K ) (h)dh  H δ(h−1 )ϕ(h) (f ∗ ∗ Lxk−1 f )(h)dkdh = K H = νϕ (f ∗ ∗ Lxk −1 f )dk. K

This in turn yields  −1 G K K indG H τ (xk )[f ]ϕ , [f ]ϕ dk = indH τ (x)[f ]ϕ , [f ]ϕ .

(6.3)

K

Now let a compact subset C of G/K and  > 0 be given. Then, since G π ≺ indG H τ and [Cc (G)]ϕ is dense in H(indH τ ), there exist f1 , . . . , fn ∈ Cc (G) such that     n    G −1 ϕ(xk −1 ) − indH τ (xk )[fj ]ϕ , [fj ]ϕ  ≤     j =1

280

Topological Frobenius properties

for all x ∈ q −1 (C) and k ∈ K. Integrating over K and using that ϕ(xk −1 ) = ϕ(x) for all x ∈ G and k ∈ K as well as equation (6.3), we get     n    G K K ϕ(x) − indH τ (x)[fj ]ϕ , [fj ]ϕ  ≤  (6.4)    j =1 for all x ∈ q −1 (C). This inequality in particular ensures the existence of some f ∈ Cc (G) such that [f K ]ϕ = 0, so of an f ∈ Cc (G) with [f K ]ϕ = 1. Recall that T = δ · ((f K )∗ ∗ (f K ))|H is a positive element of C ∗ (H ). Then, for each k ∈ H ∩ K, using again that f K (xky) = f K (xy) for all x, y ∈ G,  δ(h)((f K )∗ ∗ (f K ))(h)τ (kh)dh τ (k)τ (T ) = H = δ(k −1 h)((f K )∗ ∗ (f K ))(h)τ (h)dh H  = δ(h)((f K )∗ ∗ (f K ))(h)τ (h)dh = τ (T ) = τ (T )τ (k). H

Thus τ (T ), and hence also τ (T 1/2 ), commutes with τ (k) for all k ∈ H ∩ K. Let ξϕ be a cyclic vector for τ such that ϕ(h) = τ (h)ξϕ , ξϕ  for all h ∈ H . Because ϕ|H ∩K = 1, for arbitrary k ∈ H ∩ K, we then have τ (k)ξϕ = ξϕ and therefore τ (k)τ (T 1/2 )ξϕ , ξϕ  = τ (T )ξϕ , ξϕ   δ(h)((f K )∗ ∗ f K )(h)ϕ(h)dh = [f K ]ϕ = 1. = H

This implies that τ (k)τ (T

1/2

)ξϕ − τ (T 1/2 )ξϕ 2 = 0 and hence

τ (k)τ (h)τ (T 1/2 )ξϕ = τ (h)τ (h−1 kh)τ (T 1/2 )ξϕ = τ (h)τ (T 1/2 )ξϕ for all h ∈ H . Since τ is irreducible and τ (T 1/2 )ξϕ = 0, it follows that τ (k) = I for all k ∈ H ∩ K. This proves the first statement of the proposition. G/K To show that π˙ ≺ indH K/K τ˙ , note that ϕ|H ∩K = 1 and define ϕ˙ ∈ 1 ˙ = ϕ(h), h ∈ H. For g ∈ Cc (G), let g # ∈ Cc (G/K) be P (H K/K) by ϕ(hK) defined as in Chapter 1, i.e., g # (xK) = K g(xk)dk, x ∈ G. Then, for all h ∈ H ,  K ∗ K g K (y −1 h−1 ) g K (x −1 y −1 )dy [(g ) ∗ Lx (g )](h) = G  = g # ((hy)−1 K) g # ((yx)−1 K)d(yK) G/K  (g # )∗ (hyK)LxK (g # )(y −1 K)d(yK) = G/K

= [(g # )∗ ∗ LxK (g # )](hK).

6.2 Basic definitions and inheritance properties

281

Now, let Haar measures on G/K and H K/K be chosen so that δ(hk) = δ(h) for all h ∈ H and k ∈ K. It then follows from the preceding equation that G/K

K K indH K/K τ˙ (xK)[g # ]ϕ˙ , [g # ]ϕ˙  = indG H τ (x)[g ]ϕ , [g ]ϕ 

for all g ∈ Cc (G) and x ∈ G. Finally, this formula together with inequality G/K (6.4) above yields π˙ ≺ indH K/K τ˙ .  Proposition 6.15, when applied to quotients modulo compact normal subgroups, yields some useful inheritance results of properties (RFP), (FP1), and (FP2), as pointed out in the next two corollaries. Corollary 6.16 Let G be a locally compact group and K a compact normal subgroup of G. If G/K has property (RFP), then so does G. Proof Note first that G is amenable since both G/K and K are amenable. Therefore, it only remains to observe that if H is a closed subgroup of  is such that 1G ≺ indG G and τ ∈ H H τ , then τ = 1H . By Proposition 6.15, G/K τ (H ∩ K) = {I } and with τ˙ as in Proposition 6.15, 1G/K ≺ indH K/K τ˙ . Since (RFP) holds for G/K, it follows that τ˙ = 1H K/K and this of course implies that τ = 1H .  Corollary 6.17 Suppose that G is a projective limit of groups Gα = G/Kα , α ∈ A. If all Gα satisfy (FP1) (respectively, (FP2)), then the same is true of G. , and π ∈ G.  Since H is Proof Let H be a closed subgroup of G, τ ∈ H the projective limit of the groups H /H ∩ Kα , by Moore [111, proposition 2.3] there exists α such that π(Kα ) = {I } and τ (H ∩ Kα ) = {I }. Let K = Kα and let q : G → G/K denote the quotient homomorphism. Moreover, define  and τ˙ ∈ H π˙ ∈ G/K K/K as in Proposition 6.15. ˙ H K/K Assume first that τ ≺ π|H and that (FP1) holds for G/K. Then τ˙ ≺ π| G/K and hence π˙ ≺ indH K/K τ˙ . This implies   G/K π = π˙ ◦ q ≺ indH K/K τ˙ ◦ q = indG (6.5) H K (τ˙ ◦ q). On the other hand, since (FP1) holds for G/K, G/K is amenable and hence so is G. So H K is amenable and therefore K HK τ˙ ◦ q ≺ indH H ((τ˙ ◦ q)|H ) = indH τ

by Theorem 6.9. Combining (6.5) and (6.6) gives HK G π ≺ indG H K (indH τ ) = indH τ,

as required.

(6.6)

282

Topological Frobenius properties

Now suppose that π ≺ indG H τ and that (FP2) holds for G/K. Then ProposiG/K tion 6.15 yields that π˙ ≺ indH K/K τ˙ . Thus τ˙ ≺ π˙ H K/K by (FP2) and hence τ˙ ◦ q ≺ π˙ |H K/K ◦ q = (π˙ ◦ q)|H K = π|H K and therefore τ = (τ˙ ◦ q)|H ≺ (π |H K )|H = π |H .



We conclude this section by giving very useful sufficient conditions in terms of tensor products of representations for a pair (G, H ) to satisfy (FP1). Lemma 6.18 Let H be a closed subgroup of the amenable locally compact group G and suppose that the following two conditions are satisfied. . (1) 1H ≺ τ ⊗ τ for every τ ∈ H  then 1G ≺ π ⊗ ρ (2) If ρ is an arbitrary representation of G and π ∈ G, implies that π ≺ ρ. Then (FP1) holds for the pair (G, H ).  and τ ∈ H  be such that τ ≺ π|H . Then 1H ≺ τ ⊗ τ by (1), Proof Let π ∈ G and hence 1H ≺ τ ⊗ π|H = τ ⊗ π|H . Since G is amenable, it follows that G G 1G ≺ indG H 1H ≺ indH (τ ⊗ π|H ) = π ⊗ indH τ

(Theorem 6.9). By (2), this gives π ≺ indG H τ.



It should be pointed out that in the preceding lemma, condition (1) can be omitted. In fact, it is satisfied by any amenable locally compact group, by a remarkable result due to Bekka [16], which we feel is beyond the scope of this book.

6.3 Motion groups The first class of specific examples for which the topological Frobenius properties will be studied in this chapter is formed by the classical motion groups, that is, semidirect products Gn = Rn  SO(n), where SO(n) acts on Rn by rotation. It turns out that none of the groups Gn satisfies (RFP), and hence also not (FP2). Concerning (FP1), we show that this Frobenius property holds for Gn precisely when n = 2. Proposition 6.19 Let G = R2  SO(2). Then 1G ≺ indG SO(2) χ for every character χ of SO(2). In particular, (RFP) does not hold for G. Proof Let K = SO(2), normalize the Haar measure on K, and identify K and 2 2 R2 with the corresponding subgroups of G. Realizing indG K χ on L (R ), we

6.3 Motion groups

283

have indG K χ(x, a)f (y) = χ(a)f (a · y + x), for f ∈ L2 (R2 ), x, y ∈ R2 , a ∈ K. For f ∈ L2 (R2 ), let cf denote the associated coordinate function cf (g) = indG K χ(g)f, f , g ∈ G, of indG K χ. Fix a neighborhood U of e in K such that Reχ(a) ≥ 0 for all a ∈ U . Then there exists a nonempty open subset V of R2 with the property that a · V ∩ V = ∅ for every a ∈ K \ U . Indeed, one can take for V an open segment V = {r(cos ϕ, sin ϕ) : r > 0, 0 < ϕ < δ} for sufficiently small δ. Choose h ∈ Cc+ (R2 ), h = 0, with supp h ⊆ V . Then ch (e) = h 22 > 0 and, by the choice of V and h,  Re ch (a) = Re χ(a) h(a −1 · y)h(y)dy ≥ 0 R2

for each a ∈ K. Of course, we can assume that K → L2 (R2 ),

 K

Re ch (k)dk = 1. The map

a → indG K χ(a)h

is continuous. Define a function f on R2 by  f (x) = (indG K χ(k)h)(x)dk, K

for x ∈ R . Then f is continuous and, for a ∈ K,  cf (a) = χ (a) f (a · x)f (x)dx 2 R      G  = χ (a) indA χ(k1 )h (a · x) indG A χ(k2 )h (x)dxdk1 dk2 2 R K K = χ (a) χ(k1 k2−1 )h(k1 · x)h(k2 · x)dk1 dk2 dx R2 K K    = χ(k1 ak2−1 )h((k1 ak2−1 ) · x)h(x)dxdk1 dk2 K K R2   −1 = indG K χ(k1 ak2 )h, hdk1 dk2 K K  = ch (k1 ak2−1 )dk1 dk2 = ch (k)dk. 2

K

K

K

284

Topological Frobenius properties

 Notice next that K ch (k)dk is real. Indeed, this follows from the facts that the representation (indG K χ)|K decomposes into a direct sum of characters of K and  that K ω(k)dk = 0 whenever ω is a nontrivial character of K. Therefore we have cf (a) =  K Re ch (k)dk = 1 for all a ∈ K. This in particular shows that 1K ≺ indG K χ |K . 2 To verify that actually indG K χ ! 1G , let  > 0 and a compact subset C of R 2 2 be given. Then, as shown above, there exists f ∈ L (R ) such that cf (a) = 1 for all a ∈ K. For t > 0, define now a function ft on R2 by ft (y) = t 1/2 f (ty), y ∈ R2 . Then ft ∈ L2 (R2 ) and  ft (a · y + x)ft (y)dy cft (x, a) = χ(a) 2 R tf (a · (ty) + tx)f (ty)dy = χ(a) 2 R = χ(a) f (a · y + tx)f (y)dy = cf (tx, a) R2

for all a ∈ K and x ∈ R2 . Since the coordinate function cf is uniformly continuous, it follows that lim cft (x, a) = lim cf (tx, a) = cf (a)

t→0

t→0

uniformly on C × K. Consequently, for t sufficiently small, |cft (x, a) − 1| ≤  for all x ∈ C and all a ∈ K. Thus 1G ≺ indG K χ, and taking any χ = 1K , it follows that (RFP) fails to hold for G.  Using Proposition 6.19, we can now quickly draw the same conclusion for the higher-dimensional motion groups. Theorem 6.20 The Frobenius property (RFP), and hence (FP2), fails for all motion groups Gn = Rn  SO(n), n ≥ 2. Proof Of course, G2 = R2  SO(2) can be considered as a subgroup of Gn by identifying R2 with the subspace V = {x = (x1 , . . . , xn ) ∈ Rn : xi = 0 for i ≥ 3} of Rn and SO(2) with the subgroup consisting of all A ∈ SO(n) such that A(x) = x for all x ∈ V ⊥ . By Proposition 6.19, for every character χ of SO(2), 2 1G2 ≺ indG SO(2) χ , and hence, since Gn is amenable,   Gn Gn G2 n 1Gn ≺ indG G2 1G2 ≺ indG2 indSO(2) χ = indSO(2) χ . Thus (RFP) (and consequently, (FP2)) does not hold for Gn .



6.3 Motion groups

285

We now proceed to show that the motion group of the plane satisfies (FP1). Example 6.21 Let K = SO(2) and G = R2  K. Recall from Section 4.5 that =K  ∪ {πt : 0 < t < ∞}, G where πt is the representation induced by the character χt of R2 , where χt (x) = e2πix1 t , x = (x1 , x2 ) ∈ R2 . To prove that (FP1) holds for G, we verify conditions (1) and (2) of Lemma 6.18. . Since the groups H /H ∩ R2 and Let H be a closed subgroup of G and τ ∈ H 2 2 H R /R are algebraically isomorphic, either H /H ∩ R2 is finite or H R2 /R2 is dense in SO(2). In the first case, every irreducible representation of H is finite-dimensional and hence τ ⊗ τ contains 1H as a subrepresentation. So suppose that H R2 /R2 is dense in SO(2) and hence H R2 = R2  D for some dense subgroup D of SO(2). Then D normalizes H ∩ R2 , and since D is dense in SO(2), it follows that H ∩ R2 is normal in G. This in turn implies that either H ∩ R2 = {0} or H ⊇ R2 . If H ∩ R2 = {0}, then H is abelian and hence τ ⊗ τ = 1H . So we are left with the case R2 ⊆ H . Then H = G since H /R2 is dense in SO(2). Therefore we only have to verify condition (1) for G itself, and then it suffices to consider τ = πt = indG R2 χt , t = 0. But then  G   G 2 πt ⊗ πt = indG indR2 χt |R2 ⊗ χ−t ! indG R2 R2 (χt ⊗ χ−t ) = indR2 1R ! 1G . So condition (1) holds for G. For (2), let π be an irreducible and ρ an arbitrary representation of G such that 1G ≺ π ⊗ ρ. Suppose first that π = πt for some t. Then   G 2 1G ≺ indG R2 χt ⊗ ρ = indR2 (χt ⊗ ρ|R ) and therefore 1R2 ≺ K(χt ) ⊗ ρ|R2 . Since the set of all characters (a · χt )χ, 2 and contains 1R2 , it follows where a ∈ K and χ ∈ supp(ρ|R2 ), is closed in R that K(χt ) ∩ supp(ρ|R2 ) = ∅ and hence χt ≺ ρ|R2 . So there exists a sequence (πn )n in supp ρ such that πn |R2 → χt . Then necessarily, πn = πtn for some 0 < tn < ∞ and tn → t. This implies that πn → indG R2 χt = π, as required.  If there exists a sequence (πtn )n in supp ρ with tn → 0, then Now let π ∈ K. G  2 πtn = indG R2 χtn → indR2 1R ∼ K,

whence π ≺ ρ. If no such sequence exists, then supp ρ is a disjoint union  ∪ S, where S is closed in G.  It follows that 1K ≺ supp ρ = (supp ρ ∩ K)  π ⊗ (supp ρ ∩ K), and this in turn implies that π ≺ ρ. Lemma 6.18 now implies that G has the Frobenius property (FP1).

286

Topological Frobenius properties

In order to show that (FP1) does not hold for Gn = Rn  SO(n) whenever n ≥ 3, we need the following lemma. We remind the reader that K(F ) denotes the collection of closed subgroups of a locally compact group F and that K(F ) is equipped with Fell’s compact–open topology (Section 5.1). Lemma 6.22 Let G be a semidirect product G = N  K of second countable  let Kλ and groups N and K, where N is abelian and K is compact. For λ ∈ N,  and Gλ denote the stability group of λ in K and G, respectively. Let χ ∈ N suppose that (FP1) holds for the pair (G, Gχ ). Then the map λ → Kλ ∩ Kχ from the orbit G(χ) into K(Kχ ) is continuous at χ. Proof Toward a contradiction, assume that the map is not continuous at χ . Then there exist a proper closed subgroup H of Kχ and a sequence (an )n in K such that an → e in K and Hn = Kχ ∩ an Kχ an−1 = Kχ ∩ Kan ·χ → H K

in K(Kχ ). Choose ω ∈ supp(indHχ 1H ) with ω = 1Kχ and define two irreducible representations σ and τ of Gχ = N  Kχ by σ (x, a) = χ(x)

and

τ (x, a) = χ(x)ω(a),

x ∈ N, a ∈ Kχ .

G Then π = indG Gχ σ and ρ = indGχ τ are irreducible and inequivalent represen is a T1 -space, π is not weakly contained in tations of G. Therefore, since G G indGχ τ . We are going to show that nevertheless τ ≺ π|Gχ . Since Gχ is regularly related to itself in the sense of Mackey [101, page 127], by Fell [51, theorem 5.3],   G Gχ π|Gχ ∼ indGχχ ∩ tGχ t −1 (t · σ )|Gχ ∩tGχ t −1 : t ∈ G ! indNH (an · σ |NHn ), n

for all n ∈ N. Now, (an · σ )|NHn = (an · χ) · 1Hn and (N  Hn , (an · χ) × 1Hn ) → (N  H, χ × 1H ) in Fell’s subgroup representation topology. Inducing being continuous in the subgroup representation topology, we obtain that G

G

χ χ indNH (an · σ |NHn ) → indNH (χ × 1H ). n

K

G

χ Since ω ≺ indHχ 1H , we conclude that τ ≺ indNH (χ × 1H ) ≺ π|Gχ .



Theorem 6.23 The Frobenius property (FP1) holds for Gn = Rn  SO(n) if and only if n = 2. Proof In view of Example 6.21 we only have to show the “only if” part. Thus, let n ≥ 3, K = SO(n), and N = Rn . Then, retaining the notation of

6.4 Property (FP) for discrete groups

287

 with the property that the map λ → Lemma 6.22, it suffices to find χ ∈ N Kλ ∩ Kχ from G(χ) into K(Kχ ) is not continuous at χ . Let χ be defined by χ(x) = exp(2π ix1 ) for x = (x1 , . . . , xn ) ∈ Rn . Let Ek denote the k × k unit matrix and 0k,l the k × l zero matrix, 1 ≤ k, l ≤ n. Then (compare Section 4.5)   # 1 01,n−1 Kχ = : S ∈ SO(n − 1) . 0n−1,1 S For 0 < ϕ < π/2, let  cos ϕ Dϕ = − sin ϕ

sin ϕ cos ϕ



 and Aϕ =

Dϕ 0

0 En−2

 .

Then KAϕ ·χ = Aϕ Kχ A−1 ϕ and using this, it is easily verified that   # 02,n−2 E2 Kχ ∩ KAϕ ·χ = : T ∈ SO(n − 2) . 0n−2,2 T This shows that when ϕ → 0, Kχ ∩ KAϕ ·χ does not converge to Kχ in K(Kχ ). The reader will have observed that instead of this specific χ we could have equally well taken any nontrivial character of N . 

6.4 Property (FP) for discrete groups In this section we study the class of discrete groups for which a complete solution can be given (Theorem 6.32) to the question of when the Frobenius property (FP) holds. To start with, we have to introduce some notation and list certain structural properties of finite conjugacy class groups. For an arbitrary group G let Gf denote the set of all elements x ∈ G for which the conjugacy class Kx = {yxy −1 : y ∈ G} is finite. Then Gf is a normal subgroup of G, and G is called an FC-group if Gf = G. The structure of FC-groups is well understood. In fact, such a group has the following properties. (1) The set Gt of elements of finite order forms a normal subgroup of G, the so-called torsion subgroup of G. (2) The quotient group G/Gt is abelian and (G/Gt )t = {Gt }. In particular, the commutator group [G, G] of G is contained in Gt . (3) If G is finitely generated, then so is every subgroup of G. Moreover, Gt is finite. If G is an arbitrary group, then each finite subset F of Gf is contained in a finite, G-conjugation invariant set and the centralizer of F has finite index in G.

288

Topological Frobenius properties

All this can, for instance, be found in Robinson [136]. The following series of lemmas and propositions will yield that discrete FC-groups satisfy (FP). Lemma 6.24 Let G be a discrete FC-group, H a subgroup of G and F a finite  be such that π(F ) = {I } and let ρ be a normal subgroup of G. Let π ∈ G representation of H such that π ≺ indG H ρ. Then there exists a representation σ of H with the following properties. (1) σ ≺ ρ, π ≺ indG H σ , and σ (H ∩ F ) = {I }.  and σ˙ of H F /F are defined by π˙ (xF ) = π(x), (2) If representations π˙ ∈ G/F G/F x ∈ G, and σ˙ (yF ) = σ (y), y ∈ H , then π˙ ≺ indH F /F σ˙ . Proof For irreducible ρ, the existence of σ has been shown in Proposition 6.15. Now let ρ be an arbitrary representation of H and let   A = {α ∈ supp ρ : supp(indG H α) ∩ G/F = ∅} ⊆ H .  it follows that π is weakly contained in the set of all  is open in G, Since G/F  α, α ∈ A. If α ∈ A, then indG representations indG H H α ! β for some β ∈ G/F  and hence α(H ∩ F ) = {I } by Proposition 6.15. For α ∈ A, let α˙ ∈ H F /F be as in Section 6.2. Let σ denote the direct sum of all α ∈ A. Then σ clearly satisfies (1). G/F For (2), note that indH F /F σ˙ is weakly equivalent to the set of all represenG/F

tations indH F /F α, ˙ α ∈ A, and that, with q: G → G/F ,   G/F  ∩ supp(indG indH F /F α˙ ◦ q ! G/F H α) for each α ∈ A. Since

-

  ∩ supp(indG G/F α) , H

π≺

α∈A

it follows that π˙ ◦ q = π ≺

-

   G/F  ∩ supp(indG G/F ˙ ◦q H α) ≺ indH F /F σ

α∈A G/F

and hence π˙ ≺ indH F /F σ˙ , as was to be shown.



Since FC-groups are amenable, the following proposition in particular shows that (RFP) holds for such groups. Proposition 6.25 Let G be a discrete FC-group, H a subgroup of G, and τ an arbitrary representation of H . If 1G ≺ indG H τ , then τ ! 1H . Proof We notice first that H can be assumed to be finitely generated. To see this, let L denote the collection of all finitely generated subgroups L of H .

6.4 Property (FP) for discrete groups

289

Then τ ! 1H provided that τ |L ! 1L for every L ∈ L. On the other hand, by the amenability of H , τ ≺ indH L (τ |L ), and hence G H G 1G ≺ indG H τ ≺ indH (indL (τ |L )) = indL (τ |L ).

Thus let H be finitely generated. Since G is an FC-group, there exists a finitely generated normal subgroup N of G containing H . Then N t is finite and N/N t is abelian. We now apply Lemma 6.24 to obtain a representation σ of H with the following properties. (1) σ (H ∩ N t ) = {I } and σ ≺ τ . (2) The representation ρ of H N t /N t defined by ρ(yN t ) = σ (y), y ∈ H , satG/N t isfies 1G/N t ≺ indH N t /N t ρ. Once it has been shown that ρ ! 1H N t /N t , it will follow that σ ! 1H (compare the proof of Lemma 6.24), and hence that τ ! 1H . Thus we can and do assume that N t = {e}, so that N is abelian. Since G is an FC-group and N is finitely generated, the centralizer C(N ) of N in G has finite index in G. So there exists a finite subset A of G such that  and x ∈ G, x · χ = a · χ for some a ∈ A. This implies for each χ ∈ N     a · supp(indN supp (indG H τ )|N = H τ) . a∈A

By hypothesis, 1G ≺ τ and hence 1N ≺ (indG H τ )|N . It follows that N 1N = a · γ for some a ∈ A and γ ∈ supp(indH τ ). This forces 1N = γ ∈ supp(indN H τ ). On the other hand, since N is abelian, indG H

 supp(indN H τ ) = {λ ∈ N : λ|H ∈ supp τ }. Combining these two facts, we get   1H = γ |H ∈ supp(indN H τ ) |H = supp τ, 

so that τ ! 1H .

Lemma 6.26 Let G be a discrete FC-group and let N be a finitely generated  there exists σ ∈ N  such that normal subgroup of G. Then, given π ∈ G, supp(π|N ) = G(σ ). Proof Note that the centralizer of N has finite index in G. Therefore the  of N is a Hausdorff space and G-orbits in N  are finite. We dual space N  is now proceed as in the proof of Proposition 4.21. Since N is countable, N second countable. Let U be a countable basis for the topology of supp(π|N ),

290

Topological Frobenius properties

and for U ∈ U, set U G = ∪x∈G x · U . Then each U G is open and dense in supp(π |N ). Since supp(π|N ) is a Hausdorff space, it follows that ∩U ∈U U G = ∅. Fix σ ∈ ∩U ∈U U G . Then G(σ ) is dense in supp(π |N ). In fact, if V is a nonempty open subset of supp(π|N ), then V ⊇ x · U for some U ∈ U and x ∈ G and hence G(σ ) ∩ V = ∅. As G(σ ) is finite, we conclude that supp(π |N ) = G(σ ).  We show next that a discrete FC-group shares the tensor product properties of Lemma 6.18 and consequently satisfies (FP1). Proposition 6.27 Let G be a discrete FC-group, and let π be an irreducible and ρ an arbitrary representation of G. Then 1G ≺ π ⊗ π and 1G ≺ π ⊗ ρ implies π ≺ ρ. Proof Note that every finitely generated subgroup H of G is almost abelian and , σ ⊗ σ contains 1H as a subrepresentation. This implies hence, for each σ ∈ H that 1H ≺ π |H ⊗ π|H for every such H and, consequently, 1G ≺ π ⊗ π . For the second assertion, since every finite subset of G is contained in a G-invariant finite set, it suffices to show that π|N ≺ ρ|N for every finitely generated normal subgroup N of G. Fix such an N and let F be the subgroup generated by all commutators xnx −1 n, x ∈ G, n ∈ N . Then F is finite since Gt is a torsion group, F ⊆ [G, G] ⊆ Gt , and every subgroup of a finitely generated FC-group is finitely generated.  such that supp(π|N ) = G(σ ). Let ω ∈ F  By Lemma 6.26 there exists σ ∈ N such that ω ≤ σ |F and let H be the stabilizer of ω in N. By Mackey’s theory,  such that σ = indN there exists α ∈ H H α and α|F is a multiple of ω.  : τ |F ∼ π|F } is open in G  since F F,π = {τ ∈ G Notice next that the set G    is finite, and supp(π ⊗ τ ) ∩ G/F = ∅ for every τ ∈ G \ GF,π . Therefore, if γ F,π , then 1G ≺ π ⊗ γ and is a representation of G with supp γ = supp ρ ∩ G F,π . γ ≺ ρ. Hence we can assume that supp ρ ⊆ G We claim that ρ|H ∼ G(α) ⊗  for some closed subset  of the dual group  H /F of H /F . To see this, observe first that   /F ∼ G(α) ⊗ H /F . ρ|H ≺ π |H ⊗ H   Let  = {γ ∈ H /F : α ⊗ γ ≺ ρ|H }. Then  is closed in H /F and since, by definition of F , H /F is contained in the center of G/F , G(α) ⊗  = G(α ⊗ ) ≺ ρ|H .  Conversely, if τ ∈ supp(ρ|H ) then τ = (x · α) ⊗ γ for some γ ∈ H /F and  some x ∈ G. Indeed, this follows from the fact that the sets x · σ ⊗ H /F are

6.4 Property (FP) for discrete groups

291

. Consequently, because γ is G-invariant, open and closed in H α ⊗ γ = α ⊗ x −1 · γ = x −1 (x · α ⊗ γ ) ∼ x −1 · τ ∈ supp(ρ|H ). Choose a finite subset X of G such that G(α) = {x · α : x ∈ X}. Now 1G ≺ π ⊗ ρ implies that 1H ≺ π|H ⊗ ρ|H ∼ G(α) ⊗ G(α) ⊗  = {x · α ⊗ y · α ⊗ γ : x, y ∈ X, γ ∈ }. Recall that H is the stability group of ω. Therefore, if for some x, y ∈ X,  H /F ∩ supp(x · α ⊗ y · α) = ∅, then x · ω = y · ω and hence x · α = y · α. Moreover, since H /F is contained in the center of G/F ,   supp(x · α ⊗ y · α) ∩ H /F = supp(α ⊗ α) ∩ H /F . It follows that 1H ≺ α ⊗ α ⊗  and hence, since α is finite dimensional, 1H = β ⊗ γ0 for some β ∈ supp(α ⊗ α) and γ0 ∈ . So 1H ≺ α ⊗ (α ⊗ γ0 ) and therefore α = α ⊗ γ0 . This in turn implies N π |N ∼ G(σ ) ∼ G(indN H α) ∼ {indH (x · α) : x ∈ X}

∼ {indN H x · (α ⊗ γ0 ) : x ∈ X} N ≺ {indN H (x · α ⊗ γ ) : x ∈ X, γ ∈ } ∼ indH (ρ|H ).

Now, again using that N/F is contained in the center of G/F , we have F,π |N ∼ π |N ⊗ N/F  ∼ G(σ ) ⊗ N/F  = G(indN  G H α) ⊗ N/F  ∼ {indN H ((x · α)) ⊗ χ) : χ ∈ H /F , x ∈ X}.  and supp(ρ|N ) ⊆ G F,π |N , it follows that Since this latter set is closed in N N ρ|N ∼ indH (ρ|H ) and hence π|N ≺ ρ|N . As this holds for every finitely generated normal subgroup N of G, the proof is complete.  From Proposition 6.27 we can now quickly deduce that (FP) holds for discrete FC-groups. Theorem 6.28 Let G be a discrete FC-group. Then G has the Frobenius property (FP). Proof We first show that (FP2) holds for G. Thus, let H be a subgroup of G, , and π ∈ G  such that π ≺ indG τ ∈H H τ . Then, by Proposition 6.27, G 1G ≺ π ⊗ π ≺ π ⊗ indG H τ = indH (π|H ⊗ τ ).

292

Topological Frobenius properties

By Proposition 6.25, this implies that 1H ≺ π|H ⊗ τ or, equivalently, 1H ≺ τ ⊗ π |H . Proposition 6.27 now yields that τ ≺ π |H . Conversely, assume that τ ≺ π|H . Then, using amenability of G and Proposition 6.27, 1H ≺ τ ⊗ τ ≺ τ ⊗ π|H and hence G G G 1G ≺ indG H 1H ≺ indH (τ ⊗ τ ) ≺ indH (τ ⊗ π |H ) = π ⊗ indH τ.

Using Proposition 6.27 again, we conclude that π ≺ indG H τ . So (FP1) also holds for G.  Our aim is to establish the converse of Theorem 6.28. Actually, it will turn out that for a discrete group G even validity of property (RFP) forces G to be an FC-group. The next two lemmas are important, but somewhat technical tools. Lemma 6.29 Let G be a discrete group with property (RFP). If G = {e}, then Gf = {e}. Proof We first suppose that G is a nontrivial countable group. Toward a contradiction, assume that Gf = {e}. Then λG is a faithful factorial representation of L1 (G) and, since G is amenable, extends to a faithful representation of C ∗ (G). Now, since C ∗ (G) is separable, the kernel of any factorial representation of  C ∗ (G) is a primitive ideal [36, (3.9.1) and (5.7.6)]. So there exists π ∈ G such that ker π = {0}. Then 1G ≺ π = indG G π, so that 1G = π by property (RFP). However, this is impossible unless G = {e}. This proves the lemma for countable G. We now drop the hypothesis that G be countable. Since property (RFP) is inherited by subgroups (Lemma 6.14), by the first paragraph it suffices to show that if Gf = {e}, then there exists a nontrivial countable subgroup H of G with Hf = {e}. This can be achieved as follows. For every x ∈ G, x = e, there exists a countable subset Mx of G such that x ∈ Mx and the set {yxy −1 : y ∈ Mx } is infinite. We construct by induction a sequence (Hn )n of countable subgroups of G such that Hn ⊆ Hn+1 and Hn ∩ (Hn+1 )f = {e}. To start with, choose any x ∈ G, x = e, and let H1 be the subgroup generated by Mx . Assuming that Hn has been defined, let Hn+1 be the subgroup generated by the countable set ∪{Mx : x ∈ Hn }. Then it is easily verified that the groups Hn do have the required properties, and the group H = ∪∞ n=1 Hn is countable and satisfies  Hf = {e}. In passing, we have to introduce some more notation to be used in the sequel. If the group G is fixed and M is a subset of G, then M, C(M), and N (M) will denote the subgroup generated by M, the centralizer of M, and the normalizer of M in G, respectively. If M is a singleton, say M = {x}, then we simply write x, C(x), and N (x).

6.4 Property (FP) for discrete groups

293

Lemma 6.30 Let G be a discrete group and suppose that G has property (RFP). Then (i) The elements of G of prime order lie in Gf . (ii) (G/Gf )f is a torsion group. Proof (i) Toward a contradiction, assume that there exists x ∈ G such that x , has order p, a prime, and nevertheless x ∈ Gf . Let H = x and fix any χ ∈ H χ = 1H . Let ϕ denote the trivial extension of χ to all of G and πϕ the cyclic representation of G associated with ϕ. We show that λG ≺ πϕ = indG H χ. Since C(x i ) = C(x) for 1 ≤ i ≤ p − 1, for any given finite subset F of G we find y ∈ G such that F ∩ yHy −1 ⊆ {e}. For z ∈ F , z = e, we then have ϕ(y −1 zy) = 0 and therefore y · ϕ(z) = δe (z) for all z ∈ F . Now, λG is the cyclic representation associated with the Dirac function δe and y · ϕ is a positive definite function associated with πϕ . Since F is an arbitrary finite subset of G, it follows that λG ≺ πϕ . Finally, since G has property (RFP), 1G ≺ λG ≺ indG H χ and therefore χ = 1H . This contradiction proves that x ∈ Gf , as was to be shown. (ii) Assume that there exists x ∈ G such that xGf ∈ (G/Gf )f and xGf has infinite order. Then the subgroup K = {y ∈ G : yGf ∈ C(xGf )} has finite index in G and therefore satisfies Kf = Gf . Thus, passing to K, we can henceforth assume that xGf is contained in the center of G/Gf . The proof now proceeds similarly to the proof of (i). We are going to show that λG ≺ indG x χ for every character χ of x. To that end, let H = xGf , let ϕ denote the trivial extension of χ to G, and consider any finite subset F of G. For y ∈ G \ H and z ∈ G we then have z−1 yz ∈ x. Indeed, z−1 yz ∈ x and yGf ∈ Z(G/Gf ) would imply that y ∈ xGf . Thus z · ϕ(y) = 0 for all y ∈ G \ H and z ∈ G. As xGf has infinite order, x ∩ Gf = {e} and hence z · ϕ(y) = δe (y) for y ∈ Gf . So we only have to consider elements of F ∩ (H \ Gf ) and hence can assume that F = {y1 , . . . , ym }, where each yj is  of the form yj = x kj tj with tj ∈ Gf and kj = 0, 1 ≤ j ≤ m. Let k = m j =1 kj . If z ∈ C({t1 , . . . , tm }) is such that z−1 yj z ∈ x for some j , then [x −kj , z−1 ]tj = x −kj (z−1 yj z) ∈ x ∩ Gf , since xGf is contained in the center of G/Gf . Thus [x −kj , z−1 ] = tj−1 because x ∩ Gf = {e}. Moreover, if z and z0 are elements of G such that [x −kj , z−1 ] = [x −kj , z0−1 ], then z−1 z0 ∈ C(x kj ) ⊆ C(x k ). Now [C({t1 , . . . , tm }) : (C(x k ) ∩ C({t1 , . . . , tm }))] = ∞ since x k ∈ Gf and [G : C(tj )] < ∞ for 1 ≤ j ≤ m. Hence there exists z ∈ C({t1 , . . . , tm }) such that z−1 yj z ∈ x for 1 ≤ j ≤ m. For such z, it follows that z · ϕ(yj ) = 0, 1 ≤ j ≤ m, and therefore z · ϕ(y) = δe (y) for all y ∈ F .

294

Topological Frobenius properties

This shows that λG ≺ πϕ = indG x χ. Since χ was an arbitrary character of x and G satisfies (RFP), this leads to a contradiction as in the proof of part (i). It follows that every xGf ∈ (G/Gf )f has finite order, as was to be shown.  Proposition 6.31 Let G be a discrete group with property (RFP). Then G is an FC-group. Proof Assume that G = Gf . Then G/Gf has property (RFP) by Lemma 6.14 and hence (G/Gf )f is nontrivial by Lemma 6.29. By Lemma 6.30(ii) there exists x ∈ G such that x ∈ Gf and the element xGf of (G/Gf )f has order p for some prime number p. We then have x ∩ Gf = x p  and the quotient group H = N(x p )/x p  has property (RFP) by Lemma 6.14. We claim that xx p  ∈ Hf , equivalently, that the centralizer of xx p  in H has infinite index in H . To see this, observe first that if y ∈ N (x p ) is such that yx ∈ C(xx p ), then yxy −1 ∈ xx p  ⊆ x and hence y ∈ N (x). Thus we get [H : C(xx p )] ≥ [N (x p )/x p  : N(x)/x p ] = [N (x p ) : N (x)]. Now [G : N(x)] = ∞. Indeed, this is clear if x is finite and follows from [N (x) : C(x)] ≤ 2 if x is infinite. Since C(x p ) has finite index in G, we get [N(x p ) : N (x)] = ∞. So xx p  does not lie in Hf and has order p since x ∩ Gf = x p . This contradicts Lemma 6.30(i) and thus shows that G = Gf .  The following final result is now an immediate consequence of Theorem 6.28 and Proposition 6.31. Theorem 6.32 For a discrete group G, the following three conditions are equivalent. (i) G has the Frobenius property (FP). (ii) G has the restricted Frobenius property (RFP). (iii) G is an FC-group.

6.5 Nilpotent groups The theme of this final section is to explore the topological Frobenius properties (RFP), (FP1), and (FP2) for nilpotent locally compact groups. Property (RFP) (and hence (FP)) turns out to be extremely restrictive in that it does not hold for any nonabelian simply connected nilpotent Lie group (Theorem 6.39). A main step toward Theorem 6.39 is Theorem 6.33 below. Concerning (FP1), if G is a simply connected nilpotent Lie group and π is an irreducible representation

6.5 Nilpotent groups

295

of G which is square-integrable modulo its kernel, then (FP1) holds for π, that , τ ≺ π|H implies π ≺ indG is, for each closed subgroup H of G and τ ∈ H H τ (Theorem 6.40). In particular, (FP1) holds true for any two-step nilpotent simply connected Lie group. However, we present an example showing that (FP1) may fail for three-step nilpotent simply connected Lie groups. For any group G, let Z(G) = Z1 (G) ⊆ Z2 (G) ⊆ . . . denote the ascending central series of G, and recall that G is n-step nilpotent if Zn (G) = G but Zn−1 (G) = G. Theorem 6.33 Let G be a simply connected nilpotent Lie group and let H be any non-normal one-dimensional subgroup of G. Then 1G ≺ indG H α for every character α of H . Proof We start the proof with the following simple observation. If G is any simply connected nilpotent Lie group and H is a one-dimensional normal vector subgroup of G such that H ⊆ Z2 (G), then even H ⊆ Z(G). Indeed, [G, H ] ⊆ H ∩ Z(G) which is either trivial or equals H because H is one-dimensional and the intersection of two connected subgroups is connected [130]. Thus H ⊆ Z(G) in both cases. We now assume first that H ⊆ Z2 (G) and that dim Z(G) = 1. Then the abelian normal subgroup N = Z(G)H of G is closed because both H and Z(G) = are connected. Moreover, N = H × Z(G) since H ∩ Z(G) = {e}. Thus N N N    H × Z(G) and indH α ∼ {α} × Z(G). It suffices to show that 1N ≺ G(indH α). Indeed, since G/N is amenable, it then follows that G N G 1G ≺ indG N 1N ≺ indN (indH α) = indH α.

 and (xn )n in G such that Consequently we have to find sequences (γn )n in Z(G) . Since H ⊆ Z2 (G), for any x ∈ G, y ∈ H , z ∈ Z(G), xn · (α × γn ) → 1N in N  we have and γ ∈ Z(G), x · (α × γ )(yz) = (α × γ )((x −1 yxy −1 )yz) = α(y)γ ([x −1 , y])γ (z). It is easily verified that y → γ ([x −1 , y]) defines a character γx of H and that  = R. If γ the map  : x → γx is a continuous homomorphism of G into H is any nontrivial character of Z(G), then γx = 1H for some x ∈ G because otherwise the connected set [G, H ] = {[x, y] : x ∈ G, y ∈ H } is contained in the discrete kernel of γ , which implies that [G, H ] = {e} and hence H ⊆ Z(G),  a contradiction. It follows that (G) is a nontrivial connected subgroup of H  with . Now choose a sequence (γn )n in Z(G) and therefore coincides with H γn → 1Z(G) and γn = 1Z(G) for all n. For every n there exists xn ∈ G such that xn · γn = α and hence xn · (α × γn )(yz) = α(y)(γn )xn (y)γn (z) = γn (z) → 1 uniformly on H × C for every compact subset C of Z(G).

296

Topological Frobenius properties

Next we drop the hypothesis that dim Z(G) = 1 and argue by induction on d = dim G. If d = 3, which is the smallest possible dimension if G is nonabelian, then dim Z(G) = 1. So suppose that d ≥ 4 and dim Z(G) ≥ 2, the case dim Z(G) = 1 having been dealt with above. Then there exists a nontrivial vector subgroup V of Z(G) such that H V /V is not contained in the center of G/V . In fact, assuming the contrary and using that H ⊆ Z2 (G) and H ∩ Z(G) = {e}, we would get that [G, H ] ⊆ V for each such V , which is clearly impossible since there exist at least two such complementary subspaces V . Fix such a vector subgroup V . Then H V /V is not normal in G/V and, by G/V the inductive hypothesis, 1G/V ≺ indH V /V β for each character β of H V /V . Denoting by q : G → G/V the quotient homomorphism, this yields   G/V G 1G ≺ indH V /V β ◦ q = indG H V (β ◦ q) ≺ indH ((β ◦ q)|H ).  is of the form (β ◦ q)|H for some β ∈ H V /V . Since H ∩ V = {e}, every α ∈ H G  and thus completes the proof This proves that 1G ≺ indH α for every α ∈ H when H ⊆ Z2 (G). Finally, in the general case, let k ∈ N be minimal with the property that H ⊆ Zk+1 (G). Since the case k ≤ 1 has already been dealt with, we can assume that k ≥ 2. Let F = G/Zk−1 (G) and q : G → F the quotient homomorphism. Then q(H ) = H Zk−1 (G)/Zk−1 (G) is a closed one-dimensional subgroup of F , which is contained in Z2 (F ) and which is non-normal because it is not contained in the center of F . By what we have shown so far, 1F ≺ indFq(H ) γ for  every character γ of q(H ), and this implies that 1G ≺ indG H α for each α ∈ H by the same argument as above. This finishes the proof of the theorem.



We now recall for the reader’s convenience the following proposition from Section 5.8, which is essential for what follows. Proposition 6.34 Let G be a two-step nilpotent locally compact group and let Z be a closed subgroup of G such that Z is contained in the center of G and  let G/Z is abelian. For λ ∈ Z, Zλ = {z ∈ Z : λ(z) = 1} and Lλ = {x ∈ G : [x, G] ⊆ Zλ }, that is, Lλ /Zλ equals the center of G/Zλ . Let π be an irreducible representation  be such that π|Z is a multiple of λ. Then there exists a Gof G and let λ ∈ Z invariant character α of Lλ such that π ∼ indG Lλ α. Lemma 6.35 Suppose that G contains a compact subgroup K such that K is contained in the center of G and G/K is abelian. Then G satisfies (FP2).

6.5 Nilpotent groups

297

 and π ∈ G  be such Proof Let H be a closed subgroup of G and let τ ∈ H that π ≺ indG τ . Let C = {z ∈ K : π(z) = I }. Then τ (H ∩ C) = {I } and if H   π˙ ∈ G/C and τ˙ ∈ H C/C are defined as prior to Proposition 6.15, then π˙ ≺ G/C indH C/C τ˙ by Proposition 6.15. Suppose that this implies π˙ |H C/C ! τ˙ . Then, with q : G → G/C denoting the quotient homomorphism, π|H = (π˙ ◦ q)|H ≺ (τ˙ ◦ q)|H = τ. Therefore, passing to G/C, we can assume that π |K is faithful. Let Z denote the center of G. By Proposition 6.34, there exists a character χ of Z such that π ∼ indG Z χ . Then, since Z is the center of G, χ|H ∩Z ∼ π |H ∩Z ≺ (indG H τ )|H ∩Z ∼ τ |H ∩Z and since H is amenable, this implies H π |H ∼ indH H ∩Z (χ|H ∩Z ) ∼ indH ∩Z (τ |H ∩Z ) ! τ,



as was to be shown.

To establish (FP1) for certain two-step nilpotent groups, we need the next two lemmas. Lemma 6.36 Let G and Z be as in Proposition 6.34, and let H be an abelian  let Lγ |Z be  and for γ |Z ∈ Z, closed subgroup of G that contains Z. Let γ ∈ H defined as in Proposition 6.34. Then  G(γ ) = γ · H /H ∩ Lγ |Z . Proof Put λ = γ |Z and choose an irreducible representation π of G such that λ be as guaranteed by Proposition 6.34. Then, since α is π |Z ∼ λ. Let α ∈ L G-invariant, G G  π ∼ indG Lλ α ∼ indLλ (π|Lλ ) = π ⊗ indLλ 1Lλ ∼ π ⊗ G/Lλ .

λ |H . Clearly, the subgroup Therefore, restricting to H , π|H ∼ π |H ⊗ G/L  λ |H of H /H ∩ Lλ separates the points of H /H ∩ Lλ , and hence is dense G/L  in H /H ∩ Lλ . It follows that  ∩ Lλ ∼ π|H ⊗ indH π |H ∼ π|H ⊗ H /H H ∩Lλ 1H ∩Lλ H = indH H ∩Lλ (π|H ∩Lλ ) ∼ indH ∩Lλ (α|H ∩Lλ ).

Since γ |H ∩Lλ = α|H ∩Lλ , we conclude that  G(γ ) ∼ π |N ∼ indH H ∩Lλ (α|H ∩Lλ ) ∼ γ · H /H ∩ Lλ . This proves the lemma.



298

Topological Frobenius properties

Lemma 6.37 Let G be a two-step nilpotent locally compact group and suppose that (FP1) holds for all pairs (B, A), where B is any quotient group of G and A is an abelian closed subgroup of B. Then G satisfies (FP1). Proof Let H be any closed subgroup of G, and let π and τ be irreducible representations of G and H , respectively, such that τ ≺ π |H . Let Z be the  such that π |Z ∼ λ. By Proposition 6.34 there exist center of G and let λ ∈ Z closed normal subgroups K and L of H such that K ⊆ L and L/K equals the center of H /K and a character α of L/K such that τ ∼ indH L α. We claim that π(K ∩ Z) = {I }. In fact, since τ |K∩Z = 1 and τ |K∩Z ≺ π |K∩Z ∼ λ|K∩Z , we have λ|K∩Z = 1. Moreover, L/K ∩ Z is abelian since both L/K and G/Z are abelian. Let q : G → G/K ∩ Z denote the quotient homomorphism, and define π˙ ∈   G/K ∩ Z, τ˙ ∈ H  /K ∩ Z, and α˙ ∈ L/K ∩ Z, respectively, by π(x(K ˙ ∩ Z)) = π (x) for x ∈ G, τ˙ (h(K ∩ Z)) = τ (h) for h ∈ H , and α(l(K ˙ ∩ Z)) = α(l) for l ∈ L. Then H /K∩Z

π| ˙ H /K∩Z ! τ˙ ∼ indL/K∩Z α˙ and hence, restricting further to L/K ∩ Z, since α is invariant, ˙ π| ˙ L/K∩Z ! τ˙ |L/K∩Z ∼ α. Now, since L/K ∩ Z is abelian, by hypothesis (FP1) holds for the pair of groups (G/K ∩ Z, L/K ∩ Z). Therefore,   G/K∩Z G/K∩Z H /K∩Z G/K∩Z π˙ ≺ indL/K∩Z α˙ = indH /K∩Z indL/K∩Z α˙ ∼ indH /K∩Z τ˙ . This in turn implies

  G/K∩Z G π = π˙ ◦ q ≺ indH /K∩Z τ˙ ◦ q = indG H (τ˙ ◦ q) = indH τ.

Thus G satisfies (FP1).



At first glance the preceding lemma does not look very helpful. However, its value will become apparent when proving the following proposition, which establishes property (FP1) for a special class of two-step nilpotent groups. Proposition 6.38 Suppose that G contains a compact subgroup Z such that Z is contained in the center of G and G/Z is abelian. Then (FP1) holds for G. Proof Since every quotient group of G has the same structure, by Lemma 6.37 we only have to verify (FP1) when the subgroup of G in question is abelian.

6.5 Nilpotent groups

299

Thus let H be an abelian closed subgroup of G, and let τ be a character of H and π an irreducible representation of G such that τ ≺ π |H . Let N = H Z, which is an abelian closed subgroup of G since Z is compact and contained in the center of G and H is abelian. Once we have proved that π |N ≺ G(indN H τ ), since G is amenable it will follow that G N G π ≺ indG N (π|N ) ≺ {indN (x · (indH τ )) : x ∈ G} = {indH τ }.

By Proposition 6.34 there exist a closed subgroup L of G with Z ⊆ L ⊆ N and  a G-invariant character λ of L such that π|N ∼ indN L λ. In fact, if π |Z ∼ σ ∈ Z, then L = {x ∈ N : σ ([x, G]) = {1}}. Let  : γ |H = τ } = supp(indN  = {γ ∈ N H τ ). For every γ ∈ , γ |H ∩L = τ |H ∩L = λH ∩L , and  |L ∼ γ |L · L/L  |L = γ |L · N/H ∩ H. , let Nγ = {x ∈ N :  Therefore |L ∼ λ · L/L ∩ H . For every γ ∈ N γ (Lemma 6.36). Since λ ≺ |L , we γ ([x, G]) = {1}}. Then G(γ ) = γ · N/N find a net (γι )ι in  such that γι |L → λ. Because K(N ) is a compact space (Lemma 5.15), we can assume that Nγι → N0 for some closed subgroup N0 of N. Then N0 ⊆ L. To see this, let x ∈ N0 and select xι ∈ Nγι such that xι → x. Then, since γι |Z → λ|Z , 1 = γι ([xι , y]) → λ([x, y]) = σ ([x, y]) for all y ∈ G, whence x ∈ L. We claim that (Nγι , γι |Nγι ) → (N0 , λ|N0 ) in Fell’s subgroup representation topology (Section 5.2). To that end, we have to show that of xι ∈ Nγι and xι → x ∈ N0 , then some subnet of γι (xι ) converges to λ(x). Let xι = hι zι and x = hz, where hι , h ∈ H and zι , z ∈ Z. Since Z is compact, after passing to a subnet and relabeling, we can assume that zι → z0 for some z0 ∈ Z. Then hι → h(zz0−1 ) and zz0−1 ∈ H ∩ Z ⊆ H ∩ L. Since γι |Z → λ|Z and Z is compact, we can in addition assume that γι |Z = λ|Z for all ι. Because τ |H ∩L = λ|H ∩L , it follows that γι (xι ) = γι (hι )γι (zι ) = τ (hι )λ(zι ) → τ (h(zz0−1 ))λ(z0 ) = τ (h)λ(zz0−1 )λ(z0 ) = τ (hz), as required.

300

Topological Frobenius properties

Since inducing is continuous in the subgroup representation topology (Theorem 5.39), we conclude that N indN Nγι (γι |Nγι ) → indN0 (λ|N0 ).

(6.7)

N π |N ∼ indN L λ ≺ indN0 (λ|N0 )

(6.8)

 indN Nγι (γι |Nγι ) ∼ γι · N/Nγι ∼ G(γι ).

(6.9)

Finally, recall that

and

Since γι ≺ indN H τ , combining (6.7), (6.8), and (6.9) shows that π |N ≺ N G(indH τ ). This completes the proof of the proposition.  As an application of the preceding results, we can now identify the connected nilpotent groups for which (FP) or (RFP) holds. Theorem 6.39 Let G be a connected nilpotent locally compact group and let K be the maximal compact normal subgroup of G. Then the following are equivalent. (i) (FP) holds for G. (ii) (RFP) holds for G. (iii) G/K is abelian. Proof (i) ⇒ (ii) being trivial, suppose next that (iii) holds. Since K is a compact solvable normal subgroup of G and G is connected, K is contained in the center of G. In fact, this is well known and can be verified by using that the action of a connected group on the dual of any abelian compact normal subgroup is trivial. Hence G satisfies (FP2) by Lemma 6.35, and also (FP1) holds by Proposition 6.38. Therefore it only remains to show (ii) ⇒ (iii). Since (RFP) holds for G/K and G/K is simply connected, it suffices to show that (RFP) does not hold for any nonabelian simply connected nilpotent Lie group G. Since G is nonabelian, there exists a one-dimensional subgroup H of G which is non-normal. Then Theorem 6.33 tells us that (RFP) fails for G. So (ii) implies (iii).  Recall that if G is any locally compact group and π is an irreducible representation of G with group kernel K = {x ∈ G : π (x) = I }, then π is called square-integrable modulo K if for at least one (and hence all) pairs of nonzero vectors ξ, η in H(π ), the function xK → π(x)ξ, η is square-integrable on the quotient group G/K. We now show that (FP1) holds for irreducible representations of connected nilpotent Lie groups which are square-integrable

6.5 Nilpotent groups

301

modulo their kernels and connected closed subgroups, regardless of the degree of nilpotency of the group. In particular, (FP1) holds true for two-step nilpotent connected Lie groups and connected closed subgroups, because every irreducible representation of such a group is square-integrable modulo its kernel. The proof of the following theorem, however, exploits a deep result due to Moore and Wolf [113].  If π is Theorem 6.40 Let G be a connected nilpotent Lie group and π ∈ G. square-integrable modulo its kernel, then for every connected subgroup H of , τ ≺ π |H implies π ≺ indG G and τ ∈ H H τ.  denote the simply connected covering group of G and p : G →G Proof Let G  −1   π = π ◦ p ∈ G, and  τ =τ◦ the covering homomorphism. Let H = p (H ),   . Then   = CD, where C is the connected component of p∈H τ ≺ π |H and H  and D is the kernel of p. Let Z denote the center of G  and the identity of H let F = CZ. Then F , being the product of two connected subgroups, is closed  and   [130]. Now, since  τ |H∩Z ∼ χ|H∩Z , we can in G π |Z ∼ χ for some χ ∈ Z define an irreducible representation σ of F by setting σ (yz) = χ(z) τ (x),

y ∈ C, z ∈ Z.

τ and σ ≺  π |F . Since  π is square-integrable modulo Then σ satisfies σ |H =   its kernel, it follows from [113, theorem 1] that  π ≺ indG F σ . From this we  conclude that π ≺ indG H τ . Indeed, if ω ∈ p(F ) is defined by ω(p(x)) = σ (x), then ω|H = τ and therefore, since H ⊆ p(F ),  G   G   π ◦p = π ≺ indG F σ = indp(F ) ω ◦ p ≺ indH ω|H ◦ p   = indG H τ ◦ p. Thus π ≺ indG H τ.



We now give an example which shows that (FP1) does not hold in general for three-step nilpotent simply connected Lie groups. Being six-dimensional, this example might not be the smallest possible one from the dimension point of view, but it is conveniently accessible. Example 6.41 Let G be the three-step nilpotent group consisting of all uppertriangular 4 × 4 matrices with real entries and 1 on the main diagonal. For an element ⎛ ⎞ 1 x1 y1 z ⎜ 0 1 x 2 y2 ⎟ ⎟ g=⎜ ⎝ 0 0 1 x3 ⎠ 0 0 0 1

302

Topological Frobenius properties

of G we use the notation g = (x, y, z), where x = (x1 , x2 , x3 ) and y = (y1 , y2 ). With this convention, (x, y, z)(x  , y  , z ) = (x + x  , y + y  + (x1 x2 , x2 x3 ), z + z + x1 y2 + y1 x3 ) and (x, y, z)−1 = (−x, −y + (x1 x2 , x2 x3 ), −z + x1 y2 + y1 x3 + x1 x2 x3 ). Define subgroups H and N of G by H = {g ∈ G : x1 = x3 = y1 = 0} and N = {g ∈ G : x1 = x3 = 0}. Then N is an abelian normal subgroup of G and the action of an element ga,b = (a, 0, b, 0, 0, 0) of G on N is given by −1 (x2 , y1 , y2 , z)ga,b ga,b · (x2 , y1 , y2 , z) = ga,b

= (x2 , y1 − ax2 , y2 + bx2 , z + ay2 + by1 + abx2 ).  with R4 via (r, s, t, u) → χr,s,t,u , where As usual, we identify N χr,s,t,u (x2 , y1 , y2 , z) = exp(2π i(rx2 + sy1 + ty2 + uz)).  with R3 . Then, with ga,b as above, the action on R4 is Similarly, we identify H given by ga,b · (r, s, t, u) = (r − as + bt + abu, s + au, t + bu, u).  and α = (1, 0, 1) ∈ H . Then π = indG Now, let β = (0, 0, 0, 1) ∈ N N β is an irreducible representation of G since the stability group of β in G equals N. We claim that π|H weakly contains α, but π is not weakly contained in indG H α. Since N is normal in G, π|N ∼ G(β) and hence π |H ∼ {(ga,b · β)|H : a, b ∈ R} = {(ab, b, 1) : a, b ∈ R}. Taking b = n1 and a = n, n ∈ N, we conclude that π |H ! α. Toward a contradiction, suppose that nevertheless π ≺ indG H α. Then β ≺ π |N ≺ G(indN H α) ∼ G({1, s, 0, 1) : s ∈ R}) = {(1 − as + ab, s + a, b, 1) : a, b, s ∈ R}. It is now straightforward to verify that there are no sequences (an )n , (bn )n , and (sn )n of real numbers such that (1 − an sn + an bn , sn + an , bn , 1) → (0, 0, 0, 1) = β in R4 as n → ∞. This contradiction shows that π is not weakly contained in indG H α.

6.6 Notes and references

303

6.6 Notes and references The topological Frobenius property (FP) as well as the weaker properties, (WF1) and (WF2), were initially introduced and briefly studied by Fell [51]. He observed that (FP) holds for locally compact abelian groups and, in view of Kirillov’s theory for simply connected nilpotent Lie groups, which was just developed in the early 1960s, he felt that (FP) might be satisfied by all simply connected nilpotent Lie groups. Fell also showed that (WF2) fails to hold for SL(3, R) and that (WF1) forces the group to be amenable. He posed the question of whether property (WF1) is actually equivalent to the amenability of the group. The positive solution to this problem (Theorem 6.9) was achieved by Greenleaf [68] as an application of his investigation of amenable actions of locally compact groups on locally compact Hausdorff spaces. Subsequently, Theorem 6.9 turned out to be of immense value in the study of topological Frobenius properties. As mentioned in Section 6.2, it is not known whether property (FP) (in contrast to (RFP)) is inherited by closed subgroups. An example that (FP1) does not carry over to closed subgroups, is given in Bekka [18, example 2.4]. The extremely useful Proposition 6.15, which sometimes allows us to pass over to quotients modulo a compact normal subgroup, was proved in Henrichs [72, proposition 2.8]. The most general class of locally compact groups for which (FP) is known to hold, is formed by the so-called [FC]− -groups, the groups with relatively compact conjugacy classes [84, theorems 2.5 and 2.6]. Thus the implication (iii) ⇒ (i) of Theorem 6.28 holds in much wider generality. These [FC]− groups are natural generalizations of discrete FC-groups, which we studied in Section 6.4. Due to work of Grosser and Moskowitz [70], the structure of [FC]− -groups is fairly well understood. The question of whether conversely an arbitrary locally compact group G satisfying (FP) has to be an [FC]− -group, remains open. However, as we have seen in Section 6.4, a discrete group with property (FP) has to be an FC-group [71, theorem 2.7]. Moreover, as proved by Felix et al. [45], the answer is also affirmative if either G is a group which possesses a compact invariant neighborhood of the identity (a so-called IN-group) or G is amenable and has an open connected component of the identity. It is interesting to note in this context that [FC]− -groups also seem to be precisely those locally compact groups which have a Hausdorff primitive ideal space. In fact, [FC]− -groups have a Hausdorff primitive ideal space (see [97] and [81]) and, conversely, it was shown in Baggett and Sund [11] that for a connected locally compact group G, Hausdorffness of Prim(G) forces G to be an [FC]− -group.

304

Topological Frobenius properties

Nielsen [115] (see also [116]) was the first to discover that (FP) fails to hold for every nonabelian connected and simply connected nilpotent Lie group, thereby refuting Fell’s expectation. Theorem 6.33 is due to him. The simpler proof presented here can be found in Kaniuth [85, theorem 2.1]. Theorem 6.40 was established in Bekka and Kaniuth [18, theorem 1.3]. Example 6.41, which shows that three-step nilpotent connected Lie groups need not share (FP1), is taken from Fell [47], where Kirillov’s theory was employed. Further indication of limitation of the validity of (FP1) for nilpotent Lie groups is provided by [18, theorem 1.5], which states that if (FP1) holds for a nilpotent group of the form Rn  R, then G has to be two-step nilpotent. As shown in Henrichs [72, theorem 3.1], every group with small invariant neighborhoods has property (WF2). In this chapter we have focused mainly on the relation between the validity of any of these topological Frobenius properties and the structure of the group in question. A different viewpoint that can be taken is to fix a pair (G, N) of a locally compact group G and a closed normal subgroup N and aim for criteria when such a pair satisfies (FP1), (FP2), or (FP). This was done by Gootman [64] who has, for instance, shown that if G is second countable, then (FP2) and also (FP) hold for (G, N) if and only if G/N is amenable and G acts minimally on Prim(N), that is, each orbit closure in Prim(N) is a minimal closed G-invariant set. Moreover, when G/N is compact, this latter condition is equivalent to Prim(N ) being a T1 -space. From Gootman’s result one can deduce that (FP) holds for the pair (G, N) if G is a connected and simply connected nilpotent Lie group and N is a connected closed normal subgroup of G, a result which was earlier proved by Moscovici [114].

7 Further applications

In this final chapter, we present applications of the theory of induced representations where knowing an explicit expression for an induced representation of a specific group is used. In Section 7.1, we apply Mackey’s theory, and one realization of the induced representations giving the irreducible representations to study the asymptotic behavior of coefficient functions of those representations. The main theorem is that those coefficient functions of infinite-dimensional irreducible representations of motion groups vanish at infinity. As a consequence, one can conclude that the image of a motion group under any irreducible representation is closed in the unitary group with the weak operator topology. Section 7.2 is concerned with introducing methods for constructing self-adjoint idempotents, or projections, in L1 (G) for certain kinds of groups  of a projection must be a compact G. A key observation is that the support in G open set. After reviewing how projections arise for compact and abelian G, we turn to the noncompact, nonabelian situation. Drawing upon the theory developed in Chapters 4 and 5, we identify groups with nontrivial compact open sets in their duals. For appropriate groups G and explicit induced representations of those groups, coefficient functions of those representations can be modified to produce nontrivial projections in L1 (G). Finally, certain identities arising in the construction of projections can be exploited to produce generalizations of the continuous wavelet transform. This is explored in Section 7.3.

7.1 Asymptotic properties of irreducible representations of motion groups In this section, we are concerned with the question of whether or not the coefficient functions of a representation vanish at infinity. We give a small sample 305

306

Further applications

of this area of research by providing a complete answer for the irreducible representations of motion groups where explicitly knowing the descriptions in terms of induced representations is used. Let G be a locally compact group and π a representation of G. For ξ , π η ∈ H(π ), recall that the corresponding coefficient function ϕξ,η : G → C is given by π (x) = π(x)ξ, η, ϕξ,η

for x ∈ G. Definition 7.1 A representation π of a locally compact group G is said to π ∈ C0 (G), for all ξ, η ∈ H(π ). vanish at infinity if ϕξ,η π Remark 7.2 Using the Cauchy–Schwarz inequality, if ϕξ,ξ ∈ C0 (G), for all ξ ∈ H(π ), then π vanishes at infinity.

The following result is a now classic generalization of the Riemann– Lebesgue lemma. Proposition 7.3 The left-regular representation λG of any locally compact group G vanishes at infinity. λG Proof By density of Cc (G) in L2 (G), it suffices to show that ϕf,g ∈ C0 (G) for any f, g ∈ Cc (G). But, for such f and g, if K = supp(g) supp(f )−1 , then λG ϕf,g (x) = λG (x)f, g = 0, for any x ∈ G \ K. 

It is clear that the property of vanishing at infinity is maintained by taking subrepresentations, tensor products, and arbitrary direct sums. It is also clear that any character vanishes at infinity if and only if G is compact. In fact, the same holds for any finite-dimensional representation. Proposition 7.4 Let G be a locally compact group and let π be a representation of G with d = dim H(π ) < ∞. If π vanishes at infinity then G is compact. Proof We prove the contrapositive. Suppose G is not compact. For any compact C ⊆ G, there exists xC ∈ G \ C. Partially order the set of all compact subsets of G by inclusion. Then the net (π (xC )) in the compact d-dimensional unitary group has a convergent subnet (π(xCα )). Let U be a unitary operator on H(π ) so that π (xCα ) → U in norm. Let ξ ∈ H(π ) with ||ξ || = 1. Let η = U ξ . Then π ϕξ,η (xCα ) = π(xCα )ξ, η → 1. π ∈ / C0 (G) and π cannot vanish at infinity. Thus ϕξ,η



We know the regular representation always vanishes at infinity, but we are particularly interested in irreducible representations. As noted above, if G is

7.1 Asymptotic properties for motion groups

307

a noncompact abelian group, then it has no irreducible representations which vanish at infinity. In contrast, noncompact nonabelian groups can have irreducible representations which vanish at infinity. In light of Proposition 7.4, any such representation must be infinite-dimensional. Before stating the main theorem of this section, a positive result for motion groups, we need a vanishing-at-infinity property for the Fourier–Stieltjes transform of functions on spheres viewed as measures in the ambient Euclidean space. Let S n−1 denote the sphere of radius 1 in Rn . Let σ denote the rotationinvariant measure on S n−1 such that σ (S n−1 ) = 1. Let f ∈ L1 (S n−1 , σ ). Define  f (y)eiγ y dσ (y), Ff (γ ) = S n−1

n , which we consider as row vectors while Rn consists of column for γ ∈ R vectors. n ). Lemma 7.5 Let f ∈ L1 (S n−1 , σ ). Then F f ∈ C0 (R Proof We begin by recalling the following result for oscillatory integrals in one variable: see page 334 of Stein [145]. Suppose φ is real-valued and k-times continuously differentiable on (a, b), ψ is complex-valued and sufficiently differentiable on [a, b], and that |φ (k) (x)| ≥ 1, for all x ∈ (a, b). Then, for any λ > 0,   b    b   iλφ(x)   ≤ ck λ− 1k |ψ(b)| +  e ψ(x)dx |ψ (x)|dx (7.1)   a

a

holds when: (1) k ≥ 2, or (2) k = 1 and φ  (x) is monotonic. The bound ck is independent of φ, ψ, and λ. Clearly, Ff ∈ C(Rn ). It remains to show that F f vanishes at infinity. To start with, assume γ = (0, . . . , 0, γn ) and consider functions of the form h(y) = ˜ where θ is the angle between y and (0, . . . , 0, 1), [a, b] ⊆ χ[a,b] (θ )χE (y), ˜ θ) ∈ S n−1 is y in spherical coordinates, and E is a bounded [0, π ], y = (y, measurable subset of the first n − 1 angles. Fh(γ ) = Fh(0, . . . , 0, γn )  h(y)ei|γ | cos θ dσ (y) = S n−1  π  ˜ i|γ | cos θ (sin θ )n−2 dθ dσn−2 (y) ˜ = χ[a,b] (θ)χE (y)e  =

S n−2

0



b

˜ ˜ χE (y)dσ n−2 (y) S n−2

a

ei|γ | cos θ (sin θ )n−2 dθ,

308

Further applications

where dσn−2 represents the rotation-invariant measure on S n−2 . The first integral is bounded by the measure of S n−2 . On the second integral, we apply the estimate (7.1) with φ(θ) = cos θ, ψ(θ) = (sin θ)n−2 , and λ = |γ |. Note that it must be applied on three separate subintervals, near 0, π2 , and π , to assure the lower bounds of 1/2 on derivatives of cos θ. |F h(γ )| ≤ (c2 2|γ |−1/2 + c3 2|γ |− 3 ) + c2 2|γ |−1/2 )[1 + (n − 2)π ] 1

≤ C|γ |−1/3 ,

(7.2)

where the constant C is independent of h. If g is a linear combination of functions of the form h, then (7.2) implies |Fg(γ )| ≤ C g ∞ |γ |−1/3 ,

(7.3)

n . for all γ ∈ R Returning to f , let  > 0. Since step functions are dense in L1 (S n−1 , σ ), there exists g which is a linear combination of functions of the form h such that

g − f L1 (S n−1 , σ ) < /2. It follows that F g − Ff L∞ (Rn ) < /2. By (7.3), for any direction  ∈ S n−1 and r > 0, |Fg (r)| ≤ C g ∞ r −1/3 . Therefore, n with γ > [2C g ∞ /]3 , then if γ ∈ R |F f (γ )| ≤ Fg − Ff L∞ (Rn ) + |Fg (γ )| < . n ). Since  > 0 was arbitrary, Ff ∈ C0 (R



Using an explicit expression for the infinite-dimensional irreducible representations of a motion group enables expressing a matrix coefficient of such a representation as the Fourier–Stieltjes transform in the nature of the previous lemma. This is the key to the proof of the following theorem. Theorem 7.6 Let n ≥ 2 and let G = Rn  SO(n). Then any infinitedimensional irreducible representation of G vanishes at infinity. Proof We recall the description of Rn  SO(n) given in Example 4.42. G = {(x, S) : x ∈ Rn , S ∈ SO(n)} with the product given by (x, S)(y, T ) = (x + Sy, ST ), x, y ∈ Rn , S, T ∈ SO(n). We identify the normal subgroup N = Rn × {I } n by defining, for γ ∈ R n , χγ (x) = exp iγ x. With this  with R with Rn and N  is given by (x, S) · χγ = χγ S −1 . Thus the identification, the action of G on N  are spheres rS n−1 = {γ ∈ Rn : γ = r}, r ≥ 0. orbits in N Inducing two characters of the same sphere results in equivalent representations. Select re1 = (r, 0, . . . , 0) as a representative of rS n−1 . Let U r = indG N χre1 , for r > 0. Let H denote the subgroup of SO(n) which stabilizes re1 .

7.1 Asymptotic properties for motion groups

309

Then H can be identified with SO(n − 1) acting on the last n − 1 coordinates. n Since H stabilizes χre1 , indRRn H χre1 acts on L2 (H ) and   n indRRn H χre1 (x, S) = eire1 x λH (S) = χre1 × λH (x, S). Since every irreducible representation of the compact group H occurs as a subrepresentation of λH and, by the induction-in-stages theorem,     Rn H ind = indG χ U r = indG n n re 1 R H Rn H χre1 × λH , R  in Example 4.42 that every infiniteit follows from the description of G dimensional irreducible representation of G occurs as a subrepresentation of U r for some r > 0. Thus, it suffices to prove U r vanishes at infinity. Fix r. We will realize U r on L2 (SO(n)). In this semidirect product situation, the formula for the induced representation (Realization III) takes a particularly simple form. For (x, R) ∈ G, f ∈ L2 (SO(n)), S ∈ SO(n), U r (x, R)f (S) = eire1 S

−1

x

f (R −1 S).

Before calculating a coefficient function, we want to carefully write the Haar integral of SO(n) as an iterated integral over H and G/H = S n−1 . Define a measurable map P : S n−1 → SO(n) so that, for each ω ∈ S n−1 , ωP (ω) = e1 . For example, the matrix P (ω) can be constructed inductively by first rotating the first two coordinates of ω until the second coordinate becomes 0, then rotating the first and third coordinates of the result of the first step until the third coordinate is 0, etc. Then P provides a measurable cross-section of the H cosets in G. If μG and μH denote the normalized Haar measure on G = SO(n) and H ∼ = SO(n − 1), respectively, then for any integrable function ψ on G,    ψ(S)dμG (S) = ψ(P (ω)T )dμH (T )dσ (ω). S n−1

G

H

Let f ∈ L (SO(n)) = H(U ). Then, for (x, R) ∈ G, 2

r

r

U ϕf,f (x, R) = U r (x, R)f, f   −1 eire1 S x f (R −1 S)f (S)dμG (S) = G   −1 −1 = eire1 T P (ω) x f (R −1 P (ω)T )f (P (ω)T )dμH (T )dσ (ω) n−1 H  S irωx e f (R −1 P (ω)T )f (P (ω)T )dμH (T )dσ (ω), = S n−1

H

since T ∈ H fixes e1 and e1 P (ω)−1 = ω for each ω ∈ S n−1 . Define  gR (ω) = f (R −1 P (ω)T )f (P (ω)T )dμH (T ), H

310

Further applications

for ω ∈ S n−1 , R ∈ SO(n). Then gR ∈ L1 (S n−1 , dω) and gR 1 ≤ f 22 . Thus,  Ur ϕf,f (x, R) = eirωx gR (ω)dσ (ω) = F gR (rx). S n−1

Let  > 0. For each fixed R ∈ SO(n), Lemma 7.5 implies there exists a compact Ur (x, R)| < /2, for all x ∈ Rn \ CR . By compactness CR ⊆ Rn such that |ϕf,f of SO(n) and since R → gR from SO(n) into L1 (S n−1 , dω) is continuous, Ur there exists a compact subset C of Rn such that |ϕf,f (x, R)| < , for all (x, R) r U vanishes at infinity on G for any with x ∈ Rn \ C and R ∈ SO(n). Thus, ϕf,f f ∈ L2 (SO(n)). Therefore any infinite-dimensional irreducible representation of G vanishes at infinity.  We end the section by pointing out one of the consequences of Theorem 7.6. Recall that U(H) denotes the group of unitary operators on the Hilbert space H. Corollary 7.7 Let n ≥ 2 and let G = Rn  SO(n). For any irreducible representation π of G, π(G) is closed in U(H(π )) when U(H(π )) is equipped with the weak operator topology. Proof If π is finite-dimensional, then π can be considered as a representation of SO(n) composed with the quotient map of G onto SO(n). Since SO(n) is compact, π(G) is compact in the weak operator topology; hence a closed subset of U(H(π)). If π is infinite-dimensional, it follows from Theorem 7.6 that π(G) is closed when U(H(π)) is equipped with the weak operator topology. 

7.2 Projections in L 1 (G) For a locally compact group G, a function f ∈ L1 (G) is called a projection in L1 (G) if f ∗ f = f = f ∗ ; that is, if f is a self-adjoint idempotent in L1 (G). If π is a representation of L1 (G) and f is a projection in L1 (G), then π (f ) is a projection operator on H(π). In fact, if S is any faithful family of representations of L1 (G), then f ∈ L1 (G) is a projection if and only if π(f ) is a projection on H(π ) for each π ∈ S. Let PL1 (G) denote the set of all projections in L1 (G). Understanding PL1 (G) provides structural information about the algebra L1 (G), while a specific projection f combined with a concrete representation π constitutes an identity π(f ∗ f ) = π (f ) that can be useful.

7.2 Projections in L1 (G)

311

For certain groups G, the theory developed in Chapters 4 and 5 gives us  and its topology. Our goal in this section is to use detailed knowledge of G this knowledge to construct and analyze projections in L1 (G). We will see that the construction of a projection often starts with a coefficient function of an induced representation. The function f = 0 is a projection in L1 (G) for any G. For a discrete group G, δe is the identity for L1 (G) and, hence, a projection. We consider these to be trivial projections. We will introduce methods for constructing nontrivial projections in L1 (G). We begin with some elementary observations on PL1 (G) and then discuss the abelian and compact cases, which are relatively well known. The main body of this section is devoted to the study of certain classes of noncompact nonabelian locally compact groups G where the theory of Chapters 4 and 5 aid in the construction of nontrivial elements of PL1 (G). Remark 7.8 There is a natural partial order on PL1 (G). For f, g ∈ PL1 (G), we say f " g if f ∗ g = f . In that case, g ∗ f = f also and g − f ∈ PL1 (G). Definition 7.9 A minimal projection in L1 (G) is f ∈ PL1 (G), f = 0 such that if g ∈ PL1 (G) satisfies g " f then either g = 0 or g = f . Remark 7.10 If ψ is a character (one-dimensional representation) of G and f ∈ PL1 (G), then elementary calculations show that the pointwise product ψf is in PL1 (G) as well. For f ∈ PL1 (G), define  : π(f ) = 0}. Sf = {π ∈ G We call Sf the support of the projection f . Proposition 7.11 Let G be a locally compact group and f ∈ PL1 (G). Then  Sf is a compact open subset of G.  So Proof If f ∈ PL1 (G), then π(f ) ∈ {0, 1}, for any π ∈ G.  #  # 1 1 ∗ (G) : π(f ) ≥ ∗ (G) : π(f ) >   = π ∈C Sf = π ∈ C 2 2 is both compact and open (see 3.3.2 and 3.3.7 in Dixmier [37]).



 need not be Hausdorff, so Sf need For a general locally compact group G, G not be closed even though it is compact. Nevertheless, it is still very useful to formulate a statement ruling out the existence of nontrivial projections for many groups.

312

Further applications

Proposition 7.12 Let G be a locally compact group. If there are no proper  then there are no nontrivial projections in compact open subsets of G, L1 (G).  then the Fourier If G is an abelian locally compact group with dual group G, 1  Thus,  transform f → f is an injective ∗-homomorphism of L (G) into C0 (G). 1 2    if f is a projection in L (G), then [f (χ)] = f (χ ), for all χ ∈ G, so f(χ) must be 0 or 1. Then f = 1Sf . Proposition 7.13 Let G be an abelian locally compact group. The map f → Sf is a bijection between PL1 (G) and the collection of compact open subsets  of G. Proof By the discussion above, it is clear that f → Sf is a one-to-one map  It remains to of PL1 (G) into the collection of compact open subsets of G.  be compact and open. Since S0 = ∅, we may show this map is onto. Let C ⊆ G assume C = ∅.  This For each ψ ∈ C, ψ −1 C is a compact open neighborhood of 1 in G.   is a comimplies that G0 , the connected component of the identity in G,  Let q denote the quotient homomorphism of G  onto pact subgroup of G.     G/G0 . Since G/G0 is totally disconnected any neighborhood of the identity, such as q(ψ −1 C), contains a compact open subgroup (see [73], II.7.7),  such that we conclude that there exists a compact open subgroup Hψ of G −1 Hψ ⊆ ψ C. So ψHψ ⊆ C and C = ∪{ψHψ : ψ ∈ C}. Since C is compact, there exist ψ1 , . . . , ψn ∈ C such that C = ∪ni=1 ψi Hψi . Let H = ∩ni=1 Hψi . Then, for each i, H is an open subgroup of the compact group Hψi . This means that C is the exact union of the cosets of H which intersect nontrivially with it. By compactness of C again, there exist ω1 , . . . , ωm ∈ C such that C = ∪m i=1 ωi H and ωi H ∩ ωj H = ∅, 1 ≤ i < j ≤ m. Observe that, if f ∈ PL1 (G) and ψ is a character of G, then Sψf = ψ −1 Sf = {ψ −1 χ : χ ∈ Sf }. Let K = {x ∈ G : χ(x) = 1, for all χ ∈ H }. Since H is a compact open  K is a compact open subgroup of G by 1.85, 1.87, and 1.89. subgroup of G, Define f = |K|−1 1K . Clearly, f ∗ f = f = f ∗ . So f is a projection and Sf = H . For 1 ≤ i ≤ m, let fi = ωi−1 f . Then each fi is a projection with Sfi = ωi H . In particular, fi ∗ fj = 0 if i = j . Thus, g = f1 + · · · + fm ∈ PL1 (G) and  is the support set of a Sg = C. Therefore, any compact open subset of G 1  projection in L (G).

7.2 Projections in L1 (G)

313

These observations allow us to identify many abelian groups G for which  is connected, there are no nontrivial projections in L1 (G). If G is such that G for example G = Rn or Zn , then L1 (G) has no nontrivial projections.  is discrete and any If G is a compact (not necessarily abelian) group, then G  finite subset of G constitutes a compact open set. The orthogonality relations for coefficient functions of irreducible representations of G provide the key to constructing projections in L1 (G). For π and σ irreducible representations of the compact group G, and ξ1 , η1 ∈ H(π), ξ2 , η2 ∈ H(σ ), : .  1 η , η ξ , ξ  if π = σ 1 2 1 2 , (7.4) η1 , π (x)ξ1 η2 , σ (x)ξ2 dx = dπ 0 if π  σ G  we get, where dπ = dim π . By specializing to a single representation π ∈ G 1/2 for ξ1 = ξ2 = ξ ∈ H(π) with ξ = dπ and for any η1 , η2 ∈ H(π ),  η1 , π (x)ξ η2 , π (x)ξ dx = η1 , η2 .

(7.5)

G

This says that if we define Vξ η(x) = η, π(x)ξ , for x ∈ G, η ∈ H(π ), then Vξ is a unitary map of H(π) onto a closed subspace of L2 (G). We invite the reader to show that the range of Vξ is invariant under the left-regular representation and that Vξ intertwines π with a subrepresentation of the left-regular representation. But even more can be extracted from (7.4). For any ξ ∈ H(π), let g(x) = ξ, π (x)ξ , for all x ∈ G. Since g is continuous and G is compact, g ∈ L1 (G). Also note that g ∗ = g and consider g ∗ g. Applying (7.4), for any y ∈ G,  (7.6) g ∗ g(y) = ξ, π (x)ξ ξ, π (x −1 y)ξ dx G = ξ, π (x)ξ π(y)ξ, π (x)ξ dx G

ξ 2 1 ξ, π (y)ξ ξ, ξ  = g(y). = dπ dπ This tells us how to construct projections in L1 (G) when G is compact.  Let ξ ∈ Proposition 7.14 Let G be a compact group and let π ∈ G. 1/2 H(π), ξ = dπ . Define fξ (x) = ξ, π (x)ξ , for x ∈ G. Then the following hold. (i) fξ ∈ PL1 (G). (ii) Sfξ = {π }. (iii) π (fξ ) is the rank-one projection of H(π) onto Cξ .

314

Further applications

Proof (i) follows from (7.6) and the following verification that fξ is selfadjoint, using the fact that compact groups are unimodular. For x ∈ G, (fξ )∗ (x) = fξ (x −1 ) = ξ, π (x −1 )ξ  = π(x)ξ, ξ  = ξ, π (x)ξ  = fξ (x).  η1 , η2 ∈ H(σ ). Then (7.4) yields For (ii), let σ ∈ G,  σ (fξ )η2 , η1  = fξ (x)σ (x)η2 , η1 dx G = ξ, π (x)ξ σ (x)η2 , η1 dx G = ξ, π (x)ξ η1 , σ (x)η2 dx = 0,

(7.7)

G

if σ  π. On the other hand, (7.4) also yields, taking σ = π, η1 = η2 = ξ , π (fξ )ξ, ξ  = 1. Thus, Sfξ = {π }. Another application of (7.4) with σ = π, η2 ∈ H(π), η2 ⊥ ξ , and a calculation as in (7.7) with any η1 ∈ H(π), shows π (fξ )η2 = 0. This shows that π (fξ ) is the projection of H(π ) onto Cξ .  Clearly there is an abundance of projections in L1 (G) when G is compact. From Proposition 7.14(ii) and (iii), it is clear that each fξ as constructed is a minimal element of PL1 (G). If S is any compact open (that is, finite) subset of  select ξπ ∈ H(π), ξπ = dπ1/2 , for each π ∈ S. If π, σ ∈ S, π  σ , then G,

(7.4) implies fξπ ∗ fξσ = 0. Thus, f = π∈S fξπ is a projection in L1 (G) with Sf = S. We can use Mackey theory to find noncompact groups with open points in their duals. Let A be an abelian locally compact group and suppose that H is a locally compact group acting on A. Let G = A  H . The action of h ∈ H on  is given by (h · A is denoted a → h · a. The corresponding action of h on A −1  χ)(a) = χ (h · a), χ ∈ A. Let δ(h) denote the modulus of the automorphism of A given by h. Thus, for any integrable function g on A and h ∈ H ,   δ(h) g(h · a)da = g(a)da. A

A

Then δ is a continuous of H into R+ . The left Haar integral   homomorphism on G is given by H A f (a, h)δ(h)−1 da dh. It turns out that δ(h)−1 is the amount by which the action of h scales the  Haar measure of A.  Lemma 7.15 For h ∈ H and any integrable function ξ on A,   ξ (χ)dχ = δ(h)−1 ξ (h · χ)dχ .  A

 A

7.2 Projections in L1 (G)

315

 Proof As usual, Haar measures  on A and A are arranged so that the Plancherel  and formula holds. With h fixed, A ξ (h · χ)dχ is an invariant integral on A so must be proportional to the Haar integral. The constant of proportionality can be determined from one nonzero, non-negative, integrable function on  Select g ∈ Cc (A), g ≥ 0, g = 0; let gh (a) = g(h · a). Then ξ = | A. g |2 is a −1   Note that gh (χ) = δ(h)  g (h · χ), for χ ∈ A. nonzero integrable function on A. Then the Plancherel theorem gives    2 2 2 ξ (χ )dχ =  g = g = |g(a)| da = δ(h) |g(h · a)|2 da  A A A  2 2 = δ(h) gh = δ(h) gh = δ(h) |gh (χ)|2 dχ  A  g (h · χ)|2 dχ = δ(h) |δ(h)−1  A  −1 = δ(h) ξ (h · χ)dχ .  A



This suffices to prove the lemma.

We will make the strategic assumptions that H is σ -compact and that there  That is, assume there exists an ω ∈ A  such that exists a free open H -orbit in A. O = {h · ω : h ∈ H } is open and h · ω = ω implies h = e. Then h → h · ω is a homeomorphism of H onto O (see Proposition 4.6). Under these assumptions,  indG A ω is irreducible by Theorem 4.24 and defines an open point in G since O  Let us fix ω ∈ A  with these properties. is an open orbit in A. So H and O can be identified as topological spaces and even as left H -spaces. The map δ and Lemma 7.15 enable us to move the natural measure on O, the restriction of μA, to get a Haar measure on H . First, we define a linear map T of Cc (O) to Cc (H ) by T ξ (h) = δ(h)−1 ξ (h · ω), h ∈ H , for any ξ ∈ Cc (O). Clearly, T is a bijection. Now let ϕ : Cc (H ) → C be given by ϕ(g) = O T −1 g(χ)dχ . Lemma 7.16 With the above notation, ϕ is a nonzero left-translation-invariant positive linear functional on Cc (H ). Proof All parts are clear except for the translation invariance. For k ∈ H let Lk g(h) = g(k −1 h), h ∈ H , for all g ∈ Cc (H ) and ξk (χ) = ξ (k · χ ), χ ∈ O, for ξ ∈ Cc (O). Then Lk T ξ = δ(k)T ξk−1 . Thus, for any ξ ∈ Cc (O) and any k ∈ H ,  ξk−1 (χ)dχ ϕ(Lk T ξ ) = δ(k)ϕ(T ξk −1 ) = δ(k) O   = δ(k) ξ (k −1 · χ)dχ = ξ (χ )dχ = ϕ(T ξ ), O

O

by Lemma 7.15. Since T is onto, ϕ is left-translation-invariant on Cc (H ).



316

Further applications

The uniqueness of left Haar integration and Lemma 7.16 mean a left Haar measure on H can be selected so that, for any ξ ∈ Cc (O),   ξ (χ)dχ = ξ (h · ω)δ(h)−1 dh. (7.8) O

H

With these preliminaries, we can now turn attention to indG A ω, the irreducible representation of G associated with the orbit O. Our goal now is to construct a projection in L1 (G) with this irreducible as its support. 2 2 ω Realize U ω = indG A ω on L (H ). For (a, h) ∈ G and f ∈ L (H ), U (a, h)f is given by U ω (a, h)f (k) = (k · ω)(a)f (h−1 k), for k ∈ H . It turns out to be very illuminating to use (7.8) to move U ω over to L2 (O, μA). For ξ ∈ L2 (O, μA) define W ξ on H by W ξ (h) = δ(h)−1/2 ξ (h · ω), h ∈ H . Using (7.8), it is straightforward to show that W is a unitary map of L2 (O, μA) onto L2 (H ). Define πO to be the representation U ω transferred by W . That is, for (a, h) ∈ G, πO (a, h) = W −1 U ω (a, h)W . We write L2 (O, μA) simply as  consisting of those L2 (O) and identify it with the closed subspace of L2 (A) functions supported on O. Proposition 7.17 Let G = A  H , where A is an abelian locally compact group and H is a σ -compact locally compact group acting on A so that  Then the unique element of G  there exists an open free H -orbit O in A. 2 whose restriction to A lives on O is given by πO which acts on L (O) by, for (a, h) ∈ G, ξ ∈ L2 (O), πO (a, h)ξ (χ) = δ(h)1/2 χ(a)ξ (h−1 · χ ),  for all χ ∈ O. Moreover, πO is an open point in G. Proof The formula for πO is verified by the following calculation. For (a, h) ∈ G, ξ ∈ L2 (O), and fixed χ ∈ O, let k ∈ H be such that k · ω = χ. Then πO (a, h)ξ (χ) = W −1 U ω (a, h)W ξ (k · ω) = δ(k)1/2 U ω (a, h)W ξ (k) = δ(k)1/2 (k · ω)(a)W ξ (h−1 k) = δ(k)1/2 (k · ω)(a)δ(h−1 k)−1/2 ξ (h−1 k · ω) = δ(h)1/2 χ(a)ξ (h−1 · χ).  whose restriction The uniqueness of πO (hence of U ω ) as the only element of G to A lives on O follows from Theorem 4.24. The fact that πO is an open point  follows from continuity of restriction (Theorem 5.37). in G 

7.2 Projections in L1 (G)

317

Definition 7.18 Let L be a locally compact group. An irreducible representation σ of L is called square-integrable if there exists a pair of nonzero vectors σ ∈ L2 (L). ξ, η ∈ H(σ ) such that ϕξ,η The distinguished representation πO happens to be square-integrable. This has far-reaching implications and we formulate it as a theorem. The proof is a direct generalization of that applied to the affine group in Proposition 2.33 to show that certain infinite-dimensional representations were irreducuble. Theorem 7.19 Let A be an abelian locally compact group, let H be a σ compact locally compact group acting on A, and let G = A  H . Suppose that  The irreducible representation of G whose O is an open free H -orbit in A.  restriction to A lives on O is square-integrable. Proof In light of Proposition 7.17, we need to show that πO , whose formula is given in that proposition, is square-integrable. We start with some notation. For any measurable function ξ on O and χ ∈ O, let ξ χ (h) = ξ (h−1 · χ ), for all h ∈ H . Then ξ χ is a measurable function on H . Note that left invariance of the Haar integral on H implies that   χ 2 |ξ (h)| dh = |ξ ψ (h)|2 dh, (7.9) H

H

for all χ , ψ ∈ O. To show that πO is square-integrable, we must show there exist nonzero πO ∈ L2 (G). Let us calculate, for any ξ, η ∈ L2 (O), ξ, η ∈ L2 (O) such that ϕξ,η   πO |ϕξ,η (a, h)|2 d(a, h) = |π(a, h)ξ, η|2 d(a, h) G G = |η, π(a, h)ξ |2 d(a, h) G

2      η(χ)δ(h)1/2 χ(a)ξ (h−1 · χ )dχ  d(a, h). =   G

O

(7.10) For each h ∈ H , let vh (χ) = η(χ)ξ (h−1 · χ), if χ ∈ O and vh (χ) = 0 for χ ∈  Let vh∨ denote the inverse Fourier transform of vh  \ O. Then vh ∈ L1 (A). A  on A. That is, vh∨ (a) = A vh (χ)χ(a)dχ , for a ∈ A. Note that vh∨ 2 = vh 2 by the Plancherel theorem even if one of the sides, and hence both, is ∞. Let  C= |ξ (h−1 · χ)|2 dh ∈ [0, ∞]. H

318

Further applications

Note that C is independent of χ by (7.9). Then (7.10) continues as   πO 2 |ϕξ,η (a, h)| d(a, h) = δ(h)|vh∨ (a)|2 d(a, h) G G   = |vh∨ (a)|2 da dh H A = |vh (χ)|2 dχ dh (7.11)  H A   |η(χ)|2 |ξ (h−1 · χ )|2 dχ dh =  H A    = |η(χ)|2 |ξ (h−1 · χ )|2 dh dχ O

H

= C η 22 . If ξ is any nonzero element of Cc (O) then C = Hence, πO is a square-integrable representation.

 H

|ξ (h−1 · χ )|2 dh < ∞. 

 by Remark 7.20 If we define a representation π of G on L2 (A) π(a, h)ξ (χ ) = δ(h)1/2 χ(a)ξ (h−1 · χ ),  and (a, h) ∈ G, then obviously L2 (O) is a π -invariant  ξ ∈ L2 (A), for χ ∈ A,  subspace and πO is just π restricted to that subspace. Let F : L2 (A) → L2 (A) denote the Plancherel transform. Using this unitary, we can move π over to a representation acting on L2 (A). For (a, h) ∈ G, let ρ(a, h) = F −1 π (a, h)F. A quick computation shows that ρ(a, h)g(b) = δ(h)−1/2 g(h−1 · (a −1 b)), for any b ∈ A, g ∈ L2 (A), and (a, h) ∈ G. This ρ is just the quasi-regular representation formed by inducing the trivial character up from H . Let H2 (O) = {f ∈ L2 (A) : f ∈ L2 (O)}. Then H2 (O) is a ρ-invariant subspace and ρO , formed by restricting ρ to H2 (O), is another representation in the equivalence class of the square-integrable indG A ω for ω ∈ O. Returning to our analysis of πO , it is clear that something special may happen if the C which arises in (7.11) is 1. It is also clear, by picking any nonzero function in Cc (O) and appropriately normalizing, that it is easy to get C to be 1. This leads us to make the following definitions.  Definition 7.21 Fix a function ξ ∈ L2 (O) such that H |ξ (h−1 · χ )|2 dh = 1 for some, and hence every, χ ∈ O. Let Tξ : L2 (O) → L2 (G) be the map given πO by Tξ η = ϕξ,η , for all η ∈ L2 (O). That is, Tξ η(x) = η, πO (x)ξ , for x ∈ G. Let Kξ denote the range of Tξ .

7.2 Projections in L1 (G)

319

Proposition 7.22 Let A be an abelian locally compact group, let H be a σ compact group acting on A, and let G = A  H . Suppose that O is an open −1  Let ξ ∈ L2 (O) be such that · χ )|2 dh = 1 and let free H -orbit in A. H |ξ (h Tξ and Kξ be as in Definition 7.21. Then (i) Kξ is a closed subspace of L2 (G) invariant under the left-regular representation λG of G. (ii) Tξ is a unitary map of L2 (O) onto Kξ which intertwines πO with λG .  Proof Since H |ξ (h−1 · χ)|2 dh = 1, (7.11) implies Tξ is an isometry into L2 (G). This implies Kξ = Tξ L2 (O) is a closed subspace and Tξ : L2 (O) → Kξ is a unitary map. Suppose η ∈ L2 (O) and x ∈ G. Then λG (x)Tξ η(y) = Tξ η(x −1 y) = η, πO (x −1 y)ξ  = πO (x)η, πO (y)ξ  = Tξ πO (x)η, for all y ∈ G. Then both (i) and (ii) follow.

(7.12) 

Remark 7.23 With ξ as in Proposition 7.22 and any η1 , η2 ∈ L2 (O), then Tξ being unitary says  η1 , πO (x)ξ η2 , πO (x)ξ dx = η1 , η2 . (7.13) G

Compare with equation (7.5). Using the full form of the Haar integral on G = A  H and applying the complex conjugate to the inner product, (7.13) becomes   η1 , πO (a, h)ξ πO (a, h)ξ, η2 δ(h)−1 dadh = η1 , η2 . (7.14) H

A

This remark will be useful in the next section. In analogy with Proposition 7.14, we could let g(x) = ξ, πO (x)ξ , where  ξ ∈ L2 (O) satisfies H |ξ (h−1 · χ)|2 dh = 1. Then, if g is integrable, the identities above show that g ∗ g = g. However, g will not be self-adjoint unless G is unimodular. Recall from Section 1.1 that the modular function on G = A  H is given by G (a, h) = H (h)δ(h)−1 , (a, h) ∈ G. Thus, let fξ (x) = G (x)−1/2 ξ, πO (x)ξ , for x ∈ G. Again, assuming we know fξ ∈ L1 (G), we have for x ∈ G, fξ∗ (x) = G (x)−1 fξ (x −1 ) = G (x)−1 G (x −1 )−1/2 πO (x −1 )ξ, ξ  = G (x)−1/2 ξ, πO (x)ξ  = fξ (x).

(7.15)

320

Further applications

To make the issue of integrability of fξ clearer, use the A × H parametrization of G and for h ∈ H , let ξh (χ) = ξ (h−1 · χ) for χ ∈ O. Then fξ (a, h) = δ(h)1/2 H (h)−1/2 ξ, πO (a, h)ξ   = δ(h)H (h)−1/2 ξ (χ)χ(a)ξ (h−1 · χ )dχ O  = δ(h)H (h)−1/2 (ξ ξh )(χ )χ(a)dχ

(7.16)

O

= δ(h)H (h)−1/2 (ξ ξh )∨ (a). Definition 7.24 For ξ ∈ L2 (O), we call ξ a projection-generating function (PGF) associated with O if  (i) H |ξ (h−1 · χ)|2 dh = 1, for any χ ∈ O, and (ii) fξ ∈ L1 (G), where fξ (a, h) = δ(h)H (h)−1/2 (ξ ξh )∨ (a), for all (a, h) ∈ G. Proposition 7.25 Let A be an abelian locally compact group, let H be a σ compact group acting on A, and let G = A  H . Suppose that O is an open free  Let ξ ∈ L2 (O) be a PGF associated with O. Then fξ ∈ PL1 (G). H -orbit in A. Proof By (7.15), fξ∗ = fξ . To see that fξ is an idempotent, let (a, h) = δ(h)H (h)−1/2 . Note that  : G → (0, ∞) is a homomorphism. Then fξ (x) = (x)ξ, πO (x)ξ , for x ∈ G, in convenient notation. Using (7.13),  fξ (y)fξ (y −1 x)dy fξ ∗ fξ (x) = G  = (y)ξ, πO (y)ξ (y −1 x)ξ, πO (y −1 x)ξ dy G  = (x) ξ, πO (y)ξ ξ, πO (y −1 x)ξ dy G = (x) ξ, πO (y)ξ πO (y)ξ, πO (x)ξ dy G

= (x)ξ, πO (x)ξ  = fξ (x). Thus, fξ is a projection in L1 (G).



The most important thing at this point is to show that there actually exist projection-generating functions. In fact, they exist in abundance. Theorem 7.26 Let A be an abelian locally compact group, let H be a σ compact group acting on A, and let G = A  H . Suppose that O is an open

7.2 Projections in L1 (G)

321

 The set of projection-generating functions associated with free H -orbit in A. O is a total set in L2 (O). Proof Let C and D be any two compact subsets of O satisfying ∅ = C o and C ⊆ D o , where S o denotes the interior of the set S. Let ω ∈ C o . Since L1 (A) is a regular commutative Banach algebra, there exists a g0 ∈ L1 (A) such that 1C ≤ g0 ≤ 1D (see [86], theorem 4.4.14 and corollary  1/2 g0 (h−1 · ω)|2 dh and let g = β −1 g0 . Then ξ =  g satis4.2.9).  Set β−1= H2| fies H |ξ (h · χ)| dh = 1. We will show that ξ is a PGF. To that end, note that, since h → h−1 · ω is a homeomorphism of H onto O, there is a compact subset K of H such that ξ ξh = 0 if h ∈ H \ K. Define, for h ∈ H , gh (a) = g(h−1 · a), a ∈ A. Then gh ∈ L1 (A), for all h, and the following properties all follow from routine calculations. (a) (b) (c) (d)

gh = δ(h)ξh .

gh L1 (A) = δ(h) g L1 (A) . [δ(h)−1 g ∗ (gh )∗ ]∧ = ξ ξh . g ∗ (gh )∗ = 0 for h ∈ H \ K.

Thus fξ (a, h) = δ(h)H (h)−1/2 (ξ ξh )∨ (a) = H (h)−1/2 (g ∗ (gh )∗ )(a), for (a, h) ∈ G. Therefore    |fξ (x)|dx = H (h)−1/2 |(g ∗ (gh )∗ )(a)|δ(h)−1 dadh G H A  = H (h)−1/2 g ∗ (gh )∗ L1 (A) δ(h)−1 dh K  ≤ g L1 (A) H (h)−1/2 gh L1 (A) δ(h)−1 dh K  2 = g L1 (A) H (h)−1/2 dh < ∞, K

−1/2 H

is continuous and K is compact. We used property (b) for the last since equality. Therefore, fξ ∈ L1 (G) and, thus, ξ is a PGF. From the general nature of the pairs of nested compact sets, C and D, it is clear that the linear span of the ξ  constructed in this way is dense in L2 (O). We wish to show that fξ is a minimal projection in L1 (G) for any PGF  associated with an open free orbit in A.

322

Further applications

Lemma 7.27 Let A be an abelian locally compact group, let H be a locally  compact group acting on A, and let G = A  H . Then {indG A χ : χ ∈ A} is a faithful family of representations of L1 (G). Proof This follows from continuity of inducing, Theorem 5.39, and the fact that the regular representation of A is weakly equivalent to the set of characters of A.  χ 2 Recall that we can realize indG A χ as U acting on L (H ) by

U χ (a, h)ϕ(k) = (k · χ )(a)ϕ(h−1 k), for k ∈ H, ϕ ∈ L2 (H ), and (a, h) ∈ G. It will be convenient to shift to the right-handed form via J : L2 (H ) → L2 (H ) given by J ϕ(h) = H (h)−1/2 ϕ(h−1 ), for h ∈ H, ϕ ∈ L2 (H ). Then J is a unitary map and J 2 = I . Let T χ (a, h) = J U χ (a, h)J for each (a, h) ∈ G. Then T χ is a representation of G equivalent to indG A χ whose explicit action is calculated to be T χ (a, h)ϕ(k) = (h)1/2 (k −1 · χ )(a)ϕ(kh),

(7.17)

for k ∈ H, ϕ ∈ L2 (H ), and (a, h) ∈ G. Proposition 7.28 Let A be an abelian locally compact group, let H be a σ -compact group acting on A, and let G = A  H . Suppose that O is an  Let ξ ∈ L2 (O) be a PGF associated with O. Recall open free H -orbit in A.  For χ ∈ A,  let T χ be as in (7.17). ξ χ (h) = ξ (h−1 · χ ), h ∈ H, χ ∈ A.  \ O, T χ (fξ ) = 0. (i) For any χ ∈ A (ii) For any ω ∈ O, T ω (fξ ) is the rank-one projection of L2 (H ) onto Cξ ω .  and ϕ, ψ ∈ L2 (H ), compute Proof For any χ ∈ A T χ (fξ )ϕ, ψ  fξ (a, h)T χ (a, h)ϕ, ψd(a, h) = G    −1/2 ∨ = (h) (ξ ξh ) (a) T χ (a, h)ϕ(k)ψ(k) dk dadh H A H    = (ξ ξh )∨ (a)(k −1 · χ)(a) da ϕ(kh)ψ(k) dhdk H H A ϕ(kh)ψ(k)ξ (k −1 · χ)ξh (k −1 · χ)dkdh. = H

H

(7.18)

7.2 Projections in L1 (G)

323

 \ O, then T χ (fξ )ϕ, ψ = 0 for all ϕ, ψ ∈ L2 (H ) Since ξ is 0 off O, if χ ∈ A χ and thus T (fξ ) = 0. On the other hand, if ω ∈ O, then (7.18) becomes   T ω (fξ )ϕ, ψ = ϕ(kh)ψ(k)ξ (k −1 · ω)ξh (k −1 · ω)dkdh H H    −1 ϕ(kh)ξ ((kh) · ω) dh ξ (k −1 · ω)ψ(k) dk (7.19) = H

H ω

= ϕ, ξ ξ ω , ψ. Thus, T ω (fξ ) is the projection of L2 (H ) onto the one-dimensional subspace spanned by ξ ω .  Corollary 7.29 With the notation of Proposition 7.28, for any PGF ξ , associated with O, fξ is a minimal projection in L1 (G). \ O Proof If f ∈ PL1 (G) satisfies f " fξ , then T χ (f ) = 0 for any χ ∈ A since f = f ∗ fξ . While for a fixed ω ∈ O, T ω (f ) ≤ T ω (fξ ). Since T ω (fξ ) is rank one, either T ω (f ) = 0 or T ω (f ) = T ω (fξ ). Since T χ ∼ T ω for all χ ∈ O,  or T χ (f ) = T χ (fξ ) for all χ ∈ A.  By Lemma either T χ (f ) = 0 for all χ ∈ A 7.27, either f = 0 or f = fξ . Thus, fξ is a minimal element of PL1 (G).  There are important examples where the hypotheses of Proposition 7.28 hold. Example 7.30 For the affine group Gaff = R  R+ , there are two open free R, O + = (0, ∞) and O − = (−∞, 0). Note that O + ∪ O − is coR+ -orbits in  + −   null in R. Then π + = π O and π − = π O are open points in G aff . We will use the theory developed above to construct explicit minimal projections in L1 (Gaff ). R = {χγ : γ ∈ R}, where χγ (t) = Write Gaff = {(t, a) : t ∈ R, a ∈ R+ } and  2πiγ t e for t ∈ R. As usual, we write  g (γ ) for  g (χγ ). Let O = (0, ∞) or (−∞, 0). g ) is a compact subset of O and Let L1 (R) ∩ L2 (R) be such that supp(  ∞ g ∈ −1 2 | g (a γ )| da = 1 for some, hence any, γ ∈ O. Let ga (t) = g(a −1 t), for 0 + t ∈ R, a ∈ R . Define fg (t, a) = (g ∗ (ga )∗ )(t), for all (t, a) ∈ Gaff . Then fg is a minimal projection in L1 (Gaff ) by Corollary 7.29. In the rest of the examples, we leave it to the reader to work out explicit expressions for minimal projections.

324

Further applications

Example 7.31 Let N = R2 and fix c ∈ R, c = 0. Let  #  a 0 Hc = : a, b ∈ R, a > 0 b ac act on N by the natural matrix action. For a row vector γ = (γ1 , γ2 ) ∈ R2 , χγ is the character of N given by χγ (x) = exp iγ x. Then for h ∈ Hc , h · χγ = χγ h−1 and there are two open free H -orbits O + = Hc (χ(0,1) ) and O − = Hc (χ(0,−1) ), the upper and lower open half planes. Example 7.32 Let n be a positive integer. Let N = M(n, R) considered as 2 a group under addition, so N is just Rn . Let H = GL(n, R) and have H  by, for act on N by left matrix multiplication. We can identify N with N  A ∈ N, χA (B) = exp i tr(AB), for all B ∈ N . Then, for h ∈ H and χA ∈ N, h · χA = χAh−1 . Let O = {A ∈ N : det A = 0}. Then O is an open free H -orbit which is co-null  in N. Example 7.33 The Fell group introduced in Example 4.43 is N  K where N is the additive group of p-adic numbers, for a fixed prime number p, and K is a  is the union of the trivial orbit and countably many compact group such that N open free K-orbits. It followed that the Fell group is a noncompact locally compact group with a countable dual. There are examples of locally compact groups of the form A  H with A abelian which have nontrivial compact open sets in their dual that are not singletons or finite sets. We will not explore the most general case, but will show how to construct projections in a particularly important family of such groups. Let n be a positive integer and let M be a fixed n × n matrix with real entries. Let B = eM . Then, the power B t = etM is defined for all t ∈ R and {B t : t ∈ R} is a one-parameter subgroup of GL(n, R). We consider Rn as consisting of column vectors and B t acting on it by matrix product. If δ = det B, then δ > 0. Let GB = Rn B R = {(x, t) : x ∈ Rn , t ∈ R} with product given by (x, t)(y, s) = (x + B t y, t + s). Then (x, t)−1 = (−B −t x, −t) and the left Haar integral on GB is given by    g(x, t)d(x, t) = g(x, t)δ −t dxdt. GB

R

Rn

The modular function on GB is given by GB (x, t) = δ −t , for all (x, t) ∈ GB .

7.2 Projections in L1 (G)

325

Many calculations of the free open orbit case above carry through to this situation. Indeed, parts can be simplified because the acting group is now R (unimodular and abelian). But there are now some significant differences. Of course, there will be no nonzero projections in L1 (GB ) unless there are compact B . open subsets of G For n = 2, we studied the duals of various GB in Chapter 5. From those B exactly when both descriptions, we observe that compact open sets exist in G eigenvalues of B either lie completely inside or outside the unit circle in the complex plane. Definition 7.34 An n × n real matrix X is called a dilation matrix if all of the eigenvalues of X have absolute value greater than 1. For the construction of projections, we will be interested in GB when either B or B −1 is a dilation. Since both situations work out identically, we will usually just ask for B to be a dilation. To carry out an analysis of the dual of GB , we identify Rn with the obvious n with the space of row vectors γ = normal subgroup of GB . We identify R 2πiγ x (γ1 , . . . , γn ) via χγ (x) = e , for all x ∈ Rn . The induced action of t ∈ R −t n  on R is given by t · γ = γ B . Dilation matrices are not necessarily strictly expanding with respect to n . However, for a given dilation matrix B, there Euclidean distance on R n so that B is strictly expanding. One example is given by are norms on R defining ⎞1/2 ⎛ ∞  −j 2

γ B = ⎝

γ B 2 ⎠ , j =0

for all γ ∈ Rn . The series converges by the root test and an application of the spectral radius formula to B −1 . Notice that ⎞1/2 ⎛ ⎞1/2 ⎛ ∞ ∞  

γ B −j +1 22 ⎠ = ⎝ γ B 22 +

γ B −j 22 ⎠ > γ B ,

γ B B = ⎝ j =0

j =0

if γ is nonzero. Let SB = {γ ∈ Rn : γ B = 1}. As the boundary of the unit ball for a norm on Rn , SB is homeomorphic to the (n − 1)-sphere. Moreover, for any nonzero n , there are unique γ0 ∈ SB and t ∈ R such that γ = γ0 B −t . That is, SB γ ∈R n . We can now describe G B as a is a cross-section of the nonzero R-orbits in R topological space.

326

Further applications

n , let U γ = indGnB γ . We will realize U γ on L2 (R) via the For each γ ∈ R R formula U γ (x, t)ϕ(s) = e2πiγ B

−s

x

ϕ(s − t),

for all s ∈ R, ϕ ∈ L2 (R), (x, t) ∈ GB . B whose Let C = {U γ : γ ∈ SB }. Then C exhausts all those elements of G n is nontrivial. Of course, all of R stabilizes 0, so associated orbit in R B associated with the 0 orbit,  = {χ˜ : χ ∈  R} exhausts the elements of G B by routine where χ(x, ˜ t) = χ(t), for all (x, t) ∈ GB . We can now describe G application of the results of Chapters 4 and 5. Proposition 7.35 Let B be an n × n dilation matrix and adopt the above B = C ∪ ,  is a closed subset homeomorphic to R and C notation. Then G is a compact open subset homeomorphic to the (n − 1)-sphere. For any χ˜ ∈  and U γ ∈ C, χ˜ is in the closure of {U γ }. B has a nontrivial compact open So if B (or B −1 ) is a dilation matrix, G M B subset. Conversely, if B = e , for some general n × n real matrix M and G has a nonempty compact open subset, then either B or B −1 is a dilation matrix. n when B has eigenvalues This can be shown by an analysis of the R-orbits in R on the unit circle or at least one inside and one outside the unit circle. We will spare the reader the details. Fix a dilation matrix B for the rest of this section. Guided by Remark 7.20, we consider the quasi-regular representation ρ of GB acting on L2 (Rn ). Let H = {(0, t) : t ∈ R}, a closed subgroup of GB . The map x → (x, 0)H is an identification of Rn with GB /H and the corresponding action of (x, t) ∈ GB on the coset space Rn is given by (x, t) · y = x + B t y, for y ∈ Rn . The Lebesgue measure on Rn is quasi-invariant. For g ∈ L2 (Rn ), (x, t) ∈ GB , and y ∈ Rn , let ρ(x, t)g(y) = δ −t/2 g(B −t (y − x)). n ), The Fourier transform, considered as a unitary map from L2 (Rn ) to L2 (R n ) given by intertwines ρ with the equivalent representation, π , on L2 (R π(x, t)ξ (γ ) = δ t/2 e2πiγ x ξ (γ B t ), n , ξ ∈ L2 (R n ), and (x, t) ∈ GB . for all γ ∈ R Although there are no open orbits when n > 1, we are able to imitate to a certain extent the construction of projections we gave for the open free orbit

7.2 Projections in L1 (G)

327

case by using a modified coefficient function of the associated irreducible representation in that case. Let us experiment. n ) and guided by (7.15), consider the coefficient function fξ For ξ ∈ L2 (R defined by fξ (x, t) = δ t/2 ξ, π (x, t)ξ , for all (x, t) ∈ GB . We verify that fξ will be self-adjoint if it is integrable. For any (x, t) ∈ GB , fξ∗ (x, t) =

1 δ −t/2 fξ ((x, t)−1 ) GB (x, t)

= δ t/2 ξ, π (x, t)∗ ξ  = δ t/2 ξ, π (x, t)ξ  = fξ (x, t). Thus, if we can establish conditions on ξ that ensure fξ ∈ L1 (GB ), we will have fξ∗ = fξ . In order to make the nature of fξ more explicit, we define ξt as ξ dilated by n and t ∈ R. Note that the action of t. That is, ξt (γ ) = ξ (γ B t ), for all γ ∈ R 2  1  n n ξ ∈ L (R ) implies that ξ ξt ∈ L (R ), for every t ∈ R. Then  fξ (x, t) = δ t/2 Rn ξ (γ )π(x, t)ξ (γ )dγ = δ t Rn ξ (γ )e−2πiγ x ξ (γ B t )dγ = δ t (ξ ξt )∨ (x), for all (x, t) ∈ GB . Now (ξ ξt )∨ ∈ C0 (Rn ). With the aim of forcing fξ into 1 ∨ 1 n L (GB ), ∨we require that (ξ ξt ) ∈ L (R ), for almost all t ∈ R, and that R (ξ ξt ) 1 dt < ∞. Under these mild assumptions on ξ , we have    |fξ (x, t)|d(x, t) = |fξ (x, t)|δ −t dxdt GB R Rn   |(ξ ξt )∨ (x)|dxdt < ∞. = R

Rn

We note that the assumptions on ξ would be satisfied if ξ is a continuous n . function of compact support and is 0 on a neighborhood of 0 in R 1 So now we know how to select ξ so that fξ ∈ L (GB ). We turn to a faithful family of representations, {U γ : γ ∈ SB }, of L1 (GB ) to help us determine when n , let ξ γ (t) = ξ (γ B −t ), fξ is a projection. To facilitate notation, for each γ ∈ R γ for all t ∈ R. Assume that each ξ is square-integrable. Lemma 7.36 For any γ ∈ SB and ϕ, ψ ∈ L2 (R), U γ (fξ )ϕ, ψ = ϕ, ξ γ ξ γ , ψ.

328

Further applications

Proof

 U (fξ )ϕ, ψ = γ

U γ (fξ )ϕ(s)ψ(s)ds    = fξ (x, t)[U γ (x, t)ϕ(s)]ψ(s)δ −t dxdtds n R R R −s (ξ ξt )∨ (x)e2πiγ B x ϕ(s − t)ψ(s)dxdtds = n R R R = (ξ ξt )(γ B −s )ϕ(s − t)ψ(s)dtds R R   = ξ (γ B −s )ψ(s) ϕ(s − t)ξ (γ B t−s )dtds R  R γ = ξ (s)ψ(s) ϕ(t)ξ γ (t)dtds R

R

R

= ϕ, ξ γ ξ γ , ψ. This establishes the claim.



This shows that for quite general ξ ∈ L2 (R), U γ (fξ ) is a multiple of a rankone projection (the multiple being ξ γ 22 ) in L2 (R), for every γ ∈ SB . Theorem 7.37 Let B = eM for a real n × n matrix M and suppose that B is n ) satisfy a dilation matrix. Let ξ ∈ L2 (R ∨ 1 n (i) (ξ  ξt ) ∈ ∨L (R ), for almost all t ∈ R, (ii) R (ξ ξt ) 1 dt < ∞, n \ {0}. (iii) R |ξ (γ B t )|2 dt = 1, for all γ ∈ R

Then fξ (x, t) = δ t/2 ξ, π (x, t)ξ  defines a projection, fξ , in L1 (GB ). The supB consists of the set of infinite-dimensional irreducible repreport of fξ in G n \ {0}, U γ (fξ ) is the projection of L2 (R) sentations. Moreover, for each γ ∈ R γ onto the span of ξ . Proof Taking into account Lemma 7.36 and the calculations that preceded it, we only need to observe that property (iii) in the theorem is equivalent to ξ γ 2 = 1, for all γ ∈ SB , by invariance of the Lebesgue measure under translation and inversion, and that there is only one nonempty compact open B . subset of G  n ). We Definition 7.38 Let B = eM be an n × n dilation matrix and ξ ∈ L2 (R call ξ a projection-generating function relative to B (PGFB ) if properties (i), (ii), and (iii) of Theorem 7.37 are satisfied. Proposition 7.39 Let n ≥ 2 and let B = eM be an n × n dilation matrix. Let ξ be any PGFB . Then fξ is a minimal projection in L1 (GB ).

7.3 Generalizations of the wavelet transform

329

Proof Let f ∈ L1 (GB ) be a nonzero projection such that fξ ∗ f = f . Since n ≥ 2, the (n − 1)-sphere is connected; so there is only one nonempty compact B , Sf = {π ∈ G B : π (f ) = 0} is B . The support of f in G open subset C of G γ a compact open set, so it equals C. That is U (f ) = 0, for every γ ∈ SB . Since fξ ∗ f = f , 0 = U γ (f ) ≤ U γ (fξ ). But each U γ (fξ ) is a rank-one B , projection. Thus, U γ (f ) = U γ (fξ ), for every γ ∈ SB . Since C is dense in G  we must have f = fξ . We turn to the issue of existence of PGFB s. The following theorem shows that there are many projection-generating functions and many distinct projections in L1 (GB ). Moreover, the constructive nature of the proof can be useful in its n ). own right. Let QB denote the set of all PGFB s in L2 (R Theorem 7.40 Let B = eM be an n × n dilation matrix. For any γ ∈ SB , {ξ γ : ξ ∈ QB } is a total set in L2 (R). Proof Let I and J be any bounded open intervals in R with I ⊆ J . For any subset X of R, let SB · X = {γ B −t : γ ∈ SB , t ∈ X}. Then n \ {0}. SB · I ⊆ SB · I ⊆ SB · J ⊆ SB · J ⊆ R Moreover, SB · I is compact and SB · J is open. n ) be such that 1S ·I ≤ η ≤ 1SB ·J . For γ ∈ R n \ {0}, let Let η ∈ Cc∞ (R B  σ (γ ) = |η(γ B t )|2 dt. R

n \ {0} that is constant on R-orbits. Moreover, Then σ is a C -function on R |I | ≤ σ ≤ |J |, where |I | denotes the length of I . n \ {0} and n by ξ (γ ) = σ (γ )−1/2 η(γ ) for γ ∈ R Define a function ξ on R ∞  n ξ (0) = 0. Then ξ ∈ Cc (R ) and ξ ξt = 0 except for t in some compact set. Thus, one easily verifies that properties (i), (ii), and (iii) of a PGFB hold. Therefore ξ ∈ QB . For any γ ∈ SB and ξ as constructed, |I |−1/2 1I ≤ ξ γ ≤ |J |−1/2 1J . So it is clear that the characteristic function of any bounded interval can be approximated in L2 (R) by multiples of elements of QB . Thus, {ξ γ : ξ ∈ QB } is a total set in L2 (R).  ∞

7.3 Generalizations of the wavelet transform Let us return to the general setup of a σ -compact locally compact group H acting on an abelian locally compact group A in such a manner that there  The natural representation of the semidirect exists a free open H -orbit O in A.

330

Further applications

product G = A  H on L2 (A) is the quasi-regular representation ρ given by ρ(a, h)g(b) = δ(h)−1 g(h−1 · (a −1 b)), for any b ∈ A, g ∈ L2 (A), and (a, h) ∈ G. The closed subspace H2 (O) = {f ∈ L2 (A) : f ∈ L2 (O)} is ρ-invariant and defines, by restriction, an irreducible representation ρO of G which is equivalent to πO of Proposition 7.17. The special orthogonality relation (7.14) can be restated for ρO through this unitary equivalence. For this purpose, we introduce a definition. Definition 7.41 A function w ∈ H2 (O) is called admissible if  | w (h−1 · χ)|2 dh = 1, H

for some, and hence all, χ ∈ O. Select an admissible w ∈ H2 (O). Using the equivalence of πO and ρO , (7.14) becomes   f, ρO (a, h)wρO (a, h)w, gδ(h)−1 dadh = f, g, (7.20) H

A

for any f, g ∈ H2 (O). Interpreting integration of the Hilbert space-valued function (a, h) → f, ρO (a, h)wρO (a, h)w in the sense of weak convergence of the integral, we obtain a general reproducing formula that is presented as the main theorem of this section. Before stating the theorem, we introduce a small notational change. For a ∈ A, h ∈ H , let wa,h = ρO (a, h)w. That is wa,h (b) = δ(h)−1/2 w(h−1 · (a −1 b)), for any b ∈ A. Theorem 7.42 Let A be an abelian locally compact group and H a σ -compact locally compact group acting on A so that there exists a free open H -orbit O  Let w ∈ H2 (O) be admissible. Then, for any f ∈ H2 (O), in A.   f, wa,h wa,h δ(h)−1 dadh, (7.21) f = H

A

weakly in H (O). 2

Thus, f as an element of H2 (O) can be completely recovered from the set of coefficients {f, wa,h  : (a, h) ∈ G}. It is a consequence of Theorem 7.19, and its proof, that the coefficient function (a, h) → f, wa,h  is in L2 (G), for any f ∈ H2 (O). Define Vw : H2 (O) → L2 (G) by Vw f (a, h) = f, wa,h , for (a, h) ∈ G,

7.3 Generalizations of the wavelet transform

331

for any f ∈ H2 (O). The map Vw is a general continuous wavelet transform. Here the word continuous refers to the parameters a and h running through all of A and H as opposed to a discrete wavelet transform where one uses f, wa,h  only for (a, h) in a specified discrete subset of G. By specializing to specific examples where the hypotheses of Theorem 7.42 hold, we can obtain a variety of continuous wavelet transforms and reconstruction identities. Example 7.43 Let H = R∗ = (−∞, 0) ∪ (0, ∞), as a group under multiplication, act on A = R. We follow the common practice of using a for a generic element of R∗ . So a · t = at, for a ∈ R∗ , t ∈ R. Then G = R  R∗ is the disconnected affine group. There is one free open orbit O, the set of all nontrivial characters, in  R and O is co-null. So H2 (O) = L2 (R). Select an admissible 2 w ∈ L (R). That is, w satisfies  ∞  ∞ | w(γ )|2 da | w(a −1 · 1)|2 dγ = = 1. |γ | |a| −∞ −∞   Let wb,a (t) = |a|−1/2 w t−b , for t ∈ R, (b, a) ∈ R × R∗ . Then (7.21) states, a 2 for any f ∈ L (R),  ∞ ∞ dbda f = f, wb,a wb,a 2 , (7.22) a −∞ −∞ weakly in L2 (R). This is known as the Calder´on reproducing formula and is the basis of a classical continuous wavelet transform with wavelet w. If we replace R∗ with R+ as the acting group in this example, then there are two free open R+ -orbits, O+ = (0, ∞) and O− = (−∞, 0) in  R whose union is 2 2 2 = H2 (O± ) we have L2 (R) = H+ ⊕ H− . One deal co-null. Letting H±  ∞ can then 2 2 2 2 dγ with H+ and H− separately. For example, if w ∈ H+ satisfies 0 | w (γ )| γ = ∞∞ 2 2 , f = 0 −∞ f, wb,a wb,a dbda , weakly in H+ . 1, then, for any f ∈ H+ a2 Example 7.44 Let Hc , for c = 0, be as in Example 7.31 acting on R2 . Applying Theorem 7.42 results in a family of transforms and corresponding reconstruction identities depending on the parameter c. The reader is encouraged to verify 2 (O − ) the details of the following formulae for H2 (O + ). Of course,  the  case  of H  x a 0 of is very similar. Write [x, y, b, a] for a generic element , y b ac 2 , the open upper half plane R2  Hc . There are two open free Hc orbits in R + − O and the open lower half plane O . Thus, L2 (R2 ) = H2 (O + ) ⊕ H2 (O + ). A function w ∈ H2 (O + ) is admissible if   | w (b, a c )|2 a −c da db = 1. R

R+

332

Further applications

For an admissible w ∈ H2 (O + ) and x, y, b, a ∈ R, a > 0 (so [x, y, b, a] ∈ R2  Hc ), wx,y,b,a is defined by, for (s, t) ∈ R2 ,   s − x at − ay − bs + bx c+1 , wx,y,b,a (s, t) = a −( 2 ) w , a a c+1 where elements of R2 are written as row vectors for convenience here. Then equation (7.21) becomes, for any f ∈ H2 (O + ),     dx dy db da f, wx,y,b,a wx,y,b,a , f = a 2c+1 R R R R+ weakly in H2 (O + ).

7.4 Notes and references Section 7.1 is based on Baggett and Taylor [12], although we have provided a more elementary proof here. This sample result on the behavior at infinity of coefficient functions is included because of the application of an explicit expression for the induced representations involved and the use of Mackey analysis. Howe and Moore studied the asymptotic behavior of coefficient functions in [77]. They showed that if G is either a connected algebraic group over a local field of characteristic zero or a connected exponential solvable Lie group and π is an irreducible representation of G with projective kernel (those x ∈ G such that π(x) is a scalar) Pπ , then all coefficient functions of π vanish at infinity modulo Pπ . Proposition 7.13 is a special case of the Shilov idempotent theorem first established in Shilov [144]. The orthogonality relations for irreducible representations of compact groups (7.4) are well known and expressed in various forms. See, for example, Hewitt and Ross [74]. For groups of the form Rn B R, the concept of a projection-generating function was introduced in Gr¨ochenig et al. [69]. The material presented in Section 7.2 is derived from Bernier and Taylor [19], [69], and Kaniuth and Taylor [88]. The generalization of the continuous wavelet transform in the final section is from [19], while Example 7.44 arose naturally through extending the threedimensional Heisenberg group by dilations in Schulz and Taylor [142]. It is worth noting that the continuous shearlet transform introduced in Kutyniok and Labate [95] can be derived from Example 7.44 with a minor change of variables.

Bibliography

[1] L.W. Baggett, A weak containment theorem for groups with a quotient R-group. Trans. Amer. Math. Soc., 128 (1967), 277–90. [2] L.W. Baggett, A description of the topology on the dual spaces of certain locally compact groups. Trans. Amer. Math. Soc., 132 (1968), 175–212. [3] L.W. Baggett, A separable group having a discrete dual space is compact. J. Funct. Anal., 10 (1972), 131–48. [4] L.W. Baggett, Representations of the Mautner group. I. Pacific J. Math., 77 (1978), 7–22. [5] L.W. Baggett, On the continuity of Mackey’s extension process. J. Funct. Anal., 56 (1984), 233–50. [6] L.W. Baggett and A. Kleppner, Multiplier representations of abelian groups. J. Funct. Anal., 14 (1973), 299–324. [7] L.W. Baggett and K. Merrill, Representations of the Mautner group and cocycles of an irrational rotation. Michigan Math. J., 33 (1986), 221–9. [8] L.W. Baggett and J. Packer, C ∗ -algebras associated to two-step nilpotent groups. Contemporary Math., 120 (1991), 1–6. [9] L.W. Baggett and J. Packer, The primitive ideal space of two-step nilpotent group C ∗ -algebras. J. Funct. Anal., 124 (1994), 389–426. [10] L.W. Baggett and A. Ramsay, Some pathologies in the Mackey analysis for a certain nonseparable group. J. Funct. Anal., 39 (1980), 375–80. [11] L.W. Baggett and T. Sund, The Hausdorff dual problem for connected groups. J. Funct. Anal., 43 (1981), 60–8. [12] L.W. Baggett and K.F. Taylor, Riemann–Lebesgue subsets of Rn and representations which vanish at infinity. J. Funct. Anal., 28 (1978), 168–81. [13] L.W. Baggett and K.F. Taylor, On asymptotic behaviour of induced representations. Canad. J. Math., 34 (1982), 220–32. [14] B.A. Barnes, The role of minimal idempotents in the representation theory of locally compact groups. Proc. Edinb. Math. Soc., 23 (1980), 229–38. [15] A.O. Barut and R. Raczka, Theory of Group Representations and Applications. Warsaw: Polish Scientific Publishers, 1977. [16] M.E.B. Bekka, A characterization of locally compact amenable groups by means of tensor products. Arch. Math. (Basel), 52 (1989), 424–7.

333

334

Bibliography

[17] M.E.B. Bekka and R. Curtis, On Mackey’s irreducibility criterion for induced representations. Int. Math. Res. Not., 38 (2003), 2095–101. [18] M.E.B. Bekka and E. Kaniuth, Topological Frobenius properties for nilpotent groups. Math. Scand., 63 (1988), 282–96. [19] D. Bernier and K.F. Taylor, Wavelets from square-integrable representations. SIAM J. Math. Anal., 27 (1996), 594–608. [20] M.W. Binder, Irreducible induced representations of ICC groups. Math. Ann., 294 (1992), 37–47. [21] M.W. Binder, On induced representations of discrete groups. Proc. Amer. Math. Soc., 118 (1993), 301–9. [22] R.J. Blattner, On induced representations. Amer. J. Math., 83 (1961), 79–98. [23] R.J. Blattner, On induced representations. II. Infinitesimal induction. Amer. J. Math., 83 (1961), 499–512. [24] R.J. Blattner, On a theorem of G.W. Mackey. Bull. Amer. Math. Soc., 68 (1962), 585–7. [25] R.J. Blattner, A theorem on induced representations. Proc. Amer. Math. Soc., 13 (1962), 881–4. [26] R.J. Blattner, Positive definite measures. Proc. Amer. Math. Soc., 14 (1963), 423–8. [27] R.J. Blattner, Group extension representations and the structure space. Pacific J. Math., 15 (1965), 1101–13. ´ Norm. Sup. [28] I.D. Brown, Dual topology of a nilpotent Lie group. Ann. Sci. Ec. (4), 6 (1973), 407–11. [29] R.C. Busby and I.E. Schochetman, Compact induced representations. Canad. J. Math., 24 (1972), 5–16. [30] R.C. Busby, I.E. Schochetman and H.A. Smith, Integral operators and the compactness of induced representations. Trans. Amer. Math. Soc., 164 (1972), 461– 77. [31] L. Corwin, Induced representations of discrete groups. Proc. Amer. Math. Soc., 47 (1975), 279–87. [32] L. Corwin and F.P. Greenleaf, Intertwining operators for representations induced from uniform subgroups. Acta Math., 136 (1976), 275–301. [33] L. Corwin and F.P. Greenleaf, Representations of Nilpotent Lie Groups and their Applications. Part 1: Basic Theory and Examples. Cambridge: Cambridge University Press, 1990. [34] J. Dauns and K.H. Hofmann, Representations of rings by sections. Mem. Amer. Math. Soc., 83 (1968). [35] J. Dixmier, Sur les repr´esentations unitaires des groupes de Lie nilpotents. VI. Canad. J. Math., 12 (1960), 324–52. [36] J. Dixmier, Les Alg`ebres d’Op´erateurs dans l’Espace Hilbertien. Paris: GauthierVillars, 1969. [37] J. Dixmier, C ∗ -algebras. Amsterdam: North-Holland, 1977. [38] S. Echterhoff, Mackey’sche Theorie irreduzibler Darstellungen und Topologie im Dual lokalkompakter Gruppen. Diplomarbeit, Universit¨at Paderborn, 1986. [39] R.C. Fabec, Fundamentals of Infinite Dimensional Representation Theory. Boca Raton, FL: Chapman and Hall/CRC, 2000. [40] R.A. Fakler, On Mackey’s tensor product theorem. Duke Math. J., 40 (1973), 689–94.

Bibliography

335

[41] R.A. Fakler, Erratum to “On Mackey’s tensor product theorem”. Duke Math. J., 41 (1974), 691. [42] R.A. Fakler, Representations induced from conjugate subgroups. Indiana J. Math., 19 (1977), 167–71. [43] R.A. Fakler, An intertwining number theorem for induced representations. Nanta Math., 11 (1978), 164–73. ¨ [44] R. Felix, Uber Integralzerlegungen von Darstellungen nilpotenter Liegruppen. Manuscr. Math., 9 (1979), 279–90. [45] R. Felix, R.W. Henrichs and H. Skudlarek, Topological Frobenius reciprocity for projective limits of Lie groups. Math. Z., 165 (1979), 19–28. [46] J.M.G. Fell, The dual spaces of C ∗ -algebras. Trans. Amer. Math. Soc., 94 (1960), 365–403. [47] J.M.G. Fell, A new proof that nilpotent groups are CCR. Proc. Amer. Math. Soc., 13 (1962), 93–9. [48] J.M.G. Fell, A Hausdorff topology on the closed subsets of a locally compact non-Hausdorff space. Proc. Amer. Math. Soc., 13 (1962), 472–6. [49] J.M.G. Fell, Weak containment and induced representations of groups. Canad. J. Math., 14 (1962), 237–68. [50] J.M.G. Fell, Weak containment and Kronecker products of group representations. Pacific J. Math., 13 (1963), 503–10. [51] J.M.G. Fell, Weak containment and induced representations of groups. II. Trans. Amer. Math. Soc., 110 (1964), 424–47. [52] J.M.G. Fell, A new look at Mackey’s imprimitivity theorem. In: Conference on Harmonic Analysis, Lecture Notes in Mathematics 266. Berlin: Springer-Verlag 1972, pp. 43–58. [53] J.M.G. Fell and R.S. Doran, Representations of ∗-Algebras, Locally Compact Groups, and Banach ∗-Algebraic Bundles, Vol. 1: Basic Representation Theory of Groups and Algebras. Boston: Academic Press, 1988. [54] J.M.G. Fell and R.S. Doran, Representations of ∗-Algebras, Locally Compact Groups, and Banach ∗-Algebraic Bundles, Vol. 2: Banach ∗-Algebraic Bundles, Induced Representations, and the Generalized Mackey Analysis. Boston: Academic Press, 1988. [55] G.B. Folland, A Course in Abstract Harmonic Analysis. Boca Raton: CRC Press, 1995. ¨ [56] F.G. Frobenius, Uber Relationen zwischen den Charakteren einer Gruppe und denen ihrer Untergruppen. Sitz. Preuss. Akad. Wiss. (1898), 501–15; also pp. 104– 18 in Frobenius’s Gesammelte Abhandlungen, Vol. III. Berlin: Springer, 1968. [57] S.A. Gaal, Linear Analysis and Representation Theory. Grundlehren Math. Wiss. 198. Berlin: Springer, 1973. [58] I.M. Gelfand and M.A. Naimark, On the embedding of normed rings into the ring of operators in Hilbert space. Mat. Sb. N. S., 12(54) (1943), 197–213. [59] I.M. Gelfand and D.A. Raikov, Irreducible unitary representations of locally bicompact groups. Mat. Sb. N. S., 13(55) (1943), 301–16. [60] J.G. Glimm, Locally compact transformation groups. Trans. Amer. Math. Soc., 101 (1961), 124–38. [61] J.G. Glimm, Type I C ∗ -algebras. Ann. of Math., 73 (1961), 572–612. [62] J.G. Glimm, Families of induced representations. Pacific J. Math., 12 (1962), 885–911.

336

Bibliography

[63] R. Godement, Les fonctions de type positif et la th`eorie des groupes. Trans. Amer. Math. Soc., 63 (1948), 1–84. [64] E.C. Gootman, Weak containment and weak Frobenius reciprocity. Proc. Amer. Math. Soc., 54 (1976), 417–22. [65] E.C. Gootman, Induced representations and finite volume homogeneous spaces. J. Funct. Anal., 24 (1977), 223–40. [66] E.C. Gootman and J. Rosenberg, The structure of crossed product C ∗ -algebras: a proof of the generalized Effros–Hahn conjecture. Invent. Math., 52 (1979), 283–98. [67] F.P. Greenleaf, Invariant Means on Topological Groups. New York: van Nostrand, 1969. [68] F.P. Greenleaf, Amenable actions of locally compact groups. J. Funct. Anal., 4 (1969), 295–315. [69] K. Gr¨ochenig, E. Kaniuth and K.F. Taylor, Compact open sets in duals and projections in L1 -algebras of certain semi-direct product groups. Math. Proc. Cambridge Phil. Soc., 111 (1992), 545–56. [70] S. Grosser and M. Moskowitz, Compactness conditions in topological groups. J. Reine Angew. Math., 246 (1971), 1–40. [71] R.W. Henrichs, Die Frobeniuseigenschaft FP f¨ur diskrete Gruppen. Math. Z., 147 (1976), 191–9. [72] R.W. Henrichs, Weak Frobenius reciprocity and compactness conditions in topological groups. Pacific J. Math., 82 (1979), 387–406. [73] E. Hewitt and K.A. Ross, Abstract Harmonic Analysis. I. Berlin: Springer, 1963. [74] E. Hewitt and K.A. Ross, Abstract Harmonic Analysis. II. Berlin: Springer, 1970. [75] R.E. Howe, On representations of discrete, finitely generated, torsion-free, nilpotent groups. Pacific J. Math., 73 (1977), 281–305. [76] R.E. Howe, The Fourier transform for nilpotent locally compact groups. Pacific J. Math., 73 (1977), 307–27. [77] R.E. Howe and C.C. Moore, Asymptotic properties of unitary representations. J. Funct. Anal., 32 (1979), 72–96. [78] A. Hulanicki, Groups whose regular representation weakly contains all unitary representations. Studia Math., 24 (1964), 37–59. [79] R.V. Kadison and J.R. Ringrose, Fundamentals of the Theory of Operator Algebras. I. New York: Academic Press, 1983. [80] R.V. Kadison and J.R. Ringrose, Fundamentals of the Theory of Operator Algebras. II. New York: Academic Press, 1986. [81] E. Kaniuth, Primitive ideal spaces of groups with relatively compact conjugacy classes. Arch. Math. (Basel), 32 (1979), 16–24. [82] E. Kaniuth, On primary ideals in group algebras. Monatsh. Math., 93 (1982), 293–302. [83] E. Kaniuth, Weak containment and tensor products of group representations. II. Math. Ann., 270 (1985), 1–15. [84] E. Kaniuth, On topological Frobenius reciprocity for locally compact groups. Arch. Math. (Basel), 48 (1987), 286–97. [85] E. Kaniuth, Topological Frobenius properties for nilpotent groups. II. Math. Scand., 68 (1991), 89–107. [86] E. Kaniuth, A Course in Commutative Banach Algebras. Graduate Texts in Mathematics 246. New York: Springer, 2009.

Bibliography

337

[87] E. Kaniuth and W. Moran, The Glimm ideal space of a two-step nilpotent locally compact group. Proc. Edinb. Math. Soc., 44 (2001), 505–26. [88] E. Kaniuth and K.F. Taylor, Minimal projections in L1 -algebras and open points in the dual spaces of semi-direct product groups. J. London Math. Soc., 53 (1996), 141–57. [89] A. Kleppner, On the intertwining number theorem. Proc. Amer. Math. Soc., 12 (1961), 731–3. [90] A. Kleppner, The structure of some induced representations. Duke Math. J., 29 (1962), 555–72. [91] A. Kleppner, Multipliers on abelian groups. Math. Ann., 158 (1965), 11–34. [92] A. Kleppner, Representations induced from compact subgroups. Amer. J. Math., 88 (1966), 544–52. [93] A.W. Knapp, Representation Theory of Semisimple Groups: An Overview Based on Examples. Princeton, NJ: Princeton University Press, 1986. [94] R.A. Kunze, On the Frobenius reciprocity theorem for square-integrable representations. Pacific J. Math., 53 (1974), 465–71. [95] G. Kutyniok and D. Labate, Resolution of the wavefront set using continuous shearlets. Trans. Amer. Math. Soc., 361 (2009), 2719–54. [96] R.L. Lipsman, Group Representations. Lecture Notes in Mathematics 388. Berlin: Springer, 1974. [97] J.R. Liukkonen and R.D. Mosak, The primitive dual space of [F C]− -groups. J. Funct. Anal., 15 (1974), 279–96. [98] L.H. Loomis, Positive definite functions and induced representations. Duke Math. J., 27 (1960), 569–79. [99] G.W. Mackey, Imprimitivity for representations of locally compact groups. I. Proc. Nat. Acad. Sci. USA, 35 (1949), 537–45. [100] G.W. Mackey, On induced representations of groups. Amer. J. Math., 73 (1951), 576–92. [101] G.W. Mackey, Induced representations of locally compact groups. I. Ann. of Math., 55 (1952), 101–39. [102] G.W. Mackey, Induced representations of locally compact groups. II: The Frobenius theorem. Ann. of Math., 58 (1953), 193–220. [103] G.W. Mackey, Unitary representations of group extensions. I. Acta Math., 99 (1958), 265–311. [104] G.W. Mackey, Induced Representations of Groups and Quantum Mechanics. New York: Benjamin, 1968. [105] G.W. Mackey, The Theory of Unitary Group Representations. Chicago: The University of Chicago Press, 1976. [106] G.W. Mackey, Unitary Group Representations in Physics, Probability, and Number Theory. Mathematics Lecture Note Series 55. Reading, MA: Benjamin/Cummings, 1978. [107] F.I. Mautner, Unitary representations of locally compact groups. II. Ann. of Math., 52 (1950), 528–56. [108] F.I. Mautner, A generalization of the Frobenius reciprocity theorem. Proc. Nat. Acad. Sci. USA, 37 (1951), 431–5. [109] F.I. Mautner, Induced representations. Amer. J. Math., 74 (1952), 737–58. [110] C.C. Moore, On the Frobenius reciprocity theorem for locally compact groups. Pacific J. Math., 12 (1962), 359–65.

338

Bibliography

[111] C.C. Moore, Groups with finite dimensional irreducible representations. Trans. Amer. Math. Soc., 166 (1972), 401–10. [112] C.C. Moore and J. Repka, A reciprocity theorem for tensor products of group representations. Proc. Amer. Math. Soc., 64 (1977), 361–4. [113] C.C. Moore and J.A. Wolf, Square integrable representations of nilpotent Lie groups. Trans. Amer. Math. Soc., 185 (1973), 445–62. [114] H. Moscovici, Topological Frobenius properties for nilpotent Lie groups. Rev. Roumaine Math. Pures Appl., 19 (1974), 421–5. [115] O.A. Nielsen, The failure of the topological Frobenius property for nilpotent Lie groups. Math. Scand., 45 (1979), 305–10. [116] O.A. Nielsen, The failure of the topological Frobenius property for nilpotent Lie groups. II. Math. Ann., 256 (1981), 561–8. [117] O.A. Nielsen, Unitary Representations and Coadjoint Orbits of Low-dimensional Nilpotent Lie Groups. Queen’s Papers in Pure and Applied Mathematics 63. Kingston, Ont., 1983. ˇ [118] M. Nilsen, The Stone–Cech compactification of Prim A. Bull. Austral. Math. Soc., 52 (1995), 377–83. [119] N. Obata, Some remarks on induced representations of infinite discrete groups. Math. Ann., 284 (1989), 91–102. [120] B. Ørstedt, Induced representations and a new proof of the imprimitivity theorem. J. Funct. Anal., 31 (1979), 355–9. [121] J. Packer and I. Raeburn, On the structure of twisted group C ∗ -algebras. Trans. Amer. Math. Soc., 334 (1992), 685–718. [122] G.K. Pedersen, Analysis Now. Graduate Texts in Mathematics 118. New York: Springer-Verlag, 1989. [123] D. Poguntke, Der Raum der primitiven Ideale von endlichen Erweiterungen lokalkompakter Gruppen. Arch. Math. (Basel), 28 (1977), 133–8. [124] D. Poguntke, Simple quotients of group C ∗ -algebras for two step nilpotent groups ´ Norm. Sup. (4), 16 (1983), 151–72. and connected Lie groups. Ann. Ec. [125] L.S. Pontryagin, Topological Groups. Princeton Mathematical Series 2. Princeton, NJ: Princeton University Press, 1939. Translated from the Russian by Emma Lehmer. [126] N.S. Poulsen, On C ∞ -vectors and intertwining bilinear forms for representations of Lie groups. J. Funct. Anal., 9 (1972), 87–120. [127] J.C. Quigg, On the irreducibility of an induced representation. Pacific J. Math., 93 (1981), 163–79. [128] J.C. Quigg, On the irreducibility of an induced representation. II. Proc. Amer. Math. Soc., 86 (1982), 345–8. [129] I. Raeburn, On group C ∗ -algebras of bounded representation dimension. Trans. Amer. Math. Soc., 272 (1982), 629–44. [130] M.S. Raghunathan, Discrete Subgroups of Lie Groups. New York: Springer, 1972. [131] A. Ramsay, Non-transitive quasi-orbits in Mackey’s analysis of group extensions. Acta Math., 137 (1976), 17–48. [132] M.A. Rieffel, Unitary representations induced from compact subgroups. Studia Math., 42 (1972), 145–75. [133] M.A. Rieffel, Induced representations of C ∗ -algebras. Adv. Math., 13 (1974), 176–257.

Bibliography

339

[134] M.A. Rieffel, Unitary representations of group extensions: an algebraic approach to the theory of Mackey and Blattner. Studies in Analysis, Adv. Math. Suppl. Studies (1969), 43–82. [135] A. Robert, Introduction to the Representation Theory of Compact and Locally Compact Groups. LMS Lecture Note Series 80. Cambridge: Cambridge University Press, 1983. [136] D.J.S. Robinson, Finiteness Conditions and Generalized Soluble Groups. I. Berlin: Springer, 1972. [137] J.M. Rosenberg, The C ∗ -algebras of some real and p-adic solvable groups. Pacific J. Math., 65 (1976), 175–92. [138] W. Rudin, Fourier Analysis on Groups. Interscience Tracts in Pure and Applied Mathematics 12. New York: Interscience, 1960. [139] I.E. Schochetman, Topology and the duals of certain locally compact groups. Trans. Amer. Math. Soc., 150 (1970), 477–89. [140] I.E. Schochetman, Compact and Hilbert–Schmidt induced representations. Duke Math. J., 41 (1974), 89–102. [141] I.E. Schochetman, The dual topology of certain group extensions. Adv. Math., 35 (1980), 113–28. [142] E. Schulz and K.F. Taylor, Extensions of the Heisenberg group and wavelet analysis in the plane. In: Spline Functions and the Theory of Wavelets, CRM Proceedings and Lecture Notes 18, eds., D. Serge and D. Gilles (1999), 99–107. [143] I.E. Segal, Irreducible representations of operator algebras. Bull. Amer. Math. Soc., 53 (1947), 73–88. [144] G.E. Shilov, On the decomposition of a commutative normed ring into a direct sum of ideals. Mat. Sb., 32 (1954), 353–64. [145] E.M. Stein, Harmonic Analysis: Real Variable Methods, Orthogonality, and Oscillatory Integrals. Princeton, NJ: Princeton University Press, 1993. [146] E.M. Stein and G. Weiss, Introduction to Fourier Analysis on Euclidean Spaces. Princeton, NJ: Princeton University Press, 1971. [147] A.I. Stern, The connection between the topologies of a locally bicompact group and its dual space. Funkc. Anal. Pril., 5 (1971), 56–63. [148] M. Takesaki, Theory of Operator Algebras. I. Berlin: Springer, 1979. [149] A. Valette, Minimal projections, integrable representations and property (T). Arch. Math. (Basel), 43 (1984), 397–406. [150] A. Weil, L’int´egration dans les Groupes Topologiques et ses Applications. Actualit´es Sci. et Ind. 869. Paris: Herman et Cie., 1941. [151] D.P. Williams, The topology on the primitive ideal space of transformation group C ∗ -algebras and CCR transformation group C ∗ -algebras. Trans. Amer. Math. Soc., 266 (1981), 335–59. [152] D.P. Williams, The structure of crossed products by smooth actions. J. Austral. Math. Soc., 47 (1989), 226–35. [153] D.P. Williams, Crossed Product C ∗ -algebras. Math. Surveys and Monographs 134. Providence, RI: American Mathematical Society, 2007. [154] A. Zygmund, Trigonometric Series, II. Cambridge: Cambridge University Press, 1959.

Index

commutant, 25 compact–open topology, 204 continuity of inducing, 232 continuity of restriction, 231 continuous wavelet transform, 82, 331 convergence to topological invariance strongly, 272 weakly, 272 convolution of functions, 4 of measures, 5 coset space, 5 cross-section, 73 cyclic representation, 26 cyclic vector of a representation, 26

admissible function, 330 affine group, 8, 80, 148 associated wavelet transform, 331 irreducible representations, 83 its dual, 162 projections in its L1 -algebra, 323 topology on its dual, 235 algebra Banach ∗, 4 C*, 35 group C*, 37 subgroup C*, 214 L1 (G), 4 almost abelian group, 141 almost Hausdorff space, 154 amenable group, 272 approximate identity, 4 Banach algebra subgroup algebra, 214 Bochner’s theorem, 43 C*-algebra, 35 group C*-algebra, 37 subgroup C*-algebra, 214, 221 Calder´on reproducing formula, 331 character, 40 circle group T, 6 cocycle, 180 measurable, 180 similar cocycles, 180 trivial, 180 cocycle representation, 180 coefficient function of a representation, 24, 30, 306 vanishing at infinity, 306

Dauns–Hofmann, 186 dihedral 4 group, 170 dilation matrix, 325 dual group, 40 of R, 40 of T, 40 of Z, 40 of p , 264 dual space of a C*-algebra, 35 of a generalized motion group, 170 of a locally compact group, 27 of the affine group, 162 of the classical motion group, 172 of the Heisenberg group, 167 dual space topology, 35 affine group, 235 Heisenberg group, 237 of the motion group, 254

340

Index

equivalence of representations, 24 equivariant map, 119 FC-group, 287 Fell group, 173 Fell topology, 204 finite conjugacy class subgroup, 287 Fourier transform, 41 free group on two generators, 56 free open orbit, 315 Frobenius properties restricted topological Frobenius property, 277 topological Frobenius properties (FP), 277 (FP1), 277 (FP2), 277 weak Frobenius properties (WF1), 277 (WF2), 277 Frobenius reciprocity theorem, 107 function admissible, 330 modular, 3 of positive type, 30 rho-function, 13 positive definite, 34 generalized limit, 216 GNS construction, 31 group ax + b, 8, 80, 148 affine, 8, 80, 148 almost abelian, 141 amenable, 272 C*-algebra, 37 dihedral, 170 FC, 287 Fell, 173 finite conjugacy class subgroup, 287 free, 56 generalized motion, 172 Heisenberg, 11, 167, 237 integer Heisenberg, 12, 265 locally compact, 1 Mautner, 151, 175 motion, 11, 172, 254 nilpotent Lie, 168 of unitary operators, 21 semidirect product, 8 two-step nilpotent, 204

341

unimodular, 3 GL(n, R), 10 p-adic integers, 173 p-adic numbers, 173 SL(2, R), 80 SO(n), 11 Haar measure, 2 Heisenberg group, 11, 167 as a semidirect product, 149 discrete Heisenberg group, 12, 153 its dual, 167 topology on its dual, 237 HomG (π, σ ), 24 hull–kernel topology, 35 imprimitivity theorem, 126 induced representation, 65, 72, 74 from an open subgroup, 47 irreducibility, 51, 53, 145, 159 various realizations, 70 induction in stages theorem, 96 inner hull–kernel topology, 206 inner tensor product theorem, 106 intertwine, 24 intertwining operator, 24 invariant subspace, 23 inverse Fourier transform, 42 inversion theorem, 42 involution on L1 (G), 4 irreducible representation, 23 Jacobson topology, 35 kernel of a representation, 35 left action, 6 left coset space, 5 left Haar measure, 2 left-invariant mean, 272 left-regular representation, 22 locally compact group, 1 product groups, 7 Mackey analysis, 140 abelian normal Mackey compatible subgroup, 160 nonabelian normal subgroup, 185, 191 Mackey compatible normal subgroup, 159 Mackey machine, 140 Mackey’s tensor product theorem, 102

342

Mautner group, 151, 175 mean left invariant, topologically left invariant, 272 measure absolutely continuous, 18 Haar, 2 of positive type, 75 positive definite, 75 projection-valued, 117 quasi-invariant, 16 Radon, 2 regular Borel, 2 measure algebra, 5 measure of positive type, 75 minimal projection, 311 modular function, 3 modulation representation, 42 motion group, 254 multiplicity of a subrepresentation, 26 multiplier representation, 180 nondegenerate representation, 29 normal subgroup Mackey-compatible, 159 regularly embedded, 154 orbit under a group action, 6 orthogonality relations, 313 outer tensor product of representations, 27 Peter–Weyl theorem, 172 Plancherel theorem, 41 Plancherel transform, 41 Pontryagin duality theorem, 42 positive definite function, 34 positive definite measure, 75 positive linear functional on L1 (G), 30 Prim(A), 35 Prim(G), 38 primitive ideal, 35 primitive ideal space of a C ∗ -algebra, 35 of a locally compact group, 38 of a two-step nilpotent group, 260 of the integer Heisenberg group, 265 principal series representations, 84 projection minimal, 311 projection-generating function, 320

Index

projection-valued measure, 117 projective representation, 180 quasi-invariant measure, 16 quasi-regular representation, 318 quotient group, 5 quotient homomorphism, 5 Radon measure, 2 left invariant, 2 right invariant, 2 Radon–Nikodym derivative, 16 regular Borel measure, 2 regularly embedded normal subgroup, 154 representation associated function of positive type, 33 cocycle, 180 commutant of, 25 cyclic, 26 induced, 65, 72, 74 induced from an open subgroup, 47 irreducible, 23 kernel of, 35 left regular, 22 lives on an orbit, 154 modulation, 42 multiplier, 180 nondegenerate, 29 of a locally compact group, 21 of a normed algebra, 28 outer tensor product, 27 principal series, 84 projective, 180 quasi-regular, 318, 326, 330 right regular, 23 square integrable modulo its kernel, 300 square-integrable, 317 subrepresentation, 23 multiplicity, 26 tensor product, 27 unitary equivalence, 24 vanishing at infinity, 306 weakly equivalent, 35 rho-function, 13 right-regular representation, 23 semicompact set, 224 semicompact–open topology, 224 semidirect product, 8, 9 separating vector, 26 sesquilinear form, 29

Index

Shilov idempotent theorem, 332 smooth choice of Haar measures, 215 square-integrable representation, 317 stability subgroup, 6 stabilizer, 6 subgroup algebra, 214 subgroup C*-algebra, 214 subgroup representation topology, 222 support of a projection, 311 of a representation, 35 system of imprimitivity, 114, 119 equivalent, 119 induced, 120 tensor product of representations, 27 theorems Bochner’s, 43 Dauns–Hofmann, 186 dual of a generalized motion group, 170 Frobenius reciprocity for finite groups, 107 Gelfand–Raikov, 34 imprimitivity, 126 induction in stages, 96 inner tensor product, 106 inversion, 42 Mackey theorem abelian normal Mackey compatible subgroup, 160 abelian subgroup of finite index, 145

343

for semidirect products, 161 nonabelian normal subgroup, 185, 191 Peter–Weyl, 172 Plancherel, 41 Pontryagin duality, 42 Shilov idempotent, 332 topologically left-invariant mean, 272 topology compact–open, 204 Fell, 204 hull–kernel, 35 inner hull–kernel, 206 Jacobson, 35 semicompact–open, 224 strong operator, 22 subgroup representation, 222 uniform convergence on compacta, 34 weak operator, 21 transform continuous wavelet, 82, 331 Fourier, 41 inverse Fourier, 42 Plancherel, 41 unimodular, 3 unitary equivalence of representations, 24 universal net, 205 wavelet transform, 82 weak containment, 35 Weil’s integration formula, 18