No title

Author: Jonathan Wahl (Academic Editor)

103 downloads 635 Views 3MB Size Report

This content was uploaded by our users and we assume good faith they have the permission to share this book. If you own the copyright to this book and it is wrongfully on our website, we offer a simple DMCA procedure to remove your content from our site. Start by pressing the button below!

Report copyright / DMCA form

DOWNLOAD PDF

Vol. 103, No. 1

DUKE MATHEMATICAL JOURNAL

© 2000

EQUIVARIANT K-THEORY, WREATH PRODUCTS, AND HEISENBERG ALGEBRA WEIQIANG WANG 0. Introduction. Given a ﬁnite group G and a locally compact, Hausdorff, paracompact G-space X, the nth direct product Xn admits a natural action of the wreath product Gn = G ∼ Sn , which is a semidirect product of the nth direct product Gn of G and the symmetric group Sn . The main goal of the present paper is to study the equivariant topological K-theory KGn (X n ) for all n together, and discuss several applications that are of independent interest. We ﬁrst show that a direct sum ᏲG (X) = KGn (X n ) C n≥0

carries several wonderful structures. More explicitly, we show that ᏲG (X) admits a natural Hopf algebra structure with a certain induction functor as multiplication and a certain restriction functor as comultiplication (cf. Theorem 2). When X is a point, KGn (X n ) is theGrothendieck ring R(Gn ), and we recover the standard Hopf algebra structure of n≥0 R(Gn ) (cf., e.g., [M2], [M3], [Z]). A key lemma used here is a straightforward generalization to equivariant K-theory of a statement in the representation theory of ﬁnite groups concerning the restriction of an induced representation to a subgroup. We show that ᏲG (X) is a free λ-ring generated by KG (X) C (cf. Proposition 3). We write down explicitly the Adams operations ϕ n ’s in ᏲG (X). Incidentally, we also obtain an equivalent way of deﬁning the Adams operations in KG (X) C (not over Z) by means of the wreath products, generalizing a deﬁnition by Atiyah [A1] in terms of the symmetric group in the ordinary (i.e., nonequivariant) K-theory setting. When X is a point, we recover the λ-ring structure of n≥0 R(Gn ) (cf. [M2]). As a graded algebra, ᏲG (X) has a simple description as a certain supersymmetric algebra in terms of KG (X) C (cf. Theorem 3). The proof uses a theorem in [AS] and the structures of the centralizer group of an element in Gn and of the ﬁxed-point set of the action of a ∈ Gn on X n , which we work out in Section 1. In particular, this description indicates that ᏲG (X) has the size of a Fock space of a certain inﬁnitedimensional Heisenberg superalgebra that we construct in terms of natural additive maps in K-theory (cf. Theorem 4). Our results above generalize Segal’s work [S2], and our proofs are direct generalizations of those in [S2] (also see [Z], [M1]). What Segal studied in [S2], partly Received 5 February 1999. Revision received 16 August 1999. 2000 Mathematics Subject Classiﬁcation. Primary 19L47, 17B65. 1

2

WEIQIANG WANG

motivated by remarks in Grojnowski [Gr], is the space n≥0 KSn (X n ) C for compact X, which corresponds to our special case when G is trivial and then Gn is the symmetric group Sn . The paper [Gr] was in turn motivated by a physical paper of Vafa and Witten [VW]. Our present work grew out of an attempt to understand Segal’s outlines [S2] and was also stimulated by Nakajima’s lecture notes on Hilbert schemes [N2] (also cf. [N1], [Gr]). Our ﬁrst main observation in this paper is that there is a natural way to add the group G into Segal’s scheme and this allows several different applications, as discussed below. These applications are of independent interest as well. We expect that there is also a natural way to incorporate G into the remaining part of [S2]. Our addition of G already has highly nontrivial consequences, even in the case when X is a point. By tensoring ᏲG (pt) with the group algebra of the Grothendieck ring R(G), we obtain the underlying vector space for the vertex algebra associated to the lattice R(G) (cf. [B], [FLM]). When G is a ﬁnite subgroup of SL2 (C), this leads to a group-theoretic construction of the Frenkel-Kac-Segal vertex representation of an afﬁne Kac-Moody Lie algebra, which can be viewed as a new form of McKay correspondence [Mc]. Details along these lines will be developed in a forthcoming paper. An interesting case of our study is that X is the complex plane C2 acted upon 2 /G denote the minimal resolution of sinby a ﬁnite subgroup G of SL2 (C). Let C 2 gularities of C /G (cf., e.g., [N2]). Via the McKay correspondence [Mc], we show 2 /G)n ) ⊗ C has the same dimension as the that either KGn ((C2 )n ) ⊗ C or KSn ((C 2 /G (cf. [G]). This fact has homology group of the Hilbert scheme of n points on C a straightforward generalization (cf. Remark 6). Our message here is that the wreath product plays an important role in the study of the Hilbert scheme of n points on 2 /G in exactly the way a symmetric group S does for the Hilbert scheme of n C n

points on C2 (cf. [N2], [BG]), which is in turn a special case of the former when G is trivial. We will discuss these in more detail on another occasion. For a smooth manifold X acted upon by G, Dixon, Harvey, Vafa, and Witten introduced a notion of orbifold Euler characteristic e(X, G) in their study of string theory of orbifolds [DHVW]. We show that the orbifold Euler characteristic e(X n , Gn ) is uniquely determined by e(X, G) and n. In terms of a generating function, our formula reads as follows (see Theorem 5): n≥1

e(X n , Gn )q n =

∞

(1 − q r )−e(X,G) .

(1)

r=1

By putting G = 1 and thus e(X, G) = e(X), we recover a formula of HirzebruchHöfer [HH]. By using (1) and Göttsche’s formula [G], we show that Xn /Gn admits a resolution of singularities whose Euler characteristic coincides with the orbifold Euler characteristic e(X n , Gn ), assuming that X is a smooth, quasi-projective surface and X/G has a resolution of singularities whose Euler characteristic is e(X, G).

EQUIVARIANT K-THEORY AND WREATH PRODUCTS

3

In this paper, the language of equivariant K-theory is used. We should also mention the very relevant construction of Heisenberg superalgebra on a direct sum over n of the homology group H (X[n] ) of Hilbert scheme of n points on a smooth quasi-projective surface X, due to Nakajima [N1] and Grojnowski [Gr] independently. However, the constructions and computations in terms of K-theory are simpler and work for more general spaces. Bezrukavnikov and Ginzburg [BG] have proposed a way to obtain a direct isomorphism from KSn (X n ) C to H (X [n] ) for an algebraic surface X. Independently, de Cataldo and Migliorini have recently established this isomorphism for complex surfaces [CM]. This paper is organized in the following manner. In Section 1 we give a presentation of the wreath product Gn and study its action on X n . In Section 2 we construct a Hopf algebra structure on ᏲG (X). In Section 3 we give a description of ᏲG (X) as a graded algebra. In Section 4 we give a λ-ring structure on ᏲG (X). In Section 5 we construct the Heisenberg superalgebra, which acts on ᏲG (X) irreducibly. In Section 6 we calculate the orbifold Euler characteristic e(X n , Gn ) and study in detail the special case when X is the complex plane acted upon by a ﬁnite subgroup of SL2 (C). We have included some details that are probably trivial to experts in hopes that this may beneﬁt readers with different backgrounds. Acknowledgments. The starting point of this work is the insight of Segal [S2]. I am grateful to him for his permission to use his unpublished results. I am also grateful to Igor Frenkel, who ﬁrst emphasized to me the importance of [S2], and to Mikio Furuta, whose generous help was indispensable to me in understanding [S2] and whose discussions were inspiring and fruitful. I thank Mark de Cataldo for helpful conversations on Hilbert schemes. I also thank the referee for his comments and suggestions, which helped to improve the presentation of the paper. This paper is a modiﬁed version of my Max-Planck Institut für Mathematik preprint “Equivariant K-Theory and Wreath Products” in August 1998. It is a pleasure to acknowledge the warm hospitality of the Max-Planck Institut für Mathematik at Bonn. 1. The wreath product and its action on X n . Let G be a ﬁnite group. We denote by G∗ the set of conjugacy classes of G and by R(G) the Grothendieck ring of G. R(G) Z C can be identiﬁed with the ring of class functions C(G) on G by taking the character of a representation. Denote by ζc the order of the centralizer of an element lying in the conjugacy class c in G. We deﬁne an inner product on C(G) as usual: (χ | ψ) =

1 χ(g)ψ(g), |G|

χ, ψ ∈ C(G).

(2)

g∈G

Let Gn = G × · · · × G be the direct product of n copies of G. Denote by |G| the order of G and denote by [g] the conjugacy class of g ∈ G. The symmetric group Sn acts on Gn by permuting the n factors: s(g1 , . . . , gn ) = (gs −1 (1) , . . . , gs −1 (n) ). The wreath product Gn = G ∼ Sn is deﬁned to be the semidirect product of Gn and

4

WEIQIANG WANG

Sn ; namely, the multiplication on Gn is given by (g, s)(h, t) = (g · s(h), st) where g, h ∈ Gn , s, t ∈ Sn . Note that Gn is a normal subgroup of Gn by identifying g ∈ Gn with (g, 1) ∈ Gn . Take a = (g, s) ∈ Gn where g = (g1 , . . . , gn ); we write s ∈ Sn as a product of disjoint cycles; if z = (i1 , . . . , ir ) is one of the cycles the cycle-product gir gir−1 . . . gi1 of a corresponding to the cycle z is determined by g and z up to conjugacy. For each c ∈ G∗ and each integer r ≥ 1, let mr (c) be the number of r-cycles in s whose cycle-product lies in c. Denote by ρ(c) the partition having mr (c) parts equal to r (r ≥ 1) and denote by ρ = (ρ(c))c∈G∗ the corresponding partition-valued function on G∗ . Note that ρ := c∈G∗ |ρ(c)| = c∈G∗ ,r≥1 rmr (c) = n, where |ρ(c)| is the size of the partition ρ(c). Thus, we have deﬁned a map from Gn to ᏼn (G∗ ), the set of partition-valued function ρ = (ρ(c))c∈G∗ on G∗ such that ρ = n. The function ρ is called the type of a = (g, s) ∈ Gn . Denote ᏼ(G∗ ) = n≥0 ᏼ n (G∗ ). Given a partition λ with mr r-cycles (r ≥ 1), deﬁne zλ = r≥1 r mr mr !. This is the order of the centralizer in Sn of an element of cycle-type λ. We denote by l(λ) = r≥1 mr the length of λ. Given a partition-valued function ρ ∈ ᏼ(G∗ ), we deﬁne l(ρ) = c∈G∗ l(ρ(c)) and zρ(c) ζcl(ρ(c)) . Zρ = c∈G∗

Denote by σn (c) the class function of Gn that takes value nζc at an n-cycle whose cycle-product lies in c ∈ G∗ and is zero otherwise. For ρ = {mr (c)}c,r ∈ ᏼ(G∗ ), we deﬁne σρ = σr (c)mr (c) . c∈G∗ ,r≥1

σρ

as the class function on Gn that takes value Zρ at elements of type ρ We regard (where n = ρ) and is zero elsewhere. We formulate some well-known facts below (cf., e.g., [M3]), which are needed later. Proposition 1. Two elements in Gn are conjugate to each other if and only if they have the same type. The order of the centralizer in Gn of an element of type ρ is Zρ . We want to calculate the centralizer ZGn (a) of a ∈ Gn . First we consider the typical case that a has one n-cycle. As the centralizers of conjugate elements are conjugate subgroups, we may assume that a is of the form a = ((g, 1, . . . , 1), τ ) by Proposition 1, % (g), or Z %n (g) when it is necessary to specify where τ = (1, 2, . . . , n). Denote by ZG G n, the following diagonal subgroup of Gn (and thus a subgroup of Gn ):

% (g) = (h, . . . , h), 1 ∈ Gn | h ∈ ZG (g) . ZG The next lemma follows from a direct computation.

EQUIVARIANT K-THEORY AND WREATH PRODUCTS

5

% (g) · a, Lemma 1. The centralizer ZGn (a) of a in Gn is equal to the product ZG % (g) and n where a is the cyclic subgroup of Gn generated by a. Moreover, a ∈ ZG |ZGn (a)| = n|ZG (g)|.

Take a generic element a = (g, s) ∈ Gn of type ρ = (ρ(c))c∈G∗ , where ρ(c) has mr (c) r-cycles (r ≥ 1). By Proposition 1, we may assume (by taking a conjugation if necessary) that the mr (c) r-cycles are of the form

gur (c) = (g, 1, . . . , 1), iu1 , . . . , iur , 1 ≤ u ≤ mr (c), g ∈ c. . . , 1), (1, 2, . . . , r)). Throughout the paper, c,r is underDenote gr (c) = ((g, 1, . stood to be the product c∈G∗ ,r≥1 . Lemma 2. The centralizer ZGn (a) of a ∈ Gn is isomorphic to a direct product of the wreath products

ZGr gr (c) ∼ Smr (c) . (3) c,r

%r (g) · gr (c). Furthermore, ZGr (gr (c)) is isomorphic to ZG

Proof. It follows from the ﬁrst part of Lemma 1 that the centralizer ZGn (a) should contain a certain subgroup naturally isomorphic to (3). By the second part of Lemma 1, we can count that the order of (3) is equal to Zρ . The lemma now follows by comparing with the order of ZGn (a) given in Proposition 1. We use ' to denote the multiplication in C(Gn ) that corresponds to the tensor product in R(Gn ). We denote by n the trivial representation of Gn , and by 1n the sign representation of Gn in which Gn acts trivially and Sn acts by ±1 depending on whether a permutation is even or odd. By abuse of notation, we also use the same symbols to denote the corresponding characters. The next lemma follows easily from the deﬁnitions. ρ Lemma 3. (1) Given ρ, ρ

∈ ᏼn (G∗ ), σ ρ ' σ ρ = δρ, ρ Zρ σ . In particular, Zρ−1 σ ρ , n=

(4)

ρ=n

1n =

ρ=n

(−1)n−l(ρ) Zρ−1 σ ρ .

(5)

ρ (2) (σ ρ | σ ρ ) = δρ, ρ Zρ . In other words, σ takes value Zρ at the elements in Gn of type ρ and is 0 elsewhere.

For a G-space X, we deﬁne an action of Gn on X n as follows: given a = ((g1 , . . . , gn ), s), we let

a · (x1 , . . . , xn ) = g1 xs −1 (1) , . . . , gn xs −1 (n) (6)

6

WEIQIANG WANG

where x1 , . . . , xn ∈ X. Next we want to determine the ﬁxed-point set (X n )a for a ∈ Gn . Let us ﬁrst calculate in the typical case a = ((g, 1, . . . , 1), τ ) ∈ Gn . Note that the centralizer group ZG (g) preserves the g ﬁxed-point set X g . Lemma 4. The ﬁxed-point set is (X n )a = (x, . . . , x) ∈ X n | x = gx , which can be naturally identiﬁed with X g . The action of ZGn (a) on (Xn )a can be identiﬁed canonically with that of ZG (g) on Xg , together with the trivial action of the cyclic group a (cf. Lemma 1). Thus (X n )a /ZGn (a) ≈ X g /ZG (g). Proof. Let (x1 , . . . , xn ) be in the ﬁxed-point set (Xn )a . By (6), we have

x1 , x2 , x3 , . . . , xn = a · x1 , x2 , x3 , . . . , xn = gxn , x1 , x2 , . . . , xn−1 . So all xi (i = 1, . . . , n) are equal to, say, x, and gx = x. The remaining statements follow from Lemma 1. All ZG (g) are conjugate and all X g are homeomorphic to each other for different representatives g in a ﬁxed conjugacy class c ∈ G∗ . Also, the orbit space X g /ZG (g) can be identiﬁed with each other by conjugation for different representatives of g in c ∈ G∗ . We make a convention to denote ZG (g) (resp., Xg , X g /ZG (g)) by ZG (c) (resp., X c , Xc /ZG (c)) by abuse of notation when the choice of a representative g in c is immaterial. Lemma 5. Retain thenotation of Lemma 2. The ﬁxed-point set (Xn )a can be naturally identiﬁed with c,r (X c )mr (c) . Furthermore, the orbit space (Xn )a /ZGn (a) can be naturally identiﬁed with

S mr (c) X c /ZG (c)

c,r

where S m (·) denotes the mth symmetric product. Proof. The ﬁrst part easily follows from Lemma 4. By Lemmas 2 and 4, the action of ZGn (a) on (X n )a can be naturally identiﬁed with that of %r (g) · gr (c) ∼ Smr (c) ZG

c,r

on c,r (X c )mr (c) where Smr (c) acts by permutation and gr (c) acts on Xc trivially. Thus the second part of the lemma follows.

EQUIVARIANT K-THEORY AND WREATH PRODUCTS

7

2. The Hopf algebra structure on ᏲG (X). Given a compact Hausdorff G-space 0 (X) is the Grothendieck group of G-vector X, we recall from [A2] and [S1] that KG 1 bundles over X. We can deﬁne KG (X) in terms of the K 0 functor and a certain suspension operation, and put 0 1 (X) KG (X). KG (X) = KG The tensor product of vector bundles gives rise to a multiplication on KG (X) that is super (i.e., Z2 -graded) commutative. In this paper, we are only concerned about the free part K(X) ⊗ C, which is subsequently denoted by K G (X). We denote by i (X)(i = 0, 1) the dimension of K i (X) ⊗ C. dim KG G If X is a locally compact, Hausdorff, and paracompact G-space, take the one-point 0 (X) to compactiﬁcation X + with the extra point ∞ ﬁxed by G. Then we deﬁne KG be the kernel of the map 0 0 KG (X + ) −→ KG ({∞}) induced by the inclusion map {∞} *→ X + . It is clear that this deﬁnition is equivalent 1 (X) = K 1 (X + ). to the earlier one when X is compact. We also deﬁne KG G Note that KG (pt) is isomorphic to the Grothendieck ring R(G) and K G (pt) is isomorphic to the ring C(G) of class functions on G. The bilinear map ' induced from the tensor product KG (X) −→ KG (X) KG (pt) gives rise to a natural K G (pt)-module structure on K G (X). Thus, K G (X) naturally decomposes into a direct sum over the set of conjugacy classes G∗ of G. The following theorem (from [AS]; see also [BC]) gives a description of each direct summand. Theorem 1. There is a natural Z2 -graded isomorphism

K X g /ZG (g) . φ : K G (X) −→ [g]

Given c ∈ G∗ , we denote by σc the class function that takes value ζc at an element of G lying in the conjugacy class c and zero otherwise. Then an element in K G (X) can be written as of the form c∈G∗ ξc σc , where ξc ∈ K(X g /ZG (g)). More explicitly, the isomorphism φ is deﬁned as follows when X is compact: if E is a G-vector bundle over X, its restriction to X g is acted by g with base points ﬁxed. Thus, E|Xg splits into a direct sum of subbundles Eµ consisting of eigenspaces of g ﬁberwise for each eigenvalue µ of g. ZG (g) acts on Xg and we may check that µ µEµ indeed lies in the ZG (g)-invariant part of K 0 (X g ). Put

φg (E) = µEµ ∈ K 0 X g /ZG (g) = K 0 (X g )ZG (g) . µ

The isomorphism φ on the K 0 part is given by φ =

[g]∈G∗ φg . Then we easily extend

8

WEIQIANG WANG

1 (X) can be identiﬁed with the kernel of the map from the isomorphism φ to K 1 as KG 0 0 1 KG (X × S ) to KG (X) given by the inclusion of a point in S 1 . When X is a point, the isomorphism φ becomes the map from a representation of G to its character. By some standard arguments using compact pairs [A2], the isomorphism remains valid for a locally compact, Hausdorff, and paracompact G-space. The following lemma is well known.

Lemma 6. Given a ﬁnite group G, a subgroup H of G, and a G-space X, there is a natural induction functor Ind = IndG H : KH (X) → KG (X). In particular, when X is G a point, the functor IndH reduces to the familiar induction functor of representations. Proof. Note that there is a G-equivariant isomorphism Iso

G ×H X −−→ G/H × X

(7)

by sending (g, x) ∈ G ×H X to (gH, gx). We remark that although both sides of (7) remain well deﬁned for an H -space X without a G-action, the map Iso makes sense only for a G-space X. Denote by p : G×H X → X the composition of the projection G/H × X to X with the isomorphism (7). As is well known (see [S1]), we have a natural isomorphism

KH (X) −→ KG G ×H X by sending an H -equivariant vector bundle V on X to the G-equivariant vector bundle G ×H V . The composition p ◦ π of the projection p : G ×H X → X with the bundle map π : G×H V → G×H X sends (g, v) to (g, π(v)). We easily check that this gives rise to a well-deﬁned, G-equivariant vector bundle on X, which induces the induction functor IndG H : KH (X) → KG (X). We denote by ResG H (or ResH , or even Res, if there is no ambiguity) the restriction functor from KG (X) to KH (X) by regarding a G-equivariant vector bundle as an H -equivariant vector bundle. Denote q ᏲG (X) = K Gn (X n ), ᏲG (X) = q n K Gn (X n ), n≥0

n≥0

where q is a formal variable counting the graded structure of ᏲG (X). We introduce the notion of q-dimension: q n dim KGn (X n ). dimq ᏲG (X) = n≥0

Deﬁne a multiplication · on ᏲG (X) by a composition of the induction map and the Künneth isomorphism k: k Ind K Gm (X m ) −→ K Gn ×Gm (X n+m ) −−→ K Gn+m (X n+m ). (8) K Gn (X n ) We denote by 1 the unit in KG0 (X 0 ) ≈ C.

EQUIVARIANT K-THEORY AND WREATH PRODUCTS

9

On the other hand, we can deﬁne a comultiplication % on ᏲG (X), given by a composition of the inverse of the Künneth isomorphism and the restriction from Gn to Gk × Gn−k : K Gn (X n ) −→

n m=0

k −1

K Gm ×Gn−m (X n ) −−→

n m=0

K Gm (X m )

K Gn−m (X n−m ).

We deﬁne the counit 3 : ᏲG (X) → C by sending K Gn (X n ) (n > 0) to 0 and 1 ∈ KG0 (X 0 ) ≈ C to 1. The antipode can be also easily deﬁned (see Remark 2). Theorem 2. With various operations deﬁned as above, ᏲG (X) is a graded Hopf algebra. To prove Theorem 2, we need some preparation. Given two subgroups H and L of a ﬁnite group 6 and a 6-space Y , let V be an H -equivariant vector bundle on Y . We denote the action of H on V by ρ. Choose a set of representatives S of the double coset H \6/L. Hs = sH s −1 ∩ L is a subgroup of L for s ∈ S. We denote by Vs the Hs -equivariant vector bundle on Y that is the same as V as a vector bundle and has the conjugated action

(9) ρ s (x) = ρ s −1 xs , x ∈ Hs . Lemma 7. ResL Ind6H V is isomorphic to the direct sum of the L-equivariant vector bundles IndL Hs Vs for all s ∈ H \ 6/L. We easily show that we can extend V in Lemma 7 to the whole KH (Y ). In the case Y = pt, an H -equivariant vector bundle is just an H -module, and the induction and restriction functors become the more familiar ones in representation theory. In such a case, the above lemma is standard (cf., e.g., [Ser, Proposition 2.2]). In view of our construction of the induction functor and the restriction functor, the proof of Lemma 7 is essentially the same as in the case X = pt, which we refer to for a proof [Ser]. Proof of Theorem 2. We show below that the comultiplication % is an algebra homomorphism. The other Hopf algebra axioms are easy to check. We apply Lemma 7 to the case Y = XN , L = Gm ×Gn , H = Gl ×Gr , and 6 = GN , where l + r = m + n = N. In this case, the double coset H \6/L is isomorphic to (Sl × Sr )\SN /(Sm × Sn ) since GN = GN · SN and Gl × Gr = GN · (Sl × Sr ). Furthermore, (Sl × Sr )\SN /(Sm × Sn ) is parametrized by the 2 × 2 matrices a11 a12 (10) a21 a22 satisfying aij ∈ Z+ ,

i, j = 1, 2,

a11 + a12 = m,

a21 + a22 = n,

a11 + a21 = l,

a12 + a22 = r.

(11)

10

WEIQIANG WANG

We denote by ᏹ the set of all the 2 × 2 matrices of the form (10) satisfying the conditions (11). Given E ∈ K Gl (X l ), F ∈ K Gr (X r ), by using Lemma 7, we calculate the following: (m,n)

Res(m,n) IndN (E F ) = IndA 1a11 ⊗ T (a12 ,a21 ) ⊗ 1a22 ResA (E F ) (l,r) A∈ᏹ

=

A∈ᏹ

(12)

n Indm (a11 ,a12 ) F1 Ind(a21 ,a22 ) F2 .

Here the superscript or subscript (a, b) is a shorthand notation for Ga ×Gb . 1a stands a+b for the identity operator from K Ga (X a ) to itself, and Ind(a,b) stands for the induction a+b a+b (a,b) functor from K Ga ×Gb (X ) to K Ga+b (X ). T denotes the canonical functor from K Ga (X a ) ⊗ K Gb (X b ) to K Gb (X b ) ⊗ K Ga (X a ) by switching the factors with an appropriate sign coming from the Z2 -grading of K-theory. Given A ∈ ᏹ of the form (10), the A in the expressions ResA , IndA , and so forth, stands for GA ≡ Ga11 × Ga12 × Ga21 × Ga22 , while A stands for GA ≡ Ga11 × Ga21 × Ga12 × Ga22 . We write (1 ⊗ T (a12 ,a21 ) ⊗ 1)(ResA (E F )) as F1 F2 instead of a direct sum of the form F1 F2 in order to simplify notation, with Fi (i = 1, 2) as the corresponding elements in KGai1 ×Gai2 (X a11 +a12 ). Now it is straightforward to check that the statement that % is an algebra homomorphism is just a reformulation of the identity obtained by summing (12) over all possible (m, n) with m + n = N. 3. A description of ᏲG (X) as a graded algebra. In this section, we give an explicit description of ᏲG (X) as a graded algebra, which, in particular, tells us the dimension of K Gn (X n ). q

Theorem 3. As a (Z + × Z2 )-graded algebra, ᏲG (X) is isomorphic to the supersymmetric algebra ᏿( n≥1 q n K G (X)). In particular, 1

r dim KG (X) r≥1 1 + q dimq ᏲG (X) =

. 0 r dim KG (X) 1 − q r≥1 The supersymmetric algebra here is equal to the tensor product of the symmetric algebra S( n≥1 q n K 0G (X)) and the exterior algebra >( n≥1 q n K 1G (X)). Proof. Take a ∈ Gn of type ρ = {mr (c)}c,r as in Section 1. By Lemma 2, Lemma 5, and the Künneth formula, we have

mr (c) Smr (c)

K(X c )ZG (c) K (Xn )a /ZGn (a) ≈ c∈G∗ ,r≥1

≈

c∈G∗ ,r≥1

᏿mr (c) K X c /ZG (c) .

(13)

EQUIVARIANT K-THEORY AND WREATH PRODUCTS

11

Thus, if we take a summation of (13) over all conjugacy classes of Gn and over all n ≥ 0, we obtain

q ᏲG (X) ≈ qn ᏿mr (c) K X c /ZG (c) by Theorem 1, c,r

n≥0 {mr (c)}c,r ∈ᏼn (G∗ )

=

{mr (c)}c,r ∈ᏼ(G∗ ) c,r

=



᏿mr 

{mr }r r≥1

=

{mr }r r≥1



= ᏿

r≥1

᏿mr (c) q r K X c /ZG (c) ,



q r K X c /ZG (c) 

by letting mr =

c∈G∗

mr (c),

c∈G∗

᏿mr q r K G (X)

by Theorem 1,



q r K G (X) .

The statement concerning dimq ᏲG (X) is an immediate consequence. Remark 1. (1) Theorem 3 in the case when G is the trivial group (and so Gn = Sn ) is due to Segal [S2]. Our proof is adapted from his to the wreath product setting. (2) If G acts freely on X, so does Gn on X n . Then we have the isomorphism K(X/G) ≈ KG (X). Note that Gn is a normal subgroup of the wreath product Gn , and the quotient Gn /Gn is isomorphic to Sn . By Proposition 2.1 in Segal [S1], we see that

KGn (X n ) ≈ KGn /Gn X n /Gn = KSn (X/G)n . (14) Therefore, Theorem 3 follows from the special case G = 1 of Theorem 3 (applying to X/G) and (14). (3) When X is a point, ᏲG (pt) = n≥0 R(Gn ), and σ ρ , ρ ∈ ᏼ(G∗ ) form a linear basis for ᏲG (pt) (cf. [M3]). In particular,

−|G∗ | dimq ᏲG (pt) = . 1 − qr r≥1

(4) One may reformulate Theorem 3 in terms of the delocalized equivariant cohomology (see [BBM], [BC]) via the Chern character. Remark 2. The Hopf algebra deﬁned in Section 2 can be identiﬁed via the isomor phism in Theorem 3 with the standard one on the supersymmetric algebra ᏿( n≥1 K G (X)[n]) by showing that the sets of primitive vectors correspond to each other. Here K G (X)[n] denotes the nth copy of K G (X) (see Theorem 3). The antipode of the former space can be transfered via the isomorphism from the latter one.

12

WEIQIANG WANG

4. The λ-ring structure on ᏲG (X). Let us denote by cn the conjugacy class in Gn that has the type of an n-cycle and whose cycle product lies in the conjugacy class c ∈ G∗ . We consider the following diagram of K-theory maps:

φn n K G (X) −−→ K Gn (X n ) −−→ K (X n )a /ZGn (a) [a]∈(Gn )∗

pr

ι

ϑ c

K (X n )cn /ZGn (cn ) −→ K X /ZG (c)

c∈G∗

c∈G∗

φ

←−− K G (X). Given a G-equivariant vector bundle V , we deﬁne a Gn -action on the nth outer tensor product V n by letting

(15) (g1 , . . . , gn ), s · v1 ⊗ · · · ⊗ vn = g1 vs −1 (1) ⊗ · · · ⊗ gn vs −1 (n) , where g1 , . . . , gn ∈ G, s ∈ Sn . Clearly V n endowed with such a Gn action is a Gn equivariant vector bundle over Xn . Sending V to V n gives rise to the K-theory map n. φn is the isomorphism in Theorem 1 when applying to the case X n with the action of Gn . Let pr be the projection to the direct sum over the conjugacy classes of Gn that are of the type of an n-cycle while ι denotes the inclusion map. ϑ denotes the natural identiﬁcation given by Lemma 5. Finally, the last map φ is the isomorphism given in Theorem 1. We now deﬁne a λ-ring structure on ᏲG (X). It sufﬁces to deﬁne the Adams operations ψ n on ᏲG (X). We also introduce several other K-theory operations which are needed later. Deﬁnition 1. We deﬁne the following composition maps: ψ n := nφ −1 ◦ ϑ ◦ pr ◦φn ◦ n : K G (X) −→ K G (X), ϕ n := nφn−1 ◦ ι ◦ pr ◦ φn ◦ n : K G (X) −→ K Gn (X n ), chn := φ −1 ◦ ϑ ◦ pr ◦φn : K Gn (X n ) −→ K G (X), ωn := nφn−1 ◦ ι ◦ ϑ −1 ◦ φ : K G (X) −→ K Gn (X n ). We list some properties of these K-theory maps whose proof is straightforward. Proposition 2. The following identities hold: chn ◦ωn = n Id,

ωn ◦ ψ n = nϕ n ,

chn ◦ϕ n = nψ n .

Recall that σn (c) is the class function of Gn that takes value nζc at an n-cycle whose cycle-product lies in c ∈ G∗ and is zero otherwise.

EQUIVARIANT K-THEORY AND WREATH PRODUCTS

13

Lemma 8. (1) Let ϕ n (V ) = c∈G∗ ζc−1 V n ' σn (c). (2) Both ψ n and ϕ n are additive K-theory maps. Note that the order of the centralizer of an element lying in the conjugacy class cn is equal to nζc . The ﬁrst part of the above lemma now follows from the deﬁnition of ϕ n and Lemma 3. The second part can be proved in exactly the same way as in the symmetric group case (see [A1]). Here we record only a useful formula in the proof: given that V , W are two G-equivariant vector bundles, let [V ] denote the corresponding element of V in KG (X). (In general, we use V itself to denote the corresponding element in KG (X) by abuse of notation.) Then

[V ] − [W ]

n

=

n (n−j ) j n (−1)j IndG W V ∈ KGn (X n ). Gn−j ×Gj

(16)

j =0

Here, V (n−j ) endows the standard Gn−j -action given by substituting n with n − j in (15), and Gj acts on W j by the tensor product of the standard Gj -action with the sign representation of Gj . ψ n are the Adams operations on K G (X), giving rise to the λ-ring structure on K G (X). Theorem 3 ensures us that as a λ-ring, ᏲG (X) is free and generated by K G (X). Proposition 3. ᏲG (X) is a free λ-ring generated by KG (X) C, with ϕ n ’s as the Adams operations. Remark 3. If X is a point, then KGn (pt) = R(Gn ) and ᏲG (pt) = n≥0 R(Gn ). Our result reduces to the fact that ᏲG (pt) is a free λ-ring generated by G∗ [M2]. In the case when G = 1, the proposition is due to Segal [S2]. q

q

(X) the completion of Ᏺ (X) that allows formal inﬁnite sums. Given Denote by Ᏺ G G q (X) as follows: V ∈ K G (X), we introduce H (V , q), E(V , q) ∈ Ᏺ G q n V n , H (V , q) = n≥0

E(V , q) =

q n V n ' 1n .

n≥0

Proposition 4. We can express H (V , q) and E(V , q) in terms of ϕ r (V ) as follows: 1 r r ϕ (V )q , H (V , q) = exp r r>0 (17) 1 r r E(V , −q) = exp − ϕ (V )q . r r>0

14

WEIQIANG WANG

Proof. We prove (17) by using (4). The formula for E(V , −q) can be similarly obtained by using (5): q n V n ' n H (V , q) = n≥0

=

n

q V

n

'

ρ=n

n≥0

= =

Zρ−1 σ ρ

n≥0 ρ=n

(')

c∈G∗ ,r≥1

Zρ−1 q n V n ' σ ρ

mr (c) 1 r −1 r 1 q ζc V ' σr (c) mr (c)! r



1

= exp   = exp 

r≥1

r

1 r≥1

r

qr

c∈G∗

 ζc−1 V r ' σr (c)

by Lemma 8,



q r ϕ r (V ) .

Here the equation (') is understood by means of the multiplication in ᏲG (X) given by the composition (8). Corollary 1. The λ-operations λn on the λ ring ᏲG (X) send V ∈ K G (X) to

V n ' 1n .

Combining with the additivity of ϕ r , we have the following corollary. Corollary 2. The following equations hold for V , W ∈ K G (X): H (−V , q) = E(V , −q),

H V W, q = H (V , q)H (W, q). 5. ᏲG (X) and a Heisenberg superalgebra. We see from Theorem 3 that ᏲG (X) has the same size as the tensor product of the Fock space of an inﬁnite-dimensional 0 (X) and the Fock space of an inﬁnite-dimensional Heisenberg algebra of rank dim KG 1 Clifford algebra of rank dim KG (X). It is our next step to actually construct such a Heisenberg-Clifford algebra. We simply refer to it as Heisenberg superalgebra from now on. The dual of K G (X), denoted by K G (X)∗ , is naturally Z2 -graded as identiﬁed with 0 K G (X)∗ K 1G (X)∗ . Denote by ·, · the pairing between K G (X)∗ and K G (X). For

EQUIVARIANT K-THEORY AND WREATH PRODUCTS

15

any n, m ≥ 1 and η ∈ K G (X)∗ , we deﬁne an additive map a−m (η) : K Gn (X n ) −→ K Gn−m (X n−m )

(18)

as the composition k −1

Res

K Gn (X n ) −−→ K Gm ×Gn−m (X n ) −−→ K Gm (X m ) chm ⊗1

−−−−→ K G (X)

K Gn−m (X n−m )

η⊗1

K Gn−m (X n−m ) −−−→ K Gn−m (X n−m ).

On the other hand, for any m ≥ 1 and V ∈ K G (X), we deﬁne an additive map am (V ) : K Gn−m (X n−m ) −→ K Gn (X n )

(19)

as the composition ωm (V )·

K Gn−m (X n−m ) −−−−−−→ K Gm (X m ) k

K Gn−m (X n−m )

Ind

−→ K Gm ×Gn−m (X n ) −−→ K Gn (X n ). Let Ᏼ be the linear span of the operators a−m (η), am (V ), m ≥ 1, η ∈ K G (X)∗ , V ∈ K G (X). Clearly, Ᏼ admits a natural Z2 -gradation induced from that on K G (X) and K G (X)∗ . Below we use [·, ·] to denote the supercommutator as well. It is understood that [a, b] is the anticommutator ab + ba when a, b ∈ Ᏼ are both odd elements according to the Z2 -gradation. Theorem 4. When acting on ᏲG (X), Ᏼ satisﬁes the Heisenberg superalgebra commutation relations, namely, for m, l ≥ 1, η, η ∈ K G (X)∗ , V , W ∈ K G (X),

a−m (η), al (V ) = lδm,l η, V , am (W ), al (V ) = 0,

a−m (η), a−l (η ) = 0.

(20) (21) (22)

Furthermore, ᏲG (X) is an irreducible representation of the Heisenberg superalgebra. Proof. We may assume that V , η are homogeneous, say, of degree v and e where v, e ∈ {0, 1}, according to the Z2 -grading of K G (X) and its dual. We continue to use the notation in the proof of Theorem 2. Given E ∈ K Gr (X r ), we ﬁrst observe by the deﬁnitions (18) and (19) that a−m (η) al (V )E (resp., (−1)ve al (V )a−m (η)E) is given by the composition from the top to the bottom along the left (resp., right) side of the following diagram:

16

WEIQIANG WANG

K Gr (X r ) ωl (V )·

K Gl X l K Gr (X r ) VVVV o o VVVV1⊗Res Ind ooo VVVV oo VVVV o o o V o w

l *

N K Gl X K Gm (X m ) K Gr−m (X r−m ) K GN X T (l,m) ⊗1

K Gm (X m ) K Gl X l K Gr−m (X r−m ) K GN X N OOO h h OOO hhhh OOO hhhh h h h h 1⊗Ind OO' Res thhhh m n K Gm (X ) K Gn (X )

K G (X)

chm ⊗1

K Gn (X n ) η⊗1

K Gn (X n )

Here and below it is understood that when a negative integer appears in indices, the corresponding term is zero. To simplify notation, we put Res (resp., Ind) instead of the composition k −1 ◦ Res (resp., Ind ◦k) in the above diagram and below. We denote by ᏹ the set of all the 2 × 2 matrices of the form (10) satisfying (11), except the following two matrices: m 0 0 m . , l −m r l r −m As in the proof of Theorem 2, we apply Lemma 7 to the case Y = X N , 6 = GN , H = Gl × Gr , and L = Gm × Gn , where l + r = m + n = N:

Res(m,n) IndN (l,r) ωl (V ) E =

A∈ᏹ

=

A∈ᏹ

=

(m,n)

IndA

1a11 ⊗ T (a12 ,a21 ) ⊗ 1a22

n Indm (a11 ,a12 ) F1 Ind(a21 ,a22 ) F2

A∈ᏹ

n Indm (a11 ,a12 ) F1 Ind(a21 ,a22 ) F2

ResA ωl (V ) E

EQUIVARIANT K-THEORY AND WREATH PRODUCTS

=

F1 Indn(l,r−m) F2

A∈ᏹ

17

F1 Indn(l−m,r) F2

n Indm (a11 ,a12 ) F1 Ind(a21 ,a22 ) F2

1m ⊗ Indn(l,r−m)

1m ⊗ Indn(l−m,r)

T (l,m) ⊗ 1r−m 1l ⊗ Res(m,r−m) ωl (V ) E

Res(m,l−m) ⊗1r ωl (V ) E .

(23)

n We get zero when applying the map chm ⊗1 to Indm (a11 ,a12 ) F1 Ind(a21 ,a22 ) F2 for A ∈ ᏹ in (23) by Lemma 3. When applying (η ⊗1)◦(chm ⊗1) to the second term of the right-hand side of (23), we obtain (−1)ve al (V )a−m (η)E. When applying chm ⊗1 to the third term of the right-hand side of (23), we get zero if m = l by Lemma 3. In the case m = l, the third term of the right-hand of (23) is simply ωl (V ) E. When applying (η ⊗ 1) ◦ (chm ⊗1) to it, we get lη, V E by Proposition 2. Putting all these pieces together, we have proved (20). We may assume that W is homogeneous of degree w ∈ {0, 1} according to the Z2 grading of K G (X). (21) is a consequence of the transitivity of the induction functor: G Gl+r

IndGm+l+r ω (W ) Ind ω (V ) E m l Gl ×Gr m ×Gl+r G Gm ×Gl+r

Ind (W ) ω (V ) E ω = IndGm+l+r m l G ×G ×G ×G m m r l+r l

G = IndGm+l+r ωm (W ) ωl (V ) E m ×Gl ×Gr G Gl ×Gm+r

= (−1)vw IndGm+l+r (V ) ω (W ) E ω Ind l m ×G G ×G ×G m+r m r l l

G Gm+r = IndGm+l+r (V ) Ind (W ) E . ω ω l m Gm ×Gr l ×Gm+r

Similarly, (22) is a consequence of the transitivity of the restriction functor. The irreducibility of ᏲG (X) as a representation of Ᏼ follows immediately from the qdimension formula for ᏲG (X) given in Theorem 3. In the special case G = 1, the Heisenberg superalgebra was constructed by Segal [S2] and differs slightly from ours. Our proof follows his strategy of proof as well. Remark 4. We may consider the enlarged space VG := ᏲG (pt) C R(G) , where C[R(G)] is the group algebra of the lattice R(G). Note that VG is the underlying space for a lattice vertex algebra (see [B], [FLM]). In particular, when G is a ﬁnite subgroup of SL2 (C), the space VG is closely related to the Frenkel-Kac-Segal vertex representation of an afﬁne Lie algebra. In this way, we are able to obtain a new link

18

WEIQIANG WANG

between the subgroups of SL2 (C) and the afﬁne Lie algebras widely known as the McKay correspondence. Connections among symmetric functions, ᏲG (pt), VG , and vertex operators will be developed in a forthcoming paper. More generally, we may consider VG (X) := ᏲG (X) C KG (X) (24) when KG (X) is torsion-free and where C[KG (X)] is the group algebra of the lattice C[KG (X)]. (If KG (X) is not torsion-free, we replace KG (X) in (24) by the free part of KG (X) over Z.) 6. The orbifold Euler characteristic e(X n , Gn ). In the study of string theory on orbifolds, Dixon et al. [DHVW] came up with a notion of orbifold Euler characteristic deﬁned as follows:

1 e X g1 ,g2 , e(X, G) = |G| g g =g g 1 2

2 1

X g1 ,g2

denotes the common ﬁxed-point set of g1 where X is a smooth G-manifold. and g2 , and e(·) denotes the usual Euler characteristic. One easily shows (see [HH]) that the orbifold Euler characteristic can be equivalently deﬁned as

e X g /ZG (g) . (25) e(X, G) = [g]∈G∗

Denote by X(n) the nth symmetric product of X. Recall that Macdonald’s formula [M1] relates e(X(n) ) to e(X) as follows:

e X (n) q n = (1 − q)−e(X) . (26) n=0

The following theorem relates the orbifold Euler characteristic e(Xn , Gn ) to e(X, G). r −e(X,G) . Theorem 5. We have n≥0 e(X n , Gn )q n = ∞ r=1 (1 − q ) Proof. For an alternative proof, see Remark 5. By the deﬁnition of the orbifold Euler characteristic, Lemma 2, and Lemma 5, we have

e(X n , Gn )q n = e (X n )a /ZGn (a) q n by (25), n≥0

n≥0 [a]∈(Gn )∗

=

n≥0

=

m (c) e X c /ZG (c) r q n

r rmr (c)=n c∈G∗

c∈G∗ r≥1

mr (c)≥0

m (c) m (c) e X /ZG (c) r q r c

(A)

EQUIVARIANT K-THEORY AND WREATH PRODUCTS

=

(1 − q r )−e(X

c /Z (c)) G

19

by applying (26) to Xc /ZG (c),

c∈G∗ r≥1

1 r e(X,G) r=0 (1 − q )

= ∞

by (25).

Here (A) follows from Lemma 2 and Lemma 5. In the case when G is trivial, we recover a formula given in [HH]. Remark 5. According to Atiyah and Segal [AS], the orbifold Euler characteristic can be calculated in terms of equivariant K-theory: 0 1 (X) − dim KG (X). e(X, G) = dim KG

Theorem 5 follows from Theorem 3 by applying (27) to KGn

(27) (X n ).

We are interested (cf. [DHVW], [HH]) in ﬁnding a resolution of singularities −→ X/G X/G

with the property

. e(X, G) = e X/G

We assume that X is a smooth quasi-projective surface with such a property in the following discussions. Denote by X [n] the Hilbert scheme of n points on X. According to Göttsche [G], the Euler characteristic of X [n] is given by ∞

e X [n] q n = (1 − q r )−e(X) . n≥0

(28)

r=0

We note that Xn /Gn is naturally identiﬁed with (X/G)n /Sn . The commutative diagram [n] / X/G n /Sn X/G X n /Gn

≡

(X/G)n /Sn

[n]

implies that the Hilbert scheme X/G is a resolution of singularity of Xn /Gn and comparing with Theorem 5, (indeed it is semismall). By applying (28) to X/G we have the following corollary.

Corollary 3. Let X be a smooth quasi-projective surface and assume that there of singularities of the orbifold X/G such that exists a smooth resolution X/G = e(X, G). Then there exists a resolution of singularities of X n /Gn given e(X/G) [n] satisfying by X/G

[n] e Xn , Gn = e X/G .

20

WEIQIANG WANG

of singularities of X/G The assumption of the existence of the resolution X/G n above is necessary as this is the special case of X /Gn for n = 1. In the setting of [n] is a crepant resolution of X n /Gn provided Corollary 3, we conjecture that X/G is a crepant resolution of singularities of X/G. that X/G We consider a special case in detail. Let X be the complex plane C2 acted upon by a ﬁnite subgroup G of SL2 (C). Via the McKay correspondence [Mc], there is a one-to-one correspondence between the ﬁnite subgroups of SL2 (C) and the Dynkin diagrams of simply laced types An , Dn , E6 , E7 , and E8 . Let us denote by g the simple Lie algebra corresponding to G. From the exact correspondence, we know that the rank of g is |G∗ | − 1. The quotient C2 /G has an isolated simple singularity at zero. There exists a mini 2 /G of C2 /G well known as ALE spaces (cf., e.g., [N2]). It is known mal resolution C 2 /G, Z) is isomorphic to the root lattice of g that the second homology group H2 (C (cf., e.g., [N2]). In particular,

2 /G = dim H C 2 /G = |G |. dim K C (29) ∗ 2 /G with a trivial group action, we have So if we apply Theorem 3 to the case C n 2 /G q n = dim KSn C (1 − q r )−|G∗ | . n≥0

r≥1

On the other hand, by the Thom isomorphism [S1], we have KGn (X n ) ≈ KGn (pt) = R(Gn ). It follows from part (3) of Remark 1 and (29) that

−|G∗ | dim KGn C2n q n = . 1 − qr n≥0

r≥1

We can obtain another numerical coincidence from a somewhat different point of view as follows. For a general quasi-projective smooth surface Y , a well-known result of Fogarty says that the Hilbert scheme of n points on Y , denoted by Y [n] , is a smooth 2n-dimensional manifold. The Betti numbers of Y [n] were computed by Göttsche [G]. In particular, Göttsche’s formula yields the dimension of the homology group of Y [n] :

dim H Y [n] q n = (1 − q r )− dim H (Y ) . (30) n≥0

r≥1

2 /G. It follows from (29) that We apply (30) to the case Y = C [n] 2 /G dim H C (1 − q r )−|G∗ | . qn = n≥0

r≥1

Therefore, we have proved the following proposition.

EQUIVARIANT K-THEORY AND WREATH PRODUCTS

Proposition 5. The spaces KGn (C2n ) have the same dimension.

21

n

[n] 2 /G 2 /G C, KSn C C, and H C

2 /G of C2 /G has no odd dimensional homology, Since the minimal resolution C we have (cf. [HH], [N2])

2 /G = dim H C 2 /G = e C2 , G = |G |. e C ∗ The following corollary is a special and important case of Corollary 3. 2 /G be the minimal Corollary 4. Let G be a ﬁnite subgroup of SL2 (C) and let C 2 /G is a resolution resolution of C2 /G. Then the Hilbert scheme of n points on C

of singularities of C2n /Gn , whose Euler characteristic is equal to the orbifold Euler characteristic e(C2n , Gn ). Remark 6. The fact that the (graded) dimension of KSn (X n ) equals that of the homology group of the Hilbert scheme of n points of X for a more general surface X holds by the same argument as above. Bezrukavnikov and Ginzburg [BG] have proposed a way to establish a direct isomorphism between these two spaces for an algebraic surface X. Independently, de Cataldo and Migliorini have recently established an isomorphism for any complex surface [CM]. Proposition 5 can be generalized as follows: Assume that X is a quasi-projective surface acted upon by G and that there exists a smooth resolution of singularities X/G i of X/G such that the dimension of KG (X) (i = 0, 1) equals that of the even (resp., Then we conclude that the dimension of odd) dimensional homology group of X/G. n KGn (X ) is equal to that of the homology group of the Hilbert scheme of n points on We conjecture the existence of a natural isomorphism from KGn (X n ) C to X/G.

[n] H X/G , assuming the (necessary) existence of an isomorphism from KG (X) C or K(X/G) C. to H (X/G) We believe that this is just a ﬁrst indication of intriguing relations between the and the wreath product Gn . We will elaborate Hilbert scheme of n points on X/G on this on another occasion. When G is trivial, it reduces to well-known relations between the Hilbert scheme of n points and the symmetric group Sn (cf. [BG], [N2]). Remark 7. Let us consider a special case of Corollary 4 by putting G = Z2 . The wreath product Gn in this case is exactly the Weyl group of Bn or Cn . It is interesting to compare with a Hilbert scheme associated to a reductive group of type Bn (or Cn ) deﬁned in [BG]. Note 1 added. The idea here of relating the representation rings of wreath products associated to ﬁnite groups of SL2 (C) and vertex representations of afﬁne Lie algebras has been fully developed in “Vertex representations via ﬁnite groups and the McKay correspondence” [FJW].

22

WEIQIANG WANG

Note 2 added. The connections among Hilbert schemes, wreath products, and Ktheory have been developed in “Hilbert schemes, wreath products, and the McKay correspondence” [W]. References [A1] [A2] [AS] [BBM] [BC] [BG] [B] [CM] [DHVW] [FJW] [FLM] [G] [Gr] [HH] [M1] [M2] [M3]

[Mc]

[N1] [N2] [S1] [S2]

M. Atiyah, Power operations in K-theory, Quart. J. Math. Oxford Ser. (2) 17 (1966), 165–193. , K-Theory, lecture notes by D. W. Anderson, W. A. Benjamin, New York, 1967. M. Atiyah and G. Segal, On equivariant Euler characteristics, J. Geom. Phys. 6 (1989), 671–677. P. Baum, J. Brylinski, and R. MacPherson, Cohomologie équivariante délocalisée, C. R. Acad. Sci. Paris Sér. I Math. 300 (1985), 605–608. P. Baum and A. Connes, “Chern character for discrete groups” in A Fete of Topology, Academic Press, Boston, 1988, 163–232. R. Bezrukavnikov and V. Ginzburg, Hilbert schemes and reductive groups, unpublished notes. R. Borcherds, Vertex algebras, Kac-Moody algebras, and the Monster, Proc. Nat. Acad. Sci. U.S.A. 83 (1986), 3068–3071. M. de Cataldo and L. Migliorini, The Douady space of a complex surface, preprint. L. Dixon, J. Harvey, C. Vafa, and E. Witten, Strings on orbifolds, Nuclear Phys. B 261 (1985), 678–686. I. Frenkel, N. Jing, and W. Wang, Vertex representations via ﬁnite groups and the McKay correspondence, Internat. Math. Res. Notices 2000, 195–222. I. Frenkel, J. Lepowsky, and A. Meurman, Vertex Operator Algebras and the Monster, Pure Appl. Math. 134, Academic Press, Boston, 1988. L. Göttsche, The Betti numbers of the Hilbert scheme of points on a smooth projective surface, Math. Ann. 286 (1990), 193–207. I. Grojnowski, Instantons and afﬁne algebras. I. The Hilbert scheme and vertex operators, Math. Res. Lett. 3 (1996), 275–291. F. Hirzebruch and T. Höfer, On the Euler number of an orbifold, Math. Ann. 286 (1990), 255–260. I. Macdonald, The Poincaré polynomial of a symmetric product, Proc. Cambridge Philos. Soc. 58 (1962), 563–568. , Polynomial functors and wreath products, J. Pure Appl. Algebra 18 (1980), 173–204. , Symmetric Functions and Hall Polynomials, 2d. ed., with contributions by A. Zelevinsky, Oxford Math. Monographs, The Clarendon Press, Oxford University Press, New York, 1995. J. McKay, “Graphs, singularities and ﬁnite groups” in The Santa Cruz Conference on Finite Groups (Univ. California, Santa Cruz, Calif., 1979), Proc. Sympos. Pure Math. 37, Amer. Math. Soc., Providence, 1980, 183–186. H. Nakajima, Heisenberg algebra and Hilbert schemes of points on projective surfaces, Ann. of Math. (2) 145 (1997), 379–388. , Lectures on Hilbert Schemes of Points on Surfaces, Univ. Lecture Ser. 18, Amer. Math. Soc., Providence, R.I., 1999. G. Segal, Equivariant K-theory, Inst. Hautes Études Sci. Publ. Math. 34 (1968), 129–151. , Equivariant K-theory and symmetric products, preprint, 1996.

EQUIVARIANT K-THEORY AND WREATH PRODUCTS [Ser] [VW] [W] [Z]

23

J.-P. Serre, Linear Representations of Finite Groups, trans. L. L. Scott, Grad. Texts in Math. 42, Springer, New York, 1977. C. Vafa and E. Witten, A strong coupling test of S-duality, Nuclear Phys. B 431 (1994), 3–77. W. Wang, Hilbert schemes, wreath products, and the McKay correspondence, preprint, http://arXiv.org/abs/math.AG/9912104. A. Zelevinsky, Representations of Finite Classical Groups. A Hopf Algebra Approach, Lecture Notes in Math. 869, Springer, Berlin, 1981.

Department of Mathematics, Yale University, New Haven, Connecticut 06520, USA; [email protected]

Vol. 103, No. 1

DUKE MATHEMATICAL JOURNAL

© 2000

ORTHONORMAL BASES OF EXPONENTIALS FOR THE n-CUBE JEFFREY C. LAGARIAS, JAMES A. REEDS, and YANG WANG 1. Introduction. A compact set in Rn of positive Lebesgue measure is a spectral set if there is some set of exponentials Ꮾ := e2πiλ,x : λ ∈ , (1.1) which when restricted to gives an orthogonal basis for L2 (), with respect to the inner product f (x)g(x) dx. (1.2) f, g :=

Any set that gives such an orthogonal basis is called a spectrum for . Only very special sets in Rn are spectral sets. However, when a spectrum exists, it can be viewed as a generalization of Fourier series, because for the n-cube = [0, 1]n the spectrum = Zn gives the standard Fourier basis of L2 ([0, 1]n ). The main object of this paper is to relate the spectra of sets to tilings in Fourier space. We develop such a relation for a large class of sets and apply it to geometrically characterize all spectra for the n-cube = [0, 1]n . Theorem 1.1. The following conditions on a set in Rn are equivalent. (i) The set Ꮾ = {e2πiλ,x : λ ∈ } when restricted to [0, 1]n is an orthonormal basis of L2 ([0, 1]n ). (ii) The collection of sets {λ + [0, 1]n : λ ∈ } is a tiling of Rn by translates of unit cubes. This result was conjectured by Jorgensen and Pedersen [6], who proved it in dimensions n ≤ 3. We note that in high dimensions there are many “exotic” cube tilings. There are aperiodic cube tilings in all dimensions n ≥ 3, while in dimensions n ≥ 10 there are cube tilings in which no two cubes share a common (n − 1)-face; see Lagarias and Shor [9]. In Theorem 1.1, the n-cube [0, 1]n appears in both conditions (i) and (ii), but in functorially different contexts. The n-cube in (i) lies in the space domain Rn while the n-cube in (ii) lies in the Fourier domain (Rn )∗ , so they transform differently under linear change of variables. Thus Theorem 1.1 is equivalent to the following result. Received 15 June 1999. Revision received 24 August 1999. 2000 Mathematics Subject Classiﬁcation. Primary 42B05; Secondary 11K70, 47A13, 52C22. Wang partially supported by National Science Foundation grant number DMS-97-06793. 25

26

LAGARIAS, REEDS, AND WANG

Theorem 1.2. For any invertible linear transformation A ∈ GL(n, R), the following conditions are equivalent. (i) ⊂ Rn is a spectrum for A := A([0, 1]n ). (ii) The collection of sets {λ + DA : λ ∈ } is a tiling of Rn , where DA = T (A )−1 ([0, 1]n ). Our main result in §3 gives a necessary and sufﬁcient condition for a general set to be a spectrum of in terms of a tiling of Rn by + D, where D is a speciﬁed auxiliary set in Fourier space. This result applies whenever a suitable auxiliary set D exists. In §4 we show that this is the case when is an n-cube, with D also being an n-cube, and obtain Theorem 1.1. Spectral sets were originally studied by Fuglede [1], who related them to the problem of ﬁnding commuting self-adjoint extensions in L2 () of the set of differential operators −i ∂x∂ 1 , . . . , −i ∂x∂ n deﬁned on the common dense domain Cc∞ (). Our definition of spectrum differs from his by a factor of 2π. Fuglede showed that for sufﬁciently nice connected open regions , each spectrum of (in our sense) has 2π as a joint spectrum of a set of commuting self-adjoint extensions of −i ∂x∂ 1 , . . . , −i ∂x∂ n , and conversely. He also showed that only very special sets are spectral sets. In particular, Fuglede [1, p. 120] made the following conjecture. Spectral set conjecture. A set in Rn is a spectral set if and only if it tiles by translations.

Rn

Much recent work on spectral sets is due to Jorgenson and Pedersen (see [4]–[6], [13], and [14]), with additional work by Lagarias and Wang [10]. The spectral set conjecture concerns tilings by in the space domain. In contrast, Theorem 1.2 describes spectra for the n-cube in terms of tilings in the Fourier domain by an auxiliary set D. In general there does not seem to be any simple relation between sets of translations T used to tile in the space domain and the set of spectra for (see [5], [10], and [14]). Our main results in §3 indicate a relation between the spectral set conjecture and tilings in the Fourier domain—this is discussed at the end of §3. Theorem 1.2 also implies a result concerning sampling and interpolation of certain classes of entire functions. Given a compact set of nonzero Lebesgue measure, let B2 () denote the set of band-limited functions on , which are those entire functions f : Cn → C whose restriction to Rn is the Fourier transform of an L2 -function with compact support contained in . A countable set is a set of sampling for B2 () if there exist A, B > 0 such that for all f ∈ B2 (), Af 2 ≤

|f (λ)|2 ≤ Bf 2 .

(1.3)

λ∈

A set of sampling is always a set of uniqueness for B2 (), where a set is a set of uniqueness if for each set of complex values {cλ : λ ∈ } with |cλ |2 < ∞ there is

ORTHONORMAL BASES OF EXPONENTIALS FOR THE n-CUBE

27

at most one function f ∈ B2 () with f (λ) = cλ ,

for each λ ∈ .

(1.4)

A set is a set of interpolation for B2 () if for each such set {cλ : λ ∈ } there is at least one function f ∈ B2 () such that (1.4) holds. It is clear that a spectrum of a spectral set is both a set of sampling and a set of interpolation for B2 (), so Theorem 1.2 immediately yields the following theorem. Theorem 1.3. Given a linear transformation A in GL(n, R), set A = A([0, 1]n ) and DA = (AT )−1 ([0, 1]n ). If + DA is a tiling of Rn , then is both a set of sampling and a set of interpolation for B2 (A ). Here the set has density exactly equal to the Nyquist rate | det(A)|, as is required by results of Landau (see [11], [12]) for sets of sampling and interpolation. In the appendix we apply Theorem 1.1 to show that in dimensions n = 1 and n = 2 any orthogonal set of exponentials in L2 ([0, 1]n ) can be completed to a basis of exponentials of L2 ([0, 1]n ) but that this is not always the case in dimensions n ≥ 3. We conclude this introduction with two remarks concerning the relation of spectral sets and tilings. First, in comparison with other spectral sets, the n-cube [0, 1]n has an enormous variety of spectra . It seems likely that a “generic” spectral set has a unique spectrum, up to translations.1 Second, the tiling result in §3 applies to more general sets than linearly transformed n-cubes A = A([0, 1]n ); we give the onedimensional example = [0, 1] ∪ [2, 3]. After completing a preprint of this paper in early 1998, we learned that A. Iosevich and S. Pedersen [3] simultaneously and independently obtained a proof of Theorem 1.1, by a different approach. M. Kolountzakis [8] has proved Conjecture 2.1 below, building on the approach of our paper. Notation. For x ∈ Rn , let x denote the Euclidean length of x. We let B(x; T ) := {y : y − x ≤ T } denote the ball of radius T centered at x. The Lebesgue measure of a set in Rn is denoted m(). The Fourier transform fˆ(u) is normalized by ˆ e−2πiu,x f (x) dx. f (u) := Rn

Throughout the paper we let eλ (x) := e2πiλ,x ,

for x ∈ Rn .

(1.5)

Some other authors (see [1], [6]) deﬁne eλ (x) differently, without the factor 2π in the exponent. can be shown that a “generic” fundamental domain of a full rank lattice L in Rn has a unique spectrum = L∗ , the dual lattice. 1 It

28

LAGARIAS, REEDS, AND WANG

2. Orthogonal sets of exponentials and packings. We consider packings and tilings in Rn by compact sets of the following kind. Deﬁnition 2.1. A compact set in Rn is a regular region if it has positive Lebesgue measure m() > 0, is the closure of its interior ◦ , and has a boundary ∂ = \◦ of measure zero. Deﬁnition 2.2. If is a regular region, then a discrete set is a packing set for if the sets { + λ : λ ∈ } have disjoint interiors. It is a tiling set if, in addition, the union of the sets { + λ : λ ∈ } covers Rn . In these cases we say + is a packing or tiling of Rn by , respectively. To a vector λ in Rn , we associate the exponential function eλ (x) := e2πiλ,x ,

for x ∈ Rn .

(2.1)

Given a discrete set in Rn , we set Ꮾ := {eλ (x) : λ ∈ }.

(2.2)

Now suppose that Ꮾ restricted to a regular region gives an orthogonal set of exponentials in L2 (). We derive conditions that the points of must satisfy. Let 1, for x ∈ , (2.3) χ (x) = 0, for x ∈ be the characteristic function of , and consider its Fourier transform e−2πiu,x χ (x) dx, u ∈ Rn . χˆ (u) = Rn

(2.4)

Since is compact, the function χˆ (u) is an entire function of u ∈ Cn . We denote the set of real zeros of χˆ (u) by Z() := u ∈ Rn : χˆ (u) = 0 . (2.5) Lemma 2.1. If is a regular region in Rn , then a set gives an orthogonal set of exponentials Ꮾ in L2 () if and only if − ⊆ Z() ∪ {0}.

(2.6)

Proof. For distinct λ, µ ∈ we have χˆ (λ − µ) = e−2πiλ−µ,x χ (x) dx =

Rn

e−2πiλ,x e2πiµ,x dx = eλ , eµ .

If (2.6) holds, then eλ , eµ = 0, and conversely.

(2.7)

ORTHONORMAL BASES OF EXPONENTIALS FOR THE n-CUBE

29

This lemma implies that the points of have the property of being “well spaced” in the sense of being uniformly discrete; that is, there is some positive R such that any two points are no closer than R. Indeed, since χˆ (0) = m() > 0, the continuity of χˆ (u) implies that there is some ball B(0; R) around zero that includes no point of Z(); hence, λ − µ ≥ R for all λ, µ ∈ , λ = µ. Deﬁnition 2.3. Let be a regular region in Rn . A regular region D is said to be an orthogonal packing region for if (D ◦ − D ◦ ) ∩ Z() = ∅.

(2.8)

Lemma 2.2. Let be a regular region in Rn , and let D be an orthogonal packing region for . If a set gives an orthogonal set of exponentials Ꮾ in L2 (), then is a packing set for D. Proof. If λ = µ ∈ , then Lemma 2.1 gives λ − µ ∈ Z(). By deﬁnition of an orthogonal packing region we have D ◦ ∩ (D ◦ + u) = ∅ for all u ∈ Z(); hence, D ◦ ∩ (D ◦ + λ − µ) = ∅, as required. As indicated above, each regular region has an orthogonal packing region D given by a ball B(0; T ) for small enough T . The larger we can take D, the stronger the restrictions imposed on . Lemma 2.3. If is a spectral set and if D is an orthogonal packing region for , then m(D)m() ≤ 1.

(2.9)

Proof. Let be a spectrum for . Then is a set of sampling for B2 (), so the density results of Landau [11] (see also Gröchenig and Razaﬁnjatovo [2]) give d() = lim inf n→∞

1 # ∩ [−T , T ]n ≥ m(). (2T )n

Now + D is a packing of Rn ; hence if R = diam(D), we have

m(D) 1 # ∩ [−T , T ]n = m (λ + D) : λ ∈ ∩ [−T , T ]n (2T )n (2T )n λ n m − T + R, T + R R n ≤ = 1+ . 2T (2T )n

(2.10)

(2.11)

Letting T → ∞ and taking the inferior limit yields m(D)d() ≤ 1, and now (2.10) yields (2.9).

(2.12)

30

LAGARIAS, REEDS, AND WANG

In §3 we give a self-contained proof of Lemma 2.3. The inequality of Lemma 2.3 does not hold for general sets . In fact the set = [0, 1] ∪ [2, 2 + θ] for suitable irrational θ has a Fourier transform χˆ (ξ ) that has no real zeros; so Z() = ∅, and any regular region D, of arbitrarily large measure, is an orthogonal packing region for . In view of Lemma 2.3 we introduce the following terminology. Deﬁnition 2.4. An orthogonal packing region D for a regular region is tight if m(D) =

1 . m()

(2.13)

This deﬁnition transforms in the Fourier domain under linear transformations: If D is a tight orthogonal packing region for a regular region , then for any A ∈ GL(n, R) the set (AT )−1 (D) is a tight orthogonal packing region for A(). There are many spectral sets that have tight orthogonal packing regions. In §4 we show that D = [0, 1]n is a tight orthogonal packing region for = [0, 1]n . Another example in R1 is the region

In this case we can take

= [0, 1] ∪ [2, 3].

(2.14)

1 3 1 ∪ , . D = 0, 4 2 4

(2.15)

Indeed, χ (x) is the convolution of χ[0,1] (x) with the sum of two delta functions δ0 + δ2 . Thus χˆ (x) = 1 + e−4πix χˆ [0,1] (x). (2.16) From this it is easy to check that the zero set is given by 1 1 +Z ∪ − +Z , Z() = Z \ {0} ∪ 4 4

(2.17)

that D is an orthogonal packing region for , and, since m(D) = 1/2 = 1/m(), that D is tight. A spectrum for is = Z ∪ (Z + (1/4)). Lemma 2.3 together with the spectral set conjecture leads us to propose the following. Conjecture 2.1. If tiles Rn by translations and if D is an orthogonal packing region for , then m()m(D) ≤ 1. This conjecture has now been proved by Kolountzakis [8, Theorem 7].

(2.18)

ORTHONORMAL BASES OF EXPONENTIALS FOR THE n-CUBE

31

3. Spectra and tilings. A main result of this paper is the following criterion that relates spectra to tilings in the Fourier domain. Theorem 3.1. Let be a regular region in Rn , and let be such that the set of exponentials Ꮾ is orthogonal for L2 (). Suppose that D is a regular region with m(D)m() = 1

(3.1)

such that +D is a packing of Rn . Then is a spectrum for if and only if +D is a tiling of Rn . Proof (⇒). Suppose ﬁrst that is a spectrum for . Pick a “bump function” γ (x) ∈ Cc∞ (), and set γt (x) = e−2πit,x γ (x),

for t ∈ Rn .

By hypothesis Ꮾ = {eλ (x) : λ ∈ } is orthogonal and complete for L2 (). Thus, on , we have e2πiλ,x , γt (x) 2πiλ,x γt (x) ∼ e , (3.2) 2 e λ 2 λ∈ with coefﬁcients

e2πiλ,x , γt (x) eλ 22

1 = m()

e−2πiλ,x γt (x) dx

1 e−2πiλ+t,x γ (x) dx = m() Rn 1 = γˆ (λ + t), m()

(3.3)

where m() is the Lebesgue measure of . The rapid decrease of γˆ with increasing radius x and the well-spaced property of show that the right side of (3.2) converges absolutely and uniformly on Rn . Since γt (x) is continuous, we have γt (x) =

1 γˆ (λ + t)e2πiλ,x , m()

for all x ∈ .

(3.4)

λ∈

This yields, for all t ∈ Rn , that γ (x) = e2πit,x γt (x) =

1 γˆ (λ + t)e2πiλ+t,x , m()

for all x ∈ .

(3.5)

λ∈

The series on the right side of (3.5) converges absolutely and uniformly for all x ∈ Rn and t in any ﬁxed compact subset of Rn , but it is only guaranteed to agree with γ (x) for x ∈ .

32

LAGARIAS, REEDS, AND WANG

We now integrate both sides of (3.5) in t over all t ∈ D to obtain m(D)γ (x) = γ (x) χD (t) dt Rn

1 γˆ (λ + t)e2πiλ+t,x dt = m() D λ∈ 1 γˆ (u)e2πiu,x du, for all x ∈ . = m() +D

(3.6)

In the last step we used the fact that the translates λ + D overlap on sets of measure zero, because + D is a packing of Rn . Since m(D) = 1/m(), (3.6) yields γ (x) = h(u)γˆ (u)e2πiu,x du, for all x ∈ , (3.7) Rn

where h(u) =

1, if u ∈ + D, 0, otherwise.

Deﬁne k ∈ L2 (Rn ) by kˆ = hγˆ , so (3.7) asserts that γ (x) = k(x) for almost all x ∈ . Plancherel’s theorem on L2 (Rn ) applied to k, together with (3.7), gives 2 2 2 2 γˆ 2 ≥ hγˆ 2 = k2 ≥ |k(x)| dx = |γ (x)|2 dx = γ 22 . (3.8)

Since Plancherel’s theorem also gives γˆ 22 = γ 22 , we must have γˆ 22 = hγˆ 22 .

(3.9)

We next show that this equality implies that h(u) = 1 almost everywhere on Rn . To do this we show that γˆ (u) = 0 a.e. in Rn . Since γ has compact support, the PaleyWiener theorem states that γˆ (u) is the restriction to Rn of an entire function on Cn that satisﬁes an exponential growth condition at inﬁnity; see Stein and Weiss [16, Theorem 4.9]. Thus γˆ (u) is real analytic on Rn and is not identically zero; hence Z := u ∈ Rn : γˆ (u) = 0 has Lebesgue measure zero. Together with (3.9) this yields h(u) = 1,

a.e. in Rn .

(3.10)

Thus, + D covers all of Rn except a set of measure zero. Finally we show that +D covers all of Rn . By the well-spaced property of and the compactness of D, the set + D is locally the union of ﬁnitely many translates of D; hence + D is closed. Thus, the complement of + D is an open set. But the

ORTHONORMAL BASES OF EXPONENTIALS FOR THE n-CUBE

33

complement of + D has zero Lebesgue measure; hence, it is empty, so + D is a tiling of Rn . (⇐). Suppose + D tiles Rn . By hypothesis, Ꮾ is an orthogonal set in L2 (), and to show that is a spectrum it remains to show that it is complete in L2 (). Let S be the closed span of Ꮾ in L2 (). We show that Cc∞ () is contained in S. Since Cc∞ () is dense in L2 (), this implies S = L2 (). For each γ ∈ Cc∞ (), set γt (x) = e−2πit,x γ (x),

for t ∈ Rn .

Since the elements of Ꮾ are orthogonal, Bessel’s inequality gives γt 2 ≥

|eλ , γt |2 λ∈

eλ 2

=

1 |γˆ (λ + t)|2 , m()

(3.11)

λ∈

where the last series converges uniformly on compact sets by the rapid decay of γˆ at inﬁnity. Integrating this inequality over t ∈ D yields 1 γt 2 dt ≥ |γˆ (λ + t)|2 dt. m() D D λ∈

Since γt = γ for all t and since + D is a tiling, we obtain m(D)γ 2 ≥ γˆ 2 /m(). But m(D) = 1/m() and γ 2 = γˆ 2 , so equality must hold in (3.11) for almost all t: |eλ , γt |2 γ 2 = . (3.12) eλ 22 λ∈ Now the right side of (3.12) converges uniformly on compact sets, so (3.12) holds for all t, including t = 0. Hence, γ 2 =

|eλ , γ |2 λ∈

eλ 22

,

and so γ ∈ S. At ﬁrst glance, the ﬁrst half of this proof of Theorem 3.1 appears too good to be true because it only uses functions γt (x) supported on a ﬁxed subset of . But the relevant fact is that supp γˆt is dense in Fourier space Rn . The proof of Theorem 3.1 yields a direct proof of Lemma 2.3. If D is an orthogonal packing set, then (3.6) holds for it; hence m(D)m()γ (x) agrees with k(x) on , and hence m(D)m()γ 2 ≤ k2 ≤ γ 2 , which shows that (2.9) holds. The following result is an immediate corollary of Theorem 3.1, which we state as a theorem for emphasis.

34

LAGARIAS, REEDS, AND WANG

Theorem 3.2. Let be a regular region in Rn , and suppose that D is a tight orthogonal packing region for . If is a spectrum for , then + D is a tiling of Rn . Proof. The assumption that D is a tight orthogonal packing region guarantees that + D is a packing for all spectra , so Theorem 3.1 applies. Theorem 3.2 sheds some light on Fuglede’s conjecture that every spectral set tiles Rn . ˆ is a tight dual pair if each is a Deﬁnition 3.1. A pair of regular regions (, ) tight orthogonal packing region for the other. In §4 we show that ([0, 1]n , [0, 1]n ) is a tight dual pair of regions; it follows that if A ∈ GL(n, R), then (A([0, 1]n ), (AT )−1 ([0, 1]n )) is also a tight dual pair of regions. The sets ([0, 1] ∪ [2, 3], [0, 1/4] ∪ [1/2, 3/4]) are a tight dual pair in R1 . ˆ is a tight dual pair, then Theorem 3.1 states that if one of (, ) ˆ is a If (, ) n ˆ were also a spectral set (as the ˆ tiles R . If spectral set, say, , then the other set spectral set conjecture implies), then Theorem 3.1 would show that tiles Rn . This raises the question whether the current evidence in favor of Fuglede’s conjecture is ˆ At present we can mainly based on sets that appear in a tight dual pair (, ). only say that there are many nontrivial examples of tight dual pairs. To clarify matters, we formulate two conjectures. ˆ is a tight dual pair Conjecture 3.1 (Spectral set duality conjecture). If (, ) ˆ of regular regions and if is a spectral set, then is also a spectral set. ˆ tile Rn . The corresponding In this case Theorem 3.2 would imply that both and tiling analogue of this conjecture is as follows. ˆ is a tight dual pair of Conjecture 3.2 (Weak spectral set conjecture). If (, ) n regular regions and if one of them tiles R , then so does the other, and both and ˆ are spectral sets. 4. Spectra for the n-cube and cube tilings. We prove Theorem 1.1, using the results of §3. We use the following basic result of Keller [7], which gives a necessary condition for a set to give a cube tiling. Proposition 4.1 (Keller’s criterion). If + [0, 1]n is a tiling of Rn , then each λ, µ ∈ has λi − µi ∈ Z \ {0}

for some i, 1 ≤ i ≤ n.

(4.1)

Proof. This result was proved by Keller [7] in 1930. A detailed proof appears in Perron [15, Satz 9]. The following lemma shows that Keller’s necessary condition for a cube tiling is the same as orthogonality of exponentials in the set .

ORTHONORMAL BASES OF EXPONENTIALS FOR THE n-CUBE

35

Lemma 4.1. Ꮾ := {e2πiλ,x : λ ∈ } gives a set of orthogonal functions in if and only if, for any distinct λ, µ ∈ ,

L2 ([0, 1]n )

λj − µj ∈ Z \ {0},

for some j, 1 ≤ j ≤ n.

(4.2)

Proof. For = [0, 1]n and u ∈ Rn , χˆ (u) =

[0,1]n

e

−2πiu,x

dx =

n

h0 (uj ),

j =1

where h0 (ω) := (1 − e−2πiω )/(2πiω), ω ∈ R, and h0 (0) := 1. Note that h0 (ω) = 0 if and only if ω ∈ Z \ {0}. Hence, χˆ (u) = 0 if and only if uj ∈ Z \ {0} for some j , 1 ≤ j ≤ n. The lemma now follows immediately from Lemma 2.1. Proof of Theorem 1.1. The set D = [0, 1]n is a tight orthogonal packing region for = [0, 1]n . To see this, note that Lemma 4.1 implies that D is an orthogonal packing region for , and since each of and D has measure 1, it is tight. (i)⇒(ii). By hypothesis, Ꮾ is an orthogonal set in L2 ([0, 1]n ). We showed above that D is a tight orthogonal packing region for . Now Theorem 3.2 applies to conclude that + D is a tiling of Rn . (ii)⇒(i). By hypothesis, + D is a cube tiling, so by Proposition 4.1, Ꮾ is an orthogonal set in L2 ([0, 1]n ). Clearly, m()m(D) = 1, and since + D is a cube tiling, it is a fortiori a cube packing. So by Theorem 3.1, is a spectrum. Appendix: Extending orthogonal sets of exponentials to orthogonal bases This appendix determines in which dimensions n every orthogonal set of exponentials on the n-cube can be extended to an orthogonal basis of L2 ([0, 1]n ). Theorem A.1. In dimensions n = 1 and n = 2, any orthogonal set of exponentials can be completed to an orthogonal basis of exponentials of L2 ([0, 1]n ). In dimensions n ≥ 3, this is not always the case. Proof. We say that a cube packing , +[0, 1]n is orthogonal if for distinct γ , µ ∈ ,, γj − µj ∈ Z \ {0},

for some j, 1 ≤ j ≤ n.

(A.1)

Now Proposition 4.1 (Keller’s criterion) and Lemma 4.1 together imply that an orthogonal set of exponentials {e2πiγ ,x : γ ∈ ,} in L2 ([0, 1]n ) corresponds to an orthogonal cube packing using ,. By Theorem 1.1, the question of whether an orthogonal set of exponentials in L2 ([0, 1]n ) can be extended to an orthogonal basis of exponentials of L2 ([0, 1]n ) is equivalent to asking whether the associated orthogonal cube packing in Rn can be completed to a cube tiling by adding extra cubes. Using the known structure of one- and two-dimensional cube tilings, it is straightforward to check that a completion of any orthogonal cube packing is always possible.

36

LAGARIAS, REEDS, AND WANG

(Two-dimensional cube tilings always partition into either all horizontal rows of cubes or all vertical columns of cubes.) We omit the details. To show that extendability is not always possible in dimension 3, consider the set of four cubes {v (i) + [0, 1]3 : 1 ≤ i ≤ 4} in R3 , given by 1 v (1) = − 1, 0, − , 2 1 (2) v = − , −1, 0 , 2 1 (3) v = 0, − , −1 , 2 1 1 1 , , . v (4) = 2 2 2 The orthogonality condition (A.1) is easily veriﬁed. The cubes corresponding to v (1) through v (3) contain (0, 0, 0) on their boundary and create a corner (0, 0, 0). Any cube tiling that extended {v (i) + [0, 1]3 : 1 ≤ i ≤ 3} would have to ﬁll this corner by including the cube [0, 1]3 . However, [0, 1]3 has nonempty interior in common with v (4) + [0, 1]3 . This construction easily generalizes to Rn for n ≥ 3. References [1] [2]

[3] [4] [5] [6] [7] [8] [9] [10] [11]

B. Fuglede, Commuting self-adjoint partial differential operators and a group theoretic problem, J. Funct. Anal. 16 (1974), 101–121. K. Gröchenig and H. Razaﬁnjatovo, On Landau’s necessary density conditions for sampling and interpolation of band-limited functions, J. London. Math. Soc. (2) 54 (1996), 557–565. A. Iosevich and S. Pedersen, Spectral and tiling properties of the unit cube, Internat. Math. Res. Notices 1998, 819–828. P. E. T. Jorgensen and S. Pedersen, Spectral theory for Borel sets in Rn of ﬁnite measure, J. Funct. Anal. 107 (1992), 72–104. , Group-theoretic and geometric properties of multivariable Fourier series, Exposition. Math. 11 (1993), 309–329. , Spectral pairs in Cartesian coordinates, J. Fourier Anal. Appl. 5 (1999), 285–302. O. H. Keller, Über die lückenlose Einfüllung des Raumes mit Würfeln, J. Reine Angew. Math. 163 (1930), 231–248. M. Kolountzakis, Packing, tiling, orthogonality and completeness, preprint, http://xxx.lanl. gov/abs/math.CA/9904066. J. C. Lagarias and P. Shor, Keller’s cube-tiling conjecture is false in high dimensions, Bull. Amer. Math. Soc. (N.S.) 27 (1992), 279–283. J. C. Lagarias and Y. Wang, Spectral sets and factorizations of ﬁnite abelian groups, J. Funct. Anal. 145 (1997), 73–98. H. Landau, Necessary density conditions for sampling and interpolation of certain entire functions, Acta Math. 117 (1967), 37–52.

ORTHONORMAL BASES OF EXPONENTIALS FOR THE n-CUBE [12] [13] [14] [15] [16]

37

, Sampling, data transmission and the Nyquist rate, Proc. IEEE 55 (1967), 1701–1706. S. Pedersen, Spectral theory of commuting selfadjoint partial differential operators, J. Funct. Anal. 73 (1987), 122–134. , Spectral sets whose spectrum is a lattice with a base, J. Funct. Anal. 141 (1996), 496–509. O. Perron, Über lückenlose Ausfüllung des n-dimensionalen Raumes durch kongruente Würfel, Math. Z. 46 (1940), 1–26; II, Math. Z. 46 (1940), 161–180. E. M. Stein and G. Weiss, Introduction to Fourier Analysis on Euclidean Spaces, Princeton Math. Ser. 32, Princeton Univ. Press, Princeton, 1971.

Lagarias and Reeds: Information Sciences Research, AT&T Labs, Florham Park, New Jersey 07932, USA Wang: School of Mathematics, Georgia Institute of Technology, Atlanta, Georgia 30332, USA

Vol. 103, No. 1

DUKE MATHEMATICAL JOURNAL

© 2000

DISTRIBUTION OF ALMOST DIVISION POINTS SHOU-WU ZHANG

1. Introduction. In [10], we proved an equidistribution theorem for small points on abelian varieties, based on the ideas in [7] and [8]. In this paper, we want to generalize this result to almost division points. In the following, we describe our main theorem and its application to the discreteness of almost division points on subvarieties. Let A be an abelian variety deﬁned over a number ﬁeld K. Let xn (n ∈ N) be a ¯ We assume this is a sequence of almost division sequence of distinct points in A(K). points, which means lim sup xnσ − xn = 0. n→∞ σ ∈G

¯ Here, G = Gal(K/K), and · is the square root of the Neron-Tate height function, with respect to some ample and symmetric line bundle on A. Obviously, the notion of almost division does not depend on the choice of the Neron-Tate height functions. If we drop the limit in the above equality, then all xn are division points for A(K). ¯ can be considered a subgroup of We ﬁx an embedding σ : K¯ → C; then A(K) G A(C) := Aσ (C). The Galois orbits xn therefore deﬁne a sequence δxnG of probability measures on A(C); if f is a continuous function on A(C), then 1 f δxnG = G f (y). x A(C) n

y∈xnG

In this paper, we address the convergence of δxnG . More precisely, we want to know whether there is a measure dµ on A(C) such that, for any continuous function f on A(C), G f dxn = f dµ. lim n→∞ A(C)

A(C)

Obviously, such a measure dµ does not exist in general; but, since the space of the continuous functions on A(C) can be topologically generated by countably many functions, dµ does exist if (xn , n ∈ N) is replaced with a subsequence. So, our purpose becomes to describe the following: • the property of the sequence (xn , n ∈ N) which can be obtained by replacing it with a subsequence; Received 5 November 1998. Revision received 9 June 1999. 2000 Mathematics Subject Classiﬁcation. Primary 11G, 14G. This research was supported by National Science Foundation grant number DMS-9796021 and by a Sloan Research Fellowship. 39

40

SHOU-WU ZHANG

• the measure dµ. Let B be an abelian subvariety of A. We deﬁne the degree dB (xn ) of xn modulo B as the degree [K(x¯n ) : K], where x¯n is the image of xn in A/B. Since A has only countably many abelian subvarieties, if we replace (xn , n ∈ N) with a subsequence, we may assume that, for any abelian subvariety B, either dB (xn ) remains bounded or limn→∞ dB (xn ) = ∞. Obviously, there is a minimal abelian subvariety C such that dC (xn ) remains bounded. This C is unique. Let yn denote the image of xn in A/C, via the projection π : A −→ A/C. Then the elements ynσ − yn with n ∈ N, σ ∈ G have the bounded degree and heights going to 0. By the Northcott theorem, these elements are in a ﬁnite list of torsion points for n sufﬁciently large. With (yn , n ∈ N) replaced by a subsequence, we may assume the following: • there is a ﬁxed subset T of torsion points such that, for any n, σ yn − yn : σ ∈ G = T ; • the sequence (yn , n ∈ N) has a limit b ∈ A(C)/C(C) in C-topology. We call xn (n ∈ N), obtained in the above manner, a sequence of almost division points with the coherent limit (C, b+T ). The following is the main result of this paper. Theorem 1.1. Let xn , n ∈ N be a sequence of almost division points with the coherent limit (C, b + T ) as above. Then δxnG converges to the measure dµ =

1 δπ −1 (b+t) , |T | t∈T

where π is the projection A → A/C and δπ −1 (b+t) is the C(C)-invariant probability measure supported in π −1 (b + t). As an application, we show the following theorem about subvarieties. Theorem 1.2. Let X be a subvariety of AK¯ that is not a translation of an abelian subvariety. Then there is an > 0 such that the subset ¯ : d x, A(K) ⊗ R ≤ x ∈ X(K) ¯ ⊗ R is given by a ﬁxed is not Zariski-dense. Here the distance function in A(K) Neron-Tate height pairing. Remarks. (1) Theorem 1.2 has previously been conjectured by B. Poonen. Recently, he proved it independently in [5], using a slightly different argument. When dim X = 1, the above theorem is a special case of a conjecture in [10]. (2) Since the above subset contains the division points of the Mordell-Weil group, the above theorem therefore implies the Mordell-Lang conjecture (see [6], [4]). As in

DISTRIBUTION OF ALMOST DIVISION POINTS

41

M. Raynaud’s proof, we assume Faltings’s theorem on Lang’s conjecture. However, our arguments are purely in “height theory,” rather than Galois theory, on torsion or division points. (3) Both Theorem 1.1 and Theorem 1.2 can be generalized to semiabelian varieties if we assume the equidistribution theorem in this case. Here, A(K) is replaced by any ﬁnitely generated subgroup of A(K), and almost division points (with respect to ) mean that lim d(xn ; ⊗ Q) = 0. n→∞

This is true, for example, for multiplicative group by a result of Y. Bilu [1], and for the split case communicated to me by A. Chambert-Loir. What can we say about the points with large distance to A(K)⊗Q? Using Faltings’s proof in [2] and [3], we can strengthen Theorem 1.2 to the following theorem. Theorem 1.3. There are positive numbers α and β such that the subset ¯ : d x, A(K) ⊗ R ≤ αx + β x ∈ X(K) is not Zariski-dense in X. By Theorem 1.2, it sufﬁces to prove that the subset ¯ : x ≥ H, d x, A(K) ⊗ R ≤ x x ∈ X(K) is not Zariski-dense for some positive numbers H and . Since v∈A(K)⊗R d x, A(K) ⊗ R = x inf v=1 sin (x, v), where (x, v) ∈ [0, π ] denotes the angle between x and v, it sufﬁces to show that, for any unit vector v of A(K) ⊗ R, the set ¯ : x ≥ H, (x, v) ≤ x ∈ X(K) is not Zariski-dense for some > 0. If this is not true, then we have a unit vector v ∈ A(K)⊗R, and a sequence of points (yn , n ∈ N) such that we have the following: • limn→∞ yn = ∞; • limn→∞ (yn , v) = 0; • the set {yn , n ∈ N} has ﬁnite intersection with any proper subvariety of X. Now we can copy Faltings’s proof [3, Theorem 4.1] for points x1 , . . . , xm chosen in {yn , n ∈ N}. The only difference is that xm ’s are no longer deﬁned over K. Acknowledgment. I want to thank Bjorn Poonen for interesting discussions and for his lemma on the limit measures, which is used in my proof of Theorem 1.2. 2. Some reductions. In this section we want to reduce Theorem 1.1 into a statement about the equidistribution of small points. First, we notice that the condition

42

SHOU-WU ZHANG

almost division with the coherent limit (C, b + T ) on xn and the convergence of δxnG depend only on the sets {xn , n ∈ N} and {δxnG , n ∈ N}, respectively, rather than particular orders put on them. So, we may simply talk about notions of a set of almost division points with the coherent limit (C, b +T ) and convergence of set of measures. First reduction. We may reduce the theorem to the case T = {0}. Indeed, since yn ¯ we may ﬁnd a ﬁnite extension L is included in a ﬁnite generated subgroup of A(K), of K, such that all yn (n ∈ N) and all t ∈ T are rational over L. Now we can apply the theorem to AL = A ⊗ L and the sets x ∈ xnG , π(x) = yn + t , t ∈ T . n

These are sets of almost division points with the coherent limits (C, b + t) (t ∈ T ). Second reduction. Now we assume T = {0} and reduce the theorem to the equidistribution of Dn in C(C), where 1 Dn := δx−y , n ∈ N. |Gxn |2 x,y∈Gxn

Indeed, if δxnG does not converge to dµ, then, after replacing xn by a subsequence, we may assume that δxnG converges to a measure dµ∗ not equal to dµ. It is easy to show that dµ∗ is supported in π −1 (s) and that Dn has the limit measure deﬁned by f −→ f (x − y) dµ∗ (x) dµ∗ (y), A(C) A(C)

for any continuous function f on A(C). Assume Dn is equidistributed; then we must have f (x − y) dµ∗ (x) dµ∗ (y) = f dx, A(C) A(C)

C(C)

where dx is the Haar measure on C(C). Let a be a ﬁxed point in π −1 (b); then π −1 (b) = a + C. So dµ∗ is induced by a measure dµ such that f dµ∗ = f (a + x) dµ (x). A(C)

C(C)

Since every continuous function on a + C can be extended to a continuous function on A(C), the above formula implies that f (x − y) dµ (x) dµ (y) = f dx C(C) C(C)

C(C)

for any continuous function f on C(C). In particular, if χ is any nontrivial character on C(C), we obtain 2 χ dµ = 0. C(C)

DISTRIBUTION OF ALMOST DIVISION POINTS

In other words,

C(C)

χ dµ =

43

C(C)

χ dx

for all characters. Since the space of continuous function on C(C) is generated topologically by all characters of C(C), we must have dµ = dx. In other words, the measure dµ∗ is C(C)-invariant. We therefore obtain a contradiction. 3. Equidistribution of Dn . For Theorem 1.1, it remains to prove the following proposition. Proposition 3.1. The sequence of measures Dn converges to the Haar measure dx of C(C). Proof. First, we note that Dn is a linear combination of the uniform probability measures of some Galois orbits of small points. Indeed, for each n, let Hn be a ﬁnite quotient that corresponds to a ﬁnite Galois extension Ln of K such that xn is rational over Ln . Then we can rewrite Dxn as Dxn =

1 1 σ −x )τ = δ δ(x − xn )G . (x n n x G |Hn |2 n G σ ∈H τ ∈H n

x∈xn

n

Here, as before, δ(x − xn )G denotes the uniform probability measure of the Galois orbits of x − xn . There are only countably many closed reduced subvarieties of C which are unions of varieties of the form B + C[N], with B a proper abelian subvariety of C, and N a positive integer. We may ﬁnd a sequence Xi of reduced closed subvarieties of C over K such that we have the following: • each Xi is a union of subvarieties of the form B + C[N]; • Xi ⊂ Xi+1 ; • for any B + C[N], there is an Xi including B + C[N]. By the equidistribution theorem for small points [9], the set / Xn δ(y − xn ) : y ∈ xnG , y − xn ∈ of measures converges to the Haar measure of C(C). We apply this fact to some subsequence of xn . For any proper closed subvariety X deﬁned over K, we deﬁne a positive rational number αX,n by the formula G x ∩ X n αX = G . x n

We want to reduce the equidistribution of Dn to the fact lim αX,n = 0,

n→∞

44

SHOU-WU ZHANG

for each X of the form B + C[N]. As for any two closed subvarieties X and Y of C, αX∪Y,n ≤ αX,n + αY,n , we see that lim αXi ,n = 0,

n→∞

for any Xi . Now by the diagonal process, we may ﬁnd a subsequence ni of N, such that lim αXi ,ni = 0. i→∞

If Dn does not have the limit

dµ, then there

is a continuous function f , such that f Dn does not converges to f dµ. Since f Dn has its absolute value bounded by

f sup , if we replace xn with a subsequence, we may assume that f D converges n

to a number not equal to f dµ. Now consider the expression

x∈x G

x∈x G

i

i

n n 1 i 1 i f Dni = G f δ(x − xni )G + G f δ(x − xni )G . x x C(C) C(C) ni x∈X C(C) ni x ∈X /

The ﬁrst summand in the right-hand side has the absolute value bounded by αXi ,ni f sup , while the second summand approaches 1 − αXi ,ni f dx, C(C)

by the equidistribution theorem of small points. The right-hand side therefore has the limit C(C) f dx. This shows that the subsequence Dni (i ∈ N) has the limit dx. This gives a contradiction. Now we ﬁx an X of the form B + C[N ] and prove that lim αX,n = 0.

n→∞

Replacing A by A/B, we may assume that B = 0. Our assumption on C implies that deg xn → ∞. Consider the multiplication by N: u : xnG −→ (Nxn )G . Then we have

−1 u (Nxn ) =

deg xn , deg(Nxn )

since the Galois group G acts transitively on the set of ﬁbers of u. It follows that, for n 0, −1 u (Nxn ) 1 N 2 dim C αX,n = = ≤ −→ 0. x G deg(Nxn ) deg xn n

DISTRIBUTION OF ALMOST DIVISION POINTS

45

4. Proof of Theorem 1.2. Replacing K by a large ﬁeld, we may assume that X is deﬁned over K. Assume that the theorem is not true. Then we can ﬁnd a sequence ¯ and zn ∈ A(K) ⊗ Q such that we have (xn , zn ) of pairs of points, with xn ∈ X(K) the following: • limn→∞ xn − zn = 0; • the sequence xn converges to the generic point of X with respect to the Zariski topology on X. As the Galois group acts trivially on A(K) ⊗ Q and preserves the Neron-Tate height ¯ we see that (xn , n ∈ N) is a sequence of almost division points in pairing on A(K), ¯ A(K). With (xn , n ∈ N) replaced by a subsequence, we may assume that this sequence has the coherent limit (C, b + T ). Again, we may enlarge K and replace (xn , n ∈ N) by a subsequence in the union of xnG , as in §2, we may assume that T = {0}. Now Theorem 1.1 implies that the measures δxnG converges to the Haar measure dµ in the ﬁber over b of the map π : A → B := A/C. Now consider the following diagram with Y = π(X): X

Y

/A / B.

Our assumption implies that the sequence yn = π(xn ) is rational over K and Zariskidense in Y . By Faltings’s theorem, Y must be the translate of an abelian subvariety. As X itself is not the translate of any abelian subvariety, X = π −1 (Y ). This implies that dim X < dim Y + dim C. As the measures δxnG are supported in the ﬁber X(C) over yn , we obtain a contradiction by the following lemma. Lemma 4.1 (B. Poonen). Let π : X → Y be a surjective morphism of projective and integral complex varieties. Let (yn , n ∈ N) be a sequence of points in Y that converges to the generic point of Y , with respect to the Zariski topology. Let dµn be a sequence of probability measures of X that converges (weakly) to a measure dµ on X. Assume each dµn is supported in the ﬁber of π over yn . Then dµ is supported in a closed subvariety of X of the dimension = dim X − dim Y . Proof. We take an embedding X → PN and consider dµn and dµ as measures in Let P be the Hilbert polynomial of the generic ﬁber of π and let ᐄ → PN × ᐅ be the universal family of the subvarieties of PN with the Hilbert polynomial P . Then, for n sufﬁciently large, π is ﬂat at points over yn , and the ﬁber Xyn is therefore given by a point pn in ᐅ. Since the Hilbert scheme ᐅ is projective, pn converges to a point p in ᐅ, with respect to the C-topology. So the measure dµ, as the limit of some measures on the ﬁbers of ᐄ → ᐅ over pn , is supported in the ﬁber over p. This PN .

46

SHOU-WU ZHANG

shows that the Zariski closure (as a subvariety of PN ) of the support of dµ has the dimension less than or equal to dim X − dim Y . References [1] [2] [3]

[4] [5] [6] [7] [8] [9] [10]

Y. Bilu, Limit distribution of small points on algebraic tori, Duke Math. J. 89 (1997), 467–476. G. Faltings, Diophantine approximation on abelian varieties, Ann. of Math. (2) 133 (1991), 549–576. , “The general case of S. Lang’s conjecture” in Barsatti Symposium in Algebraic Geometry (Abano Term, 1991), Perspect. Math. 15, Academic Press, San Diego, 1994, 175–182. M. McQuillan, Division points on semiabelian varieties, Invent. Math. 120 (1995), 143–159. B. Poonen, Mordell-Lang plus Bogomolov, Invent. Math. 137 (1999), 413–425. M. Raynaud, “Sous-variétés d’une variétés abélienne et points de torsion” in Arithmetic Ᏹ Geometry, Vol. I, Progr. Math. 35, Birkhäuser, Boston, 1983, 327–352. L. Szpiro, E. Ullmo, and S. Zhang, Equirépartition des petits points des courbes, Invent. Math. 127 (1997), 337–347. E. Ullmo, Positivité et discrétion des points algébriques, Ann. of Math. (2) 147 (1998), 167–179. S. Zhang, Equidistribution of small points on abelian varietes, Ann. of Math. (2) 147 (1998), 159–165. , “Small points and Arakelov theory” in Proceedings of the International Congress of Mathematicians ’98, Vol. II (Berlin, 1998), Doc. Math. (1998), 217–225.

Department of Mathematics, Columbia University, New York, New York 10027 USA; [email protected]

Vol. 103, No. 1

DUKE MATHEMATICAL JOURNAL

© 2000

HECKE THEORY AND EQUIDISTRIBUTION FOR THE QUANTIZATION OF LINEAR MAPS OF THE TORUS PÄR KURLBERG and ZEÉV RUDNICK

1. Introduction 1.1. Background. One of the key issues of “Quantum Chaos” is the nature of the semiclassical limit of eigenstates of classically chaotic systems. When the classical system is given by the geodesic ﬂow on a compact Riemannian manifold M (or rather, on its cotangent bundle), one can formulate the problem as follows: The quantum Hamiltonian is, in suitable units, represented by the positive Laplacian − on M. To measure the distribution of its eigenstates, we start with a (smooth) classical observable, that is, a (smooth) function on the unit cotangent bundle S ∗ M; via some choice of quantization from symbols to pseudodifferential operators, we form its quantization Op(f ). This is a zero-order pseudodifferential operator with principal symbol f . The expectation value of Op(f ) in the eigenstate ψ is Op(f )ψ, ψ. Let ψj be a sequence of normalized eigenfunctions: ψj +λj ψj = 0, M |ψj |2 = 1. The problem then is to understand the possible limits as λj → ∞ of the distributions (1.1) f ∈ C ∞ (S ∗ M) −→ Op(f )ψj , ψj . In the case where the geodesic ﬂow is chaotic, it is assumed that the eigenfunctions are random, for instance, in the sense that the expectation values converge as λj → ∞ to the average of f with respect to Liouville measure on S ∗ M. The validity of this for almost all eigenmodes if the classical ﬂow is ergodic (so a very weak notion of chaos!) is asserted by Schnirelman’s theorem [21],1 a fact sometimes referred to as quantum ergodicity. The case where there are no exceptional subsequences is called “quantum unique ergodicity” (QUE). Its validity seems to be a very difﬁcult problem, which is to date unsolved in any case where the dynamics are truly chaotic (see, however, Marklof and Rudnick [16], where QUE is proved for an ergodic, though nonmixing, model case). 1.2. Cat maps. In order to shed some light on the validity of QUE, we look at a “toy model” of the situation—the quantization of linear hyperbolic automorphisms Received 22 June 1999. Revision received 25 August 1999. 2000 Mathematics Subject Classiﬁcation. Primary 81Q50; Secondary 11F27. Authors’ work partially supported by Israel Science Foundation grant number 192/96 and the USIsrael Binational Science Foundation grant number 97-00157. Kurlberg’s work partially supported by the European Commission-Training and Mobility of Researchers network “Algebraic Lie Representations” EC-contract number ERB FMRX-CT97-0100. 1 See Zelditch [24] and de Verdiere [5] for proofs. 47

48

KURLBERG AND RUDNICK

of the 2-dimensional torus T2 . Here the phase space T2 is compact, and instead of a Hamiltonian ﬂow, we consider the discrete time dynamics generated by the iterations of a single map A ∈ SL(2, Z). If A is hyperbolic, that is, | tr A| > 2, then this map is a paradigm of chaotic dynamics. Such maps are sometimes called cat maps in the physics literature. A quantization of these cat maps was proposed by Hannay and Berry [9] and elaborated in [6], [7], [12], [13], and [25]. We review this in some detail in Sections 2 and 3. In particular, the admissible values of Planck’s constant are inverse integers h = 1/N, and the Hilbert space of states ᏴN L2 (Z/NZ) of the quantum system is ﬁnite-dimensional, of dimension N = h−1 . To every classical observable f ∈ C ∞ (T2 ), we associate an operator OpN (f ) on ᏴN , the corresponding quantum observable. The quantization of the cat map is a unitary operator UN (A) on ᏴN , the quantum propagator, unique up to a phase factor, characterized by an exact version of Egorov’s theorem2 UN (A)−1 OpN (f )UN (A) = OpN (f ◦ A), ∀f ∈ C ∞ T2 . (1.2) The eigenvectors φ of the quantum propagator UN (A) are the analogues of the eigenmodes of the Laplacian, and to study their concentration properties, one forms the distributions f −→ OpN (f )φ, φ . In particular, we want to understand the quantum limits as N → ∞. An analogue of Schnirelman’s theorem in this setting was proven in [3] and [25]. We would like to know if QUE holds, that is, if the only quantum limit is the uniform measure on T2 . The spectrum of the quantum propagator UN (A) has degeneracies, which renders the study of possible quantum limits difﬁcult. The degeneracies are systematic and are inversely related to the order of A mod 2N. Degli Esposti, Grafﬁ, and Isola [7] showed that if, instead of looking at all integer values of N, one restricts to the sparse subsequence consisting of primes for which the degeneracies are bounded,3 and, moreover, split in the quadratic extension of the rationals containing the eigenvalues of A, then the only limit is indeed the uniform measure. Our ﬁrst goal in this paper is to show that the degeneracies are coupled to the existence of quantum symmetries. There is a commutative group of unitary operators on ᏴN that commute with UN (A) and therefore act on each eigenspace of UN (A). We call these Hecke operators in analogy with the setting of the modular surface4 (see [10], [15], [20]). We may thus consider eigenfunctions of the desymmetrized 2 This

exact version of Egorov’s theorem is very special and is a consequence of the map being linear. 3 It is an open problem to show that there are inﬁnitely many primes where the degeneracy is bounded. This is known, assuming the generalized Riemann hypothesis, which, in fact, guarantees that a positive proportion of the primes satisfy the assumption. 4 A notable difference between our setting and the modular surface is that in the latter one expects few, if any, degeneracies.

HECKE THEORY AND EQUIDISTRIBUTION FOR LINEAR MAPS

49

quantum map, that is, eigenstates of both UN (A) and of all the Hecke operators. We call these Hecke eigenfunctions. Our second goal is to show that these become equidistributed with respect to Liouville measure, that is, the expectation values of quantum observables in Hecke eigenstates converge to the classical phase-space average of the observable. 1.3. Results. We turn to a detailed description of our results. We ﬁrst carry out a systematic study of the quantum propagator. We deﬁne UN (A) so that it only depends on the remainder of A mod 2N and satisﬁes (1.2). One gets a projective representation A → UN (A) of the subgroup of quantizable elements in the ﬁnite modular group SL(2, Z/2N Z). In Section 4, we explain that it can be made into an ordinary representation if we further restrict to the subgroup (4, 2N) given by g = I mod 4 for N even, g = I mod 2 for N odd. Thus, for A, B ∈ (4, 2N), we have UN (AB) = UN (A)UN (B). Consequently, if AB = BA mod 2N, then their propagators commute. This is the basic principle that we use to form the Hecke operators. Fix a hyperbolic matrix A, which we further assume lies in the congruence subgroup (4) = g ∈ SL(2, Z) : g = I mod 4 so that its reduction modulo 2N lies in (4, 2N) for all N. To ﬁnd matrices commuting with A modulo 2N, we use the connection with the theory of real quadratic ﬁelds (see Section 5). If α is an eigenvalue of A, form O = Z[α], which is an order in the real quadratic ﬁeld K = Q(α). There is an O-ideal I so that the action of α on I by multiplication has A as its matrix in a suitable basis. Thus the action of O on I by multiplication gives us an embedding ι : O → Mat 2 (Z) and induces a map ι : O/2N O → Mat2 (Z/2N Z). Under this map, the images of elements β ∈ O/2N O whose Galois norm is 1 mod 2N lie in SL(2, Z/2NZ) and commute with A modulo 2N . If we further require that β = 1 mod 4O, then we get a group of commuting matrices ι(β) ∈ (4, 2N), whose quantum propagators UN (ι(β)) commute with UN (A) and with each other. These are our Hecke operators. Since the Hecke operators commute with UN (A), they act on its eigenspaces, and since they commute with each other, there is a basis of ᏴN consisting of joint eigenfunctions of UN (A) and the Hecke operators, whose elements we call Hecke eigenfunctions. Our main theorem is the following: Theorem 1. Let A ∈ (4) be a hyperbolic matrix, and let f ∈ C ∞ (T2 ) be a smooth observable. Then for all normalized Hecke eigenfunctions φ ∈ ᏴN of UN (A), the expectation values OpN (f )φ, φ converge to the phase-space average of f as N → ∞. Moreover, for all > 0, we have OpN (f )φ, φ = f (x) dx + Of, N −1/4+ , as N −→ ∞. T2

Remark 1.1. It is easy to extend Theorem 1 to give similar results for matrix elements of OpN (f ). When N is such that the degeneracies in the spectrum of UN (A)

50

KURLBERG AND RUDNICK

are sufﬁciently small, this implies, as in [7], that the expectation values of OpN (f ) in all eigenstates converge to T2 f (x)dx. Remark 1.2. The exponent of 1/4 in our theorem is certainly not optimal, and more likely the correct exponent is 1/2. That is the exponent given in [7], where the problem is reduced to one-variable exponential sums, which can be estimated using Weil’s theorem—the Riemann hypothesis for a curve over a ﬁnite ﬁeld. What we in fact show (see Theorem 9) is that if φi , i = 1, . . . , N is an orthonormal basis of ᏴN consisting of Hecke eigenfunctions, then N

Op (f )φi , φi − N

i=1

4

f (x) dx

N −1+ , 2

T

from which we deduce Theorem 1 by taking an orthonormal basis with φ1 = φ and omitting all but one term on the left-hand side. If all terms on the left-hand side are of roughly the same size, then we would expect this to give the exponent 1/2. The proof of Theorem 1 is reduced to a counting problem in Section 6. This in turn comes down to counting solutions of the congruence β1 − β2 + β3 − β4 = 0 mod N O in norm-one elements βi ∈ O/N O. The number of such norm-one elements is O(N 1+ ) (see Lemma 8), and since this equation has three degrees of freedom, the trivial bound of the number of solutions is O(N 3+ ), ∀ > 0. To get any result in Theorem 1, we need to show that the number of solutions is O(N 3−δ ) for some δ > 0, that is, any saving over the trivial bound would do. This is accomplished in Section 7, where we show that the number of solutions is O(N 2+ ), the optimal bound. Acknowledgments. We thank J. Bernstein, D. Kazhdan, J. Keating, J. Marklof, F. Mezzadri, P. Sarnak, and S. Zelditch for helpful discussions concerning various points in the paper. 2. Background on quantization of maps. In this paper, we consider the quantization of linear (orientation-preserving) automorphisms of the torus T2 = R2 /Z2 , that is, elements of the modular group SL(2, Z), which for the most part are assumed to be hyperbolic (known as cat maps in some of the literature). For this, we ﬁrst review a procedure (one of several) for quantization of maps. The ﬁrst to quantize the cat map were Hannay and Berry [9]. We follow in part an approach by means of representation theory that was developed by Knabe [13] and Degli Esposti, Grafﬁ, and Isola [6] and [7]. See also [3], [12], and [25] for other approaches. 2.1. The quantization procedure. We start by describing some desiderata for a quantization procedure for a symplectic map A of a phase space. In the literature it is

HECKE THEORY AND EQUIDISTRIBUTION FOR LINEAR MAPS

51

customary to distinguish two components of the quantization procedure—a kinematic component and a dynamical one. In the kinematic component, one constructs a Hilbert space Ᏼh of states of the quantum system and an algebra of operators on the space—the algebra of quantum observables.5 Smooth functions f on the classical phase space of the system (that is, classical observables) are mapped to members Oph (f ) of this algebra. To make the connection with the classical system, it is required that in the limit h → 0, the commutator of the quantization of two observables f, g reproduce the quantization of their Poisson bracket {f, g} = j (∂f/∂pj )(∂g/∂qj ) − (∂f/∂qj )(∂g/∂pj ):

i Oph (f ), Oph (g) − Oph ({f, g}) −−→ 0. h→0 h¯ (We do not specify the sense of convergence.) The dynamical part of quantization amounts to prescribing a discrete time evolution of the algebra of quantum observables, that is, a unitary map Uh (A) of Ᏼh , that reproduces the classical map A in the limit h → 0 in the sense that (2.1)

(2.2)

Uh (A)−1 Oph (f )Uh (A) − Oph (f ◦ A) −−→ 0. h→0

(This is the analogue of Egorov’s theorem.) In our case, the classical phase space is the torus T2 . The classical observables are smooth functions on T2 . We ﬁnd that Planck’s constant h is restricted to be an inverse integer: h = 1/N, N ≥ 1. The state-space Ᏼh is ᏴN = L2 (Z/NZ). To each observable f ∈ C ∞ (T2 ), we assign, by an analogue of Weyl quantization, an operator OpN (f ) on ᏴN so that (2.1) holds where convergence is in the space of N The dynamics are given by a linear map A ∈ SL(2, Z) so that x = p× N matrices. 2 → Ax is a symplectic map of the torus. Given an observable f ∈ C ∞ (T2 ), ∈ T q the classical evolution deﬁned by A is f → f ◦ A, where f ◦ A(x) = f (Ax). It turns out that for a certain subset of matrices A, there is a unitary map UN (A) on L2 (Z/NZ) so that an exact form of (2.2) holds: UN (A)−1 OpN (f )UN (A) = OpN (f ◦ A), ∀f ∈ C ∞ T2 . This is our discrete time evolution. We describe these procedures in detail below. 2.2. Kinematics: The space of states. As the Hilbert space of states, we take distributions ψ(q) on the line R that are periodic in both the position and the momentum representation. As is well known, this restricts Planck’s constant to take only inverse integer values. We review the argument: recall that the momentum representation of a wave-function ψ is ∞ 1 Ᏺh ψ(p) = √ ψ(q)e−2πiqp/ h dq. h −∞ 5h

stands for Planck’s constant.

52

KURLBERG AND RUDNICK

We then require Ᏺh ψ(p + 1) = Ᏺh ψ(p)

ψ(q + 1) = ψ(q),

(one may just require that this hold up to a phase). From periodicity in the position representation, we get cn e(nq), ψ(q) = n∈Z

where e(z) := e2πiz . In the momentum representation, that is, applying Ᏺh , we get √ Ᏺh ψ(p) = h cn δ(p − nh). n∈Z

Now, in order that Ᏺh ψ(p +1) = Ᏺh ψ(p), we clearly need 1/ h ∈ Z, that is, for some integer N ≥ 1, that 1 h= . N In that case, we also need cn+N = cn . Thus, we ﬁnd that h = 1/N and the space of states is ﬁnite dimensional, of dimension N = 1/ h, and consists of periodic point-masses at the coordinates q = Q/N, Q ∈ Z. We may then identify ᏴN with the N -dimensional vector space L2 (Z/NZ), with the inner product ·, · deﬁned by φ, ψ =

1 N

φ(Q)ψ(Q).

Q mod N

2.3. Quantizing observables. Next we construct quantum observables: for a free particle on the line, we would take as the basic observables the position and momentum operators h¯ dψ qψ(q) ˆ := qψ(q), pψ(q) ˆ := (q) i dq (h¯ = h/2π ). For our periodic phase space, we take the basic observables to be e(q) ˆ = e2πi qˆ and e(p), ˆ which correspond to the phase space translations e(q)ψ(q) ˆ = e(q)ψ(q),

e(p)ψ(q) ˆ = ψ(q + h).

Corresponding to the commutation relation [q, ˆ p] ˆ = i h¯ = −

h , 2πi

HECKE THEORY AND EQUIDISTRIBUTION FOR LINEAR MAPS

53

we ﬁnd that e(q)e( ˆ p) ˆ = e−2πih e(p)e( ˆ q). ˆ Writing t1 := e(p), ˆ

t2 := e(q) ˆ

(so that t2 t1 = e−2πih t1 t2 ), we put, for n = (n1 , n2 ) ∈ Z2 , (2.3)

TN (n) := eiπn1 n2 /N t2n2 t1n1 .

Their action on a wave function ψ ∈ L2 (Z/NZ) is n2 Q TN (n)ψ(Q) = eiπn1 n2 /N e ψ(Q + n1 ). (2.4) N These are clearly of period 2N in n: TN (n + 2Nm) = TN (n),

n, m ∈ Z2 .

The adjoint of TN (n) is given by (2.5)

TN (n)∗ = TN (−n).

They also satisfy (2.6)

TN (m)TN (n) = eiπω(m,n)/N TN (m + n),

where ω(m, n) = m1 n2 − m2 n1 . Now we can ﬁnally construct quantum observables. For any smooth classical observable f ∈ C ∞ (T2 ) with Fourier expansion p ∈ T2 , fn e(n · x), x = f (x) = q n∈Z2

we deﬁne its quantization OpN (f ) as OpN (f ) :=

fn TN (n).

n∈Z2

The veriﬁcation of (2.1) is an easy calculation using (2.6). 2.4. The Heisenberg group. We now digress to connect this construction to the representation theory of a certain Heisenberg group H2N .

54

KURLBERG AND RUDNICK

For vectors x = (x1 , x2 ), y = (y1 , y2 ), deﬁne ω(x, y) := x1 y2 − x2 y1 . This is a nondegenerate symplectic form. The Heisenberg group H2N is deﬁned to be the set (Z/2NZ)2 × Z/2NZ with multiplication (x, z) · (x , z ) := x + x , z + z + ω(x, x ) . This is at odds with the standard convention where one multiplies ω by 1/2, but is essential for us because 2 is not invertible in Z/2N Z. It is useful to record various facts about the multiplication in H2N : the inverse of (x, z) is (2.7)

(x, z)−1 = (−x, −z).

The commutator of two elements is given by (2.8)

(x, z)(x , z )(x, z)−1 (x , z )−1 = 0, 2ω(x, x ) .

From this commutator identity and the fact that ω is nondegenerate, we immediately ﬁnd the following lemma. Lemma 2. The center of H2N is (NZ/2NZ)2 × Z/2N Z, that is, Cent(H2N ) = (N, Nη, z) : , η = 0, 1, z ∈ Z/2NZ . We deﬁne a representation of H2N on L2 (Z/NZ) by setting z π(n, z) = e TN (n). 2N From the relation (2.6), it follows that π(h)π(h ) = π(hh ), that is, we do indeed get a representation. The center of H2N then acts via the character χ given by z + x 0 y0 χ (x0 , y0 , z) = e 2N (that is, π(x0 , y0 , z) = χ (x0 , y0 , z)I ). The basic facts about π and the representation theory of H2N are covered in the following proposition. Proposition 3. (i) All irreducible representations of H2N have dimension at most N. (ii) The representation π is irreducible and is the unique, irreducible N -dimensional representation with central character χ. We omit the details of the proof; the main point (which is easy to verify from the deﬁnitions) is the following lemma.

HECKE THEORY AND EQUIDISTRIBUTION FOR LINEAR MAPS

55

Lemma 4. The trace of TN (n) is given by

tr TN (n) = N, if n ≡ (0, 0) mod N, 0, otherwise. √ Proof. Let φi = Nδi where δi is the Dirac delta function supported at i, so that 2 {φi }N i=1 is an orthonormal basis of L (Z/NZ). Then tr TN (n) =

N

TN (n)φi , φi ,

i=1

and by equation (2.4), n1 n2 + 2n2 Q φi (Q + n1 ) TN (n)φi (Q) = e 2N n1 n2 + 2n2 Q φi−n1 (Q) =e 2N −n1 n2 + 2n2 i φi−n1 (Q). =e 2N

Therefore, tr TN (n) = 0 unless n1 ≡ 0 mod N, in which case, N −n1 n2 n2 i . e TN (n)φi , φi = e 2N N

N i=1

The result now follows since otherwise.

i=1

N

i=1 e(n2 i/N )

equals N if n2 ≡ 0 mod N, and is zero

2.5. Description of π as an induced representation. Let Y be the subgroup of elements Y = (x0 , y, z) : y, z ∈ Z/2NZ, x0 ∈ NZ/2NZ . It is easily seen to be a normal, maximal abelian subgroup, of index N, containing the center. For (x0 , y, z) ∈ Y , set z + x0 y . τ (x0 , y, z) := e 2N This is a character of Y (we need to use 2x0 ≡ 0 mod 2N in verifying this), restricting to the character χ (x0 , y0 , z) = e(z + x0 y0 /2N) of the center. H We consider the induced representation IndY 2N τ of the Heisenberg group. The basic model for it is the space of functions 7 : H2N → C satisfying 7(ah) = τ (a)7(h) for a ∈ Y , h ∈ H2N . The action of the group is by right multiplication

56

KURLBERG AND RUDNICK

h7(h ) := 7(h h). By restricting to the subgroup X = {(x, 0, 0)}, we can realize this induced representation as functions on Z/2NZ that are N-periodic (since the element (N, 0, 0) lies in X ∩ Y ). We can identify this space of functions with L2 (Z/NZ). Let us compute the action of a group element h = (x, y, z) ∈ H2N in this model. For this we need to write (x , 0, 0) · h as a · (x , 0, 0), a ∈ Y . The relevant identity is (x , 0, 0)(x, y, z) = 0, y, z + xy + 2x y (x + x, 0, 0). Thus, the element h = (x, y, z) acts as z + xy + 2x y φ(x + x). hφ(x ) = e 2N In particular, (x, 0, 0) acts as translation by x and (0, y, 0) as a multiplication operator φ(x ) → e(x y/N)φ(x ). The center acts by the character (x0 , y0 , z) → e(z + H x0 y0 /2N). These show that π coincides with the induced representation IndY 2N τ . 3. Dynamics: Quantized cat maps. We now show how to assign to (certain) linear automorphisms A of the torus T2 , a unitary operator UN (A) on L2 (Z/NZ) that satisﬁes the following statement: for all observables f ∈ C ∞ (T2 ), UN (A)−1 OpN (f )UN (A) = OpN (f ◦ A). The ﬁnite modular group SL(2, Z/2N Z) acts by automorphisms on the Heisenberg groups H2N via (x, z)A := (xA, z), A ∈ SL(2, Z/2NZ). That this is indeed an auA tomorphism (i.e., (h1 h2 )A = hA 1 h2 ) follows from A preserving the symplectic form A B ω. Moreover, we have (h ) = hAB . Composing the representation π of H2N with A gives a new representation π A (h) := π(hA ), which is clearly still an irreducible N-dimensional representation. Its central character χ A can be easily computed as follows: if x0 , y0 ∈ N Z/2N Z and (x1 , y1 ) = (x0 , y0 )A, then χ A is given by z + x 1 y1 . χ A (x0 , y0 , z) = χ (x0 , y0 )A, z = e 2N This is the same character as χ if and only if x1 y1 ≡ x0 y0 mod 2N for all x0 , y0 ∈ NZ/2NZ. Writing A = ac db and x0 = N, y0 = Nη, , η ∈ Z/2Z, this is equivalent to requiring N ab 2 + cdη2 ≡ 0 mod 2, ∀, η ∈ Z/2Z, or Nab ≡ Ncd ≡ 0 mod 2. This is only a restriction if N a θ (2N) = c

is odd and is satisﬁed by the elements of the theta group b ∈ SL(2, Z/2N Z) : ab ≡ cd ≡ 0 mod 2 . d

HECKE THEORY AND EQUIDISTRIBUTION FOR LINEAR MAPS

57

Therefore, if A ∈ θ (2N), we get a unitarily equivalent representation π A of H2N . Thus, there is a unitary map UN (A), the quantum propagator associated to A, so that π hA = UN (A)−1 π(h)UN (A), ∀h ∈ H2N . In particular, we ﬁnd UN (A)−1 TN (n)UN (A) = TN (nA),

(3.1)

and consequently, for all observables f ∈ C ∞ (T2 ), OpN (f ◦ A) = UN (A)−1 OpN (f )UN (A). Now for any quantizable element A ∈ SL(2, Z) (that is, A = ac db with ab ≡ cd ≡ ¯ where 0 mod 2), we deﬁne the quantum propagator (or quantized cat map) to be UN (A) A¯ ∈ SL(2, Z/2NZ) is the reduction of A modulo 2N . Thus, by its construction, UN (A) only depends on the reduction A mod 2N. (This is a difference from the construction in Hannay and Berry [9].) (3.2)

4. Multiplicativity. The quantum propagators UN (A) are uniquely deﬁned up to a phase factor, because of the irreducibility of π (Schur’s lemma). Thus, they deﬁne a projective representation of θ (2N); that is, UN (AB) = eiφN (A,B) UN (A)UN (B)

A, B ∈ θ (2N).

Deﬁne the subgroup

g = I mod 4, (4, 2N) = g ∈ SL(2, Z/2N Z) : g = I mod 2,

(N even) . (N odd)

The goal of this section is to show that there is a choice of phases for the propagators UN (A) so that on the subgroup (4, 2N), the map A → UN (A) is a homomorphism. Theorem 5. There is a choice of quantum propagators so that UN (AB) = UN (A)UN (B),

A, B ∈ (4, 2N).

As a consequence, we ﬁnd the following corollary. Corollary 6. If A, B ∈ (4, 2N) commute mod 2N, then their propagators also commute: UN (A)UN (B) = UN (B)UN (A). Theorem 5 is essentially known in various guises and arose out of the study of theta functions and the Weil representation. One form is due to Kubota [14] (see also [8]). There are also treatments purely at the ﬁnite level [1] and [18]. Since Corollary 6 is absolutely crucial to our work, and we did not ﬁnd a good reference for the exact form

58

KURLBERG AND RUDNICK

that we need, we sketch a proof (or more precisely, a veriﬁcation) of Theorem 5. We wish to note that Theorem 5 is a priori more subtle than Corollary 6, since once we know that there is some choice of phases for which Corollary 6 holds, then it holds for all choices; this is not the case with Theorem 5.6 4.1. Reduction to prime powers. Factor 2N = p p kp = 2k p>2 p kp = 2k M, with M odd. The Chinese remainder theorem gives an isomorphism Z/p kp Z, Z/2NZ p

given by with inverse

x −→ x mod pkp p 2N rp xp mod 2N, xp mod p kp p −→ p kp

where rp is the inverse of 2N/p kp modulo p kp . Correspondingly, we have a bijection L2 (Z/2NZ) L2 Z/p kp Z . p

We deﬁne the phase space translations T (p) on L2 (Z/p kp Z) as in (2.4) by rp (n1 n2 + 2n2 Q) T (p) (n)ψ(Q) = e ψ(Q + n1 ). p kp It is then a simple matter to see that TN (n) = ⊗p T (p) (n), that is, if ψ = ⊗p ψp ∈ 2 kp p L (Z/p Z) is decomposable, then TN (n)ψ(Q) =

T (p) (n)ψ Q mod p kp .

p

This allows us to express the quantum propagators UN (A) as tensor products. Indeed, if we already have propagators U (p) (A) that satisfy U (p) (A)−1 T (p) (n)U (p) (A) = T (p) (nA),

(4.1) we then set

UN (A) := ⊗U (p) (A),

(4.2) which still satisﬁes 6 We

UN (A)−1 TN (n)UN (A) = TN (nA)

thank Jon Keating for emphasizing this point to us.

HECKE THEORY AND EQUIDISTRIBUTION FOR LINEAR MAPS

59

for all n ∈ Z2 , and therefore UN (A) coincides up to a phase with any other map satisfying this. We use this procedure to deﬁne UN (A) (that is, choose a phase) so that UN is an honest representation of a subgroup (4, 2N) of SL(2, Z/2NZ), not merely a projective representation. From the factorization property (4.2), it follows that it is enough to show that U (p) is a representation of SL(2, Z/p kp Z) when p > 2 is odd, and of (4, 2k ) if N = 2k−1 M is even. 4.2. Gauss sums. We need some preliminary information on Gauss sums. We deﬁne normalized Gauss sums −rax 2 1 Sr a, p k = e (4.3) . pk p k x mod pk For p odd, these are fourth roots of unity. To describe them, deﬁne for t ∈ (Z/p k Z)∗ , Sr t, p k . >r,pk (t) = Sr 1, p k Note that if t = t12 ∈ (Z/pk Z)∗ is a square, then >r,pk (t) = 1, since from (4.3) we ﬁnd after the change of variables x1 = t1 x that Sr (t, p k ) = Sr (1, p k ). For p odd, >r,pk is given in terms of the Legendre symbol as >r,pk (t) =

k t p

and is a character of (Z/pk Z)∗ : >r,pk (tt ) = >r,pk (t)>r,pk (t ). When p = 2, we have

>r,2k (t) =

−2k −r(t¯2 −1)/8 i , t

where t¯ is the smallest positive residue of t mod 4. In that case, it is not quite a character of the whole multiplicative group of Z/2k Z, but instead satisﬁes (4.4)

>r,2k (tt ) = (t, t )2 >r,2k (t)>r,2k (t ),

where (t, t )2 is the Hilbert symbol. In particular, if t, t = 1 mod 4, then the Hilbert symbol is trivial, and so we get a character of the subgroup {t = 1 mod 4} ⊂ (Z/2k Z)∗ (this is relevant for k ≥ 2) given simply by 1, t = 1 mod 8, >r,2k (t) = k (−1) , t = 5 mod 8.

60

KURLBERG AND RUDNICK

For p odd, we also need to know the normalized Gauss sum (4.3) when t = −1, in which case, we have  1, k even, Sr − 1, p k = r (p) , k odd, p where

(p) =

1, p = 1 mod 4, i, p = 3 mod 4.

4.3. p odd. We describe how to deﬁne U (p) on SL(2, Z/p k Z) so that it gives a representation (see Nobs [18] for details). This group is generated by the matrices t 1 1 b (4.5) , , , −1 1 t −1 and so it sufﬁces to specify U (p) on such matrices, provided we preserve all relations between them. This is done by the formulas (4.6) (4.7) (4.8)

rbx 2 1 b ψ(x) = e U ψ(x), 1 pk t ψ(x) = >r,pk (t)ψ(tx), U (p) t −1 1 2rxy 1 ψ(x) = Sr − 1, p k ψ(y)e . U (p) −1 pk pk k (p)

y mod p

It is easy to check that these satisfy (4.1). To see a veriﬁcation that this prescription does indeed give a consistent deﬁnition (that is, that all relations between the generators (4.5) are satisﬁed), see, for example, [18]. Once we have this, then we get U (p) (AB) = U (p) (A)U (p) (B) automatically. Remark 4.1. It is in fact the case that any projective representation of SL(2, Z/ p k Z), p odd, can be modiﬁed to give a representation (and more generally, SL(2, Z/ mZ) if m = 0 mod 4)—this is due to Schur [22] when k = 1. See [17] and [2] for the general case. 4.4. p = 2. Here we restrict to the subgroup (4, 2k ), k ≥ 2. The literature in this case is harder to come by, so we include complete proofs. We start by describing generators and relations for this group. More generally, let p be any prime and let k ≥ 2. Let p 2 , p k := g ∈ SL 2, Z/p k Z : g = I mod p 2 .

HECKE THEORY AND EQUIDISTRIBUTION FOR LINEAR MAPS

61

Lemma 7. (p 2 , p k ) has a presentation with generators u+ (x), u− (y), s(t), where x, y, t ∈ Z/p k Z, x, y ≡ 0 mod p 2 , t ≡ 1 mod p 2 , and relations (4.9)

u+ (x)u+ (x ) = u+ (x + x ),

(4.10)

u− (y)u− (y ) = u− (y + y ),

(4.11) (4.12) (4.13) (4.14)

s(t)s(t ) = s(tt ), s(t)u+ (x)s(t)−1 = u+ t 2 x , s(t)u− (y)s(t)−1 = u− t −2 y , s(d)u+ (a)u− (b) = u− d −1 b u+ (da),

d := (1 + ab)−1 .

Proof. Let G be the abstract group with the above presentation. We get a map B from G into (p 2 , p k ) by taking t 1 x 1 , u− (y) −→ , s(t) −→ . B : u+ (x) −→ 1 y 1 t −1 We verify that the relations hold in SL(2, Z/p k Z) so that B is a homomorphism. Next, note that we have a Bruhat decomposition for (p 2 , p k ): every element can be uniquely written in the form t 1 1 x , γ= 1 y 1 t −1 which follows from the formula −1 a b d =γ = c d

1 1 bd c/d 1 d

1

(note that since d = 1 mod p 2 , it is particularly invertible). This implies that the map B is surjective. To see that B is an isomorphism, it sufﬁces to show that every element of the abstract group G can also be written in the form g = s(t)u+ (x)u− (y), since then by the uniqueness of the decomposition in (p 2 , p k ), B is also one-to-one. With the aid of the ﬁrst ﬁve relations, every word W ∈ G can be written as a product: W = s(t1 )u+ (x1 )u− (y1 ) · . . . · s(tn )u+ (xn )u− (yn ), for some n ≥ 1. We prove by induction on n that we can write W = s(t)u+ (x)u− (y) for x, y = 0 mod p2 , t = 1 mod p2 . When n = 1, this holds trivially, and for n > 1, we use the relations (4.13) and (4.14) to write u− (yn−1 )s(tn )u+ (xn ) = s(tn )u− tn2 yn−1 u+ (xn ) = s(tn )s(t )u+ (x )u− (y ),

62

KURLBERG AND RUDNICK

and so W = s(t1 )u+ (x1 )u− (y1 ) · · · s(tn−1 )u+ (xn−1 )s(tn )s(t )u+ (x )u− (y )u− (yn ) = s(t1 )u+ (x1 )u− (y1 ) · · · s(tn−1 )u+ (xn−1 )u− (yn−1 )

after a further application of the ﬁrst ﬁve relations. The result now follows by induction. We now specify the propagators U (2) (A) for the generators: for t 1 a and , 1 t −1 they are given by the same formulas (4.6) and (4.7). For the matrices

−1 1 1 1 −b 1 , = −1 −1 1 b 1

we conjugate (4.6) by an analogue of the Fourier transform (4.8) and deﬁne r − bz2 + 2z(y − x) 1 (2) 1 ψ(x) = ψ(y) k e (4.15) U . b 1 2 2k k k y mod 2

z mod 2

To show that this deﬁnes a representation, we have to check that all the relations of Lemma 7 are satisﬁed. The ﬁrst ﬁve are fairly straightforward, bearing in mind that > is a character of the multiplicative group of residues t = 1 mod 4 (see (4.4)). The last relation (4.14) requires verifying an identity of Gauss sums: unwinding the action of the right and left-hand sides in (4.14), we must show that r >(d) ψ(y)e k 2yz − bz2 − 2dxz + ad 2 x 2 2 k k z mod 2 y mod 2

=

z mod 2k y mod 2k

ψ(y)e

r −1 2 2 bz − 2xz + ady 2yz − d . 2k

Now d ≡ 1 mod 16 implies that >(d) = 1 since d is then a square modulo 2k , and if the identity is to hold for all ψ and all values of x, we obtain that for all x, y, r e k − bz2 + 2z(y − dx) + ad 2 x 2 2 z mod 2k r = e k − d −1 bz2 + 2z(y − x) + ady 2 . (4.16) 2 k z mod 2

We verify this in Appendix A.

HECKE THEORY AND EQUIDISTRIBUTION FOR LINEAR MAPS

63

5. Hecke operators. We now introduce a commutative group of unitary operators on L2 (Z/NZ) that commute with UN (A). For this, we have to bring in the theory of quadratic ﬁelds (see [19] for a survey in connection to cat maps). 5.1. Integral matrices and quadratic ﬁelds. Let A ∈ SL2 (Z) be a hyperbolic matrix: | tr A| > 2. The eigenvalues α, α −1 of A generate a ﬁeld extension K = Q(α), which is a real quadratic ﬁeld since tr(A)2 > 4. We denote by OK the ring of integers of K. The eigenvalues α, α −1 of A are units in OK . Adjoining α to Z gives an order O = Z[α] ⊆ OK in K. We claim that there is an O-ideal I ⊂ O so that the action of α by multiplication on I is equivalent to the action of A on Z2 , in the sense that there is a basis of I with respect to which the matrix of α is precisely A. The construction is as follows (refer to [23]): since α is an eigenvalue of A, there is a vector v = (v1 , v2 ) such that vA = αv and v ∈ O2 . Let I := Z[v1 , v2 ] ⊂ O. Then I is in an O-ideal, and the matrix of α acting on I by multiplication in the basis v1 , v2 is precisely A. Remark 5.1. It is easy to check that the above construction sets up a bijection between GL2 (Z)-conjugacy classes of elements in SL2 (Z) with eigenvalues α, α −1 and ideal classes in the order O. (Recall that two ideals, I1 , I2 ; are said to be in the same ideal class if there exist nonzero a, b ∈ O so that aI1 = bI2 .) In the same way, the action of O by multiplication on I gives us an embedding ι : O → Mat 2 (Z) so that γ = x + yα ∈ O corresponds to xI + yA. Moreover, the determinant of xI + yA equals ᏺ(γ ) = γ γ¯ , where ᏺ : K → Q is the Galois norm. In particular, if γ ∈ O has norm 1, then γ corresponds to an element in SL2 (Z), and if in addition γ ≡ 1 mod 4O, then γ corresponds to an element in (4). 5.2. Hecke operators. Given an integer M ≥ 1, the embedding ι : O → Mat 2 (Z) induces a map ιM : O/M O → Mat2 (Z/MZ), and the norm ᏺ : K → Q gives a well-deﬁned map ᏺ : O/M O −→ Z/MZ. We let ᏯA (M) be the group of norm-one elements in O/M O:

ᏯA (M) = ker ᏺ : (O/M O)∗ −→ (Z/MZ)∗ . Similarly, replacing the order O by the maximal order OK , we set

ᏯK (M) = ker ᏺ : (OK /M OK )∗ −→ (Z/MZ)∗ to be the norm-one elements in OK /M OK . If M = 2N is even, we set ᏯθA (M) to be the elements of ᏯA (2N) that are congruent to 1 modulo 4O (resp., 2O) if N is even (resp., odd). For M odd, we set ᏯθA (M) = ᏯA (M).

64

KURLBERG AND RUDNICK

By construction, the image of ᏯθA (2N) in Mat 2 (Z/2NZ) lies in (4, 2N). Since α commutes with all elements in ᏯθA (2N), we see that A commutes, modulo 2N, with the elements in ι(ᏯθA (2N)). Thus, by Corollary 6, the quantizations UN (ι(β)) of β ∈ ᏯθA (2N) commute with UN (A) and with each other. We call these Hecke operators. We need to know the number of Hecke operators. Lemma 8. The number of elements of ᏯθA (2N) satisﬁes

N 1− ᏯθA (2N) N 1+ , ∀ > 0. Proof. Since the reduction map O → O/4O has image of size 42 , ᏯθA (2N) has bounded index in ᏯA (2N ). The inclusion O ⊂ OK induces a map O/M O → OK /M OK , which has kernel and cokernel of size at most [OK : O], independent of M. Therefore, the induced map ᏯA (M) → ᏯK (M) on norm-one elements also has bounded kernel and cokernel. Thus, it sufﬁces to prove the lemma in the case of the maximal order OK . By the Chinese remainder theorem, it sufﬁces to prove it in the case of prime powers, which is given in Appendix B by Lemma 19. 5.3. Hecke eigenfunctions. The Hecke operators UN (ι(β)), β ∈ ᏯθA (2N), commute with each other and with UN (A). Therefore, the eigenspaces of the unitary map UN (A) break up into joint eigenspaces of the Hecke operators. Such a joint eigenfunction we call a Hecke eigenfunction. In other words, there exist an orthonormal basis {φi } of L2 (Z/NZ) and characters λi of ᏯθA (2N) such that φi are eigenfunctions of UN (A) and UN ι(β) φi = λi (β)φi , ∀β ∈ ᏯθA (2N). We call such a basis of L2 (Z/NZ) a Hecke basis. 6. Ergodicity of Hecke eigenfunctions. In the next two sections, we show that if φ ∈ L2 (Z/NZ) is a normalized Hecke eigenfunction, then the expectation values OpN (f )φ, φ converge to the classical phase-space average T2 f for all smooth observables (see Theorem 1). In fact, we show something stronger. Theorem 9. Let φi ∈ L2 (Z/NZ), i = 1, . . . , N be any orthonormal basis of Hecke eigenfunctions of UN (A). Then N

Op (f )φi , φi − N

i=1

4

f (x) dx

f, N −1+ . 2

T

6.1. Proof of Theorem 9. To prove this theorem, it sufﬁces to prove it for the basic observables f (x) = e(nx), 0 = n ∈ Z2 , that is, to show that the following theorem holds.

65

HECKE THEORY AND EQUIDISTRIBUTION FOR LINEAR MAPS

Theorem 10. Let 0 = n ∈ Z2 , and let φi ∈ L2 (Z/NZ), i = 1, . . . , N be any orthonormal basis of Hecke eigenfunctions of UN (A). Then N

TN (n)φi , φi 4 |n|16 N −1+ ,

N −→ ∞.

i=1

The proof of Theorem 9 from Theorem 10 is easy using the rapid decay of the Fourier coefﬁcients of f . Indeed, write f (x) = n∈Z2 f(n)e(nx), so that OpN (f ) = n∈Z2 f (n)TN (n). Therefore,

4 N

Op (f )φ N , φ N − f (x) dx

N i i

i=1

T2

4

4 N N

f(nk ) TN (nk )φi , φi .

≤ = (n)φ , φ f (n) T N i i

i=1 0 =n∈Z2 i=1 n1 ,...,n4 =0 k=1

For notational convenience, we write

ti (n) := TN (n)φi , φi .

Now interchange the order of summation and apply Cauchy-Schwartz twice. For ﬁxed n1 , n 2 , n 3 , n 4 , N

ti (n1 )ti (n2 )ti (n3 )ti (n4 )

i=1

≤

N

2

ti (n1 )ti (n2 )

1/2 N

i=1

2

1/2

ti (n3 )ti (n4 )

≤

i=1

Now use Theorem 10. For nk = 0, N ti (nk )4

N 4 k=1

1/4

ti (nk )

i=1

1/4

|nk |4 N −1/4+ ,

i=1

and so we get N

ti (n1 )ti (n2 )ti (n3 )ti (n4 ) N −1+

i=1

4

|nk |4 .

k=1

Now sum over all possible nk = 0 to ﬁnd N

Op (f )φi , φi − N

i=1

which proves Theorem 9.

 4

4

f(n)|n|4  , f (x) dx

N −1+  2

T

n=0

4

.

66

KURLBERG AND RUDNICK

6.2. Reduction to a counting problem. We ﬁrst reduce Theorem 10 to a counting problem. Proposition 11. Fix 0 = n = ι(ν) ∈ Z2 , ν ∈ I . Then for any Hecke basis of eigenfunctions φi , N

TN (n)φi , φi 4 i=1

N θ ≤

4 # βi ∈ ᏯA (2N) : ν β1 − β2 + β3 − β4 = 0 mod NI . θ

Ꮿ (2N)

A

In order to prove Proposition 11, we deﬁne for n = ι(ν), 0 = ν ∈ I , 1

D = D(n) = θ

Ꮿ (2N)

A

−1 UN ι(β) TN (n)UN ι(β) .

β∈ᏯθA (2N)

If (tij ) is the matrix coefﬁcients of TN (n) expressed in the eigenvector basis {φk } so that tij = TN (n)φi , φj , then we see that 1

Dij = θ

Ꮿ (2N)

A

λi (β)λj (β)tij .

β∈ᏯθA (2n)

Since the sum of a nontrivial character over all elements in a group vanishes, we have tij , if λi = λj , (6.1) Dij = 0, otherwise. Lemma 12. With D deﬁned as above, we have |tij |4 ≤ tr (D ∗ D)2 . λi =λj

Proof. Let D = (dij ) = (vi ) where the vi ’s are the column vectors of D. Examining the (k, k)-entry of (D ∗ D)2 , we get

∗ 2

vi , vk 2 , vi , vk vk , vi = (D D) kk = i

and hence,

i

vk , vk 2 ≥ |dij |4 . tr (D ∗ D)2 ≥ k

The result now follows from (6.1).

i,j

HECKE THEORY AND EQUIDISTRIBUTION FOR LINEAR MAPS

67

Lemma 13. We have

N

βi ∈ Ꮿθ (2N) : ν(β1 − β2 + β3 − β4 ) ≡ 0 mod NI . tr (D ∗ D)2 ≤

A 4

Ꮿθ (2N)

A Proof. Recall that by (3.1), since n · ι(β) = ι(νβ) for β ∈ O, n = ι(ν), −1 UN ι(β) TN (n)UN ι(β) = TN ι(νβ) . Also note that TN (w)∗ = TN (−w) for all w by (2.5). Substituting the deﬁnition of D and expanding, we see that (D ∗ D)2 is given by 1/|ᏯθA (2N)|4 times a sum, ranging over all β1 , β2 , β3 , β4 ∈ ᏯθA (2N ), of terms TN ι(νβ1 ) TN − ι(νβ2 ) TN ι(νβ3 ) TN − ι(νβ4 ) = γ (β1 , β2 , β3 , β4 )TN ι ν(β1 − β2 + β3 − β4 ) , where γ (β1 , β2 , β3 , β4 ) has absolute value 1 (see (2.6)). Now take the trace; by Lemma 4, the absolute value of the trace of TN (n) equals N if n ≡ (0, 0) mod N, and equals zero otherwise. The result now follows by taking absolute values and summing over all β1 , β2 , β3 , β4 ∈ ᏯθA (2N). It remains to estimate the number of solutions of (6.2)

ν(β1 − β2 + β3 − β4 ) ≡ 0 mod NI,

βi ∈ ᏯθA (2N).

Proposition 14. The number of solutions to (6.2) is bounded by O(|ᏺ(ν)|8 N 2+ ). 6.3. Proof of Theorem 10: Conclusion. By Proposition 11, we need a suitable upper bound for the number of solutions of (6.2) and a lower bound for the number of elements of ᏯθA (2N). By Proposition 14, the number of solutions is at most |ᏺ(ν)|8 N 2+ . Note that |ᏺ(ν)| |n|2 . From Lemma 8, we obtain that |ᏯθA (2N)| # N 1− and the result follows. 7. Counting solutions. In this section, we prove Proposition 14. 7.1. A reduction. Since NI ⊆ N O ⊆ N OK , the number of solutions to (6.2) is bounded by the number of solutions to ν β1 − β2 + β3 − β4 ∈ N OK , βi ∈ ᏯθA (2N). Moreover, at the cost of increasing slightly the number of solutions, we may omit the parity condition on βi , replacing ᏯθA (2N) by ᏯA (2N). The inclusion O ⊂ OK induces a map O/M O → OK /M OK , which has kernel and cokernel of size at most [OK : O], independent of M. Therefore, the induced map

ᏯA (M) = ker (O/M O)∗ −→ (Z/MZ)∗ −→ ᏯK (M)

= ker (OK /M OK )∗ −→ (Z/MZ)∗

68

KURLBERG AND RUDNICK

on norm-one elements also has bounded kernel and cokernel. Thus, up to a bounded factor (depending on A but not on N or ν), the number of solutions to (6.2) is bounded by the number of solutions of ν β1 − β2 + β3 − β4 = 0 mod N OK , βi ∈ ᏯK (2N). (7.1) At the cost of increasing the number of solutions, we multiply (7.1) by the Galois conjugate ν¯ to get ᏺ(ν) β1 − β2 + β3 − β4 = 0 mod N OK , βi ∈ ᏯK (2N). Setting N =

N , gcd(N, ᏺ(ν))

this equation is equivalent to (7.2)

β1 − β2 + β3 − β4 = 0 mod N OK ,

βi ∈ ᏯK (2N).

Next, note that the reduction map OK /rs OK → OK /r OK has kernel r OK /rs OK OK /s OK of size s 2 , and so the induced map on norm-one elements ᏯK (rs) → ᏯK (r) has kernel of order at most s 2 . (This is crude, but sufﬁcient for our purposes.) Thus, the reduction map ᏯK (2N) → ᏯK (N ) has kernel of size at most 4 gcd(N, ᏺ(ν))2 ≤ 4|ᏺ(ν)|2 . Therefore, the number of solutions of (7.2) is bounded by (4|ᏺ(ν)|2 )4 times the number of solutions of the equation (7.3)

β1 − β2 + β3 − β4 = 0 mod N OK ,

βi ∈ ᏯK (N ).

Equation (7.3) is invariant under Galois conjugation, and we obtain a second equation (note that β¯ = β −1 since ᏺ(β) = 1 mod N ): (7.4)

β1−1 − β2−1 + β3−1 − β4−1 ≡ 0 mod N OK .

7.2. A transformation. We thus have a system of equations (7.3) and (7.4), which we transform using the following lemma. Lemma 15. If x, y, z, w are invertible, then the system of equations x +y = z+w x −1 + y −1 = z−1 + w −1 is equivalent to the system (z − x)(z − y)(x + y) = 0 w = x + y − z.

HECKE THEORY AND EQUIDISTRIBUTION FOR LINEAR MAPS

69

Proof. From the second equation, we get x +y z+w = , xy zw or (x + y)zw = (z + w)xy. The ﬁrst equation gives us that w = x + y − z; inserting it in (x + y)zw = (z + w)xy, we get (x + y)z(x + y − z) = (x + y)xy or

0 = (x + y) zx + zy − z2 − xy = −(z − x)(z − y)(x + y).

Thus, by Lemma 15, the system of equations (7.3) and (7.4) is equivalent to the system (7.5)

(β3 − β1 )(β3 − β2 )(β1 + β2 ) ≡ 0 mod N OK ,

(7.6)

β4 ≡ β1 − β2 + β3 mod N OK ,

with βi ∈ ᏯK (N ). Since β4 is determined by β1 , β2 , β3 , we may ignore (7.6) (at the cost of increasing the number of solutions, since being in ᏯK (N ) is a nonempty condition). Multiplying equation (7.5) by β3−3 and letting βi = βi /β3 , we obtain (7.7)

1 − β1 1 − β2 β1 + β2 ≡ 0 mod N OK .

Since β3 is arbitrary, the number of solutions of (7.5) is bounded by |ᏯK (N )| times the number of solutions in β1 , β2 ∈ ᏯK (N ) to (7.7). 7.3. Prime powers. By the Chinese remainder theorem, the number of solutions to (7.7) is multiplicative, and we may concentrate on the prime power case. Thus, we need to count the solutions to the equation 1 − β1 1 − β2 β1 + β2 ≡ 0 mod p k OK (7.8) with βi ∈ OK /p k OK , ᏺ(βi ) = 1 mod p k . We ﬁrst recall some properties of primes in quadratic extensions: let P |p be a prime in OK lying above p, and let e denote the ramiﬁcation index, that is, the largest integer e such that P e |pOK . Since K is quadratic, e ∈ {1, 2} and e = 1 for all but ﬁnitely many primes p. If e = 2, then p is said to be ramiﬁed. If e = 1, then p is called unramiﬁed, and one of two things can happen: either p OK = P is still a prime ideal, in which case p is said to be inert, or pOK = P P , in which case p is said to split.

70

KURLBERG AND RUDNICK

Now, ﬁx a prime p with ramiﬁcation index e, be it 1 or 2. The norm map ᏺ : OK → Z gives a well-deﬁned homomorphism: × × OK /P ek −→ Z/p k . We let

OK /P ek

1

be the kernel of this map, that is, the group of norm-one elements. For l ≤ ek, we let

1+P l

%

1 + P ek

1

be the norm-one elements in the subgroup (1+P l )/(1+P ek ); these are precisely the norm-one elements that reduce to 1 modulo P l . Lemma 16. There is a constant c > 1 so that the number of solutions of (7.8) is at most ckp k . Proof. Equation (7.8) is invariant under Galois conjugation; therefore, its solutions in OK /p k OK correspond bijectively to solutions βi ∈ OK /P ek , ᏺ(βi ) = 1 mod pk (this is, of course, only an issue in the split case where OK /p k OK OK /P k × k OK /P ). Thus, we need to count solutions of 1 − β1 1 − β2 β1 + β2 ≡ 0 mod P ek (7.9) with βi ∈ OK /P ek , ᏺ(βi ) = 1 mod p k . We ﬁrst assume that p is odd. Since β1 ≡ β2 ≡ 1 mod P implies that β1 + β2 ≡ 2 ≡ 0 mod P , we see that at most two of the factors in (7.9) can be congruent to zero modulo P . Moreover, we may assume that the third factor is nonzero by multiplying by a suitable β and permuting the variables. (Of course, we must then compensate by multiplying the number of solutions by 23 .) Now if the product is zero modulo P ek , then there is some 0 ≤ n ≤ ek such that one factor is zero modulo P n and the other is zero modulo P ek−n . Thus, the number of solutions to (7.9) equals ek−1

% %

% ek 1

3

n ek 1

ek−n ek 1

1 + P 1 + P × 1 + P + 2 O 1 + P

K P

. 2 n=1

Using Lemma 20, we obtain

% 1

% 1

1 + P n 1 + P ek × 1 + P ek−n 1 + P ek ≤ p k+e−1 , and by Lemma 19, we obtain

% 1

OK P ek ≤ 2(p + 1)p k−1 .

HECKE THEORY AND EQUIDISTRIBUTION FOR LINEAR MAPS

71

Hence, for p odd, the total number of solutions to (7.9) is bounded by 4(p + 1)p k−1 + 3(ek − 1)p e−1 p k kp k (since e = 1 for all but ﬁnitely many primes). If p = 2, it is no longer true that only two factors can be zero modulo P . However, β1 ≡ β2 ≡ 1 mod P e+1 implies that β1 + β2 ≡ 2 mod P e+1 . Since 2OK = P e , we see that if two factors are zero modulo P e+1 , then the third factor can be congruent to 0 at most modulo P e . We may thus bound the number of solutions by counting the number of ways the product of two factors can be equal to zero modulo P ek−e . This we can do as we did for odd primes, and we obtain the same bound as before, except that we lose an additional factor of at most

% 1

4

1 + P ek−e 1 + P ek 2O(e) = O(1). This proves Lemma 16. 7.4. Proof of Proposition 14. By multiplying over all primes, we see from Lemma 16 that the number of solutions of equation (7.7) is O((N )1+ ). Therefore, we see that the number of solutions of (7.5) is O((N )2+ ) since |ᏯK (N )| (N )1+ by Lemma 19. This gives a bound for the solutions of (7.3), and multiplying by |ᏺ(ν)|8 gives a bound for the number of solutions of (7.2). In turn, by the reasoning in Section 7.1, we get a bound of O(|ᏺ(ν)|8 N 2+ ) on the solutions of (6.2). Appendices Appendix A. An identity of Gauss sums. For Section 4, we need to prove (4.16). To prove it we need a lemma about Gauss sums. Given an integer x, we deﬁne its dyadic valuation, v(x), by x = 2v(x) x0 , where x0 is an odd integer. Let r G(b, c) = e k − bz2 + 2cz . 2 k z mod 2

Lemma 17. If v(c) < v(b) < k, then 2k , if v(b) = k − 1 and v(c) = k − 2, G(b, c) = 0, otherwise. Proof. We may write G(b, c) =

z mod 2

2cr 2 e − βz + z , 2k k

where β is an integer satisfying 2cβ ≡ b mod 2k . Let n = k −1−v(c); it is the smallest integer n such that e((2cr/2k )x) = 1 for all x ≡ 0 mod 2n .

72

KURLBERG AND RUDNICK

First, assume that n > 1. Let = 0 2n−1 be such that e((2cr/2k )) = 1. Making the change of variables z → z + , we see that 2cr 2 2cr 2 G(b, c) = e + 2z + − β z + z + = G(b, c)e 2k 2k k z mod 2

since 2z + 2 ≡ 0 mod 2n . But e((2cr/2k )) = 1, and therefore, G(b, c) = 0. If n ≤ 1, then as n = k −1−v(c) and v(c) < v(b) < k, we must have n = 1, v(c) = k − 2, and v(b) = k − 1. Hence, β ≡ 1 mod 2. Moreover, if n = 1, we must have e(2crx/2k ) = e(x/2). Thus G(b, c) =

z mod 2

z2 + z e 2 k

= 2k

since z2 + z ≡ 0 mod 2 for all z. Proposition 18. The following equality holds for all x, y: r e k − bz2 + 2z(y − dx) + ad 2 x 2 2 z mod 2k r = e k − d −1 bz2 + 2z(y − x) + ady 2 . 2 k z mod 2

Proof. The case v(b) ≥ k, that is, b ≡ 0 mod 2k , implies that d ≡ 1 mod 2k and the equality holds trivially. We may thus assume that v(b) < k. We begin by noting that since y −dx = d(d −1 y −x) = d(y −x +aby), we see that v(y − x) < v(b) implies that v(y − dx) < v(b); putting x = d −1 x, we see that the converse holds, and hence, v(y − x) < v(b) if and only if v(y − dx) < v(b). First case: v(y − x) < v(b). Putting c = y − x, c = y − dx, respectively, and applying Lemma 17, we see that both sides are zero except when v(c) = k − 2 and v(b) = k − 1. For the exceptional case, we note that v(b) = k − 1 implies that d −1 = 1 + ab ≡ 1 mod 2k , and the same holds for d. Moreover, v(c) = k − 2 means that x ≡ y mod 2k−2 , and since 4 | a, we see that r r LHS = 2k e k ad 2 x 2 = 2k e k ady 2 = RHS. 2 2 Second case: v(y − x) ≥ v(b). As remarked above, this means that v(y − dx) ≥ v(b). We may thus complete the squares inside the exponentials, and we get LHS =

z mod 2

r y − dx 2 (y − dx)2 2 2 e k −b z− + + ad x 2 b b k

HECKE THEORY AND EQUIDISTRIBUTION FOR LINEAR MAPS

73

and RHS =

z mod 2k

e

r d(y − x) 2 d(y − x)2 −1 2 + ady b z − + − d . 2k b b

After changing variables and taking constants outside, we get r r (y − dx)2 2 2 2 + ad x e k − bz LHS = e k 2 b 2 k z mod 2

and

r d(y − x)2 r 2 −1 2 RHS = e k + ady e k − d bz . 2 b 2 k z mod 2

k Now, √ d ≡ 1 mod 16 means that d is a square modulo 2 . Changing variables by z → dz in the second sum, we see that the sums are equal, and we are left to prove that r (y − dx)2 r d(y − x)2 2 2 2 e k + ad x + ady =e k . 2 b 2 b

This follows from the equality (y − dx)2 d(y − x)2 + ad 2 x 2 = + ady 2 . b b Collecting terms, it is equivalent to 0 = ad y 2 − dx 2 + b−1 dy 2 + dx 2 − 2dxy − y 2 − d 2 x 2 + 2dxy = ad y 2 − dx 2 + b−1 y 2 (d − 1) + x 2 d − d 2 = ad y 2 − dx 2 + (d − 1)b−1 y 2 − dx 2 , which follows from the identity (d − 1) 1 − 1/d 1 − (1 + ab) ab ad + = d a+ = d a+ = d a− = 0. b b b b Appendix B. Counting norm-one elements. Let e be the ramiﬁcation index of a prime p in OK , that is, the largest integer such that P e | p OK , where P ⊂ OK is any prime ideal dividing p OK . Since K is quadratic, e ∈ {1, 2}. If e = 2, then p is said to be ramiﬁed. If e = 1, then p is called unramiﬁed, and one of two things can happen: either p OK = P , in which case p is said to be inert, or pOK = P P , in which case p is said to be split.

74

KURLBERG AND RUDNICK

Now ﬁx a prime p with ramiﬁcation index e, be it 1 or 2. The norm map ᏺ : OK −→ Z

descends modulo pk and gives a homomorphism % × % × OK P ek −→ Z p k . We let (OK /P ek )1 be the kernel of this map, that is, the group of norm-one elements. For l ≤ ek, we let % 1 1 + P l ) 1 + P ek be the norm-one elements in the subgroup (1+P l )/(1+P ek ); these are precisely the norm-one elements that reduce to 1 modulo P l . Lemma 19. We have

 (p − 1)p k−1 , if p is split,

  1

OK /P ek = (p + 1)pk−1 , if p is inert,    k if p is ramiﬁed. 2p ,

Proof. Recall ﬁrst from class ﬁeld theory [4] that the index (in Z× p ) of the image of the units in the p-adic completion of OK under the norm map equals the ramiﬁcation index e. We split the proof into three parts. The split case. If p splits in K, then pOK = P1 P2 where P1 , P2 are prime ideals in OK , and where P2 = P1 . The map x → x gives an isomorphism between OK /P1k and OK /P2k . This, together with the Chinese remainder theorem, gives % % % % % OK p k OK OK P1k × OK P2k OK P1k × OK P1k , where x ∈ OK /p k OK is mapped to (x, x) ∈ OK /P1k × OK /P1k . Furthermore, OK /P1k Z/p k Z, and therefore, % % % OK p k OK Z p k Z × Z p k Z. (B.1) Under this isomorphism, Galois conjugation maps (x, y) ∈ Z/p k Z×Z/p k Z to (y, x). Thus the natural embedding of Z/p k Z in OK /p k OK Z/p k Z × Z/p k Z consists of elements of the form (x, x) and the image of (x, y) under the norm map is (xy, xy). Hence, the norm-one elements in OK /p k OK correspond to elements of the form (x, y) ∈ Z/p k Z × Z/p k Z such that xy = 1, and the number of such elements is (p − 1)p k−1 . The inert case. Here e = 1 and the local norm map is onto Z× p . Reducing modulo p, we get an exact sequence

HECKE THEORY AND EQUIDISTRIBUTION FOR LINEAR MAPS

75

% 1 % × % × 1 −→ OK P k −→ OK P k −→ Z pk −→ 1. Hence,

% ×

% k 1

OK P k

OK P = % × = (p + 1)p k−1 .

Z pk

The ramiﬁed case. Here the image of the norm map in Z× p is of index 2, and thus k × k−1 the image of the norm in (Z/p ) has cardinality (p − 1)p /2. Consequently,

%

OK P ek ×

% ek 1

.

OK P

=2 (p − 1)p k−1 Now

% % %

OK P ek × = OK P × × (1 + P ) 1 + P ek = (p − 1)p ek−1 , and since e = 2, we get

% 1

(p − 1)p 2k−1

= 2p k .

OK P ek = 2 (p − 1)p k−1 We also need to know the number of norm-one elements that reduce to 1 modulo P l . Lemma 20. We have

% if p is split or inert, p k−l , 1

1 + P ek =

1+P l k+$l/2%−l Kp × p , if p is ramiﬁed, where Kp = 1 if p is odd, and K2 = 1 or 2. Proof. The split case. From the previous discussion of the isomorphism in (B.1), we see that norm-one elements congruent to 1 modulo P1l correspond to elements (x, x −1 ) ∈ Z/p k Z×Z/pk Z, such that x ≡ 1 mod p l . The number of such elements is |(1 + p l )/(1 + p k )| = pk−l . The inert case. If p is odd, then x → x 2 is an automorphism of (1+P l )/(1+P k ) since the order of the group is odd. Thus, the norm is locally onto in the sense that the map % % 1 + P k −→ 1 + p l 1 + pk ᏺ : 1+P l is onto. If p is even (and inert), then squaring is not an automorphism as (1 + x)2 = 1 + 2x + x 2 . However, 1 + p l ⊂ 1 + P l and squaring maps (1 + p l )/(1 + p k ) onto (1 + p l+1 )/(1 + p k ). Thus, % % 1 + p l+1 1 + p k ⊂ ᏺ 1 + P l 1+P k ,

76

KURLBERG AND RUDNICK

which shows that the image of the norms must be either (1 + p l+1 )/(1 + p k ) or (1 + p l )/(1 + p k ). (There are no subgroups in between!) We show that the√former holds; since 2 is unramiﬁed, the discriminant of K is odd and OK = Z[1 + dk /2]. Hence, tr(OK ) = Z, and there exists x ∈ OK with odd trace. Now ᏺ 1 + p k x = 1 + p k tr(x) + p 2k ᏺ(x) shows that the image must be (1 + p l )/(1 + p k ). Thus, whether p is even or odd, the norm map is locally onto, and hence,

% ×

% 1 + P l 1+P k

l k 1

k−l 1+P

1+P

= % × = p . k

1 + pl

1+p The ramiﬁed case. First, we note that % % ᏺ 1+P l (B.2) 1 + P ek ⊂ 1 + p $l/2% 1 + p k . Arguing as before that squares are in the image of the norm, we see that equality holds for p odd, and we obtain

% 1

1 + P ek

1+P l

% ×

%

OK P 2k−l

1+P l

1 + P ek p 2k−l = = k−$l/2% = p k+$l/2%−l . % × = k−$l/2%

1 + p $l/2% 1 + p k

p p For p even, the squaring argument shows that % % 1 + p $l/2%+1 1 + p k ⊂ ᏺ 1 + P l 1 + P ek , which gives a lower bound on the image. This gives the same result as for the odd case, except for a factor of 2. References [1] [2] [3] [4]

[5] [6]

R. Balian and C. Itzykson, Observations sur la mécanique ﬁnie, C. R. Acad. Sci. Paris Sér. I Math. 303 (1986), 773–778. F. R. Beyl, The Schur Multiplicator of SL(2, Z/mZ) and the congruence subgroup property, Math. Z. 191 (1986), 23–42. A. Bouzouina and S. De Bièvre, Equipartition of the eigenfunctions of quantized ergodic maps on the torus, Comm. Math. Phys. 178 (1996), 83–105. J. W. S. Cassels and A. Frohlich, eds., Algebraic Number Theory, Proceedings of an instructional conference organized by the London Mathematical Society (University of Sussex, Brighton, 1965), Thompson, Washington, 1967. Y. Colin de Verdière, Ergodicité et fonctions propres du laplacien, Comm. Math. Phys. 102 (1985), 497–502. M. Degli Esposti, Quantization of the orientation preserving automorphisms of the torus, Ann. Inst. H. Poincaré 58 (1993), 323–341.

HECKE THEORY AND EQUIDISTRIBUTION FOR LINEAR MAPS [7] [8] [9] [10] [11] [12] [13] [14]

[15] [16] [17] [18] [19] [20] [21] [22] [23]

[24] [25]

77

M. Degli Esposti, S. Grafﬁ, and S. Isola, Classical limit of the quantized hyperbolic toral automorphisms, Comm. Math. Phys. 167 (1995), 471–507. S. Gelbart, Weil’s Representation and the Spectrum of the Metaplectic Group, Lecture Notes in Math. 530, Springer, Berlin, 1976. J. H. Hannay and M. V. Berry, Quantization of linear maps on a torus-Fresnel diffraction by a periodic grating, Phys. D 1 (1980), 267–290. D. Jakobson, Quantum unique ergodicity for Eisenstein series on PSL2 (Z)\ PSL2 (R), Ann. Inst. Fourier (Grenoble) 44 (1994), 1477–1504. J. P. Keating, The cat maps: quantum mechanics and classical motion, Nonlinearity 4 (1991), 309–341. S. Klimek, A. Lésniewski, N. Maitra, and R. Rubin, Ergodic properties of quantized toral automorphisms, J. Math. Phys. 38 (1997), 67–83. S. Knabe, On the quantisation of Arnold’s cat, J. Phys. A 23 (1990), 2013–2025. T. Kubota, On automorphic functions and the reciprocity law in a number ﬁeld, Lectures in Mathematics (Department of Mathematics, Kyoto University) 2, Kinokuniya, Tokyo, 1969. W. Luo and P. Sarnak, Quantum ergodicity of eigenfunctions on PSL2 (Z)\H2 , Inst. Hautes Études Sci. Publ. Math. 81 (1995), 207–237. J. Marklof and Z. Rudnick, Quantum unique ergodicity for parabolic maps, to appear in Geom. Funct. Analysis. J. Mennicke, On Ihara’s modular group, Invent. Math. 4 (1967), 202–228. A. Nobs, Die irreduziblen Darstellungen der Gruppen SL2 (Zp ), insbesondere SL2 (Z2 ). I, Comment. Math. Helv. 51 (1976), 465–489. I. Percival and F. Vivaldi, Arithmetical properties of strongly chaotic motions, Phys. D 25 (1987), 105–130. Z. Rudnick and P. Sarnak, The behaviour of eigenstates of arithmetic hyperbolic manifolds, Comm. Math. Phys. 161 (1994), 195–213. A. Schnirelman, Ergodic properties of eigenfunctions, Uspekhi Math. Nauk 29 (1974), 181– 182. I. Schur, Untersuchungen über die Darstellung der endlichen Gruppen durch gebrochene lineare Substitutionen, J. Reine Angew. Math. 132 (1907), 85–137. O. Taussky, “Connections between algebraic number theory and integral matrices”, appendix to A Classical Invitation to Algebraic Numbers and Class Fields, by H. Cohn, Universitext, Springer, New York, 1978. S. Zelditch, Uniform distribution of eigenfunctions on compact hyperbolic surfaces, Duke Math. J. 55 (1987), 919–941. , Index and dynamics of quantized contact transformations, Ann. Inst. Fourier (Grenoble) 47 (1997), 305–363.

Kurlberg: Raymond and Beverly Sackler School of Mathematical Sciences, Tel Aviv University, Tel Aviv 69978, Israel; Current: Department of Mathematics, University of Georgia, Athens, Georgia 30602, USA; [email protected] Rudnick: Raymond and Beverly Sackler School of Mathematical Sciences, Tel Aviv University, Tel Aviv 69978, Israel; [email protected]

Vol. 103, No. 1

DUKE MATHEMATICAL JOURNAL

© 2000

INTERSECTION COHOMOLOGY OF S 1 SYMPLECTIC QUOTIENTS AND SMALL RESOLUTIONS E. LERMAN and S. TOLMAN 1. Introduction. Let a compact Lie group G act effectively on a compact connected symplectic manifold M with a moment map : M → g∗ . When 0 is a regular value of the moment map, the symplectic quotient (reduced space) Mred := −1 (0)/G is an orbifold and its rational cohomology ring is fairly well understood (see [GK], [JK], [Ka], [Ki1], [TW], [Wi], [Wu]). However, many interesting spaces arise as reduced spaces at singular values of the moment map. Some examples include the moduli space of ﬂat connections, some polygon spaces, many physical systems, and singular projective toric varieties. Since the symplectic quotient at a singular value is a stratiﬁed space [SL], a natural invariant to compute is the intersection cohomology (with middle perversity). Less is known in this case. Kirwan has provided formulas to compute the Betti numbers in the algebraic case (see [Ki1], [Ki2], [Ki3]); Woolf extended this work to the symplectic case [Wo]. Moreover, Jeffrey and Kirwan computed the pairing in the intersection cohomology of particular symplectic quotients [Ki5]. The main result of this paper is two explicit formulas for the intersection cohomology (as a graded vector space with pairing) of the symplectic quotient by a circle in terms of the S 1 equivariant cohomology of the original symplectic manifold and the ﬁxed-point data. More precisely, these formulas depend on the image of the restriction map in equivariant cohomology from the original manifold to the ﬁxed-point set, 1 HS∗1 (M; R) → HS∗1 (M S ; R). Additionally, we show that the intersection cohomology of the reduced space admits a compatible ring structure. Theorem 1. Let the circle S 1 act on a compact connected symplectic manifold M with moment map : M → R so that 0 is in the interior of (M). Let Mred := −1 (0)/S 1 denote the reduced space. There exists a surjective map κ from the equivariant cohomology ring HS∗1 (M; R) to the intersection cohomology IH ∗ (Mred ; R). Moreover, given any equivariant cohomology classes α and β in HS∗1 (M), the pairing of κ(α) and κ(β) in IH ∗ (Mred ) is given by the formula i ∗ (αβ) F . κ(α), κ(β) = Res0 eF + F F ∈Ᏺ

Received 3 December 1998. Revision received 8 September 1999. 2000 Mathematics Subject Classiﬁcation. Primary 53D20; Secondary 55N33. Lerman and Tolman supported by National Science Foundation grant number DMS-980305. Tolman also supported by an Alfred P. Sloan Foundation Fellowship. 79

80

LERMAN AND TOLMAN

Here, eF denotes the equivariant Euler class of the normal bundle of the manifold F , and Ᏺ+ denotes the set of components F of the ﬁxed-point set S 1 such that either (1) (F ) > 0 or (2) (F ) = 0 and index F ≤ (1/2)(dim M − dim F ), where the index of F is the dimension of the negative eigenspace of the Hessian of the moment map at a point of F . The meaning of the right-hand side is as follows: the map iF∗ is the restriction to F ; the equivariant cohomology ring HS∗1 (F ) is naturally isomorphic to the polynomial ring in one variable H ∗ (F )[t]. The equivariant Euler class eF is invertible in the localized ring H ∗ (F )(t); thus, iF∗ (α)/eF is an element of this ring. The integral ∗ F : H (F )(t) → R(t) acts by integrating each coefﬁcient in the series. Finally, Res 0 denotes the operator that returns the coefﬁcient of t −1 . Our convention is that the pairing in intersection cohomology between two classes α ∈ IH p (Mred ) and β ∈ IH q (Mred ) is zero if p + q = dim(Mred ) = dim M − 2. Note that since κ is surjective, this theorem determines the pairing for all pairs of elements in IH ∗ (Mred ). Additionally, by Poincaré duality in intersection cohomology, it determines the kernel of κ. We now provide an alternate version of our main result. Theorem 1 . Let the circle S 1 act on a compact connected symplectic manifold M with moment map : M → R so that 0 is in the interior of (M). Let Mred := −1 (0)/S 1 denote the reduced space. Then there exists a ring structure on the intersection cohomology IH ∗ (Mred ; R) so that we have the following: • The ring structure on IH ∗ (Mred ) is compatible with the pairing, in the sense that there exists an isomorphism from the top-dimensional intersection cohomology to R so that α · β = α, β for all α, β ∈ IH ∗ (Mred ). • As a graded ring, IH ∗ (M; R) is isomorphic to HS∗1 (M; R)/K, where K := α ∈ HS∗1 (M) | α|F = 0 ∀F ∈ Ᏺ+ ⊕ α ∈ HS∗1 (M) | α|F = 0 ∀F ∈ Ᏺ− . Here, Ᏺ+ denotes the set of components F of the ﬁxed-point set M S such that either (1) (F ) > 0 or (2) (F ) = 0 and index F ≤ (1/2)(dim M − dim F ), where the index of F is the dimension of the negative eigenspace of the Hessian of the moment map at a point of F . Conversely, Ᏺ− denotes the set of all other components of the ﬁxed-point set. 1

In principle, these two formulas for the intersection cohomology give almost exactly the same information. We include both, because, in practice, one or the other might be better suited to tackle a particular problem.

INTERSECTION COHOMOLOGY OF S 1 SYMPLECTIC QUOTIENTS

81

We prove these two theorems simultaneously. First, we construct an orbifold M red , which we call the perturbed quotient. The perturbed quotient is a small resolution of the symplectic quotient; thus, as a graded vector space with pairing, IH ∗ (−1 (0)/S 1 ) is isomorphic to H ∗ (M red ). The construction of the perturbed quotient is fairly straightforward. The singularities of the reduced space Mred := −1 (0)/S 1 correspond to components Y of the 1 ﬁxed-point set M S lying on the zero level set −1 (0). In the setting of projective varieties, it is known that the neighborhoods of these singularities have small resolutions [H].1 Although these resolutions are only local, it is possible to piece them together into a global resolution. We construct the perturbed quotient M red as the quotient of the zero ﬁber of a ˜ −1 (0)/S 1 . By con˜ : M → R of the original moment map: M perturbation red = ˜ is Morse-Bott, and its critical points are exactly the ﬁxed points struction the map 1 of the action of S on M. Hence, the standard techniques used to compute the cohomology of symplectic quotients can also be applied to compute the cohomology of the perturbed quotient. In Section 2 we deﬁne the notion of a simple stratiﬁed space and introduce a complex of intersection differential forms which computes the intersection cohomology of a simple stratiﬁed space. In Section 3 we describe the structure of a neighborhood the zero level set for an S 1 moment map and show that the reduced space at zero is a simple stratiﬁed space. In Section 4 we construct the perturbed quotient, and show that it is a small resolution of the reduced space at zero, and compute its cohomology. In the last section we construct explicitly an isomorphism in the derived category between the de Rham complex on the perturbed quotient and the complex of the intersection forms on the reduced space. Acknowledgments. The work in this paper was inspired by the lectures of Frances Kirwan at the Newton institute in the fall of 1994. We thank Reyer Sjamaar for many helpful discussions; In particular, we thank him for the idea that the intersection cohomology of S 1 quotients should be very simple to compute. We thank Sam Evens for a number of useful discussions. We also thank the two referees for suggestions on the exposition and for pointing out numerous typographical errors. 2. Simple stratiﬁed spaces and intersection cohomology. In this section, we introduce the two main concepts that we need in this paper: simple stratiﬁed spaces and intersection cohomology. The notion of a simple stratiﬁed space is not standard; it is, however, convenient for our purposes. We use the notion of an orbifold (also called a V -manifold) introduced by Satake [S]. For more information on Hamiltonian group actions on symplectic orbifolds, see [LT]. 1 It

is claimed in [H] that the resolutions are global, which need not be the case.

82

LERMAN AND TOLMAN

In this paper we deﬁne intersection cohomology via the complex of intersection differential forms due to Goresky and MacPherson (see §1.2 of [GM]), except that we allow the strata to be orbifolds and only consider simple stratiﬁed spaces. Recall that an open cone on a topological space L is the space ◦

c(L) := L × [0, 1)/∼, ◦

where (x, 0) ∼ (x , 0) for all x, x ∈ L. Equivalently c(L) = L × [0, ∞)/ ∼. Deﬁnition 2.1. A simple stratiﬁed space is a Hausdorff topological space X with the following properties. • The space X is a disjoint (set-theoretic) union of even-dimensional orbifolds, called strata. • There exists an open dense oriented stratum X r , called the top stratum. r r • The complement of X in X is a disjoint union of connected orbifolds, X X = Yi , called the singular strata. • For each singular stratum Y there is a neighborhood T of Y in X and a retraction ◦ map π : T → Y which is a C 0 ﬁber bundle with a typical ﬁber c(L) for some orbifold L, which depends on Y . (Thus Y embeds into T as the vertex section.) • There exists a diffeomorphism from the complement T Y to Q × (0, 1), where Q → Y is a C ∞ ﬁber bundle of orbifolds with typical ﬁber L, such that the following diagram commutes: T Y π

Y

/ Q × (0, 1) Y.

In particular, π : T Y → Y is a smooth ﬁber bundle of orbifolds with a typical ﬁber L × (0, 1). Thus, we write a simple stratiﬁed space X as a decomposition X = Xr Yi together with a collection of maps {πi : Ti → Yi }. A simple stratiﬁed space is a stratiﬁed space with oriented even-dimensional orbifold strata of depths 0 and 1 only. If all the strata of a simple stratiﬁed space X are manifolds, then X is a pseudomanifold. Remark 2.2. Note that the composite T Y → Q × (0, 1) → (0, 1), where Q × (0, 1) → (0, 1) is the obvious projection, extends to a continuous map r : T → [0, 1). Deﬁnition 2.3 (Cartan ﬁltration of forms relative to a submersion). Let π : E → B be a smooth submersion of orbifolds. The Cartan ﬁltration Fk $∗ (E) of the complex of forms $∗ (E) on E is given by

Fk $∗ (E) = ω ∈ $∗ (E) | i(ξ0 ) ◦ · · · ◦ i(ξk ) ω(e) = 0 ,

INTERSECTION COHOMOLOGY OF S 1 SYMPLECTIC QUOTIENTS

83

where i(ξ0 ) ◦ · · · ◦ i(ξk )(dω(e)) = 0, for all e ∈ E and for all vectors ξ0 , . . . , ξk ∈ ker dπe . By convention, i(ξ0 ) ◦ · · · ◦ i(ξk )(σ ) = 0 if deg σ ≤ k. Note that F0 $∗ (E) consists of basic forms. Let X = X r Yi be a simple stratiﬁed space. A perversity p¯ : {Yi } → N is a function that assigns a nonnegative integer to each singular stratum Yi . The middle perversity m is deﬁned by m(Yi ) =

1

dim Xr − dim Yi − 1. 2

Deﬁnition 2.4. Let (X = Xr Yi , {πi : Ti → Yi }) be a simple stratiﬁed space, and let p : {Yi } → N be a perversity. The complex of intersection differential forms I $∗p¯ (X) is a sub complex of the complex of differential forms on the top stratum ∗ r $∗ (X r ) deﬁned as follows: ω ∈ I $∗p¯ (X) if and only if ω|Ui ∈ Fp(Y ¯ i ) $ (Ui ∩ X ) on some neighborhood Ui ⊂ Ti (which may depend on ω) of every singular stratum Yi , ∗ r where the ﬁltration Fp(Y ¯ i ) $ (Ui ∩ X ) is deﬁned relative to the submersion πi |Ui : Ui ∩ X r = Ui Yi → Yi . The coboundary map is the exterior differentiation d. The intersection cohomology IH p∗¯ (X) of the simple stratiﬁed space X with perversity p¯ is the cohomology of the complex (I $∗p¯ (X), d). We now deﬁne the pairing on the middle perversity intersection cohomology of a compact simple stratiﬁed space X. Note that if q > dim(Yi ) + p(Yi ), then every q α ∈ I $p (X) vanishes near Yi . In particular, if dim Xr > dim Yi +p(Yi ) for all singular Xr (X) has compact support. Therefore, since the top strata Yi then every α ∈ I $dim p Xr (X) → stratum Xr is oriented, there is a well-deﬁned integration map : I $dim p R, α → Xr α. Similarly, if dim X r − 1 > dim Yi + p(Yi ) for all i, then any β ∈ r I $pdim X −1 (X) is also compactly supported. Thus, in this case, integration descends r to a well-deﬁned map on cohomology : IH pdim X (X) → R; we extend this by zero to a map : IH ∗p (X) → R. Given α and β in I $∗m (X), notice that α ∧ β ∈ I $2m (X), where 2m is twice the middle perversity. This follows from this property of the Cartan ﬁltration: for any α ∈ Fk $∗ (E) and β ∈ Fl $∗ (E), α ∧ β ∈ Fk+l $∗ (E). By deﬁnition 2m(Yi ) = dim Xr − dim Yi − 2 for all singular strata Yi . Thus, there is a well-deﬁned bilinear p q p q pairing IH m(X) × IH m (X) → R which sends [α] ∈ IH m (X) and [β] ∈ IH m (X) to the integral Xr α ∧ β. 3. The structure of the symplectic quotient. In this section, we recall a normal form for neighborhoods of ﬁxed points on symplectic manifolds with Hamiltonian circle actions. Using this, we give a normal form for neighborhoods of the singularities in a symplectic quotient. In particular, we show that the quotient is a simple stratiﬁed space.

84

LERMAN AND TOLMAN

This last statement is a special case of a theorem of Sjamaar and Lerman [SL], who show that every symplectic quotient by a compact Lie group is a stratiﬁed space. Note, however, that in [SL] the stratiﬁcation is by orbit type, whereas here we use the slightly coarser stratiﬁcation by inﬁnitesimal orbit type. Let a circle act on a symplectic manifold M in a Hamiltonian fashion with a moment map : M → R. Recall that the symplectic quotient (also known as the reduced space) is Mred := −1 (0)/S 1 . If 0 is a regular value for , then the quotient 1 is a symplectic orbifold. More generally, is regular on M M S , and

1 r := −1 (0) ∩ M M S /S 1 Mred is an orbifold; this is the top stratum. Moreover, since the restriction of the symplectic 1 r , the latter form on M to −1 (0)∩(M M S ) descends to a symplectic form on Mred is naturally oriented. 1 The moment map is constant on each component of the ﬁxed-point set M S . Thus, every component Y of the ﬁxed-point set that intersects the zero level set −1 (0) is entirely contained in the level set −1 (0) and gives rise to a stratum of Mred diffeomorphic to Y . To see how these strata ﬁt together, we need the following lemma. Lemma 3.1. Let S 1 act on a symplectic manifold (M, ω) in a Hamiltonian fashion with a moment map : M → R. Fix a connected component Y of the ﬁxed-point 1 set M S . The index of both and − at Y are even, say, 2l and 2k, respectively. There exist • a faithful unitary representation ρ : S 1 → (S 1 )k+l ⊂ U (k + l) with positive weights κ1 , . . . , κk ∈ Z and negative weights κk+1 , . . . , κk+l , • a principal G bundle P over Y (where G is a subgroup of U (k) × U (l) that commutes with ρ(S 1 )), such that there is a diffeomorphism σ from a neighborhood U of Y in M to a neighborhood U0 of the zero section in the associated bundle P ×G Ck+l → Y . This diffeomorphism is equivariant with respect to the circle action on P ×G Ck+l induced by ρ. The diffeomorphism pulls back the moment map to the map µ : P ×G Ck+l → R given below; that is, ◦ σ = µ, where 1

κi |zi |2 + (Y ). µ p, (z1 , . . . , zk+l ) = 2 Proof. Let T Y and T M denote the tangent bundles of Y and M, respectively. Consider the symplectic perpendicular bundle E = T Y ω of T Y in T M. Since Y is a symplectic submanifold of M, we have T M|Y = T Y ⊕T Y ω . So E is the normal bundle of Y , and E is a symplectic vector bundle. The group S 1 acts on the bundle E by ﬁber-preserving vector bundle maps. We may choose an S 1 invariant complex structure on E compatible with the symplectic structure. Up to an equivariant homotopy, this complex structure is unique.

INTERSECTION COHOMOLOGY OF S 1 SYMPLECTIC QUOTIENTS

85

The codimension of Y is 2n, where n = k + l. A ﬁber Cn of E splits into the direct sum C⊕· · ·⊕C of one-dimensional representations of S 1 , so that the action of λ ∈ S 1 on the ith summand is given by multiplication by λκi for some weight κi ∈ Z. Under the above identiﬁcation of the ﬁber of E with Cn , the symplectic structure is the imaginary part of the standard Hermitian inner product. Hence, a moment map for the S 1 action on the ﬁber is 1 Cn " (z1 , . . . , zn ) −→ κi |zi |2 . 2 The structure group of the vector bundle E reduces to the subgroup of U (n) consisting of transformations that commute with the action of S 1 described above. Consequently, E = P ×G Cn for some principal G bundle P over Y . The equivariant symplectic embedding theorem (see, for example, Theorem 2.2.1 in [GLS] and the subsequent discussion) implies that we may identify a neighborhood of the submanifold Y in M with a neighborhood of the zero section of E in such a way that a moment map µ : E → R is given by

1 µ p, (z1 , . . . , zn ) = κi |zi |2 + a constant 2 for all [p, (z1 , . . . , zn )] ∈ P ×G Cn . Thus, it is clear that the number of negative weights is l, which is half the index of , and that the number of positive weights is k. So we may assume that κ1 , . . . , κk > 0, whereas κk+1 , . . . , κk+l < 0. We now use Lemma 3.1 to show that the reduced space Mred is a simple stratiﬁed space. Proposition 3.2. Let the circle S 1 act effectively on a compact connected symplectic manifold M in a Hamiltonian fashion with a moment map : M → R. Assume that 0 is in the interior of the image of the moment map. Then the reduced space Mred = −1 (0)/S 1 is a simple stratiﬁed space. In particular every singular stratum Y of Mred is a connected component Y of the 1 ﬁxed-point set M S with (Y ) = 0. Moreover, let the integers l and k, the subgroup G ⊂ U (k)×U (l), and the principle G bundle P over Y be those given in Lemma 3.1. ◦ Then a neighborhood T of Y in Mred is the associated cone bundle P ×G c(S 2k−1 ×S 1 S 2l−1 ), where S 1 acts via the representation ρ : S 1 → U (k + l) given in Lemma 3.1. Proof. It follows from Lemma 3.1 that the zero level set −1 (0) near Y is isomorphic to the zero level set of the model map µ : P ×G Cn → R. The zero level set of µ is the set   k k+l   ◦

q, (z1 , . . . , zk+l ) κi |zi |2 = −κi |zi |2 # P ×G c S 2k−1 × S 2l−1 ,   i=1 i=k+1 and the lemma follows.

86

LERMAN AND TOLMAN

Remark 3.3. By the statement “a neighborhood T of Y in Mred is the associated ◦ cone bundle P ×G c(S 2p−1 ×S 1 S 2q−1 )” we mean that there exists a stratum-preserving homeomorphism from a neighborhood T of Y in Mred to the associated bundle P ×G ◦ c(S 2p−1 ×S 1 S 2q−1 ); this restricts to a diffeomorphism on each stratum and commutes ◦ with the retractions π : T → Y and P ×G c(S 2p−1 ×S 1 S 2q−1 ) → Y . The stratiﬁcation ◦ ◦ 2p−1 of P ×G c(S ×S 1 S 2q−1 ) comes from the stratiﬁcation of the cone c(S 2p−1 ×S 1 S 2q−1 ) into the vertex and the complement of the vertex. 4. The perturbed quotient. In this section we construct an orbifold M red , which we call the perturbed quotient, together with a map f : M → M . These objects red red have two key properties: it is straightforward to explicitly compute the cohomology ring of M red ; and the map f is a small resolution and thus induces a pairing-preserving isomorphism between the cohomology ring of the perturbed quotient and the intersection cohomology (middle perversity) of the reduced space. The key idea. The key idea that makes this work is this observation (due to Hu [H]): Assume that 0 is a singular value of an S 1 moment map : M → R and that 0 lies in the interior of the image (M). Then, for each component Y of the ﬁxed-point 1 set M S with (Y ) = 0, for either all small positive 5 or all small negative 5, there is a neighborhood U of Y in M such that there is a natural isomorphism (4.1)

IH ∗m −1 (0) ∩ U S 1 ∼ = H ∗ −1 (5) ∩ U S 1 .

Whether equation (4.1) holds for negative or positive 5 depends on whether the index of the moment map at Y is at least or at most the index of −. If these signs are all the same for different ﬁxed-point sets Y in the zero level set −1 (0), then there is 5 = 0 so that −1 (5)/S 1 is a global small resolution of Mred . (In fact this is what Hu assumed.) However, it is easy to construct examples where the signs are different. We ˜ : M → R which, like , ﬁx Hu’s mistake by constructing a perturbed moment map is a Morse-Bott function, and has exactly the ﬁxed points of the action of S 1 on M as critical points. It also has the following property: there exists 5 > 0 such that the level ˜ −1 (0) agrees with −1 (5) near ﬁxed-point components with large Morse index and ˜ −1 (0) agrees with −1 (−5) near ﬁxed-point components with small Morse index ˜ −1 (0) → −1 (0), (see Figure 1). Moreover, we construct an S 1 equivariant map −1 1 −1 ˜ which induces a small resolution f : Mred := (0)/S → (0)/S 1 = Mred . To understand why equation (4.1) is true, let us consider a simple special case. Suppose a component Y of the ﬁxed-point set is an isolated point p and that the action of S 1 in a neighborhood of p is free away from p. Then, by the equivariant Darboux theorem, a neighborhood of p is equivariantly symplectomorphic to a neighborhood of 0 in Ck ×Cl with the standard symplectic form and with the action of S 1 given by

λ · (z, w) = λz, λ−1 w

INTERSECTION COHOMOLOGY OF S 1 SYMPLECTIC QUOTIENTS

87

R

M

˜ 0

˜ The fat dots denote Figure 1. The zero ﬁber of the perturbed moment map . the critical manifolds of the moment on −1 (0).

for λ ∈ S 1 , (z, w) ∈ Ck × Cl (the weights of the representation of S 1 are ±1 because we assume that the action is free away from 0). A corresponding moment map is given by (z, w) = |z|2 − |w|2 . Hence, Therefore,

◦

−1 (0) = (z, w) | |z|2 = |w|2 = c S 2k−1 × S 2l−1 . ◦

−1 (0)/S 1 = c S 2k−1 ×S 1 S 2l−1 ,

which is a cone on a sphere bundle over the projective space CPk−1 . Similarly, −1 (5) = (z, w) | |z|2 = |w|2 + 5 . Assuming that 5 > 0, we have an S 1 -equivariant diffeomorphism ψ(ζ, w) = |w|2 + 5 ζ, w . ψ : S 2k−1 × Cl −→ −1 (5),

88 Hence,

LERMAN AND TOLMAN

−1 (5)/S 1 = S 2k−1 ×S 1 Cl ,

which is a rank l complex vector bundle over the projective space CPk−1 . Similarly −1 (−5)/S 1 = Ck ×S 1 S 2l−1 , which is a rank-k complex vector bundle over the projective space CPl−1 . It is straightforward to check that (4.2)



◦

C[x]/x k = H ∗ S 2k−1 ×S 1 Cl ; C , if k ≤ l, ∗ 2k−1 2l−1 ×S 1 S ;C = IH m c S C[x]/x l = H ∗ Ck × 1 S 2l−1 ; C, if k ≥ l, S

which is exactly the content of equation (4.1). Notice also that there are two blow-down maps ◦

S 2k−1 ×S 1 Cl −→ c S 2k−1 ×S 1 S 2l−1 and

◦

Ck ×S 1 S 2l−1 −→ c S 2k−1 ×S 1 S 2l−1 ,

which map the zero sections of the vector bundles to the vertex of the cone. If k ≤ l, then the ﬁrst map is a small resolution in the sense of Deﬁnition 4.3; if k ≥ l, then the second map is a small resolution. Thus, equation (4.2) also follows from the fact that small resolutions induce isomorphisms in intersection cohomology (cf., [GM, p. 121]). Finally note that the two blow-down maps are induced by an S 1 equivariant map from −1 (5) to −1 (0). This map is constructed as follows: the complex linear action of S 1 on Ck × Cl extends to an action of C× , which gives us an action of R× : λ · (z, w) = (λz, λ−1 w) for λ ∈ R× and (z, w) ∈ Ck × Cl . If a point (z, w) lies in −1 (5) and z = 0 = w, then there is a unique λ ∈ (0, ∞) with λ · (z, w) ∈ −1 (0). We then map (z, w) to λ · (z, w). If either z = 0 or w = 0, we map (z, w) to (0, 0). This is the desired map that induces the blow-down maps discussed above. In the next subsection we carry out these constructions in full generality. The construction

Deﬁnition 4.3. Let X = X r Yi be a simple stratiﬁed space. A resolution h : ˜ such that h−1 (X r ) X˜ → X is a continuous surjective map from a smooth orbifold X, −1 r r ˜ is dense in X and h : h (X ) → X is a diffeomorphism. A resolution h : X˜ → X is small if and only if for all r > 0 codim x ∈ X | dim f −1 (x) ≥ r > 2r.

INTERSECTION COHOMOLOGY OF S 1 SYMPLECTIC QUOTIENTS

89

Lemma 4.4. Let the circle S 1 act effectively on a compact connected symplectic manifold M with a moment map : M → R so that 0 is in the interior of the image ˜ : M → R and a map f : ˜ −1 (0)/S 1 → (M). There exists a Morse-Bott function −1 1 (0)/S with the following properties. ˜ are exactly the ﬁxed points of S 1 on M. (1) The critical points of ˜ (2) 0 is a regular value of . 1 ˜ ) > 0 exactly if either (3) For a component Y of the ﬁxed-point set M S , (Y (Y ) > 0, or (Y ) = 0 and index Y ≤ (1/2)(dim M − dim Y ). ˜ −1 (0)/S 1 → −1 (0)/S 1 is a small resolution and, hence, in(4) The map f : duces an isomorphism in cohomology

−1 1

−1 1 ∗ ˜ (0) S . (0) S ∼ IH m = H∗ ˜ −1 (0)/S 1 the perturbed quotient. Deﬁnition 4.5. We call the subquotient M red := ˜ guarantee that the perturbed quotient M The ﬁrst three properties of red is an orbifold, and that it is possible to compute the cohomology ring H ∗ (M ) in a fairly red straightforward manner. This is treated explicitly in the next subsection. Remark 4.6. We see no reason to suspect that the perturbed quotient possesses a symplectic structure. Proof of Lemma 4.4. By Lemma 3.1 for each connected critical manifold Y of in −1 (0), there is a neighborhood U of Y in M that is equivariantly isomorphic to a neighborhood of the zero section in the model P ×G Ck+l . Let V + and V − be the positive and negative weight spaces for the action of S 1 on We may deﬁne G×S 1 invariant norms on V + and V − by |z+ |2 = ki=1 κi |zi |2 for all z+ ∈ V + , and by |z− |2 = ni=l+1 −κi |zi |2 for z− ∈ V − . We may assume that the neighborhoods U for distinct critical manifolds do not intersect. Furthermore, there exists δ > 0 so that 0 is the only critical value of in (−δ, δ) and that each neighborhood U is the image of the set

Ck+l .

P ×G (z+ , z− ) | |z+ |2 + |z− |2 < 3δ . Therefore, we simply give our construction on the vector space V + × V − . As ˜ and f are G-invariant, this construction can be naturally long as our deﬁnition of ˜ = and f is the identity extended to the local model. Additionally, as long as outside the set {(z+ , z− ) | |z+ |2 + |z− |2 < 3δ}, they can be extended globally by ˜ = and f = id outside of these neighborhoods U . taking Choose a smooth function ρ : R → R such that ρ(t) = 1 for all t < δ, ρ(t) = 0 for all t > 2δ, and ρ (t) ≤ 0 for all t. Let C = sup |ρ (t)|. Choose 5 ∈ R so that 5 = 0,

90

LERMAN AND TOLMAN

|5| < C −1 , |5| < δ, and 5 > 0 if and only if dim V + ≥ dim V − . We now deﬁne our ˜ as new function ˜ + , z− ) := (z+ , z− ) + 5ρ (z+ , z− )2 = |z+ |2 − |z− |2 + 5ρ (z+ , z− )2 . (z The norm

+ − 2 (z , z ) = |z+ |2 + |z− |2

is G×S 1 -invariant by construction (see Lemma 3.1, Proposition 3.2, and subsequent ˜ is discussion). Therefore, the function ρ(|(z+ , z− )|2 ), and hence also the function , 1 + − + − 2 + − + ˜ G×S -invariant. Moreover, for (z , z ) with |(z , z )| > 2δ, (z , z ) = |z |2 − − 2 |z | . Therefore, ˜ −1 (0) ∩ (z+ , z− )2 > 2δ = −1 (0) ∩ (z+ , z− )2 > 2δ . ˜ + , z− ) = |z+ |2 − |z− |2 + 5. Thus In contrast, for (z+ , z− ) with |(z+ , z− )|2 < δ, (z (0, 0) is a nondegenerate critical point, and ˜ −1 (0) ∩ (z+ , z− )2 < δ = −1 (−5) ∩ (z+ , z− )2 < δ . Note that |5| < δ guarantees that −1 (−5) ∩ {|(z+ , z− )|2 < δ} = ∅. Moreover, (0, 0) ˜ because is the only critical point of , ˜ = d|z+ |2 − d|z− |2 + 5dρ (z+ , z− )2 d 2

+ 2 2

− 2 d|z | − 1 − 5ρ (z+ , z− ) d|z | = 1 + 5ρ (z+ , z− ) and |1 ± 5ρ (t)| ≥ 1 − |5|(sup |ρ (t)|) > 0, since |5|(sup |ρ (t)|) < 1 by the choice of ˜ is a Morse-Bott function, and that 0 is a regular value of ˜ (since 5. It follows that ˜ (0, 0) = 5 = 0). 1 We now construct a resolution f : M red → Mred . We start by considering a G×S equivariant map ψ : V + × V − → −1 (0) deﬁned by 1/4 + 2 1/4 |z− |2 |z | + − + z , z− if z+ , z− = 0, ψ(z , z ) = + 2 − |2 |z | |z (4.7) ψ(0, z− ) = ψ(z+ , 0) = (0, 0). We let f : M red → Mred be the G-equivariant map induced by the restriction ˜ −1 (0) −→ −1 (0). ψ|˜ −1 (0) : To prove that f is a resolution, it is enough to show that ˜ −1 (0) ψ −1 (0, 0) −→ −1 (0) {(0, 0)} ψ|˜ −1 (0)ψ −1 (0,0) :

INTERSECTION COHOMOLOGY OF S 1 SYMPLECTIC QUOTIENTS

91

is a diffeomorphism. It follows from (4.7) that for (0, 0) = (z+ , z− ) ∈ −1 (0),

ψ −1 (z+ , z− ) = λz+ , λ−1 z− | λ > 0 . (4.8) ˜ −1 (0) ψ −1 (0, 0) → −1 (0) {(0, 0)} is oneConsequently, ψ|˜ −1 (0)ψ −1 (0,0) : to-one and onto. Therefore, it remains to prove that dψ|T (˜ −1 (0)ψ −1 (0,0)) is one to ˜ −1 (0) ψ −1 (0, 0), one, or, equivalently, that for any (z+ , z− ) ∈ ˜ ˜ −1 (0) = ker dψ ∩ ker d . 0 = ker dψ ∩ T(z+ ,z− ) d By (4.8), the kernel of dψ at (z+ , z− ) is spanned by the vector dλ (λz+ , λ−1 z− ). λ=1 ˜ −1 (0) ψ −1 (0, 0) we have Thus, it remains to show that for any (z+ , z− ) ∈

d ˜ λz+ , λ−1 z− = 0. dλ λ=1 Now,

d λz+ 2 − λ−1 z− 2 + 5ρ λz+ 2 + λ−1 z− 2 dλ λ=1 2 2 = 2λ|z+ |2 + 2λ−3 |z− |2 + 5ρ λz+ + λ−1 z−

× 2λ|z+ |2 − 2λ−3 |z− |2

− 2

λ=1

= 2 |z | + |z | + 25ρ |z | + |z− |2 |z+ |2 − |z− |2 . + 2

+ 2

˜ −1 (0), we have |z+ |2 − |z− |2 = −5ρ(|(z+ , z− )|). Since ρ (t) ≤ 0 For (z+ , z− ) ∈ 2 for all t, −5 ρ(t)ρ (t) ≥ 0 for all t. Moreover, (z+ , z− ) = (0, 0). Hence,

d ˜ λz+ , λ−1 z− = 2 |z+ |2 + |z− |2 − 25ρ |z+ |2 + |z− |2 5ρ |z+ |2 + |z− |2 dλ λ=1

≥ 2 |z+ |2 + |z− |2 > 0. A dimension count shows that the map f constructed above is a small resolution. This ﬁnishes the proof of Lemma 4.4. The following lemma is needed in Section 5. Lemma 4.9. Let a circle S 1 act on a symplectic manifold M with a moment map ˜ : M → R be the : M → R so that 0 is in the interior of the image (M). Let 1 −1 ˜ perturbed moment map, and let f : Mred = (0)/S → Mred = −1 (0)/S 1 be the resolution constructed in Lemma 4.4 above. Then, for each singular stratum Y of Mred , there exist the following:

92

LERMAN AND TOLMAN

• an even-dimensional orbifold vector bundle E → N over a compact orbifold N with a sphere bundle L → N such that dim N ≤

1 dim E − 1; 2

• a principal G bundle P → Y ; • an action of G on E by vector bundle maps; • an isomorphism from a neighborhood of the vertex section of the cone; bundle ◦ P ×G c(L) → Y to a neighborhood T of Y in Mred ; • an isomorphism from a neighborhood of the zero section of the vector bundle P ×G E → P ×G N to the neighborhood f −1 (T ) of f −1 (Y ) in M red such that the diagram /M P ×G E o f −1 (T ) red h

◦ P ×G c(L) o

f

T

f

/ Mred ◦

commutes. Here, the map h is induced by the natural blow-down map E → c(L) taking the zero section to the vertex. Proof. We keep the notation of Lemma 4.4. A singular stratum Y of Mred is 1 a component of the ﬁxed-point set M S . A neighborhood of Y in M is equivariantly diffeomorphic to a neighborhood of the zero section in the associated bundle P ×G (V + × V − ), and the moment map : P ×G (V + × V − ) → R is given by ([p, z+ , z− ]) = |z+ |2 − |z− |2 . It is no loss of generality to assume that dim V + ≥ ˜ −1 (0) near Y is S 1 -equivariantly diffeomordim V − . Then, by construction, the set −1 phic to an open subset of (−5) for some small 5 > 0. Now, −1 (0) = [p, z+ , z− ] | |z+ |2 = |z− |2 = P ×G (z+ , z− ) ∈ V + × V − | |z+ |2 = |z− |2 and similarly

−1 (−5) = P ×G (z+ , z− ) ∈ V + × V − | |z+ |2 + 5 = |z− |2 .

The set {(z+ , z− ) ∈ V + × V − | |z+ |2 + 5 = |z− |2 }/S 1 is the desired orbifold vector bundle E, and the set {(z+ , z− ) ∈ V + × V − | |z+ |2 = |z− |2 }/S 1 is the cone bundle on the sphere bundle of E. Indeed, let S − denote the ellipsoid {z− ∈ V − | |z− |2 = 1}. The set {(z+ , z− ) ∈ + V × V − | |z+ |2 + 5 = |z− |2 } is diffeomorphic to V + × S − ; the diffeomorphism is given by V + ×S − " (z+ , ζ ) → (z+ , 5 + |z+ |2 ζ ). Thus E = S − ×S 1 V + , an orbifold vector bundle over the weighted projective space S − /S 1 . One the other hand, by a ◦ similar argument, {(z+ , z− ) ∈ V + × V − | |z+ |2 = |z− |2 }/S 1 = c(S + ×S 1 S − ) where S + is the ellipsoid {z+ ∈ V + | |z+ |2 = 1}.

INTERSECTION COHOMOLOGY OF S 1 SYMPLECTIC QUOTIENTS

93

Cohomology of the perturbed quotient. We can now compute the cohomology of the perturbed quotient by adapting techniques used to compute the cohomology ring of a symplectic quotient at a regular value. We begin by reviewing those techniques. Let a circle S 1 act on a compact connected symplectic manifold M with a moment map . Assume that 0 is a regular value. There is a natural restriction from HS∗1 (M; R) (the equivariant cohomology of M) to HS∗1 (−1 (0); R) (the equivariant cohomology of the preimage of 0). Since 0 is a regular value, the stabilizer of every point in −1 (0) is discrete. Therefore, there is a natural isomorphism from HS∗1 (−1 (0), R) to H ∗ (Mred ), the ordinary cohomology of the symplectic quotient Mred := −1 (0)/S 1 . The composition of these two maps gives a natural map, κ : HS∗1 (M) → H ∗ (Mred ), called the Kirwan map. Theorem 4.10 (Kirwan, [Ki1]). Let a circle S 1 act on a compact connected symplectic manifold M with a moment map so that 0 is a regular value. The Kirwan map κ : HS∗1 (M; R) → H ∗ (Mred ; R) is surjective. Thus, assuming we know the ring structure on HS∗1 (M), the ring structure on red ) can be computed from the kernel of κ. By Poincaré duality, in order to compute the kernel it is enough to compute the integral of κ(α) over the reduced spaces for every equivariant cohomology class α on M. We take one formula for this integral from Kalkman [Ka]; slightly different but morally equivalent formulas were proved by Wu [Wu], and a more general version by Jeffrey and Kirwan [JK]. See also [GK]. All of these results were inspired by a paper of Witten [Wi]. H ∗ (M

Theorem 4.11. Let a circle S 1 act on a compact connected symplectic manifold M with a moment map so that 0 is a regular value. Let Ᏺ+ denote the set of 1 components F of the ﬁxed-point set M S such that φ(F ) > 0. Given an equivariant cohomology class α ∈ HS∗1 (M), the integral of κ(α) over Mred is given by the formula Mred

κ(α) = Res 0

i ∗ (α) F , eF F +

F ∈Ᏺ

where eF denotes the equivariant Euler class of the normal bundle of F . The right-hand side of this formula requires some explanation. The map iF∗ is simply the restriction to F . The equivariant cohomology ring HS∗1 (F ) is naturally isomorphic to H ∗ (F )[t]. The equivariant Euler class eF is invertible in the local ized ring H ∗ (F )(t); thus, iF∗ (α)/eF is an element of this ring. The integral F : H ∗ (F )(t) → R(t) acts by integrating each coefﬁcient in the series. Finally, Res0 denotes the operator that returns the coefﬁcient of t −1 . An alternate method of computing the kernel is given by a theorem of Tolman and Weitsman.

94

LERMAN AND TOLMAN

Theorem 4.12 (Tolman and Weitsman [TW]). Let the circle S 1 act on a compact connected symplectic manifold M with a moment map : M → R so that 0 is a regular value. Let Ᏺ+ denote the set of components F of the ﬁxed-point set such that (F ) > 0; let − Ᏺ denote the set of components F of the ﬁxed-point set such that (F ) < 0. Deﬁne K± := α ∈ HS∗1 (M) | α|F = 0 ∀F ∈ Ᏺ± . The kernel of the Kirwan map is K+ ⊕ K− . In our case, closely analogous propositions are true. Proposition 4.13. Let the circle S 1 act on a compact connected symplectic manifold M with a moment map : M → R. Assume that 0 is in the interior of (M). Let M red denote the perturbed quotient. Then, there is a surjective ring homomorphism κ : HS∗1 (M) → H ∗ (M red ). Moreover, we have the following: • The kernel of κ is K+ ⊕ K− , where K± := {α ∈ HS∗1 (M) | α|G = 0 ∀G ∈ Ᏻ± }. • Given an equivariant cohomology class α ∈ HS∗1 (M), the integral of κ(α) over Mred is given by Kalkman’s formula i ∗ (α) G κ(α) = Res 0 , eG M red + G G∈Ᏻ

where eG denotes the equivariant Euler class of the normal bundle of G and iG : G → M denotes the inclusion. Here, Ᏻ+ denotes the set of components G of the 1 ﬁxed-point set M S such that (1) (G) > 0 or (2) (G) = 0 and 2 index G + dim G ≤ dim M. Conversely, Ᏻ− denotes the collection of all other components of the ﬁxed-point set. Proof. The reason that this proposition is true is that the perturbed quotient M red is deﬁned in a way very similar to the ordinary reduced space. Thus, for example, Kalkman’s formula follows from the fact that there exists a smooth invariant function ˜ : M → R so that 0 is a regular value and M ˜ −1 (0)/S 1 . red is deﬁned to be −1 ˜ Kalkman’s proof relies only on the fact that ([0, ∞)) is a manifold with boundary. Thus, one only needs to note that Ᏻ+ does indeed correspond to the components G ˜ of the ﬁxed-point set such that (G) > 0. ∗ To see that κ : HS 1 (M) → H ∗ (M red ) is surjective, and the Tolman and Weitsman ˜ is a Morse-Bott formula for the kernel of κ holds, we must also use the fact that ˜ function and that the critical points of are exactly the ﬁxed points of the action. This is sufﬁcient to prove both theorems, as was pointed out in [TW]. Combining Proposition 4.13 with Lemma 4.4 and the fact that small resolutions induce isomorphisms in cohomology (cf. [GM, p. 121]), we now obtain the main

INTERSECTION COHOMOLOGY OF S 1 SYMPLECTIC QUOTIENTS

95

results of the paper: Theorem 1 and Theorem 1 . 5. The isomorphism. The goal of this section is to construct explicitly an isomorphism between the intersection cohomology of a singular symplectic quotient and the ordinary cohomology of the perturbed quotient. This section is not, in the strictest sense, necessary. However, we include the proof below for two reasons. First, it is conceptually simple and fairly elementary. Second, the techniques used in the proof are useful in computing the intersection cohomology of singular symplectic quotients of torus actions. Theorem 5.1. Let the circle S 1 act on a compact connected symplectic manifold M with moment map : M → R so that 0 is in the interior of (M). Let Mred := −1 (0)/S 1 denote the reduced space and let M red denote the perturbed reduced space. There is a natural pairing-preserving isomorphism between the intersection cohomology of the symplectic quotient Mred and the cohomology of the perturbed quotient M red . ∗ More precisely, there exists an isomorphism ψ : H ∗ (M red ) → IH (Mred ) of graded p q vector spaces such for any α ∈ H (Mred ) and β ∈ H (Mred ) with p + q = dim M red we have α ∪β = ψ(α), ψ(β) . M red

Mred

Instead of trying to construct the isomorphism between the intersection cohomology of the reduced space and the (ordinary) cohomology of the perturbed quotient directly, ∗ we introduce a new complex A∗m (M red ) = Am (f : M red → Mred ) and show that the ∗ ∗ cohomology of Am (M ) is naturally isomorphic to both H ∗ (M red red ) and IH m (Mred ). Deﬁnition 5.2. Let f : X˜ → X be a resolution of a simple stratiﬁed space. Let X r be the top stratum of X, let X˜ r be its preimage f −1 (X r ), and let ι : X˜ r ?→ X˜ denote the inclusion. By construction, there are maps of complexes f ∗ : I $∗m (X) → $∗ (X˜ r ) ˜ → $∗ (X˜ r ). We deﬁne the complex of resolution forms to be and ι∗ : $∗ (X) (5.3)

˜ = A∗ f : X˜ −→ X := f ∗ I $∗ (X) ∩ ι∗ $∗ (X) ˜ . A∗m (X) m m

Note that f ∗ and ι∗ are both injective. Therefore, we may think of a resolution form ˜ This gives as an intersection form on Xr that extends to a globally deﬁned form on X. ˜ → I $∗ (X) and A∗ (X) ˜ → $∗ (X), ˜ which us the inclusions of complexes A∗m (X) m m ∗ ∗ ∗ ∗ ∗ ˜ ˜ → induce maps in cohomology j : H (Am (X)) → IH m (X) and i : H (Am (X)) ∗ ∗ ˜ Note that the graded vector space H (Am (X)) ˜ has a pairing deﬁned by H (X). taking the exterior product of the representatives of the classes and then integrating ˜ Clearly, the maps i and j are pairing-preserving. Thus, to prove the product over X. Theorem 5.1 it is enough to show that the maps i and j are isomorphisms.

96

LERMAN AND TOLMAN

Local issues. We start with a simple calculation. Lemma 5.4. Let E → N be an even-dimensional orbifold vector bundle over an orbifold N , and let L denote the sphere bundle of E. Then the obvious blow-down ◦ map f : E → c(L) is a resolution. If dim N ≤ (1/2)E − 1, then the maps A∗m (E) ?→ $∗ (E) and

◦ A∗m (E) ?→ I $∗m c(L)

induce isomorphisms in cohomology. Proof of Lemma 5.4. Note ﬁrst that the middle perversity of the vertex ∗ of the ◦ ◦ cone c(L) is m(∗) = (1/2) dim E − 1, since E is even-dimensional. Recall that c(L) is a stratiﬁed space with two strata: the vertex ∗ and the complement L × (0, ∞) # E N. It follows from the deﬁnitions that  q  for q < m(∗), $ (E N), 

q ◦ I $m c(L) = α ∈ $q (E N) | dα = 0 near ∗ , for q = m(∗),    for q > m(∗). α ∈ $q (E N) | α = 0 near ∗ , Consequently,   for q < m(∗), $q (E),   q Am (E) = α ∈ $q (E) | dα = 0 near the zero section , for q = m(∗),    for q > m(∗). α ∈ $q (E) | α = 0 near the zero section , Its not hard to check that if α is a closed k form on the cylinder L × (0, ∞) that vanishes on L × (0, a) for some a, then α = dβ for some k − 1 form β which also vanishes on L × (0, a). It follows that for q ≤ m(∗) the map H q (A∗m (E)) → H q (E) is an isomorphism and that H q (A∗m (E)) = 0 for q > m(∗). Since H ∗ (E) = H ∗ (N ) and since m(∗) ≥ dim N by assumption, the map H q (A∗m (E)) → H q (E) is an isomorphism for all q. Similarly, " H q (E N), for q ≤ m(∗), q ◦ IH m c(L) = 0, for q > m(∗). Consider the Gysin sequence · · · −→ H q−λ (E) −→ H q (E) −→ H q (E N) −→ H q−λ+1 (E) −→ · · · , where λ = dim E − dim N . Since for q − λ + 1 ≤ −1 (i.e., for q ≤ λ − 2) we have H q−λ+1 (E) = 0 = H q−λ (E), the pull-back map H q (E) → H q (E N) is

INTERSECTION COHOMOLOGY OF S 1 SYMPLECTIC QUOTIENTS

97

an isomorphism. In particular, the pull-back is an isomorphism for q ≤ m(∗) = (1/2) dim E − 1 = dim E − ((1/2) dim E − 1) − 2 ≤ dim E − dim N − 2 = λ − 2. Proposition 5.5. Let the circle S 1 act on a compact connected symplectic manifold M with moment map : M → R. Assume that 0 is in the interior of the image (M). Let Mred := −1 (0)/S 1 denote the reduced space, and let f : M red → Mred denote its resolution by the perturbed quotient. There exists a cover ᐁ of Mred such that the natural inclusions A∗m (f −1 (Uα1 ) ∩ ∗ (f −1 (U ) ∩ · · · ∩ f −1 (U )) → · · · ∩ f −1 (Uαk )) → I $∗m (Uα1 ∩ · · · ∩ Uαk ) and Am α1 αk $∗ (f −1 (Uα1 ) ∩ · · · ∩ f −1 (Uαk )) induce isomorphisms in cohomology for all k-tuples {Uα1 , . . . , Uαk } of elements of ᐁ. r Proof. We have seen in Proposition 3.2 that Mred = Mred Yi , where Yi are compact manifolds. By Lemma 4.9 for each singular stratum Yi there exists a tubular ◦ neighborhood Ti of Yi in Mred with the following properties: Ti = Pi ×Gi c(Li ), f −1 (Ti ) = Pi ×Gi Ei , and f : f −1 (Ti ) → Ti is induced by the obvious blow-down ◦ map Ei → c(Li ). Here, as in Lemma 4.9, Pi → Yi is a principal Gi bundle, Ei is an orbifold vector bundle with an action of Gi , and Li is the sphere bundle of ◦ Ei . Note that Ti = Pi ×Gi c(Li ) minus the vertex section Yi is (Pi ×Gi Li ) × (0, 1), and the projection ri : (Pi ×Gi Li ) × (0, 1) → (0, 1) extends to a continuous map ◦ Pi ×Gi c(Li ) → [0, 1), which we also denote by ri (cf. Remark 2.2). Let U0 be the open set Mred ∪ri−1 ([0, 1/2]). Since each singular stratum Yi is a compact manifold, it possesses a ﬁnite good cover {Vαi }. Moreover, we may assume ◦ ◦ that πi−1 (Vαi ) # c(L) × Vαi , where πi : Ti = Pi ×Gi c(Li ) → Yi is the projection. We take Uαi := πi−1 (Vαi ) ⊂ T . This gives us a cover ᐁ = {U0 } ∪ {Uαi } of Mred . Now reindex the elements of ᐁ so that ᐁ = {Uα }K α=0 for some integer K. Note that by construction for a k-tuple {Uα1 , . . . , Uαk } of elements of ᐁ we have either that Uα1 ∩ · · · ∩ Uαk does not intersect any singular stratum Y (in which case f −1 (Uα1 )∩· · ·∩f −1 (Uαk ) and Uα1 ∩· · ·∩Uαk are diffeomorphic) or that it intersects ◦ a unique stratum Y . In the latter case Uα1 ∩ · · · ∩ Uαk # D × c(L), f −1 (Uα1 ) ∩ · · · ∩ f −1 (Uαk ) # D × E, and f : f −1 (Uα1 ) ∩ · · · ∩ f −1 (Uαk ) → Uα1 ∩ · · · ∩ Uαk is ◦ equivalent to the map h × id : E × D → c(L) × D, where D is a disk in Y and ◦ h : E → c(L) is the resolution. Since disks are contractable, the result now follows easily from the previous lemma. Proposition 5.6. Let the circle S 1 act on a compact connected symplectic manifold M with moment map : M → R. Assume that 0 is in the interior of the image (M). Let Mred := −1 (0)/S 1 denote the reduced space, and let f : M red → Mred denote its resolution by the perturbed quotient. ∗ ∗ ) → $∗ (M The inclusions A∗m (M red ) → I $m (Mred ) and Am (M red red ) induce isomorphisms in cohomology.

98

LERMAN AND TOLMAN

Proof. Let ᐁ = {Uα } be the cover of Mred constructed in the proof of Proposition 5.5. One can construct a continuous partition of unity ρα subordinate to the cover ᐁ with the following properties. r . • The functions ρα restrict to smooth functions on Mred • The functions ρα are constant along the ﬁbers of π : T → Y on some neighborhood of every singular strata Y . These properties ensure the following: −1 (U )} of M • {f ∗ ρα } is a partition of unity on M α red subordinate to the cover {f red . ∗ ∗ • For any intersection form γ ∈ I $m (Mred ) or resolution form δ ∈ Am , the products ρα γ and (f ∗ ρα )δ are in I $∗m (Mred ) and A∗m , respectively. The proof is now a standard spectral sequence argument. This completes the proof of Theorem 5.1. By combining Theorem 5.1 with Proposition 4.13 we obtain the main results of the paper, Theorems 1 and 1 . References [BBD]

[BT] [B] [GM] [GK] [GLS] [H] [JK] [Ka] [Ki1] [Ki2] [Ki3] [Ki4] [Ki5] [LT] [S]

A. Beilinson, J. Bernstein, and P. Deligne, “Faisceaux pervers” in Analyse et topologie sur les espaces singuliers, I (Luminy, 1981), Astérisque 100, Soc. Math. France, Montrouge, 1982, 5–171. R. Bott and L. Tu, Differential Forms in Algebraic Topology, Grad. Texts in Math. 82, Springer, Berlin, 1982. J.-L. Brylinsky, “Equivariant intersection cohomology” in Kazhdan-Luszting Theory and Related Topics (Chicago, IL, 1989), Contemp. Math. 139 (1992), 5–32. M. Goresky and R. MacPherson, Intersection homology. II, Invent. Math. 72 (1983), 77–129. V. Guillemin and J. Kalkman, The Jeffrey-Kirwan localization theorem and residue operations in equivariant cohomology, J. Reine Angew. Math. 470 (1996), 123–142. V. Guillemin, E. Lerman, and S. Sternberg, Symplectic Fibrations and Multiplicity Diagrams, Cambridge Univ. Press, Cambridge, 1996. Y. Hu, The geometry and topology of quotient varieties of torus actions, Duke Math. J. 68 (1992), 151–184. L. Jeffrey and F. Kirwan, Localization for nonabelian group actions, Topology 34 (1995), 291–327. J. Kalkman, Cohomology rings of sympletic quotients, J. Reine Angew. Math. 458 (1995), 37–52. F. Kirwan, Cohomology of Quotients in Symplectic and Algebraic Geometry, Math. Notes 31, Princeton Univ. Press, Princeton, 1984. , Rational intersection cohomology of quotient varieties. I, Invent. Math. 86 (1986), 471–505. , Rational intersection cohomology of quotient varieties. II, Invent. Math. 90 (1987), 153–167. , An Introduction to Intersection Homology Theory, Pitman Res. Notes Math. Series 187, Longman, Harlaw, 1988. , Newton Institute lectures, fall 1994. E. Lerman and S. Tolman, Hamiltonian torus actions on symplectic orbifolds and toric varieties, Trans. Amer. Math. Soc. 349 (1997), 4201–4230. I. Satake, On a generalization of the notion of manifold, Proc. Nat. Acad. Sci. U.S.A. 42

INTERSECTION COHOMOLOGY OF S 1 SYMPLECTIC QUOTIENTS

[SL] [TW] [Wi] [Wo] [Wu]

99

(1956), 359–363. R. Sjamaar and E. Lerman, Stratiﬁed symplectic spaces and reduction, Ann. of Math. (2) 134 (1991), 375–422. S. Tolman and J. Weitsman, The cohomology rings of abelian symplectic quotients, preprint, http://xxx.lanl.gov/abs/math.DG/9807173. E. Witten, Two-dimensional gauge theories revisited, J. Geom. Phys. 9 (1992), 303–368. J. Woolf, The intersection Betti numbers of singular symplectic quotients, preprint, 1998. S. Wu, An integration formula for the square of moment maps of circle actions, Lett. Math. Phys. 29 (1993), 311–328.

Lerman: Department of Mathematics, University of Illinois, Urbana, Illinois 61801, USA; [email protected] Tolman: Department of Mathematics, University of Illinois, Urbana, Illinois 61801, USA; [email protected]

Vol. 103, No. 1

DUKE MATHEMATICAL JOURNAL

© 2000

PERTURBATION OF SCATTERING POLES FOR HYPERBOLIC SURFACES AND CENTRAL VALUES OF L-SERIES YIANNIS N. PETRIDIS 1. Introduction. Let \ H be a noncompact hyperbolic surface of ﬁnite area. In the analysis of the Laplace-Beltrami operator on it, apart from the L2 spectrum, a crucial role is played by the scattering poles. They appear in the analog of Weyl’s law, in the Selberg zeta function, and in the determinant of the Laplace operator; see [18]. More important is the fact that imbedded eigenvalues become scattering poles under perturbation (see [25]). In this work we study variation formulas arising from perturbations of scattering poles. A scattering pole is a pole of the analytic continuation of the determinant of the scattering matrix (s) to the left half-plane s < 1/2. These poles show among the poles of the analytic continuation of the resolvent on the same half-plane. But, generally, they are not exactly the same. There can be only ﬁnitely many exceptions for 0 ≤ s < 1/2, where the resolvent has a pole at s0 , corresponding to a cuspidal eigenvalue s0 (1−s0 ) ∈ [0, 1/4), and where det (s) has a zero at s0 . We assume that this does not happen, or, alternatively, we look at scattering poles that do not lie on the interval [0, 1/2). Let E(z, s) = (E1 (z, s), . . . , En (z, s))T be the vector of Eisenstein series indexed by the cusps. Let m be the multiplicity of the pole of det (s) at s0 . We set s() to be the weighted mean of scattering poles; that is, if s1 (0) = s2 (0) = · · · = sm (0) = s0 and if the scattering poles split as s1 (), s2 (), . . . , sm (), when the perturbation is switched on, then m 1 s() = si (). m i=1

The ﬁrst theorem concerns the ﬁrst variation of s() at = 0: s˙ =

d s(0). d

Throughout this work dµ denotes the invariant hyperbolic measure dx dy/y 2 . Theorem 1.1. Assume all the Eisenstein series have a pole of order at most 1 at s0 . For a compactly supported perturbation of the metric, the ﬁrst variation of the Received 16 October 1998. Revision received 18 August 1999. 2000 Mathematics Subject Classiﬁcation. Primary 11F72; Secondary 58J50, 35P25. Author partially supported by National Science Foundation grant number DMS-9600111, Centre interuniversitaire en calcul mathématique algébrique, and the McGill University Department of Mathematics and Statistics. 101

102

YIANNIS N. PETRIDIS

weighted mean of the scattering pole at s0 of multiplicity m is 1 ˙ s˙ = (1.1) E(z, 1 − s0 )T Ress=s0 (s)E(z, 1 − s0 ) dµ. m(2s0 − 1) \H Remark 1.2. Let the space of residues of Eisenstein series at s0 have dimension m ≤ n. According to Lemma 3.1 the order of the pole of det (s) at s0 is m. We can ﬁnd an invertible matrix M such that A(z, s0 ) = M lim (s − s0 )E(z, s) s→s0

has the ﬁrst m elements Ai (z, s0 ) linearly independent and Ai (z, s0 ) = 0 for i > m. Extend M to a holomorphic matrix M(s) close to s0 in any way desired. Let ˜ s) = M(s)E(z, s) = (e1 (z, s), . . . , en (z, s))T be a new basis of the Eisenstein E(z, ˜ s). We have ( + 1/4)Ai (z, s0 ) = λ2 Ai (z, s0 ) with series. Set A(z, s) = (s − s0 )E(z, 0 λ0 = s0 − 1/2. Set V = V (s0 ) = (vj i ) = M(s0 )−1 . It is easy to see that all entries of (s) have a pole of order at most 1 at s0 by looking at the zero Fourier coefﬁcient of Ei (z, s) at the j -cusp. We can write (1.1) as (1.2) s˙ =

1 m(2s0 − 1)

\H

˜ 1 − s0 ) dµ, ˙ E(z, E(z, 1 − s0 )T Ress=s0 (s)M(1 − s0 )−1

if one continues the new basis analytically to the point 1−s0 . In the important example of congruence subgroups, M(s) is explicitly given as the change of basis matrix from the Eisenstein series indexed by the cusps to the Eisenstein series with characters. Corollary 1. For a compactly supported perturbation of the metric, the ﬁrst variation of a scattering pole at s0 with multiplicity 1 for a surface with one cusp is 1 ˙ s˙ = Ress=s0 (s) E(z, 1 − s0 ) E(z, 1 − s0 ) dµ. 2s0 − 1 \H Corollary 2. For a compactly supported conformal perturbation of the metric g = ef g0 , where f ∈ Cc∞ ( \H), the ﬁrst variation of the scattering pole at s0 with multiplicity 1 for a surface with one cusp is s0 (1 − s0 ) s˙ = Ress=s0 (s) f (z)E(z, 1 − s0 )2 dµ. (1.3) 2s0 − 1 \H Since we assume that the scattering pole s0 has multiplicity m, we are also interested in breaking the degeneracy under perturbation. We have the following theorem. Theorem 1.3. Assume that all the Eisenstein series have a pole of order at most 1 at s0 , which is a scattering pole of multiplicity m. Consider the matrix n i ˙ i (z, s0 ) aj = A Ek (z, 1 − s0 )vkj dµ, i, j = 1, 2, . . . , m. \H

k=1

PERTURBATION OF SCATTERING POLES

103

If its eigenvalues are distinct and nonzero, then the degeneracy of the scattering pole at s0 is removed by the compactly supported perturbation and the branches of scattering poles are distinct at s0 . Remark 1.4. The restriction that all the Eisenstein series have a pole of order at most 1 at s0 may seem too strict. However, in all examples where the scattering matrix has been computed, that is, certain congruence subgroups of SL(2, Z), it is easily veriﬁed, assuming that the zeros of the L-series involved are simple. The simplest hyperbolic surface with a cusp is SL(2, Z) \ H. The scattering matrix for SL(2, Z) \ H is (1.4)

φ(s) =

π 2s−1 (1 − s)ζ (2 − 2s) . (s)ζ (2s)

The function φ(s) plays a special role for other congruence subgroups, because it appears as a factor of det (s). Condition A. The nontrivial zeros ρ of the Riemann zeta function are simple, and they lie on s = 1/2. The second part of the condition is the Riemann hypothesis (RH). It implies that the poles of φ(s) are at s0 = ρ/2 = 1/4 + iγ /2, where γ ∈ R. The simplicity of the zeros is also a natural condition to assume. Numerical evidence by Odlyzko [20] suggests that this is indeed true. Montgomery [17] proved that the pair correlation conjecture implies that almost all zeros are simple, and RH implies that at least 2/3 of the zeros are simple. Unconditionally Conrey [3] proved that at least 2/5 of the zeros are simple, while under RH and the generalized Lindelöf hypothesis at least 19/27 of the zeros are simple (see [4]). The following weak form of the Mertens hypothesis implies that all the zeros of the zeta function on the critical line are simple (see [31, Th. 14.29, p. 376]). Let µ(n) be the Möbius function, and let M(x) = n≤x µ(n). Then X M(x) 2 dx = O(log X). x 1 We apply the perturbation results of Theorems 1.1 and 1.3 to the character varieties of 0 (q), where q is a prime number. The perturbations are generated by holomorphic cusp forms of weight 2 for 0 (q). Let g(z) be such a form that has real coefﬁcients, is a Hecke eigenform, and has eigenvalue q for the Fricke involution Wq (q = ±1). We have the following theorem. Theorem 1.5. For 0 (q), q prime, the ﬁrst variation of the weighted mean of scattering poles at any s0 is zero. (a) If the central value L(g, 1) of the L-series of g vanishes (which happens automatically if q = 1), then ﬁrst-order degenerate perturbation theory does not remove the degeneracy at any s0 .

104

YIANNIS N. PETRIDIS

(b) If L(g, 1) = 0, then the degeneracy is resolved, L(g, 1) is a factor of the ﬁrst variation of the branches of scattering poles at any s0 , and the branches are not constant, that is, the scattering poles move. In fact, L(g, 1) and L(g, 2 − 2s0 ) are factors of all the variation formulas, and the second value is on the line of convergence of L(g, s). Examples of groups that have a newform g(z) with L(g, 1) = 0 are 0 (q) with q = 37, 43, 53, 61, 79, 83, 89. The groups with q = 37 and 89 have two newforms and only one has the central value vanishing. The other groups have only one newform. In all cases the order of vanishing is exactly 1 and q = 1. Examples of groups with all newforms g(z) having L(g, 1) = 0 are 0 (q), with q = 11, 17, 19, 67, 73. In fact they all have only one newform. The special value at 1 of the L-series L(g, s) has been studied extensively in relation to the rank of the group of rational points of the elliptic curve C/.g , where .g is the period lattice of g(z). The order of vanishing of L(g, s) at s = 1 is, according to the Birch-Swinnerton-Dyer conjecture, equal to the rank of the elliptic curve. There has been extensive numerical evidence for the conjecture (see [5]), from where we collected the data above. Theorem 1.5 relates the nonvanishing of L(g, s) at s = 1 to the movement of the scattering poles away from the position described by the zeros of the Riemann zeta function. Remark 1.6. The central value of the L-series of a cusp form of weight 2 or 4 also appears in the ﬁrst variation of resonances at 1/4 in [22]. Remark 1.7. In Section 4.2 we explain how Theorems 1.1 and 1.3 can be applied to the groups 0 (N ), N composite, and (N) to produce similar results as in Theorem 1.5. The values of the L-series involved are central values of twists L(g, χ, s), where the conductor of χ divides N. Remark 1.8. One can also study perturbations of the metric in Teichmüller space. In this case the tangent direction is speciﬁed by a holomorphic cusp form g of weight 4. One gets similar results to Theorem 1.5. The central value L(g, 2) shows up as a factor of the variation formulas. We intend to study this phenomenon elsewhere. One would like to know which direction (most of) the scattering poles move. One hopes that most will move off the line s = 1/4 under perturbations in character varieties. Let us assume the character variety has two tangent directions generated by two distinct newforms f (z) and g(z) of 0 (q), which have nonvanishing central values for their L-series: L(f, 1)L(g, 1) = 0. An extra technical assumption is the following condition. Condition B. There exists a prime p such that the corresponding Fourier coefﬁcients b(p) and a(p) satisfy (1.5)

a(p)2 = b(p)2 .

PERTURBATION OF SCATTERING POLES

105

This condition is mild. Two cusp forms with equal coefﬁcients (except possibly ﬁnitely many of them) are identical. In the case when one of the f and g is a quadratic twist of the other, condition (1.5) fails, but the levels of the forms do not agree. Let the curves of characters generated by f and g be χf, and χg, , respectively; see (4.1). We prove that, under Conditions A and B and for at least one of the forms f , g, a positive proposition of the scattering poles move off the line prescribed by the Riemann hypothesis. More precisely, we prove the following theorem. Theorem 1.9. For at least one of the two curves χf, , χg, and for T sufﬁciently large, we can ﬁnd a δ = δ(T ) > 0 such that there exists a positive proportion of the scattering poles with |s0 | ≤ T that are to the left of s = 1/4 and a positive proportion that are to the right of it for characters in the δ-neighborhood of the trivial character and on the curve. Remark 1.10. Selberg [29] has constructed a family χ(α) of characters of 0 (4) with the following property: for a given vertical line, one can ﬁnd α such that there are scattering poles for the character χ(α) to the left of the vertical line, and, in fact, approximately c · T of them have imaginary parts γ satisfying |γ | < T , for T large. In the case of Theorem 1.9 the number of scattering poles with |γ | < T is asymptotic to c · T log T . Extensions of this result can be found in [2]. Remark 1.11. The assumption of RH is technical but clariﬁes the picture on the side of the scattering theory. If RH fails in the strong sense that there is a positive proportion of zeros of ζ (s) off s = 1/2, then the corresponding scattering poles are off s = 1/4 before the deformation and remain off for δ(T ) sufﬁciently small. If zero proportion of the zeros of ζ (s) are off s = 1/2, the proof in Section 5 needs to be modiﬁed. We can work as in [8], where the main theorem on discrete mean values of ζ (s) and its derivatives is proved without assuming RH. Remark 1.12. For hyperbolic surfaces and for a character χ that is singular with respect to κ1 > 0 cusps, one has

√ det (s) =

κ1 π s − 1/2 a(χ)b(χ)1−2s L(s, χ), (s)

where L(s, χ ) is a Dirichlet series with constant term 1. It follows that det (s) does not vanish for s sufﬁciently large, which implies through the functional equation that the scattering poles are contained in a vertical strip σ < s < 1/2. The constant σ depends on the group and character. Müller [18] raised the question of whether the same is true for a general surface with cusps. Froese and Zworski [7] gave a counterexample, which is a rotational symmetric surface. The motivation of this work was to see whether conformal perturbations of hyperbolic surfaces keep the scattering poles in a vertical strip or not. We study the size of s˙ for conformal perturbations of SL(2, Z) \ H in [23].

106

YIANNIS N. PETRIDIS

Remark 1.13. Selberg [29] has shown that if we denote the scattering poles by s0 , then 1 κ1 − s0 = T log T + O(T ), 2 4π 0≤s0 ≤T

0≤s0 ≤T , s0 <1/4

and

1 κ1 − s0 < T log log T + c(a, b, σ )T + O(log T ), 4 2π

(σ − s0 ) < c (a, b, σ )T + O(log T )

0≤s0 ≤T , s0 <σ

for σ < 1/4. This shows that the distance of the scattering poles to the unitary axis s = 1/2 is on the average not greater than 1/4. All the estimates are uniform in the characters for ﬁxed, as long as the number a = a(χ) is bounded away from zero. It is not known whether other surfaces with cusps satisfy these estimates. Acknowledgments. The author would like to thank W. Müller, who suggested the problem and (independently) proved Corollary 2. The author also thanks P. Sarnak, R. Murty, and H. Darmon for various suggestions. 2. Preliminary material. We recall some standard facts about the resolvent and its analytic continuation, which is due to Faddeev [6] (see also [15, Ch. XIV]), and deﬁne the operators and norms of the Lax-Phillips scattering theory as applied to automorphic functions. Let R(s) = (−−s(1−s))−1 be the resolvent of the Laplace operator. Its kernel is constructed as follows. The fundamental point-pair invariant 1 is u(z, z ) = |z − z |2 /(4yy ) for z, z ∈ H. We set ϕ(u, s) = 0 [t (1 − t)]s−1 (t + u)−s dt/(4π ) for σ > 0, u > 0, s = σ + it, and k(z, z ; s) = ϕ(u(z, z ), s). The kernel k(z, z ; s) is the Green function for the problem h+s(1−s)h = f at least for σ > 1 (see [15, p. 275]). For a discrete coﬁnite subgroup of SL(2, R), we set (2.1)

r(z, z ; s) =

1 ϕ u(z, γ z ), s 2 γ ∈

for σ > 1. This is the resolvent kernel. We decompose the fundamental domain F of into n

F = F0 ∪ Fj , j =1

where the Fj ’s are isometric to the standard cusp. Let the cusps be z1 , z2 , . . . , zn . There exists a gj with gj ∞ = zj . One can choose gj ∈ SL(2, R) so that z → gj z maps C = {z; −1/2 ≤ z ≤ 1/2, z ≥ a} one-to-one onto Fj . Each function f on F

PERTURBATION OF SCATTERING POLES

107

has n + 1 components f0 (z) = f (z) for z ∈ F0 and fj (z) = f (gj z) for z ∈ C. One has the decomposition L2 ( \ H) = L2 (F0 ) ⊕

n

L2 (Fj ),

j =1

but it turns out that one has to use weighted spaces to deﬁne the analytic continuation of the resolvent. Faddeev [6] introduced the Banach spaces Bµ , which consist of complex-valued functions f (z) whose components f0 (z) and fj (z), j = 1, . . . , n, are continuous on F0 and C, respectively, with |fj (z)| ≤ cy µ for z ∈ C. The µ-norm is n fj (z) f µ = max f0 (z) + max . z∈F0 z∈C yµ j =1

One can attach a meaning to the analytic continuation of the resolvent kernel on a Riemann surface that is a 2-sheeted covering of the z-plane. We set z = s(1 − s), and then the z-plane cut along the ray [0, ∞) corresponds to the right half-plane s > 1/2 cut along 1/2 ≤ s ≤ 1. For s, 1 − s nonsingular, we have the limiting absorption principle (2.2)

r(z, z ; s) − r(z, z ; 1 − s) =

n

1 Ej (z, s)Ej (z , 1 − s), 2s − 1 j =1

a proof of which is given in [15, p. 344]. After obtaining the analytic continuation of the resolvent kernel, one deﬁnes R(s) in the following manner. Fix µ ≤ 1/2; then R(s) : Bµ → B1−µ is deﬁned for s > µ as the integral operator with kernel r(z, z ; s). A slightly different approach that works for all surfaces with cusps and where the resolvent is considered as an operator-valued function with values in the bounded operators between weighted L2 -spaces was worked out by Müller [19]. It should be remarked that in (2.2) the Eisenstein series are indexed by the cusps and are deﬁned as s Ej (z, s) = gj−1 σ z , σ ∈j \

where j is the stabilizer of the cusp zj . This way the zero Fourier coefﬁcient of 1−s . The scattering matrix is (s) = (φ (s)). Ej (z, s) at the cusp zi is δij y s +φ ij 0ij 1(s)y We set L = + (1/4), A = L 0 , and E the energy form for the wave equation utt = Lu, that is, E (f1 , f2 )T = −(f1 , Lf1 )L2 + (f2 , f2 )L2 . Let HG be the completion of the space of pairs of C ∞ functions with compact support in the norm G (f1 , f2 )T = E (f1 , f2 )T + cf1 22

108

YIANNIS N. PETRIDIS

for c sufﬁciently large. Let P be the E-orthogonal projection to H , which is the complement of the space D+ ⊕D− , in HG , where D± are the spaces of outgoing and incoming data (see [16, p. 121]). The operator P may only change the zero Fourier coefﬁcients of data at each cusp. The operator B is the inﬁnitesimal generator of the semigroup P U (t)P , where U (t) is the standard wave operator. We denote by RF (z) the resolvent of an operator F , that is, RF (z) = (F − z)−1 . We have (2.3)

RB (λ) = P RA (λ)P

for λ sufﬁciently large (see [16, p. 29]). A calculation with matrices gives

λRL (λ2 ) RL (λ2 ) RA (λ) = LRL (λ2 ) λRL (λ2 ) (2.4)

−λR(λ + 1/2) −R(λ + 1/2) = , I − λ2 R(λ + 1/2) −λR(λ + 1/2) λ = s − 1/2. since RL (λ2 ) = −R(s) iand (z,s0 ) . Then Afi = λ0 fi . The eigenvector of the cutoff wave Call fi the vector λA 0 Ai (z,s0 ) operator B is Pfi , since BPfi = P Afi = λ0 Pfi . 3. Proof of Theorems 1.1 and 1.3. The idea of the proof of Theorem 1.1 is to use perturbation theory for the cutoff wave operator B, which is an operator with discrete spectrum (see [25, p. 6]). However, since B is not selfadjoint, we choose to use variational formulas that use traces (see [14, p. 90]), instead of energy inner products, as was done in [25, p. 24]. A similar method was used in [21] and [22] to study the variation of the resonances at 1/4 and Fermi’s golden rule. An outgoing eigenfunction of A with eigenvalue λ0 is a pair f = (f1 , f2 )T that satisﬁes the following: (1) (A − λ0 )k f = 0 for some integer k; (0) (0) (2) f is outgoing; that is, the zero Fourier coefﬁcients f1 , f2 of f1 , f2 satisfy (0) (0) f2 = −y 3/2 ∂y (f1 /y 1/2 ) in the cusps; and (3) Aj f minus its zero Fourier coefﬁcient in the cusps lies in HG for j = 0, 1, . . . , k − 1. Lemma 3.1. All the Eisenstein series have a pole of order at most 1 at s0 if and only if s0 − 1/2 is a semisimple eigenvalue of the cutoff wave operator B. Proof. Assume that λ0 is a semisimple eigenvalue of B. According to [25, Th. 3.1] the semisimplicity of the eigenvalue s0 −1/2 for B is equivalent to the semisimplicity of the outgoing eigenspace of A at λ0 = s0 − 1/2. We show that any pole of order greater than 1 for any Eisenstein series produces an outgoing eigenfunction g = (g1 , g2 )T of A with (A − λ0 )2 g = 0 but (A − λ0 )g = 0, which contradicts the semisimplicity of the eigenspace for A. If the pole of the Eisenstein series

PERTURBATION OF SCATTERING POLES

109

E(z, s) at s0 has order k ≥ 2, set A(z, s) = (s − s0 )k E(z, s). By differentiation of A(z, s) + s(1 − s)A(z, s) = 0 in s, we get LB(z, s0 ) − λ20 B(z, s0 ) = 2λ0 A(z, s0 ), where B(z, s0 ) = dA(z, s0 )/ds. Set g1 = B(z, s0 ) and g2 = A(z, s0 ) + λ0 B(z, s0 ). Then (A − λ0 )g = (A(z, s0 ), λ0 A(z, s0 ))T = 0. However, (A − λ0 )2 g = 0. It is easy to check condition (2) for g. We now prove the converse. Assume the Eisenstein series have poles of order at most 1 and the space of residues has dimension m as in Remark 1.2. We have at least m eigenvectors for A, the fi , i ≤ m: Afi = λ0 fi . If we prove that the order of the pole of det (s) at s0 is at most m, since by [25, Th. 4.1] the order of the pole is the multiplicity of the outgoing eigenspace of A, then the dimension of the eigenspace is exactly m. Therefore the eigenvalue is semisimple. We see that A(z, s0 ) = M(s0 ) Ress=s0 (s)E(z, 1 − s0 ). Since Ai (z, s0 ) = 0 for i > m and Ej (z, 1 − s0 ) are linearly independent, the matrix M(s0 ) Ress=s0 (s) has zero entries on the rows m + 1, m + 2, . . . , n. Set N(s) = M(s)(s). Its entries on the rows m + 1, m + 2, . . . , n should be regular, while the other entries have a pole of order at most 1. By multiplying the ﬁrst m rows by s − s0 , we see that (s − s0 )m det N(s) = (s − s0 )m det M(s) det (s) remains bounded close to s0 . However, det M(s) = 0, so (s − s0 )m det (s) remains bounded close to s0 . The ﬁrst variation of the weighted mean for a scattering pole, which is also the ﬁrst variation of the weighted mean for the eigenvalue λ0 of B, is given by (3.1)

s˙ =

1 ˙ Tr(BQ), m

where Q is the projection to the eigenspace of B generated by the Pfi ’s (see [14, 2.33, p. 90]). If is a contour enclosing only λ0 = s0 − 1/2 among the eigenvalues of B, then, using (2.3) and (2.4) one gets 1 1 P RA (λ) dλP RB (λ) dλ = − Q=− 2π i 2πi (3.2) −λR(λ + 1/2) −R(λ + 1/2) 1 =− dλP . P 2 2π i I − λ R(λ + 1/2) −λR(λ + 1/2) n With the standard inner product on Rn , we have j =1 Ej (z, s)Ej (z , 1 − s) = ˜ s). E(z , 1 − s)T · E(z, s) = E(z , 1 − s)M(s)−1 E(z, By (2.2), since r(z, z , 1 − s) is regular at s0 , the contour integral in (3.2) is an operator with integral kernel

E(z , 1 − s0 )T V A(z, s0 )/(2s0 − 1) E(z , 1 − s0 )T V A(z, s0 )/2 − . (s0 − 1/2)E(z , 1 − s0 )T V A(z, s0 )/2 E(z , 1 − s0 )T V A(z, s0 )/2

110

YIANNIS N. PETRIDIS

If g = (g1 , g2 )T is any pair of data supported in some compact set of the surface and with zero Fourier coefﬁcient of g1 and g2 vanishing above the cut at y = a, then P g = g. Let K be the integral operator with kernel 1 K(z, z ) = E(z , 1 − s0 )T V A(z, s0 ). 2

Then Qg = P

Kg1 + 1/(s0 − 1/2)Kg2

(s0 − 1/2)Kg1 + Kg2

.

Notice that Kg1 and Kg2 are linear combinations of the Aj (z, s0 )’s, j ≤ m, and Qg is a linear combination of the Pfj ’s, j ≤ m, as it should be. Since B˙ = L0˙ 00 , we have 0 ˙ Bfi = ˙ (3.3) LAi (z, s0 )

˙ i (z, s0 ) K LA 1 0 P = Q ˙ . ˙ i (z, s0 ) LAi (z, s0 ) s0 − 1/2 (s0 − 1/2)K LA

and

We have ˙ i (z, s0 ) = K LA where

ai

(3.4) Then (3.5) Finally,

\H

˙ i (z , s0 )K(z, z ) dµ(z ) = LA

1 i a · A(z, s0 ), 2

is the row vector i ˙ i (z , s0 )E(z , 1 − s0 )T V dµ(z ). a = LA

ai · A(z, s0 ) 1 0 Q ˙ P = . LAi (z, s0 ) 2s0 − 1 (s0 − 1/2)ai · A(z, s0 )

1 0 0 ˙ BQ ˙ = ˙ LAi (z, s0 ) s0 ) 2s0 − 1 ai · LA(z,

˙ maps H into the space spanned by ˙ 0 for j ≤ m. The operator BQ LAi (z,s0 ) . The ˙ i (z, s0 ), i ≤ m, may not be linearly independent, but, still, Tr(BQ) ˙ functions LA = m i 1/(2s0 − 1) i=1 ai . This follows from elementary linear algebra: assume that only 0 the ﬁrst k out of the m vectors LA ˙ i (z,s0 ) are linearly independent and extend them to any basis of the whole space. If ˙ j (z, s0 ) = LA

k t=1

˙ t (z, s0 ), bj t LA

j = k + 1, . . . , m,

111

PERTURBATION OF SCATTERING POLES

then k m k 0 0 0 i i ˙ aj ˙ aj bj t ˙ = + . (2s0 − 1)BQ ˙ LAj (z, s0 ) LAt (z, s0 ) LAi (z, s0 ) j =1

j =k+1

˙ The contribution to Tr(BQ) is (aii + ˙ Tr(BQ) =

However,

k

i i=1 aj bj i

t=1

m

1 2s0 − 1

i j =k+1 aj bj i )/(2s0 − 1)

k i=1

 aii +

and



m j =k+1

aji bj i  .

j

= aj for j = k + 1, . . . , m. This is so because (3.5) gives

k 0 0 bj t Q ˙ = (2s0 − 1) (2s0 − 1)Q ˙ LAj (z, s0 ) LAt (z, s0 ) =

k

bj t

t=1

m p=1

t=1

apt P

Ap (z, s0 )

λ0 Ap (z, s0 )

=

m

j ap P

p=1

Ap (z, s0 )

λ0 Ap (z, s0 )

.

Using (3.4), one gets n

(3.6)

1 1 aii = 2s0 − 1 2s0 − 1

j =1 \H

˙ i (z , s0 ) dµ(z ), Ej (z , 1 − s0 )vj i A

˙ Since Ai (z, s0 ) = 0 for i = m + 1, . . . , n, we can take the same formula since L˙ = . as valid for all i = 1, . . . , n. Therefore, ˙ (2s0 − 1) Tr(BQ) =

n i,j =1 \H

=

\H

˙ i (z , s0 ) dµ(z ) Ej (z , 1 − s0 )vj i A

˙ E(z , 1 − s0 )T M(s0 )−1 A(z , s0 ) dµ(z ).

Now we switch back to the original basis of Eisenstein series indexed by the cusps, and we use the functional equation for E(z, s) to get (1.1). Proof of Corollary 1. It is obvious from Theorem 1.1. ˙ = −f 0 , where Proof of Corollary 2. If g = ef g0 , then = e−f 0 and 0 is the Laplacian of the unperturbed metric. Since the Eisenstein series E(z, 1−s0 ) corresponds to the eigenvalue s0 (1 − s0 ), formula (1.3) follows.

112

YIANNIS N. PETRIDIS

Proof of Theorem 1.3. Since the eigenvalue λ0 of B is semisimple, [14, Th. 2.3, p. 93] implies that the eigenvalue branches at λ0 are continuously differentiable at = 0. They are of the form (3.7)

(1)

λ0 + λi + o(),

i = 1, 2, . . . , m,

(1) ˙ If we can solve the eigenvalue where λi are the eigenvalues of B (1) = QBQ. (1) problem for B on the image of Q, that is, B (1) w = λw, then we remove the (1) degeneracy and, if the eigenvalues λi are distinct, then the perturbation produces distinct power series at λ0 . Let

w=

m

xi Pfi

i=1

be any vector in the image of Q. Then m 0 (1) ˙ B w = QBw = xi Q ˙ LAi (z, s0 ) i=1

m m m ai · A(z, s0 ) 1 1 i xi P x aj Pfj . = = i 2s0 − 1 2s0 − 1 λ0 ai · A(z, s0 ) i=1 i=1 j =1 Since the Pfj ’s are linearly independent, the equation B (1) w = λw gives m

1 i xi aj = λxj , 2s0 − 1

j = 1, 2, . . . , m.

i=1

This is the eigenvalue equation for the matrix (aji ), i, j = 1, . . . , m, with eigenvalue λ(2s0 − 1). 4. Character varieties 4.1. General theory of character perturbations. For \ H, the ﬁrst homology group is isomorphic to /[, ]. Its dual group consists of the unitary characters χ of and Acusp = {χ | χ (p) = 1, p ∈ , p parabolic}. The cohomology classes in the ﬁrst de Rham cohomology which can be represented by forms of compact support have a square integrable harmonic representative (which can be taken to be cuspidal, that is, if w = w0 dy + w1 dx, then C w0 = C w1 = 0, where C is a path corresponding to a parabolic). Fix z0 ∈ \ H, and let π : → /[, ] be the natural projection from π1 ( \ H) → H1 ( \ H, R). For any cuspidal harmonic square integrable form w, we set χw (γ ) = exp 2πi (4.1) w , π(γ )

PERTURBATION OF SCATTERING POLES

113

which is a cuspidal character in the connected component of the trivial character in Acusp . The deformation we consider depends on a real parameter , and the corresponding spectral problem concerns L2 -functions satisfying h(γ z) = χw (γ )h(z). Let us denote the corresponding L2 space by L2 ( \H, χw ). We conjugate this space to the ﬁxed space L2 ( \ H) as follows. Set z w h(z) (U h)(z) = exp 2πi z0

so that U : L2 ( \ H) → L2 ( \ H, χw ). We set L() = U−1 U , which now acts on L2 ( \ H). We deﬁne δ(p dx + q dy) = −y 2 (px + qy ), p dx + ¯ and |p dx + q dy|2H = y 2 (|p|2 + |q|2 ). Then it is q dy, f dx + g dy = y 2 (p f¯ + q g), easy to see that L()u = u + 4πidu, w − 4π 2 2 |w|2H u − 2πi(δw)u. If w is harmonic, the last term vanishes. Let f (z) now be a holomorphic cusp form of weight 2, and let w be the real-valued harmonic form (f (z) dz). So f (z) = w1 − iw0 . As usual we deﬁne for a function f (z) of weight k and T ∈ GL(2, R), (f | T )(z) = (det T )k/2 (cz + d)−k f (T z).

Let f | U (z) = n>0 an e2πinz be its expansion at the cusp zi , where U = gi in the notation of Section 2. Let u(z) be any of the Eisenstein series ek (z, 1 − s0 ) with Fourier expansion at the zi cusp of the form (4.2) cn y 1/2 K1/2−s0 (2π|n|y)e2πinx . u | U (z) = Ay s0 + By 1−s0 + n=0

Then

˙ u(z) = 4πidu, w.

The equation (1.2) shows that the variation of the weighted mean of the scattering pole is a linear combination of ˙ Ei (z, 1 − s0 )u(z) dµ \H

and this is R(1 − s0 ), where R(s) =

\H

˙ Ei (z, s)u(z) dµ.

114

YIANNIS N. PETRIDIS

Since 2w = f (z) dz + f (z) dz, we have du, w = y 2 uz f (z) + uz¯ f (z) . We also set f | U (z) − f | U (z) =

bn e−2π|n|y e2πinx

n=0

so that bn = an for n > 0 and bn = −a−n for n < 0. Then for s 0 we have R(s) = 4π i du, w | U (z)Ei (U (z), s) dx dy/y 2 −1 U U \H = 4π i du, w | U (z)y s dx dy/y 2 \H ∞ = 4π i (u | U )z f | U (z) + (u | U )z¯ f | U (z) y s dx dy. ∞ \H

An integration by parts gives ∞ 1 R(s) = 2π (u | U )(z)sy s−1 f | U (z) − f | U (z) dx dy 0 0 ∞ cn b−n = 2π s e−y y s−1/2 K1/2−s0 (y) dy (4.3) (2π|n|)s+1/2 0 n=0

∞

=

s (s + 1 − s0 )(s + s0 ) −cn a¯ n + c−n an , s s−1 (s + 1) 4π ns+1/2 n=1

using [11, 6.621.3, p. 733]. In the case that c−n = cn and the an are purely imaginary, the Dirichlet series in (4.3) becomes 2 n>0 cn an n−(s+1/2) . Remark 4.1. The case cn = c−n is the only case we are interested in because it is relevant to congruence subgroups (see (4.4) below). If the an are real, then the Dirichlet series in (4.3) vanishes. This can also be explained in the following manner. The Eisenstein series Ei (z, s) is even in x, and so is u and uy , while ux is odd. If the an ’s are real, w1 = f is even, while w0 = −f is odd. So du, w = y 2 (ux w1 + uy w0 ) is odd and R(s) vanishes. 4.2. Congruence subgroups and central values of L-series. We now assume that g(z) = −if (z) is a Hecke eigenform for some congruence subgroup of level N with Hecke eigenvalues α1 (p) and α2 (p) for Tp and all coefﬁcients a(n) real. Then (1 − a(p)p −s + p 1−2s ) = (1 − α1 (p)p −s )(1 − α2 (p)p −s ) for p N. If p | N, the Euler factor is (1 − a(p)p −s ). We also assume that for n > 0, 1/2−s0 k (4.4) χ1 (c)χ¯ 2 (k) , c±n = c ck=n

PERTURBATION OF SCATTERING POLES

115

where χ1 and χ2 are primitive characters modulo q1 and q2 . It is easy to see that the coefﬁcients cn are multiplicative. In fact ek (z, 1 − s0 ) is also a Hecke eigenform and we easily ﬁnd the Euler factors for its L-series. It is −1 −1 L(s) = . 1 − χ1 (p)p −s−1/2+s0 1 − χ¯ 2 (p)p −s+1/2−s0 p

For example, one can explicitly compute k≥0 cpk p −ks . Now we get the Euler factors of cn a(n)n−s using [30, Lemma 1]. We get ∞ n=1

where

cn a(n)n−s =

Xp (s)Yp (s)−1 ,

p

Xp (s) = 1 − χ1 (p)χ¯ 2 (p)p −2s+1 ,

pN

using α1 (p)α2 (p) = p and Xp (s) = 1 for p | N. Also Yp (s) = 1 − α1 (p)χ1 (p)p −s−1/2+s0 1 − α2 (p)χ1 (p)p −s−1/2+s0 · 1 − α1 (p)χ¯ 2 (p)p −s−s0 +1/2 1 − α2 (p)χ¯ 2 (p)p −s−s0 +1/2 . We denote the twisted L-series of g(z) by a character χ as L(g, χ, s). Then (4.5) ∞ cn a(n) n=1

ns

=

1 − χ1 (p)χ¯ 2 (p)p −2s+1 L g, χ1 , s + 1/2 − s0 L g, χ¯ 2 , s + s0 − 1/2 . p N

We notice that in (4.3) we are interested in the Rankin-Selberg convolution at s +1/2 and that in the ﬁrst variation formula (1.1) we have s = 1−s0 . So we set s = 3/2−s0 in (4.5). We get (4.6)

∞ −1 cn a(n) = L χ1 χ¯ 2 ωN , 2 − 2s0 · L g, χ1 , 2 − 2s0 · L g, χ¯ 2 , 1 , 3/2−s 0 n n=1

where ωN is the trivial character modulo N. We notice that the factor L(g, χ¯ 2 , 1) shows up irrespective of which scattering pole s0 we consider, provided it is a pole of ek (z, s). For the congruence subgroups (N) and 0 (N), which consist of matrices in SL(2, Z) which are lower triangular mod N , the discussion above applies. We notice that 0 (N ) is conjugate to 0 (N), so they have the same spectral theory. According to Huxley [13], the space of nonholomorphic Eisenstein series is spanned by Eisenstein series with characters χ1 (c)χ2 (d)y s Eχχ12 (z, s) = , |cz + d|2s (c,d)=1

116

YIANNIS N. PETRIDIS

where χ1 and χ2 are primitive characters modulo q1 and q2 , respectively, where q1 and q2 are appropriate divisors of N and χ1 (−1) = χ2 (−1). χ (1) For (N ) we take Eχ12 (m1 z/m2 , s), where m1 q1 | N and m2 q2 | N . χ 0 (2) For (N ) we take Eχ (z/m, s), χ primitive modulo q, and q | m, mq | N. χ The Fourier expansion of Eχ12 (z, s) at ∞ is √ s − 1/2 L χ1 , 2s − 1 1−s 1 s 2δ0 (q1 − 1)L(χ2 , 2s)y + 2δ0 (q2 − 1) π y (s) L χ1 χ2 , 2s s−1/2 1/2 4π s τ (χ2 ) k y + χ1 (c)χ¯ 2 (k) (s)q2s c q2 ck=n n>0 2π ny 2πnx × Ks−1/2 cos . q2 q2 The scattering poles appear at s0 , where 2s0 is a zero of L(χ1 χ2 , s). These zeros are conjecturally simple and different from the zeros of the other L-series. It follows that the coefﬁcients of ek (z, 1 − s0 ) are exactly of the form (4.4), up to a factor 2(π/q2 )1−s0 τ (χ2 )/((1 − s0 )L(χ1 χ2 , 2 − 2s0 )). √ Notice that the coefﬁcients cn in [13] differ from (4.4) by√a factor 1/ n, because Huxley includes in the Bessel function Ks−1/2 (u) a factor u. This does not affect (4.6). 4.3. The group 0 (q): Proof of Theorem 1.5. We concentrate now on the Hecke congruence subgroups 0 (q), where q is a prime number. These groups have only two cusps: at ∞ of width 1 and at 0 of width q. The space of Eisenstein series is spanned by the usual Eisenstein series E(z, s) for SL(2, Z) and E(qz, s). They are oldforms. Let Wq be the Fricke involution given by the matrix q0 −1 0 . Then E(z, s) | Wq = E(qz, s). The Fourier expansion of E(z, s) at ∞ is given by (4.7) s

E(z, s) = y + φ(s)y

∞

1−s

2y 1/2 s−1/2 + n σ1−2s (n)Ks−1/2 (2πny) cos(2πnx), ξ(2s) n=1

where ξ(s) = π −s/2 (s/2)ζ (s), φ(s) = ξ(2s − 1)/ξ(2s), and σν (n) = d|n d ν . We set An to be the coefﬁcients in this expansion. Let T and S be the standard generators of√ SL(2, Z) inducing the maps T (z) = −1/z and S(z) = z + 1 on H. Let B = q 0 √ −1 . Then T B conjugates the stabilizer of zero in 0 (q) to ∞ , the standard q 0 parabolic subgroup of SL(2, Z), and it induces the same map as Wq on H. The expansion of E(z, s) at zero is the expansion of E | (T B)(z) = E(qz, s) at inﬁnity.

117

PERTURBATION OF SCATTERING POLES

We have (4.8) s

E(qz, s) = (qy) + φ(s)(qy)

1−s

+

∞

An (qy)1/2 Ks−1/2 (2πnqy) cos(2πnqx).

n=1

For the Fourier expansion of E(qz, s) at zero, we notice that E(qz, s) = E | B(z), which implies that (E | B) | (T B)(z) = E | T (z) = E(z, s). So the Fourier expansion of E(qz, s) at zero is the Fourier expansion of E(z, s) at inﬁnity. Now we can ﬁnd the matrix M(s). Let E(z, s) = (E∞ (z, s), E0 (z, s))T be the vector of Eisenstein series indexed by the cusps. Set (E(z, s), E(qz, s))T = M(s)E(z, s). To ﬁnd M(s) we look at the zero Fourier coefﬁcients in this matrix equation and recall that the scattering matrix is symmetric to get y s + φ(s)y 1−s = m11 (s) y s + φ∞∞ (s)y 1−s + m12 (s)φ0∞ (s)y 1−s , q s y s + q 1−s φ(s)y 1−s = m11 y 1−s φ0∞ (s) + m12 (s) y s + φ00 (s)y 1−s , q s y s + q 1−s φ(s)y 1−s = m21 y s + φ∞∞ (s)y 1−s + m22 (s)φ0∞ (s)y 1−s , y s + φ(s)y 1−s = m21 (s)φ0∞ (s)y 1−s + m22 (s) y s + φ00 (s)y 1−s .

This system gives M(s) =

1

qs qs 1

and

q −1 1 φ(s) s (s) = 2s q −1 q − q 1−s

q s − q 1−s

q −1

as in [12, p. 536]. Consequently det (s) = φ(s)2 (q 2−2s −1)/(q 2s −1). The scattering poles are at s0 = ρ/2, where ρ is a nontrivial zero of ζ (s), and they have multiplicity 2, assuming Condition A. The q 2s − 1 do not give scattering poles, since zeros of −2s φ(s) has a factor 1/ζ (2s) = p (1 − p ). Now assume that g(z) is a holomorphic cusp form of weight 2 for 0 (q), which is also an eigenform of the whole Hecke algebra, with eigenvalue q for Wq . It follows from [1] that q = ±1, a(q) = −q , and a(nq) = a(n)a(q). Let g(z) = 2πinz be the Fourier expansion of g(z) at inﬁnity. For its Fourier expansion n>0 a(n)e at zero, we have 1 −1 g | (T B)(z) = 2 g = g | Wq (z) = q g(z). qz qz So the coefﬁcients at zero are q a(n). We are interested in the Rankin-Selberg convolutions of E(z, 1−s0 ) and E(qz, 1− s0 ) with g(z) expanded at both cusps, inﬁnity and zero. The convolution of E(qz, 1−

118

YIANNIS N. PETRIDIS

s0 ) with g(z) at zero is q times the convolution of E(z, 1 − s0 ) with g(z) at inﬁnity. The convolution of E(z, 1 − s0 ) with g(z) at zero is q times the convolution of E(qz, 1−s0 ) with g(z) at inﬁnity. For the Rankin-Selberg convolution R of E(z, 1− s0 ) with g(z) at inﬁnity, we have χ1 = χ2 = 1, cn = n1/2−s0 σ2s0 −1 (n), and An = 2cn /ξ(2 − 2s0 ). Using (4.6) we get at s = 3/2 − s0 , R=

2π −s0 +1 L(g, 2 − 2s0 )L(g, 1). (1 − s0 )ζ (2 − 2s0 )2 1 − q 2−2s0

Consequently, by (4.3), 2i(1 − s0 )(2 − 2s0 ) ˙ (4.9) R(1 − s0 ) = R. E∞ (z, 1 − s0 )E(z, 1 − s0 ) dµ = 1−s −s 4 0 π 0 (2 − s0 ) \H From (4.8) the Fourier coefﬁcients Bm of E(qz, 1−s0 ) are Bm = 0, if q m and Bnq = An q 1/2 . The Rankin-Selberg convolution of E(qz, 1 − s0 ) with g(z) at inﬁnity is ∞ ∞ Bm a(m) An a(n) 1/2−s = − q = −q q 1/2−s R. q s m ns

(4.10)

m=1

n=1

Using (1.2) we get that the ﬁrst variation of the weighted mean of the scattering poles at s0 is (4.11) s˙ = A(1 + q ) q 2−2s0 − 1 + q q − q 2s0 −1 L(g, 2 − 2s0 )L(g, 1), where A=

2q 2s0 −1

4−s0 π 2s0 i(1 − s0 )(2 − 2s0 ) . 2 1 − q 2−2s0 (s0 )(2 − s0 )ζ (2 − 2s0 )ζ (2s0 )m(2s0 − 1)

We notice that if q = 1, then the functional equation for g(z), which has a sign −q , forces L(g, 1) = 0. In both cases q = ±1, we have s˙ = 0. This proves the ﬁrst statement in Theorem 1.5. To apply Theorem 1.3 we notice that (A1 (z, s0 ), A2 (z, s0 )) = Ress=s0 φ(s)(E(z, 1− s0 ), E(qz, 1 − s0 )). Also

1 −q s0 1 −1 V = M(s0 ) = . 1 − q 2s0 −q s0 1 Using (4.9) and (4.10) we get (4.12)

1 a1

a21

a12

a22

1 + q 2s0 −1 R(1 − s0 ) = Ress=s0 φ(s) 1 − q 2s0 − q s0 − q s0 −1 q

−q s0 − q s0 −1 . 1 + q 2s0 −1 q

PERTURBATION OF SCATTERING POLES

119

If L(g, 1) = 0, then R = 0 and we cannot solve the eigenvalue problem for the j zero matrix (ai ). If q = −1, the matrix in (4.12) has eigenvalue equation λ2 = (1+q 2s0 −1 )2 −(q s0 +q s0 −1 )2 . If λ = 0 is an eigenvalue, then q s0 = ±1 or q s0 −1 = ±1. However, since 2s0 is a zero of ζ (s), this is impossible (|q s0 | = q 1/4 , |q s0 −1 | = q −3/4 ). Also the eigenvalues are distinct and have sum zero. Since (2 − 2s0 ) = 3/2, the values L(g, 2 − 2s0 ) are at the edge of the critical strip; using the argument of de la Vallée Poussin and Hadamard proving that ζ (1 + it) = 0 (see [26]), we see that this value is nonzero. This completes the proof of Theorem 1.5. 5. Proof of Theorem 1.9. The idea is to prove that, for a positive proportion of the (1) scattering poles, the λi ’s in (3.7) for one of the two directions are not imaginary. For that we should prove that the quotient of the ones corresponding to f and g is not real for a positive proportion of the scattering poles. From (4.9) and (4.12) it is clear that, when we take the quotient, we are left with L(g, 1)L(g, 3/2−iγ )/(L(f, 1)L(f, 3/2− iγ )) and all the other factors involving the zeta, gamma functions, and q cancel because they are the same irrespective of the tangent direction. The values L(f, 1) and L(g, 1) are real for newforms with real coefﬁcients. This reduces the issue to proving that, for a positive proportion of γ ’s, the quotient L(f, 3/2 + iγ )/L(g, 3/2 + iγ ) is not real. This is equivalent to the nonvanishing of L(g, 3/2 + iγ )L(f, 3/2 − iγ ) − L(g, 3/2−iγ )L(f, 3/2+iγ ). We prove weighted mean value results for these values. Let the Hecke eigenvalues for f (z) be β1 (p) and β2 (p), and let the Hecke eigenvalues for g(z) be α1 (p) and α2 (p). We introduce weights B(s, P ) = 1 − α1 (p)p −s 1 − α2 (p)p −s 1 − β1 (p)p −s 1 − β2 (p)p −s p≤P

for P a suitable prime. Let us denote by ∗ the operation of Rankin-Selberg convolution on two Dirichlet series, and let N(T ) be the number of zeros ρ = 1/2 + iγ of ζ (s) with 0 < γ ≤ T . Proposition 1. Under the RH, B 3/2 + iγ , P L g, 3/2 + iγ L f, 3/2 − iγ (5.1) 0<γ ≤T ∼ B(·, P )L(g, ·) ∗ L(f, ·)(3)N(T ) and, by symmetry, (5.2)

B 3/2 + iγ , P L f, 3/2 + iγ L g, 3/2 − iγ

0<γ ≤T

∼ B(·, P )L(f, ·) ∗ L(g, ·)(3)N(T ). Proposition 2. Let A(γ ) = B 3/2+iγ ,P L g, 3/2+iγ L f, 3/2−iγ −L f, 3/2+iγ L g, 3/2−iγ .

120

YIANNIS N. PETRIDIS

Under RH,

(5.3)

|A(γ )|2 $ N(T ).

0<γ ≤T

Remark 5.1. The discrete mean value results in Propositions 1 and 2 correspond to mean value theorems for ζ (s)4 and ζ (s)8 , respectively, on the line of convergence s = 1. It is well known that on this line one has mean value theorems for all powers of ζ (s) (see [31, p. 148, 7.7.1]). Proposition 3. Under RH and (1.5) there exists a P such that the two limits in (5.1) and (5.2) are different, that is, (5.4) A(γ ) ∼ C · N(T ), C = 0. 0<γ ≤T

In fact, we need the weights B(s, P ) here. Without them the mean values in (5.1) and (5.2) are equal. The rest of the proof of Theorem 1.9 is easy. By the CauchySchwarz inequality, (5.3), and (5.4), we have 2 |C|2 N(T )2 0<γ ≤T A(γ ) = |C|2 N(T ). 1≥ 2 N(T ) 0<γ ≤T |A(γ )| 0<γ ≤T , A(γ )=0

This proves that a positive proportion of the A(γ )’s are nonzero; in particular, for the same γ ’s, the quotient L(g, 3/2 + iγ )/L(f, 3/2 + iγ ) is not real. Proof of Proposition 1. The idea is to imitate the method used by Gonek [9] to prove discrete mean value formulas for the zeta function. More precisely, [9, Th. 1] states T .(x) + O x log(2xT ) log log(3x) xρ = − 2π 0<γ ≤T (5.5) x 1 + O log x min T , + O log(2T ) min T , , x log x where x, T > 1, and x denotes the distance from x to the nearest prime power other than x itself. This is a uniform version of a theorem by Landau (see [9]). We use the approximate functional equation for L(f, s) and L(g, s), as stated in [10, Kor. 2, p. 333]. If s = 3/2 + it, we have L(g, s) = an n−s + χ(s) an ns−2 + O |t|−1/2+ (5.6) n≤|t|q/2π

n≤|t|q/2π

χ (s) = (2π/q)2s−2 (2−s)/ (s).

with The weights B(s, P ) are given by a Dirichlet polynomial of ﬁxed length depending on P , say, B(s, P ) = n≤R cn n−s . Let B(s, P )L(g, s) =

∞ dn ns n=1

121

PERTURBATION OF SCATTERING POLES

and

B(s, P )

m≤|t|q/2π

am = ms

n≤|t|qR/2π

dn . ns

Clearly, if n ≤ |t|q/2π , we have dn = dn . Moreover, dn and dn grow no more quickly than an , that is, an , dn , dn $ n1/2+ by the Ramanujan conjecture. The approximate functional equation for L(f, s) gives

L(f, s) =

(5.7)

bn n−s + χ(s)

n≤|t|qR/2π

bn ns−2 + O |t|−1/2+ .

n≤|t|q/(2πR)

The main term in

(5.8)

B 3/2 + iγ , P L g, 3/2 + iγ L f, 3/2 − iγ

0<γ ≤T

comes from

dn n−3/2−iγ ·

0<γ ≤T n≤γ qR/2π

=

 

0<γ ≤T

n≤γ qR/2π

bn n−3/2+iγ

n≤γ qR/2π

 γ qR/2π d bn n iγ dn bn m  = Z 1 + Z2 . + n3 (nm)3/2 m n=m

We have

Z1 =

 

0<γ ≤T

∞ dn bn n=1

n3

−

n>γ qR/2π

dn bn + n3

n≤γ qR/2π

 (dn − dn )bn  n3

= N(T )B(·, P )L(g, ·) ∗ L(f, ·)(3) + C1 + C2 , where C1 $

0<γ ≤T n>γ

n−2+ $

γ −1+ = o N(T )

0<γ ≤T

and C2 $

0<γ ≤T n>γ q/2π

dn − dn bn $ n−2+ = o N(T ) . 3 n n>γ 0<γ ≤T

122

YIANNIS N. PETRIDIS

Using (5.5) we get

Z2 =

n≤T qR/2π m
=−

T 2π

n≤T qR/2π



+O 

n≤T qR/2π

 +O 

n≤T qR/2π

 +O 

n≤T qR/2π

b n ρ dn bm n ρ dm n + n2 m m n2 m m

d bn + d bm n m n . 2m m n m


+ O log(2T )

n≤T qR/2π

 d bn + d bm 1 m n  min T , 2m log(n/m) n m
=Z21 + Z22 + Z23 + Z24 + Z25 . To estimate Z21 and Z22 , we set n = km. We get Z21 $

T 2π

k≤T qR/2π m
.(k) k 3/2− m2−

= O(T ),

since .(k) $ k and the sums are partial sums of two convergent series. Moreover, Z22 $

k≤T qR/2π m
.(k) T .(k) $ $ T 1/2+ . k 1/2− m1− k 1/2− k k$T

Similarly, one easily gets Z23 $ T 1/2+ log T log log T . For Z24 we set n = lm + r, where −m/2 < r ≤ m/2. This implies that  |r|   , if l is a prime power and r = 0, r = m l+  m ≥ 1 , otherwise. 2

123

PERTURBATION OF SCATTERING POLES

Together with n/m ≤ n ≤ cT , c = qR/(2π), this gives Z24 $ log T

n≤cT m

$ log T

1 m3/2− n1/2− n/m

m≤cT l≤%cT /m&+1 −m/2

$ log T

.(l)

m≤cT l≤%cT /m&+1

$ log T

log m m1−2

m≤cT

$ T 1/2+ log T

m3/2−

1 1/2− ! lm + r l + r/m

m log m m + m3/2− (lm)1/2− m3/2− (lm)1/2−

l

l$cT /m

l 1/2−

log m $ T 1/2+ log T . m3/2−

m≤cT

For Z25 we set m = n − r, 1 ≤ r ≤ n − 1, so that log(n/m) > r/n. This gives Z25 $ log T

n≤cT r≤n−1

$ log T

1 n/r 1 $ log T r n3/2− (n − r)1/2− n1/2− n≤cT

r≤n−1

log n $ T 1/2+ log T . n1/2−

n≤cT

The analysis above depends only on the order of growth of dn and bn , so we can get the bounds 2 (5.9) dn n−3/2−iγ $ N(T ), γ ≤T n≤γ qR/2π

(5.10)

bn n

γ ≤T n≤γ q/(2πR)

2 −3/2+iγ

$ N(T ).

Moreover, we can repeat the argument with trivial modiﬁcations to estimate 2 an n−1/2+iγ $ T 1+ N(T ), (5.11) γ ≤T n≤γ q/2π

(5.12)

γ ≤T n≤γ q/(2πR)

2 bn n−1/2−iγ $ T 1+ N(T ).

124

YIANNIS N. PETRIDIS

We also need to analyze the order of growth of the derivative of |χ(3/2 + iγ )|2 . We have 2 2 d χ 3/2 + iγ = i(2π/q)2 χ 3/2 + iγ dγ · ψ 1/2 − iγ + ψ 1/2 + iγ − ψ 3/2 + iγ − ψ 3/2 − iγ , where ψ(z) = (z)/ (z). We can differentiate the asymptotics of log (z) (see [11, 8.344, p. 949]), to get 1 ψ(z) = log z − + O(z−2 ). 2z We also use the asymptotic formula √ (5.13) (x + iy) ∼ 2πe−π|y|/2 |y|x−1/2 , |y| −→ ∞, x, y ∈ R (see [11, p. 945, 8.328.1]), to get 2 d χ 3/2 ± iγ $ γ −3 . dγ

(5.14)

We use summation by parts, (5.14), (5.11), and (5.12) to estimate 2 2 −3/2−iγ cn n χ 3/2 + iγ

γ ≤T

n≤R

$ T = o N(T ) ,

n≤γ q/2π

2 χ 3/2 − iγ

γ ≤T

an n

2 −1/2+iγ

bn n

2 −1/2−iγ

n≤γ q/(2πR)

$ T = o N(T ) .

Finally we use the Cauchy-Schwarz inequality, (5.9), (5.10), and the two equations above to estimate all other product terms in (5.8) as o(N (T )). Lemma 5.2. For 0 ≤ a < 3/2, there exists a constant c > 0 such that |an |

(5.15)

n≤x

na

$

x 3/2−a logc x

and (5.16)

|an | log n n≤x

Similar estimates hold for bn .

na

$ x 3/2−a log1−c x.

PERTURBATION OF SCATTERING POLES

125

√ Proof. Set A(x) = n≤x |an |, where an = an n. Rankin’s estimate [27] implies that A(x) $ x(log x)−c for some c > 0. We have x |an | |a | A(x) n = = + (a − 1/2) A(t)t −a−1/2 dt a a−1/2 a−1/2 n n x 1 n≤x n≤x using partial summation. Notice that ond estimate is proved similarly.

x 2

t 1/2−a log−c t dt $ x 3/2−a log−c x. The sec-

Proof of Proposition 2. Since B(3/2 + iγ , P ) is a ﬁnite Dirichlet polynomial, it is bounded independently of T . To estimate γ ≤T |A(γ )|2 , it sufﬁces to estimate 2 2 L g, 3/2 + iγ L f, 3/2 + iγ $ N(T ).

(5.17)

γ ≤T

As in the proof of Proposition 1, we use the approximate functional equation on s = 3/2 for L(g, s) and L(f, s) (see (5.6), (5.7)), bn n−s + χ(s) bn ns−2 + O |t|−1/2+ L(f, s) = n≤|t|q/2π

n≤|t|q/2π

= W1 + χ (s)W2 + O |t|−1/2+ , L(g, s) = Y1 + χ (s)Y2 + O |t|−1/2+ . We have (5.18)

Y1 Y1 W1 W1 =

0<γ ≤T

0<γ ≤T m,n,µ,ν≤γ q/2π

am bn aµ bν µν iγ . (mnµν)3/2 mn

The main term comes again from the contribution of the diagonal terms, and the number of solutions to mn = µν = r is less than or equal to d(r)2 , where d(r) is the divisor function. The main term can be estimated as (5.19)

q/2π ∞ γ am bn aµ bν d(r)2 r 1+2 $ $ N(T ), (mn)3 r3 mn=µν

γ ≤T

γ ≤T r=1

since the inner series converges. We set µν = r and mn = s. We can treat the case s < r and s > r separately. The range of the following sums is subject to the restriction m, n, µ, ν ≤ T q/(2π ). For s < r the other terms in (5.18) contribute (5.20)

Z2 =

r≤(T q/2π)

am bs/m aµ br/µ r 3/2 s 3/2 2 s
r iγ , s

K≤γ ≤T

126

YIANNIS N. PETRIDIS

where K = min(T , (2π/q) max(m, s/m, µ, r/µ)). We apply (5.5) to Z2 :   r ρ r ρ am bs/m aµ br/µ   − Z2 = s s r 2s 2 s
r$T

=

γ
0<γ ≤T

r

am bs/m aµ br/µ K − T . 2π s r 2s 2 s
r$T

m|s, µ|r



 am bs/m aµ br/µ 3r 2T r  +O  log log log 2 s s rs 2 s
m|s, µ|r

r$T

m|s, µ|r

r$T

m|s, µ|r



 r am bs/m aµ br/µ r/s  +O  log min T , s r/s r 2s 2 s
 am bs/m aµ br/µ 1  +O  log(2T ) min T , 2s log(r/s) r 2 s

.(k)s 1/2+ (sk)1/2+ $ T, s 3k2 2

k$T 2 s$T /k

since both series converge. Using (5.15), we get µ,ν≤T q/2π

aµ bν T $ . µν log2c T

We have Z23 $ log T log log T $ log T log log T

an bm aµ bν µνm2 n2 m,n,µ,ν T log T 2c

= o N(T ) ,

since the series am m−2 converges. For Z24 we set r = ls + t, −s/2 < t ≤ s/2, and we distinguish two cases as in Proposition 1. Case 1 occurs when l is a prime power and t = 0, and case 2 happens

PERTURBATION OF SCATTERING POLES

127

otherwise. The contribution from case 2 is $ log T

am bn aµ bν $ T log1−2c T , 2 n2 µνm m,n,µ,ν

as for the estimate of Z23 . In case 1 we distinguish two subcases depending on whether T is larger than (r/s)/r/s or not. If T is larger, T > µν/|t|, which implies T > l. The contribution Z24,1 of these terms is Z24,1 $

am bn aµ bν log l + t/(mn) mnµν|t| m,n,µ,ν T >l

$

m,n

$

am bn m,n

mn

log l (lmn + t)1/2− |t|

0=|t|<mn/2 l≤T

am bn log l log(mn) 3/2− (mn) (l − 1/2)1/2− l≤T

$ T 1/2+

m,n≤T q/2π

am an log(mn) $ T 1/2+3 log2−2c T , (mn)3/2−

using (5.16) and log(mn) ≤ log n log m for log n, log m ≥ 2. If T is smaller than (r/s)/r/s, we have T ≤ µν/|t| and this implies l > T |t|/(2mn). Let the contribution of these terms be Z24,2 . We ﬁrst analyze the summation over µ, ν. We see that aµ bν $ µ2 ν 2

µν≥T |t|

1 (lmn + t)3/2−

0=|t|≤mn/2 l>T |t|/(2mn)

1 $ (mn)3/2− $ T −1/2+

0=|t|≤mn/2

1 (mn)1−

T |t| 2mn

−1/2+

|t|−1/2+ $ (T mn)−1/2+ .

0=|t|≤mn/2

The summation over m, n now gives Z24,2 $ T log T

m,n≤T q/2π

am bn T −1/2+ (mn)3/2−

2 $ T 1/2+ log T T / logc T = o N(T ) .

128

YIANNIS N. PETRIDIS

This takes care of Z24 . Using again the Ramanujan conjecture for an , bµ , we get

Z25 $ log T

m,n,µ,ν≤T q/2π,s

$ T log T

s

$ T log T

s
bn am aµ bν 2 2 µ ν mn log(µν/(mn))

d(r)d(s) 3/2 s 1/2 log(r/s) r 2 2 1 $ T log T log T 2 , sr log(r/s) 2

where in the last line we used [31, Lemma 7.2, p. 139], with the obvious modiﬁcations for σ = 1. The rest of the proof follows as in the proof of Proposition 1. In fact one can easily see that we can not only get upper bounds for γ ≤T |A(γ )|2 , but we can also identify the main term in the asymptotics of it. Proof of Proposition 3. It is clear that the product of two Dirichlet series with multiplicative coefﬁcients also has multiplicative coefﬁcients and this is also true for their Rankin-Selberg convolution. This allows us to work the Euler factors separately for each prime. Since B(s, P )L(g, s) =

"

#−1 1 − α1 (p)p −s 1 − α2 (p)p −s

p>P

(5.21)

×

1 − β1 (p)p −s 1 − β2 (p)p −s ,

p≤P

the convolution B(s, P )L(g, s) ∗ L(f, s) has the same Euler factors for p > P as L(g, s)∗L(f, s). The same is true for the Euler factors with p > P for B(s, P )L(f, s)∗ L(g, s). So when we subtract (5.2) from (5.1) to get (5.4) we get a factor −1 1 − α1 (p)α2 (p)β1 (p)β2 (p)p −2s 1 − αi (p)βj (p)p −s p>P

1≤i, j ≤2

using [30, Lemma 1]. The value of this at s = 3 is nonzero, since 3 is in the domain of convergence. To show that the asymptotics in (5.4) have C = 0 for some P , let us assume that for all P prime the difference in the other Euler factors with p ≤ P in (5.1) and (5.2) is zero at s = 3. We analyze these Euler factors. Fix p ≤ P . For B(s, P )L(g, s) ∗ L(f, s) we get ∞ 1 − b(p)p −s + p 1−2s ∗ b p k p −ks = 1 − b(p)2 p −s + pb p 2 p −2s

k=0

PERTURBATION OF SCATTERING POLES

129

while for B(s, P )L(f, s) ∗ L(g, s) we get ∞ 1 − a(p)p −s + p 1−2s ∗ a(p k )p −ks = 1 − a(p)2 p −s + pa(p 2 )p −2s . k=0

The Hecke relations give b(p 2 ) = b(p)2 − p and a(p 2 ) = a(p)2 − p. If for all P we have 1 + b(p)2 p −5 − p −3 − p −4 = 1 + a(p)2 p −5 − p −3 − p −4 , p≤P

p≤P

we get equality for the individual terms, by considering successive primes P and by dividing the corresponding relations. This gives b(p)2 = a(p)2 for all p, contradicting the assumption (1.5). The case p = q is even simpler. References [1] [2]

[3] [4] [5] [6]

[7] [8] [9]

[10] [11] [12] [13]

[14]

A. Atkin and J. Lehner, Hecke operators on 0 (m), Math. Ann. 185 (1970), 134–160. E. Balslev and A. Venkov, Stability of character resonances, preprint, Center for Mathematical Physics and Stochastics, Department of Mathematical Sciences, Univ. of Aarhus, January 1999. J. B. Conrey, More than two ﬁfths of the zeros of the Riemann zeta function are on the critical line, J. Reine Angew. Math. 399 (1989), 1–26. J. B. Conrey, A. Ghosh, and S. M. Gonek, Simple zeros of the Riemann zeta-function, Proc. London Math. Soc. (3) 76 (1998), 497–522. J. Cremona, Algorithms for Modular Elliptic Curves, Cambridge Univ. Press, Cambridge, 1992. L. Faddeev, Expansion in eigenfunctions of the Laplace operator on the fundamental domain of a discrete group on the Lobacevskii plane, Trans. Moscow Math. Soc. 17 (1967), 357–386. R. Froese and M. Zworski, Finite volume surfaces with resonances far from the unitarity axis, Internat. Math. Res. Notices 1993, 275–277. S. M. Gonek, Mean values of the Riemann zeta function and its derivatives, Invent. Math. 75 (1984), 123–141. , “An explicit formula of Landau and its applications to the theory of the zeta-function” in A Tribute to Emil Grosswald: Number Theory and Related Analysis, Contemp. Math. 143, Amer. Math. Soc., Providence, 1993, 395–413. A. Good, Approximative Funktionalgleichungen und Mittelwertsätze für Dirichletreihen, die Spitzenformen assoziiert sind, Comment. Math. Helv. 50 (1975), 327–361. I. S. Gradshteyn and I. M. Ryzhik, Table of Integrals, Series, and Products, 5th ed., ed. Alan Jeffrey, Academic Press, Boston, 1994. D. Hejhal, The Selberg Trace Formula for PSL(2, R), Vol. 2, Lecture Notes in Math. 1001, Springer, Berlin, 1983. M. N. Huxley, “Scattering matrices for congruence subgroups” in Modular Forms (Durham, 1983), Ellis Horwood Ser. Math. Appl. Statist. Oper. Res., Horwood, Chichester, 1984, 141–156. T. Kato, A Short Introduction to Perturbation Theory for Linear Operators, Springer, New York, 1982.

130 [15] [16] [17]

[18] [19] [20] [21] [22] [23]

[24] [25] [26]

[27] [28]

[29] [30] [31]

YIANNIS N. PETRIDIS S. Lang, SL2 (R), Grad. Texts in Math. 105, Springer, New York, 1985. P. Lax and R. Phillips, Scattering Theory for Automorphic Functions, Ann. of Math. Stud. 87, Princeton Univ. Press, Princeton, 1976. H. Montgomery, “The pair correlation of zeros of the zeta function” in Analytic Number Theory (St. Louis, Mo., 1972), Proc. Sympos. Pure Math. 24, Amer. Math. Soc., Providence, 1973, 181–193. W. Müller, Spectral geometry and scattering theory for certain complete surfaces of ﬁnite volume, Invent. Math. 109 (1992), 265–305. , On the analytic continuation of rank one Eisenstein series, Geom. Funct. Anal. 6 (1996), 572–586. A. Odlyzko, On the distribution of spacings between zeros of the zeta function, Math. Comp. 48 (1987), 273–308. Y. Petridis, On the singular set, the resolvent and Fermi’s golden rule for ﬁnite volume hyperbolic surfaces, Manuscripta Math. 82 (1994), 331–347. , Spectral data for ﬁnite volume hyperbolic surfaces at the bottom of the continuous spectrum, J. Funct. Anal. 124 (1994), 61–94. , “Variation of scattering poles for conformal metrics” in Spectral Problems in Geometry and Arithmetic (Iowa City, Iowa, 1997), Contemp. Math. 237, Amer. Math. Soc., Providence, 1999, 149–158. R. Phillips and P. Sarnak, On cusp forms for co-ﬁnite subgroups of PSL(2, R), Invent. Math. 80 (1985), 339–364. , Perturbation theory for the Laplacian on automorphic functions, J. Amer. Math. Soc. 5 (1992), 1–32. R. Rankin, Contributions to the theory of Ramanujan’s function τ (n) and similar arithmetical s functions, I: The zeros of the function ∞ n=1 τ (n)/n on the line s = 13/2, Proc. Cambridge Philos. Soc. 35 (1939), 351–356. , Sums of powers of cusp form coefﬁcients, II, Math. Ann. 272 (1985), 593–600. P. Sarnak, “On cusp forms, II” in Festschrift in Honor of I. I. Piatetski-Shapiro on the Occasion of his Sixtieth Birthday (Ramat Aviv, 1989), Part II, Israel Math. Conf. Proc. 3, Weizmann, Jerusalem, 1990, 237–250. A. Selberg, “Remarks on the distribution of poles of Eisenstein series” in Collected Papers, Vol. 2, Springer, Berlin, 1991, 15–45. G. Shimura, The special values of the zeta functions associated with cusp forms, Comm. Pure Appl. Math. 29 (1976), 783–804. E. Titchmarsh, The Theory of the Riemann Zeta-Function, 2d ed., ed. D. R. Heath-Brown, Oxford Univ. Press, New York, 1986.

Department of Mathematics and Statistics, McGill University, 805 Sherbrooke Street West, Montreal, Quebec H3A 2K6, Canada Current: Department of Mathematics and Statistics, Queen’s University, Kingston, Ontario K7L 3N6, Canada; [email protected]

Vol. 103, No. 1

DUKE MATHEMATICAL JOURNAL

© 2000

ZERO-CYCLES ON HILBERT-BLUMENTHAL SURFACES ANDREAS LANGER

Introduction. The main object of study in this paper with respect to zero-cycles is a special class of Hilbert-Blumenthal surfaces X, which are deﬁned over Q as smooth compactiﬁcations of quasi-projective varieties S/Q, more precisely, of coarse moduli schemes S that represent the moduli stack of polarized abelian surfaces with real √ multiplication by the ring of integers in a real quadratic ﬁeld F = Q( d). We assume that d = q is a prime ≡ 1(4) and that the class number of F is 1. Then S(C), the complex points of S, can be described as H×H/ SL2 (OF ), where H is the upper halfplane. In the early seventies, Hirzebruch and Zagier [HZ] deﬁned for each integer N a curve TN on S (called Hirzebruch-Zagier cycles) and showed that their intersection numbers occur as Fourier coefﬁcients of modular forms of level q with Nebentypes εq , the quadratic character of F /Q. In this connection with modular forms, HirzebruchZagier cycles reveal very similar properties to Hecke correspondences on the selfproduct of the modular curve X0 (q). This crucial observation of Hirzebruch and Zagier, together with Tunnell’s proof of the Tate conjecture for a product of modular curves, inspired Harder, Langlands, and Rapoport [HLR] to prove the Tate conjecture for divisors on Hilbert-Blumenthal surfaces over abelian number ﬁelds. (The proof of the Tate conjecture was then accomplished by Klingenberg [Kl] and Murty and Ramakrishnan [MR] in the general case.) From the new strategy to study torsion zero-cycles on algebraic surfaces as developed in [LS] and used in [L1], [L2] (compare also [LR]), it is clear that a crucial point is the Tate conjecture in characteristic p at good reduction primes. As one of our main results, we prove the Tate conjecture in characteristic p for primes p that split in F for a certain class of Hilbert-Blumenthal surfaces. For this we recall that to each modular cusp form f of weight 2, level q, and Nebentypes εq , that is, f ∈ S2 (0 (q), εq ), we associate a Hilbert modular cusp form fˆ ∈ S2 (SL2 (OF )) under the Doi-Naganuma lift DN F (compare [vdG]). Then we prove the following theorem. Theorem A. Let p be a prime that splits in F and such that X has good reduction at p. Let Xp be the closed ﬁber of a smooth, proper model ᐄ/Zp of X. Assume that DN F is surjective. Then the Tate conjecture holds for divisors on Xp . Examples of such surfaces can be found in [O]. From the work of [HLR], we know that the interesting part in the second étale cohomology of X is intersection cohomology that decomposes into isotypic components Received 2 June 1998. Revision received 24 September 1999. 2000 Mathematics Subject Classiﬁcation. Primary 10D21, 14C25, 14C13; Secondary 10D20, 18F25. 131

132

ANDREAS LANGER

under the action of the Hecke algebra. The local Tate classes in these isotypic components, that is, the invariants under Frobenius, can be easily computed. It turns out that for primes p that are inert in F , the dimension of the space of local Tate classes is three times the one of global Tate classes. It seems to be very difﬁcult to construct cycles that cover these Tate classes. For split primes p, we get in each isotypic component two local Tate classes, so there are also new cycles to construct in characteristic p. To bound the dimension of local Tate classes from above, we use a result of Coleman and Edixhoven [CE] that states that the eigenvalues of Frobenius acting on the 2-dimensional Galois representation associated to a modular form are different. In order to construct these new cycles in characteristic p for split primes p, we ﬁrst remember the behaviour of the Hecke correspondence on the self-product of a modular curve: It satisﬁes an Eichler-Shimura identity, that is, its reduction mod p can be expressed by the graphs of Frobenius and its transpose. In our situation, there is a birational morphism Y0 (p) → Tp , where Tp is the Hirzebruch-Zagier cycle on S. Then from the reduction of Y0 (p) mod p, we get two components F and V of Tp mod p, which is a curve on Sp = ᏿Zp ⊗Zp Fp where ᏿Zp is the model of S over Zp considered as a coarse moduli scheme. Now these components F and V deﬁne cuspidal parts F c , V c in the intersection cohomology. From its decomposition under the action of the Hecke algebra into isotypic components, we get, for each isotypic component, components of F c and V c , and we have to show that they are linearly independent. This is achieved by considering a suitable Galois covering SKN of S (p N) that represents the moduli stack of abelian surfaces with real multiplication and full-level N -structure. It is a ﬁne moduli scheme, and we perform the construction of Hirzebruch-Zagier cycles on SKN by giving an explicit modular description of these cycles, in particular, in characteristic p. Here a combination of Frobenius and the real multiplication on E × E/Fp , where E is an elliptic curve over Fp , plays a crucial role. The ﬁnal argument consists of linear algebra of Hecke operators acting on the isotypic components of intersection cohomology. We again use the result of Coleman and Edixhoven, which is applicable because under our assumption, each isotypic component is, noncanonically, isomorphic to the vector space of endomorphisms of the 2-dimensional Galois representation associated to a modular form of level q (the discriminant of F ). In the second section, we pass to the study of torsion zero-cycles on our HilbertBlumenthal surface X. Here the boundary map ∂ : H 1 (X, ᏷2 ) −→

⊕

X good red. at

Pic(X ),

obtained from the localization sequence in algebraic K-theory, plays a crucial role. Concerning ∂, we have the following partial result. Let S/Q be the Borel-Satake compactiﬁcation of S, and let U ⊂ Spec Z be the open subscheme such that the morphism X → S extends to a morphism ᐄ → ᏿ over U , where ᐄ is smooth and proper, that is, the quotient and cusp singularities of the

ZERO-CYCLES ON HILBERT-BLUMENTHAL SURFACES

133

singular surface S can be resolved simultaneously over U (compare the beginning of Section 2). Theorem B. Let X be the Hilbert-Blumenthal surface deﬁned as before, and let

ᐄ be a smooth, proper model over U as above. Furthermore, assume that the Doi-

Naganuma lift DN F is surjective. Then the boundary map in (2.1), when restricted to all split primes, that is, ∂

H 1 (X, ᏷2 ) ⊗ Q −→

⊕

∈U splits in F

Pic(X ) ⊗ Q,

is surjective. To get new elements in H 1 (X, ᏷2 ), we observe that the modular unit constructed by Flach and Mildenhall on X0 (p) also deﬁnes a modular unit on the Hirzebruch-Zagier cycle Tp because they have the same function ﬁelds. By applying a construction that was ﬁrst given by Ramakrishnan [R1], [R2], and [R3], that is, by adding a certain formal sum i (Di , fi ), where Di is a rational curve in the resolutions of the cusp or quotient singularities and fi ∈ k(Di ), we get a new, well-deﬁned (i.e., indecomposable) element in H 1 (X, ᏷2 ) for each split prime p. Their images under Hecke correspondences and the decomposable elements coming from Pic(X) ⊗ Q∗ generate a subspace in H 1 (X, ᏷2 )⊗Q, the boundary of which at p covers the whole vector space Pic(Xp ) ⊗ Q. As a corollary, we derive the following local ﬁniteness result on torsion zero-cycles. Theorem C. Let p ∈ U be a split, good reduction prime of X in U , and let XQp := X ×Q Qp . Then Ch0 (XQp )tor , the prime-to-p torsion in Ch0 (XQp ), is ﬁnite. In the global situation, we need additional assumptions in terms of well-known conjectures on values of L-functions. They are speciﬁed in the conditions (H1), (H2), (H3) preceding Theorem 2.5. (H1) is an S-integral version of Beilinson’s conjecture [R4, Conjecture 6.8.7] relating the order of pole of the partial L-function LS (H 2 (X), s) at s = 1 to the dimension of the S-integral motivic cohomology. Here S is a ﬁnite set of primes including all bad reduction primes. (H2) is the Tate conjecture for divisors in characteristic p at inert good reduction primes. (H3) is a suitable analogous condition for inert bad reduction primes. Then we show the following theorem. Theorem D. Let X be as in Theorem A, let ᐄ be a smooth, proper model over U , and assume that the Doi-Naganuma lift DN F is surjective. Assume furthermore that the hypotheses (H1) for varying S, (H2) for inert good primes, and (H3) for inert primes not lying in U are satisﬁed. Then Ch0 (X){p} is ﬁnite for all primes p ∈ U , p 6. For the proof of this theorem, we would like to apply the general framework that was developed in [LS] in the case of the self-product of a modular elliptic curve. So one has to elaborate the following two tasks: derive the surjectivity of ∂ ⊗ Q, and prove the ﬁniteness of the Selmer group associated to the motive H 2 (X)(2).

134

ANDREAS LANGER

It turns out that the surjectivity of ∂ ⊗ Q is a consequence of (H1) and (H2). By using our assumption on X, the ﬁniteness of the Selmer group can be reduced, by a norm argument, to the ﬁniteness of the Selmer group over F (!) associated to the adjoint representation of a modular form, which is known due to the recent work of Fujiwara [Fu] who extends an analogous result of Wiles [W] from Q to totally real ﬁelds. Of course, another crucial point in the proof of Theorem D is the description of (2) the image of the p-adic (étale) regulator map on K1 in terms of what Bloch and 2 Kato call the local points of the motive H (X)(2). The result is based on Schneider’s p-adic points conjecture [S] relating syntomic cohomology to Hf1 , and a description of the cohomology of the truncated complex of p-adic vanishing cycles in terms of Hg1 in the sense of [BK]. The latter result is a consequence of the main theorem in [L3], which treats a semistable analog of Schneider’s conjecture. We hope to convince the reader, through the proof of Theorems B, C, and D, how closely related the study of algebraic cycles is to p-adic Hodge theory and the theory of modular forms and their associated Galois representations. On the other hand, the deep theorems of Fontaine and Messing, Kato, Tsuji, and Faltings on the comparison of p-adic cohomology theories now have an application in the cycle world. The interplay of these different branches in arithmetic geometry deserves to be explored much more. Theorem C and the conditional Theorem D provide a ﬁrst example of a Shimura variety (of dimension greater than 1) where we can obtain a ﬁniteness result on higher Chow groups. Acknowledgements. First, I would like to thank T. Saito for pointing out a gap related to the Tate conjecture at inert primes in an earlier version of this work. I then want to thank U. Jannsen for his suggestion to study Hilbert-Blumenthal surfaces with respect to zero-cycles. It is a pleasure for me to thank M. Rapoport and A. J. Scholl for several very helpful discussions on this work and for their interest and encouragement. Finally, I thank G. Kings, K. Künnemann, H. Reimann, S. Saito, and Th. Zink for useful comments while I was writing this paper. 1. Modular description of Hirzebruch-Zagier cycles and the Tate conjecture in ﬁnite characteristic. Let H = {z ∈ C, Im z > 0} be the upper half-plane. Fix a √ prime q ≡ 1(4), and let F = Q( q) with ring of integers OF . Assume that there is a unit of norm −1 and that the class number of F is 1. Then SL2 (OF ) operates on H × H in the following way: az1 + b a z2 + b a b (z1 , z2 ) = , , c d cz1 + d c z2 + d where means conjugation by the nontrivial element σ in Gal(F /Q). The quotient H × H/ SL2 (OF ) is a noncompact complex surface with ﬁnitely many (quotient) singularities. Following [HZ], for each integer N, we deﬁne curves TN on X = H × H/ SL2 (OF ) as follows.

ZERO-CYCLES ON HILBERT-BLUMENTHAL SURFACES

135

Given N, consider all points (z1 , z2 ) ∈ H2 satisfying some equation of the form √ √ a q z1 z2 + λz2 − λ z1 + b q = 0, with a, b ∈ Z, λ ∈ OF , λλ + abq = N. This set is invariant under SL2 (OF ) and the curve TN is deﬁned as its image in X. The curves TN are called Hirzebruch-Zagier cycles. We have that the property (N/q) = −1 ⇔ TN is nonempty and TN is compact if N is not the norm of an ideal of OF . For primes p that split in OF , we look at the construction of Tp from a slightly different view, which becomes useful when we want to give a modular description of these curves (compare [HZ], [vdG, Chapter V]). Let ℘ be a prime in OF above p, ℘ = (π), such that π is totally positive and π · π = p. a b We look at the matrix group SL(OF ⊕ ℘) = {A = c d , a, d ∈ OF , c ∈ ℘, b ∈ ℘ −1 , −1 det A = 1} and consider the element g = π0 01 , which acts on H × H as follows: −1

0 π (z1 , z2 ) = π −1 z1 , π −1 z2 . 0 1 Since g −1 SL(OF ⊕ ℘)g = SL2 (OF ), we have an isomorphism φp = H × H/ SL(OF ⊕ ℘) ∼ = H × H/ SL2 (OF ). The subgroup of SL2 (Z) that ﬁxes the diagonal in H × H/ SL(OF ⊕ ℘) is 0 (p). Tp is the image of the diagonal under φp , and we have a birational morphism α : Y0 (p)(C) = H/ 0 (p) −→ Tp ⊂ X. This construction of Tp allows us to give a modular description of the points lying on them. First, it is well known (see [Ra]) that H × H/ SL2 (OF ) parametrizes polarized abelian surfaces A/C together with an endomorphism (called real multiplication): m : OF −→ End(A),

such that -20 H1 (X, Z) ∼ = OF .

The diagonal point z = (z1 , z1 ) in H × H/ SL(OF ⊕ ℘) corresponds to the abelian surface A = F ⊗ C/Lz (OF ) where Lz (OF ) is the lattice 2πi(℘ + zOF ). Then the

point φp (z) = (π −1 z1 , π −1 z1 ) ∈ H×H/ SL2 (OF ) corresponds to the abelian surface

A = F ⊗ C/2π i(OF + (π −1 z1 , π −1 z1 )OF ). Let E = Ez1 = C/2πi(Z + z1 Z), and let E ⊗ OF be the following abelian surface with real multiplication m, as an abelian √ surface E ⊗ OF is E ×C E and the real multiplication m is deﬁned by letting q act 0 q through the matrix 1 0 : E × E → E × E. Lemma 1.1. The abelian surface F ⊗ C/2πi(OF + (z1 , z1 )OF ), together with its (naturally deﬁned) real multiplication, is isomorphic to (E ⊗ OF , m). Proof. We have an exact sequence j i 0 −→ E − → C2 /2πi OF + (z1 , z1 )OF −→ E −→ 0, ← h

136

ANDREAS LANGER

where the ﬁrst map i is induced by the diagonal embedding C → C2 , the second map j is induced by C2 → C, (x, y) → (x − y)/2, and E is identiﬁed with the elliptic √ √ curve C/2π i( q Z + z1 q Z). The map E → C2 /2πi(OF + (z1 , z1 )OF ) induced by C → C2 , x → (x, −x) deﬁnes a section h of j . Therefore, C2 /OF + (z1 , z1 )OF is isomorphic to E × E as abelian surfaces. It remains to show that under these isomorphisms, the real multiplications are identiﬁed. Let z = (x, x) ∈ C2 /OF + √ √ √ √ (z1 , z1 )OF . Then q ·z = ( q x, − q x) = h( q x), where we use the isomorphism ∼ √ √ √ √ = C/2π i(Z + z1 Z) −→ C/2π i( q Z + z1 q Z), a → q a. We see that q(x, 0) = (0, x) ∈ E × E. On the other hand, when we start with a point x in C/2πi(Z + √ √ z1 Z) and use the above isomorphism, we see that h(x) = ( q x, − q x). Then √ √ √ √ q( q x, − q x) = (qx, qx) = i(qx). Therefore, q(0, x) = (qx, 0) ∈ E × E, as desired. This ﬁnishes the proof of Lemma 1.1.

The point (π −1 z1 , π −1 z1 ) lies in the kernel of the map m(π) : Ez1 ⊗ OF → Ez1 ⊗OF as well as in the kernel of 2p : Ez1 ⊗OF → Ep−1 z1 ⊗OF , since π ·π = p.

Therefore, the abelian surface A corresponding to (π −1 z1 , π −1 z1 ) is isomorphic to Ez1 ⊗ OF /U where U = ker m(π) ∩ ker 2p , which is a subgroup scheme of order p. This yields the modular description of the points on Tp we are looking for. Lemma 1.2. A point on Tp corresponds to an abelian surface A of the form E ⊗ OF /U , where U = ker(m(π ))∩ker 2p ; here 2p : E ⊗OF → E ⊗OF is an isogeny of degree p 2 induced by an isogeny E → E of degree p. The real multiplication on E ⊗ OF /U is the one induced by m on E ⊗ OF . We can also understand the map α : Y0 (p)(C) → Tp from the modular point of view. If x ∈ Y0 (p)(C) correpsonds to an isogeny E → E of degree p, then α(x) corresponds to the abelian surface E ⊗OF /U with the notation given in Lemma 1.2. Let S/Q be the coarse moduli scheme deﬁned over Q that was constructed by Rapoport [Ra]. It represents the moduli stack of polarized abelian surfaces with real multiplication by OF . It is known that S/Q is a quasi-projective variety with S(C) = H × H/ SL2 (OF ). The map α descends to a morphism α : Y0 (p) → S of coarse moduli schemes deﬁned over Q. α extends to a map X0 (p) → S, which maps the two cusps of X0 (p) to the unique cusp of S, which lies in the closure T p of Tp on S, because Tp is not compact. Here S is the Borel-Satake compactiﬁcation, which contains just one cusp because we assume that the class number of F is 1. Since we also deal with Hilbert-Blumenthal surfaces classifying abelian surfaces with real multiplication and level structures, we brieﬂy recall the deﬁnition of the underlying Shimura variety in the adelic language. Consider the reductive group F G = G deﬁned as ﬁbre product FG

/ ResF /Q GL2 /F

Gm

det

/ ResF /Q Gm .

ZERO-CYCLES ON HILBERT-BLUMENTHAL SURFACES

137

We also consider the homomorphism of R-algebraic groups h : ResC/R Gm → F G/R, which is deﬁned on real points by a b a b ∈ G(R) ⊂ GL2 (R) × GL2 (R). , h(a + ib) = −b a −b a a b ∗ Let L∞ = { −b a ∈ GL2 (R)} = ZGL2 (R) · O(2) = C , and let K∞ = L∞ × L∞ ∩ G(R). The theory of canonical models yields a quasi-compact separated scheme ˆ Sˆ = S(G, h)/Q (called the Shimura variety attached to G and h) with a continuous ˆ = G(Q)\G(A)/K∞ . For each compact open Gf = G(Af )-action such that S(C) ˆ such that Sˆ = lim subgroup K of Gf , we have a surface with SK (C) = S(C)/K ← −SK . If K = K0 is the full modular group, we have S(C) := SK0 (C) = H × H/ SL2 (OF ). (One uses the strong approximation theorem and the assumption that the class number of F is 1; compare [P] and [R3, (2.24)–(2.26)].) As before, let S be the Borel-Satake compactiﬁcation. By resolving the cusp and the ﬁnite quotient singularities of S = S K0 , we obtain a projective smooth surface X = S˜K0 over Q, which is our main object of study with respect to torsion zero-cycles and the Tate conjecture in characteristic at a good reduction prime . In order to proceed, we need to consider Hilbert-Blumenthal surfaces with level structures. The reason for this is that they are ﬁne moduli spaces that represent the moduli stack of polarized abelian surfaces with real multiplication by OF and additional level structures. The moduli property is needed to produce extra cycles in characteristic at good reduction primes in order to prove the Tate conjecture in characteristic . For an integer N, we consider the congruence subgroup KN = {g ∈ K0 , g ≡ 1(N)}. Then SKN , the Hilbert modular surface associated to the compact open subgroup KN , is a ﬁne moduli space representing the moduli stack of polarized abelian surfaces A with real multiplication by OF and level N-structures, that is, an isomorphism (1/N )ϑ −1 /ϑ −1 ∼ = A[N], where ϑ −1 is the inverse different that is a projective OF module of rank 1. For further details, compare [Ra, (1.21) and (1.22)]. SKN is, for N ≥ 3, a smooth quasi-projective variety over Q. Let Y = S˜KN be a smooth projective model over Q obtained by taking the Borel-Satake compactiﬁcation and resolving all the cusps. Y has good reduction at all primes p Nq. We have a Galois covering SKN → S with Galois group H = K0 /KN . Now we mimic the construction of Hirzebruch-Zagier cycles for the surface SKN , as at the beginning of the section. For p N, let Y0 (p)∩(N) be the ﬁne moduli space representing the moduli stack of isogenies 6p : (E, E[N ]) → (E , E (N)) of degree p between elliptic curves E and E that induce an isomorphism of level N-structures E[N] ∼ = E [N] [DR, Chapter V, 1.17]. We deﬁne a morphism of ﬁne moduli schemes σ : Y0 (p)∩(N) −→ SKN

138

ANDREAS LANGER

with scheme-theoretic image denoted by Tp[N] , which is given on the moduli stacks as follows. If 6p : (E, E[N]) → (E , E [N]) is an object in Y0 (p)∩(N) , then its image in SKN is (A = E × E/ ker(6p × 6p ) ∩ ker π, A[N]), where the real multiplication on A is deﬁned as in Lemma 1.2, and the level N-structure A[N ] is induced by E[N]×E[N]. The morphism σ extends to a morphism of ﬁne moduli schemes over Zp , 7 : ᐅ0 (p)∩(N) −→ ᏿KN /Zp , in the evident way. We observe that if 6p : Ᏹ → Ᏹ is an isogeny of elliptic curves over a Zp -scheme S of degree p, then the ﬁnite group scheme ker(6p × 6p ) ∩ ker π is a direct summand of the ﬂat group scheme ker 6p × 6p = ker 6p × 6p ∩ ker π ⊕ ker 6p × 6p ∩ ker π , and therefore also ﬂat. Then the projective group scheme Ꮽ = Ᏹ ×S Ᏹ/ ker(6p × 6p ) ∩ ker π is again an abelian surface over S. This is because we have isogenies Ᏹ ×S Ᏹ → Ꮽ and Ꮽ → Ᏹ ×S Ᏹ of degree p, which force Ꮽ to be an abelian scheme. (N)

Lemma 1.3. The scheme-theoretic image ᐀p ᏿KN /Zp .

(N)

of 7 is the closure of Tp

in

Proof. From [DR] we know that ᐅ0 (p)∩(N) ⊗Zp Fp has two components (N)

and (N) , where the isogenies of elliptic curves 6p : E → E of degree p corres ponding to points on (N) are Frobenius F : E → E (p) , and on (N) are Verschiebung V : E → E 1/p such that F ◦ V = p, multiplication by p on E. If x is a point on ᏿KN ⊗Zp Fp that lies in the image of 7 ⊗Fp , then it corresponds to the abelian surface E ×E/ ker F ∩ker π or to E ×E/ ker V ∩ker π (with induced real multiplication and level N -structure), where E is an elliptic curve over Fp . It is evident from the modular (N) property that x is the specialization of some point on Tp . Lemma 1.4. If E is an elliptic curve over Fp and F : E ×E → E (p) ×E (p) is the Frobenius, we have an isomorphism ∼ =

E (p) × E (p) / ker V ∩ ker π −→ E × E/ ker F ∩ ker π . Proof. We have ker F = (ker F ∩ ker π) ⊕ (ker F ∩ ker π ). Therefore, F induces an isomorphism ∼ = E × E/ ker π ⊕ ker F ∩ ker π −→ E (p) × E (p) / ker V ∩ ker π, because F (ker π ) = ker V ∩ ker π. But multiplication by π induces an isomorphism (note that ker π + ker F = ker π ⊕ (ker F ∩ ker π )) ∼ =

E × E/ ker π + ker F −→ E × E/ ker F ∩ ker π . π

The lemma follows.

ZERO-CYCLES ON HILBERT-BLUMENTHAL SURFACES

139

(N)

Lemmas 1.3 and 1.4 show that ᐀p ⊗Zp Fp has two components deﬁned as

the images of (N) and (N) under the map 7, and we may call them F℘ and (N) (N) (N) F℘ . If Sp = ᏿KN ⊗Zp Fp , and i : Sp → Sp denotes the involution given by i(A, A[N], m) = (A, A[N], m) where m = m ◦ σ, σ ∈ Gal(F /Q), σ = id, then Lemma 1.4 implies i(F℘ ) = F℘ . The Galois covering SKN → S extends to a Galois covering of the models ᏿KN −→ ᏿

over Zp ,

which induces a ﬁnite ﬂat covering of curves ᐀p(N) −→ ᐀p

over Zp ,

where ᐀p is the closure of the Hirzebruch-Zagier cycle Tp on ᏿. Since the composite map ᐅ0 (p)∩(N) −→ ᏿KN −→ ᏿ forgets the level structure, it factors through ᐅ0 (p) −→ ᐀p ⊂ ᏿Zp ,

where ᐅ0 (p) is the regular model of Y0 (p) over Zp as a coarse modular scheme [DR]. Denote by F and V the images of the two components of ᐅ0 (p) ⊗Zp Fp under this map, so ᐀p mod p = F + V . Let X = S˜K0 be as above, a smooth projective variety over Q. For certain X, we are going to prove the Tate conjecture for divisors in characteristic p at split primes p, that is, on the closed ﬁber Xp of a smooth proper model ᐄ of X over Zp . It is clear that there exists an open scheme U ⊂ Spec Z such that X has good reduction at p ∈ U . The conjecture of Tate in characteristic p has the following shape (here NS(Xp ) denotes the Neron-Severi group of Xp , and is a prime = p). Conjecture (Tate). We have the following isomorphisms: F r F r NS Xp ⊗ Q ∼ = Het2 X, Q (1) p ∼ = Het2 Xp , Q (1) p . The isomorphism on the right-hand side is clear from the smooth and proper base change theorem. We recall (compare [vdG, Chapter XI, Section 2] and [HLR, Proposition 5.3]) that Het2 (X, Q ) decomposes as Gal(Q/Q)-modules Het2 X, Q ∼ (1.5) = H2 S, Q ⊕ Q (−1)s , where Q (−1)s = HX2 ∞ (X, Q ) corresponds to the cycle classes (twisted by −1) of the rational curves in the resolution of the cusp or quotient singularities, the union of which is denoted by X∞ , and H2 S, Q := Im Hc2 S × Q, Q −→ H 2 S × Q, Q Q

Q

140

ANDREAS LANGER

is the image of the compactly supported cohomology that coincides with intersection cohomology (compare [HLR, 1.8] and the references given there). After extending the coefﬁcients to Q , we have a semisimple action of the Hecke algebra ᏴK0 on H2 (S, Q ) that commutes with the Galois action of GQ = Gal(Q/Q). Let p ) ⊗ Q denote the projection of NS(Xp ) ⊗ Q into H2 (S, Q (1)) given by NS(X the decomposition (1.5). Then the Tate conjecture in characteristic p is equivalent to the assertion that the inclusion Xp ⊗ Q ;→ H2 S, Q (1) F rp NS (1.6) is an isomorphism. For the convenience of the reader, we recall the deﬁnition of Hilbert modular forms. A holomorphic function f (z) on H × H is called a Hilbert modular form of weight k if it satisﬁes −k −k f γ (z) = cz1 + d c z2 + d f (z) for any γ = ac db ∈ SL2 (OF ) and z = (z1 , z2 ) ∈ H × H. Any Hilbert modular form has a Fourier expansion at each cusp of H × H/ SL2 (OF ). Since the class number of F is 1, by our assumption, there is only one cusp and f (z) is called a cusp form, if and only if the constant term a((0)) of its Fourier expansion at the cusp

f (z) = a((ν)) exp 2πi νωz1 + ν ω z2 ν∈OF + ∪{0}

is zero. Here ω is a totally positive generator of δF−1 (the inverse different), and OF + is the set of totally positive integers in F . The semisimple ᏴK0 -module H2 (S, Q ) decomposes into isotypic components as follows: (1.7)

H2 S, Q ∼ =

⊕

g∈{g1 ,...,gr }

Wg ⊗ Q ⊕ W 0 ⊗ Q . Q

Kˆ g

Here {g1 , . . . , gr } is a basis of Hilbert modular cusp forms of weight 2 for SL2 (OF ) that are normalized eigenforms under the Hecke algebra. Kg is the number ﬁeld obtained by adjoining the Fourier coefﬁcients of g, Kˆ g ⊂ Q is the completion of Kg at a prime above , Wg is a 4-dimensional Kˆ g -vector space on which Gal(Q/Q) acts, and W0 (1) is a 2-dimensional Q -vector space generated by the étale cycle classes of two line bundles L1 and L2 , both deﬁned over F , with Chern classes dzr ∧ dzr /yr2 , xr + iyr = zr , r = 1, 2, and such that L1 ⊗ L2 is deﬁned over Q (compare [HLR, 4.6] and [O]). The sections of L1 and L2 are, when considered over C, holomorphic modular forms for SL2 (OF ). The direct sum H2cusp S, Q := Wg ⊗ Q ⊕ g∈{g1 ,...,gr }

Kˆ g

ZERO-CYCLES ON HILBERT-BLUMENTHAL SURFACES

141

is called the cuspidal cohomology of S. In the classical language, the decomposition of the cuspidal cohomology in isotypic components is given in terms of irreducible cuspidal automorphic representations of G(A), whose archimedean components lie in the discrete series of weight 2 (see [vdG, Theorem 2.1]). However, since we almost exclusively work with the HilbertBlumenthal surface associated to the full modular group, it is more convenient to work with Hilbert modular cusp forms. For the correspondence of Hilbert modular forms and automorphic representations, we refer to [G, Section 3.10]. A basic principle behind the decomposition (1.7) is the multiplicity one theorem [O, Section 2.2] and [Miy], which says that two normalized Hecke eigenforms in S2 (SL2 (OF )) that have the same eigenvalues for all Hecke operators coincide. We recall the DoiNaganuma lift [vdG, Chapter VI, Section 4], DN F : S2 0 (q), εq −→ S2 SL2 (OF ) , (1.8) f −→ fˆ, which associates to a modular form f of level q and Nebentypus εq (the quadratic character of F /Q) a Hilbert modular cusp form g = fˆ. DN F factors through an injection DN F : S2+ 0 (q), εq ;→ S2 SL2 (OF ) , where S2+ (0 (q), εq ) = {f ∈ S2 (0 (q), εq ), aN (f ) = 0 if (N/q) = −1} (see [vdG, Proposition 4.3]). In terms of automorphic representations, DN F corresponds to the 0 of GL (A ) from Q to F . lift π of a cuspidal automorphic representation πQ 2 Q We also recall the following fact due to Harder, Langlands, and Rapoport [HLR] (compare with [vdG, Theorem 3.13]). If g = fˆ, denote by H˜ f the 2-dimensional Kf -vectorspace in H 1 (X1 (q), Kf ) associated to f that was constructed by Eichler and Shimura. Let Kˆ f be the completion of a prime above such that Kˆ f ⊂ Q , and let Hf = H˜ f ⊗Kf Kˆ f together with its Gal(Q/Q)-action. Then we have a (noncanonical) isomorphism (1.9)

Wfˆ (1) ∼ = End(Hf ) of GF -modules.

The right-hand side decomposes further, (1.10)

End(Hf ) ∼ = Sym2 (Hf )(1) ⊕ Kˆ f ,

where Kˆ f has trivial Galois action and corresponds to the cycle class of a HirzebruchZagier cycle. In the following theorem, we prove the Tate conjecture in characteristic p at split primes p for a special class of Hilbert-Blumenthal surfaces. Theorem 1.11 (Theorem A). Let p be a prime that splits in F and such that X has good reduction at p. Let Xp be the closed ﬁber of a smooth, proper model ᐄ/Zp

142

ANDREAS LANGER

of X. Assume that DN F is surjective. Then the Tate conjecture holds for divisors on Xp . Remark. Examples of such Hilbert-Blumenthal surfaces can be found in Oda’s book [O, p. 109]. Proof. In the ﬁrst part of the proof, we allow p to be an arbitrary, good reduction prime until we have computed the dimension of local Tate classes. Let NS(Xp )c ⊗Q denote the projection of NS(Xp ) ⊗ Q to H2cusp (S, Q (1)). Since, according to the decompositions (1.5) and (1.7), H2cusp (S, Q (1)) has a complement in Het2 (X, Q (1)), that is, over Q , generated by algebraic cycle classes that certainly specialize to Xp , it sufﬁces to show that F r c NS Xp ⊗ Q ∼ = H2cusp S, Q (1) p . According to (1.7), we can decompose NS(Xp )c ⊗ Q as c ⊕ NS Xp ⊗ Q ∼ NS Xp (f ) ⊗ Q , = f ∈{f1 ,...,fr }

Kˆ f

where {f1 , . . . , fr } is the basis of Hecke eigenforms in S2+ (0 (q), εq ) and NS(Xp )(f ) is contained in Wfˆ (1). Here we use the fact that under the action of the smaller p p p Hecke algebra Ᏼ(G(Af ), K0 ) ⊆ ᏴK0 , where G(Af ) are the adelic points with pp p component 1, and K0 = G(Af ) ∩ K0 , the decomposition (1.7) remains the same, p p that is, the isotypic components Wg are already determined by Ᏼ(G(Af ), K0 ). This is a consequence of the strong multiplicity one theorem for GL2 (see [Miy] and, for the precise argument, [BL, remarks after Lemma 3.4.1]). This is needed because on p p NS(Xp )c ⊗Q , we only have a well-deﬁned action of Ᏼ(G(Af ), K0 ), so NS(Xp )(f ) p p is an isotypic component under the action of Ᏼ(G(Af , K0 )) and really consists of cycle classes. To prove the Tate conjecture, it sufﬁces to show that (1.12) NS Xp (f ) ∼ = Wfˆ (1)F rp . To the GQ -module Wfˆ , we can associate an L-function in the usual way. Its Euler factor at p is given by I −1 Lp Wfˆ , s = det 1 − p −s F rp | W ˆp , f

where F rp denotes the geometric Frobenius and Ip denotes the inertia group at p. The L-function is a factor of the Hasse-Weil L-function L(H 2 (X), s) and coincides, up to the Euler factor at q, with the L-series G(fˆ, s) = ξ(2s − 2)

∞ m=1

c((m))m−s

ZERO-CYCLES ON HILBERT-BLUMENTHAL SURFACES

143

studied by Asai. Here the coefﬁcient c(a) is, for an integral ideal a ⊂ OF , the Fourier coefﬁcient of fˆ (compare [vdG, p. 140]). The coincidence of these L-functions was shown by Brylinski and Labesse [BL] in a more general context (see the remark below). We can also associate to fˆ the L-function −1 , 1 − c(℘)N (℘)−s + N(℘)1−2s L(fˆ, s) = ℘⊂OF

where ℘ runs through all prime ideal in OF . If we factor X2 − c(℘)X + N(℘) = (X − α℘ )(X − β℘ ), then α℘ , β℘ are the eigenvalues of Frob℘ acting on Hf , and the Euler product of G(fˆ, s) at p is given by the inverse of (1 − Ap p −s ), where  α ℘ α℘       α℘ β℘ 0         0 α ℘ β℘      0 Ap =     0 α℘       0 α℘           β 0 ℘    0 β℘

0

    if (p) = ℘ · ℘ splits, 

β℘ β℘

if (p) = ℘ is inert.

All this can be found in [vdG, p. 140]. Remark. If π is a cuspidal automorphic representation of G(A), we deﬁne for π an automorphic L-function L(s, π, r) where r is a certain representation of the underlying L-group L G (compare [HLR, paragraph 2]). If π corresponds to a Hilbert modular cusp form g ∈ S2 (SL2 (OF )), which is a common eigenform for all Hecke operators (not necessarily a lift), then L(s − 1, π, r) coincides, up to ﬁnitely many Euler factors, with the L-function G(g, s) studied by Asai (the deﬁnition is the same as for a lift fˆ). Its Euler factor at an unramiﬁed prime p is again given by the inverse of (1−Ap p −s ), where the matrix Ap is deﬁned as above, with α℘ , β℘ being the zeros of the polynomial X 2 −c(℘)X +N(℘). Then Brylinski and Labesse [BL] and Harder, Langlands, and Rapoport [HLR, (2.4)] showed, by comparing the Lefschetz ﬁxedpoint formula with the Selberg trace formula, that the above Euler factor coincides with the Euler factor of the L-function associated to the GQ -representation Wg . If g is not a lift, then it follows from [HLR] that there are no global Tate classes in Wg (1) deﬁned over Q. However, it is possible that there are local Tate classes; that is, it may happen that some diagonal entries in Ap are equal to p. It seems to be very difﬁcult to control these local Tate classes. If they occur, then the construction of new cycles in characteristic p that we perform is probably not sufﬁcient to cover these Tate classes.

144

ANDREAS LANGER

This is the reason why we restrict ourselves to the case where all Hilbert modular cusp forms are lifts of 1-variable modular forms. From the deﬁnition of α℘ , β℘ , we get the following lemma. Lemma 1.13. Let g = fˆ as above. Then we have α ℘ · α℘ · β℘ · β℘ = p 2

if p is split,

α℘ · β℘ = p 2

if p is inert.

Now we are able to compute the dimension of the space of local Tate classes. Lemma 1.14. We have

dim Wfˆ (1)

F rP

=

3 2

p inert, p split.

Proof. We assume that fˆ is a lift of a modular form f ∈ S2+ (0 (q), εq ). For the Fourier coefﬁcient c(℘) of a lift fˆ, we have the following formulas: c(℘) = ap (f ) 2 c(℘) = ap (f ) + 2p

if p is split, if p is inert.

(See [vdG, (4.1) and (4.2)].) If p is inert, then ap (f ) = 0 by our assumption on f , so α℘ , β℘ are the zeros of the polynomial X 2 − 2p + p2 . So they are both equal to p, which proves Lemma 1.14 in the inert case. Now assume that (p) = ℘ · ℘ is split. Since there is a global cycle class, one of the diagonal entries in Ap must be p, let us say α℘ · α℘ = p. From Lemma 1.13, we get β℘ ◦ β℘ = p, so dim Wfˆ (1)F rp ≥ 2. If dim Wfˆ (1)F rp > 2, then the eigenvalues of F rp acting on Hf would be the same. Since F rp acts semisimply on Hf , F rp|Hf would be a homothety. By a recent result of Coleman and Edixhoven [CE, Theorem 2.1], this is impossible. From now on, let p be a split prime. Let Tpc be the projection of the cycle class of Tp on NS(Xp )c ⊗ Q . We have a decomposition (1.15)

Tpc =

⊕

f ∈{f1 ,...,fr }

f

Tp ,

f

Tp ∈ Wfˆ (1).

Since Tp = F + V , we have Tpc = F c + V c , and we also get decompositions Fc =

⊕

f ∈{f1 ,...,fr }

Ff,

Vc =

⊕

f ∈{f1 ,...,fr }

Vf

for the cuspidal parts of the cycle classes of the two components F and V of Tp mod p. To show that (1.12) implies Theorem 1.11, we have to show that for

145

ZERO-CYCLES ON HILBERT-BLUMENTHAL SURFACES

each f ∈ {f1 , . . . , fr }, the f -components F f and V f are linearly independent in Wfˆ (1) ∼ = End(Hf ). (Note that this is an isomorphism of Gal(Qp /Qp )-modules because p is a split prime.) Since we have to work with the modular description of the Hirzebruch-Zagier cycles, we pass to the ﬁne moduli space SKN where N ≥ 3 is π an integer with p N. Recall the Galois coverings π : SKN → S and ᏿KN → ᏿ of (N) their models over Zp . We have π(᐀p ) = ᐀p , π(F℘ ) = F , π(F℘ ) = V . Let H2 (S KN , Q ) = Im(Hc2 (SKN , Q ) → H 2 (S KN , Q )), which coincides with intersection cohomology by [HLR, (1.8)]. The Hecke algebra ᏴKN acts on H2 (S KN , Q ), and again we get a decomposition into isotypic components

Wg ⊗ Q ⊗ πgKN ⊕ ᐂ1 ⊕ ᐂ0 , (1.16) ⊕ H2 S KN , Q = g∈{g1 ,...,gr }

Kˆ g

Q

where Wg is deﬁned as in (1.7) and πgKN is the module under the Hecke algebra ᏴKN consisting of KN -invariants in the space of automorphic forms of the cuspidal automorphic representation that is associated to g. Here g runs through a basis {g1 , . . . , gr } of normalized Hecke eigenforms in S2 (SL2 (OF )). πgKN can be realized over Q. ᐂ1 is a direct sum of isotypic components that are associated to Hilbert modular cusp forms that are not modular forms for the full Hilbert modular group SL2 (OF ). ᐂ0 is a Q -vectorspace such that ᐂ0 (1) is generated by the cohomology classes of the line bundles L1 and L2 restricted to the connected components of SKN (see [vdG, (2.7)]). (N) (N) Let c Tp be the projection (the cuspidal component) of the cycle class of Tp KN on ⊕g∈{g1 ,...,gr } [Wg ⊗Kˆ g Q ] ⊗Q πg ⊕ ᐂ1 . Since all g are lifts of modular forms f ∈ S2+ (0 (q), εq ), we may write

(1.17)

c

Tp(N) =

⊕

f ∈{f1 ,...,fr }

Tp(N) (f ) ⊕ T˜p(N) (N)

according to the decomposition of the cuspidal part of H2 (S KN , Q (1)). So Tp (f ) (N) lies in [Wfˆ (1)⊗Kˆ g Q ]⊗Q π Kˆ N , and T˜p lies in ᐂ1 (1). Similarly, we have decom f positions f c c F℘ = ⊕F℘f + F˜℘ ; F℘ = ⊕F℘ + F˜℘ f

f

such that f

F℘f + F℘ = Tp(N) (f ), F˜℘ + F˜℘ = T˜p(N) . It is clear that under the homomorphism π∗ : H2 (S KN , Q (1)) → H2 (S, Q (1)), we f f f (N) have π∗ (Tp (f )) = mTp , π∗ (F℘ ) = mF f , π∗ (F℘ ) = mV f for some m ∈ N (m is (N)

the degree of the covering Tp

→ Tp ).

146

ANDREAS LANGER

We normalize the isomorphism Wfˆ (1) ∼ = End(Hf ) of Gal(Qp /Qp )-modules in f such a way that T1 = id. In the Hecke algebra ᏴK (resp., ᏴKN ), we have a Hecke operator tq for each prime ideal q in OF , satisfying tq(g) = aq(g) · g for each Hilbert modular eigenform g with Fourier coefﬁcient aq(g). If g = fˆ is a lift and p = ℘ · ℘ is split, we have t℘ (fˆ) = a℘ (fˆ)· fˆ = ap (f )· fˆ. Then t℘ also acts on the cohomology H2 (S, Q ) and we have t℘ (T1c ) = Tpc (compare [vdG, (1.5)] and the construction of Tp at the beginning of this section). The class of the Hirzebruch-Zagier cycle Tp is obtained by applying the Hecke operator t℘ to the diagonal curve T1 = 2. Since the action of t℘ on the cohomology part associated to g = fˆ is the same as on the modular form, we get f f Tp = t℘ T1 = ap · id in End(Hf ), (1.18) where ap = ap (f ) is the Fourier coefﬁcient of f . (N) Let Sp be the closed ﬁber of ᏿KN /Zp . (N) We deﬁne morphisms Ᏺ℘ , Ᏺ℘ on Sp as follows: Ᏺ℘ : Sp(N) −→ Sp(N) ,

A −→ A/ ker π ∩ ker Frob; Ᏺ℘ : Sp(N) −→ Sp(N) ,

A −→ A/ ker π ∩ ker Frob . (We omit here the additional data of real multiplications and level N-structures.) As morphisms of ﬁne moduli schemes (and also the underlying moduli stack), we have Ᏺ℘ ◦ Ᏺ℘ = Frob . (N)

If 2(N) denotes the diagonal curve on Sp , we have Ᏺ℘∗ 2(N) = F℘ , Ᏺ℘∗ 2(N) = F℘ and Frob∗ 2(N) = p · 2(N) . It is clear that Ᏺ℘ and Ᏺ℘ descend to morphisms of the coarse moduli scheme Sp and the closed ﬁber of the model ᏿/Zp of S, and the same formulas hold for the diagonal curve T1 = 2 in Sp (we replace F℘ and F℘ with F and V ). Furthermore, Ᏺ℘ and Ᏺ℘ act on the intersection cohomology H2 (S, Q ), which coincides with the intersection cohomology of S p (compare [BL, p. 408]). Since this action commutes with the action of the Hecke algebra, Ᏺ℘ and Ᏺ℘ induce homomorphisms of Kˆ f -vectorspaces f

f Ᏺ℘ , Ᏺ℘ : End(Hf ) −→ End(Hf ).

Assume that F f = (α · id), V f = (α · id) are homotheties in End(Hf ). (If not, we

ZERO-CYCLES ON HILBERT-BLUMENTHAL SURFACES

147

have found a new cycle class in End(Hf ). Since F f +V f = ap ·id, F f and V f must be linearly independent in End(Hf ), and we are done). Then we have the following identity in End(Hf ): f f f f f f f ◦ Ᏺ℘ T1 = Ᏺ℘ α · id = α Ᏺ℘ T1 = α · α · id . p · id = Frob T1 = Ᏺ℘ We get the following equations: f (i) F f + V f = Tp , that is, α + α = ap , (ii) α · α = p; thus, α, α are the two roots of X 2 − ap x + p, so they coincide with the eigenvalues of F rp acting on Hf . Applying again the result of Coleman and Edixhoven [CE, Theorem 2.1] we see that α = α . On the other hand, we know that the involution ι : Sp → Sp that is induced from the involution on the underlying moduli stack by permuting the real multiplication leaves the diagonal curve T1 = 2 invariant and permutes F and V . We can compute the trace of F f and V f by taking the intersection number with the diagonal 1 f T r F f = F f , 2f = V f , 2 = T r Tp = ap . 2 So ap = 2α = 2α , which is a contradiction. This proves the isomorphism (1.12) and ﬁnishes the proof of Theorem 1.11. 2. Construction of new elements in K-theory and torsion zero-cycles. In this section we prove, by assuming certain conjectures on values of L-functions, a ﬁniteness result on the torsion subgroup of the Chow group of zero-cycles on X. The expository paper [L4] spells out a general axiomatic and conjectural framework that relates the ﬁniteness of torsion algebraic cycles and the methods developed in [LS] to Beilinson’s conjecture, the Tate conjecture, and the Bloch-Kato conjecture, and can be applied to any smooth projective variety over a number ﬁeld. Some of the axioms or claims can be found in the conditional Theorem 2.5. Since the geometry of Hilbert-Blumenthal surfaces and their cohomological properties are rather special, the axioms can, in part, be formulated more explicitly (e.g., the treatment of bad primes) or they can even be proven (e.g., the ﬁniteness of the Selmer group of the motive H 2 (X)(2), which is related to the Bloch-Kato conjecture). Therefore, we have decided to give here a coherent presentation that is appropriate to the particular situation and independent of the more general approach treated in [L4]. Let U ⊂ Spec Z be an open subscheme such that there exists a smooth proper model ᐄ of X over U with the following property: If ᏿ is the coarse moduli scheme that is a quasi-projective model of S over U , then there exists a morphism ᐄ → ᏿ of schemes over U , where ᏿ is the compactiﬁcation of ᏿ constructed in [Ra], which extends the morphism X → S obtained by resolving the cusp and quotient singularities on S. The existence of U and ᐄ/U is clear. We believe that the only primes that have to

148

ANDREAS LANGER

be inverted to get U are the primes dividing the discriminant of F and the orders of the cyclic quotient singularities on S (in our situation, 2 or 3). However, we cannot prove that, and this is the reason why we work with the unspeciﬁed set U . For a regular scheme Z, let H i (Z, ᏷j ) be the Zariski cohomology of the Quillen algebraic K-sheaf ᏷j associated to the presheaf V → Kj (V ), let Chj (Z) be the Chow group of cycles of codimension i modulo rational equivalence, and let Pic(Z) = Ch1 (Z) be the Picard group. Note that Ch2 (X) = Ch0 (X), the Chow group of zerocycles on X, and let Ch0 (X){p} be its p-primary torsion subgroup for a prime p. From the works of Bloch [Bl1] and Sherman, we have the following exact localization sequence in algebraic K-theory, using Gersten’s conjecture for ᏷2 in a mixed characteristic setting that was proven by Bloch [Bl2]: (2.1)

∂ H 1 ᐄ, ᏷2 −→ H 1 X, ᏷2 −→ ⊕ Pic(X ) −→ Ch2 (ᐄ) −→ Ch2 (X) −→ 0, ∈U

where X ;→ ᐄ is the closed ﬁber at the good reduction prime . It is expected that the cokernel of ∂ is a torsion group. This requires the construction of new elements in K-theory. In this direction, we get the following partial result. Theorem 2.2 (Theorem B). Let X be the Hilbert-Blumenthal surface deﬁned as before, and let ᐄ be a smooth, proper model over U as above. Furthermore, assume that the Doi-Naganuma lift DN F is surjective. Then the boundary map in (2.1), when restricted to all split primes, that is, ∂ H 1 X, ᏷2 ⊗ Q −→

⊕

∈U splits in F

Pic(X ) ⊗ Q,

is surjective. Proof. In order to prove Theorem 2.2, it sufﬁces to construct, for each split prime p ∈ U , nondecomposable elements in H 1 (X, ᏷2 ), which together with the image of Pic(X) ⊗ p Z in H 1 (X, ᏷2 ), generate a subspace Up in H 1 (X, ᏷2 ) ⊗ Q such that ∂p (Up ) = Pic Xp ⊗ Q, ∂ (Up ) = 0 for = p. This can be achieved by using the modular units Flach and Mildenhall constructed as modular units on Hirzebruch-Zagier cycles and taking the translates of the so-obtained elements in K-theory under Hecke operators. The a priori construction only works on the singular variety S, not on X, because the Hecke operators are not deﬁned on the resolutions of singularities. In fact, in order to obtain new elements in H 1 (X, ᏷2 ), we use a construction that was ﬁrst discovered by Ramakrishnan [R1], [R2], and [R3]. He generalized Beilinson’s construction for the self-product of a modular curve to Hilbert-Blumenthal surfaces (compare [Sch]). Note that the Brown-Gersten spectral sequence on ᏿, E1r,s (᏿) = x∈᏿r K−r−s k(x)

⇒ K−r−s (᏿), and the analogous sequence for S and Sp yield, by functorial properties,

ZERO-CYCLES ON HILBERT-BLUMENTHAL SURFACES

149

a boundary map ∂p : E21,−2 (S) −→ Ch1 (Sp ). Now let u2 be the modular unit on X0 (p) constructed by Flach and Mildenhall from the unique cusp form 2 of weight 12 for SL2 (Z), that is, u2 (τ ) = p −6 (2(pτ )/2(τ )). The map X0 (p) → T p ⊂ S is birational and maps the two cusps p1 , p2 of X0 (p) to the unique cusp of S (because the class number of F is 1). Since ordp1 (u2 ) = − ordp2 (u2 ), and u2 has no other poles or zeros, the pair (Tp , u2 ) (resp., (T p , u2 )) deﬁnes an element of E21,−2 (S) (resp., E21,−2 (S)). Then we have the following lemma. Lemma 2.3. ∂p ((Tp , u2 )) = κ(p)(F − V ) for some constant κ(p) where F and V are the two components of ᐀p mod p. Proof. The proof is the same as for the Hecke correspondences on X0 (N)×X0 (N) in [Fl1] and [Fl2]. Now assume that F f (and also V f ) is a cycle class with coefﬁcient in Q, that is, ∈ Pic(Xp )c ⊗ Q. If this happens, then ap (f ) ∈ Q. Then there exists an element Q (the Hecke algebra generated by the Hecke operators Tg for g ∈ G(Af ) 7αg Tg ∈ ᏴK 0 over Q!) such that f 7αg Tg Tpc = Tp , 7αg Tg (F c ) = F f .

Ff

Q -module H2 (S, Q (1)) This is clear from the decomposition of the semisimple ᏴK 0 in isotypic components. Let L = gKg −1 ∩ K0 . Then Tg is deﬁned via the diagram

RL/K0 (1)

S

SL A AA RL/K (g) AA 0 AA S,

where both morphisms are ﬁnite and ﬂat, and RL/K0 (g) is induced from the right p multiplication by g on the complex points. Let G(Af ) be the set of adelic points p of G where the p-component is trivial. Then if g ∈ G(Af ), the Hecke operator Tg is deﬁned on the models over Zp . Since the Brown-Gersten spectral sequence is covariant for proper and contravariant for ﬂat morphisms, Tg deﬁnes a morphism Tg# : E21,−2 (S) −→ E21,−2 (S), p

which, for g ∈ G(Af ), is compatible with ∂p : E21,−2 (S) −→ Ch1 (Sp ). Since the isotypic component Wfˆ (1) in H2cusp (S, Q (1)) is already determined by p p the smaller Hecke algebra Ᏼ(G(Af ), K0 ) (compare the proof of Theorem 1.11), we

150

ANDREAS LANGER

may assume that all g for which αg = 0 in the above projector p g ∈ G(Af ), and we get

g αg Tg

satisfy

∂ 7αg Tg# Tp , u2 = 7αg Tg# κ(p)(F − V ). We know that the Brown-Gersten spectral sequence is covariant for proper and contravariant for ﬂat morphisms of noetherian schemes (see [Q, Section 5]). We apply this to the proper morphism ᐄZp → ᏿Zp and the open immersion ᏿Zp → ᏿Zp , and get maps H 1 (X, ᏷2 ) → E21,−2 (S), Pic(Xp ) → Ch1 (Sp ). Now a comparison of the Brown-Gersten resolutions of ᏷2 /ᐄZp , ᏷2 /X, and ᏷1 /Xp (where they are exact) and of ᏷2 /᏿Zp , ᏷2 /S, and ᏷1 /Sp (where they are not exact) yields the following commutative diagram: E21,−2 (S) O

∂p

/ Ch1 (Sp ) O

∂p

/ Pic Xp .

(∗) H 1 X, ᏷2

The exact argument for the commutativity is a tedious diagram chase in a very large diagram and is therefore left out. We lift (Tp , u2 ) to an element in H 1 (X, ᏷2 ) as follows. Consider (T p , u2 ) in E21,−2 (S) and let T˜p be the strict transform of T p on X. Then u2 lifts to a rational function on T˜p with possible zeros or poles at the intersection points of T˜p with the resolution cycles of the cusp or quotient singularities. Since the resolutions of the singularities consistof polygons or strings of rational curves, it is possible to ﬁnd a ﬁnite formal sum i (Ci , fi ), where Ci is a rational curve in the resolutions and fi ∈ k(Ci ), such that div(u2 ) + i div(fi ) = 0 on X, that is, (T˜p , u2 ) + i (Ci , fi ) deﬁnes an element in H 1 (X, ᏷2 ). This construction is due to Ramakrishnan [Ra1, Section 12.10] and can be applied to 7αg Tg# (Tp , u2 ) as well, which is a formal sum of pairs consisting of a modular curve and a modular unit (the translates of (Tp , u2 ) under Tg ). There is an element z ∈ H 1 (X, ᏷2 ) such that the restriction of z to S is (7αg Tg# )(Tp , u2 ). The image of z under the composite map H 1 X, ᏷2 −→ E21,−2 (S) −→ Ch1 (Sp ) is κ(p)( g αg Tg# )(F − V ). The diagram (∗) shows that the image of z under the composite map c H 1 X, ᏷2 −→ Pic(Xp ) −→ Pic Xp ⊗ Q is

κ(p) 7αg Tg# F c − V c = κ(p) F f − V f .

ZERO-CYCLES ON HILBERT-BLUMENTHAL SURFACES

151

We have seen that the new elements in H 1 (X, ᏷2 ) are lifts of (7αg Tg# ) (Tp , u2 ) ∈ E21,−2 (S), αg ∈ Q obtained by adding a certain formal sum i (Ci , fi ) supported in the resolutions of cusp and quotient singularities. They have the desired property, that is, they kill the new cycles in characteristic p. This construction and the fact that all the other elements in Pic(Xp ) arise from specializing cycles on X imply that the boundary map ∂ ⊕ Pic(X ) ⊗ Q H 1 X, ᏷2 ⊗ Q −→ ∈U splits in F

is surjective, and Theorem 2.2 follows. As an immediate application, we can prove the following local ﬁniteness result. Corollary 2.4 (Theorem C). Let p be a split, good reduction prime of X in U , and let XQp = X ⊗Q Qp . Then Ch0 (XQp )tor , the prime-to-p torsion in Ch0 (XQp ), is ﬁnite. Proof. By the argument of Raskind in [Ras, Theorem 1.9], the reduction map Ch2 ᐄZp {} −→ Ch2 XFp {} is injective for all = p. Then the exact sequence (2.1), Theorem 2.2, and the fact that Ch2 (XFp )tor s is ﬁnite [CTSS, Theorem 1] imply the corollary. In order to proceed further globally, we need to assume certain conjectures that are standard in the context of values of L-functions. Let S be a ﬁnite set of primes including all bad reduction primes of X, and let 3 Hᏹ ᐄZ[1/S] , Q(2) := Im H 1 ᐄZ[1/S] , ᏷2 ⊗ Q −→ H 1 (X, ᏷2 ) ⊗ Q . Let

I −1 L H 2 (X), s := det 1 − F rp p −s /H 2 X, Q p = Pp (X, s)−1 p

p

be the Hasse-Weil L-function of the motive H 2 (X). Let LS (X, s) := p∈S Pp(X, s)−1 . Then an S-integral version of Beilinson’s conjecture [R4, Conjecture 6.8.7] predicts the following equality: 3 ords=1 LS (X, s) = dim Hᏹ (H1) ᐄZ[1/S] , Q(2) . If p is an inert good reduction prime, we consider Tate’s conjecture (H2) rk NS Xp = − ords=1 Pp (X, s)−1 , where Xp is the closed ﬁbre of ᐄ at p. Our methods developed in [LS] to study torsion zero-cycles over an algebraic number ﬁeld also force us to look at the local p-adic regulator map at primes not lying

152

ANDREAS LANGER

in U and to establish a surjectivity property there. For this, we need an additional assumption, the necessity of which becomes clear in the proof. Let p be an inert prime not lying in U . As explained at the beginning of this section, we certainly expect p = 2 or p = 3, if 2 or 3 are inert in F , to be such a prime. We take an integer N ≥ 3 such that p N, and we consider the following hypothesis for the Hilbert-Blumenthal surface SKN and its smooth projective model Y = S KN : ∂p

(H3)

(i) The boundary map H 1 (SKN , ᏷2 ) ⊗ Q −−→ Pic(Sp(N) ) ⊗ Q is surjective, (ii) the Tate conjecture holds for Y in characteristic p.

Then we have the following theorem. Theorem 2.5 (Theorem D). Let X be as in Theorem A, let ᐄ be a smooth, proper model over U , and assume that the Doi-Naganuma lift DN F is surjective. Assume furthermore that the hypotheses (H1) for varying S, (H2) for inert good primes, and (H3) for inert primes not lying in U are satisﬁed. Then Ch0 (X){p} is ﬁnite for all primes p ∈ U , p 6. Remark. We do not need the full force of (H1), but it makes the argument easier. What we really need here in view of Theorem 2.2 is to ensure the existence of sufﬁciently many elements in H 1 (X, ᏷2 ) that cover all cycles in characteristic p at inert primes. We have not yet constructed the new cycles at inert primes nor the corresponding elements in K-theory. We note that our hypotheses (H1) and (H2) imply that in the exact sequence (2.1) the cokernel of ∂ is a torsion group. Indeed, for S ⊃ S, we have the equality of L-values ords=1 LS (X, s) = ords=1 LS (X, s) + ords=1 Pp (X, s)−1 . p∈S \S

Letting S vary and applying (H1), (H2), and Theorem 1.11 (Tate’s conjecture for split primes), we see that coker ∂ must be a torsion group (compare [M, pp. 389–390]). If p ∈ U is a prime different from q, ᐄ = ᐄ ×U U [1/p], we have, by Bloch [Bl1] and Merkurjev and Suslin [MS], an exact sequence (2.6) 0 −→ H 1 ᐄ , ᏷2 ⊗ Qp /Zp −→ NH 3 ᐄ , Qp /Zp (2) −→ Ch2 (ᐄ ){p} −→ 0, where NH 3 ᐄ , Qp /Zp (2) := ker Het3 ᐄ , Qp /Zp (2) −→ H 3 k(ᐄ ), Qp /Zp (2) . Here k(ᐄ ) denotes the function ﬁeld of k(ᐄ ). Since Het3 (ᐄ , Qp /Zp (2)) is coﬁnitely generated (its Qp /Zp -dual is a ﬁnitely generated Zp -module), the exact sequence shows that Ch2 (ᐄ ){p} is coﬁnitely generated as a Zp -module. The remark after

ZERO-CYCLES ON HILBERT-BLUMENTHAL SURFACES

153

Theorem 2.5 shows that Ch2 (X){p} is coﬁnitely generated, too. In analogy to the exact sequence above, we have an exact sequence for the generic ﬁber X: (2.7) 0 −→ H 1 (X, ᏷2 ) ⊗ Qp /Zp −→ NH 3 X, Qp /Zp (2) −→ Ch2 (X){p} −→ 0. For an abelian group M, let Mdiv be its maximal divisible subgroup. Let G KN H 3 X, Qp /Zp (2) = ker NH 3 X, Qp /Zp (2) −→ H 3 X, Qp /Zp (2) Q . Since H 3 (X, Qp /Zp (2))GQ is ﬁnite by Deligne’s proof of the Weil conjectures, we have KN H 3 X, Qp /Zp (2) div = NH 3 X, Qp /Zp (2) div . Due to the fact that Ch2 (X){p} is coﬁnitely generated, Theorem 2.5 follows from the next proposition. Proposition 2.8. Under the assumptions of Theorem 2.5, we have ∼ = H 1 (X, ᏷2 ) ⊗ Qp /Zp −→ KN H 3 X, Qp /Zp (2) div . Consider the Hochschild-Serre spectral sequence i,j E2 = H i Q, H j X, Qp /Zp (2) '⇒ H i+j X, Qp /Zp (2) . Lemma 2.9. H 2 (Q, H 1 (X, Qp /Zp (2))) is of ﬁnite exponent and vanishes for almost all p. Proof. We have H 1 (X, Qp /Zp (2)) = NS(X){p}(1) because the arithmetic genus of X is zero, and therefore the Picard-variety vanishes. It is known that NS(X) is ﬁnitely generated. Therefore, H 1 (X, Qp /Zp (2)) is ﬁnite and vanishes for almost all p. The lemma easily follows. Corollary 2.10. The kernel of the composite map λ KN Het3 X, Qp /Zp (2) −→ H 1 GQ , H 2 X, Qp /Zp (2) obtained from the Hochschild-Serre spectral sequence has ﬁnite exponent and vanishes for almost all p. Let V = H 2 (X, Qp (2)), T = H 2 (X, Zp (2)), A = H 2 (X, Qp /Zp (2)). For each prime , let He1 Q , V ⊂ Hf1 Q , V ⊂ Hg1 Q , V ⊂ H 1 Q , V be as deﬁned in [BK, (3.7)]. Let H?1 (Q , T ) ⊂ H 1 (Q , T ) be the inverse image of H?1 (Q , V ) and put H?1 Q , A := H?1 Q , T ⊗ Qp /Zp ⊂ H 1 Q , A .

154

ANDREAS LANGER

Write - = H 1 (Q , T )/Hf1 (Q , T ). We have - ⊗ Qp /Zp = H 1 (Q , A)div / Hf1 (Q , A). Let H 1 GQ , A 1 α : H GQ , A −→ ⊕ 1 all Hf GQ , A be the natural map, the kernel of which deﬁnes the Selmer group S(Q, A) = Sp∞ H 2 (X)(2) . Proposition 2.8 follows from the next two lemmas. Lemma 2.11. Under the assumptions of Theorem 2.2, S(Q, A) is ﬁnite. Lemma 2.12. The image of the composite map

α◦λ H 1 GQ , A H X, ᏷2 ⊗ Qp /Zp −→ KN H X, Qp /Zp (2) −−−→ ⊕ all Hf1 GQ ,A 1

3

is ⊕=p - ⊗ Qp /Zp ⊕ (Hg1 (Qp , A)/Hf1 (Qp , A)) and coincides with the image of KN H 3 (X, Qp /Zp (2))div under the map α ◦ λ. Proof of Lemma 2.11. By the ﬁniteness of the class number of F , it is easy to see that it sufﬁces to show Lemma 2.11 after replacing A by A = H 2 (X, Qp /Zp (2))div ∼ = T ⊗ Qp /Zp . Recall the decomposition of GQ -modules in (1.5): H 2 X, Qp (2) = H2 S, Qp (2) ⊕ HX2 ∞ X, Qp (2) . Choose GQ -equivariant Zp -lattices T1 , T2 , such that T1 ⊗ Qp = H2 S, Qp (2) , T2 ⊗ Qp = HX2 ∞ X, Qp (2) . Then A decomposes as T1 ⊗ Qp /Zp ⊕ T2 ⊗ Qp /Zp . By the ﬁniteness of the class number of F , it is clear that the Selmer group of T2 ⊗ Qp /Zp is ﬁnite. It remains to show that the Selmer group of the GQ -module T1 ⊗ Qp /Zp is ﬁnite. Let M be the algebraic number ﬁeld obtained by adjoining all Fourier coefﬁcients of Hecke eigenforms f ∈ S2+ (0 (q), εq ), OM the ring of integers, Mˆ the completion of M at a prime above p, and Oˆ the ring of integers of ˆ ˆ After choosing ˆ Consider the cohomology H2 (S, M(2)) with coefﬁcients in M. M. ˆ ˆ Lˆ in H2 (S, M(2)), it is clear that the ﬁniteness of the a GQ -equivariant O-lattice Selmer group of T1 ⊗ Qp /Zp follows from the ﬁniteness of the Selmer group of ˆ ˆ Again consider the decomposition (1.7): L. H2 (S, M(2))/

∼ ˆ Wfˆ (1) ⊗ Mˆ ⊕ W0 (1), ⊕ H2 S, M(1) = f ∈{f1 ,...,fr }

Kˆ f

where Wfˆ (1) is isomorphic to End(Hf ) ⊗Kˆ f Mˆ as Gal(Q/F )-modules.

ZERO-CYCLES ON HILBERT-BLUMENTHAL SURFACES

155

ˆ By Poitou-Tate duality, the ﬁniteness of the Selmer group of H2 (S, M(2))/ Lˆ fol2 ˆ ˆ lows from the ﬁniteness of the Selmer group of H (S,M(1))/L(−1). By an easy norm ˆ ˆ L(−1) argument, it sufﬁces to show the ﬁniteness of the Selmer group of H2 (S,M(1))/ over F , the real quadratic ﬁeld. The Selmer group of W0 (1) is treated in the same way as the cohomology of cycles supported in the resolutions of cusp and quotient singularities. Let Oˆ f be the ring of integers in Kˆ f (compare with Section 1). There are Oˆ f -lattices Lf in End(Hf ) constructed by taking the natural lattices Yf contained in H 1 (X0 (q), Oˆ f ), which after tensoring with Kˆ f , yield Hf . Let End0 (Hf ) be the endomorphisms of trace zero, and let L0f be the corresponding lattice. We have a decomposition (1.10), End(Hf ) = End0 (Hf ) ⊕ Kˆ f , and Lemma 2.11 follows from the next theorem, due to Fujiwara [Fu, Theorem 5.1 and its proof], which extends an analogous result of Wiles [W, Theorem 3.1], who works over Q (compare also Flach [Fl2, Theorem 1]. Theorem 2.13 (Fujiwara). The Selmer group S(F, End0 (Hf )/L0f ) is ﬁnite. Remark. In our situation, all Hilbert modular forms satisfy Fujiwara’s condition on the existence of a minimal modular lift (compare [Fu, Deﬁnition 2.2.2]). Let ᐄZp be a smooth, proper model of XQp over Zp , and let j : XQp ;→ ᐄZp be the inclusion of the generic ﬁber. Then we have the following lemma. Lemma 2.14. We have the inclusion NHet3 XQp , Z/p n (2) ⊂ Het3 ᐄZp , τ≤2 Rj∗ Z/p n (2) . This is shown in [LS, Lemma 5.4]. From Lemma 2.14, we get inclusions NHet3 XQp , Qp (2) ⊂ Het3 ᐄZp , τ≤2 Rj∗ Qp (2) ⊂ Het3 XQp , Qp (2) . The main result in [L3] on a comparison of log-syntomic cohomology and p-adic points of motives implies the following proposition. Proposition 2.15. Under the isomorphism Het3 XQp , Qp (2) ∼ = H 1 Qp , V , the image of Het3 (ᐄZp , τ≤2 Rj∗ Qp (2)) is Hg1 (Qp , V ). Corollary 2.16. The image of the composite map NHet3 XQp , Qp (2) −→ Het3 XQp , Qp (2) ∼ = H 1 Qp , V is contained in Hg1 (Qp , V ).

156

ANDREAS LANGER

To show Lemma 2.12, it sufﬁces to prove that the map Hg1 Qp , V H 1 Q , V 1 3 ⊕ 1 H (X, ᏷2 ) ⊗ Qp −→ Hcont X, Qp (2) −→ ⊕ 1 =p Hf Q , V Hf Qp , V 3 (X, Q (2)) the continuous étale cohomology is surjective. (Here we denote by Hcont p introduced by Jannsen [J]; note that H 3 (X, Qp (2))GQ is zero due to Deligne’s proof of the Weil conjectures, and therefore the second map is well deﬁned.)

Lemma 2.17. Let ∈ U . There is a canonical isomorphism 1 Q ,V Hg1 Qp , V H for = p ) (resp., Pic Xp ⊗ Qp ∼ Pic(X ) ⊗ Qp ∼ = 1 = 1 Hf Q , V Hf Qp , V ﬁtting into a commutative diagram ∂ / Pic(X ) ⊗ Qp H 1 X, ᏷2 ⊗ Qp NNN NNN ∼ NNN = NNN & H 1 Q , V Hf1 Q , V

(resp.,

Hg1 Qp , V Hf1 (Qp , V )

).

This is proven in [LS, Lemma 4.5]. The crucial point for the isomorphism is the Tate conjecture in characteristic that we show for the Hilbert-Blumenthal surface X for split primes in Section 1 and that we assume for inert primes. We recall here the main arguments for this isomorphism. For = p, we have H 1 (Q , V )/Hf1 (Q , V ) = H 1 (I , H 2 (X, Qp (2))F r ) by deﬁnition, and it is easy to see that the right-hand side is isomorphic to H 2 (X, Qp (1))F r ∼ = Pic(X )⊗Qp by Theorem 1.11. For = p, we have by local Tate duality (see [BK, Section 3.8]) that GQ Hf1 Qp , V (−1) Hg1 Qp , V p ∼ is Q B -dual to ⊗ V (−1) /(1 − f ), = p crys He1 Qp , V (−1) Hf1 Qp , V Qp where f = ϕ ·p−1 and ϕ is the Frobenius on Bcrys . By Poincaré duality for crystalline cohomology and the crystalline conjecture proven by Fontaine and Messing and by Faltings, we have an isomorphism Hg1 Qp , V 2 ϕ=p ∼ Xp /W ⊗ Qp , = Hcrys 1 Hf Qp , V where ϕ is the crystalline Frobenius. The isomorphism Pic(Xp ) ⊗ Qp ∼ = 2 ϕ=p follows from the crystalline Tate conjecture, which is (Hcrys (Xp /W ) ⊗ Qp )

157

ZERO-CYCLES ON HILBERT-BLUMENTHAL SURFACES

equivalent to the Tate conjecture using -adic cohomology = p by [Mil, Theorem 4.1] and therefore known by Theorem 1.11 for split and assumed by (H2) for inert . To see the commutativity of the diagram in Lemma 2.17, for = p we recall the commutative diagram (see [LS, (5.5)]) 0

/ H 1 ᐄZ , ᏷2 ⊗ Qp p

0

/ H 3 ᐄ, S [2] syn Q p

/ H 1 XQ , ᏷2 ⊗ Qp p

∂p

/ Pic Xp ⊗ Qp ∼ =

1 X , W P1 H / H 3 ᐄZ , τ≤2 Rj∗ Qp (2) / lim p n log ⊗ Qp . ← − p n

Here the lower exact sequence is obtained from the long exact cohomology sequence to a distinguished triangle due to Kurihara [Ku], which is recalled in [LS, paragraph 6]. We then observe that under the inclusion H 3 (ᐄZp , τ≤2 Rj∗ Qp (2)) ⊂ H 1 (Qp , V ), the 3 (ᐄ, S [2] ) corresponds to H 1 (Q , V ). This is Schneider’s syntomic cohomology Hsyn p f Qp p-adic points conjecture proven in [N] and, under some restrictions that are satisﬁed in our situation, also in [LS, Theorem 6.5]. We then use Proposition 2.15 to see ∼ = that the isomorphism Pic(Xp ) ⊗ Qp −→ Hg1 (Qp , V )/Hf1 (Qp , V ) constructed above makes the diagram in Lemma 2.17 commutative in the case = p. Now the remark after Theorem 2.5 and Lemma 2.17 imply that the homomorphism Hg1 Qp , V H 1 Q , V 1 ⊕ 1 H X, ᏷2 ⊗ Qp −→ ⊕ 1 ∈U Hf Q , V Hf Qp , V =p

is surjective. It remains to consider primes ∈ U . For this, we need additional arguments. We differentiate between two cases: = q and = q. Case 1: = q. We recall the GQ -equivariant decomposition in (1.5): V = H 2 X, Qp (2) = H2 S, Qp (2) + HX2 ∞ X, Qp (2) . Let Z be the subgroup of Pic(X) generated by cycles in the resolutions of cusp or quotient singularities. Then it is easy to see that the map H 1 Q , HX2 ∞ X, Qp (2) Z 1 Z ⊗ ⊗ Qp −→ H X, ᏷2 ⊗ Qp −→ 1 Hf Q , HX2 ∞ X, Qp (2) is surjective (compare with the argument in [LS, Lemma 4.1]). Now H2 (S, Qp (2)) decomposes as in (1.7) under the action of ᏴK0 , H2 S, Qp (2) ∼ ⊕ Wfˆ (2) ⊗ Qp ⊕ W0 (2) ⊗ Qp . = ϕ∈{f1 ,...,fr }

Kˆ f

Qp

It is clear that H 1 (Q , W0 (2))/Hf1 (Q , W0 (2)) is again covered by the image of

158

ANDREAS LANGER

(Y ⊗ Z ) ⊗ Qp ⊂ [Pic(X) ⊗ Q∗ ] ⊗ Qp in H 1 (X, ᏷2 ) ⊗ Qp where Y is the subgroup generated by the line bundle L1 ⊗ L2 . For Wfˆ (2), we use the decomposition (1.10) as Gal(Q/F )-modules Wfˆ (2) ∼ = End(Hf )(1) ∼ = End0 (Hf )(1) ⊕ Kˆ f (1). Here Kˆ f corresponds to the cycle class of a Hirzebruch-Zagier cycle C ∈ Pic(X) ⊗ Qp , and if C is deﬁned over Q, then the image of (C ⊗Z )⊗Z Qp in H 1 (X, ᏷2 )⊗Qp covers the 1-dimensional Qp -vectorspace H 1 Q , Kˆ f (1) H 1 Q , Qp (1) ⊂ . Hf1 Q , Qp (1) H 1 Q , Kˆ f (1) f

For End0 (Hf )(1), we have the following result of Flach that ﬁnishes the case = q. (We denote by Fq the completion of F at the prime above q.) Lemma 2.18 [Fl2, Lemma 5.5.2]. The inﬂation map Hf1 Fq, End0 (Hf )(1) −→ H 1 Fq, End0 (Hf )(1) is an isomorphism. Case 2: = q. We have the following lemma. Lemma 2.19. We have

H 1 Q , Wfˆ (2) ∼ W ˆ (1) F r . = f 1 Hf Q , Wfˆ (2)

Proof. This is proven in the same way as Lemma 2.17 by interpreting Wfˆ (1) as a Gal(Q/F )-subrepresentation of H 2 (X0 (q)×X0 (q), Kˆ f (1)) and using the fact that the Tate conjecture is known for the abelian variety J0 (q) × J0 (q) in char = q, where J0 (q) is the Jacobian variety of X0 (q). By Lemma 1.14, we have

F r

dim Wfˆ (1)

=

3 2

inert, split.

The decomposition

V = H2cusp S, Qp (2) ⊕ W0 (2) ⊕ HX2 ∞ X, Qp (2)

yields as before that

Z˜ ⊗

Z

H 1 Q , W0 (2) ⊕ HX2 ∞ X, Qp (2) ⊗ Qp −→ H X, ᏷2 ⊗ Qp −→ 1 Hf Q , W0 (2) ⊕ HX2 ∞ X, Qp (2) 1

ZERO-CYCLES ON HILBERT-BLUMENTHAL SURFACES

159

is surjective where Z˜ is the subgroup in Pic(X) generated by cycles in the resolutions of singularities and the class of the line bundle L1 ⊗ L2 . Let N ≥ 3 be an integer such that N and such that (H3) is satisﬁed if is inert, and let π : SKN → S be the Galois covering considered in Section 1. We have a commutative diagram 1

H SKN , ᏷2 ⊗ Qp O

H 1 Q , H2cusp S KN , Qp (2) / Hf1 Q , H2cusp S KN , Qp (2) O

c ∂N

π∗

(2.20)

E21,−2 (S) ⊗ Qp O

π∗

w

H 1 (X, ᏷2 ) ⊗ Qp

∂c

H 1 Q , H2cusp S, Qp (2) , Hf1 Q , H2cusp S, Qp (2) /

where w is the map obtained by functorial properties of K-theory that was already used in the proof of Theorem 2.2 (compare with the commutative diagram (∗)). The map π ∗ on the right is induced by the inclusion H2 (S, Qp (2)) ;→ H2 (S KN , Qp (2)). The maps ∂ c and ∂Nc are obtained from the -adic regulator map and the projection on the cuspidal part of the cohomology. For functorial reasons, the commutativity of the diagram is clear. Now it is easy to see that the map ω is surjective. If i (Di , fi ), fi ∈ k(Di ) is an element in E21,−2 (S), then take the closure Di i on X with fi ∈ k(D i ). By adding a certain ﬁnite on S and the strict transform D sum j (Cj , gj ) where Cj is a rational curve in the resolution of cusp or quotient with the proof of Theorem 2.2), we obtain an singularities and gi ∈ k(Ci ) (compare element y ∈ H 1 (X, ᏷2 ) with ω(y) = i (Di , fi ). Furthermore, it is clear that ker ω consists of elements supported on cycles in the resolution of singularities. Therefore, ∂ c (ker ω) = 0 and ∂ c factors through E21,−2 (S) ⊗ Qp . Now assume that = q is an inert prime. By our hypothesis (H3), ∂Nc is surjective. From the commutative diagram H1

SKN , ᏷2 ⊗ Qp

c ∂N

π∗

E21,−2 (S) ⊗ Qp

/

H 1 Q , H2cusp S KN , Qp (2) Hf1 Q , H2cusp S KN , Qp (2) π∗

∂c

we see that ∂ c must also be surjective.

H 1 Q , H2cusp S, Qp (2) / , Hf1 Q , H2cusp S, Qp (2)

160

ANDREAS LANGER

Finally, let be a split prime, let (T , u2 ) be the element in E21,−2 (S) considered as before (it exists for all split primes), and let z be a lift of (T , u2 ) in H 1 (X, ᏷2 ), which is constructed in the proof of Theorem 2.2. We also have the element T˜ ⊗ ∈ Pic(X)⊗Q∗ → H 1 (X, ᏷2 ) where T˜ is the strict transform of the Hirzebruch-Zagier cycle T on X. We deﬁne 1 c ˜ 1 c c ∂ T ⊗ + ∂ (z) , F = 2 κ() (2.21) 1 c ˜ 1 c Vc = ∂ T ⊗ − ∂ (z) , 2 κ() where κ() is the constant appearing in [Fl2, Lemma 2]. As before, let M be the algebraic number ﬁeld generated by all Fourier coefﬁcients of modular forms f ∈ S2+ (0 (q), εq ), and let Mˆ be the completion at a prime above ˆ p such that Mˆ ⊂ Qp . Using the decomposition (1.7) for H2cusp (S, M(2)), we get

F r ˆ H 1 Q , H2cusp S, M(2) = ⊕ End(Hf ) ⊗ Mˆ ˆ f ∈{f1 ,...,fr } Hf1 Q , H2cusp S, M(2) Kˆ f by Lemma 2.19 and the fact that splits. According to the decomposition F c and V c , we have components F f and V f in End(Hf )F r . The surjectivity of ∂ c follows if we can show that F f and V f are linearly independent in End(Hf ). In (N) H 1 (SKN , ᏷2 ), we have elements (T , u2 ) and we deﬁne similar elements FNc , VNc in H 1 (Q , H2cusp (S KN , Qp (2)))/Hf1 (Q , H2cusp (S KN , Qp (2))) by the formulas (N)

in (2.21), using the elements T (2.22)

(N)

⊗ and (T , u2 ). Then we have # (N) T g T , u 2 , π ∗ ω(z) = π ∗ T , u2 = g

where g runs through a ﬁnite set of elements in K0 /KN such that

(N) # g Tg ((T , u2 )) π

is invariant under the action of K0 /KN (the Galois group of SKN → S). From the commutative diagram (2.20), we get π ∗F c = (2.23) Tg FNc , π ∗V c = Tg VNc . g

g

f

f

In analogy to the situation over S, we get components FN , VN in [End(Hf ) ⊗Kˆ f

Qp ⊗ π KN ]F r using the decomposition (1.16) where F r acts on End(Hf ). From (2.23), we get f f Tg F N , π ∗V f = Tg V N . π ∗F f = g

g

161

ZERO-CYCLES ON HILBERT-BLUMENTHAL SURFACES

We check the linear independence of F f and V f in [End(Hf )⊗Kˆ f Qp ⊗Q π KN ]F r , p

that is, we show the linear independence of π ∗ F f , π ∗ V f . The proof is very similar to the proof of the Tate conjecture in characteristic p at good reduction primes in Section 1. Assume that F f and V f are homotheties in End(Hf ), that is, α 0 α 0 f f . , V = F = 0 α 0 α

Then α + α = a (f ), the th Fourier coefﬁcient of f . Then we have (N) Tg T1 f , π ∗ F f = απ ∗ (id) = α · g

(2.24)

∗

f

∗

π V = α π (id) = α ·

g

(N) Tg T1 f ,

(N)

where (T1 )f is the f -component of the diagonal cycle class in SKN . Since SKN has good reduction at , we can again use the morphisms Ᏺ℘ , Ᏺ℘ : SN → SN , where SN is the closed ﬁber of ᏿KN /Z and () = ℘ · ℘ (see (1.19)). These induce homomorphisms on End(Hf ) ⊗Kˆ f Qp ⊗ π KN , and we have the relations

Ᏺ℘ T1N

f

f

= FN ,

Ᏺ℘ T1N

f

f

= VN ,

as in Section 1. Indeed, from the deﬁnitions of FNc , VNc , we see that FNc and VNc are the cuspidal parts of the cycle classes of F℘ and F℘ deﬁned in Section 1. (N) (N) Since Ᏺ℘ ◦ Ᏺ℘ ((T1N )f ) = Frob((T1 )f ) = · (T1 )f , we get from (2.24) that α ·α = . Thus α and α are the two different eigenvalues of F r acting on Hf (again f f use [CE, Theorem 2.1]). Since (F f , T1 ) = (V f , T1 ) = (1/2)T r(F f +V f ) = a (f ), we again get 2α = 2α = a (f ) ⇒ α = α . This is a contradiction. Summarizing, we see that the map ∂ c in diagram (2.20) is surjective. This ﬁnishes Case 2 ( = q) and the proof of Lemma 2.12, which implies Proposition 2.8. Theorem 2.5 now follows. References [Bl1] [Bl2]

[BK]

[BL]

S. Bloch, Lectures on Algebraic Cycles, Duke Univ. Math. Ser. 4, Duke University, Durham, N.C., 1980. , “A note on Gersten’s conjecture in the mixed characteristic case” in Applications of Algebraic K-Theory to Algebraic Geometry and Number Theory, Part I, II (Boulder, Colo., 1983), Contemp. Math. 55, Amer. Math. Soc., Providence, 1986, 75–88. S. Bloch and K. Kato, “L-functions and Tamagawa numbers of motives” in The Grothendieck Festschrift, Vol. 1, Progr. Math. 86, Birkhäuser, Boston, 1990, 333–400. J.-L. Brylinski and J.-P. Labesse, Cohomologie d’intersection et fonctions L de certaines variétés de Shimura, Ann. Sci. École Norm. Sup. (4) 17 (1984), 361–412.

162 [CE] [CTR]

[CTSS] [D]

[DR]

[Fl1] [Fl2]

[Fu] [vdG] [G] [HLR] [HZ] [J] [Kl] [Ku] [L1]

[L2] [L3] [L4]

[LR] [LS] [MS]

ANDREAS LANGER R. Coleman and S. J. Edixhoven, On the semisimplicity of the Up -operator on modular forms, to appear in Math. Ann. J.-L. Colliot-Thélène and W. Raskind, Groupe de Chow de codimension deux des variétés déﬁnies sur un corps de nombres: un théorème de ﬁnitude pour la torsion, Invent. Math. 105 (1991), 221–245. J.-L. Colliot-Thélène, J.-J. Sansuc, and C. Soulé, Torsion dans le groupe de Chow de codimension deux, Duke Math. J. 50 (1983), 763–801. P. Deligne, Formes modulaires et représentations -adiques, (Séminaire Bourbaki 1968/69, exp. no. 355), Lecture Notes in Math. 179, Springer, New York, 1971, 139–172. P. Deligne and M. Rapoport, “Les schémas de modules de courbes elliptiques” in Modular Functions of One Variable, II (Antwerp, 1972), Lecture Notes in Math. 349, Springer, Berlin, 1973, 143–316. M. Flach, A ﬁniteness theorem for the symmetric square of an elliptic curve, Invent. Math. 109 (1992), 307–327. , “Annihilation of Selmer groups for the adjoint representation of a modular form” in Seminar on Fermat’s Last Theorem (Toronto, 1993–1994), CMS Conf. Proc. 17, Amer. Math. Soc., Providence, 1995, 249–265. K. Fujiwara, Deformation rings and Hecke algebras in the totally real case, preprint, 1996. G. van der Geer, Hilbert Modular Surfaces, Ergeb. Math. Grenzgeb. (3) 16, Springer, Berlin, 1988. S. Gelbart, Automorphic forms on adèle groups. Ann. of Math. Stud. 83, Princeton Univ. Press, Princeton, 1975. G. Harder, R. P. Langlands, and M. Rapoport, Algebraische Zyklen auf HilbertBlumenthal-Flächen, J. Reine Angew. Math. 366 (1986), 53–120. F. Hirzebruch and D. Zagier, Intersection numbers of curves on Hilbert modular surfaces and modular forms of Nebentypus, Invent. Math. 36 (1976), 57–113. U. Jannsen, Continuous étale cohomology, Math. Ann. 280 (1988), 207–245. C. Klingenberg, Die Tate-Vermutungen für Hilbert-Blumenthal-Flächen, Invent. Math. 89 (1987), 291–317. M. Kurihara, A note on p-adic étale cohomology, Proc. Japan Acad. Ser. A Math. Sci. 63 (1987), 275–278. A. Langer, Selmer groups and torsion zero cycles on the selfproduct of a semistable elliptic curve, Doc. Math. 2 (1997), 47–59, available from http://www.mathematik. uni-bielefeld.de/documenta/. , 0-cycles on the elliptic modular surface of level 4, Tohoku Math. J. (2) 50 (1998), 291–302. , Local points of motives in semistable reduction, Compositio Math. 116 (1999), 189–217. , “Finiteness of torsion in the codimension-two Chow group: An axiomatic approach,” to appear in Proceedings of the Banff Conference on Arithmetic and Geometry of Algebraic Cycles, Kluwer. A. Langer and W. Raskind, 0-cycles on the selfproduct of a CM-elliptic curve over Q, J. Reine Angew. Math. 516 (1999), 1–26. A. Langer and S. Saito, Torsion zero-cycles on the self-product of a modular elliptic curve, Duke Math. J. 85 (1996), 315–357. A. S. Merkurjev and A. A. Suslin, K-cohomology of Severi-Brauer varieties and the norm residue homomorphism (in Russian), Izv. Akad. Nauk. SSSR Ser. Mat. 46 (1982), 1011–1046, 1135–1136.

ZERO-CYCLES ON HILBERT-BLUMENTHAL SURFACES [M] [Mi] [Miy] [MR] [N] [O] [P] [Q] [R1] [R2] [R3] [R4]

[Ra] [Ras]

[S] [Sch] [W]

163

S. Mildenhall, Cycles in a product of elliptic curves, and a group analogous to the class group, Duke Math. J. 67 (1992), 387–406. J. S. Milne, On a conjecture of Artin and Tate, Ann. of Math. (2) 102 (1975), 517–533. T. Miyake, On automorphic forms on GL(2) and Hecke operators, Ann. of Math. (2) 94 (1971), 174–189. V. K. Murty and D. Ramakrishnan, Period relations and the Tate conjecture for Hilbert modular surfaces, Invent. Math. 89 (1987), 319–345. J. Nekováˇr, Syntomic cohomology and p-adic regulators, in preparation. T. Oda, Periods of Hilbert Modular Surfaces, Progr. Math. 19, Birkhäuser, Boston, 1982. G. Prasad, Strong approximation for semi-simple groups over function ﬁelds, Ann. of Math. (2) 105 (1977), 553–572. D. Quillen, “Higher algebraic K-theory, I” in Algebraic K-Theory, I: Higher K-Theories (Seattle, 1972), Lecture Notes in Math. 341, Springer, Berlin, 1973, 85–147. D. Ramakrishnan, Periods of integrals arising from K1 of Hilbert-Blumenthal surfaces, preprint, 1984. , Valeurs de fonctions L des surfaces d’Hilbert-Blumenthal en s = 1, C. R. Acad. Sci. Paris Sér. I Math. 301 (1985), 809–812. , “Arithmetic of Hilbert-Blumenthal surfaces” in Number Theory (Montreal, 1985), CMS Conf. Proc. 7, Amer. Math. Soc., Providence, 1987, 285–370. , “Regulators, algebraic cycles, and values of L-functions” in Algebraic K-Theory and Algebraic Number Theory (Honolulu, 1987), Contemp. Math. 83, Amer. Math. Soc., Providence, 1989, 183–310. M. Rapoport, Compactiﬁcations de l’espace de modules de Hilbert-Blumenthal, Compositio Math. 36 (1978), 255–335. W. Raskind, “Torsion algebraic cycles on varieties over local ﬁelds” in Algebraic KTheory: Connections with Geometry and Topology (Lake Louise, Alberta, 1987), NATO Adv. Sci. Inst. Ser. C Math. Phys. Sci. 279, Kluwer, Dordrecht, 1989, 343–388. P. Schneider, “p-adic points of motives” in Motives (Seattle, 1991), Proc. Sympos. Pure Math. 55, Part 2, Amer. Math. Soc., Providence, 1994, 225–249. A. J. Scholl, Integral elements in K-theory and products of modular curves, preprint, 1999. A. Wiles, Modular elliptic curves and Fermat’s last theorem, Ann. of Math. (2) 141 (1995), 443–551.

Mathematisches Institut, Universität Köln, Weyertal 86-90, D-50931 Köln, Germany

Vol. 103, No. 1

DUKE MATHEMATICAL JOURNAL

© 2000

QUANTUM DETERMINANTAL IDEALS K. R. GOODEARL and T. H. LENAGAN

Introduction. Fix a base ﬁeld k. The quantized coordinate ring of n × n matrices over k, denoted by ᏻq (Mn (k)), is a deformation of the classical coordinate ring of n × n matrices, ᏻ(Mn (k)). As such, it is a k-algebra generated by n2 indeterminates Xij , for 1 ≤ i, j ≤ n, subject to relations which we state in (1.1). Here, q is a nonzero element of the ﬁeld k. When q = 1, we recover ᏻ(Mn (k)), which is the commutative polynomial algebra k[Xij ]. The algebra ᏻq (Mn (k)) has a distinguished element Dq , the quantum determinant, which is a central element. Two important algebras ᏻq (GLn (k)) and ᏻq (SLn (k)) are formed by inverting Dq and setting Dq = 1, respectively. The structures of the primitive and prime ideal spectra of the algebras ᏻq (GLn (k)) and ᏻq (SLn (k)) have been investigated recently (see, for example, [2], [7], and [10]). Results obtained in these investigations can be pulled back to partial results about the primitive and prime ideal spectra of ᏻq (Mn (k)). However, these techniques give no information about the closed subset of the spectrum determined by Dq . In this paper, we begin the study of this portion of the spectrum. In the classical commutative setting, much attention has been paid to determinantal ideals: that is, the ideals generated by the minors of a given size. In particular, these are special prime ideals of ᏻ(Mn (k)) containing the determinant. Moreover, there are interesting geometrical and invariant theoretical reasons for the importance of these ideals (see, for example, [4]). In order to put our results into context, it may be useful to review some highlights of the commutative theory. Let Ml,m (k) denote the algebraic variety of l × m matrices over k. For t ≤ n, the general linear group GLt (k) acts on Mn,t (k) × Mt,n (k) via g · (A, B) := Ag −1 , gB . Matrix multiplication yields a map µ : Mn,t (k) × Mt,n (k) −→ Mn (k), the image of which is the set of matrices with rank at most t. Received 1 March 1998. Revision received 1 April 1999. 2000 Mathematics Subject Classiﬁcation. Primary 16P40, 16W30, 16W35, 16S15, 13C40, 20G42. Goodearl and Lenagan partially supported by National Science Foundation grant number DMS9622876 and NATO Collaborative Research grant number 960250. 165

166

GOODEARL AND LENAGAN

There is an induced map µ∗ : ᏻ Mn (k) −→ ᏻ Mn,t (k) × Mt,n (k) = ᏻ Mn,t (k) ⊗ ᏻ Mt,n (k) . The ﬁrst fundamental theorem of invariant theory identiﬁes the ﬁxed ring of the coordinate ring ᏻ(Mn,t (k)×Mt,n (k)) under the induced action of GLt (k) as precisely the image of µ∗ . The second fundamental theorem states that the kernel of µ∗ is Ᏽt , the ideal generated by the (t +1)×(t +1) minors of ᏻ(Mn (k)), so that the coordinate ring of the variety of n × n matrices of rank at most t is ᏻ(Mn (k))/Ᏽt . As a consequence, since this variety is irreducible, the ideal Ᏽt is a prime ideal of ᏻ(Mn (k)). Our main result, Theorem 2.5, is a quantum analog of the second fundamental theorem. (We make the conjecture that there is also a quantum analog of the ﬁrst fundamental theorem, but do not address that problem in the present paper.) If I and J are subsets of {1, . . . , n} with |I | = |J |, then D(I, J ) denotes the quantum minor obtained by evaluating the quantum determinant of the subalgebra of ᏻq (Mn (k)) generated by those Xij with i ∈ I and j ∈ J . The ideal Ᏽt is then the ideal generated by the (t + 1) × (t + 1) quantum minors of ᏻq (Mn (k)). Theorem 2.5 states that ᏻq (Mn (k))/Ᏽt is an integral domain, for 0 ≤ t ≤ n − 1. The case (t = n − 1) of this result was proved by Jordan [9] and Levasseur and Stafford [11, p. 182]; the case (t = 1) was obtained by Rigal [15]. The case (t = 0) holds trivially. The classical commutative appproach to the second fundamental theorem is as follows: By geometrical considerations, the variety of n × n matrices of rank at most t is an irreducible √ variety, and it is easy to see that the coordinate ring of this variety is ᏻ(Mn (k))/ Ᏽt . Thus, the main problem is to show that Ᏽt is a radical ideal of ᏻ(Mn (k)). This is achieved via the notion of algebras with straightening laws (see [3] or [4]). In order to simplify the problem, the algebra ᏻ(Mn (k)) is replaced temporarily by ᏻ(Mn,2n (k)). The subalgebra B of ᏻ(Mn,2n (k)) generated by the maximal minors of ᏻ(Mn,2n (k)) is the coordinate ring of the Grassmannian of the n-dimensional subspaces of k 2n . The products of maximal minors span B, but do not form a basis— the famous Plücker relations generate the relations between the maximal minors. The Plücker relations are used to produce straightening laws leading to a standard basis of B. All this is now specialised by setting the rightmost n × n block of Xij ’s equal to the identity matrix. The images of the maximal minors become all of the minors of ᏻ(Mn (k)), and the standard basis of B induces a standard basis of ᏻ(Mn (k)) consisting of certain products of minors of ᏻ(Mn (k)). This establishes that ᏻ(Mn (k)) is an algebra with a straightening law. The conclusion that Ᏽt is radical then follows easily. The classical approach breaks down completely in the quantum setting. There is no group acting, and setting noncentral elements equal to 0 or 1 produces a homomorphic image that is far too small. The action of the group GLt (k) can be replaced by a coaction of the Hopf algebra ᏻq (GLt (k)). Otherwise, the only thing that survives is the idea of a basis constructed of products of (quantum) minors and straightening laws. However, an added complication appears: As well as straightening laws to deal with linear dependencies, it is also necessary to generate commutation laws, in order

QUANTUM DETERMINANTAL IDEALS

167

to deal with reordering products. This latter problem leads us to choose a different ordering of variables than the ordering chosen in the classical case, so that we can give preference to the best approximation to central elements—namely, the normal elements that occur in profusion in the quantum case. As a result, we call our basis a preferred basis rather than a standard basis. The second main ingredient in the proof is the exploitation of the fact that ᏻq (Mn (k)) is a bialgebra. Quantum minors behave well under the comultiplication map ; using this fact, we produce an embedding of ᏻq (Mn (k))/Ᏽt into the algebra ᏻq (Mn,t (k))⊗ ᏻq (Mt,n (k)). This latter algebra is an iterated Ore extension of k and is thus a domain, thereby establishing our theorem. In the latter part of the paper, we use the twisting methods of Artin, Schelter, and Tate [1] to show that our results also hold for multiparameter coordinate rings of quantum matrices. Acknowledgments. We thank James Zhang for several very helpful conversations, and Jacques Alev and Laurent Rigal for useful comments. 1. A basis for quantum matrices. This section is devoted to establishing the existence of a basis for ᏻq (Mn (k)) which is built from products of quantum minors. This basis is crucial to our calculations with quantum determinantal ideals. A basis of this type was constructed in [8] for a class of quantum matrix superalgebras, which includes the ᏻq (Mn (k)) for q not a root of unity. Our modiﬁcation of their construction allows q to be an arbitrary nonzero scalar. For convenience of notation and when applying results from the literature, we work mainly with the quantum coordinate rings of square matrices. At the end of the section, we see that our basis theorem readily carries over to the case of ᏻq (Mm,n (k)). The calculations involved in constructing and verifying our basis rely on several general identities concerning products of quantum minors. Although some of these identities are of standard types, they are not available in the literature in precisely the forms we require; thus we derive them from known forms. In order not to disrupt the line of this section, we relegate the discussions of the identities to the appendices. 1.1. Throughout this section, we ﬁx an integer n ≥ 2, a base ﬁeld k, and a nonzero scalar q ∈ k × . No other restrictions are assumed; in particular, k need not be algebraically closed, and q is allowed to be a root of unity. We work with the oneparameter quantized coordinate ring of n × n matrices over k, namely, the algebra Ꮽ = ᏻq (Mn (k)) with generators Xij for i, j = 1, . . . , n and relations Xij Xlj = qXlj Xij ,

when i < l;

Xij Xim = qXim Xij ,

when j < m;

when i < l and j < m; Xim Xlj = Xlj Xim , Xij Xlm − Xlm Xij = q − q −1 Xim Xlj , when i < l and j < m.

168

GOODEARL AND LENAGAN

As is well known, this algebra is in fact a bialgebra, with comultiplication : Ꮽ → Ꮽ ⊗ Ꮽ and counit : Ꮽ → k such that n Xij = Xil ⊗ Xlj

and

Xij = δij

l=1

for all i, j . 1.2. We need several partial-order relations on index sets. Let A, B ⊆ {1, . . . , n}, not necessarily of the same cardinality. First, we deﬁne a row ordering, denoted by ≤r . To describe this, we write A and B in descending order: A = a 1 > a2 > · · · > a α and B = b 1 > b2 > · · · > b β . Deﬁne A ≤r B to mean that α ≥ β and ai ≥ bi , for i = 1, . . . , β. For the column ordering, denoted by ≤c , we write A and B in ascending order: A = a 1 < a2 < · · · < a α and B = b 1 < b2 < · · · < b β . Deﬁne A ≤c B to mean that α ≥ β and ai ≤ bi , for i = 1, . . . , β. With the term index pair we denote a pair (I, J ) where I, J ⊆ {1, . . . , n} and |I | = |J |. Order index pairs by (≤r , ≤c ); that is, deﬁne (I, J ) ≤ (I , J ) if and only if I ≤r I and J ≤c J . For example, when n = 3 the poset of index pairs can be drawn as in Figure 1, where we have abbreviated the descriptions of index sets by eliminating braces and commas. 1.3. The basis we construct is indexed by certain bitableaux (pairs of tableaux) with speciﬁcations as below. Recall that, in general, a tableau consists of a Young diagram with entries in each box. We consider only tableaux with entries from {1, . . . , n} and no repetitions in any row. Allowable bitableaux are pairs (T , T ) where the following are true: (a) T and T have the same shape; (b) T has strictly decreasing rows; (c) T has strictly increasing rows. Rows of T or T can be identiﬁed with subsets of {1, . . . , n} listed in descending or ascending order. Hence, allowable bitableaux can be labeled in the form   I 1 J1  I 2 J2    (••)  .. ..  , . . Il Jl where (I1 , J1 ), . . . , (Il , Jl ) are index pairs such that |I1 | ≥ |I2 | ≥ · · · ≥ |Il |. The pair (I1 , J1 ) is called the top row of (T , T ).

169

QUANTUM DETERMINANTAL IDEALS

(∅, ∅)

pp ppp p p pp

qq qqq q q qq

(1, 1) M MMM MMM MM

(1, 2) N NNN NNN NN

(1, 3) N NNN NNN NN pp ppp p p pp

(2, 3) M MMM MMM MM

(2, 2) N (3, 3) NNN pp qq p q N p q NNN pp qq N qqq ppp (2, 1) N (21, 23)N (3, 2) NNN ppp NNN ppp NNpN p pp NNNN p NN ppp ppp (21, 13)N (3, 1) (31, 23)M MMM NNN pp qq p q MMM N p q NNN p q p q MM q p N qq pp (21, 12)M (31, 13)N (32, 23) MMM NNN pp qq p q MMM N p q NNN pp qq MM N qqq ppp (31, 12)N (32, 13) NNN pp p NNN p pp NN ppp (32, 12) (321, 123) Figure 1

We say that a bitableau (T , T ) is preferred if it is allowable, the columns of T are nonincreasing, and the columns of T are nondecreasing. In the format (••) above, (T , T ) is preferred if and only if (I1 , J1 ) ≤ (I2 , J2 ) ≤ · · · ≤ (Il , Jl ). For induction purposes, we also need an ordering on bitableaux. Suppose that     I 1 J1 K 1 L1  I2 J2   K 2 L2      and (T , T ) =  . (S, S ) =  .  . ..  ..   ..  .. .  Is

Js

Kt

Lt

are bitableaux presented in the format (••). Deﬁne (S, S ) ≺ (T , T ) if and only if one of the following conditions is true. Either the shape of S is larger than the shape

170

GOODEARL AND LENAGAN

of T , relative to the lexicographic ordering on shapes, that is, |I1 |, . . . , |Is | >lex |K1 |, . . . , |Kt | . Else, the shapes of S and T coincide and (I1 , J1 ), . . . , (Is , Js )
QUANTUM DETERMINANTAL IDEALS

171

can be expressed as a linear combination of products [T | T ] of the same bidegree as P , where each (T , T ) is a bitableau of the form   R 1 C1  .. ..   . .     R g Cg     K1 K  , 1   K2 K  2  .. .. . . with |K1 | = |Rj +1 | and (K1 , K1 ) ≤ (Rj +1 , Cj +1 ). Proof. We proceed by induction on j . In view of Proposition A.3, [Rj | Cj ][Rj +1 | Cj +1 ] can be written as a linear combination of products

K1 | K1 K2 | K2 · · · Ks | Ks of the same bidegree as [Rj | Cj ][Rj +1 | Cj +1 ], such that |K1 | = |Rj +1 | and (K1 , K1 ) ≤ (Rj +1 , Cj +1 ). (Here we write any Xij occurring in a monomial M as a 1 × 1 quantum minor [i | j ].) Substituting this linear combination into P , we obtain an expression for P as a linear combination of products

[R1 | C1 ] · · · Rj −1 | Cj −1

× K1 | K1 K2 | K2 · · · Ks | Ks Rj +2 | Cj +2 · · · [Rl | Cl ] with the same bidegree as P . After expanding each of [Rj +2 | Cj +2 ], . . . , [Rl | Cl ] as a linear combination of monomials, we can express P as a linear combination of products

[R1 | C1 ] · · · Rj −1 | Cj −1 K1 | K1 K2 | K2 · · · Kt | Kt (†) of the same bidegree as P , such that |K1 | = |Rj +1 | and (K1 , K1 ) ≤ (Rj +1 , Cj +1 ), while |Ki | = 1 for i > 1. If either j = 1 or |Rj −1 | ≥ |K1 |, these products can be written in the form [T | T ] for bitableaux (T , T ) of the desired type, and we are done. If j > 1 and |Rj −1 | < |K1 |, the induction hypothesis applies to each of the products (†); after collecting terms, we are again done. 1.7.

Recall the (nontotal) ordering ≺ on bitableaux deﬁned in 1.3.

Lemma. Let (S, S ) be a bitableau with bicontent γ and top row (I1 , J1 ), and suppose that (S, S ) is not preferred. (a) (S, S ) is not minimal with respect to ≺ among bitableaux with bicontent γ .

172

GOODEARL AND LENAGAN

(b) [S | S ] can be expressed as a linear combination of products [T | T ], where each (T , T ) is a bitableau with bicontent γ such that (T , T ) ≺ (S, S ). Further, each (T , T ) can be chosen with a top row (X1 , Y1 ) such that either |X1 | > |I1 | or (X1 , Y1 ) ≤ (I1 , J1 ). Although part (a) can be obtained as a consequence of part (b), we ﬁnd it clearer to give an explicit proof of (a). Proof. Let δ denote the bidegree of [S | S ]. Since (S, S ) is not preferred, it must have at least two rows. Write   I 1 J1  I2 J2    (S, S ) =  . ..   .. . Il

Jl

in the format (••) of 1.3. Then either Ij ≤r Ij +1 or Jj ≤c Jj +1 for some j . Case I. Suppose that Ij ≤r Ij +1 for some j . We may assume that j is minimal with respect to this property, so that I1 ≤r I2 ≤r · · · ≤r Ij . Write Ij = a 1 > a 2 > · · · > a α and Ij +1 = b1 > b2 > · · · > bβ . Then α ≥ β (by the shape of S), but ai < bi for some i ≤ β. We may assume that i is minimal, so that a1 ≥ b1 , . . . , ai−1 ≥ bi−1 . Set A1 = a1 > a2 > · · · > ai−1 , A2 = bi+1 > · · · > bβ , and K = b 1 > · · · > b i > ai > · · · > a α . (a) Since {b1 , . . . , bi } has one more element than A1 , there must be an index p ≤ i such that bp ∈ / A1 . In addition, bp ≥ bi > ai > · · · > aα , and so bp ∈ / Ij . Similarly, there is an index q ≥ i such that aq ∈ / Ij +1 , and bp ≥ bi > ai ≥ aq . Now set Ij = Ij ∪ {bp } \ {aq } and Ij +1 = Ij +1 ∪ {aq } \ {bp }.

QUANTUM DETERMINANTAL IDEALS

173

Observe that Ij and Ij +1 have the same cardinalities as Ij and Ij +1 , respectively. Further, Ij ∪ Ij +1 = Ij ∪ Ij +1 , and Ij aq . Set 

I1  ..  .  Ij −1   I  (R, R ) =  j Ij +1  Ij +2   ..  . Il

 J1 ..  .   Jj −1   Jj    Jj +1   Jj +2   ..  .  Jl

and note that (R, R ) is a bitableau with the same shape and bicontent as (S, S ). Since Ij

± q • A1 K | Jj K A2 | Jj +1 K=K K

(†)

=

Jν =Jν Jν

±q • A1 | Jj K | Jj Jj+1 A2 | Jj +1

with all terms of the same bidegree. Note that [Ij | Jj ][Ij +1 | Jj +1 ] occurs on the left-hand side of (†) when K = {ai > · · · > aα } and K = {b1 > · · · > bi }. In any other term on the left, K contains at least one of b1 , . . . , bi , from which we see that A1 K

±q • [I1 | J1 ] · · · Ij −1 | Jj −1 A1 K | Jj K A2 | Jj +1

K=K K

=

× Ij +2 | Jj +2 · · · [Il | Jl ]

±q • [I1 | J1 ] · · · Ij −1 | Jj −1 A1 | Jj K | Jj Jj+1 A2 | Jj +1

Jν =Jν Jν (ν=j,j +1)

× Ij +2 | Jj +2 · · · [Il | Jl ]

with all terms of bidegree δ. On the left-hand side, a term ±q • [S | S ] occurs once, and all other terms are of the form ±q • [T | S ] with (T , S ) ≺ (S, S ). Moreover, if j > 1 the top row of (T , S ) equals (I1 , J1 ). If j = 1, the top row of (T , S ) equals (A1 K , J1 ); in this case (A1 K , J1 ) < (I1 , J1 ) because A1 K
174

GOODEARL AND LENAGAN

Note that |A1 | = i − 1 < α while |K| = α + 1. Let g < j be the largest index such that |Ig | ≥ α + 1, or g = 0 if |I1 | ≤ α. Applying Lemma 1.6 to each term, we can express the right-hand side of (∗) as a linear combination of products [T | T ] of bidegree δ where each (T , T ) is a bitableau of the form 

I1  ..  .   Ig   K1  K  2 .. .

 J1 ..  .   Jg  , K1   K2   .. .

with |K1 | = α +1. Since |Ig+1 | ≤ α, the shape of T is larger than the shape of S, and so (T , T ) ≺ (S, S ). If g ≥ 1, the top row of (T , T ) equals (I1 , J1 ), while if g = 0, the top row of (T , T ) equals (K1 , K1 ), and in this case |K1 | = α +1 > α ≥ |I1 |. This establishes part (b) in case I. Case II. Suppose that Jj ≤c Jj +1 for some j . This case can be handled in the same manner as case I, by using B.2(a) rather than B.2(b). Corollary 1.8. Let (S, S ) be a bitableau with bicontent γ and top row (I1 , J1 ). Then [S | S ] can be expressed as a linear combination of products [T | T ] where we have the following: (a) (T , T ) is a preferred bitableau with bicontent γ ; (b) (T , T ) has a top row (X1 , Y1 ) such that either |X1 | > |I1 | or (X1 , Y1 ) ≤ (I1 , J1 ). Proof. This follows from Lemma 1.7 by induction with respect to ≺. Part (b) of this corollary is a weak form of the straightening law for a classical standard basis and is useful in computing preferred bases for certain ideals of ᏻq (Mn (k)). Theorem 1.9. Let δ = (r1 , . . . , rn , c1 , . . . , cn ) ∈ (Z+ )n × (Z+ )n , and let V be the homogeneous component of Ꮽ with bidegree δ. Set γ = (1r1 2r2 · · · nrn , 1c1 2c2 · · · ncn ). The products [T | T ] form a basis for V , as (T , T ) runs over all preferred bitableaux with bicontent γ . Proof. Observe that [S | S ] ∈ V for all bitableaux (S, S ) with bicontent γ , and that there are only ﬁnitely many such bitableaux. Further, such products [S | S ] include all monomials Xi1 j1 Xi2 j2 · · · Xir jr with bidegree δ, and these monomials span V . Hence, Corollary 1.8 implies that V is spanned by the products [T | T ] as (T , T ) runs over all preferred bitableaux with bicontent γ . It remains to show that these products are linearly independent. To see this, it sufﬁces to prove that the number of preferred bitableaux with bicontent γ is equal to the dimension of V .

QUANTUM DETERMINANTAL IDEALS

175

We may write Ꮽ as an iterated Ore extension with the variables Xij in the order Xnn , Xn,n−1 , . . . , Xn1 , Xn−1,n , Xn−1,n−1 , . . . , Xn−1,1 , . . . , X1n , X1,n−1 , . . . , X11 . Hence, Ꮽ has a basis consisting of monomials Xi1 j1 Xi2 j2 · · · Xir jr satisfying the following conditions: (a) i1 ≥ i2 ≥ · · · ≥ ir ; (b) jl ≥ jl+1 , whenever il = il+1 . Since Xim Xij = q −1 Xij Xim when m > j , we can reverse any product of generators with the same row index, at the cost of a nonzero scalar coefﬁcient. Hence, Ꮽ has a basis Ꮾ consisting of monomials Xi1 j1 Xi2 j2 · · · Xir jr such that we have the following: (a) i1 ≥ i2 ≥ · · · ≥ ir ; (b ) jl ≤ jl+1 , whenever il = il+1 . Note that under conditions (a) and (b ), the list i1 j1 , . . . , ir jr of double indices is in lexicographic order, provided we write our row alphabet in reverse order (i.e., n, n−1, . . . , 1) while keeping our column alphabet 1, 2, . . . , n in the usual order. With this convention, the monomials in Ꮾ are in bijection with those two-rowed matrices i 1 i2 · · · i r j1 j2 · · · j r having entries from {1, . . . , n} and columns in lexicographic order. Note that the monomial Xi1 j1 Xi2 j2 · · · Xir jr has bidegree δ if and only if the pair of multisets ({i1 , . . . , ir }, {j1 , . . . , jr }) coincides with γ . By the Robinson-Schensted-Knuth theorem [5, p. 40], the two-rowed matrices corresponding to monomials from Ꮾ, with bidegree δ, are in bijection with standard bitableaux (Q, P ) of bicontent γ . In this result, standard tableaux are required to be nondecreasing on each row and strictly increasing on each column, relative to the total orders given on the two alphabets. Note that this means Q has nonincreasing rows and strictly decreasing columns, relative to the usual ordering of integers. Thus, the ordering conditions on (Q, P ) hold precisely when the pair (Qtr , P tr ) is preferred in our sense 1.3. Therefore, there exists a bijection between the monomials of bidegree δ in Ꮾ and the preferred bitableaux with bicontent γ . Since the former make a basis for V , we conclude that the number of preferred bitableaux with bicontent γ is precisely dimk V , as required. Corollary 1.10. The products [T | T ], as (T , T ) runs over all preferred bitableaux, form a basis for Ꮽ = ᏻq (Mn (k)). 1.11. The existence of analogous bases for rectangular quantum matrix algebras follows easily from Corollary 1.10. For m < n, we may deﬁne ᏻq (Mm,n (k)) as the ksubalgebra of ᏻq (Mn (k)) generated by the Xij with i ≤ m; the case m > n is handled by writing ᏻq (Mm,n (k)) as a subalgebra of ᏻq (Mm (k)). Note that in the ﬁrst case,

176

GOODEARL AND LENAGAN

there is a k-algebra retraction π : ᏻq (Mn (k)) → ᏻq (Mm,n (k)) such that π(Xij ) = Xij when i ≤ m and π(Xij ) = 0 when i > m. Corollary. Let m, n be any positive integers, and let Ꮾm,n be the set of all products [T | T ] where (T , T ) runs over all preferred bitableaux in which the entries of T lie in {1, . . . , m}, while the entries of T lie in {1, . . . , n}. Then Ꮾm,n is a basis for ᏻq (Mm,n (k)). Proof. We prove only the case (m < n); the other case is identical. By Corollary 1.10, the set Ꮾn,n is a basis for ᏻq (Mn (k)). On one hand, Ꮾm,n ⊆ Ꮾn,n and so Ꮾm,n is linearly independent. On the other hand, π(Ꮾn,n ) = Ꮾm,n ∪{0}, and therefore, Ꮾm,n spans ᏻq (Mm,n (k)). 2. One-parameter quantum determinantal ideals. In this section, we prove that quantum determinantal ideals in ᏻq (Mm,n (k)) are completely prime. The case of the ideal generated by all 2 × 2 quantum minors has been proved by Rigal [15], using different methods. 2.1. As in the previous section, we ﬁx n ≥ 2, a ﬁeld k, a scalar q ∈ k × , and set [t] Ꮽ = ᏻq (Mn (k)). Fix t ∈ {1, . . . , n − 1}, and let Ᏽt = Iq (Mn (k)) denote the ideal of Ꮽ generated by all (t +1)×(t +1) quantum minors. Again, it is convenient to remain with this case until the main result is proved, and to derive the corresponding result for ᏻq (Mm,n (k)) as an easy corollary. We proceed by establishing a quantized version of the theorem stating that, in the classical case, Ᏽt equals the kernel of the k-algebra homomorphism µ∗ : ᏻ Mn (k) −→ ᏻ Mn,t (k) × Mt,n (k) = ᏻ Mn,t (k) ⊗ ᏻ Mt,n (k) discussed in the introduction. First, some labels. Set

Ꮽnt = ᏻq Mn,t (k) = k Xij | j ≤ t ⊆ Ꮽ

and Ꮽtn = ᏻq Mt,n (k) = k Xij | i ≤ t ⊆ Ꮽ.

For τ = nt or tn, let πτ : Ꮽ → Ꮽτ denote the natural k-algebra retraction. Thus Xij (j ≤ t), Xij (i ≤ t), πtn Xij = πnt Xij = 0 (j > t); 0 (i > t). The kernels of these homomorphisms are the ideals Xij | j > t and Xij | i > t, respectively. Finally, deﬁne the k-algebra homomorphism

πnt ⊗πtn

θt : Ꮽ −→ Ꮽ ⊗ Ꮽ −−−−−→ Ꮽnt ⊗ Ꮽtn ,

QUANTUM DETERMINANTAL IDEALS

177

where denotes the comultiplication on the bialgebra Ꮽ. By [13, (1.9)], comultiplication of quantum minors is given by the rule [I | J ] = [I | K] ⊗ [K | J ]. |K|=|I |

Since all (t + 1) × (t + 1) quantum minors are killed by πnt , we see that Ᏽt ⊆ ker θt . We prove equality in Proposition 2.4. Note that any product [T | T ] for which the shape of T has more than t columns lies in Ᏽt . Hence, Ꮽ/Ᏽt is spanned by the images of those products [T | T ] indexed by preferred bitableaux (T , T ) with shapes having at most t columns. 2.2. Consider an allowable bitableau (T , T ). For l = 1, . . . , n, let ρl (T ) be the number of rows of T of length ≥ l, and set ρ(T ) = (ρ1 (T ), ρ2 (T ), . . . , ρn (T )). Let µ(T ) and µ (T ) denote the tableaux with the same shape as T and entries as follows: each row of length l is ﬁlled 1, 2, . . . , l in µ(T ) and is ﬁlled l, l −1, . . . , 1 in µ (T ). If (T , T ) is preferred, then (T , µ(T )) and (µ (T ), T ) are preferred bitableaux. For any homogeneous element x ∈ Ꮽ, label the bidegree of x as r(x), c(x) = r1 (x), r2 (x), . . . , rn (x), c1 (x), c2 (x), . . . , cn (x) . Thus, with respect to the usual Poincaré-Birkhoff-Witt (PBW) basis of ordered monomials, rl (x) records the number of Xl? factors in each monomial in x, and cl (x) the number of X?l factors. For [T | T ] as in the previous paragraph, rl [T | T ] is the number of l’s in T and cl [T | T ] is the number of l’s in T . Note that cl [T | µ(T )] = rl [µ (T ) | T ] = ρl (T ). We write

θt T | T = T | µ(T ) ⊗ µ (T ) | T + Xi ⊗ Y i , i

where the Xi and Yi are homogeneous with c(Xi ) = r(Yi ) >rlex ρ(T ). Proof. Write



I1  I2  (T , T ) =  .  .. Is

where the (Ij , Jj ) are index pairs. Then

θt I j | J j = |K|=|Ij | K⊆{1,...,t}

 J1 J2   ..  , . Js

Ij | K ⊗ K | J j

178

GOODEARL AND LENAGAN

for each j . Hence, θt [T | T ] is the sum of all possible terms

Xi ⊗ Yi = I1 | K1 I2 | K2 · · · Is | Ks ⊗ K1 | J1 K2 | J2 · · · Ks | Js , where each Kj ⊆ {1, . . . , t} and |Kj | = |Ij |. Obviously Xi and Yi are homogeneous. Let i = i0 label the special case where Kj = {1, 2, . . . , |Ij |} for all j . This yields the term Xi0 ⊗ Yi0 = [T | µ(T )] ⊗ [µ (T ) | T ]. Now assume that i = i0 . Obviously c(Xi ) = r(Yi ), so it remains to show that c(Xi ) >rlex ρ(T ). Note that cl (Xi ) = ρl (T ) = 0 for l > t. We claim that if cl (Xi ) = ρl (T ) for l = n, n−1, . . . , h+1, then Kj = {1, 2, . . . , |Ij |} for all j such that |Ij | ≥ h. This is vacuously true for h > t. Now suppose that h ≤ t and that Kj = {1, 2, . . . , |Ij |} whenever |Ij | > h. For l > h, there are ρl (T ) indices j for which |Ij | ≥ l, and l ∈ Kj for each such j . Since cl (Xi ) = ρl (T ), this uses up all the available column l’s in Xi , and so l ∈ / Kj for any j with |Ij | < l. Thus, Kj ⊆ {1, 2, . . . , h} for all j with |Ij | ≤ h. In particular Kj = {1, 2, . . . , h} for all j with |Ij | = h, verifying the induction step. This establishes the claim. Since we are in the case i = i0 , we cannot have Kj = {1, 2, . . . , |Ij |} for all j , and so the claim shows that we cannot have cl (Xi ) = ρl (T ) for all l. Hence, there is an index g ≥ 1 such that cg (Xi ) = ρg (T ), while cl (Xi ) = ρl (T ) for all l > g. By the claim, Kj = {1, 2, . . . , |Ij |} for all j such that |Ij | ≥ g. Hence, g ∈ Kj for all j with |Ij | ≥ g, and so cg (Xi ) ≥ ρg (T ). By our choice of g, we thus must have cg (Xi ) > ρg (T ). Therefore c(Xi ) >rlex ρ(T ), as required. Proposition 2.4. Ᏽt = ker θt . Proof. If ker θt properly contains Ᏽt , then ker θt contains a nonzero element of the form m

αi Ti | Ti , x= i=1

where the αi are nonzero scalars and the (Ti , Ti ) are distinct preferred bitableaux with shapes having at most t columns. Let ρ be the minimum of the n-tuples ρ(Ti ) under reverse lexicographic order. Without loss of generality, there exists m such that ρ(Ti ) = ρ for i ≤ m and ρ(Ti ) >rlex ρ for i > m . Applying Lemma 2.3 to each θt [Ti | Ti ] and collecting terms, we see that

0 = θt (x) =

m i=1

αi Ti | µ(Ti ) ⊗ µ (Ti ) | Ti + Xj ⊗ Y j , j

where the Xj and Yj are homogeneous elements with c(Xj ) = r(Yj ) >rlex ρ. Since c[Ti | µ(Ti )] = ρ for i ≤ m , all of the Xj belong to different homogeneous components than the [Ti | µ(Ti )] for i ≤ m . Consequently,

m i=1

αi Ti | µ(Ti ) ⊗ µ (Ti ) | Ti = 0.

QUANTUM DETERMINANTAL IDEALS

179

For 1 ≤ i < j ≤ m , either Ti = Tj or Ti = Tj , so (Ti , µ(Ti )) = (Tj , µ(Tj )) or (µ (Ti ), Ti ) = (µ (Tj ), Tj ). Thus, it follows from the linear independence of the preferred products [• | •] in the algebras Ꮽnt and Ꮽtn (Corollary 1.11) that the terms [Ti | µ(Ti )]⊗[µ (Ti ) | Ti ] are linearly independent. But then αi = 0 for i = 1, . . . , m , contradicting our assumptions. Theorem 2.5. Ꮽ/Ᏽt = ᏻq (Mn (k))/Iq[t] (Mn (k)) is an integral domain. Proof. By Proposition 2.4, Ꮽ/Ᏽt embeds in Ꮽnt ⊗ Ꮽtn . Now Ꮽnt and Ꮽtn are iterated Ore extensions of k, with respect to k-algebra automorphisms and k-linear skew derivations. In particular, both of these algebras are domains. Further, Ꮽnt ⊗ Ꮽtn is an iterated Ore extension of Ꮽnt , and so it is a domain too. Therefore, Ꮽ/Ᏽt is a domain. Corollary 2.6. Let m, n, t be any positive integers such that t < min{m, n}, and let Iq[t] (Mm,n (k)) be the ideal of ᏻq (Mm,n (k)) generated by all (t + 1) × (t + 1) quantum minors. Then ᏻq (Mm,n (k))/Iq[t] (Mm,n (k)) is an integral domain. Proof. Consider the case (m < n), and put I = Iq[t] (Mm,n (k)) and J = Iq[t] (Mn (k)). Obviously I ⊆ J ∩ ᏻq (Mm,n (k)). For the reverse inclusion, we use the retraction π : ᏻq (Mn (k)) → ᏻq (Mm,n (k)) discussed in 1.11. Note that the image of any quantum minor [X | Y ] under π is either [X | Y ] or 0, and hence π(J ) ⊆ I . Since π is the identity on ᏻq (Mm,n (k)), it follows that I = J ∩ ᏻq (Mm,n (k)). Therefore, the corollary follows from Theorem 2.5. 3. Twisting. Artin, Schelter, and Tate showed in [1] that multiparameter quantum matrix algebras ᏻλ,p (Mn (k)) can be obtained from the one-parameter versions by a process of twisting by 2-cocycles. In this section, we recall some details of this process and determine its effect on quantum minors. 3.1. Let k be a ﬁeld, and let p = (pij ) be a multiplicatively antisymmetric matrix −1 over k × , that is, pii = 1 and pj i = pij for all i, j . Let λ ∈ k × \ {−1}. The algebra ᏻλ,p (Mn (k)) is the k-algebra with generators Yij for i, j = 1, . . . , n and the following relations:   pli pj m Yij Ylm + (λ − 1)pli Yim Ylj , when l > i and m > j ; Ylm Yij = λpli pj m Yij Ylm , when l > i and m ≤ j ;   pj m Yij Ylm , when l = i and m > j . We denote this algebra Ꮽλ,p for short. 3.2. The quantum exterior algebra 8p = 8p (k n ) is the k-algebra with generators η1 , . . . , ηn and relations ηi2 = 0,

ηi ηj = −pij ηj ηi

180

GOODEARL AND LENAGAN

for all i, j . For any subset I ⊆ {1, . . . , n}, write the elements of I in ascending order, say I = {i1 < i2 < · · · < ir }, and set ηI = ηi1 ηi2 · · · ηir . By convention, η∅ = 1. The elements ηI form a k-basis for 8p . As is well known (and easily checked), there is a k-algebra homomorphism Lλ,p : 8p −→ Ꮽλ,p ⊗ 8p such that Lλ,p (ηi ) = j Yij ⊗ ηj for all i (cf. [12, Chapter 6, Theorem 3], [1, (11) and (12)]). For any nonempty subset I ⊆ {1, . . . , n}, the image of ηI under Lλ,p has the form Lλ,p (ηI ) = UI J ⊗ η J |J |=|I |

(see [1, Lemma 1]). The elements UI J ∈ Ꮽλ,p are unique due to the linear independence of the ηJ . Each UI J is the quantum minor corresponding to the rows i ∈ I and columns j ∈ J ; we use the notation Dλ,p (I, J ) = UI J to indicate the dependence on the parameters λ, pij . Explicit formulas for Dλ,p (I, J ) are given in [1, Lemma 1]. 3.3. The quantum minors in Ꮽq = ᏻq (Mn (k)) can be obtained as in 3.2, of course. Since we must consider both settings simultaneously, let us use ξi for the generators of the quantum exterior algebra in this case. Thus, 8q = 8q (k n ) is the k-algebra with generators ξ1 , . . . , ξn and relations ξi2 = 0

(all i),

ξj ξi = −qξi ξj

(i < j ).

There is a basis consisting of elements ξI = ξi1 ξi2 · · · ξir where I = {i1 < i2 < · · · < ir } runs through all subsets of {1, . . . , n}. There is a k-algebra homomorphism Lq : 8q → Ꮽq ⊗ 8q such that Lq (ξi ) = j Xij ⊗ ξj for all i. The quantum minors in Ꮽq , which we now denote Dq (I, J ) to indicate the dependence on q, arise in the formulas Lq (ξI ) = Dq (I, J ) ⊗ ξJ |J |=|I |

(cf. [14, Lemma 4.4.2]; [13, Remark, p. 36]). 3.4. As observed in [1, p. 889], the algebra Ꮽλ,p can be obtained as a cocycle twist of Ꮽq provided λ = q −2 (we take q −2 rather than q 2 to account for the difference q ↔ q −1 between [1, (43)] and our choice of relations for Ꮽq ). We thus carry out the twisting process under the assumption that λ has a square root in k; the general cases of our results require a passage to k. Since we must simultaneously work with twists

QUANTUM DETERMINANTAL IDEALS

181

of Ꮽq , 8q , and a subalgebra of Ꮽq ⊗ 8q , it is helpful to give the appropriate cocycle explicitly. For the remainder of this section, p = (pij ) is an arbitrary multiplicatively antisymmetric matrix over k × , and we take λ = q −2 for some (ﬁxed) q ∈ k × . Deﬁne a map c : Zn × Zn → k × by the rule a b c (a1 , . . . , an ), (b1 , . . . , bn ) = qpj i i j . i>j

Then c is a multiplicative bicharacter on Zn (that is, c(a + a , b) = c(a, b)c(a , b), and similarly in the second variable), and hence also a 2-cocycle. Note that qpj i (i > j ), c(i , j ) = 1 (i ≤ j ), where 1 , . . . , n denotes the standard basis for Zn . 3.5. Recall the Zn × Zn -bigrading on Ꮽq from 1.5. Following [1, Theorem 4], we simultaneously twist Ꮽq on the left by c−1 and on the right by c. This results in a new algebra, denoted Ꮽq , as follows. As a graded vector space, Ꮽq is isomorphic to Ꮽq via an isomorphism a ! → a . The multiplication in Ꮽq is given by a b = c(u1 , v1 )−1 c(u2 , v2 )(ab) for homogeneous elements a, b ∈ Ꮽq of bidegrees (u1 , u2 ) and (v1 , v2 ). In particular,   pil pmj Xij Xlm (i > l, j > m),     q −1 pil Xij Xlm (i > l, j ≤ m), Xlm = Xij  (i ≤ l, j > m), qpmj Xij Xlm      Xij Xlm (i ≤ l, j ≤ m). Observe that 8q has a natural Zn -grading, where ξi has degree i . We twist 8q by to obtain a new algebra 8q . Note that

c−1

ξi ξj

=

q −1 pij (ξi ξj ) (ξi ξj )

(i > j ), (i ≤ j ).

Lemma 3.6. There are k-algebra isomorphisms φ : Ꮽλ,p −→ Ꮽq

and

ψ : 8p −→ 8q

and ψ(η ) = ξ for all i, j . such that φ(Yij ) = Xij i i

182

GOODEARL AND LENAGAN

Proof. The existence of φ follows from [1, Theorem 4], and the existence of ψ is ∈ Ꮽ and ξ ∈ 8 proved in the same manner. One ﬁrst checks that the elements Xij q q i satisfy the same relations as the Yij and the ηi . For instance, for i < j we have ξj ξi = −qξi ξj and so ξj ξi = q −1 pj i (ξj ξi ) = −pj i (ξi ξj ) = −pj i ξi ξj . and Hence, there exist k-algebra homomorphisms φ and ψ sending Yij ! → Xij ηi ! → ξi . Since Ꮽq has a basis of ordered monomials Xi1 j1 · · · Xit jt , and since each (Xi1 j1 · · · Xit jt ) is a nonzero scalar multiple of Xi1 j1 · · · Xit jt , we see that Ꮽq has a basis of ordered monomials Xi1 j1 · · · Xit jt . In addition, Ꮽλ,p has a basis of ordered monomials Yi1 j1 · · · Yit jt , which φ maps to the Xi1 j1 · · · Xit jt . Therefore φ is an isomorphism, and similarly so is ψ.

3.7.

There is a Zn -graded subalgebra Ꮾq ⊆ Ꮽq ⊗ 8q , where (Ꮾq )u = (Ꮽq )uv ⊗ (8q )v v∈Zn

for u ∈ Zn . Using this grading, we twist Ꮾq by c−1 to obtain a new algebra Ꮾq . Note that there is a vector space embedding Ꮾq → Ꮽq ⊗ 8q where (a ⊗ b) ! → a ⊗ b for a ∈ (Ꮽq )uv and b ∈ (8q )v . We identify Ꮾq with its image in Ꮽq ⊗ 8q via this embedding. Lemma. Under the above identiﬁcation, Ꮾq is a k-subalgebra of Ꮽq ⊗ 8q . Proof. It sufﬁces to show that the product of any two homogeneous elements from

Ꮾq is the same in both algebras. Given x ∈ (Ꮾq )u1 and y ∈ (Ꮾq )u2 , we can write x = i xi and y = j yj , where each xi = ai ⊗ bi ∈ (Ꮽq )u1 vi ⊗ (8q )vi for some vi ∈ Zn and yj = dj ⊗ ej ∈ (Ꮽq )u2 wj ⊗ (8q )wj for some wj ∈ Zn . It is enough to

compare the products of any xi with any yj . Hence, there is no loss of generality in assuming that x = a ⊗ b ∈ (Ꮽq )u1 v1 ⊗ (8q )v1 and y = d ⊗ e ∈ (Ꮽq )u2 v2 ⊗ (8q )v2 . Under the product in Ꮾq , we have x y = c(u1 , u2 )−1 (xy) . On the other hand, under the product in Ꮽq ⊗ 8q , we have x y = (a ⊗ b )(d ⊗ e ) = a d ⊗ b e

= c(u1 , u2 )−1 c(v1 , v2 )(ad) ⊗ c(v1 , v2 )−1 (be) = c(u1 , u2 )−1 (ad) ⊗ (be) = c(u1 , u2 )−1 (xy) . Therefore, the two products do coincide, as required. 3.8. Observe that the k-algebra homomorphism Lq : 8q → Ꮽq ⊗ 8q actually maps 8q to Ꮾq . Viewed as a map from 8q to Ꮾq , the homomorphism Lq is homogeneous of degree 0 with respect to the Zn -gradings on these algebras. Since we have

183

QUANTUM DETERMINANTAL IDEALS

twisted both algebras by the same cocycle (namely, c−1 ), we see that Lq induces a k-algebra homomorphism Lq : 8q → Ꮾq , where Lq (a ) = Lq (a) for a ∈ 8q . The various k-algebra homomorphisms we have been discussing ﬁt into the following diagram: Lλ,p

8p

/ Ꮽλ,p ⊗ 8p

ψ ∼ =

8q

∼ = φ⊗ψ

Lq

/ Ꮾq

/ Ꮽq ⊗ 8q .

⊆

This diagram commutes, since (φ ⊗ ψ)Lλ,p (ηi ) =

φ(Yij ) ⊗ ψ(ηj ) =

j

Lq ψ(ηi ) = Lq (ξi )

=

j

j

Xij

Xij ⊗ ξj

⊗ ξj

for all i. Proposition 3.9. φ(Dλ,p (I, J )) = Dq (I, J ) for all I, J . Proof. We ﬁrst show that ψ(ηH ) = ξH for all H ⊆ {1, . . . , n}. This is clear for |H | ≤ 1. If H = {h1 < h2 < · · · < hr } = {h1 } J for some r ≥ 2, we may assume by induction that ψ(ηJ ) = ξJ . Hence, −1 ψ(ηH ) = ψ(ηh1 ηJ ) = ξh 1 ξJ = c h1 , h2 + · · · + hr (ξh1 ξJ ) . But c(h1 , h2 + · · · + hr ) = c(h1 , h2 )c(h1 , h3 ) · · · c(h1 , hr ) = 1 because h1 < h2 < · · · < hr , and so ψ(ηH ) = (ξh1 ξJ ) = ξH . This establishes the induction step for our claim. Now let I be an arbitrary nonempty subset of {1, . . . , n}. In view of the commutativity of the diagram in 3.8, φ Dλ,p (I, J ) ⊗ ξJ = (φ ⊗ ψ)Lλ,p (ηI ) = Lq ψ(ηI ) |J |=|I |

= Lq (ξI ) =

|J |=|I |

Dq (I, J ) ⊗ ξJ .

Since the ξJ are linearly independent, the proposition follows. 4. Multiparameter quantum determinantal ideals. Using the twisting method discussed in the previous section, we extend our main result from quantum determinantal ideals in one-parameter quantum matrix algebras ᏻq (Mm,n (k)) to those in multiparameter quantum matrix algebras ᏻλ,p (Mm,n (k)).

184

GOODEARL AND LENAGAN

4.1. Let Ꮽλ,p = ᏻλ,p (Mn (k)) be an arbitrary multiparameter quantum matrix algebra over an arbitrary base ﬁeld k, as in 3.1. Fix t ∈ {1, . . . , n − 1}, and let Ᏽλ,p = [t] (Mn (k)) denote the ideal of Ꮽλ,p generated by all (t + 1) × (t + 1) quantum Iλ,p minors, that is, all Dλ,p (I, J ) with |I | = |J | = t + 1. [t] Theorem. ᏻλ,p (Mn (k))/Iλ,p (Mn (k)) is an integral domain. [t] Proof. First set Ꮽλ,p = ᏻλ,p (Mn (k)) and Ᏽλ,p = Iλ,p (Mn (k)). We identify Ꮽλ,p with Ꮽλ,p ⊗ k. Since the quantum minors in Ꮽλ,p and Ꮽλ,p are the same, Ᏽλ,p = Ᏽλ,p ⊗k. As a result, Ꮽλ,p /Ᏽλ,p ∼ = (Ꮽλ,p /Ᏽλ,p )⊗k; in particular, Ꮽλ,p /Ᏽλ,p embeds in Ꮽλ,p /Ᏽλ,p . Thus, it sufﬁces to show that the latter algebra is a domain, and hence we may pass to the case where k is algebraically closed. Now there exists q ∈ k × such that q −2 = λ. Let c be the 2-cocycle deﬁned in 3.4, and set Ꮽq = ᏻq (Mn (k)). Twist Ꮽq on the left by c−1 and on the right by c as in 3.5. In view of Lemma 3.6 and Proposition 3.9, there is a k-algebra isomorphism φ : Ꮽλ,p → [t] Ꮽq such that φ(Ᏽλ,p ) = Ᏽq , where Ᏽq = Iq (Mn (k)). Thus, Ꮽλ,p /Ᏽλ,p ∼ = (Ꮽq /Ᏽq ) , a twist of Ꮽq /Ᏽq . Since Ꮽq /Ᏽq is a domain by Theorem 2.5, it only remains to check that the property of being a domain is preserved in the twist (Ꮽq /Ᏽq ) . We may view Ꮽq /Ᏽq as graded by Z2n , which can be made into a totally ordered group; then (Ꮽq /Ᏽq ) is graded by the same totally ordered group. To see that the product of any two nonzero elements of (Ꮽq /Ᏽq ) is nonzero, it sufﬁces to show that the product of their highest terms is nonzero. Hence, we just need to show that the product of any two nonzero homogeneous elements a , b ∈ (Ꮽq /Ᏽq ) is nonzero. But that is clear since a b is a nonzero scalar multiple of (ab) , while ab = 0 because Ꮽq /Ᏽq is a domain. Therefore (Ꮽq /Ᏽq ) is a domain, as required.

4.2. Just as in Corollary 2.6, the rectangular case follows directly from Theorem 4.1. Corollary. Let m, n, t be positive integers such that t < min{m, n}, and let [t] Iλ,p (Mm,n (k)) be the ideal of ᏻλ,p (Mm,n (k)) generated by all (t + 1) × (t + 1) quan[t] tum minors. Then ᏻλ,p (Mm,n (k))/Iλ,p (Mm,n (k)) is an integral domain.

4.3. The method of proof used in the previous theorem can also be applied to the other results of Sections 1 and 2. In particular, we obtain a basis of products of quantum minors for Ꮽλ,p in the following manner. Deﬁne preferred bitableaux as in 1.3. For any preferred bitableau   I 1 J1  I2 J2    (T , T ) =  . ..  ,  .. . Il Jl where (I1 , J1 ) ≤ (I2 , J2 ) ≤ · · · ≤ (Il , Jl ) are index pairs, deﬁne

QUANTUM DETERMINANTAL IDEALS

185

T | T λ,p = Dλ,p (I1 , J1 )Dλ,p (I2 , J2 ) · · · Dλ,p (Il , Jl ). Theorem. The products [T | T ]λ,p , as (T , T ) runs over all preferred bitableaux, form a basis for ᏻλ,p (Mn (k)). Proof. First note that the symbols [T | T ]λ,p stand for the same elements in the algebras Ꮽλ,p and Ꮽλ,p = ᏻλ,p (Mn (k)) = Ꮽλ,p ⊗k. If these elements form a k-basis for Ꮽλ,p , then they must also form a k-basis for Ꮽλ,p . Hence, there is no loss of generality in assuming that k is algebraically closed. Now choose q ∈ k × such that q −2 = λ, and twist Ꮽq as in 3.5. In view of Lemma 3.6 and Proposition 3.9, there is a k-algebra isomorphism φ : Ꮽλ,p → Ꮽq such that φ([T | T ]λ,p ) is a nonzero scalar multiple of [T | T ] for all preferred bitableaux (T , T ). Since the products [T | T ] form a basis for Ꮽq by Corollary 1.10, the theorem follows. Appendices Appendix A. Commutation relations. We derive some commutation relations for quantum minors in ᏻq (Mn (k)), expressed using the notation and conventions of 1.1–1.4. A.1. We begin by restating some identities from [14], given there for generators and maximal minors, in a form that applies to minors of arbitrary size. Note the difference between our choice of relations for ᏻq (Mn (k)) (see 1.1) and that in [14, p. 37]. Because of this, we must interchange q and q −1 whenever carrying over a formula from [14]. Lemma. Let r, c ∈ {1, . . . , n} and I, J ⊆ {1, . . . , n} with |I | = |J | ≥ 1. (a) If r ∈ I and c ∈ J , then Xrc [I | J ] = [I | J ]Xrc . (b) If r ∈ I and c ∈ / J , then

Xrc [I | J ] − q −1 [I | J ]Xrc = q −1 − q (−q)−|J ∩[c,j ]| I | J ∪ {c} \ {j } Xrj . j ∈J j >c

(c) If r ∈ / I and c ∈ J , then

Xrc [I | J ] − q[I | J ]Xrc = q − q −1 (−q)|I ∩[i,r]| I ∪ {r} \ {i} | J Xic . i∈I i
(d) If r ∈ / I and c ∈ / J , then Xrc [I | J ] − [I | J ]Xrc = 1 − q −2 





 |I ∩[i,r]| × I ∪{r} \{i} | J Xic − (−q)|J ∩[c,j ]| Xrj I | J ∪{c} \{j }   (−q) . i∈I i
j ∈J j >c

186

GOODEARL AND LENAGAN

Proof. Let t = |I | = |J |. (a) There is a k-algebra isomorphism

∼ =

ᏻq Mt (k) −→ k Xij | i ∈ I, j ∈ J ⊆ ᏻq Mn (k) ,

which sends the quantum determinant of ᏻq (Mt (k)) to [I | J ]. Since the quantum determinant is central in ᏻq (Mt (k)), part (a) follows. (b) Pick r0 ∈ {1, . . . , n}\I . Set I = I ∪{r0 } and J = J ∪{c}, and label the elements of these sets in ascending order, say, I = i1 < i2 < · · · < it+1 J = j1 < j2 < · · · < jt+1 . and There exists a k-algebra embedding φ : ᏻq (Mt+1 (k)) → ᏻq (Mn (k)) such that φ(Xab ) = Xia jb for a, b = 1, . . . , t + 1. Let ρ, γ , σ be the indices such that iρ = r, jγ = c, and iσ = r0 . Then φ(Xργ ) = Xrc and φ(A(σ γ )) = [I | J ], where A(σ γ ) is (in the notation of [14]) the t × t quantum minor in ᏻq (Mt+1 (k)) obtained by deleting the σ th row and γ th column. By the second part of [14, 4.5.1(2)], Xργ A(σ γ ) − q −1 A(σ γ )Xργ = q −1 − q (†) (−q)γ −δ A(σ δ)Xρδ . δ>γ

Note that δ − γ = |J ∩ (c, jδ ]| = |J ∩ [c, jδ ]| for δ = γ + 1, . . . , t + 1. Thus, part (b) results from applying φ to (†). (c)(d) These follow, in the same manner, from the ﬁrst part of [14, 4.5.1(4)] and the ﬁrst part of [14, 5.1.2], respectively. Corollary A.2. Let r, c ∈ {1, . . . , n} and I, J ⊆ {1, . . . , n} with |I | = |J |. Then the term Y := Xrc [I | J ] − q δ(c,J )−δ(r,I ) [I | J ]Xrc is a linear combination of terms [I | J ]Xij , with the same bidegree as Xrc [I | J ], such that |I | = |I | and (I , J ) < (I, J ). Proof. We allow the trivial case I = J = ∅ for completeness. Now assume that |I | = |J | ≥ 1. The cases in which r ∈ I or c ∈ J (or both) are clear from the ﬁrst three parts of Lemma A.1. Hence, we may assume that r ∈ / I and c ∈ / J. If the corollary fails, we may suppose that we have a counterexample in which J is minimal with respect to ≤c . By Lemma A.1(d), Y is a linear combination of the following: (i) terms [I | J ]Xic of the desired form; (ii) terms Xrj [I | J ∪ {c} \ {j }] with j ∈ J and j > c. Note that the terms in (ii) have the form Xrj [I | J ] with the same bidegree as Xrc [I | J ], and with J
QUANTUM DETERMINANTAL IDEALS

187

as Xrj [I | J ], such that |I | = |I | and (I , J ) < (I, J ) < (I, J ). But if these expressions for the terms in (ii) are substituted in our initial expression for Y , we have written Y in the desired form, contradicting the assumption of a counterexample. Therefore, the corollary holds. Proposition A.3. Let R, C, I, J ⊆ {1, . . . , n} with |R| = |C| and |I | = |J |. If M is any element of ᏻq (Mn (k)) of bidegree (χR , χC ), then the term Z := M[I | J ] − q |C∩J |−|R∩I | [I | J ]M is a linear combination of terms [I | J ]M such that (a) [I | J ]M has the same bidegree as M[I | J ]; (b) M is a monomial of length |R|; (c) |I | = |I | and (I , J ) < (I, J ). In particular, this holds for M = [R | C]. Proof. The proposition holds trivially if either R, C or I, J are empty. Now assume that R, C, I, J are all nonempty. Write M as a linear combination of monomials Ml with length |R| and bidegree (χR , χC ). If each of the terms Zl := Ml [I | J ] − q |C∩J |−|R∩I | [I | J ]Ml is a linear combination of terms [I | J ]M satisfying (a), (b), and (c), then so is Z. Thus, we may assume that M is a monomial. We now induct on the length of M, namely, |R|. The case |R| = 1 is given by Corollary A.2. If |R| > 1, write M = Xrc N for some r, c and some monomial N of length |R|−1. Note that N has bidegree (χQ , χB ) where Q = R \{r} and B = C \{c}. By induction, (1)

N[I | J ] = q |B∩J |−|Q∩I | [I | J ]N + lin. comb. of terms [I1 | J1 ]N1

such that • [I1 | J1 ]N1 has the same bidegree as N [I | J ]; • N1 is a monomial of length |R| − 1; • |I1 | = |I | and (I1 , J1 ) < (I, J ). Multiplying (1) on the left by Xrc , we obtain (∗)

M[I | J ] = q |B∩J |−|Q∩I | Xrc [I | J ]N + lin. comb. of terms Xrc [I1 | J1 ]N1 .

Next, apply Corollary A.2 to both Xrc [I | J ] and Xrc [I1 | J1 ]. In the ﬁrst case, (2)

Xrc [I | J ] = q δ(c,J )−δ(r,I ) [I | J ]Xrc + lin. comb. of terms [I2 | J2 ]Xij

such that [I2 | J2 ]Xij has the same bidegree as Xrc [I | J ], while |I2 | = |I | and (I2 , J2 ) < (I, J ). Since |B ∩ J | − |Q ∩ I | + δ(c, J ) − δ(r, I ) = |C ∩ J | − |R ∩ I |,

188

GOODEARL AND LENAGAN

it follows that (†)

q |B∩J |−|Q∩I | Xrc [I | J ]N = q |C∩J |−|R∩I | [I | J ]M + lin. comb. of terms [I2 | J2 ]Xij N.

In the second case, for each term Xrc [I1 | J1 ] we have an expression of the following type, where we incorporate the [I1 | J1 ]Xrc term with the remaining terms: (3)

Xrc [I1 | J1 ] = lin. comb. of terms [I3 | J3 ]Xst ,

such that [I3 | J3 ]Xst has the same bidegree as Xrc [I1 | J1 ], while |I3 | = |I1 | = |I | and (I3 , J3 ) ≤ (I1 , J1 ) < (I, J ). Consequently, (‡)

Xrc [I1 | J1 ]N1 = lin. comb. of terms [I3 | J3 ]Xst N1 .

Finally, substitute (†) and (‡) in (∗), which yields (∗∗)

M[I | J ] = q |C∩J |−|R∩I | [I | J ]M + lin. comb. of terms [I2 | J2 ]Xij N and [I3 | J3 ]Xst N1 .

Observe that the terms [I2 | J2 ]Xij N and [I3 | J3 ]Xst N1 have the same bidegree as M[I | J ], and that the terms Xij N and Xst N1 are monomials of length |R|. We already have |I2 | = |I3 | = |I | while (I2 , J2 ) < (I, J ) and (I3 , J3 ) < (I, J ). Therefore, (∗∗) gives us the desired relation. Appendix B. Laplace and exchange relations. We adapt some of the relations derived in [13]. (Although the base ﬁeld is taken to be C in that paper, the arguments are valid over any ﬁeld.) For subsets I, J ⊆ {1, . . . , n}, set H(I ; J ) = (i, j ) ∈ I × J | i > j . In the following formulas, we use to denote disjoint unions. Notation and conventions from 1.1–1.4 are again in force. Lemma B.1 (Laplace expansions). Let I, J ⊆ {1, . . . , n} with |I | = |J |. (a) If J = J1 J2 , then (−q)H(I1 ;I2 ) [I1 | J1 ][I2 | J2 ]. [I | J ] = (−q)−H(J1 ;J2 ) I1 I2 =I |I1 |=|J1 |

(b) If I = I1 I2 , then [I | J ] = (−q)−H(I1 ;I2 )

(−q)H(J1 ;J2 ) [I1 | J1 ][I2 | J2 ].

J1 J2 =J |J1 |=|I1 |

Proof. The nontrivial cases (J1 , J2 = ∅ in (a), and I1 , I2 = ∅ in (b)) are given in [13, Proposition 1.1]. The trivial cases are clear.

QUANTUM DETERMINANTAL IDEALS

189

B.2. The sums in the next formulas run over certain partitions of index sets; we take these sums to run over only those partitions for which the terms in the formulas are deﬁned. For instance, in part (a) the only allowable partitions K K = K are those for which J1 ∩K = J2 ∩K = ∅ while |J1 |+|K | = |I1 | and |K |+|J2 | = |I2 |. Observe that in each formula, all terms on both sides of the equation have the same bidegree. Proposition (Exchange formulas). Let I1 , I2 , J1 , J2 , K ⊆ {1, . . . , n}. (a) If |Jν | ≤ |Iν | and |K| = |I1 | + |I2 | − |J1 | − |J2 |, then (∗)

(−q)H(J1 ;K )+H(K ;K

K K =K

=

)+H(K ;J ) 2

I1 | J1 K I2 | K J2

(−q)H(I1 ;I1 )+H(I1 ;I2 )+H(I2 ;I2 ) I1 | J1 I1 I2 | K I2 | J2 .

Iν Iν =Iν

(b) If |Iν | ≤ |Jν | and |K| = |J1 | + |J2 | − |I1 | − |I2 |, then

(−q)H(I1 ;K )+H(K ;K

K K =K

=

)+H(K ;I ) 2

I1 K | J1 K I2 | J2

(−q)H(J1 ;J1 )+H(J1 ;J2 )+H(J2 ;J2 ) I1 | J1 K | J1 J2 I2 | J2 .

Jν Jν =Jν

Proof. (a) The case in which 1 ≤ |Jν | < |Iν | is given in the proof of [13, Proposition 1.2]; our version of this case includes only the terms with nonzero coefﬁcients. The same proof yields the general case, as follows. First expand the left-hand side of (∗) by applying Lemma B.1(a) to both [I1 | J1 K ] and [I2 | K J2 ]. This yields (†)

(−q)H(I1 ;I1 )+H(I2 ;I2 )+H(K ;K

)

I1 | J1 I1 | K I2 | K I2 | J2 .

K K =K I1 I1 =I1 I2 I2 =I2

We can also expand the right-hand side of (∗) by applying Lemma B.1(b) to the term [I1 I2 | K]. Since this also yields (†), part (a) is proved. (b) This is proved in the same manner. B.3. Note that if |I1 ∪ I2 | < |K| in Proposition B.2, there do not exist disjoint subsets Iν ⊆ Iν such that |I1 I2 | = |K|, and so the right-hand side of formula (a) is zero. Similarly, if |J1 ∪ J2 | < |K|, the right-hand side of (b) is zero. These cases are called generalized Plücker relations [13, Proposition 1.2]. Note added in proof: The conjectured quantum analog of the ﬁrst fundamental theorem (see the Introduction) has been proved in [6].

190

GOODEARL AND LENAGAN

References [1] [2] [3] [4] [5] [6]

[7] [8] [9] [10] [11] [12] [13]

[14] [15]

M. Artin, W. Schelter, and J. Tate, Quantum deformations of GLn , Comm. Pure Appl. Math. 44 (1991), 879–895. K. A. Brown and K. R. Goodearl, Prime spectra of quantum semisimple groups, Trans. Amer. Math. Soc. 348 (1996), 2465–2502. W. Bruns and U. Vetter, Determinantal Rings, Lecture Notes in Math. 1327, Springer, Berlin, 1988. C. De Concini, D. Eisenbud, and C. Procesi, Young diagrams and determinantal varieties, Invent. Math. 56 (1980), 129–165. W. Fulton, Young Tableaux, London Math. Soc. Stud. Texts 35, Cambridge Univ. Press, Cambridge, 1997. K. R. Goodearl, T. H. Lenagan, and L. Rigal, The ﬁrst fundamental theorem of coinvariant theory for the quantum general linear group, preprint, http://xxx. lanl.gov/abs/math.QA/9910035; to appear in Publ. Res. Inst. Math. Sci. T. J. Hodges and T. Levasseur, Primitive ideals of Cq [SL(n)], J. Algebra 168 (1994), 455– 468. R. Q. Huang and J. J. Zhang, Standard basis theorem for quantum linear groups, Adv. Math. 102 (1993), 202–229. D. A. Jordan, notes, 1993. A. Joseph, Quantum Groups and Their Primitive Ideals, Ergeb. Math. Grenzgeb. (3) 29, Springer, Berlin, 1995. T. Levasseur and J. T. Stafford, The quantum coordinate ring of the special linear group, J. Pure Appl. Algebra 86 (1993), 181–186. Yu. I. Manin, Quantum Groups and Noncommutative Geometry, Publications du Centre de Recherches Mathématiques, Université de Montréal, Montréal, 1988. M. Noumi, H. Yamada, and K. Mimachi, Finite-dimensional representations of the quantum group GLq (n; C) and the zonal spherical functions on Uq (n−1)\Uq (n), Japan. J. Math. (N.S.) 19 (1993), 31–80. B. Parshall and J. P. Wang, Quantum linear groups, Mem. Amer. Math. Soc. 89 (1991), no. 439. L. Rigal, Normalité de certains anneaux déterminantiels quantiques, Proc. Edinburgh Math. Soc. (2) 42 (1999), 621–640.

Goodearl: Department of Mathematics, University of California, Santa Barbara, California 93106, USA; [email protected] Lenagan: Department of Mathematics, University of Edinburgh, James Clerk Maxwell Building, King’s Buildings, Mayﬁeld Road, Edinburgh EH9 3JZ, Scotland; [email protected]. ac.uk

Vol. 103, No. 2

DUKE MATHEMATICAL JOURNAL

© 2000

AFFINE MAPPINGS OF TRANSLATION SURFACES: GEOMETRY AND ARITHMETIC EUGENE GUTKIN and CHRIS JUDGE

1. Introduction. Translation surfaces naturally arise in the study of billiards in rational polygons (see [ZKa]). To any such polygon P , there corresponds a unique translation surface, S = S(P ), such that the billiard ﬂow in P is equivalent to the geodesic ﬂow on S (see, e.g., [Gu2], [Gu3]). There is also a classical relation between translation surfaces and quadratic differentials on a Riemann surface S. Namely, each quadratic differential induces a translation structure on a ﬁnite puncturing of S or on a canonical double covering of S. Quadratic differentials have a natural interpretation as cotangent vectors to Teichmüller space, and this connection has proven useful in the study of billiards (see, e.g., [Ma2], [V1]). With a translation surface, S, one associates various algebraic and geometric objects: the induced afﬁne structure of S and the group of afﬁne diffeomorphisms, Aff(S); the holonomy homomorphism, hol : π1 (S) → R2 and the holonomy group Hol(S) = hol(π1 (S)); the ﬂat structure on S and the natural cell decompositions of its metric completion S. In the present paper, we study the relations between these objects, as well as relations among different translation surfaces. Our main focus is the group Aff(S) and the associated group of differentials, (S) ⊂ SL(2, R). The study of these groups began as part of W. Thurston’s classiﬁcation of surface diffeomorphisms in [Th2]. This study continued with the work of W. Veech in [V1] and [V2]. Veech produced explicit examples of translation surfaces S for which (S) is a nonarithmetic lattice. He showed that if (S) is a lattice, then the geodesic ﬂow on S exhibits remarkable dynamical properties. For these reasons, we call (S) the Veech group of S, and if this group is a lattice, then we call S a Veech surface. We now describe the structure of the paper and our main results. In §2, we establish the setting. In particular, we recall the notion of a G-manifold and associated objects: the developing map, the holonomy homomorphism, and the holonomy group. We introduce the notion of the differential of a G-map with respect to a normal subgroup H ⊂ G. We also introduce the spinal triangulation, one of several cell decompositions canonically associated to a ﬂat surface with cone points. Received 8 March 1999. Revision received 27 October 1999. 2000 Mathematics Subject Classiﬁcation. Primary 30F60; Secondary 37D40, 53D25, 30F30, 11F72. Gutkin partially supported by the National Science Foundation grant numbers DMS-9013220, DMS9400295. Judge partially supported by a National Science Foundation postdoctoral fellowship. 191

192

GUTKIN AND JUDGE

In §3, we study G-coverings of G-manifolds. Given such a covering, p : X → Y , we characterize the group of G-automorphisms of Y (resp., X) that lift to X (resp., descend to Y ). As an application, we show that if Y has a ﬁnitely generated fundamental group, then the subgroup of homeomorphisms that lift to X has ﬁnite index in the full homeomorphism group of Y (see Proposition 3.3). This fact and the technical Proposition 3.4 are crucial to the proof of the results of §4.2. Also in §3, we establish the existence and uniqueness of the minimal H -covering of a G-manifold for a subgroup H ⊂ G (Theorem 3.6). In §4, we specialize to the case where G is a group of afﬁne transformations of the Euclidean plane. Using the functoriality of the spinal triangulation (Proposition 4.1), we obtain a topological bound on the number of ﬂat coverings, p : X → Y , where X is a given ﬂat surface (Theorem 4.3). We then apply this result, in combination with results of §3, to afﬁne coverings of translation surfaces. In particular, we show that if p : R → S has ﬁnite degree, then (R) and (S) are commensurable (see Theorem 4.9). It follows that Veech surfaces are preserved by ﬁnite degree coverings, a result that is anticipated in [V2]. In §5, we discuss arithmetic properties of translation surfaces. We introduce the set of developed cone points, the cross-ratio of saddle connections, and the notion of a translation tiling. Our main result, Theorem 5.5, gives several geometric and algebraic characterizations of translation surfaces that have arithmetic Veech groups. For example, we show that (S) is commensurable to SL(2, Z) if and only if S is (translation) tiled by a parallelogram. Translation surfaces tiled by parallelograms are the historical precursors of Veech surfaces. They were the ﬁrst examples of translation surfaces having a large group of afﬁne diffeomorphisms (see [Th2]). The so-called Veech dichotomy (a geodesic is either ﬁnite or uniformly distributed) was ﬁrst proven for these surfaces (see [Gu1]). In §6, we take up the counting of simple closed geodesics on a Veech surface S. Let NS (x) be the number of lengths of closed geodesics on S that are less than x. H. Masur (see [Ma1], [Ma2]) showed that there are constants 0 < c1 ≤ c2 < ∞ such that c1 x 2 ≤ NS (x) ≤ c2 x 2 holds for large x. Remarkably, if S is a Veech surface, then NS (x) ∼ c · x 2 for some constant c = c(S) (see [V1]). We call c(S) the quadratic constant of S. Veech explicitly computed the quadratic constant for a family of right triangles in [V1] and for regular polygons in [V2]. The main result of §6, Theorem 6.5, gives an explicit general formula for the quadratic constant. The proof is based on the counting of vectors in the orbit of a lattice (Theorem 6.1). We reduce the latter to a counting of horocycles in the hyperbolic plane. Our approach to counting closed geodesics is more elementary than Veech’s original approach. In particular, our proof depends on neither Eisenstein series nor Tauberian theorems, but rather on the mixing of the geodesic ﬂow on surfaces of constant negative curvature and ﬁnite volume. Independently, Veech discovered another “Eisenstein series free” approach in [V3]. However, [V3] does not contain a formula for the quadratic constant of a Veech surface.

AFFINE MAPPINGS OF TRANSLATION SURFACES

193

In §7, we show that the cross-ratio ﬁeld of a Veech surface is equal to the trace ﬁeld of its Veech group (Theorem 7.1). From results in [Th2], it is easy to deduce that the trace ﬁeld of a Veech surface is a number ﬁeld. Combining this fact with Theorem 7.1, we show that the quadratic constant of a Veech surface S multiplied by π ·Area(S) is an algebraic number (Theorem 7.4). The proof depends on the fact that the ratio of the lengths of two parallel saddle connections belongs to the cross-ratio ﬁeld. In [GuJ], we announced and sketched proofs of several of our results from the present paper and gave applications to polygonal billiards. At the same time, an independent announcement, [Vo1], appeared. Some of the theorems announced in [Vo1] overlap with results in [GuJ]. More speciﬁcally, the overlap concerns those results whose detailed proofs appear below in §4. Although no proofs were given in [Vo1], they have since appeared in [Vo2]. The methods of both §4 below and [Vo2, §5] have their genesis in the work of Veech [V2]. Acknowledgments. It is a pleasure to thank the following individuals for stimulating conversations concerning this material: A. Eskin, N. Haydn, G. Mess, and Z. Rudnick. We also thank the referee for helpful comments. 2. Preliminaries and notation 2.1. Group structures on manifolds. Recall the notion of a (G, X)-manifold (refer to [Th3]), where G is a subgroup of the group of self-homeomorphisms of a manifold X. In the present paper, X = Rn , and hence we use the term G-manifold. We denote by (M, µ) a G-structure µ on a manifold M. A mapping f : (M, µ) → (N, ν) is called a G-map of G-manifolds if for each µi : Ui → Rn and νj : Vj → Rn , there exists gj i ∈ G such that νj ◦f = gj i ◦µi . Let X be a topological manifold, let (M, µ) be a Gmanifold, and let f : X → M be a local homeomorphism. Then there exists a unique G-structure f ∗ (µ) on X, the pullback of µ, such that f : (X, f ∗ (µ)) → (M, µ) is a G-map. From now on, we consider only connected G-manifolds where G has the following (unique continuation) property: If g1 (x) = g2 (x) for all x belonging to a nonempty open set U ⊂ Rn , then g1 = g2 .1 Let (M, µ) be a simply connected G-manifold, and let µ0 : U0 → Rn be a chart. Then there exists a unique G-map, dev : (M, µ) → Rn , satisfying dev|U0 = µ0 (see [Th3]). This map is the developing map of (M, µ). For arbitrary (M, µ), the developing map associated to (M, µ) is the map dev : ˜ p ∗ (µ)) → (M, µ) is the universal covering. ˜ p ∗ (µ)) → Rn , where p : (M, (M, For each G-map f of a simply connected G-manifold into itself, there is a unique element hol(f ) ∈ G, the holonomy of f , satisfying dev ◦ f = hol(f ) ◦ dev. For example, if M is a G-manifold, then each deck transformation γ ∈ π1 (M) is a Gmap from M˜ to itself.2 Thus, (1) 1 For 2 We

dev(γ · m) ˜ = hol(γ ) · dev(m). ˜ example, any group of real-analytic diffeomorphisms has this property. suppress the base point of π1 (M) from notation.

194

GUTKIN AND JUDGE

The map hol : π1 (M) → G is the holonomy homomorphism (see [Th3]). The group Hol(M) ≡ hol(π1 (M)) ⊂ G is the holonomy group of M. Let Str G (X) denote the set of G-structures on X. An inclusion H ⊂ G of groups G : Str (X) → Str (X), which is often implicit in the yields a forgetful functor iH H G sequel. Let H be a normal subgroup of G. For a G-map, f : (X, µ) → (Y, ν), between H -manifolds, we deﬁne the differential D(f ) ∈ G/H as follows. Choose µi : Ui → Rn and νj : Vj → Rn to be charts associated, respectively, to µ and ν such that f (Ui ) ∩ Vj = ∅. Then D(f ) = νj ◦ f ◦ µ−1 (2) H i is independent of the choices. The name “differential” is justiﬁed by the example of afﬁne diffeomorphisms of translation manifolds (see §2.3). Next, we deﬁne a natural left action G/H × Str H (X) → Str H (X). Let g ∈ G, and let ᐂ = {µi : Ui → Rn } be an atlas of µ ∈ Str H (X). Consider the atlas g · ᐂ = {g ◦µi : Ui → Rn }. Because H is a normal subgroup, g · ᐂ is an H -atlas whose equivalence class in Str H (X) depends only on the coset gH . In this paper, we consider afﬁne, ﬂat, and translation manifolds. These three types of G-manifolds correspond, respectively, to G equal to • the group of afﬁne transformations, Aff(Rn ) ∼ = GL(n, R) × Rn ; n n ∼ • the group of isometries of R , Isom(R ) = O(n) × Rn ; • the group of translations of Rn , Trans(Rn ) ∼ = Rn . A pseudogroup Ᏻ of homeomorphisms between open subsets of Rn gives rise to the concepts of Ᏻ-structure and Ᏻ-manifold (see [Th3]). Since every group G of self-homeomorphisms of Rn deﬁnes a pseudogroup, every G-manifold is also a Ᏻmanifold. Some of the notions deﬁned above extend to pseudogroup manifolds. For example, we have the notion of a Ᏻ-map and the forgetful functor, which maps Ᏼstructures to Ᏻ-structures when Ᏼ ⊂ Ᏻ. Note that every pseudogroup of homeomorphisms is a subset of ᐀ op, the pseudogroup of homeomorphisms of open subsets of Rn . In particular, every Ᏻ-manifold is a topological manifold. 2.2. Flat surfaces. A ﬂat surface X is a 2-dimensional G-manifold such that G = Isom(R2 ). The standard metric tensor on R2 pulls back to a ﬂat Riemannian metric on X. A curve γ ⊂ X is a geodesic if dev(γ˜ ) is a line (segment) in R2 for some lift γ˜ of γ . Let X denote the metric completion of the ﬂat surface X with respect to the Riemannian distance. A ﬂat map f : X → Y has a unique continuous extension f : X → Y . For example, each deck transformation γ : X˜ → X˜ has an extension γ . The developing map extends to a map dev from the completion of X˜ to R2 , and equation (1) extends to (3)

dev(γ · x) = hol(γ ) · dev(x).

Assumption 2.1. For each ﬂat surface X appearing in the sequel, the set X\X is assumed to be discrete.

AFFINE MAPPINGS OF TRANSLATION SURFACES

195

The set "(X) ≡ X\X is the set of cone points of X. Let U denote the universal cover of the punctured plane with the metric induced by the covering. The ﬂat surface U is isometric to R+ ×R with the metric dr 2 +r 2 dθ 2 . For each cone point c ∈ "(X), there exist s and α such that the quotient of [0, s)×R ⊂ U by the group generated by (r, θ ) → (r, θ + α) is isometric to a neighborhood of c in X. The number α is called the angle at c. Recall the notion of a cell complex. We only consider 2-dimensional cell complexes whose 2-cells are simple Euclidean polygons that are glued along their edges via isometries. Let Ꮿ and Ᏸ be cell complexes. Then a polygonal map f : Ꮿ → Ᏸ is a continuous map of the underlying topological spaces such that f maps each 2-cell of Ꮿ isometrically onto a 2-cell of Ᏸ. If the underlying topological space of a cell complex Ꮿ is a closed surface S, then Ꮿ is a cell decomposition of S. Note that a cell decomposition of S induces a ﬂat structure on the complement of the 0-skeleton, making the puncturing of S a ﬂat surface. The completion X of a ﬂat surface has several canonical cell decompositions. Here we deﬁne the spinal triangulation and the Voronoi decomposition. Let Spine(X) ⊂ X be the set of points x ∈ X such that there exist at least two geodesic segments realizing the distance, d(x, "(X)) (see [Bo]). Let K ⊂ Spine(X) consist of points for which there exist exactly two such segments. Then K is a disjoint union of open geodesic segments called vertebrae. The set Spine(X) has a natural interpretation as a graph: The edges are the vertebrae, and the vertices, v ∈ V , are the points for which there are at least three distinct geodesic segments realizing d(v, "). The connected components of X\ Spine(X) are the umbrellas. Each cone point c ∈ "(X) is contained in a unique umbrella Uc . For each point x ∈ Uc , there is a unique minimal geodesic segment joining x to c ∈ Uc . The Voronoi decomposition, ᐂX , is the cell decomposition of X whose 2-cells are the umbrellas (see [Th1]). Let v ∈ ∂Uc be a vertex of the spine. Each geodesic segment joining v to c is called a rib. The ribs partition Uc into a ﬁnite number of spinal triangles each bounded by two ribs and an edge of the spine. The Gauss-Bonnet theorem implies that each spinal triangle is isometric to a Euclidean triangle. The spinal triangulation, ᐀X , is the cell decomposition of X whose 2-cells are the spinal triangles. 2.3. Translation and half-translation surfaces. A translation surface is a topological surface X together with µ ∈ Str G (X), where G = Trans(R2 ). Note that Trans(R2 ) ∼ = R2 is a normal subgroup of Aff(R2 ) with quotient GL(2, R). Hence, the remarks of §2.1 apply. In particular, we use the left action GL(2, R) × Str R2 (X) → Str R2 (X) and the differential D(f ) ∈ GL(2, R) of an afﬁne map f : (X, µ) → (Y, ν) of translation surfaces. Let R be a precompact translation surface. Let Aff(R) denote the group of orientation-preserving, afﬁne diffeomorphisms of R. Any φ ∈ Aff(R) is volume preserving, and thus the differential is a homomorphism D : Aff(R) → SL(2, R). Deﬁne the Veech group of R to be (R) ≡ D(Aff(R)) ⊂ SL(2, R). For R = (X, µ), let g · R = (X, g · µ) denote the GL(2, R)-action on translation

196

GUTKIN AND JUDGE

surfaces. By deﬁnition of G-map, we have Aff(R) = Aff(g · R). Let DR and Dg·R denote the respective differential homomorphisms. It follows from equation (2) that for any φ ∈ Aff(R), we have Dg·R (φ) = g · DR (φ) · g −1 . Therefore, (g · R) = g · (R) · g −1 . There is a natural mapping from the set of geodesics on R into the projective space ˜ P (R2 ): Let γ ⊂ R be a geodesic and let γ˜ be a lift of γ to the universal cover R. 2 Deﬁne [γ ] ∈ P (R ) to be the line through the origin parallel to dev(γ˜ ). Equation (1) implies that [γ ] is independent of the choice of lift. The preceding discussion applies to the slightly larger class of half-translation surfaces. These are the G-surfaces where G is the group of half-translations, 1/2Trans(R2 ) ≡ {± Id} × R2 . Each half-translation structure either reduces to a translation structure or lifts uniquely to a translation structure on a double covering. The study of half-translation surfaces is equivalent to the study of meromorphic quadratic differentials with poles of order at most one (see, e.g., [Ma1]). In [V1], half-translation structures were called “F-structures.” Our analysis of translation surfaces is based on the fact that Trans(R2 ) ⊂ Isom(R2 ) is a normal subgroup of Aff(R2 ). Since 1/2Trans(R2 ) also has this property, we have the following remark. Remark 2.2 (On quadratic differentials). All of the results of this paper concerning translation surfaces are valid for half-translation surfaces, provided we replace SL(2, R) with PSL(2, R). 3. Group structures and coverings 3.1. Lifting and pushing down homeomorphisms. Let Homeo(M) denote the group of self-homeomorphisms of a topological manifold M. Deﬁnition 3.1. Let M and N be topological manifolds. Two coverings, pi : M → N, i = 1, 2, are lower equivalent (resp., upper equivalent3 ) if there exists φ ∈ Homeo(N ) (resp., ψ ∈ Homeo(M)) such that p2 = φ ◦ p1 (resp., p1 = p2 ◦ ψ). For a covering p : M → N, let [p]∗ (resp., [p]∗ ) denote its lower (resp., upper) equivalence class. Deﬁne a right (resp., left) action of Homeo(M) (resp., Homeo(N)) on the set of lower (resp., upper) equivalence classes by [p]∗ · ψ = [p ◦ ψ]∗ (resp., φ · [p]∗ = [φ ◦ p]∗ ). Let Ᏻ be a pseudogroup, and let µ be a Ᏻ-structure on a topological manifold X. We let Aut(X, µ) ⊂ Homeo(X) denote the subgroup of homeomorphisms of X that preserve µ. If the structure µ is implicit, then we use the notation Aut(X). To each Ᏻ-covering of Ᏻ-manifolds, p : X → Y , we associate two groups: Aut p (Y ) ⊂ Aut(Y ) consists of the Ᏻ-homeomorphisms that lift via p to X, and Aut p (X) ⊂ Aut(X) consists of the Ᏻ-homeomorphisms that descend via p to Y . Lemma 3.2. Let p : X → Y be a Ᏻ-covering of Ᏻ-manifolds. Then Aut p (X) (resp., Autp (Y )) is the stabilizer of [p]∗ (resp., [p]∗ ) under the action of Aut(X) 3 “Upper

equivalence” is the usual topological equivalence (see [M]).

AFFINE MAPPINGS OF TRANSLATION SURFACES

197

(resp., Aut(Y )) on the set of lower (resp., upper) equivalence classes of coverings. Proof. The proofs of the two claims are dual, and hence we prove only the ﬁrst: Let ψ belong to Aut(X). We have [p]∗ = [p ◦ ψ]∗ if and only if there exists φ ∈ Homeo(Y ) such that φ ◦ p = p ◦ ψ. Since both p and ψ are Ᏻ-maps, then φ lies in Aut(Y ) ⊂ Homeo(Y ). Hence, ψ ∈ Aut(X) descends if and only if [p]∗ = [p]∗ ·ψ. The following is probably known if Ᏻ is the pseudogroup of homeomorphisms or diffeomorphisms. Proposition 3.3. Let X and Y be Ᏻ-manifolds with π1 (Y ) ﬁnitely generated. Let p : X → Y be a ﬁnite Ᏻ-covering. Then Aut p (Y ) has ﬁnite index in Aut(Y ). Proof. The standard theory (see [M]) yields a one-to-one correspondence between upper equivalence classes of Ᏻ-coverings and conjugacy classes of subgroups of π1 (Y ). Moreover, if a covering q has degree d, then the conjugacy class C(q) consists of subgroups of index d in π1 (Y ). Let φ ∈ Aut(Y ). Then φ ◦ p has the same degree as p. Hence, | Aut(Y ) · [p]∗ | is at most the number of conjugacy classes of subgroups whose index is equal to the degree of p. A ﬁnitely generated group has only ﬁnitely many index d subgroups. Thus, the claim follows from Lemma 3.2. We say that a covering p : X → Y pushes down a G-structure µ on X to a GG : Str (X) → structure ν on Y , provided p ∗ (ν) = µ. Recall the “forgetful functor,” iH H Str G (X), associated to the inclusion of groups H ⊂ G. The following proposition is used in §4.2. Proposition 3.4. Let H ⊂ G be a normal subgroup and let p : (X, µ) → (Y, ν) be an H -covering. Let A be the set of lower equivalence classes [q]∗ of (topological) coverings q : X → Y that push µ down to an H -structure on Y . Then the index G (µ)) : Aut (X, i G (µ))] ≤ |A|. [Aut(X, iH p H Proof. By Lemma 3.2, it sufﬁces to show that the orbit [p]∗ ·Aut(X) is a subset of A. Let q = p ◦ ψ be a representative of [p]∗ · ψ. Since both p : (X, µ) → (Y, ν) and ψ : (X, µ) → (X, µ) are G-maps, q : (X, µ) → (Y, ν) is a G-map of H -manifolds. Let D(q) ∈ G/H be the differential of q, and let ν = D(q)−1 ·ν (see §2.1). Then the differential of q : (X, µ) → (Y, ν ) is equal to the identity, and hence q is an H -map. Therefore, q ∗ (ν ) = µ, and thus [p]∗ · ψ = [q]∗ ∈ A. G consists of G3.2. The minimal H -covering of a G-manifold. The range of iH structures µ on X that can be represented by an H -atlas. In this case, we say that µ is an H -representable G-structure. The following proposition is standard. (See, e.g., [R, Theorem 8.4.5].) We include the proof here for the convenience of the reader.

Proposition 3.5. Let H ⊂ G be a subgroup. A G-structure µ on a topological manifold X is H -representable if and only if Hol(X, µ) ⊂ H .

198

GUTKIN AND JUDGE

Proof. We shall prove the nontrivial implication. Suppose Hol(X, µ) ⊂ H , and let ˜ p ∗ (µ)) → (X, µ) denote the universal covering. The developing map provides p : (X, ˜ The deck group π1 (X) acts properly discontinuously on an H -atlas for p∗ (µ) on X. ˜ and by equation (1), each deck transformation is an H -map. It follows that the X, ˜ 1 (X). H -structure p ∗ (µ) uniquely descends to an H -structure ν on X = X/π G ∗ ∗ ∗ By construction, p (ν) is an H -structure, and iH (p (ν)) = p (µ). Using the fact that the pullback and the forgetful functor commute, and the injectivity of the pullback, G (ν). we conclude that µ = iH Let X be a G-manifold and let H ⊂ G be a subgroup. Then hol−1 (H ) is a subgroup of π1 (X). The standard theory (see [M]) provides a covering pˆ : Xˆ → X such that ˆ = hol−1 (H ). Thus, (X, ˆ pˆ ∗ (µ)) is a G-manifold whose holonomy group pˆ ∗ (π1 (X)) belongs to H . Hence, Proposition 3.5 implies that there exists an H -structure µˆ on Xˆ G (µ) ˆ µ) ˆ = pˆ ∗ (µ). We call (X, ˆ the minimal H -covering manifold of (X, µ). such that iH ˆ Note that the G-covering pˆ : (X, µ) ˆ → (X, µ) is uniquely determined up to upper equivalence, and the degree of pˆ is equal to the index [Hol(X, µ) : H ∩ Hol(X, µ)]. The following theorem justiﬁes the choice of terminology. Theorem 3.6. Let (X, µ) be a G-manifold, let H ⊂ G be a subgroup, and let ˆ µ) (X, ˆ be as above. Suppose that there is an H -manifold, (Y, ν), and a G-covering G (ν)) → (X, µ). Then p is the composition of an H -covering q : (Y, ν) → p : (Y, iH ˆ µ) ˆ µ) ˆ µ) (X, ˆ and a G-covering pˆ : (X, ˆ → (X, µ). The H -manifold (X, ˆ is uniquely determined by this property. G (ν)) ⊂ H , and hence Proof. Since (Y, ν) is an H -manifold, we have Hol(Y, iH ˆ µ) ˆ and the π1 (Y ) ⊂ hol−1 (H ). The claims now follow from the deﬁnition of (X, theory of topological coverings (see [M]).

4. Afﬁne coverings of ﬂat surfaces 4.1. Flat coverings and cell decompositions. Recall from §2.2 the spinal triangulation ᐀X associated to a ﬂat surface X with nonempty cone set "(X). Proposition 4.1. Let X and Y be ﬂat surfaces with "(X) = ∅ (or "(Y ) = ∅). Each ﬂat covering p : X → Y deﬁnes a unique polygonal map ᐀(p) : ᐀X → ᐀Y . The mapping p → ᐀(p) is injective. Proof. To prove the ﬁrst claim, it sufﬁces to show that the restriction of p to each k-cell of ᐀X is an isometry onto a k-cell of ᐀Y for k = 0, 1, 2. Let I = [0, 1], let x ∈ X, and set y = p(x). Each path α : I → Y , with α(0) = y and α(1) ∈ "(Y ), lifts uniquely to a path α˜ : I → X, with α(0) ˜ = x and α(1) ˜ ∈ "(X). Because p is a local isometry, any path in X has the same length as its projection in Y . Thus, since the distance d(x, "(X)) (resp., d(y, "(Y ))) is obtained by minimizing length over the paths joining x and "(X) (resp., y and "(Y )), we have d(x, "(X)) = d(y, "(Y )).

AFFINE MAPPINGS OF TRANSLATION SURFACES

199

Thus, a geodesic γ realizes the distance d(x, "(X)) if and only if p ◦γ realizes the distance d(y, "(Y )). Two distinct geodesics based at x are mapped to two distinct geodesics based at y. Therefore: (1) The inverse image of the vertex set V (Y ) of Spine(Y ) is the vertex set of Spine(X); (2) the inverse image of Spine(Y ) \ V (Y ) is Spine(X) \ V (X); (3) the inverse image of a rib in ᐀Y is a union of ribs in ᐀X . Indeed, vertices of the spine are points for which there are at least three geodesics— the ribs—realizing the distance to ", and their complement in the spine is the set of points for which there are exactly two such geodesics. Consequently, (1) and the fact p("(X)) = "(Y ) together imply that the 0-skeleton of ᐀X is mapped onto the 0-skeleton of ᐀Y . Since p : Spine(X) \ V (X) → Spine(Y ) \ V (Y ) is continuous and surjective, each vertebra of X is mapped onto a vertebra of Y . This and (3) above imply that each 1-cell e of ᐀X is mapped onto some 1-cell e of ᐀Y . Since e is simply connected and p is a covering, p : e → e is injective. Thus, the local isometry p : e → e is an isometry. Let E(X) and E(Y ) denote the respective 1-skeletons of ᐀X and ᐀Y . We have shown that p−1 (E(Y )) = E(X). The 2-cells of ᐀X (resp., ᐀Y ) are the connected components of X \ E(X) (resp., Y \ E(Y )). Arguing as above, we conclude that each 2-cell of ᐀X is mapped isometrically onto a 2-cell of ᐀Y . In sum, p deﬁnes a polygonal map ᐀(p), and p is determined by ᐀(p). As a corollary of the proof, we have the following. Corollary 4.2. Let X and Y be ﬂat surfaces with "(X) = ∅. Let p : X → Y be a ﬂat covering. Then p : Spine(X) → Spine(Y ) is a topological covering of graphs. Let X be a topological manifold. The number of topological equivalence classes of degree d coverings of X is bounded in terms of d and the number, k(X), of generators of π1 (X). Under Assumption 2.1, a precompact ﬂat surface X is a ﬁnite puncturing of the closed surface X. Hence k(X) < ∞ is determined by the Euler characteristic, χ (X), and the number of punctures, |"(X)|. Thus, there is a bound on the number of upper equivalence classes of ﬂat coverings of a ﬂat surface X in terms of the number of cone points, the Euler characteristic of X, and the degree of the covering. Here we consider a dual problem. Theorem 4.3. Let X be a precompact, ﬂat surface, with "(X) = ∅. The number of ﬂat coverings with X as the covering space is bounded in terms of |"(X)| and χ(X). Proof. Proposition 4.1 reduces the desired bound to an estimate of the number of polygonal maps from ᐀X onto arbitrary cell complexes. Lemma 4.5 below gives a bound on the number of 2-cells of ᐀X in terms of |"(X)| and χ(X). Thus, the claim follows from Lemma 4.4 below. Let f (Ꮿ), e(Ꮿ), and v(Ꮿ), respectively, denote the number of 2-cells, 1-cells, and

200

GUTKIN AND JUDGE

0-cells of a ﬁnite cell complex Ꮿ. Let n(C) denote the number of sides of a 2-cell C ∈ Ꮿ, and let N(Ꮿ) denote the maximum of n(C) taken over all C ∈ Ꮿ. Lemma 4.4. Let Ꮿ be a ﬁnite cell complex. There is a bound on the number of surjective polygonal maps, p : Ꮿ → Ᏸ, which depends only on N(Ꮿ) and f (Ꮿ). Proof. If g : Ꮿ → Ᏸ is a surjective polygonal map, then each 2-cell in Ᏸ is isometric to a 2-cell in Ꮿ, and f (Ᏸ) ≤ f (Ꮿ). In particular, the cell complex Ᏸ can be constructed by gluing together 2-cells belonging to Ꮿ. It follows that the number of cell complexes Ᏸ that can be the image of Ꮿ under a polygonal map is bounded in terms of f (Ꮿ) and N(Ꮿ). Fix Ᏸ and let g : Ꮿ → Ᏸ be a surjective polygonal map. Then g is determined by the following data: a map G from the index set I of the 2-cells of Ꮿ, onto the index set J of the 2-cells of Ᏸ, and for i ∈ I , an isometry gi : Ci → CG(i) . The number of possible isometries is bounded by 2N(Ꮿ). Since f (Ᏸ) ≤ f (Ꮿ), the number of maps G : I → J is bounded by f (Ꮿ)f (Ꮿ) . Lemma 4.5. Let X be a precompact ﬂat surface with "(X) = ∅. Then f (᐀X ) ≤ 6(|"(X)| − χ (X)). Proof. The vertex set of the spinal triangulation is the disjoint union of the cone point set "(X) and the vertex set V (X) of Spine(X). Since the 2-cells of ᐀X are triangles, Lemma 4.6 below implies f (᐀X ) = 2|"(X)| + 2|V (X)| − 2χ(X). On the other hand, V (X) is the vertex set of the Voronoi decomposition ᐂX . The set of 2-cells in ᐂX is in one-to-one correspondence with "(X). Applying inequality (5) below to ᐂX , we obtain |V (X)| ≤ 2|"(X)|−2χ(X). Substituting this into the equality above, we obtain the claim. Lemma 4.6. Let Ꮿ be a cell decomposition of a closed surface S. Then we have the inequality (4)

f (Ꮿ) ≤ 2v(Ꮿ) − 2χ(S),

and its dual (5)

v(Ꮿ) ≤ 2f (Ꮿ) − 2χ(S).

If each 2-cell of Ꮿ is a triangle, then equality holds in (4). Proof. The claim follows from Euler’s formula χ(S) = f (Ꮿ) − e(Ꮿ) + v(Ꮿ) and elementary combinatorics. Remark 4.7. The bounds in the results above can be made explicit. We leave this to the reader. More efﬁcient bounds on the number of coverings can be obtained by exploiting the fact that the polygonal maps considered here are induced by coverings of ﬂat surfaces, and by using, for instance, the dual of the Voronoi decomposition, the Delaunay decomposition (see [Th1], [MaS]). Efﬁcient bounds would be useful in estimating the relative sizes of Veech groups of translation surfaces (see Theorem 4.9).

AFFINE MAPPINGS OF TRANSLATION SURFACES

201

4.2. Afﬁne coverings and Veech groups. In this section, we show that if two translation surfaces are related by a covering map, then the respective groups of afﬁne diffeomorphisms are closely related. Theorem 4.8. Let p : R → S be an afﬁne covering of precompact translation surfaces such that "(R) = ∅ (or, equivalently, "(S) = ∅). Then the indices [Aff(S) : Aff p (S)] and [Aff(R) : Aff p (R)] are ﬁnite. Proof. By hypothesis, S is a ﬁnitely punctured closed surface and hence, π1 (S) is ﬁnitely generated. Since R is compact, p has ﬁnite degree. Hence, Proposition 3.3 gives the ﬁrst claim. For gp = (Dp)−1 ∈ GL(2, R), we have Aff(gp · R) = Aff(R). As an afﬁne map from gp · R to S, the covering p has differential equal to the identity. Therefore, by replacing R by gp · R, we may assume that p is a translation covering. Let H = Trans(R2 ) and let G = Aff(R2 ). By Proposition 3.4, it sufﬁces to demonstrate the ﬁniteness of the set of (lower equivalence classes of) translation coverings with R as a covering space. Since every translation covering is a ﬂat covering and R is precompact, the claim follows from Theorem 4.3. Recall that two subgroups , ⊂ G are commensurate if ∩ has ﬁnite index in and . They are commensurable if is commensurate with gg −1 , for some g ∈ G. Theorem 4.9. Let p : R → S be an afﬁne covering of precompact translation surfaces. Then the groups (R) and (S) are commensurable. If p is a translation covering, then (R) and (S) are commensurate. Proof. In the proof of Theorem 4.8, we showed that p : g · R → S is a translation covering for some g ∈ GL(2, R). Thus, since (g ·R) = g ·(R)·g −1 , the ﬁrst claim follows from the second. Let p be a translation covering. If "(R) = ∅, then R (resp., S) is a ﬂat torus 2 R /L(R) (resp., R2 /L(S)), where L(R) (resp., L(S)) is an abelian lattice in R2 . The translation covering p induces a ﬁnite index inclusion L(R) ⊂ L(S). It follows that the stabilizers of the respective lattices are commensurate. The Veech group of a ﬂat torus is equal to the corresponding stabilizer, and the claim follows. Suppose that "(R) = ∅. Theorem 4.8 gives that both [Aff(S) : Aff p (S)] and [Aff(R) : Aff p (R)] are ﬁnite. Let DR (resp., DS ) denote the differential homomophism, and let p (R) ≡ DR (Aff p (R)) (resp., p (S) ≡ DS (Aff p (S))). Since DR (resp., DS ) is surjective, [(R) : p (R)] (resp., [(S) : p (S)]) is ﬁnite. The operation of “pushing down” an afﬁne homeomorphism is a surjective homomorphism: p∗ : Aff p (R) → Aff p (S). Since p is a translation covering, we have DS ◦ p∗ = DR . It follows that p (R) = p (S), and hence (R) and (S) are commensurate. The preceding theorem motivates the following deﬁnition.

202

GUTKIN AND JUDGE

Deﬁnition 4.10. Two precompact translation surfaces are close relatives if either both surfaces are afﬁnely covered by a common precompact translation surface, or both surfaces afﬁnely cover a common translation surface. Two translation surfaces are called relatives if they are equivalent under the equivalence relation generated by the relation of being close relatives. Finiteness of index is a property that is closed under intersection, and hence both commensurateness and commensurability are equivalence relations. Therefore, Theorem 4.9 implies the following. Theorem 4.11. Let R1 and R2 be precompact translation surfaces, and let 1 and 2 be the corresponding Veech groups. If R1 and R2 are relatives, then 1 and 2 are commensurable. Corollary 4.12. Let R1 and R2 be precompact translation surfaces, and let 1 and 2 be the corresponding Veech groups. If R1 and R2 are relatives, then (1) the surface R1 is a Veech surface if and only if R2 is a Veech surface; (2) the group 1 is ﬁnite if and only if 2 is ﬁnite; (3) the group 1 is nonelementary if and only if 2 is nonelementary; (4) the group 1 contains a pair of noncommuting parabolic elements if and only if 2 does. Proof. The ﬁrst two assertions are immediate. The class of elementary subgroups of SL(2, R) is closed under the equivalence relation of commensurability (see [B]), thus implying assertion (3). Let ⊂ SL(2, R) be arbitrary, and let ⊂ be a subgroup of ﬁnite index. Let g, h ∈ be noncommuting parabolics. Then for some n ≥ 1, the group elements g n , hn are noncommuting parabolics in . Thus, the class of groups in question is closed under commensurability. The last assertion follows. 5. Arithmetic translation surfaces. Let S be a translation surface with "(S) = ∅, let S˜ be its universal covering, and let dev : S˜ → R2 be the developing map. Deﬁne ˜ ˜ the set of developed cone points to be "(S) ≡ dev("(S)). Lemma 5.1. Let S be a precompact translation surface with "(S) = ∅. The set of developed cone points is a ﬁnite union of Hol(S) orbits. Proof. Let p be the extension of the universal covering, p : S˜ → S, to the metric ˜ in the ﬁbre completions of S˜ and S. For each c ∈ "(S), choose exactly one c˜ ∈ "(S) −1 ˜ under the action of the group of p (c). The resulting ﬁnite set, A, generates "(S) ˜ = ∪a∈A {γ · a | γ ∈ π1 (S)}. The claim then follows from deck transformations: "(S) equation (3). Associated to each saddle connection γ ⊂ S, there is a unique line [γ ] ∈ P (R2 ). Let ᏸ(S) ⊂ P (R2 ) denote the set of such lines. We deﬁne the cross-ratio of saddle connections, [γ1 , γ2 , γ3 , γ4 ], to be the cross-ratio of the corresponding lines [γ1 ], . . . , [γ4 ]

AFFINE MAPPINGS OF TRANSLATION SURFACES

203

(see, for example, [Be]). Let K(S) denote the ﬁeld of fractions of the cross-ratios of saddle connections on S. Note that K(g · S) = K(S) for any g ∈ GL(2, R). The developing map associates to each (oriented) saddle connection γ , a unique vector γ ∈ R2 . Let V (S) ⊂ R2 denote the K(S)-vector space spanned by these vectors. Proposition 5.2. Let S be a precompact translation surface with "(S) = ∅. Then the following are true. (1) The vector space V (S) contains Hol(S). ˜ (2) Any two elements of "(S) differ by an element of V (S). (3) The vector space V (S) is 2-dimensional over K(S). Proof. Let Ꮿ be a triangulation of S such that every 1-cell is a saddle connection (for example, a Delaunay triangulation). Let A(Ꮿ) be the group generated by the vectors γ , where γ is a 1-cell of Ꮿ. Now let γ ⊂ S be an arbitrary saddle connection. By developing Ꮿ along γ , one ﬁnds that γ belongs to A(Ꮿ). In particular, V (S) is spanned by the vectors associated to 1-cells of Ꮿ. β2

α2

α1

T

T γ

β1

Figure 1. Adjacent triangles

The holonomy homomorphism, hol : π1 (S) → Hol(S), factors through π1 (S), and each class in π1 (S) is represented by a loop in the 1-skeleton of Ꮿ. It follows that Hol(S) ⊂ A(Ꮿ) ⊂ V (S). Let c, c ∈ "(S), the vertex set of Ꮿ, and let γ be a path consisting of 1-cells that joins c to c . The development of γ is an element of A(Ꮿ) ⊂ V (S), and the second claim follows. It remains to show that V (S) is 2-dimensional over K(S). Let T0 be a triangle belonging to Ꮿ, and let α and β be 1-cells belonging to ∂T0 . It sufﬁces to show that for any 1-cell γ ∈ Ꮿ, the vector γ is a K(S)-linear combination of α and β. We prove the claim by induction. Namely, since S is connected, we may order the triangles, T0 , . . . , Tn ∈ Ꮿ, so that Ti is adjacent to Ti+1 for i = 1, . . . , n−1. The claim

204

GUTKIN AND JUDGE

is trivial for each 1-cell in ∂T0 . Suppose that the claim holds for all 1-cells in ∂Ti . By developing Ti and Ti+1 and applying the lemma below, we prove the claim for each side of Ti+1 . To each oriented ﬁnite line segment γ ⊂ R2 , there is a unique vector γ ∈ R2 . If γ is the side of a Euclidean polygon P , then γ is called a side vector of P . Lemma 5.3. Let T , T be two adjacent triangles in R2 . Suppose that the crossratios of any four sides of these triangles belongs to a ﬁeld K. Then each side vector is a K-linear combination of any pair of side vectors. Proof. Let γ be the common side, and let α1 , α2 (resp., β1 , β2 ) be the other sides of T (resp., T ). See Figure 1. Orient each side, and let α1 , α2 , β1 , β2 , and γ be the corresponding side vectors. We have the obvious relations γ = ±α1 ± β1 = ±α2 ± β2 , where the signs depend on the choice of side orientations. By applying paragraph 6.5.6 of [Be], we obtain γ = ± α 1 , β 1 , α2 , γ · α 1 ± β 1 , α1 , β 2 , γ · β 1 , (6) γ = ± α2 , β2 , α1 , γ · α2 ± β2 , α2 , β1 , γ · β2 . (7) Since the cross-ratios appearing above belong to K, the claim follows. Lemma 5.4. Let S be a precompact translation surface with "(S) = ∅. If K(S) = Q, then Hol(S) is a lattice in R2 . Moreover, there is a lattice L ⊂ R2 containing ˜ Hol(S) such that any two points in "(S) differ by an element of L. Proof. Since S is precompact, it has ﬁnite topological type. It follows that Hol(S) ˜ = ∪ni=1 (xi + is generated by a ﬁnite set h1 , . . . , hk . Lemma 5.1 implies that "(S) 2 ˜ Hol(S)), where x1 , . . . , xn ∈ "(S). Let L ⊂ R be the group generated by the set {h1 , h2 , . . . , hk , x2 − x1 , x3 − x1 , . . . , xn − x1 }. By Proposition 5.2, this generating set belongs to V (S), and hence L is isomorphic to a ﬁnitely generated subgroup of Q2 . Such a group has rank at most two. Since S is precompact, the rank of Hol(S), and hence the rank of L, is at least two. Therefore, both Hol(S) and L are lattices in R2 with Hol(S) ⊂ L. Let S be a ﬂat surface, and let P be the interior of a Euclidean polygon. A P -tile in S is an isometrically embedded copy of P . The polygon P tiles S if there exists a disjoint set of P -tiles, P1 , . . . , Pn , such that S = ∪P i and such that each intersection "(S) ∩ P i is contained in the set of vertices of Pi . If, in addition, each adjacent pair of tiles is related by a translation, then P tiles S by translations. Let S be a ﬂat surface and let I ⊂ S be a discrete set. We call SI ≡ S \ I , with the induced ﬂat structure, a puncturing of S. For a torus T , we let T0 denote its puncturing at a single point. Theorem 5.5. Let S be a precompact translation surface, and let (S) be its Veech group. The following statements are equivalent.

AFFINE MAPPINGS OF TRANSLATION SURFACES

205

(1) The groups (S) and SL(2, Z) are commensurable. (2) Every cross-ratio of saddle connections is rational (equivalently, K(S) = Q). (3) There exists a translation covering from a puncturing of S onto a oncepunctured ﬂat torus. (4) There is a Euclidean parallelogram that tiles S by translations. Proof. (1⇒2) There exists g ∈ SL(2, R) such that (g · S) = g(S)g −1 is commensurate to SL(2, Z). It follows that the set, F ⊂ R ∪{∞}, of parabolic ﬁxed points of (g · S) is equal to Q. Propositions 2.4 and 2.10 of [V1] imply that F is equal to the set of slopes of lines in ᏸ(g · S). Hence, K(S) = K(g · S) = Q. ˜ (2⇒3) Lemma 5.4 provides a lattice L containing Hol(S) such that "(S) ⊂ L+x 2 2 ˜ for some x ∈ "(S). Let ψ : R / Hol(S) → R /L be the corresponding translation covering. Puncture the ﬂat torus T = R2 / Hol(S) at the set J consisting of points corresponding to the orbit L + x ⊂ R2 . The restriction of ψ to TJ is a translation covering onto T0 , where T = R2 /L. To prove the claim, it sufﬁces to show that there is a translation covering from a puncturing of S onto TJ . Let π : S˜ → S be the universal covering. Puncture S˜ at I˜ = dev−1 (L + x) (the set of “regular” points of S˜ that correspond to L + x), and puncture S at I = π(I˜). The restriction of the developing map to S˜I˜ is a translation covering onto the puncturing of R2 at the orbit L+x. Equation (1) implies that this covering descends to a translation covering φ : SI → TJ . (3⇒1) Theorem 4.9 implies that for some ﬁnite set I ⊂ S and ﬂat torus T , the group (SI ) is commensurable to (T0 ) and hence to SL(2, Z). In particular, the area of H2 / (SI ) is ﬁnite, and it follows that the covering H2 / (SI ) → H2 / (S) associated to the inclusion (SI ) ⊂ (S) is ﬁnite. Therefore, (S) is commensurable with (SI ) and hence with SL(2, Z). (3⇒4) Let p : SI → T0 be the given translation covering. Let L be the lattice corresponding to T . Let P ⊂ R2 be an (open) fundamental parallelogram for L. Then S is tiled by P . (4⇒3) Let I ⊂ S be the set of vertices of the given tiling. Associated to the given parallelogram P , there is a ﬂat torus T = R2 /L such that P is the fundamental domain for L. Let T0 be the corresponding punctured torus. Since P tiles S by translations, the isometries mapping each tile onto P induce a translation covering from SI onto T0 . A precompact translation surface is called arithmetic if it satisﬁes one (and hence all) of the four conditions in Theorem 5.5. As mentioned in the introduction, these surfaces constitute a subclass of Veech surfaces that were studied prior to [V1]. For instance, the geodesic ﬂow on arithmetic translation surfaces was completely analyzed in [Gu1]. ˜ Remark 5.6. Suppose that Hol(S) is a lattice. This does not imply that "(S) belongs to the orbit of a lattice. For example, let S = T \ {t, t }, where T is a ﬂat

206

GUTKIN AND JUDGE

˜ torus, and t and t are points lying on a closed geodesic γ ⊂ T . Then "(S) belongs to the orbit of a lattice if and only if the distance along γ from t to t is commensurate to >(γ ). Yet we have Hol(S) = Z2 regardless of the relative positions of t and t . In particular, the assumption that Hol(S) is a lattice does not imply any of the statements in Theorem 5.5. 6. Asymptotics of closed geodesics on Veech surfaces. In this section, we derive an explicit asymptotic formula for the number of cylinders on a Veech surface. The proof of the main result, Theorem 6.5, is based on an asymptotic count of vectors in the orbit of a nonuniform lattice in SL(2, R). The antipodal map v → −v acts freely on R2 \ {0} with quotient space ᐂ. The elements of ᐂ are called vectors. The standard Euclidean norm and the action of SL(2, R) on R2 descend, respectively, to a norm " · " on ᐂ and an action on ᐂ. Let ᐂ = ᐂ ∪ {0} be the metric completion. For vectors u, v ∈ ᐂ, the absolute value of the inner product |#u, v$| is well deﬁned. Given two functions f, g : R → R, we write f (x) ∼ g(x) to indicate that limx→∞ (f (x)g(x)−1 ) = 1. Theorem 6.1. Let ⊂ SL(2, R) be a discrete subgroup with ﬁnite covolume, and let v ∈ ᐂ. Let v ⊥ ∈ ᐂ be orthogonal to v. Suppose that the stabilizer of v in is not {± id}, and let g be a generator of this stabilizer. Then the number of vectors, Nv (x), in the orbit v with norm less than x is well deﬁned, and ⊥ 2 −1 #gv , v$ Nv (x) ∼ Area(H / ) · x2. (8) "v"3 · "v ⊥ " Proof. The group SL(2, R) acts on the hyperbolic upper half plane, H2 = {(x, y) ∈ R2 : y > 0}, via Möbius transformations. This action induces an action, h → g · h, on the space, Ᏼ, of horocycles in H2 . The manifolds Ᏼ and ᐂ are isomorphic as SL(2, R) homogeneous spaces. We choose a speciﬁc √ isomorphism F : Ᏼ → ᐂ by requiring that the distance, d(i, h), between i = −1 and the horocycle h ∈ Ᏼ be equal to | log("F (h)"2 )|. Let h = F −1 (v). Let A(t) be the set of horocycles g ·h ∈ ·h with dist(i, g ·h) < t. Then F (A(t)) is the set of vectors in v lying in the annulus with the radii e−t/2 and et/2 . Since the stabilizer of h in is nontrivial, the set · h is discrete in Ᏼ. Hence, |A(t)| < ∞ and the orbit v is discrete in ᐂ. Thus, by Lemma 10 in [G], the set v ⊂ ᐂ is bounded away from 0. Hence, for t sufﬁciently large, Nv (et/2 ) = |A(t)|. The asymptotics of the latter are known (see, e.g., [EM, Theorem 7.2]). Namely, if h ⊂ H2 / is the closed horocycle corresponding to h, then |A(t)| ∼ length(h) · Area(H2 / )−1 · et . Therefore, (9)

Nv (x) ∼

length(h) · x2. Area(H2 / )

AFFINE MAPPINGS OF TRANSLATION SURFACES

207

It remains to show that (10)

length(h) = #gv ⊥ , v$ · "v"−3 · "v ⊥ "−1 .

Let B ∈ SL(2, R) be the orthogonal matrix that maps h to a horizontal horocycle h = {(x, y) : y = y0 } of height y0 in H2 . Then g = B ◦ g ◦ B −1 stabilizes h and v = B · v = F (h ). Because B preserves both the Euclidean inner product and hyperbolic arclength, it sufﬁces to show that (10) holds with h, g, v, and v ⊥ , replaced with h , g , v , and (v )⊥ . Thus, without loss of generality, the horocycle h is horizontal, the matrix g is upper triangular, v = (λ, 0), and v ⊥ = (0, µ). Letting b denote the upper right entry of g, we have length(h) = y0−1 · |b|. By the deﬁnition of F , we have | log(y0 )| = d(i, h) = | log(λ2 )|. Therefore, length(h) = λ−2 |b|, and the claim follows by calculating the right-hand side of (10). In applying Theorem 6.1 to the counting of cylinders, we reinterpret the constant in (8). Let g ∈ (S) belong to the stabilizer v of the vector v ∈ R2 . Let w ∈ R2 be an independent vector. Deﬁne x(g, v, w) implicitly by the equation g · w = w + x(g, v, w)v. Note that the absolute value, |x(g, v, w)|, depends only on the class of v, w ∈ ᐂ. Suppose that w = v ⊥ is perpendicular to v, and let R be the rectangle spanned by v and v ⊥ . A calculation gives the desired reinterpretation: ⊥ #gv , v$ x(g, v, v ⊥ ) = (11) . Area(R) "v"3 · "v ⊥ " The function x(g, v, w) satisﬁes the following equivariance properties: If h ∈ SL(2, R), then we have (12)

x hgh−1 , hv, hw = x(g, v, w).

If λ1 , λ2 ∈ R∗ , then (13)

λ2 x g, λ1 v, λ2 w = x(g, v, w). λ1

A (translation) cylinder is a translation surface (with boundary) that is homeomorphic to a cylinder. We associate to a cylinder C its length, >(C), and its width, w(C). Let S be a translation surface. From here on, we only consider embedded cylinders, C ⊂ S, which are “maximal” in the sense that C is not properly contained in another cylinder C ⊂ S. The group Aff(S) acts naturally on the set, Ꮿ(S), of embedded cylinders. The developing map induces a mapping v : Ꮿ(S) → ᐂ, which is equivariant, v(φ · C) = D(φ) · v(C), and satisﬁes "v(C)" = >(C). Let v(C)⊥ ∈ ᐂ denote the unique vector perpendicular to v with "v ⊥ (C)" = w(C).

208

GUTKIN AND JUDGE

A cylinder decomposition, Ᏸ, of S is a subset of Ꮿ(S) consisting of nonoverlapping cylinders that form a partition of S. The span of {v(C) : C ∈ Ᏸ} is one-dimensional and hence deﬁnes a line l(Ᏸ) ∈ P (R2 ), the direction of Ᏸ. Proposition 6.2. Let C be a cylinder in a cylinder decompostion Ᏸ. Let g = ± id be a generator of the stabilizer of v(C) in (S). The number i(C) ≡ |x(g, v(C), v ⊥ (C))| is a rational number that does not depend on the choice of g. Proof. Let φ ∈ Aff(S) be such that Dφ = g. By [V1, Proposition (2.4)], some power φ m maps each cylinder C ∈ Ᏸ onto itself with φ|∂C = id. (Indeed, the set ᏿ of saddle connections in Ᏸ is ﬁnite. Let m be the order of the permutation of ᏿ induced by φ.) Let ψ : C → C be a basic Dehn twist for C. Then φ m |C = ψ n for some n ∈ Z. By developing C into a strip—see Figure 2—we ﬁnd that g m v ⊥ = v ⊥ + n · v; that is, x(g m , v, v ⊥ ) = n. We have x(g m , v, v ⊥ ) = m · x(g, v, v ⊥ ), and the claim follows.

gmv⊥

v

v⊥ 0

Figure 2. Development of a multiple of a Dehn twist

Theorem 6.3. Let S be a Veech surface. Let Ᏸ be a cylinder decomposition and let C ∈ Ᏸ. Let NC (x) denote the number of cylinders φ · C ∈ Aff(S) · C of length less than x. Let Ᏹ(C) = Ᏸ ∩ Aff(S) · C. Then −1 i(C) · |Ᏹ(C)| 2 ·x . (14) NC (x) ∼ Area H2 / (S) Area(C) Proof. By equivariance, the restriction of the map v to Aff(S) · C is a |Ᏹ(C)|-to-1 mapping onto (S)·v(C). Hence, NC (x) = |Ᏹ(C)|·Nv(C) (x). The claim then follows from Theorem 6.1, equation (11), and the deﬁnition of i(C). The group Aff(S) acts naturally on the collection of all cylinder decompositions of S. It also acts by conjugation on the set ᏼ of maximal parabolic subgroups of (S).

AFFINE MAPPINGS OF TRANSLATION SURFACES

209

Each P ∈ ᏼ preserves a unique line that corresponds, in turn, to a unique cylinder decompostion Ᏸ (see [V1]). This map is equivariant with respect to the actions above. For a Veech surface, this map is bijective. Hence each orbit Aff(S) · Ᏸ corresponds to a unique conjugacy class of maximal parabolic subgroups of (S)—that is, a cusp of (S). Proposition 6.4. Let Ᏸ be a cylinder decomposition of a Veech surface S. The number |i(C)| c(Ᏸ) = (15) Area(C) C∈Ᏸ

depends only on the cusp, u, of (S) associated to Ᏸ. We set c(u) = c(Ᏸ), where Ᏸ is any cylinder decomposition corresponding to u. Proof. Let φ ∈ Aff(S), h = Dφ, and C = φ(C). It sufﬁces to show that c(φ · Ᏸ) = c(Ᏸ). Since det(h) = 1, we have Area(C ) = Area(C), and hence it is enough to show that i(C ) = i(C). Let v, v ⊥ , and g (resp., v , (v )⊥ , and g ) be the data associated to C (resp., C ) as in Proposition 6.2. Then g = hgh−1 , v = hv, and (v )⊥ = hv ⊥ . By equation (12), x(g , v , (v )⊥ ) = x(g, v, v ⊥ ), and the claim follows. Theorem 6.5. Let S be a Veech surface, and let u1 , u2 , . . . , un be the cusps of (S). Let N(x) be the number of cylinders C ⊂ S with >(C) < x. Equivalently, N(x) is the number of lengths of closed geodesics in S, counted with multiplicities, that are less than x. Then N(x) ∼

(16)

c(u1 ) + · · · + c(un ) 2 ·x . Area H2 / (S)

Proof. Let Ᏸi be a cylinder decomposition such that the orbit Aff(S) · Ᏸi corresponds to the cusp ui . By the results of [V1], each cylinder of S lies in the Aff(S)-orbit of a cylinder C ∈ Ᏸi for a unique i. Let NᏰ (x) denote the number of cylinders of length less than x lying in the orbit of some C ∈ Ᏸ. Then, N(x) = i NᏰi (x). Let Ᏹ1 , . . . , Ᏹk ⊂ Ᏸ be the equivalence classes deﬁned by C ∼ C if and only if C, C ∈ Ᏸ lie in the same Aff(S)-orbit. Using Theorem 6.3 and equation (15), we ﬁnd that NᏰ (x) =

k

|Ᏹj |−1

j =1

NC (x)

C∈Ᏹj

k i(C) · |Ᏹj | −1 · x2 |Ᏹj |−1 ∼ Area H2 / (S) Area(C)

= Area H2 / (S) The claim follows.

−1

i=1

· c(Ᏸ) · x 2 .

C∈Ᏹj

210

GUTKIN AND JUDGE

The number c(S) = Area(H2 / (S))−1 (c(u1 ) + · · · + c(un )) in (16) is called the quadratic constant of the Veech surface S. By Theorem 6.5, this constant determines the main term of the asymptotics of the lengths of closed geodesics on S. Note that c(g · S) = c(S) for any g ∈ SL(2, R). Thus, c(S) depends only on the afﬁne structure of S, that is, on the Teichmüller disc to which S belongs. 7. The arithmetic of the quadratic constant. Let tr : SL(2, R) → R denote the trace. For each subgroup ⊂ SL(2, R), deﬁne the trace ﬁeld of to be T () = Q(tr()). Theorem 7.1. Let S be a translation surface. Then the trace ﬁeld of (S) is contained in the cross-ratio ﬁeld, K(S). If S is a Veech surface, then T ((S)) = K(S). Proof. By Proposition 5.2, elements of (S) have entries in K(S) with respect to some basis of R2 . Thus, T ((S)) ⊂ K(S) for any translation surface. Now suppose that S is a Veech surface. Then the slope of each saddle connection is ﬁxed by some parabolic matrix in (S) acting projectively on R ∪ {∞} (see [V1]). Thus, to show that K(S) ⊂ T ((S)), it sufﬁces to prove that each cross-ratio of parabolic ﬁxed points belongs to the trace ﬁeld. Let p, q, r, and s be four parabolic ﬁxed points. Since [p, q, r, s] + [p, r, q, s] = 1, we have 2 2 p, q, r, s − 2 · p, q, r, s + 1 = p, r, q, s . (17) By applying Lemma 7.2 to the squares in (17), we obtain the claim. ˜ G) ≡ tr(F G) − 1/2 · tr(F ) · tr(G). For F, G ∈ SL(2, R), set tr(F Lemma 7.2. Let A, B, C, and D be parabolic matrices in SL(2, R), and let a, b, c, d ∈ R ∪ {∞} be their respective ﬁxed points. Then (18)

a, b, c, d

2

=

˜ · C) · tr(B ˜ · D) tr(A . ˜ · D) · tr(B ˜ · C) tr(A

Proof. Use the conjugation invariance of cross-ratio and trace to bring three of the matrices into a standard form, and then make a direct computation. Corollary 7.3. Let S be a Veech surface. Then K(S) is an algebraic number ﬁeld. Proof. It is well known that if ⊂ SL(2, R) is ﬁnitely generated, and the set tr() consists of algebraic numbers, then T () is an algebraic number ﬁeld (see, for example, [T, Lemma 2]). The trace of each matrix in the Veech group of any translation surface is an algebraic integer (see [Th2]), and hence Theorem 7.1 implies the claim. Theorem 7.4. Let S be a Veech surface, and let c(S) be the quadratic constant. Then π · Area(S) · c(S) is an algebraic number.

AFFINE MAPPINGS OF TRANSLATION SURFACES

211

Proof. Let u be a cusp of (S). By Theorem 6.5, it sufﬁces to prove that π · Area(H2 / (S))−1 · Area(S) · c(U ) is an algebraic number. By the Gauss-Bonnet theorem, π ·Area(H2 / (S))−1 is rational, and it remains to show that Area(S)·c(u) is algebraic. Hence, by Corollary 7.3, we only need to prove that Area(S) · c(u) belongs to the cross-ratio ﬁeld K(S). Let Ᏸ be a cylinder decomposition associated to u. In view of equation (15) and Proposition 6.2, it sufﬁces to show that Area(C)/ Area(C ) ∈ K(S) for any pair of cylinders C, C ∈ Ᏸ. For any cylinder C, we set µ(C) = >(C) · w(C)−1 . Note that µ(C) = "v(C)"/"v ⊥ (C)", where v ⊥ (C) ∈ ᐂ is the unique vector perpendicular to v(C) with length w(C). We have Area(C) >(C) 2 µ(C ) = (19) . · Area(C ) >(C ) µ(C) Let g be the generator of the stabilizer of the line corresponding to Ᏸ. By (13), we have

x g, v(C ), v(C )⊥ v ⊥ (C ) / v ⊥ (C)

µ(C) = (20) = . x g, v(C), v(C)⊥ "v(C )"/"v(C)" µ(C ) Thus, by Proposition 6.2, µ(C )/µ(C) is rational. Hence, it sufﬁces to show that >(C)/>(C ) ∈ K(S). The length of any cylinder is a rational linear combination of the lengths of the saddle connections in its boundary. The claim then follows from the lemma below. Lemma 7.5. Let S be a translation surface with "(S) = ∅, and let C ⊂ S be a cylinder. Let γ and γ be two saddle connections in the boundary of C. Then the ratio of lengths, >(γ )/>(γ ), belongs to K(S). Proof. “Develop” the cylinder C into a strip in R2 . Let p, q, r, s denote the developed cone points such that the line segment pq corresponds to γ and the line segment rs corresponds to γ . Either pq and rs belong to the same boundary line or they do not. The two cases are illustrated by Figures 3 and 4. In the ﬁrst case, choose c to be a developed cone point on the line not containing p, q, r, s. It follows from [Be, paragraph 6.5.6] that4 (21)

|pq| |pq| |qr| = = cp, cr, pq, cq · cq, cs, qr, cr . |rs| |qr| |rs|

In the second case, choose a such that arsp is a parallelogram. It then sufﬁces to show that |pq|/|ap| belongs to K(S). By [Be, paragraph 6.5.6], |pq|/|ap| = [ra, rq, pq, rp]. Since ra is parallel to sp, we have |pq|/|ap| = [sp, rq, pq, rp]. 4 We

have extended the deﬁnition of cross-ratio from lines to (nontrivial) line segments in the obvious way.

212

GUTKIN AND JUDGE

c

s r q p Figure 3. Saddle connections and cross-ratios: Case 1

s r

q p a

Figure 4. Saddle connections and cross-ratios: Case 2

Corollary 7.6. Let S be an arithmetic translation surface (see §5). Then π Area(S) · c(S) is a rational number. Proof. By the proof of Theorem 7.4, it sufﬁces to show that K(S) = Q. This follows from Theorem 5.5. References [B] [Be]

A. F. Beardon, The Geometry of Discrete Groups, Grad. Texts in Math. 91, Springer, New York, 1983. M. Berger, Geometry, Universitext, Springer, Berlin, 1994.

AFFINE MAPPINGS OF TRANSLATION SURFACES [Bo] [EM] [G]

[Gu1] [Gu2] [Gu3] [GuJ] [K]

[Ka] [M] [Ma1]

[Ma2] [MaS] [R] [T] [Th1] [Th2] [Th3] [V1] [V2] [V3] [Vo1] [Vo2] [ZKa]

213

B. Bowditch, Singular euclidean structures on surfaces, J. London Math. Soc. (2) 44 (1991), 553–565. A. Eskin and C. McMullen, Mixing, counting, and equidistribution in Lie groups, Duke Math. J. 71 (1993), 181–209. L. Greenberg, “Discrete groups with dense orbits [Appendix]” in L. Auslander, L. Green, and F. Hahn, Flows on Homogeneous Spaces, Ann. of Math. Stud. 53, Princeton Univ. Press, Princeton, 1963. E. Gutkin, Billiards on almost integrable polyhedral surfaces, Ergodic Theory Dynam. Systems 4 (1984), 569–584. , Billiards in polygons, Phys. D 19 (1986), 311–333. , Billiards in polygons: Survey of recent results, J. Statist. Phys. 83 (1996), 7–26. E. Gutkin and C. Judge, The geometry and arithmetic of translation surfaces with applications to polygonal billiards, Math. Res. Lett. 3 (1996), 391–403. R. Kirby, “Problems in low dimensional manifold theory” in Algebraic and Geometric Topology (Stanford, Calif., 1976), Part 2, Proc. Sympos. Pure Math. 32, Amer. Math. Soc., Providence, 1978, 273–312. I. Kra, On the Nielsen-Thurston-Bers type of some self-maps of Riemann surfaces, Acta Math. 146 (1981), 231–270 W. S. Massey, Algebraic Topology: An Introduction, Grad. Texts in Math. 56, Springer, New York, 1977. H. Masur, “Lower bounds for the number of saddle connections and closed trajectories of a quadratic differential” in Holomorphic Functions and Moduli I (Berkeley, Calif., 1996), Math. Sci. Res. Inst. Publ. 10, Springer, New York, 1988, 215–228. , The growth rate of trajectories of a quadratic differential, Ergodic Theory Dynam. Systems 10 (1990), 151–176. H. Masur and J. Smillie, Hausdorff dimension of sets of nonergodic measured foliations, Ann. of Math. (2) 134 (1991), 455–543. J. G. Ratcliffe, Foundations of Hyperbolic Manifolds, Grad. Texts in Math. 149, Springer, New York, 1994. K. Takeuchi, Arithmetic triangle groups, J. Math. Soc. Japan 29 (1977), 91–106. W. Thurston, Shapes of polyhedra, preprint, 1987. , On the geometry and dynamics of diffeomorphisms of surfaces, Bull. Amer. Math. Soc. 19 (1988), 417–431. , Three-Dimensional Geometry and Topology, Princeton Math. Ser. 35, Princeton Univ. Press, Princeton, 1997. W. Veech, Teichmüller curves in moduli space, Eisenstein series, and an application to triangular billiards, Invent. Math. 97 (1989), 553–583. , The billiard in a regular polygon, Geom. Funct. Anal. 2 (1992), 341–379. , Siegel measures, Ann. of Math. (2) 148 (1998), 895–944. Ya. Vorobets, Plane structures and billiards in rational polyhedra, Russ. Math. Surv. 51 (1996), 177–178. , Plane structures and billiards in rational polygons: The Veech alternative, Russ. Math. Surv. 51 (1996), 779–817. A. Zemlyakov and A. Katok, Topological transitivity of billiards in polygons, Math. Notes 18 (1975), 760–764.

Gutkin: Department of Mathematics, University of Southern California, Los Angeles, California 90089-1113, USA; [email protected], http://math.usc.edu/˜egutkin Judge: Department of Mathematics, Indiana University, Bloomington, Indiana 47405, USA; [email protected], http://poincare.math.indiana.edu/˜cjudge

Vol. 103, No. 2

DUKE MATHEMATICAL JOURNAL

© 2000

ANALYTIC STRATIFICATION IN THE PFAFFIAN CLOSURE OF AN O-MINIMAL STRUCTURE JEAN-MARIE LION and PATRICK SPEISSEGGER Introduction. Let U ⊆ Rn be open and ω = a1 dx1 + · · · + an dxn a nonsingular, integrable 1-form on U of class C 1 , and let Ᏺ be the foliation on U associated to ω. A leaf L ⊆ U of Ᏺ is a Rolle leaf if any C 1 curve γ : [0, 1] → U with γ (0), γ (1) ∈ L is tangent to Ᏺ at some point, that is, ω(γ (t))(γ (t)) = 0 for some t ∈ [0, 1]. Note that while a leaf of Ᏺ is in general only an immersed manifold, any Rolle leaf of Ᏺ is an embedded and closed submanifold of U . of the ﬁeld of Throughout this paper, we ﬁx an arbitrary o-minimal expansion R , then a leaf of Ᏺ is real numbers. Whenever U and a1 , . . . , an are deﬁnable in R . We use R 1 to denote the expansion of R by all Rolle leaves called a leaf over R over R . For example, the expansion Ran of the real ﬁeld generated by all globally semianalytic sets is o-minimal; in fact the sets deﬁnable in Ran are exactly the globally subanalytic sets (see [7], [4]). Building on Khovanski˘ı’s theory of fewnomials [10] and subsequent work by Moussu and Roche [14], Lion and Rolin [12] showed that (Ran )1 is also o-minimal. Adapting the various ideas involved to the general o-minimal setting, Speissegger [15] proved the following statement. 1 is o-minimal. Fact. The structure R is said to admit analytic cell decomposition if, for any The o-minimal structure R , there is a decomposition ﬁnite collection A1 , . . . , Ak ⊆ Rn of sets deﬁnable in R , such that each Ai is a union of Rn into ﬁnitely many analytic cells deﬁnable in R of cells in . In this paper we establish the following statement. admits analytic cell decomposition, then so does R 1. Theorem. If R We assume that the reader is familiar with the terminology introduced in [6] (for instance, “C k cell,” “cell decomposition,” “Whitney stratiﬁcation,” etc.). By the general results on o-minimal expansions of the real ﬁeld described there, the theorem can be restated as follows, thereby generalizing the results obtained by Cano, Lion and Moussu in [3]. Received 30 October 1998. Revision received 19 September 1999. 2000 Mathematics Subject Classiﬁcation. Primary 14P10, 58A17; Secondary 03C98. Authors partially supported by Centre National de la Recherche Scientiﬁque, Mathematical Sciences Research Institute, Natural Sciences and Engineering Research Council of Canada, and Swiss Academy of Engineering Sciences. 215

216

LION AND SPEISSEGGER

admits analytic Whitney stratiﬁcation, then so does R 1. Corollary. If R The paper is organized in the following manner. In Section 1 we introduce the notions of distribution and Pfafﬁan system, and we show how integrability can be combined with deﬁnability. These ideas are then used in Section 2 to study the lifting of deﬁnable distributions to jet space. Section 3 contains a ﬁber cutting lemma for sets 1 . In Section 4, we relate Hausdorff convergence to C k convergence in deﬁnable in R a sufﬁciently stratiﬁed situation. The proof of the main theorem is given in Section 5. The proof of the main theorem relies on a certain model completeness result for 1 , given by Theorem 5.1. This result may be of independent interest, and we extend R ) of R , which is obtained as follows. By it in Corollary 5.2 to the Pfafﬁan closure ᏼ( R i−1 generated by all Rolle leaves induction on i > 1 we let R i be the expansion of R that expands over R i−1 , and we deﬁne ᏼ( R ) to be the smallest expansion of R each R i . (Note that each R i , and hence ᏼ( R ), is o-minimal, and the theorem above ) in place of R 1 ; however, the clearly implies the corresponding statement with ᏼ( R ) is less immediate.) generalization of Theorem 5.1 to ᏼ( R We thank the referee for many helpful suggestions on the writing of this paper. In particular, we were advised to include an informal description of the proof of the main theorem, which we give at the end of Section 3. Conventions. Throughout this paper “deﬁnable” means “deﬁnable with parameters from R.” To simplify terminology, we also simply say “deﬁnable” in place of .” “deﬁnable in R The spaces Rn are equipped with the standard Euclidean metric. All manifolds in this paper occur as subsets of some ambient space Rn , and are assumed throughout to be embedded submanifolds of class (at least) C 1 in this ambient space. The only exception is in Deﬁnition 1.3, where we allow immersed submanifolds. The C 1 assumption is also in force for the use of the terms “map,” “diffeomorphism,” “cell,” and “1-form.” For a manifold M ⊆ Rn and any x ∈ M, we identify each tangent space Tx M with a subspace of Rn in the obvious way. For any X ⊆ Rn we write X for the closure of X and put ∂X := X \ X. For our purposes, a stratiﬁcation is a ﬁnite collection of disjoint manifolds in Rn for some ﬁxed (but arbitrary) n, such that if A, B ∈ satisfy A ∩ B = ∅, then A ⊆ B. A stratiﬁcation (or partition, or cell decomposition) is compatible with a set S ⊆ Rn if S is a union of members of . Let k, n ∈ N with k ≤ n. We denote by nk : Rn → Rk the projection on the ﬁrst k coordinates. More generally, if ι : {1, . . . , k} → {1, . . . , n} is strictly increasing, we let nι : Rn → Rk be the projection deﬁned by nι (x1 , . . . , xn ) := (xι(1) , . . . , xι(k) ). Whenever n is clear from context, we simply write k and ι instead of nk and nι . p

1. Distributions and Pfafﬁan systems. Let p ≤ n. We denote by Gn the Grassp mannian of all p-dimensional vector subspaces of Rn . This Gn is an analytic, real algebraic variety with a natural analytic embedding into the vector space Mn of all

ANALYTIC STRATIFICATION

217

real valued (n × n)-matrices. Each p-dimensional vector space E is identiﬁed with the unique matrix A (with respect to the standard basis of Rn ) corresponding to the orthogonal projection on the orthogonal complement of E (see [2, Section 3.4.2]); in 2 p particular, E = ker(A). We identify Mn with Rn and Gn with its image in Mn under 2 p this natural embedding; note that Gn ⊆ [−1, 1]n . p 0 n The sets Gn , . . . , Gn are the connected components of Gn := np=0 Gn . We denote by Ᏻ : Mn → Gn the map that assigns to every A ∈ Mn the unique matrix Ᏻ(A) ∈ Gn satisfying ker(Ᏻ(A)) = ker(A). Note that Ᏻ is a semialgebraic map; moreover, the p p p restriction of Ᏻ to Mn is analytic, where Mn := Ᏻ−1 (Gn ) = {A ∈ Mn : rk(A) = n − p}. We ﬁx k ≥ 1 and let M ⊆ Rn be a C k manifold; in particular, the tangent bundle T M of M is of class C k−1 . Deﬁnition 1.1. A map Ᏸ : M → Gn is called a distribution on M if Ᏸ(x) ⊆ Tx M p for all x ∈ M. If Ᏸ is a distribution on M and Ᏸ(M) ⊆ Gn for some p, then Ᏸ is a p-distribution on M. Note that if Ᏸ is a continuous distribution on M, and M is connected, then Ᏸ is a p-distribution for some p. Following the identiﬁcation above, we do not explicitly distinguish between the matrix Ᏸ(x) and its kernel, where x ∈ M. Example 1.2. Let U ⊆ Rn be open. A Pfafﬁan system of class C k on U is a family = (ω1 , . . . , ωq ) of 1-forms of class C k on U ; we call q the order of . Let S() := x ∈ U : ω1 (x) ∧ · · · ∧ ωq (x) = 0 . The Pfafﬁan system is called nonsingular if S() = ∅. Let = (ω1 , . . . , ωq ) be a Pfafﬁan system of class C k on U , with q ≤ n and ωi = ai1 dx1 + · · · + ain dxn . We associate to a distribution Ᏸ on U by putting Ᏸ (x) := Ᏻ A(x) , where A(x) ∈ Mn is the matrix qwhose ith row is (ai1 (x), . . . , ain (x)) if i ≤ q and is zero if i > q; thus Ᏸ (x) = i=1 ker ωi (x). Note that if is nonsingular, then Ᏸ is an (n − q)-distribution of class C k . Deﬁnition 1.3. Let W ⊆ M be an immersed manifold and Ᏸ a distribution on M. We say that Ᏸ is tangent to W if Ᏸ(x) ⊆ Tx W for all x ∈ W . Conversely, W is tangent to Ᏸ if Tx W ⊆ Ᏸ(x) for all x ∈ W . The distribution Ᏸ is transverse to W if dim Tx W + dim Ᏸ(x) = dim(M) for all x ∈ W . The manifold W is an integral manifold of Ᏸ if Tx W = Ᏸ(x) for all x ∈ W . A leaf of Ᏸ is a maximal connected integral manifold of Ᏸ. Assume now that Ᏸ is of class C l with 1 ≤ l < k. A section v : M → T M is a vector ﬁeld on M; v is tangent to Ᏸ if v(x) ∈ Ᏸ(x) for all x ∈ M. Let ᐂ1 (M, Ᏸ) be the collection of all vector ﬁelds on M of class C 1 and tangent to Ᏸ, and put I (Ᏸ) := x ∈ M : [v, w](x) ∈ Ᏸ(x) for all v, w ∈ ᐂ1 (M, Ᏸ) ,

218

LION AND SPEISSEGGER

where [v, w] denotes the Lie bracket of the vector ﬁelds v and w. The distribution Ᏸ is called integrable if I (Ᏸ) = M. Remark. An easy argument left to the reader shows that if M and Ᏸ are deﬁnable, then I (Ᏸ) is deﬁnable. Example 1.4. Let = (ω1 , . . . , ωq ) be a nonsingular Pfafﬁan system of class C k on an open set U ⊆ Rn . Each notion of Deﬁnition 1.3 is said to hold for if it holds for the distribution Ᏸ . Note that in this situation I (Ᏸ ) = I (), where I () := x ∈ U : ω1 (x) ∧ · · · ∧ ωq (x) ∧ dωr (x) = 0, r = 1, . . . , q . If is integrable, then deﬁnes a foliation on U whose leaves are exactly the leaves of in the above sense. In particular, the leaves of the foliation Ᏺ in the introduction are exactly the leaves of the Pfafﬁan system (ω), and henceforth we simply call them the leaves of ω. Let W ⊆ U be a submanifold. A subsystem of is a basis of along W if

is transverse to W and Tx W ∩ Ᏸ (x) = Tx W ∩ Ᏸ (x) for all x ∈ W . Deﬁnition 1.5. Let N ⊆ Rl be a C 2 manifold, and let f : N → M be a C 2 map and Ᏸ a distribution on M. The pullback of Ᏸ on N by f is the distribution f ∗ Ᏸ on N deﬁned by f ∗ Ᏸ(x) := (dfx )−1 Ᏸ f (x) . The following lemma is crucial to this paper; its proof can be found in [8, Chapter III, Corollary 6.19]. Lemma 1.6. Assume that k ≥ 3, and let Ᏸ be a p-distribution on M of class C k−1 . Then any integral manifold of Ᏸ is contained in I (Ᏸ). 2. Retrieving equations. The ideas of this section are based on the use of “jet space” (introduced as J j below) in combination with the notion of deﬁnable sets in an o-minimal structure; they considerably extend corresponding applications outlined in [11]. A traditional development of the notion of jet space can be found in [1] or [9]. Throughout this section we ﬁx p ≤ n and k > 0. We deﬁne analytic submanifolds j J = J j (n, p) ⊆ Rnj by induction on j in the following manner. We let n0 := n and J 0 := Rn , and for j > 0 we put nj := nj −1 + n2j −1 , and let p

J j := J j −1 × Gnj −1 ⊆ Rnj . j

For i ≤ j we denote by σi : J j → J i the canonical projection on the ﬁrst ni coordinates. Deﬁnition 2.1. Let W ⊆ Rn be a manifold of class C k and dimension p. The Gauss map gW associated to W is the distribution gW on W deﬁned by g(x) := Tx W . Note that gW is of class C k−1 , so that the manifold T 1 (W ) := graph(gW )

ANALYTIC STRATIFICATION

219

is a p-dimensional submanifold of J 1 of class C k−1 . We deﬁne the p-dimensional submanifolds T j (W ) of J j for j = 0, . . . , k as follows: We let T 0 (W ) := W , and for j > 0 we put T j (W ) := T 1 T j −1 (W ) . Note that each T j (W ) is of class C k−j , and if W is bounded, then each T j (W ) is bounded. Let M ⊆ Rn be a submanifold of class C k+1 and dimension m ≥ p, and let Ᏸ be a p-distribution on M of class C k . Deﬁnition 2.2. We deﬁne by induction on j = 0, . . . , k a manifold M j ⊆ J j of class C k+1−j and a distribution bj (Ᏸ) on M j (called the j th blow-up of Ᏸ) as follows: M 0 := M and b0 (Ᏸ) := Ᏸ, and for j > 0 we put M j := graph(bj −1 (Ᏸ)) and

∗ j bj (Ᏸ) := σj −1 M j bj −1 (Ᏸ). Note that if W ⊆ M is an integral manifold of Ᏸ, then each T j (W ) ⊆ M j is an integral manifold of bj (Ᏸ). Proposition 2.3. Let A ⊆ J k be deﬁnable and of dimension d ≤ k + p, and let l ≥ k + 3. Then there is a partition Ꮿ of A into distinct deﬁnable C l cells C1 , . . . , Cr , D1 , . . . , Ds such that k (D ) of (1) each Di is the graph of an integrable p-distribution Ᏸi on σk−1 i class C l−1 ; (2) if W ⊆ Rn is a C k manifold of dimension p such that T k (W ) ⊆ E for some E ∈ Ꮿ, then E = Di for some i ∈ {1, . . . , s} and T k−1 (W ) is an integral manifold of Ᏸi . Remark. Let A and Ꮿ be as in Proposition 2.3, and let W ⊆ Rn be a C k manifold of dimension p such that T k (W ) ⊆ A. It then follows from Proposition 2.3 that, for every i, any manifold contained in T k (W ) ∩ Ci has dimension less than p. Proof of Proposition 2.3. We proceed by induction on d ≤ k + p (simultaneously for all k, l, and p). The case d = 0 is trivial, so we assume that d > 0 and that the proposition holds for lower values of d. By C l cell decomposition, we may assume k (A) and d := dim(A ). We distinguish that A is a deﬁnable C l cell. Let A := σk−1 the following two cases. Case 1. Take d = d; then A is the graph of a map Ᏸ : A → Gdnk−1 of class C l−1 . By cell decomposition, there is a partition Ꮿ of A into deﬁnable C l cells, such that for each C ∈ Ꮿ one of the following holds: (i) Ᏸ(x) Tx C for all x ∈ C ; (ii) Ᏸ is a distribution on C and C ∩ I (Ᏸ) = ∅;

220

LION AND SPEISSEGGER

(iii) Ᏸ is a distribution on C and C ⊆ I (Ᏸ). Let Ꮿ := {graph(Ᏸ|C ) : C ∈ Ꮿ }; then Ꮿ is a partition of A into deﬁnable C l cells, and we claim that Ꮿ works. Let W ⊆ Rn be a C k manifold of dimension p such that T k (W ) ⊆ C for some k (C). We need to show that C satisﬁes (iii) above. Note that C ∈ Ꮿ, and put C := σk−1 T k−1 (W ) is contained in C and is tangent to Ᏸ. Therefore, Ᏸ(x) = Tx (T k−1 (W )) ⊆ Tx C for all x ∈ T k−1 (W ), so that Ᏸ is a distribution on C by our choice of Ꮿ . By Lemma 1.6 we also have T k−1 (W ) ⊆ I (Ᏸ), that is, C ⊆ I (Ᏸ). Case 2. Take d < d; then d ≤ k − 1 + p and l + 1 ≥ k − 1 + 3, and we choose a partition Ꮿ of A into distinct deﬁnable C l+1 cells C1 , . . . , Cr , D1 , . . . , Ds according to two cases. Subcase a. Take k = 1, so that d ≤ p. Let Ꮿ be any partition of A into deﬁnable cells, and let C1 , . . . , Cr be the list of all distinct cells in Ꮿ of dimension less than p. Let D1 , . . . , Ds the list of all distinct cells in Ꮿ of dimension equal to p.

C l+1

Subcase b. Take k > 1. By the inductive hypothesis, we can ﬁnd Ꮿ such that k−1 (1 ) each Di is the graph of an integrable p-distribution Ᏸ i on σk−2 (Di ) of class l C; (2 ) if W ⊆ Rn is a C k−1 manifold of dimension p such that T k−1 (W ) ⊆ E for some E ∈ Ꮿ , then E = Di for some i ∈ {1, . . . , s } and T k−2 (W ) is an integral manifold of Ᏸ i . After choosing Ꮿ , we let Ꮿ be a partition of A into deﬁnable C l cells obtained as follows: k )−1 (C ) be an element of Ꮿ. (i) For each Ci we let A ∩ (σk−1 i

(ii) For each Di we put 1 if k = 1; T Di , Bi := 1 graph b Ᏸi , if k > 1. Since dim(A∩Bi ) < d, we use the inductive hypothesis to partition A∩Bi as desired, and we add each cell of this partition to Ꮿ. Finally, using C l cell decomposition we k )−1 (D ) and add each cell of this partition to Ꮿ. Note partition the set (A\Bi )∩(σk−1 i k (C) is contained in that Ꮿ is a partition of A into deﬁnable C l cells, and that σk−1 some element of Ꮿ for all C ∈ Ꮿ. Let now W ⊆ Rn be a C k manifold of dimension p such that T k (W ) ⊆ C for k (C) ⊆ D for some i. Now, if k = 1, then some C ∈ Ꮿ. Then T k−1 (W ) ⊆ σk−1 i

k−1 (W ) is an integral manifold of the Gauss map dim(W ) = p = dim(Di ); hence, T gDi . If k > 1, then T k−2 (W ) is an integral manifold of Ᏸ i ; therefore, T k−1 (W ) is an integral manifold of b1 (Ᏸ i ). Thus, in both cases, T k (W ) is contained in one of the cells of Ꮿ contained in Bi . This ﬁnishes the proof of case 2 and hence the proof of Proposition 2.3.

ANALYTIC STRATIFICATION

221

3. A ﬁber cutting lemma for Pfafﬁan limits. In this section we develop the notion of a Pfafﬁan limit (Deﬁnition 3.2) and prove a related ﬁber cutting lemma (Lemma 3.6). The notion of Pfafﬁan limit was ﬁrst introduced in [12] in the subanalytic context; here we adapt and extend some of the results there to the o-minimal setting of [15]. 1 denotes the o-minimal expansion of R generated by Throughout this section, R (as deﬁned in the introduction). all Rolle leaves over R Deﬁnition 3.1. Let U ⊆ Rn be open and deﬁnable. A Pfafﬁan system = if each ωi is deﬁnable, nonsingular, and inte(ω1 , . . . , ωq ) on U is called R-Pfafﬁan grable. Note that in this case Ᏸ is also deﬁnable. A set W ⊆ Rn is called basic R-Pfafﬁan if there are an open deﬁnable set U ⊆ Rn , an R-Pfafﬁan system = (ω1 , . . . , ωq ) on U , a deﬁnable set A ⊆ U , and Rolle leaves Li of ωi for each i, such that W = A ∩ L1 ∩ · · · ∩ Lq . Remark. The notion of basic R-Pfafﬁan set deﬁned above is more restrictive than in [15, Section 2], because we assume here that the 1-forms involved are all integrable. However, one easily veriﬁes using Lemma 1.6 and cell decomposition that the collection of all ﬁnite unions of basic R-Pfafﬁan sets as deﬁned here is the same as the collection of R-Pfafﬁan sets deﬁned there. -Pfafﬁan set. A set X ⊆ Rn Deﬁnition 3.2. Let W ⊆ Rm+n be a bounded basic R is called an R-Pfafﬁan limit obtained from W if there is a sequence (1j )j ∈N with 1j ∈ Rm such that X = lim W1j , j →∞

∈ Rn

where we put Wz := {x : (z, x) ∈ W } for all z ∈ Rm . (Here the limit is taken in the topological space of all compact subsets of Rn equipped with the usual Hausdorff distance. In this space the empty set is an isolated point.) Without loss of generality, we always assume that 1i → 1 ∈ Rm ; in particular, {1} × X = limi {1i } × W1i . limit obtained from W ; this is equivalent to X Take X ⊆ Rn to be an R-Pfafﬁan being a point in the closure of {Wz : z ∈ Rm } in the space of compact subsets of Rn . 1 . Conversely, We show in Lemma 3.3 that any R-Pfafﬁan limit is deﬁnable in R it follows from Corollary 3.7 and the o-minimality of R 1 that every bounded set 1 is a ﬁnite union of sets of the form X \ X , where both X and X are deﬁnable in R projections of R-Pfafﬁan limits. For the remainder of this section, we ﬁx a bounded nonempty R-Pfafﬁan set W ⊆ m+n . We let X = limi W1i be an R-Pfafﬁan limit obtained from W and put R p := max dim(Wz ) . z∈Rm

Passing to a subsequence if necessary, we may assume that the sequence (∂W1i )i converges to a compact set X ⊆ Rn .

222

LION AND SPEISSEGGER

1 , and dim(X) ≤ p, while Lemma 3.3. Both X and X are deﬁnable in R

dim(X ) < p. Proof. We use the Marker-Steinhorn theorem [13], as in [5]. We just consider X, since the case of X is similar. Since X = limi W1i , there is for every ﬁnite set F ⊆ Rn an 1 > 0 and a z ∈ Rm such that for every x ∈ F , x ∈ X if and only if d(x, Wz ) < 1. It follows from model 1 with underlying theoretic compactness that there is an elementary extension ᏾ of R 1+m n set R, and there is an (1, z) ∈ R , such that for all x ∈ R we have x ∈ X if and only if d(x, Wz∗ ) < 1, where W ∗ is the subset of R m+n deﬁned by the same formula of 1 that deﬁnes W . It follows that 1 is inﬁnitesimal and X = st(W ∗ ). the language of R z 1 and Corollary 1.3 and Proposition 1.10 of [5] now imply that X is deﬁnable in R dim(X) ≤ p. Deﬁnition 3.4. We call X a proper R-Pfafﬁan limit obtained from W if dim(X) = s limit if Y is a proper R-Pfafﬁan limit obtained p. A set Y ⊆ R is a proper R-Pfafﬁan from some bounded nonempty basic R-Pfafﬁan set V ⊆ Rr+s . Remark 3.5. Let ᏾ be an o-minimal expansion of the ﬁeld of real numbers, and let S ⊆ Rk be deﬁnable in ᏾ with l := dim(S). Then there is a set Y ⊆ S, deﬁnable in ᏾, such that S ⊆ Y , and for every x ∈ Y there is a strictly increasing λ : {1, . . . , l} → {1, . . . , k} such that x is isolated in S ∩ −1 λ (λ (x)). To see this, let be a stratiﬁcation of Rk into cells deﬁnable in ᏾ such that is compatible with S. Then we can take Y := Yl ∪ Yl−1 ∪ · · · ∪ Y0 , where Yl := C ∈ : C ⊆ S and dim(C) = l , and for i = 0, . . . , l − 1, Yi := C ∈ : C ⊆ S \ Yl ∪ · · · ∪ Yi+1 and dim(C) = i . Lemma 3.6 (Fiber cutting). Let r ≤ n. Then there are proper R-Pfafﬁan limits n X1 , . . . , XK ⊆ R such that r (X) = r (X1 ) ∪ · · · ∪ r (XK ) and dim(r (Xi )) = dim(Xi ) for each i. Proof. We proceed by induction on d := dim(W ); the case d = 0 is trivial, so we assume that d > 0 and that the lemma holds for lower values of d. Let = system on a deﬁnable open set U ⊆ Rm+n , A ⊆ U a (ω1 , . . . , ωq ) be an R-Pfafﬁan bounded deﬁnable set, and Li Rolle leaves of ωi for each i, such that W := A ∩ L1 ∩ · · · ∩ Lq . By [15, Lemma 2.8], we reduce to the case that A is a deﬁnable C 2 manifold and that every subsystem of (ω1 , . . . , ωq , dz1 , . . . , dzm , dx1 , . . . , dxn ) has a basis along A, where (z1 , . . . , zm , x1 , . . . , xn ) denotes the usual coordinate system on U ⊆ Rm+n . Thus, by [15, Corollary 2.9], we have for all z ∈ Rm that either Wz = ∅ or

ANALYTIC STRATIFICATION

223

dim(Wz ) = p, and for every s ≤ r and every strictly increasing map λ : {1, . . . , s} → {1, . . . , r} the rank rm,λ := rk(m,λ |W ) is constant. (Here m,λ : Rm+n → Rm+s denotes the projection on the coordinates (z1 , . . . , zm , xλ(1) , . . . , xλ(s) ).) Let s := dim(r (X)); if s = p, then we are done by Lemma 3.3, so we assume that s < p. Let λ : {1, . . . , s} → {1, . . . , r} be strictly increasing. Since s < p ≤ d and n −1 m+s , we have dim(W ∩ W ∩ −1 m,λ (z, y) = {z} × (Wz ∩ (λ ) (y)) for all (z, y) ∈ R −1 −1 m,λ (z, y)) ≥ p − s > 0 whenever W ∩ m,λ (z, y) = ∅; that is, rm,λ < d. Hence, by [15, Proposition 2.2], there is an R-Pfafﬁan set W λ ⊆ W such that dim(W λ ) < d and for all (z, y) ∈ Rm+s , each component of the ﬁber W ∩−1 m,λ (z, y) intersects the −1 λ ﬁber W ∩ m,λ (z, y). In particular, nλ (Wzλ ) = nλ (Wz ), and every component of r (Wz ) ∩ (rλ )−1 (y) intersects the ﬁber r (Wzλ ) ∩ (rλ )−1 (y). Passing to a subsequence if necessary, we may assume that the sequence (W1λi )i converges, and we put Xλ := limi W1λi . By the inductive hypothesis the lemma holds with X λ in place of X for each strictly increasing λ : {1, . . . , s} → {1, . . . , r}. Therefore, it remains to show that r (X) = λ λ r (X ). To see this, we ﬁx a strictly increasing λ : {1, . . . , s} → {1, . . . , r}; since each r (X λ ) is closed, it sufﬁces by Remark 3.5 to establish the following claim. Claim. Let y ∈ λ (X), and let x ∈ r (X) ∩ (rλ )−1 (y) be isolated. Then x ∈ r (Xλ ). To prove the claim, note that r (X) = limi r W1i , since W is bounded. Let xi ∈ r (W1i ) be such that xi → x, and put yi := λ (xi ). Let Ci ⊆ Rr be the component of r (W1i ) ∩ (rλ )−1 (yi ) containing xi , and let xi belong to Ci ∩ r (W1λi ). Since we also have r (X λ ) = limi r W1λi , we may assume, after passing to a subsequence if necessary, that xi → x ∈ r (X λ ). We show that x = x, which then proves the claim. Assume for a contradiction that x = x, and let δ > 0 be such that δ ≤ d(x, x ) and (3.1)

−1 B(x, δ) ∩ r (X) ∩ rλ (y) = {x},

where B(x, δ) is the open ball with center x and radius δ. Then for all sufﬁciently large i, there is an xi

∈ Ci such that δ/3 ≤ d(xi

, xi ) ≤ 2δ/3, because xi , xi ∈ Ci and Ci is connected. Passing to a subsequence if necessary, we may assume that xi

→ x

∈ r (X). Then x

∈ B(x, δ) with x

= x, and since xi

∈ Ci implies that λ (xi

) = yi , we get λ (x

) = y, contradicting (3.1). This establishes the claim, and hence the lemma is proved. 1 . Then there exist Corollary 3.7. Let S ⊆ RN be a compact set deﬁnable in R n proper R-Pfafﬁan limits X1 , . . . , XK ⊆ R for some n ≥ N such that S = N (X1 ) ∪ · · · ∪ N (XK ) and dim(N (Xi )) = dim(Xi ) for each i.

224

LION AND SPEISSEGGER

Proof. Since S is a <∞ -set and hence a ﬁnite union of basic <∞ -sets (see [15]), we may assume that there is a basic <∞ -set S ⊆ RN such that S = S . Let W ⊆ m+N+l be an R-Pfafﬁan set such that S is a <∞ -set obtained from W , say, S = R i j N W1i,j with each W1i,j compact. Using a semialgebraic diffeomorphism τ : R → (−1, 1), we may assume that W is bounded. Since W is a ﬁnite union of basic R-Pfafﬁan sets, we may further reduce to the case that W is a basic R-Pfafﬁan set

and that S = i j N (W1i,j ). Thus S = S = N (limi limj W1i,j ). The corollary now follows from Lemma 3.6, in view of a remark in Deﬁnition 3.2. We have now gathered enough information to give a description of the proof of the admits analytic cell decomposition, and let X ⊆ Rn main theorem. Suppose that R be deﬁnable in R 1 . Since R 1 is o-minimal, we know that X can be partitioned into 1 . In order to conclude that ﬁnitely many C 1 manifolds W1 , . . . , WK , deﬁnable in R we can choose each Wl to be analytic, we then try to ﬁnd an integrable, analytic of which Wl is an integral manifold. However, we do not distribution deﬁnable in R know if this is possible in general; instead, we show that Wl is the diffeomorphic projection of such an integral manifold. Since by the theorem of Frobenius integral manifolds of integrable, analytic distributions are analytic manifolds, we can then conclude the theorem. imply that we only need to ﬁnd distribuThe analytic stratiﬁcation properties of R 1 tions of class at least C ; to do so, we proceed by induction on p := dim(X). Given any integer k > 0, we can reduce by Corollary 3.7 to the case where X is the Hausdorff limit of a sequence of the closures of leaves Vi , i ∈ N, of some p-distribution , where M ⊆ Rn is a bounded C k+1 manifold of Ᏸ on M of class C k deﬁnable in R dimension m ≥ p. The problem is that, while for any given point x ∈ M there is at most one leaf of Ᏸ passing through x, a point x in the boundary of M may belong to many different Hausdorff limits all associated to this Ᏸ. To separate the various limits, we choose k ≥ m + 3 and work with M k , bk (Ᏸ), and T k (Vi ) in place of M, Ᏸ, and Vi . What we have gained now is the following (see and dim(A) ≤ m, then Proposition 2.3): If A ⊆ J m+1 is a C 1 cell deﬁnable in R

there is a p-distribution Ᏸ on σm (A). This distribution has the following property: Whenever W ⊆ σ0 (A) is a C m+1 submanifold of dimension p such that T m+1 (W ) ⊆ A, then T m (W ) is an integral manifold of Ᏸ . 1 and Finally, we show in the next section that there is a set Z ⊆ X deﬁnable in R of dimension at most p − 1, such that X \ Z can be partitioned into ﬁnitely many 1 with the property that T m+1 (W ) ⊆ M m+1 . C m+1 submanifolds W deﬁnable in R Since dim(M m+1 ) = n, the previous paragraph, combined with routine stratiﬁcation arguments in o-minimal structures, as well as the inductive hypothesis applied to Z, allow us to conclude that X is of the desired form (see Theorem 5.1). 4. Hausdorff convergence vs. C k convergence. We relate here the notion of convergence in the space of all compact subsets of Rn with that of C k convergence,

ANALYTIC STRATIFICATION

225

in stratiﬁed situations, as encountered in the context of o-minimal structures. We begin with a general lemma. Lemma 4.1. Let U ⊆ Rn be open, let X and Vi , i ∈ N, be closed submanifolds of p U of dimension p, and let N > 0 and g1 , . . . , gN : X → Gn be continuous functions whose graphs 1 , . . . , N are pairwise disjoint. Assume that for every compact set K ⊆ U we have p p lim T 1 (Vi ) ∩ K × Gn = 1 ∪ · · · ∪ N ∩ K × Gn . i

Then N = 1 and 1 = T 1 (X). Proof. The following quick proof was suggested to us by the referee. Let x ∈ X. p We take K = {x} and then note that T 1 (Vi )∩(K ×Gn ) is empty if x ∈ / Vi and equals the singleton {(x, Tx Vi )} if x ∈ Vi . It follows from the convergence assumption that x ∈ Vi for all but ﬁnitely many i, and hence N = 1. Next, take for K a compact ball in Rn centered at x such that K ⊆ U . By the previous argument, X ∩ K ⊆ i ( j ≥i Vj ∩ K), so by Baire there is a nonempty open V ⊆ X and an i such that V ⊆ Vj ∩ K for all j ≥ i. Then Ty X = Ty V = Ty Vj for y ∈ V and j ≥ i. Hence, by the convergence assumption, Ty X = g1 (y) for y ∈ V . As K shrinks, the points y ∈ V approach x, so Tx X = g1 (x). The lemma follows since x was arbitrary.

be an o-minimal expansion of the ﬁeld of real numbers, and Deﬁnition 4.2. Let R n let M ⊆ R be a bounded submanifold. Let Ᏸ be a p-distribution on M with p ≤ n, and let (Vi )i∈N be a sequence of integral manifolds of Ᏸ. Also let Y, Y be compact

system of class C k subsets of J k . The tuple (M, Ᏸ, (Vi )i , Y, Y ) is called an R-limit n in R if the following are true:

M is of class C k+1 , and Ᏸ is of (i) M, Ᏸ, each Vi , Y , and Y are deﬁnable in R, class C k ; (ii) the sequence (T k (Vi ))i converges to Y and dim(Y ) = p; (iii) the sequence (∂T k (Vi ))i converges to a compact set contained in Y and dim(Y ) < p.

Remark 4.3. Let (M, Ᏸ, (Vi )i , Y, Y ) be an R-limit system of class C k in Rn . k k

and satis(1) By C cell decomposition, there is a set Z ⊆ σ0 (Y ), deﬁnable in R k fying dim(Z) < p, and there are disjoint C k cells Y1 , . . . , YK ⊆ σ0 (Y ), deﬁnable in

and of dimension p, such that (a) σ k (Y ) = Z ∪ Y1 ∪ · · · ∪ YK , and (b) for each j , R 0 we have Yj ∩σ0k (Y ) = ∅ and each component of Y ∩(σ0k )−1 (Yj ) is the graph of a C k p map gj : Yj → Gn .

sys(2) Since M is bounded, the tuple (M, Ᏸ, (Vi )i , σjk (Y ), σjk (Y )) is an R-limit j n tem of class C in R whenever 1 ≤ j ≤ k. Similarly, if k > 1, then the tuple

system of class C k−1 in Rn1 . (graph(Ᏸ), b1 (Ᏸ), (T 1 (Vi ))i , Y, Y ) is an R-limit Example 4.4. Let W ⊆ Rm+n be a bounded R-Pfafﬁan set and X ⊆ Rn a proper R-Pfafﬁan limit of dimension p obtained from W (say, X = limi W1i with 1i ∈ Rm ).

226

LION AND SPEISSEGGER

Next, we let = (ω1 , . . . , ωq ) be an R-Pfafﬁan system on a deﬁnable open set U ; let Li be a Rolle leaf of ωi for each i; and let A ⊆ U be bounded deﬁnable such that W = A ∩ L1 ∩ · · · ∩ Lq . In addition, we assume that A is a C k+1 manifold M and (ω1 , . . . , ωq , dz1 , . . . , dzm ) has a basis @ along M, and we denote by Ᏸ the pullback on M, via the inclusion map ι : M → U , of the p-distribution Ᏸ@ associated to @. By [15, Lemma 1.6], each Vi := {1i } × W1i is an integral manifold of Ᏸ. We suppose furthermore that both sequences (T k (Vi ))i and (∂T k (Vi ))i converge; let Y and Y be 1 -limit system of class C k in their respective limits. Then (M, Ᏸ, (Vi )i , Y, Y ) is an R m+n R . To see this, note that each W k := graph(bk−1 (Ᏸ|W )) is a basic R-Pfafﬁan set such that T k (Vi ) = {1i } × (W k )1i , so (ii) and (iii) from Deﬁnition 4.2 follow from Lemma 3.3.

be an o-minimal expansion of the ﬁeld of real numbers; let k ≥ 1 and Let R

system of class C k ; and put p := dim(Y ). (M, Ᏸ, (Vi )i , Y, Y ) be an R-limit

and of dimension less than Proposition 4.5. There is a set Z ⊆ Rn , deﬁnable in R k n

and of dimension p, and there are disjoint C cells Y1 , . . . , YK ⊆ R , deﬁnable in R k p, such that σ0 (Y ) = Z ∪ Y1 ∪ · · · ∪ YK and, for each j , −1 (Yj ). T k (Yj ) = Y ∩ σ0k In particular, each T k (Yj ) is contained in M k . Proof. We proceed by induction on k.

Case 1. Take k = 1. By Remark 4.3(1), there is a set Z ⊆ Rn , deﬁnable in R 1

and of with dim(Z) < p, and there are disjoint C cells Y1 , . . . , YK , deﬁnable in R dimension p, such that (a) σ01 (Y ) = Z ∪ Y1 ∪ · · · ∪ YK , and (b) for all j , we have p Yj ∩ σ01 (Y ) = ∅ and there are C 1 maps gj,1 , . . . , gj,ν(j ) : Yj → Gn whose graphs j,1 , . . . , j,ν(j ) are disjoint such that −1 Y ∩ σ01 (Yj ) = j,1 ∪ · · · ∪ j,ν(j ) . Fix j ∈ {1, . . . , K}; we claim that T 1 (Yj ) = Y ∩ (σ01 )−1 (Yj ). To see this, we ﬁx an x ∈ Yj and let U be an open neighbourhood of x, such that cl(U ) ∩ σ01 (Y ) = ∅. Passing to a subsequence and shrinking U if necessary, we may therefore assume that Yj ∩U and each Vi ∩U is closed in U and that, for every compact p K ⊆ U , the sequence (T 1 (Vi ) ∩ (K × Gn ))i converges to Y ∩ (σ01 )−1 (Yj ∩ K). Thus T 1 (Yj ∩ U ) = Y ∩ (σ01 )−1 (Yj ∩ U ), by Lemma 4.1 applied to X = Yj ∩ U . Since x ∈ Yj was arbitrary, this ﬁnishes case 1. Case 2. Take k > 1. We assume the proposition holds for lower values of k. Thus by Remark 4.3(2) and the inductive hypothesis, we have the following.

ANALYTIC STRATIFICATION

227

with dim(Z ) < p, and there are disjoint (a) There is a set Z ⊆ Rn , deﬁnable in R

n

cells Y1 , . . . , Yr ⊆ R , deﬁnable in R and of dimension p, such that σ0k (Y ) =

Z ∪ Y1 ∪ · · · ∪ Yr , and for each j ,

C1

−1 T 1 Yj = σ1k (Y ) ∩ σ01 Yj .

with dim(Z

) < p, and there are (b) There is a set Z

⊆ J 1 , deﬁnable in R

k−1

1

and of dimension p, such that cells Y1 , . . . , Ys ⊆ J , deﬁnable in R disjoint C k

σ1 (Y ) = Z ∪ Y1 ∪ · · · ∪ Ys , and for each j , −1

T k−1 Yj

= Y ∩ σ1k Yj .

and We let ᏿ := {S1 , . . . , SL } be a cell decomposition of J 1 into C k cells deﬁnable in R k

1 1

compatible with σ1 (Y ), T (Y1 ), . . . , T (Yr ) as well as Y1 , . . . , Ys and Z . We also put Yj := σ01 (Sj ) for each j . Then {Y1 , . . . , YL } is compatible with σ0k (Y ) and Y1 , . . . , Yr , and for each Yj with Yj ⊆ Y1 ∪ · · · ∪ Yr and dim(Yj ) = p we have T 1 (Yj ) ∈ ᏿. Thus, k −1 k for each such Yj , we have, by (a) and (b) above, that T (Yj ) = Y ∩ (σ0 ) (Yj ). We therefore put Z := {Yj : dim(Yj ) < p} and let {Y1 , . . . , YK } be the list of those Yj that are of dimension p and contained in Y1 ∪ · · · ∪ Yr . The last assertion follows from the fact that Y ⊆ M k , which ﬁnishes the proof of the proposition. 5. Analytic cell decomposition. We now formulate a description of the sets 1. deﬁnable in R 1 . There are cells S1 , . . . , SM , deTheorem 5.1. Let S ⊆ Rn be deﬁnable in R ﬁnable in R 1 , such that S = S1 ∪ · · · ∪ SM and for each S = Si the following holds: There is an N ≥ n, a deﬁnable manifold M ⊆ RN , a deﬁnable, integrable 1 , such p-distribution Ᏸ on M with p := dim(S ), and a cell V ⊆ M deﬁnable in R that V is an integral manifold of Ᏸ and S = n (V ). 1 is model Remark. In model-theoretic terminology, this proposition implies that R complete in the language of all sets of the form of V above. We do not know if each V in the theorem can be taken to be an R-Pfafﬁan set. Proof of Theorem 5.1. Using a semialgebraic, analytic diffeomorphism from R onto (−1, 1), we may assume that S is bounded. We proceed by induction on p := dim(S); if p = 0, the proposition is trivial, so we assume that p > 0. By o-minimality and the inductive hypothesis, it sufﬁces to prove the proposition with S in place of S; that is, we may assume that S is compact. By Corollary 3.7, we further reduce to the

case where S = n (X) for some proper R-Pfafﬁan limit X ⊆ Rn , with n ≥ n and dim(X) = dim(S).

228

LION AND SPEISSEGGER

To streamline notation, we now rename n to n. We show the following: (∗)

1 and of dimension less than p, such There is a set Z ⊆ Rn , deﬁnable in R that the proposition holds with X \ Z in place of S.

This ﬁnishes the proof: given r ≤ n, each cell r (Si ) of dimension p is of the required nature, while r (Z) and the cells r (Si ) of dimension less than p are handled by the inductive hypothesis. set and (1i )i ⊆ Rm To prove (∗), we let W ⊆ Rm+n be a bounded R-Pfafﬁan such that 1i → 1 and X = limi W1i . Furthermore, we let = (ω1 , . . . , ωq ) be an R-Pfafﬁan system on a deﬁnable open set U ⊆ Rm+n , Li a Rolle leaf of ωi for each i, and A ⊆ U bounded deﬁnable such that W = A ∩ L1 ∩ · · · ∩ Lq . Let := (ω1 , . . . , ωq , dz1 , . . . , dzm ). By [15, Lemma 2.1 and Theorem 2.6], we may assume that is of class C k for some ﬁxed k ≥ d −p +3, that A is a C k+1 manifold M, and 1 -limit system that there is a basis @ of along M. Let (M, Ᏸ, (Vi )i , Y, Y ) be the R k m+n of class C in R associated to X as in Example 4.4. To streamline notation again, we now rename m + n to n and {1} × X to X; in particular, (M, Ᏸ, (Vi )i , Y, Y ) is an 1 -limit system of class C k in Rn . R Note that X = σ0k (Y ), and let Z, Y1 , . . . , YK be the subsets of X obtained from Proposition 4.5. Since dim(Z) < p, it sufﬁces to prove (∗) with each Yi in place of 1 such that T k (X) is X. Thus, we may assume that X is a C k cell deﬁnable in R contained in M k . Since dim(M k ) = d, we can apply Proposition 2.3 with A = M k to obtain a partition Ꮿ of M k into distinct deﬁnable C k+3 cells C1 , . . . , Cr , D1 , . . . , Ds and pdistributions Ᏸ1 , . . . , Ᏸs associated to D1 , . . . , Ds . Let Ꮿ be a stratiﬁcation of T k (X) 1 and compatible with each element of Ꮿ, and put into C k cells deﬁnable in R Z := σ0k

C ∈ Ꮿ : dim(C ) < p

.

Then dim(Z) < p. Now let V ∈ Ꮿ be such that dim(V ) = p. Since also dim(T k (X)) = p and T k (X) is the graph of a continuous function deﬁned on X, the cell V is the graph of a continuous function deﬁned on W := σ0k (V ). Furthermore, both T k (X) and V are cells, so V is open in T k (X); hence W is open in X. It follows that V = T k (W ). By part (2) of Proposition 2.3, we conclude that T k (W ) ⊆ Di for some k (V ) is an integral manifold of Ᏸ . We therefore i ∈ {1, . . . , s} and T k−1 (W ) = σk−1 i k (V ) such that V ∈ Ꮿ and dim(V ) = p, and the let V1 , . . . , VM be the list of all σk−1 theorem is proved. ) in place of R 1. Corollary 5.2. The previous proposition holds with ᏼ( R K . We proceed by induction on Proof. Let K ∈ N be such that S is deﬁnable in R K and p := dim(S). If p = 0, the statement is trivial; if K = 1, the corollary follows

ANALYTIC STRATIFICATION

229

from the previous proposition. So we assume that K > 1 and p > 0, and that the corollary holds for lower values of K or p. K−1 in place of R , we may assume that there By the inductive hypothesis, with R

m is an m ≥ n and a manifold M ⊆ R deﬁnable in R K−1 ; there is an integrable K−1 ; and there is a cell W ⊆ M , p-distribution Ᏹ on M that is deﬁnable in R K , such that W is an integral manifold of Ᏹ and S = m (W ). In deﬁnable in R n particular, S is a cell of dimension p. Let d := dim(M ). By using the inductive hypothesis again, we may assume that there is an N ≥ m + m2 and a deﬁnable manifold M ⊆ RN ; there is a deﬁnable, K−1 , integrable d-distribution Ᏹ on M; and there is a cell W ⊆ M, deﬁnable in R N

such that graph(Ᏹ ) = m+m2 (W ) and W is an integral manifold of Ᏹ.

Let σ : Rm+m → Rm be the projection on the last m2 coordinates. Note that for p any x ∈ M we have σ ◦ N (x) ∈ Gm , and denote by H (x) the corresponding m+m2 p m N p-subspace of Rm . We have N m (W ) ⊆ R × Gm , and for all x ∈ W , m |Ᏹ(x) has N rank d and m (Ᏹ(x)) contains H (x). We may thus assume by cell decomposition and the inductive hypothesis that 2

2

p

N N (M) ⊆ Rm × Gm and for all x ∈ M, N m |Ᏹ(x) has rank d and m (Ᏹ(x)) m+m2 contains H (x).

Therefore, we obtain a deﬁnable p-distribution Ᏸ on M by putting

Ᏸ(x) := Ᏹ(x) ∩ N m

−1

H (x) .

−1 We now let V := (N n ) (S) ∩ W . Then

−1 m −1

V = N (S) . n W

m W N Since m n |W and m |W are diffeomorphisms, it follows that V is a submanifold of K , such that N (V ) = S. On the other hand, for W of dimension p, deﬁnable in R n

(x)); hence T every x ∈ W we have H (x) = Ᏹ (N x V = Ᏸ(x), for all x ∈ V . m

has analytic cell decomposition. We assume for the remainder of this section that R ∞ (Alternatively, we might assume that R has C cell decomposition; then Lemma 5.3 and Corollary 5.4 go through with “C ∞ ” in place of “analytic.”) 1 . Then S is a ﬁnite union of analytic Lemma 5.3. Let S ⊆ Rn be deﬁnable in R manifolds that are deﬁnable in R 1 . Proof. The proof is by induction on p := dim(S). The case p = 0 is trivial, so we assume that p > 0 and the lemma holds for lower values of p. By Theorem 5.1, we

230

LION AND SPEISSEGGER

may assume that S = N n (V ), where N, M, Ᏸ, and V are as stated in that theorem. By the inductive hypothesis, we may assume that dim(V ) = p = dim(S). Using analytic cell decomposition and the inductive hypothesis, an easy argument (by induction on the dimension of M) shows that M and Ᏸ can be chosen to be analytic. The Frobenius integrability theorem in the analytic setting then implies that V is an analytic manifold. N Since V is a C 1 cell and dim(N n (V )) = dim(V ), n |V is a diffeomorphism. Hence, N n |V is an analytic diffeomorphism and S an analytic manifold. 1 has analytic cell decomposition. Corollary 5.4. R Proof. We show by induction on n that if ᏿ is a ﬁnite collection of subsets of Rn 1 , then there is a decomposition of Rn into analytic cells deﬁnable in deﬁnable in R R 1 that is compatible with each member of ᏿. The cases n = 0, 1 are trivial, so we 1 , with S ⊆ Rn−1 assume that n > 1. Let f : S → R be a function deﬁnable in R a cell. By cell decomposition, it now sufﬁces to show that S can be partitioned into 1 , such that f |S is analytic for each j . analytic cells S1 , . . . , SK deﬁnable in R j To see this, we apply Lemma 5.3 to graph(f ). The resulting analytic manifolds S1 , . . . , SL ⊆ Rn are the graphs of analytic functions gj : n−1 (Sj ) → R. Now we use the inductive hypothesis to obtain a partition of n−1 (S) into analytic cells 1 compatible with each n−1 (S ). S1 , . . . , SK deﬁnable in R j References [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13] [14]

V. I. Arnold, Geometrical Methods in the Theory of Ordinary Differential Equations, 2d ed., Grundlehren Math. Wiss. 250, Springer, New York, 1988. J. Bochnak, M. Coste, and M.-F. Roy, Géométrie algébrique réelle, Ergeb. Math. Grenzgeb. (3) 12, Springer, Berlin, 1987. F. Cano, J.-M. Lion, and R. Moussu, Frontière d’une hypersurface pfafﬁenne, Ann. Sci. École Norm. Sup. (4) 28 (1995), 591–646. J. Denef and L. van den Dries, p-adic and real subanalytic sets, Ann. of Math. (2) 128 (1988), 79–138. L. van den Dries, T -convexity and tame extensions, II, J. Symbolic Logic 62 (1997), 14–34. L. van den Dries and C. Miller, Geometric categories and o-minimal structures, Duke Math. J. 84 (1996), 497–540. A. Gabrièlov, Projections of semianalytic sets, Funct. Anal. Appl. 2 (1968), 282–291. C. Godbillon, Géométrie différentielle et mécanique analytique, Hermann, Paris, 1969. M. Gromov, Partial Differential Relations, Ergeb. Math. Grenzgeb. (3) 9, Springer, Berlin, 1986. A. Khovanski˘ı, Fewnomials, Transl. Math. Monogr. 88, Amer. Math. Soc., Providence, 1991. J.-M. Lion, Exemples de sous-ensembles sous-pfafﬁens et contact entre sous-ensembles souspfafﬁens, preprint, 1997. J.-M. Lion and J.-P. Rolin, Volumes, feuilles de Rolle de feuilletages analytiques et théorème de Wilkie, Ann. Fac. Sci. Toulouse Math. (6) 7 (1998), 93–112. D. Marker and C. Steinhorn, Deﬁnable types in O-minimal theories, J. Symbolic Logic 59 (1994), 185–198. R. Moussu and C. Roche, Théorie de Khovanski˘ı et problème de Dulac, Invent. Math. 105 (1991), 431–441.

ANALYTIC STRATIFICATION [15]

231

P. Speissegger, The Pfafﬁan closure of an o-minimal structure, J. Reine Angew. Math. 508 (1999), 189–211.

Lion: Université de Bourgogne, Laboratoire de Topologie, BP 400, 21011 Dijon Cedex, France; [email protected] Speissegger: University of Wisconsin-Madison, Department of Mathematics, 480 Lincoln Drive, Madison, Wisconsin 53706, USA; [email protected]

Vol. 103, No. 2

DUKE MATHEMATICAL JOURNAL

© 2000

ANALOGS OF WIENER’S ERGODIC THEOREMS FOR SEMISIMPLE LIE GROUPS, II G. A. MARGULIS, A. NEVO, and E. M. STEIN §0. Introduction. Given a measure-preserving action Tv : X → X, v ∈ Rd of the group G = Rd on a probability space (X, m), and a function f ∈ L1 (X), consider the averaging operators 1 f (Tv x) dv, π(βt )f (x) = vol Bt v∈Bt where Bt = {v ∈ Rd , v ≤ t}. Wiener’s pointwise ergodic theorem asserts that π(βt )f (x) converges to a limit as t → ∞ for almost every x ∈ X. The limit is given by the average of f on X, namely, X f dm, provided the action is ergodic. The main tool used in the proof of this result is Wiener’s maximal inequality, which asserts that the maximal function fβ∗ (x) = supt>0 |π(βt )f (x)| satisﬁes m{x : fβ∗ (x) ≥ δ} ≤ (C/δ)f L1 (X) . Consider the following generalization of the foregoing setup. Let G be a connected Lie group G, and let K be a compact subgroup. Assume there exists a G-invariant Riemannian metric on the homogeneous space S = G/K. The (bi-K-invariant) ball averages βt on G are deﬁned to be the probability measures 1 δg dmG (g), βt = mG (Bt ) g∈Bt where mG is a left-invariant Haar measure on G, Bt = {g ∈ G | d(gK, K) ≤ t}, d is the Riemmanian distance on S = G/K, and δg is the delta measure at g. βt give rise to canonical averaging operators, denoted π(βt ), in every measure-preserving action of G. We can now formulate the following problem. Ball averaging problem. Determine whether, for any ergodic measurepreserving action of G on a probability space (X, m), the averaging operators π(βt )f (x) converge to X f dm, for f ∈ L1 (X), or at least for f ∈ Lp (X), p > 1. Also, determine whether the maximal inequality fβ∗ Lp (X) ≤ Cp f Lp (X) holds. Received 13 May 1999. Revision received 12 November 1999. 2000 Mathematics Subject Classiﬁcation. Primary 22D40, 22E30, 28D10; Secondary 43A10, 43A90. Margulis partially supported by National Science Foundation grant numbers DMS-9424613 and DMS-9800607. Stein partially supported by National Science Foundation grant number DMS-9706889. 233

234

MARGULIS, NEVO, AND STEIN

A particularly interesting class to consider is that of semisimple groups with ﬁnite center. Here when we take K to be a maximal compact subgroup, there is a canonical choice of the Riemannian metric on G/K, namely, the one induced by the restriction of a Cartan-Killing form. The problem posed above was introduced in [N2] and solved afﬁrmatively for the ball averages βt on the groups of isometries of hyperbolic spaces Hn of dimension n ≥ 3, for functions f ∈ L2 (X) (see [N2]). Subsequently, in [N3] and [NS2] it was established that βt is a pointwise ergodic family in Lp , for all p > 1 and every simple Lie group of real rank one (with ﬁnite center). One of the main results (see Theorem 1) of the present paper constitutes a solution (in the semisimple setting) to the problem posed above; it establishes an analog of Wiener’s pointwise ergodic theorem for the (bi-K-invariant) ball averages on all connected noncompact semisimple Lie groups with ﬁnite center. The maximal inequality fβ∗ Lp (X) ≤ Cp f Lp (X) , for p > 1, is also established. We note, however, that the analogy with Wiener’s theorem is still incomplete since we restrict to f ∈ Lp , p > 1. The case of f ∈ L1 remains open. Our analysis relies in part on spectral considerations, which we apply to the commutative Banach ∗-algebra M(G, K) of all bounded bi-K-invariant Borel measures on G. A continuous unitary representation (τ, Ᏼτ ) of G extends canonically to a ∗-algebra homomorphism τ : M(G, K) → End(Ᏼτ ). The spectrum of the closure of the algebra τ (M(G, K)), namely, the set of continuous characters of this algebra, can be identiﬁed with a subset τ of the set of positive-deﬁnite spherical functions ϕλ on G (see, e.g., [HC1], [HC2], [GV], [H2]). To describe our second main result, we ﬁrst recall that in the theory of positivedeﬁnite spherical functions on connected semisimple Lie groups, there appears a fundamental distinction, which was introduced by D. Kazhdan [K] and can be described in the following manner. Given a probability measure µ ∈ M(G, K), consider the following expression: µT = sup ϕλ (µ); ϕλ a nonconstant positive-deﬁnite spherical function . Let τ be any continuous unitary representation of G that does not contain G-invariant unit vectors, or equivalently, the unit character of M(G, K) does not appear in the spectral decomposition. Then by the spectral theorem, the following operator norm estimate is valid: τ (µ) ≤ µT . Consider now µ = β1 , say. The distinction we refer to is whether β1 T < 1 or β1 T = 1. Groups satisfying the “spectral gap” condition β1 T < 1 are said to have Kazhdan’s property T. As is well known (see, e.g., [M]), all connected semisimple Lie groups with ﬁnite center satisfy property T unless one of their factors is locally isomorphic to SO(n, 1) or SU(n, 1). Property T of Kazhdan has been given an explicit quantitative formulation by M. Cowling (see [C2]). Cowling’s fundamental spectral estimate asserts (in particular) that in fact, for a connected semisimple group with ﬁnite center and no compact factors satisfying property T, there exist positive constants b > 0, B > 0 depending only on G, such that βt T ≤ B exp(−bt).

WIENER’S ERGODIC THEOREMS FOR SEMISIMPLE LIE GROUPS

235

This fact gives rise to a remarkable phenomenon (formulated as Theorem 2), which has no analog in the ergodic theory of amenable groups. Namely, for the family βt of ball averages on a connected semisimple Lie group with ﬁnite center and no compact factors possessing property T, the ergodic averages π(βt )f (x) converge exponentially fast to the ergodic mean X f dm. Furthermore, the rate of convergence is independent of the action and can be calculated explicitly. Similar results for several other families of radial averages, including some that are highly singular, are also proved (see Theorem 5). Exponentially fast convergence to the ergodic mean is established also for actions of general semisimple groups, not possessing property T, provided the associated unitary representation admits a spectral gap (see Theorem 4). For further results on the ergodic theory of semisimple Kazhdan groups, including some regarding nonradial and even discrete averages, we refer to [N4]. We note that the methods developed in this paper can be applied without any signiﬁcant change to the case of radial averages on noncompact semisimple algebraic groups over locally compact nondiscrete ﬁelds, provided the groups possess property T. The metric used to deﬁne radiality is obtained from the distance function on the associated Bruhat-Tits building. It should be remarked, however, that it is not always the case that the family of ball averages on a simple algebraic group constitutes a pointwise ergodic family, or even a mean ergodic family in L2 . Indeed, taking G = PGL2 (Qp ), and the natural radial ball averages βn deﬁned via the action of G on the p + 1-regular tree, the sequence π(βn )f generally fails to converge in the L2 -norm. The sequence π(β2n)f does converge in the norm and pointwise, but the limit is not the ergodic mean X f dm. For the analysis of radial averages on simple algebraic groups of split rank one, we refer to [N1] and [NS1]. Finally, we note that another natural problem is that of establishing pointwise ergodic theorems and maximal inequalities for the sphere averages deﬁned by the Riemannian structure. This problem was completely resolved for the case of real rank one groups (except those locally isomorphic to SL2 (R)) in [N2], [N3], and [NS2]. The case of sphere averages on complex groups is considered in [CN]. The general case remains unresolved. §1. Deﬁnitions, statements of results, and some remarks 1.1. Deﬁnitions. Let G denote a connected semisimple Lie group with ﬁnite center and without nontrivial compact factors, g its Lie algebra, g = ⊕N i=1 gi , where gi are the simple factors of g. Let θ denote a Cartan involution on G and g, and let g = k ⊕ p be the corresponding Cartan decomposition. Let a ⊂ p denote a maximal Abelian subalgebra, and let &(a, g) = & ⊂ a∗ denote the (real) root system of a in g. Let gα denote the root space corresponding to α ∈ &, and let g = m+a+ α∈& gα denote the root space decomposition. Fix a system of simple roots ( ⊂ &, the corresponding ordering of a∗ = hom(a, R), and the system of positive roots &+ , and let ρ denote half the sum of the positive roots. k is the Lie subalgebra corresponding to a connected

236

MARGULIS, NEVO, AND STEIN

maximal compact subgroup K. Let W = W (a, g) denote the Weyl group of the root system, a+ the positive Weyl chamber, and a+ its closure. Let A = exp a denote the Lie subgroup corresponding to a, A+ = exp a+ , and let A+ denote its closure. The Cartan (or polar coordinates) decomposition in G is given by G = KA+ K and g = k1 eH (g) k2 , where H (g) is the a+ component of g, and is uniquely determined for regular g ∈ G. The expression H (g) = d(gK, K) is uniquely determined by g. Let , denote the Cartan-Killing form on g, and let d denote the induced Riemannian metric on the symmetric space √ G/K. The restriction of , to a is an inner product, and we have d(exp(H )o, o) = H, H for all H ∈ a, where o = [K] denotes our choice of origin in G/K. Let (∨ = {H1 , . . . , Hr } ⊂ a denote the basis of a dual to the chosen basis ( ⊂ a∗ with respect to the inner product. We recall that the Cartan polar coordinates decomposition yields the following integration formula for Haar measure on G (see [H2, p. 186] or [GV]):

G

f (g) dmG (g) =

K

a+

K

f keH k ξ(H ) dmK (k) dH dmK (k ).

Here ξ(H ) = α∈&+ (sinh α(H ))mα , H ∈ a+ , mα = dimR gα , and mG , mK denote Haar measures on G and K, and dH denotes Lebesgue measure on a. We note that |ξ(H )| is the W -invariant extension of ξ(H ) from a+ to all of a, and W is the Weyl group of a in g, since ξ is alternating under W : ξ(w(H )) = (−1)det w ξ(H ). We let M(G) denote the convolution algebra of complex bounded Borel measures on G, and let M(G, K) denote the convolution subalgebra of measures that are biinvariant under K. For g ∈ G, we let δg denote the Dirac delta measure at g. 1.2. Statement of results. Let (X, Ꮾ, m) denote a standard Borel space with a Borel measurable G-action preserving the probability measure m. Recall that the Gaction is called ergodic if the only G-invariant functions in L2 (X) are the constant functions. Given any ﬁnite measure µ on G, consider the operator π(µ)f (x) = G f (g −1 x) dµ(g), acting on Lp (X), 1 ≤ p ≤ ∞. Deﬁnition 1. Let νt , t ∈ R+ be a one-parameter family of probability measures on G, such that t → νt is continuous in the w∗ -topology on M(G) = C0 (G)∗ . (1) Given an ergodic probability-measure-preserving action of G on (X, Ꮾ, m), νt is called a pointwise ergodic family in Lp (X) if, for every f ∈ Lp (X), lim π(νt )f (x) =

t→∞

X

f (x) dm,

where the convergence is pointwise almost everywhere and in the Lp norm. (2) The family is called a pointwise ergodic family in Lp if the foregoing condition holds for every ergodic G-space (X, Ꮾ, m).

WIENER’S ERGODIC THEOREMS FOR SEMISIMPLE LIE GROUPS

237

We also introduce the following notation. (1) Mν f (x) = supt≥0 |π(νt )f (x)| is the maximal function associated with νt and a function f . (2) fν∗ (x) = supt≥1 |π(νt )f (x)| is the global part of the maximal operator. Consider the natural family of ball averages on G, deﬁned as follows: Let B˜ r = H ∈ a | d exp(H )o, o ≤ r = {H ∈ a | H ≤ r}, and let βr =

mK ∗

ξ(H ) dH ∗ mK . B˜ r ξ(H ) dH

B˜ r δexp H

βr are bi-K-invariant probability measures on G projecting to the normalized volume on a ball of radius r in G/K under the canonical map τ : G → G/K. As noted in the introduction, the following result constitutes a direct analog of Wiener’s (see [W]) pointwise ergodic theorem and maximal inequality for ball averages on Rn in the setup of actions of semisimple Lie groups (for p > 1). The case of simple Lie groups of R-rank one was established in [N2], [N3], and [NS2]. Theorem 1. Let G be a connected semisimple Lie group with ﬁnite center and no compact factors. Then the following statements hold. (1) The family βt of ball averages satisﬁes the strong maximal inequality in Lp , 1 < p ≤ ∞,

Mβ f = sup π(βt )f (x) ≤ Cp f p .

p t≥0

p

(2) βt is a pointwise ergodic family in Lp , 1 < p < ∞. For semisimple Lie groups satisfying Kazhdan’s property T (see [K] and §0), we establish an exponential-maximal inequality and show that the convergence of the ball averages to the ergodic mean is at an exponential rate. As noted in the introduction, such results have no analog for actions of amenable groups, or for general actions of simple Lie groups of real rank one, without further spectral assumptions (see Section II below for a further discussion of this point). Theorem 2. Let G be a connected semisimple Lie group with ﬁnite center and no compact factors. Assume G satisﬁes Kazhdan’s property T. Then there exists a positive constant b(G), such that for every θ < b(G), the following holds. (1) For every f ∈ Lp (X), 1 < p < ∞,

sup exp uθ t π(βt )f (x) − f dm

≤ Bf Lp (X)

r 4 X t≥1 L (X) provided p, r, and 0 < u < 1 satisfy 1/p = (1 − u)/q and 1/r = ((1 − u)/q) + u/2 for some 1 < q < ∞.

238

MARGULIS, NEVO, AND STEIN

(2) βt f (x) converges to the ergodic mean at an exponential rate; namely, for every f ∈ Lp (X), 1 < p < ∞; for almost every x ∈ X, and for q, r, θ, and u as in (1), π(βt )f (x) − f (x) dm ≤ B(x, f ) exp − uθt , 4 X where the estimator B(x, f ) satisﬁes B(·, f )r ≤ Bf p for all f ∈ Lp (X). It is natural to generalize these results in the following two directions. I. General radial averages. First, we can consider other families of bi-K-invariant averages on G, including some singular averages. To facilitate the discussion, we introduce the following deﬁnitions. Deﬁnition 2. A one-parameter family νt , t ≥ 0 of probability measures on G is called: (1) monotone, if νt ≤ Bν[t]+1 , for t ≥ 1, as measures on G. ([t] is the largest integer ≤ t); (2) roughly monotone, if νt ≤ B [t]+N k=max([t]−N,0) ν[t]+k , for t ≥ 1 and some ﬁxed B and N; (3) uniformly continuous, if t → νt is uniformly continuous, as a map R+ → M(G), with respect to the total variation norm. Equivalently, for every ε > 0, there exists δ > 0 such that νt+τ − νt 1 ≤ ε if τ ≤ δ, for all t ≥ 1; (4) uniformly Hölder continuous, if t → νt satisﬁes νt+7 −νt 1 ≤ C|7|a , for some 0 < a ≤ 1 and all 0 < 7 ≤ 1/2, t ≥ 1. We remark that when the measures νt are bi-K-invariant, the conditions above are satisﬁed by νt if and only if they are satisﬁed by the natural projections ν˜ t of νt to measures on A+ . II. Actions with a spectral gap. Second, we can consider actions of general connected semisimple Lie groups (not necessarily satisfying property T) under the additional assumption that the spectrum of the action has a spectral gap. To make this notion precise, we recall the following result, which is due to M. Cowling. Theorem A Estimate of spherical functions (see [C2]). Let G be a connected simple Lie group with ﬁnite center and no compact factors. Let (τ, Ᏼτ ) denote a strongly continuous unitary representation of G. Then the following two conditions are equivalent. (1) Ᏼτ does not contain a sequence of unit vectors asymptotically invariant under τ (G), that is, a sequence satisfying τ (g)vk − vk → 0 uniformly on compact subsets of G. (2) There exist κτ > 0 and C > 0, such that for g ∈ G, the following holds for every H ∈ a+ , any two vectors v, w ∈ Ᏼτ , and every κ < κτ :

τ mK ∗ δexp H ∗ mK v, w ≤ Ce−κρ(H ) vw.

WIENER’S ERGODIC THEOREMS FOR SEMISIMPLE LIE GROUPS

239

The representation (τ, Ᏼτ ) is said to have a spectral gap if these equivalent conditions hold. We note that the ﬁrst condition in the theorem above is Kazhdan’s original deﬁnition of the property that the representation τ is isolated from the trivial representation. Now consider the following deﬁnition. Deﬁnition 3. (1) A group G is said to have Kazhdan’s property T (see [K]) if every unitary representation τ of G that does not contain invariant unit vectors is isolated from the trivial representation (i.e., Ᏼτ does not contain an asymptotically G-invariant sequence of unit vectors). (2) Given a unitary representation τ of a group G satisfying the conditions of Theorem A, we call the maximum value of κτ that satisﬁes condition (2) the parameter of the spectral gap. Remarks. (1) We note that in the context of simple Lie groups, Theorem A turns property T into a very concrete estimate. Property T amounts to the assertion that every continuous unitary representation of G without invariant unit vectors is isolated from the trivial representation uniformly over all such representations (e.g., the sum of all equivalence classes of unitary representations without invariant unit vectors is isolated from the trivial representation). For simple noncompact Lie groups, equivalently, there exists κ(G) > 0 such that the estimate in (2) holds for every such representation, provided κ < κ(G). (2) Deﬁnition 3(1) above of property T agrees with the one given in §0. Indeed, if a representation τ has asymptotically invariant unit vectors, then clearly τ (µ) = 1 for every probability measure on G. Conversely, if τ (β1 ) = 1, then by the spectral theorem, the selfadjoint operator τ (β1 ) has a sequence vn of unit approximate eigenvectors with eigenvalue 1, namely, τ (β1 )vn − vn → 0. The same is true of every power of τ (β1 ). It follows that B1 τ (g)vn , vn dβ1 → 1. Choosing a subsequence wn of vn , we can assume that B1 Reτ (g)wn , wn dβ1 ≥ 1 − 2−2n . By a standard argument, we conclude that β1 {g; Reτ (g)wn , wn ≥ 1 − 2−n } ≥ 1 − 2−n . Using the Borel-Cantelli lemma, it follows that for β1 -almost all g ∈ B1 , τ (g)wn , wn → 1. Hence the sequence wn is asymptotically invariant under a.e. g ∈ B1 ; and since B1 generates G, standard arguments imply that wn is asymptotically invariant under G. (3) If τ has a spectral gap with parameter κτ , then typically for a family of bi-Kinvariant averages νt , there exists a constant θ > 0 satisfying τ (νt ) ≤ B exp(−θt). Indeed, to verify such a norm estimate, it sufﬁces to check that for κ < κτ , exp − κρ(H ) d ν˜ t (H ) ≤ B exp(−tθ). a+

In the sequel, the assumption of exponential decay of the operator norms π(νt ) will be the essential ingredient of our analysis. (4) Consider now the semisimple case where G = N i=1 Gi is a product group and Gi are connected, almost simple, and noncompact. A representation (τ, Ᏼτ ) of G is said to have a spectral gap if there is no τ (G) asymptotically invariant sequence of

240

MARGULIS, NEVO, AND STEIN

unit vectors. It is said to have a strong spectral gap if the restriction of τ to every simple component Gi has no τ (Gi ) asymptotically invariant sequence. To obtain that a certain sequence of averages νt has exponentially decaying operator norm, it is often sufﬁcient to assume just the existence of a spectral gap, rather than a strong spectral gap. Indeed, given νt ∈ M(G1 ×G2 , K1 ×K2 ), it sufﬁces to establish that (in the obvious notation) e−κ1 ρ1 (H1 ) e−κ2 ρ2 (H2 ) d ν˜ t (H1 , H2 ) ≤ B exp(−tθ) (H1 ,H2 )∈a+

under the additional assumption that κ1 + κ2 ≥ κτ > 0. Before proceeding, we recall the notation L20 (X) = {f ∈ L2 (X) | X f dm = 0}. The unitary representation π of G in L2 (X) restricts to a unitary representation π0 in L20 (X), and in the ergodic case, L20 (X) does not contain π0 (G)-invariant unit vectors. We can now state the following theorem. Theorem 3. Let (X, Ꮾ, m) be a probability-measure-preserving action of a connected semisimple Lie group with ﬁnite center and no compact factors. Assume that the averages νt are bi-K-invariant, roughly monotone, uniformly continuous, and satisfy π0 (νt ) ≤ B exp(−tθ ), where θ > 0. Then the following statements hold. (1) fν∗ p = supt≥1 |π(νt )f |p ≤ Cp f p , for 1 < p ≤ ∞. (2) νt is a pointwise ergodic family in Lp (X), 1 < p < ∞. When the averages νt are uniformly Hölder continuous (Deﬁnition 2(4)), it is possible to strengthen the previous result considerably, and establish an exponential rate of convergence to the ergodic mean, as well as an exponential-maximal inequality. Theorem 4. Let (X, Ꮾ, m) be a probability-measure-preserving action of a connected semisimple Lie group with ﬁnite center and no compact factors. Assume that the averages νt are bi-K-invariant, roughly monotone, uniformly Hölder continuous (with exponent a), and satisfy π0 (νt ) ≤ B exp(−tθ), where θ > 0. Then the following statements hold. (1) fν∗ p = supt≥1 |π(νt )f |p ≤ Cp f p , for 1 < p ≤ ∞. (2) νt is a pointwise ergodic family in Lp (X), 1 < p < ∞. (3) For every f ∈ Lp (X), 1 < p < ∞,

uaθ t

sup exp π(νt )f (x) − f dm

4 t≥1

X

Lr (X)

≤ Bf Lp (X)

provided p, r, and 0 < u < 1 satisfy 1/p = (1 − u)/q and 1/r = ((1 − u)/q) + u/2 for some 1 < q < ∞. (4) π(νt )f (x) converges to the ergodic mean at an exponential rate; namely, for every f ∈ Lp (X), 1 < p < ∞, for almost every x ∈ X, and for q, r, θ, and u as in (1),

WIENER’S ERGODIC THEOREMS FOR SEMISIMPLE LIE GROUPS

241

π(νt )f (x) − f (x) dm ≤ B(x, f ) exp − uaθt . 4 X

The estimator B(x, f ) satisﬁes B(·, f )r ≤ Bf p for all f ∈ Lp (X). We now introduce some natural bi-K-invariant averages on G. Deﬁnition 4. (1) Sphere averages. We denote by σt the unique bi-K-invariant probability measure on G whose image in G/K is the induced normalized Riemannian area on a sphere St of radius t with center o = [K]. It is explicitly given by H =t ξ(H ) mK ∗ δexp H ∗ mK dωt (H ) , σt = ξ(H ) dωt (H ) H =t

where ωt is the rotation-invariant probability measure on a sphere of radius t in a with respect to the restriction of the Cartan-Killing form. Alternatively, we can write H H =1 ξ(tH ) σt dω1 (H ) , σt = ξ(tH ) dω1 (H ) H =1

where σtH = mK ∗ δexp tH ∗ mK . (2) Ball averages. The normalized measures βt on Riemannian balls introduced in §1.2 are continuous convex combinations of sphere averages, with a positive density given by t Dt (s)σs ds βt = 0 t . 0 Dt (s) ds Here Dt (s) equals the total area of the sphere Ss if s ≤ t; otherwise, Dt (s) = 0. (3) Shell averages. We deﬁne the average on a shell of ﬁxed width one, as follows: 1 γt = σt−s ds, t ≥ 1 0

d

in analogy with the real rank one case [NS2]. Deﬁne also γt = γ1 for 0 ≤ t ≤ 1. (4) Directional sphere and directional shell averages. Let H ∈ a+ , H = 1, and deﬁne 1 H σtH = mK ∗ δexp tH ∗ mK , γtH = σt−s ds, t ≥ 1, γt = γ1 , 0 ≤ t ≤ 1. 0

(5) Directional ball averages. For any b > 0, t ≥ 1, deﬁne t ebs σtH ds βtH = exp(−bt) 0

again in analogy with the real rank one case [NS2].

242

MARGULIS, NEVO, AND STEIN

We can now formulate the following theorem. Theorem 5. Let G be a connected semisimple Lie group with ﬁnite center and no compact factors. The following bi-K-invariant families are roughly monotone and uniformly Hölder continuous: (1) shell averages γt , (2) directional ball averages βtH and directional shell averages γtH , (3) mK ∗νtL ∗mK , where νtL are the ball or shell averages on a connected semisimple subgroup L ⊂ G with no compact factors, and which are invariant under the Cartan involution. In any action of G in which the estimate π0 (νt ) ≤ B exp(−θt), θ > 0 holds, they satisfy the conclusion of Theorem 4. The foregoing results do not establish an exponential-maximal inequality and exponential pointwise convergence to the ergodic mean for the sphere averages σt . In view of this fact, we formulate the following elementary result, which (in particular) establishes such results for the sequence of sphere averages with integer radii. Theorem 6. Let (X, Ꮾ, m) be a probability-measure-preserving action of a connected semisimple Lie group with ﬁnite center and no compact factors. Let νt be any family of bi-K-invariant averages, satisfying the estimate π0 (νt ) ≤ B exp(−θt), θ > 0. Then νn satisﬁes the following statements. (1) An exponential-maximal inequality holds in Lp (X), 1 < p < ∞, namely,

sup exp θp n π(νn )f − f dm ≤ Cp f p .

n

2 X p (2) For f ∈ Lp , 1 < p < ∞, the rate of convergence to the ergodic mean is exponential with rate (1/2)θp , namely, for every f ∈ Lp (X) and almost every x ∈ X, π(νn )f (x) − f (x) dm ≤ Cp (x, f )e−(1/2)θp n . X

The estimator C(x, f ) satisﬁes C(·, f )p ≤ Cp f p for all f ∈ Lp (X). Here θp = θp = 2θ (1 − (1/p)) if 1 ≤ p ≤ 2 and θp = (2θ/p) if p ≥ 2, θp = θ if p ≥ 2. A similar result holds for any sequence νtn , provided only that ∞ n=1 exp(−θtn /2) < ∞. In particular, the conclusion holds in the case of νt = σt , the sphere averages on G. We note that Theorem 4 implies Theorem 2, since the family βt of ball averages is monotone and uniformly Hölder continuous, as we see in §4.2. Furthermore, the family of ball averages on semisimple Kazhdan groups satisﬁes π0 (βt ) ≤ B exp(−θt) for any θ < lim inf t→∞ −(1/t) log βt T . As noted above, the fact that the foregoing limit inferior is positive depends on the spectral estimates in [C2]. Several other

WIENER’S ERGODIC THEOREMS FOR SEMISIMPLE LIE GROUPS

243

relevant references establishing exponential decay estimates for K-ﬁnite matrix coefﬁcients associated with continuous unitary representations of semisimple groups are [H], [HT], [KS], [KM], and [L]. A sharper estimate than that of [C2] is given in [O]. These estimates imply that the exponential rate of convergence to the ergodic mean in Theorems 2, 4, 5, and 6 can be computed explicitly for a given family νt . Remarks. (1) Inﬁnite measure spaces. All the maximal inequalities stated above are of course valid whenever the group G acts on a measure space, preserving a σ -ﬁnite measure m. When the measure m is inﬁnite, the space L20 (X) should be interpreted as the space of functions in L2 (X) that are orthogonal to the space of G-invariant functions. π0 denotes the unitary representation of G in L20 (X). With this interpretation, all statements and proofs of maximal inequalities apply without change. (2) General locally compact groups. Given a locally compact second countable group G, and a one-parameter family of probability measures νt on G, the assumption of exponential decay of the operator norms π(νt ), together with monotonicity (or rough monotonicity), and uniform Hölder continuity (or uniform continuity) of νt , may hold. In that case, the conclusions of Theorems 3, 4, and 6 (correspondingly) hold also. In particular, ball averages on simple algebraic groups (without compact factors) over locally compact nondiscrete ﬁelds satisfy Theorem 6, provided the split rank of the group is at least two, since in that case the group satisﬁes property T. Furthermore, many natural actions of semisimple algebraic groups admit continuous or Hölder-continuous families of averages with exponentially decaying operator norms, without the group possessing property T. A particularly interesting class is that of homogeneous spaces with ﬁnite invariant measure, where the spectral gap condition holds (but it is unknown whether the strong spectral gap condition holds). Under these assumptions, the obvious versions of Theorems 3, 4, and 6 hold true. 1.3. Preliminary reductions. As is well known, to prove a pointwise ergodic theorem in Lp , 1 < p < ∞ for a family of averages νt , it sufﬁces to (1) prove the mean ergodic theorem in L2 , (2) ﬁnd a dense subspace of functions f ∈ L2 for which π(νt )f (x) converges almost everywhere, and (3) prove the strong maximal inequality in Lp . Indeed, then it follows that for every f ∈ L2 (X), π(νt )f (x) converges almost everywhere, to the limit given by the mean ergodic theorem. Since L2 ∩ Lp is dense in Lp , 1 ≤ p < ∞, the maximal inequality in Lp , 1 < p < ∞ implies the pointwise convergence for f ∈ Lp (X). For more on these arguments, see, for example, [N2], [N3], and [NS2]. We note the following facts regarding these ingredients. (1) The mean ergodic theorem. In Theorems 3, 4, 5, and 6, the estimate π0 (νt ) ≤ B exp(−θ t), θ > 0 is part of the assumptions. Of course, this estimate is equivalent to π(νt )f (x)− X f dmL2 (X) ≤ B exp(−θt), and so the mean ergodic theorem holds, with an exponential rate of convergence to the ergodic mean. It follows that the same holds also in every Lp (X), 1 < p < ∞ (with a rate θp > 0, see §2.1). Approximat ing f ∈ L1 (X) by a bounded function f , and using π(νt )f − X f dmL1 (X) ≤

244

MARGULIS, NEVO, AND STEIN

π(νt )f − X f dmL2 (X) , it follows that the mean ergodic theorem for νt holds in L1 (X) also; but in L1 (X), there is no exponential rate of convergence. Consider now the mean ergodic theorem for ball averages stated in Theorem 1. If G is simple, then limt→∞ |ϕλ (σt )| = 0 for every nonconstant positive-deﬁnite spherical function ϕλ , by the Howe-Moore mixing theorem (see [HM]). It follows by an elementary computation that the same property holds for the sphere averages on any connected semisimple group with ﬁnite center and no compact factors (see §4.2 below for more on this point). By the spectral theory of the commutative algebra M(G, K), it follows that limt→∞ π(σt )f − X f dmL2 (X) = 0. It follows immediately that the same is true for the ball averages βt . Hence, as in the previous paragraph, the mean ergodic theorem holds for βt (and σt ) in every Lp , 1 ≤ p < ∞. (2) The maximal function for singular averages. For singular averages, it is not immediately clear that the maximal function is even measurable. However, for all singular averages considered here, this is in fact the case. Indeed, as noted in [N2], given a probability space (X, m), without loss of generality, X can be assumed to be a compact metric space, and the G-action to be jointly continuous. Clearly, when f is continuous, the maximal function supt≥1 |π(νt )f (x)| = supq≥1,q∈Q |π(νq )f (x)| is measurable, since we always assume that t → νt is w ∗ -continuous. If, in addition, the strong Lp maximal inequality is satisﬁed for f ∈ C(X), then the arguments of [N2, App. A] apply, and fν∗ is measurable for any nonnegative f ∈ Lp . We note that when the averages are assumed to be roughly monotone and the action has a spectral gap, a much more elementary argument sufﬁces. The maximal inequality for νt acting on f ∈ Lp (X) follows from the maximal inequality supn≥1 |π(νn )f (x)|, which is established ﬁrst, using the exponential decay of π(νn )f − X f dm (see §2.1 for more on this point). (3) The local maximal inequality and local ergodic theorem. For the averages βt , the full maximal operator Mβ f (x) = supt≥0 |π(βt )f (x)| satisﬁes a strong maximal inequality in Lp , 1 < p < ∞. We prove the maximal inequality for the global operator fβ∗ = supt≥1 |π(βt )f (x)|. As to the local operator, we refer to §5 of [N3], where it is shown, using the local transfer principle, that it sufﬁces to establish the local maximal inequality for convolutions. The maximal inequality for the convolution case was proved in [CS]. We have stated the global maximal inequalities in the form fν∗ (x) = supt≥1 |π(νt )f (x)|. Of course, there is in all cases considered also a maximal inequality for the operator supt≥a |π(νt )f (x)| for any ﬁxed a > 0. (4) Pointwise convergence on a dense subspace. In contrast to the classical case of Rn -actions, on semisimple groups even for ball averages it is not a straightforward matter to establish pointwise almost everywhere convergence for functions that lie in a dense subspace of L2 . Furthermore, it is not straightforward to deduce the validity of Theorem 1 for a product group G1 ×G2 from its validity on the component groups G1 and G2 . In the Euclidean case, one can use the fact that a ball in a product group Rd1 × Rd2 is comparable to a square. Here, however, the exponential volume growth implies that the analogous averages are not comparable.

WIENER’S ERGODIC THEOREMS FOR SEMISIMPLE LIE GROUPS

245

This problem necessitates restricting attention to averages with exponentially decaying operator norm, satisfying the uniform continuity condition in Theorem 3, and satisfying the uniform Hölder condition in Theorem 4. We consider these matters in §4. The paper is organized in the following manner. In §2, we give the proofs of Theorems 4 and 6. In §3, we give the proofs of Theorems 2 and 5. In §4, we give the proof of Theorem 1. In §5, we give the proof of Theorem 3. §2. Maximal and exponential-maximal inequalities via spectral theory: Proof of Theorem 4 2.1. The strong Lp -maximal inequality: Proof of Theorem 4, part (1). Let us now consider a probability-measure-preserving action of G on X with the property that the associated representation π restricted to L20 (X) has a spectral gap (in the sense of Deﬁnition 3, §1.1). Consider the spectral decomposition of the algebra π(M(G, K)) = Aπ , given by

π(µ)f, f = ϕ(µ) dηf (ϕ), spAπ

where ηf is the spectral measure, and ϕ : Aπ → C are the ∗-characters of M(G, K). In particular,

π(µ)f 2 = ϕ(µ)2 dηf (ϕ). 2 spAπ

Therefore, if f ∈ L20 (X), and νt are bi-K-invariant averages on G (of the form νt = mK ∗ ν˜ t ∗ mK ), by our assumption on their operator-norm bound,

π(νt )f 2 = 2

Therefore,

spAπ

a+

2 ϕ(exp H ) d ν˜ t (H ) dηf (ϕ) ≤ B 2 e−2θt f 22 .

∞

π(νn )f 2 ≤ C 2 f 2 . n=1

2

2

But since

2 sup π(νn )f (x) dm(x) ≤

X n≥1

we conclude that

∞ π(νn )f (x)2 dm(x), X n=1

sup π(νn )f ≤ Cf 2 .

n≥1

2

246

MARGULIS, NEVO, AND STEIN

Therefore, for every f ∈ L2 (X) ,

sup π(νn )f ≤ sup π(νn ) f − f dm + f dm n≥1

so that

X

n≥1

X

sup π(νn )f ≤ C f − f dm + f dm ≤ (2C + 1)f 2 .

2 X

X

2

[t]+N

Now, since νt is roughly monotone, νt ≤ B k=max([t]−N,0) ν[t]+k ; and so for every f ∈ L2 (X),

sup π(νt )f (x) ≤ B(2N + 1) sup π(νn )|f |(x) ≤ C2 f 2 .

t≥1

n≥1

2

2

L2 -maximal

inequality. To prove the maximal This completes the proof of the strong inequality in Lp , p > 1, consider again the inequality, for f ∈ L20 (X),

π(νn )f ≤ Ce−θn f 2 . 2 p

It sufﬁces to prove that for f ∈ L0 (X) and some θp > 0,

π(νn )f ≤ Cp e−θp n f p . p

Then arguing as before, we can conclude that supt≥1 |π(νt )f |p ≤ Cp f p . The π(νt0 )f = π(νt )f proof proceeds by Riesz-Thorin interpolation: The operator − X f dm is deﬁned and bounded as an operator π(νt0 ) : Lp (X) → Lp (X), 1 ≤ p ≤ ∞, and of course, π(νt0 )1 ≤ 2, π(νt0 )∞ ≤ 2. Furthermore, π(νt0 )f 2 ≤ Ce−θ t f 2 . Hence

(2/p)−1 0 2(1−(1/p))

0

π ν

π ν ≤ Bp π ν 0

1 ≤ p ≤ 2, t t t p 1

2

0

2/p 1−(2/p)

π ν ≤ Bp π ν 0 π ν 0

t t t p ∞

2

2 ≤ p ≤ ∞.

We conclude that θp = 2θ (1 − (1/p)) if 1 < p ≤ 2, and θp = 2θ/p if 2 ≤ p ≤ ∞, where θ2 = θ > 0 is the parameter appearing in the estimate of the operator norm of νt . This concludes the proof of Theorem 4(1). 2.2. Pointwise convergence: Proof of Theorem 4, part (2). We have established in §2.1 the strong Lp -maximal inequality for the averages νt of Theorem 4. We now show that π(νt )f (x) converges pointwise almost everywhere with a ﬁxed exponential rate for every f ∈ L∞ (X). To begin, ﬁx any function f ∈ L2 (X). Since π(νt ) satisﬁes the operator-norm bound, we have

π(νt ) f − f dm ≤ B exp(−θt) f − f dm .

X

2

X

2

WIENER’S ERGODIC THEOREMS FOR SEMISIMPLE LIE GROUPS

247

Hence, for every increasing sequence ti ∈ R+ , satisfying ∞ i=0 exp(−θti /2) < ∞, we have 2 ∞ exp θ ti π(νt ) f (x) − f dm dm(x) < ∞, i 2 X X i=1

and hence

2 ∞ exp θ ti π(νt ) f (x) − f dm < ∞ i 2 X

i=1

almost everywhere. 2 Now deﬁne for h ∈ L20 (X), C(x, h)2 = ∞ i=1 exp(ti θ/2)π(νti )h(x) . We note the following fact, which is used in §2.3: ∞

C(·, h) 2 ≤ B 2 exp(−θti ) h2 . 2

2

i=1

We can now write ≤ C x, f − f dm exp − θti π νt f (x) − f dm i 2 X

X

for almost every x ∈ X, and estimate as follows: π(νt )f (x) − f dm ≤ π νt − νt f (x) + π νt f (x) − f dm . i i X

X

Clearly, for any sequence ti satisfying the conditions above, the second summand on the right-hand side can be estimated by the preceding inequality. It remains to estimate the ﬁrst summand, and for that, we now assume that f is a bounded function. We choose the sequence ti as follows: divide the interval [n, n + 1] to 2 + [exp(nθ/4)] intervals of equal length, and take ti to be the increasing sequence consisting of their endpoints. It is then clear that ∞ ∞ θn θ ti nθ ≤ + 2 exp − < ∞. exp − exp 2 4 2 n=0 n≤ti
n=0

Moreover, given any t, let [t] = n, and then t belongs to some subinterval with length equal to ([exp(nθ/4)]+2)−1 ≤ 1/2. Since we have assumed that t → νt is uniformly Hölder-continuous in every interval of length 1/2 (with the same modulus a), we conclude that for almost every x ∈ X, −a π(νt )f (x) − π νt f (x) ≤ f ∞ t − ti a ≤ f ∞ exp nθ + 2 i 4 naθ . ≤ f ∞ exp − 4

248

MARGULIS, NEVO, AND STEIN

Therefore, combining these two estimates, we conclude that for a bounded function f , π(νt ) f (x) − f dm ≤ C x, f − f dm + f ∞ exp − [t]aθ . 4 X X Now since L∞ is norm-dense in each Lp (X), 1 ≤ p ≤ ∞, we have established that each of these spaces has a norm-dense subspace of functions f with the property that π(νt )f (x) converges to X f dm for almost all x ∈ X. Since the maximal inequality holds for νt in Lp (X), 1 < p ≤ ∞ (by §2.1), it follows that νt is a pointwise ergodic family in Lp (X), 1 < p < ∞. For further details on this standard argument, we refer, for example, to [N2], [N3]. This concludes the proof of Theorem 4(2). 2.3. Exponential-maximal inequalities, and exponential pointwise convergence: Proof of Theorem 4, parts (3) and (4). In §2.2, it was shown that the following holds for bounded functions f : π(νt )f (x) − f dm ≤ C x, f − f dm + f ∞ exp − aθt . 4 X X Furthermore, C(·, f )2 ≤ Bf ∞ , since, as was noted in §2.2, for h = f − X f dm, we have C(·, h)2 ≤ B h2 ≤ Bf ∞ . Hence, for every f ∈ L∞ (X),

aθ t

π(ν )f (x) − f dm ≤ Af ∞ .

sup exp t

t≥1 4 X 2

The strong Lq -maximal inequality (for every 1 < q ≤ ∞) established in §2.1 for the averages νt , implies, for every f ∈ Lq (X),

sup π(νt )f (x) − f dm ≤ Cq f q .

t≥1 X q

We can therefore use the complex interpolation theorem to establish an exponentialmaximal inequality for all 1 < p < ∞, in the following manner. Let τ : X → [1, ∞) be an arbitrary measurable function, let z be a complex parameter, and deﬁne the analytic family of operators as follows: aθzt τ π ντ (x) f (x) − f dm . Uz f (x) = exp 4 X Now note that when Re z = 1, Uzτ : L∞ (X) → L2 (X) is a bounded linear operator, with a ﬁxed bound (namely, the constant A above) independent of the index function τ , Indeed, for each function τ , the operator Uzτ is bounded by the maximal operator,

WIENER’S ERGODIC THEOREMS FOR SEMISIMPLE LIE GROUPS

249

which was shown to be bounded in §2.2, as follows: τ U f (x) = exp aθzt π ντ (x) f (x) − f dm z 4 X aθt ≤ sup exp π(νt )f (x) − f dm. 4 X t≥1 Similarly, when Re z = 0, Uzτ : Lq (X) → Lq (X) is bounded (by Cq ) for every index function τ . By the complex interpolation theorem (see, e.g., [S1]), it follows that for every u satisfying 0 < u < 1 , the operator Uuτ is deﬁned and bounded as an operator Uuτ : Lp (X) → Lr (X), provided that 1/p = (1 − u)/q, where 1/r = ((1 − u)/q) + u/2. The bound depends only on the bounds at Re z = 0 and Re z = 1; hence, it is independent of the index function τ . Taking the supremum over all such functions yields a bound for the maximal operator (see, e.g., [S1]), and we conclude that for all f ∈ Lp (X) and some constant M = M(a, θ, p, q),

uaθ t

π(νt )f (x) − f dm

≤ Mf Lp (X) .

sup exp

t≥1

r 4 X L (X)

This concludes the proof of part (3) of Theorem 4. To prove pointwise exponential convergence of π(νt )f (x) to the ergodic mean for Lp functions, it remains only to note that the foregoing exponential-maximal inequality implies r uaθ t dm ≤ M r f r ; sup exp π(ν )f (x) − f dm t p 4 X

t≥1

X

and hence for almost every x ∈ X, uaθt d B(x, f ) = sup exp π(νt )f (x) − f dm < ∞. 4 X t≥1 Therefore, for almost every x ∈ X, π(νt )f (x) − f dm ≤ B(x, f ) exp − uaθt . 4 X Furthermore, B(·, f )r ≤ Mf p . This completes the proof of Theorem 4. 2.4. Sequences of averages: Proof of Theorem 6. Consider the family νt , acting as p p operators νt : L0 (X) → L0 (X) and satisfying an exponential decay estimate given, p for f ∈ L0 (X), by νt f p ≤ Bp exp − θp t f p , 1 ≤ p ≤ ∞.

250

MARGULIS, NEVO, AND STEIN

This fact follows by Riesz-Thorin interpolation, as noted in §2.1, where θp is comp puted explicitly. Now let 1 < p < ∞, 0 < 7 < 1. If f ∈ L0 (X), then ∞

p p e(1−7)pθp n νn f p ≤ Bp (7)f p .

n=0

Therefore, the function p d e(1−7)θp n νn f (x) = Cp7 (x, f )p

∞ n=0

is ﬁnite almost everywhere. It follows immediately that the following exponentialmaximal inequality holds:

(1−7)θp n νn f (x) ≤ Cp7 (·, f ) p ≤ Bp f p .

sup e

n≥1 p

As a consequence, we have νn f (x) ≤ exp − θp (1 − 7)n C 7 (x, f ) −→ 0; p so the convergence to the mean is exponential. For an arbitrary function f ∈ Lp (X), the same conclusion holds: νn f (x) − f dm ≤ exp − θp (1 − 7)n C 7 x, f − f dm . p X

X

Clearly, exponential convergence to the mean holds for ηtn f (x), f ∈ Lp (X) whenever ∞ tn satisﬁes n=1 exp(−tn 7θp ) < ∞ for some 0 < 7 < 1. In Theorem 6, parts (1) and (2) are proved by taking 7 = 1/2 in the foregoing conclusions. For the fact that σt satisﬁes the exponential decay estimate of the operator norms, one uses the spectral estimates of [C2] or [KM, Theorem 2.4.3]. §3. Ball averages and directional averages in actions with a spectral gap 3.1. Proof of Theorem 2. Theorem 2 follows from Theorem 4 once it is established that the family βt of ball averages is indeed a monotone family that is uniformly Hölder continuous with exponent a = 1, which has exponentially decaying operator norms. We consider these properties separately. (1) Uniform Hölder continuity. This property is veriﬁed using the explicit form of the density of Haar measure on G in polar coordinates, given by the formula in §1.1, as we proceed to show.

WIENER’S ERGODIC THEOREMS FOR SEMISIMPLE LIE GROUPS

251

In general, if νt = (m(At ))−1 χAt for some measure m and some family of sets At of positive measure, then At ∪As m(As )χAt − m(At )χAs νt − νs 1 = m(At )m(As ) m(At ∩ As )m(As ) − m(At ) + m(At \ As )m(As ) + m(As \ At )m(At ) = . m(At )m(As ) Now consider βt and note the fact that the derivative of vol Bs at s = t is the area of the t sphere St , denoted area St , since vol Bt = 0 area Ss ds. Of course, area St is a positive monotonically increasing continuous function. Taking for (t, s) the values (t + ε, t), it is easily seen that βt+ε − βt 1 = (vol Bt+ε − vol Bt )/ vol Bt+ε . We claim that vol Bt+ε − vol Bt area St+ε ≤ε ≤ Cε, vol Bt+ε vol Bt+ε where C is independent of t ≥ 1. Indeed, the ﬁrst inequality follows from the mean value theorem and monotonocity of area St . For the second, note that since vol Bt = t area Ss ds, we have 0 area St area St ≤ . vol Bt (1/2) area St−(1/2) We claim that the latter expression is bounded for t ≥ 1. To see that, deﬁne ﬁrst M = max{α(H ) | α ∈ &+ , H ∈ a+ , H = 1}. Now note that there exists C > 0 such that for all 0 < α(H ) ≤ M and t ≥ 1, we have sinh(tα(H )) ≤ C sinh((t − (1/2))α(H )). Indeed, when t ≥ 1 and 0 < (t − 1/2)α(H ) ≤ 1, this fact follows from limα(H )→0 sinh(tα(H ))/ sinh((t −1/2)α(H )) = t/(t −1/2). If (t −1/2)α(H ) ≥ 1, it follows from the obvious inequality C1 exp y ≤ sinh y ≤ C2 exp y, valid for y ≥ 1. Hence, since area St = H =1 |ξ(tH )| dω1 (H ), it follows that area St ≤ C area St−(1/2) , as claimed above. Also, we see that the Hölder exponent of the ball averages is in fact a = 1. (2) Monotonicity. To show that βt ≤ Cβ[t]+1 , (for t ≥ 1), it sufﬁces to show that vol Bt+1 / vol Bt ≤ C for all t ≥ 1. This follows easilyfrom the explicit expression t for area St considered in (1), and the fact that vol Bt = 0 area Ss ds. (3) Exponential decay of operator norm. The estimate given in Theorem A (§1.2) due to M. Cowling (see [C2]; see also R. Howe [H], [KS], [KM], [HT]) implies immediately that the ball averages βt on connected semisimple Kazhdan groups (with ﬁnite center and no compact factors) satisfy π(βt ) ≤ B exp(−θt) in every representation π that has a spectral gap (see §4 for more on this point). The parameter θ > 0 depends only on G and can be determined from an estimate on the slowest possible decay for a bi-K-invariant matrix coefﬁcient of G (see, e.g., [O] for such estimates).

252

MARGULIS, NEVO, AND STEIN

This concludes the proof of Theorem 2. 3.2. Proof of Theorem 5. The fact that the shell averages are roughly monotone is self-evident, as is the fact that they are uniformly Hölder continuous, with Hölder exponent 1. Again the operator norm estimate is clear from the exponential decay estimates that were cited in Theorem A, §1.2, and §3.1(3). As to the directional ball averages and directional shell averages on G, the arguments used for the ball and shell averages apply without any signiﬁcant change. The same holds for ball and shell averages on semisimple subgroups (with ﬁnite center and without compact factors) L of G, stable under the Cartan involution. This concludes the proof of Theorem 5. §4. Proof of Theorem 1 4.1. Radial averages on product groups. In order to prove the maximal inequalities (as well as pointwise convergence on a dense subspace of L2 ) for product groups, we consider products of radial averages in the algebra M(G, K). Let G = G1 · G2 be a (almost direct) product, where each factor is a connected semisimple Lie group with ﬁnite center and no compact factors (and G1 ∩G2 is ﬁnite). M(G, K) is isometrically isomorphic to the tensor product of M(G1 , K1 ) and M(G2 , K2 ). This is a simple consequence of the obvious formula for g1 ∈ G1 and g2 ∈ G2 : mK1 ×K2 ∗ δg1 g2 ∗ mK1 ×K2 = mK1 ∗ δg1 ∗ mK1 ∗ mK2 ∗ δg2 ∗ mK2 . By deﬁnition, a = a1 ⊕ a2, H = H1 + H2 , and ξ(H1 , H2 ) = ξ1 (H1 )ξ2 (H2 ). Let us (i) deﬁne σ˜ Hi = mKi ∗δexp Hi ∗mKi , for i = 1, 2. Recall that σt denotes the normalized bi-Ki -invariant measure projecting to the Riemannian measure on a sphere of radius t in the symmetric space Gi /Ki . Hence, writing r 2 + s 2 = t 2 , 1 σt = area St

=

1 area St

H1 2 +H1 2 =t 2

t

= 0

1 ×K2

∗ δexp H1 exp H2 ∗ mK1 ×K2 dωt H1 , H2 t ξ1 (H1 )σ˜ H1 dωr (H1 )j˜t (s) H1 2 =r 2

0

×

ξ1 (H1 )ξ2 (H2 )mK

H2 2 =s 2

(1)

jt (s)σ√ 2

t −s 2

ξ2 (H2 )σ˜ H2 dωs (H2 ) ds

σs(2) ds.

Here jt is a positive density on the interval (0, t), which depends on G1 , G2 .

WIENER’S ERGODIC THEOREMS FOR SEMISIMPLE LIE GROUPS

253

Similarly, we can write 1 βt = vol Bt

t 0

H1

t

2 =r 2

ξ1 (H1 )σ˜ H1 dωr (H1 )J˜t (s)

H2 ≤s

ξ2 (H2 )σ˜ H2 ds

1 ξ1 (H1 )σ˜ H1 dω√ (H1 )J˜t (s) vol Bs(2) βs(2) ds t 2 −s 2 vol Bt 0 H1 2 =t 2 −s 2 t (1) = Jt (s)σ√ 2 2 βs(2) ds,

=

0

t −s

where again Jt (s) is a positive density on (0, t), depending √ on G1 , G2 . By deﬁnition, Jt (s) is the volume of a ball of radius t 2 − s 2 in G2 , multiplied by the area of a sphere of radius s in G2 , divided by the volume of a ball of radius t in t G1 × G2 . In particular, 0 Jt (s) ds = 1. We recall also the following well-known facts (see, e.g., [St], [BCG, App. C], [CN]). There exist positive constants c1 , c2 , and D depending only on G, satisfying, for t ≥ 1, c1 ≤

area St ≤ c2 tρ

t D exp

and c1 ≤

vol Bt ≤ c2 . t D exp tρ

Proof of Theorem 1(1). Let G be a connected semisimple group with ﬁnite center and no compact factors, and write G = G1 · G2 . We can assume that both G1 and G2 are nontrivial, since if G is simple, the maximal inequality follows from [NS2] when G has real rank one, and from Theorem 4 when G has higher real rank. We ﬁrst note that we can assume without loss of generality that the representation π is nontrivial when restricted to each simple component of G. Indeed, if π has nontrivial kernel, say, G1 , then the ball averages on G act via their projection to G2 . The projected measures are of course bi-K2 -invariant. Furthermore, they are convex (2) averages of the measures βt on G2 , by the foregoing formula for ball averages on a (2) product group. If a strong maximal inequality holds for the measures βt on G2 , it holds also for their convex averages. Hence we can assume that π is faithful. Let us consider ﬁrst the maximal inequality for ball averages on a group G, which is a product of real rank one groups. Ball averages on a real rank one group L satisfy, t d for t ≥ 1 (see [NS2]), βtL ≤ CL exp(−ρL t) 0 exp(ρL s)σsL ds = btL . The ball averages

254

MARGULIS, NEVO, AND STEIN

on a product G = L1 · L2 of two groups of real rank one therefore satisfy (for t ≥ 1) (1) (2) u2 +v 2 ≤t 2 exp(ρ1 u) exp(ρ2 v)σu σv du dv G2 βt ≤ C u2 +v 2 ≤t 2 exp(ρ1 u) exp(ρ2 v) du dv ≤B

(1) (2) n2 +m2 ≤([t]+1)2 exp(ρ1 n) exp(ρ2 m)γn γm

n2 +m2 ≤([t]+1)2 exp(ρ1 n) exp(ρ2 m)

,

(i)

where γt are the shell averages on a real rank one group Li . Here we have replaced the integral on a disc of radius t in the plane by an integral on a square of side [t]+1, and the integral over the square is then written as a sum of integrals on ([t] + 1)2 unit squares. The estimate of the density in each unit square is the obvious one; we estimate the function exp(ρ1 u) exp(ρ2 v) on the square whose upper right-hand corner is (n, m) by C exp(ρ1 n) exp(ρ2 m). We have shown in §3.1 that β[t]+1 ≤ Cβt , and therefore we can bound the union of the squares from above and from below by two discs in the obvious fashion, and the foregoing inequalities are established. Since the shell averages satisfy the strong maximal inequality in Lp , 1 < p ≤ ∞ by [NS2, §3], it follows that the same holds for the operators above. This argument clearly extends to the general case where G is an almost direct product of an arbitrary number of groups of real rank one, and concludes the proof of Theorem 1(1) in this case. To conclude the proof of Theorem 1(1), it remains to establish the maximal inequality in the case G = G1 · G2 , where G1 has property T and every simple factor of G2 has real rank one or is trivial. Consider for a nonnegative function f the expression t (1) √ π(βt )f (x) = Jt (s)π σ 2 2 π βs(2) f (x) ds. 0

t −s (2)

Since, as noted in the previous paragraph, π(βs )f (x) satisﬁes the strong maximal, it t (1) remains to show that the operators π(ηt )F (x) = 0 Jt (s)π σ√ 2 2 F (x) ds satisfy t −s it also. Now G1 satisﬁes property T, and the restriction of π to each one of its simple components is nontrivial. We claim that π0 (ηt ) ≤ C exp(−δt) for some δ > 0. Indeed, this estimate is a consequence of the following two facts. First, by the explicit estimate of the area of the sphere and the volume of the ball given in §4.1, together with the deﬁnition of Jt (s), we see that Jt (s) ≤ exp(−δ1 t) provided s ≤ c1 t for some positive δ1 and c1 . Second, the spectral estimates stated in Theorem A (§1.2) imply that the integral of any nonconstant spherical function on the surface of a sphere of radius t in G1 converges to zero faster than exp(−δ2 t). Here δ2 > 0 and is independent of the spherical function, since G1 has property T (for more on this point, see §4.2). Putting these two facts together, we obtain the norm estimate π0 (ηt ) ≤ exp(−δt). Finally, ηt is a roughly monotone family, since it is obtained as the projection of the roughly monotone family βt on G to the factor group G1 , and projection is clearly

WIENER’S ERGODIC THEOREMS FOR SEMISIMPLE LIE GROUPS

255

order-preserving. Hence, by the argument in §2.1, ηt satisﬁes the maximal inequality. This concludes the proof of Theorem 1(1). Remark. It is clear that the argument used in the foregoing proof can be utilized to prove a maximal inequality for averaging on a family of (positive measure) sets Ut in a product of rank one groups whenever the sets can be suitably covered by unit squares. As an example, consider the dilations Ut = tU of a ﬁxed convex open neighbourhood U of the origin in a. In this case, the comparison of the average χUt / vol(Ut ) to the average formed by the corresponding convex combination of shell averages again implies a maximal inequality in a similar fashion. This observation can be used systematically to bound certain maximal functions on symmetric spaces (and measure-preserving actions) for which the analogous Euclidean maximal function is not bounded, thus giving non-Euclidean extensions of Wiener’s theorem. We refer to [MNS] for more on these results. 4.2. Pointwise convergence on a dense subspace: Proof of Theorem 1, part (2). We now prove pointwise convergence on a dense subspace of L2 (X) for the ball averages βt . If G has property T, Theorem 4 gives a much stronger conclusion. Here we consider the case of a general semisimple group. We ﬁrst note that it sufﬁces to establish that for every positive-deﬁnite spherical function ϕλ , there exists a ﬁnite constant C(λ) satisfying

d ϕλ (σt )dt ≤ C(λ) < ∞. dt

∞ 1

Indeed, then we can consider the sets ε consisting of the characters ϕλ satisfying C(λ) ≤ 1/ε. We can then deﬁne Ᏼε as the space of functions whose spectral measures are supported in ε . Then the subspace ∪ε>0 Ᏼε together with the subspace of constant functions is dense in the subspace of K-invariant functions. In the complementary subspace, π(mK ) acts as the zero operator, and therefore the same holds for all the operators π(σt ). In the space Ᏼε , we can apply the method of [N2, §7.1] and conclude that pointwise convergence holds almost everywhere for functions in Ᏼε . The foregoing estimate certainly holds if there exist positive constants C(λ) and c(λ) satisfying d ϕλ (σt ) ≤ C(λ) exp − c(λ)t . dt For positive-deﬁnite spherical functions on simple Kazhdan groups, we have the following estimate, which is proved in detail in [CN, Proposition 2.6]. For all H ∈ a+ , t ≥ 0, k ≥ 0, k d ≤ C 1 + λ k (1 + t)d+k exp − κρ(tH ) , ϕ exp(tH ) dt k λ

256

MARGULIS, NEVO, AND STEIN

where κ > 0 and depends only on G (C depend on G and k). It is well known that the same estimate holds for every simple group of real rank one; but here κ > 0 may depend on λ, not just on G (see, e.g., [NS2, Theorem 5]). Every positive-deﬁnite spherical function on G = N i=1 Gi is of the form ϕλ N (i) (i) (g1 · · · gN ) = i=1 ϕλi (gi ). Here ϕλi is a positive-deﬁnite spherical function on the (i) group Gi . Clearly, ϕλ is nonconstant if and only if at least one of the functions ϕλi (i) is nonconstant. In that case, we have an exponential decay estimate for ϕλi (and its derivatives), so that ϕλ (and its derivatives) decays exponentially in some, but not necessarily all, directions in a+ . To complete the argument, we need to establish the estimate ξ(tH ) d d ϕλ (σt ) = ϕλ (tH ) dω1 (H ) ≤ C(λ) exp − c(λ)t , dt dt {H ∈a+ ;H =1} area St where ω1 is the normalized rotation-invariant measure on a sphere of radius 1 in a. This estimate is a straightforward consequence of the exponential decay of the spherical function and its derivatives noted above. The proof of Theorem 1 is complete, using §4.1 and §4.2. §5. Proof of Theorem 3. We now assume that t → νt is a uniformly continuous and roughly monotone family, and that the action of G on X satisﬁes π0 (νt ) ≤ exp(−θ t), θ > 0. Under these assumptions, the maximal inequality for νn and then νt follows as in §2.1. The main point in completing the proof of Theorem 3 is the following section. 5.1. Pointwise convergence for uniformly continuous families. We now establish the existence of a dense subspace of functions in L2 (X), where π(νt )f (x) converges almost everwhere, adapting an argument from [EMM, §3.6]. As noted in §2.2, for f ∈ L20 (X), π(νt )f 2 ≤ Ce−θt f 2 ; therefore, for any ﬁxed δ > 0, ∞ π(νnδ )f (x)2 dm(x) < ∞, ∞

X n=1

so that n=1 |π(νnδ )f (x)|2 < ∞, and π(νnδ )f (x) → 0 almost everwhere. Now ﬁx ε > 0, and also δ > 0, which is determined later. Denote the largest element of N · δ smaller than or equal to t by [t], δ. Writing π(νt )f (x) ≤ π νt − ν[t],δ f (x) + π ν[t],δ f (x), we see that

x : lim supπ(νt )f (x) > ε t→∞

WIENER’S ERGODIC THEOREMS FOR SEMISIMPLE LIE GROUPS

257

is contained in the union of ε Aε = x : lim sup π νt − ν[t],δ f (x) > 2 t→∞ ε . Bε = x : lim sup π ν[t],δ f (x) > 2 t→∞ For every N ≥ 1, we have the following inclusion: ∞ 2 2 ε π(νnδ )f (x) > ε ⊂ x . x sup π ν[t],δ f (x) > 2 2 t≥N

and

n=N

The measure of the second set in this inclusion is bounded by the expression

2 ∞

C 2 −2Nδθ n=N π(νnδ )f 2 ≤ e f 22 . (ε/2)2 ε2 Since the set in question contains Bε for every choice of N, taking N → ∞, we conclude that Bε has measure zero. Aε is also contained in a set of measure zero, as we now show. First, note that the assumption of uniform continuity of νt implies that given a bounded function f ∈ L∞ (X), for every ε > 0, there exists δ > 0 such that if |τ | ≤ δ, then |π(νt )f (x) − π(νt+τ )f (x)| < ε/2 for almost every x ∈ X and t ≥ 1. Indeed, the functions Fx (g) = f (g −1 x) are all bounded by the same constant f ∞ and are well deﬁned in L∞ (G, νt +νt+τ ) for almost all x ∈ X. Furthermore, the norm of νt+τ −νt as a functional is bounded by its total variation norm. We therefore choose δ determined by the choice of ε and the uniform continuity of νt . Now let [t], δ ∈ N · δ, and then since π(ν[t],δ )f ≤ C exp(−θδ[t]), we have ε C2 m x : π ν[t],δ f (x) > exp − 2θδ[t] f 22 . ≤ 2 2 (ε/2) Therefore, deﬁning the set Et = x : for some τ ∈ [0, δ], π ν(t,δ)+τ f (x) > ε , we have, for bounded functions f , using the uniform continuity of νt , the estimate m(Et ) ≤ (C 2 /ε 2 ) exp(−2θ δt)f 22 . Consequently, ∞ ∞ Ekδ = 0. m n=1 k=n

It follows that the measure of the set Aε is zero, and hence also the measure of the set {x : lim supt→∞ |π(νt )f (x)| > 7} is zero, provided f is a bounded function in L20 (X).

258

MARGULIS, NEVO, AND STEIN

It follows that there exists a dense set of functions in L2 (X), where the operators π(νt ) converge pointwise almost everywhere, to the limit given by the mean ergodic theorem. This concludes the proof of Theorem 3. References [BCG] [CS] [C1] [C2]

[CN] [EM] [EMM] [GV] [HC1] [HC2] [H1] [H2] [H]

[HM] [HT] [KS]

[K] [KM]

[Kn] [L] [M]

G. Besson, G. Courtois, and S. Gallot, Volume et entropie minimale des espaces localement symétriques, Invent. Math. 103 (1991), 417–445. J. L. Clerc and E. M. Stein, Lp -multipliers for non-compact symmetric spaces, Proc. Nat. Acad. Sci. U.S.A. 71 (1974), 3911–3912. M. Cowling, The Kunze-Stein phenomenon, Ann. of Math. (2) 107 (1978), 209–234. , “Sur les coefﬁcients des représentations unitaires des groupes de Lie simples” in Analyse harmonique sur les groupes de Lie (Sém. Nancy-Strasbourg, 1976–1978), II, Lecture Notes in Math. 739, Springer, Berlin, 1979, 132–178. M. Cowling and A. Nevo, Uniform estimates for spherical functions on complex semisimple Lie groups, preprint. A. Eskin and C. McMullen, Mixing, counting, and equidistribution in Lie groups. Duke Math. J. 71 (1993), 181–209. A. Eskin, G. A. Margulis, and S. Mozes, Upper bounds and asymptotics in a quantitative version of the Oppenheim conjecture, Ann. of Math. (2) 147 (1998), 93–141. R. Gangolli and V. S. Varadarajan, Harmonic Analysis of Spherical Functions on Real Reductive Groups, Ergeb. Math. Grenzgeb. 101, Springer, Berlin, 1988. Harish-Chandra, Spherical functions on a semi-simple Lie group, I, Amer. J. Math. 80 (1958), 241–310. , Spherical functions on a semi-simple Lie group, II, Amer. J. Math. 80 (1958), 553–613. S. Helgason, Differential Geometry, Lie Groups, and Symmetric Spaces, Pure Appl. Math. 80, Academic Press, New York, 1978. , Groups and Geometric Analysis, Pure Appl. Math. 113, Academic Press, Orlando, Fla., 1984. R. Howe, “On a notion of rank for unitary representations of the classical groups” in Harmonic Analysis and Group Representations, Centro Internazionale Matematico Estivo, Liguori, Naples, 1982, 223–331. R. Howe and C. C. Moore, Asymptotic properties of unitary representations, J. Funct. Anal. 32 (1979), 72–96. R. Howe and E. C. Tan, Non-Abelian Harmonic Analysis, Universitext, Springer, New York, 1992. A. Katok and R. Spatzier, First cohomology of Anosov actions of higher rank abelian groups and applications to rigidity, Inst. Hautes Etudes Sci. Publ. Math. 79 (1994), 131–156. D. A. Kazhdan, On a connection between the dual space of a group and the structure of its closed subgroups, Funct. Anal. Appl. 1 (1967), 63–65. D. Y. Kleinbock and G. A. Margulis, “Bounded orbits of nonquasiunipotent ﬂows on homogeneous spaces” in Sinai’s Moscow Center on Dynamical Systems, Amer. Math. Soc. Transl. Ser. 2 171 (1996), 141–172. A. W. Knapp, Lie Groups Beyond an Introduction, Progr. Math. 140, Birkhäuser, Boston, 1996. J-S. Li, “The minimal decay of matrix coefﬁcients for classical groups” in Harmonic Analysis in China, Math. Appl. 327, Kluwer Acad. Publ., Dordrecht, 1995, 146–169. G. A. Margulis, Discrete Subgroups of Semisimple Lie Groups, Ergeb. Math. Grenzgeb.

WIENER’S ERGODIC THEOREMS FOR SEMISIMPLE LIE GROUPS

[MNS] [N1] [N2] [N3] [N4] [NS1] [NS2] [O] [P] [S1] [S2] [S3] [SW] [St] [W]

259

17, Springer, Berlin, 1991. G. A. Margulis, A. Nevo, and E. M. Stein, in preparation. A. Nevo, Harmonic analysis and pointwise ergodic theorems for noncommuting transformations, J. Amer. Math. Soc. 7 (1994), 875–902. , Pointwise ergodic theorems for radial averages on simple Lie groups, I, Duke Math. J. 76 (1994), 113–140. , Pointwise ergodic theorems for radial averages on simple Lie groups, II, Duke Math. J., 86 (1997), 239–259. , Spectral transfer and pointwise ergodic theorems for semi-simple Kazhdan groups, Math. Res. Lett. 5 (1998), 305–325. A. Nevo and E. M. Stein, A generalization of Birkhoff’s pointwise ergodic theorem, Acta Math. 173 (1994), 135–154. , Analogs of Wiener’s ergodic theorems for semi-simple Lie groups, I, Ann. of Math. (2) 145 (1997), 565–595. H. Oh, Tempered subgroups and representations with minimal decay of matrix coefﬁcients, Bull. Soc. Math. France 126 (1998), 355–380. K. Petersen, Ergodic Theory, Cambridge Stud. Adv. Math. 2, Cambridge Univ. Press, New York, 1983. E. M. Stein, Topics in Harmonic Analysis Related to the Littlewood-Paley Theory, Ann. of Math. Stud. 63, Princeton Univ. Press, Princeton, 1970. , Analytic continuation of group representations, Adv. Math. 4 (1972), 172–207. , Harmonic Analysis: Real-Variable Methods, Orthogonality, and Oscillatory Integrals, Princeton Math. Ser. 45, Princeton Univ. Press, Princeton, 1993. E. M. Stein and S. Wainger, Problems in harmonic analysis related to curvature, Bull. Amer. Math. Soc. 84 (1978), 1239–1295. J-O. Stromberg, Weak type L1 estimates for maximal functions on non-compact symmetric spaces, Ann. of Math. (2) 114 (1981), 115–126. N. Wiener, The ergodic theorem, Duke Math. J. 5 (1939), 1–18.

Margulis: Department of Mathematics, Yale University, New Haven, Connecticut 06520, USA; [email protected] Nevo: Department of Mathematics, Technion Israel Institute of Technology, Haifa 32000, Israel; [email protected] Stein: Department of Mathematics, Princeton University, Princeton, New Jersey 08544, USA; [email protected]

Vol. 103, No. 2

DUKE MATHEMATICAL JOURNAL

© 2000

TOPOLOGICAL ENTROPY ON SADDLE SETS IN P2 JEFFREY DILLER and MATTIAS JONSSON 0. Introduction. The Fatou set Ᏺ of a holomorphic map f : Pk is the largest open subset of Pk on which iterates of f form a normal family. The complement of Ᏺ is called the Julia set when k = 1, and it is well known that the Julia set is the closure of the set of repelling periodic points. When k = 2, however, even product maps sufﬁce to show that the structure of P2 \ Ᏺ is more intricate. For instance, P2 \ Ᏺ contains both repelling periodic points and periodic points of “saddle” type, with one expanding and one contracting direction. In nice situations, these distinct types of periodic points occupy distinct regions in P2 \ Ᏺ, and each of these smaller regions legitimately vies with P2 \ Ᏺ for the designation of Julia set. Our concern in this paper is with what we call saddle sets of a holomorphic map f : P2 . These generalize the notion of a saddle periodic point, and while we defer the precise deﬁnition until Section 1, the following description should sufﬁce for the moment. A closed invariant set = f () ⊂ P2 is a saddle set of f if f acts transitively and hyperbolically on with one contracting and one expanding direction, and if is in some sense both maximal and isolated as a hyperbolic set. Important examples of saddle sets are given by the basic sets of saddle type for an Axiom A map f : P2 . Indeed, the paper [FS2] of Fornæss and Sibony on Axiom A holomorphic maps of P2 inspired much of the work on which this paper is based. Given a history pˆ = (pj )j ≤0 in (i.e., pj ∈ and f (pj −1 ) = pj for all j ≤ 0) and a small ﬁxed δ > 0, the associated local unstable manifold is u Wloc (p) ˆ = q ∈ P2 : dist(qj , pj ) < δ for j ≤ 0 and some qˆ with q0 = q . As is well known, f expands local unstable manifolds near . Nevertheless, we call u (p) terminal if for each history pˆ in , iterates of f act normally on Wloc ˆ − . We believe that terminal saddle sets play a distinguished role in the global dynamics of f . Our main result is the following theorem. Theorem A. Let be a saddle set of a holomorphic map f : P2 of degree d. Then the topological entropy htop (f | ) of f restricted to is no greater than log d. Equality holds if and only if is terminal. Received 19 July 1999. 2000 Mathematics Subject Classiﬁcation. Primary 32H50; Secondary 37F15, 37D05, 37D20. Diller supported by National Science Foundation grant number DMS98-96370. Jonsson supported by the Swedish Foundation for International Cooperation in Research and Higher Education (STINT). 261

262

DILLER AND JONSSON

Topological entropy is, roughly speaking, a nonnegative number that measures the complexity of orbits of f contained in . We do not actually use the precise deﬁnition of topological entropy in this paper, so we refer the reader to [KH] for such a deﬁnition, but we do point out that for any saddle set , 1 htop (f | ) = lim log # p = f n (p) ∈ . n→∞ n So if, as in the Axiom A case, all saddle periodic points lie in saddle sets of f , then either the vast asymptotic majority of these points lie in terminal saddle sets, or (in the absence of terminal saddle sets) saddle periodic points are comparatively scarce. Notice that Theorem A connects the behavior of f on to the global behavior of f on P2 . That is, topological entropy is a quantity purely intrinsic to the action of f on . On the other hand, large iterates of f might a priori take the local unstable manifolds of anywhere in P2 . Thus, the normal families criterion underlying terminality is really a global condition on the behavior of f . Despite the topological nature of the hypothesis and the conclusion of Theorem A, the proof actually relies largely on measure theory. This dependence happens in two ways. First of all, there is a natural positive closed (1, 1)-current T associated with any holomorphic map of P2 with degree d greater than 1. This current is globally deﬁned and has the transformation property f ∗ T = d · T . It also has the property that supp T is the complement of the Fatou set. Secondly, as Ruelle and Sullivan [RS] observed in the case of Axiom A diffeomorphisms, there is a canonical local current σ u supported on the local unstable manifolds of a saddle set . This local current satisﬁes f∗ σ u = λ · σ u where log λ = htop (f | ). It is possible to understand the wedge product σ u ∧ T as a positive measure deﬁned near , and the proof of Theorem A proceeds by considering the support and invariance of this measure. Bedford and Smillie [BS] were the ﬁrst to realize the importance of currents like σ u for multivariable complex dynamics. At least in the terminal case, the current σ u that we use here is identical to the current σ constructed by very different means in [FS2, Theorem 5.10]. We plan to explore this and other properties of σ u further in a future paper. It is not difﬁcult to give examples of terminal saddle sets. We present several in Section 3. It is harder to ﬁnd examples of nonterminal saddle sets. Nevertheless, we have the following theorem. Theorem B. There exist holomorphic maps of P2 with nonterminal saddle sets. More precisely, given integers d ≥ 2 and 0 < k < d, we can ﬁnd a holomorphic map of degree d with a saddle set such that htop (f | ) = log k. The proof of this theorem is constructive. That is, we actually manufacture examples of the desired type. These examples all come from the family of skew product maps, which have been studied in their own right by the second author (see [J2], [J3]) and Heinemann (see [H1], [H2]). The fact that our examples fail to be terminal follows from Theorem A.

TOPOLOGICAL ENTROPY ON SADDLE SETS IN P2

263

The contents of the rest of this paper are arranged as follows. Section 1 contains most of the background needed for this paper, including a review of the hyperbolic theory, the constructions of σ u and T , and the (precise) deﬁnition of a saddle set. Section 2 provides the proof of Theorem A. Section 3 presents examples of both terminal and nonterminal saddle sets. In particular, it contains the proof of Theorem B. 1. Background. In this section, we review the theory of hyperbolic dynamical systems. In particular, we describe the transversal measure and laminar current associated to the local unstable manifolds of a hyperbolic set. Our presentation is biased in two important ways. First of all, we are interested in noninvertible maps, and this leads to extra subtlety in the deﬁnition and properties of a hyperbolic set. The ﬁrst paper in the second author’s thesis [J1] provides further detail about hyperbolicity in the noninvertible setting. Secondly, we seek to prove results that are largely semilocal in nature. That is, we are usually concerned with assertions that are valid only in a small neighborhood of a hyperbolic set. After generalities about hyperbolicity, we discuss the hyperbolic sets of particular interest to us and indicate how such sets arise naturally for maps satisfying the (global) Axiom A condition. We close the section with a brief review of the deﬁnition and properties of the global current T associated with a holomorphic map f : P2 . 1.1. Hyperbolicity. Let f : X be a holomorphic (possibly branched), ﬁnite-to-1 map of a compact complex manifold X. Suppose that = f () ⊂ X is a compact ˆ to be the induced map on histories subset. We deﬁne the natural extension fˆ : ˆ contained in . That is, a point in is a sequence pˆ = (pj )j ≤0 of points pj ∈ ˆ such that f (pj ) = pj +1 , and fˆ is the shift map sending (pj ) to (pj +1 ). We give ∞ ˆ the product topology induced from X so that f is a homeomorphism of a compact ˆ → by πj (p) ˆ = pj . metric space. For every j ≤ 0, we deﬁne the projections πj : ˆ We obtain a vector bundle T over by using π0 to pull back the tangent bundle of X. Points in this bundle are speciﬁed by (p, ˆ v), where v is tangent to X at p0 . We say that f is hyperbolic on if there is a continuous fˆ-invariant splitting T = E s ⊕ E u and constants C > 0, ρ > 1 such that n (Df n )−1 w ≤ Cρ −n w, Df v ≤ Cρ −n v, for n ≥ 0 and all v ∈ E s and w ∈ E u . The inequalities imply the choice of a Hermitian metric on X, but any such metric will do. After a continuous change of metric near , we can assume that the constant C is 1. Note that the deﬁnition of hyperbolicity requires only that Df be invertible on the space E u . While E u (p) ˆ necessarily depends on the entire history pˆ of the point p = p0 , characterization in terms of forward iteration implies that E s (p) ˆ = E s (p) depends only on p and can therefore be considered a subspace of T Xp . In particular, the dimensions of E s and E u are constant on if we assume that f | is topologically transitive—an assumption we now adopt for the sake of simplicity. As advertised in

264

DILLER AND JONSSON

the introduction, our concern in this paper is with hyperbolic sets in P2 , where both E u and E s have dimension 1. The primary consequence of hyperbolicity is the existence of local stable and unstable manifolds attached to points in the hyperbolic set (see, for example, [SFL] ˆ the sets or [PS]). For small enough δ > 0 and for any history pˆ ∈ , Wδs (p) = q ∈ X : dist f n (q), f n (p0 ) < δ for all n ≥ 0 ˆ = q ∈ X : there exists a history qˆ ∈ Xˆ of q Wδu (p) such that dist(qn , pn ) < δ for all n ≤ 0 are f -invariant complex submanifolds tangent to E s (p) and E u (p), ˆ respectively. These are the local stable and unstable manifolds of p and p, ˆ respectively. As the notation implies, there is a unique local stable manifold but possibly many local unstable manifolds passing through a point p ∈ . Where it is not important to be s (p) and W u (p). explicit about δ, we write Wloc loc ˆ If we restrict to , then we can lift the local stable and unstable manifolds to the ˆ provided that we are willing to settle for a bit more asymmetry natural extension , s (p) = π −1 (W s (p) ∩ ) simply by pulling back. On in the results. We deﬁne Wˆ loc 0 loc u (p) ˆ : qj ∈ W u (fˆj (p)) the other hand, we let Wˆ loc ˆ = {qˆ ∈ ˆ for all j ≤ 0} consist loc u (p) of only those histories qˆ of points q ∈ Wloc ˆ that are backward asymptotic to p. ˆ s ˆ The projection π0 is not generally injective on Wloc (p), but since f acts injectively u (p) u (p) on local unstable manifolds, π0 : Wˆ loc ˆ → Wloc ˆ is a homeomorphism onto its image. We observe that it is possible to take a more intrinsic approach to deﬁning ˆ but for our purposes, it is more convenient to use the local stable/unstable sets in , above deﬁnitions. Continuous variation of the splitting and compactness of guarantee that a local s (p) intersects a local unstable manifold W u (q) stable manifold Wloc loc ˆ in at most one point. It also guarantees that the intersection is nonempty if q0 and p0 are close enough. It turns out to be quite useful to know that the intersection lies in and, ˆ stronger still, that the intersection has a unique history lying in . ˆ has local product structure if there exist constants Deﬁnition 1.1. We say that ˆ such that dist(p, q0 ) < δ , the intersecδ, δ > 0 such that for all p ∈ and all qˆ ∈ def ˆ tion {[p, q]} ˆ = Wˆ δs (p) ∩ Wˆ δu (q) ˆ consists of a unique point in . Local product structure is equivalent to the following local maximality condition (see [J1]): there exists a neighborhood ᏺ of such that any full orbit in ᏺ is actually contained in . We remark that [·, ·] commutes with f . That is, ˆ has local product structure, then continuous variation fˆ([p, q]) ˆ = [f (p), fˆ(q)]. ˆ If u (p)× s (p) → ˆ ˆ Wˆ loc of local stable/unstable manifolds implies that the map [·, ·] : Wloc u ˆ ˆ is a homeomorphism onto a neighborhood (in ) of p. ˆ In particular, π0 : Wloc (p) ˆ → u (p) Wloc ˆ ∩ is surjective.

TOPOLOGICAL ENTROPY ON SADDLE SETS IN P2

265

1.2. Transversal measures and laminar currents. In this subsection, we continue our discussion with the additional assumption that f is topologically mixing on . ˆ allows us to deﬁne holonomy along unstable maniLocal product structure on s (p) on a folds. Given p ∈ , we deﬁne the unstable holonomy map χˆ pu : R → Wˆ loc u of χˆ u to ˆ of Wˆ s (p) by χˆ pu (q) neighborhood R ⊂ ˆ = [p, q]. ˆ The restriction χˆ q,p p loc s (q) is a homeomorphism onto its image. R ∩ Wˆ loc Ruelle and Sullivan [RS] and Bowen and Marcus [BM] considered the notion of transversal measures for Axiom A diffeomorphisms. The following is a translation of their results into our setting. Theorem 1.2. Given f, as above, there exists for each p ∈ a positive measure s (p) with the following properties: µˆ up on Wˆ loc u (E) = F for Borel sets E ⊂ W ˆ s (q) and F ⊂ Wˆ s (p), then µˆ uq (E) = (i) if χˆ p,q loc loc u µˆ p (F ); (ii) fˆ∗ µˆ up = λµˆ uf (p) |fˆ(Wˆ s (p)) , where log λ is the entropy of f | ; loc (iii) supp µˆ up = Wˆ s (p). loc u µˆ p are

known as (unstable) transversal measures. Bowen and MarThe measures cus [BM] showed that transversal measures are unique up to rescaling by a factor independent of p. In fact, they are unique given only that they satisfy a more global variant of the ﬁrst item in the conclusion of Theorem 1.2. We do not actually need this fact here, so we pursue it no further. We remark that the restriction is important on the right side of the equation in the second item of Theorem 1.2. Under our deﬁnition, there need not be an ! > 0 such that Wˆ !s (f (p)) ⊂ fˆ(Wˆ δs (p)). Rather, if ! > 0 is small enough, we have fˆ Wˆ δs (p ) , (1.1) Wˆ !s f (p) ⊂ p ∈f −1 (f (p))

where, since fˆ is a homeomorphism, the sets in the union on the right side are mutually disjoint. The proofs of existence and uniqueness for transversal measures were originally given for basic sets of Axiom A diffeomorphisms. These proofs rest principally on the existence and properties of so-called Markov partitions of a basic set. Careful examination of the literature (see [A], [B2], [KH], and [SFL]) reveals that one can establish all relevant results about Markov partitions (with proofs nearly unchanged) for any expansive homeomorphism h : S of a compact metric space S with the following shadowing property: given any ! > 0, there exists δ > 0 such that for any δ-pseudoorbit {sj } ⊂ S, there exists a unique point s ∈ S such that dist(f j (s), sj ) < δ for all ˆ holds as a consequence of local product j ∈ Z. The shadowing property for fˆ : structure (see [J1] for a proof). For those interested in further discussion of notions like hyperbolicity, shadowing, and local product structure for homeomorphisms, we recommend the last chapter of [A].

266

DILLER AND JONSSON

Ruelle and Sullivan [RS] observed that transversal measures can be used to deﬁne a current σ u supported on the local unstable manifolds of . For instance, if ϕ is a test (1, 1)-form supported on a small neighborhood of p, then the action of the laminar current σ u on ϕ is given by u

σ , ϕ =

s (p) Wˆ loc

u (q) Wloc ˆ

ϕ d µˆ up (q). ˆ

(1.2)

Then σ u is extended to an entire neighborhood ᏺ of using a partition of unity. Holonomy invariance of transversal measures guarantees that the result is well deu ()∩ ᏺ. The property f σ u = λ·σ u ﬁned. Clearly, σ u is positive and supp σ u = Wloc ∗ on ᏺ inherits from equation (1.1) the pushforward property of µˆ up and the fact that unstable manifolds are expanded by f . Finally, σ u is closed because any local unu (q) stable manifold Wloc ˆ intersects small neighborhoods of p in relatively compact subsets when q is close to p. The preceding discussion can be modiﬁed to deﬁne holonomy along stable manifolds, stable transversal measures, and a laminar current supported on stable manifolds. Since we do not need these objects in this paper, we omit the details. 1.3. Saddle sets in P2 . Now we describe the situation of particular interest in this u () to denote the union of all local unstable manifolds associated paper. We use Wloc ˆ with points pˆ ∈ . Deﬁnition 1.3. Suppose that f : P2 is holomorphic. We call a saddle set for f if the following are true. (1) is a hyperbolic set for f , and both E s and E u are 1-dimensional. (2) f | is topologically transitive. ˆ has local product structure. (3) u ())∩ ᏺ = W u ()∩ ᏺ (4) There exists a neighborhood ᏺ of such that f n (Wloc loc for all n ≥ 0. Note that except for the last condition on , the requirements of this deﬁnition are semilocal in nature; that is, they only apply to the behavior of f near . Such sets arise naturally if we place a global restriction on the behavior of f . Recall that the nonwandering set $ of f consists of those points p ∈ P2 such that f n (U ) ∩ U = ∅ for any neighborhood U p and arbitrarily large n. The map f is Axiom A if f is hyperbolic on $ and if, in addition, periodic points are dense in $. Under these conditions, the set $ decomposes into a ﬁnite number of closed sets, on each of which f is topologically transitive. These are called the basic sets for f . By passing to a higher iterate and further decomposing, we can assume that f is actually topologically mixing on each basic set. If is a basic set for an Axiom A holomorphic map f : P2 with dim E s () = dim E u () = 1, then it turns out that is a saddle set. Condition (3) is proved in [J1, Proposition 3.3]. A proof of (4) is given for diffeomorphisms in [BM, pp. 46–47].

TOPOLOGICAL ENTROPY ON SADDLE SETS IN P2

267

Stable and unstable manifolds from different saddle sets , of an Axiom A map s () ∩ f n (W u ( )) = ∅ for some n ≥ 0, can intersect in complicated ways. If Wloc loc then we say that ≺ . When this relation actually orders the basic sets, then f is said to satisfy the no-cycles condition. We are interested in singling out basic sets of saddle type such that ≺ for all other such . These basic sets ﬁgured importantly in the study (where they were referred to as “minimal”) of hyperbolic holomorphic maps of P2 by Fornæss and Sibony [FS2]. Since we do not wish to restrict ourselves to the Axiom A setting, we phrase our condition in terms of normal families. Deﬁnition 1.4. Suppose that is a saddle set for a holomorphic map f : P2 . ˆ the iterates of f restricted to W u (p) We call terminal if for any pˆ ∈ , loc ˆ − form a normal family. Our use of the word terminal is motivated by the case of Axiom A maps. If f u (p) is Axiom A and an unstable manifold Wloc ˆ of does not intersect the stable u manifolds of some other basic set, then Wloc (p)− ˆ must lie in the basins of attracting cycles. Hence, basic sets of an Axiom A map that are minimal with respect to ≺ are necessarily terminal. On the other hand, iterates of f cannot form a normal family in a neighborhood of any point in a stable manifold, because a disk transverse to the stable u ()− lies manifold will eventually be expanded. Therefore, if we assume that Wloc in the Fatou set of f , then must be minimal with respect to ≺. It would be interesting u () is to know whether (or when) this apparently slightly stronger condition on Wloc actually equivalent to terminality. Before moving on, we mention that the spectral decomposition discussed above for basic sets of an Axiom A diffeomorphism applies to any saddle set, regardless of whether or not f is Axiom A (see [KH, Theorem 18.3.1]). In particular, we lose no generality by assuming that f is topologically mixing on saddle sets. 1.4. Pluripotential theory and holomorphic maps of P2 . A fundamental tool for understanding complex dynamics on Pk is the use of pluripotential theory to construct and study positive closed currents with good transformation properties. The papers [HP], [U1], and [FS1] present early applications of pluripotential theory to dynamics, and they remain excellent references. A degree d holomorphic map f : P2 acts linearly by pullback on the middle cohomology group. The group is freely generated by the cohomology class of the Fubini-Study form ω, and f ∗ multiplies this class by d. It is an interesting and very useful fact that there is a representative for the class of ω that is canonical for f . Namely, the sequence 1 n∗ f ω dn converges weakly to a positive closed (1, 1)-current T such that f ∗ T = d · T . Positivity means that locally, T = dd c u where u is a plurisubharmonic function. In the particular case of T , the local potentials u are always continuous. This allows us

268

DILLER AND JONSSON

to consider the slice measure T |R = dd c (u ◦ ι) of T along an embedded Riemann surface ι : R → P2 . The following proposition concerning slice measures of T is well known. Proposition 1.5. Slice measures of T vary continuously. That is, if ϕ is a test function in P2 and Rj ⊂ P2 are Riemann surfaces converging uniformly to a surface R ⊂ P2 , then Rj

ϕT |Rj −→

R

ϕ T |R .

Slice measures transform according to the formula f∗ (T |R ) =

1 · T |f (R) , d

provided that f is injective on R. There is a remarkable characterization of T in terms of normal families. Theorem 1.6. The support of T is equal to the complement of the Fatou set of f , that is, of the largest open set on which iterates of f form a normal family. Likewise, if R ⊂ P2 is a Riemann surface, then supp T |R is the complement of the largest open subset of R on which iterates of f act normally. A more recent paper of Fornæss and Sibony [FS2] gives another method of constructing positive closed (1, 1)-currents from iterates of f . In order to state the next result, we recall that the mass M[S] of a current S is given by M[S] = sup S, ϕ : ϕ∞ ≤ 1 . Theorem 1.7. Let S be a positive closed (1, 1)-current deﬁned on an open set U ⊂ P2 , and let ψ : U → C be a smooth, compactly supported function. Then there is a constant C such that M[f∗n (ψS)] ≤ Cd n and M[∂f∗n (ψS)] ≤ Cd n/2 for all n. Therefore, the sequence f∗n (ψS)/d n has weak limit points, and all such points are closed. A construction of Bedford and Taylor [BT] allows us to understand the wedge product S ∧ T , where S is a positive closed (1, 1)-current on U ⊂ P2 , as a positive measure. Locally, we choose a continuous potential u for T and set ψ S ∧ T = S, u dd c ψ . U

We get the following proposition from [FS2]. Proposition 1.8. We have the transformation property f∗ (S ∧ T ) =

f∗ S ∧T. d

TOPOLOGICAL ENTROPY ON SADDLE SETS IN P2

269

In particular, if S is deﬁned on all of P2 and f∗ S = d · S, then S ∧ T is an invariant measure. 2. Proof of Theorem A. Given a small neighborhood ᏺ of , let σ u be the laminar current in ᏺ supported on local unstable manifolds. As indicated above, it is possible to interpret σ u ∧ T as a positive measure on ᏺ. In this section, we study this measure carefully, ultimately proving Theorem A. Let us ﬁrst show that the wedge product σ u ∧ T commutes with the laminar structure of σ u . Proposition 2.1. If ϕ is a test function supported on a small neighborhood of p ∈ , then

u u (q) ϕ σ ∧T = ϕ T |Wloc ˆ d µˆ up (q). ˆ s (p) Wˆ loc

U

u (q) Wloc ˆ

Proof. Note that by Proposition 1.5, the right side of the equation in the conclusion of this proposition deﬁnes a positive distribution (acting on ϕ), that is, it deﬁnes a Radon measure. Let v be a local potential for T on a neighborhood U of the support of ϕ. Let vj be smooth plurisubharmonic functions that decrease uniformly locally to v as j goes to inﬁnity. We can assume without loss of generality that ϕ is smooth. Then ϕ σ u ∧ T = lim σ u , vj dd c ϕ j →∞ U

= lim

j →∞ Wˆ s (p) loc

=

s (p) Wˆ loc

=

s (p) Wˆ loc

u (q) Wloc ˆ

vj dd c ϕ d µˆ up (q) ˆ

c

u (q) Wloc ˆ

u (q) Wloc ˆ

v dd ϕ d µˆ up (q) ˆ u (q) d µˆ up (q), ϕ T |Wloc ˆ ˆ

as desired. The ﬁrst equality takes advantage of continuity of the wedge product operation under decreasing limits of plurisubharmonic functions (see [BT]). The third equality relies on the fact that, viewed as a function of q, ˆ the inner integral on the second line converges uniformly to the inner integral on the third line. The last equality holds by deﬁnition. Corollary 2.2. The support of σ u ∧T contains . The support of σ u ∧T equals if and only if is terminal. In particular, if is terminal, then σ u ∧ T is independent of the choice of the neighborhood ᏺ ⊃ on which σ u is deﬁned. u (p) Proof. Pick p ∈ and let pˆ be a history. The support of T |Wloc ˆ contains u ˆ do not form a normal family on any p because iterates of f applied to Wloc (p)

270

DILLER AND JONSSON

neighborhood of p. If ϕ ≥ 0 is a continuous function equal to 1 at p, then u (p) ϕ T |Wloc ˆ > 0. u (p) Wloc ˆ

s (p) near p. By continuity of slice measures, the same is true for all qˆ ∈ Wˆ loc ˆ Therefore, u u Proposition 2.1 and the fact that supp µˆ p contains pˆ imply that ϕ σ ∧ T > 0, that is, p ∈ supp σ u ∧ T , and the ﬁrst assertion is proved. u (p) If is terminal, then we have that T |Wloc ˆ is zero outside , so another application u ˆ be a of Proposition 2.1 shows that supp T ∧ σ ⊂ . If is not terminal, let pˆ ∈ u point such that supp T |Wloc (p) / . Continuous variation of slice ˆ contains a point q ∈ measures and Proposition 2.1 again allow us to conclude that q ∈ supp σ u ∧ T .

Transversal measures are only determined up to constant multiples, so there is no canonical way to completely ﬁx σ u in general. In the terminal case, however, we can normalize by rescaling σ u so that σ u ∧ T is a probability measure. The preceding results show that this deﬁnes the measure unambiguously. We are now ready to prove the main result of this paper. Proof of Theorem A. Let log λ be the entropy of f | , and let σ u be the laminar current supported on local unstable manifolds. We can assume that σ u is deﬁned on the neighborhood ᏺ of provided for in the deﬁnition of a saddle set. Suppose that χ is a cutoff function equal to 1 on some smaller neighborhood of but vanishing outside ᏺ. Then by expansion along unstable manifolds and the transformation property of σ u , we have a neighborhood ᏺ ⊂ ᏺ of on which (1/d n )f∗n (χσ u ) ≥ (λn /d n )σ u . Now Theorem 1.7 gives that pushforwards of χσ u have mass bounded by Cd n for some C, so we conclude that λ ≤ d. If is terminal, then the results above show that ν := σ u ∧T deﬁnes a probability measure on . Now σ u has the invariance property f∗ σ u = λ · σ u near ᏺ, so by Proposition 1.8, we see that f∗ ν = (λ/d) · ν. But the mass of a positive measure is preserved under pushforward, so λ = d. Finally, suppose that λ = d and that χ is as before. By Theorem 1.7, the sequence σnu =

n

1 1 j f∗ (χσ u ) n dj j =1

u on has a subsequence that converges weakly to a positive closed (1, 1)-current σ∞ 2 u u u u u P . Furthermore, f∗ σ∞ = d · σ∞ . Since f∗ σ = d · σ near , we have σ∞ ≥ σ u . u = σ u near . The last condition in the deﬁnition of a saddle set gives, in fact, that σ∞ u u (). u But the set Wloc () is locally closed in ᏺ, so we have that supp σ∞ ∩ ᏺ ⊂ Wloc u u Because of the transformation property f∗ σ = d ·σ , Proposition 1.8 gives us that u ∧ T is f -invariant. In particular, the Poincaré recurrence the (ﬁnite) measure ν = σ∞ u () theorem tells us that ν at almost every point is recurrent. But supp ν ∩ ᏺ ⊂ Wloc for the neighborhood ᏺ ⊃ described above, and the last condition in the deﬁnition

TOPOLOGICAL ENTROPY ON SADDLE SETS IN P2

271

u () are those in . of a saddle set guarantees that the only recurrent points in Wloc u ∧ T = σ u ∧ T , so it We conclude that supp ν ∩ ᏺ ⊂ . Near we have that ν = σ∞ follows from Corollary 2.2 that is terminal.

We remark that a more general version of Theorem A can be had with essentially the same proof we have just given. If f : Pk is a holomorphic map and ⊂ Pk is a saddle set with a single expanding direction, then we can again show that htop (f | ) ≤ log d and equality holds if and only if is terminal. The deﬁnitions of saddle set and terminal are, moreover, exactly the same as the ones we have used here. However, our proof does not work as is for saddle sets with l > 1 expanding directions. The problem here is that the appropriate slice measures to consider are of ˆ u (p) the form T l |Wloc ˆ , pˆ ∈ . Whereas the support of a slice of T is equal to the set on which iterates of f fail to be normal, no such relationship is known for slices of T l . In particular, there is no guarantee that slices of T l are nonzero on any of the local unstable manifolds. 3. Examples: Proof of Theorem B. In this section, we give several examples of saddle sets in P2 , both terminal and nonterminal. The latter ones are signiﬁcantly harder to construct and their existence constitutes the statement of Theorem B. 3.1. Terminal saddle sets. Terminal saddle sets in P2 are easy to ﬁnd. Example 3.1. Let p and q be polynomial mappings of C of common degree d ≥ 2, and let f (z, w) = (p(z), q(w)) be the product map. Assume that q is hyperbolic and let Aq be the set of attracting periodic points of q. If z is an attracting ﬁxed point of p, and Jq is the Julia set of q, then := {z} × Jq is a terminal, mixing saddle set for ˆ then W u (ˆr ) − = {z} × (C − Jq ), so if U ⊂⊂ W u (ˆr ) − is f . Indeed, if rˆ ∈ , loc loc n connected, then f converges uniformly on U to some point in ({z} × Aq ) ∪ {[0 : 1 : 0]}. Thus, is terminal. The next two examples are quite similar to the ﬁrst one. Example 3.2. This example uses a construction of Ueda [U2]. Namely, let g be ˆ # P1 of degree d ≥ 2. There exists a branched covering π : a rational map of C 1 1 2 P × P → P , which semiconjugates g × g to a holomorphic mapping f : P2 of degree d. If g is hyperbolic with Julia set Jg and z is an attracting ﬁxed point for g, then := π({z} × Jg ) is a terminal, mixing saddle set for f . Example 3.3. Let f (z, w) = (p(z, w), q(z, w)) be a polynomial mapping of C2 of degree d ≥ 2. Assume that f is regular, that is, that f extends to a holomorphic mapping of P2 (see [BJ]). The line at inﬁnity 5 := P2 − C2 # P1 is completely invariant and the restriction f5 of f to 5 is a rational map f5 , the Julia set of which we denote by J5 . If f5 is hyperbolic, then := J5 is a terminal, mixing saddle set for f in P2 . Remark 3.4. In the above three examples, the unstable currents σ u are given as currents of integration on the curves {z} × C, π({z} × P1 ) and 5, respectively.

272

DILLER AND JONSSON

Perturbations of these examples yield new terminal saddle sets. Indeed, if is a u () − is in terminal saddle set for a holomorphic mapping f : P2 such that Wloc the union of basins of attraction of ﬁnitely many sinks, then any small perturbation of f has a terminal saddle set close to . For perturbations of the three examples, the current σ u is, in general, not a current of integration on an analytic set. Example 3.5. Consider a polynomial automorphism of C2 of the form f0 (z, w) = (p(z)+aw, z), where p is a polynomial of degree d ≥ 2. Assume that f0 is hyperbolic, that is, that J is a hyperbolic set (see [BS]). Let f (z, w) = (p(z) + aw, z + !wd ) for small ! > 0. Then f has a terminal saddle set , which is a perturbation of J . 3.2. Nonterminal saddle sets. We now prove Theorem B by constructing mappings with nonterminal saddle sets in P2 . The examples that we describe belong to the family of polynomial skew products on C2 —a class of nontrivial holomorphic mappings on P2 with tractable dynamics. We start by recalling some facts about skew products on C2 . Then we state sufﬁcient conditions for a skew product to be Axiom A and have a nonterminal saddle set. Finally, we show how to construct explicit examples where these conditions are satisﬁed. 3.2.1. Polynomial skew products on C2 . These are mappings of C2 of the form f (z, w) = p(z), q(z, w) , (3.1) where p and q are polynomials of the same degree d ≥ 2, and q has nonvanishing wd term. Polynomial skew products have been studied by Heinemann (see [H1], [H2]), by Sester (see [Se1], [Se2]), and by the second author (see [J2], [J3]). We recall a few deﬁnitions and results from [J2]. The ﬁrst component of (3.1) deﬁnes a polynomial mapping of C, the Julia set and ﬁlled Julia set of which are denoted by Jp and Kp , respectively. The polynomial p is said to be uniformly expanding on Jp (or hyperbolic) if there exist c > 0 and λ > 1 such that |Dp n (z)| > cλn for z ∈ Jp and n ≥ 1. We write qz for the polynomial mapping q(z, ·) of C, deﬁned by (3.1), and we denote the composition qzn−1 ◦ · · · ◦ qz by Qnz . If z ∈ Kp , then we denote by Kz the ﬁlled Julia set of {Qnz }n≥1 , that is, the set where this family is bounded. Also, we set Jz = ∂Kz . If Z ⊂ Kp is compact and p(Z) ⊂ Z, then we say that f is vertically expanding over Z if there exists c > 0 and λ > 1 such that |DQnz (w)| ≥ cλn for z ∈ Z, w ∈ Jz , and n ≥ 1. A polynomial skew product (3.1) of C2 extends to a holomorphic mapping of P2 . The line at inﬁnity 5 := P2 − C2 # P1 is completely invariant and the restriction f5 of f to 5 is a polynomial map f5 , the Julia set of which we denote by J5 . The following result characterizes Axiom A for polynomial skew products. Theorem 3.6 [J2, Theorem 8.2]. A polynomial skew product (3.1), viewed as a holomorphic mapping of P2 , is Axiom A if and only if the following are true.

TOPOLOGICAL ENTROPY ON SADDLE SETS IN P2

(i) (ii) (iii) (iv)

273

p is uniformly expanding on Jp ; f is vertically expanding over Jp ; f is vertically expanding over Ap , the set of attracting periodic points for p; f5 is uniformly expanding on J5 .

Conditions (i)–(iv) are most easily checked in terms of the postcritical set. Indeed, it is a standard 1-dimensional result that a rational map is uniformly expanding on its Julia set if and only if its postcritical set is disjoint from the Julia set. A generalization of this holds for skew products. Let Cz denote the critical set of qz for z ∈ C. Let Z ⊂ Kp be compact and invariant, and let DZ denote the postcritical set over Z; that is,

DZ = fn {z} × DZ,z , {z} × Cz =: z∈Z

n≥1

z∈Z

where the last equation deﬁnes DZ,z . Also, let ∗ JZ,z = lim sup Jz ζ →z, ζ ∈Z

in the Hausdorff metric (z → Jz is not continuous in general). Theorem 3.7 [J2, Theorem 3.1]. f is vertically expanding over Z if and only if ∗ = ∅ for all z ∈ Z. DZ,z ∩ JZ,z The most important situation is when Z = Jp . We then write Dz := DJp ,z and Jz∗ := JJ∗p ,z . 3.2.2. A sufﬁcient condition. The following proposition gives a sufﬁcient condition for a polynomial skew product on C2 to be Axiom A and to have a nonterminal saddle set with topological entropy log k for some integer k less than the degree of the skew product. In Section 3.2.3, we show how to construct skew products satisfying the conditions in the proposition below. Taken together, this provides a proof of Theorem B. Proposition 3.8. Let 1 ≤ k < d and let f (z, w) = (p(z), q(z, w)) be a polynomial skew product of degree d on C2 such that qz (w) = w d +r(z) for some polynomial r of degree less than d and such that the following are true. (i) There exist λ > 1, a compact set D ⊂ C, and d disjoint subsets Dj of D such that p −1 (D) = D1 ∪ · · · ∪ Dd and p : Dj → D is a homeomorphism with |Dp(z)| ≥ λ, z ∈ Dj , 1 ≤ j ≤ d. (ii) There exist numbers M ≥ 3, A ≥ M + 2 such that if we write E = D1 ∪ · · · ∪ Dk

and

E = Dk+1 ∪ · · · ∪ Dd ,

274

DILLER AND JONSSON

then |r(z)| ≤ 1/5 for z ∈ E, 3A ≤ |r(z)| ≤ MA for z ∈ E . Then f is Axiom A on P2 and has a nonterminal basic set with htop (f | ) = log k. The idea is that if z ∈ E, then qz (w) ≈ wd , whereas if z ∈ E , then qz (w) = + r(z), where r(z) is large. The set is the closure of the set of all periodic saddle points (z, w) with z ∈ E. Proof of Proposition 3.8. It follows easily from (i) that Jp = n≥0 p −n (D) and that p is uniformly expanding on Jp . In fact, p|Jp is isomorphic to the full shift on d symbols. Let L be the set of z ∈ Jp such that zn ∈ E for all n ≥ 0. Then L is compact and p|L is isomorphic to the full shift on k symbols. We estimate the position of the Julia set Jz and the postcritical set Dz for z ∈ Jp . Using the triangle inequality, we easily show that wd

z ∈ E ∪ E , |w| ≥ A %⇒ |qz (w)| ≥ 2|w|, 1 z ∈ E , |w| ≤ %⇒ |qz (w)| ≥ 2A, 3 1 14 z ∈ E, |w| ≤ %⇒ |qz (w)| ≤ . 3 45

(3.2) (3.3) (3.4)

It follows from this that z ∈ Jp %⇒ Kz ⊂ DA , z ∈ L %⇒ D1/3 ⊂ Kz , z ∈ Jp − L %⇒ D1/3 ∩ Kz = ∅.

(3.5) (3.6) (3.7)

Since Cz = {0} for all z ∈ C, we also get z ∈ Jp %⇒ Dz ⊂ D14/45 ∪ (C − D2A ).

(3.8)

We are now in position to prove that f is Axiom A, using Theorem 3.6. Indeed, p is uniformly expanding on Jp by (i) and f5 is uniformly expanding on J5 , since f5 is afﬁnely conjugate to ζ → ζ d . Furthermore, (3.6)–(3.8) imply that d(Dz , Jz∗ ) ≥ 1/45 for all z ∈ Jp , so by Theorem 3.7, f is vertically expanding over Jp . Since p has no attracting periodic points, f must be Axiom A on P2 by Theorem 3.6. From the proof of Theorem 3.6, it follows that the nonwandering set of f can be written as $ = A5 ∪ J5 ∪ J2 ∪ . Here A5 is the set of sinks of f5 , and J5 is the Julia set of f5 . Furthermore, J2 = z∈Jp {z} × Jz is the closure of the repelling periodic points of f , and is the closure of the periodic saddle points (z, w) with z ∈ Jp . We analyze the set in more detail. If (z, w) is a periodic point in , say, of period n, then p n (z) = z and w ∈ int Kz . Suppose that z ∈ / L. Then it follows from

TOPOLOGICAL ENTROPY ON SADDLE SETS IN P2

275

(3.2)–(3.4) that all critical points of Qnz have unbounded orbits. Thus, int Kz = ∅, which is a contradiction. We conclude that all periodic points (z, w) ∈ have z ∈ L. By (3.4), we have qz (D1/3 ) ⊂ D14/45 for z ∈ L. A theorem of Sester [Se2] applied to the restriction of f to L × C then shows that Kz is a quasidisk for all z ∈ L. ˆ then we ˆ p) Let (L, ˆ be the natural extension of p|L . If zˆ = (zi )i≤0 is a history in L, deﬁne θ (ˆz) ∈ by qzk−k (D1/3 ). θ(ˆz) = k≥0

ˆ Write This is well deﬁned by (3.4) and θ is continuous on L. ˜ := θ(ˆz). zˆ ∈Lˆ

˜ is compact. We claim that ˜ = . To see this, ﬁrst notice that if zˆ is a periodic Then history, say, of order n, then θ (ˆz) coincides with the (unique) ﬁxed point of Qnz . Thus, ˜ so ⊂ . ˜ every periodic saddle point in L × C is contained in ˜ The To prove the reverse inclusion, we show that f is topologically mixing on . 2 ˜ = ∅ argument is adapted from [JW]. Let U1 and U2 be two open sets in C with Ui ∩ n ˜ for i = 1, 2. We show that f (U1 ) ∩ U2 = ∅ for large n. Pick x ∈ U2 ∩ and write ˆ We can ﬁnd k ≥ 0 such that f k ({z−k } × Vz−k ) ⊂⊂ U2 . By x = θ (ˆz) for some zˆ ∈ L. continuity there is an open subset ω of z−k in L such that f k ({z} × Vz−k ) ⊂⊂ U2 for z ∈ ω. Let ω1 be the projection of U1 on the z-axis. Since p is topologically mixing on L, there exists N ≥ 0 such that if n ≥ N, then pn (ω1 ) ∩ ω = ∅. It follows that ˜ and ˜ = . f n (U1 ) ∩ U2 for n ≥ k + N. Thus f is topologically mixing on Finally, we claim that the topological entropy of f | equals log k. Since d < k, Theorem A then shows that is nonterminal. On the one hand, the projection (z, w) → z semiconjugates f | to p|L , so htop (f | ) ≥ htop (p|L ) = log k, since p|L is isomorphic to the full shift on k symbols. On the other hand, by a result of Bowen [B1], we have htop (f | ) ≤ htop (p|L ) + sup htop f, ∩ {z} × C = log k, z∈L

since htop (f, ∩{z}×C) = 0 for z ∈ L by contraction. This completes the proof that htop (f | ) = log k. 3.2.3. Satisfying the hypotheses. We now show that we can ﬁnd, for every d and k, a polynomial skew product on C2 that satisﬁes the assumptions of Proposition 3.8. The following example, which is similar to [J2, Example 9.6], does the job for d = 2, k = 1.

276

DILLER AND JONSSON

Example 3.9. Let f (z, w) = (z2 − 9900, w 2 + (99 √ + z)/6). We take D = [−100, 100], D1 = [−100, −b], D2 = [b, 100] where b = 9800 ≈ 98.99. Then |Dp(z)| = 2|z| ≥ 196 for z ∈ D1 ∪ D2 . Furthermore, r(z) = (99 + z)/6, so we have |r(z)| ≤ 1/5 when z ∈ D1 and 32 ≤ |r(z)| ≤ 34 when z ∈ D2 . Thus we can take A = 10, M = 4. In this case, the nonterminal basic set is the single saddle ﬁxed point (−99, 0). We now turn to the general case with 1 ≤ d < k. Let p0 (z) be a polynomial of degree d with exactly d different zeros ξ1 · · · ξd . Let a > 0 be a large number and deﬁne p(z) = a · p0 (z). Fix a closed disk D centered at the origin and containing all the points ξj in its interior. If a is large enough, then p−1 (D) is contained in the interior of D and consists of d disjoint Jordan domains Dj , j = 1, . . . , d. Furthermore, the restriction of p maps Dj homeomorphically onto D and |Dp| ≥ 2 on Dj . Notice that diam(Dj ) ∼ 1/a. J5

5

Figure 1. A nonterminal basic set. The picture shows how forward images of a local unstable manifold of a history in intersect local stable manifolds of points in J5 , and hence accumulate on all of 5.

Pick an integer k with 1 ≤ k < d, let E be the union of k of the sets Dj , and let E be the union of the remaining sets Dj . Deﬁne (z − ξj ). r0 (z) = ξj ∈E

Then there exist constants m0 , m1 , and m2 such that m2 for z ∈ E, |r0 (z)| ≤ a m0 ≤ |r0 (z)| ≤ m1 for z ∈ E for all large enough a. Now let r(z) = (a/5m2 )r0 (z), A = am0 /15m2 , M = 3m1 /m0 . Then for large a, we have M ≥ 3, A ≥ M + 2, 1 5 3A ≤ |r(z)| ≤ MA |r(z)| ≤

for z ∈ E, for z ∈ E .

TOPOLOGICAL ENTROPY ON SADDLE SETS IN P2

277

This completes the construction and hence the proof of Theorem B. Remark 3.10. It is possible to say more precisely how the set in Proposition 3.8 ˆ then f n (W u (ˆr )) has transverse intersections fails to be terminal. Namely, if rˆ ∈ , loc u (p)) with local stable manifolds of points in J5 for large n. This implies that f n (Wloc ˆ n accumulates on all of 5 (see Figure 1). In particular, {f } fails to be normal on u (ˆ Wloc r ) − . References [A] [BJ] [BS] [BT] [B1] [B2] [BM] [FS1]

[FS2] [H1] [H2] [HP] [J1] [J2] [J3] [JW] [KH] [PS] [RS] [Se1] [Se2] [SFL]

E. Akin, The General Topology of Dynamical Systems, Grad. Stud. Math. 1, Amer. Math. Soc., Providence, 1993. E. Bedford and M. Jonsson, Regular polynomial endomorphisms of Ck , preprint, 1998. E. Bedford and J. Smillie, Polynomial diffeomorphisms of C2 : Currents, equilibrium measure and hyperbolicity, Invent. Math. 103 (1991), 69–99. E. Bedford and B. A. Taylor, The Dirichlet problem for a complex Monge-Ampère equation, Invent. Math. 37 (1976), 1–44. R. Bowen, Entropy for group endomorphisms and homogeneous spaces, Trans. Amer. Math. Soc. 153 (1971), 401–414. , Equilibrium States and the Ergodic Theory of Anosov Diffeomorphisms, Lecture Notes in Math. 470, Springer, Berlin, 1975. R. Bowen and B. Marcus, Unique ergodicity for horocycle foliations, Israel J. Math. 26 (1977), 43–67. J. E. Fornæss and N. Sibony, “Complex dynamics in higher dimension. II” in Modern Methods in Complex Analysis (Princeton, N.J., 1992), 135–182, Ann. of Math. Stud. 137, Princeton Univ. Press, Princeton, 1995. , Hyperbolic maps on P2 , Math. Ann. 311 (1998), 305–333. S.-M. Heinemann, Julia sets for holomorphic endomorphisms of Cn , Ergodic Theory Dynam. Systems 16 (1996), 1275–1296. , Julia sets of skew products in C2 , Kyushu J. Math. 52 (1998), 299–329. J. H. Hubbard and P. Papadopol, Superattractive ﬁxed points in Cn , Indiana Univ. Math. J. 43 (1994), 321–365. M. Jonsson, Dynamical studies in several complex variables, Ph.D. thesis, Royal Institute of Technology, November 1997. , Dynamics of polynomial skew products on C2 , to appear in Math. Ann. , Ergodic properties of ﬁbered rational maps and applications, preprint. M. Jonsson and B. Weickert, A nonalgebraic attractor in P2 , to appear in Proc. Amer. Math. Soc. A. Katok and B. Hasselblatt, Introduction to the Modern Theory of Dynamical Systems, Encyclopedia Math. Appl. 54, Cambridge Univ. Press, Cambridge, 1995. C. Pugh and M. Shub, Ergodic attractors, Trans. Amer. Math. Soc. 312 (1989), 1–54. D. Ruelle and D. Sullivan, Currents, ﬂows and diffeomorphisms, Topology 14 (1975), 319–327. O. Sester, Étude dynamique des polynômes ﬁbrés, Ph.D. thesis, Université de Paris-Sud, 1997. , Hyperbolicité des polynômes ﬁbrés, preprint. M. Shub, A. Fathi, and R. Langevin, Global Stability of Dynamical Systems, trans. J. Christy, Springer, New York, 1987.

278 [U1] [U2]

DILLER AND JONSSON T. Ueda, Fatou sets in complex dynamics on projective spaces, J. Math. Soc. Japan 46 (1994), 545–555. , Complex dynamical systems on projective spaces, preprint.

Diller: Department of Mathematics, University of Notre Dame, Notre Dame, Indiana 46556, USA; [email protected] Jonsson: Department of Mathematics, University of Michigan, Ann Arbor, Michigan 48109-1109, USA; [email protected]

Vol. 103, No. 2

DUKE MATHEMATICAL JOURNAL

© 2000

ARRANGEMENT OF HYPERPLANES, II: THE SZENES FORMULA AND EISENSTEIN SERIES MICHEL BRION and MICHÈLE VERGNE

To Victor Guillemin, for his 60th birthday 1. Introduction. Consider a sequence (α1 , α2 , . . . , αk ) of linear forms in r complex variables, with integral coefﬁcients. The linear forms αj need not be distinct. For example, r = 2 and α1 = α2 = z1 , α3 = α4 = z2 , α5 = α6 = z1 +z2 . For any such sequence, D. Zagier [5] introduced the series n∈Zr ,αj ,n=0

k

1

j =1 αj , n

.

Assuming convergence, its sum is a rational multiple of π k . For example (see [5]), we have (2π)6 1 . = 30240 n2 n2 (n + n2 )2 n =0,n =0,n +n =0 1 2 1 1

2

1

2

These numbers are natural multidimensional generalizations of the value of the Riemann zeta function at even integers. A. Szenes gave in [3, Theorem 4.4] a residue formula for these numbers, relating them to Bernoulli numbers. The formula of Szenes [3] is the multidimensional analogue of the residue formula 1 1 2l B2l l 2l . = (−1) (2π) Resz=0 2l = (2π ) (2l)! n2l z (1 − ez ) n=0

A motivation for computing such sums comes from the work of E. Witten [4]. In the special case where αj are the positive roots of a compact connected Lie group G, each of these roots being repeated with multiplicity 2g − 2, Witten expressed the symplectic volume of the space of homomorphisms of the fundamental group of a Riemann surface of genus g into G, in terms of these sums. In [2], L. Jeffrey and F. Kirwan proved a special case of the Szenes formula leading to the explicit computation of this symplectic volume, when G is SU(n). Our interest in such series comes from a different motivation. Let us consider ﬁrst the case. By the Poisson formula, for Re(z) > 0, the convergent 1-dimensional −mz is also equal to 2 series ∞ me m=1 n∈Z 1/(z+2iπn) . Similarly, sums of products Received 5 March 1999. 2000 Mathematics Subject Classiﬁcation. Primary 52C35; Secondary 11B68, 40H05. 279

280

BRION AND VERGNE

of polynomial functions with exponential functions over all integral points of an r-dimensional rational convex cone are related to functions of r complex variables of the form 1 ψ(z) = . k j =1 αj , z + 2iπn n∈Zr When this series is not convergent, introduce the oscillating factor et,2iπn and deﬁne the Eisenstein series et,z+2iπn ψ(t, z) = , k j =1 αj , z + 2iπn n∈Zr a generalized function of t ∈ Rr . In Section 3, we construct a decomposition of an open dense subset of Rr into alcoves such that t → ψ(t, z) is given on each alcove by a polynomial in t, with rational functions of ez as coefﬁcients. Our ﬁrst theorem (see Theorem 19) gives an explicit residue formula for ψ(t, z). It follows easily from the obvious behaviour of ψ(t, z) under differentiation in z. This formula allows us to give a residual meaning “ψ(t, 0)” for the value of ψ(t, z) at z = 0, although ψ(t, z) clearly has poles along all hyperplanes αj , z = 0. An alternate way to deﬁne ψ(t, 0) is to remove all inﬁnities 1/αj in the series ψ(t, 0) =

n∈Zr

k

et,2iπn

j =1 αj , 2iπn

.

Indeed, we prove that the residue formula for “ψ(t, 0)” coincides with the renormalized sum: et,2iπn “ψ(t, 0)” = . k α , 2iπn j r j =1 n∈Z ,α ,n=0 j

This equality gives another proof of the Szenes residue formula, as a “limit” of a natural formula for ψ(t, z) when z → 0 along a generic line. To illustrate our method, let us consider the 1-dimensional case. For k ≥ 2, we can deﬁne the Eisenstein series 1 . Ek (z) = (z + 2iπn)k n∈Z

Clearly, Ek (z) is periodic in z with respect to translation by the lattice 2iπZ. From the residue theorem, when y is not in 2iπZ, we have the kernel formula 1 (1) . Ek (y) = Resz=0 k z (1 − ez−y ) Observe that the right-hand side has a meaning when y = 0, and equals, by deﬁnition, the Bernoulli number Bk /k!. The function

ARRANGEMENT OF HYPERPLANES, II

Ek (y) =

1 + yk

n∈Z,n=0

281

1 (y + 2iπn)k

has a Laurent expansion at y = 0, with 1/y k as Laurentnegative part. We see from the residue formula that the constant term CT(Ek ) = n∈Z,n=0 1/(2iπn)k equals Resz=0 (1/(zk (1 − ez ))). In view of this example, we call the value “ψ(t, 0)” of ψ(t, y) at y = 0 the constant term of the Eisenstein series et,z+2iπn . k j =1 αj , z + 2iπn n∈Zr Acknowledgments. We thank A. Szenes and the referees of our paper for several suggestions. 2. Kernel formula. In this section, we brieﬂy recall results of [1] with slightly modiﬁed notation. Let V be an r-dimensional complex vector space. Let V ∗ be the dual vector space, and let ⊂ V ∗ be a ﬁnite subset of nonzero linear forms. Each α ∈ determines a hyperplane {α = 0} in V . Consider the hyperplane arrangement Ᏼ= {α = 0}. α∈

An element z ∈ V is called regular if z is not in Ᏼ. If S is a subset of V , we write Sreg for the set of regular elements in S. The ring R of rational functions with poles on Ᏼ is the ring −1 S(V ∗ ) generated by the ring S(V ∗ ) of polynomial functions on V , together with inverses of the linear functions α ∈ . The ring R has a Z-gradation by the homogeneous degree that can be positive or negative. Elements of R are deﬁned on the open subset Vreg . (Our notation differs from [1] in that the roles of V and V ∗ are interchanged.) In the one-variable case, the function 1/z is the unique function that cannot be obtained as a derivative. There is a similar description of a complement space to the space of derivatives in the ring R , which we recall now. A subset σ of is called a basis of if the elements α ∈ σ form a basis of V . We denote by Ꮾ() the set of bases of . An ordered basis is a sequence (α1 , α2 , . . . , αr ) of elements of such that the underlying set is a basis. We denote by OᏮ() the set of ordered bases. For σ ∈ Ꮾ(), set 1 . φσ (z) := α∈σ α(z) We call φσ a simple fraction. Setting zj = z, αj , we have φσ (z) =

1 . z1 z2 · · · zr

282

BRION AND VERGNE

Deﬁnition 1. The subspace S of R spanned by the elements φσ , σ ∈ Ꮾ(), will be called the space of simple elements of R : Cφσ . S = σ ∈Ꮾ()

The space S consists of homogeneous rational functions of degree −r. However, not every homogeneous element of degree −r of R is in S (e.g., in the preceding notation, if r ≥ 2, both functions 1/z1r and z2 /z1r+1 are not in S ). Furthermore, we must be careful, as the elements φσ may be linearly dependent. For example, if V = C2 and = {z1 , z2 , z1 + z2 }, we have S = C

1 1 1 +C +C z1 z2 z1 (z1 + z2 ) z2 (z1 + z2 )

and we have the relation 1 1 1 + . = z1 z2 z1 (z1 + z2 ) z2 (z1 + z2 ) A description due to Orlik and Solomon of all linear relations between the elements φσ is given in [1, Proposition 13]. Deﬁnition 2. A basis B of Ꮾ() is a subset of Ꮾ() such that the elements φσ , σ ∈ B, form a basis of S : Cφσ . S = σ ∈B

We let elements v of V act on R by differentiation:

d f (z + #v)|#=0 . ∂(v)f (z) := d#

Then the following holds (see [1, Proposition 7]). Theorem 3. We have R = ∂(V )R ⊕ S . Thus, we see that only simple fractions cannot be obtained as derivatives. As a corollary of this decomposition, we can deﬁne the projection map Res : R −→ S . The projection Res f (z) of a function f (z) is a function of z that we call the Jeffrey-Kirwan residue of f . By deﬁnition, this function can be expressed as a linear combination of the simple fractions φσ . The main property of the map Res is that it vanishes on derivatives, so that for v ∈ V , f, g ∈ R ,

Res ∂(v)f g = − Res f ∂(v)g . (2)

ARRANGEMENT OF HYPERPLANES, II

283

If oσ ∈ OᏮ() is an ordered basis, an important functional Resoσ can be deﬁned on R : the iterated residue with respect to the ordered basis oσ . If we write an element z ∈ V on the basis oσ = (α1 , α2 , . . . , αr ) as z = (z1 , . . . , zr ), then

Resoσ (f ) = Resz1 =0 Resz2 =0 · · · Reszr =0 f (z1 , z2 , . . . , zr ) · · · . The map Resoσ depends on the order oσ chosen on σ and not only on the basis σ underlying oσ . The restriction of the functional Resoσ to S is called r oσ . We have (3)

Resoσ = r oσ Res .

Indeed, we have only to check that Resoσ vanishes on derivatives. If oσ = (α1 , α2 , . . . , αr ) and z = (z1 , . . . , zr ), the iterated residue Resoσ vanishes at the step Reszj =0 on ∂R /∂zj . Recall the following deﬁnition from A. Szenes (see [3, Deﬁnition 3.3]). Deﬁnition 4. A diagonal basis is a subset OB of OᏮ() such that the following are true. (1) The set of underlying (unordered) bases forms a basis B of Ꮾ(). (2) The dual basis to the basis (φσ , oσ ∈ OB) is the set of linear forms (r oσ , oσ ∈ OB): r oτ (φσ ) = δστ . In [3, Proposition 3.4], it is proved that a total order on gives rise to a diagonal basis. (This is proved again in more detail in [1, Proposition 14].) In the 1-dimensional case, S = Cz−1 , and the space G = k≤−1 Czk of negative Laurent series is the space obtained from the function 1/z by successive derivations. In the case of several variables, we can also characterize the space generated by simple fractions under differentiation. Let κ be a sequence of (not necessarily distinct) elements of . The sequence κ is called generating if the α ∈ κ generate the vector space V ∗ . We denote by G the subspace of R spanned by the φκ :=

1 α∈κ α

,

where κ is a generating sequence. Finally, we denote by S(V ) the ring of differential operators on V , with constant coefﬁcients. This ring acts on S(V ∗ ) and on R . Proposition 5 [1, Theorem 1]. The space G is the S(V )-submodule of R generated by S . For example, if = {z1 , z2 , z1 + z2 }, we have 1 1 ∂ 1 ∂ ∂ + =− . − z1 z2 (z1 + z2 ) ∂z1 z1 z2 ∂z1 ∂z2 z1 (z1 + z2 )

284

BRION AND VERGNE

In particular, every element of G can be expressed as a linear combination of elements 1 , nα α∈σ α where σ is a basis and the nα are positive integers. For example, the above equality is equivalent to 1 1 1 . = 2 − 2 z1 z2 (z1 + z2 ) z1 z2 z1 (z1 + z2 ) The ring S(V ∗ ) operates by multiplication on R . It is also useful to consider the action of the ring Ᏸ(V ) of differential operators with polynomial coefﬁcients, generated by S(V ) and S(V ∗ ). The following lemma is an obvious corollary of the description of G . Lemma 6. The space R is generated by G as an S(V ∗ )-module. It is generated by S as a Ᏸ(V )-module. Consider now the space ᏻ of holomorphic functions on V deﬁned in a neighborhood of zero. Let ᏻ = −1 ᏻ be the space of meromorphic functions in a neighborhood of zero, with products of elements of as denominators. The space ᏻ is a module for the action of differential operators with constant coefﬁcients. Via the Taylor series at the origin of elements of ᏻ, the residue Res f (z) still has a meaning if f (z) ∈ ᏻ ; indeed, Res f (z) = 0 if f ∈ R is homogeneous of degree not equal to −r. If y ∈ V is sufﬁciently near zero and f ∈ ᏻ , the function

᐀(y)f (z) := f (z − y) is still an element of ᏻ . Moreover, if y is regular, then f (z − y) is deﬁned for z = 0 and thus is an element of ᏻ. If f ∈ R , we denote by m(f ) the operator of multiplication by f :

m(f )φ (z) := f (z)φ(z). It operates on ᏻ . Finally, we denote by C the operator (Cf )(z) := f (−z) on ᏻ . Theorem 7 (Kernel theorem). Let A : R → ᏻ be an operator commuting with the action of differential operators with constant coefﬁcients. For y ∈ V regular, sufﬁciently near zero, and for f ∈ G , we have the formula

(Af )(y) = Tr S Res m(f )C ᐀(y)A Res .

ARRANGEMENT OF HYPERPLANES, II

285

∗ More explicitly, choose a basis B of Ꮾ(), and let (φ σ , σ ∈ B) be the basis of S dual to the basis (φσ , σ ∈ B) of S . Then we have the kernel formula

(Af )(y) = φ σ , Res f (z)Aσ (y − z) , σ ∈B

where Aσ (z) = A(φσ )(z). Concretely, this formula has the following meaning. Let f be homogeneous of degree d. We ﬁx y regular and small. The function z → Aσ (y − z) is deﬁned near z = 0. The Jeffrey-Kirwan residue Res of the function z → f (z)Aσ (y − z) is a function of z belonging to the space S . We pair it with the linear form φ σ on S , and we obtain a certain complex number depending on y. More precisely, consider the Taylor expansion Aσ (y − z) = Aσ (y) +

∞ j =1

Ajσ (y, z),

j

where Aσ (y, z) is the part of the Taylor expansion at zero of the holomorphic function z → Aσ (y − z), which is homogeneous of degree j in z. We have Ajσ (y, z) = (−1)j

(k),|(k)|=j

A(k) σ (y)

z(k) , (k)!

(k)

where (k) = (k1 , . . . , kr ) is a multi-index, and Aσ (y) = ((∂/∂y)(k) Aσ )(y). Then, as the Jeffrey-Kirwan residue vanishes on homogeneous terms of degree not equal to −r, we obtain

Res f (z)Aσ (y − z) = Res f (z)Aσ−d−r (y, z) z(k) d+r (k) . Aσ (y) Res f (z) = (−1) (k)! (k),|(k)|=−d−r

Thus, φ σ , Res (f (z)Aσ (y − z)) is equal to z(k) σ (−1)d+r . A(k) (y) φ , Res f (z) σ (k)! (k),|(k)|=−d−r

(k)

f

Set cσ (f ) = φ σ , Res (f (z)(z(k) /(k)!)). Let Pσ (∂/∂y) be the differential operator with constant coefﬁcients deﬁned by (k) ∂ ∂ f d+r (k) = (−1) cσ (f ) . Pσ ∂y ∂y (k),|(k)|=−d−r

286

BRION AND VERGNE f

Then Pσ depends linearly on f , and

σ

φ , Res f (z)Aσ (y − z) =

The claim of the theorem is that (Af )(y) =

σ ∈B

Pσf

Pσf

∂ Aσ (y). ∂y

∂ · Aσ (y). ∂y

We now prove this theorem. Proof. Deﬁne an operator A : R → ᏻ by

(A f )(y) = φ σ , Res f (z)Aσ (y − z) . σ ∈B

We ﬁrst check that A

commutes with the action of differential operators with constant coefﬁcients. Using the equation

∂y (v)φ (y − z) = − ∂z (v)φ (y − z)

and the main property (2) of Res , we obtain

∂y (v) · φ σ , Res f (z)Aσ (y − z) = φ σ , Res f (z) ∂y (v) · Aσ (y − z)

= − φ σ , Res f (z) ∂z (v) · Aσ (y − z)

= φ σ , Res ∂z (v) · f Aσ (y − z) . It remains to see that A and A coincide on S . For this, we use the following formula. If P is a polynomial and φ a simple fraction, then Res (P φ) = P (0)φ.

(4)

To see this, recall that the function φ is homogeneous of degree −r. As P ∈ S(V ∗ ), P − P (0) is a sum of homogeneous terms of positive degree. Thus, for homogeneity reasons, Res ((P − P (0))φ) = 0. Let y be regular, and let σ, τ ∈ B. As the function z → Aσ (y − z) is an element of ᏻ, by formula (4) we obtain

Res φτ (z)Aσ (y − z) = Aσ (y)φτ (z). Thus, A (φτ )(y) =

φ σ , Res φτ (z)Aσ (y − z)

σ ∈B

=

σ ∈B

φ σ , φτ Aσ (y) =

σ ∈B

δστ Aσ (y) = Aτ (y) = A(φτ )(y).

Choosing a diagonal basis OB and using equation (3), we obtain an iterated residue formula for (Af )(y).

ARRANGEMENT OF HYPERPLANES, II

287

Corollary 8. For any diagonal basis OB of Ꮾ(), we have, for f ∈ G ,

(Af )(y) = Resoσ f (z)Aσ (y − z) , oσ ∈OB

where Aσ (z) = A(φσ )(z). Corollary 8 applies to the identity operator A : R → R . If f ∈ G , we obtain f (y) = oσ ∈OB Resoσ (f (z)φσ (y − z)). But if f ∈ NG , then clearly Resoσ (f (z) φσ (y − z)) = 0, as the Taylor series of f (z)φσ (y − z) at z = 0 is also in NG . As a consequence, we obtain a formula for the Jeffrey-Kirwan residue as a function of iterated residues. Lemma 9. For any f ∈ R , we have (Res f )(y) = Resoσ (f )φσ (y). oσ ∈OB

Similarly, if Z : R → ᏻ is an operator commuting with the action of differential operators with constant coefﬁcients, the formula

Z(f )(y) = Tr S Res m(f )C ᐀(y)Z Res is valid for all elements y ∈ V sufﬁciently near zero and for all f ∈ G . In particular, we have the following proposition. Proposition 10. Let Z : R → ᏻ be an operator commuting with the action of differential operators with constant coefﬁcients. Then we have, for f ∈ G ,

Z(f )(0) = Tr S Res m(f )CZ Res , where (CZ)(φ)(z) = Z(φ)(−z). Choosing a diagonal basis of OᏮ(), we can express the preceding formula as a residue formula in several variables:

Z(f )(0) = Resoσ f (z)Zσ (−z) , oσ ∈OB

with Zσ (z) = Z(φσ )(z). For later use, we prove a vanishing property of the linear form Resoσ . Let oσ be an ordered basis. We write oσ = (α1 , α2 , . . . , αr ) and z = (z1 , z2 , . . . , zr ). Set oσ = (α2 , . . . , αr ) and z = (z2 , . . . , zr ); then z = (z1 , z ). Let ψ(z ) in ᏻ be a meromorphic function with a product of linear forms α(z ), where α ∈ is not a multiple of α1 , as a denominator. Lemma 11. For any f ∈ G and for any ψ ∈ ᏻ , 1 Resoσ f (z1 , z )ψ(z ) = 0. z1

288

BRION AND VERGNE

Proof. We have

1 1 oσ oσ Res f (z1 , z )ψ(z ) = Resz1 =0 Res f (z1 , z )ψ(z ) . z1 z1

In computing Resoσ (f (z1 , z )ψ(z )), the variable z1 is ﬁxed to a nonzero value. The result Resoσ (f (z1 , z )ψ(z )) is a meromorphic function of z1 . It is thus sufﬁcient to prove that Resoσ (f (z1 , z )ψ(z )) belongs to the space G = k≤−1 Cz1k . We check this for f = φκ , where 1 φκ (z) = α∈κ α, z and κ is a generating sequence. Let κ1 := α ∈ κ, α, (z1 , 0) = 0 and

κ = α ∈ κ, α, (z1 , 0) = 0 .

As κ is generating, the set κ1 is nonempty. We ﬁx z1 = 0. We have φκ (z1 , z )ψ(z ) = φκ1 (z1 , z )φκ (z )ψ(z ) and φκ ∈ ᏻ . For α ∈ κ1 , we set α, (z1 , z ) = cα z1 + β, z , with cα = 0. We consider the Taylor expansion at z = 0 of the holomorphic function of z : 1 1 1

. = = α, (z1 , z ) cα z1 + β, z cα z1 1 + β, z /(cα z1 ) This is of the form

∞

z1−k Pk−1 (z ),

k=1

where Pk−1 (z ) is homogeneous of degree k − 1 in z . Let n = |κ1 |; then n ≥ 1. We see that the function 1 z −→ φκ1 (z1 , z ) = α∈κ1 α, (z1 , z ) has a Taylor expansion of the form

z1−k Qk−1 (z ),

k≥n

is homogeneous of degree k − 1 in z . Thus −k

Resoσ φκ1 (z1 , z )φκ (z )ψ(z ) = z1 Resoσ Qk−1 (z )φκ (z )ψ(z ) .

where Qk−1

(z )

k≥n

z

Via the Taylor series at = 0, the function φκ (z )ψ(z ) can be expressed as an inﬁnite sum of homogeneous elements with ﬁnitely many negative degrees.As the iterated residue Resoσ vanishes on elements ofdegree not equal to −(r − 1) and as

ARRANGEMENT OF HYPERPLANES, II

289

Qk−1 (z ) is homogeneous of degree k − 1, we see that the sum is ﬁnite and that Resoσ (φκ1 (z1 , z )φκ (z )ψ(z )) is in the space G as claimed. 3. Eisenstein series. Results of Section 2 are used for a complex vector space that is the complexiﬁcation of a real vector space. Thus, we slightly change the notation in this section. Let V be a real vector space of dimension r equipped with a lattice N. The complex vector space VC is the space to which we apply the results of Section 2. We consider the dual lattice M = N ∗ to N. We consider the compact torus T = iV /(2iπ N) and its complexiﬁcation TC = VC /(2iπN). The projection map VC → TC is denoted by the exponential notation v → ev . If {e1 , e2 , . . . , er } is a Z-basis of N, we write an element of VC as z = z1 e1 + z2 e2 + · · · + zr er with zj ∈ C. We can identify TC with C∗ × C∗ × · · · × C∗ by z → (ez1 , ez2 , . . . , ezr ). If m ∈ M, we denote by em the character of T deﬁned by em , ev = em,v . We extend em to a holomorphic character of the complex torus TC . The ring of holomorphic functions on TC generated by the functions em is denoted by R(T ). A quotient of two elements of R(T ) is called a rational function on the complex torus TC . Via the exponential map VC → TC , a function on TC is sometimes identiﬁed with a function on VC , invariant under translation by the lattice 2iπN. If {e1 , e2 , . . . , er } is a Z-basis of N, a rational function on TC written in exponential coordinates is a rational function of ez1 , ez2 , . . . , ezr . We brieﬂy say that it is a rational function of ez . Let us consider a ﬁnite set of nontrivial characters of T . We identify with a subset of M; for α ∈ , we denote by eα the corresponding character of TC . Deﬁnition 12. We denote by R(T ) the subring of rational functions on T generated by R(T ) and the inverses of the functions 1 − e−α with α ∈ . Observe that R is left unchanged when each element of is replaced by a nonzero scalar multiple, but that R(T ) strictly increases when (say) each α ∈ is replaced by 2α. We assume from now on that all elements of are indivisible in the lattice M. Via the exponential map, we consider elements of R(T ) as periodic meromorphic functions on VC . On VC , the function α, z 1 − e−α,z is deﬁned at z = 0, so it is an element of ᏻ. Writing α, z 1 1 = , −α,z 1−e α, z 1 − e−α,z we see that R(T ) is contained in ᏻ . We see furthermore from the formula 1 −e−z 1 d = = dz 1 − e−z (1 − ez )(1 − e−z ) (1 − e−z )2 that R(T ) ⊂ ᏻ is stable under differentiation.

290

BRION AND VERGNE

Our aim is to ﬁnd a natural map from R to R(T ) commuting with the action of differential operators with constant coefﬁcients. In particular, we want to force a rational function of z ∈ VC to become periodic, so that it is natural to deﬁne the Eisenstein series E(f )(z) = f (z + 2iπn). n∈N

We need to be more careful, as the sum is usually not convergent for an arbitrary f ∈ R . We introduce an oscillating factor et,2iπn with t ∈ V ∗ in front of each term of this inﬁnite sum. Let U = z ∈ VC , α, z + 2iπn = 0 for all n ∈ N and for all α ∈ . Then R(T ) consists of periodic holomorphic functions on U . Let f ∈ R ; then f (z + 2iπ n) is deﬁned for each n ∈ N if z ∈ U . For z ∈ U , we consider the function on V ∗ deﬁned by t −→ et,z+2iπn f (z + 2iπn). n∈N

If n → f (z + 2iπ n) is sufﬁciently decreasing at inﬁnity, the series is absolutely convergent and sums up to a continuous function of t with value at t = 0 equal to f (z + 2iπn). n∈N

In any case, it is easy to see that this series of functions of t converges to a generalized function of t. Proposition 13. For each f ∈ R and z ∈ U , the function on V ∗ deﬁned by t −→ et,z+2iπn f (z + 2iπn) n∈N

is well deﬁned as a generalized function of t, which depends holomorphically on z for z in the open set U . Proof. Indeed, if s(t) is a smooth function on V ∗ with compact support, consider the series f (z + 2iπ n) et,z+2iπn s(t) dt = c(z, n)f (z + 2iπn). n∈N

V∗

n∈N

The coefﬁcient c(z, n) =

V∗

e2iπt,n et,z s(t) dt

is rapidly decreasing in n, as the function t → et,z s(t) is smooth and compactly supported. Thus, c(z, n)f (z + 2iπn) is also a rapidly decreasing function of n.

ARRANGEMENT OF HYPERPLANES, II

291

Furthermore, c(z, n)f (z + 2iπn) depends holomorphically on z ∈ U . So the result of the summation c(z, n)f (z + 2iπn) n∈N

exists and is a holomorphic function of z. We write E(f )(t, z) =

et,z+2iπn f (z + 2iπn)

n∈N

for this generalized function of t depending holomorphically on z. We analyze this function of (t, z), t ∈ V ∗ , z ∈ U . We ﬁrst summarize some of the obvious properties of E(f )(t, z). Proposition 14. The following equations are satisﬁed. (1) For every P ∈ S(V ∗ ) and f ∈ R , E(Pf )(t, z) = P (∂t )E(f )(t, z). (2) For every v ∈ V and f ∈ R ,

E ∂(v)f (t, z) = ∂z (v)E(f )(t, z) − t, vE(f )(t, z). (3) For every m ∈ M and z ∈ U , E(f )(t + m, z) = em,z E(f )(t, z). As R is generated by S under the action of S(V ) and S(V ∗ ), we see that the operator E is completely determined by the functions E(φσ )(t, z) (σ ∈ Ꮾ()). A wall of is a hyperplane of V ∗ generated by r − 1 linearly independent vectors of . We consider the system of afﬁne hyperplanes generated by the walls of ∗ together with their translates by M (the dual lattice of N ). We denote by V,areg the complement of the union of these afﬁne hyperplanes. A connected component of ∗ V,areg is called an alcove and is denoted by a. ∗ Proposition 15. The function E(f )(t, z) is smooth when t varies on V,areg and when z ∈ U . More precisely, let a be an alcove. Assume that f is homogeneous of degree d. Then, on the open set a × U , the function E(f )(t, z) is a polynomial in t of degree at most −d − r, with coefﬁcients in R(T ) . ∗ is R − Z. Let [t] be Proof. Consider ﬁrst the one-variable case. The set V,areg the integral part of t. Fix z ∈ C − 2iπZ. Consider the locally constant function of t ∈ R − Z deﬁned by e[t]z t −→ . 1 − e−z We extend this function as a locally L1 -function on R (deﬁned except on the set Z of measure zero).

292

BRION AND VERGNE

Lemma 16. We have the equality of generalized functions of t: et (z+2iπn) n∈Z

z + 2iπn

=

e[t]z . 1 − e−z

Proof. We compute the derivative in t of the left-hand side. It is equal to et (z+2iπn) = etz δZ (t), n∈Z

where δZ is the delta function of the set of integers. We compute the derivative in t of the right-hand side. This function of t is constant on each interval (n, n + 1). The jump at the integer n is enz e(n−1)z − = enz . 1 − e−z 1 − e−z It follows that the derivative in t of the right-hand side is also equal to etz δZ (t). Thus, et (z+2iπn) n∈Z

z + 2iπn

= c(z) +

e[t]z , 1 − e−z

where c(z) is a constant. We verify that c(z) is equal to zero by using periodicity properties in t. It is clear that e−tz

et (z+2iπn) n∈Z

z + 2iπn

=

e2iπnt z + 2iπn n∈Z

is a periodic function of t as is e−tz

e([t]−t)z e[t]z = . 1 − e−z 1 − e−z

It follows that e−tz c(z) is also a periodic function of t. This implies c(z) = 0. Consider now, for k ∈ Z, Ek (t, z) =

et (z+2iπn) (z + 2iπn)k .

n∈Z

We just saw that

e[t]z . 1 − e−z To determine Ek (t, z) for k ≤ −1, we use the differential equation in z, E−1 (t, z) =

∂z Ek (t, z) = tEk (t, z) + kEk−1 (t, z).

293

ARRANGEMENT OF HYPERPLANES, II

Using decreasing induction over k, we see that Ek (t, z) is an L1 -function of t, equal to a polynomial function of t of degree −k − 1 on each interval (n, n + 1) and with rational functions of ez as coefﬁcients. For example, we obtain the value of the convergent series et (z+2iπn) e[t]z e[t]z . = (t − [t]) − 1 − e−z (1 − e−z )(1 − ez ) (z + 2iπ n)2 n When k ≥ 0, we use the differential equation ∂t Ek (t, z) = Ek+1 (t, z) so that, as we have already used, E0 (t, z) =

et (z+2iπn) = etz δZ (t).

n∈Z

More generally, Ek (t, z) = (∂t )k (etz δZ (t)) is supported on Z; in particular, it is identically zero on R − Z. We return to the proof of Proposition 15. For a simple fraction φ, consider the function t −→ E(φ)(t, z). We ﬁrst prove that it is a locally L1 -function, which is constant when t varies in an alcove. Let σ = {α1 , α2 , . . . , αr } be a basis of . Let t ∈ V ∗ . If t = j tj αj is the decom position of t on the basis σ , set [t]σ = j [tj ]αj . The function t → [t]σ is constant when t varies in an alcove. Consider the sublattice Z α ⊆ M. Mσ = α∈σ

We say that σ is a Z-basis if Mσ = M. In general, the quotient M/Mσ is a ﬁnite set; let ᏾ be a set of representatives of this quotient. We can choose ᏾ in the following standard way. We consider the box Qσ = [0, 1)α = u ∈ V ∗ , [u]σ = 0 . α∈σ

Then we can take

᏾ = Qσ ∩ M = u ∈ M, [u]σ = 0 .

Deﬁne

᏾(t, σ ) = (t − Qσ ) ∩ M = u ∈ M, [t − u]σ = 0 .

The set ᏾(t, σ ) is also a set of representatives of M/Mσ . If σ is a Z-basis of M, this set is reduced to the single element [t]σ . Remark that the set ᏾(t, σ ) is constant when t varies in an alcove a. We denote it by ᏾(a, σ ).

294

BRION AND VERGNE

Deﬁnition 17. If a is an alcove and if σ is a basis of , we set M −1 em m∈᏾(a,σ ) . −α Mσ α∈σ (1 − e )

Fσa =

Thus, an alcove a together with a basis σ ∈ Ꮾ() produces a particular element Fσa of R(T ) . ∗ Consider on the set V,areg the locally constant function of t deﬁned by Fσ (t, z) = a Fσ (z) when t is in the alcove a. This deﬁnes a locally L1 -function of t, still denoted ∗ by Fσ (t, z), deﬁned except on the set V ∗ − V,areg of measure zero. This locally 1 L -function of t deﬁnes a generalized function of t which depends holomorphically on z. Lemma 18. We have the equality of generalized functions of t ∈ V ∗ : E(φσ )(t, z) = Fσ (t, z). Proof. If σ is a Z-basis of M, this follows from the formula in dimension 1. In general, we consider Mσ ⊆ M and the dual lattice Nσ = Mσ∗ . Then N ⊆ Nσ . We set Eσ (φσ )(t, z) :=

et,z+2iπ7 φσ (z + 2iπ7).

7∈Nσ

For any set of representatives ᏾ of M/Mσ , we have u∈᏾ e−u,2iπ7 = 0 if 7 ∈ Nσ is not in N, while this sum equals |M/Mσ | if n ∈ N. Thus, E(φσ )(t, z) =

φσ (z + 2iπn)et,z+2iπn

n∈N

=

7∈Nσ

φσ (z + 2iπ7)e

t,z+2iπ7

M M

σ

−1 −u,2iπ7 e u∈᏾

M = M

−1 φσ (z + 2iπ7)et−u,z+2iπ7 eu,z

M = M

−1 u,z e Eσ (φσ )(t − u, z).

σ

σ

u∈᏾ 7∈Nσ

u∈᏾

This holds as an equality of generalized functions of t. Further, we have the following, by the 1-dimensional case: Eσ (φσ )(t, z) =

e[t]σ ,z

. −α,z α∈σ 1 − e

ARRANGEMENT OF HYPERPLANES, II

295

It follows that E(φσ )(t, z) is a locally L1 -function of t, as is Eσ (φσ ). It remains to determine the value of this function when t is in an alcove. For m ∈ Mσ , we have Eσ (φσ )(t + m, z) = em,z Eσ (φσ )(t, z), so that the sum u∈᏾ eu,z Eσ (φσ )(t − u, z) is independent of the choice of the system of representatives ᏾ of M/Mσ . We choose ᏾ = ᏾(t, σ ). Then u,z M −1 u∈᏾(t,σ ) e

E(φσ )(t, z) = −α,z Mσ α∈σ 1 − e because [t − u]σ = 0 for all u ∈ ᏾(t, σ ). Every function f ∈ R , homogeneous of degree d, is obtained from an element of S by the action of a differential operator with polynomial coefﬁcients. This operator is of degree d + r, if multiplication by zj is given degree 1, while derivation ∂/∂zj is given degree −1. Using Proposition 14, we see that Proposition 15 follows from the fact that the function t → E(φσ )(t, z) is constant on each alcove. a (z) ∈ R(T ) such that From Proposition 15, we see that there exist functions φ(k) we have the equality for t in the alcove a: a E(f )(t, z) = et,z+2iπn f (z + 2iπn) = t (k) φ(k) (z), n∈N

(k)

where the sum is over a ﬁnite number of multi-indices (k). This deﬁnes an operator E t : R −→ R(T ) ,

f −→ E(f )(t, z)

obtained by ﬁxing the regular value t. The operator E t satisﬁes the following relation, which is just relation (2) in Proposition 14: For v ∈ V and f ∈ R ,

E t ∂(v)f (z) = ∂z (v)E t (f )(z) − t, vE t (f )(z). Let B be a basis of Ꮾ(). Let (φσ , σ ∈ B) be the corresponding basis of S , and ∗ . For σ ∈ B and an alcove a, consider the let (φ σ , σ ∈ B) be the dual basis of S a element Fσ of R(T ) ⊂ ᏻ associated to σ, a. We obtain a kernel formula for the operator E t . Theorem 19. Let f ∈ G . For y ∈ U and t ∈ a, we have

E t (f )(y) = Tr S Res m et,· f C ᐀(y)E t Res =

σ ∈B

φ σ , Res et,z f (z)Fσa(y − z) ,

296

BRION AND VERGNE

where Fσa is given by Deﬁnition 17. Moreover, if B is the underlying basis of a diagonal basis OB, then

E t (f )(y) = Resoσ et,z f (z)Fσa(y − z) . oσ ∈OB

Proof. By a method entirely similar to the proof of Theorem 1, we see that the operator

φ σ , Res et,z f (z)Fσa(y − z) At (f )(y) = σ ∈B

satisﬁes the relation

At ∂(v)f (z) = ∂z (v)At (f )(z) − t, vAt (f )(z)

for v ∈ V , f ∈ R . Thus, to prove that E t = At on G , it is sufﬁcient to prove that they coincide for f = φτ . In this case, we obtain At (φτ )(y) = φ σ , φτ (z) Fσa(y) = Fτa(y) = E t (φτ )(y). σ ∈B

In view of the kernel formula for the Eisenstein series E t , it is natural to introduce the following deﬁnition. Deﬁnition 20. The constant term of the Eisenstein series E t is the linear form f → CT(f )(t) deﬁned for f ∈ R and t in the alcove a by

CT(f )(t) = Tr S Res m et,· f CE t Res . More explicitly, if OB is a diagonal basis of Ꮾ(), then

CT(f )(t) = Resoσ et,z f (z)Fσa(−z) . oσ ∈OB

4. Partial Eisenstein series. Let Nreg = N ∩ Vreg be the set of regular elements of N. The aim of this section is to prove that the function ENreg (f )(t, z) = et,z+2iπn f (z + 2iπn) n∈Nreg

is analytic in (t, z) when t is in an alcove and z ∈ VC is close to zero. In the next section we prove the Szenes residue formula for ENreg (f )(t, 0) = et,2iπn f (2iπn). n∈Nreg

Let 8 be a subset of N . We can deﬁne, for f ∈ R , the generalized function of t, E8 (f )(t, z) = et,z+2iπn f (z + 2iπn). n∈8

ARRANGEMENT OF HYPERPLANES, II

297

Introduce the set

U,8 = z ∈ VC , α, z + 2iπn = 0 for all α ∈ and n ∈ 8 .

The generalized function E8 (f )(t, z) depends holomorphically on z, when z ∈ U,8 . Let W be a rational subspace of V . Then N ∩ W is a lattice in W . Consider, for f ∈ R , et,z+2iπn f (z + 2iπn). EN∩W (f )(t, z) = n∈N∩W

We analyze the singularities in (t, z) of EN∩W (f )(t, z). If W is zero, then E{0} (f )(t, z) = et,z f (z) is analytic in (t, z) when z is regular in VC . Assume that W is nonzero and consider the subspace W ⊥ of V ∗ . Notice that if u ∈ M +W ⊥ , we have the relation EN∩W (f )(t + u, z) = eu,z EN∩W (f )(t, z). It is clear that the singular set of EN∩W (f )(t, z) is stable by translation by M + W ⊥ . Deﬁne a (W, )-wall in V ∗ as a hyperplane generated by W ⊥ together with dim W −1 ∗ vectors of . We introduce the set ᏴW,,M consisting of the union of all (W, )-walls ∗ as the complement of and of their translates by elements of M. We deﬁne VW,,areg ∗ ∗ ∗ ᏴW,,M in V . This set VW,,areg is invariant by translation by M + W ⊥ . Lemma 21. For f ∈ R , the function EN∩W (f )(t, z) is analytic in (t, z) when t ∗ ∗ varies on VW,,areg and z ∈ U,N∩W . Furthermore, if t ∈ VW,,areg and z is near zero, the function z → EN∩W (f )(t, z) deﬁnes an element of ᏻ . Proof. Let σ be a basis of . Although we are not able to give a nice formula for the function EN∩W (φσ )(t, z), we can still obtain an inductive expression that sufﬁces ∗ to give some information on it. Consider the set VW,σ,areg , that is, the complement of the union of (W, σ )-walls together with their translates by M. Let Uσ,N∩W be the set of all z ∈ VC such that α, z+2iπn = 0 for all α ∈ σ and n ∈ N ∩W . The intersection of this set with a small neighborhood of zero is contained in the complement of the union of the complex hyperplanes {z ∈ VC , α, z = 0}, for α ∈ σ . ∗ and z ∈ Lemma 22. The function EN∩W (φσ )(t, z) is analytic in t ∈ VW,σ,areg ∗ Uσ,N∩W . Furthermore, when t ∈ VW,σ,areg , the function α, z EN∩W (φσ )(t, z) z −→ α∈σ

is holomorphic at z = 0. We prove this by induction on the codimension of W . If W = V , this follows from the explicit formula for E(φσ )(t, z). Let α be an indivisible element of M such that W is contained in the real hyperplane Hα = y ∈ V , α, y = 0 .

298

BRION AND VERGNE

We assume ﬁrst that α is an element of σ . We number it the ﬁrst vector α1 of the basis σ . We set σ = (α2 , . . . , αr ), z = (z2 , . . . , zr ), and so on; then z = (z1 , z ). Our subspace W is contained in V = V ∩ {z1 = 0}. Thus, we have et1 z1 et,z+2iπn φσ (z + 2iπn) = EN ∩W (φσ )(t , z ). EN∩W (φσ )(t, z) = z1

n∈N∩W By induction, EN ∩W (φσ )(t , z ) is analytic in (t , z ) for z ∈ Uσ ,N , except if there exist m ∈ M such that t + m is in a hyperplane generated by W ⊥ (the orthogonal

of W in V ) and some vectors of σ . As W ⊥ = W ⊥ ⊕ Rα1 , we see that the singular ∗ set of EN∩W (φσ )(t, z) is contained in ᏴW,σ,M . Furthermore, the function z1 z2 · · · zr EN∩W (φσ )(t, z) = et1 z1 z2 · · · zr EN ∩W (φσ )(t , z ) is holomorphic in z near z = 0. Assume now that α is not an element of σ . We add it to the system if α is not an element of . Writing α = j cj αj , we obtain one of the Orlik-Solomon relations of the system ∪ {α}, cj φ σ j , φσ = j

σj

(W, σ j )-wall

= σ ∪ {α} − {αj }. A is a hyperplane of V ∗ generated by W ⊥ where j and dim W −1 vectors of σ ; then these vectors are distinct from α, because α ∈ W ⊥ . Thus, all W -walls for the basis σ j are also W -walls for the basis σ . By our ﬁrst calculation, it follows that EN∩W (φσ j )(t, z) is analytic when t is not on a translate of a (W, σ )-wall. Moreover, we have EN∩W (φσ )(t, z) = cj EN∩W (φσ j )(t, z), j

so that the function

 z −→ α, z 

r

 αj , z EN∩W (φσ )(t, z)

j =1

is holomorphic in z in a neighborhood of zero. By the induction hypothesis applied to W ⊆ V = {α = 0}, the function z → EN∩W (φσ )(t, z) is holomorphic on a nonempty open subset of VC . So this function, considered as a function of z ∈ VC , has no pole along α = 0. This proves Lemma 22 and, hence, Lemma 21 when f is a simple fraction. The operator EN∩W satisﬁes also the commutation relation of Proposition 14. Thus, using differential operators with polynomial coefﬁcients, we obtain the statement of Lemma 21 when f is any element in R . Let I be a subset of , and let WI = ∩i∈I Hαi . This is a rational subspace of V , and the (WI , )-walls are some of the walls of . Then it follows from Lemma 21 ∗ and z ∈ U . that EN∩WI (f )(t, z) is a fortiori analytic when t ∈ Vareg

ARRANGEMENT OF HYPERPLANES, II

299

Deﬁnition 23. A subset 8 of N is admissible if the characteristic function of 8 is a linear combination of characteristic functions of sets N ∩ WI , where I ranges over subsets of . Then we have the following by Lemma 21. Lemma 24. If 8 is an admissible subset of N, the function (t, z) → E8 (f )(t, z) ∗ is analytic when t ∈ V,areg and z ∈ U,8 . Furthermore, when z is near zero and ∗ t ∈ V,areg , the function z → E8 (f )(t, z) deﬁnes an element of ᏻ . If 8 is an admissible subset of N, we can take the value at t of the generalized function E8 (f )(t, z) = et,z+2iπn f (z + 2iπn) n∈8

provided that t is in an alcove a. Thus, for t ∈ a, we can deﬁne the operator E8t : R → ᏻ , f → E8 (f )(t, z). Now the argument of Theorem 19 proves the following proposition. ∗ Proposition 25. For f ∈ G , t ∈ V,areg , and y ∈ U,8 , we have

E8t (f )(y) = Tr S Res m et,· f C ᐀(y)E8t Res . More explicitly, if we choose a diagonal basis OB, then

t E8t (f )(y) = Resoσ f (z)et,z F8,σ (y − z) , oσ ∈OB

t (z) = E (φ )(t, z). where F8,σ 8 σ

5. Witten series and the Szenes formula. For f ∈ R , let us form the series Z(f )(t, z) = et,z+2iπn f (z + 2iπn), n∈Nreg

where Nreg is the set of regular elements of N. Then Z(f )(t, z) is deﬁned as a generalized function of t. As n varies in Nreg , this generalized function of t depends holomorphically on z when z varies in a neighborhood of zero. As Nreg is an admissible subset of N, we obtain the following from Lemma 24. Proposition 26. For any alcove a, Z(f )(t, z) is an analytic function of (t, z) when t ∈ a and z is in a neighborhood of zero. We have Z(f )(t, 0) =

n∈Nreg

et,2iπn f (2iπn).

300

BRION AND VERGNE

This is well deﬁned as a generalized function of t when t is in an alcove. If n → f (2iπ n) is sufﬁciently decreasing, then Z(f )(t, 0) is a continuous function of t; it generalizes the Bernoulli polynomial Bk (t) =

e2iπnt , (2iπn)k n=0

where 0 < t < 1. We reformulate the Szenes formula as an equality between Z(f )(t, 0) and the constant term of the Eisenstein series E(f )(t, z). Theorem 27. For any f ∈ R and t in an alcove a, we have

Z(f )(t, 0) = CT(f )(t) = Tr S Res m et,· f CE t Res . In particular, Z(f )(t, 0) is a polynomial function of t when t varies in an alcove a. As a consequence, if OB is a diagonal basis, then we recover the following residue formula (see [3, Theorem 4.4]):

et,2iπn f (2iπn) = Resoσ et,z f (z)Fσa(−z) . oσ ∈OB

n∈Nreg

Thus, when f = k

1

j =1 αj

is sufﬁciently decreasing, this formula expresses the series n∈Zr ,αj ,n=0

k

1

j =1 αj , 2iπn

as an explicit rational number. Proof. From the deﬁnitions of Z(f )(t, z) and CT(f )(t), we obtain, for any P ∈ S(V ∗ ), P (∂t )Z(f )(t, 0) = Z(Pf )(t, 0),

P (∂t ) CT(f )(t) = CT(Pf )(t).

Thus, it is enough to prove that Z(f )(t, 0) = CT(f )(t) for f ∈ G , because G generates R as a S(V ∗ )-module by Lemma 6. For t in an alcove a, we can deﬁne the operator Z t : R → ᏻ by Z t (f )(z) = et,z+2iπn f (z + 2iπn). n∈Nreg

ARRANGEMENT OF HYPERPLANES, II

301

The kernel formula holds for the operator Z t . In particular, we obtain, for f ∈ G ,

Z t (f )(0) = Tr S Res m et,· f CZ t Res . We thus need to prove that, for f ∈ G ,

Tr S Res m et,· f C E t − Z t Res = 0. But E t is given by a sum over the full lattice N, while Z t is only over the regular elements of N. Thus, we can write (in many ways) E t − Z t as a linear combination of operators E8t α , where each 8α is an admissible subset of N contained in the real hyperplane Hα . Now the Szenes formula follows from the next proposition. Proposition 28. Let 8 be an admissible subset of N contained in the real hyperplane Hα . Then, for f ∈ G ,

Tr S Res m et,· f CE8t Res = 0. Proof. It sufﬁces to prove that

Resoσ et,z f (z)E8t (φσ )(−z) = 0 oσ ∈OB

for some diagonal basis OB. A total order on provides us with a special diagonal basis OB of OᏮ() (see, for example, [1, Proposition 14]). We choose this order such that α is minimal. In this case, every element of OB is of the form oσ = (α1 , α2 , . . . , αr ) with α1 = α. We claim that for each oσ ∈ OB,

Resoσ et,z f (z)E8t (φσ )(−z) = 0. Indeed, we use the notation of Lemma 11 and write V = Hα . Then our set 8 is contained in V . Thus, et1 z1 et ,z +2iπγ . E8t (φσ )(z1 , z ) = r z1 j =2 αj , z + 2iπγ γ ∈8

We see that for t ﬁxed and regular, et,z f (z)E8t (φσ )(−z) =

1 f (z1 , z )ψ(z ), z1

where f ∈ G and ψ(z ) has poles at most on the complex hyperplanes αj = 0 for j = 2, . . . , r. Thus the claim follows from Lemma 11. Therefore, both Theorem 27 and Proposition 28 are proved. References [1] [2]

M. Brion and M. Vergne, Arrangement of hyperplanes, I: Rational functions and JeffreyKirwan residue, Ann. Sci. École Norm. Sup. (4) 32 (1999), 715–741. L. Jeffrey and F. Kirwan, Intersection theory on moduli spaces of holomorphic bundles of arbitrary rank on a Riemann surface, Ann. of Math. (2) 148 (1998), 109–196.

302 [3] [4] [5]

BRION AND VERGNE A. Szenes, Iterated residues and multiple Bernoulli polynomials, Internat. Math. Res. Notices 1998, 937–956. E. Witten, On quantum gauge theories in two dimensions, Comm. Math. Phys. 141 (1991), 153–209. D. Zagier, “Values of zeta functions and their applications” in First European Congress of Mathematics (Paris, 1992), Vol. II, Progr. Math. 120, Birkhäuser, Basel, 1994, 497–512.

Brion: Institut Fourier, Boîte Postale 74, F-38402 Saint-Martin d’Hères Cedex, France Vergne: Centre de Mathématiques, École Polytechnique, F-91128 Palaiseau Cedex, France

Vol. 103, No. 2

DUKE MATHEMATICAL JOURNAL

© 2000

NEW COMPLEX- AND QUATERNION-HYPERBOLIC REFLECTION GROUPS DANIEL ALLCOCK

To my father, John Allcock, 1940–1991 1. Introduction. In this paper, we carry out complex and quaternionic analogues of some of Vinberg’s extensive study of reﬂection groups on real hyperbolic space. In [25] and [26], Vinberg investigated the symmetry groups of the integral quadratic forms diag[−1, +1, . . . , +1] or, equivalently, the Lorentzian lattices In,1 . He was able to describe these groups very concretely for n ≤ 17, and extensions of his work by Borcherds [7] and with Kaplinskaja [27] provide similar descriptions for all n ≤ 23. In particular, the subgroup of Aut In,1 generated by reﬂections has ﬁnite index just when n ≤ 19. In this paper, we study the symmetry groups of Lorentzian lattices over the rings Ᏻ and Ᏹ of Gaussian and Eisenstein integers and the ring Ᏼ of Hurwitz integers (a discrete subring of the skew ﬁeld H of quaternions). Most of the paper is devoted to the most natural of such lattices, the self-dual ones. The symmetry groups of these lattices provide a large number of discrete groups generated by reﬂections and acting with ﬁnite-volume quotient on the hyperbolic spaces CH n and HH n . We construct a total of 19 such groups, including groups acting on CH 7 and HH 5 . At least one of our groups has been discovered before, in the work of Deligne and Mostow [18], Mostow [22], and Thurston [24], but our largest examples are new. To the author’s knowledge, quaternion-hyperbolic reﬂection groups have not been studied before. Our results and techniques have found important application in work by the author, Carlson, and Toledo on the moduli space of complex cubic surfaces [3], [4]. Namely, this space is isomorphic to the Satake compactiﬁcation of the quotient of CH 4 by one of the reﬂection groups studied here. Furthermore, the moduli space of “marked” cubic surfaces may be realized as the Satake compactiﬁcation of the quotient of CH 4 by a congruence subgroup, which is also a reﬂection group in its own right. The techniques used by Vinberg and others for the real hyperbolic case rely heavily on the fact that if a discrete group G is generated by reﬂections of RH n , then the mirrors of the reﬂections of G chop RH n into pieces, and each piece may be taken as a fundamental domain for G. Work with complex or quaternionic reﬂection groups is much more complicated since hyperplanes have real codimension 2 or 4, and so the mirrors fail to chop hyperbolic space into pieces. Our solution to this problem is to Received 11 June 1999. Revision received 8 November 1999. 2000 Mathematics Subject Classiﬁcation. Primary 22E40; Secondary 11F06, 11E39, 53C35. Author’s work supported by National Science Foundation graduate and postdoctoral fellowships. 303

304

DANIEL ALLCOCK

avoid fundamental domains altogether. Each of our groups is deﬁned as the subgroup Reﬂec L of Aut L generated by reﬂections, where L is a Lorentzian lattice over Ᏻ, Ᏹ, or Ᏼ. (A Lorentzian lattice is a free module equipped with a Hermitian form of signature − + · · · +.) Since Aut L is an arithmetic group, to show that the quotient of CH n or HH n by Reﬂec L has ﬁnite volume, it sufﬁces to show that Reﬂec L has ﬁnite index in Aut L. In this case, we say that L is reﬂective. Our basic strategy for proving a lattice L to be reﬂective is ﬁrst to prove that Reﬂec L acts with only ﬁnitely many orbits on the vectors of L of norm 0, and second to prove that the stabilizer in Reﬂec L of one such vector has ﬁnite index in the stabilizer in Aut L. We work mostly arithmetically, avoiding use of such tools as the bisectors introduced by Mostow for his study of reﬂection groups on CH 2 [21]. However, there are certain steps in our constructions where geometric ideas play a key role. We express each of our Lorentzian lattices L in the form ⊕II1,1 , where is positive-deﬁnite and II1,1 is a certain 2-dimensional lattice, the hyperbolic plane, with inner product matrix 01 01 . It turns out that this description of L allows us to easily write down a large collection of reﬂections of L, parameterized by (a central extension of) the lattice . It turns out that if has enough vectors of norms 1 and 2 and provides a good covering of Euclidean space by balls, then we can automatically deduce that L is reﬂective. This implication is the content of Theorem 6.1. The rest of Section 6 is devoted to the application of this theorem and related ideas in the study of various examples. In particular, we prove that each of the self-dual Lorentzian lattices Ᏹ In,1

n = 1, 2, 3, 4, 7,

Ᏻ I In,1 Ᏼ In,1

n = 1, 5,

n = 1, 2, 3, 5,

is reﬂective. (These lattices are deﬁned in Section 3 and characterized in Theorem 7.1.) For some of these, we obtain more detailed information. In particular, we prove that Ᏹ Ᏹ Ᏼ Reﬂec In,1 = Aut In,1 for n = 2, 3, 4, or 7 and that Reﬂec In,1 has index at most 4 in Ᏼ Aut In,1 for n = 2, 3, or 5. We also give explicit descriptions of the reﬂection groups Ᏹ Ᏻ and I I1,1 as subgroups of certain Coxeter groups acting on CH 1 ∼ of I1,1 = RH 2 and HH 1 ∼ = RH 4 . We note that the geometric ideas used here, namely, that good coverings of Euclidean space lead to hyperbolic reﬂection groups, apply even when C or H is replaced by the nonassociative ﬁeld O of octaves (or octonions or Cayley numbers). In [1] we constructed two octave reﬂection groups acting on OH 2 and one acting on OH 1 ∼ = RH 8 , and interpreted these groups as the stabilizers of lattices over a certain discrete subring of O. We provide background information on lattices in Section 2 and examples of them in Section 3; the latter should be referred to only as needed. Section 4 establishes our conventions regarding hyperbolic geometry. In Section 5, we relate certain geometric

NEW HYPERBOLIC REFLECTION GROUPS

305

properties of a positive-deﬁnite lattice to the reﬂection group of ⊕ II1,1 and lay the foundations for Section 6, where we construct all of our examples. In Section 7, we explain the correspondence between primitive isotropic sublattices of In+1,1 and positive-deﬁnite self-dual lattices of dimension n. We use this correspondence to provide a quick geometric proof of the classiﬁcation of self-dual lattices over Ᏹ and Ᏼ in dimensions less than or equal to 6 and 4, respectively. The only examples besides the lattices Ᏹn and Ᏼn are the Coxeter-Todd lattice Ᏹ6 and a quaternionic form Ᏼ 4 of the Barnes-Wall lattice. In Section 8, we show that our largest three groups, namely, Ᏹ Ᏹ Ᏼ Reﬂec L for L = I7,1 , I4,1 , and I I5,1 , are not among the 94 groups constructed in Ᏹ does appear among these [18], [22], and [24]. We also sketch a proof that Reﬂec I3,1 groups. Ᏻ Ᏻ = E8Ᏻ ⊕I I1,1 , which The easiest route to a new reﬂection group is our study of I I5,1 5 acts on CH . This requires only Lemmas 5.1 and 5.2 and Theorem 6.2. Most of our other examples require the more complicated Lemma 5.3 in place of Lemma 5.2. The Ᏹ Ᏼ arguments for I7,1 and I5,1 also require fairly involved space-covering arguments, involving embeddings of the Coxeter-Todd and Barnes-Wall lattices into the famous Leech lattice 24 . It is pleasing that 24 plays a role here, because our basic approach is inspired by Conway’s elegant description [10] of the isometry group of the Z-lattice II25,1 = 24 ⊕ II1,1 in terms of the combinatorics of 24 . The embeddings of the Coxeter-Todd and Barnes-Wall lattices into 24 have also been used by Borcherds [8] to produce interesting real hyperbolic reﬂection groups, acting on RH 13 and RH 17 . Finally, the Leech lattice plays a much more direct role in [2], which constructs several other complex and quaternionic hyperbolic reﬂection groups, including one on CH 13 and one on HH 7 . Most of this paper is derived from my Ph.D. thesis at the University of California, Berkeley. I would like to thank my dissertation advisor, R. Borcherds, for his interest and suggestions—in particular, for suggesting that the quaternionic Barnes-Wall lattice would provide a reﬂection group on HH 5 . 2. Lattices. We denote by ᏾ any one of the rings Ᏻ, Ᏹ, and Ᏼ—the Eisenstein, Gaussian, and Hurwitz integers. That is, Ᏻ = Z[i] and Ᏹ = Z[ω], where ω = (−1 + √ −3)/2 is a primitive cube root of unity. The ring Ᏼ is the integral span of its 24 units ±1, ±i, ±j, ±k, and (±1 ± i ± j ± k)/2 in the skew ﬁeld H of quaternions. We write K for the ﬁeld (C or H ) naturally containing ᏾. Conjugation x → x¯ denotes complex or quaternionic conjugation, as appropriate. For any element x of K, we write Re x = (x + x)/2 ¯ and Im x = (x − x)/2 ¯ for the real and imaginary parts of x, and we say that x is imaginary if Re x = 0. If X ⊆ K, then we write Im X for the set of imaginary elements of X. For any x ∈ K, x x¯ is a positive real number, and the absolute value √ |x| of x is deﬁned to be (x x) ¯ 1/2 . It is convenient to deﬁne the element θ = ω − ω¯ = −3 of Ᏹ. We sometimes also consider ω and θ as elements of Ᏼ, via the embedding Ᏹ → Ᏼ deﬁned by ω → (−1 + i + j + k)/2 or equally well by θ → i + j + k.

306

DANIEL ALLCOCK

A lattice over ᏾ is a free (right) module over ᏾ equipped with a Hermitian form, which is to say a Z-bilinear pairing (the inner product) · | · : × → K such that x | y = y | x

and

x | yα = x | yα

for all x, y ∈ and α ∈ ᏾. We use right modules and right-linear Hermitian forms so that lattice automorphisms can be described by matrices acting on the left. A Hermitian form on a (right) vector space over K is deﬁned similarly. Section 3 deﬁnes a number of interesting lattices and lists some of their properties. Sometimes we indicate that a lattice is an ᏾-lattice by writing ᏾ . If S ⊆ , then we denote by S ⊥ its orthogonal complement: those elements of whose inner products with all elements of S vanish. We say that is nonsingular if ⊥ = {0} and that is integral if for all x, y ∈ , the inner product x | y lies in ᏾. All lattices that we consider are integral and nonsingular unless otherwise speciﬁed. The dual ∗ of is the set of all ᏾-linear maps from to ᏾. An integral lattice is called self-dual if the natural (antilinear) map from to ∗ is onto. A self-dual lattice is sometimes called “unimodular,” because the matrix of inner products of any basis for has determinant ±1; we use “self-dual” to avoid discussing determinants of quaternionic matrices. The norm of a vector v ∈ V is deﬁned to be v 2 = v | v; some authors call this the squared norm of v, but our convention is better for indeﬁnite forms. We say that v is isotropic, or null, if v 2 = 0. A lattice is isotropic, or null, if each of its elements is. A lattice is called even if each of its elements have even norm and are odd otherwise. A sublattice of is called primitive if = ∩ ( ⊗ Q). A vector v of is called primitive if v = wα for w ∈ , and α ∈ ᏾ implies that α is a unit. Because the rings Ᏻ, Ᏹ, and Ᏼ are principal ideal domains, a nonzero vector is primitive if and only if its ᏾-span is primitive as a sublattice. We sometimes write v for the ᏾-span of v ∈ . We sometimes deﬁne an ᏾-lattice by describing a Hermitian form on ᏾n . We do this by giving an n × n matrix (φij ) with entries in ᏾ such that φij = φj i . Then the Hermitian form is given by n x¯i φij yj . (x1 , . . . , xn ) | (y1 , . . . , yn ) =

i,j =1

We may also view a lattice as a subset of the vector space V = ⊗ R over the ﬁeld K = R ⊗ ᏾. The Hermitian form on gives rise to one on V . If is nonsingular, then ∗ may be identiﬁed with the set of vectors in V having ᏾-integral inner product with each element of . Every nonsingular Hermitian form on a vector space V over K is equivalent under GL(V ) to one given by a diagonal matrix, with each diagonal entry being ±1. The signature of is the ordered pair (n, m) where n (resp., m) is the number of +1’s (resp., −1’s). This characterizes up to equivalence under GL(V ). We write Kn,m for the vector space Kn+m equipped with the standard Hermitian form of signature (n, m); the isometry group of Kn,m is the unitary group U (n, m; K). The

NEW HYPERBOLIC REFLECTION GROUPS

307

term “Lorentzian” is applied to various concepts in the study of real Minkowski space Rn,1 . By analogy with this, we call a lattice Lorentzian if its signature is (n, 1). Any isotropic sublattice of a Lorentzian lattice has dimension less than or equal to 1. If is positive-deﬁnite, then ⊗ R is a copy of Euclidean space under the metric d(x, y) = (x − y)2 . Points of ⊗ R at maximal distance from are called deep holes of . The maximal distance is called the covering radius of because closed balls of that radius placed at lattice points exactly cover ⊗ R. The lattice points nearest a deep hole are called the vertices√of the hole. √ The covering radii of the Z-lattices Im Ᏻ, Im Ᏹ, and Im Ᏼ are 1/2, 3/2, and 3/2, respectively. The ﬁrst two are obvious and the last follows because Im Ᏼ is the 3-dimensional cubic lattice spanned by i, j, and k. Any two deep holes of Im ᏾ are equivalent under translation by some element of Im ᏾. Suppose that V is a Hermitian vector space over K, ξ ∈ K is a root of unity, and v ∈ V has nonzero norm. We deﬁne the ξ -reﬂection in v to be the map v −→ v − r(1 − ξ )

r | v . r2

(2.1)

This is an isometry of the right vector space V , which ﬁxes r ⊥ pointwise and carries r to rξ . (Warning: If K = H, then although the reﬂection acts by right scalar multiplication on r, it does not act this way on the entire H-span of r. This is due to the noncommutativity of multiplication in H.) Unless otherwise speciﬁed, we use the term “reﬂection” to mean “reﬂection in a vector of positive norm.” Under the conventions of Section 4, a reﬂection in a negative norm vector acts on hyperbolic space with an isolated ﬁxed point rather than a hyperplane of ﬁxed points. This is why we focus on positive-norm vectors. We call r ⊥ the mirror of the reﬂection. Reﬂections of order 2, 3, . . . are sometimes called biﬂections, triﬂections, and so on. A ξ -reﬂection is a biﬂection only if ξ = −1; in this case, we recover the classical notion of a reﬂection. Suppose that L is an integral lattice. If v ∈ L has norm 1 (resp., 2), then we say that v is a short (resp., long) root of L. Inspection of (2.1) reveals that if ξ is a unit of ᏾, then ξ -reﬂection in any short root of L preserves L. Furthermore, biﬂections in long roots of L also preserve L. We deﬁne the reﬂection group Reﬂec L to be the subgroup of Aut L generated by reﬂections (in positive-norm vectors), and we say that L is reﬂective if Reﬂec L has ﬁnite index in Aut L. In general, a group generated by reﬂections is called a reﬂection group. Since Aut L is an arithmetic subgroup of the semisimple real Lie group U (L ⊗ R; K), a theorem of Borel and Harish-Chandra [9] implies that it has ﬁnite covolume. It follows that L is reﬂective if and only if Reﬂec L also has ﬁnite covolume. It may happen that Reﬂec L contains reﬂections other than those in its roots, but we do not use them. 3. Reference: Examples of lattices. This section contains background information on the various complex and quaternionic lattices we use; it should be referred to only as necessary. We brieﬂy deﬁne each lattice, list a few important properties,

308

DANIEL ALLCOCK

and give references to the literature. The main source is [15, Chapter 4]. All lattices n described here are integral. When lattices are described as subsets of K , it should be understood that the Hermitian form is (x1 , . . . , xn ) | (y1 , . . . , yn ) = x¯i yi . The simplest lattice is ᏾n , which is obviously self-dual. Its symmetry group contains the left-multiplication by each diagonal matrix, all of whose diagonal entries are units of ᏾. It is easy to see that the group of these coincides with the group generated by the reﬂections in the short roots. Adjoining to this group the permutations of coordinates, which are generated by biﬂections in long roots such as (1, −1, 0, . . . , 0), we see that Aut ᏾n is a reﬂection group. If is a lattice, then its real form is the Z-module equipped with the inner product (x, y) = Rex | y. Here are three forms of the E8 root lattice: 1 (x1 , . . . , x8 ) ∈ Z8 | xi ≡ xj (mod 2), xi ≡ 0 (mod 4) , 2 1 E8Ᏻ = (x1 , . . . , x4 ) ∈ Ᏻ4 | xi ≡ xj (mod 1 + i), xi ≡ 0 (mod 2) , 1+i

Ᏼ E8 = (x1 , x2 ) ∈ Ᏼ2 | x1 + x2 ≡ 0 (mod 1 + i) . E8 =

It is straightforward to identify the real forms of these lattices with each other; each has covering radius 1 and minimal norm 2. Often the dimension of a lattice is indicated by a subscript. Unfortunately, this sometimes refers to its dimension as a Z-lattice and sometimes to its dimension as an ᏾-lattice. There seems to be no universal solution to this problem. Some other useful even Gaussian lattices are Ᏻ = (x1 , . . . , xn ) ∈ Ᏻn D2n xi ≡ 0 (mod 1 + i) , whose real forms are the D2n root lattices. The D4 lattice is also the real form of Ᏼ, scaled up by a factor of 21/2 . The covering radius of D2n is (n/2)1/2 . The Eisenstein lattice

D3 (θ ) = (x, y, z) ∈ Ᏹ3 | x + y + z ≡ 0 (mod θ) √ is one of the lattices Dn ( −3) introduced by Feit in [19]. It has 54 long roots and 72 vectors of norm 3; biﬂections in the former and triﬂections in the latter preserve the lattice. Its covering radius is 1; this can be seen as follows. According to [15, p. 126], the real form of the lattice {(x, y, z) ∈ Ᏹ3 | x ≡ y ≡ z (mod θ)} is the E6 root lattice scaled up by (3/2)1/2 . This identiﬁcation can be used to show that the real form of D3 (θ ) is the real form of E6∗ scaled up by (3/2)1/2 , where E6∗ is the dual (over Z) of E6 . By [15, p. 127], the covering radius of E6∗ is (2/3)1/2 , so the covering radius of D3 (θ ) is 1. The Coxeter-Todd lattice Ᏹ6 is a self-dual Ᏹ-lattice that is spanned by its long roots, which are also its minimal vectors. It is discussed at length in [14]; we quote

309

NEW HYPERBOLIC REFLECTION GROUPS

just one of the deﬁnitions given there: Ᏹ6 =

1 (x1 , . . . , x6 ) ∈ Ᏹ | xi ≡ xj (mod θ), xi ≡ 0 (mod 3) . θ

Its automorphism group is the ﬁnite complex reﬂection group 6·U4 (3) : 2, and Ᏹ6 shares many interesting properties with E8 and the Leech lattice 24 . We refer to [14] for details. The quaternionic Barnes-Wall lattice is Ᏼ 4 =

1 (x1 , . . . , x4 ) ∈ Ᏼ | xi ≡ xj mod (1 + i)Ᏼ , xi ∈ 2 Ᏼ . 1+i

We may recognize the real form of 21/2 Ᏼ 4 by identifying the vector

a1 + b1 i + c1 j + d1 k, . . . , a4 + b4 i + c4 j + d4 k

with the vector in R16 whose coordinates we arrange in the square array a1 4 d1 √ 8 b1 c1

a2 c2 d2 b2

a3 d3 b3 c3

a4 c4 d4 b4

where the inner product is the usual one on R16 . This array may be taken to be, say, the left four columns of the 4×6 array in the miracle octad generator description (see [11, p. 97]) of the Leech lattice 24 , and then the real form of 21/2 Ᏼ 4 is visibly the real Barnes-Wall lattice BW 16 (see [15, Chapter 4]). Theorem 3.1. The lattice Ᏼ 4 is self-dual and spanned by its minimal vectors, which have norm 2. Its automorphism group is generated by the biﬂections in its Ᏼ minimal vectors. Each class of Ᏼ 4 modulo 4 (1 + i) is represented by a vector of Ᏼ 2 norm at most 3. The deep holes of 4 coincide with the set {λ(1+i)−1 | λ ∈ Ᏼ 4, λ ≡ 1 (mod 2)}. Proof. Proofs of all claims except the last appear in [5, Section 4.6]. Most of the rest of the work has been done for us by Conway and Sloane [13, Section 5]. They showed that the deep holes of BW 16 nearest 0 are the halves of certain vectors v ∈ BW 16 of norm 12, and further that such v are not congruent modulo 2 to minimal vectors of BW 16 . (They write 16 for BW 16 .) After rescaling, we ﬁnd that the deep Ᏼ holes of Ᏼ 4 nearest 0 are the halves of certain elements v of norm 6 in 4 . Since 2 each such v has even norm and is not congruent modulo 2 = −(1 + i) to any root, Ᏼ it must map to 0 in Ᏼ 4 /4 (1 + i). Therefore, v = λ(1 + i) for some λ of norm 3 in −1 Ᏼ 4 , and so the deep holes nearest 0 have the form v/2 = λ(1 + i)/2 = (λi)(1 + i) .

310

DANIEL ALLCOCK

The deep holes of Ᏼ 4 are the translates by lattice vectors of the deep holes nearest zero. That is, the set of deep holes coincides with the set

λ(1 + i)−1 | λ ∈ Ᏼ 4 is congruent modulo 1 + i to a norm 3 lattice vector . The norms of any two lattice vectors that are congruent modulo 1 + i have the same parity. Since each lattice vector is congruent to some vector of norm 0, 2, or 3, the set above coincides with the one in the statement of the theorem. ᏾ Now we describe some indeﬁnite self-dual lattices. The lattice In,m is the ᏾-module equipped with the inner product given by the diagonal matrix

᏾n+m

diag[+1, . . . , +1, −1, . . . , −1] ᏾ is the module ᏾2 with inner with n (resp., m) +1’s (resp., −1’s). The lattice II1,1 0 1 ᏾ ∼ ᏾ product matrix 1 0 . If ᏾ = Ᏹ or Ᏼ, then II1,1 = I1,1 because we can ﬁnd a norm 1 vector in the former lattice. If ᏾ = Ᏻ, then II1,1 is even, whereas I1,1 is odd. We Ᏻ to be the lattices deﬁne the Gaussian lattices I I4m+n,n Ᏻ Ᏻ Ᏻ II4m+n,n = E8Ᏻ ⊕ · · · ⊕ E8Ᏻ ⊕ I I1,1 ⊕ · · · ⊕ I I1,1 , Ᏻ where there are m summands E8Ᏻ and n summands I I1,1 . These lattices are even and self-dual. By Theorem 7.1, every indeﬁnite self-dual lattice over ᏾ appears among Ᏹ ∼ Ᏹ Ᏼ ∼ Ᏼ the examples just given. In particular, Ᏹ6 ⊕ I I1,1 = I7,1 and Ᏼ 4 ⊕ I I1,1 = I5,1 .

4. Hyperbolic space. The hyperbolic space KH n+1 (n ≥ 0) is deﬁned as the image in projective space KP n+1 of the set of vectors of negative norm in Kn+1,1 ; its boundary ∂KH n+1 is the image of the (nonzero) null vectors. We write elements of Kn+1,1 in the form (λ; µ, ν) with λ ∈ Kn,0 and µ, ν ∈ K, with inner product (λ1 ; µ1 , ν1 ) | (λ2 ; µ2 , ν2 ) = λ1 | λ2 + µ¯ 1 ν2 + ν¯ 1 µ2 . This corresponds to a decomposition Kn+1,1 ∼ = Kn,0 ⊕ 01 01 . We often refer to points in projective space by naming vectors in the underlying vector space. It is convenient to distinguish the isotropic vector (0; 0, 1) and give it the name ρ. Every point of KH n+1 ∪ ∂KH n+1 except ρ has a unique preimage in Kn+1,1 with inner product 1 with ρ, and so we may make the identiﬁcations

KH n+1 = (λ; 1, z) : λ ∈ Kn , λ2 + 2 Re(z) > 0 , (4.1)

∂KH n+1 {ρ} = (λ; 1, z) : λ ∈ Kn , λ2 + 2 Re(z) = 0 . We deﬁne the height of a vector v ∈ Kn+1,1 to be ht v = ρ | v. For v = (λ; µ, ν), this is simply µ. For vectors of any ﬁxed norm, the height function measures how

NEW HYPERBOLIC REFLECTION GROUPS

311

far away from ρ the corresponding points in projective space are—the smaller the height, the closer to ρ. We sometimes say that a vector v has height less than that of another vector v. By this we mean | ht v | < | ht v|. We say that the vector (λ; µ, ν) of height µ ! = 0 lies over λµ−1 ∈ Kn . It is obvious that all the scalar multiples of any given vector of nonzero height lie over the same point of Kn , so we may think of points in projective space (except for those in ρ ⊥ ) as lying over elements of Kn . The geometric content of this deﬁnition is that the lines in KP n+1 passing through ρ and meeting KH n+1 are in one-to-one correspondence with Kn . The points in the line associated to λ ∈ Kn are the scalar multiples of those of the form (λ; 1, z) with z ∈ K, which are precisely the points of KP n+1 lying over λ. We gave two special cases in (4.1). In particular, the family of height 1 isotropic vectors lying over λ is parameterized by the elements of Im K. This description of ∂KH n+1 {ρ} as a bundle over Kn with ﬁber Im K helps us relate the properties of lattices in Kn to properties of groups acting on KH n+1 . The subgroup of U (n + 1, 1; K) ﬁxing ρ contains transformations Tx,z (with x ∈ Kn , z ∈ Im K) deﬁned by ρ −→ ρ, Tx,z : (0; 1, 0) −→ x; 1, z − x 2 /2 , (λ; 0, 0) −→ λ; 0, −x | λ for each λ ∈ Kn .

(4.2)

(The map is deﬁned in terms of some unspeciﬁed but ﬁxed inner product on Kn .) We call these maps translations. If we regard elements of Kn+1,1 as column vectors, then Tx,z acts by multiplication on the left by the matrix 

In  0 −x ∗

 x 0 1 0 . z − x 2 /2 1

We have written x ∗ for the linear function y → x | y on Kn,0 deﬁned by x. We have the relations Tx,z ◦ Tx ,z = Tx+x ,z+z +Imx |x ,

(4.3)

−1 = T−x,−z , Tx,z

(4.4)

−1 ◦ Tx−1 Tx,z ,z ◦ Tx,z ◦ Tx ,z = T0,2 Imx |x ,

(4.5)

which are most easily veriﬁed in the order listed. These relations make it clear that the translations form a group and that its center and commutator subgroup coincide and consist of the T0,z . We call elements of this subgroup central translations. The translations form a (complex or quaternionic) Heisenberg group that acts freely and transitively on ∂KH n+1 {ρ}. If v ∈ Kn+1,1 lies over λ ∈ Kn , then Tx,z (v) lies over

312

DANIEL ALLCOCK

λ + x. That is, the translations act in the natural way (by translations) on the points of Kn over which vectors in Kn+1,1 lie. We note that these constructions all make sense when K = R, and even simplify. Since Im R = 0, the translations form an abelian group, which is just the obvious set of translations in the usual upper half-space model for RH n+1 . The obvious projection map from the upper half-space to Rn carries points of RH n+1 to the points of Rn over which they lie, in the sense deﬁned above. This is the source of the terminology. The simultaneous stabilizer of (0; 1, 0) and (0; 0, 1) is the unitary group U (n, 0; K), which ﬁxes pointwise the second summand of the decomposition Kn+1,1 = Kn,0 ⊕ K1,1 . If S is an element of this unitary group, then matrix computations reveal S ◦ Tx,z ◦ S −1 = TSx,z .

(4.6)

5. Reﬂections in Lorentzian lattices. The Lorentzian lattices we consider all have the form ⊕ II1,1 , where is a positive-deﬁnite ᏾-lattice and II1,1 is the 2dimensional self-dual lattice deﬁned by the matrix II1,1 = 01 01 . In general, we write L for a Lorentzian lattice ⊕ II1,1 , where and even ᏾ may be left unspeciﬁed, except that is always positive-deﬁnite. We write elements of L = ⊕ II1,1 in the form (λ; µ, ν) with λ ∈ and µ, ν ∈ ᏾. This embeds L in the description of Kn+1,1 given in Section 4 and allows us to transfer to L several important concepts deﬁned there. In particular, ρ = (0; 0, 1) is an element of L, and we deﬁne the height of elements of L as before. For v ∈ L of nonzero height, the point of ⊗ R over which v lies is in ⊗ Q, but not necessarily in . There are two basic ideas in this section. First, this description of L provides a way to write down a large collection of reﬂections of L, essentially parameterized by the elements of a discrete Heisenberg group of translations. The second idea is that if r is a root of L and v is a null vector in Kn+1,1 , and if r and v lie over points of Kn that are sufﬁciently close, then by applying a reﬂection of L, we can reduce the height of v. (This reﬂection might be in some root other than r.) Both of these ideas can be found in the simpler setting of real hyperbolic space in Conway’s study [10] of the automorphism group of the Lorentzian Z-lattice II25,1 = 24 ⊕II1,1 . Here 24 is the Leech lattice. Conway found a set of reﬂections permuted freely by a group of translations naturally isomorphic to the additive group of 24 . By using facts about the covering radius of 24 together with the second idea described above, he was able to prove that these reﬂections generate the entire reﬂection group of II25,1 . The major complication in transferring this approach to our setting is that the discrete group of translations is no longer a copy of but a central extension of by Im ᏾. This issue dramatically complicates the precise formulation (Table 1) of the second main idea. For example, it is complicated to state exactly what happens when we cannot quite reduce the height of v ∈ Kn+1,1 by using a reﬂection. We begin by ﬁnding the translations in Aut L and showing that under simple conditions, Reﬂec L contains a large number of them. The translation Tx,z preserves L

313

NEW HYPERBOLIC REFLECTION GROUPS

just if x ∈ and z − x 2 /2 ∈ ᏾. If ᏾ = Ᏹ or Ᏼ, then for any given x ∈ , we may choose z ∈ Im K such that Tx,z ∈ Aut L, by taking z = 0 or θ/2 according to whether x 2 is even or odd. If ᏾ = Ᏻ, then such a z exists if and only if x 2 is even; z may then be taken to be 0. The different rings behave differently because Ᏹ and Ᏼ contain elements with half-integral real parts, while Ᏻ does not. All the central translations T0,z with z ∈ ᏾ lie in Aut L; they ﬁx pointwise and act by isometries of II1,1 . The assertions of the next lemma are precise formulations of the idea that if Aut contains many reﬂections, then Reﬂec L contains many translations. Lemma 5.1. Let L = ⊕ II1,1 for some positive-deﬁnite ᏾-lattice . Deﬁne

0 = x ∈ | Tx,z ∈ Reﬂec L for some z ∈ Im K ,

᏿ = z ∈ Im ᏾ | T0,z ∈ Reﬂec L . (i) If ᏾ = Ᏹ or Ᏼ, then 0 contains the short roots of . (ii) If r is a long root of , then 2r ∈ 0 . Furthermore, if r has inner product 1 with some element of , then r itself lies in 0 . (iii) ᏿ contains the integral span of the elements 2 Imx | y with x, y ∈ 0 . (iv) If the roots of ! = {0} span up to ﬁnite index, then the stabilizer of ρ in Reﬂec L has ﬁnite index in the stabilizer in Aut L. Proof. Let R be a ξ -reﬂection of with mirror M. We regard R as acting on L, −1 ◦ R ◦ T ﬁxing the summand II1,1 pointwise. If Tx,z ∈ Aut L, then Tx,z x,z ∈ Reﬂec L. By (4.4), (4.6), and (4.3), −1 Tx,z ◦ R ◦ Tx,z ◦ R −1 = T−x,−z ◦ TRx,z = TRx−x,− ImRx|x ,

proving that Rx − x ∈ 0

(5.1)

for all x ∈ and all reﬂections R of . The geometric picture behind this computation −1 pass through ρ and are parallel there. We have is that both M and its translate by Tx,z constructed a translation out of reﬂections in two parallel mirrors. (i) If r is a short root of , then we let x = rω, and let R be the (−ω)-reﬂection in r. Then 0 contains Rx − x = r(−ω)ω − rω = r(−ω¯ − ω) = r. (ii) If r is a long root of , then we let x = −r, let R be the biﬂection in r, and observe that Rx −x = 2r. To prove the second claim, suppose x ∈ has inner product −1 with r and take R to be the biﬂection in r. Then Rx − x ∈ 0 is proportional to r and has inner product 2 with r, so it coincides with r. (iii) This follows immediately from (4.5) by taking commutators of translations of Reﬂec L. (iv) The null vectors of height 1 in L are exactly those vectors (λ; 1, z) with λ ∈ , z ∈ ᏾, and Re z = −λ2 /2, and the translations in Aut L permute them transitively. Since the simultaneous stabilizer of ρ and one of these, say, (0; 1, 0), is the ﬁnite group Aut , it sufﬁces to prove that the group of translations in Reﬂec L has ﬁnite

314

DANIEL ALLCOCK

index in the group of those in Aut L. This follows from (i)–(iii): 0 has ﬁnite index in and ᏿ has ﬁnite index in Im ᏾. Now we exhibit a large number of reﬂections of L, namely, the reﬂections in the roots of L of small height. It is straightforward to enumerate the roots of L of any given height h. For h = 1, we ﬁnd that these are the vectors: Norm 2 : (λ; 1, z), Norm 1 : (λ; 1, z),

(2 − λ2 ) , 2 (1 − λ2 ) Re z = , 2 Re z =

with λ ∈ and z ∈ ᏾. If ᏾ = Ᏹ or Ᏼ, then height 1 roots of both norms lie over each λ ∈ , and the translations of L act simply transitively on each set. If ᏾ = Ᏻ, then height 1 roots lie over each λ ∈ : long roots over λ of even norm and short roots over λ of odd norm. Again, the translations act simply transitively on each set. The differing behavior of the different rings is another manifestation of the fact that Ᏹ and Ᏼ have elements with half-integer real part, while Ᏻ does not. We may also enumerate roots of larger heights. For example, if is an Ᏹ-lattice, then there are short roots of L of height θ over each λθ −1 ∈ θ −1 with λ2 ≡ 1 modulo 3. Now we discuss the second idea of this section: the effects of reﬂections in roots of small height h. This occupies the rest of the section. Lemma 5.2. Suppose that is a positive-deﬁnite Ᏻ-lattice and L = ⊕ II1,1 . Suppose that r is a long root of L of height 1 lying over λ ∈ , and that v is an isotropic√ vector of L ⊗ R of height 1 that lies over 1 ∈ ⊗ R. Suppose that (1 − λ)2 < 3. Then there is another long root r ∈ L of height 1, also lying over λ, such that the biﬂection in r reduces the height of v. Proof. Since v has height 1 and norm 0 and lies over 1, we know that for some w ∈ Im K, we have v = (1; 1, w − 12 /2). Similarly, we deduce that 2 − λ2 (5.2) r = λ; 1, z0 + 2 for some z0 ∈ Im K. Every other height 1 long root of L lying over λ has the form r = r + (0; 0, z) for some z ∈ Im Ᏻ. We will obtain the theorem by choosing z appropriately. We have (5.3) r | v = λ | 1 + w − 12 /2 + z¯ 0 + z¯ + 1 − λ2 /2 1 (5.4) = 1 − 12 − 2λ | 1 + λ2 + w + z¯ 0 + z¯ 2 1 1 = 1 − 12 − λ | 1 − 1 | λ + λ2 + λ | 1 − 1 | λ + w + z¯ 0 + z¯ (5.5) 2 2

NEW HYPERBOLIC REFLECTION GROUPS

1 = 1 + (1 − λ)2 + Imλ | 1 + w + z¯ 0 + z¯ 2 = a + B,

315 (5.6) (5.7)

where a is the ﬁrst bracketed expression and B is the second. The important thing to observe here is that a depends on (1 − λ)2 , which is bounded by hypothesis, and B depends on z, over which we have some control. Let v be the image of v under biﬂection in r . Since v = v − r r | v, we have ρ | v = ρ | v − ρ | r r | v = 1 − (1)(a + B) =

(1 − λ)2 − B. 2

Since the covering radius of Im Ᏻ is 1/2, we may choose z so that |B| ≤ 1/2. Then √ 2 2 2 ρ | v 2 = (1 − λ) + |B|2 < 3 + 1 = 1, 2 2 2 so that ht v < ht v. Lemma 5.3. Suppose that is a positive-deﬁnite ᏾-lattice and L = ⊕ II1,1 . Let h = 1 if ᏾ = Ᏻ, h = 1 or θ if ᏾ = Ᏹ, and h = 1 or 1 + i if ᏾ = Ᏼ. Suppose that r is a short root of L of height h lying over λh−1 , with λ ∈ . Let v ∈ L ⊗ R be isotropic, have height 1, and lie over 1 ∈ ⊗ R. Set D 2 = (1 − λh−1 )2 and suppose that D 2 ≤ 1/|h|2 . Then there exists a short root r of L, also of height h and lying over λh−1 , such that one of the following holds: (i) some reﬂection in r carries v to a vector of smaller height than v, (ii) D 2 = 1/|h|2 and r | v = 0, or (iii) ᏾ = Ᏼ, h = 1 + i, D 2 = 1/|h|2 = 1/2, and r | v = (1 + i)/2. Proof. From the given norms and heights of v and r, together with the fact that they lie over 1 and λh−1 , we deduce that 1 − λ2 2 and r = λ; h, z0 + h (5.8) v = 1; 1, w − 1 /2 2|h|2 ¯ 0 ) = 0. The other height h short roots for some w ∈ Im K and z0 ∈ K such that Re(hz ¯ = 0. of L lying over λh−1 have the form r = r +(0; 0, z) for z ∈ ᏾ such that Re(hz) The basic idea is similar to that of Lemma 5.2. We try to choose z, together with a unit ξ of ᏾, such that the ξ -reﬂection in r carries v to a vector of smaller height. It may happen that no such choice is possible, which leads to the cases (ii) and (iii) of Theorem 5.3. A calculation similar to (5.3)–(5.7) reveals that

316

DANIEL ALLCOCK

12 (1 − λ2 )h ¯ r | v = λ | 1 + h w − + z0 + z + 2 2|h|2 2 1 |h|2 2 −1 −1 2 − D + |h| Imλh | 1 + |h| w + h¯z0 + h¯z =h 2 2

= h−1 [a + B],

(5.9)

where a = (1 − |h|2 D 2 )/2 is the real part of the term in brackets and B is the imaginary part. The slight difference between the terms a in (5.7) and (5.9) is due to the replacement of 2 − λ2 in (5.2) with 1 − λ2 in (5.8), which is due to the fact that r is now a short root. We take v to be the image of v under ξ -reﬂection in r (we choose ξ later). Since v = v − r (1 − ξ )r | v, we have ρ | v = ρ | v − ρ | r (1 − ξ )r | v = 1+

h(ξ − 1)h¯ [a + B]. |h|2

(5.10)

By hypothesis, D 2 ≤ 1/|h|2 , so a ∈ [0, 1/2]. We may change the value of B by h¯z, ¯ where z may be any element of ᏾ ∩ Im(hK). That is, we may change B by any element of ¯ h · ᏾ ∩ Im(hK) = h · ᏾ ∩ (h · Im K) = h · ᏾ ∩ (Im K) · h¯ = (h᏾) ∩ h · (Im K) · h¯ = (h᏾) ∩ Im K = Im(h᏾). We try to choose ξ and z so that (5.10) has absolute value less than 1 = ρ | v. This requires treating the different possibilities for ᏾ and h separately. We treat only the case ᏾ = Ᏼ, h = 1 + i, which is much more involved than the other four cases. We write B as bi + cj + dk with b, c, d ∈ R. We ﬁrst carry out a computation that allows us to use the 24 units of Ᏼ effectively. We claim that there is a unit ξ of Ᏼ with Re ξ = −1/2 such that 2 2 2 2 1 + ξ (a + B) 2 = a − 1 + |b| − 1 + |c| − 1 + |d| − 1 . (5.11) 2 2 2 2 For any unit ξ , the left side is just the square of the distance between a + B and −ξ¯ (proof: left-multiply by 1 = | − ξ¯ |2 ). Setting −ξ¯ = (1 ± i ± j ± k)/2, with each of its i, j, and k components having the same sign as the corresponding component of B (or a random sign if that component vanishes), the right-hand side becomes another expression for this squared distance, proving the claim.

317

NEW HYPERBOLIC REFLECTION GROUPS

Next we investigate our freedom to choose B. By choice of z, we may vary B by any element of Im(hᏴ). It is easy to check that

Im (1 + i)Ᏼ = bi + cj + dk | b, c, d ∈ Z, b + c + d ≡ 0 (mod 2) . That is, Im(hᏴ) is spanned by j + k, k + i , and 2i, so by choice of z, we may take b ∈ (−1, 1] and c, d ∈ [0, 1). Suppose for a moment that there is a unit ξ of Ᏼ such that ξ =

h(ξ − 1)h¯ , |h|2

(5.12)

where ξ is as in (5.11). Then by (5.10), (5.12), and (5.11), 2 | ht v |2 = 1 + ξ (a + B)

1 = a− 2

2

1 + |b| − 2

2

1 + |c| − 2

2

1 + |d| − 2

2 .

(5.13)

We have already shown that a ∈ [0, 1/2]. By this and the constraints on b, c, and d obtained above, we see that the right-hand side of (5.13) is less than 1 = | ht v|2 (so that conclusion (i) applies) unless a = 0, b ∈ {0, 1}, and c = d = 0. In this exceptional case, a = 0 implies that D 2 = 1/|h|2 , and r | v can be read from (5.9). If b = 0, we have r ⊥v and conclusion (ii) applies, and if b = 1, then conclusion (iii) applies. It only remains to show that given a unit ξ of Ᏼ with Re ξ = −1/2, there is another unit ξ of Ᏼ satisfying (5.12). We simply solve for ξ : (1 − i)ξ (1 + i) ξ = |h|2 · h−1 ξ h¯ −1 + 1 = √ √ + 1. 2 2

(5.14)

The most straightforward way to show that ξ is a unit of Ᏼ is to simply evaluate the right-hand side of (5.14) for each of the eight possibilities ξ = (−1 ±√ i ± j ± k)/2. (What is really going on here is that the units of Ᏼ together with (1+i)/ 2 generate the binary octahedral group, which normalizes the binary tetrahedral group consisting of the units of Ᏼ.) The idea used in the proofs of the last two lemmas can also be applied for some other values of h. The cases stated above are the ones that are used later, but for completeness we summarize in Table 1 all the cases we have been able to treat with this method. The table should be read as follows. Suppose that is an ᏾-lattice, L = ⊕ II1,1 , and r is a short root of L whose height h appears in the table, lying over λh−1 , with λ ∈ . Suppose that v is a primitive null vector of L ⊗ R of height 1, lying over 1 ∈ ⊗ R, and that D 2 = (1 − λh−1 )2 satisﬁes D 2 ≤ R 2 , where R 2 is given by the table. Then there is another root r of L, of the same height and norm as r and also lying over λh−1 , such that either some reﬂection in r preserves L and

318

DANIEL ALLCOCK

Table 1. Summary of Lemmas 5.2 and 5.3 and generalizations thereof The ring ᏾

Ᏻ

r | v √ 1 − ( 3/2) + (i/2)

1

1

0

1+i

1/2

0 or h−1 i

2

1/4

0 or h−1 i

1

1

−ω¯

θ

1/3

−h−1 ω¯

1

1

0

θ

1/3

0

2

1/4

0 or h−1 θ

2θ

1/12

0 or h−1 θ

1

1

−ω¯

1

1

0

1+i

1/2

0 or (1 + i)/2

2

1/4

(1 + ai + bj + ck)/2 for

Height h

Long Short

Long Ᏹ

Short

Long Ᏼ

1

R2 √ 3

Root length

Short

a, b, c ∈ {0, 1} reduces the height of v, or else D 2 = R 2 and r | v takes one of the values given in the table. Note that R 2 = 1/|h|2 in all cases except that of long roots of height 1 in Gaussian lattices. 6. The reﬂection groups. This section is the heart of the paper. We apply the results of Section 5 to ﬁnd Lorentzian lattices that are reﬂective. We begin by providing a general criterion for a lattice to be reﬂective, and we give a number of examples (see Theorems 6.1–6.3). Then we study in much greater detail the lattices Ᏹn,1 and Ᏼn,1 for small n (see Lemma 6.4 through Theorem 6.8) and also two high-dimensional examples, acting on CH 7 and HH 5 (see Lemma 6.9 through Theorem 6.14). At the ᏾ Ᏻ end of the section, we return to low dimensions, discussing the lattices I1,1 and I I1,1 . We begin with the most basic of our results. Theorem 6.1. Suppose that is a positive-deﬁnite ᏾-lattice that is spanned up to ﬁnite index by its roots and has covering radius less than or equal to 1. Then L = ⊕ II1,1 is reﬂective. Furthermore, if the covering radius is less than 1, then any two primitive isotropic vectors of L are equivalent (up to a scalar) under Reﬂec L.

NEW HYPERBOLIC REFLECTION GROUPS

319

Proof. According to Lemma 5.1, the stabilizer of ρ in Reﬂec L has ﬁnite index in the stabilizer in Aut L. Now we study the Reﬂec L orbits of primitive null vectors in L. Suppose that v is such a vector, that it is not a multiple of ρ, and that it has minimal height in its Reﬂec L orbit. Let 1 be the element of ⊗ Q over which it lies, let λ be an element of nearest 1, and let r be a short root of L of height 1 lying over λ (or a long root if ᏾ = Ᏻ and λ2 is even). We must have (1−λ)2 ≥ 1, for otherwise Lemma 5.3 (or Lemma 5.2 if ᏾ = Ᏻ and r is long) assures us that v is not of minimal height in its Reﬂec L orbit. In particular, if has covering radius less than 1, then v cannot exist, and we have proven that every null vector of L is equivalent under Reﬂec L to a multiple of ρ. This is the second part of the theorem. In the case when the covering radius of is exactly 1, we can still deduce that there are only ﬁnitely many Reﬂec L orbits of primitive null vectors in L. For if we cannot reduce the height of v by a reﬂection, then 1 is a deep hole of , and if λ is any vertex of the hole, then there is a short root r of L of height 1 that lies over λ and is orthogonal to v. Now v is determined up to a unit scalar by the point 1 of ⊗ Q and the root r (lying over λ) to which it is orthogonal. Since the stabilizer of ρ in Reﬂec L contains a ﬁnite-index subgroup of the translations of L, we may take r to lie in some ﬁxed ﬁnite set of roots. Then 1 is a deep hole nearest λ, for which there are only ﬁnitely many possibilities. That is, there are only ﬁnitely many Reﬂec L orbits of primitive null vectors in L. The fact that Reﬂec L has ﬁnite index in Aut L follows from this, together with the fact that for one particular primitive null vector, namely, ρ, its stabilizer in Reﬂec L has ﬁnite index in its stabilizer in Aut L. Essentially the same argument, using Lemma 5.2 in place of Lemma 5.3, proves the following theorem. -lattice that is spanned Theorem 6.2. Suppose that is an even positive-deﬁnite Ᏻ √ 4 up to ﬁnite index by its roots and has covering radius less than 3. Then L = ⊕II1,1 is reﬂective and any two primitive null vectors of L are equivalent (up to a scalar) under Reﬂec L. Corollary 6.3. Let be any of the ᏾-lattices Ᏻ, 21/2 Ᏻ, D4Ᏻ , D6Ᏻ or E8Ᏻ

if ᏾ = Ᏻ,

Ᏹ, Ᏹ ,

Ᏹ or D4 (θ)

if ᏾ = Ᏹ, or

Ᏼ,

21/2 Ᏼ, Ᏼ2 or E8Ᏼ

if ᏾ = Ᏼ.

2

3

Then L is reﬂective. Furthermore, if appears in the ﬁrst column of the list, then any two primitive null vectors of L are equivalent (up to a scalar) under Reﬂec L. Remark. The lattices appearing here are all described in Section 3. Theorems 6.6 and 6.8 give much more precise information about Reﬂec L for = Ᏹ, Ᏹ2 , Ᏹ3 , Ᏼ, or Ᏼ2 .

320

DANIEL ALLCOCK

Proof. All these lattices by their roots. The covering radii of the √ are spanned √ Gaussian lattices are 1/ 2, 1, 1, 3/2, and √ 1, and √ all but the ﬁrst are even. The 3, 2/3, 1, and 1, and those of the covering radii of the Eisenstein lattices are 1/ √ Hurwitz lattices are 1/ 2, 1, 1, and 1. The result follows from Theorems 6.1 and 6.2. We now study in more detail the reﬂection groups of some low-dimensional, selfdual Lorentzian lattices over Ᏹ and Ᏼ. If L is any lattice, we write Reﬂec0 L for the subgroup of Reﬂec L generated by the reﬂections in the short roots of L. Lemma 6.4. Suppose that ᏾ = Ᏹ, = Ᏹn (n > 0), and L = ⊕ II1,1 . Then the following hold. (i) Reﬂec0 L contains all the translations of L. (ii) Reﬂec0 L contains a transformation acting trivially on and as ω on II1,1 . (iii) The stabilizers of ρ in G and Aut L coincide, where G is the group generated by Reﬂec0 L and the central involution −I of L. Furthermore, G ⊆ Reﬂec L. Proof. (i) By Lemma 5.1(i), Reﬂec L contains a translation Tx,z for each x ∈ Ᏹn . The proof shows that these translations actually lie in Reﬂec0 L. Taking commutators as in Lemma 5.1(iii) shows that Reﬂec0 L contains all the translations of L. (ii) We have T0,−θ ∈ Reﬂec0 L by (i). Let F be the transformation composed of T0,−θ followed by (−ω)-reﬂection in the short root (0; 1, −ω). It is obvious that F ᏾ acts trivially on and computation reveals that it acts on II1,1 by left multiplication by the matrix 0 ω¯ . ω¯ 0 The square of this matrix is the scalar ω of II1,1 , which proves the claim. (iii) Since Reﬂec0 L contains the central involution of , G contains the central involution J of II1,1 . The biﬂection B in b = (0, . .3. , 0; 1, 1) acts trivially on and 0 −1 on II1,1 as −1 0 . It can be checked that J = F B, where F ∈ Reﬂec0 L is as in (ii). This proves that B ∈ G and also that G = Reﬂec0 L, B, hence G ⊆ Reﬂec L. We also note that since G contains J and also F 2 , it contains all the scalars of II1,1 , so it sufﬁces to show that G contains the full stabilizer in Aut L of ρ. In light of (i) it sufﬁces merely to show that G contains Aut . If n = 1, then Aut is generated by reﬂections in its short roots, as desired. If n > 1, then it sufﬁces to prove that G contains the coordinate permutations with respect to the chosen basis of . That is, we must show that G contains the biﬂections in vectors like x = (1, −1, 0, . . . , 0; 0, 0). It sufﬁces to show that x and b are equivalent under G. To see this, observe that T(1,0,...,0),θ/2 followed by F , followed by the scalar −ω, followed by T(−1,1,0,...,0),0 , carries x to b. Ᏹ Remarks. The condition n > 0 is necessary. We can show that Reﬂec I I1,1 contains no scalars except the identity and that it does not contain all the translations.

NEW HYPERBOLIC REFLECTION GROUPS

321

Lemma 6.5. Suppose that r and r are short roots in a lattice over ᏾ = Ᏹ or Ᏼ and |r | r | = 1. Then r and r are equivalent under the group generated by the reﬂections in them. Proof. We check that the (−ω)-reﬂections R and R in r and r satisfy the braid relation RR R = R RR . (Because the Hermitian form is degenerate on the span of r and r , we must check that this relation holds by using (2.1), not by just multiplying matrices for the actions of R and R on the span of r and r .) Rewriting this as R −1 RR = RR R −1 , we see that R and R are conjugate in the group they generate, which implies the lemma. Remark. The proof suggests connections between the braid groups and complex reﬂection groups. This connection was ﬁrst observed by Coxeter [16], and the braid groups play a central role in the work of Deligne and Mostow [18], Mostow [22], and Thurston [24]. They are also important in work by the author, J. Carlson and D. Toledo on moduli of cubic surfaces [4], [3]. Ᏹ Theorem 6.6. Let ᏾ = Ᏹ, = Ᏹn , and L = ⊕ I I1,1 . (i) If n = 1, then Reﬂec0 L acts with exactly 2 orbits of primitive null vectors, represented by ±ρ. If n = 2 or 3, then Reﬂec0 L acts transitively on the primitive null vectors of L. (ii) If n = 1, 2, or 3, then Aut L = Reﬂec L = Reﬂec0 L × {±I }.

Proof. First we show that Reﬂec0 L acts transitively on the 1-dimensional primitive null lattices in L. For n = 1 or 2, this follows from Theorem 6.3. So suppose that n = 3 and that v ∈ L is a primitive null vector not proportional to ρ and of smallest height in its orbit under Reﬂec0 L. Since the covering radius of Ᏹ3 is 1, Lemma 5.3(ii) implies that v is orthogonal to a short root of height 1. By applying a translation (see Lemma 6.4(i)) we may suppose that this root is r1 = (0, 0, 0; 1, −ω). Taking r2 = (0, 0, 1; 0, 1) and r3 = (0, 0, 1; 0, 0), we see that r1 | r2 = r2 | r3 = 1, so Lemma 6.5 shows that r1 is equivalent to r3 under Reﬂec0 L. Thus, v is equivalent Ᏹ to an element of r3⊥ , which is a copy of Ᏹ2 ⊕ I I1,1 . Applying the n = 2 case, we see that Reﬂec0 L acts transitively on the primitive null sublattices of L. It follows from Lemma 6.4(iii) that Aut L is generated by Reﬂec0 L and {±I }. Next we show that −I ∈ / Reﬂec0 L, from which follows the equality Aut L = Reﬂec0 L × {±I }. Then (ii) follows, because Lemma 6.4(iii) shows that −I ∈ Reﬂec L. To prove −I ∈ / Reﬂec0 L, we must consider the ﬁnite vector space V = L/Lθ over F3 = Ᏹ/θ Ᏹ; we write q for both natural maps L → V and Ᏹ → F3 . The Hermitian form on L gives rise to a symmetric bilinear form on V , given by q(v) | q(w) = q(v | w). Ᏹ ∼ Ᏹ Since I I1,1 = I1,1 , each of L and V admits an orthogonal basis with n + 1 vectors of norm 1 and one of norm −1. There is a homomorphism called the spinor norm from Aut V to the group {±1} of nonzero square classes in F3 . This is characterized by the property that the reﬂection in a vector of V of norm ±1 has spinor norm ±1. It is clear that −I acts on V with spinor norm −1. Since a reﬂection of L in a short root r acts

322

DANIEL ALLCOCK

on V either trivially or as the reﬂection in q(r), of norm q(r 2 ) = 1, every element of Reﬂec0 L acts on V with spinor norm +1. Hence, −I ∈ / Reﬂec0 L, as desired. This also characterizes Reﬂec0 L as the subgroup of Aut L whose elements act on V with spinor norm +1. Now we establish Theorem 6.6(i). By the ﬁrst part of the proof, it sufﬁces to determine which multiples of ρ are equivalent to each other. By Lemma 6.4(ii), it sufﬁces to determine whether ±ρ are equivalent. If n = 1, then they are inequivalent, because the stabilizers of ρ in Reﬂec0 L and Aut L coincide. If ρ and −ρ were equivalent, then we would have transitivity on primitive null vectors and Reﬂec0 L = Aut L would follow. Since this is not true, ρ and −ρ are inequivalent. On the other hand, if n = 2 or 3, then ±ρ are equivalent. To see this, apply the product of −I and the biﬂection in any long root of L that is orthogonal to ρ. The product exchanges ±ρ, and by spinor norm considerations, it lies in Reﬂec0 L. Remark. The case n = 3 arises in algebraic geometry. The quotient of CH 4 by Ᏹ Reﬂec0 I4,1 may be identiﬁed with the moduli space of stable cubic surfaces in CP 3 . We can also construct the moduli space of marked stable cubic surfaces by taking Ᏹ the quotient of CH 4 by the congruence subgroup of Reﬂec0 I4,1 associated to the Ᏹ prime θ ∈ Ᏹ. The quotient of Reﬂec0 I4,1 by this normal subgroup is the E6 Weyl group, also known as “the group of the 27 lines on a cubic surface.” See [4] and [3] for details. Lemma 6.7. Suppose that ᏾ = Ᏼ, = Ᏼn (n > 0), and L = ⊕ II1,1 . Then the following hold. (i) Reﬂec0 L contains a translation Tx,z for each x ∈ , and also the central translations T0,ai+bj+ck with a ≡ b ≡ c (mod 2). In particular, coset representatives for the translations of Reﬂec0 L in those of Aut L may be taken from {T0,0 , T0,i , T0,j , T0,k }. (ii) Reﬂec0 L contains transformations acting trivially on and on II1,1 by left scalar multiplication by any given unit of Ᏼ. (iii) The stabilizer of ρ in Reﬂec0 L has index less than or equal to 4 in the stabilizer in Aut L; coset representatives may be taken from the set given in (i). Proof. (i) The ﬁrst part follows immediately from Lemma 5.1(i). The second part may be obtained by taking commutators: if λ, λ ∈ and z, z ∈ Im K are such that Tλ,z , Tλ ,z ∈ Reﬂec0 L, then T0,±2 Imλ|λ ∈ Reﬂec0 L by (4.5). Since contains vectors λ and λ with λ | λ = α for any given unit α of Ᏼ, we see that Reﬂec0 L contains T0,2i , T0,2j , T0,2k , and T0,i+j+k . These generate the group of central translations given in the statement of the lemma. (ii) The argument of Lemma 6.4(ii) shows that Reﬂec0 L contains an element acting trivially on and on II1,1 as left multiplication by ω. Taking conjugates of this by the group Aut Ᏼ acting on II1,1 , which normalizes Reﬂec0 L even though it does not act Ᏼ-linearly, we see that Reﬂec0 L contains elements acting on II1,1 as left multiplication by any of the units (−1 ± i ± j ± k)/2 of Ᏼ. These generate the group

NEW HYPERBOLIC REFLECTION GROUPS

323

of all units of Ᏼ, proving (ii). (iii) This follows immediately from (i) and (ii) and the arguments given for Lemma 6.4(iii). Note the curious fact that Reﬂec0 L contains the biﬂection B, which it did not in the Eisenstein case. Ᏼ Theorem 6.8. Let ᏾ = Ᏼ, = Ᏼn for n = 1 or 2, and L = ⊕ I I1,1 . Then Reﬂec0 L acts transitively on the primitive null vectors of L and has index at most 4 in Aut L; coset representative may be taken from {T0,0 , T0,i , T0,j , T0,k }.

Proof. We ﬁrst claim that Reﬂec0 L acts transitively on primitive null lattices in L. For n = 1, this follows from Theorem 6.3. For n = 2, it follows from an argument similar to the n = 3 case of Theorem 6.6. That is, the covering radius of = Ᏼ2 is 1, so if v ∈ L is primitive, isotropic, and of smallest height in its orbit under Reﬂec0 L, then by Lemma 5.3(ii), we see that v is either proportional to ρ or orthogonal to a short root r1 of height 1. In the latter case, after applying a translation of Reﬂec0 L, courtesy of Theorem 6.7(i), we may take r = (0, 0; 1, x − ω), where x is one of 0, i, j, and k. In any of these cases, upon taking r2 = (0, 1; 0, 1) and r3 = (0, 1; 0, 0), we have r1 | r2 = r2 | r3 = 1. By Lemma 6.5, v is equivalent under Reﬂec0 L to an Ᏼ element of r3⊥ . Since r3⊥ is a copy of Ᏼ1 ⊕I I1,1 , the transitivity follows from the case n = 1. The transitivity on primitive null vectors follows from Lemma 6.7(ii). The rest of the theorem follows from Lemma 6.7(iii). Now we move on to higher-dimensional examples. We construct a group acting on CH 7 and another acting on HH 5 . These arise from our basic construction by taking = Ᏹ6 or Ᏼ 4. Lemma 6.9. Suppose that v, r1 , . . . , rm ∈ Kn ⊕ K1,1 lie over 1, λ1 , . . . , λm ∈ Kn , respectively. Suppose that v 2 = 0, that ri | v = 0 for all i, and that the vectors λi − 1 are linearly independent in Kn . Then the images of the ri in v ⊥ /v are linearly independent. Proof. We may obviously replace v and the ri by any scalar multiples of themselves and so suppose that they have height 1. Thus v = (1; 1, ?) and ri = (λi ; 1, ?) where the question marks denote irrelevant (and possibly distinct) elements of K. Let T be the translation carrying v to (0; 1, 0), so T (ri ) = (λi − 1; 1, 0). (The last coordinate vanishes because T (ri ) | T v = 0.) Since the image of T (ri ) in (T v)⊥ /T v may be identiﬁed with its ﬁrst coordinate, namely, λi − 1, the images of the T (ri ) in (T v)⊥ /T v are linearly independent. The lemma follows immediately. Lemma 6.10. Ᏹ6 ⊗R is covered by the closed balls of radius 1 centered at points √ of Ᏹ6 , together with those of radius 1/ 3 centered at points λθ −1 with λ ∈ Ᏹ6 and λ2 ≡ 1 (mod 3). Proof. Conway and Sloane [14, Section 7] deﬁne a linear “gluing map” g : (1/θ )Ᏹ6 /Ᏹ6 → (1/θ )Ᏹ6 /Ᏹ6 with the property that the Leech lattice 24 , scaled

324

DANIEL ALLCOCK

down by 21/2 , is the real form of the lattice of vectors (x1 , x2 ) ∈ ((1/θ)Ᏹ6 )2 satisfying g(x1 + Ᏹ6 ) = x2 + Ᏹ6 . Identifying Ᏹ6 with the set of such (x1 , x2 ) with x2 = 0, we see that the only points of 2−1/2 24 at distance less than 1 from Ᏹ6 ⊗R are those in Ᏹ6 and those of the form (x1 θ −1 , x2 θ −1 ) with x2 a minimal vector of Ᏹ6 (a long root). The deﬁnition of g (see [14]) shows that x12 ≡ 1 (mod 3) if and only if there is a long root x2 of Ᏹ6 such that (x1 θ −1 , x2 θ −1 ) ∈ 2−1/2 24 . By [12], the covering radius of 2−1/2 24 is 1. Therefore, the balls of radius 1 centered at the points of Ᏹ6 , and at the points (x1 θ −1 , x2 θ −1 ) with x12 ≡ 1 (mod 3) and x22 = 2, cover Ᏹ6 ⊗ R. Computing the radius of the intersection of a ball of the second family with Ᏹ6 ⊗ R yields the lemma. Ᏹ ∼ Ᏹ Theorem 6.11. Let = Ᏹ6 and L = ⊕ I I1,1 = I7,1 . (i) If v ∈ L is primitive, isotropic, and not equivalent to a multiple of ρ under Reﬂec0 L, then v ⊥ /v ∼ = Ᏹ6 . (ii) Aut L coincides with Reﬂec L. In particular, L is reﬂective.

Proof. (i) Suppose that v is a primitive isotropic vector of smallest height in its orbit under Reﬂec0 L. Suppose that v is not a multiple of ρ, so that it lies over some 1 ∈ ⊗ R. By Lemma 5.3 and the minimality of the height of v, 1 lies at distance greater than or equal to 1 from each lattice point λ ∈ and at distance greater than or equal to 3−1/2 from each λθ −1 with λ ∈ and λ2 ≡ 1 (mod 3). By Lemma 6.10, the set S of such points in ⊗ R is discrete. Let µ1 , . . . , µn be the elements of with (1 − µi )2 = 1 and let ν1 , . . . , νm be those vectors of the form λθ −1 with λ ∈ and λ2 ≡ 1 (mod 3) such that (1−νi )2 = 1/3. Over each µi (resp., νi ) there is a short root of L, say, ri (resp., si ), of height 1 (resp., θ). By Lemma 5.3, we may suppose that the ri and si are orthogonal to v. Because S is discrete, the vectors µi −1 and νi −1 span ⊗R, and therefore there are 6 among them that are linearly independent over C. By Lemma 6.9, this implies that among the images in v ⊥ /v of the vectors ri and si are 6 short roots that are linearly independent over Ᏹ. Since v ⊥ /v is positive-deﬁnite, it follows that v ⊥ /v ∼ = Ᏹ6 . (ii) We have seen the transitivity of Reﬂec0 L on the primitive null sublattices that, Ᏹ ∼ 6 like ρ, are orthogonal to no short roots. Since L ∼ = Ᏹ ⊕II1,1 , Lemma 6.4(iii) = I7,1 implies that Reﬂec0 L, −I lies in Reﬂec L and contains the scalars of L. Therefore, Reﬂec L acts transitively on the primitive null vectors that, like ρ, are orthogonal to no short roots. Now it sufﬁces to show that Reﬂec L contains the full stabilizer of ρ. Since is self-dual and spanned by its long roots, Lemma 5.1(ii) and (iii) imply that Reﬂec L contains all the translations of L. Since Aut is a reﬂection group, the proof is complete. Ᏼ We now study the quaternionic lattice I5,1 . The analysis is surprisingly similar to Ᏹ our study of I7,1 . In particular, Lemma 6.12 is very similar to Lemma 6.10. Conway and Sloane [13, Section 5] describe an embedding of the real form BW 16 of 21/2 Ᏼ 4

NEW HYPERBOLIC REFLECTION GROUPS

325

into the Leech lattice 24 . When we refer to concepts involving 24 while discussing Ᏼ 4 , we implicitly refer to this embedding. (Up to isometry of 24 , there is only one embedding.) the closed balls of radius 1 centered at points Lemma 6.12. Ᏼ 4 ⊗R is covered by √ , together with those of radius 1/ 2 centered at points λ(1 + i)−1 with λ ∈ Ᏼ of Ᏼ 4 4 of odd norm. Any point of Ᏼ 4 ⊗ R not in the interior of one of these balls is a deep hole of 2−1/2 24 . −1/2 is a copy of the E lattice. Proof. The orthogonal complement of Ᏼ 24 8 4 in 2 Properties of the embedding are described in [13] and include the following. If (x, y) ∈ −1/2 , then y ∈ (1/2)E and hence y has norm n/2 (Ᏼ 24 8 4 ⊗ R) × (E8 ⊗ R) lies in 2 for some nonnegative integer n. We write B(x, y) for the ball of radius 1 with center (x, y) ∈ 2−1/2 24 . Only if the norm of y is 0 or 1/2 does the interior of B(x, y) −1/2 meet Ᏼ 24 for some y of norm 1/2 are 4 ⊗ R. Those x for which (x, y) ∈ 2 Ᏼ exactly the deep holes of 4 . By Theorem 3.1, the set of such x coincides with Ᏼ 2 {λ(1+i)−1 | λ ∈ Ᏼ 4 , λ ≡ 1 (mod 2)}. For such (x, y), the ball B(x, y) meets 4 ⊗R in a ball of radius 2−1/2 . The theorem follows from the fact that the covering radius of 2−1/2 24 is 1 (see [12]).

Lemma 6.13. Any deep hole of 24 that lies in BW 16 ⊗R has a vertex in BW 16 . Proof. The natural language for discussing the deep holes of 24 is that of afﬁne Coxeter-Dynkin diagrams, using the slightly nonstandard √ conventions of [12]. If h is a deep hole of 24 , then its vertices vi lie at distance 2 from h, and we deﬁne the diagram 8 of h to be the graph whose vertices are the vi , with vi and vj unjoined, singly joined, or doubly joined according to whether (vi − vj )2 is 4, 6, or 8. Each component of 8 is an afﬁne diagram of type An , Dn , or En . For the rest of the proof, we take h as the origin. The deﬁnition of 8 and the fact that (vi − h)2 = 2 for all i means that the inner product of vi and vj is 0, −1, or −2 according to whether the corresponding vertices of 8 are unjoined, singly joined, or doubly joined. It follows that the subspaces spanned by different components of 8 are orthogonal, and that the vertices corresponding to each component form a system of simple roots of the corresponding type, together with the lowest root, which corresponds to the extending node in the diagram. Now BW 16 is the ﬁxed-point set of an involution φ of 24 . Since φ ﬁxes h, it acts on 8. We show that φ preserves a vertex v of 8, which forces v to lie in BW 16 , which proves the lemma. For each component C of 8, we write SC for its real span in R24 . If φ preserves C, then we write FC for the subspace of SC ﬁxed pointwise by φ. We write F for the subspace of R24 ﬁxed pointwise by φ. It is easy to see that dim F =

φ(C)=C

dim FC +

dim SC , 2

φ(C)!=C

(6.1)

326

DANIEL ALLCOCK

where the sums are over the components C of 8 that are (resp., are not) preserved by φ. If φ preserves C, then its action on C determines FC . The explicit description in terms of root systems allows us to deduce that dim FC equals the number of vertices of C preserved by φ, plus half the number not preserved, minus 1. It follows that if φ permutes the v i freely, then each term in each sum in (6.1) is bounded by (1/2) dim SC . Since C dim SC = 24, we would obtain dim F ≤ 12, which is impossible since BW 16 ⊆ F . Remark. A more involved analysis shows that any deep hole of 24 lying in BW 16 ⊗R has at least nine vertices in BW 16 and that this bound cannot be improved. Ᏼ ∼ Ᏼ Theorem 6.14. Let = Ᏼ 4 , and let L = ⊕ I I1,1 = I5,1 . (i) If v ∈ L is a primitive isotropic vector not equivalent under Reﬂec0 L to a multiple of ρ, then v ⊥ /v contains a short root. (ii) The index of Reﬂec L in Aut L is at most 4, so that L is reﬂective. More precisely, coset representatives for Reﬂec L in Aut L may be taken from {T0,0 , T0,i , T0,j , T0,k }.

Proof. (i) This is very similar to the proof of Theorem 6.11(i). Suppose that v is a primitive isotropic vector of L of smallest height in its orbit under Reﬂec L. Suppose that v is not a multiple of ρ, so that v lies over some 1 ∈ ⊗ R. By Lemma 5.3, 1 lies at distance greater than or equal to 1 from each lattice point λ ∈ and at distance greater than or equal to 2−1/2 from each point λ(1 + i)−1 with λ ∈ and λ2 ≡ 1 (mod 2). By Lemma 6.12, 1 must be a deep hole of 2−1/2 24 . By Lemma 6.13, 2 there is a vertex λ ∈ Ᏼ 4 of the hole with (1−λ) = 1. There is a short root of L lying over λ, and by Lemma 5.3 there is also one orthogonal to v. (ii) Since is self-dual and spanned by its long roots, Lemma 5.1(ii) shows that Reﬂec L contains a translation Tx,z for each x ∈ . Taking commutators as in Lemma 6.7(i) shows that Reﬂec L contains the central translations T0,ai+bj+ck with a ≡ b ≡ c (mod 2). Then the proof of Lemma 6.7(ii) shows that Reﬂec L contains elements acting on II1,1 by left multiplication by the units of Ᏼ. Together with (i), this proves the transitivity of Reﬂec L on primitive null vectors that, like ρ, are orthogonal to no short roots. Then (ii) follows from the facts that Aut is a reﬂection group and Reﬂec L contains the translations just discussed. Ᏹ Ᏼ Remark. It would be nice to understand the groups Reﬂec0 I7,1 and Reﬂec0 I5,1 .

We close this section by returning to low dimensions and studying the 2-dimensional self-dual Lorentzian lattices. If ᏾ = Ᏹ or Ᏻ, then we can obtain very explicit descriptions of the groups by drawing pictures in CH 1 ⊆ CP 1 . In particular, if we represent a point (a, b) ∈ II1,1 by a/b ∈ CP 1 , then the hyperbolic space becomes the right half-plane and ρ becomes the point at inﬁnity. It is easy to ﬁnd the points of CP 1 corresponding to the roots of L of small height; then we can work out the group Reﬂec L. Ᏹ For example, we can check that Reﬂec I I1,1 acts as the triangle group (2, 6, ∞). We Ᏻ 1 can also show that Aut I I1,1 acts on CH as (2, 3, ∞) and its subgroup of index 2,

NEW HYPERBOLIC REFLECTION GROUPS

327

consisting of elements with determinant +1, is conjugate in GL2 (Ᏻ) to SL2 (Z). The Ᏻ group Reﬂec I I1,1 is generated by three biﬂections that act by rotations by π around the three ﬁnite corners of a quadrilateral in CH 1 with corner angles π/2, π/2, π/2, Ᏻ and π/∞. For completeness, we mention that Aut I1,1 acts on CH 1 as (2, 4, ∞), and its reﬂection subgroup acts as (4, 4, ∞). See [17] for descriptions of the groups (p, q, r) and other information. We can also treat the quaternionic case: An adaptation of the argument of [1, Ᏼ Theorem 5.3(i)] shows that Reﬂec I I1,1 acts on HH 1 ∼ = RH 4 as the rotation subgroup of the real hyperbolic reﬂection group with the Coxeter diagram below. Note that the six outer nodes generate an afﬁne reﬂection group, so this graph is a special case of the usual procedure of “hyperbolizing” an afﬁne reﬂection group by adjoining an extra node: ∞ •........................................................•...

... . ... ... ... .. . . ... ... ... .. ... ... ... . . ... .. ... ... •.........................................................•.............................................................•... . . ... .. ..... .. ... ... ... ... ... . . . . . ... . . ∞ ...... ... ... ... ∞ ... ... .. .. ... .... ... .... ... .. ... .. ...... ...... . .

•

•

7. Enumeration of self-dual lattices. As we explain below, the orbits of primitive ᏾ isotropic lattices in the Lorentzian lattice In+1,1 are in natural one-to-one correspondence with the equivalence classes of positive-deﬁnite self-dual lattices of dimension ᏾ . Since n over ᏾. This means that we may classify such lattices by studying Aut In+1,1 we have made such a study in the previous section, in terms of the geometry of various special lattices, we can now classify self-dual lattices in low dimensions. We begin with an analogue of a well-known result for lattices over Z. Theorem 7.1. An indeﬁnite self-dual lattice L over ᏾ = Ᏹ or Ᏼ is characterized up to isometry by its dimension and signature. An indeﬁnite self-dual lattice L over ᏾ = Ᏻ is characterized up to isometry by its dimension, signature, and whether it is even; if L is even with signature (n, m), then n − m is divisible by 4. Proof. First we show that any indeﬁnite self-dual ᏾-lattice L contains an isotropic vector. If ᏾ = Ᏼ or if dim L > 2, then the real form of L ⊗ Q is an indeﬁnite, rational, bilinear form of rank greater than 4, so Meyer’s theorem [20, Chapter 2] asserts the existence of an isotropic vector. If dim L = 2 and ᏾ = Ᏻ or Ᏹ, then we consider the 2 × 2 matrix of inner products of the elements of a basis for L. This may be diagonalized by row and column operations over ᏾ ⊗ Q to a diagonal matrix [a, −a −1 ] with a ∈ Q. (Each term is real because the matrix is Hermitian, and

328

DANIEL ALLCOCK

each determines the other because the determinant is −1.) Then the vector (1, a) is isotropic. Having obtained an isotropic vector in L⊗Q, we may multiply by a scalar to obtain one in L. ᏾ If L is odd, then the proof of [20, Theorem 4.3] applies, and L ∼ for some n = In,m and m. This completes the proof of the ﬁrst claim, since any self-dual lattice over Ᏹ or Ᏼ is odd: If v, w ∈ L satisfy v | w = ω, then v 2 , w2 , and (v +w)2 cannot all have the same parity. This also proves that an odd indeﬁnite self-dual Gaussian lattice is characterized by its dimension and signature. We may construct lattices N from an odd Gaussian lattice M by considering the sublattice M e consisting of the elements of M of even norm and by considering the Ᏻ three lattices N such that Me ⊆ N ⊆ Me . When M is I1,1 , then N may be chosen Ᏻ to be I I1,1 . Now consider an indeﬁnite even self-dual Ᏻ-lattice L. We know that L Ᏻ . contains an isotropic vector, and as in [20] there is a decomposition L = ⊕ I I1,1 We see that L arises by applying the construction above to the odd self-dual lattice Ᏻ Ᏻ Ᏻ ⊕ I1,1 . Since ⊕ I1,1 is isomorphic to In,m , it is clear that all possible L can be Ᏻ . No even lattices arise constructed by applying our construction to the various In,m unless n − m ≡ 0 (mod 4), when two isometric ones do. Ᏹ ∼ Ᏹ Ᏹ Ᏼ ∼ Ᏼ Ᏼ Special cases of Theorem 7.1 are I7,1 and I5,1 , which = 6 ⊕ I I1,1 = 4 ⊕ I I1,1 are the lattices studied in Theorems 6.11 and 6.14. Theorem 7.1 also provides the ᏾ correspondence mentioned above: If V is a primitive isotropic lattice in In+1,1 , then ⊥ it is easy to check that V /V is an n-dimensional, positive-deﬁnite, self-dual lattice, and that this establishes a one-to-one correspondence between orbits of primitive ᏾ isotropic lattices of In+1,1 and isometry classes of self-dual, positive-deﬁnite lattices in Ᏻ correspond dimension n. Similarly, the orbits of primitive isotropic lattices of I In+1,1 to the classes of positive-deﬁnite even self-dual Gaussian lattices of dimension n.

Theorem 7.2. The positive-deﬁnite self-dual Ᏹ-lattices in dimensions less than or equal to 6 are Ᏹn and Ᏹ6 . The positive-deﬁnite self-dual Ᏼ-lattices in dimensions less than or equal to 4 are Ᏼn and Ᏼ 4 . The positive-deﬁnite even self-dual Ᏻ-lattices in dimensions less than or equal to 4 are {0} and E8Ᏻ . Ᏹ ∼ Proof. By Theorem 6.11(i), any primitive null vector v of I7,1 satisﬁes v ⊥ /v = Ᏹ ⊥ 6 ∼ 6 or v /v = Ᏹ ; the ﬁrst claim follows immediately. To see the last claim, suppose that is an even self-dual Ᏻ-lattice of dimension less than or equal to 4. By the signature condition, the dimension is either 0 or 4. In the latter case, the isomorphism Ᏻ ∼ = = E8Ᏻ follows from the equivalence of any two primitive null vectors in I I5,1 Ᏻ E8 ⊕ II1,1 (see Theorem 6.3). We only sketch the quaternionic case. By Theorem 6.14(i), any 4-dimensional positive-deﬁnite self-dual Ᏼ-lattice besides Ᏼ 4 has a short root. We claim that, in fact, has a pair of independent (and hence orthogonal) short roots. This follows from the remark after Lemma 6.13: If 1 is a deep hole of 2−1/2 24 lying in Ᏼ 4 ⊗ R,

NEW HYPERBOLIC REFLECTION GROUPS

329

then it has nine vertices vi in Ᏼ 4 . By considering the hole diagram of 1, we can show that the vi − 1 span a space of real dimension greater than or equal to 5, hence of dimension greater than or equal to 2 over H. Then the argument of Theorem 6.14(i) establishes the claim. Therefore, is the direct sum of Ᏼ2 and a 2-dimensional selfᏴ dual Ᏼ-lattice. The second summand must also be Ᏼ2 , by the treatment of I3,1 in Theorem 6.8. These results have been obtained before, by very different means. Feit [19] found examples of many positive-deﬁnite self-dual Ᏹ-lattices. He derived a version of the mass formula to verify that his list was complete for dimensions n ≤ 12. Conway and Sloane [14, Theorem 3] provide a nice proof of this classiﬁcation in dimensions n ≤ 6 based on theta series and modular forms. (Their proof does not apply for 6 < n < 12; in the second to last sentence of the proof, 12 should be replaced by 7.) Although self-dual Ᏻ-lattices have not been tabulated, it would be easy (and boring) to enumerate them through dimension 12 by using the fact that the real form of a self-dual Ᏻ-lattice is self-dual over Z. An enumeration of positive-deﬁnite self-dual Ᏼ-lattices for dimensions n ≤ 7 has recently been completed by Bachoc [5] and for n = 8 by Bachoc and Nebe [6]. These enumerations are based on a generalization of Kneser’s notion of “neighboring” lattices, together with a suitable version of the mass formula. 8. Comparison with the groups of Deligne and Mostow. In this section, we justify the word “new” in our title, by showing that our largest three reﬂection groups do not appear on the lists of Mostow [23] and Thurston [24]. Deligne and Mostow [18] and Mostow [22] constructed 94 reﬂection groups acting on CH n for various n = 2, . . . , 9 by considering the monodromy action of the braid groups on families of hypergeometric functions. Thurston [24] constructed the same set of groups in terms of moduli of ﬂat metrics (with speciﬁed sorts of singularities) on the sphere S 2 . We generally refer to these groups as the DM groups. We show here (see Theorem Ᏹ Ᏻ 8.4) that none of the groups Reﬂec In,1 (n ≥ 4) or Reﬂec I I4n+1,1 (n ≥ 1) appear Ᏹ Ᏹ Ᏻ on their lists. In particular, our groups Reﬂec I7,1 , Reﬂec I4,1 , and Reﬂec I I5,1 are Ᏹ new. We also identify Reﬂec I3,1 with one of the DM groups. We leave open the question of whether our other groups appear on their lists and also the question of commensurability. We distinguish our groups from the DM groups by considering the orders of the reﬂections in the groups. We begin by showing that the only reﬂections of the self-dual lattices are the obvious ones, a result well known for lattices over Z. Lemma 8.1. Any reﬂection R of a self-dual lattice M over ᏾ = Ᏹ or Ᏻ is either a reﬂection in a lattice vector of norm ±1 or a biﬂection in a lattice vector of norm ±2. Proof. By considering the determinant of R, we discover that its only nontrivial eigenvalue is a unit of ᏾, so M contains an element of the corresponding eigenspace, so R is the α-reﬂection in some lattice vector v, where α is a unit of ᏾. Taking v to be

330

DANIEL ALLCOCK

primitive, every lattice vector in the complex span of v lies in the ᏾-span of v. (This uses the fact that ᏾ is a principal ideal domain.) Furthermore, by the self-duality of M, there exists w ∈ M satisfying v | w = 1. Then R(w) = w − v(1 − α)/v 2 and so w − R(w) = v(1 − α)/v 2 lies in M. Therefore, (1 − α)/v 2 ∈ ᏾. Unless α = −1, this requires v 2 = ±1, and if α = −1, then it requires that v 2 divide 2. In order to compare our groups to the DM groups, we also need to consider the transformations of projective space that arise from linear reﬂections, which we call projective reﬂections. If L is a Lorentzian lattice, then P Aut L may contain projective reﬂections that are not represented by any reﬂection of L. For an example, consider Ᏻ Aut I I1,1 . The subgroup of elements of determinant 1 is conjugate to SL2 (Z) and hence contains an element acting on CH 1 as a triﬂection. This happens despite the Ᏻ fact that the only reﬂections of I I1,1 are biﬂections (see Lemma 8.1). The following lemma assures us that this is merely a low-dimensional phenomenon. Lemma 8.2. Suppose that M is an n-dimensional lattice over ᏾ = Ᏹ or Ᏻ and that R is a projective reﬂection in P Aut M, of order m < n. Then R is represented by a reﬂection of M. Proof. We also write R for any element of Aut M representing R. Since R acts on CP n−1 as a projective reﬂection, it has two distinct eigenvalues λ and λ , with one (say, λ) having multiplicity n − 1. Furthermore, since R m preserves M and acts trivially on CP n−1 , we see that there is a unit α of ᏾ such that λm = λ m = α. The characteristic polynomial of R is (λ − x)n−1 (λ − x), and since R ∈ GLn ᏾, the coefﬁcients must all lie in ᏾. We write y and z for the coefﬁcients of x n−1 and x n−m−1 , and compute n−1 n−1 λ + λ = (−1)n−1 y, 1 0 n−1 m n − 1 m+1 + λ λ = (−1)n−m−1 z. λ m m+1 Because λm = α ∈ ᏾, the second equation reduces to a linear equation in λ and λ . For n > m, this is a nonsingular system of equations, so λ, λ ∈ ᏾ ⊗ Q. Since λ, λ are roots of unity, they must actually lie in ᏾. Then λ−1 R ∈ Aut M has eigenvalues 1 (with multiplicity n − 1) and λ−1 λ , completing the proof. Now we consider the DM groups. If ? is a group acting on CH n , then a projective reﬂection in ? is called primitive if it is not a power of a projective reﬂection in ? of larger order. The construction of the DM groups allows us to ﬁnd primitive projective reﬂections in them. This requires a sketch of the construction, for which we use Thurston’s approach. Let n ≥ 4 and let α = (α1 , . . . , αn ) be an n-tuple of numbers in the interval (0, 2π) that sum to 4π. Let P (α) be the moduli space of pairs (p, g) where p is an injective map from {1, . . . , n} to an oriented sphere S 2

NEW HYPERBOLIC REFLECTION GROUPS

331

and g is a singular Riemannian metric on S 2 that is ﬂat except on the image of p, with p(i) being a cone point of curvature αi . We also denote p(i) by pi . Two such pairs are considered equivalent if they differ by an orientation-preserving similarity that identiﬁes the corresponding points pi with each other. This moduli space is a manifold of 2(n − 3) real dimensions and admits a metric that is locally isometric to CH n−3 . Let H be the group of elements σ of the symmetric group Sn satisfying ασ (i) = αi for all i = 1, . . . , n. Then H acts by isometries of P (α) by permuting the points pi . We denote the quotient orbifold by C(α). The fundamental group of P (α) is the pure (spherical) braid group on n strands, and the orbifold fundamental group of C(α) is the subgroup of the full (spherical) braid group that maps to H under the usual map from the braid group to the symmetric group. ¯ If the αi satisfy certain conditions, then the metric completion C(α) of C(α) turns n−3 out to be the quotient of CH by a reﬂection group ?(α). There are only 94 choices for α (with n ≥ 5) satisfying these conditions, and the corresponding ?(α) are the DM ¯ groups. The points of C(α) C(α) are the images of the mirrors of certain reﬂections of ?(α). We can ﬁgure out the orders of the primitive reﬂections associated to these ¯ mirrors by ﬁnding the cone angle at each generic point of C(α) C(α): if the cone angle is 2π/m, then the corresponding primitive projective reﬂections have order m. (This cone angle should not be confused with the cone angles at the points pi ∈ S 2 .) ¯ The generic points of C(α) C(α) are associated to “collisions” between pairs of 2 points pi and pj on S for which αi +αj < 2π. We quote Thurston’s Proposition 3.5, ¯ C(α). which provides a way to compute the cone angles at these points of C(α) ¯ 1 , . . . , αn ) where two cone points of Proposition 8.3. Let S be the stratum of C(α 2 S of curvature αi and αj collide. If αi = αj , then the cone angle around S is π −αi ; otherwise it is 2π − αi − αj . For example, take α to be the 10-tuple (2π/3, 2π/3, π/3, π/3, π/3, π/3, π/3, π/3, π/3, π/3), which is number 13 on Thurston’s list and number 4 on Mostow’s. Then at ¯ the singular strata of C(α) where 2 cone points of curvature 2π/3 (resp., 2 of curvature π/3, resp., 1 of each curvature) collide, the cone angle is π − (2π/3) = π/3 (resp., π − (π/3) = 2π/3, resp., 2π − (2π/3) − (π/3) = π). We deduce that ?(α) contains primitive projective reﬂections of orders 6, 3, and 2. Ᏹ Ᏻ Theorem 8.4. If L is In,1 (n ≥ 4) or I I4n+1,1 (n ≥ 1), then Reﬂec L does not appear among the Deligne-Mostow groups.

Proof. By Lemmas 8.1 and 8.2, P Aut L contains no primitive projective reﬂections of order 3 or 4. Also, Aut L is not cocompact because L contains isotropic vectors. Turning to the DM groups, Proposition 8.3 and the list of n-tuples α provided in [23] or [24] make it easy to compute the cone angles at all the generic points of ¯ C(α) C(α) for each n-tuple α with n ≥ 7. The author wrote a short computer program to do this and also performed the computation by hand. The only one for which none of the cone angles are 2π/4 or 2π/3 is number 50 on Thurston’s list (number 21 on Mostow’s). According to Thurston’s table, ?(α) is cocompact for this choice

332

DANIEL ALLCOCK

of α. Therefore, each DM groups acting on CH n for n ≥ 4 is either cocompact or contains a primitive projective reﬂection of order 3 or 4. The theorem follows. Ᏹ We close by sketching a proof that Reﬂec I3,1 is one of the DM groups; it is the group ?(α) with α = (2π/3, 2π/3, 2π/3, 2π/3, 2π/3, 2π/3), which is number 1 on Thurston’s list and number 23 on Mostow’s. Because all the αi are equal, the orbifold fundamental group of C(α) is the spherical braid group B6 on 6 strands. A standard generator for B6 , braiding 2 points pi and pi+1 , corresponds to a loop in C(α) ¯ encircling the singular stratum S of C(α) associated to a collision between pi and pi+1 . Since the cone angle at S is π/3, we ﬁnd that the standard generators map to 6fold reﬂections. This fact, together with the braid relations and the fact that the image of B6 is not ﬁnite, speciﬁes the representation uniquely up to complex conjugation. The 5 standard generators may be taken to map to (−ω)-reﬂections in short roots of Ᏹ I3,1 , which are orthogonal if the corresponding braid generators commute and have inner product +1 otherwise. We may then use the techniques of Sections 5 and 6 Ᏹ . The arguments we have sketched here to show that the image of B6 is Reﬂec0 I3,1 concerning the braid group representation are carried out in detail in [4].

References [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11]

[12] [13] [14] [15] [16]

D. Allcock, Reﬂection groups on the octave hyperbolic plane, J. Algebra 213 (1999), 467–498. , The Leech lattice and complex hyperbolic reﬂections, to appear in Invent. Math. D. Allcock, J. Carlson, and D. Toledo, A complex hyperbolic structure for moduli of cubic surfaces, C. R. Acad. Sci. Paris Sér. I Math. 326 (1998), 49–54. , The complex hyperbolic geometry of the moduli space of cubic surfaces, preprint, 2000. C. Bachoc, Voisinage au sens de Kneser pour les réseaux quaternioniens, Comment. Math. Helv. 70 (1995), 350–374. C. Bachoc and G. Nebe, Classiﬁcation of two genera of 32-dimensional lattices of rank 8 over the Hurwitz order, Experiment. Math. 6 (1997), 151–162. R. E. Borcherds, Automorphism groups of Lorentzian lattices, J. Algebra 111 (1987), 133–153. , Lattices like the Leech lattice, J. Algebra 130 (1990), 219–234. A. Borel and Harish-Chandra, Arithmetic subgroups of algebraic groups, Ann. of Math. (2) 75 (1962), 485–535. J. H. Conway, The automorphism group of the 26-dimensional even unimodular Lorentzian lattice, J. Algebra 80 (1983), 159–163. J. H. Conway, R. T. Curtis, S. P. Norton, R. A. Parker, and R. A. Wilson, Atlas of Finite Groups: Maximal Subgroups and Ordinary Characters for Simple Subgroups, Oxford Univ. Press, Oxford, 1985. J. H. Conway, R. A. Parker, and N. J. A. Sloane, The covering radius of the Leech lattice, Proc. Roy. Soc. London Ser. A 380 (1982), 261–290. J. H. Conway and N. J. A. Sloane, Laminated lattices, Ann. of Math. (2) 116 (1982), 593–620. , The Coxeter-Todd lattice, the Mitchell group, and related sphere packings, Math. Proc. Cambridge Philos. Soc. 93 (1983), 421–440. , Sphere Packings, Lattices and Groups, Grundlehren Math. Wiss. 290, Springer, New York, 1988. H. S. M. Coxeter, “Factor groups of the braid groups” in Proceedings of the Fourth Canadian Mathematical Congress (Banff, 1957), Toronto Univ. Press, 1959, 95–122.

NEW HYPERBOLIC REFLECTION GROUPS [17] [18] [19] [20] [21] [22] [23] [24]

[25] [26] [27]

333

H. S. M. Coxeter and W. O. J. Moser, Generators and Relations for Discrete Groups, 4th ed., Ergeb. Math. Grenzgeb. (3) 14, Springer, Berlin, 1980. P. Deligne and G. D. Mostow, Monodromy of hypergeometric functions and nonlattice integral monodromy, Inst. Hautes Études Sci. Publ. Math. 63 (1986), 5–89. √ W. Feit, Some lattices over Q( −3), J. Algebra 52 (1978), 248–263. J. Milnor and D. Husemoller, Symmetric Bilinear Forms, Ergeb. Math. Grenzgeb. (3) 73, Springer, New York, 1973. G. D. Mostow, On a remarkable class of polyhedra in complex hyperbolic space, Paciﬁc J. Math. 86 (1980), 171–276. , Generalized Picard lattices arising from half-integral conditions, Inst. Hautes Études Sci. Publ. Math. 63 (1986), 91–106. , On discontinuous action of monodromy groups on the complex n-ball, J. Amer. Math. Soc. 1 (1988), 555–586. W. P. Thurston, “Shapes of polyhedra and triangulations of the sphere” in The Epstein Birthday Schrift, Geom. Topol. Monogr. 1, Geom. Topol., Coventry, 1998, 511–549 (electronic). É. B. Vinberg, The groups of units of certain quadratic forms (in Russian), Mat. Sb. (N.S.) 87 (1972), 18–36; English translation in Math. USSR-Sb. 87 (1972), 17–35. , On unimodular integral quadratic forms, Funct. Anal. Appl. 6 (1972), 105–111. É. B. Vinberg and I. M. Kaplinskaja, On the groups O18,1 (Z) and O19,1 (Z) (in Russian), Dokl. Akad. Nauk SSSR 238 (1978), 1273–1275; English translation in Soviet Math. Dokl. 19 (1978), 194–197.

Department of Mathematics, Harvard University, 1 Oxford Street, Cambridge, Massachusetts 02138, USA; [email protected]

Vol. 103, No. 2

DUKE MATHEMATICAL JOURNAL

© 2000

HOLOMORPHIC CURVES AND HAMILTONIAN SYSTEMS IN AN OPEN SET WITH RESTRICTED CONTACT-TYPE BOUNDARY DAVID HERMANN

Contents 1. Introduction and main results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 335 2. Open sets with restricted contact-type boundary . . . . . . . . . . . . . . . . . . . . . . . . . . 340 2.1. Deﬁnitions and notation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 340 2.2. Gromov width . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 342 2.3. Pseudoconvexity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 344 3. Symplectic homology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 347 3.1. Trajectory spaces, compactness, and transversality . . . . . . . . . . . . . . . . . . . 347 3.2. Floer homology and its functorial properties . . . . . . . . . . . . . . . . . . . . . . . . . 350 4. Local symplectic homology for RCT open sets . . . . . . . . . . . . . . . . . . . . . . . . . . . 354 4.1. The coﬁnal family . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 354 4.2. Generic perturbation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 357 4.3. Description of the symplectic homology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 359 5. Proof of the main results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 361 5.1. The Floer-Hofer capacity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 361 5.2. The minmax characterizations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 363 5.3. Comparison with the displacement energy . . . . . . . . . . . . . . . . . . . . . . . . . . . 365 5.4. Construction of a holomorphic curve . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 366 1. Introduction and main results. Throughout this paper, we ﬁx an integer n ≥ 2 and consider the standard symplectic space (R2n , ω = dλ0 ), with n ≥ 2 and n

λ0 =

1 xk ∧ dyk − yk ∧ dxk . 2

(1.1)

k=1

We identify R2n with Cn , setting zk = xk +iyk . To each (time-dependent) Hamiltonian H ∈ Ᏼt = C ∞ (S 1 × Cn ), we can associate a Hamiltonian vector ﬁeld given by iXH ω = −dH (t, ·),

(1.2)

where iXH ω denotes the contraction of ω by XH . The ﬂow φtH of XH is called the Hamiltonian ﬂow of H and is a symplectic isotopy. We also denote by Ᏸ the group Received 20 November 1998. Revision received 16 August 1999. 2000 Mathematics Subject Classiﬁcation. Primary 53D40; Secondary 32Q65. 335

336

DAVID HERMANN

of compactly supported Hamiltonian diffeomorphisms Ᏸ = φ1H /H ∈ C0∞ S 1 × Cn .

(1.3)

The symplectic size of subsets of R2n is measured by symplectic capacities, introduced by Gromov in [12], and developed by Ekeland and Hofer in [5]. Deﬁnition 1.1. A symplectic capacity on (R2n , ω) associates to each subset U of R2n a number c(U ) ∈ [0, ∞] satisfying U ⊂ V ⇒ c(U ) ≤ c(V ) (monotonicity), c φ(U ) = c(U ) for φ ∈ Ᏸ (symplectic invariance),

c(αU ) = α 2 c(U ) for α ∈ R (homogeneity), c B 2n (1) = c B 2 (1) × Cn−1 = π (nontriviality and normalization). Several symplectic capacities can be deﬁned using different approaches. (For a nice introduction to these subjects, see [25], [16], and [23].) However, they are in general not computable, and hence the interest in obtaining estimates for them. We focus on three of these capacities. The ﬁrst one is the symplectic width of Gromov (see [12], [13]) deﬁned as w(U ) = sup sup inf u∗ ω, (1.4) x∈U J ∈᏶ u∈Hol(J,U,x) S

where ᏶ denotes the set of almost complex structures on R2n calibrated by ω (meaning that gJ (u, v) = ω(u, J v) deﬁnes a Riemannian metric on R2n ), and Hol(J, U, x) denotes the space of open J -holomorphic curves proper in U and going through x. Here U has to be a connected and bounded open set, but standard procedures allow us to extend such a capacity to all subsets of R2n . We also consider Hofer’s displacement energy, deﬁned in [15] as follows. Deﬁnition 1.2. Let φ ∈ Ᏸ. The energy of φ is the number e(φ) = inf max H − min H /H ∈ Ᏼt satisﬁes φ1H = φ . The displacement energy of U is the number d(U ) = inf e(φ)/φ ∈ Ᏸ satisﬁes φ(U ) ∩ U = ∅ . Finally, we denote by cFH the Floer-Hofer capacity. This capacity is deﬁned via symplectic homology (see [11], [8]). The Floer-Hofer capacity can be viewed as a variant of Ekeland-Hofer capacity cEH deﬁned in [5]: It is related with periodic orbits of Hamiltonian systems. Given a connected and bounded open set U , we

HOLOMORPHIC CURVES AND HAMILTONIAN SYSTEMS

337

consider “admissible” Hamiltonians H ∈ Ᏼt which are negative near U and quadratic at inﬁnity, that is, the class     H ∈ Ᏼt such that H (t, z) < 0 for z ∈ U   Ᏼad (U ) = . (1.5) and there exist R and µ > 0 with     H (t, z) = µ|z|2 for |z| > R Each of these Hamiltonians deﬁne on the loop space an action functional 1 ∗ AH (γ ) = γ λ0 − H t, γ (t) dt for γ ∈ ( = C ∞ S 1 , Cn , S1

(1.6)

0

whose critical points are 1-periodic orbits of XH and are denoted by ᏼ(H ). In order to deﬁne both capacities cEH and cFH , we ﬁrst select a critical value c(H ) of the action functional AH for each admissible Hamiltonian H . This critical value is obtained by a variational process, which is based on the Lyapounov-Schmidt reduction in the case of the capacity cEH and on Floer homology in the case of the capacity cFH . Then we can deﬁne the capacity of U as cFH (U ) =

inf

H ∈Ᏼad (U )

c(H ).

(1.7)

Our goal is to compare the capacities cFH , w, and d. We therefore have to locate 1-periodic orbits of admissible Hamiltonians. This can be done when the open set U has restricted contact-type boundary, as introduced by Weinstein long ago as a symplectically invariant generalization of convexity. Deﬁnition 1.3. A connected hypersurface ) is said to be of restricted contact-type (or RCT) if there exists a globally deﬁned Liouville vector ﬁeld transversal to ), that is, a vector ﬁeld η satisfying η ) and Lη ω = ω on R2n , where L denotes the Lie derivative. A bounded open set with RCT boundary is called an RCT open set. Convex domains (or more generally star-shaped domains) have RCT boundary, because η0 (z) = (1/2)z is Liouville. It can be shown (see [27], [14]) that from the point of view of symplectic capacities, convexity is much more restrictive than the RCT condition: all symplectic capacities are equivalent for convex sets, and this is not the case for RCT open sets. We can now state our main result. Theorem 1.4. If U is an RCT open set, then w(U ) ≤ cFH (U ) ≤ d(U ). In order to give an idea behind the proof of Theorem 1.4, we need to give a more precise deﬁnition of the capacity cFH . In [8], Floer homology is deﬁned as a Morse theory for the action functional AH on the loop space ( in the sense of ThomSmale-Witten (see [20]). Considering a time-dependent, almost complex structure J ∈ ᏶t = C ∞ (S 1 , ᏶) and an admissible Hamiltonian H ∈ Ᏼad (U ), all of whose periodic orbits are nondegenerate (we call such an H a regular Hamiltonian), we study

338

DAVID HERMANN

the following elliptic partial differential equation (PDE), whose solutions are called Floer trajectories, and correspond to gradient lines in Morse theory (time dependence of J is needed for transversality reasons): ∂u ∂u + J (t, u) = J (t, u)XH (t, u) for u ∈ C ∞ R × S 1 , Cn . ∂s ∂t

(1.8)

Combinatorics of these solutions give rise to Floer homology groups Sk[a,b[ (H, J ), obtained by considering orbits with Conley-Zehnder index equal to k and action in the interval [a, b[, which turn out to be independent of the generic J . Functorial properties of Floer homology allow us to deﬁne the symplectic homology groups Sk[a,b[ (U ) as the direct limit of Floer homology for regular admissible Hamiltonians. It can be viewed as the limit of Sk[a,b[ (Hλ , Jλ ), where Hλ is a coﬁnal family, that is, a 1-parameter family satisfying ∀K ∈ Ᏼad (U ), ∃A ∈ R such that λ > A ⇒ Hλ ≥ K.

(1.9)

In [11], the symplectic capacity cFH is deﬁned as follows. First, the symplectic homology of open balls is computed, and it is shown that [a,b[ 2n Sn+1 B (r) = Z2 for 0 < a ≤ πr 2 < b ≤ +∞, [a,b[ 2n otherwise. B (r) = 0 Sn+1 Then consider a sufﬁciently small ball B(z0 , r) ⊂ U and 0 < ε < πr 2 . The inclusion morphism [ε,b[ [ε,b[ (U ) −→ Sn+1 B(z0 , r) Z2 σUb : Sn+1 for 0 < ε < π r 2 < b turns out to depend only on b and U and to be onto for large enough b. This allows us to deﬁne the Floer-Hofer capacity of U as cFH (U ) = inf b/σUb is onto . (1.10) The ﬁrst step in the proof of Theorem 1.4 is to get a nice Floer trajectory from this deﬁnition. Following ideas of Viterbo (see [26]), we construct a regular coﬁnal family (Hλ , Jλ ) for the symplectic homology of U such that • the only 1-periodic orbits of Hλ with positive action are the critical points of Hλ in U with action near zero and the (perturbed) closed characteristics of ∂U whose actions are near their symplectic areas; • the other 1-periodic orbits of Hλ have very negative actions; • the Floer trajectories connecting orbits with positive action stay in a neighborhood of U . In this construction, it is crucial to assume that U is an RCT open set. This way we get a local description of the symplectic homology of U , which coincides indeed with Viterbo’s “intrinsic” version of symplectic homology in [26]. This shows that,

HOLOMORPHIC CURVES AND HAMILTONIAN SYSTEMS

339

for ε > 0 small enough, S∗[0,ε[ (U ) H∗+n (U, ∂U ).

(1.11)

The reason is that for action near zero, the Floer complex of (Hλ , Jλ ) equals the Thom-Smale-Witten complex of (Hλ , gJλ ). In particular, we get Sn[0,ε[ (U ) Z2 for ε small enough, the associated periodic orbit being a ﬁxed minimum z0 of Hλ . Using the functorial properties of symplectic homology, we consider the natural map iUb : Sn[0,ε[ (U ) −→ Sn[0,b[ (U ), and we show cFH (U ) = inf b/iUb (1) = 0 .

(1.12)

This implies the following minmax characterization of cFH : For large enough λ, there exists a 1-periodic orbit γλ of Hλ located near ∂U and with action near cFH (U ), and there exists a Floer trajectory uλ for (Hλ , Jλ ) satisfying and lim uλ (s, ·) = γλ in C ∞ S 1 , Cn . (1.13) lim uλ (s, ·) = z0 s→−∞

s→+∞

The next step for proving the ﬁrst inequality is an equivariant problem. If we could assume that Hλ is constant in U and that Jλ is arbitrary (and time-independent) in U , we would be done. Indeed, we could cut uλ along ∂U and get a Jλ -holomorphic curve. By the theorem on removable singularity (see [24]), this curve can be made proper, and its area is less than the action of γλ , implying the ﬁrst inequality. As (Hλ , Jλ ) has to be regular in order to get the trajectory uλ , we have to perturb this ideal situation and to show the convergence of the regular trajectory to the ideal trajectory we need. The whole problem is to control the behavior of uλ near z0 , which is done by a construction used in [10] and a rigidiﬁcation-relaxation process. The second inequality is relatively easy, and the proof is similar to the case of the capacity cEH (see [15], [23]). We show that given 0 < b < cFH (U ), a regular coﬁnal family Kλ ∈ Ᏼad (U ), and large enough λ, any Hamiltonian function Lλ satisfying Kλ − b ≤ Lλ ≤ Kλ on S 1 × Cn has a 1-periodic orbit with positive action. Let ψtλ denote the ﬂow of Kλ . Given ε > 0, by classical arguments, we can ﬁnd a Hamiltonian D ∈ C ∞ (S 1 × Cn ) satisfying φ1D (U ) ∩ U = ∅ and −d(U ) − ε < D ≤ 0 on S 1 × Cn . The Hamiltonian (1.14) Lλ (t, z) = Kλ (t, z) + D t, (ψtλ )−1 (z) has ﬂow ψtλ ◦ φtD and satisﬁes Kλ − d(U ) − ε ≤ Lλ ≤ Kλ . On the other hand, we can choose Kλ in such a way that all 1-periodic orbits of Lλ have very negative actions. Arguing by contradiction, we get the second inequality. Remark 1.5. Throughout this work, we could easily replace Cn by a symplectic manifold M with contact-type boundary, which is symplectically aspherical and

340

DAVID HERMANN

satisﬁes the strong algebraic Weinstein conjecture of [26]. Very similar results are proved in [22] for a compact aspherical symplectic manifold without boundary. Acknowledgments. I wish to thank Claude Viterbo for countless stimulating discussions. This paper was partially written during my stay at the Eidgenössische Technische Hochschule in Zurich, and I thank everyone for their warm hospitality. I also thank the participants in the seminar on symplectic geometry at the École Polytechnique for their patient listening and their comments. Finally, I wish to thank Emmanuel Giroux and Jean-Claude Sikorav for their numerous corrections and improvements to the proofs. 2. Open sets with restricted contact-type boundary. We recall very elementary properties of RCT hypersurfaces, such as symplectic conformality and pseudoconvexity, and the various deﬁnitions of Gromov width. We ﬁx also the notation we use in the whole paper. 2.1. Deﬁnitions and notation. Let U ⊂ Cn be an RCT open set, and let ) = ∂U . Let η be a Liouville vector ﬁeld transversal to ). Due to Deﬁnition 1.3, the 1-form λ = iη ω

(2.1)

∗ λ is a contact form on ), that is, σ ∧ (dσ )n−1 is a satisﬁes dλ = ω. The form σ = i) volume form. The contact ﬁeld on ) is the hyperplane ﬁeld ξ = Ker σ , on which dσ is a symplectic form. The Reeb vector ﬁeld of σ is the vector ﬁeld X on ) deﬁned by

iX dσ = 0

and

σ (X) = 1.

(2.2)

A closed orbit γ of X is called a closed characteristic of ), and the number Ꮽ(γ ) = γ ∗σ > 0 S1

is called the area of γ ; it is also its period. Notice that we also include multiple covered orbits. The action spectrum of ) is the set ᏿()) = Ꮽ(γ )/γ closed characteristic of ) . Example 2.1. The sphere S 2n−1 (R) is an RCT hypersurface with Liouville ﬁeld η0 (z) = (1/2)z, Liouville form λ0 , Reeb vector ﬁeld X0 (z) = (2/R 2 )iz, and action spectrum ᏿(S 2n−1 (R)) = {kπR 2 /k ∈ N∗ }. In the general case, it is easy to make η standard off a compact set, and we ﬁx R0 > 0 such that U ⊂ B 2n (R0 − 1)

and

η(z) = η0 (z) for |z| > R0 .

(2.3)

HOLOMORPHIC CURVES AND HAMILTONIAN SYSTEMS

341

The ﬂow ψs of η is then deﬁned on R×Cn and satisﬁes ψs∗ λ = es λ for all s. Consider ˆ = R∗+ × ), and denote by S and p the projections onto the factors. the manifold ) ˆ with the symplectic form ωˆ = d λ, ˆ where λˆ = S · p ∗ σ . The map Let us endow ) n ˆ 9 : ) → C given by 9(S, p) = ψln(S) (p) satisﬁes 9 ∗ λ = λˆ and is therefore a symplectic embedding. For any S > 0, we consider the open set US = ψln(S) (U ) and the hypersurface )S = 9({S} × )) = ∂US . Similarly, for any interval I ⊂ R+ , we use the notation U I = 9(I × )) with U [0,S[ := US . Finally, we consider the smooth function τ on U ]0,+∞[ given by τ 9(S, p) = S. √ Example 2.2. For ) = S 2n−1 (R), we get ψs (z) = es/2 z and Ua = B 2n (R a) for a > 0. On Cn \ {0}, the function τ is given by τ (z) = (1/R 2 )|z|2 . In the general case, (2.3) implies ψs (z) = es/2 z for s > 0 and z ∈ B 2n (R0 ), which implies √ Ua ⊂ B 2n (R0 a) for all a ≥ 1. (2.4) Deﬁnition 2.3. Let I ⊂ R∗+ . A Hamiltonian H ∈ Ᏼt is special in U I if there exists h ∈ C ∞ (I ) such that H (t, z) = h(τ (z)) for z ∈ U I . We denote by ᏴsI the set of such H . In this situation, the Hamiltonian vector ﬁeld of H satisﬁes XH = (h! ◦ τ )Xτ

on U I

(2.5)

with Xτ = 9∗ X. Moreover, due to (2.1) and (2.2), we have dτ (η) = λ(Xτ ) = τ.

(2.6)

Let us now consider a 1-periodic orbit γ for XH contained in U I ; it has to be contained in a level surface for H , that is, in some hypersurface )S . We infer 9 −1 (γ ) = (S, γˆ ), where γˆ satisﬁes (d/dt)γˆ = h! (S)X(γˆ ). Up to orientation, the curve γˆ is therefore a closed characteristic of ), with period |h! (S)|. Both λ and λ0 are global primitives of ω, which implies 1 γ ∗λ − H t, γ (t) dt. AH (γ ) = S1

0

ˆ via the symplectomorphism 9, we infer Computing in ) AH (γ ) = S · γˆ ∗ σ − h(S) = Sh! (S) − h(S). S1

We can summarize these computations as the following lemma.

342

DAVID HERMANN

Lemma 2.4. Any 1-periodic orbit γ of H = h ◦ τ in U I satisﬁes γ (S 1 ) ⊂ )S

with |h! (S)| ∈ ᏿()),

AH (γ ) = Sh! (S) − h(S).

Let us now consider a closed characteristic γ for ). As the ﬂow of X preserves ξ , the linearized ﬂow along γ deﬁnes an endomorphism LXγ of ξγ (0) . We say that γ is nondegenerate if 1 is not an eigenvalue of LXγ . If all closed characteristics are nondegenerate, then ᏿()) is discrete: ᏿()) = 0 < T0 < T1 < T2 < · · · < Tn < · · · . We say that ) has a nice action spectrum if this holds and if, moreover, each period is associated to a unique closed characteristic. The following proposition is proved in [19]. Proposition 2.5. The property of having a nice action spectrum is C ∞ -generic among hypersurfaces. Moreover, due to the symplectic invariance and homogeneity properties of any symplectic capacity c, the ﬂow ψs of η satisﬁes c ψs (P ) = es c(P ) for any P ⊂ Cn . In particular, we infer c(US ) = S · c(U ), which is referred to as symplectic conformality. This enables us to make a nondegeneracy assumption on the action spectrum of ) and to have enough room to do the constructions. Lemma 2.6. In order to prove Theorem 1.4, we can assume that ᏿()) is nice, and it sufﬁces to prove that given ε > 0, w(U1−ε ) ≤ cFH (U ) ≤ d(U1+ε ). Proof. In fact, C 1 -genericity would sufﬁce: given any RCT hypersurface ) = ∂U , ˜ with nice action spectrum in U [1−a,1+a] , for any a > 0. we can ﬁnd a C 1 -close ) ˜ has RCT, and its interior U˜ satisﬁes Since the RCT condition is C 1 -open, ) (1 − ε)w(U˜ ) ≤ cFH (U˜ ) ≤ (1 + ε)d(U˜ ) for any ε > 0. First we infer w(U˜ ) ≤ cFH (U˜ ) ≤ d(U˜ ), and then (due to monotonicity) w(U1−a ) ≤ cFH (U1+a )

and

cFH (U1−a ) ≤ d(U1+a ),

which implies Theorem 1.4. 2.2. Gromov width. Historically, the ﬁrst symplectic capacity was introduced by Gromov in [12] (see also [13], [1], and [25]). It is called the symplectic width, and its construction is based on the existence of holomorphic curves in a symplectic manifold seen as an almost complex manifold. In this section, we consider a (not necessarily compact) symplectic manifold without boundary (M, ω).

HOLOMORPHIC CURVES AND HAMILTONIAN SYSTEMS

343

Deﬁnition 2.7. An almost complex structure calibrated by ω is a ﬁber endomorphism J of T M satisfying Jx2 = −I dTx M

for all x in M

such that gJ (u, v) = ω(u, J v) is a Riemannian metric on M, that is, ω(J u, J v) = ω(u, v) for all u, v ∈ T M, ω(u, J u) > 0

for all u ∈ T M, u = 0.

We denote by ᏶M the space of calibrated almost complex structures on M. Deﬁnition 2.8. Let J ∈ ᏶M , and let S be a connected Riemann surface, possibly noncompact, but without boundary. A J -holomorphic curve is a map u : S → M that satisﬁes du ◦ i = J ◦ du. Ꮽ(u) denotes its (positive) symplectic area Ꮽ(u) = u∗ ω. S

Given a connected and bounded open set U ⊂ M (bounded here means with compact closure in M), a calibrated almost complex structure J ∈ ᏶M , and a point x ∈ U , we denote by Hol(J, U, x) the space of nonconstant J -holomorphic curves through x that are properly mapped in a neighborhood of U . With this notation, we can set the following deﬁnition (see [12]). Deﬁnition 2.9. For any connected and bounded open set U ⊂ M, wM (U ) = inf a > 0/∀J ∈ ᏶M , ∀x ∈ U, ∃u ∈ Hol(J, U, x)/Ꮽ(u ∩ U ) ≤ a , or equivalently, wM (U ) = sup sup

inf

J ∈᏶M x∈U u∈Hol(J,U,x)

Ꮽ(u ∩ U ).

(2.7)

By transitivity of symplectic isotopies, it can be easily seen that for all x ∈ U , wM (U ) = sup

inf

J ∈᏶M u∈Hol(J,U,x)

Ꮽ(u ∩ U ),

because this quantity is independent of x. Moreover, it is increasing under inclusion, which makes the following exhaustion procedure natural. Deﬁnition 2.10. For any open set U ⊂ M, wM (U ) = sup wM (V )/V ⊂ U, with V open, connected, and bounded , and ﬁnally, for any subset U of M, the width is deﬁned by thickening as wM (U ) = inf wM (V )/U ⊂ V , with V open .

(2.8)

With these deﬁnitions, the width can be easily seen to be monotone, invariant

344

DAVID HERMANN

by Ᏸ, and homogeneous. The normalization is due to the existence theorems for holomorphic curves in [12], implying the following theorem. Theorem 2.11. The width wM is a symplectic capacity in M. Even for regular open sets in Cn , it is interesting to get an intrinsic capacity, that is, a quantity invariant under equidimensional symplectic embedding and not only under Ᏸ. The simplest way to do it is to make exhaustion by compact sets, that is, to deﬁne the following with the previous notation. Deﬁnition 2.12. The symplectic width of (M, ω) is w(M, ω) = wM (M). The following two important remarks have been pointed out to me by Jean-Claude Sikorav. Remark 2.13. The original deﬁnition of the width in [13] is w(M, ω) = sup

inf

J ∈᏶M u∈Hol(J,M,x)

Ꮽ(u),

that is, we consider holomorphic curves in the whole M. It is nontrivial that this deﬁnition coincides with 2.12; this is due to a theorem of Chang (see [2]), and it requires the language of currents. However, as in the previous section, symplectic conformality allows us to prove that for an RCT open set U ⊂ M, we have wU (U ) = wM (U ), that is, that intrinsic and relative width (as we deﬁned it) coincide. Moreover, the theorem by Chang shows that this is the same width as the one in [13]. Remark 2.14. We could also deﬁne a width with bounded genus. Denoting by Holk (J, U, x) the space of nonconstant proper J -holomorphic curves through x with genus at most k, we can set wk (U ) = sup

inf

J ∈᏶M u∈Holk (J,U,x)

Ꮽ(u ∩ U )

and get a decreasing sequence of capacities. As we only consider disks with holes, we show actually that, if U is of RCT in Cn , then w0 (U ) ≤ cFH (U ). 2.3. Pseudoconvexity. We recall the important connections between pseudoconvexity and contact type (see [17], [6]). We see that for certain special almost complex structures, the maximum principle applies not only to holomorphic curves, but also to Floer trajectories. Let ) be a connected hypersurface in Cn , and let J ∈ ᏶. We denote by ζ the complex tangent space to ): ζx = Tx ) ∩(J ·Tx )). This ﬁeld is naturally co-oriented and is given by a Pfaff equation ζ = Ker α, where α is a nonzero form uniquely determined up to a nonnegative function. The same is true for dα|ζ , and for the Levi form q(v) = dα(v, J v) for v ∈ ζx .

HOLOMORPHIC CURVES AND HAMILTONIAN SYSTEMS

345

Deﬁnition 2.15. ) is said to be J -convex when q is positive-deﬁnite. A crucial fact is the maximum principle (see [17], [6]). Lemma 2.16. A J -holomorphic curve cannot touch a J -convex hypersurface from the interior. It is interesting later on to know that sufﬁciently small spheres are pseudoconvex. Lemma 2.17. Let M > 0. There exists r0 > 0 such that for all J ∈ ᏶ satisfying J (0) = i and #J #C 1 (B(3r0 )) ≤ M, for all z1 ∈ B(r0 ), and for all r1 ≤ 2r0 , the sphere S(z1 , r1 ) is J -convex. The proof is an easy computation. With F (z) = |z − z1 |2 and J (z) = i + m(z), the Levi form for S(z1 , r1 ) is given by q(v) = 4|v|2 + 4 ω(v, mv) − [d(dF ◦ m)](v, J v). As #m# ≤ 3Mr0 and # dm #≤ M, an easy estimate gives the lemma. Now we go back to the case where ) = ∂U has restricted contact-type. Deﬁnition 2.18. Let I ⊂ R∗+ . We say that J ∈ ᏶t is special in U I if it is timeindependent and ψs -invariant in U I and if there exists a constant C > 0 such that J (z)η(z) = C · Xτ (z) for all z ∈ U I . We denote by ᏶Is the set of such J . ˆ is to assume that Jˆ(S∂S ) = C ·X and An equivalent deﬁnition with Jˆ = 9 ∗ J on ) 1 ˆ that J is independent on t and S in S × I × ). Since ω is J -invariant, we easily compute in U I λ = −Cdτ ◦ J = −Cα.

(2.9)

We infer that ξ = ζ is stable under J . Consequently, in order to get J ∈ ᏶Is , we can do the following (see [3]). Pick a calibrated almost complex structure Jξ on the ˆ by symplectic ﬁber bundle (ξ, dσ ) → ). Extend it to ) Jˆξ = Jξ

on ξ,

Jˆξ · S∂S = X,

and

Jˆξ · X = −S∂S

(2.10)

(i.e., we choose C = 1). Push-forward by 9 gives a calibrated almost complex structure J˜ξ on U ]0,+∞[ . After restriction to U I , we can extend it to Cn as a calibrated almost complex structure. This shows that ᏶Is is nonempty and contractible. This construction allows us to control the area of holomorphic curves in the following way. Lemma 2.19. There exists a constant C1 (Jξ ) > 0 such that for all S0 > 0 and for all J ∈ ᏶ satisfying J = J˜ξ on U [(1/2)S0 ,S0 ] , if the J -holomorphic curve u is proper

346

DAVID HERMANN

in U [(1/2)S0 ,S0 ] and touches both )(1/2)S0 and )S0 , then Ꮽ(u) ≥ C1 (Jξ )S0 . Proof. Let s0 = ln(S0 ) and consider J1 = ψs∗0 J on U [1/2,1] . It satisﬁes J1 = J˜ξ on and u1 = ψs−1 (u) is J1 -holomorphic and touches both )1/2 and ). Let 0

U [1/2,1] ,

1 r = d()1/2 , )), 3 where d is the Euclidian distance. Then there is a point z0 in the source of u satisfying B 2n (u1 (z0 ), r) ⊂ U ]1/2,1[ , and u is proper in B 2n (u1 (z0 ), r). The monotony lemma (see, for instance, [24, Proposition 4.3.1]) implies that there exists C1 > 0 depending only on r and J1 with Ꮽ(u1 ) ≥ C1 . Moreover, ψs∗0 ω = S0 ω, which proves the lemma. Moreover, given J ∈ ᏶Is , we can compute the Levi form for )S , S ∈ I as q(v) = dα(v, J v) = Cω(v, J v) > 0, which implies that )S is J -convex for S ∈ I . Example 2.20. When ) = S 2n−1 (R), J (z) = i is special in Cn \ {0}. A crucial result for us is the following version of the maximum principle. Lemma 2.21. Let I ⊂ R∗+ . If (H, J ) ∈ ᏴsI × ᏶Is , a Floer trajectory for (H, J ) such that ∂u/∂s ≡ 0 cannot touch )S from the interior, for S ∈ Int(I ). Proof. Recall that u ∈ C ∞ (R × S 1 , Cn ) is a solution of ∂u ∂u + J (t, u) = J (t, u)XH (t, u). ∂s ∂t If there were an interior tangency, the function f = τ ◦u would have a local maximum. Let us compute dd c f = (−Df )ds ∧ dt in u−1 (U I ), where J is time-independent: ∂u ∂u ∂u ∂u c d f = dτ − dt + ds = dτ − J XH − J dt + J + XH ds , ∂s ∂t ∂t ∂s and because of (2.5), (2.6), and (2.9), d c f = dτ ◦ J ◦ du − dτ J h! (τ )Xτ dt = −C −1 u∗ λ + C −1 τ h! (τ ) dt. It follows that

CDf = ω

∂ ∂u ∂u ,J + h! (τ )Xτ − τ (u)h! τ (u) , ∂s ∂s ∂s

which implies

CDf = ω

∂u ∂u ∂u ,J − τ h!! (τ ) dτ . ∂s ∂s ∂s

HOLOMORPHIC CURVES AND HAMILTONIAN SYSTEMS

As a result, on u−1 (U I ), we have CDf + g(f )

∂f = gJ ∂s

∂u ∂u , ∂s ∂s

347

≥0

with g(S) = Sh!! (S). If f had a local maximum, it would have to be constant in view of the maximum principle. But since we assume ∂u/∂s ≡ 0, [9, Lemma 4.1] implies that the set of points where ∂u/∂s = 0 is discrete. Therefore, the map f cannot be constant, which implies the lemma. 3. Symplectic homology. We recall here the construction of symplectic homology in [8], as well as certain useful results from the classical literature on this subject (mainly [3], [4], [7], [9], [11], [16], [18], [20], and [21]). For simplicity, we restrict ourselves to Z2 -homology. We focus on local convergence and transversality. 3.1. Trajectory spaces, compactness, and transversality. As mentioned earlier, Floer homology can be viewed as Morse theory for the action functional AH on the loop space (. The required metric on ( is given by a time-dependent, calibrated, almost complex structure J ∈ ᏶t : we then consider the L2 -metric on ( induced by gJ . The gradient of AH is given by ∇J AH (γ ) = J XH − J γ˙ . This does not deﬁne a ﬂow on (; nevertheless, we can consider the gradient lines as solutions of the following elliptic equation: ∂u ∂u + J (t, u) = J (t, u)XH (t, u) for u ∈ C ∞ (Z, Cn ), ∂s ∂t

(3.1)

with Z = R×S 1 , which is referred to as the Floer equation. Given two critical points x− and x+ for AH , the space of Floer trajectories connecting them is (3.2) ᏹ(x− , x+ , J, H ) = u ∈ C ∞ (Z, Cn )/(3.1) and lim u(s, ·) = x± s→±∞

(the above limit is a priori in C 1 (S 1 )). The difference of action between the ends is given by the energy of the trajectory: ∂u ∂u AH (x+ ) − AH (x− ) = gJ (3.3) , dt ds =: EJ (u), ∂s ∂s Z and it is important to notice that the function s ( → AH (u(s, ·)) is increasing (because the trajectories are gradient lines). The following result is obvious but very useful. Lemma 3.1. If u is a solution of (3.1) with EJ (u) = 0, then u depends only on t and is a periodic orbit of H . Next observe that (3.1) is invariant under translation in the s direction, which gives

348

DAVID HERMANN

rise to a free R-action on the trajectory spaces obtained by the reparametrization u ( → u(s + s0 , t). We denote by ˜ (x− , x+ , J, H ) = ᏹ(x− , x+ , H, J )/R ᏹ

(3.4)

the quotient spaces of this action. A crucial feature is that Floer trajectories can be viewed as holomorphic curves in a bigger space. Indeed, consider the following almost complex structure on Z × Cn : J˜(s,t;z) (h, k; x) = − k, h; J(t,z) x + h − kJ(t,z) XH (t, z) . Then the map u(s, ˜ t) = (s, t, u(s, t)) is a J˜-holomorphic curve: this motivates the following local compactness theorems. We ﬁrst have to stop the trajectories from going to inﬁnity in Cn . This is achieved by considering convex pairs (H, J ) in the following sense. Deﬁnition 3.2. The pair (H, J ) ∈ Ᏼt × ᏶t is said to be standard at inﬁnity if there exists R > 0 and µ > 0 with µ ∈ πZ such that H (t, z) = µ|z|2

and

J (t, z) = i

for z ∈ B 2n (R).

We denote this space by ᏴR × ᏶R . Considering U = B 2n (R), we have ᏴR × ᏶R ⊂ Ᏼs[1,+∞[ × ᏶[1,+∞[ . The maximum s principle (see Lemma 2.21) then implies the following lemma. Lemma 3.3. If (H, J ) ∈ ᏴR × ᏶R , then ᏼ(H ) ⊂ B 2n (R) and ᏹ(x, y, J, H ) ⊂ B 2n (R)

for all x, y ∈ ᏼ(H ).

It is therefore relevant to deﬁne the space of bounded trajectories as ᏹ(H, J ) = u ∈ C ∞ Z, B 2n (R) (3.1) and EJ (u) < ∞ . Elliptic regularity for the solutions of (3.1) (or, alternatively, Gromov compactness results for holomorphic curves) implies the following local compactness theorem. Theorem 3.4. Let (Hn , Jn ) ∈ ᏴR × ᏶R . Assume that (Hn , Jn ) converges to (H, J ) in C ∞ (S 1 × Cn ). Let un ∈ ᏹ(Hn , Jn ) with EJn (un ) bounded. Then there exist a ∞ (Z, Cn ). trajectory u ∈ ᏹ(H, J ) and a subsequence unp of un with unp → u in Cloc The proof in [7] (or [16]) is based on the nonexistence of holomorphic spheres and on elliptic bootstrapping argument. Under weaker regularity, the same proof leads to the following theorem. Theorem 3.5. In Theorem 3.4, if we assume that Hn is bounded in C 2 (S 1 × Cn ) and goes to H in C 1 (S 1 × Cn ), and that Jn is bounded in C 1 (S 1 × Cn ) and goes to 1 (Z, Cn ). J in C 0 (S 1 × Cn ), then we can conclude that unp → u in Cloc

HOLOMORPHIC CURVES AND HAMILTONIAN SYSTEMS

349

A direct consequence of Theorem 3.4 is the following. Lemma 3.6. For a given u ∈ ᏹ(H, J ), there exist γ± ∈ ᏼ(H ) and a sequence sn ∈ R with lim sn = ±∞

n→±∞

lim u(sn , ·) = γ±

and

n→±∞

in C ∞ (S 1 ).

Moreover, we have EJ (u) = AH (γ+ ) − AH (γ− ). See, for instance, [10] for a proof. We call the orbits γ+ and γ− the ends of the trajectory u. Notice that, in general, these ends have no reason to be unique, because they are deﬁned in a weak sense. In order to prove the existence of ends of bounded trajectories in a stronger sense, we have to make the following generic assumption: A 1-periodic orbit γ ∈ ᏼ(H ) is said to be nondegenerate if 1 is not an eigenvalue of the linearized ﬂow of XH along γ . In this situation, a Floer trajectory has ends in the following strong sense. Lemma 3.7. If γ+ (resp., γ− ) is a nondegenerate 1-periodic orbit of H , then lim u(s, t) = γ+ (resp., lim u(s, t) = γ− ) in C ∞ (S 1 ).

s→+∞

s→−∞

See [7] or [16] for a proof. If the orbits x+ and x− are nondegenerate, they have well-deﬁned Conley-Zehnder indices µ(x± , H ) (see [21]). In fact, our sign conventions lead us to consider the opposite of the index in [21]. Their computation of the Fredholm index of (3.2) gives the virtual dimension of the trajectory spaces. Proposition 3.8. If x+ and x− are nondegenerate periodic orbits, then we have dimv ᏹ(x− , x+ , J, H ) = µ(x+ , H ) − µ(x− , H ). Now let H be an autonomous Hamiltonian, and let Fix(H ) be its critical points. A point z ∈ Fix(H ) is a nondegenerate periodic orbit if it is a Morse critical point and if the second differential of H at z is sufﬁciently small. Let ᏼ∗ (H ) = ᏼ(H ) \ Fix(H ). As S 1 acts freely on ᏼ∗ (H ), these orbits have to be degenerate. In this situation, γ is called transversally nondegenerate if the eigenvalue 1 of the linearized ﬂow of XH along γ has multiplicity 1. In this case, γ is isolated, and the following perturbation lemma is proved in [4]. Lemma 3.9. Let H ∈ C ∞ (Cn ) be a Hamiltonian such that each 1-periodic orbit γ ∈ ᏼ∗ (H ) is transversally nondegenerate. Let V be a tubular neighborhood of the images of ᏼ∗ (H ). There exists a function k with support in S 1 × V such that for all δ < 1, each γ ∈ ᏼ∗ (H ) splits into two nondegenerate orbits γ + and γ − of Kδ (t, x) = H (x) + δk(t, x) satisfying γ± ⊂ V, AKδ γ ± = AH (γ ) ∓ δ, and µ(γ + , Kδ ) = µ(γ − , Kδ ) + 1, without creating other 1-periodic orbits.

350

DAVID HERMANN

This is essentially achieved by adding a Morse function on the S 1 factor. Now consider H ∈ ᏴR and J1 ∈ ᏶R , and assume that x− , x+ ∈ ᏼ(H ) are nondegenerate. It is proved in [9] that for a residual set ᏶reg (H ) ⊂ ᏶R of almost complex structures J , the space ᏹ(x− , x+ , J, H ) is a manifold of dimension given by Proposition 3.8. However, we want to keep the initial J1 ﬁxed in certain regions, and we make an explicit use of [9, Remark 5.2] (see also [18]). Let V be an open set separating the periodic orbits x− and x+ such that if v ∈ C 0 [−1, 1] × S 1 , Cn satisﬁes v(±1, ·) = x± , then v −1 (V ) = ∅. Consider the space ᏶(J1 , V ) = {J ∈ ᏶t /J = J1 off S 1 ×V }. Our aim is the following result. Proposition 3.10. For J in a residual set ᏶reg (H, J1 , V , x− , x+ ) ⊂ ᏶(J1 , V ), the space ᏹ(x− , x+ , J, H ) is a manifold of dimension given by Proposition 3.8. Proof. The proof of Proposition 3.10 follows the same lines as the proof of Theorem 5.1 in [9]. We brieﬂy present the main points of the proof and indicate the necessary changes. Consider p > 2 and l > 2, and let ᏶l be the completion of ᏶R for the C l -topology. (Banach spaces must be used, but C l -genericity for all l will ﬁnally imply C ∞ -genericity.) Fix u0 ∈ C ∞ (Z, Cn ) with u0 (s, t) = x− (t) for s < −1 and u0 (s, t) = x+ (t) for s > 1, and let Ꮾ = u0 + W 1,p (Z, Cn ).

Consider the map Ᏺ : ᏶l × Ꮾ → Lp (Z, Cn ) deﬁned by Ᏺ(J, u) =

∂u ∂u + J (t, u) − J (t, u)XH (t, u). ∂s ∂t

The key point is to show that D Ᏺ(J, u) has a dense range at every zero. Choose q satisfying q −1 + p −1 = 1 and (J, u) ∈ Ᏺ−1 (0). Let η ∈ Lq (Z, Cn ) be in the annihilator of the range of D Ᏺ(J, u): we have to prove that η = 0. By unique continuation, it sufﬁces to ﬁnd a nonempty open set where η vanishes, since η satisﬁes an elliptic equation (adjoint to the linearization of Ᏺ with respect to u). In [9], the authors consider the set ᏾(u) of regular points of u (the points z ∈ Z satisfying (∂u/∂s)(z) = 0 and u−1 (u(z)) = {z}). They show that this set is open and dense, and that η vanishes on it. In our case, we have only to replace ᏶l by the C l -completion of ᏶(J1 , V ), and their proof shows that η vanishes on ᏾(u) ∩ u−1 (V ), which is a nonempty open set by assumption. This ﬁnishes the proof of Proposition 3.10. As usual, we simply denote by ᏶reg (H ) the set of J ∈ ᏶R for which all trajectory spaces are manifolds of the right dimension. 3.2. Floer homology and its functorial properties. Let us now consider a regular pair, that is, (H, J ) ∈ ᏴR × ᏶R such that all 1-periodic orbits of H are nondegenerate and J ∈ ᏶reg (H ). In this situation, the previous results prove the following:

HOLOMORPHIC CURVES AND HAMILTONIAN SYSTEMS

351

˜ (x− , x+ , J, H ) is a compact zero-dimensional If µ(x+ , H ) − µ(x− , H ) = 1, then ᏹ manifold. It also makes sense to deﬁne ˜ (x− , x+ , J, H ) mod 2. δ(x− , x+ ) = #ᏹ The Floer complex can be deﬁned as follows. For I ⊂ R and k ∈ Z, consider the set ᏼIk (H ) of 1-periodic orbits of H with action in I and index equal to k, and let ]−∞,a[ ᏼak (H ) = ᏼk (H ) for a ∈ R. Consider the Z2 vector space they span, Cka (H ) = Z2 · x, x∈ᏼak (H )

a (H ) by the matrix δ: and deﬁne the boundary operator ∂k : Cka (H ) → Ck−1 ∂k x = δ(y, x) · y for x ∈ Cka (H ). a (H ) y∈Ck−1

For −∞ < a ≤ b ≤ +∞, consider the quotient space Ck[a,b[ (H ) = Ckb (H )/Cka (H ) and the induced operator ∂k[a,b[ (we are then considering orbits with action in [a, b[ ). The key point is the following result, due to Floer. [a,b[ = 0. Theorem 3.11. ∂ [a,b[ is a boundary operator, that is, it satisﬁes ∂k[a,b[ ◦∂k−1

˜ consist of broken This result is due to the fact that the ends of the spaces ᏹ trajectories. This makes possible the following deﬁnition. Deﬁnition 3.12. The Floer cohomology groups of the regular pair (H, J ) are [a,b[ Sk[a,b[ (H, J ) = Ker ∂k[a,b[ / Im ∂k+1 , which turns out to be independent of J ∈ ᏶reg (H ). Symplectic homology arises from functorial properties of Floer homology. Given generic pairs (H1 , J1 ) and (H2 , J2 ) in ᏴR × ᏶R such that H1 ≤ H2 on S 1 × Cn , consider a monotone homotopy connecting them, that is, functions L ∈ C ∞ (R, ᏴR ) and J˜ ∈ C ∞ (R, ᏶R ) such that L(s), J˜(s) = (H2 , J2 ) for s ≤ −s0 , L(s), J˜(s) = (H1 , J1 ) for s ≥ s0 . Moreover, we have L(s, t, z) = µ(s)|z|2 for |z| ≥ R, and we also require ∂L ≤0 ∂s

on R × S 1 × Cn

and

µ(s) ∈ πZ ⇒ µ! (s) < 0.

We now consider the equation ∂u ˜ ∂u + J (s, t, u) = J˜(s, t, u)XL(s) (t, u) for u ∈ C ∞ (Z, Cn ). ∂s ∂t

(3.5)

352

DAVID HERMANN

Again, the action AL(s) (u(s, ·)) is increasing along a solution. In a similar way as before (but more difﬁcult), we can construct a chain map between the two Floer complexes by counting the number of solutions of (3.5) going from an orbit of H2 to an orbit of H1 . This deﬁnes a monotonicity morphism m(H1 , H2 ) : Sk[a,b[ (H1 , J1 ) −→ Sk[a,b[ (H2 , J2 ),

(3.6)

which is indeed independent of the given monotone homotopy. These morphisms behave functorially; that is, m(H2 , H3 ) ◦ m(H1 , H2 ) = m(H1 , H3 ) for H1 ≤ H2 ≤ H3 . This functorial property allows us to deﬁne the symplectic homology of a nonempty bounded open set U as the direct limit of the Floer homology of regular pairs (H, J ) with H ∈ Ᏼad (U ). Deﬁnition 3.13. The symplectic homology groups associated to the open set U are Sk[a,b[ (U ) = lim Sk[a,b[ (H, J ). − → This direct limit comes together with a limit morphism M(H, J ) : Sk[a,b[ (H, J ) −→ Sk[a,b[ (U ). It is important to notice that we always require a > −∞ and that the convexity radius R is not ﬁxed. Remark 3.14. The groups S∗[a,b[ (H, J ) and the monotonicity morphisms are well deﬁned as soon as all x ∈ ᏼ[a,b] (H ) are nondegenerate and all trajectory spaces ∗ connecting them are regular. If we start with a pair (H1 , J1 ) ∈ ᏴR × ᏶R such that all x ∈ ᏼ[a,b] (H1 ) are transversally nondegenerate, Lemma 3.9 and Proposition 3.10 ∗ allow us to perturb this pair locally in order to get a pair (H2 , J2 ) such that S∗[a,b[ (H2 , J2 ) and M(H2 , J2 ) are well deﬁned. We call such a pair a regular pair for Sk[a,b[ (U ). Given −∞ < a ≤ b ≤ c ≤ +∞, we have a short exact sequence given by inclusions: 0 −→ Ck[a,b[ (H, J ) −→ Ck[a,c[ (H, J ) −→ Ck[b,c[ (H, J ) −→ 0, which gives rise to an exact triangle Da,b,c (H, J ): [a,b[ Sk[a,b[ (H, J ) −→ Sk[a,c[ (H, J ) −→ Sk[b,c[ (H, J ) −→ Sk−1 (H, J ).

This triangle commutes with the monotonicity morphisms from (3.6), which gives rise to an exact triangle Da,b,c (U ): [a,b[ Sk[a,b[ (U ) −→ Sk[a,c[ (U ) −→ Sk[b,c[ (U ) −→ Sk−1 (U ).

(3.7)

HOLOMORPHIC CURVES AND HAMILTONIAN SYSTEMS

353

Similarly, given numbers a ≤ a ! , b ≤ b! , and c ≤ c! , we have a natural map !

!

Ck[a,b[ (H, J ) −→ Ck[a ,b [ (H, J ) given by inclusions, which deﬁnes a map !

!

Sk[a,b[ (H, J ) −→ Sk[a ,b [ (H, J ).

(3.8)

This map is also compatible with monotonicity morphisms and deﬁnes a map !

!

Sk[a,b[ (U ) −→ Sk[a ,b [ (U ). This map commutes with the triangle (3.7) and gives natural homomorphisms Da,b,c (U ) −→ Da ! ,b! ,c! (U ).

(3.9)

On the other hand, for U ⊂ V , the inclusion Ᏼad (V ) I→ Ᏼad (U ) and the monotonicity morphisms in Ᏼad (U ) deﬁne an inclusion morphism ∗ : Sk[a,b[ (V ) −→ Sk[a,b[ (U ), iU,V

(3.10)

∗ ∗ = iU,V ◦ iV∗ ,W for U ⊂ V ⊂ W , and which which behaves functorially, that is, iU,W commutes with (3.9). Moreover, given φ ∈ Ᏸ, pullback by φ of pairs (H, J ) gives an isomorphism ∼

φ# : Sk[a,b[ (φ(U )) −−→ Sk[a,b[ (U ). If φ(U ) ⊂ V , the composition of φ# with the inclusion morphism gives a pullback morphism φ ∗ : Sk[a,b[ (V ) −→ Sk[a,b[ (U ), which is compatible with all previous arrows. One of the most important results is the following isotopy invariance theorem. Theorem 3.15. If ψs is a smooth path in Ᏸ such that ψs (U ) ⊂ V for all s, then we have ψs∗ = ψ0∗ . Given a regular pair (H, J ) and a constant C ≥ 0, (3.8) gives us a map σ (H, C) : Sk[a−C,b−C[ (H, J ) −→ Sk[a,b[ (H, J ). On the other hand, as AH −C = AH + C, we have an isomorphism ∼

φ(H − C, H ) : Sk[a,b[ (H − C, J ) −−→ Sk[a−C,b−C[ (H, J ).

354

DAVID HERMANN

The composition m(H ˆ − C, H ) = σ (H, C) ◦ φ(H − C, H ) gives a map m(H ˆ − C, H ) : Sk[a,b[ (H − C, J ) −→ Sk[a,b[ (H, J ).

(3.11)

By (3.6), we obtain a monotonicity map m(H − C, H ) : Sk[a,b[ (H − C, J ) −→ Sk[a,b[ (H, J ), and we now prove the following lemma. Lemma 3.16. For any constant C ≥ 0, m(H ˆ − C, H ) = m(H − C, H ). Indeed, let ρ be a smooth function with ρ(s) = 0 for s < −1 and ρ(s) = −1 for s > 1, and consider a monotone homotopy (L, J˜) where J˜(s, t, z) is a perturbation of J and L(s, t, z) = H (t, z) + C · ρ(s). By setting Lε (s, t, z) = L(ε · s, t, z)

and

J˜ε (s, t, z) = J˜(ε · s, t, z) for ε > 0,

the proof of Theorem 3.15 (see [8, Theorem 36]) shows that for small enough ε, the solutions of (3.5) correspond bijectively to Floer trajectories associated to (H, J ), implying the lemma. 4. Local symplectic homology for RCT open sets. In this section, we consider an open set U with restricted contact-type boundary ), and we assume that ᏿()) is nice. This assumption is not restrictive in view of Lemma 2.6. We construct our coﬁnal family, which is essentially the same as the one used in [26]. The construction is done in two steps. First, we construct an autonomous coﬁnal family (Hλ , Jλ ) with nice geometric properties, but with degenerate critical orbits. Then we perturb it as described in Section 3.1 in order to get a regular coﬁnal family. This can be done without destroying the geometric features of (Hλ , Jλ ). We also deduce the local description of the symplectic homology of U . 4.1. The coﬁnal √ family. We use the notation of Section 2. In particular, we have U a ⊂ B 2n (R0 a) for a ≥ 1 and ᏿()) = {0 < T0 < T1 < T2 < · · · < Tn < · · · }. µ,R We consider the Hamiltonian Hλ,A deﬁned as follows (see Figure 1): Choose √ λ ∈ ᏿()), A > 1, R > R0 3A, and µ > 0 with µ ∈ πZ and sufﬁciently small extra parameters ε > 0 and ν > 0. We only consider the case where ε −→ 0,

ν −→ 0,

R −→ +∞,

and

µ −→ +∞

as λ −→ +∞.

(4.1)

Deﬁne the numbers B=

R2 > 3A R02

and

C = −ε + (A − 1)λ,

(4.2)

HOLOMORPHIC CURVES AND HAMILTONIAN SYSTEMS

355

µ C

λ

τ (z)

1 −ε

A

B

|z|2 (R + 1)2

µ,R

Figure 1. General Hamiltonian Hλ,A

and consider smooth functions g and h on R+ with • h! (S) = λ for S ∈ [1 + ν, A], • h(S) = −ε for S ∈ [0, 1 − ν] and h(S) = C for S ≥ A + ν, • h is convex on [1 − ν, 1 + ν] and concave on [A, A + ν], • h(1) < 0 and h! (1) = (T0 /2), • g(S) = C for S < (R + 1)2 − ν and g ! (S) = µ for S > (R + 1)2 , • g is convex on [(R + 1)2 − ν, (R + 1)2 ]. µ,R As UB ⊂ B 2n (R) in view of (4.2), we can deﬁne a smooth function Hλ,A on Cn by µ,R

µ,R

• Hλ,A = h ◦ τ on U [1−ν,B] and Hλ,A (z) = g(|z|2 ) for |z| > R, µ,R

µ,R

• Hλ,A = −ε in U1−ν and Hλ,A = C in B 2n (R) \ UB . As we impose λ ∈ ᏿()), we get a number ηλ with 0 < ηλ < 1 such that ]λ − ηλ , λ] ∩ ᏿()) = ∅. µ,R

(4.3)

Lemma 2.4 shows that the 1-periodic orbits of Hλ,A are (1) constants in U1−ν , with action equal to a1 (λ) = ε, (2) closed characteristics on )Sk , Sk ∈ [1, 1 + ν], with period Tk < λ and action a2k (λ) = Sk Tk − h(Sk ) ∈ T0 , (1 + ν)λ + ε , (3) closed characteristics on )Sk! , Sk! ∈ [A, A + ν], with action less than a3 (λ, A) = (λ − ηλ )(A + ν) − (C − λν) = λ(1 + 2ν) + ε − (A + ν)ηλ ,

356

DAVID HERMANN

(4) constants in B 2n (R + 1) \ UA+ν , with action equal to a4 (λ, A) = −C, (5) closed characteristics of the sphere, with action less than a5 (λ, A, µ, R) = (R + 1)2 µ − C = (R + 1)2 µ + ε − (A − 1)λ. Moreover, it is proved in [4] that orbits of type 2 are transversally nondegenerate because h is convex there. Conditions (4.1) allow the following estimates for λ large enough: a3 (λ, A) ≤ 2λ − Aηλ ,

a4 (λ, A) ≤ a5 (λ, A, µ, R) ≤ 2R 2 µ − (A − 1)λ.

(4.4)

Now choose J1 ∈ ᏶ and ε0 > 0. We pick a calibrated almost complex structure Jξ on the symplectic bundle (ξ, dσ ) → ) that deﬁnes a special almost complex structure J˜ξ on U ]0,+∞[ by (2.10). Since UB ⊂ B 2n (R) in view of (4.2), we can consider an almost complex structure JR ∈ ᏶ satisfying JR = J 1

in U1−2ε0 ,

JR = J˜ξ

in U [1−ε0 ,B] ,

JR = i

off B 2n (R).

The following is a crucial estimate for us. µ,R

Lemma 4.1. Let (H, J ) ∈ Ᏼt × ᏶t with (H, J ) = (Hλ,A , JR ) off S 1 × U1+ν . Let x, y ∈ ᏼ(H ) ∩ U1+ν , and let u ∈ ᏹ(x, y, J, H ) such that u(Z) ⊂ U¯ 1+ν . Then AH (y) − AH (x) ≥ C1 (Jξ ) · B where C1 (Jξ ) > 0 is given by Lemma 2.19 and B is given by (4.2). Proof. As in Lemma 3.3, the maximum principle (see Lemma 2.21) ﬁrst implies that u(Z) ⊂ B 2n (R). It also implies that u(Z) ⊂ UB , because τ ◦ u can have no maximum. This implies that u must touch both )B and )B/2 by (4.2). Now consider the Riemann surface Z+ = u−1 (U ](1/2)B,B[ ). As H is constant on U ](1/2)B,B[ , the restriction of u to Z+ is a J -holomorphic curve, and Lemma 2.19 implies u∗ ω ≥ C1 (Jξ )B. Z+

It follows that

EJ (u) ≥

Z+

gJ

∂u ∂u , dt ds = u∗ ω. ∂s ∂s Z+

Combined with (3.3), this implies the lemma.

357

HOLOMORPHIC CURVES AND HAMILTONIAN SYSTEMS

We are now in position to choose the parameters. In order to get a coﬁnal family for λ → +∞, we need ε → 0, C/R 2 → +∞, and µ → +∞, which is ensured by (4.1) and (A − 1)λ −→ +∞. R2

(4.5)

The orbits of types 3, 4, and 5 have actions going uniformly to −∞ provided that Aηλ − 2λ → +∞ and (A − 1)λ − 2R 2 µ → +∞ (see (4.4)), which is ensured by (4.1) and A≥

3λ ηλ

and

1 µR 2 ≤ Aλ. 4

(4.6)

Finally, Lemma 4.1 implies that all Floer trajectories between orbits of type 1 or 2 stay in U1+ν provided that C1 (Jξ )B > (1 + ν)λ, which is ensured by R2 −→ +∞. λ

(4.7)

We therefore choose ηλ → 0 satisfying (4.3) and, for instance, A(λ) =

3λ , ηλ

R 2 (λ) = A(λ)λ1/2 ,

1 µ(λ) = λ1/2 , 4

and

ε(λ) = ν(λ) = λ−1/2 , µ(λ),R(λ)

which satisﬁes (4.1), (4.5), (4.6), and (4.7). We now set Hλ := Hλ,A(λ) Jλ := JR(λ) .

and

4.2. Generic perturbation. Given a > −∞, for large enough λ, the only orbits of Hλ with action larger than a are the orbits of type 1 and 2, that is, constants in U and closed characteristics of )S , with 1 < S < 1 + ν. Following Remark 3.14, it sufﬁces to perturb Hλ near these orbits, and Jλ in an open set separating these orbits, in order to get a regular pair for Sk[a,b[ (U ). In order to perturb orbits of type 1, choose a point z0 ∈ U1−3ε0 . Then pick a negative function f on U such that • f is a Morse function, and f = τ 2 − 1 on U [1−ε0 ,1[ , • f has a unique local minimum, attained in z0 with f (z0 ) = −1, • the other critical points for f are in U ]1−2ε0 ,1−ε0 [ . This is possible because U is connected. For 0 < α < (1/4)T0 , we can easily glue Hλ and −ε(λ) + αf to get a smooth function Hλα satisfying • Hλα = −ε(λ) + αf in U1−ε0 and Hλα = Hλ off U , • Hλα = k ◦ τ in U [1−ε0 ,1] for a convex function k. For small enough α, the only 1-periodic orbits in ᏼ[a,T0 [ (Hλα ) are the critical points

358

DAVID HERMANN

of f (see, for instance, [21]). As they are nondegenerate, this way we get a regular [a,T [ Hamiltonian for Sk 0 (U ). Moreover, following our sign conventions, their ConleyZehnder indices are given by µ x, Hλα = n − iM x, Hλα = n − iM (x, f ) for x ∈ Crit(f ),

(4.8)

where iM denotes the Morse index (see [21]). Proposition 3.10 allows us to perturb Jλ in S 1 × U ]1−2ε0 ,1−ε0 [ in order to get a regular almost complex structure Jλα for Hλα . Moreover, we can do it in such a way that the gradient of f for the associated metric is Morse-Smale (see [21]), that is, the stable and unstable manifolds intersect transversally. On the other hand, Lemma 4.1 implies that all Floer trajectories between the critical points of f stay in U1+ν . As (Hλα , Jλα ) is a special pair in U [1−ε0 ,1+ν[ , Lemma 2.21 implies further that they stay in U1−ε0 . Therefore, we get the following proposition. Proposition 4.2. For b < T0 , (Hλα , Jλα ) is a coﬁnal family of regular pairs for For large enough λ and small enough α, ᏼ[a,b[ (Hλα ) = Crit(f ). For any x, y ∈ Crit(f ) and u ∈ ᏹ(x, y, Jλα , Hλα ), we have u(Z) ⊂ U1−ε0 .

S∗[a,b[ (U ).

Next, since orbits of type 2 are transversally nondegenerate, Lemma 3.9 gives us a function kλ with support in S 1 × U ]1,1−ν[ such that these orbits split into pairs of nondegenerate orbits of α Hλ,δ = Hλα + δkλ

for small enough δ. For large enough λ and small enough α, δ, the orbits in ᏼ[a,+∞[ α ) are all nondegenerate. Critical points for f have actions near ε(λ) (we call (Hλ,δ ˜ and the perturbed closed characteristics have actions near Tk ± δ for Tk < λ them 1), ˜ Moreover, all trajectories between orbits of type 1˜ or 2˜ must cross (we call them 2). U ]1−2ε0 ,1+ν[ , and Proposition 3.10 allows us to perturb Jλ in S 1 × U ]1−2ε0 ,1+ν[ in α for H α . Finally, Lemma 4.1 order to get a regular, almost complex structure Jλ,δ λ,δ and the above computations show that the associated Floer trajectories stay in U1+ν . α , J α ) is a coﬁnal family of regular pairs for S [a,b[ (U ). Proposition 4.3. (Hλ,δ ∗ λ,δ α ) consists of orbits of For large enough λ and small enough α and δ, ᏼ[a,b[ (Hλ,δ ˜ and for any x, y ∈ ᏼ[a,b[ (H α ) and u ∈ ᏹ(x, y, J α , H α ), we have type 1˜ and 2, λ,δ λ,δ λ,δ u(Z) ⊂ U1+ν .

The same perturbations give rise to the Hamiltonian 0 Hλ,δ = Hλ + δkλ = Hλ,δ ,

for which the constant orbits in U are degenerate, but the orbits of type 2˜ (i.e., closed characteristics of ∂U with positive action) are nondegenerate.

HOLOMORPHIC CURVES AND HAMILTONIAN SYSTEMS

359

Remark 4.4. Since the perturbations depend not only on α and δ but also on λ, they are valid for α < α(λ) and δ < δ(λ). We can easily assume that α(λ) and δ(λ) go to zero for λ going to +∞, and in the remainder of this paper, “λ going to +∞” means “(λ, α, δ) going to (+∞, 0, 0).” 4.3. Description of the symplectic homology. We are now in position to get the local description of S∗[a,b[ (U ) as in [26]. We ﬁrst recall Viterbo’s intrinsic point of view on symplectic homology for an aspherical symplectic manifold (M, L) with contact-type boundary (“aspherical” means that [L] vanishes on π2 (M)). The word “intrinsic” does not mean here that it depends a priori only on the symplectic manifold (Int M, L), but rather that it depends only on the symplectic manifold with boundary (M, L) (for a really intrinsic symplectic homology, see [4]). Viterbo’s construction is indeed a cohomology rather than a homology like ours, because he studies gradient lines instead of minus-gradient lines, but this makes no real difference. With the notation of Section 2, he considers the symplectic manifold Mˆ = M [0,+∞[ , and pairs (K, J ) satisfying • K(t, z) = k∞ · τ (z) in S 1 × M [S0 ,+∞[ with k∞ ∈ ᏿(∂M), • J is special in S 1 × M [S0 ,+∞[ , • K(t, z) < 0 if z ∈ M, for which the Floer homology FH [a,b[ (K, J ) ≡ S∗[a,b[ (K, J ) is well deﬁned, because ∗ all orbits and trajectories stay in a compact set. Given two such pairs (K1 , J1 ) and ˆ he considers a map (K2 , J2 ) with K1 ≤ K2 on S 1 × M, (K1 , J1 ) −→ FH [a,b[ m(K ˆ 1 , K2 ) : FH [a,b[ (K2 , J2 ) k k

(4.9)

obtained essentially as the map m ˆ in (3.11). These maps behave functorially, which allows us to deﬁne (M, L) = lim FH [a,b[ (K, J ). FH [a,b[ k k − →

(4.10)

In our situation, we get the following proposition. Proposition 4.5. If ∂U is of RCT and has a nice action spectrum, then (U, ω) = Sk[a,b[ (U ). FH [a,b[ k α , J α ) from our coﬁnal family, and choose S in the Proof. Consider a pair (Hλ,δ 0 λ,δ interval ]1 + ν, A[. We get an admissible pair in the sense of Viterbo by α α Kλ,δ , Jˆλ,δ = λ · τ, J˜ξ in M [S0 ,+∞[ , α α α α Kλ,δ , Jˆλ,δ = Hλ,δ , Jλ,δ in M [0,S0 ] .

360

DAVID HERMANN

Due to Proposition 4.3, we get α α α α , Jλ,δ Kλ,δ , Jˆλ,δ = Sk[a,b[ Hλ,δ , FH [a,b[ k because the orbits and trajectories are the same. Moreover, the proof of Lemma 3.16 implies that the monotony maps m ˆ from (4.9) and m from (3.6) coincide. As both α , J α ) and (K α , Jˆα ) are coﬁnal, this implies the lemma. families (Hλ,δ λ,δ λ,δ λ,δ Remark 4.6. Due to Proposition 4.3, we have α α , Jλ,δ . Sk[a,b[ (U ) = lim Sk[a,b[ Hλ,δ − → Observe that given −∞ < a ≤ b < +∞, there are ﬁnitely many orbits with action in [a, b[. If we assume, in addition, that a and b do not belong to ᏿(∂U ), the actions of these orbits do not cross {a, b} for λ → +∞. This implies that this direct limit is in fact a true limit, that is, that the associated morphism [a,b[ α α α α , Jλ,δ : Sk Hλ,δ , Jλ,δ −→ Sk[a,b[ (U ) M Hλ,δ is an isomorphism for large enough λ. The same result is true for M(Hλα , Jλα ) when −∞ < a ≤ b < T0 . Recall that given −∞ < a ≤ b ≤ +∞ and −∞ < a ! ≤ b! ≤ +∞ with a ≤ a ! and b ≤ b! , (3.9) gives us a natural morphism !

!

σ : Sk[a,b[ (U ) −→ Sk[a ,b [ (U ). Proposition 4.7. If ∂U is of RCT and has a nice action spectrum, then • if [a, a ! ] and [b, b! ] do not meet ᏿(∂U ) ∪ {0}, then σ is an isomorphism, • Sk[a,b[ (U ) = 0 for −∞ < a ≤ b ≤ 0, • Sk[0,b[ (U ) = Hn+k (U, ∂U ) for 0 < b < T0 . Proof. The ﬁrst two points are direct consequences of Proposition 4.3 because orbits of type 1˜ and 2˜ have actions near ᏿(∂U )∪{0}. To prove the third point, we use Proposition 4.2. We know that ᏼ[a,b[ (Hλα ) = Crit(f ) for large enough λ and small enough α. Now ﬁx a large enough λ, and let α → 0. We know that all the Floer trajectories we have to consider stay in U1−ε0 , where Hλα = −ε(λ) + αf. In this situation, [21, Theorem 7.3] guarantees that for small enough α, these Floer trajectories are the gradient lines of −αf for the metric gJλα . This shows that the Floer complex we consider coincides up to a change of indices with the ThomSmale-Witten complex associated to (−αf, gJλα ). As the corresponding gradient ﬂow is Morse-Smale, its Morse homology groups MH l (−αf, gJλα ) are well deﬁned. As the gradient of −αf enters U near ∂U , we have MH l − αf, gJλα = Hl (U, ∂U ).

361

HOLOMORPHIC CURVES AND HAMILTONIAN SYSTEMS

The normalization (4.8) implies that for any x ∈ Crit(f ), µ x, Hλα = n − iM (x, f ) = iM (x, −αf ) − n, which implies Sk[0,b[ Hλα , Jλα = MHn+k − αf, gJλα = Hn+k (U, ∂U ), and we conclude by Remark 4.6. 5. Proof of the main results 5.1. The Floer-Hofer capacity. We recall here the results of [11] that allow us to deﬁne the capacity cFH . Let us start with an easy lemma. Lemma 5.1. Sk[α

2 a,α 2 b[

(αU ) = Sk[a,b[ (U ) for any α > 0 and U ⊂ Cn .

Indeed, if (H, J ) is a regular admissible pair for U , the pair J (t, z) = J t, α −1 z Hα (t, z) = α 2 H t, α −1 z , is a regular admissible pair for αU . From now on, Sk[a,b[ (r) = Sk[a,b[ (B 2n (r)). By studying special Hamiltonians for S 2n−1 (1), we get (see [11, Corollary 2]) the following lemma. Lemma 5.2. The symplectic homology groups of an open ball satisfy Sn[a,b[ (r) = Z2

for − ∞ < a ≤ 0 < b ≤ πr 2 ,

Sn[a,b[ (r) = 0

otherwise,

[a,b[ Sn+1 (r) = Z2

for 0 < a ≤ πr 2 < b ≤ +∞,

[a,b[ Sn+1 (r) = 0

otherwise.

Next, given 0 < r ≤ R, consider the inclusion morphism ir,R : Sn[0,ε[ (R) −→ Sn[0,ε[ (r). We then have the following lemma (see [11, pages 596 and 598]). Lemma 5.3. If 0 < ε < π r 2 , then ir,R is an isomorphism. Remark 5.4. As in [26], we can interpret ir,R as the pullback H2n B 2n (R), S 2n−1 (R) −→ H2n B 2n (r), S 2n−1 (r) . Moreover, consider the inclusion morphism [ε,b[ [ε,b[ (R) −→ Sn+1 (r). Ir,R : Sn+1

By studying the diagram given by inclusion morphism D0,ε,b (R) → D0,ε,b (r), we easily deduce the following from Lemmas 5.2 and 5.3.

362

DAVID HERMANN

Lemma 5.5. If 0 < ε < π r 2 ≤ πR 2 < b, then Ir,R is an isomorphism. These results allow us to deﬁne the capacity of any bounded and connected open set U ⊂ Cn as follows (see [11]). The natural map Sk[ε1 ,b[ (U ) −→ Sk[ε2 ,b[ (U ) for 0 < ε1 ≤ ε2 < b is functorial, which deﬁnes Sk]0,b[ (U ) = lim Sk[ε,b[ (U ). ← −

(5.1)

From Lemma 5.2, we infer ]0,b[ Sn+1 (r) = Z2

for b > πr 2 > 0.

Moreover, for 0 < π r12 < π r22 < b, the isomorphism Ir2 ,r1 from Lemma 5.5 gives us an isomorphism ]0,b[ ]0,b[ I˜r2 ,r1 : Sn+1 (r2 ) −→ Sn+1 (r1 ),

which is also functorial. The homology of a point can then be deﬁned as ]0,b[ 2n M = lim Sn+1 B (r) Z2 , − →

(5.2)

which is independent of b > 0. Now consider φ ∈ Ᏸ with φ(B 2n (r)) ⊂ U . The pullback morphism [ε,b[ [ε,b[ 2n φ ∗ : Sn+1 (U ) −→ Sn+1 B (r) gives, after taking inverse and direct limits, a morphism ]0,b[ σUb : Sn+1 (U ) −→ M,

(5.3)

which is independent of φ in view of the isotopy invariance (see Theorem 3.15). With notation, we have the following deﬁnition. Deﬁnition 5.6. The Floer-Hofer capacity of U is cFH (U ) = inf{b/σUb is onto}. It extends to any subset of Cn by exhaustion and thickening as in Section 2.2. This deﬁnes a symplectic capacity: the monotonicity of cFH is due to the functoriality of ∗ the inclusion morphisms, implying that σVb = iU,V ◦ σUb for U ⊂ V . The symplectic invariance is due to Theorem 3.15, and the homogeneity is due to Lemma 5.1. The computation in [11] of the symplectic homology of an ellipsoid n 2 z k <1 E(r1 , . . . , rn ) = z ∈ Cn r k k=1

with r1 < · · · < rn shows that cFH (E(r1 , . . . , rn )) = πr12 , implying the normalization.

HOLOMORPHIC CURVES AND HAMILTONIAN SYSTEMS

363

5.2. The minmax characterizations. In this section, we assume that ∂U is of RCT and has a nice action spectrum. As in (5.1), the natural map Sk[0,ε1 [ (U ) −→ Sk[0,ε2 [ (U ) for 0 < ε1 ≤ ε2 allows us to deﬁne Sk0 (U ) = lim Sk[0,ε[ (U ). ← − Proposition 4.7 then implies that

(5.4)

Sk0 (U ) = Hn+k (U, ∂U ).

(5.5)

The natural morphism Sn[0,ε[ (U ) −→ Sn[0,b[ (U ), after taking the inverse limit, gives us a morphism iUb : Sn0 (U ) −→ Sn[0,b[ (U )

(5.6)

with Sn0 (U ) = H2n (U, ∂U ) = Z2 . We now prove the following proposition. Proposition 5.7. If U is an RCT open set, then cFH (U ) = inf{b/iUb (1) = 0}. Proof. Due to the isotopy invariance, we can assume that B 2n (r) ⊂ U ⊂ B 2n (R). Choose ε and b with 0 < ε < πr 2 < b and consider the associated inclusion morphisms for D0,ε,b (in view of Lemma 5.2 and Proposition 4.7): [0,b[ (R) Sn+1

/ S [ε,b[ (R) n+1

/ S [0,ε[ (R) = Z 2 n

/ S [0,ε[ (U ) = Z 2 n

/ S [0,b[ (R) n

iR

[0,b[ Sn+1 (U )

/ S [ε,b[ (U ) n+1

[0,b[ Sn+1 (r) = 0

/ S [ε,b[ (r) = Z2 n+1

∂U

σUb

b iU

ir

∂r

/ S [0,ε[ (r) = Z 2 n

/ S [0,b[ (U ) n / S [0,b[ (r), n

where the lines are exact: we infer that ∂r is an isomorphism. By Lemma 5.3, we know that iR,r = ir ◦iR is an isomorphism, so ir is an isomorphism too. It results that σUb onto ⇐⇒ ∂U onto ⇐⇒ iUb (1) = 0, which ﬁnishes the proof. The following is an easy corollary of Propositions 5.7 and 4.7. Corollary 5.8. The capacity cFH satisﬁes the representation theorem: if ∂U is of RCT and has a nice action spectrum, then cFH (U ) ∈ ᏿(∂U ).

364

DAVID HERMANN

With the same hypothesis, we get the following corollary. Corollary 5.9. Let 0 < b < cFH (U ), and let Kλ ∈ Ᏼad (U ) be any regular coﬁnal family. Assume that Lλ ∈ C ∞ (S 1 × Cn ) satisﬁes Kλ − b ≤ Lλ ≤ Kλ . Then, for large enough λ, Lλ has a 1-periodic orbit with action in [0, cFH (U )]. Proof. Given Jλ ∈ ᏶reg (Kλ ) and 0 < ε < cFH (U )−b, in view of Propositions 5.7 and 4.7, we get a commutative diagram from (3.8) and (3.9), Sn[−b,ε[ (Kλ , Jλ )

σ (Kλ ,b)

Sn[−b,ε[ (U ) = Z2

/ S [0,b+ε[ (K , J ) λ λ n / S [0,b+ε[ (U ) = Z , 2 n

∼

in which the vertical arrows (which are the direct limit morphisms) are surjective for large λ. The map σ (Kλ , b) must therefore be nonzero. It follows from (3.11) that the map m(K ˆ λ − b, Kλ ) : Sn[0,b+ε[ (Kλ − b, Jλ ) −→ Sn[0,b+ε[ (Kλ , Jλ ) is nonzero. In view of Lemma 3.16, we have m(K ˆ λ −b, Kλ ) = m(Kλ −b, Kλ ), which implies that the monotonicity map m(Kλ − b, Kλ ) is nonzero. We now argue by contradiction. If Lλ had no 1-periodic orbit with action in [0, cFH (U )], the Floer homology Sn[0,b+ε[ (Lλ , Jλ ) would be well deﬁned and equal to zero. We know that m(Kλ −b, Kλ ) = m(Lλ , Kλ )◦m(Kλ −b, Lλ ): the map m(Kλ −b, Kλ ) would therefore vanish, which is a contradiction. α , J α ) we get the following corollary. Finally, for our coﬁnal family (Hλ,δ λ,δ α of type Corollary 5.10. For large enough λ, there exists a 1-periodic orbit γλ,δ α 2˜ for Hλ,δ satisfying α α α α α γ = n+1 and AHλ,δ µ γλ,δ , Hλ,δ λ,δ = c Hλ,δ α with lim c Hλ,δ = cFH (U ), λ−→+∞

α , J α , H α ). and there exists a Floer trajectory uαλ,δ ∈ ᏹ(z0 , γλ,δ λ,δ λ,δ

Proof. For 0 < ε ≤ b, we have a commutative diagram from (3.7): α α Sn[0,ε[ Hλ,δ , Jλ,δ

Sn[0,ε[ (U ) = Z2

b iλ,δ,α

b iU

/ Sn[0,b[ H α , J α λ,δ λ,δ / S [0,b[ (U ), n

HOLOMORPHIC CURVES AND HAMILTONIAN SYSTEMS

365

where the vertical arrows are isomorphisms by Remark 4.6. By Proposition 4.7, α , J α ) is generated by a critical point for f with Morse index equal to Sn[0,ε[ (Hλ,δ λ,δ zero. This point must therefore be the only minimum z0 for f . Let b α (z0 ) = 0 . = inf b/iλ,δ,α c Hλ,δ Remark 4.6 and Proposition 5.7 easily imply that α lim c Hλ,δ = cFH (U ). λ→+∞

Moreover, by deﬁnition of the Floer complex, we have α [0,b[ α α

⇒ x ∈ Cn+1 Hλ,δ , Jλ,δ /∂n+1 x = z0 b < c Hλ,δ and α [0,b[ α α

⇒ ∃x ∈ Cn+1 Hλ,δ , Jλ,δ /∂n+1 x = z0 , b > c Hλ,δ α must be of type 2 ˜ because it implying the corollary by contradiction (the orbit γλ,δ has positive action).

5.3. Comparison with the displacement energy. Here we prove the second inequality of Theorem 1.4 with the help of Corollary 5.9. Recall from Lemma 2.6 that we can assume that ᏿(∂U ) is nice, and it sufﬁces to prove that cFH (U ) ≤ d(U1+ε0 ) for all ε0 > 0.

(5.7)

Here we have to construct another coﬁnal family, more adapted to the situation. µ,R Consider again the Hamiltonian Hλ,A of Section 4.1. This time we choose A near 1. Recall from (4.1) and (4.5) that we need (A − 1)λ −→ +∞, R2

R −→ +∞,

and

µ −→ +∞

as λ −→ +∞

in order to get a coﬁnal family, and recall from (4.4) that the orbits of type 4 and 5 have actions less than a5 (λ, A, µ, R) ≤ 2R 2 µ − (A − 1)λ. We set A(λ) = 1 + λ−1/2 ,

R 2 (λ) = λ1/6 ,

and

µ(λ) = λ1/6 ,

and we choose, for instance, ε(λ) = ν(λ) = λ−1 . This way we get a coﬁnal family with lim a5 λ, A(λ), µ(λ), R(λ) = −∞. λ→+∞

366

DAVID HERMANN µ,R

As in Section 4.2, we can perturb the function Hλ,A near its orbits of types 1, 2, and µ,R

3 and get a coﬁnal family Kλ ∈ Ᏼad (U ) satisfying Kλ (t, z) = Hλ,A (z) if z ∈ UA+ν , and all 1-periodic orbits of Kλ in UA+ν are nondegenerate. Let us now prove (5.7) by contradiction. Assume that d(U1+ε0 ) < cFH (U ). We can D then choose a Hamiltonian D0 ∈ C0∞ (S 1 ×Cn ) such that φ1 0 (U1+ε0 )∩U1+ε0 = ∅ and max(D0 ) − min(D0 ) < cFH (U ). Let ρ ∈ C ∞ (R+ , [0, 1]) with ρ(t) = 1 for t ≤ 1/2 and ρ(t) = 0 for t ≥ 1. For R0 > 0, consider the function |z|2 D0 (t, z) − max(D0 ) . D(t, z) = ρ 2 R0 D

For large enough (but ﬁxed) R0 , we have φ1D = φ1 0 on U1+ε0 . Moreover, we have −cFH (U ) ≤ D ≤ 0 on S 1 × Cn and D(t, z) = 0 for |z| ≥ R0 . Denote by ψtλ the ﬂow of Kλ , and consider the function Lλ (t, z) = Kλ (t, z) + D t, (ψtλ )−1 (z) . Its ﬂow φtλ can be easily computed as φtλ = ψtλ ◦ φtD . Now consider a ﬁxed point z1 for φ1λ , and let γ (t) = φtλ (z1 ). For large enough λ, ψ1λ preserves both U1+ε0 and its complement, and φ1D (U1+ε0 ) ∩ U1+ε0 = ∅, implying that z1 ∈ U1+ε0 . Moreover, R(λ) > R0 for large enough λ. If z1 ∈ B 2n (R), then γ is a 1-periodic orbit for Kλ , ALλ (γ ) ≤ a5 (λ). If z1 ∈ B 2n (R) \ U1+ε0 , then γ is a 1-periodic orbit for D, and ALλ (γ ) = AD (γ ) − C(λ). As D is compactly supported, AD (ᏼ(D)) is bounded. This implies that for large enough λ, all 1-periodic orbits for Lλ have very negative actions. On the other hand, Corollary 5.9 implies that for large enough λ, Lλ has a 1-periodic orbit with positive action, which contradicts the previous results and proves the second inequality. 5.4. Construction of a holomorphic curve. We prove the ﬁrst inequality of Theorem 1.4 with the help of Corollary 5.10. In view of Lemma 2.6, we can assume that ∂U has a nice action spectrum, and it sufﬁces to prove that w(U1−3ε0 ) ≤ cFH (U ),

(5.8)

because ε0 in Section 4.1 is arbitrary. This estimate follows from the next proposition. Proposition 5.11. For large enough λ, there exists a 1-periodic orbit γλ,δ of type 2˜ for Hλ,δ satisfying lim sup AHλ,δ (γλ,δ ) ≤ cFH (U ), λ→+∞

and there exists a Floer trajectory uλ,δ ∈ ᏹ(z0 , γλ,δ , Jλ , Hλ,δ ).

HOLOMORPHIC CURVES AND HAMILTONIAN SYSTEMS

uλ,δ

γλ,δ

367

)1−2ε0

z0

)1−ε0

u

Figure 2. Cutting the minmax trajectory

Postponing the proof of Proposition 5.11, let us ﬁnish the proof of Theorem 1.4. Recall from Sections 4.1 and 4.2 that Hλ,δ = −ε(λ) and Jλ = J1 in U1−2ε0 , with arbitrary J1 ∈ ᏶ and z0 ∈ U1−3ε0 . By deﬁnition, we have lim uλ,δ (s, ·) = z0

s→−∞

and

lim uλ,δ (s, ·) = γλ,δ

s→+∞

in C 1 (S1 ).

Since we know that γλ,δ is off U , we can cut uλ,δ along )1−2ε0 in the following way (see Figure 2). Let ᐆ be the connected component of (uλ,δ )−1 (U1−2ε0 ) containing ] − ∞, −S0 ] × S 1 for large enough S0 . We have ∂ ∂ uλ,δ + J1 (uλ,δ ) uλ,δ = 0 ∂s ∂t and moreover,

in ᐆ,

∂ ∂ uλ,δ , uλ,δ dt ds gJ1 ∂s ∂s ᐆ ∂ ∂ ≤ gJλ uλ,δ , uλ,δ dt ds = AHλ,δ (γλ,δ ) − ε(λ). ∂s ∂s Z

The restriction u of uλ,δ to ᐆ is then a J1 -holomorphic curve, with area less than AHλ,δ (γλ,δ ) − ε(λ). It is properly mapped into U1−2ε0 , except the singularity in −∞ with lim u(s, ·) = z0

s→−∞

in C 1 (S1 ).

368

DAVID HERMANN

Let N(s, t) = e2π(s+it) for (s, t) ∈ Z. The map v = u ◦ N−1 is J1 -holomorphic on D(e−2πS0 ) \ 0 and has a continuous extension to D(e−2πS0 ) with v(0) = z0 . It is therefore holomorphic on D(e−2πS0 ) by [24, Theorem 4.5.1]. We can therefore remove the singularity of u in −∞ by considering the Riemann surface S = ᐆ \ ] − ∞, −S0 [×S 1 ∪ D e−2πS0 ∼ ᐆ ∪ {−∞}, and get a proper J1 -holomorphic curve uˆ : S → U1−2ε0 satisfying u(−∞) ˆ = z0 and Ꮽ(u) ˆ = Ꮽ(u) ≤ EJ (uλ,δ ) = AHλ,δ (γλ,δ ) − ε(λ).

Since J1 and z0 are arbitrary, this proves that w(U1−3ε0 ) ≤ AHλ,δ (γλ,δ )−ε(λ), which proves (5.8) for λ going to inﬁnity, and we get Theorem 1.4. It now remains to prove Proposition 5.11. For α > 0, Corollary 5.10 gives us α , J α , H α ). We want to obtain the trajectory u a trajectory uαλ,δ ∈ ᏹ(z0 , γλ,δ λ,δ of λ,δ λ,δ Proposition 5.11 as the limit of these trajectories for α going to zero. The whole problem is to control the behavior of uαλ,δ near z0 . Since this control is easier to obtain in a rigid situation, that is, when J1 is constant near z0 , we prove Proposition 5.11 in two steps. In the ﬁrst step, we prove Proposition 5.11 in the rigid case (see Proposition 5.12). In the second step, we prove the general case by deforming the initial almost complex structure J1 into J r ∈ ᏶, which is constant on B 2n (z0 , r). We then get the trajectory of Proposition 5.11 by taking the limit of the trajectories of Proposition 5.12 for r going to zero. Proposition 5.12. Assume that z0 = 0 and that there exists some r > 0 with J1 (z) = i on B 2n (r). Then for large enough λ, there exists a Floer trajectory uλ,δ ∈ ᏹ(0, γλ,δ , Jλ , Hλ,δ ), where γλ,δ is a 1-periodic orbit of type 2˜ for Hλ,δ satisfying lim sup AHλ,δ (γλ,δ ) ≤ cFH (U ). λ→+∞

Proof. Notice ﬁrst that up to taking a smaller r, we can assume that the function f in Section 4.2 additionally satisﬁes f (z) = −1 + |z|2

for z ∈ B 2n (r).

Choose ε > 0. Due to Corollary 5.10, for large enough λ, we get a Floer trajectory α , J α , H α ) with uαλ,δ ∈ ᏹ(0, γλ,δ λ,δ λ,δ α α α u EJλ,δ λ,δ = c Hλ,δ − ε(λ) − α ≤ cFH (U ) + ε. We ﬁx a large enough λ and a small enough δ, and we choose a sequence αn going to zero. Let αn Hn = Hλ,δ ,

J = Jλ,δ ,

αn Jn = Jλ,δ ,

αn γn = γλ,δ ,

H = Hλ,δ , n un = uαλ,δ .

HOLOMORPHIC CURVES AND HAMILTONIAN SYSTEMS

369

Then γn is off U , zero is in U1−3ε0 , and un : Z → U1+ν satisﬁes lim un (s, ·) = 0,

lim un (s, ·) = γn

s→−∞

s→+∞

in C 1 (S1 )

(5.9)

as well as ∂un ∂un + Jn (un ) = Jn (un )XHn (un ) ∂s ∂t and EJn (un ) ≤ cFH (U ) + ε. Moreover, our constructions imply that Jn (t, z) = i

and

Hn (t, z) = −ε(λ) − αn + αn |z|2

for z ∈ B 2n (r). (5.10)

Due to (5.9), we can assume up to reparametrization that for some tn ∈ S 1 , un (0, tn ) ∈ S 2n−1 (r)

and

s < 0 ⇒ un (s, t) ∈ B 2n (r).

(5.11)

Since (Hn , Jn ) → (H, J ) in C ∞ (S 1 × Cn ), Theorem 3.4 implies that up to taking a subsequence, un −→ u

∞ in Cloc (Z, Cn )

(5.12)

where u ∈ ᏹ(H, J ) satisﬁes EJ (u) ≤ cFH (U ) + ε

and

u(Z) ⊂ U1+ν .

(5.13)

Up to taking a subsequence, we can assume that tn → t∞ ∈ S 1 , and (5.11) and (5.12) imply u(0, t∞ ) ∈ S 2n−1 (r).

(5.14)

Moreover, Lemma 3.6 implies that u has ends in ᏼ(H ). However, as H is constant in U1−ν , the map u could, for instance, be constant. The key point in the proof of Proposition 5.12 is the following lemma. Lemma 5.13. We have lims→−∞ u(s, ·) = 0 in C 0 (S 1 ). Proof. If we had ∀r1 > 0, ∃S0 ∈ R

such that ∀n ∈ N, ∀s < S0 , ∀t ∈ S 1 ,

un (s, t) ∈ B 2n (r1 ),

then Lemma 5.13 would result from (5.12). Now we argue by contradiction and assume that there exist r1 > 0 and a sequence (Sn , Tn ) with Sn → −∞ such that un (Sn , Tn ) ∈ B 2n (r1 ).

370

DAVID HERMANN

Since we know that in C ∞ (S 1 ),

lim un (s, ·) = 0

s→−∞

we can assume that 0 < r1 < (1/3)r, un (Sn , Tn ) ∈ S 2n−1 (r1 ),

and

s < Sn ⇒ un (s, t) ∈ B 2n (r1 ).

(5.15)

Now consider the sequence vn (s, t) = un (s + Sn , t). Up to taking a subsequence, the same arguments as before prove that for some v ∈ ᏹ(H, J ), we have vn −→ v

∞ in Cloc (Z, Cn ).

(5.16)

Moreover, (5.11) implies that vn (] − ∞, −Sn [×S 1 ) ⊂ B 2n (r). As Sn goes to −∞, we infer that v(Z) ⊂ B 2n (r). Lemma 3.6 implies that there exist γ± ∈ ᏼ(H ) and a sequence sn ∈ R with lim sn = ±∞

lim v(sn , ·) = γ±

and

n→±∞

n→±∞

in C ∞ (S 1 ).

Since H is constant in B 2n (r), it follows that the orbits γ+ and γ− are constants, with EJ (v) = AH (γ+ ) − AH (γ− ) = 0. From Lemma 3.1 and (5.15), we infer that v is equal to a constant z1 ∈ S 2n−1 (r1 ). Given any r2 > 0, we deduce from (5.16) that for large enough n, we have vn (0, t) ∈ B 2n (z1 , r2 ) for all t ∈ S 1 . On the other hand, since vn (] − ∞, −Sn [×S 1 ) ⊂ B 2n (r), it results from (5.10) that vn satisﬁes ∂ ∂ vn + i vn = −2αn · vn ∂s ∂t

for s ≤ −Sn .

Let us consider the map wn : Z → Cn given by wn (s, t) = e2αn ·s vn (s, t), which satisﬁes, since αn > 0, lim wn (s, t) = 0

s→−∞

and

wn {0} × S 1 ⊂ B 2n (z1 , r2 ).

Moreover, due to the rigid situation, we have ∂ ∂ wn + i wn = 0 ∂s ∂t

for s ≤ −Sn .

As before, consider N(s, t) = e2π(s+it) for (s, t) ∈ Z. The map w˜ n = wn ◦ N−1 is holomorphic on D(1)\{0} and extends continuously to D(1) with w˜ n (0) = 0. By the theorem on removable singularity, it is holomorphic on D(1). On the other hand, we get w˜ n (∂D) ⊂ B 2n (z1 , r2 ). The maximum principle implies 0 = w˜ n (0) ∈ B 2n (z1 , r2 ), which contradicts r2 < r1 . This concludes the proof of Lemma 5.13.

HOLOMORPHIC CURVES AND HAMILTONIAN SYSTEMS

371

We can now ﬁnish the proof of Proposition 5.12. The crucial point is that due to (5.14) and Lemma 5.13, the trajectory u is nonconstant. Now consider the curve u˜ = u ◦ N−1 ; it is again holomorphic on D(1) \ {0} and continuous on D(1) with u(0) ˜ = 0. By the theorem on removable singularity, it is holomorphic on D(1) and therefore is smooth. Going back to Z, the curve u = u˜ ◦ N therefore satisﬁes in C ∞ (S 1 ).

lim u(s, ·) = 0

s→−∞

Let us now consider the other end of u. Lemma 3.6 implies the existence of a sequence sn ∈ R and of γ+ ∈ ᏼ(H ) with lim sn = +∞ and lim u(sn , ·) = γ+ in C ∞ (S 1 ). We know by (5.13) that ε(λ) ≤ AH (γ+ ) ≤ cFH (U ) + ε + ε(λ). ˜ The ﬁrst case It follows that γ+ is either a constant in U1−ν , or an orbit of type 2. is impossible because it would imply as before that u is constant (its energy would vanish). We are thus in the second case, and Lemma 3.7 implies in C ∞ (S 1 ),

lim u(s, ·) = γ+

s→+∞

which ﬁnishes the proof of Proposition 5.12. Proof of Proposition 5.11. In the general case, we can assume with no restriction that z0 = 0 and J1 (0) = i. Choose a function ρ ∈ C ∞ (R, [0, 1]) such that ρ(x) = 0 for x ≤ 1 and ρ(x) = 1 for x ≥ 4. Let 2 |z| J r (z) = J1 z · ρ r2 for r > 0. Then J r ∈ ᏶, and it satisﬁes Jr = i

in B 2n (r)

and

J r = J1

off B 2n (2r).

Moreover, a simple computation shows that for some constant C(J1 ) > 0, r J − J 0 ≤ C(J1 ) · r. #J r #C 1 ≤ C(J1 ) and C Denote by Jλr the almost complex structure obtained from J r in Section 4.1, and choose ε > 0. From Proposition 5.12, we get for large enough λ a Floer trajectory r , J r , H ) where γ r is a 1-periodic orbit of type 2 ˜ for Hλ,δ satisfyurλ,δ ∈ ᏹ(0, γλ,δ λ,δ λ λ,δ ing r r r EJλ,δ uλ,δ = AHλ,δ γλ,δ − ε(λ) ≤ cFH (U ) + ε. We again ﬁx a large enough λ and a small enough δ, and this time we choose a sequence rn going to zero. Let H = Hλ,δ ,

J = Jλ,δ ,

rn Jn = Jλ,δ ,

n un = urλ,δ .

372

DAVID HERMANN

For large enough n, we have #Jn #C 1 ≤ C(J )

and

Jn − J

C0

≤ C(J ) · rn .

From Lemma 2.17, we get r0 > 0 which depends only on C(J ), such that S(z1 , r1 ) is Jn -convex for all z1 ∈ B 2n (r0 ) and all r1 ≤ 2r0 .

(5.17)

We can also assume that B 2n (3r0 ) ⊂ U1−3ε0 . As before, we can assume that for some tn ∈ S 1 , un (0, tn ) ∈ S 2n−1 (r0 )

and

s < 0 ⇒ un (s, t) ∈ B 2n (r0 ).

1 (Z, Cn ), where Theorem 3.5 implies that up to taking a subsequence, un → u in Cloc u ∈ ᏹ(H, J ) again satisﬁes EJ (u) ≤ cFH (U ) + ε and u(Z) ⊂ U1+ν . We can also assume that tn → t∞ ∈ S 1 , and we get

u(0, t∞ ) ∈ S 2n−1 (r0 ).

(5.18)

As in the proof of Proposition 5.12, the key point is the following lemma. Lemma 5.14. We have lims→−∞ u(s, ·) = 0 in C 0 (S 1 ). Proof. Argue again by contradiction and assume that there exist 0 < r1 < r0 and a sequence (Sn , Tn ) with Sn → −∞ such that un (Sn , Tn ) ∈ S 2n−1 (r1 )

and

s < Sn −→ un (s, t) ∈ B 2n (r1 ).

1 (Z, Cn ) to As in Lemma 5.13, the sequence vn (s, t) = un (s + Sn , t) converges in Cloc 2n−1 1 2n (r1 ), and given any r2 > 0, we get vn ({0} × S ) ⊂ B (z1 , r2 ) for some z1 ∈ S large enough n. This time, for s ≤ −Sn , the map vn satisﬁes

∂ ∂ vn + Jn (vn ) vn = 0. ∂s ∂t We set directly v˜n = vn ◦ N−1 ; this map is Jn -holomorphic on D(1) \ {0} and continuous on D(1) with v˜n (0) = 0. By [24, Theorem 4.5.1], it is Jn -holomorphic on D(1). Moreover, it satisﬁes v˜n (D(1)) ⊂ B 2n (r0 ) and v˜n (∂D) ⊂ B 2n (z1 , r2 ). Due to (5.17), the maximum principle (see Lemma 2.16) implies that 0 = v˜n (0) ∈ B 2n (z1 , r2 ), which again contradicts r2 < r1 . This ﬁnishes the proof of Lemma 5.14. The end of the proof of Proposition 5.11 is almost identical to that of Proposition 5.12. [24, Theorem 4.5.1] implies that lim u(s, ·) = 0

s→−∞

in C ∞ (S 1 ).

As u is nonconstant due to (5.18) and Lemma 5.14, its “positive end” γ+ cannot be

HOLOMORPHIC CURVES AND HAMILTONIAN SYSTEMS

373

˜ which implies that a constant in U1−ν and has to be an orbit of type 2, lim u(s, ·) = γ+

s→+∞

in C ∞ (S 1 ),

and its action is at most cFH (U ) + ε + ε(λ), which concludes the proof. References [1]

[2] [3] [4] [5] [6]

[7] [8] [9] [10] [11] [12] [13]

[14] [15] [16] [17] [18] [19]

D. Bennequin, Problèmes elliptiques, surfaces de Riemann et structures symplectiques (d’après M. Gromov), Astérisque 145–146 (1987), 111–136, Séminaire Bourbaki, 1985/86, exp. no. 657. S. X.-D. Chang, Two-dimensional area minimizing integral currents are classical minimal surfaces, J. Amer. Math. Soc. 1 (1988), 699–778. K. Cieliebak, A. Floer, and H. Hofer, Symplectic homology, II: A general construction, Math. Z. 218 (1995), 103–122. K. Cieliebak, A. Floer, H. Hofer, and K. Wysocki, Applications of symplectic homology, II: Stability of the action spectrum, Math. Z. 223 (1996), 27–45. I. Ekeland and H. Hofer, Symplectic topology and Hamiltonian dynamics, Math. Z. 200 (1989), 355–378. Y. Eliashberg, “Filling by holomorphic discs and its applications” in Geometry of LowDimensional Manifolds (Durham, 1989), Vol. 2, London Math. Soc. Lecture Note Ser. 151, Cambridge Univ. Press, Cambridge, 1990, 45–67. A. Floer, The unregularized gradient ﬂow of the symplectic action, Comm. Pure Appl. Math. 41 (1988), 775–813. A. Floer and H. Hofer, Symplectic homology, I: Open sets in Cn , Math. Z. 215 (1994), 37–88. A. Floer, H. Hofer, and D. Salamon, Transversality in elliptic Morse theory for the symplectic action, Duke Math. J. 80 (1995), 251–292. A. Floer, H. Hofer, and C. Viterbo, The Weinstein conjecture in P × Cl , Math. Z. 203 (1990), 469–482. A. Floer, H. Hofer, and K. Wysocki, Applications of symplectic homology, I, Math. Z. 217 (1994), 577–606. M. Gromov, Pseudoholomorphic curves in symplectic manifolds, Invent. Math. 82 (1985), 307–347. , “Soft and hard symplectic geometry” in Proceedings of the International Congress of Mathematicians, Vol. 1, 2 (Berkeley, Calif., 1986), Amer. Math. Soc. Providence, 1987, 81–98. D. Hermann, Nonequivalence of symplectic capacities for open sets with restricted contact type boundary, preprint, 1998. H. Hofer, On the topological properties of symplectic maps, Proc. Roy. Soc. Edinburgh Sect. A 115A (1990), 25–38. H. Hofer and E. Zehnder, Symplectic Invariants and Hamiltonian Dynamics, Birkhäuser, Basel, 1994. D. McDuff, Symplectic manifolds with contact type boundaries, Invent. Math. 103 (1991), 651–671. D. McDuff and D. Salamon, J -Holomorphic Curves and Quantum Cohomology, Univ. Lecture Ser. 6, Amer. Math. Soc., Providence, 1994. R. C. Robinson, “A global approximation theorem for Hamiltonian systems” in Global Analysis, Proc. Sympos. Pure Math., Providence, 1970, 233–243.

374 [20] [21] [22] [23] [24] [25] [26] [27]

DAVID HERMANN D. Salamon, Morse theory, the Conley index and Floer homology, Bull. London Math. Soc. 22 (1990), 113–140. D. Salamon and E. Zehnder, Morse theory for periodic solutions of Hamiltonian systems and the Maslov index, Comm. Pure Appl. Math. 45 (1992), 1303–1360. M. Schwarz, Continuous sections in the action spectrum for closed symplectically aspherical manifolds, preprint, 1998. J. C. Sikorav, Systèmes Hamiltoniens et topologie symplectique, Dottorato Ric Mat., Dipartimento di Matematica, Università di Pisa, ETS Editrice, Pisa, 1990. , “Some properties of holomorphic curves in almost complex manifolds” in Holomorphic Curves in Symplectic Geometry, Progr. Math. 117, Birkhäuser, Basel, 1994. C. Viterbo, Capacités symplectiques et applications (d’après Ekeland-Hofer, Gromov), Astérisque 177–178 (1989), 345–362, Séminaire Bourbaki, 1988/89, exp. no. 714. , Functors and computations in Floer homology with applications, I, to appear in Geom. Funct. Anal. , Metric and isoperimetric problems in symplectic geometry, to appear in J. Amer. Math. Soc.

Institut de Mathématiques, Université Paris 7, 2 place Jussieu, 75251 Paris Cedex 05, France; [email protected]

Vol. 103, No. 3

DUKE MATHEMATICAL JOURNAL

© 2000

ONE-DIMENSIONAL SYMMETRY OF BOUNDED ENTIRE SOLUTIONS OF SOME ELLIPTIC EQUATIONS H. BERESTYCKI, F. HAMEL, and R. MONNEAU 1. Introduction. This article is devoted to the classiﬁcation of the functions u that are solutions of the semilinear elliptic equation u + f (u) = 0

in Rn

(1.1)

and that satisfy |u| ≤ 1 together with the asymptotic conditions u(x , xn )

xn →±∞

/ ±1

uniformly in x = (x1 , . . . , xn−1 ).

(1.2)

The given function f = f (u) is Lipschitz-continuous in [−1, 1]. Clearly, for (1.1), (1.2) to have a solution, f has to be such that f (±1) = 0. Here we assume furthermore that there exists δ > 0 such that f is nonincreasing on [−1, −1 + δ] and on [1 − δ, 1];

f (±1) = 0.

(1.3)

We prove that any solution u of the multidimensional equation (1.1) with the limiting conditions (1.2) has one-dimensional symmetry. Theorem 1. Let u be a solution of (1.1), (1.2) such that |u| ≤ 1. Then u(x , xn ) = u0 (xn ), where u0 is a solution of u0 + f (u0 ) = 0 in R, (1.4) u0 (±∞) = ±1, and u is increasing with respect to xn . In particular, the existence of a solution u of (1.1), (1.2) such that |u| ≤ 1 implies the existence of a solution u0 of (1.4). Lastly, this solution u is unique up to translations of the origin. For the 1-dimensional problem, we refer to [5], [11], [19], or [23]. For the low dimensions case n = 2, 3 (assuming also that f is C 1 ), the same result had been obtained by Ghoussoub and Gui [21]. Their method relies on spectral properties of some Schrödinger operators and is different from the one we use in this paper in any dimension n. We have recently learned that a result similar to Theorem 1 has been proved independently by Barlow, Bass, and Gui [7], using a very different method relying on probabilistic arguments. Let us point out that Theorem 1 is related to a more difﬁcult question, known as a conjecture of De Giorgi. Received 17 February 1999. 2000 Mathematics Subject Classiﬁcation. Primary 35J60; Secondary 35B05, 35B40, 35B50. 375

376

BERESTYCKI, HAMEL, AND MONNEAU

Conjecture (De Giorgi [17]). If u is a solution of u + u − u3 = 0 such that |u| ≤ 1 in Rn , limxn →±∞ u(x , xn ) = ±1 for all x ∈ Rn−1 , and ∂u/∂xn > 0, then there exists a vector a ∈ Rn−1 and a function u1 : R → R such that u(x , xn ) = u1 (a · x + xn ) in Rn . In the particular case where f = u−u3 , we see that this conjecture is stronger than Theorem 1 in the sense that for the conjecture of De Giorgi, the limits as xn → ±∞ are only simple in x , whereas they are uniform in x for Theorem 1. In fact, for a general nonlinearity f , the conjecture of De Giorgi has been proved in dimension n = 2 by Ghoussoub and Gui [21] (see also a presentation of Berestycki, Caffarelli, and Nirenberg [10]), and, very recently, it has been proved in dimension n = 3 by Ambrosio and Cabré [3]. See also earlier works by Modica and Mortola [24] for dimension 2 and by Caffarelli, Garofalo, and Segala [15] for general inequalities related to this problem. Recently, some new results in higher dimensions have been obtained by Farina [18] and Barlow, Bass, and Gui [7]. Farina proves one-dimensional symmetry for the solutions of (1.1) provided that they minimize a certain energy in a cylinder ω × R included in Rn . Barlow, Bass, and Gui, with probabilistic arguments, derive this symmetry result from a Liouville-type theorem, assuming monotonicity in a cone of directions. We also refer to the papers of Berestycki, Caffarelli, and Nirenberg [10] and Barlow [6] about the connection between spectral properties of Schrödinger operators and the conjecture of De Giorgi. However, the conjecture of De Giorgi, in its general form, remains open in dimensions greater than 3. Let us now turn to more general semilinear elliptic equations of the type Lu + g(xn , u) = 0

in Rn ,

(1.5)

where Lu = aij (x)∂ij u + bj (x)∂j u (here we use standard summation conventions). This operator is not necessarily selfadjoint. We assume that the coefﬁcients aij (x), bj (x) are continuous functions and that ∃c0 ≥ c0 > 0, ∀x ∈ Rn , ∀ξ ∈ Rn ,

c0 |ξ |2 ≤ aij (x)ξi ξj ≤ c0 |ξ |2 .

(1.6)

Here it is natural to ask whether the one-dimensional symmetry still holds for the solutions of (1.5), (1.2) with a general elliptic operator L instead of the Laplace operator. Nothing has been known so far about this problem, even in low dimensions. The following two theorems show that the qualitative results actually depend on the structure of the coefﬁcients aij and bj .

ONE-DIMENSIONAL SYMMETRY OF ELLIPTIC EQUATIONS

377

In the following results, g(xn , u) is required to be deﬁned and continuous on R × [−1, 1] and to satisfy the conditions g is nondecreasing in xn , ∀xn ∈ R, ∃δ > 0

(1.7)

g(xn , ±1) = 0,

(1.8)

such that (xn , s) −→ g(xn , s) is nonincreasing in s

on R × [−1, − 1 + δ] ∪ R × [1 − δ, 1], ∃C0 > 0, ∀xn ∈ R, ∀s, s˜ ∈ [−1, 1], g(xn , s˜ ) − g(xn , s) ≤ C0 |˜s − s|.

(1.9) (1.10)

We ﬁrst consider the case where the coefﬁcients aij and bj are constant; we prove the same symmetry result as in Theorem 1. Theorem 2. Assume that L and g satisfy (1.6) and (1.7)–(1.10), and assume that the coefﬁcients aij , bj , i, j = 1, . . . , n are constant. Let u be a solution of (1.5), (1.2), such that |u| ≤ 1. Then u(x , xn ) = u0 (xn ), where u0 is a solution of ann u0 + bn u0 + g(xn , u0 ) = 0 in R, (1.11) u0 (±∞) = ±1, and u is increasing with respect to xn . In particular, the existence of a solution u of (1.5), (1.2), such that |u| ≤ 1 implies the existence of a solution u0 of (1.11). Furthermore, this solution u is unique up to translations of the origin, and if g is increasing in xn , then u is unique. For general operators with nonconstant coefﬁcients, however, this symmetry property does not hold. For example, it is natural to ask if a solution of the equation u + b(x1 )∂x1 u − c∂x2 u + f (u) = 0

in R2

(1.12)

together with the uniform limiting conditions (1.2) actually satisﬁes u = u(x2 ) (and therefore the term b(x1 )∂x1 u drops). This is not the case, as the following counterexample in dimension 2 shows. Theorem 3. There exist some real numbers c, some functions f (s) fulﬁlling the assumptions of Theorem 1, and some continuous functions b(x1 ) such that the twodimensional equation (1.12), together with the uniform limiting conditions (1.2), admits both a planar solution u0 and inﬁnitely many nonplanar solutions (i.e., solutions whose level sets are not parallel lines). Remark 1.1. It is natural to ask whether the one-dimensional symmetry holds or not if the coefﬁcients of the operator only depend on xn . Recently, Alessio, Jeanjean, and Montecchiari [2] actually proved the existence of solutions that satisfy (1.2) and that do not depend on xn only, for some equations of the type a(xn )u + f (u) = 0

in Rn .

378

BERESTYCKI, HAMEL, AND MONNEAU

Lastly, whereas Theorems 1 and 2 state symmetry properties for the solutions of some elliptic equations in Rn , the following theorem, which can be proved in the same way as Theorems 1 and 2 (see Section 4), deals with the case of the half-space Rn+ = {xn > 0}. Theorem 4. Let L satisfy (1.6), and let the coefﬁcients aij , bj , i, j = 1, . . . , n be constant. Assume that the function (xn , s) → g(xn , s) is deﬁned and continuous on [0, ∞) × [0, 1] and satisﬁes g is nondecreasing in xn , ∀xn ≥ 0, ∃δ > 0

(1.13)

g(xn , 1) = 0,

such that (xn , s) −→ g(xn , s) is nonincreasing in s on [0, +∞) × [1 − δ, 1],

∃C0 > 0, ∀xn ∈ [0, +∞), ∀˜s , s ∈ [0, 1],

|g(xn , s˜ ) − g(xn , s)| ≤ C0 |˜s − s|,

g(0, 0) ≥ 0.

(1.14)

Let u ∈ C(Rn+ ) be a solution of Lu + g(xn , u) = 0

in Rn+

(1.15)

satisfying 0 ≤ u ≤ 1 together with the following boundary and limiting conditions:  u = 0 on {xn = 0}, (1.16)  lim u(x , xn ) = 1 uniformly in x = (x1 , . . . , xn−1 ) ∈ Rn−1 . xn →+∞

Then u(x , xn ) = u0 (xn ), where u0 is a solution of ann u0 + bn u0 + g(xn , u0 ) = 0 in (0, +∞),

(1.17)

u0 (0) = 0, u0 (+∞) = 1,

and u is increasing in xn . In particular, the existence of a solution u of (1.15), (1.16), such that 0 ≤ u ≤ 1, implies the existence of a solution u0 of (1.17). Lastly, this solution u is unique. The following theorem extends to more general operators and equations a result of Clément and Sweers [16], who also considered the case of uniform limits as xn → +∞. Theorem (Clément and Sweers [16]). Let f ∈ C 1,γ for some γ ∈ (0, 1) satisfy ∃ρ1 < 1 such that f (ρ1 ) = f (1) = 0 and f > 0 1 ∀ρ ∈ [0, 1), f (s) ds > 0, ∃δ > 0

ρ

such that f ≤ 0

in (1 − δ, 1).

in (ρ1 , 1),

ONE-DIMENSIONAL SYMMETRY OF ELLIPTIC EQUATIONS

379

Let u ∈ C 2 (Rn+ ) ∩ C(Rn+ ) be a solution of in Rn+

u + f (u) = 0

that satisﬁes 0 ≤ u < 1 in Rn+ together with (1.16). Then u(x , xn ) = u0 (xn ), where u0 is a solution of u0 + f (u0 ) = 0 in (0, +∞), u0 (0) = 0, u0 (+∞) = 1, and u is monotonic in xn . The method to prove this theorem is different from the one we use in this paper. It relies on comparisons with suitable one-dimensional sub- and supersolutions and on shooting-type arguments. Other problems in half-spaces have been considered by Angenent [4] and Berestycki, Caffarelli, and Nirenberg [8], [10], where no assumption is imposed on the limiting behaviour of u as xn → +∞. These symmetry results can also be thought of as extensions of the Gidas, Ni, and Nirenberg [20] symmetry result for spheres. The main device to prove Theorems 1 and 2 (and also Theorem 4) is the sliding method, which was developed by Berestycki and Nirenberg [12] and has been used in various works of Berestycki, Caffarelli, and Nirenberg [8], [9], and [10]. For another semilinear elliptic equation of the type (1.5) in Rn with conical limiting conditions, Bonnet, Hamel, and Monneau have also applied this method to state some monotonicity and uniqueness results (see [14], [22]). 2. Proof of Theorem 1. The proof uses a sliding method and a version of the maximum principle in unbounded domains. Let us start by stating the following comparison result, which directly follows from [9, Lemma 1] (based on the maximum principle). Lemma 2.1 [9]. Let f be a Lipschitz-continuous function, nonincreasing on [−1, −1 + δ] and on [1 − δ, 1] for some δ > 0. Assume that u1 , u2 are solutions of ui + f (ui ) = 0

in

and are such that |ui | ≤ 1 (i = 1, 2). Furthermore, assume that u2 ≥ u 1

on ∂

u2 ≥ 1 − δ

in

and that either

or u1 ≤ −1 + δ

in .

If ⊂ Rn is an open connected set such that Rn \ contains an inﬁnite open connected cone, then u2 ≥ u1 in .

380

BERESTYCKI, HAMEL, AND MONNEAU

Here this result is applied for domains that are half-spaces. Let us now consider a solution u of (1.1), (1.2) such that |u| ≤ 1, and let f satisfy (1.3). We are ﬁrst going to prove that u is increasing in any direction ν = (ν1 , . . . , νn ) such that νn > 0. In order to do so, for any t ∈ R, we deﬁne the function ut by ut (x) = u(x + tν). From (1.2), there exists a real a > 0 such that u(x , xn ) ≥ 1 − δ for all x ∈ Rn−1 and xn ≥ a and u(x , xn ) ≤ −1+δ for all x ∈ Rn−1 and xn ≤ −a. For any t ≥ 2a/νn , the functions u and ut are such that  t  for all x ∈ Rn−1 and for all xn ≥ −a, u (x , xn ) ≥ 1 − δ (2.1) for all x ∈ Rn−1 and for all xn ≤ −a, u(x , xn ) ≤ −1 + δ   t n−1 u (x , −a) ≥ u(x , −a) for all x ∈ R . Consequently, u and ut fulﬁll the assumptions of Lemma 2.1 in both = Rn−1 × (−∞, −a) and = Rn−1 × (−a, +∞). Therefore, it follows that ut ≥ u in Rn . Let us now decrease t. We claim that ut ≥ u for all t > 0. Indeed, deﬁne τ = inf{t > 0, ut ≥ u in Rn }. By continuity, we see that uτ ≥ u in Rn . Let us now argue by contradiction and suppose that τ > 0. Two cases may occur. Case 1. Suppose that inf

Rn−1 ×[−a,a]

(uτ − u) > 0.

(2.2)

From standard elliptic estimates, u is globally Lipschitz-continuous. Hence, there exists a real η0 small enough, which can be chosen smaller than τ , such that for all τ ≥ t > τ − η0 , we have ut (x , xn ) − u(x , xn ) > 0

for all x ∈ Rn−1

and

xn ∈ [−a, a].

(2.3)

Since u ≥ 1 − δ in Rn−1 × [a, +∞), it follows that ut (x , xn ) ≥ 1 − δ

for all x ∈ Rn−1 , xn ≥ a

and for all t > 0.

We may now apply Lemma 2.1 in the two half-spaces + = {xn > a} and − = {xn < −a}. We then infer that, for all η ∈ [0, η0 ], uτ −η (x , xn ) ≥ u(x , xn ) for all x ∈ Rn−1 and for all xn ∈ (−∞, −a)∪(a, +∞) and so for all xn ∈ R owing to (2.3). This is in contradiction with the minimality of τ . Hence (2.2) is ruled out. Case 2. Suppose that inf

Rn−1 ×[−a,a]

(uτ − u) = 0.

(2.4)

Then there exists a sequence (x k )k∈N ∈ Rn−1 ×[−a, a] such that uτ (x k )−u(x k ) → 0 as k → ∞. Set uk (x) = u(x k + x). By standard elliptic estimates and the Sobolev

ONE-DIMENSIONAL SYMMETRY OF ELLIPTIC EQUATIONS

381

injections, up to extraction of a subsequence, the functions uk approach locally a solution u∞ of (1.1) as k → ∞. We have uτ∞ (0) = u∞ (0) and uτ∞ ≥ u∞ because uτk ≥ uk for any k ∈ N. The function z = uτ∞ − u∞ satisﬁes   z + c(x)z = 0 z≥0   z(0) = 0

in Rn , in Rn ,

(2.5)

for some bounded function c(x) deﬁned by f uτ∞ (x) − f u∞ (x) c(x) = uτ∞ (x) − u∞ (x) if uτ∞ (x) = u∞ (x) and, say, c(x) = 0 if uτ∞ (x) = u∞ (x). The strong maximum principle yields that z ≡ 0. This means that u∞ (x) ≡ u∞ (x + τ ν). Letting ξ = τ ν, we see that u∞ is periodic with respect to the vector ξ . Recalling that −a ≤ xnk ≤ a, we see that the function u∞ also satisﬁes the uniform limiting conditions (1.2). Hence, since ξn > 0, the function u∞ cannot be ξ -periodic. So Case 2 with (2.4) is also ruled out. Therefore, we have proved that τ = 0. The function u is then increasing in any direction ν = (ν1 , . . . , νn ) such that νn > 0. From the continuity of ∇u, we deduce that ∂ν u ≥ 0 for any ν such that νn = 0. If νn = 0, by taking ν and −ν, we ﬁnd that ∂ν u = 0. Since this is true for all ν with νn = 0, this implies that u(x) = u(xn ). Since the solutions of (1.4) are unique up to translations, it then follows that the solutions u of (1.1), (1.2) such that |u| ≤ 1 are unique up to translations of the origin. The proof of Theorem 1 is complete. 3. More general elliptic operators. In this section, we consider solutions u with |u| ≤ 1 of more general equations Lu + g(xn , u) = 0, where L is a general linear elliptic second-order operator with no zero-order term: Lu = aij ∂ij u + bj ∂j u. We treat separately the case of constant coefﬁcients where symmetry holds (see Theorem 2) and the case of nonconstant coefﬁcients where the symmetry may be lost (see Theorem 3). 3.1. Constant coefﬁcients Proof of Theorem 2. Assume that L and g satisfy (1.6) and (1.7)–(1.10) and assume that the coefﬁcients aij , bj , i, j = 1, . . . , n, are constant. Let us consider a

382

BERESTYCKI, HAMEL, AND MONNEAU

solution u of (1.5), (1.2) such that |u| ≤ 1. As in Theorem 1, we prove that the function u depends on xn only. The scheme of the proof is similar to that of Theorem 1, apart from the fact that instead of the maximum principle stated in Lemma 2.1 for the Laplace operator, we use an extended version of the maximum principle for general second-order elliptic operators in inﬁnite slab-type domains. We prove that u is increasing in any direction ν = (ν1 , . . . , νn ) such that νn > 0. For any t ∈ R, let ut be the function ut (x) = u(x + tν). We ﬁrst observe that for all t ≥ 0, the function ut is a supersolution for (1.5). Indeed, for all t ≥ 0 and for all x ∈ Rn , we have Lut + g xn , ut = Lu(x + tν) + g xn , u(x + tν) ≤ Lu(x + tν) + g xn + tνn , u(x + tν) by (1.7) (3.1) ≤ 0. Next, as in Section 2, there exists a real a such that for any t ≥ 2a/νn ,  t  for all x ∈ Rn−1 and xn ≥ −a,  u (x , xn ) ≥ 1 − δ u(x , xn ) ≤ −1 + δ for all x ∈ Rn−1 and xn ≤ −a,   ut (x , −a) ≥ u(x , −a) for all x ∈ Rn−1 .

(3.2)

We now want to say that ut ≥ u in Rn . To this end, we use the following version of the maximum principle in inﬁnite slab-type domains for general second-order elliptic operators. Lemma 3.1. Let w be a function satisfying in = Rn−1 × (b, c),

ᏸw ≤ 0

where b, c ∈ R and where ᏸu = αij (x)∂ij u + βj (x)∂j u + γ (x)u.

Assume that the coefﬁcients αij (x), βj (x) are uniformly continuous in and that the αij satisfy (1.6). Furthermore, assume that −C ≤ γ (x) ≤ 0

for all x ∈

for some positive real number C. The function w is required to be continuous in and to satisfy ᏸw ∈ L∞ ()

and m≤w≤M for some m, M ∈ R. If w ≥ 0 on ∂, then w ≥ 0 in .

in

ONE-DIMENSIONAL SYMMETRY OF ELLIPTIC EQUATIONS

383

Postponing the proof of the above lemma, let us conclude the proof of Theorem 2. Let us ﬁrst prove that ut ≥ u in Rn−1 ×(−a, +∞) for all t ≥ 2a/νn . Set z = ut −u. Owing to (3.2), we already know that z ≥ 0 on Rn−1 ×{−a}. We now show that z ≥ 0 in Rn−1 × (−a, +∞). Due to (3.1) and (1.10), the function z satisﬁes Lz + c(x)z ≤ 0

in Rn−1 × (−a, +∞)

for some bounded function c(x) deﬁned by g xn , ut (x) − g xn , u(x) c(x) = ut (x) − u(x) if ut (x) = u(x) and, say, c(x) = 0 if ut (x) = u(x). Set γ (x) = min(c(x), 0). If x ∈ Rn−1 × (−a, +∞) is such that z(x) ≤ 0, then 1 − δ ≤ ut (x) ≤ u(x), whence, owing to (1.9), we have c(x) ≤ 0 and γ (x) = c(x). If z(x) ≥ 0, then Lz + γ (x)z ≤ Lz + c(x)z ≤ 0. Therefore, it follows that Lz + γ (x)z ≤ 0

in Rn−1 × (−a, +∞),

(3.3)

where the function γ (x) is bounded and nonnegative in Rn−1 × (−a, +∞). We now apply Lemma 3.1 in slabs of the type h = Rn−1 × (−a, h) with h > −a. Due to (1.2), there exists a function ε(h) ≥ 0 such that z(x , h) ≥ −ε(h) for all x ∈ Rn−1 and ε(h) → 0 as h → +∞. Choose any h > −a and set w = z + ε(h). The function w is bounded and, from standard elliptic estimates, it is continuous in . Setting ᏸ = L + γ (x), we have ᏸw = Lz + γ (x)z + γ (x)ε(h)

in h

≤ γ (x)ε(h) by (3.3) ≤0 since γ ≤ 0 and ε(h) ≥ 0. Furthermore, by the deﬁnition of w, ᏸw = −g xn + tνn , u(x + tν) + g xn , u(x) + γ (x)w

∈ L∞ (h )

because g, γ , and w are bounded (the boundedness of g resorts to (1.8) and (1.10)).

384

BERESTYCKI, HAMEL, AND MONNEAU

Lemma 3.1 can then be applied to the function w and the operator ᏸ in h . We have w ≥ 0 on ∂h . Therefore, it follows that w ≥ 0 in h . By passing to the limit h → +∞ and recalling that w = ut − u + ε(h), we conclude that ut (x , xn ) ≥ u(x , xn ) for all x ∈ Rn−1

and

xn ≥ −a.

and

xn ≤ −a,

Similarly, we could show that ut (x , xn ) ≥ u(x , xn ) for all x ∈ Rn−1

whence ut ≥ u in Rn . Deﬁne τ = inf{t > 0, ut ≥ u in Rn }. By arguing as in the proof of Theorem 1, it then follows that τ = 0. More precisely, if we suppose that τ > 0, then under the same notation as in the proof of Theorem 1, Case 1 is ruled out. Moreover, Case 2 is also ruled out. Indeed, if Case 2 occurs, we can then assume that up to extraction of a subsequence, xnk → x n ∈ [−a, a] and the functions uk (x) = u(x + x k ) approach a function u∞ solving Lu∞ + g(xn + x n , u∞ ) = 0

in Rn .

As we did in (3.1), the function uτ∞ satisﬁes Luτ∞ + g(xn + x n , uτ∞ ) ≤ 0. Eventually, z = uτ∞ − u∞ veriﬁes  Lz + c(x)z ≤ 0 in Rn ,  z≥0 in Rn ,   z(0) = 0 for some bounded function c. The impossibility of Case 2 then follows, as in the proof of Theorem 1, from the strong maximum principle and from the uniform limiting conditions (1.2). Hence, u is increasing in any direction ν such that νn > 0. This implies that u = u(xn ) and that u is a solution of (1.11). The same sliding method also allows us to conclude that if u(xn ) and v(xn ) are two solutions of (1.11), then there exists a real number τ such that u(xn + τ ) = v(xn ) for all xn ∈ R. The function v(xn ) then satisﬁes ann v + bn v + g(xn , v) = 0, ann v + bn v + g(xn + τ, v) = 0. Therefore, if g is increasing in xn , it follows that τ = 0, whence we get u = v. Proof of Lemma 3.1. Let ᏸ and w fulﬁll the assumptions of Lemma 3.1. Suppose that inf w = −λ < 0.

(x k )

n−1 ×(b, c) such that w(x k ) → −λ as k → Then there exists a sequence k∈N ∈ R ∞. From standard elliptic estimates, the function w is globally Lipschitz-continuous

ONE-DIMENSIONAL SYMMETRY OF ELLIPTIC EQUATIONS

385

in . Recalling that w ≥ 0 on ∂, then there exists ε > 0 such that up to extraction of a subsequence, xnk −→ x n ∈ [b + ε, c − ε] Set

as k −→ ∞.

(3.4)

k w k (x , xn ) = w x + x , xn

and αijk (x , xn ) = αij (x +x k , xn ), βjk (x , xn ) = βj (x +x k , xn ), γ k (x , xn ) = γ (x + x k , xn ) for all (x , xn ) ∈ . The functions w k satisfy αijk ∂ij w k + βjk ∂j w k ≤ −γ k w k

in

≤ −γ k w k − γ k λ

since γ k ≤ 0

and

λ≥0

k

≤ C (w + λ) since w k + λ ≥ 0 and −γ k ≤ C. Up to extraction of subsequences, from Ascoli’s theorem the functions αij , βj locally converge to some functions α ij , β j , and from standard elliptic estimates the functions wk locally approach a function w as k → +∞. By passing to the limit k → ∞, the function z = w + λ satisﬁes Mz − Cz ≤ 0

in ,

where M = α ij ∂ij + β j ∂j . Due to the deﬁnition of λ, we have z ≥ 0 in . Furthermore, from (3.4) it follows that z(0, x n ) = 0 with x n ∈ [b + ε, c − ε]. The strong maximum principle then yields that z = w+λ ≡ 0

in .

(3.5)

On the other hand, since w is globally Lipschitz-continuous, there exists a real number δ > 0 such that, say, w(x , xn ) ≥ −λ/2 for all x ∈ Rn−1 and b ≤ xn ≤ b +δ. As a consequence, z ≥ λ/2 > 0 in Rn−1 × [b, b + δ]. This is ruled out by (3.5) and the proof of the lemma is complete. Let us now observe that Theorem 2 does not hold in general if instead of the uniform limiting conditions (1.2), we only assume that u(x , xn ) → ±1 as xn → ±∞ for each x ∈ Rn−1 . Consider the equation u − c∂x2 u + f (u) = 0

in R2

(3.6)

with u(x1 , x2 ) −→ ±1 as x2 −→ ±∞,

pointwise, for all x1 ∈ R.

(3.7)

386

BERESTYCKI, HAMEL, AND MONNEAU

Let us further assume that ∂u >0 ∂x2

in R2 .

(3.8)

Here c is a constant parameter and f is some C 1 -function. The limits in (3.7) are only pointwise and are not required to be uniform. When c = 0, it follows from the result of Ghoussoub and Gui [21] that u is a function of one variable only. This does not hold for (3.6)–(3.8) as soon as c = 0. Indeed, Bonnet and Hamel [14] have constructed, for some particular function f and for some c > 0, a solution u such that  π π  u λk λ→+∞ / − 1 for all k = (cos ϕ, sin ϕ) with − − α < ϕ < − + α, 2 2 3π π  / u λk −α + 1 for all k = (cos ϕ, sin ϕ) with − + α < ϕ < λ→+∞ 2 2 for each angle α ∈ (0, π/2]. Such a solution cannot have one-dimensional symmetry (with level sets being parallel lines). This problem arises in the modelling of Bunsen burner ﬂames (see [14] and [22] for details). Therefore, from this example we learn that for some functions f (u), De Giorgi’s conjecture cannot be extended to elliptic operators with nonzero ﬁrst-order terms, even in dimension 2. 3.2. Nonconstant coefﬁcients. Our goal in this section is to prove Theorem 3. More precisely, we prove that for an equation of the type (1.12) u + b(x1 )∂x1 u − c∂x2 u + f (u) = 0

in R2

together with the limiting conditions (1.2), there exist both a solution depending on only x2 and inﬁnitely many nonplanar solutions, that is, solutions whose level sets are not parallel lines. The construction is somewhat involved and technical. It ﬁrst relies on the choice of special types of functions b(x1 ) and f . Next we construct a family of nonplanar solutions of (1.12), (1.2) that are between suitably chosen sub- and supersolutions. Let us ﬁrst state the type of b and f we consider. We choose a continuous function x1 → b(x1 ) such that for some ξ ∈ R and χ0 > 0, the function x1

y e− 0 b(s)ds dy veriﬁes χ(±∞) = ±χ0 . χ (x1 ) = (3.9) ξ

A constant function b(x1 ) ≡ b0 does not fulﬁll this condition. In contrast, all the functions of the type b(x1 ) = α tanh x1 + β (with α > |β|) or of the type b(x1 ) = αx1 + β (with α > 0 and β ∈ R) fulﬁll this condition. The function f is chosen so as to satisfy the following conditions: f ∈ C 1 ([−1, 1]), ∃θ ∈ (−1, 1)

such that f ≤ 0

f (±1) = 0,

in [−1, θ],

f ≥0

(3.10) in [θ, 1],

(3.11)

ONE-DIMENSIONAL SYMMETRY OF ELLIPTIC EQUATIONS

387

and either f ≤0

in [−1, θ],

f >0

in (θ, 1),

f <0

in (−1, θ ),

f ≥0

in [θ, 1],

1 −1

f (s) ds > 0,

(3.12)

f (s) ds < 0,

(3.13)

or

1

−1

or f <0

in (−1, θ),

f >0

in (θ, 1).

(3.14)

Furthermore, assume that if f is positive somewhere in [−1, 1], then inf

{f (v)>0}

f (v) = f (1) < 0,

(3.15)

and that if f is negative somewhere in [−1, 1], then inf

{f (v)<0}

f (v) = f (−1) < 0.

(3.16)

On the one hand, condition (3.12) includes the case where f has an ignition temperature proﬁle (f ≡ 0 in [−1, θ] and f > 0 in (θ, 1)). On the other hand, case (3.14) corresponds to the so-called bistable proﬁle.

1 From [19], [23], there exist a unique real c whose sign is that of −1 f (s) ds, and a function z(x2 ) solving the one-dimensional problem: z − cz + f (z) = 0 in R, (3.17) z(±∞) = ±1. The solution z of (3.17) is unique up to translations and is increasing. Furthermore, it has the following asymptotic behaviour as x2 → ±∞ (see [5], [13], [19]): z(x2 ) = −1 + Ceλx2 + o eλx2 (3.18) as x2 −→ −∞, z (x2 ) = Cλeλx2 + o eλx2 ˜ −µx2 + o(e−µx2 ) z(x2 ) = 1 − Ce as x2 −→ +∞, (3.19) ˜ −µx2 + o(e−µx2 ) z (x2 ) = Cµe where

c2 − 4f (0) + c λ= , 2

µ=

c2 − 4f (1) − c 2

(3.20)

and C, C˜ are two positive constants. Under the assumptions (3.12)–(3.16), we can see that λ and µ are always positive. Theorem 3 is a consequence of the following proposition.

388

BERESTYCKI, HAMEL, AND MONNEAU

Proposition 3.2. Under the previous assumptions, for any a ∈ (−1, 1), there exist functions ψ + (x1 ) and ψ − (x1 ) such that (i) ψ − ≤ ψ + ; (ii) the function ua (x1 , x2 ) = z(x2 + ψ + (x1 )) is a supersolution of (1.12), and the function u a (x1 , x2 ) = z(x2 + ψ − (x1 )) is a subsolution of (1.12); (iii) ψ + and ψ − are increasing if a > 0 and decreasing if a < 0, and if a = 0, then ψ + ≡ ψ − ≡ 0; (iv) ψ + (−∞) = ψ − (−∞) ∈ R and ψ + (+∞) = ψ − (+∞) ∈ R; (v) l− = l− (a) := ψ ± (−∞) is decreasing with respect to a, and l+ = l+ (a) := ψ ± (+∞) is increasing. Remark 3.3. Since the function z is increasing, assertion (i) implies that u a (x1 , x2 ) ≤ ua (x1 , x2 ) for all (x1 , x2 ) ∈ R2 . Remark 3.4. In the case where f is positive somewhere, we can show that Proposition 3.2 is still true if assumption (3.15) is replaced with f (1) < 0. To this end, we approximate f in L∞ ([−1, 1]) norm by a sequence of functions satisfying (3.15). In the case where f is negative somewhere, Proposition 3.2 is also true if (3.16) is replaced with f (−1) < 0. Postponing the proof of this proposition, let us ﬁrst state two preliminary lemmas and conclude the proof of Theorem 3. Lemma 3.5. If a function u(x1 , x2 ) is such that u a ≤ u ≤ ua with a = 0, then u is not a function of only x2 . Moreover, it is not a planar function (i.e., a function whose level sets are parallel lines). Proof. First assume that there exists a function x2 → v(x2 ) such that u(x1 , x2 ) = v(x2 ) for all (x1 , x2 ) ∈ R2 . By the deﬁnitions of u a and ua , we have z x2 + ψ − (x1 ) ≤ v(x2 ) ≤ z x2 + ψ + (x1 ) for all (x1 , x2 ) ∈ R2 . Choose x2 = 0 and take the limits x1 → ±∞. By Proposition 3.2(iv), it then follows that v(0) = z(l− ) = z(l+ ). Since z is increasing, we ﬁnd that l− = l+ . This is ruled out by (iii). Assume now that there exist a function t → v(t) and two reals α and β such that u(x1 , x2 ) = v(αx1 + βx2 ) for all (x1 , x2 ) ∈ R2 . Then z x2 + ψ − (x1 ) ≤ v(αx1 + βx2 ) ≤ z x2 + ψ + (x1 ) for all (x1 , x2 ) ∈ R2 . From what precedes, only the case α = 0 remains to be treated. Now choose x1 = γ x2 , where γ = −β/α. We have z x2 + ψ − (γ x2 ) ≤ v(0) ≤ z x2 + ψ + (γ x2 ) for all x2 ∈ R. Since the functions ψ ± are bounded and z(±∞) = ±1, the limits as x2 → ±∞ imply that v(0) = −1 and v(0) = 1. This is impossible.

ONE-DIMENSIONAL SYMMETRY OF ELLIPTIC EQUATIONS

389

Lemma 3.6. If two functions u(x1 , x2 ) and v(x1 , x2 ) are such that u b ≤ u and ua ≥ v with a = b, then u = v. Proof. Assume that u ≡ v and write ua and u b as ua (x1 , x2 ) = z(x2 + ψa+ (x1 )) and u b (x1 , x2 ) = z(x2 + ψb− (x1 )). We then have z x2 + ψb− (x1 ) ≤ u(x1 , x2 ) = v(x1 , x2 ) ≤ z x2 + ψa+ (x1 )

for all (x1 , x2 ) ∈ R2 .

Therefore, since z is increasing, it follows that ψb− (x1 ) ≤ ψa+ (x1 ) for all x1 ∈ R. By taking the limit as x1 → −∞, we ﬁnd that l− (b) ≤ l− (a). By (v), this implies that a ≤ b. Similarly, the limit as x1 → +∞ yields that a ≥ b. Eventually, a = b. This is in contradiction with the assumption a = b, and the proof of the lemma is complete. Proof of Theorem 3. Choose any a ∈ (−1, 1) and, under the notation of Proposition 3.2, consider the functions ψ + , ψ − and ua , u a . By Remark 3.3, we know that u a ≤ ua . Since u a and ua are respectively sub- and supersolutions for (1.12), there then exists a solution ua of (1.12) such that u a ≤ ua ≤ ua ; that is, z x2 + ψ − (x1 ) ≤ ua (x1 , x2 ) ≤ z x2 + ψ + (x1 )

for all (x1 , x2 ) ∈ R2 .

Due to (iv), the functions ψ + and ψ − are bounded. As a consequence, the function ua still satisﬁes the uniform limiting conditions (1.2). Therefore, for each a ∈ (−1, 1), there exists a solution ua of (1.12), (1.2). If a = 0, we simply have u0 = z. By Lemma 3.5, the function ua is not planar if a = 0. By Lemma 3.6, we have ua = ub if a = b. Hence, (1.12) together with the limiting conditions (1.2) has a family of solutions ua parametrized by a ∈ (−1, 1) that are different from one another and are not planar for a = 0. Let us now turn to the proof of Proposition 3.2. Proof of Proposition 3.2. Choose a real a ∈ (−1, 1). By deﬁnition, the function χ (x1 ) is increasing; it then satisﬁes |χ(x1 )| < χ0 for all x1 ∈ R. We can then consider the functions

 χ(x1 ) 1  − −  ψ = ψa (x1 ) = − ln 1 − a ,   µ χ0

  χ(x1 ) 1  ψ + = ψa+ (x1 ) = ln 1 − α + β, λ χ0

390

BERESTYCKI, HAMEL, AND MONNEAU

where the positive real numbers λ and µ have been deﬁned in (3.20) and where

λ −1 α = αa = tanh − tanh (a) ∈ (−1, 1), µ β = βa = −

1 1 ln(1 + a) − ln(1 + α). µ λ

Proof of (iii). If a = 0, the conclusion is obvious. Take now a > 0. We have (ψ − ) (x1 ) =

a χ (x1 ) >0 µχ0 1 − a χ(x1 )/χ0

for all x1 ∈ R

since a, µ, and χ0 are positive and the function χ is increasing. As far as the function ψ + is concerned, we have (ψ + ) (x1 ) = −

α χ (x1 ) λχ0 1 − α χ(x1 )/χ0

for all x1 ∈ R.

Like a, µ, and χ0 , the real number λ is positive. Therefore, α is negative and ψ + is increasing. The case a < 0 can be treated similarly. Proof of (iv). The proof is straightforward owing to the deﬁnitions of ψ ± and to the fact that χ (±∞) = ±χ0 . Proof of (v). We have l− (a) = −(1/µ) ln(1 + a) and l+ (a) = −(1/µ) ln(1 − a). Since µ is positive, this yields (v). Proof of (i). The case a = 0 is obvious. Now choose a = 0 and deﬁne v(x1 ) = ψ + (x1 ) − ψ − (x1 ). Part (iv) says that v(±∞) = 0. To prove that v is nonnegative in R, it is then sufﬁcient to show that v (x1 ) is positive in an interval of the type (−∞, γ ) and negative in (γ , +∞). A straightforward calculation leads to v (x1 ) = A(x1 )B(x1 ) for all x1 ∈ R, where A(x1 ) =

1 χ (x1 ) 1 >0 λµχ0 1 − a χ(x1 )/χ0 1 − α χ(x1 )/χ0

for all x1 ∈ R,

and where B(x1 ) = −(aλ + αµ) + aα(λ + µ)

χ(x1 ) χ0

for all x1 ∈ R.

ONE-DIMENSIONAL SYMMETRY OF ELLIPTIC EQUATIONS

391

The product aα is always negative whatever the sign of a may be. Moreover, remember that λ and µ are positive and that χ is increasing. Hence the function B is (strictly) decreasing. If B did not change sign, then v would be monotone and then identically zero. That would yield v ≡ 0 and B ≡ 0. The latter is impossible since B is decreasing. Hence the function B changes sign. Since it is decreasing, there exists a real γ such that B(x1 ) > 0 in (−∞, γ ) and B(x1 ) < 0 in (γ , +∞). The conclusion follows. Proof of (ii). Choose a ∈ (−1, 1) and consider the function u a (x1 , x2 ) = z x2 + ψ − (x1 ) . Owing to its deﬁnition, it is easy to check that the function ψ = ψ − is a solution of the following ordinary differential equation: µψ − ψ − b(x1 )ψ = 0. 2

(3.21)

Set I (u) := u + b(x1 )∂x1 u − c∂x2 u + f (u). We have 2 I (u a ) = 1 + ψ z + − c + ψ + bψ z + f (z) 2 2 = 1 + ψ cz − f (z) + − c + µψ z + f (z) by (3.17) and (3.21) = −ψ f (z) + (µ + c)ψ z

f (z) f (1) 2 ψ z since µ2 + cµ + f (1) = 0. =− + z µ 2

We now claim that

2

f z(y) f (1) + ≤0 z (y) µ

for all y ∈ R.

(3.22)

Indeed, ﬁrst the function v(y) = f (z(y))/z (y) satisﬁes v = v 2 − cv + f (z). If the supremum of v were reached at a point b ∈ R, then c2 − 4f z(b) c ± f z(b) = v(b) = . z (b) 2 Owing to (3.10) and (3.11), we always have f (1) ≤ 0. Therefore, if f (z(b)) ≤ 0, then v(y) ≤ v(b) ≤ 0 for all y ∈ R and the claim (3.22) follows. Let us now consider the case where f (z(b)) > 0. By the deﬁnition of µ and by (3.15), it follows that c + c2 − 4f (1) f (1) v(b) ≤ =− . 2 µ

392

BERESTYCKI, HAMEL, AND MONNEAU

Moreover, lim supy→−∞ v(y) ≤ 0 owing to (3.11) and z(−∞) = −1. On the other hand, v(+∞) = −f (1)/µ > 0 by (3.19). Consequently, we have supR v ≤ −f (1)/µ. This yields (3.22). This implies that I (u a ) ≥ 0 in R2 , that is, u a is a subsolution of (1.12). Similarly, we can show that the function ua is a supersolution of (1.12). The proof of Proposition 3.2 is complete. Remark 3.7. This counterexample shows that there are inﬁnitely many nonplanar solutions ua to (1.12). We can see that for any a = 0, these solutions are not symmetric with respect to any vertical axis {x1 = b}. In fact, we conjecture that u0 = z is the unique solution that is symmetric with respect to a vertical axis. For an equation of the type (1.12), u + b(x1 )∂x1 u − c∂x2 u + f (u) = 0

in R2 ,

and for some functions f , as we said earlier, there are nonplanar solutions with b ≡ 0 and c = 0 satisfying u(x , xn ) → ±1 as xn → ±∞ for each x ∈ Rn−1 . If uniform limits (1.2) are satisﬁed, then we know from Theorem 2 that any solution u has one-dimensional symmetry whenever b is constant. Nevertheless, Theorem 3 shows that this symmetry property does not hold for some nonconstant and yet bounded functions b and some functions f . More precisely, the nonplanar solutions ua of (1.12) we have constructed are such that, say, for a > 0, z− (x2 ) := z(x2 + l− ) ≤ u(x1 , x2 ) ≤ z+ (x2 ) := z(x2 + l+ ) and

 u(x1 , x2 ) u(x1 , x2 )

x2 →±∞ x1 →±∞

/ ±1

uniformly in x1 , (3.23)

/ z± (x2 ),

where l− < l+ and z± are solutions of (3.17). The proﬁle of a function saﬁsfying these properties is drawn in Figure 1. Recently, similar results have been proved for different equations by Alessio, Jeanjean, and Montecchiari [2] and Alama, Bronsard, and Gui [1]. Alessio, Jeanjean, and Montecchiari, with methods based on Hamiltonian systems, have proved the existence of nonplanar functions u(x1 , x2 ) satisfying the same kind of limits as in (3.23) and solving the equation −u + a(x2 )W (u) = 0

in R2

for some functions a(x2 ) that are positive and periodic. Here W is a multiple well potential. Alama, Bronsard, and Gui [1], with energy methods, have proved the existence of nonplanar solutions U = (u1 , u2 ) for a system of two equations of the type −U + ∇W (U ) = 0, x = (x1 , x2 ) ∈ R2

ONE-DIMENSIONAL SYMMETRY OF ELLIPTIC EQUATIONS

393

1 0.5 u 0 −0.5 −1 −10

−10

−5 0 x2

5

10

−20

0 x1

10 20

Figure 1. Proﬁle of a function u(x1 , x2 ) satisfying (3.23)

satisfying asymptotic limiting conditions as x1 , x2 → ±∞ similar to (3.23). There W : R2 → R is also a multiple well potential. Let us now consider De Giorgi’s nonlinearity f (u) = u − u3 . It satisﬁes the con 1 ditions (3.10), (3.11), (3.14), (3.15), and (3.16). Furthermore, −1 f (s) ds = 0. The unique speed c that is a solution of (3.17) is then equal to zero. Now choose a function b(x1 ) satisfying (3.9). As a consequence of the preceding results, the bidimensional equation u + b(x1 )∂x1 u + f (u) = 0

in R2 ,

(3.24)

together with the uniform limiting conditions (1.2), admits both a planar solution and inﬁnitely many nonplanar solutions. The same result obviously holds in any dimension n ≥ 2 by considering the same equation (3.24) in Rn and choosing special solutions of the type v(x1 , . . . , xn ) = u(x1 , x2 ). As a conclusion, in any dimension n ≥ 2 and even if uniform limits (1.2) are required, De Giorgi’s conjecture cannot be extended for a class of nonconstant functions b(x1 ) (including some bounded functions) to equations of the type (3.24) involving the additional ﬁrst-order term b(x1 )∂x1 u. 4. Half-space case. Let L and g satisfy the assumptions of Theorem 4, and let u ∈ C(Rn+ ) be a solution of (1.15), (1.16). As in the proofs of Theorems 1 and 2, we

394

BERESTYCKI, HAMEL, AND MONNEAU

prove that u is increasing in any direction ν = (ν1 , . . . , νn ) such that νn > 0. For any t ≥ 0, we deﬁne the function ut in {xn ≥ −tνn } by ut (x) = u(x + tν). As we did in (3.1), we have, for any t ≥ 0, Lut + g xn , ut ≤ 0 in {xn > −tνn } ⊃ Rn+ . (4.1) x

Owing to (1.16), there exists a real a > 0 such that u(x , xn ) ≥ 1 − δ for all ∈ Rn−1 and xn ≥ a. For all t ≥ a/νn , the function ut is then such that ut (x , xn ) ≥ 1 − δ for all x ∈ Rn−1 and xn ≥ 0, ut (x , 0) ≥ 0 = u(x , 0) for all x ∈ Rn−1 .

As we did in the proof of Theorem 2, using especially Lemma 3.1, it then follows that ut ≥ u in Rn+ . Let us now decrease t. We claim that ut ≥ u in Rn+ for all t > 0. Deﬁne τ = inf{t > 0, ut ≥ u in Rn+ }. By continuity, we see that uτ ≥ u in Rn+ = {xn ≥ 0}. Let us now argue by contradiction and suppose that τ > 0. Two cases may occur. Case 1. Suppose that inf

Rn−1 ×[0,a]

(uτ − u) > 0.

In this case, as in the proof of Theorem 1, there would exist a real η0 ∈ (0, τ ) such that ut ≥ u in Rn+ for all t ∈ [τ − η0 , τ ]. This would be in contradiction with the minimality of τ . Case 2. Suppose that inf

Rn−1 ×[0,a]

(uτ − u) = 0.

Then there exists a sequence (x k )k∈N ∈ Rn−1 × [0, a] such that uτ (x k ) − u(x k ) → 0 as k → ∞. Up to extraction of a subsequence, two subcases may occur. Subcase 2.1. Suppose that xnk → x n ∈ (0, a] as k → ∞. This subcase is ruled out as Case 2 in the proof of Theorem 2. More precisely, the functions uk (x , xn ) = u(x + x k , xn ) then approach locally in n R+ a function u∞ solving Lu∞ + g(xn , u∞ ) = 0

in Rn+ .

The function uτ∞ satisﬁes Luτ∞ + g(xn , uτ∞ ) ≤ 0 in Rn+ . Furthermore, uτ∞ ≥ u∞ in Rn+ and uτ∞ (0, x n ) = u∞ (0, x n ). From the strong maximum principle, it then follows that uτ∞ ≡ u∞ in Rn+ . The function u∞ is then periodic with respect to the vector ξ = τ ν. From elliptic regularity theory, the function u is globally Lipschitz-continuous in n R+ . Since u satisﬁes (1.16) and since the uk are obtained from u by shifting it with respect to the x -variables, it follows that the function u∞ satisﬁes (1.16) as well. Hence, since ξn > 0, it cannot be ξ -periodic.

ONE-DIMENSIONAL SYMMETRY OF ELLIPTIC EQUATIONS

395

Subcase 2.2. Suppose that xnk → 0 as k → ∞. Since u = 0 on {xn = 0} and u is globally Lipschitz-continuous in {xn ≥ 0}, it then follows that u x k + τ ν −→ 0 as k −→ ∞. Set uk (x) = u(x + x k ). This function is deﬁned in {xn ≥ −xnk } ⊃ {xn ≥ 0}. By standard elliptic estimates, up to extraction of a subsequence, the (nonnegative) functions uk approach locally in {xn > 0} a function u∞ ≥ 0 as k → ∞. We have u∞ (τ ν) = 0. Furthermore, as we did in (3.1) or (4.1) and since xnk ≥ 0, we have Luk (x) + g xn , uk (x) ≤ 0 for all x ∈ Rn−1 and xn > −xnk . As a consequence, for all x ∈ Rn−1 and xn > −xnk , we have Luk (x) + g xn , uk (x) − g(xn , 0) ≤ −g(xn , 0) ≤ 0 by (1.13) and (1.14). Finally, there exists then a bounded function c(x) such that Lu∞ + cu∞ ≤ 0

in Rn+ = {xn > 0}.

Since u∞ is nonnegative and vanishes at the interior point τ ν ∈ Rn+ , the strong maximum principle implies that u∞ ≡ 0 in Rn+ . Recalling that 0 ≤ xnk ≤ a, we see that the function u∞ is such that u∞ (x , xn ) → 1 as xn → +∞ (uniformly in x ∈ Rn−1 ). So Subcase 2.2 is also ruled out. Consequently, τ = 0, and as in the proof of Theorem 1, the function u then depends only on xn and solves (1.17). Lastly, if u(xn ) and v(xn ) are two solutions of (1.17), then the previous proof implies that we simultaneously have u ≥ v and v ≥ u. As a conclusion, the solution u of (1.15), (1.16) is unique, and the proof of Theorem 4 is complete. References [1]

[2] [3] [4] [5]

[6]

S. Alama, L. Bronsard, and C. Gui, Stationary layered solutions in R2 for an Allen-Cahn system with multiple well potential, Calc. Var. Partial Differential Equations 5 (1997), 359–390. F. Alessio, L. Jeanjean, and P. Montecchiari, Stationary layered solutions in R2 for a class of non-autonomous Allen-Cahn equations, preprint. L. Ambrosio and X. Cabré, Entire solutions of semilinear elliptic equations in R3 and a conjecture of De Giorgi, preprint. S. B. Angenent, Uniqueness of the solution of a semilinear boundary value problem, Math. Ann. 272 (1985), 129–138. D. G. Aronson and H. F. Weinberger, “Nonlinear diffusion in population genetics, combustion, and nerve propagation” in Partial Differential Equations and Related Topics (Tulane Univ., New Orleans, 1974), Lectures Notes in Math. 446, Springer, Berlin, 1975, 5–49. M. T. Barlow, On the Liouville property for divergence form operators, Canad. J. Math. 50 (1998), 487–496.

396 [7] [8]

[9] [10] [11] [12] [13] [14] [15] [16] [17]

[18] [19] [20] [21] [22]

[23] [24]

BERESTYCKI, HAMEL, AND MONNEAU M. T. Barlow, R. Bass, and C. Gui, The Liouville property and a conjecture of De Giorgi, preprint. H. Berestycki, L. Caffarelli, and L. Nirenberg, “Symmetry for elliptic equations in a half space” in Boundary Value Problems for Partial Differential Equations and Applications, Rech. Math. Appl. 29, Masson, Paris, 1993, 27–42. , Monotonicity for elliptic equations in unbounded Lipschitz domains, Comm. Pure Appl. Math. 50 (1997), 1089–1111. , Further qualitative properties for elliptic equations in unbounded domains, Ann. Scuola Norm. Sup. Pisa Cl. Sci. (4) 25 (1997), 69–94. H. Berestycki, B. Nicolaenko, and B. Scheurer, Traveling wave solutions to combustion models and their singular limits, SIAM J. Math. Anal. 16 (1985), 1207–1242. H. Berestycki and L. Nirenberg, On the method of moving planes and the sliding method, Bol. Soc. Brasil. Mat. (N.S.) 22 (1991), 1–37. , Travelling fronts in cylinders, Ann. Inst. H. Poincaré Anal. Non Linéaire 9 (1992), 497–572. A. Bonnet and F. Hamel, Existence of non-planar solution of a simple model of premixed Bunsen ﬂames, to appear in SIAM J. Math. Anal. L. Caffarelli, N. Garofalo, and F. Segala, A gradient bound for entire solutions of quasilinear equations and its consequences, Comm. Pure Appl. Math. 47 (1994), 1457–1473. Ph. Clément and G. Sweers, Existence and multiplicity results for a semilinear elliptic eigenvalue problem, Ann. Scuola Norm. Sup. Pisa Cl. Sci. (4) 14 (1987), 97–121. E. De Giorgi, “Convergence problems for functionals and operators” in Proceedings of the International Meeting on Recent Methods in Nonlinear Analysis (Rome, 1978), Pitagora, Bologna, 1979, 131–188. A. Farina, Some remarks on a conjecture of De Giorgi, Calc. Var. Partial Differential Equations 8 (1999), 233–245. P. C. Fife and J. B. McLeod, The approach of solutions of nonlinear diffusion equations to travelling front solutions, Arch. Rational Mech. Anal. 65 (1977), 335–361. B. Gidas, W. N. Ni, and L. Nirenberg, Symmetry and related properties via the maximum principle, Comm. Math. Phys. 68 (1979), 209–243. N. Ghoussoub and C. Gui, On a conjecture of De Giorgi and some related problems, Math. Ann. 311 (1998), 481–491. F. Hamel and R. Monneau, Solutions d’équations elliptiques semi-linéaires dans R N ayant des courbes de niveau de forme conique, C.R. Acad. Sci. Paris Sér. I, Math. 327 (1998), 645–650. Ja. I. Kanel’, Certain problems on equations in the theory of burning, Sov. Math. Dokl. 2 (1961), 48–51. L. Modica and S. Mortola, Some entire solutions in the plane of nonlinear Poisson equations, Boll. Un. Mat. Ital. (5) 17 (1980), 614–622.

Berestycki: Laboratoire d’Analyse Numérique, Université Paris VI, 4 place Jussieu, 75252 Paris cedex 05, France; [email protected] Hamel: Laboratoire d’Analyse Numérique, Université Paris VI, 4 place Jussieu, 75252 Paris cedex 05, France; [email protected] Monneau: CERMICS-ENPC, 6-8 avenue B. Pascal, Cité Descartes, Champs-sur-Marne, 7455 Marne-La-Vallée cedex 2, France; [email protected]

Vol. 103, No. 3

DUKE MATHEMATICAL JOURNAL

© 2000

MONODROMY OF AIRY AND KLOOSTERMAN SHEAVES ONDREJ ŠUCH

1. Introduction. Our work began with calculations of the numbers of points on a family of hyperelliptic curves given by the following equation over a ﬁnite ﬁeld k of characteristic 2: y 2 − y = x n + an−1 x n−1 + · · · + a1 x + a0 . The simplicity of this equation makes these hyperelliptic curves particularly well suited for explicit computer calculations requiring higher genus curves. On the other hand, the special form of the equation raises the question of how generic or random, if at all, these curves are. For instance, all of these curves have 2-rank 0. The question of genericity can be restated more precisely as the question of the monodromy group of a family of curves. Let us denote by K the ﬁeld k(ai ), where ai are the variable parameters of a family C of hyperelliptic curves. Choosing l = 2, the l-adic monodromy group is the image of the action of Gal(K sep /K) on H 1 (C ⊗K ¯ Ql ). Many properties of a family C of curves are described by the geometric K, monodromy group Ggeom , which is the Zariski closure of image of the subgroup ¯ For instance, if the geometric monodromy group of a family C is Gal(K sep /K k). ﬁnite, all curves in the family C are supersingular. We prove that quite the opposite is true. Theorem (Corollary 13.3). For any g ≥ 3, the geometric monodromy group of hyperelliptic curves of genus g and 2-rank zero over F2 is Sp(2g). Of several possible approaches to studying the monodromy group of the above family, we chose the one using Fourier transforms. In naive terms, a Fourier transform ¯ From a higher corresponds to a varying coefﬁcient a1 of the family over values in k. viewpoint, the cohomology sheaf Ᏺ of this subfamily is the Fourier transform of a ¯ General theory of Fourier transforms rank-1 Ql sheaf lisse on the afﬁne line over k. guarantees that irreducibles transform into irreducibles, and therefore the geometric monodromy group of this subfamily is always irreducible. Another key restriction on the sheaf Ᏺ arises from the action of the inertia group. Again, the general theory of local Fourier transforms guarantees that the inertia group at ∞ of A1 acts irreducibly. It is, however, hard to describe this action, because even the action of wild inertia does not factor via an abelian quotient. Still, it is possible to show that the sheaf of traceless endomorphisms End0 (Ᏺ) has no factors of dimension less than Ᏺ. This property of Ᏺ is used to derive strong restrictions on its Ggeom (see Received 16 July 1998. 2000 Mathematics Subject Classiﬁcation. Primary 14D; Secondary 11L. 397

398

ONDREJ ŠUCH

Proposition 11.1, Theorem 5.1, Proposition 6.2, and Corollary 6.3). These two properties are shared more generally by any Airy sheaf, which, by deﬁnition, is the Fourier transform of any rank-1 sheaf lisse on A1 . Using our approach, we are able to prove the following result. Theorem (Proposition 11.6). If p > 2, and Ᏺ is a Lie irreducible Airy sheaf over A1 ⊗ F¯ p , then the Lie algebra of Ggeom is sl or sp in its standard representation. This was known previously [K1], when the characteristic p > 2 rank Ᏺ + 1. Our ﬁnal result (see Proposition 11.6) determines the geometric monodromy group of an Airy sheaf when p > 2, modulo recognition of ﬁnite Airy sheaves. The result extends even to p = 2 when certain ranks are excluded (see Proposition 11.7). Although we developed our methods for studying the geometric monodromy of Airy sheaves, they apply equally well to Kloosterman sheaves. The reason is that like Airy sheaves, Kloosterman sheaves admit an intrisic characterization based on the action of inertia at 0 and ∞. Theorem (Corollary 12.4). Let Ᏺ be a Lie irreducible Kloosterman sheaf on A1 √ ¯ over Fp . If p > rank Ᏺ + 1, then the Lie algebra of Ggeom is either sl or sp or so in its standard representation. Again, this result was previously proven by Katz when p > 2 rank Ᏺ + 1. Our work still leaves many open questions about Airy and Kloosterman sheaves. We would like to highlight two of them. Question 1. How does one effectively recognize ﬁniteness of an Airy or a Kloosterman sheaf ? Question 2. What algebraic groups occur as Ggeom of an Airy sheaf in characteristic p = 2? Lastly, but most importantly, I would like to express my gratitude to Professor Katz, who introduced me to the subject of l-adic sheaves and guided me through my investigations. 2. Properties of inertial representations. Let K be the function ﬁeld of a smooth, geometrically connected curve over a separably closed ﬁeld k of characteristic p > 0. For any discrete valuation v of K, we denote by Kv the completion of K with respect sep to v. Let I (v) denote the inertia group of v, that is, Gal(Kv /Kv ). It is customary to equip I (v) with the upper numbering ﬁltration (see [Se, §3]) by closed subgroups I (v)(r) , where r ranges through all nonnegative real numbers. The closure of the union I (v)(r) r>0

is called the wild inertia group, and we denote it by P (v). It is a p-Sylow subgroup

MONODROMY OF AIRY AND KLOOSTERMAN SHEAVES

399

of I (v). There is a short exact sequence of groups 0 −→ P (v) −→ I (v) −→ Zl −→ 0. l=p

The procyclic quotient in the sequence corresponds to the maximal tamely ramiﬁed sep subextension of Kv /Kv . For any integer N prime to p, there exists a canonical extension of degree N. Namely, choosing a local parameter x at v, we can identify ∼ Kv −→ k((x)), and then the extension is k((x 1/N ))/k((x)). We denote by I (v)(N) the corresponding normal subgroup of I (v). Note that the p-Sylow subgroups of I (v) and I (v)(N ) coincide. Moreover, the upper numbering on I (v)(N) satisﬁes I (v)(N)r = I (v)r/N . Now choose a prime l = p. Let V be any continuous ﬁnite-dimensional Ql representation of I (v). We have canonical splitting of V as Vwild , V = Vtame where P (v) acts trivially on Vtame and without invariants on Vwild . This is the simplest consequence of more general break decomposition (see [K2, Lemma 1.8 and Proposition 1.1]) of V : V (x), V = x≥0

where V (x) is 0 for all but ﬁnitely many x, and V (x) has the property that x is the inﬁmum of all r such that I (v)(r) acts trivially on V (x). In terms of this decomposition, Vtame = V (0). We call nonnegative real numbers x such that V (x) = 0 the breaks of V and deﬁne Swan(V ) = x dim V (x) . x≥0

The following is a trivial but very useful fact. Lemma 2.1. Using the same notation as above, let be the image of I (v) in V . If V had breaks less than or equal to r, then any Ql representation of I (v) obtained by composing with a representation of has all breaks less than or equal to r. The structure of I (v) can be exploited to give some information about irreducible representations of I (v). Proposition 2.2 (Katz). Let V be an irreducible representation of I (v). Write dim(V ) as n0 p k , where n0 is prime to p. Then V is induced from a pk -dimensional representation W of I (v)(n0 ), and the restriction of W to P (v) is irreducible. Furthermore, the restriction of V to P (v) is the direct sum of n0 pairwise inequivalent irreducible pk -dimensional representations of P (v), whose isomorphism classes are ﬁxed by I (v)(n0 ) and cyclically permuted by I (v)/I (v)(n0 ).

400

ONDREJ ŠUCH

Proof. See [K2, 1.14.2]. Corollary 2.3. Using the hypotheses in Proposition 2.2, V is absolutely irreducible as a P (v) representation if and only if dim(V ) is a power of p. Lemma 2.4. If V is a ﬁnite-dimensional Ql representation of a group G induced from a subgroup H of G of ﬁnite index, then End(V ) contains IndG H 1. Proof. Quite generally, if A and B are G-representations induced from representations V and W of H , then σ A ⊗ B = ⊕σ ∈G/H IndG H (A ⊗ B ),

where ρ(h) | B σ := ρ(σgσ −1 ). Lemma 2.5 (Burnside). If G is a group of odd order, then G does not have a nontrivial irreducible self-dual representation. Proof. See [Is, 3.16]. It seems difﬁcult to analyze the decomposition of End V in complete generality. However, we always have the following. Lemma 2.6. Using the hypotheses in Proposition 2.2, the largest tame subrepre∼ sentation (End V )tame of End V is the regular representation of I (v)/I (v)(n0 ) −→ Z/n0 . In particular, it has dimension n0 . If p = 2 and V is self-dual, then (End V )tame is contained in Sym2 V or !2 V based on whether or not V is orthogonal. If p = 2 and V is self-dual, then n0 is even, and only characters factoring via I (v)/I (v)(n0 /2) are contained in Sym2 V or !2 V based on whether or not V is orthogonal. Proof. Since V restricted to P (v) decomposes as the sum of n0 distinct irreducible representations, it follows from Schur’s lemma that End V has n0 invariants under P (v), that is, that the dimension of End(V )tame is n0 . On the other hand, since V was induced from a representation W of I (v)(n0 ), the module End V contains the induction of End W , which in turn contains the induction of the trivial representation, which is the regular representation of I (v)/I (v)(n0 ), and has dimension n0 (see Lemma 2.4). Now suppose p = 2, so that n0 is odd. As a representation of P (v), V is the sum of an odd number of distinct irreducibles Wi ; hence, each of these irreducibles is self-dual and its autoduality is of the same sign as that of V . Therefore, the tame subrepresentation of End V is the space of invariants of those representations. Suppose p = 2, and suppose for the sake of simplicity that V is symplectic. As a representation of I (v)(n0 ), V is the sum of n0 distinct irreducibles Wi , none of which are self-dual, because each is irreducible when restricted to P (v), and P (v) is a group of odd order (see Lemma 2.5). Since the sum of the Wi ’s is self-dual, it has to be possible to pair them into self-dual representations. Hence, n0 is even. Let Wτ (i) be

MONODROMY OF AIRY AND KLOOSTERMAN SHEAVES

401

the dual of Wi . Then Zi = Wi + Wτ (i) is self-dual. In fact, Zi ⊗ Zi = End0 Zi carries two invariants of wild inertia, namely, 1Wi + 1Wτ (i) and 1Wi − 1Wτ (i) . The former is symmetric and the latter is symplectic. It follows that n0 /2 characters of End(V )tame lie in Sym2 V and n0 /2 lie in !2 V . Since I (v)/I (v)(n0 ) permutes the Wi ’s and its action commutes with duality, any element of I (v)/I (v)(n0 ) that maps some Wi to its dual has to be of order 2, but there is such a unique element in I (v)/I (v)(n0 ): W _i

∨

τ

Wi∨

/ W∨ _i τ −1 =τ

∨

/ Wi .

In particular, the decomposition of V into Zi is precisely the decomposition of V when restricted to I (v)(n0 /2), so that V is induced from any of Zi . Therefore, !2 V contains the induction of !2 Zi , which is the regular representation of I (v)/I (v)(n0 /2). The argument in the orthogonal case is completely analogous. We now recall the well-known correspondence between cyclic Z/p r Z coverings of afﬁne scheme and a certain quotient of the additive group of Witt vectors. For the deﬁnition and basic properties of Witt vectors, we refer the reader to [Se, §6]. For any integer r ≥ 1, the scheme of Witt vectors of length r is denoted by Wr . It is an afﬁne scheme isomorphic to Ar , but it has also a structure of a ring. There is a distinguished set of coordinates X0 , . . . , Xr−1 . For any afﬁne scheme A, an element f of Wr (A) can be written as f = (f0 , . . . , fr−1 ), where fi = Xi (f ) are elements of A. We deﬁne a map of schemes V : Wr → Wr by V (X0 ) = 0, V (Xi ) = Xi−1 . It is an additive map of W to W . For any pair of integers r, s ≥ 1, we have a short exact sequence of additive group schemes Vr

0 −→ Wr −−→ Wr+s −→ Ws −→ 0.

(1) p

Another map F : W → W of schemes is given by F (Xi ) = Xi . If A is an afﬁne scheme of characteristic p, then F is a ring endomorphism of W (A) as well as of Wn (A), for n ≥ 1. Moreover, F V = V F = p. There is an exact sequence of étale abelian sheaves over Fp : F −1

0 −→ Z/p n Z −→ Wn −−−→ Wn −→ 0. Taking the long exact sequence of étale cohomology, we obtain 0 −→ Z/pn Z −→ Wn (A) −→ Wn (A) −→ Hom π1 (A), Z/p n Z −→ H 1 A, Wn (A) . Since A is afﬁne, 0 = H 1 (A, OA ) = H 1 (A, W1 (A)), so that using (1), we obtain H 1 A, Wn (A) = 0. Thus, we arrive at the following result.

402

ONDREJ ŠUCH

Proposition 2.7. Let A be an afﬁne scheme over Fp , and let r be an integer greater than or equal to 1. Then Hom π1 (A), Z/p r Z = Wr (A)/(F − 1)Wr (A). It is of interest to be able to determine explicitly the conductor of a character obtained from a Witt vector. Theorem 2.8 (Brylinski). Let k be an algebraically closed ﬁeld of characteristic p. Let f = (f0 , . . . , fr−1 ) be an element of Wr (k[[x]]). Then the Swan conductor of the cyclic extension of k((x)) corresponding to f is equal to min max deg(gi )p n−1−i , g 0≤i≤n−1

where g = (g0 , . . . , gn−1 ) runs through elements of Wr (k[[x]]) such that f = g in Wr (k[[x]])/(F − 1). Proof. See [Bry, pp. 24–27]. Corollary 2.9. Let χ be any Ql -valued character of inertia of k((x)) of Swan conductor N = N0 p k , where p N0 . Let φ be an arbitrary faithful Ql character of Z/pk+1 . Then there exists a constant a in k, such that χ = φ((a · x N0 , 0, . . . , 0)) · χ , where χ has Swan conductor less than N. Proof. See [Bry, Proposition 1 and corollaire to Théorème 1]. We are able to describe the decomposition of End(V ) only in very special cases. Lemma 2.10. Suppose that the break of V is a/N, where N = dim V , and (pa, N) = 1. Then N−1 End V = (End V )tame + Wi , i=1

where each Wi is an N-dimensional irreducible representation of I (v) of break a/N. Proof. Since p dim V , it follows that V is induced from a character χ of I (v)(N) of Swan conductor a. Write a = pk a0 , where p a0 . Then by Corollary 2.9, k times

χ = ψ(c · x , 0, . . . , 0) · χ , a0

where χ has Swan conductor less than a. By Frobenius reciprocity, V | I (v)(N) =

N ia ψ c · ζN 0 x a0 , 0, . . . , 0 · χi , i=1

where χi have Swan conductor less than a. In particular,

MONODROMY OF AIRY AND KLOOSTERMAN SHEAVES

403

ja End V | I (v)(N ) = ψ c · ζNi x a0 , 0, . . . , 0 · χi · ψ¯ c · ζN 0 x a0 , 0, . . . , 0 · χ¯ j i,j

= 1N +

j ψ c ζNi − ζN x a0 , 0, . . . , 0 · χi χ¯ j i=j

N

=1 +

ρi=j .

Every character ρi,j of I (v)(N) in the above expression has Swan conductor a. Since all breaks of End V are ≤ a/N, it follows from the integrality of the Swan conductor that each one those characters has to have an orbit of size at least N under the action of I (v) on it, by conjugation. On the other hand, since ρi,j are characters of I (v)(N ), the orbit of any of them can be at most equal to the index of I (v)(N ) in I (v), which is N, so that every one lies in an N-dimensional irreducible subrepresentation of I (v). 3. Sheaves lisse on A1 . Let k be an algebraically closed ﬁeld of characteristic p, and let C be a smooth, irreducible projective curve over k. If Ᏺ is a constructible sheaf on C, then the Euler-Poincaré formula (see [R, Théorème 1]) reads χ (C, Ᏺ) = χ (C) rank Ᏺ − Swanp (Ᏺp ) − dropp (Ᏺ), p∈C

p∈C

where dropp (Ᏺ) is deﬁned as dim Ᏺ − dim Ᏺp . The following lemma is a useful application of the formula. Lemma 3.1. Let Ᏻ be a sheaf lisse on A1 such that H 0 (A1 , Ᏻ) = Hc2 (A1 , Ᏻ) = 0. Then the Swan conductor of Ᏻ is greater than or equal to the rank of Ᏻ, and the equality holds if and only if any of the following conditions hold: (1) H ∗ (A1 , Ᏻ) = 0, (2) Hc∗ (A1 , Ᏻ) = 0, (3) H 1 (A1 , Ᏻ) = 0, (4) Hc1 (A1 , Ᏻ) = 0. Proof. By the Euler-Poincaré formula, −h1 = −h1c = rank(Ᏻ) − Swan∞ (Ᏻ), which implies the conclusion. To formulate some of our results we use the Fourier transform. To keep things as simple as possible, we use the naive Fourier transform as deﬁned in [K2, 8.2.3]. The Fourier transform depends (in a nonessential way) on a choice of a nontrivial additive character ψ : Fq −→ Ql , where Fq is a ﬁnite subﬁeld of k. We denote by ᏸψ the lisse rank-1 Ql sheaf on

404

ONDREJ ŠUCH

A1 ⊗ k obtained by pushing out the Lang torsor. For any a ∈ k = A1 (k), we denote ᏸψ(ax) := (x −→ ax)∗ (ᏸψ ).

The naive Fourier transform is well behaved on the class of Fourier sheaves (see [K2, 8.2.1.2]). A constructible Ql sheaf Ᏺ on A1 is Fourier if it satisﬁes the following. (F1) For j : U → A1 ⊗ k, the inclusion of any nonempty ouvert de lissité of Ᏺ, we ∼ have Ᏺ −→ j∗ j ∗ Ᏺ. (F2) For any a ∈ k, H 0 A1 ⊗ k, Ᏺ ⊗ ᏸψ(ax) = 0. If Ᏺ is a Fourier sheaf, then the stalk of FT ψ (Ᏺ) at a point a ∈ k is Hc1 A1 , Ᏺ ⊗ ᏸψ(ax) . The following proposition demonstrates an interesting feature of Fourier sheaves that are lisse on the whole of A1 . Lemma 3.2. Let Ᏻ be a nonzero Fourier sheaf lisse on A1 . Then one of the breaks of Ᏻ at ∞ is greater than 1. Proof. If every break of Ᏻ were at most 1, then the same would hold for every sheaf Ᏻ ⊗ ᏸψ(wt) , with w ∈ k. Therefore, Swan Ᏻ ⊗ ᏸψ(wx) ≤ rank Ᏻ ⊗ ᏸψ(wx) . Since Ᏻ is Fourier,

H 0 A1 , Ᏻ ⊗ ᏸψ(wx) = Hc2 A1 , Ᏻ ⊗ ᏸψ(wx) = 0.

It follows from Lemma 3.1 that h1 (Ᏻ ⊗ ᏸψ(wt) ) = 0, so that the Fourier transform vanishes. We arrive at a contradiction by applying the Fourier inversion formula. Corollary 3.3. Let Ᏻ be a semisimple Ql sheaf lisse on A1 , all of whose breaks are less than or equal to 1. Then Ᏻ is a sum of ᏸψ(ai x) , where ai ∈ k. Proof. We proceed by induction on the rank of Ᏻ. If rank Ᏻ is 1, then by integrality of Swan conductor, the break of Ᏻ(∞) is either 0 or 1. Since A1 is tamely simply connected, the former case implies that Ᏻ is the constant sheaf. The latter case follows from Corollary 2.9. Now suppose dim Ᏻ > 1. By Lemma 3.2, we see that Ᏻ is not Fourier; hence, by virtue of its semisimplicity, it has a direct summand of the form ᏸψ(ax) . Writing Ᏻ = ᏸψ(ax) ⊕ Ᏻ ,

the sheaf Ᏻ is lisse on A1 , and all of its breaks are less than or equal to 1; hence, we may apply an induction hypothesis.

MONODROMY OF AIRY AND KLOOSTERMAN SHEAVES

405

Lemma 3.4. Let Ᏺ be a Fourier sheaf lisse on A1 , all of whose breaks are less than or equal to (n + 1)/n. Then the rank of its Fourier transform is at most (rank Ᏺ)/n. If, furthermore, the average break of Ᏺ is 1, then the rank of its Fourier transform is less than or equal to (rank Ᏺ)/(n + 1) with equality occurring if and only if Ᏺ has (rank Ᏺ)/(n+1) breaks 0 at ∞, and all the rest of the breaks of Ᏺ(∞) are (n+1)/n. Proof. The rank of the Fourier transform is (λ − 1). λ>1

Finally, we include the following result on 1-dimensional representations of π1 (A1 ). Lemma 3.5. Let χ be a Ql -valued character of π1 (A1 ). Then χ is of ﬁnite order

pr .

Proof. We use the fact that the tame fundamental group of A1 is trivial. Suppose χ takes values in the ring of integers OE of a ﬁnite extension of Ql . We may consider the sequence of ﬁnite characters χn , obtained by composition χn : π1 (A1 ) −→ OE× −→ OE× /λn , where λ is the prime of OE over l. Each χn corresponds to a cyclic covering of A1 of degree d(n) with Galois group (n). Write d(n) = pv(n) m(n), where p m(n). Then we can write (n) = G(n) × H (n), where G(n), H (n) are cyclic groups of orders pv(n) and m(n), respectively. The covering of A1 corresponding to H (n) has to be trivial, since it is abelian of degree prime to p. We are ﬁnished once we show that the increasing sequence of v(n) has an upper bound. But this holds because for sufﬁciently large n, the index of OE× /λn+1 in OE× /λn is a power of l = p. 4. Some facts about Lie groups. We use standard facts about the representation theory of semisimple Lie algebras and Lie groups. We follow standard terminology and notation (see, e.g., [Ja]), except for the following. We denote Lie algebras A1 , B2 by C1 , C2 , respectively. Given a choice of Cartan subalgebra h of a semisimple Lie algebra g, and given a choice of simple roots R of g, let π denote the corresponding system of fundamental weights of g. Any dominant weight ! is a linear combination with integral coefﬁcients ! = !1 π1 + !2 π2 + · · · + !l πl . We refer to these coefﬁcients as numerical labels of !. We denote by R(!) an irreducible representation of g with highest weight !. The following lemma is presumably well known, but we were unable to ﬁnd a satisfactory reference. The lemma describes the normalizer of a connected semisimple linear algebraic group in the ambient GL. To formulate our result, we recall that if G is a connected semisimple algebraic group, then elements of the subgroup of Aut

406

ONDREJ ŠUCH

Lie G generated by exp(ad x), x ∈ G are called invariant automorphisms. (See [Ja, Chapter IX].) Lemma 4.1. Let G be a semisimple algebraic group, and let ρ : G → GL(V ) be a faithful Lie irreducible representation of G. Denote by g the Lie algebra of G, and choose a Cartan subalgebra h of g, together with a system of simple roots π. Every element r in G/G0 has a representative r in G such that conjugation by r ﬁxes (h, π) and preserves the highest weight of V . Conversely, given an element σ of Aut(g) and an invariant automorphism τ such that σ = σ τ ﬁxes both (h, π) and the highest weight of V , there exists an element r in GL(V ) inducing σ and normalizing G0 . Proof. Conjugation by any element of G is an automorphism of g; hence, upon multiplication by an invariant automorphism (which is induced by conjugation by an element of G0 ), we may assume it preserves (h, π). It follows from the uniqueness of the highest weight vector that the highest weight of V does not change under the action of r. Suppose without loss of generality that σ ﬁxes (h, π). Since σ ﬁxes the highest weight, the representations ρ, σ ◦ρ are isomorphic; hence, there exists an intertwining element r in GL(V ) translating one into the other. We also need some results on certain ﬁnite subgroups of exceptional Lie groups. To that end, we prove two results. Lemma 4.2. Let p be any prime. For any ﬁnite ﬁeld Fλ such that λ ≡ 1 mod p, the p-Sylow subgroups of the following pairs of ﬁnite (split) Chevalley groups coincide: (1) E6 (Fλ ) and (SL(2) × SL(6))(Fλ ), when p > 3, (2) E7 (Fλ ) and SL(8, Fλ ) when p > 3, (3) E8 (Fλ ) and SL(9, Fλ ) when p > 5, (4) F4 (Fλ ) and SO(9, Fλ ) when p > 3, (5) G2 (Fλ ) and SL(3, Fλ ) when p > 3. Proof. First we know by the classiﬁcation of maximal reductive subalgebras of exceptional Lie algebras (see [MPR, pp. 57–65]) that the groups listed map to the exceptional Lie groups. Then we use explicit formula for the number of points on these Chevalley groups (see [CCNP, p. 14]) to see that their ratio, which is a polynomial in λ, is prime to p. Explicitly, we have #E6 (Fλ ) λ36 (λ12 − 1)(λ9 − 1)(λ8 − 1)(λ6 − 1)(λ5 − 1)(λ2 − 1) = #(A1 + A5 )(Fλ ) λ(λ2 − 1) · λ15 (λ2 − 1)(λ3 − 1)(λ4 − 1)(λ5 − 1)(λ6 − 1) = λ20 (1 + λ2 )(1 − λ + λ2 )(1 + λ + λ2 )(1 + λ4 ) × (1 − λ2 + λ4 )(1 + λ3 + λ6 ), #E7 (Fλ ) λ63 (λ18 − 1)(λ14 − 1)(λ12 − 1)(λ10 − 1)(λ8 − 1)(λ6 − 1)(λ2 − 1) , = #A7 (Fλ ) λ28 (λ8 − 1)(λ7 − 1)(λ6 − 1)(λ5 − 1)(λ4 − 1)(λ3 − 1)(λ2 − 1)

MONODROMY OF AIRY AND KLOOSTERMAN SHEAVES

407

#E7 (Fλ ) = λ35 (λ + 1)3 (λ2 − λ + 1)2 (λ2 + λ + 1)(λ4 − λ2 + 1) #A7 (Fλ ) × (λ6 − λ5 + λ4 − λ3 + λ2 − λ + 1)(λ4 − λ3 + λ2 − λ + 1) × (λ6 − λ3 + 1)(λ6 + λ3 + 1), #E8 (Fλ ) #A8 (Fλ ) λ120 (λ30 − 1)(λ24 − 1)(λ20 − 1)(λ18 − 1)(λ14 − 1)(λ12 − 1)(λ8 − 1)(λ2 − 1) = λ36 (λ2 − 1)(λ3 − 1)(λ4 − 1)(λ5 − 1)(λ6 − 1)(λ7 − 1)(λ8 − 1)(λ9 − 1) = λ84 (1 + λ4 )(1 + λ2 )2 (1 − λ + λ2 )3 (1 + λ + λ2 )(1 + λ4 )(λ − λ2 + λ4 )2 × (1 − λ + λ2 − λ3 + λ4 )2 (1 + λ + λ2 + λ3 + λ4 )(1 − λ3 + λ6 ) × (1 − λ + λ2 − λ3 + λ4 − λ5 + λ6 )(1 − λ4 + λ8 )(1 − λ2 + λ4 − λ6 + λ8 ) × (1 − λ + λ3 − λ4 + λ5 − λ7 + λ8 )(1 + λ − λ3 − λ4 − λ5 + λ7 + λ8 ), #F4 (Fλ ) λ24 (λ1 2 − 1)(λ8 − 1)(λ6 − 1)(λ2 − 1) = 16 2 #B4 (Fλ ) λ (λ − 1)(λ4 − 1)(λ6 − 1)(λ8 − 1) = λ8 (1 − λ + λ2 )(1 + λ + λ2 )(1 − λ2 + λ4 ), #G2 (Fλ ) λ6 (λ6 − 1)(λ2 − 1) = λ3 (1 + λ)(1 − λ + λ2 ). = #A2 (Fλ ) λ3 (λ2 − 1)(λ3 − 1) Corollary 4.3. Let P be a ﬁnite 7-group inside E7 (C). Then the adjoint representation of E7 (C) has a P -invariant. Proof. Choose a basis of the adjoint representation V such that P acts by matrices with coefﬁcients in a number ﬁeld F . Choose a prime λ of F (ζ7 ) such that the residual characteristic of λ is not 7, and the kernel of the reduction map p : GL F (ζ7 )133 −→ GL V , Fλ133 has no elements of ﬁnite order. By [MPR, p. 118], we may also choose a map φ: φ : SL 8, F (ζ7 ) −→ GL V , F (ζ7 )133 , such that the image of φ lies inside E7 (F (ζ7 )). By Lemma 4.2, the 7-Sylow subgroups of r(E7 (F (ζ7 ))) and r ◦φ(F (ζ7 )) have the same size. Since the latter lies inside the former, it is sufﬁcient to show that the 7-Sylow subgroup of r ◦φ(F (ζ7 )) has an invariant. We know [MPR, p. 118] that under the embedding φ, V contains the adjoint representation of SL(8, Fλ ). Since no 7-group can act irreducibly on an 8-dimensional vector space, the 7-Sylow subgroup of SL(8, Fλ ) has an invariant when acting on the adjoint representation of SL(8), and a fortiori an invariant when acting on V .

408

ONDREJ ŠUCH

Corollary 4.4. The p-Sylow subgroup of a split exceptional Chevalley group G over a ﬁnite ﬁeld F of characteristic not equal to p is abelian if (1) G = E6 and p > 11, (2) G = E7 and p > 7, (3) G = E8 and p > 7, (4) G = F4 and p > 7, (5) G = G2 and p > 3. Proof. We apply Lemma 4.2 and note that in each case, the p-Sylow subgroup of G admits a faithful less than p-dimensional representation. Proposition 4.5. Let G be a connected semisimple linear algebraic group over a ﬁnite ﬁeld F, given in an irreducible representation V . Suppose that the Dynkin diagram G has no nontrivial automorphisms. For any element g ∈ G, the Galois group of the characteristic polynomial char(g) of g over the ﬁeld of coefﬁcients of g is a subgroup of the Weyl group of G. Proof. Any semisimple algebraic group over a ﬁnite ﬁeld is automatically F-quasisplit, that is, it has a Borel subgroup deﬁned over F (see [Hu, Theorem 35.2]). Since G has no nontrivial outer automorphisms, it is by [Hu, 35.1] split; that is, it has an F-split maximal torus together with associated admissible isomorphisms. Since the Galois group decreases under specialization, it is sufﬁcient to prove our claim for the generic point of G. Denote by K the ﬁeld of coefﬁcients of char(η), where η is the generic point of G. The splitting ﬁeld L of K is the ﬁeld generated by the roots of G. Any automorphism of L/K induces an automorphism of the root system of G. Quite generally, the automorphism group of a root system R of G is the semidirect product of the Weyl group of R with the group of outer automorphisms of G, which proves our result since our root system has no nontrivial automorphisms. 5. A Lie simplicity theorem. The following theorem gives a sufﬁcient condition for a semisimple algebraic group either to have a simple Lie algebra or to be on a short list of exceptional cases. Theorem 5.1. Let G be a semisimple algebraic group, and let V be a faithful, Lie irreducible, N-dimensional representation of G. Suppose further that End0 (V ) decomposes into irreducibles, all of which have dimension at least N. Then one of the following is true: (1) The Lie algebra of G is simple. √ √ (2) The N is a square, the Lie algebra of G is isomorphic to sl( N)×sl( N), and there exists an element of G that interchanges the two simple factors V . As a representation of the Lie algebra of G, V is the tensor product of standard representations of the factors. The subgroup G ∩ (Gm · G0 ) has index 2 in G. (3) The Lie algebra of G is sl(2) × sl(2) × sl(2). As a representation of the Lie algebra of G, V is the tensor product of three standard representations of sl(2). The index of G ∩ (Gm · G0 ) in G is either 3 or 6.

MONODROMY OF AIRY AND KLOOSTERMAN SHEAVES

409

Proof. Let g be the Lie algebra of G, and let g = ;gi , i ≤ k be the decomposition of g into simple factors. Choose a Cartan subalgebra h of g and a system of simple roots π . Since V is Lie irreducible, there is a corresponding decomposition V = ⊗Vi , i ≤ k, where Vi is an irreducible represention of gi of highest weight wi . The group G acts on its connected component by conjugation, and this action can be interpreted as a permutation action on gi , or even as a permutation of the set 1, . . . , k. For any i ≤ k, denote by orb(i) the number of simple factors in the orbit of gi in this action. As g-representation, we may write V ⊗ V ∨ = ⊗ Vi ⊗ Vi∨ = ⊗ 1 ⊕ End0 (Vi ) . Choose any j ≤ k. By abuse of notation, we denote by ι the involution of any semisimple Lie algebra that carries an irreducible representation to its dual. Denote by Ej the irreducible g-submodule of V ⊗ V ∨ with highest weight wj + ιwj . Let Wj be the G-irreducible component containing Ej . It is a sum of representations factoring through simple quotients of g isomorphic to gj , namely, those in the orbit of gj under the action of G on the Dynkin diagram. Since any element of G acts trivially on the highest weight of V , any element of G stabilizing gi acts trivially on the highest weight of Vi and can map gr to gs only if there is an isomorphism i : gr → gs such that i(wr ) = ws . Therefore, if we denote by dj the dimension of Vj , dim Wj ≤ orb(j ) · dim End0 (Vj ) = orb(j ) · dj2 − 1 . By assumption,

N ≤ dim Wj ≤ orb(j ) · dj2 − 1 . orb(j )

On the other hand, dj

divides the dimension N of V . Therefore, orb(j )

dj

≤ orb(j ) · dj2 − 1 .

For n ≥ 3, consider the function φn (d) := d n /(d 2 − 1) on the interval [2, ∞). Its derivative is the function nd n−1 (d 2 − 1) − 2d · d n (n − 2)d n+1 − nd n−1 = 2 2 (d − 1) (d 2 − 1)2 (n − 2)d n−1 2 n > 0. d = − n−2 (d 2 − 1)2

φn (d) =

It follows that the function φn (d) is monotonously increasing on this interval, so that if orb(j ) ≥ 3, then 2orb(j ) ≤ 3 · orb(j ), and that implies orb(j ) ≤ 3. Now consider three cases.

410

ONDREJ ŠUCH

Suppose orb(j ) = 3. Then

dj3 ≤ N ≤ 3 dj2 − 1 ,

which implies dj <

dj3 dj2 − 1

≤ 3,

orb(j )

so that dj = 2. Since dj divides N, it follows that N = 8. There is a unique simple Lie algebra with a 2-dimensional representation, namely, sl(2) in its standard representation. This is the case (3). The group of outer automorphisms of sl(2) × sl(2) × sl(2) is the symmetric group on three elements, and we know that G operates transitively on factors of g. This proves the claim about the index. Suppose orb(j ) = 2. Then dj2 ≤ N ≤ 2 dj2 − 1 , and dj2 divides N ; therefore, N = dj2 , which implies that g is the direct product of two isomorphic simple algebras. We claim that End0 (Vi ) is an irreducible representation of gi , i ≤ 2. Suppose not. Then in the decomposition of End0 (Vi ) into g-irreducibles there are members of at least two distinct orbits of StabG (gi ) because the weight wi + ι(wi ) occurs only once (see [Bou, §7]) in End0 (Vi ). One of these orbits has dimension less than dj2 /2, and the G-stable subspace of End0 (V ) containing it has dimension less than d 2 , contradicting the assumption. It is known that End0 (Vi ), being irreducible, implies that Vi is the special linear algebra √ in its standard representation. If N > 4, the group of outer automorphisms of sl( N) has order 2 and is generated by the canonical involution. Moreover, neither V1 nor V2 is self-dual, so the index of (Gm · G0 ) ∩ G in G is 2. If N = 4, then the group of outer automorphisms is trivial; hence, the index is also 2. Finally, suppose that for all j , orb(j ) = 1. We still have √ N ≤ dj2 − 1 ⇒ dj ≥ N + 1. However, ;dj = N; therefore, N≥

√

k

N + 1 ⇒ k = 1,

and the Lie algebra of G is simple. 6. A classiﬁcation theorem. We now give general classiﬁcation theorems for pairs (g, V ), where g is a simple Lie algebra and V is a representation of g that satisﬁes a hypothesis similar to that of Theorem 5.1. The proof requires us to compare dimensions of various representations—a tedious task in general. However, the following lemma is often sufﬁcient. First, we introduce

MONODROMY OF AIRY AND KLOOSTERMAN SHEAVES

411

partial ordering on the set of numerical labels as follows. Let ! = (!i ), ! = (!i ) be two sets of numerical labels with respect to the same choice of Cartan algebra and numbering of simple roots. Then we say ! ≥ ! if !i ≥ !i . Lemma 6.1. Let ! and ! be dominant weights of a semisimple Lie algebra g. Suppose ! ≥ ! . Then dim R(!) ≥ dim R(! ), and the equality holds if and only if ! = ! . Proof. The proof follows from Weyl’s dimension formula. Proposition 6.2. Suppose V is an irreducible representation of a simple Lie algebra g, and let w be the highest weight of g in V . Suppose that V has dimension not exceeding the dimension of g, or, equivalently, that the dimension of V is smaller than or equal to the dimension of the adjoint representation of g. Then (g, w) is one of the following: (1) g = Al , l ≥ 2, w = π1 , 2π1 , π2 , πl−1 , πl , 2πl , π1 + πl , (2) g = Al , l = 5, 6, 7, w = π3 , πl−2 , (3) g = Bl , l ≥ 3, w = π1 , π2 , (4) g = Bl , l = 3, 4, 5, 6, w = πl , (5) g = Cl , l ≥ 1, w = π1 , 2π1 , π2 , (6) g = Dl , l ≥ 4, w = π1 , π2 , (7) g = Dl , l = 4, 5, 6, 7, w = πl−1 , πl , (8) g = E6 , w = π1 , π6 , π7 , (9) g = E7 , w = π1 , π7 , (10) g = E8 , w = π8 , (11) g = F4 , w = π1 , π4 , (12) g = G2 , w = π1 , π2 . Proof. The exceptional Lie algebras are handled by inspection (see [MPR, Chapter 6, Table 2]) and by repeatedly applying Lemma 6.1. We therefore deal only with the classical Lie algebras. Until further mention, suppose that g is of type Al , l ≥ 2. We make comparisons of dimensions of various representations using Lemma 6.1 and Table 1, which, for brevity, is not mentioned explicitly. The Dynkin diagram has a symmetry that maps πi to πl+1−i . Therefore, without loss of generality, we may assume that for some i ≤ l/2, the numerical label !i is not 0. Following Leitfaden 1, we distinguish several cases, which we box for clarity. !1 > 0, !2 = !l−1 = !l = 0, and w is not a multiple of π1 Then one of the labels !3 , . . . , !l−2 is nonzero so that l ≥ 5. Since dim R(πi ) ≥ dim R(π3 ) (3 ≤ i ≤ l − 2), dim R(π3 ) > dim g

(l ≥ 8),

412

ONDREJ ŠUCH

5 ≤ l< ≤ 7 zz zz z z zz 5 !l = D0D kkkk DD k k k DD kk k k k DD kkkk " !l−1 = 0 w = kπ , k = 1, 2 1 4 SSSS jjjj SSSS j j j SSSS jj SSS jjjj ) jjjj ! = 0 ⇒ w = π1 + πl !: 2 = 0 TT l TTTT vv TTTT vv v TTTT vv TTT* v v !l−1 = 0 ⇒ l ≤ 2 ⇒ kπ1 , k = 1, 2 !1 N= 0H HHH HH HH H$ = 0 ⇒ l ≤ 2 ⇒ w = π1 + πl ! 2 Al , l ! ≥ 2 !! !! !! !! !2 = 0 ⇒ 5 ≤ l ≤ 7 !! v: vv !! v v ! vvvv !l−1 = 0 ⇒ l ≤ 2, contradiction !1 = 0H 4 HH jjjj j HH j j HH jj HH jjjj $ jjjj !l = 0 ⇒ l ≤ 2 ⇒ w = π2 !2 = 0 T TTTT k5 k k TTTT k kkk TTTT kkkk TTTT k k k k * !l−1 = 0 w = kπ2 ⇒ w < = π2 TTTT zz TTTT zz TTTT z z TTTT zz ) !l = D0 DD DD DD D" 5≤l≤7

Leitfaden 1

413

MONODROMY OF AIRY AND KLOOSTERMAN SHEAVES

Table 1 Dimensions of Some Representations of the Lie Algebra Al Highest Weight 3π1 2π2 π1 + π 2 π1 + πl−1 π1 + π 3 π2 + π 3 π2 + πl−1 2π3 π3 + πl−1 π3 + π l π3 π4 π1 + π l

l

2

3

4

5

6

7

8

10

20

35

56

84

120

165

(l + 2)l(l + 1)2 /12

l +2 2 3

6

20

50

105

196

336

540

8

20

40

70

112

168

240

(l + 2)(l + 1)(l − 1)/2

l +2 3 4

l +2 (l + 1) 4

6

20

45

84

140

216

315

15

45

105

210

378

630

20

75

210

490

1008

1890

(l + 1)2 (l + 2)(l − 2)/4

l +2 l +1 1/4 3 3

20

75

189

392

720

1215

10

50

175

490

1176

2520

(l + 2)(l + 1)2 l(l − 3)/12

l +2 (l − 2) 3

l +1 3

l +1 4

20

50

210

588

1344

2700

10

40

105

224

420

720

4

10

20

35

56

84

5

15

35

70

126

24

35

48

63

80

l +3 3

l(l + 2)

8

15

we conclude that 5 ≤ l ≤ 7. We have dim R(π4 ) > dim g

(l = 7),

dim R(π1 + π3 ) > dim g

(l ≥ 4),

dim R(π1 + πl−2 ) = dim R(πl + π3 ) > dim g

(l ≥ 4),

and hence we conclude that this case could not have occurred.

414

ONDREJ ŠUCH

w = kπ1 Since dim R(3π1 ) > dim g

(l ≥ 3),

we conclude that w = π1 , or 2π1 . !1 , !l > 0, !2 = !l−1 = 0 Since dim R(π1 + πl ) = dim g, we conclude that w = π1 + πl . !1 , !l−1 > 0, !2 = 0 Since dim R(π1 + πl−1 ) > dim g

(l ≥ 3),

we conclude that l = 2, and therefore w = kπ1 , which is a case considered above. !1 , ! 2 > 0 Since dim R(π1 + π2 ) > dim g (l ≥ 3), we conclude that l = 2, and since dim R(π1 + π2 ) = dim g, we have w = π1 + π2 . !1 = ! 2 = 0 Since we assumed !i = 0 for some i ≤ l − 2, we have l ≥ 5. Since dim R(πi ) ≥ dim R(π3 ) dim R(π3 ) > dim g

3≤i≤

(l ≥ 8),

l , 2

we conclude that l ≤ 7. We have dim R(2π3 ) = dim R(2πl−2 ) > dim g dim R(π4 ) = 70 > 63 = dim g

(l ≥ 4),

(l = 7),

dim R(π3 + π4 ) = 784 > 48 = dim g dim R(π3 + π5 ) = 2352 > 63 = dim g

(l = 6), (l = 7),

which implies that for 5 ≤ l ≤ 7, only π3 , πl−2 may occur as highest weights of V . !2 , !l−1 > 0, !1 = 0 Since dim R(π2 + πl−1 ) > dim g

(l ≥ 3),

we conclude that l = 2, which is a contradiction because 0 = !1 and 0 < !1 . !2 , !l > 0, !1 = 0 Since dim R(π2 + πl ) = dim R(π1 + πl−1 ) > dim g

(l ≥ 3),

MONODROMY OF AIRY AND KLOOSTERMAN SHEAVES

415

we conclude that l = 2, so that w = k ·π2 , but since dim R(3π2 ) = 10 > 8 = rank A2 , we conclude that w = π2 = πl , or w = 2π2 = 2πl . w = k · π2 Since dim R(2π2 ) > dim g (l ≥ 3), we conclude that either l = 2, which was the case considered immediately above, or w = π2 . !2 > 0, !1 = !l−1 = !l = 0, w = k · π2 This case implies that one !3 , . . . , !l−2 does not vanish, so that l ≥ 5. If we have l ≥ 8, 3 ≤ i ≤ l − 2, then dim R(πi ) ≥ dim R(π3 ) > dim g, so that 5 ≤ l ≤ 7. We have dim R(π2 + π3 ) > dim g

(l ≥ 3),

dim R(π4 ) = 70 > 63 = dim g

(l = 7),

dim R(π2 + πl−2 ) = dim R(π3 + πl−1 ) > dim g

(l ≥ 3),

which shows that this case could not have happened. Suppose now that g is of type Bl , l ≥ 3. Then dim R(2π1 ) = 2l 2 + 3l > 2l 2 + l = dim g

(for l ≥ 3),

dim R(π2 ) = dim g, dim R(πi ) > dim g

(3 ≤ i ≤ l − 1),

dim R(πl ) > dim g

(l ≥ 7),

dim R(π1 + πl ) = l · 2l+1 > dim g (l ≥ 3),

2l + 1 > dim g (l ≥ 3), dim R(2πl ) = l which implies the desired conclusion for an algebra of type Bl . Now suppose that g is of type Cl , l ≥ 1. Then l(2l + 1)(2l + 2) > 2l 2 + l = dim g (l ≥ 1), 3 l(l − 1)(2l − 1)(2l + 3) dim R(2π2 ) = > dim g (l ≥ 2), 3 2l(2l − 2)(2l + 2) dim R(π1 + π2 ) = > dim g (l ≥ 2), 3 dim R(π3 ) > dim g (l ≥ 3), dim R(3π1 ) =

dim R(πi ) ≥ dim R(π3 ) ≥ dim g

(i, l ≥ 3),

which implies the desired conclusion for an algebra of type Cl .

416

ONDREJ ŠUCH

Now suppose that g is of type Dl , l ≥ 4. Then dim R(2π1 ) = (2l − 1)(l + 1) > (2l − 1)l = dim g dim R(π2 ) = dim g if 3 ≤ i ≤ l − 2,

(l ≥ 4),

(l ≥ 4), l ≥ 4 dim R(πi ) > dim R(π2 ) = dim g,

dim R(πl−1 ) = dim R(πl ) > dim g

(l ≥ 8),

dim R(π1 + πl ) = dim R(π1 + πl−1 ) = 2l−1 (2l − 1) > dim g (l ≥ 4),

2l > dim g (l ≥ 4), dim R(πl−1 + πl ) = l −1

2l + 1 dim R(2πl−1 ) = dim R(2πl ) = > dim g (l ≥ 4), l which implies the desired conclusion for an algebra of type Dl . One knows that most representations of simple Lie algebras are orthogonal (in fact, only Lie algebras of type Al , D2k+1 , and E6 admit non-self-dual representations, and only Lie algebras of type A4q+1 , B4q+1 , B4q+2 , Cl , and D4q+2 admit any symplectic representations. Therefore, it seems appropriate to provide a sublist of representations of Proposition 6.2 that are not orthogonal. Corollary 6.3. Suppose that V is an irreducible representation of a simple Lie algebra g that is not orthogonal. Suppose further that V has dimension not exceeding the dimension of g, or equivalently, the dimension of V is smaller than or equal to the dimension of the adjoint representation of g. Denote by w the highest weight of g in V . If V is symplectic, then (g, w) is one of the following: (1) A5 , w = π3 , (2) Bl , l = 5, 6, w = πl , (3) Cl , l ≥ 1, w = π1 , (4) D6 , w = π6 , (5) E7 , w = π7 . If V is not self-dual, then (g, w) is one of the following: (1) Al , l ≥ 2, w = π1 , 2π1 , πl , 2πl , (2) Al , l ≥ 4, w = π2 , πl−1 , (3) Al , l = 6, 7, w = π3 , πl−2 , (4) Dl , l = 5, 7, w = πl−1 , πl , (5) E6 , w = π1 , π6 . Proof. The proof follows from Proposition 6.2 by examination of Table 2. 7. Airy sheaves. Let k be the algebraic closure of a ﬁnite ﬁeld of characteristic p > 0. A Ql sheaf is called an Airy sheaf if it is lisse on A1k and irreducible as an I (∞) representation with Swan conductor one bigger than its rank. In this section, we derive some properties of Airy sheaves.

417

MONODROMY OF AIRY AND KLOOSTERMAN SHEAVES

Table 2 Lie Algebra Al Bl

Order of the Center

Self-Duality

Symplecticity

Conditions

Conditions

(l +1)/ gcd l + 1, li=1 i!i

!i = !l+1−i

2, if !l odd

None

1, otherwise 2, if i !2i+1 odd

Cl

None

1, otherwise

l ≡ 1 mod 4 !(l−1)/2 odd l ≡ 1, 2 mod 4 !l odd i

!2i+1 odd

4, if !l−1 + !l odd Dl , l odd

1, if !l−1 + !l even (l−3)/2 !2i+1 i=0

!l = !l−1

Orthogonal

+1/2(!l−1 + !l ) even

Dl , l even

2, otherwise (l−2)/2 1, if !l−1 + !l , !2i+1 even i=0

None

2, otherwise E6

1, if !1 + !5 − !3 − !6 ≡ 0 mod 3

!1 = ! 6

3, otherwise

!3 = ! 5

1, if !2 + !5 + !7 ≡ 0 mod 2

E7

1

!l + !l−1 odd Orthogonal

None

!2 + !5 + !7 odd

None

Orthogonal

2, otherwise

E8 , F 4 , G 2

l ≡ 2 mod 4

Lemma 7.1. The Fourier transform of an Airy sheaf is a character of π1 (A1 ) of Swan conductor greater than 1. Conversely, the Fourier transform of any character of π1 (A1 ) of Swan conductor greater than 1 is an Airy sheaf. Proof. The class of sheaves lisse on A1 with all ∞ breaks greater than 1 is stable under Fourier transform (see [K3, Class (1)]). Similarly, the class of sheaves lisse on A1 with an irreducible representation of I (∞) of break greater than 1 of exact denominator equal to the rank of the sheaf is also stable under Fourier transform (see [K3, Class (1ter)]). If Ᏺ is in either of these classes, then the Euler-Poincaré formula gives (FT ψ Ᏺ)0 = H 1 (A1 , Ᏺ), rank FT ψ Ᏺ = rank FT ψ Ᏺ0 = dim H 1 (A, Ᏺ) = Swan∞ Ᏺ − rank Ᏺ.

418

ONDREJ ŠUCH

The lemma now follows from the fact that for either of the above classes, the Swan conductors of Ᏺ(∞) and FT ψ Ᏺ(∞) are the same (see [K3, p. 222]). To avoid trivialities (see Example 7.3.1) we assume that the rank of an Airy sheaf is greater than 1. The length of an Airy sheaf Ᏺ is deﬁned to be the exponent r, such that the Fourier transform of Ᏺ is a character of exact order p r . This terminology is intended to suggest the correspondence of characters of π1 (A1 ) with Witt vectors (see Proposition 2.7). In view of the preceding lemma, it seems natural to start with an overview of characters of π1 (A1 ). Proposition 7.2. Let k be an algebraic closure of Fp , and let ᏸ be a rank-1 Ql sheaf lisse on A1 . (1) ᏸ is of ﬁnite order p r . (2) Given any faithful Ql -valued character φ of Z/p r Z, there exists a Witt vector f in Wr (k[x]) such that ᏸ corresponds to φ(f ) in the sense of Proposition 2.7. (3) ᏸφ(f ) is a Ql (ζpr ) sheaf lisse on A1k whose Swan conductor at ∞ is min max deg(fi )p r−1−i , f 0≤i≤r−1

where f = (f0 , . . . , fr−1 ) runs through all Witt vectors in Wr (k[x]) satisfying (2). (4) Suppose that fi , 0 ≤ i ≤ r − 1 have coefﬁcients in a ﬁnite subﬁeld F of k. Then ᏸφ(f ) is a lisse sheaf on A1F , pure of weight 0. For any ﬁnite overﬁeld E of F, and any x in E = A1 (E), the trace of Frobenius at x is given by trace Frobx,E | ᏸφ(f ) = φE f (x) := φ traceWr (E)/Wr (Fp ) f (x) . (5) The linear dual of ᏸφ(f ) is the sheaf ᏸφ(−f ) . When p > 2, then −f = − f0 , . . . , −fr−1 . (6) Two sheaves ᏸφ(f ) , ᏸφ(g) are isomorphic if and only if f − g lies in (F − 1)Wr (k[x]). Proof. The proof of (1) is Lemma 3.5. The proofs of (2) and (3) follow from Theorem 2.8 and Proposition 2.7 except for the fact that ᏸφ(f ) is a Ql (ζpr ) sheaf, which is clear since we pushed out a Z/p r Z torsor by a character of Z/pr Z. (4) is a standard result about pushouts of torsors (see [De]). To prove (5), we note that for any commutative ring A, there is a natural map r : A → Wr (A), given by r(x) = (x, 0, . . . , 0), and r−1 r(x) · (a0 , . . . ) = xa0 , x p a1 , . . . , x p ar−1 . (6) follows from Proposition 2.7. For future reference, we record the following result.

MONODROMY OF AIRY AND KLOOSTERMAN SHEAVES

419

Lemma 7.3. The following are equivalent: (1) H 0 (A1 , ᏸφ(f ) ) = 0, (2) h0 (A1 , ᏸφ(f ) ) = 1, (3) f lies in (F − 1)Wr (k[x]), (4) the character φ ◦χf corresponding to ᏸφ(f ) is trivial, (5) Hc2 (A1 , ᏸφ(f ) ) = 0, (6) h2c (A1 , ᏸφ(f ) ) = 1, (7) if f = (f0 , . . . , fr−1 ) and fi lies in F[x], then there exists no constant C, such that for every overﬁeld E of F, we have φE f (x) ≤ C · |E|1/2 . x∈E

Proof. Equivalence of (4) with (1), (2), (5), (6) follows directly from the deﬁnition of H 0 (A1 , Ᏻ) (resp., Hc2 (A1 , Ᏻ)) as the space of coinvariants (resp., invariants) of π1 (A1 ) when acting on a Ql sheaf Ᏻ. Equivalence of (3) with (4) follows from Proposition 2.7. Equivalence of (5) and (7) follows from the Lefschetz trace formula and the fact that Hc2 (A1 , ᏸφ(f ) ) is pure of weight 2, whereas Hc1 (A1 , ᏸφ(f ) ) is mixed of weight less than or equal to 1. In what follows, we use the notation Ᏺ = Ᏺψ,φ(f ),l := FT ψ ᏸφ(f ) .

Example 7.3.1. When Swan∞ (ᏸ) = 2, it is possible to compute the Fourier transform of ᏸ explicitly. We distinguish two subcases. First suppose that p > 2, so that by Theorem 2.8, r = 1. If we take f of the form ax 2 + bx + c, we may also assume that φ = ψ. We compute the trace of Frobenius at an E-valued point t of A1 : 2 trace Frobt,E | F Ft¯ = φE ax + bx + c φE (tx) x∈E

b+t 2 b+t 2 = φE a x + +c− 2a 2a x∈E  

b+t 2 2   = φE (a · y ) · φE c − . 2a

y∈E

This computation shows that the Fourier transform is again a character of order p of Swan conductor 2. When p = 2, then by Theorem 2.8, r = 2, and f may be taken of the form (ax, bx). Furthermore, we may assume that φ 2 (cx) = ψ(x). We can compute the

420

ONDREJ ŠUCH

trace of Frobenius at an E-valued point t of A1 : trace Frobt,E | Ᏺt¯ = φE (ax, bx) ψE (tx) x∈E

=

x∈E

=

x∈E

φE (ax, bx) φE (0, ctx) φE (ax, bx + ctx)

bc c c2 2 bc c t, t + 2 t φE ax + t, bx + 2 t + = a a a2 a a x∈E

bc c c2 2 bc = y, y · φE t, t + 2 t φE , a a a2 a y∈E

and again we see the Fourier transform is a character of order 4, of Swan conductor 2. Proposition 7.4. Let k be an algebraic closure of Fp . Suppose that the Swan conductor d of ᏸφ(f ) is greater than 1, and let Ᏺ = FT ψ ᏸφ(f ) be the corresponding Airy sheaf on A1 ⊗ k. (1) The sheaf Ᏺ has rank d − 1 and is lisse on A1 . (2) As I (∞) representation, all the breaks of Ᏺ are d/(d − 1), and therefore the I (∞) representation is irreducible. The Swan conductor of Ᏺ is d. (3) The sheaf Ᏺ is irreducible. (4) When f = (f0 , . . . , fn−1 ) with fi in F[x] for a ﬁnite ﬁeld F and i ≤ n − 1, then Ᏺ is naturally a lisse sheaf on A1F , and there it is pure of weight 1. For any ﬁnite overﬁeld E of F, and any t in E = A1 (E), the trace of Frobenius at t is given by trace Frobt,E | Ᏺt = − φE f (x) ψE (tx). x∈E

(5) The linear dual of Ᏺ = FT ψ (ᏸφ(f (x) ) is the sheaf FT ψ (ᏸφ(−f (−x)) )(1). (6) The sheaf Ᏺ is a Fourier sheaf. (7) Ᏺφ(f ) and Ᏺφ(g) are isomorphic as representations of π1 (A1 ⊗ k) if and only if f − g lies in (F − 1)Wn (k[x]). Proof. Since the unique break of ᏸφ(f ) is by assumption greater than 1, the Fourier transform of ᏸφ(f ) is lisse on A1 , and its rank is equal to Swan(ᏸφ(f ) ) − 1, which proves (1). Since ᏸφ(f ) is lisse on A1 , and FT ψ (ᏸψ(f ) )(∞) is the local Fourier transform FT loc(∞, ∞)ᏸφ(f ) (∞), they both have the same Swan conductor, and if one is irreducible, so is the other. Since any character is irreducible, Ᏺ(∞) is irreducible of rank d −1 and Swan conductor d;

MONODROMY OF AIRY AND KLOOSTERMAN SHEAVES

421

hence, it has d − 1 breaks d/(d − 1), which proves (2). (3) follows from (2). (4) is a direct consequence of Proposition 7.2(2). (5) follows from Proposition 7.2(3) and the fact that D FT (Ᏻ) = FT [−1]∗ D(Ᏻ) (1). (6) follows from (2). Finally, (7) is a direct consequence of Proposition 7.2(4). Most of our conclusions about Ᏺ are obtained by a careful study of local and global properties of the sheaf End0 (Ᏺ) := (Ᏺ ⊗ Ᏺ∨ )/1. In view of Proposition 7.4(1), it is then natural to assume that d = Swan(Ᏺφ(f ) ) > 2. Proposition 7.5. Let k be the algebraic closure of Fp . Assume that the Swan conductor d of ᏸφ(f ) is greater than 2, and let Ᏺ := FT ψ ᏸφ(f ) be the corresponding Airy sheaf. (1) The sheaf End0 (Ᏺ) has rank (d − 1)2 − 1 and is lisse on A1 . (2) All the breaks of the I (∞) representation of End0 (Ᏺ) are less than or equal to d/(d − 1). (3) The I (∞) representation of End0 (Ᏺ) has no nonzero invariants. (4) The Swan conductor of End0 (Ᏺ) is (d − 1)2 − 1. (5) The sheaf End0 (Ᏺ) is semisimple as a representation of π1 (A1k ). (6) We have H ∗ A1 , End0 (Ᏺ) = Hc∗ A1 , End0 (Ᏺ) = 0. (7) When f has coefﬁcients in a ﬁnite ﬁeld F, then End0 (Ᏺ) is naturally a lisse sheaf on A1F , and there it is pure of weight 0. For any ﬁnite overﬁeld E of F, and any t in E = A1 (E), the trace of Frobenius at t is given by 1 φE f (x) − f (−y) ψE t (x + y) . trace Frobt,E | End0 (Ᏺ)t = −1 + |E| x,y∈E

(8) The sheaf End0 (Ᏺ) is self-dual. × (9) The sheaf End0 (Ᏺ) is a Fourier sheaf if and only if for all w in k , the Witt vector f (x + w) − f (x) does not lie in (F − 1)Wn (k[x]). Proof. (1) and (2) follow from Proposition 7.4(1), (2), respectively. (3) follows from Proposition 7.4(2). (5) follows from Proposition 7.4(3). (7) follows from Proposition 7.4(4) and (5). (8) can be seen from the fact that End(Ᏺ) = Ᏺ ⊗ Ᏺ∨ = Ᏺ∨ ⊗ Ᏺ. Denote by j : A1 → P1 the natural compactiﬁcation of the afﬁne line. It follows from (3) that ∼ ∼ j! End0 (Ᏺ) −→ Rj∗ End0 (Ᏺ) −→ j∗ End0 (Ᏺ). It follows that Hci (A1 , End0 (Ᏺ)) = H i (A1 , End0 (Ᏺ)) are pure of weight i on F. By the Lefschetz trace formula, for any overﬁeld E of F, we have (−1)i trace FrobE | Hci A1 , End0 (Ᏺ) = trace Frobt,E | End0 (Ᏺ)t i

t∈E

422

ONDREJ ŠUCH

= −|E| +

t,x,y∈E

1 φE f (x) − f (−y) ψE t (x + y) |E|

1 φE f (x) − f (−y) ψE t (x + y) |E| t,x=−y∈E = −|E| + φE f (x) − f (−y) = 0.

= −|E| +

x=−y∈E

It follows that hic (End0 (Ᏺ)) = hi (End0 (Ᏺ)) = 0, which proves (6). (An observant reader will certainly notice that we proved the rigidity of Ᏺ, a sheaf property introduced and examined in more detail in [K4].) Inserting this information into the Euler-Poincaré formula yields rank End0 (Ᏺ) = Swan End0 (Ᏺ) , which proves (4). We established that End0 (Ᏺ) is a lisse sheaf on A1F , semisimple, with no inertial × invariants at ∞, and self-dual. Such a sheaf is Fourier if and only if, for all w ∈ F , we have Hc2 A1 , End0 (Ᏺ) ⊗ ᏸψ(wt) = 0. Since End0 (Ᏺ) is pure of weight 0, Hc2 is pure of weight 2, and Hc1 is mixed of weight less than or equal to 1. Therefore, by the Lefschetz trace formula, End0 (Ᏺ) is Fourier if and only if there exists a real constant Cw such that for any overﬁeld E of F(w), trace Frobt,E | End0 (F) ⊗ ᏸψ(wt) t ≤ Cw · |E|1/2 . t∈E

However, t∈E

trace Frobt,E | End0 (F) ⊗ ᏸψ(wt) t 1 φE f (x) − f (−y) ψE t (x + y + w) |E| t,x,y∈E = φE f (x) − f (−y) =

x,y∈E x+y+w=0

=

φE f (x) − f (x + w) .

x∈E

To require that the last expression to be less than or equal to Cw |E|1/2 is equivalent to f (x) − f (x + w) not lying in (F − 1)Wn (k[x]) (see Lemma 7.3).

MONODROMY OF AIRY AND KLOOSTERMAN SHEAVES

423

8. Geometric interpretation of Airy sheaves and duality. We give a geometric interpretation of Airy sheaves, which helps us demonstrate some of their more subtle properties. The following is a deep result on endomorphism algebras of abelian varieties. Theorem 8.1 (Zarhin, Mori). Let A and B be abelian varieties over a ﬁeld K, ﬁnitely generated over its prime ﬁeld. Then the natural map HomK (A, B) ⊗ Zl −→ HomZl [Gal(K sep /K)] Tl (A), Tl (B) is an isomorphism, and End(Tl (A) ⊗ Ql ) is a semisimple Ql [Gal(K sep /K)]-module. Proof. See [Mo, Théorème 2.5]. As a corollary of the preceding theorem, we have the following result on the decomposition of the Tate module of an abelian variety. Proposition 8.2. Let A be an abelian ri variety over a ﬁeld K, ﬁnitely generated over its prime ﬁeld. Suppose A = Ai , where Ai are pairwise nonisogenous Ksimple abelian varieties. Suppose that the endomorphism algebra End(Ai ) ⊗ Q is a central simple algebra of rank di over a number ﬁeld Ei . Let ei = [Ei : Q], and let Bi be the set of embeddings of Ei into Ql . Then rd Tl (A) ⊗ Ql = Vi,σi i , i

σ ∈Bi

where Vi,σ is an irreducible Ql [Gal(K sep /K)]-module of dimension 2 dim Ai /(di ni ). The traces of Gal(K sep /K) acting on Vi,σ lie in σ (Ei ). Proof. Clearly, it is sufﬁcient to consider the case of a K-simple abelian variety A. If dj Vj Tl (A) ⊗ Ql = j

is a decomposition of Tl (A) ⊗ Ql into its irreducible components, then the algebra End(Tl (A)) ⊗ Ql is the direct product of Mdj (Ql ). On the other hand, Theorem 8.1 says that End(Tl (A)) ⊗ Ql is End(A) ⊗ Ql , which is the direct sum of e copies of Md (Ql ) if End(A) ⊗ Q is a division algebra of rank d central simple over F , where [F : Q] = e. It follows that the number of isotypical components in Tl (A) ⊗ Ql is e, and each decomposes into d pairwise isomorphic irreducibles. To show that all irreducibles have the same dimension and prove the statement about traces, it is sufﬁcient to do the same for isotypical components themselves. Now quite generally, if we have a Ql vector space V together with an action of a group ring E[G], where E is a ﬁnite extension of Q, then V decomposes naturally into components on which E acts via different embeddings of E into Ql or via 0. Since A is assumed to be K simple, only nonzero embeddings occur, and since the traces of E lie in Q, all embeddings occur with equal multiplicity (see [ST, Chapter II, Lemma 1]).

424

ONDREJ ŠUCH

Now consider a 1-parameter family of Artin-Schreier-Witt curves, given by p

r

C −→ P1 ⊗ U −→ U, where U is a dense open subset of A1 and C admits a Z/p r Z action compatible with p, whose quotient is P1 ⊗ U . If q = r ◦p, then we obtain cohomology sheaves Ᏺ = R 1 q! Ql

on U . The Z/p r Z action carries over to Ᏺ, and we can decompose Ᏺ according to the characters of Z/p r Z. Suppose that the family C has an additional automorphism τ of order 2, such that if σ is the generator of Z/pr Z, then τ σ τ = σ −1 . ¯

¯

Then τ interchanges Ᏺφ and Ᏺφ . Since Ᏺφ and Ᏺφ are self-dual, that means that Ᏺφ is self-dual, since α, β := (α, τβ) is a pairing on Ᏺφ . The pairing is easily shown to be symplectic: β, α = (β, τ α) = (τβ, τ τ α) = (τβ, α) = −(α, τβ) = −α, β . To give the above construction concrete meaning, we suppose that C corresponds to a family ft (x) of Witt vectors, where ft (x) = (f0 (t, x), . . . , fr (t, x)) are polynomials in F[x, t], where F is a ﬁnite ﬁeld. We can compute, using the Lefschetz trace formula, that for any E-rational point t ∈ A1 , φ trace Frobt , E | Ᏺt¯ = − traceE/Fp φ ft (x) . x∈E

Example 8.2.1. When ft (x) = (f0 (x), f1 (x), . . . , fr (x)+tx), Ᏺφ is an Airy sheaf. To see this, compare its trace function with the one computed in Proposition 7.4(4). Assume that the sheaf Ᏺφ is self-dual. Note that this is automatic when p = 2 and r = 1, and never happens if p = 2 and r > 1. Indeed, a necessary and sufﬁcient criterion for an Airy sheaf to be self-dual is that it be the Fourier transform of a character χ of π1 (A1 ) such that [−1]∗ χ = χ −1 . In characteristic 2, [−1]∗ is the identity functor, and only characters of order 2 are self-dual. Moreover, if p = 2 and r = 1, then clearly the Airy sheaf is symplectic, since it is the whole of the cohomology of an Artin-Schreier curve. So suppose that p > 2, so that by Proposition 7.4(5) and (7), f0 (x) = −f0 (−x) + (F − 1)g(x). For a point (y, x, t) in C, the automorphism σ maps it to (y + 1, x, t), where (x, t) is a point of A1 × U . The map τ : (y, x, t) → (−y, −x, t) is then an example of an automorphism of a family C as above, and we conclude that any self-dual Airy sheaf is symplectic. Example 8.2.2. Suppose that p > 2 and let f (x) be an “odd” Witt vector, that is, f (x) = −f (−x) + (F − 1)g(x).

MONODROMY OF AIRY AND KLOOSTERMAN SHEAVES

425

Let ft (x) = f (x) + f (t + x). As in Example 8.2.1, we obtain a family of curves. The map τ : (y, x, t) → (−y, −t − x, t) is then an automorphism τ satisfying the assumptions above, so we conclude that any component Ᏺφ is symplectic. Example 8.2.3. Let f (x) be an arbitrary Witt vector in Wr (k[x]), where k is the algebraic closure of Fp . Take ft (x) = f (x) − f (x − t). As remarked above, the trace of Ᏺφ at an E-valued point t of A1 is φ φ f (x) − f (t − x) . trace Frobt,E | Ᏺt¯ = x∈E

On the other hand, using Proposition 7.5(7), we ﬁnd that for t = 0, trace Frobt,E | FT ψ¯ End0 Ᏺφ(f ),t¯ 1 φE f (u) − f (−v) ψE x(u + v) ψ(−xt) = |E| u,v,x∈E

1 = |E|φ f (u) − f (−v) |E| u,v∈E u+v−t=0

φ f (x) − f (x − t) . = x∈E

This shows that the Fourier transform FT ψ¯ End0 (Ᏺφ(f ) ) is the sheaf Ᏺφ for the family of Witt-Artin-Schreier curves given by ft (x) = f (x) − f (x − t). Example 8.2.4. The p = 2 case of Examples 8.2.2 and 8.2.3 is of particular interest. Let ft (x) = f (x) + f (t + x), but make no assumption on f (x). Note that the ﬁbers over x and t +x of this family are canonically isomorphic. Let σ be a generator n−1 of Z/2n Z. Then the map τ : (y, x, t) → (σ 2 y, x + t, t) is an automorphism τ as above, so we conclude that any component Ᏺφ is symplectic. We note that the trace function of this sheaf is φ f (x) + f (t + x) = φ f (x) + f (t − x) , x

x

which for all x = 0 is identical with the Fourier transform of End0 (Ᏺφ(f ) ) (see Proposition 7.5). Therefore, if End0 (Ᏺφ(f ) ) is Fourier, then its Fourier transform is symplectic. It is convenient to use the following deﬁnitions. Deﬁnition. Let K be an algebraically closed ﬁeld of characteristic 0. A ﬁnitedimensional K-representation of a group G is called Lie irreducible if it is irreducible, and it stays irreducible when restricted to every subgroup of ﬁnite index of G. Equivalently, an irreducible representation ρ is Lie irreducible when the restriction to the Lie algebra of the Zariski closure of the image of ρ is irreducible. A K-representation of a group G is called Lie self-dual (resp., Lie orthogonal, Lie symplectic) if it becomes

426

ONDREJ ŠUCH

self-dual (resp., orthogonal, symplectic) when restricted to a subgroup of ﬁnite index of G. Proposition 8.3. Let k be an algebraic closure of Fp , and let Ᏻ be a Lie irreducible lisse sheaf on a curve C/k. Then the following are equivalent: (1) Ᏻ is self-dual when viewed as a representation of a subgroup of π1 (C) of ﬁnite index. (2) Ᏻ ⊗ Ᏻ has a 1-dimensional quotient on which π1 (C) acts through ﬁnite quotient. Furthermore, if C = A1 and Ᏻ is a Lie irreducible lisse sheaf, all of whose ∞breaks are less than 2, then (1) is equivalent to the following: (3) There exists w in k such Ᏻ ⊗ Ᏻ has the sheaf ᏸψ(wt) for a quotient. If, in addition, Ᏻ = Ᏺ := FT ψ ᏸφ(f (x)) , and Ᏻ is Lie irreducible, then (1) is equivalent to the following: (4) There exists w in k such that f (x) + f (−w − x) lies in (F − 1)Wr (k[x]). (5) There exists w in k such that Ᏺφ(f (x−w )) is self-dual. (6) Ᏺ is a symplectic representation of a subgroup of π1 (A1k ) of ﬁnite index. Proof. (1) and (2) are equivalent for any Lie irreducible representation of any group. (3) clearly implies (2). To prove the converse, we note that any Ql -valued character of π1 (A1k ) of break less than 2 has a break 1 or 0, hence, by the “break depression lemma” of [K2, 8.5.7] is of the form ᏸψ(wx) for some w ∈ k. If the coefﬁcients of f lie in a ﬁnite ﬁeld F, then the sheaf Ᏺ ⊗ Ᏺ is a lisse sheaf on A1F , where it is pure of weight 2. We need to check whether or not Hc2 A1k , Ᏺ ⊗ Ᏺ ⊗ ᏸψ(wt) = 0. This is equivalent to the existence of a constant C such that for all ﬁnite overﬁelds E of F, the sum trace Frobt,E | Ᏺ ⊗ Ᏺ ⊗ ᏸψ(wt) = φE f (x) + f (y) ψE t (x + y + w) t∈E

t,x,y,∈E

= |E|

φE f (x) + f (y)

x,y=−x−w∈E

= |E|

φE f (x) + f (−w − x)

x∈E

is bounded by |E|3/2 . By Lemma 7.3, this is the case if and only if the Witt vector f (x) + f (−w − x) lies in (F − 1)Wr (k[x]), which shows the equivalence of (3) and (4). Now suppose that (3) holds, so that there is a w in k such that Ᏺφ(f ) has ᏸψ(wt) for a quotient. If p > 2, then Ᏻ := FT ψ [w/2]∗ (ᏸφ(f ) ) is Ᏺ ⊗ ᏸψ(−wx/2) so that Ᏻ ⊗ Ᏻ = Ᏺ ⊗ ᏸψ(−wx)

MONODROMY OF AIRY AND KLOOSTERMAN SHEAVES

427

has a constant sheaf for a quotient, so Ᏻ is self-dual. If p = 2, then any Airy sheaf of level 1 is self-dual, and by Example 8.2.1, no Airy sheaf of higher level is self-dual. Conversely, if Ᏺφ(f (x−w )) is self-dual, then Ᏺφ(f ) contains ᏸψ(2wx) as a quotient. Finally, suppose (1) holds, so by (5) we may assume that Ᏺ is self-dual. Then the result is proved in Example 8.2.1. 9. Kloosterman sheaves. Let k be an algebraic closure of Fp . Any sheaf Ᏺ lisse on Gm ⊗ k that is tame at 0, totally wild at ∞, and of Swan conductor 1 is called a Kloosterman sheaf. Such sheaves were studied in great detail in [K2]. There Katz introduced sheaves Kl n (ψ, χ1 , . . . , χn ) and showed that they are prototypes for more general Kloosterman sheaves in the following sense. Theorem 9.1 (Katz). Let k be an algebraic closure of Fp and let Ᏺ be a Kloosterman sheaf on Gm ⊗ k. Suppose that all the eigenvalues of geometrical local monodromy at 0 are q − 1’st roots of unity. Then there exist  ×  n = rank(Ᏺ) multiplicative characters χi : Fq −→ Ql , a point b ∈ F× = Gm (F),   a lisse, rank 1, geometrically constant Ql − sheaf T , such that

∼

Ᏺ ⊗ T −→ (Transb )∗ Kl(ψ; χ1 , . . . , χn ) .

Proof. See [K2, Theorem 8.7.1]. 10. General approach to computation of monodromy. Let k be an algebraically closed ﬁeld of characteristic p > 0, let C be a curve over k, and let Ᏺ be a lisse Ql sheaf on C. Choosing a geometric point x in C, Ᏺ corresponds to a representation ρ of the fundamental group π1 (C) of the curve on the ﬁnite-dimensional Ql -vector space Ᏺx . We denote by Ggeom the Zariski closure in GL(Ᏺx ) of the image of the fundamental group in this representation. Sometimes one expects that the group Ggeom is “big” in the sense that Ggeom is one of the classical groups SL(V ), SO(V ), Sp(V ). A ﬁrst step towards establishing such a result is determining whether the action of the Lie algebra ggeom of Ggeom is irreducible. We use the following result of Katz. Proposition 10.1 (Katz). Suppose that Ᏺ is a rank n irreducible Ql -sheaf on an afﬁne curve over an algebraically closed ﬁeld k of characteristic p > 0. Then either ρ is induced from a representation of a proper open subgroup of π1 , or ρ is Lie irreducible, or there exists an integer d ≥ 2 dividing n and a factorization of ρ as a tensor product τ ⊗ ω, where τ is a Lie irreducible representation of π1 of dimension n/d, and where ω is an irreducible representation of π1 of dimension d that factors through a ﬁnite quotient of π1 . Proof. See [K1, Proposition 1].

428

ONDREJ ŠUCH

The following corollary is simple to prove but very useful in applications. Corollary 10.2. Using the assumptions as in Proposition 10.1, suppose Ᏺ is neither Lie irreducible, nor a ﬁnite sheaf (in the sense that ρ does not factor through a ﬁnite quotient of π1 ). Then End0 (Ᏺ) contains a nonzero subsheaf of dimension less than n. More precisely, if Ᏺ = g∗ Ᏻ, then End0 (Ᏺ) ⊃ g∗ Ql /Ql , and if Ᏺ = Ᏻ ⊗ Ᏼ, then End0 (Ᏺ) ⊃ End0 Ᏻ, where Ᏻ is chosen so that rank Ᏻ ≤ rank Ᏼ. Proof. By Proposition 10.1, we may assume that ρ is either induced or a tensor product, both in a nontrivial way. If ρ is induced, then Ᏺ = g∗ Ᏻ, where g is a ﬁnite étale cover of U , say, of degree d dividing n. Then by Lemma 2.4, End Ᏺ = Ᏺ ⊗ Ᏺ∨ = g∗ Ᏻ ⊗ g∗ Ᏻ∨ ⊃ g∗ End Ᏻ ⊃ g∗ Ql , which implies End0 Ᏺ contains g∗ Ql /Ql , which has rank deg g − 1 < n. If ρ is a tensor product Ᏺ = Ᏻ ⊗ Ᏼ, then End Ᏺ = Ᏺ ⊗ Ᏺ∨ = Ᏻ ⊗ Ᏻ∨ ⊗ Ᏼ ⊗ Ᏼ∨ = 1 + End0 (Ᏻ) ⊗ 1 + End0 (Ᏼ) = 1 + End0 Ᏻ + End0 Ᏼ + End0 Ᏻ ⊗ End0 Ᏼ . √ Without loss of generality, we may assume that rank Ᏻ ≤ n, so that rank End0 Ᏻ ≤ n − 1, which ﬁnishes our proof. A general procedure for showing that Ggeom is “big” is to proceed as follows: (1) show that Lie Ggeom acts irreducibly using Corollary 10.2; (2) show that Lie Ggeom is simple using Theorem 5.1; (3) show that Lie Ggeom is a classical group using Proposition 6.2 or Corollary 6.3. In the next two sections, we apply this procedure to Airy and Kloosterman sheaves. In the course of doing so, several lemmas prove useful. All of them deal with the following situation. Given a linear representation of Ggeom , say, ϒ : Ggeom −→ GL(d), the composite representation ϒ ◦ρ : π1 −→ GL(d) gives rise to a lisse Ql -sheaf of rank d on C, denoted by Ᏺ(ϒ). Lemma 10.3 (Highest slope lemma). Using the same notation as above, suppose that the kernel of ϒ : Ggeom → GL(d) is a ﬁnite subgroup of order prime to p. Then at every point x ∈ C¯ − C, Ᏺ and Ᏺ(ϒ) have the same highest break. Proof. (See [K3, Lemma 7.2.4]).

MONODROMY OF AIRY AND KLOOSTERMAN SHEAVES

429

Lemma 10.4 (Lifting lemma). Let k be an algebraically closed ﬁeld, let U/k be an afﬁne curve, and let l be a prime different from characteristic p of k. Suppose that ρ : G → H is a surjective homomorphism of linear algebraic groups over Ql , whose kernel is a ﬁnite central subgroup of G. Then any continous homomorphism φ : π1 (X) → H (Ql ) lifts to a homomorphism φ˜ : π1 (X, x) −→ G(Ql ) ˜ with φ = ρ φ. Proof. See [K3, Lemma 7.2.5]. It is somewhat awkward to use previous lemmas, because in practice we do not know what the geometric monodromy group is and only have some (possibly incorrect) guess as to its Lie algebra. In these situations, we use the following combined version of Lemmas 10.4 and 10.3. To formulate it, we recall that automorphisms of Dynkin diagram of a semisimple Lie algebra act naturally on the weights of the Lie algebra. Lemma 10.5. Suppose that k is an algebraically closed ﬁeld of characteristic p > 0. Consider quadruples (Ᏺ, g, !1 , !2 ) with the following: Ᏺ is a lisse Lie irreducible Ql sheaf on an afﬁne curve C, g is a semisimple Lie algebra, which is Lie Ggeom , !1 is a dominant weight of g, which is the highest weight of Lie Ggeom in its representation in GL(Ᏺx ), x is any geometric point of C, and !2 is another dominant weight of g. Suppose further that (i) R(!1 ) and R(!2 ) as representations of the universal covering group of G, and both have kernels of ﬁnite order prime to p; (ii) the geometric monodromy group G of Ᏺ lies inside Gm · G0 . Then there exists a lisse sheaf Ᏻ on C such that (1) the Lie algebra of the geometric monodromy group of Ᏻ is g in the representation R(!2 ), (2) at every point p of C, Ᏻ(p) and Ᏺ(p) have the same highest break. Proof. Suppose that we ﬁnd linear algebraic groups G1 , G2 , their faithful representations ϒ1 , ϒ2 , and homomorphisms φ1 : G1 → Ggeom (Ᏺ), φ2 : G1 → G2 satisfying the following: (P1) Lie G1 = Lie G2 = g, (P2) ϒ1 , ϒ2 are faithful representations, (P3) ϒ2 has highest weight !2 , (P4) φ1 , φ2 have kernels prime to p. Then by Lemma 10.4, we may construct a sheaf Ᏻ# with monodromy group G1 , and then deﬁne Ᏻ := Ᏻ(ϒ2 ). By construction, Ᏻ satisﬁes (1), and by (P4), it satisﬁes (2). Let n := [G : ((Gm · G) ∩ G)]. Let c1 , c2 be the order of centers of the universal cover G#1 of (Ggeom )0 and of the image G#2 of G#1 in R(!2 ). We deﬁne n2 = c2 n/Z (Ggeom )0 . n1 = c1 n/Z (Ggeom )0 ,

430

ONDREJ ŠUCH

For i = 1, 2, we deﬁne Gi as the central product G#i ×Z(G# ) Z/ni . i We let φ1 and φ2 be the natural homorphisms guaranteed by the universal property of group products. We take for ϒ1 any faithful representation of G1 . We take for ϒ2 the tensor product representation of the product group G# × Z/ni . Properties (P1)–(P4) are satisﬁed by construction. 11. Monodromy of Airy sheaves. Our ﬁrst goal is to show that without loss of generality, we may restrict our attention to sheaves that are Lie irreducible. Proposition 11.1. Let k be the algebraic closure of Fp . Let Ᏺ be an Airy sheaf on 1 Ak , and assume that Ᏺ is not ﬁnite. Then Ᏺ is either Lie irreducible or Artin-Schreierinduced. Moreover, if it is induced, then it is induced from another Airy sheaf. Finally,

Ᏺ is induced if and only if End0 (Ᏺ) is not Fourier.

Proof. Assume that Ᏺ is neither Lie irreducible nor ﬁnite. Our ﬁrst task is to show that Ᏺ is induced. By Corollary 10.2, End0 Ᏺ has a subsheaf of rank < n := rank Ᏺ. Let Ᏻ be any subsheaf of End0 Ᏺ of rank less than n. Since all breaks of Ᏺ at ∞ are equal to (n+1)/n, all the breaks of Ᏻ are less than or equal to (n+1)/n. But the rank of Ᏻ is less than n; therefore, by the integrality of Swan conductors, all the breaks of Ᏻ are less than or equal to 1. It follows by Corollary 3.3 that Ᏻ is a direct sum of various ᏸψ(ai x) and the constant sheaf. Since End0 Ᏺ does not contain a nonzero constant subsheaf, it follows that Ᏻ is a sum of characters of order p. Suppose that Ᏺ is a tensor product Ᏼ1 ⊗ Ᏼ2 , where rank Ᏼ1 ≤ rank Ᏼ2 . Denote by the image of π1 in Ᏼ1 . Since det Ᏼ1 is a character of π1 (A1 ), it is ﬁnite by Lemma 3.5. Hence, the center Z of is cyclic and isomorphic to some group of roots of unity, and /Z is abelian and isomorphic to a subgroup of (Z/p)r . Therefore, is supersolvable, so that Ᏼ1 is induced from a character. Therefore, Ᏺ itself is induced, so we may write Ᏺ = g∗ Ᏼ ,

where g is a ﬁnite étale cover of A1 , through which Ᏺ is induced. In this case, we may take Ᏻ := g∗ Ql /Ql . Now g is a subcover of the covering trivializing the representation g∗ Ql /Ql , which by the above discussion of Ᏻ, is a sum of characters of Swan conductor 1. We note that any such character has order p. Therefore, g is a subcover of the cover trivializing all of the characters occurring in g∗ Ql . The cover trivializing those characters is abelian with Galois group of type (p, . . . , p). Therefore, the cover itself is abelian with elementary abelian Galois group, and factoring it as a succession of p-covers, we see that Ᏺ = g∗ Ᏼ, where g is a ﬁnite étale Z/p-cover of A1 , of Swan conductor 1. The sheaf Ᏼ is lisse on A1 , and because Ᏺ(∞) is irreducible, Ᏼ(∞) is also irreducible. In particular, Ᏼ(∞) has a unique break λ at ∞. If λ ≤ 1, then by Corollary 3.3, Ᏼ would be a sum of ᏸψ(ai x) , and hence ﬁnite, but that contradicts our assumption that Ᏺ is not ﬁnite. By local Fourier transform t, the Fourier transform of Ᏼ is lisse on A1 , and to compute its rank it is

MONODROMY OF AIRY AND KLOOSTERMAN SHEAVES

431

sufﬁcient to compute its rank at a single point. We do so at the point 0, where the ﬁbre of FT ψ Ᏼ is H 1 A1 , Ᏼ = H 1 A1 , g∗ Ᏼ = (FT ψ Ᏺ)0 , which is 1-dimensional by Lemma 7.1. Therefore, by Lemma 7.1, Ᏼ is an Airy sheaf. We already showed that if Ᏺ is induced, then End0 (Ᏺ) contains a subsheaf consisting of characters of break 1; hence, End0 (Ᏺ) is not Fourier. Conversely, if End0 (Ᏺ) is not Fourier, then End0 (Ᏺ) (which is semisimple because of its purity (see Proposition 7.5(7))), contains characters of π1 (A1 ) of ﬁnite order, and hence invariants of ggeom , which shows that the ggeom does not act irreducibly on Ᏺ. Deﬁnition. Let V be a completely reducible representation of a group. We say that an irreducible summand W is dimensionally unique if it has dimension that is unique among dimensions of irreducible summands of V . The following lemma, special to characteristic 2, is used many times. Lemma 11.2. Suppose k is an algebraic closure of F2 and Ᏺ is an Airy sheaf on A1 ⊗ k, such that End0 (Ᏺ) is Fourier. Every dimensionally unique irreducible summand of End0 (Ᏺ) has rank at least 2(rank Ᏺ + 1), and its Fourier transform has rank at least 2. Proof. By Lemma 3.4 and Proposition 7.5(1) and (4), it is sufﬁcient to prove the second claim. Assume the contrary. Then we have a subsheaf W of End0 (Ᏺ) such that FT ψ W has rank 1. On the other hand, the Fourier transform of W is a subsheaf of the symplectic sheaf constructed in Example 8.2.4. Since FT ψ W is not symplectic, FT ψ End0 (Ᏺ) also contains its dual D(FT ψ W ), and by Fourier inversion, End0 (Ᏺ) would contain another summand of the same dimension as W . Starting with an Airy sheaf that is not ﬁnite, and applying Proposition 11.1 inductively, we arrive at a Lie irreducible Airy sheaf. The next step is to show that the Lie algebra of this sheaf is simple. Proposition 11.3. Suppose that Ᏺ is a Lie irreducible Airy sheaf. Then ggeom is a simple Lie algebra. Proof. Denote by n the rank of Ᏺ. Since Ᏺ is Lie irreducible, by Proposition 11.3, End0 (Ᏺ) is Fourier. So by Lemma 3.2, every irreducible summand of End0 (Ᏺ) has a break λ > 1. But by Proposition 7.5(2), λ ≤ (n+1)/n. Therefore, the assumptions √ √ of Theorem 5.1 are satisﬁed, and we have that ggeom is either simple or sl( n)×sl( n) or sl(2) × sl(2) × sl(2). √ √ Suppose that ggeom = sl( n) × sl( n). By Theorem 5.1, the group Ggeom has a subgroup of index 2, which corresponds to a ﬁnite étale cover of A1 of degree 2, which is possible only when the characteristic is 2. In that case, End0 (Ᏺ) contains a Ggeom irreducible summand of dimension less than 2n with multiplicity 1, namely, it is the

432

ONDREJ ŠUCH

component that, as a ggeom representation, is the sum of two adjoint representations of simple factors of ggeom . This is a contradiction by Lemma 11.2. Finally, suppose that ggeom = sl(2) × sl(2) × sl(2). By Theorem 5.1, the group Ggeom has a normal subgroup with quotient either S3 or A3 . In the former case, Ggeom has a normal subgroup of index 2, so that p = 2, whereas in the latter it has a normal subgroup of index 3, so that p = 3. Let W be the subsheaf of End0 (Ᏺ) that corresponds to the sum of the adjoint representations of the simple factors of ggeom . When p = 2, the existence of such a W contradicts Lemma 11.2. If p = 3, then p 8 = rank Ᏺ, so by Lemma 2.10 and Proposition 7.5(1) and (4) we may write W (∞) = χ + W0 , where W0 has break 9/8 and χ is a tame character of I (∞). Then End W (∞) = (χ + W0 ) ⊗ χ¯ + W0∨ = 1 + W0 ⊗ χ¯ + W0∨ ⊗ χ + End(W0 ). By Lemma 2.10, it follows that breaks of End(W ) are 0 and 9/8. When restricted to (Ggeom )0 , W splits into three distinct representations of (Ggeom )0 , so that W is induced from a subgroup of index 3, say, W = g∗ W1 . Hence, End W contains K = g∗ Ql /Ql , but since it can have only breaks 0 or 9/8, it is tame, and hence constant, but that is impossible, since g was a connected cover of A1 . Lemma 11.4. Let Ᏺ be a Lie irreducible Airy sheaf. Then the pair consisting of ggeom and its highest weight in the representation that is Ᏺ is one of those listed in Corollary 6.3. Proof. From Proposition 11.3, we deduce that ggeom is a simple Lie algebra. Since End0 (Ᏺ) is Fourier (see Proposition 11.1), by Lemma 3.2, every irreducible summand has a break λ > 1, λ ≤ 1 + 1/ rank Ᏺ. Since any outer automorphism ﬁxes the adjoint representation of the Lie algebra, by the integrality of Swan conductors, the representation is one of those listed in Proposition 6.2. In view of the duality results obtained in Proposition 8.3, it is actually one of those listed in Corollary 6.3. Lemma 11.5. Let Ᏺ be a Lie irreducible Airy sheaf. Then Ggeom is a subgroup of Gm · (Ggeom )0 . The index of (Ggeom )0 in Ggeom is 1 or p. Proof. If there were an element σ in Ggeom not lying in Gm · (Ggeom )0 , then conjugation by σ would induce a nontrivial outer automorphism of ggeom . By Lemma 11.4, the pair ggeom and its highest weight is one of those listed in Corollary 6.3, and among those only sl(6) in its third fundamental representation admits such an automorphism. The index of Ggeom ∩ (Gm · (Ggeom )0 ) in Ggeom then divides 2, since there is only one nontrivial automorphism of the Dynkin diagram that preserves the third fundamental weight. But π1 (A1 ) has no normal subgroups of index prime to p, hence p = 2. But that leads to a contradiction since End0 (Ᏺ) contains a 35dimensional summand, contradicting Lemma 11.2. Consider the cyclic étale cover corresponding to the subgroup (Ggeom )0 of Ggeom = (Gm ·(Ggeom )0 )∩Ggeom . Since π1 (A1 ) has no normal subgroups of order prime to p,

MONODROMY OF AIRY AND KLOOSTERMAN SHEAVES

433

the index is a power of p. Since all the ∞-breaks of Ᏺ are less than 2, we conclude by Theorem 2.8 that the covering is of degree p. Proposition 11.6. Suppose k is an algebraic closure of Fp , p > 2, and Ᏺ is a Lie irreducible Airy sheaf. Then the Lie algebra Lie Ggeom is either sl(d) or sp(d) in its standard representations, d = rank Ᏺ. Proof. By Lemma 11.4, Lie Ggeom and its weight is one of those listed in Corollary 6.3. We can exclude the symplectic cases (2) and (4), as well as the non-self-dual cases (2), (3), (4), and partially the non-self-dual case (1) of those from Corollary 6.3. Indeed, upon application of Lemma 10.5 to one of (Ᏺ, Bl , πl , π1 ), (Ᏺ, D6 , π6 , π1 ), (Ᏺ, Al , 2π1 , π1 ), (Ᏺ, Al , π2 , π1 ), (Ᏺ, Al , π3 , π1 ) l = 6, 7, (Ᏺ, Dl , πl , π1 ), (Ᏺ, Dl , πl−1 , π1 ), we arrive at a contradiction of the integrality of the Swan conductor. By virtue of Lemma 11.5, we can also apply Lemma 10.5 to (Ᏺ, A5 , π3 , π1 ) excluding the symplectic case (1) as well. The remaining possibilities are ggeom = sl(d) or ggeom = sp(2d) in their standard representations, or ggeom = E7 or ggeom = E6 in their 56- and 27-dimensional representations, respectively. We now prove that the latter two possibilities cannot arise. First, we restrict characteristics p in which these E7 or E6 cases could possibly occur. Recall from Proposition 7.5(6) and Lemma 3.1 that the average break of any irreducible summand of End0 (Ᏺ) is 1. On the other hand, if p rank Ᏺ, then in the break decomposition of End0 (Ᏺ)(∞), only 0 and 1 + 1/ rank Ᏺ occur (see Lemma 2.10). We conclude that if p rank Ᏺ, any irreducible summand of End0 (Ᏺ) has dimension divisible by rank Ᏺ +1. Examining Table 3, we see that if ggeom = e7 , then p = 7, and if ggeom = e6 , then p = 3. Suppose that ggeom = e7 in its 56-dimensional representation. Then Sym2 contains a 133-dimensional, irreducible summand V , namely, the adjoint representation of e7 . By Lemma 2.6, Sym2 (Ᏺ) also contains four tame characters at ∞, all of order 8. By Corollary 4.3, at least one of these characters lies in V . Taking the Fourier transform of V , we obtain a sheaf W of rank less than or equal to 2 (see Lemma 3.4). By local Fourier transform theory, inertia at 0 acts on W by a character χ0 of order 8. If W is 1-dimensional, then the ﬁeld generated by the traces of inertia is the ﬁeld Q(ζ8 ) ⊃ Q(i). If W is 2-dimensional, let χ1 denote the other character through which I (0) acts. Let α ∈ I (0) be an element such that χ0 (α) is an eighth root of unity, and suppose χ1 (α) is an mth root of unity, where m = 2ν m0 , where 2 m0 . Then χ0 (α m0 ) is again an eighth root of unity and χ1 (α m0 ) is a 2ν th root of unity ξ . Therefore, the ﬁeld generated by traces of inertia contains (ξ + ζ8 )2 − ξ 2 − ζ82 = 2ξ ζ8 . If Q(ξ ζ8 ) does not contain Q(ζ8 ), then ξ is an eighth root of unity, which is the complex conjugate to ζ8 . Therefore, we have that the ﬁeld generated by traces of I (0) acting on W contains Q(i).

434

ONDREJ ŠUCH

Table 3 Decompositions of End0 (V ) of Lie Algebras from Proposition 6.2 Lie Algebra

Highest Weight of V

Dimension of V

A5

π3

20

B5

π5

32

B6

π6

64

Cl

π1

2l

D6

π5 π6

E7

π7

Al

π1 , π l

Al

π2 , πl−1

A6

π3 , π4

35

A7

π3 , π5

56

D5

π5 , π4

16

D7

π6 , π7

64

E6

π1 , π6

27

32

56 l +1 l +1 2

Decomposition of End0 (V ) π1 + π 5 π2 + π 3 2π3 π1 , π 2 , π 3 π4 , 2π5 π1 , π 2 , π 3 π3 , π 4 π5 , 2π6 2π1 π2 2π5 2π6 π4 π2 2π7 π6 π1

35 189 175 11, 55, 165 330, 462 13, 78, 286 286, 715 1287, 1716 (2l + 1)l (2l + 1)(l − 1)

π1 + π l

(l + 2)l

π2 + πl−1

(l + 1)2 (l + 2)(l − 2)/4

π1 + π l π3 + π 4 π2 + π 5 π1 + π 6 π3 + π 5 π2 + π 6 π1 + π 7 π5 + π 4 π2 π6 + π 7 π2 π4 π2 π1 + π 6

l(l + 2) 784 392 48 2352 720 63 210 45 3003 91 1001 78 650

462 495 66 1466 1539 133

MONODROMY OF AIRY AND KLOOSTERMAN SHEAVES

435

Next we claim that W is a dimensionally unique summand of FT ψ End0 (Ᏺ). Suppose that was not the case, so that FT ψ End0 (Ᏺ) contained another ≤ 2-dimensional subspace W1 . We know (see Example 8.2.3) that FT ψ End0 (Ᏺ) is the φcomponent of the cohomology of a family of curves given by the Witt vector in variable t: ft (x) = f (x) − f (x − t). Moreover, since dim Ᏺ = 56, f has form (. . . , x 57 + · · · ), ft (x) = (. . . , . . . + cx 50 t 7 + · · · ), so that the generic rank of FT ψ (End0 (Ᏺ)) is greater than or equal to 49. It follows that the rank of the remaining summand of FT ψ (End0 (Ᏺ)) is greater than or equal to 44, which implies by Lemma 3.4 that End0 (Ᏺ) has a direct summand of dimension greater than or equal to 2, 464. This contradicts the decomposition End0 (Ᏺ) = Sym2 (Ᏺ) + !2 (Ᏺ), in which both summands have dimension less than or equal to 1, 596. Having established that W is dimensionally unique, it follows that there is a splitting of the Jacobian J = A×B of the family of curves and W = Tl (A)φ . In particular, the traces of Frobenius at 0 of FT ψ V lie in the ﬁeld obtained by adjoining all 7-power roots of unity to Q. But Q(i) is a subﬁeld of the ﬁeld generated by the traces, which is a contradiction. Suppose, ﬁnally, that ggeom = E6 in its 27-dimensional representation, so that p = 3. It is known [MPR, p. 293] that Ᏺ∨ is a direct summand of Ᏺ ⊗ Ᏺ. Therefore, ∨ FT ψ Ᏺ∨ = [−1]∗ ᏸφ(f ) is a direct summand of FT ψ¯ (Ᏺ ⊗ Ᏺ), so that H 0 A1 , ᏸφ(f (−x)) ⊗ FT ψ¯ (Ᏺ ⊗ Ᏺ) = 0. Since FT ψ¯ (Ᏺ ⊗ Ᏺ) is pure of weight 2, and hence semisimple, this implies that Hc2 A1 , ᏸφ(f (−x)) ⊗ FT ψ¯ (Ᏺ ⊗ Ᏺ) = 0. Since Hc2 is pure of weight 4, we can examine whether this holds by using the Lefschetz trace formula. If f has coefﬁcients in a ﬁnite ﬁeld F, then for any overﬁeld E of F, trace Frobx,E | ᏸφ(f (−x)) ⊗ FT ψ¯ Ᏺ ⊗ Ᏺ = ψ f (u) + f (v) + f (u + v) . x∈E

u,v∈E

It follows that Hc2 = 0 if and only if f (u)+f (v)+f (u+v) lies in (F −1)Wr (k[u, v]). Since Swan(ᏸψ(f ) ) = 28, f = (f0 , . . . , fr−1 ), where fr−1 is a polynomial of degree 28, and hence f (u) + f (v) + f (u + v) has its last coordinate of degree 28, and hence it cannot be in (F − 1)Wr (k[u, v]) Proposition 11.7. Suppose that p = 2 and Ᏺ is a Lie irreducible Airy sheaf. If Ᏺ is Lie symplectic, then ggeom is one of the following:

436

ONDREJ ŠUCH

(1) sp in its standard representation, (2) e7 in its 56-dimensional representation. If Ᏺ is Lie non-self-dual, then Ggeom is one of the following: (1) sl in its standard representation, (2) sl in its second fundamental representation, and rank Ᏺ = 2k−1 (2k − 1). Proof. We apply Lemma 11.4 together with Lemma 11.2 and examine Table 3 to conclude that if the conclusion of Proposition 11.7 does not hold, then ggeom = D6 , or ggeom = E6 , or ggeom = Al in their 32nd, 27th, and second fundamental representations, respectively. We consider these cases separately. ggeom = D6 We see from Table 3 that End0 (Ᏺ) has a 66-dimensional subspace V whose Fourier transform is, by Lemma 3.4, less than or equal to 2-dimensional, but because of Lemma 11.2, is actually 2-dimensional. By Lemma 3.4, Proposition 7.5(1) and (4), and Lemma 3.1, it follows that V has two breaks 0. This contradicts Lemma 2.6 since the rank of Ᏺ is 32, which is a perfect power of p = 2. ggeom = E6 In this case, p rank Ᏺ, so by Lemma 2.10, only the breaks 0 and 28/27 can possibly occur in the break decomposition of End0 (Ᏺ). By Lemma 3.1 and Proposition 7.5(6), we conclude that the average break of every irreducible summand of End0 (Ᏺ) is 1. Therefore, the rank of every irreducible summand of End0 (Ᏺ) has to be divisible by 28, which is not the case, as we see from Table 3. ggeom = Al , w = π2 In this case, we have to prove that l +1 is a power of 2. Our ﬁrst remark is that l = 3, since then the second fundamental representation is orthogonal. By Lemmas 10.4 and 11.5, there exists a sheaf Ᏼ such that !2 Ᏼ = g ∗ Ᏺ, where g is an étale cover of A1 of degree dividing 2. Since the inertial representation at ∞ of Ᏺ is irreducible, the inertia representation of g ∗ Ᏺ has at most two components. If Ᏼ(∞) were reducible, say, W1 +W2 , then g ∗ Ᏺ(∞) is !2 W1 +!2 W2 +W1 ⊗W2 . Hence, Ᏼ(∞) is irreducible. By Proposition 2.2, if l + 1 is not a power of p = 2, then we can write l + 1 = 2m n, with n odd and greater than or equal to 1, such that Ᏼ(∞) is induced from a 2m dimensional representation of I (∞)(n). When restricted to I (∞)(n), Ᏼ(∞) has n irreducible summands Zi , each of dimension 2m . Suppose the Zi ’s were 1-dimensional. Then since n is odd and l = 2, we have n = l + 1 ≥ 5. In particular, by Lemma 2.6, End0 (Ᏼ) contains n − 1 ≥ 4 tame characters of ﬁnite order. Since End0 (Ᏼ) is a summand of End0 (Ᏺ) that is Fourier by Proposition 11.1, End0 (Ᏼ) itself is Fourier. Since

l +1 0 = 3 rank Ᏺ, dim End (Ᏼ) = l(l + 2) < 3 2 we have that FT ψ End0 (Ᏼ) has rank less than or equal to 2 by Lemma 3.4. On the other

MONODROMY OF AIRY AND KLOOSTERMAN SHEAVES

437

hand, the presence of n − 1 ≥ 4 tame characters at ∞ of Ᏺ implies by local Fourier transform that FT ψ End0 (Ᏼ) has n − 1 ≥ 4 characters acting at 0, so FT ψ End0 (Ᏼ) has rank at least 4, which is a contradiction. Therefore, the Zi ’s are greater than 1-dimensional and !2 Ᏼ(∞) = g ∗ Ᏺ(∞) has at least n irreducible summands, namely, the !2 Zi , which are permuted by the action of I (∞)(n). Let U be an irreducible summand of g ∗ Ᏺ(∞) containing one !2 Zi . We have

1 l +1 (l + 1)/n 2 = dim Ᏺ/2 ≤ dim U ≤ n dim ! Zi = n . 2 2 2 In particular,

  l ≤ 1−n,

 1 l +1 (l + 1)/n ≤n ⇒ 2 n − 2  2 2 2 n ≤ 2.

Therefore, n = 1, and l + 1 is a perfect power of 2, say, 2m , and hence, rank Ᏺ = 2m−1 (2m − 1). 12. Monodromy of Kloosterman sheaves. In [K1], Katz computes the geometric monodromy groups of Kloosterman sheaves Ᏺ when p > 2 rank Ᏺ +1. In this section, we extend his results somewhat. The ﬁrst of our results is an analogue of Proposition 11.1, which shows that without loss of generality we may restrict our attention to Lie irreducible Kloosterman sheaves. Proposition 12.1. Let k be the algebraic closure of Fp , and let Ᏺ be a Kloosterman sheaf on Gm ⊗k. Then Ᏺ is either ﬁnite, or Kummer induced, or Lie irreducible. If it is induced, it is induced from a Kloosterman sheaf. It is not induced if and only if no irreducible summand of End0 (Ᏺ) is tame at ∞. Proof. Assume that Ᏺ is neither Lie irreducible nor ﬁnite, so that by Corollary 10.2, End0 Ᏺ has a subsheaf Ᏻ of rank less than n. Since Ᏺ is tame at 0, so is Ᏻ. Since all breaks of Ᏺ at ∞ are equal 1/n, all the breaks of Ᏻ at ∞ are less than or equal to 1/n. Since the rank of Ᏻ is less than n, it cannot have nonzero breaks in the interval (0, 1/n]. Therefore, Ᏻ is tame at ∞. The fundamental group of Gm is l=p Zl (1), a procyclic group. Since Ᏺ is a pure sheaf (see [K2, Theorem 4.1.1]), and hence semisimple, Ᏻ is a sum of characters of ﬁnite order prime to p. If Ᏺ were a tensor product of sheaves of lower rank Ᏼ1 ⊗ Ᏼ2 , then Ᏻ could be taken as End0 Ᏼ1 if Ᏼ1 has lower rank than Ᏼ2 . Since wild inertia P∞ acts trivially on Ᏻ = End0 (Ᏼ1 ), it must act via scalars on Ᏼ1 , which contradicts Proposition 2.2. If Ᏺ is induced, we may take Ᏻ = g∗ Ql /Ql , where g is the cover of U via which Ᏺ is induced. Then Ᏻ is a subcover of the compositum of all characters occurring in Ᏻ. But the whole compositum corresponds to a ﬁnite Kummer covering of Gm , so g is itself Kummer.

438

ONDREJ ŠUCH

Now suppose that Ᏺ is induced from Ᏻ, so that Ᏺ = g∗ Ᏻ. Since g is a Kummer covering, Ᏻ is necessarily lisse on Gm and tame at 0. To compute the Swan conductor of Ᏻ, we use the Euler-Poincaré formula. Since Ᏺ was irreducible as I (∞)-representation, the same holds for Ᏻ. Moreover, χ (Ᏺ) = χ(Gm ) · rank Ᏺ − Swan∞ (Ᏺ) = −1 −1 = χ (Ᏺ) = χ(Ᏻ) = χ(Gm ) · rank Ᏻ − Swan∞ (Ᏻ). Therefore, Ᏻ is itself a Kloosterman sheaf. Lemma 12.2. Let k be an algebraic closure of Fp , and let Ᏺ be a Lie irreducible Kloosterman sheaf on Gm ⊗ k. Suppose that (1) p rank Ᏺ, (2) p > rank ggeom + 1, (3) rank Ᏺ > [Ggeom : ((Gm · (Ggeom )0 ) ∩ Ggeom )], (4) [Ggeom : ((Gm · (Ggeom )0 ) ∩ Ggeom )] < p. (If denotes the outer automorphism group of ggeom , then the last two assumptions are automatically satisﬁed if both rank Ᏺ > || and p > ||.) Then ggeom is one of the following: (1) sl in its standard representation, (2) so in its standard representation, (3) sp in its standard representation. Proof. We would like to apply [K1, Theorem 7], which has slightly different assumptions. Upon closer examination of its proof, one ﬁnds that only the following facts are used: (1) the inertia representation at ∞ has exact denominator equal to the rank of the sheaf, (2) the rank of the sheaf is not divisible by p, (3) p > rank ggeom + 1, (4) the index of [Ggeom : (Ggeom )0 ] is prime to p. Of these, only (4) is not immediately clear from our assumptions. However, our assumptions guarantee that the covering C of Gm corresponding to the subgroup Ggeom ∩(Gm ·(Ggeom )0 ) is tamely ramiﬁed at ∞, and hence tame everywhere, hence a Kummer covering. Moreover, the Swan conductor at ∞ of the pullback of Ᏺ to that covering has Swan conductor less than 1. Therefore, the cyclic covering of C needed to pass to the connected component of Ggeom is tamely ramiﬁed. Altogether, we get that the index of (Ggeom )0 in Ggeom is prime to p, and hence our conclusion follows. Finally, we note that Ggeom /(Ggeom ∩ (Gm · (Ggeom )0 )) injects into . Theorem 12.3. Let p be a prime greater than 7, and let Ᏺ be a Lie irreducible Kloosterman sheaf. Then ggeom is one of the following: (1) sl in its standard representation,

MONODROMY OF AIRY AND KLOOSTERMAN SHEAVES

(2) (3) (4) (5)

439

so in its standard representation, sp in its standard representation, sl in its adjoint representation, and rank Ᏺ = p 2k − 1, sl(d) × sl(d) in the tensor product of standard representations, and p | d 2 = rank Ᏺ.

Proof. Since Ᏺ is Lie irreducible, Lie Ggeom acts nontrivially on every irreducible summand of End0 (Ᏺ). Since the tame fundamental group of Gm is procyclic, it follows that no irreducible summand of End0 (Ᏺ) is tame at ∞. Since all the breaks of Ᏺ(∞) are 1/ rank Ᏺ, it follows from integrality of Swan conductors that all irreducible summands of End0 (Ᏺ) have dimension greater than or equal to rank Ᏺ. In particular, this holds for the adjoint representation of ggeom . First suppose that ggeom is a simple Lie algebra. Then we may apply Proposition 6.2, since any outer automorphism ﬁxes the isomorphism class of the adjoint representation (so even though the adjoint representation inside End0 (Ᏺ) begins as only a representation of ggeom , it is actually a representation of Ggeom ). By Corollary 4.4, we know that in cases (8), (9), (10), (11), and (12) of Proposition 6.2, p cannot divide rank Ᏺ, because then by Proposition 2.2, the wild inertia at ∞ would not act on Ᏺ through an abelian quotient. Therefore, in cases (2), (4), (7), (8), (9), (10), (11), and (12) of Proposition 6.2, p rank Ᏺ, and we exclude them by applying Lemma 12.2. It remains to consider cases (1), (3), (5), and (6) of Proposition 6.2. ggeom = Al If l = 3 and the highest weight of ggeom is 2π1 , π2 , πl−1 , or 2πl , we may apply Lemma 10.5 to the quadruples (Ᏺ, Al , 2π1 , π1 ), (Ᏺ, Al , π2 , π1 ), (Ᏺ, Al , πl−1 , π1 ), and (Ᏺ, Al , 2πl , π1 ), respectively, to arrive at a representation whose highest break has a denominator greater than its dimension. The case of l = 3 is again excluded by application of Lemma 12.2. If w = π1 or πl , then we are in case (1) of Theorem 12.3. If w = π1 + πl , then we are in case (4) of Theorem 12.3 and we have to prove that rank Ᏺ = p 2k−1 . To do that, consider the subsheaf Ᏺ1 of End0 (Ᏺ), which is the adjoint representation of ggeom . By construction, Ᏺ1 is tame at 0 and has all ∞ breaks less than or equal to 1/ rank Ᏺ, but since rank Ᏺ = rank Ᏺ1 , the breaks are 1/ rank Ᏺ. Therefore, Ᏺ1 is again a Kloosterman sheaf. Moreover, the geometric monodromy group of Ᏺ1 has trivial center, hence the index of its connected component divides 2. Let g be the Kummer covering corresponding to passing to the connected component of the geometric monodromy group of Ᏺ1 . Applying Lemma 10.5 to the quadruple (g ∗ Ᏺ1 , sl(l + 1), π1 + πl , π1 ), we obtain a sheaf Ᏻ, lisse on Gm , such that Ᏺ1 = End0 (Ᏻ). Since g ∗ (Ᏺ)(∞) has at most two irreducible components, Ᏻ(∞) is irreducible. Since Ᏺ was totally wild at ∞, the same holds for Ᏺ1 , g ∗ Ᏺ1 , and hence by Lemma 2.6, the sheaf Ᏻ(∞) has to have rank that is a power of p, so that

440

ONDREJ ŠUCH

rank(Ᏺ) = rank(Ᏻ)2 − 1 = p 2k − 1. ggeom = Bl If the highest weight w of ggeom is π1 , we are in case (2) of Theorem 12.3. If the highest weight were π2 , we could apply Lemma 10.5 to the quadruple (Ᏺ, Bl , π2 , π1 ) to arrive at a representation of highest break 1/ rank Ᏺ, but dimension less than rank Ᏺ, which is a contradiction. ggeom = Cl If the highest weight w of ggeom is π1 , we are in case (3) of Theorem 12.3. If the highest weight of ggeom were 2π1 or π2 , we could apply Lemma 10.5 to the quadruples (Ᏺ, Cl , 2π1 , π1 ) and (Ᏺ, Cl , π2 , π1 ), respectively, to arrive at a representation of highest break 1/ rank Ᏺ, but dimension less than rank Ᏺ, which is a contradiction. ggeom = Dl If the highest weight w of ggeom is π1 , we are in case (2) of Theorem 12.3. Now suppose that the weight w of ggeom is π2 . Then End0 (Ᏺ) contains a subsheaf Ᏻ that is the adjoint representation of ggeom . Since Ᏻ has the same rank as Ᏺ and is wild with breaks less than or equal to 1/ rank Ᏺ, all its breaks at ∞ are in fact 1/ rank Ᏺ. The center of the geometric monodromy group of Ᏻ is trivial, and Ᏻ = !2 Ᏼ, where Ᏼ has rank less than rank Ᏺ (the case of ggeom of type D4 is again excluded by Theorem 12.3.) Since Ᏼ has the same highest break as Ᏻ, we arrive at a contradiction. It remains to consider the case when the Lie algebra of ggeom is not simple. In that case, we apply Theorem 5.1 and conclude that ggeom is either sl(d) × sl(d) or sl(2) × sl(2) × sl(2) in its d 2 or 8-dimensional representation, respectively. The latter case can again be excluded by applying Lemma 12.2. The former case is case (5) of Theorem 12.3, so we have to show that p | rank Ᏺ. Suppose that was not the case. Then by Lemma 2.10, only the breaks 0 and 1/ rank Ᏺ could occur as breaks at ∞ of any irreducible summand of End0 (Ᏺ). In particular, this holds for the summand Ᏻ, which as a representation of ggeom is the sum of the adjoint representations of the two simple factors of ggeom . Since Ᏺ was Lie irreducible, Ᏻ is not tame at ∞, and hence it has break 1/d 2 with multiplicity d 2 and break 0 with multiplicity d 2 −2. The sheaf Ᏻ is not Lie irreducible, in fact, it is induced via a Kummer covering of degree 2. If Ᏻ1 is a sheaf on Gm from which Ᏻ is Kummer induced, then Ᏻ1 has d 2 /2 breaks 2/d 2 , and the rest 0. Applying Lemma 10.5 to (Ᏻ1 , sl(d), π1 + πl , π1 ), we obtain a sheaf Ᏼ, lisse on Gm , of rank d, whose highest break at ∞ is 2/d 2 . Since its rank is d, by the integrality of Swan conductor, we have d = 2. But the case of sl(2) × sl(2) contradicts Theorem 12.3. √ Corollary 12.4. If p > rank Ᏺ + 1, then the ggeom of any Lie irreducible Kloosterman sheaf is one of the following: (1) so in its standard representation, (2) sp in its standard representation, (3) sl in its standard representation.

MONODROMY OF AIRY AND KLOOSTERMAN SHEAVES

441

Proof. The proof follows directly from Theorem 12.3, since in cases (4) and (5) of Theorem 12.3, we have √ p ≤ rank Ᏺ + 1. 13. Results on large monodromy. The following lemma gives sufﬁcient conditions for an Airy sheaf not to be ﬁnite. Lemma 13.1. Given p a prime, and r ≥ 1 an integer, choose an integer j ≥ 1 so that j (p − 1)pr−1 > 2. Let f (x) = (f0 (x), 0, . . . , 0), where f0 (x) is a polynomial in Fp [x] with the following properties: (1) f0 (x) = c0 for x in Fp×j , (2) f0 (0) = 0. Suppose that traceFpj /Fp (c0 ) is 1. Then the Airy sheaf Ᏺφ(f ) is not ﬁnite. Proof. We compute the trace at 0 of Frobenius acting on Ᏺφ(f ) , φ (f0 (x), 0, . . . , 0) trace Frob0,Fpj | Ᏺφ(f ),0 = − x∈Fpj

= − (p − 1) · traceFpj /Fp (c0 , 0, . . . , 0) + 1 ≡ −1 + ζpr mod p. If Ᏺφ(f ) is ﬁnite, then by the numerical criterion for ﬁniteness [K3, Theorem 8.14.4], j vp trace Frob0,Fpj | Ᏺφ(f ),0 ≥ . 2 But the above computation shows that the valuation is 1/(p r−1 (p − 1)), which is a contradiction. We note that a sufﬁcient condition for a Kloosterman sheaf Kl n (χ1 , . . . , χn ) to be inﬁnite is that two characters χi , χj coincide (indeed, this guarantees that local monodromy at 0 is inﬁnite order [K2, Theorem 7.3.2]). We ﬁx p and let r(d) := d/pvp (d) . Suppose that N ≥ 1 is an integer such that k−1 p ≤ N < pk . We can identify the parameter space of characters of π1 (A1 ) with CN := Gm × AN−1 as follows. Given any (aN , aN−1 , . . . , a1 ) ∈ CN , we associate to it the character k−1 N r(d) pk−1−vp (j ) φ ak x , 0, . . . , 0 , j =1

where φ is any faithful Ql character of Z/p k Z. By Corollary 2.9, every character of Swan conductor N corresponds to a (unique) element of CN . There exists a universal Airy sheaf Ᏺuniv lisse of rank N − 1 on CN × A1 , such that for every point c ∈ CN , Ᏺc = c∗ Ᏺuniv .

442

ONDREJ ŠUCH

Theorem 13.2. Let r ≥ 1, and let p be a prime. Let N be any integer satisfying the following: (1) p N, (2) N ≥ 7 if p = 2, (3) N ≥ 8 if p = 3, (4) N ≥ (p − 1) if p > 3, (5) N ≥ p r . Then the connected component of the geometric monodromy group of the universal Airy sheaf Ᏺuniv of length r and Swan conductor N is Sp(N − 1)

if p = 2 and r = 1,

SL(N − 1) otherwise. Proof. The sets of closed points corresponding to characters of order less than or equal to p l are closed in CN . Consider the statement S(N, r): “The theorem is true for N, r.” If p > 2, then by [K3, 8.18.2], to prove that S(N, r) holds, it is sufﬁcient to show that S(N, vp (N )+1) holds. If p = 2 and N is even, to show that S(N, r) holds, it is still sufﬁcient to show that S(N, vp (N) + 1) holds. If p = 2 and N is odd, it is still sufﬁcient to show that S(N, 1) holds because then we will know the following: (1) Ggeom for the parameter space acts Lie irreducibly, (2) the semisimple rank of Ggeom is at least (N − 1)/2, √ (3) (by Proposition 11.7) the alternative to SL(N −1) is SL(d +1) where d ≤ 2N. If r = vp (N )+1, then all characters in the set CN have order p r . As in [K3, Theorem 8.18.2], consider the group (η), ¯ where η¯ is the generic point of the CN . It follows from Theorem 13.2 that if (η) ¯ were ﬁnite, then the monodromy group of every Airy sheaf of rank N − 1 is ﬁnite. If Ggeom η¯ were induced, then the monodromy of every Airy sheaf of rank N − 1 would be induced. Taking into account Propositions 11.6 and 11.7, if (p, N, r) = (2, 57, 1), and we exhibit a pair of points c1 , c2 in CN such that the monodromy group of the Airy sheaves Ᏺc1 and Ᏺc2 are not induced and not ﬁnite, respectively, we have shown that the generic monodromy of Airy sheaves of length pr is as big as claimed by Theorem 13.2. (N, r, p) = (57, 1, 2) In this case, c1 = c2 = φ(x 57 + x 3 ). The characteristic polynomial of Frobenius at 1 acting on Ᏺφ(f ) is p(x) = x 56 + 4x 50 + 8x 48 + 32x 44 − 32x 43 + 64x 42 + 64x 41 − 128x 39 + 256x 38 − 512x 37 + 512x 36 − 1024x 33 − 4096x 31 + 4096x 30 + 16384x 26 − 32768x 25 − 32768x 23 + 131072x 20 − 262144x 19 + 262144x 18 − 262144x 17 + 524288x 15 + 1048576x 14 − 1048576x 13 + 2097152x 12 + 8388608x 8 + 16777216x 6 + 268435456.

MONODROMY OF AIRY AND KLOOSTERMAN SHEAVES

443

We see immediately that Ᏺφ(f ) cannot be ﬁnite, since vp (Frob1,F64 | Ᏺ1 ) = 2 < 3. We check by Proposition 7.5 that End0 Ᏺ is Fourier, so that the geometric monodromy of Ᏺφ(f ) is not induced. Finally, p(x) factors over F7 as p(x) =(4 + x)4 4 + 6x + 3x 2 + 3x 3 + x 4 4 + x + 2x 2 + 4x 3 + x 4 3 + x + 4x 2 + x 5 6 + 3x 3 + 3x 4 + x 5 2 + 2x + 4x 2 + x 3 + 2x 4 + 4x 5 + x 6 + 2x 7 + x 8 2 + 4x 4 + x 5 + 2x 6 + 4x 7 + 2x 8 + 4x 9 + x 10 + 2x 11 + 2x 12 + 3x 13 + x 14 + 4x 15 + x 16 + 2x 17 + 4x 18 + x 20 + 2x 21 + 4x 22 + x 26 . Now suppose the monodromy group of Ᏺφ(f ) were E7 . Then, upon choosing a square root of 2 in Ql , Frobenius at 1, acting on Ᏺφ(f ) (1/2), would be an element of E7 (F2 ), but that contradicts Proposition 4.5. By Proposition 11.7, we obtain that the monodromy of Ᏺφ(f ) is Sp(56). By [K3, 8.18.2], we conclude that Theorem 13.2 holds in the universal case. In all the remaining cases, the veriﬁcation that Ᏺc1 is not induced is a computation using Propositions 11.1 and 7.5(9), and the veriﬁcation that Ᏺc2 is not ﬁnite is accomplished using Lemma 13.1. (p, r) = (2, 1), N = 57 Here we take c1 = φ(x N ) if (N −1)/2 is odd, and c1 = φ(x N +x N−2 ) if (N −1)/2 is even. We take c2 = φ(x 7 + x N + x N mod 7 ) where N mod 7 ∈ {1, 2, . . . , 7}. (p, r) = (3, 1) Here we take c1 = φ(x N ) if p N − 1, and c1 = φ(x N + x N−2 ) if p | N − 1. We take c2 = φ(x 8 + x N − x N mod 8 ) where N mod 8 ∈ {1, 2, . . . , 8}. p>3 Here we take c1 = φ(x N ) if p N − 1, and c1 = φ(x N + x N−2 ) if p | N − 1. We take c2 = φ(x p−1 + x N − x N mod(p−1) ) where N mod(p − 1) ∈ {1, 2, . . . , p − 1}. Corollary 13.3. If g ≥ 3, then the geometric monodromy of hyperelliptic curves of genus g of 2-rank zero over F2 is Sp(2g). References [Bou] [BrW] [Bry] [CCNP]

[De]

N. Bourbaki, Groupes et algèbres de Lie, Hermann, Paris, 1975, ch. 8. R. Brauer and H. Weyl, Spinors in n dimensions, Amer. J. Math. 57 (1935), 425–449. J.-L. Brylinski, Théorie du corps de classes de Kato et revêtements abéliens de surfaces, Ann. Inst. Fourier (Grenoble) 33 (1983), 23–38. J. H. Conway, R. T. Curtis, S. P. Norton, R. A. Parker, and R. A. Wilson, Atlas of Finite Groups: Maximal Subgroups and Ordinary Characters for Simple Groups, Oxford Univ. Press, Oxford, 1985. P. Deligne, Cohomologie étale, Séminaire de Géométrie Algébrique du Bois-Marie (SGA 4 1/2), Lecture Notes in Math. 569, Springer, Berlin, 1977.

444 [Hu] [Is] [Ja] [K1] [K2] [K3] [K4] [MPR]

[Mo] [OV]

[R]

[Se] [ST]

ONDREJ ŠUCH J. E. Humphreys, Linear Algebraic Groups, Grad. Texts in Math. 21, Springer, New York, 1975. M. Isaacs, Character Theory of Finite Groups, Pure Appl. Math. 69, Academic Press, New York, 1976. N. Jacobson, Lie Algebras, Interscience Tracts in Pure Appl. Math. 10, Interscience, New York, 1962. N. M. Katz, On the monodromy groups attached to certain families of exponential sums, Duke Math. J. 54 (1987), 41–56. , Gauss Sums, Kloosterman Sums, and Monodromy Groups, Ann. of Math. Stud. 116, Princeton Univ. Press, Princeton, 1988. , Exponential Sums and Differential Equations, Ann. of Math. Stud. 124, Princeton Univ. Press, Princeton, 1990. , Rigid Local System, Ann. of Math. Stud. 139, Princeton Univ. Press, Princeton, 1996. W. G. McKay, J. Patera, and D. W. Rand, Tables of Representations of Simple Lie Algebras, Vol. I: Exceptional Simple Lie Algebras, Université de Montréal, Centre de Recherches Mathématiques, Montréal, 1990. L. Moret-Bailly, Pinceaux de variétés abéliennes, Astérisque 129 (1985), 1–266. A. L. Onishchik and E. B. Vinberg, eds., Lie Groups and Lie Algebras, III: Structure of Lie Groups and Lie Algebras, Encyclopaedia Math. Sci. 41, Springer, New York, 1994. M. Raynaud, Caractéristique d’Euler-Poincaré d’un faisceau et cohomologie des variétés abéliennes, Séminaire Bourbaki 9, Soc. Math. France, Paris, 1995, exp. no. 286, 129–147. J.-P. Serre, Local Fields, Grad. Texts in Math. 67, Springer, New York, 1979. G. Shimura and Y. Taniyama, Complex Multiplication of Abelian Varieties and Its Applications to Number Theory, Publ. Math. Soc. Japan 6, Math. Soc. of Japan, Tokyo, 1961.

Microsoft Corporation, One Microsoft Way, Redmond, Washington 98052; ondrejs@ microsoft.com

Vol. 103, No. 3

DUKE MATHEMATICAL JOURNAL

© 2000

REMARKS ON A HOPF ALGEBRA FOR DEFINING MOTIVIC COHOMOLOGY JIANQIANG ZHAO 1. Introduction. Let F be a ﬁeld. In [1], [3], Beilinson et al. considered the groups An of pairs of conﬁguration of hyperplanes, subject to a set of relations, in projective spaces Pn (F ). We call these groups generalized scissors congruence groups (see Deﬁnition 2.2). The deﬁning relations essentially reﬂect the functional equations of Aomoto polylogarithms, which, in turn, are connected with multivalued multiple polylogarithms deﬁned in [5]. They were led to study these groups when they were considering the cohomology H n (CP n \ L, M \ L) by using Deligne’s theory of mixed Hodge structures, where (L, M) is an admissible pair of simplices in CP n (see Deﬁnition 2.1). From the Hodge-theoretic point of view, they realized that the groups An ⊗ Q should have a Hopf algebra structure over Q. Then they studied the degree 1 and 2 parts in detail and related the degree 2 part to the Bloch group. It is now known that in degree 2 and 3 the generalized scissors congruence groups are intimately related to dilogarithms and trilogarithms essentially through their functional equations (see [6]). These groups are also related to hyperbolic geometry in a recent paper by Goncharov [8]. Historically, the relation between K-theory and scissors congruences in hyperbolic spaces ﬁrst appeared in the paper [4] by Dupont and Sah. In this paper, we prove that the comultiplication ν of the Hopf algebra structure on the generic part of the generalized scissors congruence groups is well deﬁned. Then we prove it is coassociative. We do not attempt to solve the problem of deﬁning ν on all of An , which is seen to be quite complicated at present (see [9]). Let A0n be the generic part of the grade n-piece of the generalized scissors congruence groups. The coassociativity of ν is important in that it yields the following A0• complex (see Corollary 3.5): ν

A0n −→

n−1 i=1

ν⊗id − id ⊗ν

A0n−i ⊗ A0i −−−−−−−−→

k1 +k2 +k3 =n k1 ,k2 ,k3 ≥1

A0k1 ⊗ A0k2 ⊗ A0k3 −→ · · · ,

which is conjectured to have the same cohomology as that of the A-complex, though no strong evidence has been found so far. But one should not only consider the generic part of the generalized scissors congruence groups. Calculations by Goncharov show that some deﬁning relations Received 1 March 1999. Revision received 5 October 1999. 2000 Mathematics Subject Classiﬁcation. Primary 14F42; Secondary 16W35. 445

446

JIANQIANG ZHAO

(additivity) of the nongeneric part nontrivially contribute some functional equations of the trilogarithm which cannot be obtained by the generic part alone. See the remark at the end of [6, §3.4]. The relation between the generalized scissors congruence groups with the algebraic K-theory is reﬂected in the following conjecture on [3, p. 550] (not exactly in the same form). Conjecture 1.1. The restricted coproduct provides a complex A>0 −→ A>0 ⊗ A>0 −→ A>0 ⊗ A>0 ⊗ A>0 −→ · · · , whose graded n-piece An −→

n−1

Ak ⊗ An−k −→ · · ·

k=1

gives

γ i (A• , Q) ∼ H(n) = gr n K2n−i (F )Q , γ

where gr n K2n−i (F )Q is the γ -ﬁltration of K-groups. Beilinson et al. proved this up to K3 (F ) in [1], [3]. According to Tannakian formalism, the category MTM(F ) of mixed Tate motives over a ﬁeld F is supposed to be equivalent to the category of graded modules over a certain graded commutative Hopf algebra Ꮽ• (see [2] and [7, Ch. 3]). Therefore, the Ext groups in the category MTM(F ) of mixed Tate motives over Spec(F ) are isomorphic to the cohomology of the Hopf algebra Ꮽ• , which can be computed using the reduced cobar complex Ꮽ• (F ) −→ Ꮽ>0 (F ) ⊗ Ꮽ>0 (F ) −→ Ꮽ>0 (F ) ⊗ Ꮽ>0 (F ) ⊗ Ꮽ>0 (F ) −→ · · · .

It was conjectured in [2] and [1] that Ꮽ• (F ) = A• (F ). This leads to Conjecture 1.1. 2. Deﬁnitions. We begin with some notation. Let F be a ﬁeld. One may simply put F = C throughout the paper. Let L be an n-simplex with faces L0 , . . . , Ln . Let li be the projective dual of Li . For any index set I = {i1 , . . . , ik } ⊆ Sn := {1, . . . , n}, we set LI = i∈I Li or (Li1 , . . . , Lik ) depending on the context. Similarly, lI is the projection from li1 , . . . , lik successively or the face lI = (li1 , . . . , lik ) depending on the context. No confusion should occur. Now let us recall the deﬁnition of admissible pairs (L; M) of n-simplices in PnF given in [1, §2]. Deﬁnition 2.1. We say (L; M) is admissible or M is admissible relative to L if L and M have no common faces of the same dimension.

REMARKS ON A HOPF ALGEBRA

447

We are now well prepared to deﬁne the generalized scissors congruence groups An (F ) introduced in [1]. We add the trivial intersection axiom for the sake of completeness and simplicity. Deﬁnition 2.2. Deﬁne A0 (F ) = Z. If n > 0, then An (F ) is the abelian group with generators being admissible pairs of n-simplices (L; M) subject to the following relations. (1) Nondegeneracy: (L; M) = 0 if and only if L or M is degenerate. (2) Trivial intersection: Suppose L0 , . . . , Ln and M0 , . . . , Mn are hyperplanes in Pn+1 . If N = Li or N = Mi for some i, then (N | L; M) := (L0 ∩ N, . . . , Ln ∩ N); (M0 ∩ N, . . . , Mn ∩ N) = 0. (3) Skew-symmetry: For every permutation σ ∈ Sn , (σ L; M) = (L; σ M) = sgn(σ )(L; M), where σ L = (Lσ (0) , . . . , Lσ (n) ). (4) Additivity in L and M: For any n+2 hyperplanes L0 , . . . , Ln+1 and n-simplex M in Pn , n+1 j , . . . , Ln+1 ; M = 0 (−1)j L0 , . . . , L j =0

j , . . . , Ln+1 ); M) is admissible. A similar relation is satisﬁed if every pair ((L0 , . . . , L for M. (5) Projective invariance: For every g ∈ PGLn+1 (F ), (gL; gM) = (L; M). Denote by [L; M] the class of (L; M) in An (F ). We explain the deﬁnition in depth by the following remarks. Remarks 2.3. (a) Trivial intersection, in fact, does not provide a pair of simplices. But we include this relation for the sake of convenience. (b) When L and M are in suitable positions, the above relations reﬂect properties of Aomoto polylogarithms (see [1]). (c) Geometrically, the skew-symmetry means that if M and M have the same faces with different orientation, then [L; M] = −[L; M ]; while the additivity says that if one calculates the volume of the same polyhedron (which may or may not be convex) bounded by n + 2 planes twice but using different orientations, then one would get opposite signs. (d) Using projective invariance and additivity one can show that An can be generated by admissible pairs (L; M) such that L is the standard coordinate simplex and none of the vertices of M lies on the inﬁnite plane. (e) One cannot dispense with admissibility. The following example shown to me

448

JIANQIANG ZHAO

by Goncharov (see Figure 1) essentially shows that any prism (or cylinder as called by Sah and others) would be zero if one allows nonadmissible pairs of simplices. The shaded area is the difference of two simplices that are equal to each other under projective invariance. y ∞ L2

M1

M2

M0 L0

M2

L1

0

∞

x

Figure 1. Prism is zero if nonadmissible pairs occur

(f) It is well known that A1 is isomorphic to F ∗ by the cross-ratio map r : A1 −→ F ∗ , (a − d)(b − c) ∀a = c, d and b = c, d. (a − c)(b − d) Proposition 2.4 (Intersection additivity). For any n + 1 hyperplanes L0 , . . . , Ln and n hyperplanes M1 , . . . , Mn in Pn , [a, b; c, d] −→

n i , . . . , Ln ; M = 0 (−1)i Li | L0 , . . . , L i=0

i , . . . , Ln ); M) is admissible on Li ∼ if every pair (Li | (L0 , . . . , L = Pn−1 . A similar relation holds for M. Remarks 2.5. (a) Relation (5) in Deﬁnition 2.2 is the key to making this proposition meaningful. Since Aut(Pn (F )) = PGLn+1 (F ), we can identify all Li ’s with Pn−1 (F ). (b) In [6] Goncharov stated the proposition as one of the deﬁning relations of An . He told me that it should follow from other relations, and therefore his deﬁnition of An would be the same as the one given in other papers (see, e.g., [1]). Proof of Proposition 2.4. Without loss of generality we may assume that Li = {zi = 0} for 1 ≤ i ≤ n and L0 = {z1 + · · · + zn = z0 }. We can further assume that

REMARKS ON A HOPF ALGEBRA

449

m = nj=1 Mj does not lie in L0 . Let P = L0 ∩ m, o, where o = ni=1 Li = [1, 0, . . . , 0] ∈ Pn . Let lj = 0≤i≤n,i=j Li . By additivity and skew-symmetry, [L0 | L1 , . . . , Ln ; M] =

n

P , l1 , . . . , li , . . . , ln ; L0 ∩ M

i=1

=

n (−1)i−1 l1 , . . . , li−1 , P , li+1 , . . . , ln ; L0 ∩ M . i=1

If m ∈ Li , then both (l1 , . . . , li−1 , P , li+1 , . . . , ln ) and Li ∩ M are degenerate, so we i , . . . , Ln ); M) = 0. If get ((l1 , . . . , li−1 , P , li+1 , . . . , ln ); L0 ∩ M) = (Li | (L0 , . . . , L m∈ / Li , then by projecting ((l1 , . . . , li−1 , P , li+1 , . . . , ln ); L0 ∩ M) from m to Li we i , . . . , Ln ); M). Thus get exactly (Li | (L0 , . . . , L n i , . . . , Ln ; M (−1)i−1 Li | L0 , . . . , L L0 | L 1 , . . . , L n ; M =

i=1

as desired. 3. Comultiplication on the generic part. We now assume that all the tensor products appearing in the following are taken over Z. Let A0n (F ) ⊆ An (F ) be the subgroup generated by all the pairs of simplices in generic position. In this section we deﬁne the coalgebra structure on An (F ) by modifying the deﬁnition of comultiplication given on [2, p. 708]. As pointed out in [1], there are several misprints on [2, p. 708]: jk should be jn−k while jn−k should be jk . Let A00 = A0 , and let A0n (n > 0) be the subgroup of An whose elements can be represented by admissible pairs of simplices in generic position. Notation 3.1. For any index sets I = {i1 < · · · < ik } ⊆ {1, . . . , n}, J = {j1 < · · · < jn−k } ⊆ {1, . . . , n}, we deﬁne sgn(I ; J ) = sgn(I, I¯) · sgn(J, J¯), where sgn(I, I¯) is the sign of the permutation (I, I¯) → {1, . . . , n} (similarly for sgn(J, J¯)) and where I¯ (resp., J¯) is the set complementary to I (resp., J ) in {1, . . . , n}. Deﬁnition 3.2. We deﬁne the comultiplication map as follows: ν : A0 −→ A0 ⊗ A0 , 1 −→ 1 ⊗ 1; and when n > 0, ν : A0n −→ [L; M] −→

n k=0 n k=0

A0n−k ⊗ A0k , νn−k,k ([L; M]),

450

JIANQIANG ZHAO

where νn,0 = id ⊗1F , ν0,n = 1F ⊗ id; and for 0 < k < n: νn−k,k : A0n −→ A0n−k ⊗ A0k , (I ) [L; M] −→ sgn(I )νn−k,k ([L; M]), |I |=k

where (I )

νn−k,k ([L; M]) =

|J |=n−k

sgn(J ) LI | L0 , LI¯ ; M0 , MJ ⊗ MJ | L0 , LI ; M0 , MJ¯ .

Note that both factors in each term are admissible since L and M are in generic (J ) position. We similarly deﬁne νn−k,k for any ﬁxed index set J and deﬁne νn−k,k =

|J |=n−k

(J )

sgn(J )νn−k,k .

Proposition 3.3. The comultiplication on the generic part in Deﬁnition 3.2 is well deﬁned. Proof. We need to show that νn−k,k maps all of the deﬁning relations of A0n (F ) to zero. Clearly we may assume 1 ≤ k ≤ n − 1. (1) Nondegeneracy: Notice that a simplex L in Pn is degenerate if and only if there are s + t faces of L whose intersection is of dimension greater than n − s − t. For any ﬁxed I , without loss of generality, we may assume exactly s of them appear (say, Lαi for 1 ≤ i ≤ s) in LI and t of them do not. Write any of the ﬁrst factors of (J ) νn−k,k ([L; M]) as L0 , . . . , Lα1 , . . . , Lαt , . . . , ; — , where Lj = Lj ∩ i∈I Li . Then by assumption dim Lα1 ∩ · · · ∩ Lαt > n − t − k in Pn−k . Thus (L0 , LI¯ ) is degenerate. (2) Trivial intersection: This is indeed trivial since none of the ﬁrst factors is a simplex of the right dimension. (3) Skew-symmetry: We only need to prove that for any ﬁxed 1 ≤ i ≤ n − 1 and index set J , (J ) (J ) −νn−k,k L0,...,n ; M = νn−k,k Li,1,...,i−1,0,i+1,...,n ; M . (1) For k = 0 or k = n this follows directly from the deﬁnition. Let us assume 1 ≤ k ≤ n − 1 in the rest of the proof. For such ﬁxed i we put

T = |I | = k : i ∈ /I , Sj = I = {i1 , . . . , ik } : ij = i (1 ≤ j ≤ i).

REMARKS ON A HOPF ALGEBRA

451

In the rest of the proof we use “—” to mean the elements of M that are unchanged. Then, by deﬁnition, the right-hand side of (1) equals I ∈T

+

sgn(I ) − Li1 ,...,ik | L0,...,i1 ,...,ik ,...,n ; — ⊗ — | Li,i1 ,...,ik ; —

i j =1 I ∈Sj

sgn(I ) L0,i1 ,...,ij ,...,ik | Lij

= Li , L1,...,i1 ,...,ik ,...,n ; — ⊗ − — | L0,i1 ,...,ik ; — ,

(2)

(3)

where I := {1 ≤ i1 < · · · < ik }. Applying additivity to [— | L0 , Li , Li1 , . . . , Lik ; —] we see that (2) is equal to −

I ∈T

+

I ∈T

sgn(I ) Li1 ,...,ik | L0,...,i1 ,...,ik ,...,n ; — ⊗ — | L0,i1 ,...,ik ; —

(4)

sgn(I ) Li1 ,...,ik | L0,...,i1 ,...,ik ,...,n ; — 

 k ⊗  (−1)j — | L0 , Li , Li1 , . . . , L ij , . . . , Lik ; —  .

(5)

j =1

Applying the intersection additivity property (see Proposition 2.4) to the ﬁrst factor / I ) we see that (3) is equal to in equation (3) (rotate Li and Ls for all s ∈ −

i j =1 I ∈Sj

+

i

sgn(I ) Li1 ,...,ik | L0,...,i1 ,...,ik ,...,n ; — ⊗ — | L0,i1 ,...,ik ; — sgn(I )

j =1 I ∈Sj

(−1)i−j +(s−t)

s

× Ls,i1 ,...,ij ,...,ik +

i j =1 I ∈Sj

sgn(I )

| L0,..., ; — ⊗ — | L0,i1 ,...,ik ; — it ,..., s,...,i t+1 ,...,i=ij ,...,n

(−1)i−j +(s−t)+1

s>i

× Ls,i1 ,...,ij ,...,ik | L0,...,i=ij ,..., ; — ⊗ — | L0,i1 ,...,ik ; — . it ,..., s,...,i t+1 ,...,n

Now (4) + (6) = LHS of (1). To see (5) + (7) + (8) = 0,

(6)

(7)

(8)

452

JIANQIANG ZHAO

we let



 i1 , . . . , it , s, it+1 , . . . , ij , . . . , ik if s < i, I =

 i , . . . , i, . . . , i , s, i , . . . , i if s > i. 1 j t t+1 k  sgn I, I¯ (−1)s−t+i−j −1 sgn I , I¯ = sgn I, I¯ (−1)s−t+i−j

Then

(9)

if s < i, if s > i,

which means that each term from equation (7) cancels one and only one term from equation (5). In fact, when I runs through all the subsets of {1, . . . , n} such that |I | = k and i ∈ I , I runs through all the subsets of T by the correspondence in (9) after taking all possible s there. (4) Additivity in L: By symmetry the same proof works for additivity in M. Letting i0 = −1 and ik+1 = n + 1, we have for any ﬁxed index set J , n+1 β=0

=

(J )

(−1)β νn−k,k

I

sgn(I )

β , . . . , Ln+1 ; M L0 , . . . , L

it+1 k

(−1)β

t=0 β=it +1

× — | L0,...,i,..., ; — ⊗ MJ | L0,i1 ,...,it ,it+1 +1,ik +1 ; — . ,...,i i ,...,β +1,..., i +1,...,n+1 1

t

t+1

k

In the above and throughout this proof of (4), the ﬁrst “—” means the subset of β , . . . , Ln+1 } complementary to those Li ’s appearing in the ﬁrst factor, the {L0 , . . . , L second “—” means M0 , MJ , and the third M0 , MJ¯ . For ﬁxed α1 , . . . , αk ∈ {1, . . . , n+ 1} we want to look at all the terms having the form · · · ⊗ MJ | L0,α1 ,...,αk ; — . Case I: α1 = 1. Suppose there are exactly r jumps with the difference greater than 1; that is, 1 = α1 = α2 − 1, . . . , αs1 < αs1 +1 − 1, αs1 +1 = αs1 +2 − 1, . . . , αsr < αsr +1 − 1, αsr +1 = αsr +2 − 1, . . . , αk−1 = αk − 1. Then each of the following index sets contributes to some terms in the required form:

It = 1 = α1 , . . . , αst , αst +1 − 1, αst +2 − 1, . . . , αk − 1 , 0 ≤ t ≤ r + 1,

453

REMARKS ON A HOPF ALGEBRA

where s0 = 1 and sr+1 = k. Writing αsr+2 = n + 2 and αk+1 = n + 2, the total contribution is r+1

αst+1 −1

t=0 β=αst +1

(−1)β sgn It

× — | L0,2,...,α2 ,...,αs ,...,β,...,α s t

k αs+1 −1

= sgn(I )

k ,...,n+1 ; — t+1 ,...,α

⊗ MJ | L0,α1 ,...,αk ; —

(−1)β+k−s

s=0 β=αs +1

× — | L0,2,...,α2 ,...,αs ,...,β,...,α k ,...,n+1 ; — ⊗ MJ | L0,α1 ,...,αk ; — . s+1 ,...,α Case II: α1 > 1. In this case there is one more contribution from the term when α = 0 and I0 = {α1 − 1, . . . , αk − 1}. Hence, the total contribution now is (setting s0 = 0 and sr+1 = k) 0 — | L1,...,α1 ,...,αk ,...,n+1 ; — ⊗ MJ | L1,α1 ,...,αk ; — sgn(I )(−1)k L +

r+1

αst+1 −1

t=0 β=αst +1

(−1)β sgn It

⊗ MJ | L0,α1 ,...,αk ; — t   = (−1)β+k+s−1 sgn(I ) — | L0,...,α1 ,...,αs ,...,β,...,αk ,...,n+1 ; —  × — | L0,1,...,α1 ,...,αs ,...,β,...,α s

k ,...,n+1 ; — t+1 ,...,α

β=αj

k i ⊗ MJ | L0,α1 ,...,αk ; — + (−1) MJ | L0,1,α1 ,..., αi ,...,αk ; — i=1

+ sgn(I )

k

αs+1 −1

(−1)β+k−s

s=0 β=αs +1

× — | L0,1,...,α1 ,...,αs ,...,β,...,α k ,...,n+1 ; — ⊗ MJ | L0,α1 ,...,αk ; — s ,...,α 1

by additivity on the ﬁrst and second factors of the ﬁrst term. (We note that Lβ does not appear in the — part of the ﬁrst factor.) Here we set s0 = 0. Thus, after cancellation, the above is equal to k 0 — | L1,...,α1 ,...,αk ,...,n+1 ; — ⊗ sgn(I )(−1)k L (−1)s MJ | L0,1,α1 ,...,αs ,...,αk ; — . s=1

When {α1 , . . . , αk } run through all the possible choices, we obtain k each term of the above sum.

n k

terms from

454

JIANQIANG ZHAO

For any I1 = {1, α2 , . . . , αk } ∈ Sn+1 , there are n − k choices of α > 1 such that α2 , . . . , αk together with α form an index set Iα . Set α1 = 1 and αk+1 = n + 2. Then the total contribution from Case II to the terms of the form · · · ⊗ MJ | L0,1,α2 ,...,αk ; — for all the choices of α is (−1)

k

k

αs+1 −1

sgn(Iα )(−1)s

s=1 α=αs +1

0 — | L1,...,α2 ,...,αs ,..., × L α ,...,α k ,...,n+1 ; — ⊗ MJ | L0,1,α2 ,...,αk ; — . s+1 ,...,α Moreover, when {α2, .. . , αk } run through all the possible choices, there are exactly n (n − k + 1) k−1 = k nk terms. We observe that

sgn(Iα ) = sgn α2 , . . . , αs , α, . . . , αs+1 , . . . , αk

= (−1)α+k sgn α2 , . . . , αk , 1, . . . , n + 1 = (−1)α−1 sgn(I1 ). From this we see that for any ﬁxed I1 = {1, α2 , . . . , αk } ∈ Sn+1 , the total contribution from the above to the terms of the form · · · ⊗ [MJ | L0,1,α2 ,...,αk ; —] is (for the ﬁxed index set J ) 0 — | L2,...,α2 ,...,αk ,...,n+1 ; — ⊗ MJ | L0,1,α2 ,...,αk ; — sgn(I )(−1)k L = sgn(I )

k

αs+1 −1

(−1)k+β−s+1

s=0 β=αs +1

× — | L0,2,...,α2 ,...,αs ,...,β,...,α k ,...,n+1 ; — ⊗ MJ | L0,1,α2 ,...,αk ; — s+1 ,...,α by additivity. This is exactly negative to the contribution from Case I. Hence, ν sends additivity relations to zero. (5) Projective invariance: This is clear. This ﬁnishes the proof of the proposition. We are now ready to prove the coassociativity of ν. Proposition 3.4. The comultiplication ν is coassociative, that is, it induces a complex ν

A0n −→

n k=0

ν˜

A0n−k ⊗ A0k −→

k1 +k2 +k3 =n

A0k1 ⊗ A0k2 ⊗ A0k3 −→ · · · ,

455

REMARKS ON A HOPF ALGEBRA

n

where ν˜ = ν˜ k =

k=0 ν˜ k

and

n−k

k νn−k−p,p ⊗ id ⊕ − id ⊗νq,k−q :

p=0

q=0

    n−k k A0n−k ⊗ A0k −→  A0n−k ⊗ A0q ⊗ A0k−q  . A0n−k−p ⊗ A0p ⊗ A0k  ⊕  p=0

q=0

Proof. Clearly, we only need to show that the composite map , ν

A0n −→

n k=0

ν˜

A0n−k ⊗ A0k −→

k1 +k2 +k3 =n k1 ,k2 ,k3 ≥0

A0k1 ⊗ A0k2 ⊗ A0k3

is zero. By deﬁnition ,([L; M]) =

n k=0 I,J

sgn(I ; J )˜ν Li1 ,...,ik | L0,...,i1 ,...,ik ,...,n ; M0,j1 ,...,jn−k

= B − C, ⊗ Mj1 ,...,jn−k | L0,i1 ,...,ik ; M0,...,j1 ,...,j n−k ,...,n where [Li1 Li0 | L0,...,n ; M0,j1 ,...,jn ] = [L; M], [Mj1 ,...,jn | L0 ; M0 ] = 1, and C=

k n k=0 I,J q=0

⊗

S,T

sgn(I ; J ) Li1 ,...,ik | L0,...,i1 ,...,ik ,...,n ; M0,j1 ,...,jn−k

sgn(S; T ) Mj1 ,...,jn−k Liα1 ,...,iαk−q | L0,i1 ,...,i α ,...,iα

; M0,β1 ,...,βq k−q ,...,ik

1

⊗ Mj1 ,...,jn−k Mβ1 ,...,βq | L0,iα1 ,...,iαk−q ; M0,...,j1 ,...,j . 1 ,...,β q ,...,n n−k ,...,β ¯ · sgn(T , T¯ ) with Here sgn(S; T ) = sgn(S, S)

¯ = sgn {S, S} ¯ −→ {i1 , . . . , ik } , S = iα1 , . . . , iαk−q , sgn(S, S)

T = {β1 , . . . , βq }, sgn(T , T¯ ) = sgn {T , T¯ } −→ 1, . . . , j1 , . . . , j n−k , . . . , n . On the other hand, B=

n−k n k=0 I,J p=0

sgn(I ; J )

sgn(Q; R)

Q,R

× Li1 ,...,ik Lγ1 ,...,γp | L0,...,i1 ,...,ik ,...,γ1 ,..., γp ,...,n ; M0,jδ1 ,...,jδn−k−p ⊗ Li1 ,...,ik Mjδ1 ,...,jδn−k−p | L0,γ1 ,...,γp ; M0,j ,...,j ,...,j ,...,j 1

⊗ Mj1 ,...,jn−k | L0,i1 ,...,ik ; M0,...,j1 ,...,j , n−k ,...,n

δ1

δn−k−p

n−k

456

JIANQIANG ZHAO

¯ · sgn(R, R) ¯ and where sgn(Q; R) = sgn(Q, Q)

¯ = sgn {Q, Q} ¯ −→ 1, . . . , ik , . . . , n , i1 , . . . , Q = γ1 , . . . , γp , sgn(Q, Q)

¯ = sgn {R, R} ¯ −→ {j1 , . . . , jn−k } . R = jδ1 , . . . , jδn−k−p , sgn(R, R) Now by setting k = k − q and q = p in the expression of B, we have B=

n k k=0 I ,J q=0

sgn(I ; J )

sgn(Q ; R )

Q ,R

× Li1 ,...,ik−q Lγ1 ,...,γq | L0,...,i1 ,...,i ; M 0,j ,...,j δ δ ,..., γ ,..., γ ,...,n q k−q 1 n−k 1 ⊗ Li1 ,...,ik−q Mjδ1 ,...,jδn−k | L0,γ1 ,...,γq ; M0,j1 ,...,j ,...,j δ1 ,...,jδ n−k+q n−k ⊗ Mj1 ,...,jn−k+q | L0,i1 ,...,ik−q ; M0,...,j ,...,j ,...,n , 1

n−k+q

where

I = i1 , . . . , ik−q ,

J = j1 , . . . , jn−k+q ,

¯ = sgn Q , Q ¯ −→ 1, . . . , Q = {γ1 , . . . , γq }, sgn Q , Q i1 , . . . , i k−q , . . . , n ,

R = jδ1 , . . . , jδn−k , sgn R , R¯ = sgn R , R¯ −→ j1 , . . . , jn−k+q .

Now we change the indices as follows:

i1 , . . . , ik−q , Q iα1 , . . . , iαk−q , i1 , . . . , i α1 , . . . , i αk−q , . . . , ik ,

¯ j1 , . . . , jn−k , β1 , . . . , βq . R , j1 , . . . , j δ1 , . . . , j δn−k , jn−k+q = R , R

(10)

Under these changes, the signs behave as follows: ¯ sgn Q , Q

¯ −→ i1 , . . . , ik−q , 1, . . . , i1 , . . . , i = sgn i1 , . . . , ik−q , Q , Q k−q , . . . , n

sgn iα1 , . . . , iαk−q , i1 , . . . , i i1 , . . . , ik , . . . , n sgn I , I¯ α1 , . . . , i αk−q , . . . , ik , 1, . . . , ¯ sgn I , I¯ = sgn I, I¯ sgn(S, S) and

sgn R , R¯ sgn j1 , . . . , jn−k , β1 , . . . , βq −→ j1 , . . . , jn−k+q

= sgn j1 , . . . , jn−k , β1 , . . . , βq , 1, j1 , . . . , j n−k+q , . . . , n sgn J , J¯

= sgn j1 , . . . , jn−k , 1, j1 , . . . , j n−k , . . . , n sgn(T , T¯ ) sgn J , J¯ = sgn J, J¯ sgn T , T¯ sgn J , J¯ .

REMARKS ON A HOPF ALGEBRA

457

It follows that B = C, which yields the proposition immediately. The above proof also shows the following. Corollary 3.5. The comultiplication ν induces the complexes ν

A0n −→

n−1 k=1

ν˜

A0n−k ⊗ A0k −→

k1 +k2 +k3 =n k1 ,k2 ,k3 ≥1

A0k1 ⊗ A0k2 ⊗ A0k3 −→ · · · ,

where ν and ν˜ are the restrictions of ν and ν˜ , respectively. Example 3.6. The following are complexes: A04 −→ A03 ⊗ A01 ⊕ A01 ⊗ A03 ⊕ A02 ⊗ A02 −→ A02 ⊗ A01 ⊗ A01 ⊕ A01 ⊗ A02 ⊗ A01 ⊕ A01 ⊗ A01 ⊗ A02 −→ A01 ⊗ A01 ⊗ A01 ⊗ A01 and

A03 −→ A02 ⊗ A01 ⊕ A01 ⊗ A02 −→ A01 ⊗ A01 ⊗ A01 .

Acknowledgments. This paper is part of my Brown University Ph.D. thesis (see [9]). I would like to thank my thesis advisor, A. Goncharov, whose numerous concrete suggestions and criticisms improved this paper greatly. I would also like to thank the referee for very insightful and helpful suggestions and comments on earlier drafts of this paper. References [1]

[2] [3] [4] [5]

[6] [7]

[8]

A. A. Beilinson, A. B. Goncharov, V. V. Schechtman, and A. N. Varchenko, “Aomoto dilogarithms, mixed Hodge structures and motivic cohomology of pairs of triangles on the plane” in The Grothendieck Festschrift, Vol. 1, Progr. Math. 86, Birkhäuser, Boston, 1990, 135–172. A. A. Beilinson, R. MacPherson, and V. V. Schechtman, Notes on motivic cohomology, Duke Math. J. 54 (1987), 679–710. A. A. Beilinson, A. N. Varchenko, A. B. Goncharov, and V. V. Schechtman, Projective geometry and K-theory, Leningrad Math. J. 2 (1991), 523–576. J.-L. Dupont and C. Sah, Scissors congruences, II, J. Pure Appl. Algebra 25 (1982), 159–195; Corrigendum, 30 (1983), 217. A. B. Goncharov, “Polylogarithms in arithmetic and geometry” in Proceedings of the International Congress of Mathematicians (Zürich, 1994), Vol. 1, Birkhäuser, Basel, 1995, 374–387. , Geometry of trilogarithm, preprint, Max-Planck-Institut für Mathematik, 1997, no. 65, http://www.mpim-bonn.mpg.de/. , “Mixed elliptic motives” in Galois Representations in Arithmetic Algebraic Geometry (Durham, 1996), London Math. Soc. Lecture Note Ser. 254, Cambridge Univ. Press, Cambridge, 1998, 147–221. , Volumes of hyperbolic manifolds and mixed Tate motives, J. Amer. Math. Soc. 12 (1999), 569–618.

458 [9]

JIANQIANG ZHAO J. Zhao, Hopf algebra structure of generalized scissors congruence groups, Ph.D. thesis, Brown University, 1999.

Department of Mathematics, Brown University, Providence, Rhode Island 02912, USA Current: Department of Mathematics, University of Pennsylvania, Philadelphia, Pennsylvania 19104, USA

Vol. 103, No. 3

DUKE MATHEMATICAL JOURNAL

© 2000

MOTIVIC DECOMPOSITION AND INTERSECTION CHOW GROUPS, I ALESSIO CORTI and MASAKI HANAMURA

Contents 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 459 1.1. Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 460 1.2. Main results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 461 1.3. Summary of contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 463 1.4. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 463 2. Pure motives over a base . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 463 2.1. Cohomology theories . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 464 2.2. Grothendieck motives: A reminder . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 465 2.3. Pure motives over a base . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 468 2.4. Realisation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 474 3. Technical issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 476 3.1. Compatibility of f ! . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 476 3.2. Products and composition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 482 3.3. Ᏸbcc is pseudoabelian . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 486 4. Standard conjectures and canonical ﬁltrations . . . . . . . . . . . . . . . . . . . . . . . . . . . . 487 4.1. Standard conjectures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 488 4.2. Murre’s conjecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 490 4.3. Saito’s ﬁltration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 492 5. Grothendieck motives over a base, semisimplicity, and decomposition . . . . . . 494 5.1. Perverse sheaves and the topological decomposition theorem . . . . . . . . . . 494 5.2. Grothendieck motives over S . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 497 5.3. Semisimplicity and decomposition in ᏹ(S) . . . . . . . . . . . . . . . . . . . . . . . . . . 498 6. Filtrations for quasi-projective varieties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 505 6.1. Proof of Theorem 6.1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 507 6.2. Proof of Theorem 6.3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 511 6.3. Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 513 7. Decomposition in CH ᏹ(S) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 514 1. Introduction. We explain our original motivation for writing this paper, then summarise the main results and outline the contents of each section. Received 30 June 1998. Revision received 19 October 1999. 2000 Mathematics Subject Classiﬁcation. Primary 14F42; Secondary 55N33, 14C15. 459

460

CORTI AND HANAMURA

1.1. Motivation. [GoMa] introduced the intersection cohomology groups IH i X(C), Q of a (singular) algebraic variety X deﬁned over the ﬁeld of complex numbers (the étale version of this theory was constructed in [BBD] for algebraic varieties deﬁned over an algebraically closed ﬁeld). We recall the construction of these groups in the beginning of Section 5. We list some of their properties. • There is a factorisation H i (X, Q) −→ IH i (X, Q) −→ H2BM dim X−i (X, Q) of the Poincaré map H i (X, Q) → H2BM dim X−i (X, Q). • There is an intersection product IH i (X, Q) × IH j (X, Q) −→ H2 dim X−i−j (X, Q), which is nondegenerate for proper varieties and intersection cycles of complementary dimensions. • Cohomology acts on intersection cohomology as follows: H i (X, Q) × IH j (X, Q) −→ IH i+j (X, Q). This research started out as a program (carried out—to an extent—in part II) to deﬁne Chow theoretic analogues ICH r (X, Q) of the intersection cohomology groups, satisfying the following corresponding properties. • There should be a factorisation CHC r (X, Q) −→ ICH r (X, Q) −→ CH r (X, Q) of the natural map CHC r (X, Q) → CH r (X, Q). Here, CHC • means “Chow cohomology,” not the operational theory of [FM] (which does not have a cycle class map; see [To]), but the theory developed in [Ha3], [Ha5]. • There should be an intersection product ICH r (X, Q) × ICH s (X, Q) −→ CH r+s (X, Q). • Chow cohomology should act on intersection Chow groups as follows: CHC r (X, Q) × ICH s (X, Q) −→ ICH r+s (X, Q). Moreover, there should be a cycle class map cl : ICH r (X, Q) → IH 2r (X, Q), and the above constructions should be compatible.

MOTIVIC DECOMPOSITION AND INTERSECTION CHOW GROUPS

461

This seemed to us an important and interesting question. Since their discovery, IH • (X(C), Q), respectively, IH • (X × k, Ql ), have been shown to carry a natural pure Hodge structure, respectively, (in the case when k is a ﬁnite ﬁeld) pure Galois module structure (see [SaM], resp., [BBD]). These structures are known to arise in nature as the cohomology groups of a pure Chow motive (direct summands of cohomology groups of a smooth algebraic variety). We summarise Grothendieck’s deﬁnition of pure motives in the beginning of Section 2. If this “intersection motive” can be identiﬁed and constructed, one can then take its Chow groups. Where to look for such a motive? Let us assume for simplicity that X is a variety with a single isolated singularity, having a resolution f : Z → X with a single smooth exceptional divisor E. According to the decomposition theorem (see [BBD], [SaM]), Rf∗ QZ = ᏵᏯX ⊕ V . Here, ᏵᏯX —in Borel’s notation [Bo]—is the intersection complex of X (its hypercohomology groups are the IH i (X, Q)) and V is a sheaf supported on the singular point. For the convenience of the reader, we recall the deﬁnition of intersection complexes and the statement of the decomposition theorem in Section 5. If the decomposition theorem is to hold for motives, we expect the “intersection motive” IX of X to be a direct summand of the motive hZ of the desingularisation. If the decomposition theorem is to hold for motivic sheaves on X, we expect IX to be of the form (Z, i∗ P ), where P ∈ CH dim X (E × E) is a projector in the correspondence ring of E and i : E × E → Z × Z the inclusion. It is relatively easy to ﬁgure out what the cohomology class of P must be. E is naturally polarised by the dual of its normal bundle, and P must induce the Hodge operator (see the beginning of Section 4) relative to this polarisation. In other words, in this particular case, the motivic decomposition theorem is equivalent to the standard conjecture of Lefschetz type for E. Actually, there is a further quite subtle problem that forms an important thread in the paper, namely, to justify that the Chow motive (Z, i∗ P ) is independent on the choice of P . On one hand this is disappointing; there seems no way to have a reasonable theory without proving the standard conjecture. On the other hand, we can now place the problem in its natural framework. 1.2. Main results. This paper is not actually concerned with intersection Chow groups; we plan to treat those in part II. Our ﬁrst result is the following theorem. Theorem 1.1. (See Section 2 for precise statements.) Let S be a quasi-projective variety, deﬁned over an algebraically closed ﬁeld k. There is a category CH ᏹ(S) of pure Chow motives over S, with realisation in Ᏸbcc (S), which is the relative analogue of the category CH ᏹ deﬁned by Grothendieck. As CH ᏹ, CH ᏹ(S) arises from a correspondence category CH Ꮿ(S), whose objects are smooth varieties X (together with a projective morphism X → S), and whose morphisms are deﬁned as HomCH Ꮿ S (X, Y ) = ⊕ CH dim Yα (X ×S Yα ),

462

CORTI AND HANAMURA

the sum being taken over all irreducible components Yα of Y . The composition of morphisms is deﬁned by means of Fulton and MacPherson’s reﬁned Gysin map. The construction of CH ᏹ(S) is easy and generalises an earlier idea by [DM]. We are convinced that CH ᏹ(S) is a useful language (see, for instance, [DM], [Sc1]). Our second main result is the following theorem. Theorem 1.2. (See Section 7 for the precise statement.) Consider a quasiprojective variety S, deﬁned over k = C or a ﬁnite ﬁeld Fq . If k = Fq , assume that resolutions of singularities exist for varieties deﬁned over k. Then, assuming Grothendieck’s standard conjectures and Murre’s conjecture, a decomposition theorem holds in CH ᏹ(S), which realises to the (topological) decomposition theorem in Ᏸbcc (S) of [BBD]. The assumption on resolutions of singularities may be unnecessary; it is conceivable that the modiﬁcations of [DJ1] and [DJ2] are enough for our purposes. We have not pursued this, since it is hardly the point of the paper. We assume k = C, or the closure of a ﬁnite ﬁeld, because the proof relies on the topological decomposition theorem, which has only been proved in these cases. The result is part of a larger program, due to M. H., to construct a triangulated category Ᏸ(S) of “mixed motivic sheaves” on S, and show that, assuming the standard conjectures and Murre’s conjecture (and, in addition, the vanishing conjecture for Kgroups of Soulé and Beilinson), it possesses the expected t-structure. The decomposition “theorem” allows us to make sense of intersection complexes and intersection Chow groups. In part II we shall use the ideas and result of part I to propose an unconditional deﬁnition of intersection Chow groups, and study some of their properties, sometimes with the aid of the conjectures. We are sure that the reader will perceive our liberal use of various long-standing conjectures to be a signiﬁcant weakness of our study. As a partial answer to this possible objection, we would like to make two remarks. First, we are making the point in this paper, following an insight of M. H., that the standard conjectures (which were designed primarily to deal with motives over the point) are indeed enough to determine the behaviour of pure motivic sheaves. Indeed, we think of our category CH ᏹ(S) as the category of pure motivic sheaves over S. Second, there are interesting concrete contexts where the conjectural assumptions are satisﬁed, or could conceivably be shown. These include families of curves, surfaces, abelian varieties, toric and toroidal morphisms, and many examples of conﬁguration spaces, moduli spaces, and Hilbert schemes and their compactiﬁcations. We believe that our theory is useful in these situations. Before giving a quick detailed description of the contents of each section, we would like to say that we invested an inordinate amount of time to make this paper as self-contained and accessible as possible. We assume a basic knowledge of algebraic geometry and intersection theory, as can be accessed through the ﬁrst six chapters of [F1], and a working knowledge of Verdier duality, as can be obtained by looking, for

MOTIVIC DECOMPOSITION AND INTERSECTION CHOW GROUPS

463

example, into [KS] or [Iv]. We recall or summarise everything else we need, including the bivariant theory of [FM], the standard conjectures, Murre’s conjecture on the natural ﬁltration on the Chow groups of smooth and projective varieties, perverse sheaves, the decomposition theorem, and so on. We especially hope that this paper will be accessible to nonexperts and graduate students seeking an introduction to the ﬁeld. 1.3. Summary of contents. Section 2 is devoted to the construction of CH ᏹ(S) and the study of its ﬁrst properties. Among these, the most important is the existence of a realisation functor real : CH ᏹ(S) → Ᏸbcc (S), which is a very natural but rather technical result stated in Section 2 and proven there, modulo some technical issues dealt with in Section 3. In Section 4 we recall the standard conjectures of Grothendieck, Murre’s conjecture on the canonical ﬁltration of the Chow groups of smooth and projective varieties, and S. Saito’s proposed unconditional deﬁnition of this ﬁltration. In Section 5 we deﬁne a category ᏹ(S) of “Grothendieck” motives over S, and, assuming the standard conjectures, we show that it is abelian and semisimple. This is an intermediate step in the direction of the decomposition theorem in CH ᏹ(S), which we ﬁnally prove in Section 7, after extending Saito’s ﬁltration to Chow groups of quasi-projective varieties in Section 6. For more information on the material covered in the various sections and the logical dependencies between them, the reader is invited to consult the short summary that we provide at the beginning of each section. 1.4. Acknowledgments. The authors would like to thank the mathematical institute of the University of Warwick for making it possible for them to meet in the fall of 1995 during the Warwick Algebraic Geometry special year. A. C. would like to express his immense gratitude to M. H. for teaching him the vast body of knowledge comprised in [Ha1]–[Ha6] and for his inﬁnite patience during the long time of preparation of the manuscript. N. Fakhruddin provided some very helpful feedback on a preprint version. We would also like to thank the referee for helping us iron out some signiﬁcant faults in the presentation. 2. Pure motives over a base. The main object of this section is to generalise Grothendieck’s construction of pure motives to the relative situation over a quasiprojective variety S. In short, we deﬁne a category CH ᏹ(S) of Chow motives over S. Our point of view in this paper is that CH ᏹ(S) is the conjectural category of pure motivic sheaves over S. After reviewing our conventions for cohomology theories, we recall Grothendieck’s construction of the category CH ᏹ of Chow motives, then deﬁne CH ᏹ(S) along very similar lines: the new element is the deﬁnition of composition of morphisms in CH ᏹ(S), using Fulton and MacPherson’s reﬁned Gysin map. We then construct a realisation functor CH ᏹ(S) → Ᏸbcc (S), leaving some of the

464

CORTI AND HANAMURA

technical details of the construction to Section 3, to avoid interrupting the ﬂow of ideas. 2.1. Cohomology theories. In this section, we explain our notation and conventions for cohomology theories. We need some properties that are not shared by all Weil cohomology theories (in particular de Rham theory), especially the bivariant formalism, but also perverse sheaves and the existence of specialisation homomorphisms. For this reason we must work either with Betti cohomology or with étale cohomology. (1) We ﬁx an algebraically closed ﬁeld k, and consider quasi-projective varieties X deﬁned over k. The notation H i X means either • H i (X(C), Q), if k has characteristic 0, where we always assume to have chosen an embedding σ : k → C, or • H i (X, Ql ), if char k = l, where X = X ⊗ k. These are vector spaces over Q = Q or Ql , depending on the context. The mixed Hodge structure or mixed Galois module structure on these spaces is (mostly) unimportant for us; for this reason, we do not keep track of Tate twists in our notation for cohomology groups. We denote HiBM X the Borel-Moore homology theory companion to H i X. Over the complex numbers, the reader can ﬁnd a useful elementary treatment in [F2, Appendix B], whereas the étale theory is treated in [La]. Homology and cohomology are part of the more general bivariant formalism of [FM], which we summarise when needed. (2) We use CH r X = CH r (X, Q) to denote the Chow group of cycles of dimension r, with coefﬁcients in the ﬁeld Q of rational numbers, modulo rational equivalence. Similarly, we denote CH r X = CH r (X, Q) the group of cycles of codimension r. We assume that the reader understands Chow theory as explained in the ﬁrst six chapters of [F1]. BM X, constructed in [F2, There is a cycle class homomorphism cl : CH r X → H2r Appendix B], in the case k = C, using only the most elementary properties of BorelMoore homology (these properties are shown in [La] for the étale theory), and in [F1, Chapter 19.1], [La, §6]. When the same construction exists both in Chow theory and in Borel-Moore homology, we say that the construction is compatible, meaning that it is under the cycle class homomorphism. It is not difﬁcult to show that the following operations are compatible: (a) pushforward by proper morphisms, (b) pullback by smooth morphisms, (c) intersection with Cartier divisors (and, consequently, Chern classes of vector bundles). For the proof of these compatibilities the reader is referred to [F2, Appendix B], or [F1, Chapter 19]. Fulton uses singular homology and the topological intuition that comes with it, but in fact he only needs the most basic properties of Borel-Moore homology and we trust the reader to ﬁll in the arguments. The relevant constructions are also contained in [La] and [Ve].

MOTIVIC DECOMPOSITION AND INTERSECTION CHOW GROUPS

465

(3) If S is a quasi-projective variety deﬁned over k, Ᏸbcc (S) denotes either Ᏸbcc (S(C), Q), the derived category of cohomologically constructible (for the Euclidian topology) sheaves, or Ᏸbcc (S, Ql ), the category of [D] and [BBD]. This has the six operations of Grothendieck, Verdier duality, and so on. QS , respectively, DS , denote the constant sheaf, respectively, dualising sheaf; that is, QS is either QS(C) or QS,l , depending on the context. Cohomology, Borel-Moore homology, and the bivariant formalism arise from Ᏸbcc and the six operations in a known way [FM], which is also brieﬂy recalled below. (4) In Section 4 we need the following construction. Let T be the spectrum of a discrete valuation ring with residue ﬁeld k and quotient ﬁeld K; 0, η ∈ T is the central and generic point, and X → T is a smooth and proper morphism. There is then a specialisation isomorphism ∼ =

hsp : H i Xη −−→ H i X0 , which is compatible via the cycle class with the specialisation homomorphism for Chow groups csp / CH r X0 CH r Xη cl

H 2r Xη

hsp

cl

/ H 2r X 0

2.2. Grothendieck motives: A reminder. We recall Grothendieck’s classical construction of motives. This serves the double purpose of preparing the ground for the construction of CH ᏹ(S), and to ﬁx some notation for later convenience. The construction is in three steps: construction of a correspondence category, pseudoabelianisation, and the introduction of Tate twists. The standard references for this material are [De], [Ma], and [Sc2]. 2.2.1. Correspondences. Fix a ﬁeld k, consider smooth and projective varieties X over k, and denote C i X the group of algebraic cycles of codimension i on X, modulo an adequate equivalence relation. Examples of adequate are the following: (1) C i X = CH i X, the Chow group of cycles modulo rational equivalence; 2i X = Im(CH i X → H 2i X) is the group of algebraic cycles, modulo (2) C i X = Halg homological equivalence; (3) C i X = cycles on X, modulo numerical equivalence. We now construct the categories C Ꮿ of C-correspondences. Deﬁnition 2.1. An object of C Ꮿ is a smooth and projective, not necessarily connected, variety X. Morphisms in C Ꮿ are correspondences HomC Ꮿ (X, Y ) = ⊕C dim Xα Xα × Y,

466

CORTI AND HANAMURA

where X = Xα is the decomposition of X into its connected components Xα . Let u : X1 → X2 and v : X2 → X3 be correspondences; let pij : X1 × X2 × X3 → Xi × Xj be the projection. The composition is deﬁned as follows: ∗ ∗ v ◦ u = p13∗ p23 v · p12 u . It is easy to see that, with the above deﬁnitions, C Ꮿ is an additive category, with the disjoint union of varieties being the categorical direct sum. Since the intersection product for Chow groups is compatible with the cup product for cohomology classes, we have a forgetful functor from the category of Chow correspondences to the category of homological correspondences. 2.2.2. Chow and Grothendieck motives. We ﬁrst recall the construction of the pseudoabelianisation of an additive category. Let Ꮽ be an additive category, A an object of Ꮽ. A projector is an arrow P : A → A such that P 2 = P . It is possible to give a categorical deﬁnition of the image of a projector. Deﬁnition 2.2. The image of a projector P : A → A is an object Im P of Ꮽ, together with a factorisation of P : A → A (commutative diagram) P

/A AD < DD zz DD z z DD zz D" zz Im P

satisfying the two identities Hom(−, Im P ) = P ◦ Hom(−, A), Hom(Im P , −) = Hom(A, −) ◦ P . Ꮽ is pseudoabelian if every projector has an image.

Remark 2.3. The deﬁnition of image of P simply means that, for all objects X, Hom(X, I ) is the image, in the category of abelian groups, of P ◦ _ by means of the diagram P ◦_ / Hom(X, A) Hom(X, A) OOO o7 OOO ooo o OOO o oo OO' ooo Hom(X, I )

At the same time, Hom(I, X) is the image of _ ◦ P by means of the diagram _◦P / Hom(A, X) Hom(A, X) OOO o7 OOO ooo o OOO o oo OO' ooo Hom(I, X)

MOTIVIC DECOMPOSITION AND INTERSECTION CHOW GROUPS

467

Deﬁnition 2.4. The pseudoabelianisation of Ꮽ is the category Ꮽ˜ deﬁned in the following manner. Objects of Ꮽ˜ are pairs (A, P ) of an object A of Ꮽ and a projector P : A → A. Morphisms in Ꮽ˜ are deﬁned as follows: HomᏭ˜ (A, P ), (B, Q) = Q ◦ HomᏭ (A, B) ◦ P . It is a simple observation that this is the same as morphisms f : A → B in Ꮽ such that f = Q ◦ f ◦ P . The following result is a formal exercise. Theorem 2.5. The category Ꮽ˜ is pseudoabelian. There is a natural functor F : Ꮽ → Ꮽ˜. Let Ꮾ be a pseudoabelian category and G : Ꮽ → Ꮾ a functor. Then there exists a unique functor H : Ꮽ˜ → Ꮾ, such that G = H ◦ F . We now deﬁne the category C ᏹ of C-motives. This is made by taking the pseudoabelianisation of C Ꮿ, then inserting Tate objects and twists by them. Deﬁnition 2.6. An object of C ᏹ is a triple (X, P , r), which is also denoted (X, P )(r), where X is a smooth projective, not necessarily connected variety, P ∈ EndC Ꮿ (X, X) is a projector, and r ∈ Z is an integer. Morphisms in C ᏹ are deﬁned as HomC ᏹ (X, P , r), (Y, Q, s) = Q ◦ ⊕C dim Xα +s−r (Xα × Y ) ◦ P , where X = Xα is the decomposition of X into its connected components Xα . Composition is by means of the same formula used for composing correspondences. Remark 2.7. (1) For C = CH, the category CH ᏹ is called the category of Chow motives. If C is cycles, modulo numerical equivalence, the corresponding category of motives is simply denoted ᏹ, and is called the category of Grothendieck motives. A consequence of the standard conjectures is that motives, modulo homological equivalence, is the same as motives, modulo numerical equivalence. (2) Denoting ᐂ as the category of smooth and projective varieties over k, there are natural contravariant cohomological h : ᐂ → CH ᏹ and covariant homological h∨ : ᐂ → CH ᏹ functors. As an object, hX = X is regarded as a Chow motive, and for a morphism f : X → Y , h(f ) = cl ,ft is the cycle class of the transpose ,ft ⊂ Y ×X of the graph ,f of f . Similarly, h∨ X = ⊕Xα (dim Xα ), where X = Xα is the decomposition in connected components, and h∨ (f ) = cl ,f . (3) Similarly, there are natural contravariant cohomological H : ᐂ → ᏹ and covariant homological H ∨ : ᐂ → ᏹ functors. These are deﬁned in a similar way to h and h∨ .

468

CORTI AND HANAMURA

(4) If VecQ is the category of vector spaces over Q, where the cohomology theory H i X is taking values, there are also realisation functors H ∗ : ᏹ → VecQ sending a motive to its cohomology. 2.3. Pure motives over a base. We extend Grothendieck’s construction to the case of varieties over an arbitrary quasi-projective base variety S, expanding on some ideas of [DM]. 2.3.1. Chow correspondences over S. Consider quasi-projective varieties Z over a ﬁeld k. Denote Ci Z the group of i-dimensional algebraic cycles on Z, modulo rational or homological equivalence; that is, either (1) Ci Z = CH i Z, the Chow group of cycles, modulo rational equivalence; or BM Z = Im(CH Z → H BM Z) is the group of cycles, modulo homo(2) Ci Z = Halg,2i i 2i logical equivalence. Let us ﬁx a quasi-projective variety S. We construct in this section the category BM Ꮿ(S) CH Ꮿ(S) of Chow correspondences over S; the construction of the category Halg of homological correspondences is postponed until Section 2.3.3, following a reminder on the (topological) bivariant theory. Deﬁnition 2.8. An object of CH Ꮿ(S) is a smooth, not necessarily connected, variety X, together with a projective morphism f : X → S. Morphisms in CH Ꮿ(S) are correspondences: HomCH Ꮿ S (X, Y ) = ⊕ CH dim Yα X ×S Yα , where Y = Yα is the decomposition of Y into its connected components Yα . The composition of morphisms is realised with the help of the following ﬁbre square diagram / Y ×S Z × X × S Y X ×S Y ×S Z Y

δ

/ Y ×Y

For u : X → Y , v : Y → Z we deﬁne the composition v • u : X → Z by v • u = pXZ∗ δ ! (v × u), where pXZ is the projection on the ﬁrst and third factor pXZ : X ×S Y ×S Z −→ X ×S Z, and δ ! is Fulton’s reﬁned Gysin map for regular embeddings, in other words, the “basic construction” of [F1, Chapter 6]. We are assuming that Y is smooth; therefore the diagonal embedding Y → Y × Y is a regular embedding. It is easy to see that, with the above deﬁnitions, CH Ꮿ(S) is an additive category, with the disjoint union of varieties being the categorical direct sum.

MOTIVIC DECOMPOSITION AND INTERSECTION CHOW GROUPS

469

BM Ꮿ(S) of homological correspondences is deﬁned, Remark 2.9. The category Halg BM in the above deﬁnition. This relies on the in Section 2.3.3, replacing CH • with Halg,• following two facts. (1) Regular embeddings (and, more generally, local complete intersection morphisms) have canonical orientations. In particular, the reﬁned Gysin map δ ! is deﬁned at the level of Borel-Moore homology H•BM . (2) Reﬁned Gysin maps in Chow theory and Borel-Moore homology are compatible under the cycle class homomorphism cl : CH • → H•BM . As a consequence, the reﬁned BM , and the whole construction goes through. Gysin map δ ! is deﬁned at the level of Halg,• The reader who understands all this is advised to fast-forward to Section 2.3.4, BM Ꮿ(S). while we discuss these points as a preliminary to the construction of Halg

2.3.2. Topological bivariant theory. We begin with a very quick summary of the relevant part of [FM]. For more information, the reader is invited to consult the original source. Then we discuss canonical orientations of local complete intersection morphisms, Theorem 2.10, and a generalisation of intersection products (DeﬁnitionTheorem 2.14). To avoid interrupting the ﬂow of ideas, we postpone the proof of Theorem 2.10 to the next section. (1) The topological bivariant theory associates, to every morphism f : X → Y of algebraic varieties, a Q-vector space H i (X → Y ). Special cases are H i (X = X) = BM X (Borel-Moore homology), and, H i X (ordinary cohomology), H i (X → pt) = H−i if X → Y is the inclusion of a locally closed subvariety, H i (X → Y ) = HXi Y (local cohomology). (2) The natural operations are products, proper pushforward, and pullback. If α ∈ H i (X → Y ) and β ∈ H j (Y → Z), there is a product α ·β ∈ H i+j (X → Z). If f : X → Y is proper and Y → Z is arbitrary, we get a pushforward homomorphism f∗ : H i (X → Z) −→ H i (Y → Z). If X

/ Y

X

/Y

g

is a ﬁbre square, we get a pullback g ∗ : H i (X → Y ) −→ H i (X → Y ). (3) Proper pushforward and pullback are functorial and satisfy a number of natural compatibility axioms with products (an example is the projection formula) which we do not write down. (4) An orientation of f : X → Y is a class ω ∈ H j (X → Y ). The class ω is a strong orientation if the induced homomorphism

470

CORTI AND HANAMURA

H i (U → X) α −→ α · ω ∈ H i+j (U → Y ) is an isomorphism for all U → X. Taking the product against an orientation, we can deﬁne wrong way proper pushforward f! : H • X → H • Y and pullback f ! : H•BM Y → H•BM X. More generally, if / Y X X

f

/Y

g

is a ﬁbre square, we have a reﬁned wrong way pullback f ! : H•BM Y −→ H•BM X deﬁned as

f ! (α) = (g ∗ ω) · α.

A class of maps, closed under composition, has canonical orientations if all maps in the class are oriented in such a way that products of orientations are compatible with compositions in the class. The key example of such a class is morphisms of smooth varieties. We are about to discuss, in Theorem 2.10, the more general class of local complete intersection (lci) morphisms. (5) The bivariant theory is built on top of the derived category Ᏸbcc as follows. For a morphism f : X → Y one deﬁnes H i (X → Y ) = HomY Rf! QX , QY [i] = (Verdier duality) HomX QX , f ! QY [i] . The product is essentially given by composition. Indeed, let f : X → Y , g : Y → Z and α : Rf! QX → QY [i], β : Rg! QY → QZ [j ] be bivariant classes. The product α ·β is the composition Rg! α

β

R(g ◦ f )! QX −−−→ Rg! QY [i] −→ QZ [i + j ]. Note that (using Verdier duality) we might as well have done the composition on Y or on X, with the same outcome. If f : X → Y is proper, g : Y → Z is arbitrary, and α : R(g ◦ f )! QX → QZ [i] a bivariant class, the proper pushforward f∗ α : Rg! QY → QZ [i] uses the canonical trace map QY → Rf∗ QX and is deﬁned to be the composition Rg! tr

α

Rg! QY −−−→ Rg! Rf∗ QX = R(g ◦ f )! QX −→ QZ [i]. Finally, given a ﬁbre square X

f

g

Y

f

/X /Y

g

MOTIVIC DECOMPOSITION AND INTERSECTION CHOW GROUPS

471

and a class α : Rg! QX → QY [i], the pullback f ∗ α uses the base change isomorphism Rg! f ∗ = f ∗ Rg! : f ∗α

Rg! QX = Rg! f ∗ QX = f ∗ Rg! QX −−−→ f ∗ QY [i] = QY [i]. We now discuss orientations of lci morphisms; the proof of the next theorem is in Section 3.1. Theorem 2.10. (1) The class of local complete intersection (lci) morphisms has canonical orientations (in the sense of the topological bivariant theory). In particular, if f : Y → X is an lci morphism and if Y

/ X

Y

/X

f

is a ﬁbre square, there are functorial reﬁned Gysin maps BM BM f ! : H2k X −→ H2k−2c Y .

(2) Let f : Y → X as above be an lci morphism. Then reﬁned Gysin maps in Chow theory and Borel-Moore homology are compatible. Remark 2.11. Needless to say, the proof is an extremely technical business and, ultimately, a straightforward (if complicated) exercise on understanding the basic construction of intersection theory [F1, Chapter 6], the derived category Ᏸbcc , and the bivariant formalism. For algebraic schemes over the complex numbers and singular Borel-Moore homology, and in the case when f is a regular embedding, the statement is made and proved (very concisely) in [F1, Theorem 19.2] (modulo some facts about orientations of regular embeddings, to be found in [BFM]). In the beginning of [F1, Chapter 19], it is said that “most” results there hold for the étale BorelMoore homology theory developed in [La]. For ordinary (nonreﬁned) Gysin maps, the compatibility statement can be found in [Ve, 10.2]; to understand the proof you must read the whole book, but especially §IX.8 (specialisation homomorphism in homology), §IX.9 (functorial orientations in homology for regular embeddings), and §IX.10 (same thing for lci morphisms). The case of reﬁned Gysin maps is a formal consequence of the case of ordinary Gysin maps, so a proof of the theorem can be easily synthesised from existing statements in the literature. However, these sources are not exceedingly friendly; therefore, we sketch a proof (in Section 3) for regular embeddings (the case of lci morphisms is a relatively mild generalisation), to try to tell you what really is going on. Remark 2.12. (1) Let X, Y be smooth varieties, and let f : X → Y be any morphism. The natural trace map Rf !DX → DY can be regarded as an orientation, since DX = QX [2 dim X] and DY = QY [2 dim Y ]. On the other hand, f is an lci morphism;

472

CORTI AND HANAMURA

indeed we can factorise f as γ

/ X×Y XE EE EE p EE EE 2 f " Y

where the graph map γ (x) = (x, f (x)) is a regular embedding, and the second projection p2 is a smooth morphism. It follows from the construction that these two orientations of f are the same. (2) Let T be a smooth variety. The orientation class of the diagonal embedding δ : T −→ T × T can be described in yet a third way. Indeed the morphism T → {pt} has a canonical strong orientation class ω ∈ H −2 dim T (T → pt), by any of the two methods just discussed. For any W → T we denote Pω the associated “Poincaré duality” isomorphism Pω

H i (W → T ) α −−→ α · ω ∈ H i−2 dim T (W → pt). The diagonal δ : T → T × T also has a canonical orientation −1 or δ = Pω×ω (ω) ∈ H 2 dim T (T → T × T ),

which can be shown to be equivalent to the other two. Deﬁnition 2.13. For any T , given p : U → T and q : V → T we deﬁne a cup product ∪ : H j (V → T ) × H i (U → T ) −→ H i+j (U ×T V → T ) by the formula

α ∪ β = q ∗ (β) · α,

via the diagram U ×T V

/V

U

/T

p

/T q

(it is the same as (−1)i+j p ∗ α · β). Deﬁnition-Theorem 2.14. If T is smooth, given U → T and q : V → T , we deﬁne an intersection product • : HiBM U × HjBM V −→ H2BM dim T −i−j U ×T V in any of the two equivalent ways a • b = Pω Pω−1 a ∪ Pω−1 b = δ ! (a × b).

MOTIVIC DECOMPOSITION AND INTERSECTION CHOW GROUPS

Proof. To prove that

473

Pω Pω−1 a ∪ Pω−1 b = δ ! (a × b),

the reader is invited to stare at the following diagram, where every square is a ﬁbre square. U o

U o

y yy yy y y |yy U ×T o

T o

o T x x δ xxx xx x |x T ×T o

pt o

U ×T V ss s ss ss yss U ×V

T ×V

V s s ss ss s s sy s

V

T o

2.3.3. Homological correspondences over S BM Ꮿ(S) of homological correspondences over S Deﬁnition 2.15. The category Halg is deﬁned by replacing CH • with H•BM in Deﬁnition 2.8 of Chow correspondences over S. This relies on the orientations of regular embeddings and the compatibility of reﬁned Gysin maps (see Theorem 2.10).

Remark 2.16. Note that the notation • for the composition of morphisms is compatible with the intersection product in Borel-Moore homology of Deﬁnition 2.14. BM , we 2.3.4. Chow and homological motives over S. For C = CH or C = Halg now deﬁne the category C ᏹ(S) of pure C-motives over S. This is made by taking the pseudoabelianisation of C Ꮿ(S) and then inserting Tate objects and twists by them.

Deﬁnition 2.17. An object of C ᏹ(S) is a triple (X, P , r), also denoted (X, P )(r); X is a smooth, not necessarily connected, variety, together with a projective morphism f : X → S, P ∈ EndC ᏯS (X, X) is a projector, and r ∈ Z is an integer. Morphisms in C ᏹ(S) are deﬁned as HomC ᏹS (X, P , r), (Y, Q, s) = Q ◦ ⊕Cdim Yα +r−s (X ×S Yα ) ◦ P ,

474

CORTI AND HANAMURA

where Y = Yα is the decomposition of Y into its connected components Yα . The composition of morphisms is by means of the same formula used for composing correspondences. Remark 2.18. (1) For C = CH, the category CH ᏹ(S) is called the category of BM is cycles, modulo homological equivalence, Chow motives over S. If C = Halg BM ᏹ(S) is the category of homological motives over S. We only need C ᏹ(S) = Halg this category very brieﬂy in Sections 5 and 6. The analogue ᏹ(S) of Grothendieck motives ᏹ is constructed in Section 5. (2) Denoting ᐂ(S) as the category of smooth varieties X, projective over S, there are natural contravariant cohomological hS : ᐂ(᏿) → CH ᏹ(S) and covariant homological h∨ S : ᐂ(᏿) → CH ᏹ(S) functors. As an object, hS X = X regarded as a Chow motive, and for a morphism f : X → Y covering the identity of S, hS (f ) = cl ,ft is the cycle class of the transpose ,ft ⊂ Y ×S X of the graph ,f of f . Similarly, h∨ Xα is the decomposition in connected compoS X = ⊕Xα (dim Xα ), where X = nents, and h∨ (f ) = cl , . f S (3) If f : X → S is the morphism to S, we sometimes use the following alternative notation for the objects hS X and h∨ S X: hS X = CRf∗ QX , h∨ S X = CRf∗ DX . In the coming section, we construct a realisation functor CH ᏹ(S) → Ᏸbcc (S). The notation is meant to suggest, for instance, that hS X = CRf∗ QX realises to Rf∗ QX , and is therefore a Chow theoretic “Rf∗ ”. This notation is particularly useful in the statement of the decomposition theorem in Section 7. Similar remarks apply to the dualising sheaf DX . (4) As already said, we construct realisations in Theorem 2.19. 2.4. Realisation. We have the following fundamental result. Theorem 2.19. There is a natural realisation functor CH ᏹ(S) −→ Ᏸbcc (S). Remark 2.20. We have no special notation for the realisation functor. In the instances where we need to emphasise it, we denote it real : CH ᏹ(S) → Ᏸbcc (S). The idea is to make the object (f : X → S, r) correspond to the sheaf Rf∗ QX [2r]. We prove the result here, assuming technical lemmas whose proofs we relegate to the next section (to avoid interrupting the ﬂow of ideas). From the construction of the two categories, we have a forgetful functor CH ᏹ(S) BM ᏹ(S) from Chow motives to homological motives; therefore, the proof con→ Halg BM ᏹ(S) → Ᏸb (S), which is a purely topological centrates on exhibiting a functor Halg cc problem. We begin with the following lemma.

MOTIVIC DECOMPOSITION AND INTERSECTION CHOW GROUPS

475

Lemma 2.21. Let p : X → S, q : Y → S be morphisms of varieties, and consider the following ﬁbre square diagram: X ×S YF FF p w w FF ww FF w ww FF w {w # f X GG Y w GG ww GG w w ww p GGG G# {www q S q

Then: (1) for sheaves F ∈ Ᏸbcc X, G ∈ Ᏸbcc Y , there is a natural isomorphism Rf∗ R ᏴomX×S Y q ∗ F, p ! G = R ᏴomS Rp! F, Rq∗ G ; (2) in particular, if p is proper and Y is smooth, there is a natural isomorphism ∼ = ϕ : HomS Rp∗ QX [i], Rq∗ QY [j ] −−→ H2BM dim Y +i−j X ×S Y. Proof. (1) follows from Verdier duality and proper base change Rf∗ R ᏴomX×S Y q ∗ F, p ! G = (standard duality) Rp∗ R ᏴomX F, Rq∗ p ! G = (proper base change) Rp∗ R ᏴomX F, p ! Rq∗ G = (Verdier duality) R ᏴomS Rp! F, Rq∗ G . If p is proper and Y is smooth, Rp! = Rp∗ and DY = QY [2 dim Y ]. Setting F = QX [i], G = QY [j ] in (1), and taking H 0 , we obtain HomS Rp∗ QX [i], Rq∗ QY [j ] = HomX×S Y QX×S Y [i], p ! QY [j ] = HomX×S Y QX×S Y [i], p ! DY [−2 dim Y + j ] = HomX×S Y QX×S Y , DX×S Y [−2 dim Y − i + j ] = H2BM dim Y +i−j X ×S Y, that is, (2). Remark 2.22. In the proof of (1) we went from X ×S Y to S passing through X. We could have gone there passing through Y , getting the same isomorphism. If S is a quasi-projective variety, we may now attempt to use Lemma 2.21 to deﬁne BM Ꮿ(S) → Ᏸb (S) by the assignment a functor Halg cc (f : X → S, r) −→ Rf∗ QX [2r], BM Halg,• (X ×S Y ) u −→ ϕ −1 (u).

476

CORTI AND HANAMURA

The next lemma, whose proof we postpone to Section 3.2, completes the construction, BM Ꮿ(S) is compatible with the showing that the composition of morphisms in Halg composition in Ᏸbcc . Lemma 2.23. Let p1 : X → S, p2 : Y → S, and p3 : Z → S be morphisms of varieties with p1 , p2 proper and Y, Z smooth. Let u : Rp1∗ QX [i] → Rp2∗ QY [j ], let v : Rp2∗ QY [j ] → Rp3∗ QZ [k] be morphisms, and let ϕ be the isomorphism of Lemma 2.21(2). Then ϕ(v ◦ u) = ϕ(v) • ϕ(u). To ﬁnish the construction of the realisation functor, we need to show that the BM Ꮿ(S) → Ᏸb (S) passes to the pseudoabelianisation. This is just-constructed Halg cc immediate from Theorem 2.5 (universal property of the pseudoabelianisation) and the following lemma, whose proof we could not ﬁnd in the literature, and which we also postpone to Section 3.3. Lemma 2.24. The category Ᏸbcc (S) is pseudoabelian. 3. Technical issues. In this section, we prove the statements left from Section 2, thus completing the construction of the realisation functor. The material here is very technical and we advise the reader to skip it on ﬁrst reading. There are three sections, which are devoted to Theorem 2.10, and Lemmas 2.23 and 2.24. 3.1. Compatibility of f ! . Proof of Theorem 2.10. In this section, we prove Theorem 2.10. In Chow theory, as well as in Borel-Moore homology, Gysin maps for regular embeddings f : Y → X are the composite of specialisation to the normal cone CY X of Y in X (which exist for any inclusion Y → X of a subscheme) and pulling back along the zero-section z : Y → CY X (which is only deﬁned if CY X → Y is a vector bundle; i.e., f is a regular embedding). When the same construction is possible in Chow theory and in Borel-Moore homology, we say that the construction is compatible, meaning that it is under the cycle BM (which is deﬁned in [F1, Chapter 19.1], [F2, class homomorphism cl : CH k → H2k Appendix B], or [La, §6]). It is easy to establish properties of the cycle class map that ensure that pullback along the zero-section is compatible. Therefore, the proof concentrates on showing that specialisation is compatible. We now proceed with the following topics: • Chow theory and cycle classes, • specialisation to the normal cone (homology), • orientations of lci morphisms (homology), • compatibility of specialisation, • compatibility of reﬁned Gysin maps.

MOTIVIC DECOMPOSITION AND INTERSECTION CHOW GROUPS

477

(1) We assume that the reader understands Chow theory as explained in the ﬁrst six chapters of [F1]. If f : Y → X is any subscheme, we use the following notation throughout for the deformation to the normal cone CY X {0}

i

/ M0 o

j

i

/X

p1

π

/ A1 o

p2

U

j

V,

where V = A1 \ {0}, U = (A1 \ {0}) × X, and denote σY,X : CH k M 0 → CH k CY X as the specialisation homomorphism (we recall what this is later, when we need it). Recall that, by deﬁnition, M 0 = Bl{0}×Y A1 × X \ {0} × X is the blowup of A1 ×X along {0}×Y , minus the strict transform ({0}×X) of {0}×X. Note that we depart slightly from common usage, taking our deformation over A1 rather than P1 (we have our reasons). All this is explained in [F1, Chapter 5]. In case f is a regular embedding (of codimension c), CY X → Y is a vector bundle of rank c and, given the isomorphism CH • Y = CH • CY X, the pullback along the zero-section z : Y → CY X is well deﬁned, as well as the Gysin homomorphism f ! = z! σY,X . BM X, We assume the reader knows how to assign a cycle class cl : CH k X → H2k and that the following operations are compatible: (a) pushforward by proper morphisms, (b) pullback by smooth morphisms, (c) intersection with Cartier divisors (and, therefore, Chern classes of vector bundles). For these the reader is referred to [F2, Appendix B] or [F1, Chapter 19]. Fulton uses singular homology and the topological intuition that comes with it, but in fact he only needs the most basic properties of Borel-Moore homology; we trust the reader to ﬁll in the arguments. The relevant constructions are also contained in [La] and [Ve]. (2) Fix any embedding f : Y → X. Let M 0 , etc. be the deformation to the normal cone CY X; in addition let ρ : CY X → X be the natural map (projection to Y followed by the inclusion Y → X). We now deﬁne, for a complex K ∈ Ᏸbcc M 0 , such that K | U = p2∗ L is the pullback of a complex L on X, a specialisation homomorphism σY,X : ρ ∗ L −→ i ! K[2]. Indeed, consider Rπ∗ K. First of all (Rπ∗ K) | V = Rp1∗ (K | U ) = Rp1∗ p2∗ L = Rp1∗ p2! L[−2] = QA1 · R,L; hence, Rj∗ Rp1∗ (K | U ) = QU · R,L ⊕ i∗ R,L[−1].

478

CORTI AND HANAMURA

The specialisation map is obtained as a composition R,L[−1]P XXX PPP XXXXXX XXXXσXY,X PPP XXXXX PPP XXXXX PP( XX+ / R,i ! K[1]. Rj∗ Rp1∗ (K | U ) Actually, ρ ∗ L → i ! K[2] is obtained from the just deﬁned σY,X by localising in a neighbourhood of Y . Note that we have used base change to identify i ! Rπ∗ K = R,i ! K. In the special case where K = DM 0 , K | U = p2∗ DX [2], taking cohomology BM U and we get that H −i R,L[−2] = HiBM X, H −i i ∗ Rj∗ Rp1∗ (K | U )[−1] = Hi+1 BM −i ! H R,i K = Hi CY X. Consequently, the above diagram translates into HiBM X TT HH TTTT HH HH TTTTσY,X TTTT HH TTTT H$ * BM / H BM CY X, H U i+1

i

which deﬁnes the specialisation homomorphism for Borel-Moore homology. (3) We brieﬂy explain how the just-constructed specialisation leads to a theory of canonical orientations of lci morphisms. This is the basis for the deﬁnition of reﬁned Gysin maps in homology and for showing that they satisfy the relevant bivariant formalism. Let f : Y → X be an lci morphism of schemes of codimension c = dim X −dim Y . Following Fulton’s convention, we always assume that f factorises (globally) as i

/Z Y @ @@ @@ p @ f @@ X,

where i : Y → Z is a regular embedding, and p : Z → X is a smooth morphism. A canonical orientation of f is a natural homomorphism uf : Rf !QY −→ QX [2c] (usually thought of as a bivariant class uf ∈ H 2c (Y → X)). The orientation of f is obtained composing an orientation of p and one of i. Smooth morphisms have canonical orientations; in fact, if p : Z → X is a smooth morphism, there is a canonical and functorial identiﬁcation p! QX = QZ [−2 codim p].

MOTIVIC DECOMPOSITION AND INTERSECTION CHOW GROUPS

479

(If one is puzzled by this, recall that we are suppressing Tate twists from our notation; the real identiﬁcation is between p! QX and QZ [−2 codim p](−2 codim p). The relevant canonical trace map t : R 2d p! p ∗ QX (d) → QX is a “relative fundamental class,” and it exists under rather more general conditions, for instance, if p is ﬂat, see [La, §0].) We take the orientation to be the class up corresponding to the identity id ∈ HomᏰbcc Z (QZ , p ! QX [2 codim p]) = HomᏰbcc X (Rp! QZ , QX [2 codim p]). Let i : Y → X be a regular embedding of codimension c ; we want to construct an orientation for i. In (2), for any embedding Y → X, denoting q : CY X → Y as the natural projection, and ρ = i ◦ q : CY X → X CY XD DD ρ DD q DD D! /X Y we constructed (take K = DM 0 ) a specialisation homomorphism σY,X : ρ ∗ DX −→ DCY X , which, upon dualising, can be seen as a homomorphism γY,X : QCY X −→ R ᏴomᏰbcc CY X (ρ ∗ DX , DCY X ) = ρ ! QX (which should aptly be called a “generisation” homomorphism). In the case of a regular embedding i, the projection q : CY X → Y is smooth of codimension equal to − codim(i) and (as already observed) we may identify QCY X = q ! QY [−2c ]. Letting z : Y → CY X be the zero-section, this identiﬁcation, upon taking z! , together with the generisation homomorphism, gives ui : QY [−2c ] = z! QCY X −→ i ! QX , which we deﬁne to be the orientation class. In the case of an lci morphism f = p ◦ i, we deﬁne the orientation to be uf = ui · up . This only depends (it turns out) on f , not on the factorisation f = p ◦ i. The composition of lci morphisms is again lci, and orientation classes are functorial; see [Ve] for details. (4) We now show that specialisation is compatible. First, let us recall how specialisation works in Chow theory. Let α ∈ CH k X. Consider then [A1 \ {0}] × α = p1∗ α ∈ CH k+1 U . Let α˜ ∈ CH k+1 M 0 now be any cycle on M 0 , with the property that α˜ | U = α (for instance the Zariski closure of some representative of α mod rational equivalence); by deﬁnition σY,X α = i ! α˜ ∈ CH k CY X.

480

CORTI AND HANAMURA

We want to show that cl ◦σY,X = σY,X ◦ cl, that is, specialisation is compatible. Let ˜ It is a˜ = cl α. ˜ Intersection with a Cartier divisor is compatible; hence, i ! a˜ = cl i ! α. therefore enough to show that i ! a˜ = σY,X a. We prove that this is true, assuming that a˜ | U = p1∗ a, which is satisﬁed because pulling back by smooth morphisms is compatible. In order to compare the two sides of the last displayed equation, it is now necessary to make a small digression. The inclusion i : {0} → A1 is canonically oriented by the identiﬁcation u : Q{0} −→ i ! QA1 [2] 2 A1 , if you like). Now u itself is the (think of u as a class in H 2 (A1 , A1 \ {0}) = H{0} composite of two maps:

Q{0} VVV LL VVVVV LL LL VVVVVVuV LL VVVV LL VVVV % V+ ∗ / i ! Q 1 [2]. Rj∗ j QA1 [1] A Pulling this last diagram back to the deformation M 0 and using the exactness of π ∗ creates the diagram QCY X VV KKKVVVVV KKK VVVVVu VVVV KKK VVVV K% V* / i ! Q 0 [2]. Rj∗ QU [1] M Composing with a homomorphism a : QM 0 [2] → DM 0 [2 − i], regarded as a class a ∈ HiBM M 0 , gives HiBM M 0 U II UUUU II UUUU i ! II UUUU II UUUU I$ U* BM U / H BM CY X. Hi−1 i−2 Here i ! is intersection with the Cartier divisor CY X ⊂ M 0 and is a compatible operation. Combining the last diagram with the one in (2) that deﬁnes the specialisation

MOTIVIC DECOMPOSITION AND INTERSECTION CHOW GROUPS

481

homomorphism for Borel-Moore homology, we get the commutative diagram BM M 0 Hi+2 LLVLVVVVVV ! LLL VVVi VV VVVV LL& VV* ! BM U / H BM CY X i{1} Hi+1 h4 i 8 h h r h r hh rrr hhhhh rhrhrhhhhhh σY,X HiBM X

We denote i{1} : {1} → A1 the inclusion. The commutativity of the leftmost triangle is easy. This last diagram settles the claim (and the substance of the proof of Theorem 2.10). (5) We conclude here the proof of Theorem 2.10. For simplicity, we only deal with the case of regular embeddings, lci morphisms being a relatively straightforward generalisation. First, we show that Gysin maps are compatible. Let i : Y → X be a regular embedding of codimension c. In Chow theory, the Gysin map i ! : CH k X → CH k−c Y is deﬁned by the formula i ! = z! σY,X . In Borel-Moore homology, by deﬁnition i ! a = ui ∩ a, where ui ∈ H 2c (X, X \ Y ) = HY2c X = Hom(QY , i ! QX [2c]) is the orientation class constructed in (3). From the construction of the orientation class, it is easy to see that i ! = z! σY,X in Borel-Moore homology too. The compatibility then follows from the compatibility of z! and σY,X . Let us now consider reﬁned Gysin maps; in other words, consider a ﬁbre square Y g

Y

i

/ X g

i

/X

Now, reﬁned Gysin maps in Chow theory and Borel-Moore homology satisfy the same bivariant formalism (especially [F1, Theorem 6.2]). Therefore, to show that they are compatible, it is enough to do so in the case when i is a regular embedding of codimension c (e.g., by blowing up until i becomes a Cartier divisor, and then using compatibility with proper pushforward; see [F1, Theorem 6.2(a)]). The excess bundle is deﬁned by the exact sequence 0 −→ NY X −→ g ∗ NY X −→ E −→ 0. If e = c − c is the rank of E, the excess intersection formula in both theories says i ! a = ce (E) ∩ i ! a,

482

CORTI AND HANAMURA

which concludes the proof of Theorem 2.10 (cupping with ce (E) is compatible, and i ! is an ordinary Gysin map, just shown to be compatible). 3.2. Products and composition. The purpose of this section is to prove Lemma 2.23. The proof is rather technical, but it is a fairly natural generalisation of the wellknown statement that, on a smooth variety Y , cup product of cohomology classes is compatible (via the Poincaré duality isomorphism) with intersection of Borel-Moore homology classes. The proof is based on the following lemma. Lemma 3.1. Let T be a smooth variety, and let p : U → T , q : V → T be morphisms, with p proper. There are natural isomorphisms ∼ = λ : HomT Rp∗ QU , QT [i] −−→ H2BM dim T −i U, ∼ = µ : HomT QT , Rq∗ q ! QT [j ] −−→ H2BM dim T −j V , ∼ = ν : HomT Rp∗ QU , Rq∗ q ! QT [i + j ] −−→ H2BM dim T −i−j U ×T V , which satisfy the identity ν (v ◦ u) = δ ! µ (v) × λ (u) . Let us explain the idea, which is pretty basic. In the simplest case, where both p : U → T and q : V → T are the identity map T = T , the lemma just says that cup product in cohomology is compatible, via the Poincaré duality isomorphism, with intersection product in homology. Indeed, HomT (QT , QT [i]) = H i T , HomT (QT [i], QT [i + j ]) = H j T , and v ◦ u ∈ HomT (QT , QT [i + j ]) is the cup product v ∪ u. Here we let λ = µ = ν = P : H • T → H2BM dim T −• T be the Poincaré ! duality isomorphism. The lemma then says P (v ∪ u) = δ (P v × P u), but this is ﬁne because δ ! (P v × P u) = P v · P u is the intersection product. Proof. Step 1. We begin constructing natural isomorphisms ∼ = λ : HomT Rp∗ QU , QT [i] −−→ H i (U → T ), ∼ = µ : HomT QT , Rq∗ q ! QT [j ] −−→ H j (V → T ), ∼ = ν : HomT Rp∗ QU , Rq∗ q ! QT [i + j ] −−→ H i+j U ×T V → T , which satisfy the identity ν(v ◦ u) = µ(v) ∪ λ(u). The three isomorphisms are deﬁned in the following manner. The isomorphism λ is the identity since (p is proper) by deﬁnition, HomT Rp! QU , QT [i] = H i (U → T ).

MOTIVIC DECOMPOSITION AND INTERSECTION CHOW GROUPS

483

The isomorphism µ is deﬁned with a single application of standard duality, HomT QT , Rq∗ q ! QT [j ] = HomV QV , q ! QT [j ] = H j (V → T ). For the rest of the proof, we ﬁx the notation in the following diagram: U ×T VG GG p w w GG ww GG w GG ww w w{ # U HH V v HH vv HH v v H vv p HH H# {vvv q T q

To deﬁne ν, ﬁrst let ∼ = χ : HomV q ∗ Rp∗ QU , q ! QT [k] −−→ H k U ×T V → T be the isomorphism obtained, composing the following natural identiﬁcations HomV q ∗ Rp∗ QU , q ! QT [k]

= (base change, p proper) HomV Rp∗ q ∗ QU , q ! QT [k] = HomV Rp∗ QU ×T V , q ! QT [k] = (p proper) HomU ×T V QU ×T V , p ! q ! QT [k] = H k U ×T V → T .

As a small digression, let u : Rp∗ QU → QT [i], v : QT → Rq∗ q ! QT [j ], and let v : q ∗ QT = QV → q ! QT [j ] correspond to v under Verdier duality. At this point we would like to observe that µ(v) ∪ λ(u) = q ∗ λ(u) · µ(v) = χ v ◦ q ∗ (u) . This ends the digression. Now, to come back to the deﬁnition of ν, we just compose χ with a standard duality isomorphism HomT Rp∗ QU , Rq∗ q ! QT [k] χ = HomV q ∗ Rp∗ QU , q ! QT [k] −→ H k U ×T V → T .

484

CORTI AND HANAMURA

Look now at the commutative diagram Rp∗ QU u

QT [i] v

Rq∗ q ! QT [i + j ]

tr

tr

/ Rq∗ q ∗ Rp∗ QU

Rq∗ q ∗ λ(u)

/ Rq∗ q ∗ QT [i] Rq∗ v

Rq∗ q ! QT [i + j ]

The diagram shows that v ◦ q ∗ u corresponds to v ◦ u under the standard duality isomorphism HomT Rp∗ QU , Rq∗ q ! QT [i + j ] = HomV q ∗ Rp∗ QU , q ! QT [i + j ] , which was used in the deﬁnition of ν. Therefore ν(v ◦ u) = χ(v ◦ q ∗ u). On the other hand, as we have seen in the digression µ(v) ∪ λ(u) = χ(v ◦ q ∗ u). Combining the last two displayed formulas concludes step 1. Step 2. We now deﬁne ∼ = λ = Pω ◦ λ : HomT Rp∗ QU , QT [i] −−→ H2BM dim T −i U, ∼ = µ = Pω ◦ µ : HomT QT , Rq∗ q ! QT [j ] −−→ H2BM dim T −j V , ∼ = ν = Pω ◦ ν : HomT Rp∗ QU , Rq∗ q ! QT [i + j ] −−→ H2BM dim T −i−j U ×T V . The statement now follows from step 1, Deﬁnition-Theorem 2.14, and a simple calculation, ν (v ◦ u) = Pω ν(v ◦ u) = Pω µ(v) ∪ λ(u) = Pω Pω−1 µ (v) ∪ Pω−1 λ (u) = δ ! µ (v) × λ (u) . This ﬁnishes the proof.

MOTIVIC DECOMPOSITION AND INTERSECTION CHOW GROUPS

485

Proof of Lemma 2.23. We summarise the notation for the various spaces and maps in the following commutative diagram: X ×O S Z pXZ

X ×S Y ×SN Z NNN pp NNN ppp p NNN p p p N& p xp X ×S Y N Y ×S Z NNN p p3 pppp 1 N NNN p p2 p2 NNN ppp N' wpppp X OOO Y pZ OOO ppp p OOO p p2 ppp p1 OOO OO' wppppp p3 S The proof of the lemma results from contemplating the following commutative diagram: Rp1∗ QX [i] u

Rp2∗ QY [j ] v

Rp3∗ QZ [k] o

/ Rp2∗ p ∗ Rp1∗ QX [i] 2

Q Rp2∗ Rp1∗ X×S Y [i]

Rp2∗ u

Rp2∗ u

Rp2∗ QY [j ]

Rp2∗ v

Rp2∗ p2! Rp3∗ QZ [k]

Rp2∗ QY [j ]

Rp2∗ v

p ! Q [2 dim Y − 2 dim Z + k] Rp2∗ Rp3∗ 3 Y

Here u : p2∗ Rp1∗ QX [i] → QY [j ] corresponds to u via the standard duality HomY p2∗ Rp1∗ QX [i], QY [j ] = HomS Rp1∗ QX [i], Rp2∗ QY [j ] , and tr : Rp1∗ QX [i] → Rp2∗ p2∗ Rp1∗ QX [i] is the trace map giving rise to the duality (or arising from the duality, depending on the preference of the reader). Similar comments apply to v . The two equal signs in the right portion of the diagram are obtained from the proper base change isomorphism; for instance, the lower one (which is the hardest) is derived as follows: p2! Rp3∗ QZ [k] = (base change) Rp3∗ p2! QZ [k] p2! DZ [−2 dim Z + k] = (Z is smooth) Rp3∗ = Rp3∗ DY ×S Z [−2 dim Z + k] p3! QY [2 dim Y − 2 dim Z + k]. = (Y is smooth) Rp3∗

486

CORTI AND HANAMURA

Contemplating the diagram in the light of how ϕ is deﬁned (Lemma 2.21 and Remark 2.22), and using Lemma 3.1 (with X ×S Y , Y ×S Z, and Y in place of U , V , and T , resp.) we evince the following: ϕ(v ◦ u) = pXZ∗ ν (v ◦ u ), ϕ(u) = λ (u ), ϕ(v) = µ (v ). The result then follows immediately from Lemma 3.1. This ﬁnishes the proof. 3.3. Ᏸbcc is pseudoabelian. Proof of Lemma 2.24. We give a brief outline of a proof of Lemma 2.24, stating that Ᏸbcc (S) is pseudoabelian. In fact the proof works for the full subcategory D b of cohomologically bounded objects in any triangulated category Ᏸ with t-structure. Step 1. Let p 2 = p : M → M be a projector; we wish to construct the kernel and image K and I of p. The proof is by induction on the cohomological amplitude of M. For a suitable i, M = τ≤i M and M = τ>i M both have smaller cohomological amplitude than M. So we may assume by induction that p = τ≤i p, respectively, p = τ>i p, have kernel and image K and I , respectively, K and I . Note that we have a morphism of exact triangles M p

M

/M

/ M

p

/M

[1]

/

[1]

/

p

/ M

Step 2. With the identiﬁcation M = K ⊕I , p : M → M is the projection to the second factor, and similarly for M . Denoting δ : M → M [1] the map of degree 1, we have a commutative diagram K ⊕ I K ⊕ I

δ

δ

/ K [1] ⊕ I [1] / K [1] ⊕ I [1]

where the vertical arrows are projections on the second factor. We deduce that δ(K ) ⊂ K [1] (this has an obvious meaning in any additive category). Similarly, arguing with 1 − p instead of p, we also have that δ(I ) ⊂ I [1]. Step 3. Choose now a triangle K −→ K −→ K −→ K [1].

487

MOTIVIC DECOMPOSITION AND INTERSECTION CHOW GROUPS

There is then a morphism ε : K → M so that the following is a morphism of triangles: K

/K

M

/M

/ K

ε

/ M

[1]

/

[1]

/

Replacing ε with ε − p ◦ ε, we may assume that p ◦ ε = 0. Step 4. Let now F : D → abelian groups be any cohomological functor. Then the following sequence is exact: p

ε

0 −→ F K −→ F M −→ F M. Using p ◦ ε = 0 and step 2, the claim follows from a never-ending diagram chase along the paths and trails of the following diagram 0

0

F −1 K

/ F K

/ FK

F −1 K ⊕ F −1 I

/ F K ⊕ F I

/ FM

F −1 K ⊕ F −1 I

/ F K ⊕ F I

/ FM

ε

p

0

0

/ F K

/ F 1K

/ F K ⊕ F I

/ F 1K ⊕ F 1I

/ F K ⊕ F I

/ F 1K ⊕ F 1I

Step 5. Apply step 4 to F = Hom(U, −), where U is an arbitrary object. This shows that K = Ker(p). Then I = Ker(1 − p), which concludes the proof. 4. Standard conjectures and canonical ﬁltrations. In this section, which is intended mainly for reference, we begin recalling Grothendieck’s standard conjectures, which were introduced, among other things, to determine the behaviour of the category ᏹ of Grothendieck motives. We mainly need them in Section 5, when we deﬁne the relative analogue ᏹ(S) of ᏹ and show (assuming the conjectures) that it is an abelian semisimple category and the decomposition theorem holds in ᏹ(S). Then we recall Murre’s conjecture, which we only need in Sections 6 and 7, which implies the existence of a natural ﬁltration F • on the Chow groups of smooth and projective varieties. We explain how this conjecture can be used to understand the relationship between ᏹ and CH ᏹ, making it possible to deﬁne a noncanonical decomposition of a Chow motive into its cohomology groups. Finally, we recall S. Saito’s unconditional deﬁnition of a ﬁltration, which has all the expected categorical properties, except

488

CORTI AND HANAMURA

that it is not known to be separated. If it is separated, it coincides with Murre’s, and Murre’s conjecture holds. It is Saito’s ﬁltration that is extended, in Section 6, to the Chow groups of quasiprojective varieties. This is used in Section 7 to prove the decomposition theorem in CH ᏹ(S), and in the forthcoming part II, where we propose an unconditional deﬁnition of the intersection Chow groups ICH r X. 4.1. Standard conjectures. Before stating the conjectures, we introduce some notation and recall some well-known facts on the cohomology of algebraic varieties. Let X be a smooth projective variety of complex dimension d, with a ﬁxed ample divisor class L ∈ H 2 X. The Lefschetz theorem asserts that, for i ≤ d, the d − ith iterated cup product with L is an isomorphism of H i X to H 2d−i X, ∼ =

Ld−i : H i X −−→ H 2d−i X. For i ≤ d we then deﬁne the primitive cohomology of X to be P i X = Ker Ld−i+1 ⊂ H i X. We have the hard Lefschetz decomposition of the cohomology of X H i X = ⊕j ≥0 Lj P i−2j X and

(if i ≤ d),

H i X = ⊕j ≥i−d Lj P i−2j X

(if i > d).

Deﬁnition 4.1. The Lefschetz operator : H i X → H i−2 X relative to the ample class L is deﬁned in the following manner. Let α ∈ H i X and write, using the Lefschetz decomposition, Lj α i−2j , α= j

with

α i−2j

∈ P i−2j X.

Then by deﬁnition, Lj −1 α i−2j α = j

(i.e., removes one L). Grothendieck [Gr] proposed the following two standard conjectures. Conjecture 4.2. The operator is algebraic. Conjecture 4.3. The rational quadratic form (α, β) −→ (−1)i tr α ∪ β ∪ Ld−2i 2i X. is positive deﬁnite on P 2i ∩ Halg

MOTIVIC DECOMPOSITION AND INTERSECTION CHOW GROUPS

489

Some recent progress on these conjectures can be found in [Ja2] and [Sm]. In the proof of the decomposition theorem in ᏹ(S), Section 5, we need the following simple consequence of Conjecture 4.2. Proposition 4.4. Assume the standard Conjecture 4.2. Let S be a smooth quasiprojective variety, and let f : X → S be a smooth projective morphism, with relatively ample divisor class L ∈ H 2 X. There exists a cycle Z ∈ CH dim X+1 X ×S X , such that, for every s ∈ S and ﬁbre Xs , Z|Xs ×Xs induces the s operator (relative to the class Ls = L | Xs ) of that ﬁbre. Proof. The proof uses a standard “spreading out” argument followed by specialisation. By Conjecture 4.2, there is a cycle Zη on Xη ×Xη inducing η . Let U ⊂ S be a neighbourhood of η, let ZU be a cycle on XU ×U XU such that ZU | η = Zη , and let Z on X ×S X be its Zariski closure. We claim that for all scheme theoretic points s ∈ S, Z|Xs ×Xs induces s . By considering a chain of (scheme theoretic) points s ∈ s1 ⊂ s2 ⊂ · · · ⊂ η with codsi si+1 = 1, we are reduced to the case of the spectrum T of a discrete valuation ring with central point zero and generic point s, and a morphism T → S. Assuming that Z|Xs ×Xs induces s , we need to prove that Z|X0 ×X0 induces 0 . In this situation, letting k(s) be the function ﬁeld of T , there are well-deﬁned specialisation maps CH i Xs × Xs cl

csp

/ CH i X × X 0 0

H 2i Xs × Xs

hsp

cl

H 2i X0 × X0

Let us recall the construction of the Chow theoretic specialisation homomorphism. We have a diagram CH i−1 X0 × X0

i∗

/ CH i X × X T

j∗

/ CH i X × X s s

/0

i!

CH i X0 × X0

where the row is exact. Because X0 ∼ 0, we can deﬁne csp(α) = i ! α , where α ∈ CH i X ×T X is anything such that α |Xs ×Xs = α, and the result does not depend on

490

CORTI AND HANAMURA

α . Then, by construction of csp, Z|X0 ×X0 = csp(Z|Xs ×Xs ). On the other hand clearly 0 = hsp(s ), by what we just said about specialisation L0 = csp(Ls ); so “removing one L” specialises to “removing one L.” Therefore, cl(Z|X0 ×X0 ) = 0 . The most important consequence of the standard conjectures is the following theorem (see [Kl1], [Kl2]). Theorem 4.5. Assuming the standard Conjectures 4.2 and 4.3, we have the following. (1) H ᏹ = ᏹ; in other words, homological and numerical equivalence of algebraic cycles are the same. (2) The category ᏹ of Grothendieck motives is abelian and semisimple. Proof. See [Kl1]. 4.2. Murre’s conjecture. The standard conjectures are perfectly adequate in determining the behaviour of Grothendieck motives. There is a large gap between Grothendieck motives and Chow motives, which one begins to appreciate when trying to decompose a Chow motive hX into its pieces hi X[−i] in an unambiguous way. To address this issue, Murre [Mu1], [Mu2] proposed the following conjecture. Conjecture 4.6. Let X be a smooth variety of complex dimension d, and let ∈ H i X ⊗ H 2d−i X ⊂ H 2d X × X be the Künneth components of the diagonal. Then we have the following. (A)The π i lift to an orthogonal set of projectors Hi ∈ CH dim X (X × X) such that I = Hi . (B) The correspondences H2r+1 , . . . , H2 dim X act as zero on CH r X. (C) For each ν, F ν CH r X = Ker H2r ∩ · · · ∩ Ker H2r−ν+1 is independent of the choice of the Hi . (D) F 1 CH r X = CH rhom X is the group of cycles homologically equivalent to zero.

πi

Jannsen proved the following theorem [Ja1, p. 259, pp. 294–296]. Theorem 4.7. Assume Conjecture 4.6. The ﬁltration F • satisﬁes the following properties. (a) F 0 CH r X = CH r X, F 1 CH r X = CH rhom X. (b) F ν CH r X · F µ CH s X ⊂ F ν+µ CH r+s X. (c) If f : X → Y is a morphism of smooth projective varieties, f∗ and f ∗ respect the ﬁltration (no shifts involved). (d) Let , ∈ CH dim X X × X be a correspondence. Assume that , acts trivially on H 2r−ν X. Then ,∗ : F ν CH r X −→ F ν+1 CH r X.

MOTIVIC DECOMPOSITION AND INTERSECTION CHOW GROUPS

491

(e) Assuming, moreover, the standard conjecture of Lefschetz type, F r+1 CH r X = (0). We show momentarily that, assuming the standard conjecture of Lefschetz type, Murre’s (A)+(B)+(C)+(D) is equivalent to (A)+(B )+(D). Conjecture 4.8 (B ). Let X be a smooth projective variety, and let P ∈ CH dim X X × X be a projector. Assume P∗ H i X = 0, for i ≤ 2r; then P∗ CH r X = (0). Proposition 4.9. Assuming the standard Conjecture 4.2, Murre’s (A)+(B)+ (C)+(D) implies (B ). Proof. Assume (A)+(B)+(C)+(D). By Theorem 4.7(d), P∗ F ν CH r X ⊂ for all ν ≤ 2r. But P is a projector, so P∗ CH r X = P∗2 CH r X ⊂ r 1 P∗ F CH X = P∗2 F 1 CH r X ⊂ P∗ F 2 CH r X · · · ⊂ F r+1 CH r X = 0 by Theorem 4.7(e). F ν+1 CH r X,

With the aim of proving that (A)+(B )+(D) implies (A)+(B)+(C)+(D), we now make a small digression and discuss the decomposition of Chow motives. The decomposition theorem in CH ᏹ(S) in Section 7 follows the same strategy. Deﬁnition 4.10. A Chow motive M has cohomological degree less than or equal to m, respectively, greater than or equal to m, if the cohomology groups H iM = 0 vanish for i > m, respectively, i < m. M has degree exactly m if it has degree greater than or equal to m and less than or equal to m. Proposition 4.11. Conjecture (B ) implies the following. (1) If M has cohomological degree less than or equal to m and if N has cohomological degree greater than m, then HomCH ᏹ (M, N) = 0. (2) If M and N have cohomological degree exactly m, then the natural homomorphism HomCH ᏹ (M, N) −→ Homᏹ (M, N) is an isomorphism. Proof. Assume for simplicity that M = (X, P ) and N = (Y, Q). Let Z = X × Y and consider the projector CH i Z α −→ Kα = (idX ×Q) ◦ α ◦ (P × idY ) ∈ CH i Z. To prove both statements, it is enough to show that K = 0 if cl K = 0, but this is (B ).

492

CORTI AND HANAMURA

We draw two consequences. Corollary 4.12. Assume (A)+(B ); then for all smooth projective varieties X: (1) There is a noncanonical direct sum decomposition hX = X, Hi . (2) The monomorphisms τ≤i hX =

(X, Hm ) −→ hX

m≤i

(where the ﬁrst equality is a deﬁnition of τ≤i hX) are speciﬁed up to canonical isomorphism. In particular, so are the “subquotients” hi X[−i] = X, Hi (this is a deﬁnition of hi X[−i]) speciﬁed up to canonical isomorphism. Corollary 4.13. (A)+(B )+(D) implies (A)+(B)+(C)+(D). Proof. We can see that

F ν CH i X = HomCH ᏹ pt, (τ≤2i−ν hX)(i)

is independent on the Hi s, by Corollary 4.12. 4.3. Saito’s ﬁltration. S. Saito [SaS] gave an unconditional deﬁnition of a ﬁltration on the Chow groups of smooth projective algebraic varieties over k and proved that, assuming the standard conjectures, it coincides with Murre’s ﬁltration. We now recall Saito’s deﬁnition and his results. Deﬁnition 4.14 [SaS]. For a smooth projective variety X we deﬁne a ﬁltration CH r X = F 0 CH r X ⊃ F 1 CH r X ⊃ · · · ⊃ F ν CH r X ⊃ · · · in the following inductive way. (1) F 0 CH r X = CH r X. (2) Assume F ν CH r X, deﬁned for all X and all r. Then we set F ν+1 CH r X = ,∗ F ν CH r−q Y, Y,q,,

where Y , q, and , range over the following data. (2.1) Y is smooth and projective. (2.2) q is an integer (the operation yields nothing unless r − dim Y ≤ q ≤ r, since otherwise CH r−q Y = 0). (2.3) , ∈ CH dim Y +q (Y × X) = HomC ᏹ (Y, X(q)) is a correspondence such that ,∗ H 2r−2q−ν Y ⊂ N r−ν+1 H 2r−ν X, where N • is Grothendieck’s coniveau ﬁltration (see Reminder 4.15).

MOTIVIC DECOMPOSITION AND INTERSECTION CHOW GROUPS

493

Reminder 4.15. The coniveau ﬁltration on the cohomology of a smooth projective algebraic variety X is deﬁned as N pH i X = f∗ H i−2q Y, Y,f

where the sum ranges over all smooth projective Y with q = dim X − dim Y ≥ p and morphisms f : Y → X. Usually, N p is deﬁned to be the sum of the kernels of the H • X → H • U , as U ⊂ X vary over all the open subsets of X with complement Z of codimension greater than or equal to p. This is equivalent to our deﬁnition if resolutions of singularities exist. Theorem 4.16 [SaS]. (1) The ﬁltration deﬁned in Deﬁnition 4.14 satisﬁes the properties (b), (c), and (d) of Theorem 4.7 and F 1 CH r X = CH rhom X. (2) If the ﬁltration is separated, that is, F ν CH i X = 0 for ν large, then Murre’s conjecture holds and the ﬁltrations are the same. (3) Assuming the standard and Murre’s conjectures, the ﬁltrations are the same (in particular they are separated). Remark 4.17. It would have been possible to deﬁne F • just as in Deﬁnition 4.14, but if we replace condition (2.3) with the easier 0 = ,∗ H 2r−2q−ν Y −→ H 2r−ν X, Theorem 4.16 would still be true (in fact, slightly easier to prove) for this ﬁltration. The following property is not stated explicitly in [SaS] and is needed in Section 6. Proposition 4.18. For X, Y smooth projective and f : X → Y a surjective morphism, f∗ : CH s X −→ CH s Y is strictly compatible with the F • ﬁltration as deﬁned in Deﬁnition 4.14. Proof. Choose a diagram i

/X Z@ @@ @@ f π @@ Y

where Z is smooth projective, i : Z → X is a closed embedding, and π : Z → Y is a generically ﬁnite morphism of degree d. Let α ∈ F ν CH r Y . Then α= We can now say that by Theorem 4.7(c).

1 1 π∗ π ∗ α = f∗ i∗ π ∗ α. d d

i∗ π ∗ α ∈ F ν CH r X,

494

CORTI AND HANAMURA

5. Grothendieck motives over a base, semisimplicity, and decomposition. This section is divided into three subsections. In the ﬁrst, for the convenience of the reader and to ﬁx the notation, we recall the notion of perverse sheaves and the statement of the topological decomposition theorem. The standard references are [Bo] and [BBD]. For the expert, we say right away that we found it convenient to use Deligne’s convention for perverse sheaves, because it is better suited for taking direct images under a closed embedding, and Borel’s convention for intersection complexes. Therefore, with our conventions, intersection complexes are not perverse (but a suitable shift is). In the second section we deﬁne a category ᏹ(S), which we call the category of Grothendieck motives over a variety S. We think that this is the correct analogue of the category of Grothendieck motives over the point and is built precisely in order to have a faithful realisation in the graded category of perverse sheaves. In the third section, assuming the standard conjectures, we prove that ᏹ(S) is abelian and semisimple; as a consequence, we derive a decomposition theorem in ᏹ(S) which realises to the topological decomposition theorem. 5.1. Perverse sheaves and the topological decomposition theorem. In this section, we recall the theory of perverse sheaves and the topological decomposition theorem. The standard reference for this material is [BBD]. For ease of notation and terminology, we assume that k = C in most statements, and refer the reader to the original source for the language suitable to the étale situation. Deﬁnition 5.1. Let Ᏸ be a triangulated category. A t-structure on Ᏸ is a pair (Ᏸ≤0 , Ᏸ≥0 ) of full subcategories of Ᏸ, satisfying the following axioms. (1) Ᏸ≤0 [1] ⊂ Ᏸ≤0 and Ᏸ≥0 ⊂ Ᏸ≥0 [1]. (2) Hom(Ᏸ≤0 , Ᏸ>0 ) = 0. (3) For every object K of Ᏸ, there is a (necessarily unique up to canonical isomorphism) triangle 1

K −→ K −→ K −→ with K ∈ Ᏸ≤0 , K ∈ Ᏸ>0 . The assignment K to K = τ≤0 K is functorial and the corresponding functor is called the truncation functor relative to the t-structure. τ≤m K is deﬁned to be (τ≤0 (K[m]))[−m], similarly τ≥m , and Ᏼm (−) = (τ≤m τ≥m (−))[m] is the mth cohomology functor relative to the t-structure. The main theorem [BBD] about t-structures asserts that the heart Ᏸ≤0 ∩ Ᏸ≥0 is an abelian category. We now come to the most important example of t-structure, the perverse t-structure on Ᏸbcc (S). But ﬁrst, we provide the following deﬁnition. Deﬁnition 5.2. Let S be a quasi-projective variety over a ﬁeld k. A good stratiﬁcation of S is a stratiﬁcation S= Tk ,

MOTIVIC DECOMPOSITION AND INTERSECTION CHOW GROUPS

495

where Tk is a Zariski locally closed subset of complex dimension k, satisfying the following axioms. (1) Each stratum Tk is smooth. (2) The stratiﬁcation is topologically normally locally trivial. From now on, we assume that all varieties S are equipped with a good stratiﬁcation. Notation 5.3. If S = k Tk is a good stratiﬁcation, we denote iTk : Tk → S as the inclusion and Sk = h≤k Th as the Zariski closure of Sk . Deﬁnition 5.4. (1) Let ᐀ = {Tk } be a good stratiﬁcation; denote Ᏸ᐀ (S) = K ∈ Ᏸb (S) | K | Tk is cohomologically locally constant ∀k . (2) The bounded derived category of cohomologically constructible sheaves is deﬁned as Ᏸbcc (S) = ∪Ᏸ᐀ (S), the union being taken over all good stratiﬁcations of S. From now on, when dealing with a sheaf K ∈ Ᏸbcc (S), in connection with a preexisting good stratiﬁcation ᐀ = {Tk }, we assume that K is cohomologically locally constant along all strata Tk of ᐀. Deﬁnition 5.5. The perverse t-structure on Ᏸbcc (S) is deﬁned as p

Ᏸ≤0 = ∪p Ᏸ᐀ ≤0 ,

p

Ᏸ≥0 = ∪p Ᏸ᐀ ≥0 .

The union is taken over all good stratiﬁcations ᐀, where p Ᏸ᐀ ≤0 = K ∈ Ᏸ᐀ | Ᏼi iT∗k K = 0, i > −k , p Ᏸ᐀ ≥0 = K ∈ Ᏸ᐀ | Ᏼi iT! k K = 0, i < −k . It is well known that the above data deﬁne a t-structure, whose heart ᏼerv(S) is the category of perverse sheaves on S. We denote the truncation, respectively, cohomology functors of the perverse t-structure, with the symbol p τ≤0 , respectively, p Ᏼm . The most important construction in the theory of perverse sheaves is that of the intersection complexes. Deﬁnition-Theorem 5.6. Let ᐀ be a good stratiﬁcation, and let V be a rational local system on the largest stratum Td . The intersection complex ᏵᏯV is the complex characterised by the properties ᏵᏯV | Td = V , Ᏼi iT∗k ᏵᏯV Ᏼi iT! k ᏵᏯV

(0)

= 0,

i ≥ d − k (k < d),

(−)

= 0,

i ≤ d − k (k < d).

(+)

496

CORTI AND HANAMURA

It follows immediately from the characterisation just given that the shift ᏵᏯV [d] is a perverse sheaf on S. It is important to understand that ᏵᏯV [d] is not characterised by being a perverse sheaf and restricting to V [d] on the largest stratum. In fact, there are a lot of perverse sheaves that restrict to V [d] on Td , and ᏵᏯV [d] is built to be as much in the middle of ᏼerv(S) as possible. The intersection cohomology of S is deﬁned as IH m S = H m ᏵᏯQTd . Similarly, if Vk is a local system on the stratum Tk of dimension k, then the intersection complex ᏵᏯVk is a complex supported on the Zariski closure Sk characterised by the properties ᏵᏯVk | Tk = Vk ,

(0)

Ᏼi iT∗h ᏵᏯV = 0,

i ≥ k − h (h < k),

(−)

Ᏼi iT! h ᏵᏯV

i ≤ k − h (h < k).

(+)

= 0,

It follows immediately from the characterisation just given that ᏵᏯVk [k] is a perverse sheaf on S. The following is the topological decomposition theorem. Theorem 5.7. Assume the base ﬁeld is C or the closure of a ﬁnite ﬁeld. Let X be a smooth variety, and let f : X → S be a projective morphism. (1) There is a noncanonical direct sum decomposition p m Rf∗ QX ∼ R f∗ QX [−m] = in Ᏸbcc (S), where p R m f∗ QX denotes the mth perverse cohomology of Rf∗ QX . The decomposition itself is not unique, but the subobjects p p i τ≤m Rf∗ QX = R f∗ QX [−i] i≤m

are uniquely speciﬁed. (2) Let ᐀ = {Tk } be a good stratiﬁcation with the property that p R m f∗ QX ∈ Ᏸ᐀ (S). There are local systems Vkm on Tk and a canonical isomorphism p m R f∗ QX = ᏵᏯVkm [k]. k

5.8. The statement about the uniqueness of the subobjects p τ≤m Rf∗ QX = Remark pR i f Q [−i] is an immediate consequence of the axiom Hom(Ᏸ≤0 , Ᏸ>0 ) = 0 ∗ X i≤m for t-structures.

497

MOTIVIC DECOMPOSITION AND INTERSECTION CHOW GROUPS

BM ᏹ(S), and let M = real M ∈ 5.2. Grothendieck motives over S. Let M ∈ Halg BM ᏹ(S) denotes the category of homological Ᏸbcc (S) be its realisation. (Recall that Halg motives over S, constructed in Section 2.) BM ᏹ(S), and let u = real u : M → N be Let u : M → N be a morphism in Halg its realisation in Ᏸbcc (S). Because real and the perverse p Ᏼm are functors, we get a system of compatible morphisms p

Ᏼm u : p Ᏼm M −→ p Ᏼm N.

These considerations imply that, passing to the corresponding graded objects, we have a perverse realisation functor p

BM real : Halg ᏹ(S) −→ gr ᏼerv(S),

and the following deﬁnition makes sense. Deﬁnition 5.9. The category ᏹ(S) of Grothendieck motives over S has the same BM ᏹ(S) and morphisms objects as Halg HomᏹS (M, N ) = Im HomH BM ᏹS (M, N ) → Homgr ᏼerv S (p real M, p real N ) . alg

Lemma 5.10. Assume that the base ﬁeld is C or the closure of a ﬁnite ﬁeld, let BM ᏹ(S), and let M = real M ∈ Ᏸb (S) be its realisation. Then, the kernel of M ∈ Halg cc the ring homomorphism EndH BM ᏹS M −→ End Ᏸbcc S M alg

is a nilpotent ideal. BM ᏹ(S), and let u = real u : M → N Proof. Let u : M −→ N be a morphism in Halg b be its realisation in Ᏸcc (S). With respect to decompositions

M= N=

p

Ᏼm M[−m],

p

Ᏼm N[−m],

u = ul m , where ul m : p Ᏼm M[−m] → p Ᏼl N[−l]. Here ul m can be regarded as an extension of perverse sheaves ul m ∈ Hom Ᏸbcc S

p

Ᏼm M[−m], p Ᏼl N[−l] = Ext m−l ᏼerv S

p

Ᏼm H, p Ᏼl K .

498

CORTI AND HANAMURA

Hence, ul m = 0 if m < l, corresponding to the fact that we have morphisms p τ≤m u : pτ p ≤m M → τ≤m N; u can be therefore visualised as an upper triangular matrix   .. .. . .   · · · u1 u1 · · · 1 2     2 u = · · · . 0 u · · · 2   · · · 0 0 · · ·   .. .. . . If a square matrix is strictly upper triangular, it is nilpotent. 5.3. Semisimplicity and decomposition in ᏹ(S) Convention 5.11. In this section we assume that k = C or the closure of a ﬁnite ﬁeld (so we can use the topological decomposition theorem), and that desingularisations of varieties over k exist. Notation 5.12. (1) In this section only, underlined capital letters M denote objects in ᏹ(S), while nonunderlined letters M denote the corresponding realisation in gr ᏼerv(S). The same letter denotes a morphism in ᏹ(S) or its realisation in gr ᏼerv(S); the context always makes it clear which is meant. When we want speciﬁcally to emphasise the realisation functor, we call it p real : ᏹ(S) → gr ᏼerv(S). (2) If X is a smooth variety and if f : X → S is a projective morphism, we denote p

Rf∗ QX

the corresponding object in ᏹ(S). We plan to prove the following results. Theorem 5.13. Assuming the standard conjectures, the category ᏹ(S) is abelian and semisimple. Theorem 5.14. Assume the standard conjectures. Let X be a smooth variety, and let f : X → S be a projective morphism. (1) There is a canonical direct sum decomposition in ᏹ(S) p p m Rf∗ QX ∼ R f∗ QX [−m], = where pR m f∗ QX [−m] denotes an object in ᏹ(S), together with a given isomorphism in gr ᏼerv(S) p

∼ =

real pR m f∗ QX [−m] −−→ pR m f∗ QX [−m].

(2) There is a canonical direct sum decomposition p m R f∗ QX [−m] = ᏵᏯVkm [k − m], k

MOTIVIC DECOMPOSITION AND INTERSECTION CHOW GROUPS

499

where Vk is a local system on a Zariski locally closed subvariety Tk ⊂ S and

ᏵᏯVkm [k − m] denotes an object in ᏹ(S), together with a given isomorphism in gr ᏼerv(S) p

∼ =

real ᏵᏯVkm [k − m] −−→ ᏵᏯVkm [k − m].

Remark 5.15. The decomposition is unique (contrary to the case of the topological decomposition Theorem 5.7) because of the way ᏹ(S) is constructed as a faithful subcategory of gr ᏼerv(S). The semisimplicity Theorem 5.13 is an immediate consequence of the following Proposition 5.16 (as is the case for motives over a point; compare [Kl1]) which is shown, together with the decomposition Theorem 5.14, at the very end of this section. Proposition 5.16. Let X be a smooth variety, and let f : X → S be a projective morphism. Assuming the standard conjectures, EndᏹS pRf∗ QX is a semisimple ring, ﬁnite dimensional over Q. The proof of the proposition relies on the following decomposition mechanism. Lemma 5.17. Let Ꮽ be an abelian semisimple category, let Ꮾ be an additive category, and let Ꮽ ⊂ Ꮾ be a fully faithful embedding. Let A, A be given objects of Ꮽ, let B be an object of Ꮾ, and let i∗

i∗

A −−→ B −−→ A be morphisms in the category Ꮾ. Denote a = i ∗ i∗ : A → A the composition; by assumption it is a morphism in the category Ꮽ. There is a nonunique projector β : B → B, with image in Ꮾ, and a natural isomorphism ∼ =

ϕ(β) : Im β −−→ Im a. Proof. Let V = Im a, t : A → V , and let t : V −→ A be the natural maps. Choose s : V → A, s : A → V such that s t = I dV ,

ts = I dV ,

and let α = ss : A → A. It is immediately veriﬁable that αaα = α and aαa = a. Now let β = i∗ αi ∗ ; it is immediate that β 2 = β. Claim. We have that V = Im β, by means of the following diagram: B GG GG G s i ∗ G#

β

V

/B ; ww w w ww i∗ s

Indeed, let X be any object of Ꮾ. We wish to check that Hom(X, B), respectively, Hom(B, X) is an image of _◦β, respectively, β ◦_, in the category of abelian groups,

500

CORTI AND HANAMURA

via the diagram / Hom(X, B) Hom(X, B) OOO o7 OOO ooo o OOO o oo OO' ooo Hom(X, V )

respectively, the diagram / Hom(B, X) Hom(B, X) OOO o7 OOO ooo o OOO o oo OO' ooo Hom(V , X)

In other words, we may assume that Ꮾ is an abelian category. The claim then follows from the observation that s i ∗ is surjective (indeed, s i ∗ i∗ = t is surjective) and i∗ s is injective (indeed, i ∗ i∗ s = t is injective). Example 5.18. As an example, we use the decomposition mechanism to very brieﬂy outline the calculation of the Chow motive of a 3-fold conic bundle, under the simplifying assumption that the discriminant curve be smooth. A considerably more elaborate and thorough treatment, covering integral motives in the general case, can be found in [Be]. Let X be a smooth 3-fold; f : X → S is a conic bundle structure. As anticipated we assume, for simplicity, that the discriminant I ⊂ S is a smooth divisor, denote Y = f −1 I and i : Y → X the normalisation of Y . In an obvious way Y → I factors through a P1 -bundle p : Y → D, with a distinguished section (the conductor) s : D → Y , by which we may think D ⊂ Y , and an étale double cover D → I. Let τ : D → D be the involution associated to this double cover. Step 1. Let

a = i ∗ i∗ : hS Y (−1) −→ hS Y

in CH ᏹ(S); then

a = c1 (NI S) − s∗ (s ∗ − τ∗ s ∗ ),

where (abusing notation slightly) NI S is the pullback to Y of the normal bundle NI S of I in S. Indeed, let , = ,i ⊂ Y × X be the graph of i, and let γ ∈ CH 2 (Y × X) ∗ t γ · p∗ γ ) be its class. Then i∗ = γ and i ∗ = t γ , while by deﬁnition, a = p13∗ (p23 12 ∗ t γ · p∗ γ (see Deﬁnitions 2.1 and 2.6). We can calculate the intersection product p23 12 with the help of the following ﬁbre square diagram: / Y ×Y D Y (s,sτ )

IY

Y ×Y

(1,i)×1

1×(i,1)

/ Y ×X×Y

MOTIVIC DECOMPOSITION AND INTERSECTION CHOW GROUPS

501

Then, by the excess intersection formula a = c1 (E) + s∗ τ∗ s ∗ , where E is the excess bundle on Y , deﬁned by the exact sequence 0 −→ TY −→ i ∗ TX −→ E −→ 0. Finally, it is easy to convince oneself that E = NI S(−D), giving c1 (E) = c1 (NI S)− D = c1 (NI S) − s∗ s ∗ ; this proves our formula. Step 2. Now Y → D is a P1 -bundle; therefore we have an isomorphism (p ∗ , s∗ ) : hS D ⊕ hS D(−1) −→ hS Y . Using this isomorphism, it is quite easy to see that a : hS D(−1) ⊕ hS D(−2) −→ hS D ⊕ hS D(−1) can be written in matrix form as 1−τ∗ c1 (NI S) . a= ∨Y ⊗ τ ∗N Y 0 c1 N I S ⊗ N D D Step 3. From the previous step and the decomposition mechanism we can verify, for instance, the classical result stating that the Prym motive (h1 D, 1−τ ∗ ) is a direct summand of the intermediate motive h3 X(1). Deﬁnition 5.19. Let X be a smooth variety; f : X → S is a projective morphism. An equisingular stratiﬁcation of f is a pair (᐀, ᐅ1 ⇒ ᐅ0 → ᐅ), where the following is true. (1) ᐀ = {Tk } is a good stratiﬁcation of S. (2) ᐅ = {Yk } and Yk is deﬁned by the ﬁbre square Yk

/X

Tk

/ S.

(3) ᐅ1 = {Yk1 }, ᐅ0 = {Yk0 }, and Yk1 ⇒ Yk0 → Yk is a truncated simplicial resolution. The above data are subjected to the following condition. (4) The compositions Yk1 → Tk , Yk0 → Tk are all smooth (not necessarily equidimensional). In particular, for all t ∈ Tk , 1 Yk,t

is a truncated simplicial resolution.

/

0 / Yk,t

/ Yk,t

502

CORTI AND HANAMURA

It is a consequence of our assumption on the existence of resolutions of singularities, that equisingular stratiﬁcations of separable morphisms f : X → S exist. We need the following lemma. Lemma 5.20. Let X be a smooth variety; f : X → S is a projective morphism. Fix an equisingular stratiﬁcation (᐀, ᐅ1 ⇒ ᐅ0 → ᐅ) of the morphism f . Let T0 be the smallest stratum of ᐀. We have a natural isomorphism (cf. the notation in the statement of the topological decomposition Theorem 5.7) iT0 ∗ V0m = Im R m f∗ iY0 ∗ iY! 0 QX → R m f∗ QX . Proof. Step 1. First of all, if h < k the natural maps Ᏼj iTh ∗ iT! h I CVkm −→ Ᏼj I CVkm

are zero for all j . Warning: this is not saying that iTh ∗ iT! h ᏵᏯVkm → ᏵᏯVkm is the zero map in Ᏸbcc (S). Indeed, Ᏼj iT! h ᏵᏯVkm = 0 for j ≤ k − h, and Ᏼj iT∗h ᏵᏯVkm = 0 for j ≥ k − h. Step 2. We have a ﬁbre square Y0 T0

iY0

/X f

iT0

/ S.

From it, using the proper base change theorem and the topological decomposition theorem, we derive the following commutative diagram

k>0

Rf∗ iY0 ∗ iY! 0 QX

/ Rf∗ QX

iT0 ∗ iT! 0 Rf∗ QX

/ Rf∗ QX

m ! m iT0 ∗ iT0 ᏵᏯVk [k − m]

⊕

m m V0 [−m]

/

k>0

m m ᏵᏯVk [k − m]

⊕

m m V0 [−m].

The result then follows from step 1, upon taking Ᏼm of both sides of the bottom portion of the diagram.

MOTIVIC DECOMPOSITION AND INTERSECTION CHOW GROUPS

503

5.3.1. Proof of Proposition 5.16 and Theorem 5.14. We may assume that the morphism f is separable (the statement is clear from the deﬁnitions in the case of a purely inseparable morphism). Fix an equisingular stratiﬁcation (᐀, ᐅ1 ⇒ ᐅ0 → ᐅ) of the morphism f : X → S. The proof is by induction on dim S and the number of strata in ᐀. The basis for the induction is solid because of the following. (a) If dim S = 0, ᏹ(S) = ᏹ is semisimple by Theorem 4.5, proven in [Kl1]. The decomposition Theorem 5.14 in this case can be proven in the following manner. Again, in [Kl1] it is shown that, if the standard conjecture of Lefschetz type holds, then there are cycles Hi representing the Künneth components π i of the diagonal I ⊂ X × X; the sought for decomposition is then X= X, Hi . (b) If ᐀ has only one stratum, note that, by deﬁnition of equisingular stratiﬁcation; this happens if and only if f : X → S is a smooth morphism. By Proposition 4.4, there is a cycle Z on X ×S X inducing the operator on each ﬁbre. The same proof as in [Kl1] then shows that EndᏹS X is a semisimple ring, ﬁnite dimensional over Q. Then again, as in [Kl1], there are classes Hi ∈ CH dim X X ×S X inducing on each ﬁbre the Künneth components of the diagonal of that ﬁbre; one can get the decomposition as above X= X, Hi . Let now T0 be the smallest stratum. We apply the decomposition mechanism, Lemma 5.17, with the following setup. Ꮽ = ᏹ(T0 ), which is by inductive assumption abelian and semisimple, and Ꮾ = ᏹ(S). We take A = Cok pRf∗ DY 1 (− dim X) → pRf∗ DY 0 (− dim X) , 0 0 A = Ker pRf∗ QY 0 → p Rf∗ QY 1 , 0

and

0

B = pRf∗ QX .

A and A are objects of ᏹ(T0 ), but if we like we can think of them as being in ᏹ(S) via the obvious inclusion ᏹ(T0 ) ⊂ ᏹ(S). There are obvious maps i∗ : A → B and i ∗ : B → A . We do not need this, but we still like to say that the assignment X to A, respectively, A , is functorial; in other words it does not depend on the choice of the equisingular stratiﬁcation. As we anticipated, we denote by A, A , and B the realisations in gr ᏼerv(S). Claim 1. Let R m = Ᏼm A, R m, = Ᏼm A . Then there is a natural isomorphism V0m = Im Ᏼm a : R m → R m, . In fact we know from the key lemma that V0m = Im Ᏼm iT! 0 Rf∗ QX → Ᏼm iT∗0 Rf∗ QX .

504

CORTI AND HANAMURA

Since Ᏼm iT! 0 Rf∗ QX (resp., Ᏼm iT∗0 Rf∗ QX ) has weights greater than or equal to m (resp., less than or equal to m), we have m ! W m ∗ iT0 ∗ V0m = Im gr W m Ᏼ iT0 Rf∗ QX → gr m Ᏼ iT0 Rf∗ QX . The claim now follows from the identiﬁcations m ∗ / gr W m Ᏼ iT0 Rf∗ QX

m ! gr W m Ᏼ iT0 Rf∗ QX

Ᏼm a

Rm

/ R m,

The decomposition mechanism, together with the claim, provides a projector β ∈ EndᏹS p Rf∗ QX such that, upon setting M = Ker β, V = Im β, we have p

Rf∗ QX = M ⊕ V ,

whose realisations in gr ᏼerv(S) are iT0 ∗ V0m [−m], V = m

M=

k>0 m

ᏵᏯVkm [k − m].

Let us now prove Proposition 5.16, that is, let us show that EndᏹS pRf∗ QX is a semisimple ring, ﬁnite dimensional over Q. According to the above decomposition EndᏹS pRf∗ QX = EndᏹS M ⊕ EndᏹT0 V . Now EndᏹT0 V is semisimple ﬁnite dimensional over Q by inductive assumption on dim S. The same is true of EndᏹS M by inductive assumption on the number of strata, by the following claim. Claim 2. We have EndᏹS M = Endᏹ(S\T0 ) M|S\T0 . Indeed we have a diagram EndᏹS M

inj

/ Endgr ᏼerv S

inj

/ Endgr ᏼerv(S\T ) 0

m m,k>0 ᏵᏯVk

surj

Endᏹ(S\T0 ) M|S\T0

m m,k>0 ᏵᏯVk |S\T0

MOTIVIC DECOMPOSITION AND INTERSECTION CHOW GROUPS

505

where surjectivity of the left vertical arrow follows from the fact that cycles can be closed: / H BM X ×S X CH • X ×S X 2• surj

CH • X ×S\T0 X

/ H BM X ×S\T X 0 2•

This ﬁnishes the proof of Proposition 5.16, because EndᏹS X is direct sum of two rings, each of which is semisimple and ﬁnite dimensional over Q. Finally, we shall now prove the decomposition Theorem 5.14. By induction on the number of strata, we may assume that the decomposition theorem holds over S \ T0 : p Rf∗ QX |S\T0 = ᏵᏯVkm [k − m]. k>0 m

To give such a decomposition is equivalent to giving the projectors Hm k ∈ Endᏹ(S\T0 ) down to ᏵᏯVkm . Since, by Claim 2, EndᏹS M = Endᏹ(S\T0 ) (M|S\T0 ), this also decomposes pRf∗ QX over S into the desired pieces. 6. Filtrations for quasi-projective varieties. Throughout this section, we assume that desingularisations of varieties over k exist. For a quasi-projective variety X we put a canonical ﬁltration on its rational Chow group CH s X so that some functorial properties are satisﬁed (see Theorem 6.1). The ﬁltration is shown to satisfy additional properties if one assumes the conjectures of Grothendieck and Murre (see Theorem 6.3). As an application, assuming these conjectures, we show that the projectors of Theorem 5.14 can be lifted to an orthogonal set of projectors in the Chow group of relative self-correspondences (see Corollary 6.10). This is done by showing that the map CH dim X X ×S X −→ EndᏹS p Rf∗ QX is a surjective homomorphism with nilpotent kernel. Theorem 6.1. For a quasi-projective variety X, there is a decreasing ﬁnite ﬁltration (the canonical ﬁltration) on its Chow group CH s X CH s X = F 0 CH s X ⊃ F 1 CH s X ⊃ F 2 CH s X ⊃ · · · subject to the following conditions. (i) If f : X → Y is a proper map of quasi-projective varieties, then the induced map f∗ : CH s X → CH s Y respects the ﬁltrations; that is, f∗ F ν CH s X ⊂ F ν CH s Y for each ν. If f is proper and surjective, then f∗ F ν CH s X = F ν CH s Y . (In other words, the surjection f∗ is strictly compatible with F • .)

506

CORTI AND HANAMURA

(ii) If j : U → X is an open immersion of quasi-projective varieties, then the restriction j ∗ : CH s X → CH s U is strictly compatible with F • ; j ∗ F ν CH s X = F ν CH s U . (iii) For a smooth projective X, F 1 CH s X = CH hom,s X = Ker cl : CH s X → H2s X , where cl is the cycle class map. (iv) The external product map CH s X ⊗ CH t Y → CH s+t (X × Y ) respects F • ; if z ∈ F ν CH s X and w ∈ F µ CH t Y , then z × w ∈ F ν+µ CH s+t (X × Y ). (v) The internal product respects F • ; if X is smooth, quasi-projective, and equidimensional, z ∈ F ν CH s X, and w ∈ F µ CH t X, then z · w ∈ F ν+µ CH s+t−dim X X. (vi) Reﬁned Gysin maps respect F • . Let i : X → Y be a regular embedding of codimension d, and let X

/ Y

X

/Y

i

be a Cartesian square, where Y is an arbitrary quasi-projective variety and where Y → Y is an arbitrary map. Then the reﬁned Gysin map [F1, Chapter 6] i ! : CH s Y −→ CH s−d X respects F • . (vii) If X, Y are quasi-projective varieties, X is equidimensional, and pY : X×Y → Y is the projection, then the map pY∗ : CH s Y → CH s+dim X (X × Y ) respects F • . (viii) Let W, X be smooth projective equidimensional, , ∈ CH dim X−i (W ×X)hom , and ,∗ : CH s+i W −→ CH s X is the induced map (by (i), (v), and (vii), ,∗ respects F • ). Then the map induces zero on the F -graded pieces: Gr •F ,∗ = 0 : Gr •F CH s+i W −→ Gr •F CH s X. Remark 6.2. (1) As already noted in Deﬁnition 4.14 and Proposition 4.18, S. Saito deﬁned a ﬁltration F • on CH s (X) for X smooth projective satisfying the conditions (i), (iii), (v), (vii) (where X, Y are smooth projective), and (viii).

MOTIVIC DECOMPOSITION AND INTERSECTION CHOW GROUPS

507

If we further assume Murre’s conjecture, it follows that Saito’s ﬁltration is separated, that is, for any X and s, one has F ν CH s X = 0 for ν large. In the proof of Theorem 6.1, we take Saito’s ﬁltration and show that it uniquely extends to a ﬁltration for X quasi-projective, so that the conditions (i)–(viii) are satisﬁed. (2) (iv) and (v) follow from (vi) and (vii). (3) We use the following properties of reﬁned Gysin maps [F1, Chapter 6]: compatibility with proper pushforwards, compatibility with ﬂat pullbacks, the excess intersection formula, and the fact that if i is of codimension 1 (namely, if X ⊂ Y is a Cartier divisor), then i ! coincides with intersection with X. (4) There is an interpretation of the ﬁltrations in terms of mixed motives; we do not need this. Theorem 6.3. Assuming Grothendieck’s and Murre’s conjectures, the ﬁltration in Theorem 6.1 also satisﬁes the following properties. (ix) For any quasi-projective variety BM F 1 CH s X = CH hom,s X = Ker cl : CH s X → H2s X , BM X is the cycle map into Borel-Moore homology. where cl : CH s X → H2s (x) For each X, one has F ν CH s X = 0 for ν large.

6.1. Proof of Theorem 6.1. We take Saito’s ﬁltration CH s X for X smooth projective and attempt to extend it to X general. First consider the case X is smooth quasi-projective. Take a smooth projective variety X and an open immersion j : X → X. It induces the surjective map j ∗ : CH s X −→ CH s X, and CH s X is given the induced ﬁltration: F ν CH s X = j ∗ F ν CH s X. This ﬁltration is independent of the choice of a compactiﬁcation. In fact, let j : X → X be another smooth compactiﬁcation. Since X and X are dominated by a third compactiﬁcation, one may assume that there is a map f : X → X such that f ◦ j = j . The diagram

CH s X I II j ∗ II II II $ f∗ CH :t s X t tt tt ∗ t tt j CH s X

508

CORTI AND HANAMURA

commutes. By the strictness of f∗ for surjective maps of smooth projective varieties (recall Proposition 4.18), one has

f∗ F ν CH s X = F ν CH s X, so j ∗ and j ∗ induce the same ﬁltrations. Proposition 6.4. (1) If X and Y are smooth quasi-projective varieties and f : X → Y is proper (resp., proper surjective), then the map f∗ : CH s X → CH s Y respects F • (resp., is strictly compatible with F • ). (2) If j : U → X is an open immersion of smooth varieties, j ∗ : CH s X → CH s U is strictly compatible with F • . Proof. For (1) we take smooth compactiﬁcations X, Y of X, respectively, Y , so that f extends to a map f : X → Y . Consider the commutative diagram CHO s X CH s X

f∗

f∗

/ CH s Y O / CH Y s

where the vertical arrows are the pullbacks by open immersions. Since f ∗ respects F • (resp., is strictly compatible with F • if f is surjective), f ∗ F ν CH s X ⊂ F ν CH s Y (resp., is equal). On the other hand, by deﬁnition F ν CH s X surjects to F ν CH s X, and F ν CH s Y surjects to F ν CH s Y . Hence f∗ F ν CH s X ⊂ F ν CH s Y (resp., is equal). For (2) take a compactiﬁcation j : X → X and consider the commutative diagram ∗

j / CH s U CH s XeK 9 KK ss KK s s KK ss K ss CH s X

where all the arrows are restrictions by open immersions. Since F ν CH s X surjects to both F ν CH s X and F ν CH s U , one has j ∗ F ν CH s X = F ν CH s U . For an arbitrary quasi-projective variety X, take a desingularisation π : X˜ → X and equip CH s X with the ﬁltration induced by the surjective map π∗ : CH s X˜ → CH s X. By an argument using Proposition 6.4(1), one sees that the ﬁltration is well deﬁned, ˜ independent of the choice of X. Proposition 6.5. (1) If X, Y are quasi-projective varieties and f : X → Y is proper (resp., proper surjective), then f∗ : CH s X → CH s Y respects F • (resp., is strictly compatible with F • ).

MOTIVIC DECOMPOSITION AND INTERSECTION CHOW GROUPS

509

(2) If j : U → X is an open immersion of quasi-projective varieties, j ∗ : CH s X → CH s U is strictly compatible with F • . Proof. To prove (1), take desingularisations π : X˜ → X, π : Y˜ → Y so that f extends to a map f˜ : X˜ → Y˜ . Then π ◦ f˜ = f ◦π and one has a commutative diagram CHO s X

f∗

/ CH s Y O π∗

π∗

CH s X˜

f˜∗

/ CH Y˜ s

where the arrows are proper pushforwards. Since the map f˜∗ respects F • (resp., is strictly compatible with F • ) by Proposition 6.4(1), and so are the vertical surjective maps by deﬁnition, f∗ respects F • (resp., is strictly compatible with F • ). The proof of (2) is similar. Proof of Theorem 6.1. The properties (i) and (ii) have already been veriﬁed, and we started with the ﬁltration satisfying (iii) and (viii). (vii) This is veriﬁed by reducing ﬁrst to the case where X and Y are both smooth, and then to the case where they are smooth projective. (vi) Follows from strictness of the ﬁltration under proper maps and open immersions (i) and (ii), Lemma 6.6, and compatibility of the ﬁltration under action of correspondences on smooth projective varieties (i), (v), and (vii). This concludes the proof. Lemma 6.6. Let i : T → S be a regular embedding of codimension d, let /X

Y g

T

i

/S

f

be a ﬁbre square, and let i ! : CH r X −→ CH r−d Y be the associated reﬁned Gysin map [F1, Chapter 6]. There are: (1) smooth varieties U , V , and proper surjective maps p : U → X, q : V → Y , (2) a commutative diagram V q

Y

/U p

/ X,

510

CORTI AND HANAMURA

(3) smooth compactiﬁcations U ⊂ U , V ⊂ V and a correspondence , ∈ CH • V × U , such that for a cycle α ∈ CH r U i ! p∗ (α | U ) = q∗ (, ∗ α) | V . Proof. Step 1. In this step we reduce the problem to the case where Y = Y1 Y2 → X, X is smooth, Y1 → X is a normal crossing divisor, and Y2 → X is the inclusion of a bunch of connected components. Indeed, let δ : X → X be a resolution of singularities such that Y = Y ×X X = Y1 Y2 → X is as above. We have a commutative diagram of ﬁbre squares / X

Y ε

δ

Y

/X

g

T

i

/S

f

By compatibility of reﬁned Gysin maps with proper pushforward (see [F1, Chapter 6]), we have i ! δ∗ = ε∗ i ! . Therefore, it is enough to prove the result for Y → X . Step 2. We now assume that j : Y → X is a normal crossing divisor inside a smooth quasi-projective variety. Let E = g ∗ NT S/NY X be the excess bundle (by assumption, it has rank d − 1). The excess intersection formula reads i ! α = cd−1 E ∩ j ! α. If now U = X, U ⊂ U is a smooth projective compactiﬁcation of U such that the closure Y of Y in U is a normal crossing divisor, then denoting cd−1 E as any extension of cd−1 E to U , ν : V → Y as the normalisation, V = ν −1 Y as the normalisation of Y , and h : V → U as the composition V → Y → U , we have for a class α ∈ CH r U j ! (α | U ) = (ν | V )∗ (h∗ α) | V and

i ! (α | U ) = (ν | V )∗ cd−1 E ∩ h∗ α | V .

To conclude the proof now just take a correspondence , ∈ CH • V × U such that

cd−1 E ∩ h∗ _ = , ∗ _.

MOTIVIC DECOMPOSITION AND INTERSECTION CHOW GROUPS

511

Step 3. Finally, we treat the case where j : Y → X is the inclusion of a bunch of connected components. This is quite a bit easier than step 2; here i ! α = cd (E) ∪ j ∗ α. We let V = Y , U = X, U ⊂ U a smooth compactiﬁcation and cd E any extension of cd E to U , j : V → U the corresponding compactiﬁcation of V → U . , works if ∗ ,∗ _ = cd E ∪ j _. 6.2. Proof of Theorem 6.3. It relies on the following proposition. Proposition 6.7. Assume Grothendieck’s standard conjectures. For a quasiprojective variety X, let BM BM Halg,2s X = Im cl : CH s X → H2s X , which is a Q-vector space. (1) If j : U → X is an open immersion of quasi-projective varieties and if i : Z = X − U → X is the closed immersion of the complement of U , the exact sequence j∗

i∗

BM BM BM H2s Z −→ H2s X −→ H2s U

induces the following exact sequence on algebraic parts: j∗

i∗

BM BM BM Halg,2s Z −→ Halg,2s X −→ Halg,2s U −→ 0.

(2) Let Z

i

p

q

Z

/ X

i

/X

be a Cartesian square of quasi-projective varieties such that the horizontal maps are closed immersions, the vertical maps are proper surjective, and p induces an ∼ = isomorphism X − Z − → X − Z. Then the exact sequence i∗ ,q∗

p∗ −i∗

BM BM BM BM H2s Z −−−→ H2s X ⊕ H2s Z −−−−→ H2s X

induces an exact sequence i∗ ,q∗

p∗ −i∗

BM BM BM BM Halg,2s Z −−−→ Halg,2s X ⊕ Halg,2s Z −−−→ Halg,2s X −→ 0.

Proof. (1) We recall that HiBM X for X quasi-projective has a weight ﬁltration W• , the weights are greater than or equal to −i, and the maps i∗ , j ∗ are strictly compatible with the weight ﬁltrations.

512

CORTI AND HANAMURA

The ﬁrst exact sequence induces, upon taking Gr W −2s , the exact sequence BM BM BM W−2s H2s Z −→ W−2s H2s X −→ W−2s H2s U.

This may be viewed as an exact sequence in the category of Grothendieck motives ᏹ. More speciﬁcally in the weight spectral sequence pq W E1

BM $⇒ H−p−q X, pq

BM X, each E which induces the weight ﬁltration of H−p−q W 1 is the cohomology of a smooth projective variety and can be regarded as a Grothendieck motive denoted pq p,q are morphisms of Grothendieck motives. Deﬁne W E 1 ; the differentials d1 p,q

p,q

BM Gr W q H −p−q X = W Z 1 /W B 1 .

Taking the cohomological realisation H ∗ , we have BM W BM H ∗ Gr W q H −p−q X = Gr q H−p−q X because H ∗ is exact (by the standard conjectures, ᏹ is a semisimple abelian category) and the weight spectral sequence above degenerates at E2 . Since ᏹ is semisimple, the functor Homᏹ (pt(−s), −) : ᏹ → VecQ is also exact, and the above exact sequence induces an exact sequence under this functor. It is the desired exact sequence, except possibly at the end, because, for a quasi-projective variety X, BM Homᏹ pt(−s), W−2s H BM 2s X = Halg,2s X. The surjectivity at the end is obvious since j ∗ : CH s X → CH s U is surjective. (2) One has a commutative diagram BM Z H2s

i∗

q∗

BM Z H2s

/ H BM X 2s

i∗

p∗

/ H BM X 2s

/ H BM (X \ Z ) 2s p∗

/ H BM (X \ Z) 2s

where the rows are exact and the third vertical map is an isomorphism. This shows that the sequence of Borel-Moore homology groups is exact as stated. Applying (1) to the closed immersions i and i, respectively, one obtains a similar commutative diagram with exact rows BM Z Halg,2s

i∗

q∗

BM Z Halg,2s

/ H BM X alg,2s

i∗

p∗

/ H BM X alg,2s

/ H BM X \ Z alg,2s

p∗

/ H BM X \ Z alg,2s

MOTIVIC DECOMPOSITION AND INTERSECTION CHOW GROUPS

513

(the third vertical map is an isomorphism); the last exact sequence follows from this. Proof of Theorem 6.3. Part (x) is veriﬁed by reducing to the smooth projective case where it holds true; see Theorem 4.16. To show (ix), ﬁrst assume X is smooth quasi-projective. Take a compactiﬁcation j : X → X, let Z = X − X, and consider the commutative diagram, where the rows are exact by Proposition 6.7(1) BM Z Halg,2s O

CH s Z

i∗

i∗

/ H BM X alg,2s O / CH X s

j∗

j∗

/ H BM X alg,2s O

/0

/ CH s X

/0

where the vertical arrows are the cycle class maps. Since F 1 CH s X surjects to F 1 CH s X, the inclusion F 1 CH s X ⊂ CH hom,s X is obvious. The other inclusion follows from the above diagram. The proof in the general case is reduced to the smooth case by taking desingularisations and using Proposition 6.7(2). 6.3. Applications. As an application, let S be a quasi-projective variety, let X be a smooth quasi-projective equidimensional variety, and let f : X → S be a projective map. We recall that the group CH dim X X ×S X is a ring where the multiplication is deﬁned by v • u = δ ! (v × u); δ ! is the reﬁned Gysin map related to the diagonal embedding X → X × X. Hence, we have the following proposition. Proposition 6.8. The multiplication of the ring CH dim X X ×S X respects F • . Namely, if u ∈ F ν CH dim X X ×S X and v ∈ F µ CH dim X X ×S X, then v • u ∈ F ν+µ CH dim X X ×S X. In particular, if we assume the conjectures of Murre and of Grothendieck, the ideal F 1 CH dim X X ×S X = CH hom,dim X X ×S X is a nilpotent ideal. Theorem 6.9. Assume that the base ﬁeld is C or the closure of a ﬁnite ﬁeld. In addition, assume Grothendieck’s and Murre’s conjectures. If f : X → S is as above, then the surjective map ρ : CH dim X X ×S X −→ EndᏹS p Rf∗ QX has a nilpotent kernel.

514

CORTI AND HANAMURA

Proof. We claim ﬁrst that the map of rings H2BM dim X X ×S X −→ Endgr ᏼervS

p

R i f∗ QX [−i]

has a nilpotent kernel. If one chooses a decomposition p i R f∗ QX [−i] Rf∗ QX = in Ᏸbcc (S), an endomorphism u of Rf∗ QX may be represented by a matrix according to the decomposition. The matrix is upper-triangular since the maps p R i f∗ QX [−i] → p R j f Q [−j ] are zero if i < j . ∗ X Consider now an element u ∈ H2BM dim X X ×S X = EndᏰbcc S Rf∗ QX , which maps to zero in Endgr ᏼervS p R i f∗ QX [−i]. Then the matrix representing u as above is strictly upper-triangular. Hence, there exists N such that uN = 0 as an endomorphism of Rf∗ QX . The kernel of the homomorphism CH dim X X ×S X −→ H2BM dim X X ×S X equals CH hom,dim X (X ×S X). Under the conjectures, it is a nilpotent ideal, Proposition 6.8. Hence, the kernel of ρ is also nilpotent. Corollary Any set of orthogonal projectors {π i } of EndᏹS p Rf∗ QX , such 6.10. i that IX = π , canbe lifted to a set of orthogonal projectors {Hi } of CH dim X (X×S X) such that IX = Hi . Proof. More generally, the following proposition holds (cf. [Ja2, Lemma 5.4]). Proposition 6.11. Let φ : A → B be a surjective homomorphism of not necessarily commutative rings with nilpotent kernel. Then any orthogonal set {p1 , . . . , pm } of idempotents of B (i.e., pi pj = δi,j pi ) adding up to 1B can be lifted to an orthogonal set of idempotents of A adding up to 1A . 7. Decomposition in CH ᏹ(S) Convention 7.1. Throughout this section we assume that k = C, or the closure of a ﬁnite ﬁeld, and that desingularisations of varieties over k exist. The aim is to prove the following decomposition theorem in CH ᏹ(S). Theorem 7.2. Assume the standard conjectures and Murre’s conjecture. Let X be a smooth variety, and let f : X → S be a projective morphism. (1) There is a noncanonical direct sum decomposition in CH ᏹ(S) CRf∗ QX ∼ ᏯᏵᏯVkm [k − m], = m,k

MOTIVIC DECOMPOSITION AND INTERSECTION CHOW GROUPS

515

where Vk is a local system on a Zariski locally closed subvariety Tk ⊂ S and

ᏯᏵᏯVkm [k − m] denotes an object in CH ᏹ(S), together with a given isomorphism in Ᏸbcc (S) ∼ =

real ᏯᏵᏯVkm [k − m] −−→ ᏵᏯVkm [k − m].

(2) The monomorphisms p

τ≤m CRf∗ QX =

i≤m k

ᏯᏵᏯVki [k − i] −→ CRf∗ QX

(where the ﬁrst equality is the deﬁnition of p τ≤m CRf∗ QX ) are speciﬁed up to canonical isomorphism. In particular so are the “subquotients” p CR m f∗ QX [−m] = ᏯᏵᏯVkm [k − m] k

(where the ﬁrst equality is the deﬁnition of p CR m f∗ QX [−m]) speciﬁed up to canonical isomorphism. (3) The decompositions p CR m f∗ QX [−m] = ᏯᏵᏯVkm [k − m] k

are uniquely speciﬁed. Proof. Using Theorem 5.14, choose a decomposition in ᏹ(S), p Rf∗ QX ∼ ᏵᏯVkm [k − m], = m,k

and let

πkm ∈ EndᏹS X

be the projector onto ᏵᏯVkm [k − m]. By Corollary 6.10, the πkm lift to an orthogonal set of projectors Hm k ∈ End CH ᏹ S X. Now set

ᏯᏵᏯVkm [k − m] = X, Hm k .

This proves the existence of the sought-for decomposition. The uniqueness statement (2) is more subtle, and we deduce it from the corresponding uniqueness statements for the decomposition in ᏹ(S). According to Deﬁnition 7.3 below, p τ≤m CRf∗ QX (resp., p τ>m CRf∗ QX ) has cohomological degree less than or equal to m (resp., greater than m). Then, by Theorem 7.4(2), HomCH ᏹ S pτ≤m CRf∗ QX , pτ>m CRf∗ QX = 0 independently of the choice of the liftings Hm k . This implies (2).

516

CORTI AND HANAMURA

Finally, to prove the uniqueness statement in (3), note that by the way it has been constructed p CR m f∗ QX [−m] has cohomological degree exactly m; hence, the statement follows from the decomposition Theorem 5.14 in ᏹ(S), and Theorem 7.4(2). The rest of this section is devoted to the proof of Theorem 7.4 below. Deﬁnition 7.3. A Chow motive (X, P ) over S has cohomological degree less than or equal to m (resp., greater than m) if p

Ᏼi real(X, P ) = 0

for all i > m (resp., i ≤ m). Finally, (X, P ) has degree exactly m if it has degree greater than or equal to m and less than or equal to m. Theorem 7.4. Assume the standard and Murre’s conjectures. (1) If (X, P ) has cohomological degree less than or equal to m and (Y, Q) has cohomological degree greater than m, then HomCH ᏹ S (X, P ), (Y, Q) = 0. (2) If (X, P ) and (Y, Q) have degrees exactly m, then HomCH ᏹ S (X, P ), (Y, Q) = HomᏹS (X, P ), (Y, Q) . Proof. Let Z = X ×S Y . We consider the operators CH s Z α −→ Kα = Q • α • P ∈ CH s Z, HiBM Z a −→ ψa = [Q] • a • [P ] ∈ HiBM Z. Note that K 2 = K and ψ 2 = ψ; that is, both operators are projectors. To prove (1) and (2), it is enough to show that if ψ = 0, then K = 0. Because ψ = 0, most of the argument is spent showing that KF ν ⊂ F ν+1 for all ν. Step 1. Let T ⊂ Z be the singular set. Make a diagram RO

/W O

R

/W

T

/Z

p

where we have the following: (a) W is smooth and projective and R ⊂ W is a smooth normal crossing divisor;

MOTIVIC DECOMPOSITION AND INTERSECTION CHOW GROUPS

517

(b) (W, R) = (W − B, R − B) for some divisor B ⊂ R; (c) p : W → Z is a resolution of singularities and R = p −1 T ⊂ W . By Corollary 7.9, K is a class Ꮿ operator; hence by Lemma 7.6 there is a correspondence , ∈ CH • W × W , (W is not necessarily equidimensional) such that K p∗ (α | W ) = p∗ (,∗ α) | W

(∗)

for all α ∈ CH • Z, respectively, ψ p∗ (a | W ) = p∗ (,∗ a) | W

(∗∗)

for all a ∈ H•BM Z. Step 2. We have an exact sequence HiBM R −→ HiBM R ⊕ HiBM W −→ HiBM W. If j : W → W , j : R → R are the natural inclusions, the exact sequence arises, by means of a familiar construction, from the following morphism of distinguished triangles: DB

/D R

/ Rj DR ∗

DB

/ DW

/ Rj∗ DW

[1]

/

[1]

/

Step 3. This is the crucial step. We show that γ = cl , ∈ N 1 H•BM W × W lies in the codimension 1 piece of the coniveau ﬁltration. Let i : R → W and i : R → W be the inclusions. First of all, from the standard exact sequence HiBM R −→ HiBM T ⊕ HiBM W −→ HiBM Z we deduce that if p∗ (a | W ) = 0 for some a ∈ HiBM W , then

a | W = i∗ a

for some a ∈ HiBM R. Then, from the exact sequence in step 2, a = i∗ a

518

CORTI AND HANAMURA

for some a ∈ HiBM R. Now, for any a ∈ HiBM W p∗ (,∗ a) | W = ψ p∗ (a | W ) = 0 by assumption. So, because of what has been said ,∗ a ∈ Im H•BM R, which ﬁnishes step 3. Step 4. Let , = , ◦ , ◦ · · · (many times). Since K 2 = K and ψ 2 = ψ, equations (∗) and (∗∗) are still satisﬁed with , in place of ,. In addition we have γ = cl , ∈ N many times H• W × W = 0. Then ,∗ F ν CH • W ⊂ F ν+1 CH • W for all ν and, by the strictness properties of the F -ﬁltration, KF ν CH • Z ⊂ F ν+1 CH • Z for all ν. Step 5. We are assuming the standard conjectures. Therefore, F ν CH • Z = 0 for ν large. This implies that K = 0, which concludes the proof. The rest of the paper is devoted to settling the remaining technical issues in the proof of Theorem 7.4. Deﬁnition 7.5. (1) Let X be a quasi-projective variety. For the purpose of the following discussion, a smooth cover of X is a diagram /U

U p

X, p

sometimes simply denoted U ⊃ U → X, where U is a smooth projective variety, U ⊂ U is an open subvariety with smooth normal crossing boundary divisor U \ U , and p : U → X is a projective morphism. It is a consequence of our assumptions on existence of resolution of singularities, that smooth covers always exist. (2) Let X, Y be quasi-projective varieties. We say that an operator K : CH • X −→ CH •−c Y p

is of class Ꮿ (Ꮿ stands for “correspondence”) if there are smooth covers U ⊃ U → X q of X and V ⊃ V → Y of Y , and a correspondence , ∈ CH • U × V such that K p∗ (α | U ) = q∗ (,∗ α) | V for all α ∈ CH • U . In this case, we say that , induces K.

519

MOTIVIC DECOMPOSITION AND INTERSECTION CHOW GROUPS

The following is the basic point. p

Lemma 7.6. Let K : CH • X → CH •−c Y be of class Ꮿ, and let U ⊃ U → X, q V ⊃ V → Y be arbitrary smooth covers of X, Y . Then there exists a correspondence , ∈ CH • U × V inducing K. p1

Proof. By assumption, there are some smooth covers U 1 ⊃ U1 −→ X of X, q1 V 1 ⊃ V1 −→ Y of Y , and some correspondence ,1 ∈ CH • U 1 × V 1 such that K p1∗ (α | U1 ) = q1∗ (,1∗ α) | V1 ) for all α ∈ CH • U 1 . Now, any two smooth covers can be housed under a third: U2 ⊂ UJ2 JJJ s s s JJJ s s s JJJ s s s ys % U1 ⊂ U 1 U ⊂U So, in the end, we may assume that there is either a morphism U1 ⊂ U 1 → U ⊂ U , or the other way around, and similarly for V . Case 1. Assume that there is a morphism π : U ⊂ U → U1 ⊂ U 1 , and let , = ,1 ◦ ,π ∈ CH • U × V 1 . Then , induces K, since K p∗ (α | U ) = K p1∗ π∗ (α | U ) = K p1∗ (π∗ α) | U1 = q1∗ (,1∗ π∗ α) | V1 = q1∗ (,∗ α) | V1 . Case 2. Assume now that there is a morphism π : U1 ⊂ U 1 → U ⊂ U . Let i : W → U 1 be a smooth projective subvariety generically ﬁnite over U W

i

/U 1 U

π

and let us agree that d be the generic degree of π ◦i. Let us ﬁx ourselves a correspondence , ∈ CH • U × V 1 with the property that ,∗ =

1 ,1∗ i∗ i ∗ π ∗ . d

520

CORTI AND HANAMURA

Then , induces K, since 1 1 K p∗ (α | U ) = K p∗ π∗ i∗ i ∗ π ∗ α | U = K p1∗ i∗ i ∗ π ∗ α | U d d 1 = q1∗ ,1∗ i∗ i ∗ π ∗ α | V1 = q1∗ (,∗ α) | V1 . d To summarise, in both cases we were able to ﬁnd a , ∈ CH • U × V 1 inducing K. Working now the V s in a similar fashion, we can also ﬁnd , ∈ U × V inducing K, that is, prove the lemma. Lemma 7.7. Let K : CH • X −→ CH •−c Y and P : CH • Y → CH •−e Z be of class

Ꮿ. Then, the composition P ◦ K : CH • X → CH •−c−e Z is also of class Ꮿ.

Proof. Obvious. Lemma 7.8. (1) Let f : X → Y be a proper map. Then f∗ : CH i X → CH i Y is of class Ꮿ. (2) Let i : Y → X be a regular embedding of codimension c and / X

Y Y

i

/X

be a ﬁbre square. The reﬁned Gysin map i ! : CH • X −→ CH •−e Y is of class Ꮿ. Proof. (1) is obvious and (2) is Lemma 6.6. Corollary 7.9. The operator K in the proof of Theorem 7.4 is of class Ꮿ. Proof. Indeed, K is a composition of proper pushforward and reﬁned Gysin maps. All these are of class Ꮿ, by Lemma 7.8, and so their composition is, by Lemma 7.7.

References [BFM] [Be] [BBD]

P. Baum, W. Fulton, and R. MacPherson, Riemann-Roch for singular varieties, Inst. Hautes Études Sci. Publ. Math. 45 (1975), 101–145. A. Beauville, Variétés de Prym et jacobiennes intermédiaires, Ann. Sci. École Norm. Sup. (4) 10 (1977), 309–391. A. Beilinson, J. Bernstein, and P. Deligne, “Faisceaux pervers” in Analyse et topologie sur les espaces singuliers (Luminy, 1981), Vol. 1, Astérisque 100, Soc. Math. France, Montrouge, 1982, 5–171.

MOTIVIC DECOMPOSITION AND INTERSECTION CHOW GROUPS [Bo] [DJ1] [DJ2] [D] [De] [DM] [F1] [F2] [FM] [GoMu] [GoMa] [Gr]

[Ha1] [Ha2] [Ha3] [Ha4] [Ha5] [Ha6] [Iv] [Ja1] [Ja2] [KS] [Kl1] [Kl2] [La] [Ma]

[Mu1] [Mu2]

521

A. Borel, “Sheaf theoretic intersection cohomology” in Intersection Cohomology (Bern, 1983), Progr. Math. 50, Birkhäuser, Boston, 1984, 47–182. A. J. de Jong, Smoothness, semi-stability and alterations, Inst. Hautes Études Sci. Publ. Math. 83 (1996), 51–93. , Families of curves and alterations, Ann. Inst. Fourier (Grenoble) 47 (1997), 599–621. P. Deligne, La conjecture de Weil, II, Inst. Hautes Études Sci. Publ. Math. 52 (1980), 137–252. M. Demazure, Motifs des variétés algébriques, Lecture Notes in Math. 180 (1971), 19–38, Séminaire Bourbaki 1969/70, exp. no. 365. C. Deninger and J. P. Murre, Motivic decomposition of abelian schemes and the Fourier transform, J. Reine Angew. Math. 422 (1991), 201–219. W. Fulton, Intersection Theory, Ergeb. Math. Grenzgeb. (3) 2, Springer, Berlin, 1984. , Young Tableaux, London Math. Soc. Stud. Texts 35, Cambridge Univ. Press, Cambridge, 1997. W. Fulton and R. MacPherson, Categorical framework for the study of singular spaces, Mem. Amer. Math. Soc. 31 (1981), no. 243. B. Gordon and J. P. Murre, Chow motives of elliptic modular surfaces and threefolds, J. Reine Angew. Math. 514 (1999), 145–164. M. Goresky and R. MacPherson, Intersection homology, II, Invent. Math. 72 (1983), 77–129. A. Grothendieck, “Standard conjectures on algebraic cycles” in Algebraic Geometry (Bombay, 1968), Tata Inst. Fund. Res. Stud. Math. 4, Oxford Univ. Press, London, 1969, 193–199. M. Hanamura, Mixed motives and algebraic cycles, I, Math. Res. Lett. 2 (1995), 811–821. , Mixed motives and algebraic cycles, III, Math. Res. Lett. 6 (1999), 61–82. , Homological and cohomological motives of algebraic varieties, preprint, 1995. , Mixed motives and algebraic cycles, II, preprint, 1996. , Chow cohomology groups of algebraic varieties, in preparation. , Mixed motivic sheaves, in preparation. B. Iversen, Cohomology of Sheaves, Universitext, Springer, Berlin, 1986. U. Jannsen, Motives, numerical equivalence, and semi-simplicity, Invent. Math. 107 (1992), 447–452. , “Motivic sheaves and ﬁltrations on Chow groups” in Motives (Seattle, 1991), Proc. Sympos. Pure Math. 55, Part 1, Amer. Math. Soc., Providence, 1994, 245–302. M. Kashiwara and P. Schapira, Sheaves on Manifolds, Grundlehren Math. Wiss. 292, Springer, Berlin, 1990. S. Kleiman, “Algebraic cycles and the Weil conjectures” in Dix exposés sur la cohomologie des schémas, North-Holland, Amsterdam, 1968, 359–386. , “The standard conjectures” in Motives (Seattle, 1991), Proc. Sympos. Pure Math. 55, Part 1, Amer. Math. Soc., Providence, 1994, 3–20. G. Laumon, “Homologie étale” in Séminaire de géométrie analytique (Paris, 1974/75), Astérisque 36–37, Soc. Math. France, Montrouge, 1976, 163–188. Yu. I. Manin, Correspondences, motifs and monoidal transformations (in Russian), Mat. Sb. (N.S.) 77 (119), 1968, 475–507; English trans. in Math. USSR-Sb. 6 (1968), 439–470. J. P. Murre, On a conjectural ﬁltration on the Chow groups of an algebraic variety, I, Indag. Math. (N.S.) 4 (1993), 177–188. , On a conjectural ﬁltration on the Chow groups of an algebraic variety, II, Indag. Math. (N.S.) 4 (1993), 189–201.

522 [SaM] [SaS] [Sc1] [Sc2] [Sm] [To] [Ve]

CORTI AND HANAMURA M. Saito, Mixed Hodge modules, Publ. Res. Inst. Math. Sci. 26 (1990), 221–333. S. Saito, Motives and ﬁltrations on Chow groups, Invent. Math. 125 (1996), 149–196. A. J. Scholl, Motives for modular forms, Invent. Math. 100 (1990), 419–430. , “Classical motives” in Motives (Seattle, 1991), Proc. Sympos. Pure Math. 55, Part 1, Amer. Math. Soc., Providence, 1994, 163–187. O. N. Smirnov, Graded associative algebras and Grothendieck standard conjectures, Invent. Math. 128 (1997), 201–206. B. Totaro, Chow groups, Chow cohomology, and linear varieties, to appear in J. Algebraic Geom., available from http://www.dpmms.cam.ac.uk/ ˜ bt 219/. J.-L. Verdier, “Le théorème de Riemann-Roch pour les intersections complètes” in Séminaire de géométrie analytique (Paris, 1974/75), Astérisque 36–37, Soc. Math. France, Montrouge, 1976, 189–228.

Corti: Department of Pure Mathematics and Mathematical Statistics, University of Cambridge, Centre for Mathematical Sciences, Wilberforce Road, Cambridge CB3 0WB, United Kingdom; [email protected] Hanamura: Department of Mathematics, Kyushu University, Hakozaki, Fukuoka 812-12, Japan; [email protected]

Vol. 103, No. 3

DUKE MATHEMATICAL JOURNAL

© 2000

ON THE CHERN-GAUSS-BONNET INTEGRAL FOR CONFORMAL METRICS ON R 4 SUN-YUNG A. CHANG, JIE QING, and PAUL C. YANG

1. Introduction. An important landmark in the theory of surfaces is the introduction of the notion of complete open surface by Hopf and Rinow [HR]. Subsequently, Cohn-Vossen [CV] studied the Gauss-Bonnet integral for such a surface M with analytic metrics. Cohn-Vossen also showed that if the Gaussian curvature K is absolutely integrable, then (1.1) K dvM ≤ 2πχ(M), M

where χ (M) is the Euler number of M. Later, Huber [Hu] extended this inequality to metrics with much weaker regularity. More importantly, he proved that such a surface M is conformally equivalent to a closed surface with ﬁnitely many punctures. The deﬁcit in formula (1.1) has an interpretation as an isoperimetric constant. On a complete and open surface with Gaussian curvature absolutely integrable, one may represent each end conformally as R 2 \ K for some compact set K. We consider the isoperimetric ratio L2 (r) ν = lim , r→∞ 4πA(r) where L(r) is the length of the boundary circle ∂Br = {|x| = r}, and A(r) is the area of the annular region B(r) \ K. For a fairly large class of complete surfaces, Finn [F] showed that 1 χ (M) − (1.2) K dvM = νj , 2π M where the sum is taken over each end of M. For more recent development on the subject of complete surface of ﬁnite total curvature the reader is referred to the work of Li and Tam [LT]. Except for the work of Cheeger and Gromov [CG] on the Chern-Gauss-Bonnet formula for manifolds with bounded geometry, there is very little known about the situation in higher dimensions. Received 10 February 1999. 2000 Mathematics Subject Classiﬁcation. Primary 53A30, 35J35; Secondary 53A55, 25J6. Chang’s work partially supported by National Science Foundation grant number DMS-9706864. Qing’s work partially supported by National Science Foundation grant numbers DMS-9803399 and DMS-9706864. Yang’s work partially supported by National Science Foundation grant number DMS-9706507. 523

524

CHANG, QING, AND YANG

In this paper we consider a generalization of (1.2) in dimension 4. For conformal geometry in dimension 4, the Paneitz operator 2 P = 2 + δ RI − 2 Ric d 3 (where δ denotes the divergence, d is the differential, R is the scalar curvature, and Ric is the Ricci tensor) plays the same role as the Laplacian in dimension 2 (cf. [P], [BCY], and [CQ], for example). The Paneitz operator enjoys the following invariance property under conformal change of metric g = e2w g0 : the Paneitz operator transforms by Pg = e−4w Pg0 . For the conformal metric g = e2w g0 , the Paneitz operator applied to the conformal factor w calculates a fourth-order curvature invariant Q: (1.3) where (1.4)

Pg0 w + 2Qg0 = 2Qg e4w , 1 2 1 2 − R + R − 3|E| Q= 12 4

and E is the traceless Ricci tensor. The Q curvature invariant is related to the ChernGauss-Bonnet integral in dimension 4; we have 1 |W |2 (1.5) + Q dV , χ (M) = 8 4π 2 M where W is the Weyl tensor and M is a compact, closed 4-manifold. More generally, when the manifold has a boundary, Chang and Qing [CQ] have deﬁned a boundary operator P3 and its associated boundary curvature invariant T , as follows: (1.6)

P3 w + Tg0 = Tg e3w .

Then the Chern-Gauss-Bonnet integral is supplemented by 1 1 |W |2 χ (M) = + Q dV + (1.7) (L + T ) d , 8 4π 2 M 4π 2 ∂M where Ld is a pointwise conformal invariant. Our motivations are twofold. First, we are searching for the analogue of the previously mentioned result of Finn. Second, we would like to ﬁnd a geometric interpretation of the fourth-order curvature invariant Q (which we call Paneitz curvature, according to the relation in equation (1.3)). We are fortunate to ﬁnd both in one formula. Here we take the initial analytic step and study complete conformal metrics e2w |dx|2 on R 4 . For interesting applications in global geometry it would be desirable to establish some criteria for conformal compactiﬁcation, at least for locally conformally ﬂat 4-manifolds. We will continue to study this question in a forthcoming article.

CHERN-GAUSS-BONNET INTEGRAL FOR CONFORMAL METRICS

525

We now state our results. By solving an ODE, we ﬁrst verify the following theorem. Theorem 1.1. Suppose that w is a radial function on R 4 and e2w |dx|2 is a complete metric with R 4 |Q|e4w dx < ∞. And suppose that its scalar curvature is nonnegative at inﬁnity. Then (1.8) Qe4w dx ≤ 4π 2 χ R 4 = 4π 2 , R4

and (1.9)

1 1− 2 4π

R4

Qe

4w

4/3 vol ∂Br (0) dx = lim 1/3 . r→∞ 4 2π 2 vol Br (0)

Remark 1.2. (1) By “nonnegativity of scalar curvature at inﬁnity” we mean that the scalar curvature R(x) is nonnegative when |x| is sufﬁciently large. It is easy to see that without the additional condition on the scalar curvature Theorem 1.1 does not hold. For instance, suppose that e2v |dx|2 is the standard metric for the 4-sphere 2 S 4 . Then the metric e2(v+r ) |dx|2 is a complete metric on R 4 ; but 1 1 1 4(v+r 2 ) 2 2 Q e dx = v + r dx = 2 v 2 4π 2 R 4 v+r 4π 2 R 4 4π 2 R 4 3|S 4 | 1 4v Q e dx = = 2 > 1. = 4 4π 2 R 4 S 4π 2 (2) For domains in R n for n > 2, it turns out that there is more than one isoperimetric constant to be considered. These isoperimetric constants can be deﬁned as the ratio of pairs of “mixed volume” (see [BZ], [T]) of the domain. It turns out that in our setting, after suitable normalization, all these isoperimetric ratios tend to the same limit as in (1.9). We have established a generalized form of Theorem 1.1 (and also Theorems 1.3 and 1.4), with respect to all these isoperimetric ratios. In his attempt to prove (1.2) for any complete open surface with absolutely integrable Gaussian curvature, Finn [F] has found an interesting intermediate class of metrics for which (1.2) holds. That is the class of normal metrics, which are the conformal metric e2w |dx|2 , where w is a constant plus a potential (see Deﬁnition 3.1). We found the same phenomenon in 4-dimension. Theorem 1.3. Suppose that the metric e2w |dx|2 is a complete and normal metric on R 4 ; then both (1.8) and (1.9) hold. The notion of normal metrics arises naturally. In the recent work of several authors (see [CLi], [CL], [CY], [L], and [Xu]), they have all pointed out that it is essential to verify that a metric is normal in order to control the asymptotic behavior for metrics at inﬁnity. Fortunately, we are able to show that nonnegativity of the scalar curvature (at inﬁnity) ensures that the metric is normal.

526

CHANG, QING, AND YANG

Theorem 1.4. Suppose that the Paneitz curvature of a metric e2w |dx|2 on R 4 is absolutely integrable and suppose that its scalar curvature is nonnegative at inﬁnity; then the metric is normal. We believe that one also can verify a metric to be normal if one assumes some decay conditions of the Paneitz curvature such as those given in [CL]. Meanwhile, Theorem 1.4 sheds some light on estimating asymptotic behavior for a metric e2w |dx|2 on R 4 under appropriate geometric conditions. Finally, combining Theorems 1.3 and 1.4 we have our main theorem. Main theorem. Suppose that e2w |dx|2 on R 4 is a complete metric with its Paneitz curvature Q absolutely integrable, and suppose that its scalar curvature is nonnegative at inﬁnity. Then both (1.8) and (1.9) hold. The paper is organized in the following manner. In Section 2, we will study the radially symmetric case. Our approach is to ﬁrst ﬁnd a particular and good solution to the ODE, then to control the asymptotic behavior of the given metric. In Section 3, we prove Theorem 1.3. Our strategy is to reduce the problem to a radially symmetric one, then to apply the result in Section 2. In Section 4, we prove Theorem 1.4 and hence our main theorem. Acknowledgments. Part of this work was done while the second author was visiting UCLA and Princeton University and while the third author was visiting Princeton University. They would like to thank both institutions for their hospitality. 2. Symmetric case. Consider a conformal metric (R 4 , e2w |dx|2 ). We assume that the Paneitz curvature Q of the metric 2 w = 2Qe4w

(2.1) satisﬁes

on R 4

(2.2)

R4

|Q|e4w dx < ∞.

For convenience, we often use cylindrical coordinates |x| = r = et to rewrite equation (2.1) as 2 2 ∂ ∂ ∂ ∂ (w + t) = 2Qe4(w+t) ; (2.3) − 2 + 2 ∂t ∂t ∂t 2 ∂t 2 that is, (2.4)

∂ ∂2 −2 2 ∂t ∂t

when we denote w + t by v.

∂2 ∂ v = 2Qe4v , +2 ∂t ∂t 2

−∞ < t < ∞,

CHERN-GAUSS-BONNET INTEGRAL FOR CONFORMAL METRICS

527

In contrast to the situation on surfaces, where there is one isoperimetric ratio to consider, there exists a family of isoperimetric inequalities for mixed volumes, when the dimension of the manifold is greater than 2 (cf. [BZ], [T]). In the case when ' is a convex domain in R n , following Trudinger [T] we deﬁne, for each 1 ≤ m < n, the mixed volume 1 Hn−1−m [∂'] dH n−1 , Vm (') = n−1 n m ∂' where Hl [∂'] is the lth symmetric form of the principle curvatures of ∂' and where H n−1 is the n − 1 dimensional Hausdorff measure. For m = n, deﬁne Vn (') = 1/m volume of '. Denote ζm (') = Vm ('); then the isoperimetric inequalities for mixed volumes ζm ≤ ζ l

(2.5)

hold for all 1 ≤ l ≤ m ≤ n. Motivated by these inequalities, we may similarly deﬁne, in our case (when n = 4), t 3 V4 = vol({s ≤ t}) = |S | (2.6) e4v ds, V3 =

(2.7) (2.8) (2.9)

s=t

e3v ds,

1 H1 e3v ds = |S 3 |v (t)e2v(t) , 4 s=t 2 2 1 1 1 H1 − tr L2 e3v ds = |S 3 | v (t) ev(t) . H2 e3v ds = V1 = 12 s=t 24 s=t 4 V2 =

1 12

1 4

−∞

We also deﬁne the following isoperimetric ratios: 4/3

V3 (t) , 1/3 (1/2)π 2 V4 (t)

C3,4 (t) = (2.10)

V2 (t) 1/3 2/3 , V3 (t) (1/2)π 2

C2,3 (t) =

2/3

V1 (t) 1/3 1/3 , (1/2)π 2 V2 (t)

C1,2 (t) = and

1/3

2/3

1/4

3/4

1/9

2/9

C2,4 (t) = C3,4 (t)C2,3 (t), (2.11)

C1,3 (t) = C2,3 (t)C1,2 (t), 2/3

C1,4 (t) = C3,4 (t)C2,3 (t)C1,2 (t).

528

CHANG, QING, AND YANG

Then we have C2,3 (t) = C1,2 (t) = C1,3 (t) = v (t).

(2.12)

Furthermore, when both V4 (t) and V3 (t) tend to inﬁnity as t tends to inﬁnity, we also have, via L’Hôpital’s rule, lim C3,4 (t) = lim C1,4 (t) = lim C2,4 (t) = lim v (t).

(2.13)

t→∞

t→∞

t→∞

t→∞

In the proof of Theorem 2.1, we have also established (2.13) for the general cases. On the other hand, by the Chern-Gauss-Bonnet formula, we have 2 4v Qe = (2.14) T e3v . 4π − s≤t

s=t

As deﬁned in [CQ, Remark 3.1], in the special case of R 4 with cylindrical coordinate we have 1 (2.15) T e3v = P3 v = − v + 2v . 2 Therefore, 1 Qe4v = |S 3 | − v (t) + 4v (t) . (2.16) 4π 2 − 2 s≤t Now to relate the total Paneitz curvature with any of the isoperimetric constants Cl,m as deﬁned above, we would like to show that, under suitable conditions, lim v (t) = lim v (t) = 0.

(2.17)

t→∞

t→∞

Thus, we turn to study the behavior of v. For convenience, we denote 2Qe4v by F . Equation (2.4) is equivalent to the following ODE: (2.18) ∞

f − 4f = F,

−∞ < t < ∞,

where −∞ |F | dt < ∞. We ﬁrst ﬁnd a special solution to (2.18). Denote f = C(t)e−2t , then C(t) satisﬁes C (t) − 4C (t) = F (t)e2t , or equivalently

C (t)e4t = F (t)e−2t .

Thus, we can solve for C(t) as follows: ∞ t e4x F (y)e−2y dy dx C(t) = − (2.19)

−∞

1 = − e4t 4

x

∞ t

F (x)e−2x dx −

1 4

t −∞

F (x)e2x dx.

529

CHERN-GAUSS-BONNET INTEGRAL FOR CONFORMAL METRICS

Therefore, 1 (2.20) f (t) = − e2t 4

∞ t

F (x)e

−2x

1 dx − e−2t 4

t −∞

F (x)e2x dx,

and (2.21)

∞ 1 −2t t 2x 2t f (t) = e F (x)e dx − e F (x)e−2x dx 8 −∞ t ∞ t F (x) dx − F (x) dx , +(K1 − K2 ) +

t

−∞

where (2.22)

K1 = lim e

∞

2t

t→−∞

t

F (x)e

−2x

dx,

K2 = lim e

−2t

t→∞

t −∞

F (x)e2x dx.

Thus, we obtain v(t) = c0 + c1 t + c2 e−2t + c3 e2t + f (t),

(2.23)

for some constants c0 , c1 , c2 , c3 . Lemma 2.1. K1 = K2 = 0. Proof. We prove, for instance, K2 = 0 ﬁrst: e

−2t

t −∞

(2.24)

F (x)e dx = e 2x

≤e

−2t

T −∞

−2(t−T )

F (x)e dx +

t

2x

∞ −∞

T

|F (x)| dx +

F (x)e dx 2x

∞ T

|F (x)| dx.

Thus, if we take T < t, say, take T = (1/2)t, and let t tends to ∞, we obtain K2 = 0. Similarly we have e (2.25)

∞

2t t

F (x)e

−2x

dx = e

t

≤

T

2t T −∞

Thus, K1 = 0 by a similar argument.

F (x)e

−2x

|F (x)| dx + e

dx +

2(t−T )

∞ T

F (x)e

∞ −∞

−2x

dx

|F (x)| dx.

530

CHANG, QING, AND YANG

As a direct consequence of Lemma 2.1, we have Corollary 2.2. (2.26) (2.27)

1 ∞ F (x) dx, lim f (t) = − t→∞ 8 −∞ 1 ∞ F (x) dx. lim f (t) = t→−∞ 8 −∞

Lemma 2.3. c2 = 0. Proof. From (2.20) and Lemma 2.1, we have 1 lim f (t) = − K1 , t→−∞ 4 which is zero. Also we have (2.28)

2 r + wr r, v (t) = wrr

with r = et tending to zero as t tends to negative inﬁnity. Thus, from (2.23) we conclude that c2 = 0. ∞ Lemma 2.4. c1 = 1 − (1/8) −∞ F (x) dx. Proof. Since v (t) →1 as t →−∞, this lemma follows from (2.23) and (2.27). We now try to eliminate the c3 coefﬁcient of the e2t term in v(t). We notice that without any further assumption, in addition to Q being absolutely integrable, this is not always possible. For example, we may consider v(t) = e2t + t. Then Q = 0, and 2 the metric e2(v−t) |dx|2 = e2r |dx|2 is complete on R 4 . Lemma 2.5. Suppose we assume that the scalar curvature is nonnegative at inﬁnity; then c3 = 0. Proof. This is a consequence of the transformation formula for the scalar curvature. Denote R the scalar curvature of the metric. Then, with the cylindrical coordinate, we have (2.29)

1 −v − (v )2 + 1 = Re2v . 6

Thus, if R ≥ 0 at inﬁnity, it follows from (2.23), (2.26), (2.27), (2.29), and Lemma 2.4 that c3 = 0. Remark 2.6. It is even easier to see that if v (t) = O(1) as t → ∞, then c3 has to be zero, in light of (2.23) and (2.26). We will use this observation in the proof of Theorem 3.8 in the next section.

CHERN-GAUSS-BONNET INTEGRAL FOR CONFORMAL METRICS

531

Theorem 2.7. Suppose that w is a radial function on R 4 , and e2w |dx|2 is a complete metric with |Q|e4w dx < ∞ and the scalar curvature nonnegative at inﬁnity. Then 1 (2.30) Qe4w dx ≥ 0. lim v (t) = 1 − 2 t→∞ 4π R 4 Moreover, we have (2.31)

1 lim Cl,m (t) = 1 − 2 t→∞ 4π

R4

Qe4w dx,

for all 1 ≤ l ≤ m ≤ 4, where Cl,m are as deﬁned in (2.10) and (2.11). Proof. Under the assumptions, we have by Lemmas 2.3 and 2.4 that (2.32)

v(t) = c0 + c1 t + f (t).

Thus, we have v (t) = f (t), which tends to zero as t tends to inﬁnity. We also have 1 ∞ 1 lim v (t) = c1 + lim f (t) = 1 − F (x) dx = 1 − 2 Qe4w . t→∞ t→∞ 4 −∞ 4π R 4 From the completeness of e2w |dx|2 , we then conclude that limt→∞ v (t) ≥ 0. Thus, if this limit is strictly positive, then both V4 (t) and V3 (t) tend to inﬁnity as t tends to inﬁnity; (2.31) then follows from L’Hôpital’s rule. On the other hand, if v (t) tends to zero and limt→∞ V4 (t) is bounded, then limt→∞ e4v(t) = 0, hence limt→∞ ekv(t) = 0 for 1 ≤ k ≤ 3. We again have limt→∞ C3,4 (t) = 0 = limt→∞ v (t). Thus, we establish (2.31). 3. Normal metrics. In this section we ﬁrst deﬁne normal metrics as a generalization of the deﬁnition of normal metrics in 2-dimension given by Finn in [F]. Then we verify that, for a complete normal metric on R 4 , the generalized Chern-Gauss-Bonnet formula (1.9) holds. Suppose that e2w |dx|2 is a metric on R 4 with its Paneitz curvature Q absolutely integrable, that is, (3.1) |Q|e4w dx < ∞. R4

Deﬁnition 3.1. A conformal metric e2w g0 satisfying (3.1) is deﬁned to be normal if |y| 1 Q(y)e4w(y) dy + C. (3.2) log w(x) = |x − y| 4π 2 R 4 So what are the normal metrics on R 4 ? It turns out that this is not an easy question. But we show that a fairly large class of metrics on R 4 are normal in the following

532

CHANG, QING, AND YANG

section. More precisely, we prove that, if the scalar curvature of the metric e2w |dx|2 is nonnegative at inﬁnity, then the metric e2w |dx|2 is normal. As in the previous section, one deﬁnes the following mixed volumes on the manifold (R 4 , e2w |dx|2 ): V4 (r) =

(3.3)

V3 (r) =

(3.4) (3.5) (3.6)

1 4

Br

∂Br

e4w dx, e3w dσ (x),

1 ∂w 2w + e dσ (x), H1 e ∂r ∂Br ∂Br r 1 1 1 ∂w 2 w V1 (r) = + H2 e3u dσ (x) = e dσ (x). 12 ∂Br 4 ∂Br r ∂r

1 V2 (r) = 12

3w

1 dσ (x) = 4

We also deﬁne Cl,m (r) as in the previous section. In the following, we will adopt some techniques used in [F] to compare Vm (r) (as previously deﬁned) with V¯m (r), the mixed volume for the metric e2w¯ |dx|2 (which may be considered to be the average of the metric e2w |dx|2 ), where w¯ is deﬁned as w(r) ¯ =

1 |∂Br |

∂Br

w(x) dσ (x).

Lemma 3.2. Suppose that the metric e2w |dx|2 on R 4 is a normal metric. Then, for any number k > 0, (3.7)

1 |∂Br |

∂Br

¯ ekw dσ (x) = ek w(r) eo(1) ,

where o(1) → 0 as r → ∞. Proof. Suppose e2w |dx|2 is a normal metric. We rewrite w as 1 w(x) = 4π 2 (3.8)

+

1 4π 2

B|x|/2 (0)

log

|y| Q(y)e4w(y) dy + C |x − y|

R 4 \B|x|/2 (0)

= w1 (x) + w2 (x).

log

|y| Q(y)e4w(y) dy |x − y|

CHERN-GAUSS-BONNET INTEGRAL FOR CONFORMAL METRICS

533

Then we write w1 (x) =

1 4π 2

|y|≤(1/2)|x|

1 + 2 4π

(3.9)

log

|y|≤(1/2)|x|

|y| Q(y)e4w(y) dy + C |x|

log

|x| Q(y)e4w(y) dy |x − y|

= f (|x|) + w10 (x), where

0

w (x) ≤ C

1

(3.10)

|y|≤η|x|

+

η|x|≤|y|≤(1/2)|x|

1 ≤ C log +C 1−η

log

η|x|≤|y|≤(1/2)|x|

|x| |Q(y)|e4w(y) dy |x − y|

|Q(y)|e4w(y) dy = o(1).

Since

log |x| ≤ log 3,

|x − y|

(3.11)

R4

|Q(y)|e

4w(y)

dx < ∞,

and we may take η → 0 (while η|x| → ∞, as |x| → ∞), then |w10 (x)| = o(1) as |x| → ∞. We then have w1 (x) = f (|x|) + o(1),

(3.12)

for some function f,

which in turn implies

(3.13)

1 k |∂Br (0)|

= log

w(y) − w2 (y) dσ (y)

∂Br (0)

1 |∂Br (0)|

∂Br (0)

ek(w(y)−w2 (y)) dσ (y) + o(1).

Next, we study the term

(3.14)

1 |∂Br (0)|

∂Br (0)

1 = 8π 2

w2 (x) dσ (x)

|y|≥(1/2)r

1 |∂Br (0)|

|y| dσ (x) Q(y)e4w(y) dy. log |x − y| ∂Br (0)

534

CHANG, QING, AND YANG

We also claim that the term (3.15)

1 |y| |y|

dσ (x) ≤ dσ (x) log log |x − y| |∂Br (0)| ∂Br (0)

|x − y|

∂Br (0)

1

L=

|∂B (0)| r

is bounded for |y| ≥ (1/2)r. To prove the claim we ﬁrst rewrite L as

1

log |y| dσ (x) L≤

|∂Br (0)| ∂Br (0)\{x:|x−y|≤θ|y|} |x − y|

|y|

1 (3.16)

log dσ (x) + |∂Br (0)| ∂Br (0) {x:|x−y|≤θ|y|}

|x − y|

= L 1 + L2 , where θ is chosen to be smaller than 1/2 (for example). Then it is easily seen that L1 ≤ log

(3.17)

1 θ

and (3.18)

1

L2 ≤

∂B(r/|y|)(0)

∂B(r/|y|)(0)

{x:|x−(y/|y|)|≤θ }

1

log

dσ (x),

|x − (y/|y|)|

which is bounded because r 1 ≤ ≤ 2, 1−θ |y|

(3.19)

and log 1/|x − (y/|y|)| is certainly integrable. This proves the claim (3.15). Thus, from (3.14) we conclude that 1 (3.20) w2 (x) dσ (x) = o(1) as r → ∞, |∂Br (0)| ∂Br (0) when Q is absolutely integrable on (R 4 , e2w |dx|2 ). Finally, we consider the term 1 1 w2 (x) (3.21) (e − 1) dσ (x) = (ew2 (rσ ) − 1) dσ. |∂Br (0)| ∂Br (0) |∂B1 (0)| ∂B1 (0) Following [F], we will estimate the term EM = {σ ∈ S 3 : |w2 (rσ )| > M}. We have (3.22)

1 |y|

M · |EM | ≤ dσ |Q(y)|e4w(y) dy |w2 | dσ ≤ log |rσ − y|

8π 2 R 4 \Br/2 (0) EM

EM 1 H (y)|Q(y)|e4w(y) dy. = 8π 2 |y|≥r/2

CHERN-GAUSS-BONNET INTEGRAL FOR CONFORMAL METRICS

535

We will estimate the function H (y) for |y| ≥ r/2. Again we decompose H as

|y|

H (y) =

log |rσ − y| dσ EM \{σ :|rσ −y|≤(1/3)|y|}

|y|

(3.23)

+

log |rσ − y| dσ EM {σ :|rσ −y|≤(1/3)|y|} = H 1 + H2 . Clearly, H1 ≤ log 3 · |EM |.

(3.24)

It is more difﬁcult to estimate the term H2 . To do this, we observe that for |rσ −y| ≤ (1/3)|y|, we have

log |y| ≤ log |y| + log σ − y

≤ log 3 + log σ − y

. (3.25)

|rσ − y| r r 2 r

Thus, we may estimate H2 , when EM is a 3-dimensional disk centered at y/r, perpendicular to y, and of the size |EM |, as 1 1 H2 ≤ C · |EM | + C|EM | · log ≤ C 1 + log |EM |. (3.26) |EM | |EM | Combining (3.22), (3.23), (3.24), and (3.26), we obtain 1 M ≤ o(1) 1 + log (3.27) , |EM | where o(1) → 0 as r → ∞; this then implies |EM | ≤ Ce−M/o(1) .

(3.28) Thus,

(3.29)

1

|∂B (0)| r

=

∂Br (0)

k |∂B1 (0)|

e

kw2 (x) +∞

−∞

− 1 dσ (x)

ekM − 1 |EM | dM = o(1),

as r → ∞.

Combining (3.13), (3.20), and (3.29), we arrive at k 1 w(x) dσ (x) = log ekw(x) dσ (x) + o(1), |∂Br (0)| ∂Br (0) |∂Br (0)| ∂Br (0)

536

CHANG, QING, AND YANG

or equivalently,

∂Br (0)

¯ ekw(x) dσ (x) = eo(1) |∂Br (0)|ek w(r) .

Thus, we have established (3.7). As a direct consequence we have the following corollary. Corollary 3.3. Suppose that the metric e2w |dx|2 on R 4 is a normal metric. Then (3.30) V3 (r) = V¯3 (r) 1 + o(1) , and d V4 (r) = dr

(3.31)

d ¯ V4 (r) 1 + o(1) . dr

To deal with V1 , V2 , we will ﬁrst establish the following technical lemma. Lemma 3.4. Suppose that e2w |dx|2 is a normal metric on R 4 . Then 1 |∂Br |

(3.32)

∂Br

∂w ∂r

k

1 dσ (x) = O k , r

for k = 1, 2, 3,

and 1 |∂Br |

(3.33)

∂Br

∂w ∂r

2

dσ (x) =

∂ w¯ ∂r

2

(r) + o

1 . r2

Proof. We observe that ∂ w¯ 1 (r) = ∂r |∂Br |

∂Br

∂w dσ (x). ∂r

To prove (3.32), we have, for a normal metric e2w |dx|2 , ∂w (x) = (3.34) K(x, y)F (y) dy, ∂r R4 where K(x, y) = (1/8π 2 |x|)(|x|2 − x · y)/|x − y|2 and F (y) = 2Q(y)e4w(y) , which is integrable over R 4 . Then 1 |∂Br |

≤

∂Br

R4

∂w ∂r

k

1 |∂Br |

dσ (x) ∂Br

|K(x, y)|k dσ (x)|F (y)| dy ·

R4

|F (y)| dy

k−1

.

CHERN-GAUSS-BONNET INTEGRAL FOR CONFORMAL METRICS

Therefore, to prove (3.32) it sufﬁces to prove 1 1 k (3.35) |K(x, y)| dσ (x) = O k |∂Br | ∂Br r

537

for all y ∈ R 4 .

To verify (3.35), we write 2 |x|2 − x · y = |x − y|2 + |x|2 − |y|2 . Therefore, it leads us to verify that 1 |∂Br |

(3.36)

2

|x| − |y|2 3

|x − y|6

∂Br

≤C

for some constant independent of y ∈ R 4 . By the homogeneity of the integrand, we only need to consider, for any y ∈ R 4 ,

2

|x| − |y|2 3 (3.37) dσ (x) ≤ C |x − y|6 ∂B1 It is easily seen that (3.37) holds if |y| ≤ 1 − δ or |y| ≥ 1 − δ, for any given δ > 0. But for |y| ∈ (1 − δ, 1) ∪ (1, 1 + δ), we have the following calculus inequality for any N ≥ 3: C dσ (x) (3.38) ≤ . N |1 − |y||N−3 ∂B1 |x − y| We have thus established (3.37), and hence (3.35) and (3.32). To prove (3.33), we note that, since 1/|x − y|2 is the Green’s function for the Laplacian on R 4 , we have  1   when |y| < r;  2, 1 1 r

3

dσ (x) = (3.39)

S S 3 |x − y|2 1   r r  2 , when |y| > r. |y| It follows that we can write ∂ w¯ (r) = ∂r where

(3.40)

¯ K(x, y) =

R4

 r  ,    16π 2 |y|2    

¯ K(x, y)F (y) dy,  when |y| > r;    

 |y|2 1  2 − , when |y| < r.   2 2 16π r r

538

CHANG, QING, AND YANG

Thus,

∂w 2 ∂ w¯ 2

dσ (x) −

∂r ∂r

∂Br

2 1

¯ |F |(y) dy · ≤ K(x, y) − K(x, y) dσ (x) |F |(y) dy. R4 R 4 |∂Br | ∂Br

1 |∂Br |

We again compute



1

|y|2 − |x|2 |x − y|2 − |y|2

   

, for |y| > |x|; |x − y|2 |y|2

 16π 2 |x|

K(x, y) − K(x, ¯ y) =

  1

|y|2 − |x|2 |x − y|2 − |x|2

  

, for |y| < |x|. 16π 2 |x|

|x − y|2 |x|2

Therefore, (3.41)

2 1 1

K(x, y) − K(x,

¯ y) dσ (x) = O , |∂Br | ∂Br |x|2 and

(3.42)

as |x| → ∞ and |y| > |x|1/2 ,

1 1

K(x, y) − K(x,

¯ − y) ≤ C|x|

|x − y|2 |x|2

1 , as |x| → ∞ and |y| ≤ |x|1/2 . =o |x|

Combining (3.40), (3.41), and (3.42), we obtain

∂w 2 1 ∂ w¯ 2

− dσ (x) |∂Br | ∂Br ∂r ∂r

1 1 1 O |F |(y) dy + |F |(y) dy = o . = o |x|2 |x|2 |x|2 |y|>|x|1/2 |y|≤|x|1/2 This establishes (3.33). Now we are ready to estimate V1 (r) and V2 (r). Lemma 3.5. Suppose that e2w |dx|2 is a normal metric on R 4 and suppose that ∂ w¯ lim 1 + r r→∞ ∂r exists and is positive. Then (3.43)

V2 (r) = V¯2 (r) 1 + o(1) ,

as r → ∞,

CHERN-GAUSS-BONNET INTEGRAL FOR CONFORMAL METRICS

539

and (3.44)

V1 (r) = V¯1 (r) 1 + o(1) ,

as r → ∞.

Proof. Let us denote by a = (∂ w/∂r)(r), ¯ and b = ew¯ (r). Then 1 1 ¯ V2 (r) = |∂Br | + a b2 , 4 r and

2w 1 1 1 ∂w 2 ¯ V2 (r) − V2 (r) = +a − a e2w − b2 . e −b + 4 r 4 ∂Br ∂r ∂Br

Applying Lemmas 3.2 and 3.4, we have 1 ¯ V2 (r) − V2 (r) = |∂Br | + a b2 o(1) r 2 1/2 2w 2 1/2 ∂w (3.45) + −a · e − b2 ∂r ∂Br ∂Br 1 = V¯2 (r)o(1) + |∂Br | b2 o(1), r which implies (3.43) under the assumption that limr→∞ (1 + ra) > 0. Similarly, 2 1 1 ¯ V1 (r) = |∂Br | + a b, 4 r and

2 w 1 +a e −b ∂Br r 1 1 ∂w ∂w 2 + − a ew − b + − a 2 ew . 2r ∂Br ∂r 4 ∂Br ∂r

1 V1 (r) − V¯1 (r) = 4

Applying Hölder inequality and Lemmas 3.2 and 3.4, we again obtain b V1 (r) − V¯1 (r) = V¯1 (r)o(1) + |∂Br | 2 o(1), r which again implies (3.44) under the assumption that limr→∞ (1 + ra) > 0. So the lemma is proved. We now discuss the metric that is an average of the metric e2w |dx|2 on R 4 over the spheres ∂Br (0). For convenience, we would use the cylindrical coordinates again.

540

CHANG, QING, AND YANG

Then w(r) ¯ = w(e ¯ t ), and v(t) = w(r)+t. ¯ As we have veriﬁed in Section 2, v satisﬁes the following: 2 v − 4v = (3.46) Qe4v dσ (x) = F (t), −∞ < t < ∞, |∂Br (0)| ∂Br (0) with (3.47) and (3.48)

∞

2 F (t) dt = 3 |S | −∞

∞

−∞

R4

Qe

|F (t)| dt ≤

4w

1 π2

1 dx = 2 π

R4

Qe4w dx < ∞

R4

|Q|e4w dx < ∞.

Lemma 3.6. Suppose that (R 4 , e2w |dx|2 ) is a complete normal metric. Then its ¯ |dx|2 ) is also a complete metric. averaged metric (R 4 , e2w(r) Proof. This is basically a consequence of Lemma 3.2. By the argument given in the proof of Lemma 3.5 we have 1 ¯ (3.49) ew(rσ ) dσ = ew(r) · eo(1) , |S 3 | S 3 where o(1) → 0 as r → ∞. So r1 r1 r1 1 1 w(rσ ) w(rσ ) ¯ e drdσ = (3.50) e dσ dr = ew(r) · eo(1) dr, 3| 3 |S 3 | S 3 r0 |S r0 S r0 which proves the lemma. Recall that in the statement of Theorem 2.7 we have assumed that the sign of the scalar curvature is positive, in order to establish the fact that the coefﬁcient of the e2t term vanishes, in the expression (2.23) of the conformal factor v. We have also pointed out in Remark 2.6 that to achieve the same purpose, it sufﬁces to prove that v (t) = O(1), as t tends to inﬁnity. We will now establish this later fact for the average metric of a normal metric. Lemma 3.7. Suppose that (R 4 , e2w |dx|2 ) is a normal metric. Then w(r) ¯ ≤

(3.51)

C . r2

Proof. We compute (3.52)

1 w(r) ¯ = 3 |S |

1 w(rσ ) dσ = 3 |Sr | S3

Sr3

w(x) dσ (x).

CHERN-GAUSS-BONNET INTEGRAL FOR CONFORMAL METRICS

541

Then, by the assumption that the metric is normal, we have 1 2 1 w(r) ¯ = (3.53) dσ (x) Q(y)e4w(y) dy. 4π 2 R 4 |Sr3 | Sr3 |x − y|2 Thus, it follows from (3.40) and (3.53) that 1 w(r) ¯ ≤ (3.54) |Q|e4w dy. 2π 2 r 2 R 4 We have thus established the lemma. Theorem 3.8. Suppose that (R 4 , e2w |dx|2 ) is a complete normal metric. Then Qe4w dx ≤ 4π 2 , R4

and (3.55)

lim

r→∞

∂Br (0) e

3w dσ (x) 4/3

4w dx Br (0) e

1 = 1− 2 4π

R4

Qe4w dx ≥ 0.

Furthermore, if (3.56)

1 1− 2 4π

R4

Qe

4w

then (3.57)

1 lim Cl,m (r) = 1 − 2 r→∞ 4π

dx > 0, R4

Qe4w dx,

for all 1 ≤ l, m ≤ 4. Proof. We follow the outline indicated at the beginning of this section. We ﬁrst ¯ |dx|2 . Applying Lemmas 3.6 and 3.7 and Remark look at the averaged metric e2w(r) 2.6 in the last section, we have 1 ∂ w¯ (r) = − (3.58) Qe4w dx ≥ −1. lim r r→∞ ∂r 2|S 3 | R 4 We then apply Corollary 3.3 and get 3w dσ (x) 4/3 |S 3 |4/3 e4w¯ r 4 ∂Br (0) e lim (3.59) . = lim 4w r→∞ r→∞ V (r) dx Br (0) e Thus, it follows from (3.30) and (3.31), with a proof similar to the proof of Theorem 2.7, that (3.55) holds. By the same reasoning, (3.57) follows from Lemma 3.5 under the condition of (3.56).

542

CHANG, QING, AND YANG

Remark 3.9. In the statements of Theorem 3.8, we have established (3.55) without the additional assumption that (3.56) holds. But we have established the further isoperimetric equalities (3.57) for higher order mixed volumes only under the additional assumption (3.56). In view of the fact that the classical isoperimetric inequalities (2.5) hold in general only for convex bodies for domains in R n , it remains an interesting problem to see if (3.57) holds even in the degenerate case when the constant in (3.56) is zero. 4. Proof of main theorems. In this section we show that a metric (R 4 , e2w |dx|2 ) is a normal metric if its scalar curvature is nonnegative at inﬁnity. It follows from the result in the last section that for a fairly large class of conformal metrics on R 4 we have the generalized Chern-Gauss-Bonnet formula (3.55) and (3.57) in Theorem 3.8. Before we state and prove our main result, we ﬁrst remark that in general a complete conformal metric with integrable Paneitz curvature on R 4 may not be a normal metric. To see this, we can use the same example that was discussed at the beginning of Section 2. Recall that, in cylindrical coordinate, denote v = e2t +t; then (R 4 , e4v |dx|2 ) is a complete metric with Paneitz curvature Q = 0. It is also clearly not a normal metric. This justiﬁes the additional assumption in the statement of Theorem 4.1. Theorem 4.1. Suppose the metric (R 4 , e2w |dx|2 ) is a complete metric with the Paneitz curvature Q integrable, and the scalar curvature is nonnegative at inﬁnity. Then it is a normal metric. Proof. First denote v(x) =

(4.1)

3 2π 2

R4

log

|y| Qe4w dy, |x − y|

and let h = w − v. We will now show that the biharmonic function h on R 4 is a constant. Recall the transformation formula for the scalar curvature, R w + |∇w|2 = − e2w , 6

(4.2)

where R is the scalar curvature for the metric (R 4 , e2w |dx|2 ). Notice that h is a harmonic function. Thus, (4.3) h(x0 ) =

1 |∂Br (x0 )|

=−

∂Br (x0 )

1 |∂Br (x0 )|

h dσ

∂Br (x0 )

|∇w|2 +

1 R dσ − v dσ. 6 |∂Br (x0 )| ∂Br (x0 )

The ﬁrst term on the right of (4.3) is nonpositive when r is large enough by our assumption that R is nonnegative. We now observe that, by an argument similar to

CHERN-GAUSS-BONNET INTEGRAL FOR CONFORMAL METRICS

543

the proof of Lemma 3.7 in Section 3, we have 1 1 3 v dσ = 2 dσ Q(y)e4w(y) dy. π R 4 |S 3 | S 3 |rσ + x0 − y|2 ∂Br (x0 ) (4.4) 3 |Q|e4w dy. ≤ 2 2 π r R4 Therefore, by taking r → ∞, we have, for each x0 ∈ R 4 , h(x0 ) ≤ 0.

(4.5)

Thus, h = C0 for some nonpositive constant by Liouville theorem for harmonic functions. Thus, any partial derivative of h is harmonic; that is, hxi = 0.

(4.6)

Applying the mean value theorem again, we have (4.7)

2

1 1

|hxi (x0 )| =

hx dσ ≤ |∇h|2 dσ. |∂Br (x0 )| ∂Br (x0 ) i

|∂Br (x0 )| ∂Br (x0 ) 2

But |∇h|2 ≤ 2|∇w|2 + 2|∇v|2 = −2C0 −

(4.8)

R 2w e + 2|∇v|2 3

and (4.9) |∇v|2 ≤ C

R4

1 |Q|e4w dy |x − y|2

R4

|Q|e4w dy ≤ C

R4

1 |Q|e4w dy. |x − y|2

Similarly, we conclude that, for each x0 ∈ R 4 , (4.10)

|hxi (x0 )|2 ≤ −2C0 ,

which implies that all partial derivatives of h are constants. Then h = C0 = 0, which ﬁnally implies that all partial derivatives of h vanish by (4.10). Thus h is a constant. Main theorem. Suppose that (R 4 , e2w |dx|2 ) is a complete metric with its Paneitz curvature absolutely integrable, and its scalar curvature is nonnegative at inﬁnity. Then conclusions in Theorem 3.1 hold. In particular, 3w dσ (x) 4/3 1 ∂Br (0) e lim = 1− 2 (4.11) Qe4w dx ≥ 0. 4w dx r→∞ 4(2π 2 )1/3 4π R 4 Br (0) e

544

CHANG, QING, AND YANG

References [BCY] [BZ] [CQ] [CY] [CG]

[CLi] [CL] [CV] [F] [HR] [Hu] [LT] [L] [P] [T] [Xu]

T. Branson, S.-Y. A. Chang, and P. C. Yang, Estimates and extremals for the zeta functional determinant on four-manifolds, Comm. Math. Phys. 149 (1992), 241–262. Y. D. Burago and V. A. Zalgaller, Geometric Inequalities, Grundlehren Math. Wiss. 285, Springer Ser. Soviet Math., Springer, Berlin, 1988. S.-Y. A. Chang and J. Qing, The zeta functional determinants on manifolds with boundary, I: The formula, J. Funct. Anal. 147 (1997), 327–362. S.-Y. A. Chang and P. C. Yang, On the uniqueness of solution of n-th order equations in conformal geometry, Math. Res. Lett. 4 (1997), 91–102. J. Cheeger and M. Gromov, “On the characteristic numbers of complete manifolds of bounded curvature and ﬁnite volume” in Differential Geometry and Complex Analysis, Springer, Berlin, 1985, 115–154. W. X. Chen and C. Li, Classiﬁcation of solutions of some nonlinear elliptic equations, Duke Math. J. 63 (1991), 615–622. K.-S. Cheng and C.-S. Lin, On the asymptotic behavior of solutions to the conformal Gaussian curvature equations in R 2 , Math. Ann. 308 (1997), 119–139. S. Cohn-Vossen, Kürzeste Wege und Totalkrümmung auf Flächen, Compositio Math. 2 (1935), 69–133. R. Finn, On a class of conformal metrics, with application to differential geometry in the large, Comment. Math. Helv. 40 (1965), 1–30. V. H. Hopf and W. Rinow, Über den Begriff der vollständigen differentialgeometrischen Fläche, Comment. Math. Helv. 3 (1931), 209–225. A. Huber, On subharmonic functions and differential geometry in the large, Comment. Math. Helv. 32 (1957), 13–72. P. Li and L.-F. Tam, Complete surfaces with ﬁnite total curvature, J. Differential Geom. 33 (1991), 139–168. C.-S. Lin, A classiﬁcation of solutions of a conformally invariant fourth order equation in R n , Comment. Math. Helv. 73 (1998), 206–231. S. Paneitz, A quartic conformally covariant differential operator for arbitrary pseudoRiemannian manifolds, preprint, 1983. N. S. Trudinger, On new isoperimetric inequalities and symmetrization, J. Reine Angew. Math. 488 (1997), 203–220. X. Xu, Classiﬁcation of solutions of certain fourth order nonlinear elliptic equations in R 4 , preprint, 1996.

Chang: Department of Mathematics, Princeton University, Princeton, New Jersey 08544, USA and Department of Mathematics, UCLA, Los Angeles, California 90095, USA; chang@ math.princeton.edu Qing: Department of Mathematics, University of California, Santa Cruz, Santa Cruz, California 95064, USA; [email protected] Yang: Department of Mathematics, University of Southern California, Los Angeles, California 90089, USA; [email protected]

Vol. 103, No. 3

DUKE MATHEMATICAL JOURNAL

© 2000

A REMARK ON THE ENERGY BLOW-UP BEHAVIOR FOR NONLINEAR HEAT EQUATIONS HATEM ZAAG

1. Introduction. We are concerned with ﬁnite-time blow-up for the following nonlinear heat equation: ut = u + |u|p−1 u in × [0, T ), (1) u=0 on ∂ × [0, T ) with u(x, 0) = u0 (x), where u : ×[0, T ) → R, is a C 2,α -convex bounded domain of RN , u0 ∈ L∞ (). We assume that the following condition holds: 3N + 8 1 < p, (N − 2)p < N + 2, and u0 ≥ 0 or p < . (2) 3N − 4 Therefore, p +1 > N(p −1)/2 and the (local in time) Cauchy problem for (1) can be solved in Lp+1 () (see, for instance, [21, Theorem 3]). If the maximum existence time T > 0 is ﬁnite, then u(t) is said to blow up in ﬁnite time, and in this case lim u(t) Lp+1 () = lim u(t) L∞ () = +∞

t→T

t→T

(3)

(see [21, Corollary 3.2]). We consider such a blow-up solution u(t) in the following. From the regularizing effect of the Laplacian, u(t) ∈ L∞ ∩H01 () for all t ∈ (0, T ). We take u 2 1 = |∇u|2 dx. Using the Sobolev embedding and the fact that p H0 ()

is subcritical (p < (N + 2)/(N − 2) if N ≥ 3), we see that H01 () ⊂ Lp+1 (). Therefore, (3) implies that lim u(t) H 1 () = +∞.

t→T

0

A point a ∈ is called a blow-up point of u if there exists (an , tn ) → (a, T ) such that |u(an , tn )| → +∞. The set of all blow-up points of u(t) is called the blow-up set and denoted by S. From Giga and Kohn [8, Theorem 5.3], there are no blow-up points in ∂. Therefore, we see from (3) and the boundedness of that S is not empty. Many papers are concerned with the Cauchy problem for (1) (see, for instance, [21]) or the problem of ﬁnding sufﬁcient blow-up conditions on the initial data (see Received 15 September 1998. Revision received 2 December 1999. 2000 Mathematics Subject Classiﬁcation. Primary 35K20, 35K55, 35A20. 545

546

HATEM ZAAG

Ball [2], Levine [11], etc.). Other papers focus on the description of the blow-up set or the asymptotic behavior of u near blow-up points (Giga and Kohn [6], [7], and [8]; Herrero and Velázquez [9], [19], [18], and [20]; Merle and Zaag [13], [14], [15], and [16]). Let us mention, for instance, the following Liouville theorem for (1), recently proved in [16], which has many interesting consequences for the study of the blow-up behavior of solutions to (1) (see Fermanian, Merle, Zaag [3], [4], [16]). Proposition 1.1 (Merle-Zaag, a Liouville theorem for equation (1)). Assume that 1 < p and (N −2)p < N +2, and consider U to be a solution of (1) deﬁned for all (x, t) ∈ RN ×(−∞, T ). Assume, in addition, that |U (x, t)| ≤ C(T −t)−1/(p−1) . Then U ≡ 0 or there exist T0 ≥ T and ∈ {−1, 1} such that for all (x, t) ∈ RN ×(−∞, T ), U (x, t) = κ(T0 − t)−1/(p−1) , where κ = (p − 1)−1/(p−1) . Remark. Note that this result is valid for all subcritical p with no restrictions for N ≥ 2. For the reader’s convenience, a sketch of the proof is given in the appendix. (For more details, see [16, Corollary 1].) In this paper, we crucially use the Liouville theorem to study how the Lyapunov functional 1 1 2 |∇u| dx − |u|p+1 dx (4) E(u) = 2 p+1 associated with (1) behaves under the nonlinear heat ﬂow. It has been shown by Giga in [5] that under the positivity condition u0 ≥ 0, we have

E u(t) −→ −∞

as t −→ T .

(5)

Let us remark that Giga’s proof relies on another Liouville theorem related to (1). Assume that p > 1 and p(N −2) < N +2. Then, there is no nonnegative solution for the problem u + up = 0 in RN , u(0) > 0. In this paper, we use the new Liouville theorem stated in Proposition 1.1 and ideas from [5] to extend the validity of the limit (5) to the more general case (2). Theorem 1.2 (Limit of the energy at blow-up). Assume (2). Then E(u(t)) goes to −∞ as t goes to T . In [1], the study of critical points of E is related to the study of those of a functional J associated with E and deﬁned for all v ∈ , the unit sphere of H01 () by J (v) = sup E(λv). λ>0

THE ENERGY BLOW-UP BEHAVIOR FOR HEAT EQUATIONS

547

In other words, J (v) is the supremum of E in the direction of v. Note that J is positive by (4). The following is shown in [1, Proposition 1]: (i) J ∈ C 2 (, R); (ii) for all v ∈ , we have E (λ(v)v) = λ(v)−1 J (v), where λ(v) is the unique positive solution of J (v) = E(λv); (iii) there is a one-to-one correspondence between the critical points of J and E by means of the transformation ω ∈ −→ λ(ω)ω = ω1 ,

J (ω) = E(ω1 ).

This correspondence leads Bahri to the study of some topological properties of level sets of J . He shows, in particular, that the level sets of J have contractibility properties one into another. More precisely (see [1, Lemma 1]), if we deﬁne La = {v ∈ | J (v) ≥ a}, then for all a > 0, there exists µ(a) ≥ a such that Lµ(a) is contractible in La . Remark. According to Krasnosel’skii [10, p. 329], if B ⊂ A, then B is said to be contractible (to a point) in A if there is a continuous mapping θ(t, .) : [0, 1]×B → A such that for all x ∈ B, θ (0, x) = x, and θ(1, x) = x0 ∈ A. Our second concern in this paper is to understand the effect on J of the nonlinear heat ﬂow of (1) (composed with the projection over ). In other words, we want to understand the behavior of J (u(t)/ u(t) H 1 () ) as t → T . We claim the following. 0

Theorem 1.3 (Blow-up limit of the directional supremum of the energy). The Rayleigh quotient for the solution, u(t) H 1 / u(t) Lp+1 , goes to +∞ as t → T 0 and so does u(t) 1 2(p+1)/(p−1) u(t) p−1 H0 J = sup E λu(t) = . (6) u(t) H 1 () 2(p + 1) u(t) λ>0 Lp+1 0

Roughly speaking, one consequence of this theorem is that the nonlinear heat ﬂow of (1) (composed with the projection over ) maps any element of a given level set La into Lb , for any b > a. (Note that this mapping raises the level set of J , in the contrary of the contractibility result of [1], which lowers the value of J .) Another consequence of Theorem 1.3 is that E(u(t)) cannot tend to −∞ radially. More precisely, we have the following corollary. Corollary 1.4. We cannot have u(·, t) ∼ λ(t)ϕ in H01 () as t → T . Indeed, if this was the case, then J

u(t) u(t) H 1 () 0

∼J

ϕ ϕ H 1 () 0

as t −→ T since J is continuous.

548

HATEM ZAAG

This contradicts Theorem 1.3. In [1, Proposition 2], it is shown that J satisﬁes the following property: ∀(un ); un ∈ ; un goes weakly to 0 in H01 () ⇐⇒ J (un ) −→ +∞. Therefore, Theorem 1.3 is equivalent to the following. Proposition 1.5. As t → T , u(t)/ u(t) H 1 () goes to 0, weakly in H01 (). 0

Remark. Mueller and Weissler raised the issue of the behavior of u(t)/ u(t) H 1 () 0 in [17, p. 883] and proved the convergence in the case of positive solutions with single-point blow-up. The paper is organized as follows. In Section 2, we use the Liouville theorem of [16] and prove Theorem 1.2. In Section 3, we use results from [7] and some consequences of the Liouville theorem to prove Proposition 1.5 and Theorem 1.3. Acknowledgments. I would like to thank Professor Abbas Bahri for his invitation to Rutgers University, where this work was done and also for fruitful discussions and suggestions about this paper. Many thanks to the referee, whose work was appreciated and whose remarks were crucial to the clarity of the paper. 2. Energy blow-up behavior. We prove Theorem 1.2 in this section. We proceed in two parts. We recall some results from [7] and [16] for blow-up solutions of (1) in the ﬁrst part. Then the proof of Theorem 1.2 is presented in the second part. Part 1. L∞ estimates for blow-up solutions of (1). The following uniform L∞ bound for blow-up solutions of (1) is proved in [16, Theorem 2]. Proposition 2.1 (Giga-Kohn, a uniform L∞ bound on u(t) at blow-up). There exists C0 > 0 such that ∀t ∈ [0, T ),

u(t) L∞ ≤ C0 (T − t)−1/(p−1) .

(7)

In the following proposition, we derive the existence of a blow-up proﬁle for u(t). Proposition 2.2 (Existence of the blow-up proﬁle). There exists u∗ (x) deﬁned on ∗ \ S such that u∗ ∈ L∞ loc ( \ S), and u(t) → u uniformly on each compact set of \ S as t → T . Proof. See Merle [12], for example. In [16, Proposition 4], Merle and Zaag generalize a result by Velázquez (see [18], [19], and [20]) and prove the following result on the size of the blow-up set S. Proposition 2.3 (Size of the blow-up set). S is compact and the (N − 1)-Hausdorff measure of S is ﬁnite.

THE ENERGY BLOW-UP BEHAVIOR FOR HEAT EQUATIONS

549

Remark. Since u0 ∈ L∞ (), u(t) ∈ L∞ ∩ H01 () for all t > 0, from the regularizing effect of the Laplacian. Therefore, [16, Proposition 4] applies. Part 2. Proof of Theorem 1.2. Our proof relies strongly on the Liouville theorem presented in Proposition 1.1. We proceed by contradiction. Since E(u(t)) is decreasing in time, it goes to some ﬁnite A ∈ R as t → T . Therefore, multiplying (1) by ∂u/∂t and integrating over × [0, T ), we get

T 0

dt

2 ∂u dx (x, t) = E(u0 ) − A ≡ B < +∞. ∂t

(8)

In a ﬁrst step, we use a compactness procedure to derive a solution of (1) that satisﬁes the hypotheses of the Liouville theorem (see Proposition 1.1). In a second step, we apply Proposition 1.1 on one hand and use (8) with scaling arguments on the other hand to get a contradiction. Step 1. A compactness procedure. Let us consider a ∈ to be a blow-up point of u(t) and any sequence tk → T as k → +∞. From the uniform blow-up bound of Proposition 2.1 and Giga and Kohn [8], we know that u(a, tk ) ∼ κ(T − tk )−1/(p−1)

as k −→ +∞,

(9)

where ∈ {−1, 1}. We can assume that = 1 from the sign invariance of (1). For each k ∈ N, we deﬁne for all ξ ∈ ( − a)(T − tk )−1/2 and τ ∈ (−tk /T − tk , 1), vk (ξ, τ ) = (T − tk )1/(p−1) u a + ξ T − tk , tk + τ (T − tk ) . (10) From (1), (7), and (9), we see that vk satisﬁes for all ξ ∈ ( − a)(T − tk )−1/2 and τ ∈ (−tk /T − tk , 1), ∂vk = vk + |vk |p−1 vk , |vk (ξ, τ )| ≤ C0 (1 − τ )−1/(p−1) ∂τ

and

vk (0, 0) −→ κ

as k → +∞. Since a ∈ ∂ and tk → T as k → +∞, vk is deﬁned (at least) for all (ξ, τ ) ∈ Dn , ¯ n) × [−n, 1 − (1/n)]. Moreover, it for all n ∈ N∗ and k ≥ k0 (n), where Dn = B(0, 1/(p−1) satisﬁes vk L∞ (Dn ) ≤ C0 n . Using parabolic regularity for (1) in Dn+1 ⊃⊃ Dn , we obtain vk C 2,1 (D ) ≤ C(n), n α for all n ∈ N∗ and k ≥ k0 (n + 1), where h C 2,1 (D) = h Cα (D) + ∇h Cα (D) + ∇ 2 h Cα (D) + ∂τ h Cα (D) , α h(ξ, τ ) − h(ξ , τ ) sup h Cα (D) = h L∞ (D) + α/2 , (ξ,τ ),(ξ ,τ )∈D |ξ − ξ |2 + |τ − τ |

(11)

550

HATEM ZAAG

and α ∈ (0, 1). Using the compactness of the embedding of Cα (Dn ) into C(Dn ), we ﬁnd v(ξ, τ ) a solution of (1) deﬁned for all (ξ, τ ) ∈ RN × (−∞, 1) and satisfying 2,1 (RN ×(−∞, 1)) (up to a subsequence), for all (ξ, τ ) ∈ RN ×(−∞, 1), vk → v in Cloc ∂v = v + |v|p−1 v, ∂τ

|v(ξ, τ )| ≤ C0 (1 − τ )−1/(p−1) ,

and

v(0, 0) = κ. (12)

Step 2. Conclusion of the proof of Theorem 1.2. From the Liouville theorem of Proposition 1.1, (12) yields ∀(ξ, τ ) ∈ RN × (−∞, 1),

v(ξ, τ ) = κ(1 − τ )−1/(p−1) .

(13)

From the convergence of vk , we have for all R > 0, 2 2 0 0 ∂v ∂vk dτ dξ (ξ, τ ) = lim dτ dξ (ξ, τ ) . k→+∞ ∂τ ∂τ −R −R B(0,R) B(0,R) From (10), (8), and the scaling argument, we easily compute 2 0 ∂vk (ξ, τ ) dτ dξ ∂τ −R B(0,R) 2 tk ∂u (x, t) = (T − tk )β dt dx √ ∂t tk −R(T −tk ) B(a,R T −tk ) 2 T ∂u β ≤ (T − tk ) dt dx (x, t) ≤ B(T − tk )β , ∂t 0 where β=

p+1 N − >0 p−1 2

since p is subcritical. 0 Therefore, −R dτ B(0,R) dξ |(∂vk /∂τ )(ξ, τ )|2 → 0 as k → +∞, and so ∀R > 0,

0 −R

dτ

2 ∂v dξ (ξ, τ ) = 0. ∂τ B(0,R)

(14)

A contradiction follows from (13) and (14), and Theorem 1.2 is proved. 3. Blow-up behavior of the directional maximum of the energy. We prove Proposition 1.5 and Theorem 1.3 in this section. As stated in the introduction, Theorem 1.3 is a direct consequence of Proposition 1.5, thanks to a result of [1, Proposition 2]. Since this fact can be proved in a simple and short way, we present a proof of it in the following.

THE ENERGY BLOW-UP BEHAVIOR FOR HEAT EQUATIONS

551

Proposition 1.5 implies Theorem 1.3. Since p is subcritical, we have p +1 < 2∗ = 2N /(N − 2) whenever N ≥ 3. Hence, H01 () is compactly embedded in Lp+1 (). Therefore, assuming Proposition 1.5, we get u(t) Lp+1 −→ 0 ∇u(t) L2

as t −→ T .

(15)

The expression of the Rayleigh quotient given in (6) can be easily checked from (4). Thus, (15) yields Theorem 1.3. Now, we use information on the blow-up set S from Section 2 to prove Proposition 1.5. Proof of Proposition 1.5. It is enough to show that for all ϕ ∈ C ∞ () with supp ϕ ⊂⊂ and for all > 0, there exists t0 () < T such that for all t ∈ [t0 (), T ), we have ∇u(x, t) · ∇ϕ(x) dx ∞ () . ≤ 1 + ∇ϕ L 1/2 |∇u(x, t)|2 dx

From Proposition 2.3, we know that S is compact in and that its Lebesgue measure |S| = 0. Therefore, we may consider the open set V = {x ∈ | d(x, S) < δ }, where δ is small enough so that |V | ≤ 2 . We then write

∇u(x, t) · ∇ϕ(x) dx 1/2 2 |∇u(x, t)| dx

where

∇u(x, t) · ∇ϕ(x) dx I = 1/2 2 |∇u(x, t)| dx V

= I + I I,

and

∇u(x, t) · ∇ϕ(x) dx . 2 dx 1/2 |∇u(x, t)|

\V

II =

By Cauchy-Schwartz inequality, for all t ∈ [0, T ), we have V ∇u(x, t) · ∇ϕ(x) dx |I | = 1/2 |∇u(x, t)|2 dx

1/2 |∇u|2 dx 1/2 ≤ 1/2 ∇ϕ L∞ () |V | 2 |∇u| dx V

≤ ∇ϕ L∞ () .

(16)

552

HATEM ZAAG

According to Giga and Kohn, no blow-up occurs near the boundary ∂ (see [8, Theorem 5.3]). Therefore, using Proposition 2.2 and parabolic regularity, we ﬁnd M() > 0 such that ∀x ∈ \ V ,

∀t ∈ [T /2, T ),

|u(x, t)| + |∇u(x, t)| ≤ M().

For all t ≥ T /2, we then write \V ∇u(x, t) · ∇ϕ(x) dx M() ∇ϕ L∞ || |I I | = ≤ . 2 dx 1/2 2 dx 1/2 |∇u(x, t)| |∇u| 2 Since |∇u(x, t)| dx → +∞, we may take t ≥ t1 () large enough so that |I I | ≤ .

(17)

Combining (16) and (17) yields the following: for all t ≥ t0 () ≡ max(t1 (), T /2), ∇u(x, t) · ∇ϕ dx ≤ ∇ϕ L∞ + 1 . 2 |∇u(x, t)| dx This concludes the proofs of Proposition 1.5 and Theorem 1.3. Appendix Sketch of the proof of the Liouville theorem. In this appendix, we give a sketch of the proof of Proposition 1.1. For more details, one can ﬁnd a complete proof in [16]. Let U be a solution of (1) deﬁned for all (x, t) ∈ RN × (−∞, T ) and satisfying |U (x, t)| ≤ C(T − t)−1/(p−1) . If w(y, s) is deﬁned by the self-similar change of variables x y=√ , s = − log(T − t), w(y, s) = (T − t)1/(p−1) U (x, t), (18) T −t then w satisﬁes the following equation for all (y, s) ∈ RN × R: w 1 ∂s w = w − y · ∇w − + |w|p−1 w 2 p−1

(19)

and w L∞ (RN ×R) ≤ C. Let us introduce the following Lyapunov functional associated with (19): 1 1 1 |∇w|2 + |w|2 − |w|p+1 ρ(y) dy, Ᏹ(w) = 2(p − 1) p+1 RN 2 where ρ(y) = e−|y| /4 /(4π )N/2 . With the change of variables (18), Proposition 1.1 is equivalent to the following proposition. 2

THE ENERGY BLOW-UP BEHAVIOR FOR HEAT EQUATIONS

553

Proposition A.1. Assume that 1 < p and (N − 2)p < N + 2. Consider w to be a solution of (19) deﬁned for all (y, s) ∈ RN × R and satisfying w L∞ (RN ×R) ≤ C. Then either w ≡ 0, w ≡ κ, or for all (y, s) ∈ RN × R, w(y, s) = ϕ(s − s0 ), where κ = (p − 1)−1/p−1 , ∈ {−1, 1}, and ϕ(s) = κ(1 + es )−1/p−1 is a solution of ϕ = −

ϕ + ϕp , p−1

ϕ(−∞) = κ,

ϕ(+∞) = 0.

(20)

Therefore, we reduce to the proof of Proposition A.1. We proceed in 3 parts. In part I, we use the monotonicity of s " → Ᏹ(w(s)) to k (RN )) which are show that w(·, s) has limits w±∞ as s → ±∞ (in L2ρ (RN ) and Cloc stationary solutions of (19). From [6], we know that either w±∞ ≡ 0 or w±∞ ≡ κ where = ±1. We then focus on the nontrivial case (w−∞ , w+∞ ) = (κ, 0). In part II, we linearize (19) around the constant solution κ as s → −∞, and we show that w behaves in three possible ways. In part III, we show that one of these three ways corresponds to the case w(y, s) = ϕ(s − s0 ), where ϕ is deﬁned in (20). In the other two cases, we show that w satisﬁes a ﬁnite-time blow-up criterion for (19), which contradicts the fact that w is deﬁned for all (y, s) ∈ RN × R and w L∞ (RN ×R) ≤ C < +∞. Thus, we rule out these two cases. Part I. Existence of limits for w as s → ±∞. We have the following lemma. k (RN ) for all Lemma A.2. As s → +∞, w(·, s) → w+∞ in Hρ1 (RN ) and Cloc k ∈ N, where either w+∞ = 0 or w+∞ = κ with = ±1. An analogous statement holds for the limit as s → −∞.

Sketch of the proof. For a complete proof, see [16, Proposition 2.2] and [14, Proposition 3.4]. Since w L∞ (RN ×R) ≤ C, parabolic regularity applied to (19) implies that for all R > 0, w C 2,1 (B(0,R)×[−R,R]) ≤ M(R) where a C 2,1 (D) is deﬁned in (11). Using α α the compactness of the embedding of Cα (D) in C(D) and considering subsequences wj (y, s) = w(y, s + sj ), where sj → +∞, the identity s2 ∂s w(y, s)2 ρ(y) dy ds = Ᏹ w(s1 ) − Ᏹ w(s2 ) ∀s1 , s2 ∈ R, (21) s1

RN

allows us to ﬁnd w+∞ (y), a stationary solution of (19) such that w(·, s) → w+∞ 2 (RN ). The conclusion follows from the next result by Giga and as s → +∞ in Cloc Kohn in [6]. Claim A.3 (Giga-Kohn). If p > 1 and (N −2)p < N +2, then the only stationary solutions of (19) are 0, κ, and −κ. Letting s2 → +∞ and s1 → −∞ in (21), we obtain +∞ ∂s w(y, s)2 ρ(y) dy ds ≥ 0. Ᏹ(w−∞ ) − Ᏹ(w+∞ ) = −∞

RN

554

HATEM ZAAG

Therefore, two cases arise. Case 1. Ᏹ(w−∞ ) − Ᏹ(w+∞ ) = 0. Therefore, ∂s w ≡ 0 and w is a stationary solution of (19). Claim A.3 implies then that w ≡ 0, κ, or −κ. This corresponds to the ﬁrst cases expected in Proposition A.1. Case 2. Ᏹ(w−∞ ) − Ᏹ(w+∞ ) > 0. Since Ᏹ(κ) = Ᏹ(−κ) > 0 = Ᏹ(0), this implies that w+∞ ≡ 0 and w−∞ ≡ κ or −κ. From sign invariance of (19), we reduce to the case (w−∞ , w+∞ ) = (κ, 0). Part II. Linear behavior of w near κ. We introduce v = w − κ. From (19), v satisﬁes the following equation: ∂s v = ᏸv + f (v),

(22)

where |f (v)| ≤ C|v|2 and ᏸ = − (1/2)y · ∇ + 1 is a self-adjoint operator on Ᏸ(ᏸ) ⊂ L2ρ (RN ), whose spectrum consists of eigenvalues {1 − (m/2) | m ∈ N}. Therefore, we can expand v on the eigenspaces of ᏸ. Since v L∞ (RN ×R) ≤ C, we use hard analysis where the key point is the control of the quadratic term in (22), and we prove that one of the modes 1, 1/2, or 0 dominates the others as s → −∞. More precisely, we have the following lemma. Lemma A.4. As s → −∞, one of the following cases occur: (i) (mode λ = 1): w(y, s) − {κ − C0 es } Hρ1 (RN ) = o(es ) where C0 > 0;

(ii) (mode λ = 1/2): w(y, s) − {κ + es/2 C1 · y} Hρ1 (RN ) = o(es/2 ) where C1 ∈

RN \ {0};

(iii) (mode λ = 0): w(Qy, s) − {κ + (κ/(2ps))(l − (1/2) li=1 yi2 )} Hρ1 (RN ) = o(1/s) where Q is an orthonormal N × N matrix and l ∈ {1, . . . , N }. Proof. See [16, Proposition 2.4] and [14, Propositions 3.5, 3.6, 3.9, and 3.10]. Part III. Conclusion of the proof Case 1. Mode λ = 1 dominates, the relevant case. We remark that we already know a solution of (19) that behaves like w as s → −∞: it is ϕ(s − s0 ), where ϕ satisﬁes (20) and s0 = − log(C0 (p − 1)/κ). Therefore, w(y, s)−ϕ(s −s0 ) Hρ1 (RN ) = o(es ) as s → −∞. Let us prove that, in fact, w(y, s) = ϕ(s − s0 ),

for all (y, s) ∈ RN × R.

(23)

For this, we introduce V (y, s) = w(y, s)−ϕ(s−s0 ), which satisﬁes V (y, s) Hρ1 (RN ) = o(es ) and show that V ≡ 0. See [16, Proposition 2.5] for more details. Therefore, (23) holds, and this gives the last case expected in Proposition A.1. Case 2 and 3. Mode λ = 1/2 or 0 dominates, irrelevant cases. Here we use the

THE ENERGY BLOW-UP BEHAVIOR FOR HEAT EQUATIONS

555

invariance of (19) under the geometric transformation (a0 , s0 ) ∈ RN × R " −→ wa0 ,s0 : (y, s) " −→ w y + a0 es/2 , s + s0 , and the following blow-up criterion for (19). Lemma A.5 (A blow-up criterion for (19)). Consider W to be a solution of (19) satisfying I (W (0)) > 0 where (p+1)/2 p−1 |v(y)|2 ρ(y) dy . I (v) = −2Ᏹ(v) + p + 1 RN Then W blows up in ﬁnite time S > 0. Proof. See [16, Proposition 2.1]. Using the asymptotic expansions of Lemma A.4, we ﬁnd (a0 , s0 ) ∈ RN × R such that I (wa0 ,s0 ) > 0. Therefore, wa0 ,s0 blows up in ﬁnite time S > 0. This contradicts the fact that wa0 ,s0 is deﬁned for all (y, s) ∈ RN × R and satisﬁes wa0 ,s0 L∞ (RN ×R) = w L∞ (RN ×R) ≤ C < +∞. Thus, cases (ii) and (iii) of Lemma A.4 actually do not hold. For more details, see [16, Section 2, part II, step 2]. This concludes the sketch of the proof of Propositions A.1 and 1.1. References [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11]

[12]

A. Bahri, Topological results on a certain class of functionals and application, J. Funct. Anal. 41 (1981), 397–427. J. M. Ball, Remarks on blow-up and nonexistence theorems for nonlinear evolution equations, Quart. J. Math. Oxford Ser. (2) 28 (1977), 473–486. C. Fermanian Kammerer, F. Merle, and H. Zaag, Stability of the blow-up proﬁle of nonlinear heat equations from the dynamical system point of view, to appear in Math. Ann. C. Fermanian Kammerer and H. Zaag, Boundedness till blow-up of the difference between two solutions to the semilinear heat equation, preprint. Y. Giga, A bound for global solutions of semilinear heat equations, Comm. Math. Phys. 103 (1986), 415–421. Y. Giga and R. V. Kohn, Asymptotically self-similar blow-up of semilinear heat equations, Comm. Pure Appl. Math. 38 (1985), 297–319. , Characterizing blowup using similarity variables, Indiana Univ. Math. J. 36 (1987), 1–40. , Nondegeneracy of blowup for semilinear heat equations, Comm. Pure Appl. Math. 42 (1989), 845–884. M. A. Herrero and J. J. L. Velázquez, Blow-up behavior of one-dimensional semilinear parabolic equations, Ann. Inst. H. Poincaré Anal. Non Linéaire 10 (1993), 131–189. M. A. Krasnosel’skii, Topological Methods in the Theory of Nonlinear Integral Equations, trans. A. H. Armstrong, ed. J. Burlak, Macmillan, New York, 1964. H. A. Levine, Some nonexistence and instability theorems for solutions of formally parabolic equations of the form P ut = −Au + F (u), Arch. Rational Mech. Anal. 51 (1973), 371–386. F. Merle, Solution of a nonlinear heat equation with arbitrarily given blow-up points, Comm. Pure Appl. Math. 45 (1992), 263–300.

556 [13]

[14] [15] [16] [17] [18] [19] [20] [21]

HATEM ZAAG F. Merle and H. Zaag, “Estimations uniformes à l’explosion pour les équations de la chaleur non linéaires et applications,” Séminaire sur les Équations aux Dérivées Partielles, 1996–1997, École Polytech., Palaiseau, 1997, exp. no. 19. , Optimal estimates for blowup rate and behavior for nonlinear heat equations, Comm. Pure Appl. Math. 51 (1998), 139–196. , Reﬁned uniform estimates at blow-up and applications for nonlinear heat equations, Geom. Funct. Anal. 8 (1998), 1043–1085. , A Liouville theorem for vector-valued nonlinear heat equations and applications, Math. Ann. 316 (2000), 103–137. C. E. Mueller and F. B. Weissler, Single point blow-up for a general semilinear heat equation, Indiana Univ. Math. J. 34 (1985), 881–913. J. J. L. Velázquez, Higher-dimensional blow up for semilinear parabolic equations, Comm. Partial Differential Equations 17 (1992), 1567–1596. , Classiﬁcation of singularities for blowing up solutions in higher dimensions, Trans. Amer. Math. Soc. 338 (1993), 441–464. , Estimates on the (n − 1)-dimensional Hausdorff measure of the blow-up set for a semilinear heat equation, Indiana Univ. Math. J. 42 (1993), 445–476. F. B. Weissler, Local existence and nonexistence for semilinear parabolic equations in Lp , Indiana Univ. Math. J. 29 (1980), 79–102.

École Normale Supérieure, Département de Mathématiques et Applications, CNRS UMR 8553, 45 rue d’Ulm, 75 230 Paris cedex 05, France; [email protected] Current: Courant Institute, New York University, 251 Mercer Street, New York 10012, U.S.A.

No title

No title

No title

No title

No title

No title

No title

No title

No title

No title

No title

No title

No title

No title

No title

No title

No title

No title

No title

No title

No title

No title

No title

No title

No title

No title

No title

No title

No title

No title

Recommend Documents